Subject: Re: [EAI] Random thought on ASCII range

From: Yangwoo Ko <[email-masked]>
Date: Mon, 3 Jan 2011 17:09:08 +0900

Do you mean that "Basic Latin" is for a set of characters and "ASCII"
for encoding of them in 7bit?

On Sat, Jan 1, 2011 at 3:32 AM, Shawn Steele <[email-masked]> wrote:
> Random thought:  Unicode calls 00-7F “C0 Controls and Basic Latin.”  Given
> that the C0 Controls are pretty much illegal in email addresses, we could
> shorten that to “Basic Latin”, which isn’t terribly wordy, and evades the
> question of whether we mean the ASCII set of characters U+0000-U+007f or the
> encoding.
>
>
>
> - Shawn
>
>
>
>  
>
> http://blogs.msdn.com/shawnste
>
> (Selfhost 7903)
>
>
>
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>
>

-- 
a human known as [email-masked] @ gtalk

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 03 Jan 2011 07:23:00 -0800

On 1/3/2011 12:09 AM, Yangwoo Ko wrote:
> On Sat, Jan 1, 2011 at 3:32 AM, Shawn Steele<[email-masked]>  wrote:
>> Random thought:  Unicode calls 00-7F “C0 Controls and Basic Latin.”

Kudos to Shawn for being both creative and diligent, in finding such a 
well-founded basis for an additional label to consider.

I had the same question as Yangwoo, but on re-reading Shawn's note I see that 
"Latin" comes from a reference to Unicode.  Therefore I think yes, Latin is the 
set of characters and "ASCII" would be an encoding of those characters.

Hence, the 2x2 table would be:

                  | Representation |  Encoding  |
                  +----------------+------------+
                  |                |            |
           Legacy |     Latin      |   ASCII    |
                  |                |            |
                  +----------------+------------+
                  |                |            |
    International |    Unicode     |   UTF-8    |
                  |                |            |
                  +----------------+------------+

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Jan 2011 15:19:36 -0500

--On Monday, January 03, 2011 17:09 +0900 Yangwoo Ko
<[email-masked]> wrote:

> Do you mean that "Basic Latin" is for a set of characters and
> "ASCII" for encoding of them in 7bit?

Well, if one is going to be completely precise, "Basic Latin"
and "ASCII repertoire" aren't synonyms.   We sort of got away
with it in IDNA because we did a bit of hand waving and had
already excluded all of the inconvenient punctuation, C0 control
characters, etc., from consideration in other ways.  Unicode
gets away with it because they define things that way, but that
isn't the only common usage of that term, it is a local
definition.  We can make it a local definition here too, but
only at the risk of using a term that is used differently
elsewhere in a special way.  I don't have a problem with that if
the WG is ok with it.

But the state of the C0 controls actually is important because
they are valid, if little-used, in 821/822/5321/5322 email.  The
WG has discussed, and I think agreed on, prohibiting them if EAI
features are needed (deliberately vague-- another issue), but
they are still valid in legacy, all-ASCII, addresses.

If one is comparing to Unicode and ignores the ASCII embedding
problem, there is an ASCII repertoire and a Unicode repertoire,
a native encoding for ASCII and three native encodings for
Unicode.  

FWIW, Unicode code points are usually expressed as a range of
hexidecimal integers (with the normal notation of U+[N[N]]NNN)
and  ASCII code points have historically been expressed either
as a decimal number or in row/column form, e.g., 4/1 for upper
case A).

The ASCII repertoire, however, includes not only the letter
characters that Unicode calls "Basic Latin" but much of the
linguistic and writing system communities do not.  As Unicode's
definition and Shawn's note only partially point out, those
letters aren't Basic Latin:  classical Latin doesn't use W,
differentiate between U and V or I and J, etc.).  And the ASCII
repertoire also includes a set of Indic-Arabic digits that
classical Latin definitely does not use (and doesn't include all
of the characters needed to write classical Roman numerals), a
collection of so-called C0 controls (characters 0/1 through
1/15), and character 7/15.

We actually already have a term for ASCII in the native encoding
and IETF 8bit embedding.  We've been calling it "NVT" for many,
many years.  The only problem is that there has been an
ambiguity for almost that many years as to whether NVT includes
some or all of the C0 controls.   These problems are not new.

Once one accepts those definitions, and ignores the other two
native encodings for Unicode, we've got an updated version of
Dave's 2x2 matrix.

However, the most important, and most difficult, term needed to
make the relevant distinctions is what we've often (and
sloppily) called non-ASCII.   Due to some duplications in
Unicode, it isn't the set difference between the Unicode
repertoire and the ASCII repertoire unless one accepts that
those duplications are actually different characters by virtue
of being coded differently (an assertion that much of the
linguistic and writing system community rejects).  Instead, it
has to be defined in terms of ranges, e.g., Unicode code points
above U+007F.

Let me say differently what I've tried to say before:  If the
right solution at this point is for the WG to define its own set
of words (per Andrew's proposal or some variation), I'm ok with
that.   If, however, we are going to use words with definitions
that are slightly different from definitions and usage
elsewhere, I think we do a real disservice to the community if
those definitions are not made IETF-wide (a job that lies
outside EAI's scope).  Otherwise, people reading specs will have
to figure out what a given term means in the context of that
spec.  I think most of us have already had ample experience with
where that expectation leads -- people skip over definitions
that they are confident that they know and then guess wrong
about what the spec means.

   john

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 03 Jan 2011 15:23:44 -0800

On 1/3/2011 12:19 PM, John C Klensin wrote:
> sloppily) called non-ASCII.

Why does the current work require a formal term for that (sub)set?

The reference to that subset of Unicode is -- or, rather, should be -- actually 
quite limited.

In the specifications, almost every reference to "non-ascii" actually means 
"Unicode", rather than "the subset of Unicode that is above x07F."  That is, 
they appear to be making the reference to "non-ascii" when that should not be 
what is really meant.

I've posted more than one note asking this question.  Perhaps I missed the 
responses that provide a clear and compelling argument for needing the term?

d/
-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Jan 2011 22:41:57 -0500

--On Monday, January 03, 2011 15:23 -0800 Dave CROCKER
<[email-masked]> wrote:

>> However, the most important, and most difficult, term needed
>> to make the relevant distinctions is what we've often (and
> The reference to that subset of Unicode is -- or, rather,
> should be -- actually quite limited.
> 
> In the specifications, almost every reference to "non-ascii"
> actually means "Unicode", rather than "the subset of Unicode
> that is above x07F."  That is, they appear to be making the
> reference to "non-ascii" when that should not be what is
> really meant.
> 
> I've posted more than one note asking this question.  Perhaps
> I missed the responses that provide a clear and compelling
> argument for needing the term?

Dave, perhaps you haven't tried writing specs or developed code
in the i18n area and that is why this seems obvious to those who
have but not to you and, hence why explanations that seem "clear
and compelling" to some of the more active participants in this
work seem less so to you.  Let me try a different explanation in
the hope that it will make sense to you.

At first glance, one should only have the situation your matrix
seems to assume: ignoring encoding and embedding issues, a
string is either ASCII or it is relatively unrestricted Unicode.
And, if it is the latter, one doesn't need to worry about ASCII
as a distinct subset.  In practice, Unicode outside the ASCII
range always seems to carry a certain amount of baggage with it:
requirements for normalization, case and character comparison
situations that can use a number of different rules, alternate
collation sequences (possible with ASCII too depending, e.g., on
whether one thinks in terms of "ABC...abc" or "AaBbCc..." but
with far fewer alternate variations), and so on.   The result of
this is that, when one actually sits down to sort out the
details of specs, the situation often comes down to a need to
talk about the difference between a string containing only
characters from the ASCII repertoire (regardless of what one
decides to call that) and a string of characters from the
Unicode repertoire that contains at least one character outside
the ASCII range.

A recent discussion on this list about local parts provides a
fairly typical example.  A particular address is
internationalized, requiring EAI extensions be present, iff it
contains at least one Unicode character outside the ASCII range
in either the local-part or the domain part (or a display name
containing Unicode characters outside the ASCII range encoded in
UTF-8 rather than as encoded words).   It isn't sufficient to
write that rule as "Unicode" or "UTF-8" because the ASCII
repertoire is a proper superset of the first and the ASCII codes
(embedded in 8bit fields) are proper supersets of the latter.

Hope that helps.
    john

---

From: Yangwoo Ko <[email-masked]>
Date: Tue, 4 Jan 2011 14:51:54 +0900

This discussion reveals that we need to be clear about whether terms
are about a set of charcters or about a string. In the context of EAI,
I thought (and still think) that the latter terms are enough.

In other words, we need to have a term for "non-ASII string" not for
"set of characters > 0x7f".

---

From: John C Klensin <[email-masked]>
Date: Tue, 04 Jan 2011 10:44:50 -0500

--On Tuesday, January 04, 2011 14:51 +0900 Yangwoo Ko
<[email-masked]> wrote:

> This discussion reveals that we need to be clear about whether
> terms are about a set of charcters or about a string. In the
> context of EAI, I thought (and still think) that the latter
> terms are enough.

Well, I've been trying to stay within the bounds of Dave's
rather neat little matrix, but your note reminded me of how much
difficulty we've had with that distinction, starting with the
use of "variant" to describe both individual sets of characters
and labels that contain one or more such characters in early
versions of what became RFC 3743 (a problem that continues to
confuse ICANN even though we corrected in in 3743 before it was
published).  It adds a third dimension to Dave's table.
 
> In other words, we need to have a term for "non-ASII string"
> not for "set of characters > 0x7f".

I would say that we need a term for "string containing at least
one character outside the ASCII repertoire".   I don't think EAI
needs a special definition for the set of Unicode characters
outside the ASCII repertoire, but other protocols might.  We
should at least be aware of that possibility so as to avoid
choosing a term that would create even more confusion when the
set terminology is needed.

     john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 4 Jan 2011 17:31:31 +0000

Sorry for not being clearer, yes, I was thinking of Latin as the range of characters (after the Unicode name for 00-7f).  Not strictly accurate (there're non-script things in 00-7F).  Could also be misleading (since there are latin-x character sets), but I just threw it out since we seemed confused :)

-Shawn

-----Original Message-----
From: Dave CROCKER [mailto:[email-masked]] 
Sent: Pōʻakahi, Ianuali 03, 2011 7:23 AM
To: Yangwoo Ko
Cc: Shawn Steele; [email-masked]
Subject: Re: [EAI] Random thought on ASCII range

>> Random thought:  Unicode calls 00-7F “C0 Controls and Basic Latin.”

Kudos to Shawn for being both creative and diligent, in finding such a well-founded basis for an additional label to consider.

I had the same question as Yangwoo, but on re-reading Shawn's note I see that "Latin" comes from a reference to Unicode.  Therefore I think yes, Latin is the set of characters and "ASCII" would be an encoding of those characters.

d/
-- 

   Dave Crocker

---

From: Shawn Steele <[email-masked]>
Date: Tue, 4 Jan 2011 17:42:49 +0000

FWIW: I think that (since there's no IETF-wide definition), that we should just "pick something" and make sure it's clearly defined.  The problem we've had seems to be more along consistent use of the terms, I doubt we'll find a "perfect" term that's obvious w/o defining it.

-Shawn

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Pōʻakahi, Ianuali 03, 2011 12:20 PM
To: Yangwoo Ko; Shawn Steele
Cc: [email-masked]
Subject: Re: [EAI] Random thought on ASCII range

...

Let me say differently what I've tried to say before:  If the right solution at this point is for the WG to define its own set of words (per Andrew's proposal or some variation), I'm ok with
that.   If, however, we are going to use words with definitions
that are slightly different from definitions and usage elsewhere, I think we do a real disservice to the community if those definitions are not made IETF-wide (a job that lies outside EAI's scope).  Otherwise, people reading specs will have to figure out what a given term means in the context of that spec.  I think most of us have already had ample experience with where that expectation leads -- people skip over definitions that they are confident that they know and then guess wrong about what the spec means.

   john

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 04 Jan 2011 10:15:30 -0800

> Sorry for not being clearer, yes, I was thinking of Latin as the range of characters (after the Unicode name for 00-7f).  Not strictly accurate (there're non-script things in 00-7F).  Could also be…

In fact the IETF use of the term "ASCII" usually is imprecise in the same way.

The original term was "Network ASCII" as was defined for a Network Virtual 
Terminal.  It is a subset of full ASCII.  Interestingly, it is discussed at 
length in RFC 5198...

However although the fine-grained details of this distinction are highly 
relevant for someone writing code they are almost never relevant to other 
discussions.

I believe the same "permission" for being casual with the label "Latin" can 
reasonably apply here.  Within a community that is conducting discussions on a 
topic, it is common, efficient and productive to use labels that are approximate 
rather than precise.  Simplification is the essence of defining useful categories.

The challenge is to make sure that the scope of the community is appropriate. 
So, for example, terminology that is specific to one working group but different 
from terms used in a related working group is almost certainly a bad idea. 
Terminology in one standards group that directly conflicts with that of another 
might also be problematic.

Also, there is the potential -- and occasionally the reality -- that casualness 
in terminology masks sloppy thinking and sloppy specification.  The group 
certainly needs to be careful in make its choices.

The purpose behind the 2x2 matrix is to define a particular scope, between 
original Internet and international Internet.  That's a constrained environment, 
but useful, environment.  The constraint permits excluding quite a bit of 
related, but not essential, terminology and discussion.

Speaking of related...

I am increasingly developing the view that reference to "non-ASCII" should be 
rare and quite isolated.  From the working group specifications and from 
postings on this list, it appears to be used as a synonym for Unicode or UTF-8. 
  This can lead to the erroneous view that an all-ASCII string is not Unicode.

While the presence of "non-ASCII" does indeed define data as Unicode, data 
containing only ASCII also can be Unicode.

Hence a distinction such as "Latin" vs. "Unicode" needs to depend solely on an 
explicit label for the data and not depend upon inspecting the data before 
deciding on what label to use.

If the group moves away from inspecting the data and then defining its type 
based on that inspection, and instead moves to explicit labeling, it will have 
very restricted need for referring to "non-ascii".

For VRFY/EXPN, when a client declares it can support Unicode, the server does 
not have to worry about whether the responses contain ascii or non-ascii.  When 
the client has not made this declaration, a server can reasonably deliver only 
responses that are ASCII.  The only interesting case is when the server wants to 
provide Unicode strings that happen to be only ASCII.   In this case, 
"non-ascii" does not mean "Unicode"; it means "a character that was not ASCII 
was found".  But this is an extremely specific situation for the working group's 
specification; there are few others.

d/
-- 

   Dave Crocker

---

From: Shawn Steele <[email-masked]>
Date: Tue, 4 Jan 2011 18:19:01 +0000

> I am increasingly developing the view that reference to "non-ASCII" 
> should be rare and quite isolated.  From the working group specifications
> and from postings on this list, it appears to be used as a synonym for 
> Unicode or UTF-8. 

Actually "non-ASCII" is used (in a very few places I think) to indicate the characters >= U+0080.  Eg: the characters that force the use of EAI and which can't be used in legacy SMTP addresses.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Tue, 04 Jan 2011 14:09:47 -0500

--On Tuesday, January 04, 2011 17:31 +0000 Shawn Steele
<[email-masked]> wrote:

> Sorry for not being clearer, yes, I was thinking of Latin as
> the range of characters (after the Unicode name for 00-7f).
> Not strictly accurate (there're non-script things in 00-7F).
> Could also be misleading (since there are latin-x character
> sets), but I just threw it out since we seemed confused :)

Please, let's be _very_ careful about this.  In Unicode-speak
(at least most of the time), "Basic Latin" (the term you used)
refers to a subset of the U+0000 to U+007F range.  Which subset
is actually not well-defined: the U+0000 to U+007F code block is
titled "C0 Controls and Basic Latin".  But the category groups
within the code block are divided up as "C0 controls", "ASCII
punctuation and symbols", "ASCII digits", "ASCII punctuation and
symbols", "Uppercase Latin alphabet", "ASCII punctuation and
symbols", "Lowercase Latin alphabet", "ASCII punctuation and
symbols", and "Control character".   

There are two things that are important about this: the "ASCII
punctuation and symbols" group is not contiguous -- an
inconvenience, but probably not a problem-- and their explicit
use of the term "ASCII" to describe parts of the range in that
block in spite of the fact that the "ASCII repertoire" is the
entire block, including the C0 controls and the odd,
one-code-point title that Unicode assigns to U+007F.  These
titles are an editorial convenience in the Unicode definition;
they are neither normative nor guaranteed to be stable, so, if
we are going to depend on them, we probably need to reference a
particular version of Unicode (not a good idea when avoidable,
as several IETF activities have learned the hard way).

Second, while you said "Basic Latin", Dave's revised chart said
"Latin".  In Unicode-speak, "Latin" refers to a script (or
script family).  That script, according to Section 7.1 of
Unicode 5.0, includes the letters of Basic Latin and the Latin-1
supplement; the characters of Latin Extended-A, B, C, and D,
Extended Additional, and Ligatures.  It also includes the IPA
and Phonetic Extensions, whose inclusion in "Latin" on any basis
but derivation is debated by linguist and phoneticists.  Again,
while the block definitions (i.e., ranges of code points) are
stable, there are no guarantees about the names.  The list is,
in principle, open-ended: new characters and new blocks could be
added at any time.

But, most excitingly, "Latin Script" (as in 7.1), does not
include either the ASCII punctuation and symbols (a few of
which, like period and hyphen and maybe space and "@" are
important to EAI), nor does it include the so-called European
digits (which are also important to EAI).

>From a later note:

> FWIW: I think that (since there's no IETF-wide definition),
> that we should just "pick something" and make sure it's
> clearly defined.  The problem we've had seems to be more along
> consistent use of the terms, I doubt we'll find a "perfect"
> term that's obvious w/o defining it.

Shawn, let me suggest a few guidelines, all of them derived from
experience with IDNA, the development of the various JET and
CDNC "variant" documents, ICANN's attempts to engage with IDNs
and variants, and some closely-related work.  Perhaps they will
put the reasons I'm being careful (or resistant) into
perspective.

(0) While it implies risks and is not a strategy I prefer to
advocate, we can often get away with informal uses of terms that
are not precisely defined.  When it is clear we are doing that,
people can often figure out what is intended from context and
get that right.  While precise, formal, definitions are better
than that sort of informal/contextual approach, formal
definitions that are imprecise or that fail on one or more of
the criteria below are usually even worse.

	(1) It is a bad idea to use the same term in different
	IETF specifications for two different things.  Doing so
	causes people to skip over local definitions (even if
	they are present) and read things based on what they
	think the words mean.  That, in turn, leads to trouble.
	When feasible, we should even avoid definitional
	conflicts with non-IETF-stream RFCs.  
	
	(2) If Unicode uses a term, we should either use that
	term in exactly the same way they do or not use it at
	all.   If they use a term, but use it inconsistently or
	ambiguously, we are better off avoiding it. 
	
	(3) Borrowing a bit from an ICANN discussion, we should
	avoid affiliating ourselves with the Humpty Dumpty
	School of Philology, i.e., claiming that common words
	mean whatever we say they mean (but being willing to
	supply those definitions and being reasonably consistent
	about them).
	
	(4) Staying out of the Queen of Hearts School of
	Philology, in which words mean whatever one wants them
	to mean at the moment, with no obligation to explain the
	definitions, is even more important.

That list has some obvious corollaries:

	-- If we are going to try to define terms that are
	likely to be picked up and used throughout the IETF, we
	need to either get the work out of EAI or make certain
	we look backward at IDNA, forward at the PRECIS work,
	and sideways at Unicode and whatever other i18n work may
	be coming in the IETF.
	
	-- If we are going to try to use common terms in a
	specific and precise way, it isn't going to be easy.
	
	-- If the WG is still committed to getting this work
	done quickly, and precise terminology is an objective
	(or requirement), then we are probably better off
	defining terms that are clearly EAI-specific (following
	the lead of Andrew's suggestion) rather than trying to
	sort out precise and usable definitions for terms in
	more common use.    The alternative could easily hold us
	up for several months as authors waited while we tried
	to sort terminology, get agreement from other relevant
	groups within the IETF, etc.

best,
    john

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 04 Jan 2011 11:20:45 -0800

On 1/4/2011 10:19 AM, Shawn Steele wrote:
>> I am increasingly developing the view that reference to "non-ASCII" should
>> be rare and quite isolated.  From the working group specifications and from
>> postings on this list, it appears to be used as a synonym for Unicode or
>> UTF-8. This can lead to the erroneous view that an all-ASCII string is not
>> Unicode.
>
> Actually "non-ASCII" is used (in a very few places I think) to indicate the
> characters>= U+0080.  Eg: the characters that force the use of EAI and which
> can't be used in legacy SMTP addresses.

And my point is that the "force the use" reference is an example of having the 
label derived from the data, rather than have the label /dictate/ the data.

It's not that the derivation is wrong; it is that it is limiting.

In each of the places that the language does this sort of "force the use of EAI" 
perspective is document, the group should consider whether it is, instead, 
possible to have the language be strictly in terms of a Latin-vs-Unicode "mode". 
  A specific benefit of this will be to permit all-ASCII to be classed as 
Unicode rather than Latin.

d/
-- 

   Dave Crocker