Subject: [EAI] syntax: uAtom

From: "Jiankang YAO" <[email-masked]>
Date: Mon, 10 Jan 2011 11:21:40 +0800

----- Original Message ----- 
From: "Claudio Allocchio" <[email-masked]>
To: <[email-masked]>; <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>; "SM" <[email-masked]>
Sent: Thursday, December 23, 2010 1:24 AM
Subject: [apps-discuss] apps-team review of draft-ietf-eai-rfc5336bis

...
> 
>   3.3.  Extended Mailbox Address Syntax
> 
> The definition of
> 
>              uAtom = 1*ucharacter
>                ; Replace Atom in RFC 5321, Section 4.1.2
> 
>              ucharacter = atext / UTF8-non-ascii
> 
>              atext = <See Section 3.2.3 of RFC 5322>
> 
> permits that uAtom is either an atext OR an UTF8-non-ascii... which in the
> end allows a uMailbox to be a mixed contruction where some uAtom is in
> atext, while some other uAtom is in UTF8-non-ascii AT THE SAME TIME, e.g. a
> construct like
> 
>      [email-masked]
> 
> is allowed. However the distinction here seems to be unneeded, as ASCII is a
> subset of UTF8.
> 

thanks for your kind comments.

UTF8-ASCII includes all of ASCII characters.
atext only includes part of ASCII characters, and excludes some control characters in ASCII

if ucharacter=UTF8-characters;  UTF8-ASCII  plus UTF8-non-ascii

then ucharacters will include some control characters in ASCII.

so in original text, we intensionally excludes some control characters in ASCII.

can we include the  control characters in ASCII?

Jiankang Yao

---

From: John C Klensin <[email-masked]>
Date: Tue, 11 Jan 2011 02:55:16 -0500

--On Monday, January 10, 2011 11:21 +0800 Jiankang YAO
<[email-masked]> wrote:

> To: <[email-masked]>; <[email-masked]>;
> <[email-masked]> Cc:
> <[email-masked]>; "SM" <[email-masked]> Sent:
> Thursday, December 23, 2010 1:24 AM

>> permits that uAtom is either an atext OR an UTF8-non-ascii...
>> which in the end allows a uMailbox to be a mixed contruction
>> where some uAtom is in atext, while some other uAtom is in
>> UTF8-non-ascii AT THE SAME TIME, e.g. a construct like
>...

Exactly right, but see below.

> can we include the  control characters in ASCII?

I'm not certain I understand exactly what question you are
asking, but let me try to give some perspective on the issue
and, in the process, suggest an additional question for Joseph's
list.  I assume that, if we can figure out what we really want
to do, we can get the syntax sorted out, but there may still be
uncertainty about what we want to do.

Personal opinion only...

When RFC 821 and its predecessors were first defined, the rule
was, effectively, "any character in the ASCII repertoire" (with
a quoting convention for some of them that we now see as
undesirable).  The ASCII repertoire consists, in terminology
used by Unicode but introduced in later versions of the ASCII
standard and its international counterpart (ISO/IEC 646), of

   The C0 control set
   Space
   ASCII graphics
   DEL

"ASCII graphics" can be further subdivided, but the divisions
don't appear to be relevant to EAI.

In today's word (perhaps unlike that of 25 or 30 years ago),
there seems to be no advantage and some risks from including
control characters in email local-parts and other data elements.
We changed the quoting convention to eliminate some of the
problems and evolved the preferred (not requiring quotes) form
of local parts to depend on "atext", which is restricted to
ASCII graphics (printable characters).  Largely because of
concerns about the installed base and incompatible changes, we
have never deprecated use of non-graphic characters entirely, or
even removed the control characters (some of them potentially
quite dangerous) from what may appear in quoted strings.

Moving to Unicode does not only expand the repertoire of graphic
characters, it adds a second group of explicit control
characters (the C1 set), plus a lot of characters that are used
only for annotations of particular sorts.    Example of the
latter include such things as the Language Tag Characters that
have been deprecated by the Unicode consortium and by the IETF
in RFC 6082.

The question is what should be permitted in what I'm going to
refer to in the balance of this note as "EAI-address-characters"
(a repertoire, not an encoding) to avoid any confusion with
Unicode, UTF-8, ASCII, and other terms.

If we make it "all of Unicode", then we include both the ASCII
characters that have proven problematic and increase the risk of
mischief and mapping errors by including the C1 controls as well
as tagging, annotation, and other characters.

The WG spent some time on this and concluded that we needed to
prohibit use of both the C0 and C1 controls, the latter to avoid
adding characters with direct security consequences and no known
benefits and the former for consistency and to reduce risks that
become more severe in the context of the expanded repertoire.
So we now have (completely new terminology for clarity -- please
do not copy into document) and informally because it may make
the intent more clear:

821-address-characters = Any ASCII character, whole
	range.  Some require quoting.
	
EAI-address-characters = Any Unicode character, whole
	range, _except_ C0 and C1 controls.  Some require
	quoting.

So the first answer is that, while the Unicode repertoire is a
proper superset of the ASCII one and the set of octets that can
appear in UTF-8 encoding is a proper superset of the set of
octets that can appear in NVT encoding (NVT is always ASCII),
the set of characters allowed in EAI-address-characters is _not_
a proper superset of the set of characters allowed in
821-address-characters because the C0 set is excluded.

Now, to my personal taste and based on a number of things I
think we have learned since the EAI work got started, we would
be a lot better off further reducing the list of valid
EAI-address-characters.  It would probably be unreasonable to go
all the way to the strict letters (and characters used to
construct them), digits, and hyphen rule used by IDNA, but we
would almost certainly be better off if we confined ourselves to
graphic ("printing" in the vocabulary of 5322) characters and
perhaps a subset of those.  However the WG has _not_ discussed
such restrictions, so the intent is that EAI-address-characters
contain those described above.

And, because of the exclusion of C0 (as well as C1) controls,
all of the "[US-]ASCII is a subset of UTF-8" discussions are not
really relevant, quite independent of Dave's concern that the
statement itself represents a muddling of terminology, at least
unless one assumes that both have their meaning as "charset"
names, not a repertoire name and an encoding name.

    john

---

From: "Jiankang YAO" <[email-masked]>
Date: Tue, 11 Jan 2011 17:14:32 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Jiankang YAO" <[email-masked]>; "Claudio Allocchio" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, January 11, 2011 3:55 PM
Subject: Re: [EAI] syntax: uAtom

.....
> list. 

we add the question similar to below?

question:
what is the proper definition of uAtom or eai-local-part-characters?

I think that Chairs may have a better suggestion.

Jiankang Yao

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 11 Jan 2011 18:51:32 +0900

On 2011/01/11 18:14, Jiankang YAO wrote:
> From: "John C Klensin"<[email-masked]>

> what is the proper definition of uAtom or eai-local-part-characters?

I think this is too open. There is the potential to spend endless time 
discussing this and that (kind of) character/code point. I'd reword it as:

Do we need to tweak the definition of uAtom, and if yes, how?

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: "Jiankang YAO" <[email-masked]>
Date: Wed, 12 Jan 2011 11:37:25 +0800

----- Original Message ----- 
From: ""Martin J. Dürst"" <[email-masked]>
To: "Jiankang YAO" <[email-masked]>
Cc: "John C Klensin" <[email-masked]>; "Claudio Allocchio" <[email-masked]>; "Joseph Yee" <[email-masked]>; <[email-masked]>
Sent: Tuesday, January 11, 2011 5:51 PM
Subject: Re: [EAI] new question to the question list Re: syntax: uAtom

> Do we need to tweak the definition of uAtom, and if yes, how?

it is better than mine.

:)

Jiankang Yao