Subject: Re: [EAI] Unicode vs. UTF-8 / Encoding vs. Representation

From: John C Klensin <[email-masked]>
Date: Mon, 03 Jan 2011 23:49:36 -0500

--On Thursday, December 30, 2010 16:33 -0800 Dave CROCKER
<[email-masked]> wrote:

>...
> If a message is in legacy form, then it is ASCII (or, rather,
> ASCII-only).  If it is in Unicode form, then it is in Unicode,
> even if all the characters are x07F or below...

To amplify on my previous note with another example, consider
the VRFY/EXPN situation.  If the server receives an all-ASCII
argument, the condition in the spec is that it MUST NOT return a
string in which any characters outside the ASCII repertoire are
present.    One can figure out lots of more or less convoluted
ways to say that (the above is an example), but the obvious ones
all involve having a term for what has, in IETF i18n
terminology, traditionally been called "non-ASCII".

    john

---

From: John C Klensin <[email-masked]>
Date: Tue, 04 Jan 2011 00:04:45 -0500

--On Thursday, December 30, 2010 10:40 +0000 Charles Lindsey
<[email-masked]> wrote:

> If you are looking for a term to describe some UTF-8 text that
> is not also an ASCII text, then normal mathematical usage is
> to use the word "proper".
> 
> E.g., a is A is a "proper subset" of B, then A is not
> identical to B.
> 
> So something like "proper-utf8" would follow that concention.
> You might even use "PROPER_UTF8" as that parameter of the MAIL
> command.

At one level, harking back to the years when I pretended to be a
mathematician (or at least an aspiring one), this really appeals
to me.  At another, I believe that "proper utf-8" (no matter how
punctuated) will be read by most non-mathematicians and even
some mathematicians who haven't been alerted to the special
usage) as the opposite of "improper UTF-8" or "invalid UTF-8".  

Unfortunately, that is a real category.   A small oversight in
the original UTF-8 spec (see RFC 2044 and elsewhere) permitted
there to exist situations in which two [bitstring-] different
UTF-8 strings would map to/represent the same Unicode code
point.  That was later corrected (see RFC 2279 and 3629 and
elsewhere) so that the longer of the two encodings became
invalid.

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 05 Jan 2011 11:48:29 -0000

> usage) as the opposite of "improper UTF-8" or "invalid UTF-8".

Yes, I was a bit worried about that. So how about "UTF8_PROPER".

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Barry Leiba <[email-masked]>
Date: Fri, 7 Jan 2011 16:08:57 -0500

I've been watching the discussion about this and thinking about it
myself, weighing the arguments everyone's been giving.  And I've
decided it's time to step in and give my own opinion.

I think it's very important to make the distinction between characters
("Unicode") and encoding ("UTF-8"), and to (1) be rigorous in the
documents about using the terms we choose, and (2) choose our terms
and define them clearly.

I'm sympathetic to the arguments about what's been used, misused, and
inconsistently used in the past.  I'm sympathetic to the arguments
about trying to define terms in this document, given that the
definitions are needed more broadly, that people can't be depended
upon to read and understand *our* versions of the terms, and so on.

Nevertheless, I think it's what we have to do.  I don't much care
whether we choose (1) "Latin" characters and "ASCII" encoding, (2)
"ASCII" characters and "7-bit" encoding (which is defined in RFC 2045
in a way that I think is adequate and e-mail-related), or (3)
something else.  What I care about is that we settle on something,
define what we mean clearly, and move these documents forward without
too much more delay.

I also think it's important to maintain the concept of a string that
contains characters that are not in the ASCII character set (what
we've been calling "non-ASCII").  I think we can limit the use of the
term we choose for that to the "characters" side of things (and not
use it for the "encoding" side).  For example, we might say (contrived
example, just to make the point) "a string that contains non-ASCII
characters [or non-Latin characters, or whatever] MUST be encoded with
UTF-8 encoding.

Finally, I think that in order to make things absolutely clear, we
NEVER use the word "character" when we refer to encoding, and we
ALWAYS use the word "encoding" and treat "ASCII" or "7-bit" or "UTF-8"
as a modifier, not as a noun (in other words, we NEVER say just
"UTF-8", but ALWAYS "UTF-8 encoding").

I am willing (eager, I might even say) to draft a document section
that does the definitions, to give us something specific to talk
about.  I'll wait until someone appropriate tells me to go ahead with
that before I do it.  But I think that if we have definition text to
talk about, and if we take the approach that we need to do this as
well as we reasonably can, but understand the futility of expecting
perfection, we can get this settled and move ahead.

Barry

---

From: "Jiankang YAO" <[email-masked]>
Date: Tue, 11 Jan 2011 15:06:35 +0800

----- Original Message ----- 
From: "Ned Freed" <[email-masked]>
To: <[email-masked]>; <[email-masked]>; <[email-masked]>; <[email-masked]>; "SM" <[email-masked]>
Sent: Thursday, December 23, 2010 10:01 AM
Subject: Re: [EAI] apps-team review of draft-ietf-eai-rfc5336bis

....
> 
> (1) 3.1 item 4 and 3.6.4.2 add an optional UTF8REPLY parameter to VRFY and
>    EXPN. They do this by redefining the syntax to be:
> 
>    vrfy = "VRFY" SP ( uLocal-part / uMailbox )
>                     [ SP "UTF8REPLY" ] CRLF
> 
>    expn = "EXPN" SP ( uLocal-part / uMailbox )
>                     [ SP "UTF8REPLY" ] CRLF
> 
>    The original syntax for VRFY and EXPN given in RFC 5321 is:
> 
>    vrfy = "VRFY" SP String CRLF
>    expn = "EXPN" SP String CRLF
>    String = Atom / Quoted-string
>    Atom = 1*atext
> 
>    It's not immediately obvious, but the new syntax is in fact a superset of
>    the old. (uLocal-part subsumes local-part, which in turn allows atoms
>    and quoted strings.) However, this change does much more than hang a
>    parameter on the command; for the first time an entire address is
>    permitted as an EXPN/VRFY argument. But no semantics are given for
>    this significant syntax expansion, nor does this expansion appear to be
>    within scope for EAI.
> 
>    A more reasonable syntax would be:
> 
>    vrfy = "VRFY" SP uString [ SP "UTF8REPLY" ] CRLF
> 
>    expn = "EXPN" SP uString [ SP "UTF8REPLY" ] CRLF
> 
>    uString = uAtom / uQuoted-string
>
>

thanks for your kind comments and suggestions.

Personally, I prefer the syntax definition above. I think that it is more clear.

for the syntax definition below,
It is unlikely that there will have new parameter for vrfy or expn in the near future. even if it has, the new specification can still define the new one.

Jiankang Yao

> 
>    But this now fails along a different dimension: When adding a parameter
>    capability to a command, it should be done in a general way, so that any
>    future extensions can build on a cleanly specified foundation. Like so:
> 
>    vrfy = "VRFY" SP uString [ SP vrfy-parameters ] CRLF
>    vrfy-parameters = esmtp-param *(SP esmtp-param)
> 
>    expn = "EXPN" SP uString [ SP expn-parameters ] CRLF
>    expn-parameters = esmtp-param *(SP esmtp-param)
> 
>    uString = uAtom / uQuoted-string
> 
>    Of course this will add 555 as a possible response to VRFY/EXPN, and that
>    needs to be noted.
>