Subject: Re: Normalizing Unicode (Re: [EAI] silly states and header label)

From: Harald Alvestrand <[email-masked]>
Date: Mon, 03 Apr 2006 12:58:39 +0200

Charles Lindsey wrote:
> On Thu, 30 Mar 2006 08:32:23 +0100, Harald Alvestrand 
> <[email-masked]> wrote:
>
>> I think the discussion so far has proved just about conclusively that 
>> the standards cannot possibly require recipient systems to have email 
>> addresses that pass either the telephone number test or the fax test.
>
> Sure we cannot write rules telling people to say "zwo" rather than 
> "zwei", but we can avoid the grosser failures of the telephone and fax 
> tests (and especially the ones which will still confuse people even if 
> they take the greatest care over their pronunciation and handwriting) 
> by requiring some reaonable level of normalization of email addresses 
> (and of the local-part in particular). John's "Net-utf-8" document may 
> or may not be sufficient for the purpose, so I suggest we wait and see 
> what that covers before returning to this subject again. 
Charles,

I do not think you heard me the first time.

As far as I can tell, all participants except for you have said that the 
"fax test" and "telephone test" are NOT relevant to the EAI working group.

Unless I hear differently from someone else, I'm going to note this as a 
closed issue for the WG.

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Martin Duerst <[email-masked]>
Date: Wed, 05 Apr 2006 14:16:03 +0900

Hello Harald, Charles, others,

I agree with this in the sense that it is virtually impossible to
create something that will pass these tests in all cases.

However, I don't exactly like the quick-shot conclusion along the
lines of "if we can't be absolutely perfect, we better do nothing
at all". (That's at least one way, and my first way, of reading
Harald's text above.)

"zwei", but we can avoid the grosser failures of the telephone and fax 
tests (and especially the ones which will still confuse people even if they 
take the greatest care over their pronunciation and handwriting) by 
requiring some reaonable level of normalization of email addresses (and of 
the local-part in particular). John's "Net-utf-8" document may or may not 
be sufficient for the purpose, so I suggest we wait and see what that 
covers before returning to this subject again.
closed issue for the WG.

Just for the record, you just heard differently from someone else.
The fax test and the telephone test are relevant for the end users
of our technology. We as providers of this technology need to make
sure that they have a reasonable chance to create and use addresses
that work well for them. So these test are not directly relevant,
but they are indirectly very relevant.

It may turn out that things will work out without us doing something,
it may turn out that we only need to give some advice and direction,
or it may turn out that we need some stick in the ground.

As an example, it would be a bad thing if half of the email programs
used in Norway would encode an email address containing an A-ring as
precomposed, and the other half would encode it as decomposed, but
receiving servers would just do binary comparison.

Charles' starting point was IDNA. The main reason for why I think
we will end up with something different from IDNA is that one of
the main concerns in IDNA was conflicting registrations, or the
need for multiple (potentially a high number) of registrations to
avoid having a name being spoofed. This is considerably different
for LHS. In most cases, LHS don't get handed out simply on a
first-come-first-served-anything-goes basis. Where they do
(webmail hosting,...), the organizations in charge should be
large enough to be able to manage the problem.

The other thing that we don't need to copy from IDNA, because it
was probably a mistake (even though not really such an important one),
is the use of NFKC. This came about because some people wanted
half-width and full-width variants (as they appear in East Asian
encodings) to be treated equivalently, the same way uppercase and
lowercase are treated as equivalents in the DNS. This was only
available in NFKC, but that brought in a host of other equivalences
that were really not necessary. A good example is that in a
fully IDNA-compliant application, you can use special numbers
with circles around them in place of the usual ASCII digits,
and get to the same place. On the LHS, uppercase and lowercase
are (a priori) different, so we don't have to worry about
half-width vs. full-width, so we should be fine without NFKC.

Regards,    Martin.

---

From: John C Klensin <[email-masked]>
Date: Mon, 10 Apr 2006 12:20:29 -0400

Martin,

I don't know whether Harald and I are in agreement or not, but
my perspective on this is a bit different from yours.  I don't
know if the differences are important, but I want to identify
them.

--On Wednesday, 05 April, 2006 14:16 +0900 Martin Duerst
<[email-masked]> wrote:

>>Unless I hear differently from someone else, I'm going to
>> note this as a closed issue for the WG.
> 
> need for multiple (potentially a high number) of registrations
> to avoid having a name being spoofed. 

Actually, those concerns were not discussed very much while IDNA
was being formulated (although they were discussed).  The
original driving force was the combination between the DNS's
extremely restrictive matching rules and the protocol
requirement for case mapping (at least of ASCII characters).
There was no way to change the DNS matching rules without making
changes to the way the DNS itself worked.  And keeping the
spirit of case-matching required some considerable mapping
effort.  

By contrast, the SMTP protocol imposes a different mapping norm.
Its matching rule is effectively bitstring identity --not for
characters but for the entire LHS -- until one gets to the final
delivery server, which can apply any matching (or aliasing)
rules that it likes.   

These differences are very much key to the design of the two
protocols and are, arguably, the inevitable consequences of how
those protocols were designed and evolved.

> This is considerably  different

Also true.

> The other thing that we don't need to copy from IDNA, because
> it was probably a mistake (even though not really such an
> important one),

Yes, both to the conclusion and to the concerns that NFKC may
not have been a good idea for IDNA.  I would suggest that its
use in nameprep had another consequence, which is that some
character blocks that should simply have been banned were
permitted on the grounds that NFKC would take care of mapping
them to something more satisfactory.   In retrospect, that was
almost certainly a mistake, independent of whether the use of
NFKC was a mistake.

But the more important issue here, to me, is that the tradition
with email --both one the of causes of the "no one tries to
interpret LHS addresses other than the delivery server" rule and
its logical consequence.  So there is no protocol requirement
for ASCII case-matching.  Instead, we say "the protocol permits
you to enforce an exact, bitstring-identity, match but taking
advantage of that is normally stupid" and then we suggest
aggressive use of the robustness principle on the receiving
side.  

That leads, in practice, to matching rules that go far beyond
anything supported by (even) NFKC.  I'm on this list as
"[email-masked]".  It is up to the maintainer of the server,
not only whether "KlEnSiN" is accepted in the LHS, but also as
to whether "john.klensin", "john", "john/klensin", "jk",
transliterations into several other languages, and even
"donald-duck" are accepted and, if they are accepted, what
happens to them (before people try, some of those examples will
work, some won't, and not all of those that work will go to the
same mailbox/folder).  From the sender end, there is no right to
assume that any variant will work other than the one that
appears a Reply-To field or its equivalent, or to make
assumptions about what happens to those variations.

So, to come back to the A-ring example, I think that, in the
general case, it would be insane behavior on the part of the
receiving system to not have these match, just as I think it
would be insane to not do ASCII case matching in the general
case.  Similarly, if I were you, I'd be pretty insistent that,
on your delivery server, either "duerst" or "dürst", and all of
their case variations, worked and mapped to the same folder.
Failure to do that would not only fail the telephone test, but
would just be dumb and violating of the law of least
astonishment.  But trying to turn it into a network requirement
rather than a delivery server convention (noting that particular
mapping goes beyond even NFKC (for good reasons), just as
mapping "klensin" -> "john.klensin" does, would be a really bad
idea.

Put differently, requiring the general case would violate many
years of experience with email protocols: senders need to remain
used to the fact that specifying an address other than exactly
as given _may_ cause bounces or other bad behavior.  And I say
"remain" because it has been that way for 35 or more years now.
Our addresses have never been guaranteed to pass what Charles
refers to as the telephone number or fax tests, even in ASCII.
When they do, it is purely on the basis of robustness and
aliases in the delivery servers, not as a result of the
protocols.

     john

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 10 Apr 2006 21:41:46 +0100

On Wed, 05 Apr 2006 06:16:03 +0100, Martin Duerst <[email-masked]>  
wrote:

>  >As far as I can tell, all participants except for you have said that  
> the "fax test" and "telephone test" are NOT relevant to the EAI working  
> group.
>  >
>  >Unless I hear differently from someone else, I'm going to note this as  
> a closed issue for the WG.
>

Thank you.

> lowercase are treated as equivalents in the DNS.

You surprise me. But we have many East Asians on this list, so we need to  
hear from them. It seems to me that local-parts with characters of the  
"wrong" width would easily fail the telephone test. It it usual in those  
languages to convey information (as opposed to style) in the widths of the  
characters that are used?

> This was only
> and get to the same place.

So would we want it otherwise? Admittedly those would pass both the  
telephone and the fax tests, but it still strikes me as very bad practice,  
and if we lose those just because we wanted to keep it to "all of NFKC or  
none of it", it would not cause me any grief.

Anyway, these are all issues we need to coinsider at some time. The first  
step is to see and discuss what John is currently writing.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 10 Apr 2006 21:41:46 +0100

Thank you.

---

From: John C Klensin <[email-masked]>
Date: Tue, 11 Apr 2006 02:37:16 -0400

--On Monday, 10 April, 2006 21:41 +0100 Charles Lindsey
<[email-masked]> wrote:

>> important one), is the use of NFKC. This came about because
>> some people wanted half-width and full-width variants (as
>> they appear in East Asian encodings) to be treated
>> equivalently, the same way uppercase and lowercase are
>> treated as equivalents in the DNS.
> 
> You surprise me. But we have many East Asians on this list, so
> we need to  hear from them. It seems to me that local-parts
> with characters of the  "wrong" width would easily fail the
> telephone test. It it usual in those  languages to convey
> information (as opposed to style) in the widths of the

Charles,

Please see my recent response to some of these issues.   If I
were running an email receiving system for which half-width and
full-width variations were considered important, the robustness
principle would certainly argue for accepting them as
equivalent.  But it is neither necessary nor desirable to try to
write rules like this into the protocol.

>> This was only
>> available in NFKC, but that brought in a host of other
>> equivalences that were really not necessary. A good example
>> is that in a fully IDNA-compliant application, you can use
>> special numbers with circles around them in place of the
>> usual ASCII digits, and get to the same place.
> 
> So would we want it otherwise? Admittedly those would pass
> both the  telephone and the fax tests, but it still strikes me
> as very bad practice,  and if we lose those just because we
> wanted to keep it to "all of NFKC or  none of it", it would
> not cause me any grief.
> 
> Anyway, these are all issues we need to coinsider at some
> time. The first  step is to see and discuss what John is
> currently writing.

What John is currently writing in this area will be, for
addresses, strictly conformant with the traditional email norm
that acceptability and matching are entirely the province of the
delivery MTA.   What I'll have to say about canonicalization of
headers (still waiting for one of my collaborators) will be what
Martin and the (other) Unicode folks have recommended, which is
NFC and only NFC -- unless someone can demonstrate that it is
either overkill or insufficient.

     john

---

From: "YAO Jiankang" <[email-masked]>
Date: Tue, 11 Apr 2006 14:58:44 +0800

----- Original Message ----- 
From: "Charles Lindsey" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, April 11, 2006 4:41 AM
Subject: Re: Normalizing Unicode (Re: [EAI] silly states and header label)

> characters that are used?
Dear Charles,

    it is true that some people wanted  half-width and full-width variants (as they appear in East Asian
 encodings) to be treated equivalently. for example, in China, few email users konw what is half-width variants and  what is full-width variants; even many university students whose major is computer science don't konw, as far as I know. so it is almost impossible to recognize what is half-width variants and  what is full-width variants for some users.
So as an Aisan, I prefer that half-width and full-width variants to be treated equivalently, the same way uppercase and lowercase are treated as equivalents in the DNS.

Yao Jiankang
CNNIC