Subject: [EAI] Extending IDNA to email

From: Nick Teint <[email-masked]>
Date: Tue, 23 Mar 2010 21:16:49 +0100

Hi!

Today, I've submitted several Internet-Drafts describing a proposed
framework to use IDNA(bis) for non-domain addresses.

One of the drafts describes a profile for email addresses, which would
complement IMA. To be specific, it could be used as a method for
algorithmic downgrade of UTF8 addresses.

The drafts relating to email can be found here:
http://www.ietf.org/internet-drafts/draft-teint-xidna-base-00.txt
http://www.ietf.org/internet-drafts/draft-teint-xidna-email-00.txt

NT

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 24 Mar 2010 11:27:47 -0000

> http://www.ietf.org/internet-drafts/draft-teint-xidna-email-00.txt

Well I have glanced quickly through both of those, and either I have  
missed something, or they are incapable of preserving case in local-parts  
(contrary to the expectations of RFCs 5321 and 5322). Whilst it may be  
stupid for case to be significant in local-parts, it is the prerogative of  
the final delivery agent (the only place where local-parts may be taken  
apart) to be stupid if it is so minded.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Nick Teint <[email-masked]>
Date: Wed, 24 Mar 2010 17:03:03 +0100

2010/3/24 Charles Lindsey <[email-masked]>:
> Well I have glanced quickly through both of those, and either I have missed
> something, or they are incapable of preserving case in local-parts (contrary
> to the expectations of RFCs 5321 and 5322). Whilst it may be stupid for case
> to be significant in local-parts, it is the prerogative of the final
> delivery agent (the only place where local-parts may be taken apart) to be
> stupid if it is so minded.

That's correct. However, this limitation only applies to addresses
that contain non-ASCII characters. IMO it's not worthwhile to preserve
case-sensitiveness (which is retained "just in case" there's a system
that still needs it) for an expansion of the address space.

In theory, IDNAbis *does* allow case-sensitiveness; the mapping
document is purely optional. In practice, using that feature is almost
always a bad idea: In IDNA, some people will be unable to look up your
domain (as software maps input to lower case); in email with X-IDNA,
some people would be unable to send you mail.

NT

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 25 Mar 2010 10:45:37 -0000

On Wed, 24 Mar 2010 16:03:03 -0000, Nick Teint <[email-masked]>  
wrote:

>> missed
>> something, or they are incapable of preserving case in local-parts .....

> that still needs it) for an expansion of the address space.

I cannot see that is a satisfactory response. RFC 5321 makes it clear that  
a local-part is entitled to be conveyed case-sensitively. It is not a  
matter of "just in case" the final agent needs it - the final agent has  
absolute discretion to insist on it (unless and until RFC 5321 is revised  
to say otherwise). AFAIR, our EAI extension still preserves that  
requirement (but now it applies to lots of new characters which may well  
have alternative UC and LC versions). So your method could not be used as  
a downgrade mechanism for EAI addresses.
> some people would be unable to send you mail.
Indeed, but that feature clearly MUST NOT (or is SHOULD NOT sufficient?)  
be used for domain-names. But your draft makes provision for profiles for  
the various usages, so you have sufficient handles there to be able to  
specify which cases should use a case-sensitive version of punycode, and  
which not.

---

From: Nick Teint <[email-masked]>
Date: Fri, 26 Mar 2010 13:44:57 +0100

2010/3/25 Charles Lindsey <[email-masked]>:
> On Wed, 24 Mar 2010 16:03:03 -0000, Nick Teint <[email-masked]> wrote:
> I cannot see that is a satisfactory response. RFC 5321 makes it clear that a
> local-part is entitled to be conveyed case-sensitively. It is not a matter
> of "just in case" the final agent needs it - the final agent has absolute
> discretion to insist on it (unless and until RFC 5321 is revised to say
> otherwise).

Most final agents do not insist. The reason RFC 5321 still has this
requirement is backwards compatibility, "just in case".

X-IDNA does not remove this requirement for existing ASCII addresses.
A final agent can insist that the local-part it's being sent is
exactly "xn--Mller-kva" and treat it differently from "xn--mller-kva".
It just can't insist that a used who types "Müller" (German for
"Miller") will end up sending mail to "xn--Mller-kva"; user agents
would usually map "Müller" to "müller" and then to "xn--mller-kva".

Actually, X-IDNA *is* case-sensitive, just as IDNA2008: The mapping to
lowercase is considered a local matter, described in "Mapping
Characters in IDNA" (draft-ietf-idnabis-mappings-xx). If you don't do
it, the conversion to Punycode is case-preserving. Then, however, it
gets complicated: A case-insensitive comparison based on the ASCII
form will compare ASCII chars (encoded as-is in Punycode)
case-insensitively but non-ASCII (encoded as lower case ASCII chars)
chars case-sensitively. Further, case-insensitive comparison is not
well-defined in Unicode and depends on the locale (in Turkey, I/ı and
İ/i form pairs; in Germany, there's ẞ/ß/ss/SS; in France, there's
E/é/è/ê; etc.)

IDNA2008 (much unlike IDNA2003) avoids this mess by assuming that all
internationalised addresses are in lower case. However, this
assumption is not part of the core protocol but an assumption that can
be used by applications when mapping user input to useable strings.
The protocol only requires NFC.

This approach would also work with EAI (UTF8SMTP):

- User Agents are expected to do any mapping. They may take the user's
preferences and expectations into account.
- Mail Handling Services do not do any mapping
- "Registries" should not create local-parts that are unwise, i.e.
that contain characters commonly mapped to other characters
(upper-case letters, chars with a canonical decomposition, etc.)

- In UTF8SMTP, Mail Handling Services MAY complain about non-NFC characters.
- When downgrading using X-IDNA, Mail Handling Services do convert to
ASCII but they do not do any further mapping.

> AFAIR, our EAI extension still preserves that requirement (but
> now it applies to lots of new characters which may well have alternative UC
> and LC versions). So your method could not be used as a downgrade mechanism
> for EAI addresses.

EAI is currently silent about case-sensitivity or mapping.

However, future revisions of EAI will have to approach this issue, as
seen by this example:
Bob wants to send an email to Abdülmecid <[email-masked]>. He
types the email address as "[email-masked]", expecting that
case does not matter. The mail bounces. While Abdülmecid's ISP does
indeed match local-parts case-insensitively, it is located in Turkey.
So it treats "ABDÜLMECID" as equivalent to "abdülmecıd" but not
"abdülmecid".

NT

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 29 Mar 2010 11:22:24 +0100

> requirement is backwards compatibility, "just in case".

No, that is not what RFC 5321 says. What it actually says is:

    Verbs and argument values (e.g., "TO:" or "to:" in the RCPT command
    and extension name keywords) are not case sensitive, with the sole
    exception in this specification of a mailbox local-part (SMTP
    Extensions may explicitly specify case-sensitive elements).  That is,
    a command verb, an argument value other than a mailbox local-part,
    and free form text MAY be encoded in upper case, lower case, or any
    mixture of upper and lower case with no impact on its meaning.  The
    local-part of a mailbox MUST BE treated as case sensitive.
    Therefore, SMTP implementations MUST take care to preserve the case
    of mailbox local-parts.  In particular, for some hosts, the user
    "smith" is different from the user "Smith".  However, exploiting the
    case sensitivity of mailbox local-parts impedes interoperability and
    is discouraged.  Mailbox domains follow normal DNS rules and are
    hence not case sensitive.

There are a couple of MUSTs in there which you are not at liberty to  
violate. The only evidence of "just-in-case" is that word "discouraged",  
which is pretty weak (it could have said "SHOULD NOT" it it was really  
serious about it.

And AFAICS, RFC 5536 inherits both of those MUSTs (at least, it is silent  
on the issue, which implies no change).

Hence it follows that any I18N encoding scheme MUST preserve local-parts  
exactly. I.e. if some UTF-8 local part is encoded into ASCII, then
   a) the encoded form MUST be different from the encoding of any "tolower"  
transformation of the original UTF-8, and
   b) the decoding of that encoding form MUST recover the EXACT form of the  
original UTF-8 local part.

Any if IDNA/punycode cannot achieve this result, then your Draft MUST fix  
it so that it does.

Quoting RFC 5321 again:

    ..., the local-part MUST be
    interpreted and assigned semantics only by the host specified in the
    domain part of the address.

> EAI is currently silent about case-sensitivity or mapping.

Exactly so.
> So it treats "ABDÜLMECID" as equivalent to "abdülmecıd" but not

That is entirely a problem for Abdülmecid's ISP. What Bob typed was  
clearly wrong. So long as Bob typed the correct thing, then it should go  
through. Abdülmecid's ISP is indeed "discouraged" by RFC 5321, and if it  
persists, it would be well advised to apply some pretty strict  
normalizations. But that it its problem. Your draft MUST NOT make things  
worse by trying to change anything en route.

---

From: Nick Teint <[email-masked]>
Date: Mon, 29 Mar 2010 19:02:24 +0200

> which is pretty weak (it could have said "SHOULD NOT" it it was really

When writing a spec, you actually CAN change requirements. The only
potential issue is backwards-compatibility. However, this is not a
issue here, as I have explained.

> And AFAICS, RFC 5536 inherits both of those MUSTs (at least, it is silent on
> the issue, which implies no change).

RFC 5536 is not only silent on case mapping but also on any mapping,
including compatibility mapping, mapping away invisible characters,
etc. This is a bad thing and needs to be fixed if RFC 5536 is going
from Experimental to Standards Track.

If it is not fixed, Unicode mail WILL bounce without any apparent
reason. The easiest way to fix it is to take IDNAbis' mapping.

NT

---

From: John C Klensin <[email-masked]>
Date: Wed, 31 Mar 2010 00:24:06 -0400

--On Monday, March 29, 2010 11:22 +0100 Charles Lindsey
<[email-masked]> wrote:

>... 
> And AFAICS, RFC 5536 inherits both of those MUSTs (at least,
> it is silent on the issue, which implies no change).

And, speaking as 5321 and 4952 editor, rather than as EAI Chair,
that was a very explicit decision.

> Hence it follows that any I18N encoding scheme MUST preserve
> local-parts exactly. I.e. if some UTF-8 local part is encoded
> into ASCII, then
>    a) the encoded form MUST be different from the encoding of
> any "tolower" transformation of the original UTF-8, and
>    b) the decoding of that encoding form MUST recover the
> EXACT form of the original UTF-8 local part.

Yes.  And, given the very strong 5321 prohibition (which Charles
cites) about anything other than the final delivery server
making assumptions about what local-parts "mean", the fact that
an encoding scheme/ transformation is being used much be
reliably transmitted out of band, rather than relying on tricks
like special marker characters or prefixes.

The above has been discussed extensively, several times, in this
WG.  Please do not revisit it without, at least, studying the
archives.

>...

>> However, future revisions of EAI will have to approach this
>> issue, as seen by this example:
>> Bob wants to send an email to Abdülmecid
>> <[email-masked]>. He types the email address as
>> "[email-masked]", expecting that case does not
>> matter. The mail bounces. While Abdülmecid's ISP does indeed
>> match local-parts case-insensitively, it is located in Turkey.
>> So it treats "ABDÜLMECID" as equivalent to "abdülmecıd"
>> but not "abdülmecid".
> 
> That is entirely a problem for Abdülmecid's ISP. What Bob
> typed was clearly wrong. So long as Bob typed the correct
> thing, then it should go through. Abdülmecid's ISP is indeed
> "discouraged" by RFC 5321, and if it persists, it would be
> well advised to apply some pretty strict normalizations. But
> that it its problem. Your draft MUST NOT make things worse by
> trying to change anything en route.

Exactly.  

And this actually illustrates the problem rather well, as well
as illustrating one of several reasons why IDNA2008 took such a
conservative view of mapping: in an internationalized situation,
the sender cannot know what sort of matching rule is applied for
case because case-matching is locale-dependent.

--On Monday, March 29, 2010 19:02 +0200 Nick Teint
<[email-masked]> wrote:

>> There are a couple of MUSTs in there which you are not at
>> liberty to violate. The only evidence of "just-in-case" is
>> that word "discouraged", which is pretty weak (it could have
>> said "SHOULD NOT" it it was really serious about it.
> 
> When writing a spec, you actually CAN change requirements. The
> only potential issue is backwards-compatibility. However, this
> is not a issue here, as I have explained.

Actually, while one can imagine cases in which changes to the
base email specs are necessary, it has been a very general
assumption of the EAI work that doing something differently than
5321/5322/4409 do it requires rather strong justification, lest
users and systems be _really_ confused.

>> And AFAICS, RFC 5536 inherits both of those MUSTs (at least,
>> it is silent on the issue, which implies no change).
> 
> RFC 5536 is not only silent on case mapping but also on any
> mapping, including compatibility mapping, mapping away
> invisible characters, etc. This is a bad thing and needs to be
> fixed if RFC 5536 is going from Experimental to Standards
> Track.

While I would personally favor some clarification of this in any
5536bis (but it is not on the WG's task list), it is a bit
unusual in a protocol document to specify what one doesn't do.
I.e., the issue is not just the silence and implication to which
Charles refers, but the observation that 5536 not only doesn't
say "don't apply those mappings", it also doesn't say "don't XOR
the characters with an arbitrary string", "don't reverse the
strings right-to-left", "don't delete Latin vowels", etc.  You
may think it would be silly to do any of those things, and I'd
probably agree but, for email addresses, I'd suggest that
mapping characters into other characters (or throwing some
characters away), etc., would be equally strange given the way
email has worked for the last few decades.

> If it is not fixed, Unicode mail WILL bounce without any
> apparent reason. The easiest way to fix it is to take IDNAbis'
> mapping.

I don't understand this proposal given that IDNAbis doesn't map.
It permits client programs to apply mapping rules as needed by
local circumstances, but note that the DNS situation is very
different from the email one.  First, DNS does case-insensitive
matching for ASCII; actually changing of strings in one case
into strings in another before issuing the queries has generally
been discouraged.  Email has never done such matching as part of
the protocol and changing strings in transit is explicitly
prohibited.  So I also don't see the analogy in what you are
suggesting.

    john