Subject: [EAI] Downgrade Design Team Discussion Results Released

From: "YAO Jiankang" <[email-masked]>
Date: Tue, 16 Mar 2010 14:45:22 +0800

Dear all,

   The design team has several jabber meetings and reached a conclusion.
With the permission from co-chairs, I send the Design Team Discussion Results to WG for review and comments.
We will discuss this result and hope to get some consensus for downgrade during our meeting in Anaheim.
any comments are welcome.

below is the Design Team Discussion Results: (or see the attachment)
------------------------------------------------------------------------------
The EAI Downgrade Design Team was created to investigate the
problems around EAI downgrade.  The Design Team's conclusion is
that downgrade should be removed from the core docs.  This
allows the EAI WG to focus on moving the core RFCs to standards
track immediately, without being blocked by downgrade.
Alternatives to downgrade have been discussed, but the Design
Team recommends any further work be restricted to a
MUA/submission protocol independent of the core transmission
documents.

For precision, the terminology used in this document is
summarized at its end. 

The goal of the EAI Downgrade Design Team was to unblock the
discussion of downgrade to enable progress on the EAI
standards.  A key factor is a strong desire to move the core
documents to standards track.  Extensive debate of downgrade
could obstruct the rest of EAI for the foreseeable future.

The EAI Downgrade Design Team reached this conclusion by
examining the results of the current downgrade experiment and
the discussion and suggestions of the EAI working group.  We
grouped the possible approaches to backwards legacy SMTP
compatibility into three buckets:

1) Declared address pairs, eg: Alt-Addr.  This is the approach
   described in RFC5504 (downgrade) and variations on it.

2) Discoverable fallback address.  This includes any
   MUA/submission server protocol that derives ASCII addresses
   from a Unicode address for resubmission by legacy SMTP.  It
   also includes other algorithmic techniques, lookup of
   alternate addresses in local address books, and those using
   protocols external to SMTP processing to find those
   addresses.  The key characteristic of this bucket is that
   there is no downgrading of messages in transit; changes to
   addresses or other information are made only at the
   originating MUA or submission server, typically in response
   to external knowledge or some information received by them
   such as an NDN.  This bucket does not require Alt-Addr
   parameters, header address syntax to support them, or other
   dependencies on legacy compatibility strategies in the core
   transmission protocols. 

3) No fallback.  This devolves into "messages with
   undeliverable addresses are rejected", i.e., the
   traditional SMTP model.  A user, or programs acting on the
   user's behalf, can, of course, alter or resubmit the
   message with different address information.

Declared address pairs (e.g., with Alt-Addr) have been ruled
out as a possible solution.  The RFC5504 experiment was
invaluable in revealing the limitations of a paired address
system.  While the idea of friendly alternate addresses is
attractive, the Design Team concludes that paired address
mechanisms are inherently fragile.  The pairing has innate
problems with lost pairs, mismatched pairs, possible spoofing,
and failure in some cases.  Additionally, the extensions
provided by RFC5504 cause problems for some legacy mail
systems.

Discoverable fallback address mechanisms are potentially
interesting because they can always be determined by systems
aware of the discoverability convention.  However, discoverable
fallback address mechanisms do not require Alt-Addr and do not
have to be visible in the core documents.  There has been a lot
of discussion about the practicality of the proposed mechanisms
for discoverable fallback; we expect that those discussions
will continue, but that they will do so outside the critical
path.

In theory, discoverable fallback addresses could be discovered
by a relay server rather than at the MUA/submission server.
This approach was deemed impractical; it requires too much
coordination between parties that do not have communication
with each other.

No fallback is a technically "simple" solution.  It also does
not require Alt-Addr and requires no consideration in the core
documents.  The submission server may still resubmit using SMTP
if available.

By ruling out "declared address pairs" solutions as
impractical, and by removing any support for "downgrading at
relays", we remove the dependency of the core documents on
specific downgrade models.  This allows further progress on the
core documents by the working group.

The Design Team recognizes that "no fallback" provides a very
minimal transition mechanism, however we feel that "no
fallback" is better than allowing downgrade to block further
progress.  By supporting "no fallback" as the immediate
solution, we still allow for a "discoverable fallback address"
mechanism, should one be proposed.  However the Design Team
recommends that any such investigation to be independent of the
work on to finish the core documents.

In conclusion, the EAI Downgrade Design Team recommends:

o That we immediately recharter EAI to focus on the core
  standards track documents, i.e., RFC4952bis, RFC5335bis,
  RFC5336bis, RFC5337bis, RFC5721bis, and the in-progress
  drafts. 

o RFC5504 should be deprecated.  The core documents will not
  rely on fallback or downgrade techniques, particularly by
  relays.

o Discoverable fallback addresses may be investigated
  independently of the core documents.  Any such work would be
  part of, or connected to, an optional MUA/submission
  protocol, independent of the core transport documents. 

The design team also noted that an informational RFC regarding
selection of addresses (both Unicode and ASCII) would be
helpful, that clients SHOULD use Unicode Normalization Form C,
and that servers MUST use NFC.

Definitions:

Core Documents - RFC4952bis, RFC5335bis, RFC5336bis,
RFC5337bis, RFC5721bis, and the in-progress drafts, excluding
"downgrade".

Submission System - The user, sending MUA or submission server.

Fallback - A mechanism where the submission system attempts to
  resubmit mail using legacy SMTP if UTF8SMTP is rejected or
  returned as undeliverable.

Downgrade - RFC5504 style mechanisms that intend to ease
  transition between legacy SMTP systems and UTF8SMTP systems.

In-transit - message or address transformations performed in
  SMTP transport, after the message leaves the Submission
  Server and before delivery to the Final Delivery Server (as
  defined in RFC 5321), such as by relays.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 18 Mar 2010 23:38:23 +0000

As a side note, one of the things suggested by the EAI downgrade design 
team should be further discussed and should be coordinated with the 
NewPrep BOF:

> and that servers MUST use NFC.
Whatever normalization we recommend, it should be consistent with 
handling of nodes in XMPP (NodePrepBis) and SASLPrepBis.

---

From: John C Klensin <[email-masked]>
Date: Thu, 18 Mar 2010 19:59:19 -0400

--On Thursday, March 18, 2010 23:38 +0000 Alexey Melnikov
<[email-masked]> wrote:

> As a side note, one of the things suggested by the EAI
> downgrade design team should be further discussed and should
> be coordinated with the NewPrep BOF:
> 
> YAO Jiankang wrote:
> Whatever normalization we recommend, it should be consistent
> with handling of nodes in XMPP (NodePrepBis) and SASLPrepBis.

Alexey, FWIW, RFC 5198 suggests NFC and explains why.  Unless
there is a really strong reason to do something else, my
recommendation is that all three of these specs conform to that
recommendation.

      john

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 18 Mar 2010 22:14:02 -0600

> recommendation is that all three of these specs conform to that

Unfortunately, both Nodeprep (RFC 3920) and SASLprep (RFC 4013) used
NFKC, not NFC.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: John C Klensin <[email-masked]>
Date: Fri, 19 Mar 2010 00:59:43 -0400

--On Thursday, March 18, 2010 22:14 -0600 Peter Saint-Andre
<[email-masked]> wrote:

>> recommendation is that all three of these specs conform to
>> that recommendation.
> 
> Unfortunately, both Nodeprep (RFC 3920) and SASLprep (RFC
> 4013) used NFKC, not NFC.

So did Nameprep.  Unfortunately, NFKC is _very_ dangerous.  It
combines several different transformations, some quite
reasonable and obvious and some that are basically the UTC's
revenge on people who made them add characters they didn't want
to add.    The latter part results in characters being mapped
together that those who wanted them in Unicode think are
separate.

     john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 19 Mar 2010 10:03:18 +0100

> together that those who wanted them in Unicode think are

I don't think that makes NFKC very dangerous for email addresses. I 
think it bows to the inevitable.

Email addresses are used and processed by strangers all over the world, 
and often treated as plain old text. Suppose you send mail to A, who 
forwards your missive to B, who sees your address in the signature and 
cuts and pastes it into the To field of a new message. Don't you think 
the address may have gone through NFKC a time or two by the time B hits 
send?

This applies to jabber as well, but more strongly to email.

I'd rather have jabber-compatible syntax than offer rope to those who 
wish to hang themselves. If the rope were free I'd let them do as they 
please, but it isn't.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Fri, 19 Mar 2010 05:58:20 -0400

--On Friday, 19 March, 2010 10:03 +0100 Arnt Gulbrandsen
<[email-masked]> wrote:

> I don't think that makes NFKC very dangerous for email
> addresses. I think it bows to the inevitable.

Actually, if one is going to do internationalization properly,
one of the most often-cited goals is to permit people to spell
their names correctly.   Since many mailbox local parts reflect
personal names, that principle applies to them too.   And NFKC
eliminates a lot of characters that are needed to correctly
spell names.

    john

---

From: "Yao Jiankang" <[email-masked]>
Date: Fri, 19 Mar 2010 21:53:06 +0800

IMO:

The difference between NFKC and NFC is similar to the difference between 
IDNA2003 and IDNA2008.

the princeple of NFKC (idna2003) is to eliminate some characters (some 
characters will be invalid);
the princeple of NFC (idna2008) is to make more characters to be valid;

The answer to whether we should use NFKC or NFC is same to the answer to 
whether we should use idna 2003 or idna 2008.

Jiankang Yao

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Arnt Gulbrandsen" <[email-masked]>; <[email-masked]>
Sent: Friday, March 19, 2010 5:58 PM
Subject: Re: [EAI] Localpart normalization (was Re: Downgrade Design Team 
Discussion Results Released)

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>

---

From: Harald Alvestrand <[email-masked]>
Date: Sat, 20 Mar 2010 22:00:56 +0100

Alexey Melnikov wrote:
> As a side note, one of the things suggested by the EAI downgrade 
> design team should be further discussed and should be coordinated with 
> the NewPrep BOF:
>
> YAO Jiankang wrote:
Note:
we should be careful what we mean here when we say "MUST use NFC".

I read it as "everyone should expect mail to two left-hand-side whose 
name canonicalizes to the same string under NFC to end up in the same 
mailbox".

That's no guarantee that mail to Unicode(R with ring around it) and 
UnicodeR will NOT end up in the same mailbox. Any server can apply 
additional normalizations at will to localparts that it's responsible for.

The difference between putting NFC and NFKC in the sentence above is 
that no intermediate server, who's merely passing along the message, can 
make the assumption that (R with ring around it) and (R) are equivalent.

                        Harald