Subject: Re: [EAI] "configuration error"

From: Shawn Steele <[email-masked]>
Date: Tue, 10 Aug 2010 18:21:24 +0000

Do we need all the stuff about the experiments?  Changed the first few sentences to address Fujiwara's point?

   A somewhat more complex situation arises when the first-hop system
   supports these extensions but some subsequent server in the SMTP
   transmission chain does not.  It is important to note that some cases
   of that situation can be resolved by changing the configuration,
   especially if the last hop hosts non-ASCII addresses.  A final delivery MTA that
   accepts these extensions should not be configured with lower-
   preference MX hosts that do not.  While the experiments that preceded
   these specifications included a mechanism for passing backup ASCII
   addresses to intermediate relay systems and having those systems
   alter the relevant message header fields and substitute the addresses
   [RFC5504], the requirements and long-term implications of that system
   proved too complex to be satisfactory.  Consequently, if an
   intermediate SMTP relay that is transmitting a message that requires
   these extensions and discovers that the next system in the chain does
   not support them, it will have little choice other than to reject or
   return the message.

------------------------------------------
Or, to completely vandalize the text:
(Moved more of the overview in general stuff to 7 and the stage specific stuff to the right sub item, it's not as different as I imply).

7.  Downgrading before and after SMTP Transactions
   An important issue with these extensions is how to handle
   interactions between systems that support non-ASCII addresses and
   legacy systems that expect ASCII.  There is, of course, no problem
   with ASCII-only systems sending to those that can handle
   internationalized forms because the ASCII forms are just a proper
   subset.  But, when systems that support these extensions send mail,
   they may include non-ASCII addresses for senders, receivers, or both    
   and might also provide non-ASCII header information other than
   addresses.

   Of course, an transformation from non-ASCII to ASCII-only
   imply that the originating user or system must have ASCII-only
   addresses available for all senders and recipients.  Mechanisms by
   which such addresses may be found or identified are outside the scope
   of these specifications as are decisions about the design of
   originating systems such as whether any required transformations are
   made by the user, the originating MUA, or the Submission server.

   While the experiments that preceded
   these specifications included a mechanism for passing backup ASCII
   addresses to intermediate relay systems and having those systems
   alter the relevant message header fields and substitute the addresses
   [RFC5504], the requirements and long-term implications of that system
   proved too complex to be satisfactory.  

   Consequently, if the extension is not supported by the first-hop system
   (SMTP server accessed by the Submission server acting as an SMTP
   client), message originating systems should be prepared to either
   send conventional envelopes and message headers or to return the
   message to the originating user so the message may be manually
   downgraded to the traditional form, possibly using encoded words
   [RFC2047] in the message headers.  

    As discussed above, downgrading to an ASCII-only form may occur
   before or during the initial message submission.  It might also occur
   after the delivery to the final delivery MTA in order to accommodate
   messages stores or IMAP or POP servers or clients that have different
   capabilities than the delivery MTA.  These two cases are discussed in
   the subsections below.

7.1.  Downgrading before or during Message Submission

   Perhaps obviously, the most convenient time to find an ASCII address
   corresponding to an internationalized address is at the originating
   MUA.  This can occur either before the message is sent or after the
   internationalized form of the message is rejected.  It is also the
   most convenient time to convert a message from the internationalized
   form into conventional ASCII form or to generate a non-delivery
   message to the sender if either is necessary.  

   At that point, the
   user has a full range of choices available, including contacting the
   intended recipient out of band for an alternate address, consulting
   appropriate directories, arranging for translation of both addresses
   and message content into a different language, and so on.  While it
   is natural to think of message downgrading as optimally being a
   fully-automated process, we should not underestimate the capabilities
   of a user of at least moderate intelligence who wishes to communicate
   with another such user.

   In this context, one can easily imagine modifications to message
   submission servers (as described in [RFC4409]) so that they would
   perform downgrading, or perhaps even upgrading, operations, receiving
   messages with one or more of the internationalization extensions
   discussed here and adapting the outgoing message, as needed, to
   respond to the delivery or next-hop environment it encounters.

7.2.  Downgrading or Other Processing After Final SMTP Delivery

   When an email message is received by a final delivery MTA, it is
   usually stored in some form.  Then it is retrieved either by software
   that reads the stored form directly or by client software via some
   email retrieval mechanisms such as POP or IMAP.

   The SMTP extension described in Section 6.1 provides protection only
   in transport.  It does not prevent MUAs and email retrieval
   mechanisms that have not been upgraded to understand
   internationalized addresses and UTF-8 message headers from accessing
   stored internationalized emails.

   Since the final delivery MTA (or, to be more specific, its
   corresponding mail storage agent) cannot safely assume that agents
   accessing email storage will always be capable of handling the
   extensions proposed here, it MAY either downgrade internationalized
   emails or specially identify messages that utilize these extensions,
   or both.  If this is done, the final delivery MTA SHOULD include a
   mechanism to preserve or recover the original internationalized forms
   without information loss to support access by UTF8SMTPbis-aware
   agents.

8.  Downgrading in Transit

   A somewhat more complex situation arises when the first-hop system
   supports these extensions but some subsequent server in the SMTP
   transmission chain does not.  As the experimental RFCs demonstrated,
   if an
   intermediate SMTP relay that is transmitting a message that requires
   these extensions and discovers that the next system in the chain does
   not support them, it will have little choice other than to reject or
   return the message.
 
   It is important to note that some in-Transit cases can be resolved by
   changing the configuration,
   especially if the last hop hosts non-ASCII addresses.  A final delivery
   MTA that accepts these extensions should not be configured with
   lower-preference MX hosts that do not.  Similarly spam filtering or
   firewall servers for non-ASCII supporting mail hosts should also
   be non-ASCII email aware.

(The following sounds like a soap-box.  I know where you're coming from, but we already said there's "little choice other than to reject or return the message", so can we say "why" in less than 3 paragraphs?  If it really needs 3 paragraphs, I'm fine with that, but it seems like a bit of noise to someone trying to understand the core RFC, more like something I'd expect in an implementation note or something).

   The base SMTP specification (Section 2.3.11 of RFC 5321 [RFC5321])
   states that "due to a long history of problems when intermediate
   hosts have attempted to optimize transport by modifying them, the
   local-part MUST be interpreted and assigned semantics only by the
   host specified in the domain part of the address".  This is not a new
   requirement; equivalent statements appeared in specifications in 2001
   [RFC2821] and even in 1989 [RFC1123].

   Adherence to this rule means that a downgrade mechanism that
   transforms the local-part of an email address cannot be done in
   transit.  It can only be done at the endpoints, namely by the MUA or
   submission server or by the final delivery MTA.

   One of the reasons for this rule has to do with legacy email systems
   that use source routing in the local-part of the address field.
   Transforming the email address destroys such routing information.
   There is no way a server other than the final delivery server can
   know, for example, whether the local-part of [email-masked] is
   a route ("user" is reached via "foo") or simply a local address.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 Aug 2010 15:18:30 -0400

--On Tuesday, August 10, 2010 18:21 +0000 Shawn Steele
<[email-masked]> wrote:

> Do we need all the stuff about the experiments?

IMO, yes.  The alternative is to generate a separate document
describing the experiments and their results.  That would be
even more work.

>...
>    One of the reasons for this rule has to do with legacy
> email systems    that use source routing in the local-part of
> the address field.    Transforming the email address destroys
> such routing information.    There is no way a server other
> than the final delivery server can    know, for example,
> whether the local-part of [email-masked] is    a route
> ("user" is reached via "foo") or simply a local address.

Huh?  The use of actual source routing in the local part has
been prohibited since RFC 1123 in 1989.  Weren't you arguing
against getting into too many of the details of legacy SMTP?  

More after I've had time to study the rest of your recommended
text.

   john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 10 Aug 2010 19:28:52 +0000

> IMO, yes.  The alternative is to generate a separate document describing the experiments and their results.  That would be even more work.

I moved them "up" to the 7 to make more structural sense to me.  (Mostly I just moved things to make the structure easier for me to follow).

> Huh?  The use of actual source routing in the local part has been prohibited since RFC 1123 in 1989.  Weren't you arguing against getting into too many of the details of legacy SMTP?  

I don't like the last 3 paragraphs (that's what I meant my the (comment)).  I'm happy without them, but I thought you added/wanted them :)  I'd be happy with just the middle paragraph, or none of them, something like:

   Legacy SMTP specifies that the local part is to be interpreted only by the host specified in the domain part of the address.  Adherence to this rule means that a downgrade mechanism that
   transforms the local-part of an email address cannot be done in
   transit.  It can only be done at the endpoints, namely by the MUA or
   submission server or by the final delivery MTA.

As mentioned before, I was just trying to suggest words since people were bringing up issues and asked for words.  I could live with the existing version.

-Shawn

---

From: Shawn Steele <[email-masked]>
Date: Tue, 10 Aug 2010 19:32:15 +0000

> IMO, yes.  The alternative is to generate a separate document describing the experiments and their results.  That would be even more work.

(to be clear, I'm happy with that.  I'm also not suggesting that changes are needed, however if someone does change stuff about experiments, it is easier for me to read if those results are up-front rather than in the middle of the other stuff.  Eg:  Keep the "this is what we want to do" and "this is why" and "this is the experiment that didn't work" clear enough that the reader doesn't confuse the "what we want to do" with the "this didn't work" stuff).

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 Aug 2010 15:41:51 -0400

--On Tuesday, August 10, 2010 19:32 +0000 Shawn Steele
<[email-masked]> wrote:

> (to be clear, I'm happy with that.  I'm also not suggesting
> that changes are needed, however if someone does change stuff
> about experiments, it is easier for me to read if those
> results are up-front rather than in the middle of the other
> stuff.  Eg:  Keep the "this is what we want to do" and "this
> is why" and "this is the experiment that didn't work" clear
> enough that the reader doesn't confuse the "what we want to
> do" with the "this didn't work" stuff).

Ack.

   john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 10 Aug 2010 20:05:16 +0000

As I said I don't think changes are necessary.

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Tuesday, August 10, 2010 12:42 PM
To: Shawn Steele; [email-masked]
Subject: RE: [EAI] "configuration error" (4952bis)

--On Tuesday, August 10, 2010 19:32 +0000 Shawn Steele <[email-masked]> wrote:

>> describing the experiments and their results.  That would be even 
>> more work.
> 
> (to be clear, I'm happy with that.  I'm also not suggesting that 
> changes are needed, however if someone does change stuff about 
> experiments, it is easier for me to read if those results are up-front 
> rather than in the middle of the other stuff.  Eg:  Keep the "this is 
> what we want to do" and "this is why" and "this is the experiment that 
> didn't work" clear enough that the reader doesn't confuse the "what we 
> want to do" with the "this didn't work" stuff).

Ack.

   john