Subject: Re: [EAI] Consensus Result on Issue #1 - Parameter on MAIL FROM

From: "Jiankang YAO" <[email-masked]>
Date: Sat, 29 Jan 2011 17:36:51 +0800

----- Original Message ----- 
From: "Joseph Yee" <[email-masked]>
To: "EAI WG" <[email-masked]>
Sent: Saturday, January 29, 2011 12:40 AM
Subject: [EAI] Consensus Result on Issue #1 - Parameter on MAIL FROM

> All,
> 
> Following responses, this issue reached the consensus to add parameter on MAIL FROM.  The result will be documented at issue tracker (http://trac.tools.ietf.org/wg/eai/trac/report/1).
> 
> Subsequent  proposed questions 
> 1. What is the name of the parameter?
> 2. Does the parameter have value or not?  What kind of value?
> 
>

another question asked may be that "the parameter is to indicate that the SMTP client is EAI-aware or that 
the message sent is EAI message?";

if our intension is that the parameter is to indicate that  the message sent is EAI message,
then it seems that the smtp clients still have to scan the message headers to decide whether the message sent is
EAI or not if the MUA does not indicate it.

Jiankang Yao 

>
> New issues raised during the discussion
> 1. There are 2 different ideas of MAIL FROM interaction between MUA and MTA. 
>    1.1 MUA tells MTA that itself is EAI capable, hence all messages run in "EAI-mode" (MUA always uses the parameter when available)
>    1.2 MUA tells MTA that the following message needs to run under EAI-mode (the next message from MUA may not use the parameter, although both are EAI capable, the interaction and message run under  RFC5321/5322)
> I will start a new thread for this discussion.    
> 
> 
> Please notify me of any errors, misunderstanding, or missed opinions.
> 
> Regards,
> Joseph
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 29 Jan 2011 08:56:04 -0800

> EAI or not if the MUA does not indicate it.

This is a core question that we should answer carefully and clearly.

I think there are two different issues that are actually independent:

    1. Is the client submitting a message that should be treated as UTF-8

       A message might be in UTF-8, even if it contains only ASCII characters.

    2. Can the client handle responses that contain UTF-8?

       Even if the client is submitting a legacy ASCII message, the server might 
need to reply with information containing UTF-8.  This has been mentioned on the 
list, but I don't remember seeing the group come to consensus on whether this 
needs to be supported.

       Note that the server cannot know whether the client supports UTF-8 
replies, unless the client happens to signal that a message is in UTF-8.

I suggest that the parameter on the MAIL command should only mean that the 
message is in UTF-8, and that a separate /explicit/ flag is needed for the 
session, to let the server know that the client can handle UTF-8 in replies.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 29 Jan 2011 11:32:30 -0800

Follow-on thoughts...

> 2. Can the client handle responses that contain UTF-8?

The following presumes working group consensus to have a UTF8 parameter on the 
MAIL command...

The current work really is designed for using UTF-8 in a "pure" UTF-8 
environment.  Although we all know that gatewaying between UTF-8 and the legacy 
ASCII environment is needed, that requirement is being moved for separate 
handling:  The current effort is to define a pure UTF-8 world.

As such, all messages from the client will label submitted messages as being in 
UTF-8.  That is, the MAIL command will declare UTF-8.  And that means that the 
server knows that UTF-8 in replies are acceptable.

For now, we can ignore the possibility that a client might submit a legacy ASCII 
message but the server would want to return a response containing UTF-8.

As for the meaning of the MAIL command parameter:

 >>> Option 1: Any EAI-capable client that is talking with a server
 >>> that supports EAI extensions SHOULD send the parameter with
 >>> MAIL, indicating that it is modern ...
 >>
 >>> Option 2: ... an EAI-capable client talking with an
 >>> EAI-capable server SHOULD send the parameter only if it has
 >>> reason to believe that the message requires EAI support.
...
 >
 > (i) "MUST supply the parameter if the envelope or message
 > requires EAI facilities and SHOULD NOT supply it if the client
 > is EAI-capable but not using EAI facilities"
 >
 > (ii) "MUST supply the parameter if the envelope or message
 > requires EAI facilities and MAY supply it if the client is
 > EAI-capable but not using EAI facilities"

Unfortunately this type of language winds up requiring additional precise 
specification of the meaning of "requires EAI facilities" and "EAI-capable".

A much simpler approach is to define the option in terms of data, declaring that 
all data (commands, responses and message) are in "network" UTF-8.  That is, the 
version of UTF-8 defined in the ABNF specified by the working group.

d/
-- 

   Dave Crocker

---

From: John Levine <[email-masked]>
Date: 29 Jan 2011 20:03:40 -0000

> Even if the client is submitting a legacy ASCII message, the server
> might need to reply with information containing UTF-8.  This has
> been mentioned on the list, but I don't remember seeing the group
> come to consensus on whether this needs to be supported.

If the client hasn't set the EAI flag on MAIL FROM, as far as the
server is concerned, it's a legacy client sending an ASCII message.
If a server can't return an ASCII response, it's not compatible with
legacy clients.  That seems like a really bad idea.

Since EXPN and VRFY have their own flags, the only places I'm aware of
that it might be desirable to have non-ASCII in response text is RCPT
TO responses 251 and 551.  If we're going to do anything about that
tiny case, my suggestion would be non-normative advice to server
operators that if they know where a message should go, just send it
there and respond with 250 instead.

All other response text from MAIL FROM, RCPT TO, and DATA is just
cosmetic, so it doesn't matter.

>I suggest that the parameter on the MAIL command should only mean
>that the message is in UTF-8, and that a separate /explicit/ flag is
>needed for the session, to let the server know that the client can
>handle UTF-8 in replies.

To avoid more incompatibility with legacy clients, please don't.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 29 Jan 2011 14:44:42 -0800

> server is concerned, it's a legacy client sending an ASCII message.

That's correct.

> legacy clients.  That seems like a really bad idea.

What do you mean "a server can't return an ASCII response"?

That would mean that it cannot support legacy traffic.  If that's the case, we 
are far, far outside the IETF standards.

If you mean that the server has UTF-8 data that it wants to include in a 
response, but it won't be able to, then yes that is correct.

What are the actual, or expected, scenarios for which this is a major problem?

What is the basis for believing they will occur with any great frequency?  That 
is, how important are those scenarios to support and how do we know that?

>> needed for the session, to let the server know that the client can

If the client is sending around UTF-8 messages, it has to support UTF-8 
elsewhere in the session.  Anything else would make little sense.

The only issue occurs when a server wants to return UTF-8 data but the client 
has not added the UTF8 parameter to a command.

> To avoid more incompatibility with legacy clients, please don't.

And herein lies the core issue:

      The EAI work tried to maintain compatibility with legacy systems in its 
previous round of effort and it didn't work very well.

      For this round, it is trying to specify operation in a UTF8-clean environment.

As already noted, gatewaying between legacy and UTF8 is certainly important, but 
it needs to be specified separately.  Including it within the current work has 
apparently -- and not surprisingly -- consistently created confusion.  (As I 
fear it is doing now.)

Here's the overall system architecture, showing a number of different 
interoperability scenarios:

  +-----------+                                      +-----------+
  | MUA-ASCII |                                      | MUA-ASCII |
  +----+------+                                      +-----------+
       |                                                  ^
       v                                                  |
  +-----------+    +-----------+    +-----------+    +----+------+
  | MSA-ASCII +--->| MTA-ASCII +--->| MTA-ASCII +--->| MDA-ASCII |
  +-----------+ ^  +-----------+ ^  +-----------+ ^  +-----------+
                |                |                |
                |                |                |
                |                |                |
  +-----------+ V  +-----------+ V  +-----------+ V  +-----------+
  | MSA-UTF8  |--->| MTA-UTF8  |--->| MTA-UTF8  |--->| MDA-UTF8
  +-----------+    +-----------+    +-----------+    +----+------+
       ^                                                  |
       |                                                  V
  +-----------+                                      +-----------+
  | MUA-UTF8  |                                      | MUA-UTF8  |
  +-----------+                                      +-----------+

(This concerns SMTP and the RFC5322 header, not the MIME mechanisms for 
supporting UTF-8.)

The diagram shows an ASCII-clean service on top, with a UTF8-clean service 
below, and 3 points of gatewaying between them.  Gatewaying might be in either 
direction.

       (Note that it is entirely possible for a single piece of software to 
support two or more of the functions shown above.  But the chart is about 
network architecture, not software implementation.)

The legacy service supports only MSA-ASCII/MTA-ASCII, MTA-ASCII/MTA-ASCII, and 
MTA-ASCII/MDA-ASCII interactions.

The current specification is attempting ONLY to specify the MSA-UTF8/MTA-UTF8, 
MTA-UTF8/MTA-UTF8, and MTA-UTF8/MDA-UTF8 interactions.

Additional effort -- in a separate specification -- is required to cover the 
/six/ gatewaying activities -- one for each direction, at each point of gatewaying.

d/
-- 

   Dave Crocker

---

From: "John R Levine" <[email-masked]>
Date: 29 Jan 2011 18:19:35 -0500

>>> needed for the session, to let the server know that the client can

This suggests that there would be situtations where the server want to 
send UTF-8 replies, but the client couldn't handle them, which I think we 
agree could break legacy mail.

If you meant it the other way around, that a client can send UTF-8 
messages with UTF-8 addresses in the envelope, but not handle UTF-8 text 
in the SMTP responses, the right answer is still Don't Do That.

Again, I'm trying to think of a plausible scenario in which that would be 
useful.  Someone made all the changes for UTF-8 message headers, 
message/global bodies, and UTF-8 envelopes, but it was too hard to handle 
UTF-8 in the SMTP response comments?  I don't get it.

If the question is about relaying the text of the response back to a 
hypothetical prior link that doesn't do UTF-8, I don't see any reason to 
add extra complication to make that the server's problem rather than the 
client's.  Maybe the server has alternate text available, maybe it 
doesn't.  A client that needs to downcode has a variety of options without 
any help from the server, such as substituting a generic message 
appropriate for the response code.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: "Jiankang YAO" <[email-masked]>
Date: Sun, 30 Jan 2011 10:19:57 +0800

----- Original Message ----- 
From: "Dave CROCKER" <[email-masked]>
To: "Jiankang YAO" <[email-masked]>
Cc: "Joseph Yee" <[email-masked]>; "EAI WG" <[email-masked]>
Sent: Sunday, January 30, 2011 12:56 AM
Subject: Re: [EAI] Consensus Result on Issue #1 - Parameter on MAIL FROM

...
> session, to let the server know that the client can handle UTF-8 in replies.

Actually,  a separate /explicit/ flag is not necessary. we presume that we just need to use parameter to mean whether the message sent is eai or not.
if the smtp clients/servers support the parameter specified in rfc5336bis,  it indicates that they support EAI.

so for example,

if mail from [email-masked]  header=eai, it indicates that the client supports EAI;

if mail from [email-masked]  header=ascii, it indicates that the client supports EAI too;

any server/client which does not support rfc5336bis, will not support this parameter.
only the eai-aware server/client can set this parameter.

Jiankang Yao

---

From: Bill McQuillan <[email-masked]>
Date: Sat, 29 Jan 2011 18:46:57 -0800

> only the eai-aware server/client can set this parameter.

This syntax would certainly distinguish among the three cases we're talking
about:

  1-Legacy Client, 
  2-EAI Client & ASCII Message, 
  3-EAI Client & EAI Message.

And I have already recommended something like this.

However, if the desire is *only* to distinguish between a Legacy Client & an
EAI Client without saying anything about each message, that is an attribute
of the *session* and replicating the flag on every MAIL FROM: is bizarre!

Unless, of course, we wish to accommodate clients that mutate their
abilities during the course of a session. :-)

-- 
Bill McQuillan <[email-masked]>

---

From: Claudio Allocchio <[email-masked]>
Date: Sun, 30 Jan 2011 10:25:37 +0100 (CET)

+1 on all.

(see some more below)

On Sat, 29 Jan 2011, Dave CROCKER wrote:

> That would mean that it cannot support legacy traffic.  If that's the case, 
> we are far, far outside the IETF standards.
>
> What is the basis for believing they will occur with any great frequency? 
> That is, how important are those scenarios to support and how do we know 
> that?
>
>

correct. This UTF8 scenario is an upgrade which cannot be compatible with 
the previous version, e.g. we cannot in any case expect that legacy e-mail 
will be able to transport "without any kind of getewaying action" UTF8 
mail.

>
> As already noted, gatewaying between legacy and UTF8 is certainly important, 
> but it needs to be specified separately.  Including it within the current 
> work has apparently -- and not surprisingly -- consistently created 
> confusion.  (As I fear it is doing now.)

many +1 !

As we also need (as discussed on the ML) a more detailed document 
(referenced from the current specifications) which dscribes gatewaying 
issues, both with legacy e-mail and other (non IETF) mail standards, there 
is a very good reason for keeping this separate, too.

> interoperability scenarios:

       and here is the gateway system....(with its specification)

> below, and 3 points of gatewaying between them.  Gatewaying might be in 
> either direction.
>
> The legacy service supports only MSA-ASCII/MTA-ASCII, MTA-ASCII/MTA-ASCII, 
> and MTA-ASCII/MDA-ASCII interactions.
>
> The current specification is attempting ONLY to specify the 
> MSA-UTF8/MTA-UTF8, MTA-UTF8/MTA-UTF8, and MTA-UTF8/MDA-UTF8 interactions.
>

and this is just for THIS kindof gateway (the most important and common 
one, of course).

> d/

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca