Subject: Re: [EAI] Consensus Result on Issue #1 - Parameter on MAIL FROM

From: Shawn Steele <[email-masked]>
Date: Sat, 29 Jan 2011 21:22:07 +0000

I think that defining the flag as "UTF8 is enabled" is sufficient.

If the client can send UTF-8, then the client can handle UTF8 responses.  If UTF8 is permitted, it's sort of irrelevent if the message happens to be all-ASCII, though it may be convenient to know that additional information.

So I'd like to see the flag just mean "UTF8 is enabled."  

Clearly if the sender knows the message is all-ascii, it could avoid the flag if it wanted to.

-Shawn

------------------------------

Message: 4
Date: Sat, 29 Jan 2011 11:32:30 -0800
From: Dave CROCKER <[email-masked]>
Subject: Re: [EAI] Consensus Result on Issue #1 - Parameter on MAIL
        FROM
To: EAI WG <[email-masked]>
Message-ID: <[email-masked]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Follow-on thoughts...

On 1/29/2011 8:56 AM, Dave CROCKER wrote:
> 1. Is the client submitting a message that should be treated as UTF-8
>
> A message might be in UTF-8, even if it contains only ASCII characters.
>
> 2. Can the client handle responses that contain UTF-8?

The following presumes working group consensus to have a UTF8 parameter on the
MAIL command...

The current work really is designed for using UTF-8 in a "pure" UTF-8
environment.  Although we all know that gatewaying between UTF-8 and the legacy
ASCII environment is needed, that requirement is being moved for separate
handling:  The current effort is to define a pure UTF-8 world.

As such, all messages from the client will label submitted messages as being in
UTF-8.  That is, the MAIL command will declare UTF-8.  And that means that the
server knows that UTF-8 in replies are acceptable.

For now, we can ignore the possibility that a client might submit a legacy ASCII
message but the server would want to return a response containing UTF-8.

As for the meaning of the MAIL command parameter:

 >>> Option 1: Any EAI-capable client that is talking with a server
 >>> that supports EAI extensions SHOULD send the parameter with
 >>> MAIL, indicating that it is modern ...
 >>
 >>> Option 2: ... an EAI-capable client talking with an
 >>> EAI-capable server SHOULD send the parameter only if it has
 >>> reason to believe that the message requires EAI support.
...
 >
 > (i) "MUST supply the parameter if the envelope or message
 > requires EAI facilities and SHOULD NOT supply it if the client
 > is EAI-capable but not using EAI facilities"
 >
 > (ii) "MUST supply the parameter if the envelope or message
 > requires EAI facilities and MAY supply it if the client is
 > EAI-capable but not using EAI facilities"

Unfortunately this type of language winds up requiring additional precise
specification of the meaning of "requires EAI facilities" and "EAI-capable".

A much simpler approach is to define the option in terms of data, declaring that
all data (commands, responses and message) are in "network" UTF-8.  That is, the
version of UTF-8 defined in the ABNF specified by the working group.

d/
--

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

------------------------------

_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

End of IMA Digest, Vol 66, Issue 40
***********************************

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 29 Jan 2011 14:06:50 -0800

On 1/29/2011 1:22 PM, Shawn Steele wrote:
> If the client can send UTF-8, then the client can handle UTF8 responses.  If
> UTF8 is permitted, it's sort of irrelevent if the message happens to be
> all-ASCII, though it may be convenient to know that additional information.

Except that it's not irrelevant.

If the message is actually a UTF8 message, then the SMTP receiver is going to 
have to turn around and require support for UTF8 on the next hop.

It needs to know whether that is required.

So "enabled" is not sufficient.  In fact, that's why the option needs to be used 
to label the character encoding of the message.

The semantics need to mean that the message is actually UTF-8 and that it 
requires a UTF-8 path through to delivery.

> Clearly if the sender knows the message is all-ascii, it could avoid the flag

That's too casual.

If the message is an ASCII message -- as opposed to a UTF-8 message that only 
consumed x07F and below -- then the UTF8 option MUST NOT be used.

However if the message is UTF-8 -- even if it happens to consume only  x07F and 
below -- then the option MUST be used.

d/
-- 

   Dave Crocker

---

From: John Levine <[email-masked]>
Date: 30 Jan 2011 00:32:43 -0000

>However if the message is UTF-8 -- even if it happens to consume only
>x07F and below -- then the option MUST be used.

Hi.  King Canute here.

For the benefit of all the MUA and MTA implemeters, can you explain in
as few words as possible what the difference is between an ASCII
message and a UTF-8 message where all the bytes are ASCII, and why
it's important for them to maintain the distinction between the two?
(I think I understand the difference, but I don't think I can explain
it so as to make it sound like it's worth caring about.)

It seems to me that no matter what we say, they're going to use simple
hacks like scanning the headers and envelope to see if there's
anything with the high bit set, or maybe even at setup time asking
what the user's preferred character set is, and turning on the EAI
flag if it's anything other than ASCII.

At this point we can guess based on our limited experience what is
likely to maximize backwards compatibility.  But I expect that as soon
as one or two of the big webmail systems support EAI, which is likely
to be soon since they all operate all over the world, the normal
answer to any question that includes words like downcode will be "just
get a Gmail account and read your mail there."

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Sun, 30 Jan 2011 09:58:33 -0500

--On Saturday, January 29, 2011 2:06 PM -0800 Dave CROCKER
<[email-masked]> wrote:

>...
> If the message is an ASCII message -- as opposed to a UTF-8
> message that only consumed x07F and below -- then the UTF8
> option MUST NOT be used.
> 
> However if the message is UTF-8 -- even if it happens to
> consume only  x07F and below -- then the option MUST be used.

Dave,

With the understanding that I personally favor sending a MAIL
command announcement only if EAI capabilities are required,
please explain what the second paragraph quoted above means.
If I were taking the position I understand Shawn to be
advocating, I would say "well, the client is EAI-capable, there
is no way to distinguish between an ASCII message and a message
that is UTF-8 but only contains characters in the ASCII code
point range ('x07F and below') than I should send the parameter
in all cases".

    john

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 30 Jan 2011 08:19:13 -0800

> point range ('x07F and below') than I should send the parameter

"should"?  under what circumstances would it be acceptable not to?

d/
-- 

   Dave Crocker

---

From: Shawn Steele <[email-masked]>
Date: Sun, 30 Jan 2011 18:25:15 +0000

IMO this cannot happen consistently enough to trust the values, however I welcome your response to John's question.

ASCII messages and UTF-8 messages in the <=0x7f are identical and should be allowed to be treated as such.

Certainly, I'm typing a message now.  It could be transmitted in either ASCII or UTF-8 (Since I removed the Klingon from my signature).  The client has no clue whether the user intends ASCII or UTF-8, and it is irrelevent whether this message is sent in ASCII or UTF-8.  (Presumably it'll use whatever defaults OWA's giving me).  If this messege is intended as ASCII and sent with legacy SMTP, it'll behave.  If it's intended as UTF-8 and sent with legacy SMTP, it'll also behave.  Ditto for an EAI aware system.  So there's no need to care whether this message is ASCII or UTF-8.

I do agree that some of the current drafts could be simpler by have "ASCII encoded" (legacy) messages or "UTF-8 encoded" (EAI) messages, however, in practice, it really doesn't matter if someone uses the UTF8 encoding for an ASCII encoded message.  That's part (probably most) of why UTF-8 was designed as a superset.

If you're worried about "ASCII encoded" that happens to have data with the 8th bit set, well, you're already in a broken place.  That's illegal, there's no such thing as 8-bit ASCII with the 8th bit set, it must always be zero.  That leads to the kind of ambiguity we have today when mail is sent with system code pages.  If such mail was sent as UTF-8, it must fail, which is great.

-Shawn

http://blogs.msdn.com/shawnste

________________________________________
From: John C Klensin [[email-masked]]
Sent: Sunday, January 30, 2011 6:58 AM
To: [email-masked]; Shawn Steele
Cc: [email-masked]
Subject: Re: [EAI] Consensus Result on Issue #1 - Parameter on MAIL FROM

Dave,

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 31 Jan 2011 11:35:33 -0000

> flag if it's anything other than ASCII.

We are agreed that we DON'T want servers en route to be doing that.

But clients and submission servers MIGHT do that (or get the information  
some other way). We don't insist that they do, nor do we insist that they  
don't. So at most it is a "MAY" - or rather it is "MAY include the UTF8  
parameter on a pure ASCII message", but coupled with a NOTE (text already  
discussed) warning of the downside of doing so.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 31 Jan 2011 11:39:57 -0000

On Sun, 30 Jan 2011 18:25:15 -0000, Shawn Steele  
<[email-masked]> wrote:

> I do agree that some of the current drafts could be simpler by have  
> "ASCII encoded" (legacy) messages or "UTF-8 encoded" (EAI) messages,  
> however, in practice, it really doesn't matter if someone uses the UTF8  
> encoding for an ASCII encoded message.  That's part (probably most) of  
> why UTF-8 was designed as a superset.

Yes it DOES matter. Because the message would then be forwarded ONLY over  
routes that advertise UTF8SMTPbis, and there might not exist such a route  
to the intended destination, in which case it would bounce.

---

From: [email-masked]
Date: Mon, 31 Jan 2011 14:03:09 -0800 (PST)

> to the intended destination, in which case it would bounce.

Exactly right. A client that marks non-EAI messages as being EAI messages is
going to see those message bounce when they attempt to cross over into the
non-EAI-extended world - which right now is "everywhere".

And to those who think this is going to deploy so quickly that this won't be an
issue... dream on. Although John is quite correct when he says that we suck at
predicting how quickly things will deploy, the notion that this will happen
overnight flies in the face of every bit of experience we have from having
spent the past three decades updating email infrastructure. The best you can
hope for - and I think this is still completely unrealistic - is a transition
period of several years.

OTOH, we tried something similar to this in MIME when we said that clients
SHOULD "minimize" use of more complex charsets, e.g., don't use iso-8859-1 when
US-ASCII suffices. Some clients followed this recommendation but a lot did not,
and the users of those clients got a markedly inferior user experience as a
result.

Additionally, handling this really well at the client level is not as simple as
keeping track of EAI usage with a flag. Consider the case where you have
multiple recipients for a message, some with EAI addresses and others not, and
for whatever reason with the recipient addresses not appearing in the header. A
straightforward client would send a single message copy to all recipients with
the EAI flag set. But a more sophisticated client might notice that it can send
a non-EAI message to the non-EAI recipients without any content loss. And if
those recipients are in the non-EAI-extended world, this may be the only
approach that works in thie case.

So at its heart this is really a client quality issue. Absent near-universal
EAI-deployment, clients that take the trouble to check and only label actual
EAI messages as EAI will provide a superior user experience to ones that
blindly label everything they send as EAI. And those that don't will provide
gainful employnment for lots of call center staff.

				Ned