Subject: Re: [EAI] Consensus Issue #1: parameter on MAIL FROM?

From: Shawn Steele <[email-masked]>
Date: Wed, 19 Jan 2011 17:37:34 +0000

IMO: An EAI compliant client shouldn't be sending "ASCII" mail, it should be sending "UTF-8" mail (even if it only contains ASCII range characters).  Sure they're the same, but there's no reason the client, or the server, need to care because they're both EAI aware.

*IF* the recipient wants to forward the message, his end can worry about what to do with the encoding at that time.  And if the server wants to send it on to a non-EAI server, then it can deal with it at that time.  Forcing the client to parse and continue to support this information, forever, even after 99% of the servers support EAI, is a waste of effort.

A system of EAI aware servers shouldn't care what range of characters is encoded within a UTF-8 encoded mail.

-Shawn

------------------------------

Message: 4
Date: Tue, 18 Jan 2011 10:25:50 -0800
From: Bill McQuillan <[email-masked]>
Subject: Re: [EAI] Consensus Issue #1:  parameter on MAIL FROM?
To: IMA Discussion <[email-masked]>
Message-ID: <[email-masked]>
Content-Type: text/plain; charset=us-ascii

On Tue, 2011-01-18, John C Klensin wrote:

> --On Tuesday, 18 January, 2011 16:02 +0000 John Levine
> <[email-masked]> wrote:

>>...
>> I suppose that's hypothetically true, but I do not ever want
>> to meet a mail client that sends a UTF-8 message but can't
>> accept UTF-8 in the responses.  For EAI, you're either on the
>> bus or you aren't.
>>...

> And that should be a conformance issue elsewhere in the text.
> More generally, the WG has so far consistently taken the
> position that one is either EAI-conforming (i.e., fully
> i18n-email-capable) or not.  I think that is consistent with the
> view John expresses above.

That's certainly right. But NOT what I was concerned with.

How does an EAI compliant client express that fact when sending a message
that is ASCII only? If it marks the message as UTF8, it may limit it's
propagation unnecessarily, and if the client *doesn't* mark it as UTF8, it
may miss out on some useful responses from the server.

Case 0 - Non-EAI client; Message Header MUST be ASCII only

Case 1 - EAI client; Message Header is ASCII only

Case 2 - EAI client; Message Header is *not* ASCII only

--
Bill McQuillan <[email-masked]>

---

From: John C Klensin <[email-masked]>
Date: Wed, 19 Jan 2011 14:59:19 -0500

--On Wednesday, 19 January, 2011 17:37 +0000 Shawn Steele
<[email-masked]> wrote:

> IMO: An EAI compliant client shouldn't be sending "ASCII"
> mail, it should be sending "UTF-8" mail (even if it only
> contains ASCII range characters).  Sure they're the same, but
> there's no reason the client, or the server, need to care
> because they're both EAI aware.

That is the "announce capability and handshake" story in a
nutshell, at least IMO.

> *IF* the recipient wants to forward the message, his end can
> worry about what to do with the encoding at that time.  And if
> the server wants to send it on to a non-EAI server, then it
> can deal with it at that time.  Forcing the client to parse
> and continue to support this information, forever, even after
> 99% of the servers support EAI, is a waste of effort.

Certainly.   But if one adopts the "this messages needs EAI
capabilities" model, then that recipient (or at least a relay)
doesn't need to parse, it just needs to pass the indicator along
and reject if it hits a server that doesn't advertise EAI
capability.  Note the relationship between this and the
message/global situation: if I want to encapsulate and forward a
message, I can always encapsulate as message/global but can use
message/rfc822 only if I know that the outermost headers do not
require EAI capability.  Whether an EAI-conformant MUA should be
bothering to make that test, or should just retire
message/rfc822 in favor of message/global is a judgment call
that probably gets easier as more EAI-conformant systems deploy.

    john

---

From: Shawn Steele <[email-masked]>
Date: Wed, 19 Jan 2011 23:01:34 +0000

IMO the case that could be fixed by tracking some message state is an edge case.  If I go from one end to the other, encounter an EAI server, and then end up back in a non-EAI environment, then, IMO, something's broken.  Even if my message would work, "real" EAI messages would fail, so there's no point to having an EAI server.  Once that end's fixed, then the "is this message ASCII-only" question becomes irrelevant.  I don't think we should build in this edge case....

-Shawn

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Wednesday, January 19, 2011 11:59 AM
To: Shawn Steele; [email-masked]
Subject: Re: [EAI] Consensus Issue #1: parameter on MAIL FROM?

--On Wednesday, 19 January, 2011 17:37 +0000 Shawn Steele <[email-masked]> wrote:

> mail, it should be sending "UTF-8" mail (even if it only contains 
> ASCII range characters).  Sure they're the same, but there's no reason 
> the client, or the server, need to care because they're both EAI 
> aware.

That is the "announce capability and handshake" story in a nutshell, at least IMO.

> *IF* the recipient wants to forward the message, his end can worry 
> about what to do with the encoding at that time.  And if the server 
> wants to send it on to a non-EAI server, then it can deal with it at 
> that time.  Forcing the client to parse and continue to support this 
> information, forever, even after 99% of the servers support EAI, is a 
> waste of effort.

Certainly.   But if one adopts the "this messages needs EAI
capabilities" model, then that recipient (or at least a relay) doesn't need to parse, it just needs to pass the indicator along and reject if it hits a server that doesn't advertise EAI capability.  Note the relationship between this and the message/global situation: if I want to encapsulate and forward a message, I can always encapsulate as message/global but can use
message/rfc822 only if I know that the outermost headers do not require EAI capability.  Whether an EAI-conformant MUA should be bothering to make that test, or should just retire
message/rfc822 in favor of message/global is a judgment call that probably gets easier as more EAI-conformant systems deploy.

    john

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 20 Jan 2011 09:34:12 +0900

> IMO: An EAI compliant client shouldn't be sending "ASCII" mail, it should be sending "UTF-8" mail (even if it only contains ASCII range characters).  Sure they're the same, but there's no reason th…

I'm not sure about this. When a student of mine implemented the 
experimental version of eai a few years ago on a mail client, we 
realized that some of the counterparts of the user may be eai-capable, 
but others not. But the submission server will be eai-capable anyway.

I think this will be a very frequent situation for quite some while. 
Let's say I have some correspondents in Japan and China where eai gets 
deployed quickly. And I have correspondents in the US or Europe where 
uptake is slower. There is absolutely no need to send all outgoing mail 
as eai and have the outgoing server check everything and backpaddle 
where necessary.

So what we did was to assume eai was okay if there was at least one 
not-ascii-only (side-note: term carefully chosen to get past Dave :-) 
address. We also were looking at using other heuristics, such as "if I 
got an eai mail from somebody, it's okay to send one back" to be a bit 
more agressive with eai deployment/use and to hopefully close gaps where 
both sides are eai-capable but just don't know. But I'm not sure how far 
we got with implementing this.

I think this approach is more prudent because it avoids unnecessary 
transformations of mail in transit (and thus the potential of generating 
garbage or mail not passing, and with that some bad press for eai), and 
reduces server load.

At the very least, such an approach should be possible for clients (as 
far as I understand, it currently is, because the flag is on the MAIL 
command).

Regards,    Martin.

> From: Bill McQuillan<[email-masked]>
> To: IMA Discussion<[email-masked]>
> Message-ID:<[email-masked]>
> Bill McQuillan<[email-masked]>
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>
>

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Shawn Steele <[email-masked]>
Date: Thu, 20 Jan 2011 00:42:21 +0000

I don't mind if individual messages are tagged with "not needing EAI".  (Eg: they don't have to use the MAIL FROM ... flag), however I think the extra state is a bit more complex to manage.

Eg: my client sends a message to my submission server.  Does my client have to declare that it isn't an EAI message?  With all the hand offs, do I "just" remember for each message if it requires EAI or not?  Or do I mark it somehow so I can more easily figure that out?  (Eg: X-MYSOFTWARE-KNOWS-THIS-USES-EAI in the header?)  

(Presumably the header is the "only" interesting thing here, the body should be tagged correctly and work in EAI or non-EAI environments, so we "only" need to have this information if there's UTF-8 in the header).

I agree that the "I'm sending mail, and it'd be nice if it worked downlevel without retrying" scenario worked reasonably well.  I just fear there are too many parts to do that very effectively.  (Unless you added a THIS-MAIL-IS-EAI header, which would sort of defeat the purpose, since I could just as easily test the header for UTF-8 > ASCII code points).

-Shawn

-----Original Message-----
From: "Martin J. Dürst" [mailto:[email-masked]] 
Sent: Poʻakolu, Ianuali 19, 2011 4:34 hours
To: Shawn Steele
Cc: [email-masked]
Subject: Re: [EAI] Consensus Issue #1: parameter on MAIL FROM?

> IMO: An EAI compliant client shouldn't be sending "ASCII" mail, it should be sending "UTF-8" mail (even if it only contains ASCII range characters).  Sure they're the same, but there's no reason th…

I'm not sure about this. When a student of mine implemented the experimental version of eai a few years ago on a mail client, we realized that some of the counterparts of the user may be eai-capable, but others not. But the submission server will be eai-capable anyway.

I think this will be a very frequent situation for quite some while. 
Let's say I have some correspondents in Japan and China where eai gets deployed quickly. And I have correspondents in the US or Europe where uptake is slower. There is absolutely no need to send all outgoing mail as eai and have the outgoing server check everything and backpaddle where necessary.

So what we did was to assume eai was okay if there was at least one not-ascii-only (side-note: term carefully chosen to get past Dave :-) address. We also were looking at using other heuristics, such as "if I got an eai mail from somebody, it's okay to send one back" to be a bit more agressive with eai deployment/use and to hopefully close gaps where both sides are eai-capable but just don't know. But I'm not sure how far we got with implementing this.

I think this approach is more prudent because it avoids unnecessary transformations of mail in transit (and thus the potential of generating garbage or mail not passing, and with that some bad press for eai), and reduces server load.

At the very least, such an approach should be possible for clients (as far as I understand, it currently is, because the flag is on the MAIL command).

Regards,    Martin.

>>> I suppose that's hypothetically true, but I do not ever want to meet 
>>> a mail client that sends a UTF-8 message but can't accept UTF-8 in 
>>> the responses.  For EAI, you're either on the bus or you aren't.
>>> ...
>
>> More generally, the WG has so far consistently taken the position 
>> that one is either EAI-conforming (i.e., fully
>> i18n-email-capable) or not.  I think that is consistent with the view 
>> John expresses above.
>
> How does an EAI compliant client express that fact when sending a 
> message that is ASCII only? If it marks the message as UTF8, it may 
> limit it's propagation unnecessarily, and if the client *doesn't* mark 
> it as UTF8, it may miss out on some useful responses from the server.
>

---

From: John C Klensin <[email-masked]>
Date: Wed, 19 Jan 2011 20:12:48 -0500

--On Thursday, 20 January, 2011 00:42 +0000 Shawn Steele
<[email-masked]> wrote:

>...
> (Presumably the header is the "only" interesting thing here,
> the body should be tagged correctly and work in EAI or non-EAI
> environments, so we "only" need to have this information if
> there's UTF-8 in the header).

Or envelope although that case is presumably trivial.  Note that
it is possible to have an EAI requirement in the envelope but no
such requirement in the headers, but the envelope requirement is
easily determined by parsing.

And it is not "header" but "headers" because one could have a
main header that included "content-type: multipart/mixed" with
no data that was not 5322-compliant and various of the
subsidiary "MIME" headers having fields that would be UTF-8 and
non-MIME compliant without EAI. 

    john