Subject: Re: [EAI] UTF-8/MIME

From: Shawn Steele <[email-masked]>
Date: Wed, 18 Aug 2010 18:31:52 +0000

> my understanding is that headers are in utf-8,
> the body part might still use mime(gb2312).

I'm obviously really confused.  My understanding was that EAI required UTF-8 headers and MIME encoded bodies, which should also be UTF-8, but an appropriate MIME part.  Presumably the body could also be GB2312 MIME, though I'd discourage that as much as possible.

So if the sentence in the RFC ever applied, it would have to be in the absence of MIME, which sort of means it's already not an EAI message.  So a non-MIME message with UTF-8 headers is an anomaly, and I'd sort of expect the body to have the same encoding.  Indeed, if I find non-MIME messages now with GB2312 bodies, then likely the header is also GB2312, not UTF-8.  John will be quick to point out that's not permitted by the standards, but I've heard anecdotal stories of it working, at least in controlled environments.  Clearly "older" mail clients often sent non-MIME mail with 8 bit bodies using whatever the local code page was.  Hopefully any mail client updated to handle EAI addresses would be smart enough to at least use UTF-8 and tag it properly.

So: If the intent was to say "a non-MIME message with EAI headers should...", then I think the language isn't needed.  Non-MIME is clearly non-EAI so out of scope for this document.  If the intent was "a non-EAI message that happens to not be ASCII should....", then it's probably also not needed.  The only reason I could see to keep it would be if we intended to explicitly add support for UTF-8 mail in absence of any EAI/MIME markup, which I understand the WG doesn't want, and would still preclude your GB2312 case.

Is your intent to enable a GB2312 mail client to use non-ASCII addresses without the rest of EAI/MIME?  Although (John will shudder) I can see why a client/server might want to privately recognize that convention, I don't see how you can standardize it since code pages vary depending on the environment.  The only possible mechanism that would interoperate outside of a limited environment would be to use something like UTF-8 that is recognizable everywhere.

I think I'm still confused.

- Shawn

 
http://blogs.msdn.com/shawnste
(Selfhost 7799)

----- Original Message ----- 
From: "Shawn Steele" <[email-masked]>
To: "Jiankang YAO" <[email-masked]>; "John C Klensin" <[email-masked]>; <[email-masked]>
Sent: Wednesday, August 18, 2010 4:41 PM
Subject: RE: [EAI] rfc5336bis

> The current text would forbid GB2312 or any other local code page.  (It says it has to be ASCII).
> 
> For those other clients the headers would also likely be a local code page & not UTF-8.
> 
> -Shawn
> 
> ???? ?????
> http://blogs.msdn.com/shawnste
> 
> 
> ________________________________________
> From: Jiankang YAO [[email-masked]]
> Sent: Tuesday, August 17, 2010 8:55 PM
> To: Shawn Steele; John C Klensin; [email-masked]
> Subject: Re: [EAI] rfc5336bis
> 
> I prefer that we still keep the current text.
> 
> besides the reason mentioned by John,  if we allow that the body can be
> non-ascii data(utf-8) if there are no body parameters,  it is possible that the body may be other 8bit encoding standard such as
> chinese code standard GB2312.
> many MUAs sends the message with the local encoding.
> 
> 
> 
> ----- Original Message -----
> From: "Shawn Steele" <[email-masked]>
> To: "John C Klensin" <[email-masked]>; "Jiankang YAO" <[email-masked]>; <[email-masked]>
> Sent: Wednesday, August 18, 2010 7:41 AM
> Subject: RE: [EAI] rfc5336bis
> 
> 
> That's sort of what I was concluding as well.  Either the message is MIME, so the headers are MIME and the BODY then should also be MIME.  (So the case can't ever happen), or the MIME requirement was ignored, so raw UTF-8 headers shouldn't be happening, so the rest is also irrelevant.  (I don't mind assuming UTF-8 for non-MIME and crossing my fingers, but I realize that isn't a popular opinion).
> 
> Can we just strike that reference then?  Since it sounds like it's just saying "use MIME like we said earlier?"
> 
> -Shawn
> 
> -----Original Message-----
> From: John C Klensin [mailto:[email-masked]]
> Sent: Tuesday, August 17, 2010 4:38 PM
> To: Shawn Steele; Jiankang YAO; [email-masked]
> Subject: RE: [EAI] rfc5336bis
> 
> 
> 
> --On Tuesday, August 17, 2010 22:43 +0000 Shawn Steele <[email-masked]> wrote:
> 
>> That's sort of my point.  What I read is that "if headers are UTF-8,
>> but there's no MIME, then BODY is ASCII".  If there's no MIME, headers
>> shouldn't be in UTF-8, so we shouldn't even get here.  If there's no
>> MIME, but headers are UTF-8 and we want to recognize that, then I'd
>> think the only logical conclusion would be that the body is also
>> UTF-8.
>>
>> Maybe I completely misread what the original text meant.
> 
> I think we can't have UTF-8 headers (or any other piece of
> UTF8SMTPbis) without MIME:
> 
> Reasons:
> 
> (1) There is no reason for not requiring MIME in this
> day and age.
> 
> (2) UTF8SMTP[bis] requires 8BITMIME in order to make the
> addresses and headers work.  And 8BITMIME absolutely
> requires MIME.
> 
> (3) Absent MIME, we are back to RFC 822 (or 5322 if you
> prefer) and, for it, the only valid headers are ASCII.
> Absent MIME, there are no encoded words either.
> 
> Put differently, UTF-8 headers without MIME don't imply some sort of default UTF-8 body.  That case implies "seriously broken message" with no other options.
> 
> So I don't know what the original text meant either, but I'm
> pretty sure it leads into a dead end.   Or worse.
> 
>   john

------------------------------

Message: 5
Date: Wed, 18 Aug 2010 09:59:44 -0400
From: Andrew Sullivan <[email-masked]>
Subject: Re: [EAI] proposal - can we reach consensus? - text	for
	ascii@non-ascii issues
To: [email-masked]
Message-ID: <[email-masked]>
Content-Type: text/plain; charset=us-ascii

On Tue, Aug 17, 2010 at 11:25:19PM +0000, Shawn Steele wrote:
> John suggested:
> > "this specification applies only when a UTF8SMTPbis-aware client is trying to send a message that requires the UTF8SMTPbis extensions to a server that advertises the ability to handle those extensions as specified here.  For all other cases, and for addresses and messages that do not require the extension, SMTP clients and servers are expected to behave exactly as specified in RFC 5321 and its successors." and then drop all of this material.
> 
> Can we reach a consensus that something like the above in the introductory material would address people's concerns with UTF8SMTP and non-UTF8SMTP interoperability and then drop the text that is being hard to pin down?  Does anyone strongly object to this idea?  (I won't hold it against you?)

I prefer it to attempting to cover all the cases that someone might do
if they're not conforming to the RFCs we're working on.  To me,
attempting to list all the ways in which someone can do something
_other_ than the standard behaviour will just lead people to interpret
those descriptions as the right thing to do sometimes. 

A

-- 
Andrew Sullivan
[email-masked]

------------------------------

Message: 6
Date: Wed, 18 Aug 2010 08:56:26 -0600
From: Peter Saint-Andre <[email-masked]>
Subject: Re: [EAI] proposal - can we reach consensus? - text	for
	ascii@non-ascii issues
To: [email-masked]
Message-ID: <[email-masked]>
Content-Type: text/plain; charset=UTF-8

On 8/18/10 7:59 AM, Andrew Sullivan wrote:
> On Tue, Aug 17, 2010 at 11:25:19PM +0000, Shawn Steele wrote:
>> John suggested:
>>> "this specification applies only when a UTF8SMTPbis-aware client
>>> is trying to send a message that requires the UTF8SMTPbis
>>> extensions to a server that advertises the ability to handle
>>> those extensions as specified here.  For all other cases, and for
>>> addresses and messages that do not require the extension, SMTP
>>> clients and servers are expected to behave exactly as specified
>>> in RFC 5321 and its successors." and then drop all of this
>>> material.
>> 
>> Can we reach a consensus that something like the above in the
>> introductory material would address people's concerns with UTF8SMTP
>> and non-UTF8SMTP interoperability and then drop the text that is
>> being hard to pin down?  Does anyone strongly object to this idea?
>> (I won't hold it against you?)
> 
> I prefer it to attempting to cover all the cases that someone might
> do if they're not conforming to the RFCs we're working on.  To me, 
> attempting to list all the ways in which someone can do something 
> _other_ than the standard behaviour will just lead people to
> interpret those descriptions as the right thing to do sometimes.

Agreed.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

------------------------------

_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

End of IMA Digest, Vol 61, Issue 16
***********************************

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 19 Aug 2010 14:43:38 +0100

On Wed, 18 Aug 2010 19:31:52 +0100, Shawn Steele  
<[email-masked]> wrote:

> I'm obviously really confused.  My understanding was that EAI required  
> UTF-8 headers and MIME encoded bodies, which should also be UTF-8, but  
> an appropriate MIME part.  Presumably the body could also be GB2312  
> MIME, though I'd discourage that as much as possible.

Now you've got me confused. An EAI message (i.e. one coming from an agent  
asserting a need for UTF8SMTP) might contain headers using UTF-8, but the  
body entirely in ASCII. So no MIME stuff anywhere in it. So you cannot say  
that EAI REQUIRES MIME, though for sure it would be stupid to implement it  
without MIME.

AIUI, if you assert UTF8SMTP and have UTF8 in the headers, and then want  
to use UTF-8 in the Body (a common situation, presumably), then you are  
supposed to include a Content-Type: specifying charset=utf8 and a suitable  
Content-Transfer-Encoding.

What we MIGHT do is to state that, for such EAI messages the default body  
charset was UTF-8 and the default CTE was 8bit (and most modern MUAs would  
likely display that correctly). That's just another extension to RFC  
553[12], and we have extended those already. The downside would be that  
any attempt at downgrading would have to put those assumed-by-default  
headers back.

In fact, I suspect that default practice is going to happen anyway within  
EAI-only communities, so we might as well make it official.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Nick Teint <[email-masked]>
Date: Thu, 19 Aug 2010 16:50:38 +0200

On 2010-08-19, at 15:43, Charles Lindsey wrote:
> AIUI, if you assert UTF8SMTP and have UTF8 in the headers, and then want to use UTF-8 in the Body (a common situation, presumably), then you are supposed to include a Content-Type: specifying charset=utf8 and a suitable Content-Transfer-Encoding.
> 
> What we MIGHT do is to state that, for such EAI messages the default body charset was UTF-8 and the default CTE was 8bit (and most modern MUAs would likely display that correctly). That's just another extension to RFC 553[12], and we have extended those already. The downside would be that any attempt at downgrading would have to put those assumed-by-default headers back.

Another downside is that you can't represent a legacy non-MIME message with an 8bit body. While incorrect, these message do exist and many user agents will handle them in a reasonable way.

This has two implications:

You need to keep the UTF-8 flag intact. Otherwise (and without 8bit headers), you can't know whether the body defaults to UTF-8 or whether you need to use the old handling (probably NOT as UTF-8).

You need to "upgrade" the message before you can add a UTF-8 header (e.g. Received: … for [email-masked]; the message has been originally sent to the alias [email-masked]). This requires adding MIME headers declaring a Content-Type of "text/plain; charset=UNKNOWN-8BIT". (This also happens when an 8BITMIME message is forwarded to non-8BITMIME servers, and has received a lot of criticism.)

A default CTE of "8bit" does not have the same problems ("7bit" and "8bit" both mean identity-transformation on line-based content) but it still makes implementations more complicated: When downgrading, you must look at the content in order to know that to use.

NT

---

From: John C Klensin <[email-masked]>
Date: Thu, 19 Aug 2010 12:34:54 -0400

--On Thursday, August 19, 2010 14:43 +0100 Charles Lindsey
<[email-masked]> wrote:

>> I'm obviously really confused.  My understanding was that EAI
>> required   UTF-8 headers and MIME encoded bodies, which
>> should also be UTF-8, but   an appropriate MIME part.
>> Presumably the body could also be GB2312   MIME, though I'd
>> discourage that as much as possible.

The implication of draft-iab-idn-encoding is that we ought to be
gradually deprecating everything but ASCII and UTF-8 (I would
say "everything but UTF-8", but there is specific language in
the MIME specs stating a preference for coding body parts with
no non-ASCII characters as "us-ascii" rather than "utf-8").   So
I agree with Shawn's observation but don't think it is a matter
for EAI to discuss or offer advice about (see note 1 below).

> Now you've got me confused. An EAI message (i.e. one coming
> from an agent asserting a need for UTF8SMTP) might contain
> headers using UTF-8, but the body entirely in ASCII. So no
> MIME stuff anywhere in it. So you cannot say that EAI REQUIRES
> MIME, though for sure it would be stupid to implement it
> without MIME.

But there will always be "MIME stuff" in an EAI-conformant
message with non-ASCII header material.  Remember, UTF8SMTP[bis]
requires 8BITMIME, 8BITMIME requires MIME, and MIME requires at
least a MIME-version header field, so, even if there is only one
body part and the content type is allowed to default to
'text/plain; charset="us-ascii"' (i.e., no Content-Type header
field present), there is "MIME stuff" present.

> AIUI, if you assert UTF8SMTP and have UTF8 in the headers, and
> then want to use UTF-8 in the Body (a common situation,
> presumably), then you are supposed to include a Content-Type:
> specifying charset=utf8 and a suitable

Yes.  Since 8BITMIME is needed to transport those UTF-8 headers,
the C-T-E can reasonably be "8bit"

> What we MIGHT do is to state that, for such EAI messages the
> default body charset was UTF-8 and the default CTE was 8bit
> (and most modern MUAs would likely display that correctly).
> That's just another extension to RFC 553[12], and we have
> extended those already. The downside would be that any attempt
> at downgrading would have to put those assumed-by-default
> headers back.

I think this would be certain to cause problems with other MIME
implementations.  Remember that messages are passed outside the
transport system and that making up other defaults --body parts
not requiring Content-Type or C-T-E fields-- would violate the
8BITMIME spec.   Now, if we were to say that use of EAI required
either the normal default Content-Type (text/plain;
charset="us-ascii") or that text types were required to be
charset="utf-8", I think that would be ok.  However, I think it
would accomplish very little in practice that a simple
recommendation to use UTF-8 in body parts where possible
(labeled as the MIME and 8BITMIME specs required) would not.

In particular, if someone were determined to use GB body parts,
I don't think we are going to be able to effectively prohibit
that.  We should just try to insist that they be properly
labeled and identified.

> In fact, I suspect that default practice is going to happen
> anyway within EAI-only communities, so we might as well make
> it official.

Partially because of the wave of MTAs that, in self-defense,
took very harsh measures to deal with unlabeled 8bit content,
there is, in my experience, little current practice of using
non-ASCII body parts without Content-type labeling.   I see no
reason at all to try to reintroduce that practice, especially
when we remember how common text/html and various binary
("application") and image type body parts have to be handled by
those same MTAs and MUAs.

    john