Subject: Re: [EAI] Body parts

From: Shawn Steele <[email-masked]>
Date: Sat, 12 Jul 2008 14:38:04 -0700

> But people don't send c-t-e 8bit. They send a lot of q-p or base64, both
> of which obfuscate the data.

Part of my hope was that they'd be 8 bit aware and not need base64.  I'm also hoping to recommend UTF-8 not to save space, but to reduce some ambiguity about encodings.  In practice transmissions often loose encoding information and end up with some scrambled characters.

- Shawn
_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Sat, 12 Jul 2008 19:55:13 -0400

--On Saturday, 12 July, 2008 14:38 -0700 Shawn Steele
<[email-masked]> wrote:

>> But people don't send c-t-e 8bit. They send a lot of q-p or
>> base64, both of which obfuscate the data.
> 
> Part of my hope was that they'd be 8 bit aware and not need
> base64.  I'm also hoping to recommend UTF-8 not to save space,
> but to reduce some ambiguity about encodings.  In practice
> transmissions often loose encoding information and end up with
> some scrambled characters.

Shawn,

Really?  Your data collection sources obviously differ from
mine, but my impression is that we've seen very little
transmission loss or confusion in the last 10 or 15 years, at
least among MIME-conformant implementations that follow the
rules.  And many of the places where those losses actually show
up turn out to be among systems that either advertise 8BITMIME
but not support it, decide to send 8 bit data despite not
getting that announcement, or send MIME content-type information
and file names that are inconsistent with each other.

We also used to have a lot of problems with slight
incompatibilities among various flavors of UUENCODE, but it has
been a long time since I've seen any significant number of those.

For at least those classes of problems, sending UTF-8 directly
runs much more risk of data loss than Q-P or Base64 (although
local forms like Big5 are probably no better).

    john

---

From: Shawn Steele <[email-masked]>
Date: Sat, 12 Jul 2008 21:41:57 -0700

> rules.

Well, of course if they all play well it should work :)

The Unicode list gets mail fairly regularly that someone complains they couldn't read :)

Remember that I'm suggesting this for apps advertising UTF8STMP, so there *should* be no loss of data by complying applications, since presumably they'd all be designed correctly.  For the misbehaving ones, as you pointed out there's already a problem.

I hear 99% of the code page related issues within Microsoft, and I've encountered difficulties with code pages from Hotmail, Live Mail, Exchange & Outlook.  Many of those are probably when receiving mail without proper MIME content-type information.  Other causes seem to be varying code page implementations between the sender & recipient &/or different interpretations of the standards or code page names.

Obviously recommending UTF-8 can't stop mail clients from doing the wrong thing, but if we could recommend a course that moves towards a single encoding, then the entire encoding-related class of issues should eventually start doing better.  As someone mentioned UTF-8 is being adopted by more mail clients, but it can't hurt to inspire people to support UTF-8.

FWIW: Mark Davis was happy to show that UTF-8 is now the most common web encoding. http://www.w3.org/QA/2008/05/utf8-web-growth.html Sure, web isn't mail, but it does reflect a general trend.

- Shawn

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 13 Jul 2008 09:21:20 +0200

John C Klensin writes:
> ... send MIME content-type information and file names that are 
> inconsistent with each other.

Or content-type information which doesn't describe the charset(s) 
actually used. In my experience (I get to analyze those things when 
they arrive, and guess what happened at the sender) it happens when two 
pieces of software don't talk to each other enough. Consider typical 
webmail: A browser delivers the body text and some header fields, some 
advertising system delivers a spamfooter, and the webmail backend makes 
the rest of the header and glues it all together. Lots of opportunities 
for misunderstanding.

IMO a UTF8SMTP would make things worse, not better, since there would be 
one more thing to get wrong.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Sun, 13 Jul 2008 05:20:30 -0400

--On Sunday, 13 July, 2008 09:21 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

> John C Klensin writes:
>> ... send MIME content-type information and file names that
>> are  inconsistent with each other.
> 
> Or content-type information which doesn't describe the
> charset(s) actually used.

My intent was to cover that case too without going into great
detail.  But you are, of course, correct.

> In my experience (I get to analyze
> those things when they arrive, and guess what happened at the
> sender) it happens when two pieces of software don't talk to
> each other enough. Consider typical webmail: A browser
> delivers the body text and some header fields, some
> advertising system delivers a spamfooter, and the webmail
> backend makes the rest of the header and glues it all
> together. Lots of opportunities for misunderstanding.

Indeed.  The problem is aggravated by something else, which has
turned out to be one of the bigger problems in handling
international character sets in MIME:  Suppose someone sends a
message in charset X to a user whose environment is normally set
up for charset Y.  That second user is able to read messages
marked as being in charset X (which isn't always true).  But, if
the second user then excerpts a piece of the message and
replies, the reply typically goes out in charset Y, especially
if some of the second user's characters are needed in the reply.
In general, unless Y is UTF-8, whatever happens next is bad news
-- the message goes out labeled as charset=Y, it contains some
characters that are really in charset X, but the MUA belonging
to the recipient of the reply has no clue about what is going on
and the original characters are displayed as trash, even if that
recipient is perfectly capable of handling charset X.

It may be that we should have including charset-switching tags
in enriched text, but that format is so little used (as a
percentage of total legitimate messages sent) that it might have
made little difference.  The problem will presumably go away as
more and more of us switch to UTF-8, but that doesn't seem to be
happening quickly... and I don't see a changed default in a
standard as accelerating it very much, even if we were in a
position to do that.

> IMO a UTF8SMTP would make things worse, not better, since
> there would be one more thing to get wrong.

Maybe.  On the other hand, the drafts permit one, and only one,
charset to be used in headers and that is UTF-8.  There is no
mechanism to say "my headers are really in Slobbovian Alphabet
4" and that omission is deliberate.   So we won't have problems
of someone identifying one charset and sending another, headers
with mixed charsets, etc.  While I believe it will be some years
before we can formally deprecate all of the sundry [older]
methods for encoding non-ASCII characters into headers, I do
think that, if and when those standards are updated, it is time
to attach a "SHOULD use encoded UTF-8 and UTF-8 only" to them,
which would be, IMO, a big step in the right direction.

But _any_ extension, to anything, creates more opportunities for
the careless, stupid, or malicious to get things wrong.  It is
the price one pays.

   john

---

From: SM <[email-masked]>
Date: Sun, 13 Jul 2008 06:52:02 -0700

At 21:41 12-07-2008, Shawn Steele wrote:
>FWIW: Mark Davis was happy to show that UTF-8 is now the most common 
>web encoding. http://www.w3.org/QA/2008/05/utf8-web-growth.html 
>Sure, web isn't mail, but it does reflect a general trend.

That's because the web server implementations moved to using UTF-8 as 
the default.  It went unnoticed as the web browsers support 
it.  There are problems such as when non-UTF-8 content is served as 
UTF-8.  In the mail world, it's the reverse; UTF-8 content rendered 
as non-UTF-8 by legacy MUAs.

Regards,
-sm

---

From: John C Klensin <[email-masked]>
Date: Sun, 13 Jul 2008 13:13:08 -0400

Shawn,

I think this is out of scope for the WG so, until Harald or
XiaoDong tell me otherwise, I'm going to try to make this my
last note on the subject (or take it off list).   But...

>From when I was trying to follow that list, as well as general
experience, I would suggest that the origin of most of those
problems are not ones in which the content-type and associated
charset parameter match the actual content of the message.  They
are because a sending MUA (or web interface configuration) has a
default charset label that it applies, one that isn't UTF-8 or
because the type of copying and pasting that I described in my
note to Arnt has occurred (most often just by quoting all or
part of a reply).
 
> Remember that I'm suggesting this for apps advertising
> UTF8STMP, so there *should* be no loss of data by complying
> applications, since presumably they'd all be designed
> correctly.  For the misbehaving ones, as you pointed out
> there's already a problem.

Actually, the fact that a particular app can handle UTF-8 for
headers is not really a sufficient condition given the way may
mail systems are organized.  If a system receives mail in one
charset and format, and the user wants to paste part of its
contents into another message, ability to convert that character
set to UTF-8 and do so at the right point is required, not
merely the ability to render both charsets.  And the "at the
right point" part is significant too, as our documents on the
use of POP and IMAP with UTF8 headers and envelopes demonstrate.

> I hear 99% of the code page related issues within Microsoft,
> and I've encountered difficulties with code pages from
> Hotmail, Live Mail, Exchange & Outlook.  Many of those are
> probably when receiving mail without proper MIME content-type
> information. 

Of course, historically, some of those systems have been among
the worst offenders about not properly identifying the charsets,
code pages, and file types they are sending out.

> Other causes seem to be varying code page
> implementations between the sender & recipient &/or different
> interpretations of the standards or code page names.

Possibly, but I've seen symptoms of that far less than the above.

> Obviously recommending UTF-8 can't stop mail clients from
> doing the wrong thing, but if we could recommend a course that
> moves towards a single encoding, then the entire
> encoding-related class of issues should eventually start doing
> better.  As someone mentioned UTF-8 is being adopted by more
> mail clients, but it can't hurt to inspire people to support
> UTF-8.

I hate to say "cleaning up your own house would make more
difference than anything the IETF could do", but, if it were
made very difficult to transmit anything textual from, e.g.,
Outlook (both current and, via service packs, back to at least
to the 2003 version) in other than UTF-8, with content-type
fields (including the charset parameter) and file name fields
all consistent with the UTF-8 being transmitted, I'd guess a
large fraction of the problem would disappear in a fairly short
time.  

On the other hand, if we "merely" change the Standard, I note
that one major MUA with a very long history still does not
handle MIME properly (it uses an "attachment" model rather than
a "body part" one, leading to both subtle and gross problems)
and that the contemporary version of Office doesn't precisely
follow the standard or encourage users strongly enough to
configure systems to do so (e.g., too many things sent as
application/octet-stream who identity is perfectly well know to
the Operating System).

> FWIW: Mark Davis was happy to show that UTF-8 is now the most
> common web encoding.
> http://www.w3.org/QA/2008/05/utf8-web-growth.html Sure, web
> isn't mail, but it does reflect a general trend.

See SM's note.

   john

---

From: Shawn Steele <[email-masked]>
Date: Sun, 13 Jul 2008 10:40:51 -0700

I guess the WG scope doesn't include the body, but it does include UTF-8 :)

> difference than anything the IETF could do"

Unfortunately its not "our own house" that's the entire problem, but rather interoperability with software from a large number of other vendors.

> if it were made very difficult to transmit anything textual from, e.g.,
to the 2003 version) in other than UTF-8...

My understanding is that we did that once, and the Japanese market reacted strongly because (my understanding) the cell phone industry was stuck with shift-jis.

So I would like it to be included that ALL clients need to support UTF-8, otherwise we can't move forward.

I can't speak to the accuracy of our tagging, however I do know we've experienced incompatibilities with the JIS extensions, for data nominally in the same code page.

- Shawn