Subject: Re: [EAI] Body parts

From: Shawn Steele <[email-masked]>
Date: Thu, 10 Jul 2008 12:27:02 -0700

Charles said:
> Actually, there may be good reasons for using other charsets in body
> parts. Charsets in the higher reaches of Unicode require rather long
> strings of bytes per character in UTF-8. That is not a serious issue for
> headers, but body parts might well be significantly shorter in charsets
> more suited to the language in use.

> Can anyone provide data on the efficiency of UTF-8 and BIG5, for example?

The presumed savings isn't actually very much.  Add an attachment, jpg, html encoding or whatnot and the "savings" will be lost in the noise.  I concede that it might be interesting for a satellite text message or something where the bandwidth is amazingly poor, however that could be handled by the device's portal/gateway.

The cost of not using UTF-8 is a compatibility.  The #1 globalization issue I see is related to code page incompatibilities and the #1 place I see problems is in e-mail.  Within the windows environment it is "only" because the system code page varies between machine, but between OS's or applications I also see a large difference in the supported character ranges, and variations in the supported character ranges.  Some standards are even inconsistent.

The IETF mail lists are common examples of this problem.  Many non-ASCII characters get lost in transmission, leading people to ask for Unicode code points or resending.  The additional emails alone negate any minor savings in encoding size.  The additional noise and lost information also hurts those who received the original mail as intended.

- Shawn
_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: Shawn Steele <[email-masked]>
Date: Thu, 10 Jul 2008 12:27:12 -0700

Frank said:
> Adding a "you SHOULD do the obvious" norm does not really help, or does it ?

I think it does.  EAI requires that the client software understand UTF-8 (otherwise From: and To: look stupid), so any application must therefore be able to handle UTF-8 in the body as well.  This would help ensure that new implementations support UTF-8.  (Many modern devices, particularly phones, don't bother leading to the aforementioned compatibility issues).

I wouldn't mind language that said.  "Conforming applications MUST support UTF-8 in the body part and SHOULD use UTF-8 unless they have a really really good reason".

MUST support it so that everyone can expect that recipients will understand the code page (right now there's no *guaranteed* code page that'll work across all environments)

SHOULD use it since it's the most consistent of the encodings and if it MUST be supported, then compatibility is ensured.  I didn't suggest "MUST" because I realize that some devices may have a more difficult time with the transition, although gateways could help.

- Shawn

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 10 Jul 2008 22:58:04 +0200

Shawn Steele wrote:

>> a "you SHOULD do the obvious" norm does not really help, or does it ?
 
> I think it does.  EAI requires that the client software understand
> UTF-8 (otherwise From: and To: look stupid), so any application
> must therefore be able to handle UTF-8 in the body as well.

That's what I doubted:  An EAI-aware MUA can of course handle UTF-8,
and hopefully the user of this MUA finds an e-mail service provider
with EAI-aware MSA / IMAP / POP3.  But this user might still have
friends with less capable MUAs and service providers, or use mailing
lists and newsgroups where UTF-8 is not yet the norm.

A technical SHOULD trying to modify netiquette would make me nervous.
Limited to "SHOULD be the default" it is acceptable, this could be
added to the eai-scenarios and/or eai-email-clients draft.

> This would help ensure that new implementations support UTF-8.

As you said, when they can do EAI at all they must support UTF-8.
And anybody implementing a MUA today without supporting UTF-8 is
a hopeless case.  The email-arch draft could note that as fact, it
has no I18N considerations so far:  A point for the Last Call, the
last time when I wanted a mere informative EAI reference in this
draft a bunch of IETF mail reactionists tried to lynch me... :-)

> Many modern devices, particularly phones, don't bother leading
> to the aforementioned compatibility issues

The LEMONADE folks don't belong to the old mail guard, I think.

> right now there's no *guaranteed* code page that'll work across
> all environments

There's RFC 2277 and 5198, anybody who still hasn't heard the shot
is deaf.  But UTF-8 is IMHO no *guaranteed* code page.  On my box
characters in code page 437, 819, 850, 858, 878, 923, 1252, and
others are *guaranteed* to work, encoded as UTF-8 or in various
"legacy" charsets.  Actually I never tested if my MUA can do 437,
850, 858, or 923, but the used fonts offer all glyphs for these
and many other code pages.  When I want a *guarantee* I use 1252,
and that is my default, replacing <http://purl.net/net/cp/819>.

 Frank

P.S.: Just for fun using http://purl.net/net/cp/878 in reply to your 819 ;-)

---

From: Ernie Dainow <[email-masked]>
Date: Fri, 11 Jul 2008 16:33:12 -0400

> added to the eai-scenarios and/or eai-email-clients draft.
Section 4 in the eai-email-clients draft recommends UTF-8 as the default 
for new mail and  the same character encoding as received mail for 
reply/forward. However since this document is informational only, 
recommendations are not as strong as SHOULD in a normative document.

Ernie

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 11 Jul 2008 23:30:27 +0200

Ernie Dainow wrote:
   
> Section 4 in the eai-email-clients draft recommends UTF-8 as the
> default for new mail and  the same character encoding as received
> mail for reply/forward.

Yes, maybe s/should/SHOULD/ in this paragraph...

> since this document is informational only, recommendations are 
> not as strong as SHOULD in a normative document.

...aim higher, and you won't "SHOULD" in your own foot. ;-)  

 Frank

---

From: Claus Färber <[email-masked]>
Date: Sat, 12 Jul 2008 14:00:43 +0200

Shawn Steele schrieb:
> The presumed savings isn't actually very much.  Add an attachment,
> jpg, html encoding or whatnot and the "savings" will be lost in the
> noise.  I concede that it might be interesting for a satellite text
> message or something where the bandwidth is amazingly poor, however
> that could be handled by the device's portal/gateway.

If bandwith really is a problem, use compression.

I guess the charset does not matter so much with deflated text. 
(However, that's an educated guess; data would be nice.)

Claus

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 12 Jul 2008 17:55:44 +0200

Claus Färber writes:
> If bandwith really is a problem, use compression. I guess the charset 
> does not matter so much with deflated text. (However, that's an 
> educated guess; data would be nice.)

I didn't keep the data, but I did play around and measure things when I 
wrote RFC 4978, and my experience is that it makes a difference.

Deflate (like almost all other compression algorithms) want to see the 
atom. If it's compressing graphics, it wants to see the pixels or 
whatnot. If it's compressing text, it wants to see the characters. In 
deflate's case, it assumes that a file-format atom has some relation to 
octets, and works with octets.

If you send big5 with c-t-e 8bit, then deflate will see a series of 
half-characters, which is less good than seeing a series of characters, 
but still, there's a plain enough relationship that deflate's 
assumption works well. If you send the same text as utf-8 with c-t-e 
8bit, it sees thirds of characters, which is almost as good as seeing 
half-characters.

But people don't send c-t-e 8bit. They send a lot of q-p or base64, both 
of which obfuscate the data. If deflate is told to compress big5 with 
c-t-e base64, its first input oftet encodes 37.5% of a character, the 
second 37.5% more and the third the last 25% of the character with and 
12.5% of the next character. It goes on like that. A character is 
encoded in three different ways depending on where it is in the stream. 
Q-p obfuscates differently, but the effect is comparable.

UTF-8+base64 happens to be better than big5, for the poor reason that 
its three-byte characters match one four-byte base64 group. There's 
less obfuscation than in the case of big5.

HOWEVER. I didn't bother doing anything like 4978 for SMTP or 
Submission, because I think that wouldn't be worth the bother. IMO 
there isn't any simple change with a large benefit.

Arnt