Subject: [EAI] UTF8 POP and character sets

From: John C Klensin <[email-masked]>
Date: Mon, 20 Mar 2006 18:02:30 -0500

Chris,

It seems to me that the list of character sets that SHOULD be 
supported potentially adds considerable complexity to the 
protocol and doesn't accomplish much.

So, two questions for people to think about...

(1) Any POP client today that receives non-ASCII material is 
going to do so in encoded word form for the headers and Base64 
or Q-P form, with a character set label, for textual body parts. 
Q-P works fairly badly outside the "Latin" ISO8859 sets and some 
would claim that it never works well.  Those clients already 
have the machinery for decoding (or don't and have unhappy 
users), although future ones --the client side of the NO_RETR 
problem-- may not.  Now, getting from Base64 (or Q-P) or encoded 
words to a character coding that matches the charset label or 
encode word charset code is straightforward and mechanical.  But 
I wonder if we need to support --or if you document actually 
intends-- any encoding other than UTF-8 for addresses.  Perhaps 
now is the time to start saying "our on-the-wire data format for 
text is going to be UTF-8 and non-UTF-8 environments need to 
support that conversion, and only that conversion, going 
forward".

(2) That question is clearly more general that this WG's 
mission.    Is it time to formally ask the IAB to convene a 
process for advising the applications community whether we are 
ready to take such a step?

Any resemblance between the above concerns and this morning's 
AppArea conversation about an NVT-i18n is obviously no 
coincidence.

      john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Martin Duerst <[email-masked]>
Date: Tue, 21 Mar 2006 18:26:42 +0900

At 08:02 06/03/21, John C Klensin wrote:
 >Chris,
 >
 >It seems to me that the list of character sets that SHOULD be supported 
potentially adds considerable complexity to the protocol and doesn't 
accomplish much.
 >(1) Any POP client today that receives non-ASCII material is going to do 
so in encoded word form for the headers and Base64 or Q-P form,

Or 8bit, unless that's forbidden in the case of POP.

 >with a character set label, for textual body parts. Q-P works fairly 
badly outside the "Latin" ISO8859 sets and some would claim that it never 
works well.

It's not that it doesn't work well, it's just really inefficient.

 >Those clients already have the machinery for decoding (or don't and have 
unhappy users), although future ones --the client side of the NO_RETR 
problem-- may not.  Now, getting from Base64 (or Q-P) or encoded words to a 
character coding that matches the charset label or encode word charset code 
is straightforward and mechanical.  But I wonder if we need to support --or 
if you document actually intends-- any encoding other than UTF-8 for addresses.

I think that would be a highly unnecessary complication.

 >Perhaps now is the time to start saying "our on-the-wire data format for 
text is going to be UTF-8 and non-UTF-8 environments need to support that 
conversion, and only that conversion, going forward".

As for 'text', as far as this refers to body text, I think it's okay to
leave things as they are currently, i.e. that various encodings are
allowed as long as they are documented with a charset parameter in
the Content-Type header. Of course, my hope is that most of the
messages that use internationalized addresses will have their
body in UTF-8, but I don't think there is any reason to force
that, it will happen naturally.

 >(2) That question is clearly more general that this WG's mission.    Is 
it time to formally ask the IAB to convene a process for advising the 
applications community whether we are ready to take such a step?

Which step exactly?

 >Any resemblance between the above concerns and this morning's AppArea 
conversation about an NVT-i18n is obviously no coincidence.

For the benefit of those not in Dallas, could you explain.

Regards,    Martin.

---

From: Chris Newman <[email-masked]>
Date: Tue, 21 Mar 2006 22:45:02 -0600

> potentially adds considerable complexity to the protocol and doesn't

The charset requirements are specifically for the purpose of up-conversion from 
RFC 2047/2231 to native UTF-8.  I do not require or recommend any changes to 
bodies, body charsets or body transfer encodings.

RFC 2047 interoperability is mediocre in the field (particularly when it comes 
to whitespace handling between encoded words) and 2231 support is uncommon.  So 
the implicit intention behind the POP document (and upcoming IMAP document) 
up-conversion requirements is to obsolete the encoding mechanisms that don't 
work well as quickly as possible.  Indeed, these encoding mechanisms have 
caused my product team a truly incredible amount of pain.  The goal of 
deprecating 2047 and 2231 as quickly as possible is the primary reason why I'm 
interested in this effort.  UTF-8 local parts just happen to be a nice 
side-effect of the UTF-8 header proposal from my perspective.

The charset requirements are written so an implementer can simply use an 
open-source library such as ICU to make a POP server fully compliant, so the 
implementation burden is minor.  And it's only a "SHOULD" for the rare cases 
where a POP server can't deal with 1.5MB shared library for charset conversions.

The primary value of POP is how simple it is to write a client.  Deprecating 
the requirement to support 2047/2231 will improve the situation so it's worth 
some server effort.  Indeed, POP clients have the worst track-record on 2047 
compliance.

> (2) That question is clearly more general that this WG's mission.    Is it
> time to formally ask the IAB to convene a process for advising the

While I like the move in the direction of UTF-8, I see no reason to push it 
more than RFC 2277 does today.  MIME body charset labels and encodings work 
just fine and don't seem to cause any significant problems.  There are many 
charset conversion libraries which are competent.  I do see a reason to 
deprecate 2047/2231 as quickly as possible, but that's for interoperability 
reasons; the mandated charset conversions for header charsets are simply a 
necessary side effect of replacing the problematic encodings with UTF-8 headers.

                - Chris