Subject: Re: [EAI] UTF8 POP and character sets

From: John C Klensin <[email-masked]>
Date: Tue, 21 Mar 2006 11:06:21 -0500

--On Tuesday, March 21, 2006 18:26 +0900 Martin Duerst 
<[email-masked]> wrote:

> At 08:02 06/03/21, John C Klensin wrote:
>  >Chris,
>  >
>  >It seems to me that the list of character sets that SHOULD
> be supported potentially adds considerable complexity to the
> protocol and doesn't accomplish much.
>  >
>  >So, two questions for people to think about...
>  >
>  >(1) Any POP client today that receives non-ASCII material is
> going to do so in encoded word form for the headers and Base64
> or Q-P form,
>
> Or 8bit, unless that's forbidden in the case of POP.

Forbidden in POP (see forthcoming note)

>  >with a character set label, for textual body parts. Q-P
> works fairly badly outside the "Latin" ISO8859 sets and some
> would claim that it never works well.
>
> It's not that it doesn't work well, it's just really
> inefficient.

>From a user perspective, the "feature" of Q-P is that it is 
almost readable even without conversion back to the original 
form.  That works ok -- although preferences differ-- for text 
bodies that are mostly ASCII with a sprinkling of Latin-N 
characters.   For other text bodies, it just don't communicate 
any useful information if read without back-conversion.  If one 
can guarantee conversion (decoding) at the receiving end, then 
it is all about efficiency, as you point out.

>  >Those clients already have the machinery for decoding (or
> don't and have unhappy users), although future ones --the
> client side of the NO_RETR problem-- may not.  Now, getting
> from Base64 (or Q-P) or encoded words to a character coding
> that matches the charset label or encode word charset code is
> straightforward and mechanical.  But I wonder if we need to
> support --or if you document actually intends-- any encoding
> other than UTF-8 for addresses.
>
> I think that would be a highly unnecessary complication.

I think that is right, but we need to look carefully at existing 
standards and actual use patterns.

>  >Perhaps now is the time to start saying "our on-the-wire
> data format for text is going to be UTF-8 and non-UTF-8
> environments need to support that conversion, and only that
> conversion, going forward".
>
> As for 'text', as far as this refers to body text, I think
> it's okay to
> leave things as they are currently, i.e. that various
> encodings are
> allowed as long as they are documented with a charset
> parameter in
> the Content-Type header. Of course, my hope is that most of the
> messages that use internationalized addresses will have their
> body in UTF-8, but I don't think there is any reason to force
> that, it will happen naturally.

I think we agree

>  >(2) That question is clearly more general that this WG's
> mission.    Is it time to formally ask the IAB to convene a
> process for advising the applications community whether we are
> ready to take such a step?
>
> Which step exactly?

See below

>  >Any resemblance between the above concerns and this
> morning's AppArea conversation about an NVT-i18n is obviously
> no coincidence.
>
> For the benefit of those not in Dallas, could you explain.

One of the key design decisions in getting the ARPANET to 
function at a user/ applications level was the establishment of 
a standard for the form in which text would pass over the net. 
It is known as "net-ASCII" or "the NVT (network virtual 
terminal) character set" (the two are slightly different, but 
the difference is not important here).  The model was that 
systems sending text over the network --especially as part of 
protocol exchanges-- were required to convert to that form and 
systems receiving such text were required to convert from it -- 
both to the extent needed locally -- this avoiding all sorts of 
n-squared problems.

For non-ASCII text, we have tended (for good reasons at the 
time) to go the other way, more or less permitting any CCS to be 
used and focusing on making sure that the receiving system knows 
what has been sent to that it can take responsibility for 
remapping it locally if needed.  For some purposes that is fine, 
but it leads to lousy protocol design.

So the question that started yesterday morning was whether it is 
time to define an NVT-i18n (net-Unicode?).  Given how we have 
gotten here, that form would presumably be UTF-8 with CRLF line 
termination, but the question is whether, even if we assume a 
Unicode world, we are ready to make an NVT-like rule rather than 
having UCS-2 or UCS-4, sundry bit orders, etc., floating around 
the network as "Unicode". And, of course, for other character 
codings, the need becomes even more urgent.

    john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 21 Mar 2006 12:17:11 -0600

John C Klensin wrote:
> So the question that started yesterday morning was whether it is time 
> to define an NVT-i18n (net-Unicode?).  Given how we have gotten here, 
> that form would presumably be UTF-8 with CRLF line termination, but 
> the question is whether, even if we assume a Unicode world, we are 
> ready to make an NVT-like rule rather than having UCS-2 or UCS-4, 
> sundry bit orders, etc., floating around the network as "Unicode". 
> And, of course, for other character codings, the need becomes even 
> more urgent. 
One reading of RFC 2277 (IETF Policy on Character Sets and Languages) is 
that it allows a protocol to do exactly that, while also permitting a 
protocol to permit other choices.

It may be time to consider removing or restricting the leeway.