Subject: [EAI] draft-klensin-encoded-word-type-u-00

From: John C Klensin <[email-masked]>
Date: Thu, 24 Nov 2011 16:58:18 -0500

Hi.

One of John Levine's comments about the mailinglist document and
some issues with pop-imap-downgrade have convinced me that there
is a problem with the use of encoded words in strings that users
are expected to see and work with that we might actually know
how to fix.   

Historically, encoded words have been used in contexts where we
expected MUAs to turn them back into their original (native
character) forms, with display of those encoded forms to users
being a (hopefully infrequent) last resort.   That decoding is
less likely in the pop-imap-downgrade and mailing list
situations: in both, if the relevant clients could handle native
UTF-8 addresses, the encoded word forms would not be necessary.

The problem with encoded words in those contexts is that
encoding form "Q" is pretty useless except for mostly-ASCII text
and of somewhat dubious value then and encoding form "B" pretty
much requires a computer to decode.  %-encoded UTF-8 octets are
even worse: one needs a computer or a subtle calculation to turn
them into a Unicode code point reference which then must be
looked up in a table and one actually has to understand how
UTF-8 works to be able to tell whether %NN%MM%OO is one
character, two characters, or three characters.

One possible solution is to mix the encoded word strategy will
direct encoding of Unicode characters by code point.  I've just
posted draft-klensin-encoded-word-type-u-00 as a strawman for
doing just that.  It is not a WG document.  It is not even a
serious proposal at this stage.  But it provides a relatively
concrete proposal about "something else" that we might do to get
at least slightly dug out of the present hole that some of you
might consider worth thinking about.

I have deliberately not added normalization considerations to
that draft, but it could easily be done and I suspect it would
be necessary if the proposal was to be as useful as it might be.

    john

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 25 Nov 2011 00:13:35 +0100

On 24 November 2011 22:58, John C Klensin <[email-masked]> wrote:

> encoding form "B" pretty much requires a computer to decode.
> %-encoded UTF-8 octets are even worse: one needs a computer
> or a subtle calculation to turn them into a Unicode code point

I'm not sure what is worse, with UTF-8 I'd see where code points
begin and end, and a "manual" decoding of the tail bytes is not
impossible.  But after that step I'd give up and write a decoder,
actually I did that years ago.

> I've just posted draft-klensin-encoded-word-type-u-00 as a
> strawman

| =?iso-10646-UCS-4+de?u?Das ist verr\u'00FC'ckt?=

Yes, it must appear as ASCII-gibberish word, you have to encode
the spaces somehow.  And there are lots of MUST NOTs in section
5 bullet (3) of RFC 2047.  I vaguely recall that the USEFOR WG
also tried to do something interesting with RFC 2047 + 2231, but
I've forgotten the outcome.  Only Bruce Lilly really understands
these RFCs, and here I intentionally exclude the authors. :-)

But doing something based on RFC 5137 is attractive.  (If you
want to see me screaming look up "RFC 2231" in USEFOR archives.)

Of course we could say that rare EAI downgrading scenarios can
as well eat the existing 2231 dogfood, but I'm not sure that
"encoded words" really solve EAI problems with all those MUST
NOT in 2047.

-Frank

---

From: John C Klensin <[email-masked]>
Date: Thu, 24 Nov 2011 22:54:32 -0500

--On Friday, November 25, 2011 00:13 +0100 Frank Ellermann
<[email-masked]> wrote:

> I'm not sure what is worse, with UTF-8 I'd see where code
> points begin and end, and a "manual" decoding of the tail
> bytes is not impossible.  But after that step I'd give up and
> write a decoder, actually I did that years ago.
> 
> the spaces somehow.

Writing while tired... sorry.  2047 is very clear on the spaces
-- they have to be encoded.  So, more correctly,

  =?iso-10646-UCS-4+de?u?Das\u'0020'ist\u'0020'verr\u'00FC'ckt?=

Still ugly, but one can at least find a Unicode table and look
the code points up without having to go through an intermediate
decoding step.

>  And there are lots of MUST NOTs in section
> also tried to do something interesting with RFC 2047 + 2231,
> but I've forgotten the outcome.  Only Bruce Lilly really
> understands these RFCs, and here I intentionally exclude the
> authors. :-)
> 

Agreed.  As I've said, the draft was posted to enable a
discussion, not as a proposed solution.

    john

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 25 Nov 2011 14:52:39 +0900

Hello John,

> UTF-8 addresses, the encoded word forms would not be necessary.

My understanding is that, in this day and age, lots of (if not all) 
clients can decode "Q" and "B". There may be some restrictions, e.g. in 
Japan there are still clients that can decode iso-2022-jp (and these 
days also UTF-8), but not many other encodings. Some of them may use a 
Japanese encoding (Shift_JIS or EUC-JP) internally, even if they accept 
(with subsetting) UTF-8 externally.

There are currently not yet any clients that can handle raw UTF-8, and 
of course there are no clients that can handle encoded-word-type-u. What 
we want is to get them to handle raw UTF-8. So I really don't understand 
where encoded-word-type-u would be helpful.

> much requires a computer to decode.

Yes indeed.

> %-encoded UTF-8 octets are

Well, I might be slightly biased, but I'd choose %-encoded UTF-8 over 
base64-encoded UTF-8 any single time.

For base-64, I have:
- Convert character to bit pattern (use a table)
- Collect bit patterns into bytes
- Decode bytes according to UTF-8
- Look up characters in Unicode code table

For %-encoding, I have:
- Convert from hex to bits (trivial)
- Decode bytes according to UTF-8
- Look up characters in Unicode code table

So I don't see what would be worse in %-encoded UTF-8 octets when 
compared with base64. Actually, for %-encoding, I can use 
http://rishida.net/tools/conversion/ (fourth input field) and get 
everything done. Of course, it would be possible to do the same for 
base64, too, if somebody has the time to put together a page.

> be necessary if the proposal was to be as useful as it might be.

People don't want to look numbers up, whether it be on 
http://rishida.net/tools/conversion/ or on 
http://www.unicode.org/charts/ or wherever. The nerds on this list, me 
included, are the exception, not the rule. For 99.9% or so of users, 
things either work or they are broken. For the rest, things such as 
base64 or %-encoding are not an issue. If anything needs to be done, 
putting up a web site that automatically converts between (currently 
defined) encoded word syntax and actual text is way more useful than 
writing a draft for yet another convention. One big advantage of the 
former (and http://rishida.net/tools/conversion/) is that conversion is 
for a whole string, rather than character by character, as any human 
lookup would be.

So, let humans do what they are good at, and keep the code conversion 
for computers.

Regards,    Martin.