Subject: [IMA] draft-utf8headers-00

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 20 Oct 2005 20:17:45 +0100

Basically, I entirely approve of the general approach taken by this draft,  
and by the others in this series.If you want to see my full views on this,  
then see <http://www.imc.org/ietf-imaa/mail-archive/msg00867.html> which I  
wrote at the tail end of Paul Hoffman's attempt on the old imaa list. It  
is very similar to what John is now proposing in his freamework draft, but  
raises a few additional issues. Feel free to make use of it.

However, there are two major issues that I wish to raise, and a few lesser  
ones after that:

1. Essentially, we are defining an extension to RFC 2822. However, RFC  
2822 carefully steered clear of transport issues, and this document should  
do the same.

What we are defining could be called the "Internationalized Internet  
Message Format", so let us not be bashful - let us say outright that is  
what we are defining, and that it is a supserset of the "Internet Message  
Format" defined by RFC 2822 - its principal difference, of course, being  
that it allows header fields in UTF-8.

However, it should be clear that messages in this new format can be  
transported via any protocol with the necessary capabilities. SMTP is the  
obvious candidate (and we have proposals to extend SMTP), but these  
messages might also be sent via UUCP, or they might be gatewayed into  
X.400, or into News and then sent via NNTP, or they might be encapsulated  
using message/rfc822 and hidden inside some other message, or they might  
be bundled up in mbox format and sent by FTP, and I am sure there are many  
other possibilities.

Now UUCP is already capable of transporting such messages (I am not sure  
whether UTF-8 in Bang Paths would work, but it would not surprise me if  
they did). NNTP also already has that capability for stuff converted into  
News format. I haven't a clue about X.400, but anything involving FTP  
would already work. The biggest problems, of course, are MUAs and storage  
agents (POP3 and IMAP), but that is a matter for the people who want to  
send/receive such mail to worry about.

Therefore, any suggestion in the draft that this format is only for use in  
connection with an extended SMTP needs to come out (last para of 1.1, 2nd  
para of 1.2, last para of 1.2 - what you are defining is a "format", not a  
"protocol", and most of section 3 and its subsections). All that needs be  
said is that messages in this format MUST NOT be sent unless it is known  
that the transport to be used supports them (with a remark that, even  
then, bounces are to be expected) and with a reference to the extended  
SMTP draft as the prime example. Most (but not all) of what is in section  
3 really belongs in that companion SMTP draft.

2. Para 5 of 1.2 gives the impression that UTF-8 can be used anywhere in  
headers, though you do point to some of the exceptions later on. However,  
you really need to say very precisely where UTF-8 is to be allowed on top  
of the present RFC 2822, and where it is not.

Clearly it is allowed in <unstructured>s, <comment>s and <phrase>s (which  
immediately gives you the Subject, Comments and Keywords headers plus any  
X-headers). <date-time> and <msg-id> remain as they are for the  
foreseeable future, so that covers Date, Message-ID, In-Reply-To and  
References, plus their Resent versions. All the headers involving  
<mailbox>es need further work, as does Received. Naturally, header-names  
remain in ASCII on the wire (nobody cares if MUAs display them to users in  
translation).

I agree that you have mentioned most of these, but the only way to do it  
properly is to include the revised ABNF that will fix it precisely (which  
brings up the interesting question of how much UTF-8, if any, you let into  
the "Obsolete Syntax").

OK, this ABNF does not need to be written just yet, but it needs to be  
made clear that it _will_ be done once we have a proper WG to do it.  
Moreover, I am in a position to help with that, because at one time the  
Usefor WG was going to introduce UTF-8 headers as the norm (we backed out  
because of the difficulty of gatewaying into email, but we are scheduled  
to return to the matter as an Experimental Protocol if/when we have  
completed our primary tasks). Most of the ABNF changes we made are  
immediately applicable to the present work, so I can dig them out of our  
earlier drafts and present them here when the proper time to do so comes.

Then, turning to MIME, we can probably allow UTF-8 in <value>s. but not in  
<token>s (that gives us the Content-Type and Content-Distribution headers,  
but definitely not Content-Transfer-Encoding) and after that there are the  
List-* header used by mailing lists, many of which use URIs which could  
usefully be upgraded to IRIs.

Turning to section 2.1, you _appear_ to be saying that everything MUST be  
in UTF-8, implying that encoding as in RFC 2047/2231 and in IDNA is  
forbidden. Looking closely, I don't think that is what you meant to say,  
but it could do with some rewriting. My understanding is:
    a) Anything appearing naked in a header MUST be in UTF-8 (thus  
excluding naked use of any other charsets whatsoever).
    b) If you really want to use some other charset (and we would rather  
you didn't), then you MAY encode it using RFC 2047/2231, and indeed you  
MAY continue to use use those encodings even for stuff in UTF-8 as in  
present usage, and you MAY show domains either in UTF-8 or in punycode.
    c) That leaves <local-part>s, and I think we have work to do on those,  
which I intend to raise in another thread.
    d) There are also issues of normalization to be addressed (NFC, NFKC,  
stringprep, etc) wherever UTF-8 is allowed. But primary discussion of that  
really goes with <local-part>s, and we can see afterwards whether it needs  
mention in other contexts.

Now we are down to the niggles:

3. Since "alternative" addresses are to be provided in the SMTP extension,  
there needs to be some method for the author to specify (in or around the  
To: header) what these alternate recipient addresses are, so that the MUA  
can provide them to the SMTP (whatever) server. Likewise, he needs to be  
able to say (in or around the Reply-To: header) both addresses to be used  
to get back to him. I have not heard any proposals on this yet, so it  
needs to be noted as an omission to be attended to. I can imagine  
solutions involving tinkering with the <addr-spec> syntax, and solutions  
involving extra headers (as Paul Hoffman proposed), and I can see nasty  
snags with both of them :-( .

4. I see two mentions of encoding in quoted-printable (presumably implying  
the use of RFC 2047). It should be noted that RFC 2047 also allows base64  
for that purpose.

5. RFC 2822 had no need to distinguish between characters and octets.  
However, when it comes th the 78 character limit that RFC 2822 recommends  
for line lengths, then I think we have to make it clear that it means  
_characters_ (since it relates to what can be seen in one line of a  
reasonable window), whereas the required limit of 998 needs to be  
expressed in _octets_, because it relates to the maximum size of buffers  
that you might need to allocate. See section 4.

6. And finally, we need to decide whether to follow the RFC 2822  
convention of talking of "header fields". of whether to speak, as Yeh's  
draft does, of "headers". Usefor used to do it that way, but it was  
decided recently (against my advice) to follow the strict RFC 2822  
terminology, so I had to go through all the drafts I was responsible for  
and change them. I shall be most peeved if you are allowed to get away  
with making this change :-) .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Jeff Yeh" <[email-masked]>
Date: Fri, 21 Oct 2005 18:45:51 +0800

Dear Charles:

Thank you for your comment.
Please check my response in the content.

Jeff Yeh

> then see 
> <http://www.imc.org/ietf-imaa/mail-archive/msg00867.html> which I
> 
(snip)

Actually, we had discuss about this part on our last meeting in Beijing.
The situation is that we currently focus on the protocol of transporting
I-email.  And this draft specified the basic idea of using UTF8 header to 
carry Internationazed data. We will have to figure out whether our to 
"updating" 2822 or just as "extending" it, sooner or later.

> earlier drafts and present them here when the proper time to 

Totally agree, but which means a lot of content to add.  That will be great
If you can provide any useful materials.

> mention in other contexts.

a), b) yes.
d) no, normalization and stringprep should not performed.  Some reason
will be described in [history] document, which is not published yet.  But
you 
may refer to draft-klensin-emailaddr-i18n-03.txt, it has most content about
that.

> snags with both of them :-( .

Yes, it's really tough problem.  We are trying to solve this problem in 
The [downgrade] document.  The 00 version I-D had already come out.
But I think it will be a major revise in our next meeting.

> that you might need to allocate. See section 4.

4 and 5
Agree, will update in next version.

> Charles H. Lindsey ---------At Home, doing my own 
> thing-------------------
> -----
> Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: 
> http://www.cs.man.ac.uk
> /~chl
> Email: [email-masked]      Snail: 5 Clerewood Ave, 
> CHEADLE, SK8 3JU,
>  U.K.
> PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 
> 64 7E 14 A4 
> AB A5
>