Subject: [EAI] Status of utf8header draft

From: Jeff Yeh <[email-masked]>
Date: Thu, 21 Dec 2006 13:02:02 +0800

Hi all:

While I'm trying to update the header draft, there comes some issues 
(solved/un-solved). Listed below:

* Removing the "Header-Type" header (was UTF8SMTP)
The basic idea to have a "Header-Type" new header is for identifying a 
email is EAI or non-EAI. And the new header could also carry some useful 
information like downgrade information/language tag. However, this kind 
of self-labeling mechanism was found useless as the history of 
"MIME-Version" header. For the robustness of email software, programmers 
usually ignore the "MIME-Version" header, directly check the presents of 
MIME headers and the MIME encoded stuff. This is also happened in 
TWNIC's implementation on EAI testbed, "Header-Type" header is ignored, 
we check the presents of UTF8 instead. Thus, the demand of the 
"Header-Type" is being raise up again.
Although we already have consensus about this point on San Diego 
meeting, it's still necessary to bring on the mailing list for 
everyone's opinion.
<some quote from jabber scribe>
--harald offers choices:
      a) leave UTF8SMTP header field in:one vote
      b) take it out: 10 votes
      c) mime-version 2.0: none
      d) can't decide: 5 votes
--harald: will take proposal to list to take it out
</some quote from jabber scribe>

* The alt-addr syntax
There is a roughly consensus on <UTF8 <ASCII>>, but still some concern 
on parser updating effort. By TWNIC's EAI implementation experience, 
<UTF8 [ASCII]> working just fine with a little bit change on parser. 
Though there is a possible conflict to domain-literal, but the use of 
domain-literal is quite few, I think.

* Trace Field
Solved, will update in the upcoming version.
"Received" must put in ASCII, "for" could ignore (optional). 
"Return-Path" syntax need to update to allow UTF8 characters.

* UTF8 in MIME headers (RFC 2231)
Should be allowed. Will update in the upcoming version.

Did I miss anything related to utf8header draft?

Regards,

Jeff Yeh
-TWNIC

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Kari Hurtta <[email-masked]>
Date: 21 Dec 2006 08:54:14 +0200

Jeff Yeh <[email-masked]> writes in gmane.ietf.ima:

> email is EAI or non-EAI. And the new header could also carry some
> useful information like downgrade information/language tag. However,
> this kind of self-labeling mechanism was found useless as the history
> of "MIME-Version" header. For the robustness of email software,
> programmers usually ignore the "MIME-Version" header, directly check
> the presents of MIME headers and the MIME encoded stuff. This is also
> happened in TWNIC's implementation on EAI testbed, "Header-Type"
> header is ignored, we check the presents of UTF8 instead. Thus, the
> demand of the "Header-Type" is being raise up again.

Leaving out Header-Type is OK, if 
        - correspond new parameter is added to ESMTP 
          (ie.  HEADER=UTF8SMTP compare with BODY=8BITMIME )
        - message/rfc822 is labeled or some other type is used

        - something may required for parameters, which was added
          to Header-Type -header field

          'This messages is downgraded UTF8SMTP message' can
          not be added as ESMTP parameter. It can only be 
          on some header field   (new ESMTP parameter requires that
          UTF8SMTP is supported and if it is, then downgrading
          is not required.)

          Perhaps then is needed
                Downgrade-Type: none         (message is not downgraded (*))
          and
                Downgrade-Type: utf8smtp     (message is downgraded from UTF8SMTP)

          with parameters ?

(*) Downgrade-Type: none   do not indicate that message is UTF8SMTP message.

> "Received" must put in ASCII, "for" could ignore
> (optional). "Return-Path" syntax need to update to allow UTF8
> characters.

OK.

> Should be allowed. Will update in the upcoming version.

OK.

> Did I miss anything related to utf8header draft?

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 21 Dec 2006 17:41:53 -0000

> </some quote from jabber scribe>

How is it that a small subset if this list (those that have funding and/or  
time to travel to IETF meetings) can make "decisions" at those meetings,  
of which the rest are not informed until months afterwards? That is a most  
unsatisf\ctory way of proceeding.

AFAIAC, the Header-Type header field is VITAL, for the following reasons:

1. It is the only way to tell that a message is UTF8SMTP in the case where  
the transport mechanism is _not_ SMTP (a situation which the draft now  
claims to allow).

2. Even when the transport mechanism _is_ SMTP, you cannot tell in the  
case where all the addresses are in pure ASCII; there could still be  
Subject and other headers with UTF8 in them, and there could be embedded  
MIME objects with UTF8 in their headers (e.g. message/rfc822 or filenames  
in Content-Disposition). It would need an SMTP parameter such as  
BODY-UTF8SMTP to indicate that.

3. Once a message has passed "final delivery" (whatever that is), the SMTP  
information is no longer available, but you may still need to know whether  
it is a UTF8SMTP message. Even if IMAP and POP3 contrive to keep this  
information somehow, they are not the only storage mechanisms in use; for  
example, good ol' mbox format is still around, and has no other way to  
store that information.

4. You NEED that information before you can gateway the message to some  
other medium (gatewaying is always dificult in the general case, and the  
other medium may or may not require some downgrade, and even if it doesn't  
it may need to know so that it in turn can gateway it on to something  
else).

5. You NEED that information before you can Resend the message (or maybe  
even forward it, or encapsulate it as a message/rfc822 within something  
else).

6. Your MUA may NEED that information before it can display it (e.g. does  
it need to upgrade it first?), or else you NEED it in order to determine  
whether it requires to be downgraded before submission to the MUA, or even  
whether it needs to be upgraded first.

7. You NEED that information if the message has been DKIM-Signed, so you  
can either upgrade it before checking the signature, or else report  
"Sorry! This signature cannot be verified because of downgrading".

Note that there is a feature in DKIM whereby you can say, in the  
signature, "here is how such-and-such a header was at the point where it  
was signed". You might use that feature to record the original From, etc,  
but more importantly if you used it on the Header-Type (which itself MUST  
NOT be included in the signed hash, because it can change en route) then  
you could detect that the Header-Type was "UTF8" when it was signed but it  
now shows as "downgraded", and so you know exactly where you stand.

OK, there may still be lots of problems with DKIM-Signatures on UTF8SMTP  
messages, and we will need to look at them later on; but in the meantime  
let us not do anything that would make those propblems worse.

> domain-literal is quite few, I think.

I could live with either, but marginally prefer <UTF8 <ASCII>>. But if  
someone can show that one or the other is technically more difficult (and  
I don't think we have seen that yet), then that would settle the matter.

> "Return-Path" syntax need to update to allow UTF8 characters.

2 issues there, I think:

1. You can't have UTF8 in the "for", so you may have to omit the "for" in  
those cases.

2. What about comments? Either you allow UTF8 in them (as we do in  
comments in other headers) and accept downgrading of those cases, or you  
write explicitly that comments in References header fields MUST be in  
ASCII (which still allows RFC 2047 encoded-words if you really need them).

> Should be allowed. Will update in the upcoming version.
The 02 draft seems more or less OK on this.

> Did I miss anything related to utf8header draft?

BUG in <alt-address> (section 6.3)
----------------------------------

An <alt-address> MUST be pure ASCII. Yes?

Currently, you have

    alt-address    =  [CFWS] "[" addr-spec "]" [CFWS]
where
    addr-spec      =  local-part "@" domain
domain is OK (it uses the old dot-atom and old domain-literal) but
    local-part     =  dot-atom / quoted-string / obs-local-part
and
    quoted-string  = [CFWS]
                     DQUOTE *([FWS] qcontent) [FWS] DQUOTE
                     [CFWS]
where
    qcontent       = qtext / quoted-pair
    quoted-pair    = ("\" text) / obs-qp
BUT you have changed both <qtext> and <text> to allow UTF-8 :-( .

Unravelling that is going to be tedious, but here goes:

1. Rename your <qtext> and your <text> as follows:

    utf8-qtext = NO-WS-CTL /     ; all of <text> except
                 %d33 /          ; The rest of the US-ASCII
                 %d35-91 /       ; characters not including "\"
                 %d93-126 /      ; or the quote character
                 UTF8-xtra-char

    utf8-text =  %d1-9 /         ; all UTF-8 characters except
                 %d11-12 /       ; US-ASCII NUL, CR and LF
                 %d14-127 /
                 UTF8-xtra-char

2. Invent utf8-quoted-pair and utf8-qcontent etc.:

    utf8-quoted-pair   = ("\" utf8-text) / obs-qp
    utf8-qcontent      = utf8-qtext / utf8-quoted-pair
    utf8-quoted-string = [CFWS]
                         DQUOTE *([FWS] utf8-qcontent) [FWS] DQUOTE
                         [CFWS]
    utf8-ccontent      = ctext / utf8-quoted-pair / comment

3. Find all the places where <quoted-pair> is used, and see which version  
is now needed:

    comment         = "(" *([FWS] utf8-ccontent) [FWS] ")"
    word            = utf8-atom / utf8-quoted-string
    utf8-local-part = utf8-dot-atom / utf8-quoted-string / obs-local-part

Observe that I also fixed another bug there, because you had not allowed a  
<utf8-dot-atom> in a <display-name> and other such places.

Message-ID
----------

I thought we had always agreed this should remain in ASCII (to allow UTF8  
there might cause all sorts of problems in Netnews unless it were VERY  
heavily normalized). There is some slight mention of this in 6.2, but not  
strong enough. Also a reference to 6.4 there which I do not understand,  
since <mesg-id> does not occur in Received fields, or does it? And, in any  
case, shouldn't it be <msg-id> rather than <mesg-id>?

However, my revised syntax above has magically fixed this problem, since  
<msg-id> was based on <dot-atom-text>, <qtext> and <quoted-pair>. So all  
you now need is a remark such as:

    Observe that none of this revised syntax affects what is allowed in a
    <message-id>, which will still remain in pure ASCII.

<utf8-local-part>s
------------------

Soobok and I have been pressing for something about normalization here. So  
I would like to suggest, somewhere in 6,3:

"Although the syntax allows unfettered use of <UTF8-xtra-char>s in  
<utf8-local-part>s, it should be realized that many combinations of such  
characters (notably those not properly normalized according to the  
appropriate Unicode Normalization Forms  
[http://www.unicode.org/reports/tr15/] will be almost impossible to enter  
using typical keyboard and MUA software. Moreover, accurate conveyance of  
such sequences by means of handwriting or the spoken word is hardly  
possible.

Therefore, those responsible for allocating <utf8-local-part>s for use in  
domains where UTF8SMTP mail addresses are to be recognized SHOULD choose  
them in forms that are normalized at least to level NFC, and preferebly  
NFKC, and they MAY arrange that <utf8-local-part>s in incoming emails are  
correspondingly normalized before attempting to interpret them further."

Message/rfc822
--------------

We discussed this, but never came to any firm conclusion. What I would  
like to see, somewhere in 7.2, is:

"The media type "message/rfc822", as defined in [RFC 2046] is extended to  
allow all messages defined by this document. However, objects conforming  
to that extended media type MUST NOT be used except within UTF8SMTP  
messages, as defined in this document, and thus they will need to be  
downgraded [downgrade document] (or bounced/rejected) whenever the parent  
message is downgraded. {And then our {downgrade document] needs to pick  
this up and to say that this needs to be done

Other niggles
-------------

I think Frank Ellerman has already reported most of the typos in the  
present draft, so I will just mention the paragraph mentioning "required  
fields" in section 5. I don't understand this at all. I think you are  
trying to say that the only <parameter>s (syntax of <header-content>)  
defined by this document is the <header-language-value> that you give, but  
I cannot really deduce that from the paragraph as written, since the term  
"required fields" is nowhere defined. So what is it really all about?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Kari Hurtta <[email-masked]>
Date: 21 Dec 2006 20:12:18 +0200

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> present draft, so I will just mention the paragraph mentioning
> "required  fields" in section 5. I don't understand this at all. I
> think you are  trying to say that the only <parameter>s (syntax of
> <header-content>)  defined by this document is the
> <header-language-value> that you give, but  I cannot really deduce
> that from the paragraph as written, since the term  "required fields"
> is nowhere defined. So what is it really all about?

What was my suggestion. In short draft-ietf-eai-downgrade was meant
to say that if it removes any header field mentioned on "required fields",
downgrading fails.

/ Kari Hurtta

--------------------------------------------------------------------
From: Kari Hurtta <[email-masked]>
Subject: [EAI] draft-ietf-eai-utf8headers-01.txt /
 draft-ietf-eai-downgrade-02.txt suggestion
Newsgroups: gmane.ietf.ima
Cc: Kari Hurtta <[email-masked]>
Date: Sat, 14 Oct 2006 14:48:07 +0300 (EEST)

draft-ietf-eai-utf8headers-01.txt:

   header-content  = "UTF8SMTP:" [FWS] content-code [FWS]
                     *( ";" parameter ) CRLF
   content-code    = "UTF8" / "ASCII" / "Downgraded"
   parameter       = "Header-Language:" [FWS] Language-List
   Language-List   = Language-Tag [FWS]
                     *("," [FWS] Language-Tag [FWS])

following was suggested:

   header-content  = "Header-Type:" [FWS] content-code [FWS]
                      *( ";" parameter ) CRLF

   ; parameter is defined on RFC 2045 

   content-code    = "UTF8SMTP" / "ASCII" / "Downgraded"

   The "header-language" parameter is defined for "Header-Type"
   header-field. Value of "header-language" parameter is defined
   as following (after possible quotation of quoted-string is decode):

   header-language-value = Language-Tag [FWS]
                     *("," [FWS] Language-Tag [FWS])

I also suggest following (almost same was also suggested):

   The "required-fields" parameter is defined for "Header-Type"
   header-field. Value of "required-fields" parameter is defined
   as following (after possible quotation of quoted-string is decode):

   required-fields-value = field-name [FWS]
                     *("," [FWS] field-name [FWS])

   ; field-name is defined on RFC 2822

draft-ietf-eai-downgrade-02.txt:

  Downgrading:
      *  Check if the mail has 'UTF8SMTP' header and its value is
         "UTF8".  Otherwise, downgrading is not required.
      *  Check each header whether it contains UTF-8 characters or not.
         If the header contains UTF-8 characters,
         +  If the header is an address header which is described in
            Section 5.1.1,
            -  Preserve the header in 'Downgraded' header.
            -  Downgrade the header defined in Section 5.1.1.
         +  The other header case, encode the header by [RFC2047] with
            UTF-8 tag.
      *  Change 'UTF8SMTP' header value to "Downgraded".

Suggested changes:

   Downgrading:
      *  Check if the mail has 'Header-Type' header field and 
         its value is "UTF8SMTP".  Otherwise, downgrading is not required.
      *  Check each header field whether it contains UTF-8 characters or not.
         If the header field contains UTF-8 characters,
         +  If the header field is an address header field which is 
            described in Section 5.1.1,
            -  Preserve the header in 'Downgraded' header field.
            -  Downgrade the header field defined in Section 5.1.1.
         +  Otherwise if syntax of header field is known:
            - Encode header elements according of [RFC2047]
              and [RFC2231]
            - If encoding is not possible, preserve the header in 
              'Downgraded' header field and remove original header field.
         -  If syntax of header field is not known
            - Preserve the header in 'Downgraded' header field and 
              remove original header field.
      * If some header field was removed, which is mentioned on
        "required-fields" parameter on 'Header-Type' header field,
        downgrading fails. 
      *  Change 'Header-Type' header field value to "Downgraded".
     
      NOTE: Address header downgrading in Section 5.1.1 may also
            remove header field. If that header field is mentioned 
            on "required-fields" parameter on 'Header-Type' header 
            field, downgrading fails. 
            
draft-ietf-eai-downgrade-02.txt:

    Upgrading:
      *  Check if the mail has 'UTF8SMTP' header and its value is
         "Downgraded".  Otherwise, upgrading is not required.
      *  Decode all 'Downgraded' headers.
         +  Decode header value field string which is [RFC2047] encoded.
         +  If the header is address header described in Section 5.1.1,
            -  Apply address header downgrading to the decoded header.
            -  Remove the header line which is the same with the
               downgraded line.
         +  Remove the 'Downgraded' header.
         +  Add decoded header to mail header.
      *  If each mail header has [RFC2047] encoded part and which
         encoding is "UTF-8", it is a downgraded header, so decode it.
      *  Change 'UTF8SMTP' header value to "UTF8".

Suggested changes:

      *  Check if the mail has 'Header-Type' header field and its value is
         "Downgraded".  Otherwise, upgrading is not required.
         +  Decode header value field string which is [RFC2047] encoded.
         +  If the header is address header field described in Section 5.1.1,
            -  Apply address header field downgrading to the decoded header.
            -  Remove the header field line which is the same with the
               downgraded line.
         +  Remove the 'Downgraded' header field.
         +  Add decoded header to mail header field.
      *  For every mail header field, which syntax is known
         +  If each mail header field has [RFC2047] or [RFC2231] encoded part 
            and which encoding is "UTF-8", it is a downgraded header field, 
            so decode it.
      *  Change 'Header-Type' header value to "UTF8SMTP".

/ Kari Hurtta

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 22 Dec 2006 10:13:34 +0100

Charles Lindsey wrote:
>> The basic idea to have a "Header-Type" new header is for identifying 
>> a email is EAI or non-EAI. And the new header could also carry some 
> How is it that a small subset if this list (those that have funding 
> and/or time to travel to IETF meetings) can make "decisions" at those 
> meetings, of which the rest are not informed until months afterwards? 
> That is a most unsatisf\ctory way of proceeding. 
It's been exactly 6 weeks and one day from the WG meeting to today, but 
yes, the chairs have been remiss in getting the minutes posted and reviewed.

My apologies.

                    Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 22 Dec 2006 14:55:24 -0000

> to say that if it removes any header field mentioned on "required  

Ah! I understand now. Just needs a little more explanation in utf8headers,  
then.