Subject: Re: [EAI] Comments on draft--eai-utf8headers-00

From: Jeff Yeh <[email-masked]>
Date: Mon, 03 Jul 2006 17:53:01 +0800

Thank you for your precious comment.
I will fix the editorial change.
And some response below.

Jeff

Charles Lindsey wrote:
>    Sending MUAs that follow this protocol MUST create all header fields
>    encoded in UTF-8.  No other direct encodings are allowed.
>
> I don't see why stuff encoded by RFC 2047 should not be allowed. Not 
> particularly sensible if UTF8 is available, but you might, for 
> example, be replying to a message which already had RFC 2047 in its 
> Subject, and the easiest thing to do is just stick "Re:" on the front 
> of it (or maybe not even that).
Here we said "direct encoding" is some local encoding like Big5.
And I agree there's no bad for using RFC 2047. But, why use 2047 stuff 
if it can handle UTF8 natively?
FWIW, it will be good to have pure UTF8 mail environment in the future.
I'll put some note about RFC 2047 on this section.
>
> 5.  Identification of internationalized email
>
>    i-Email: 1.0
>
>    [Note in draft: There should be more useful information can be place
>    in the new header field. ]
>
> Yes indeed. I have suggested the following syntax:
>
>    header-content = "Header-Content:" [CFWS] content-code [CFWS]
>                         *( ";" parameter ) CRLF
> {point out that <parameter> brings some more [CFWS] with it)
>    content-code = "Utf8smtp" / "ASCII" / "Downgraded"
> which really needs to be followed by
>       Header-Language = "Header-Language" ":" [CFWS] Language-List
>       Language-List = Language-Tag [CFWS]
>                          *("," [CFWS] Language-Tag [CFWS])
> based in RFC 3282.
Good idea. Will put in next version.

>
> Of course, the following would need to be changed to suit. But there 
> are also some other issues:
>
>    o  The "i-Email" header field MUST be inserted by the originating
>       MUA.
>    o  The "i-Email" header field MUST be inserted, along with Return-
>       path, by the final delivery MTA if not presented.
>
> But how will the final delivery MTA know whether it is needed?
>
>    o  The "i-Email" header field, if present, MUST be removed as part of
>       any downgrading process that eliminates the UTF-8 header
>       information.
>    o  MTAs MAY check for duplicates of the "i-Email" header field and
>       eliminate all but one of them.  However, if a receiving MUA
>       encounters more than one of these headers, it SHOULD simply ignore
>       any excess ones.
>
> I think it is usually a bad idea for intermediate MTAs to try to 
> "correct" perceived mistakes in messages passing through them. If the 
> mistake is bad enough to break the system, then bounce it; othersise 
> pass it on. Experience with News suggested that systems which tried to 
> "correct" errors usually introduced more errors than they removed :-( .
In general case, MTA send what they got. Since sending MUA put the new 
header on when sending EAI messages.
However, if we like to make sure the new header is present.  Then this 
kind of process can not be avoided.
Recently we (TWNIC) are trying to implement a testbed of EAI. We found 
that we still have to look over all the possible place for UTF8, 
punycode and RFC 2047 stuff. I'm not that sure if this header is very 
much useful.
> 6.2.  Syntax extend from RFC 2822
>
> It's nice to see you adopted the syntax I suggested. Unfortunately, I 
> have spotted some problems :-( .
>
>    unstructured =  1*( [FWS] utext ) [FWS]
>
> Yes, that is the rule as it would be for Netnews. For email, it can be 
> empty. So:
>
>    unstructured =  *( [FWS] utext ) [FWS]
>
>
>    atext        =  ALPHA / DIGIT /
>                    "!" / "#" /     ; Any character except
>                    "$" / "%" /     ; controls, SP, and specials.
>                    "&" / "'" /     ; Used for atoms
>                    "*" / "+" /
>                    "-" / "/" /
>                    "=" / "?" /
>                    "^" / "_" /
>                    "`" / "{" /
>                    "|" / "}" /
>                    "~" /
>                    UTF8-xtra-char
>
> Here we have a problem, because that introduces Utf8 in atoms and 
> dot-atoms, which is fine for addrs-specs (where we waant it), but also 
> allows them in Message-ID and Received headers (where we most 
> certainly don't want it).
>
> There are two ways out of this:
> 1. use <strict-atom> etc for Message-ID and Received.
> 2. define a special <utf8-atom> etc for addr-spec.
>
> I leave it to you which to do, but I can see some benefit in using 
> approach #2, since we are going to tinker with the syntax of addr-spec 
> anyway, in order to include <alt-address> and "atomic".
Thank you precious information.  I'll try to do that in next version.
>
>    alt-separator  =  [FWS] "," [FWS]
>
> candidates for this are ",". ":" and "|", where "|" would require a 
> special syntax in place of <addr-spec> within <angle-addr> (though it 
> wold be OK within <alt-address>). I preter "|" over ":" and ":" over 
> "," (there are already too many other uses of "," within address-lists 
> and mailbox lists).
Yes, indeed. We found that "," leads more complexity on the mailbox 
parsing. And also "|" leads some potential confuse to pipe, some mail 
account are directly pipe to some shell script. The ":" is the separator 
of header name and header body, also possible to raise more complexity 
on parsing.  So, we are thinking about using "{", "}" to represent 
alt-address/atomic information. This will look like,
    "DISPLAY-NAME" <UTF8@UTF8 { ASCII@ASCII }>
This is obviously  better for programmers to parse the parameter, but 
I'm not sure if there's other limitation on "{" and "}".
>
> 7.  Additional issue
>
> I think you need to deal with the MIME headers here. In particular, 
> <value> (as used in <parameter>s) needs to allow Utf8 within it (with 
> RFC 2231 as the downgrade mechanism). <Token>s (being identifiers) 
> ahould, of course, remain in ASCII.
I don't quite get this problem. If a header value encoded with MIME, 
then it should be ASCII. Or I read your words wrong, please advice.
>
> 7.1.  Mailing list header fields
>
>    All mailing list and mail redistribution related header fields may
>    need further investigation.
>
> I agree. RFC 2369 and RFC 2919 are the relevants specs. RFC 2369 uses 
> URIs for everything, and these would naturally become IRIs (RFC 3987), 
> which already have a well-defined downgrade to URIs. I see that RFC 
> 2919 makes use of <phrase> and <dot-atom-tsxt> (so we need to decide 
> which kind of <dot-atom-text> that would be).
>    Because UTF-8 often requires several octets to encode a single
>    character, internationalized local parts may cause mail addresses to
>    become longer.  Then may possibly make it harder to keep lines in a
>    header under 78 octets.  Lines that are longer than 78 octets (which
>    is a SHOULD specification, not a MUST specification, in RFC 2822)
>    could possibly cause mail user agents to fail in ways that affect
>    security.
>
> I think you want to make it clear that the '78' limit is to be 
> expressed in characters, not in octets, since its purpose is to make 
> the text fit within the commonly used 80-character-wide windows. OTOH, 
> the '998' limit is definitely in octets, since it is related to the 
> size of buffers that implementors need to provide.
>
> There is a further security consideration which can arise if the utf8- 
> and alt- addresses point to totally different people. I am not sure 
> what sort of scam could make use of that, but spammers are, in 
> general, most skilled at exploiting any possible loophole, so I think 
> it needs to be mentioned.
Agree.
>
>
> 9.  IANA considerations
>
> All new headers that you invent need to be registered with IANA, in 
> accordance with RFC 3864. And any headers that you modify need to have 
> their registrations modified, so as to refer to your specification in 
> addition to their current definitions.
Will add in next version..

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 03 Jul 2006 14:42:47 +0100

On Mon, 03 Jul 2006 10:53:01 +0100, Jeff Yeh <[email-masked]> wrote:

>> particularly sensible if UTF8 is available, but you might, for example,  
>> be replying to a message which already had RFC 2047 in its Subject, and  
>> the easiest thing to do is just stick "Re:" on the front of it (or  
>> maybe not even that).

Ah! I had missed the significance of that word "direct". Sure, it would be  
stupid to use RFC 2047 when UTF-8 is available, but since when did people  
not do stupid things? They might be very attached to Big 5 :-).

> header on when sending EAI messages.

There is a general prohibition of multiple copies on the same header in  
RFC 2822, except for a very few exceptions (e.g. Keywords:). That should  
be maintained, but if some poor implementation breaks it, then it is  
better to leave it alone rather than to try to guess which was the "extra"  
one.

>     "DISPLAY-NAME" <UTF8@UTF8 { ASCII@ASCII }>

Yes, I like that.

> then it should be ASCII. Or I read your words wrong, please advice.

The most obvious example is

    Content-Disposition: attachment; filename="some utf8 name"

but there could be others. Currently, you can encode these things  
according to RFC 2231 (which is a mess, but would still be needed for the  
downgrade).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Jeff Yeh <[email-masked]>
Date: Tue, 04 Jul 2006 15:54:18 +0800

This was discussed on previous thread 
http://www1.ietf.org/mail-archive/web/ima/current/msg00432.html. But I'd 
like to here other people's opinion.

In the utf8header document right now, we use ',' as the alt-separator. 
TWNIC is recently working on EAI testbed implementation.We found that 
"," leads more complexity on the mailbox parsing. And also "|" leads 
some potential confuse to pipe, some mail account are directly pipe to 
some shell script. The ":" is the separator of header name and header 
body, also possible to raise more complexity on parsing. So, we are 
thinking about using "{", "}" to represent alt-address/atomic 
information. This will look like,
- "DISPLAY-NAME" <UTF8@UTF8 {ASCII@ASCII}> #EAI with ALT-ADDRESS
- "DISPLAY-NAME" <UTF8@UTF8 {ATOMIC}> #EAI with ATOMIC option
There are a few benefit I can see:
o not in RFC 2822 section 3.2.1 "specials" token
o making no confuse
o easier for program parsing
o can also be used in mailing list

There is another thing we though about, we use SMTP extension and new 
parameter on SMTP protocol.
That looks like (SMTP protocol):
MAIL FROM: <[email-masked]> EAI-PARAMETER=[email-masked]
This is pretty much different to mail headers (2822 header). And make 
the whole EAI solution more complex to implement due to various 
representation of "addr-spec".
What if we use:
MAIL FORM: <[email-masked] [email-masked]}>
We had try this on our testing server, works good.

I put some example after for your reference.
http://eai2.twnic.tw/smtp.png SMTP
http://eai2.twnic.tw/eai-downgrade-oe.jpg Outlook Express
http://eai2.twnic.tw/eai-downgrade-owm.png Openwebmail Display Style
============begin of example===========

Date: Tue, 04 Jul 2006 14:46:21 +0800
Downgraded: From:
	=?UTF8?B?IualiuemjuiRhiIgPOaliuemjuiRhkBlYWkxLnR3bmljLnR3IHthYmVseWFuZ
	0B0d25pYy5uZXQudHd9Pg==?=
From: =?UTF8?B?IualiuemjuiRhiIg?= <[email-masked]>
Downgraded: To:
	=?UTF8?B?IuioseS5g+aWhyBFQUk9YXRvbWljIiA86Kix5LmD5paHQOWPsOe2suS4reW/g
	y50dyB7YXRvbWljfT4sICLokYnlo6vosaogRUFJPWF0b21pYyIgPOiRieWjq+ixqkDlj7D
	ntrLkuK3lv4MudHcge2F0b21pY30+LCAi6JGJ5aOr6LGqIEVBST1BTFQiIDzokYnlo6vos
	apA5Y+w57ay5Lit5b+DLnR3IHthYmVseWFuZ0Bkb3duZ3JhZGUudHduaWMudHd9PiwgIum
	Zs+eOieiQsSBFQUktUGFyYW1ldGVyPUFMVCIgPOiRieWjq+ixqkDlj7DntrLkuK3lv4Mud
	Hcge2VyaW5AdHduaWMubmV0LnR3fT4=?=
To: =?UTF8?B?IuioseS5g+aWhyBFQUk9YXRvbWljIiA=?= <[email-masked]>,
        =?UTF8?B?ICLokYnlo6vosaogRUFJPWF0b21pYyIg?= <[email-masked]>,
        =?UTF8?B?ICLokYnlo6vosaogRUFJPUFMVCIg?= <[email-masked]>,
        =?UTF8?B?ICLpmbPnjonokLEgRUFJLVBhcmFtZXRlcj1BTFQiIA==?= <[email-masked]>
Subject: EAI Downgrade Testing Tue, 04 Jul 2006 14:46:21 +0800
i-Email: 1.0

ABC
這裏是 UTF8 編碼,可以正常看見嗎 ?
==========end of example==========================

Please provide your precious comment.

Best

Jeff Yeh
- TWNIC

---

From: Tony Finch <[email-masked]>
Date: Tue, 4 Jul 2006 12:57:42 +0100

On Tue, 4 Jul 2006, Jeff Yeh wrote:
> TWNIC is recently working on EAI testbed implementation. We found that
> body, also possible to raise more complexity on parsing.

: is also used in header field bodies, in addres groups (which are
outside addr-specs) and in source routes (inside addr-specs). 822
parsing is already quite complicated :-)

> So, we are thinking about using "{", "}" to represent

I think this is not a benefit, because the whole point of the special
characters is that they are used for this kind of thing! Non-special
characters are supposed to have no special syntactic meaning.

> This is pretty much different to mail headers (2822 header). And make the
> whole EAI solution more complex to implement due to various representation of
> "addr-spec".
> What if we use:

I've already said that I support using the same addr-spec syntax at both
the 821 and 822 levels.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
THAMES DOVER WIGHT PORTLAND PLYMOUTH: VARIABLE BECOMING WESTERLY 3 OR 4.
SHOWERS. MODERATE OCCASIONALLY POOR, WITH FOG PATCHES.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 04 Jul 2006 14:21:36 +0100

> o can also be used in mailing list

And you can add to that:

   o doesn't need any [FWS] around it to make it stand out (like '|' might  
have done).

> MAIL FORM: <[email-masked] [email-masked]}>

Much to my surprise, my MUA rendered those chinese characters (at double  
width). The only problem was that the '@' in the middle got lost (looked  
just like another chinese character) :-( .

> We had try this on our testing server, works good.

I think the only problem with this is that it breaks the current RFC 2821  
syntax for parameters to commands. So you will have to fight John on that  
one.

---

From: Tony Finch <[email-masked]>
Date: Tue, 4 Jul 2006 17:23:23 +0100

On Tue, 4 Jul 2006, Charles Lindsey wrote:
> syntax for parameters to commands. So you will have to fight John on that one.

And IMAs don't?!

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOVER WIGHT PORTLAND PLYMOUTH: VARIABLE BECOMING SOUTHWEST 3 OR 4. THUNDERY
SHOWERS. MODERATE OR GOOD WITH FOG PATCHES IN PLYMOUTH.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 04 Jul 2006 17:45:33 +0100

>> syntax for parameters to commands. So you will have to fight John on  

Not in the same way. The present proposal still fits within the basic  
syntax of RFC 2821 commands (just widens the character set a little). John  
tends to be a little protective of RFC 2821 (quite naturally :-).

I would have no problem with the change, but then I am not particularly au  
fait with the finer points of 2821.

---

From: John C Klensin <[email-masked]>
Date: Tue, 04 Jul 2006 17:36:49 -0400

--On Tuesday, 04 July, 2006 17:45 +0100 Charles Lindsey
<[email-masked]> wrote:

> Not in the same way. The present proposal still fits within
> the basic  syntax of RFC 2821 commands (just widens the
> character set a little). John  tends to be a little protective
> of RFC 2821 (quite naturally :-).
> 
> I would have no problem with the change, but then I am not
> particularly au  fait with the finer points of 2821.

I am really less protective of 2821 than I am about the edge
conditions encountered with existing systems and parsers.  2821
is deliberately very specific about the possible syntax of
extensions in order to permit implementations to lexically
analyze/ tokenize SMTP commands without (prior to) establishing
state via whatever options have been offered or accepted.  Note,
in this regard, that several (at least) SMTP-receiver
implementations provide code support for options whose
availability in operation depends on configuration options.  One
can even imagine offering some options to some SMTP-senders and
not others.

Depending on how much implementations take advantage of these
flexibilities built into 2821, this could turn out to be a very
significant change.

In that regard, note that a "never downgrade, bounce if needed"
implementation of our work requires no parser changes if the
alternative address/ atomic info is in parameters.  Depending on
how SMTPext is written, it might not even need to recognize the
parameters themselves (although I'd recommend against that and
recognizing them is fairly trivial).  By contrast, tampering
with what appears between the <...> requires parser changes even
if one does not intend to use the information.

And, FWIW, one of my personal goals in this is to make/ keep it
as simple as possible to support i18n addressing in existing
MTAs that are otherwise 8bit-clean.  Requiring parser changes in
all cases doesn't facilitate that.

So, yes, I'm not happy about the idea.  But only a small
fraction of my unhappiness involves defending 2821.

     john

---

From: Tony Finch <[email-masked]>
Date: Wed, 5 Jul 2006 11:19:22 +0100

On Tue, 4 Jul 2006, John C Klensin wrote:
>
> And, FWIW, one of my personal goals in this is to make/ keep it as
> simple as possible to support i18n addressing in existing MTAs that are
> otherwise 8bit-clean.  Requiring parser changes in all cases doesn't
> facilitate that.

I don't think you can avoid parser changes, because MTAs will have to be
modified to allow top-bit-set characters in forward and reverse paths.
They will in any case have to have modified 2822 parsers so that they can
implement header checks and fix-ups, for message submission mode as well
as for downgrading.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FAIR ISLE: SOUTH OR SOUTHEAST 3 OR 4, OCCASIONALLY 5 LATER. RAIN OR SHOWERS
LATER. MODERATE OR GOOD, WITH FOG PATCHES.