Subject: [EAI] draft-ietf-eai-utf8headers-02 : alt-address again

From: Kari Hurtta <[email-masked]>
Date: 01 Nov 2006 19:49:54 +0200

This is perhaps too late, but anyway I considered
that alt-address issue again.

There is:

   angle-addr     =  [CFWS] "<" utf8-addr-spec [alt-address] ">" [CFWS]

   utf8-addr-spec =  utf8-local-part "@" utf8-domain

   utf8-local-part=  utf8-dot-atom / quoted-string / obs-local-part

   utf8-domain    =  utf8-dot-atom / domain-literal / obs-domain

   alt-address    =  [CFWS] "[" addr-spec "]" [CFWS]

RFC 2822 have:

addr-spec       =       local-part "@" domain

local-part      =       dot-atom / quoted-string / obs-local-part

domain          =       dot-atom / domain-literal / obs-domain

domain-literal  =       [CFWS] "[" *([FWS] dcontent) [FWS] "]" [CFWS]

dcontent        =       dtext / quoted-pair

dtext           =       NO-WS-CTL /     ; Non white space controls

                        %d33-90 /       ; The rest of the US-ASCII
                        %d94-126        ;  characters not including "[",
                                        ;  "]", or "\"

So there is possible following header fields:

   From: Käyttäjätuki <Käyttäjätuki@[something]>
   From: Käyttäjätuki <Käyttäjätuki@[something][support@[something]]>
   From: Käyttäjätuki <Käyttäjätuki@[something][[email-masked]]>
   From: Käyttäjätuki <support@[something]>

How easy it is to parse ?    This is un-ambiguous, but still quite dangerous.

That dcontext syntax is quote similar to quoted strings:

qtext           =       NO-WS-CTL /     ; Non white space controls

                        %d33 /          ; The rest of the US-ASCII
                        %d35-91 /       ;  characters not including "\"
                        %d93-126        ;  or the quote character

qcontent        =       qtext / quoted-pair

quoted-string   =       [CFWS]
                        DQUOTE *([FWS] qcontent) [FWS] DQUOTE
                        [CFWS]

So [something]  is probably parsed as one token. Similar than 
"something" is parsed. Both allow most of characters and both use
\ for escaping.

That alt-address implies quite different kind parsing.

I'm not sure that this is problem, but this do not look like most easiest 
syntax to write parser for.

Possible other possibilities:

1)

   angle-addr     =  [CFWS] "<" utf8-addr-spec ">" [CFWS] [alt-address]

   alt-address    =  [CFWS] "[" addr-spec "]" [CFWS]

this was suggested by someone other, but that do not look much better if
[ ] are used.

2)

   angle-addr     =  [CFWS] "<" utf8-addr-spec [alt-address] ">" [CFWS]

   alt-address    =  [CFWS] ',' addr-spec  [CFWS]

3)

   angle-addr     =  [CFWS] "<" utf8-addr-spec [alt-address] ">" [CFWS]

   alt-address    =  [CFWS] ';' addr-spec  [CFWS]
 
4)

   angle-addr     =  [CFWS] "<" utf8-addr-spec [alt-address] ">" [CFWS]

   alt-address    =  [CFWS] "<" addr-spec ">" [CFWS]

perhaps that is not so bad as it looks?

5)

   angle-addr     =  [CFWS] "<" utf8-addr-spec ">" [CFWS] [alt-address]

   alt-address    =  [CFWS] "<" addr-spec ">" [CFWS]

Note that this all possibilities avoid characters used on atext.

/ Kari Hurtta

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Kari Hurtta <[email-masked]>
Date: 01 Nov 2006 20:28:30 +0200

> Note that this all possibilities avoid characters used on atext.

It occured to me that there is ALSO possible to use unicode characters > 127
assuming that particular character is removed from allowed utf8-addr-spec
characters.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 01 Nov 2006 22:27:49 +0100

> How easy it is to parse ?    This is un-ambiguous, but still quite 

Easy, I don't see the danger here, <dot-atom> can't be empty.  
The [something] is always a <domain-literal>.

>    alt-address    =  [CFWS] "[" addr-spec "]" [CFWS]

That has two adjacent [CFWS], you'd have to eliminate one -
the prose in 2822 talking about adjacent CFWS is tricky (in
essence saying that they're unintended, and MUST NOT result
in <CRLF><WSP><CRLF> anywhere)

Extracting the alt-addr from the angle brackets makes me
nervous.

>    alt-address    =  [CFWS] ',' addr-spec  [CFWS]

Comma is used in the obs-domain-list of an obs-angle-addr,
I'm not sure if that's a good idea.

>    alt-address    =  [CFWS] ';' addr-spec  [CFWS]

Semicolon is IMO better than comma.

> perhaps that is not so bad as it looks?

No, it's interesting.
  
>    alt-address    =  [CFWS] "<" addr-spec ">" [CFWS]

That's again the case of an extracted alt-address, and
makes me again nervous - even more than above :-)

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 02 Nov 2006 06:59:39 +0200

Frank Ellermann <[email-masked]> writes in gmane.ietf.ima:

> Kari Hurtta wrote:
>  
> >    From: KÃ¤yttÃ¤jÃ¤tuki <KÃ¤yttÃ¤jÃ¤tuki@[something]>
> >    From: KÃ¤yttÃ¤jÃ¤tuki <KÃ¤yttÃ¤jÃ¤tuki@[something][support@[something]]>
> >    From: KÃ¤yttÃ¤jÃ¤tuki <KÃ¤yttÃ¤jÃ¤tuki@[something][[email-masked]]>
> >    From: KÃ¤yttÃ¤jÃ¤tuki <support@[something]>
>  

Yes. It is un-ambiguous. I was more considered how tokenizer can be used.

   Käyttäjätuki <Käyttäjätuki@[something][support@[something]]>

Maybe someone want tokekize as

        Käyttäjätuki
        WSP
        <
        Käyttäjätuki
        @
        [something]
        [
        support
        @
        [something]
        ]

On there was problem   [   versus [something]   selection/conflict.

That is easy if top-down parser is used, but not otherwise.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 02 Nov 2006 16:56:26 +0100

Kari Hurtta wrote:
 
> I was more considered how tokenizer can be used.

What would you like better ?  I'm not sure about the proposals with
the alt-address outside of <L@D> i.e. behind the ">", and about a
comma as separator as in <L@D, A@B>.  IIRC John didn't like to add
"{" and "}".  You had more proposals:  <L@D <A@B>> and <L@B; A@B>

If that's later inserted as is in a cc= or similar of a 2368ter 
mailto: using semicolon could be a bad idea.  Folks try to replace
"&" in query parts by ";", and complex rules where some semicolons
have to be percent-encoded and others not aren't attractive.  

If that makes sense <L@D <A@B>> is an alternative for <L@D [A@D]>.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 02 Nov 2006 18:54:04 +0200

> If that makes sense <L@D <A@B>> is an alternative for <L@D [A@D]>.

That  <L@D <A@B>>  seems  workable.

( But there maybe something what I do not have thinked. 
  
  Someone may perhaps reduce <L@D <A@B>>  to <A@B>
  but I do not think that this is problem.
)

/ Kari Hurtta

---

From: Jeff <[email-masked]>
Date: Fri, 03 Nov 2006 10:09:30 +0800

>   but I do not think that this is problem.
In RFC 2822, it reads:

mailbox         =       name-addr / addr-spec

name-addr       =       [display-name] angle-addr

angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr

So, "From: [email-masked]" is good.
I'm not sure if this make confuse under a case like.
From: [email-masked] <[email-masked]>   # angle-addr with display name
to
From: [email-masked] <[email-masked]>   # only addr-spec, no 
display name, have alt-address

I like the idea to move "alt-addr" out "<" and ">". :)

Jeff Yeh
-TWNIC

---

From: Jeff <[email-masked]>
Date: Fri, 03 Nov 2006 10:31:17 +0800

Sr

Jeff wrote:
>> ( But there maybe something what I do not have thinked.     Someone 
>> may perhaps reduce <L@D <A@B>>  to <A@B>
Sorry, miss reading on <L@D <A@B>>.
But the possible situation still comes up.

Jeff

---

From: Kari Hurtta <[email-masked]>
Date: 03 Nov 2006 07:13:20 +0200

Jeff <[email-masked]> writes in gmane.ietf.ima:

> >>> have to be percent-encoded and others not aren't attractive. If
> >>> that makes sense <L@D <A@B>> is an alternative for <L@D [A@D]>.
> >>>
> >>

Syntax discussed was

Notice that alt-address is only defined for angle-address. 

Also

  From: [email-masked]

is NOT allowed.   utf8-addr-spec   requires '<' and '>'.

That means

  From: <[email-masked]>

On that it is possible to use < > also on alt-address

  From: <[email-masked] <[email-masked]>>

> Jeff

On my one possible alternatives there was  "alt-addr" outside 
of "<" and ">". That was 

    angle-addr     =  [CFWS] "<" utf8-addr-spec ">" [CFWS] [alt-address]
    alt-address    =         "<" addr-spec ">" [CFWS]

This was not liked. It will mean

  From: <[email-masked]> <[email-masked]>

I wrote these examples/alternatives because I suspected that
'[' ']' causes problem for writing tokenizer. 

This is different thing:

  From: [email-masked] <[email-masked]>

[email-masked]    is 'display-name'  :-)

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 03 Nov 2006 10:56:06 +0100

Kari Hurtta wrote:

> "alt-addr" outside of "<" and ">". That was

>   From: <KÃ¤yttÃ¤jÃ¤[email-masked]> <[email-masked]>

Yes, maybe I'm only paranoid.  If some spec. uses something like...

    weird-addr-lists = 1*angle-addr

...then it's not more clear what's an <alt-address> in such lists.

Maybe no spec. does this at the moment, but it's an "obvious" idea
that won't work as expected for the syntax shown above.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 03 Nov 2006 16:49:27 +0200

<...>
> perhaps that is not so bad as it looks?

   ( this alternative is best so far? )
 
>    alt-address    =  [CFWS] "<" addr-spec ">" [CFWS]

6)
     angle-addr     =  [CFWS] "<" utf8-addr-spec ">" [CFWS] [alt-address]
     alt-address    =  addr-spec [CFWS] 

7)

     angle-addr     =  [CFWS] "<" utf8-addr-spec [alt-address] ">" [CFWS]
     alt-address    =  [CFWS] ':' [CFWS] addr-spec  [CFWS]

  Perhaps not good.

> Note that this all possibilities avoid characters used on atext.

specials        =       "(" / ")" /     ; Special characters used in
                        "<" / ">" /     ;  other parts of the syntax
                        "[" / "]" /
                        ":" / ";" /
                        "@" / "\" /
                        "," / "." /
                        DQUOTE

There is not a lot of characters (if unicode characters > 127
are not used.)

/ Kari Hurtta

---

From: "abel" <[email-masked]>
Date: Thu, 23 Nov 2006 12:05:56 +0800

I read the sendmail source about sendmail/header.c for this issue
the chompheader() function will modify the Header value about the mail
address

An EAI mail  send headers are (such as To) :

To: "許乃文(e)" <[email-masked]<[email-masked]>>,
        "测试(e)" <[email-masked]<[email-masked]>>

They become :

To: "\"許乃文(e)\" <[email-masked]"<[email-masked]>,
        "\"测试(e)\" <[email-masked]"<[email-masked]>

If EAI use '[' ']' then without this issue , even if "<UTF8@[IP]
[ald-address]>" transmited.
I think "<L@D <A@B>>  would have some side effects on MTAs or IRI.

Abel

---

From: Kari Hurtta <[email-masked]>
Date: Thu, 23 Nov 2006 08:22:47 +0200 (EET)

>>> abel
[ Charset UTF-8 unsupported, converting... ]
> To: "???(e)" <???@??.twnic.tw<[email-masked]>>,
>         "??(e)" <??@test1.qmail.asrc.cn<[email-masked]>>
> 
> They become :
> 
> To: "\"???(e)\" <???@??.twnic.tw"<[email-masked]>,
>         "\"??(e)\" <??@test1.qmail.asrc.cn"<[email-masked]>
> 

Yes, that means that sendmail needs modify sendmail/header.c 
before it can announce UTF8SMTP

Of course that it is expected that any parser do not handle that
extended syntax (where there is fallback address), if it do not support UTF8SMTP

Problem is that extended syntax must dot leak outside of UTF8SMTP environment.
On SMTP this is easy.

On case where mails are readed from local spool (/var/mail/{user} for example)
by MUA, is difficult.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: Thu, 23 Nov 2006 08:48:13 +0200 (EET)

>>> Kari Hurtta

Perhaps most logical solution is use
            /var/mail/{user}.utf8                for UTF8SMTP messages
and         /var/mail/{user}			 for traditional messages

however that is not on scope of IETF specifications ?

/ Kari Hurtta

---

From: Yangwoo Ko <[email-masked]>
Date: Thu, 23 Nov 2006 17:50:44 +0900

> however that is not on scope of IETF specifications ?

I cannot but agree with you that we need a way to differentiate them
because stored message is not protected by ESMTP.

Differentiation is made possible based on information found either
in mail message itself (e.g. header) or out side of it (e.g. as shown
in your example). If we choose to go with the former, it will be
within the scope of IETF. Otherwise, it is as you said.

---

From: Jeff <[email-masked]>
Date: Thu, 23 Nov 2006 17:26:30 +0800

> On case where mails are readed from local spool (/var/mail/{user} for example)

I think Abel is trying to say that the cost on updating parser to fit
<UTF8 <ASCII>> syntax is more complicate then <UTF8 [ASCII]>. Since '<' 
and '>' are VERY special token in email address representation. I 
believe not only Sendmail had this problem, the other mail server also 
does. If I'm mail software vendor who trying to provide EAI feature, it 
is better for me update as few code as possible.
Not un-solvable but deployment concern I think.

Jeff Yeh
-TWNIC

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 23 Nov 2006 05:42:37 -0800

Jeff wrote:
>> Yes, that means that sendmail needs modify sendmail/header.c before 
>> it can announce UTF8SMTP
>>
> <UTF8 <ASCII>> syntax is more complicate then <UTF8 [ASCII]>. Since 
> '<' and '>' are VERY special token in email address representation. I 
> does. If I'm mail software vendor who trying to provide EAI feature, 
> it is better for me update as few code as possible.
Personal opinion: It is a Good Thing that things break as soon as 
possible if these formats leak.

The worst possible situation is where things leak, and stuff goes on 
working in an "almost reasonable" way that is just broken enough to 
cause problems.

                     Harald

---

From: Kari Hurtta <[email-masked]>
Date: 23 Nov 2006 16:23:59 +0200

> I think "<L@D <A@B>>  would have some side effects on MTAs or IRI.

Independent on what separators are used for alt-address
crackaddr() on sendmail/headers.c breaks also.  It works as following:

     1) Header address value
                phrase <address>   
        is rewritten to
                phrase <$g>

     2)  phrase <address>   passed to rulsets

     3)  result is assigned to $g
     
         and   phrase <$g>
         is evaluted

     That is routine which does address rewriting.

     For handling alt-address pass 1) need to be probally
     to be 

          phrase <address <fallback>>
     is rewritten to
          phrase < ${g_int} <$g>>

     and so on.   Issue is same if [ ] is used for separator
     of alt-address on here.   Breakage of [] may be more hidden
     but it is on here.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 23 Nov 2006 16:33:44 +0200

> <UTF8 <ASCII>> syntax is more complicate then <UTF8 [ASCII]>. Since

There is cost on [ ]  also because syntax inside of [ ] is unstructured
on RFC 2822. In other words [ ] works like " "      (sendmail do not
parse [ ] as quoted, but some others may do. At least my MUA seems
treate [ ] as one syntactic unit.    Compare syntax of
'domain-literal' and 'quoted-string' on RFC 2822. ) 

/ Kari Hurtta

---

From: John C Klensin <[email-masked]>
Date: Thu, 23 Nov 2006 10:09:38 -0500

--On Thursday, 23 November, 2006 17:50 +0900 Yangwoo Ko
<[email-masked]> wrote:

>...
>>> On case where mails are readed from local spool
>>> (/var/mail/{user} for example) by MUA, is difficult.
>> 
>>             /var/mail/{user}.utf8                for UTF8SMTP
>>             messages and         /var/mail/{user}			 for
>> traditional messages
>> 
>> 

I think not.  In order to manage mail, delivery systems and
message store interfaces regularly add headers and other
supplemental information that are not specified in the
standards.  Things like local-store expiration dates,
indications about whether particular messages have been
downloaded via POP3 and when, etc., are particularly common.
The standards cover message formats on the wire; once something
is delivered and goes into the message store, the message store
management systems are free to invent supplemental information
and identify it in any way desired (by storage location,
otherwise out of band, or in-band) that suits their purposes.
And none of that is traditionally within IETF scope.

regards,
    john

---

From: Kari Hurtta <[email-masked]>
Date: 23 Nov 2006 17:22:51 +0200

> within the scope of IETF. Otherwise, it is as you said.

Differiantion is not ennough IMHO.   Header field on mail message itself
is not recognized by MUAs which are unaware of UTF8SMTP
 
/ Kari Hurtta

PS. There is

RFC1343    A User Agent Configuration Mechanism for Multimedia Mail Format Information 

RFC1508	   Generic Security Service Application Program Interface 

RFC1509	   Generic Security Service API : C-bindings 

RFC1823	   The LDAP Application Program Interface 

RFC1961	   GSS-API Authentication Method for SOCKS Version 5 

...

RFC2133    Basic Socket Interface Extensions for IPv6 

RFC2292	   Advanced Sockets API for IPv6 

RFC2553	   Basic Socket Interface Extensions for IPv6 

RFC3493	   Basic Socket Interface Extensions for IPv6 

RFC3542	   Advanced Sockets Application Program Interface (API) for IPv6 

RFC4584	   Extension to Sockets API for Mobile IPv6

which either do not look like scope of IETF. So it is not so clear
what is on scope...

:-)

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 23 Nov 2006 15:31:04 -0000

> the chompheader() function will modify the Header value about the mail

But sendmail is known to be the most difficult existing MTA to convert to  
UTF8SMTP, because it is not yet 8bit-clean for its headers.
>         "\"测试(e)\" <[email-masked]"<[email-masked]>

But, oddly enough, that has produced a To header that would cause the  
email to be sent to the right place. And also returned to the right place  
if it works for Reply-To. A fortunate accident maybe, but don't dismiss it  
entirely.

Of course, a non-UTF8SMTP MTA should never have been allowed to see that  
header in the first place, but accidents happen :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Kari Hurtta <[email-masked]>
Date: 23 Nov 2006 17:41:04 +0200

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> But sendmail is known to be the most difficult existing MTA to convert
> to  UTF8SMTP, because it is not yet 8bit-clean for its headers.
> >
> > To: "許��(e)" <許��@�件.twnic.tw<[email-masked]>>,
> > To: "\"許��(e)\" <許��@�件.twnic.tw"<[email-masked]>,
> email to be sent to the right place. And also returned to the right
> place  if it works for Reply-To. A fortunate accident maybe, but don't
> dismiss it  entirely.

   "\"許��(e)\" <許��@�件.twnic.tw"<[email-masked]>

becomes

   "\"許��(e)\" <許��@�件.twnic.tw"<$g>

on crackaddr() on sendmail/headers.c, I think

And rulesets translate
   "\"許��(e)\" <許��@�件.twnic.tw"<[email-masked]>

to

  [email-masked]

i think

so it comes $g = [email-masked] 

:-)

> Of course, a non-UTF8SMTP MTA should never have been allowed to see
> that  header in the first place, but accidents happen :-( .
> 
> --

---

From: Kari Hurtta <[email-masked]>
Date: 23 Nov 2006 20:27:22 +0200

> to  UTF8SMTP, because it is not yet 8bit-clean for its headers.

That parts is already done. On 8.14 there is:
|        Header field values are now 8 bit clean.  Notes:
|                - header field names are still restricted to 7 bit.
|                - RFC 2822 allows only 7 bit (US-ASCII) characters in
|                  headers.

(quote is from release notes of sendmail 8.14.0.Beta1)

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 24 Nov 2006 11:35:53 -0000

> (quote is from release notes of sendmail 8.14.0.Beta1)

OK, that is good news, but it is still only a beta. But it does mean that  
sendmail.cf could, in principle, be hacked to accept alt-addresses whether  
with <...> or [...] (the hack wouldn't be a pretty sight, but sendmail  
hacks never are :-( ).

---

From: "abel" <[email-masked]>
Date: Mon, 27 Nov 2006 12:07:35 +0800

> with <...> or [...] (the hack wouldn't be a pretty sight, but sendmail
This is a good news, but on the other we do not know how the MTA is !
Because too often use in "<>",
I think we have the <A<B>> or  the "<A[B]>" For the MTA will run a number of
conflicts,
But I personally feel that the "[]" the problem should be smaller than "<>"
for eai ALT-ADDRESS identifiy,

But the problem may be more obvious IRI,
mailto:[email-masked]?cc=[email-masked]&body=hello

to be:
mailto:[email-masked]<ALT-ADDRESS>?cc=[email-masked]<...>&body=hello
or:
mailto:[email-masked][ALT-ADDRESS]?cc=[email-masked][....]&body=hello

I think this is part  "<>" of the impact is greater than the "[]".This is
what we need to consider and attention.

Abel

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Nov 2006 06:12:32 +0100

abel wrote:

 [mailto-IRI]
 
> I think this is part  "<>" of the impact is greater than the "[]".

With URLs it's clear, both <> and [] have to be percent-encoded.

For IRIs 3987 doesn't say anything special about [], I'd guess that
[] still have to be percent-encoded.  For <> 3987 has an obscure MAY,
also for some other characters like space.  This MAY in RFC 3987 is
a mystery for me, so far I hope that Martin silently removes it when
3987 goes to DS, removing options is allowed.

Frank

---

From: Martin Duerst <[email-masked]>
Date: Mon, 27 Nov 2006 16:03:10 +0900

At 14:12 06/11/27, Frank Ellermann wrote:
>[] still have to be percent-encoded.

Yes.

>For <> 3987 has an obscure MAY,

The historical background is that at one time, a draft of the IRI spec
allowed printable ASCII not allowed in URIs. The reason for this was
mainly XPointer, which is awkward to write without that provision.
Later discussions moved IRI towards having exactly the same syntax
for ASCII as URIs. Quite some specs in the XML area use something
that's essentially the same as an IRI, but need this MAY. I can
immagine wording it more narrowly, but at the moment, I don't expect
it to disappear. But it shouldn't be used for new protocol elements.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 27 Nov 2006 17:51:43 +0100

Martin Duerst wrote:

 [3987 3.1 "IRIs MAY also deal with the printable characters in US-ASCII
            that are not allowed in URIs"]
> it to disappear. But it shouldn't be used for new protocol elements.

Thanks for info.  I hope that "shouldn't be used for new protocols"
also includes the future mailto-2368ter, unless you desperately need
it for an alt-address construct.

Frank