Subject: [EAI] Issues open against SMTPEXT and UTF8HDR documents

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 19 May 2007 08:51:10 +0200

Given that we have known open issues against the documents, the chairs have 
concluded that it's a bit premature to take these documents to WG Last Call.

But we have to get these issues settled, and make sure we have raised all 
significant issues as soon as possible, if we're going to catch up with any 
reasonable schedule.

Below are the tickets created for the issues we know.
When following up, please use the following subject line convention:

 - Subject: #1483 <your text here> for addressing issue 1483
 - Subject: ISSUE: <document> <section>: <your text here> for raising a new 
issue.

The chairs will enter issues raised with the ISSUE: convention into the 
tracker if they deem the issue well defined and important; if they are 
unsure of its importance, they may ask for a second before tracking it.

Grammar and spelling notes should go directly to the editors.

PLEASE raise issues against these 2 documents as soon as possible; 
preferably before May 23 (Wednesday of next week).

Note that we're NOT calling for a final round of issues on -downgrade or 
the other documents at this time; issues will be noted, but we're not going 
to attempt to resolve all of them rapidly.

Below are the tickets created so far.

------------- Forwarded Message ------------
Subject: [psg.com #1483] AutoReply: SMTPEXT 2.7: Non-ASCII in response texts

In RFC 2821, SMTP responses are defined to be ASCII-only.
If email addresses are extended to support non-ASCII, some responses,
which today contain email addresses, will either have to not contain
email addresses or be extended in some way.
Alternatives include:

- Just send non-ASCII
- Send non-ASCII, but only in email addresses that were specified by the
client
- Send non-ASCII, but only if client enables that feature by a special
command
- Send ASCII only

There may be more.
------------ Forwarded Message ------------
Subject: [psg.com #1484] AutoReply: SMTPEXT 2.2: More exact definition of 
action choices

Section 2.2 contains the following text:

   ..... Instead, an
   SMTP client other than the Submission MTA MUST make one of the
   following three choices:
   1.  Reject or return the message as undeliverable.
   2.  Find an alternate route to the destination that permits UTF8SMTP.
   3.  If and only if an ASCII address is available for the return path
       and one or more of the specific forwarding paths being attempted,
       downgrade the message to an all-ASCII form as specified in
       [EAI-downgrading].

None of the 3 choices is fully described in detail in the document.
It may be desirable to add more detail.
------------ Forwarded Message ------------
Subject: [psg.com #1485]: UTF8HDR 4.6/DSN: Choice of body part for 
transport of UTF8SMTP messages

The choice of the body part to transport UTF8SMTP messages in a message
has proved controversial.
Suggestions include:

- message/utf8 (inconsistent with certain statements in the MIME RFCs)
- application/utf8smtp (stylistically questionable)
- message/rfc822 (incompatible with existing definition)

A decision needs to be made.

------------ Forwarded Message ------------
Subject: [psg.com #1486] AutoReply: SMTPEXT 2.7.3: Message retry

It's been suggested that changing the message retry mechanism as
specified in section 2.7.3 is a sensitive issue. We might want to either
revisit the text or remove it entirely, so that the behaviour is
identical to normal 2821 behaviour.

------------ Forwarded Message ------------
Subject: [psg.com #1487] AutoReply: UTF8HDR 4.2: Does this specification 
update MIME?

As written, section 4.2 updates the MIME specification by allowing UTF-8
in MIME headers.

The headers do not say that the MIME RFC is updated, and we need a
proper descritption of what it means for an experimental RFC to "update"
a standards-track one.

------------ Forwarded Message ------------
Subject: [psg.com #1488] AutoReply: UTF8HDR 4.2: List MIME headers that are 
modified?

Should the UTF8HDR specification list the MIME headers that are updated
by this specification?
--------------------------------------------------

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 14:45:10 +0100

> - Send non-ASCII, but only if client enables that feature by a special

Certainly the last case is wrong.
Certainly, if the response is being sent in a DSN, then non-ASCII is in  
order.
Certainly, if the original message contained non-ASCII, the originator can  
be presumed to understanda non-ASCII responses.
We seem to have been arguing against "special command's (we got rid of  
Header-Type and HDR=UTF8).

So I think either Case 1 or Case 2 are the only ppssibilites (I prefer  
Case 1)

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 14:50:56 +0100

> has proved controversial.

I have written separately on this.

But utf8headers-05.txt is fine with me as it stands, and it should not be  
altered until we have decided on this issue in connection with DSN.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 15:14:55 +0100

> Subject: [psg.com #1486] AutoReply: SMTPEXT 2.7.3: Message retry

s/SMTPEXT 2.7.3/SMTPEXT 2.7.2/
> identical to normal 2821 behaviour.

The behaviours is probably reasonable, provided the circumstances are  
appropriate, and waiting for an MX that is down to come up may be a good  
reason to queue the message for a while. But if all the available MXes do  
not support UTF8SMTP, then there is no point in further queuing.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 15:21:58 +0100

> proper descritption of what it means for an experimental RFC to "update"

And the same applies to RFC 2822, where we propose to allow things that  
2822 forbids.

I think, if an experimental RFC updates an existing RFC, then the proper  
wording would be "Experimentally Updates/Obsoletes/Whatever RFC xxxx".  
Then, if the experiment is abandoned, RFC xxxx remains unupdated. If the  
experiment later gets standardized, then RFC xxxx becomes fully updated.  
And, in the meantime, participants in the experiment do whatever the  
experimental document says.

So I think the proper course is to included the words "Experimentally  
Updates ...", and then sit back and see if the IESG will allow it (maybe  
they will suggest something better).

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 15:23:56 +0100

On Sat, 19 May 2007 07:51:10 +0100, Harald Tveit Alvestrand  
<[email-masked]> wrote:

> Subject: [psg.com #1488] AutoReply: UTF8HDR 4.2: List MIME headers that  
> are modified?
>

Yes (when we have finally agreed on the list :-( ).

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 22 May 2007 15:43:08 +0100

On Mon, 21 May 2007 15:21:58 +0100, Charles Lindsey <[email-masked]>  
wrote:

>> Subject: [psg.com #1487] AutoReply: UTF8HDR 4.2: Does this  
>> specification update MIME?
>>

> experimental document says.

On further thought, I think the wording needed is "Experimentally extends  
RFC xxxx".

Essentially, an "extension" allows you to say, or do, something that was  
not allowed by RFC xxxx. But you are only allowed to say or do it in  
envireonments that support the extension, So, for example, when speaking  
to an MTA that announces support for UTF8SMTP, it is safe to use both the  
extensions to RFC 2822 that we have defined, plus whatever extensions we  
define for RFC 204[56]. That restriction to supportive environments would  
still remain even if our experimental protocol eventually joins the  
standards track,

---

From: Kari Hurtta <[email-masked]>
Date: 24 May 2007 17:35:34 +0300

> Certainly, if the response is being sent in a DSN, then non-ASCII is
> in  order.
> Certainly, if the original message contained non-ASCII, the originator
> can  be presumed to understanda non-ASCII responses.

Because document is SMTPEXT, I think that response is smtp reply and
not a response which is sent in a DSN.  Of course also smtp reply
can be copied to DSN ultimately.

Problem is that when smtp server announce UTF8SMTP this does not 
indicate that smtp client supports UTF8SMTP. Therefore sending
UTF-8 to unprepared smtp client is risky.

On other hand this UTF8SMTP extension tries move smtp to UTF-8
so limiting smtp replies to UTF-8 does not show very wise.

Therefore alternatives

are these which make sense.

This case "special command" implies nothing about message. It only
tells capabilities of smtp client.

One of my scenarios was

EHLO xxx
250-server.example
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250 UTF8SMTP
250 HELP
EXPN root
250 2.1.5 Ville Keränen <[email-masked]>

In these case EXPN response include phrase (fullname) and mailbox with UTF-8.
Danger is that there is nothing which indicates that client supports
UTF-8.

Also usually (althoug smtp does not require it) mailbox is
often repeated on smtp reply. That case it includes UTF-8 only
if original command included UTF-8.

Another theoretical example was

      MAIL FROM:<[email-masked]>
      250 OK
      RCPT TO:<[email-masked]>
      251 <[email-masked]> is new address, will forward

     ( client crashes ?)

However 251 and 551 response codes (= new address) are not usually
used.

I was following suggestion. In that suggestion "special command" is
part of argument in MAIL, EXPN and VRFY commands.  (On other hand 
to enable  UTF-8 replies to whole smtp session special command 
before MAIL command is required -- assuming that I interpreted 
Klensin correctly :-) )

----------------------------------------------------------------------------------
To: Jiankang YAO <[email-masked]>, Wei MAO <[email-masked]>
Date: Fri, 18 May 2007 08:57:36 +0300 (EEST)
Subject: draft-ietf-eai-smtpext-05.txt: Revised text for UTF-8 reply

Revised text for UTF-8 reply (with taking account Klensin's
comments.)

== 2.1.  Framework for the Internationalization Extension

|  4.  One optional parameter,

=>

|  4.  Optional parameter,

Addition:

   X.  Optional parameter, UTF8REPLY, is added to SMTP MAIL,
       VRFY and EXPN commands. Paramater UTF8REPLY have no
       values. Parameter indicates SMTP cleant accepts UTF-8
       on replies of SMTP commands. Specially this allows
       to use UTF-8 on mailboxes which occurs on replies.

( Now I ignore change for "9. The maximum length"  text. )

== 2.2.  The UTF8SMTP Extension

Additions:

|    SMTP client following this specification MUST accept UTF-8
|    on replies of SMTP commands. However on SMTP server SHOULD
|    not use UTF-8 on replies, if SMTP client do not ask UTF-8
|    replies. Some replies includes mailbox, but usually most
|    of replies do not require that mailbox is include to reply
|    and therefore UTF-8 is not needed.
|
|    UTF8REPLY parameter on MAIL, VRFY and EXPN commands tells
|    that SMTP client is prepared for UTF-8 on SMTP replies.
|
|    VERIFY (VRFY) command syntax is changed to:
|
|        "VRFY" [SP "UTF8REPLY"] SP String CRLF
|
|    EXPAND (EXPN) command syntax is changed to:
|
|        EXPN" [SP "UTF8REPLY"] SP String CRLF
|
|    "UTF8REPLY" parameter is added to <Mail-parameters>.
|    This parameter do not have value.
|
|    If SMTP reply requires UTF-8, but SMTP client
|    is not used "UTF8REPLY" parameter on any command,
|    response code "550" is used, defined in [RFC2821],
|    meaning "mailbox unavailable".  For VERIFY (VRFY)
|    and EXPAND (EXPN) commands also response code "252"
|    can be used,  defined in [RFC2821], meaning "Cannot VRFY user,
|    but will accept message and attempt delivery". If enhanced
|    mail system status codes [RFC3463] is used, the response code
|    should be "5.6.y" or "2.6.y" [SMTP-codes], meaning that
|    "UTF-8 reply required, but UTF8REPLY parameter not used.".
|
|   [[anchor: REMOVE THIS: IANA please assign the proper error codes for
|   "5.6.y" and "2.6.y".]]
|
|   "UTF8REPLY" parameter in MAIL command command <Mail-parameters>
|   enables UTF-8 replies for SMTP commands on that mail transaction.
|   "UTF8REPLY" on  VERIFY (VRFY) and EXPAND (EXPN) enables UTF-8
|   for that command only.

== 6.  IANA Considerations

Addition:

|   IANA is requested to assign the proper error codes "5.6.y" and "2.6.y"
|   for this specification based on [SMTP-codes].

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 25 May 2007 07:41:36 +0300

There is bad typo on my message.

> so limiting smtp replies to UTF-8 does not show very wise.

There is typo. That should be:

  On other hand this UTF8SMTP extension tries move smtp to UTF-8
  so limiting smtp replies to ASCII does not show very wise.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 27 May 2007 18:42:26 +0300

> A decision needs to be made.

I prefer
    UTF8SMTP messages labeled as message/rfc822 on UTF8SMTP environment
    and then downgraded to RFC 2822 messages when then leave UTF8SMTP 
    environment.

On downgrading there is two different cases
    A)   bounce/reject is available
    B)   bounce/reject is not available

Case A is when downgrading is done by UTF8SMTP gateway on smtp when envelope 
sender is not <>.

Case B is when downgrading is done after delivery (for example on IMAP or
POP server) or when downgrading is  done by UTF8SMTP gateway on smtp when 
envelope sender is <>.

In case A if downgrading of attached message (with type message/rfc822)
fails, enclosed message is rejected or bounced (NDN generated).

In case B is it is just better leave out data what is not downgradeable
(for example if there is no alternative address for some address on header)
or on extreme case left out whole message/rfc822 body part.

( In case B also some other options exists, but they are better discussed
  when downgrading document is on subject.  Also if embedded message is not
  downgradeable, one option is do some sort encapsulation on both
  cases A and B )

For why I prefer message/rfc822 I have mentioned these reasons already:

   When working group was decided that there is no need labeling
   for UTF8SMTP messages, that quite clearly follow that UTF8SMTP
   messages does not need new label (ie. message/utf8 type) when
   they are attached to another message.

   When whole message is not need labeling on UTF8SMTP case (ie no ESMTP
   paramater), there is no need also different kind labeling if
   message is attached to another message.

   That makes "message/rfc822" just mean "mail message". And consistent
   that "message/rfc822" is just that type what occurs inside of DATA
   on smtp.

   Actually "message/rfc822" is not mean to be only mail message,
   but superset of it (rfc 2046):

      It should be noted that, despite the use of the numbers "822", a
      "message/rfc822" entity isn't restricted to material in strict
      conformance to RFC822, nor are the semantics of "message/rfc822"
      objects restricted to the semantics defined in RFC822. More
      specifically, a "message/rfc822" message could well be a News article
      or a MIME message.

   If there is reason why "message/rfc822" can not be used to
   mean any mail message on UTF8SMTP universe, same problems are
   also on top level.   

   For example if there is some problems for IMAP on UTF8SMTP universe,
   when UTF8SMTP message is attaches as "message/rfc822", same problems
   also aply to top level message. 

This issue have subject UTF8HDR 4.6. That chapter do not mention
MIME type of UTF8SMTP message at all. Instead that chapter says:

   the object defined in [EAI-dsn] is intended to be transportable as
   part of an ordinary [RFC2822] message.

That implies that UTF8SMTP messages transported inside of messages are not
downgraded. I prefer that they are downgraded.  That is important when 
this MIME type is just used for forwarding.

On situation where UTF8SMTP messages start appear is when MUA and MSA supports
UTF8SMTP, and just non-ascii subject is used. If embedded messages are 
downgraded when message is forwarded outside of UTF8SMTP environment, that
does not cause surprises.   If downgrading is not done (and encapsulation is 
done instead), recipient get different situation depending was just UTF8SMTP
MSA+MUA used.

That is: End result outside of UTF8SMTP environment should be same on 
following  cases when non ascii subject is typed:
        1) MUA sends ordinary MIME message (non UTF8SMTP message)
        2) MUA sends UTF8SMTP message
On both cases end result outside of UTF8SMTP environment should be ordinary
MIME message with MIME encoded words on subject. And that should happend
even when this message leaves UTF8SMTP environment inside of another message.

Perhaps that can be say also on following way: "Fully downgradeable UTF8SMTP
messages should be downgraded (and not encapsulated) when they leave UTF8SMTP 
environment. And that does not depend is message stand alone or is it embedded
inside of another message."

MIME type of UTF8SMTP message is mentioned on chapter 4.2 instead:

   In all those header fields, Observe that such Content-Type and other
   header fields may be found both amongst the top-level fields of a
   message and also within multiparts; and also that a complete message
   conforming to this document may now appear as a message/rfc822 (in
   both cases, subject to downgrade when that is necessary)

application/utf8smtp have some values on situations where message is
not downgradeable. In that case that type can be used for encapsulation,
but I do not recommended it for general use.

message/utf8 is bad if  UTF8SMTP content is not downgraded (and retyped
as message/rfc822) when message  leaves UTF8SMTP environment. Some reasons
I have discussed on my messages with subjects
        - Ascii user X sends mail
        - UTF8SMTP / 8BITMIME / ASCII -universe amd message/utf8smtp

/ Kari Hurtta

---

From: Yuri Inglikov <[email-masked]>
Date: Tue, 29 May 2007 10:02:07 -0700

I like the idea of changing labeling from message/rfc822 to something like application/utf8smtp as a last resort if message is not downgradeable and cannot be bounced.

> -----Original Message-----
> From: Kari Hurtta [mailto:[email-masked]]
> Sent: Sunday, May 27, 2007 8:42 AM
> To: [email-masked]
> Cc: Kari Hurtta
> Subject: [EAI] #1485: Choice of body part for transport of UTF8SMTP
> messages
>

---

From: Kari Hurtta <[email-masked]>
Date: 30 May 2007 18:56:01 +0300

> proper descritption of what it means for an experimental RFC to "update"

Also section 4.3 updates RFC 2822 specification by allowing UTF-8 on
addresses and other places.

Yes, RFC 2231 changes mime parameter syntax and encoded word syntax. And
it updates RFC 2045, 2047, 2183.

So perhaps that is similar situation.

These syntax changes are limited to UTF8SMTP environment.

There is similarity with 8BITMIME. 8BITMIME also is limited 
and changes syntax.

But here RFC 1652 does not update RFC 821 or 822 either.

So if this correspond then these UTF8SMTP related specifications does not 
need update RFC 2821, 2822, 2045, and so on either.

I think there can be found reasoning because of this that headers do not need
say that the MIME, MAIL or SMTP RFC are updated.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 31 May 2007 14:49:28 +0100

> But utf8headers-05.txt is fine with me as it stands, and it should not  
> be altered until we have decided on this issue in connection with DSN.
>

May I reiterate that it would be quite improper to issue a further draft  
of the utf8headers document removing the possibility of using  
message/rfc822 to tranport a UTF8SMTP Message (within the UTF8SMTP  
universe) until this issue (#1485) has been resolved.

I note that there is now an increasing recognition within the WG that  
message/rfc822 is the proper method to do this. Moreover, both Kari and  
myself have put forward technical arguments (as opposed to gut reactions)  
as to why message/utf8smtp is not suited to this purpose (whether within  
DSNs or otherwise), and NOBODY has yet even attempted to refute those  
arguments.

---

From: Kari Hurtta <[email-masked]>
Date: Thu, 31 May 2007 20:12:54 +0300 (EEST)

> preferably before May 23 (Wednesday of next week).

I posted May 19 following issues to you,. Nothing hear about
these. Are these all found as unimportant or was they lost?

   ISSUE: UTF8HDR 4.4: <angle-addr> should include <obs-angle-addr>
   ISSUE: UTF8HDR 4.3/4.5: Message-ID restriction not descibed
   ISSUE: SMTPEXT 2.6: Reply code so that downgrade is not possible
   ISSUE: SMTPEXT 2.4: Redefining of RCPT TO and MAIL FROM commands (ABNF grammar)
   ISSUE: SMTPEXT 2.7.3: <uFor> syntax
   ISSUE: UTF8HDR 4.5: <angle-addr> and <addr-spec> on trace field syntax
   ISSUE: UTF8HDR 4.4: <angle-addr> must always include "<" and ">" characters
   ISSUE: UTF8HDR 4.3: Incremental alternatives (ABNF grammar)
   ISSUE: UTF8HDR 4.2: Extending MIME (ABNF grammar)
   ISSUE: UTF8HDR 3: UTF-8 character defination
   ISSUE: UTF8HDR 3: "header" versus "header field"
   ISSUE: UTF8HDR 4.2: Including Content-Description to picture (grammar)

These are included to that message for reference.

Most of these I have earlier sent to authors of drafts.

Today I got message Abel Yang which included pointer to new
unpublished draft-ietf-eai-utf8headers-06.txt.  That message
was addressed also to Randall Gellens and Alexey Melnikov which
was also commented draft and then also to [email-masked]
address. That address seems to be some members-only mailing 
list (I got bounce from Mailman when I used group reply.)
Apparently these drafts are discussed also on some other 
mailing lists than on WG mailing list.

Anyway draft-ietf-eai-utf8headers-06.txt seems have fixed some
of these issues:

   ISSUE: UTF8HDR 4.4: <angle-addr> should include <obs-angle-addr>
      seems to have fixed

   ISSUE: UTF8HDR 4.3/4.5: Message-ID restriction not descibed
       not handled

   ISSUE: UTF8HDR 4.5: <angle-addr> and <addr-spec> on trace field syntax
       not handled 

   ISSUE: UTF8HDR 4.4: <angle-addr> must always include "<" and ">" characters
       not handled

   ISSUE: UTF8HDR 4.3: Incremental alternatives (ABNF grammar)
       have some changes, but there is some problems still

   ISSUE: UTF8HDR 4.2: Extending MIME (ABNF grammar)
       fixed, except some old stray text is left behind

   ISSUE: UTF8HDR 3: UTF-8 character defination
       have some changes, but may require still touch

   ISSUE: UTF8HDR 3: "header" versus "header field"
       have some changes, but "header field" is not used

   ISSUE: UTF8HDR 4.2: Including Content-Description to picture (grammar)
       not handled

/ Kari Hurtta

Subject: ISSUE: UTF8HDR 4.2: Including Content-Description to picture (grammar)
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 10:11:10 +0300

This is related to

   UTF8HDR 4.2: List MIME headers that are modified?

but actual question is that MIME headers are not defined
with RFC 2822 grammar and it is unclear that extending
of RFC 2822 gives  extended syntax for to that MIME header.

RFC 2045 defines Content-Description as

    description := "Content-Description" ":" *text

That is clearly unstructured. That correspond RFC 822 defination

    optional-field =
...
                    /  "Subject"           ":"  *text

However that is "utext" on RFC 2822:

    subject         =       "Subject:" unstructured CRLF

    unstructured    =       *([FWS] utext) [FWS]

So it is little unclear is chapter "4.3. Syntax extensions to RFC 2822"
redefination of "utext"   in force here?

Parhaps:

" To able to use UTF-8 on Content-Description -header field, following
" syntax is used
"
"    description   = "Content-Description:" unstructured CRLF
"
" <utext> syntax is extended on next chapter to allow UTF-8 characters
" on <unstructured> header fields.

Subject: ISSUE: UTF8HDR 3: "header" versus "header field"
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 11:19:09 +0300

| In this document, even ordinarg ASCII characters are UTF-8 characters
| if the bodies of those headers contain <utf8-xtra-char>s.

I found that somewhat strange defination. Anyway it is better to use
"header field".

So perhaps:

" A plain ASCII string is also a valid UTF-8 string, see [RFC3629].
" In this document, ordinary ASCII characters are UTF-8 characters
" if the bodies of those header fields contain <utf8-xtra-char>s.

But that is still strange :-(

Term "header" there is problem that "header" is ambiguous.
draft-resnick-2822upd-01.txt  uses terms "header field" and
"header section".

Subject: ISSUE: UTF8HDR 3: UTF-8 character defination
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 11:25:15 +0300

|   In this document, even ordinarg ASCII characters are UTF-8 characters
|   if the bodies of those headers contain <utf8-xtra-char>s.

RFC3629 notes:

|   UTF-8, the object of this memo, has a one-octet encoding unit.  It
|   uses all bits of an octet, but has the quality of preserving the full
|   US-ASCII [US-ASCII] range: US-ASCII characters are encoded in one
|   octet having the normal US-ASCII value, and any octet with such a
|   value can only stand for a US-ASCII character, and nothing else.

Therefore ASCII characters are also UTF-8 characters even when there
is no any <utf8-xtra-char>s.

/ Kari Hurtta

Subject: ISSUE: UTF8HDR 4.2: Extending MIME (ABNF grammar)
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 11:38:58 +0300

| The syntax of <value>, as defined in [RFC2045] is
|
| value = token / utf8-quoted-string
|
| To be able to use UTF-8 characters in MIME headers, <quoted-string>
| syntax is extended as
|
| qcontent = utf8-qtext / utf8-quoted-pair

That is strange. Defination on [RFC2045] was

   value = token / quoted-string

So perhaps it is mean

" To be able to use UTF-8 characters in MIME header field
" paramerer values, the syntax of <value>, as defined in [RFC2045],
" is extended as
"
" value      =/ utf8-quoted-string
"
" <utf8-quoted-string> is defined on chapter <REF: Syntax extensions to RFC 2822>

Subject: ISSUE: UTF8HDR 4.3: Incremental alternatives (ABNF grammar)
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 11:48:34 +0300

|  ctext   /=  NO-WS-CTL /     ; all of <text> except
|              %d33-39 /       ; SP, HTAB, "(", ")"
|              %d42-91 /       ; and "\"
|              %d93-126 /
|              UTF8-xtra-char

Is that mean just add terms to <ctext> ? 

Then it is "=/" and  not "/=".

RFC 4234:

| 3.3.  Incremental Alternatives: Rule1 =/ Rule2

Also then existing alternatives need not need to be repeated:

" ctext =/  UTF8-xtra-char

Also

|  utext   =  NO-WS-CTL /     ; Non white space controls
|              %d33-126 /      ; The rest of US-ASCII
|              UTF8-xtra-char

can be changed to

"  utext   =/  UTF8-xtra-char

Subject: ISSUE: UTF8HDR 4.4: <angle-addr> must always include "<" and ">" characters
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 11:56:17 +0300

|   angle-addr = [CFWS] "<" utf8-addr-spec<alt-address>">" [CFWS] / \
|                [CFWS] "<" utf8-addr-spec ">" [CFWS] / \
|                            [CFWS] utf8-addr-spec [CFWS]

I lost track where that last "[CFWS] utf8-addr-spec [CFWS]"
is needed. <mailbox> already includes <utf8-addr-spec>

<angle-addr> is used on

  name-addr       =       [display-name] angle-addr

  angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr

Anyway that produces VERY strange production:

  <name-addr>  may be
      [display-name] [CFWS] utf8-addr-spec [CFWS]

This definately is NOT what is wanted.

 <angle-addr>  must not include productions which do not produce
"<" and ">" characters !

|  Below list a few possible <mailbox> representation as example.
...
|  non-ASCII@non-ASCII
|                       ; without DISPLAY_NAME and quoted string
|                       ; UTF8SMTP but no ALT-ADDRESS parameter provided,
|                       ; message will bounce if UTF8SMTP extension is not supported

This was already allowed on <mailbox> production so addition of
this to <angle-addr> does not make sense.

|  mailbox        =  name-addr / addr-spec / utf8-addr-spec

Subject: ISSUE: UTF8HDR 4.5: <angle-addr> and <addr-spec> on trace field syntax
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 12:04:06 +0300

Only other place where <angle-addr> is used is

  item-value      =       1*angle-addr / addr-spec /
                          atom / domain / msg-id

Because addr-spec does not allow UTF-8 so there perhaps
utf8-addr-spec is needed, but it is better to do with

" item-value      =/      utf8-addr-spec

( However draft-resnick-2822upd-01.txt updates Received: -syntax
  as:

    received        =       "Received:" *received-token ";" date-time CRLF

    received-token  =       word / angle-addr / addr-spec / domain

  so perhaps actually it is

"  received-token  =/  utf8-addr-spec

   if RFC 2822 update is published before that document.
   [  I am not checked <CFWS> however. ]
)

Subject: ISSUE: SMTPEXT 2.7.3: <uFor> syntax
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 12:26:42 +0300

draft-ietf-eai-smtpext-05.txt:

|  uFor = "FOR" FWS 1*( uPath / uMailbox ) CFWS
|           ; Replaces For in the section 4.4 of [RFC2821]
|           ; uPath is defined in section 2.4 of this document

Compare to draft-klensin-rfc2821bis-04.txt:

|   For            = CFWS "FOR" 1*( FWS ( Path / Mailbox ))

I think that this is more correct placement of <FWS>

So that makes it to be:

"   uFor          = "FOR"  1*( FWS (uPath / uMailbox) ) CFWS

If RFC 2822 update is published before that document, then it is

"   uFor          = CFWS "FOR"  1*( FWS (uPath / uMailbox) )

Subject: ISSUE: SMTPEXT 2.4: Redefining of RCPT TO and MAIL FROM commands (ABNF grammar)
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 15:00:37 +0300

|  If the UTF8SMTP extension is offered, the syntax of the SMTP MAIL and
|  RCPT commands is extended to support the optional esmtp-keyword "ALT-
|  ADDRESS", which specifies an alternate all-ASCII address which may be
|  used when downgrading.  If the ALT-ADDRESS esmtp-keyword is used, it
|  MUST have an associated esmtp-value (ALT-ADDRESS-esmtp-value which is
|  defined below).
|
|  Based on the definition of mail-parameters in [RFC2821], the ALT-
|  ADDRESS parameter usage in the commands of "mail from" and "rcpt to"
|  is defined below.
|
|
|       "MAIL FROM:" SP <uReverse-path> [ SP <ALT-ADDRESS-parameter> ]
|                  ; Update mail command in RFC 2821, section 3.3
|                  ; The syntax for "esmtp-value" in RFC2821
|                  ; does not allow "=", SP and control characters.
|                  ; Therefore ALT-ADDRESS-paramater is extended.
|
|           "RCPT TO:" SP <uForward-path> [ SP <rcpt-parameters> ]
|              ; Update rcpt command in RFC 2821, section 3.3

This failes to take account that  ALT-ADDRESS is is valid
both on MAIL and RCPT commands.

And then there must not be SP  between  "FROM:" and <uReverse-path>
and there must not be SP between "TO:" and <uForward-path>

Corresponding RFC 2821 syntaxes are:

|  Syntax:
|
|    "MAIL FROM:" ("<>" / Reverse-Path)
|                       [SP Mail-parameters] CRLF

and

| Syntax:
|    "RCPT TO:" ("<Postmaster@" domain ">" / "<Postmaster>" / Forward-Path)
|                     [SP Rcpt-parameters] CRLF

Possible text:

" Syntax of MAIL command is altered as following:
"
"    "MAIL FROM:" ("<>" / uReverse-Path)
"                       [SP uMail-parameters] CRLF
"
" 
" Syntax of RECIPIENT (RCPT) command is altered as following:
"
"   "RCPT TO:" ("<Postmaster@" udomain ">" / "<Postmaster>" / uForward-Path)
"                     [SP uRcpt-parameters] CRLF
"
"
" These parameters are defined as following
"
"   uMail-parameters  = Mail-parameters | ALT-ADDRESS-parameter
"
"   uRcpt-parameters  = Rcpt-parameters | ALT-ADDRESS-parameter
"
"   uReverse-path = uPath
"               ; Replace Reverse-path in RFC 2821, section 4.1.2
"
"        uForward-path = uPath
"               ; Replace Forward-path in RFC 2821, section 4.1.2
"
"        uPath = "<" [ A-d-l ":" ] uMailbox ">"
"               ; Replace Path in RFC 2821, section 4.1.2
"                   ; A-d-l is defined in RFC 2821, section 4.1.2
"                   ; uMailbox is defined in section 2.3 of this document
"
"        ALT-ADDRESS-parameter="ALT-ADDRESS=" ALT-ADDRESS-esmtp-value
"                   ; The syntax for "esmtp-value" in RFC2821
"                   ; does not allow "=", SP and control characters.
"                   ; Therefore ALT-ADDRESS-paramater is extended.
"
"        ALT-ADDRESS-esmtp-value=xtext
"           ; xtext is defined in RFC 3461, section 4.2

Subject: ISSUE: SMTPEXT 2.6: Reply code so that downgrade is not possible
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 15:42:55 +0300

|   Since there is no ESMTP parameter which tells whether the message is
|   UTF8SMTP message, SMTP server needs to parse all message header
|   fields and MIME header fields in the message body to discover which
|   messages are UTF8SMTP.

There is needed reply for "." on final DATA which indicates that downgrade 
is not possible (for example <alt-address> is missing from mail headers).

Nowdays accept and then bounce do not fly. Even when reply code
is not specified on document, very much of parsing will be done
on before ACKing that mail is received (nowdays for example
virus scanning is done before ACKin that message is received). 
That means that possibility of downgrading is checked before
mail is ACKed. If reply code is not assigned, implementations will
invent own code for that.

Addition:

" Mail message may be rejected because downgrade is required and
" is is not possible. Altenatively message may be accepted and
" then DSN is send for downgarde failure.
"
" When messages are rejected because they require UTF8SMTP and
" downgrading is not available or downgrading fails, response
" code "554" is used, defined in [RFC2821], meaning "Transaction 
" failed". if enhanced mail system status codes [RFC3463] is
" used, the response code should be "5.6.a" [SMTP-codes], meaning that
" "UTF8SMTP downgrade failed".
"
" [[anchor: REMOVE THIS: IANA please assign the proper error codes for
"   "5.6.a".]]

And to IANA Considerations:

"  IANA is requested to assign the proper error codes "5.6.a" and 
" "5.6.x" for this specification based on [SMTP-codes].

Section 2.5 talks also about return code, but it seems be
for MAIL FROM and RCPT TO commands when ALT-ADDRESS paramater is
missing.

Subject: ISSUE: UTF8HDR 4.3/4.5: Message-ID restriction not descibed
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 20:11:13 +0300

|   Besides, in order to allow UTF8 characters in <addr-spec> we have to
|   change the syntax of <atext>.  However, it would also lead <msg-id>
|   to allow UTF8 characters, which is not allowed due to the limitation
|   described in Section 4.5.  So <utf8-atext> is added to meet this
|   requirement.

However section 4.5 does not talk about that <msg-id> restriction:

| 4.5.  Trace field syntax
|
|   "For" fields containing internationalized addresses are allowed, by
|   use of the new uFor syntax.  UTF-8 information in needed in Received
|   fields and such information is therefore allowed, to preserve the
|   integrity of those fields.  The uFor syntax retains the original
|   UTF-8 email address between EAI-aware MTAs.  Note that, should
|   downgrading be required, the uFor parameter is dropped per the
|   procedure specified in [EAI-downgrading].
|
|   The "Return-Path" header provides the email returning address in the
|   mail delivery.  Thus, it MUST able to carry UTF8 addresses (see the
|   revised syntax of <angle-addr> in Section 4.3 of this document).
|   This will not break the rule of trace fied integrity, because it is
|   added at the last MTA.

Message-ID  may need own chapter?

RFC 2822 uses <msg-id> on following places:

message-id      =       "Message-ID:" msg-id CRLF

in-reply-to     =       "In-Reply-To:" 1*msg-id CRLF

references      =       "References:" 1*msg-id CRLF

resent-msg-id   =       "Resent-Message-ID:" msg-id CRLF

item-value      =       1*angle-addr / addr-spec /
                         atom / domain / msg-id

So <msg-id> is actually also component on trace field,
but text on chapter 4.5 actually do not describe reason
of restriction.

I think that following sentence is mutated:
|                             UTF-8 information in needed in Received
|   fields and such information is therefore allowed, to preserve the
|   integrity of those fields.

It may be that <msg-id> is NOT allowed to include UTF-8
so that intregrety of these fields of Received -header filed
and other identity header fields is preserved.

( That <msg-id> pointer was making more sense when text was:

|        With these two restrictions, there should be no need for
|   UTF-8 information in Received fields and such information is
|   prohibited to preserve the integrity of those fields.
)

Some serious rephrasing is needed.

Subject: ISSUE: UTF8HDR 4.4: <angle-addr> should include <obs-angle-addr>
From: Kari Hurtta <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
Cc: Kari Hurtta <[email-masked]>
Date: 19 May 2007 21:53:57 +0300

|   angle-addr = [CFWS] "<" utf8-addr-spec<alt-address>">" [CFWS] / \
|                [CFWS] "<" utf8-addr-spec ">" [CFWS] / \
|                            [CFWS] utf8-addr-spec [CFWS]

RFC 2822 corresponding rule is:

| angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr

Because other rules also have retained obs-* then this should also.

That makes (ignoring utf8-addr-spec):

"   angle-addr = [CFWS] "<" utf8-addr-spec<alt-address>">" [CFWS] / \
"                [CFWS] "<" utf8-addr-spec ">" [CFWS] / \
"                 obs-angle-addr

(   Although I think that it is easier to use incremental syntax with "=/" )

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 31 May 2007 21:58:40 +0200

> These are included to that message for reference.
Thanks for sending these. I have been very slow in entering them (did so 
a few hours ago).
Four of these caused creation of tickets; some of them I considered 
editorial, one I claim is not an EAI issue.
> mailing lists than on WG mailing list.
Yes, all the document editors + the WG chairs are members of the "design 
team" list.
See RFC 2418 section 6.5 for what a design team is supposed to be.

>    ISSUE: UTF8HDR 4.2: Including Content-Description to picture (grammar)
We will see what can be resolved before submitting the next version.
If something is contentious, the WG will have to weigh in on it.

                    Harald