Subject: Re: [EAI] Comments on draft-ietf-eai-smtp-00.txt

From: John C Klensin <[email-masked]>
Date: Thu, 01 Jun 2006 05:52:41 -0400

Several comments below.  Let's make an agenda item for Beijing 
on terminology to be used.  The decisions are more or less 
arbitrary, but we need to be consistent.  And I, at least, have 
achieved confusion.

--On Thursday, June 01, 2006 11:36 AM +0800 YAO Jiankang 
<[email-masked]> wrote:

>
> ----- Original Message -----
> From: "Charles Lindsey" <[email-masked]>
> To: <[email-masked]>
> Sent: Tuesday, May 30, 2006 2:46 AM
> Subject: [EAI] Comments on draft-ietf-eai-00.txt
>...
>>     6.  Servers offering this extension MUST provide support
>>     for, and announce, the 8BITMIME extension [RFC1652].
>>
>> Hmmm! I see that this means you are supposed to include a
>> BODY=8BITMIME   parameter in your MAIL FROM command if the
>> body uses TCE 8bit (highly   probable in an EAI message). Do
>> agents currently do that in practice?
>
> can I suggest that where in the fields there use MIME now ,
> still use MIME in the email message or content. we only update
> the fields containing the UET-8 email address to UTF8.

I don't understand this response, but I'm not sure I understand 
Charles's comment, partially because there is no client practice 
yet.  My assumption is that clients or servers supporting EAI 
MUST be fully-conformant with 8BITMIME.  I assume that makes the 
answer to Charles's question not "supposed to" but "MUST".  As 
far as the transport is concerned, the 2821 headers are just 
message content so, if they are extended and transmitted in 
UTF-8 then 8BITMIME must be present.   We could, in principle, 
define these extensions as subsuming 8BITMIME, but it seems to 
me that would be error-prone and of no particular advantage. 
Others may disagree; if so, we need to discuss.

>> However, there is another possible 8BITMIME problem. Suppose
>> your message   is a multipart, and suppose one of the headers
>> of one of the multiparts   contains some UTF-8, e.g.
>>      Content-Disposition: ... filename="some utf-8 stuff"
>> (which surely we want to allow, though it is actually an
>> extension to RFC   2045 which needs to be mentioned in the
>> utf8headers document). Now this   message encounters a server
>> which supports neither IEmail nor 8BITMIME.   RFC 1652 tells
>> you to downgrade to 7bit or else to bounce, so in this case
>> the "some utf-8 stuff" needs to be downgraded accordig to RFC
>> 2231, and   this needs to be mentioned somewhere in our
>> drafts. Maybe the downgrade   document is the place to do it,
>> but I mention it here to make sure it does   not get
>> forgotten.

The downgrade document is the place to do it.  This runs into 
the "no multiple encapsulations" rule and requires very careful 
consideration.  My guess is that the UTF-8 headers doc needs to 
note that it updates 2045 and that the downgrade one needs to 
note that it updates 1652... but the differences in how headers 
are handled might make a case for defining these extensions on 
their own, rather than by reference to 8BITMIME, despite what 
I've said above.

>...
>>     An SMTP Client that receives the IMA extension keyword
>>     MAY transmit a mailbox name as an internationalized
>>     string in UTF-8 form and MAY
>>
>> s/receives the IMA extension keyword MAY/receives the IMA
>> extension   keyword in response to EHLO MAY/
>
> ok, will update it.

Harmless, but I think unnecessary.  The thing that is necessary 
is a requirement that the keyword MUST NOT be sent unless the 
extension is advertised, and we have that.

>...
>     ....  If the IMA SMTP extension is not offered by
>>     the Server, the SMTP Client MUST NOT transmit an
>>     internationalized address and MUST NOT transmit a mail
>>     body which contains internationalized mail headers
>>     [IMA-utf8header].  Instead, it MUST either return the
>>     message to the user as undeliverable or replace it with
>>     the alternate ASCII address.  .....
>>
>> Are you trying to cover addresses in both SMTP commands and
>> in email   headers here? If so, then the words "a mail body"
>> are wrong (the body of a   message does not include its
>> headers, apart from in multiparts as I   mentioned earlier).
>> So I think you need
>>
>> s/a mail body/a message/
>
>
> yes, will use "a message".

Still not quite right, because either "message" or "mail body" 
also contains content.  E.g., we have

    Envelope
    Headers

    Body

the restriction above applies properly to the first two, but it 
would be a bad idea to accidentally try to specify how addresses 
are represented in the message body itself.   For example, while 
the address would not be usable, nothing prohibits
    ....
    Header-stuff...
    Content-type: text/plain; charset=UTF-8

    Hi there, my preferred email address is 
[email-masked]
    .

today, with unextended SMTP (although a CTE would presumably be 
required)

>> but in either case, "it" is too vague, so
>>
>> s/it/all addresses (both in the envelope and in the message's
>> header   fields)/
>
> will consider it carefully.

>>     ... If it is replaced, the replacement
>>     MUST be either the ASCII-only address specified with the
>>     ALT-ADDRESS parameter or with an address obtained from
>>     some algorithmic conversions of the primary address that
>>     conforms to the syntax rules of RFC 2821, which is
>>     defined in [IMA-downgrading].
>>
>> That wording is fine if the client is an earlier
>> IEmail-compliant server,   but if the the client is the
>> originating MUA it doesn't have an   ALT-ADDRESS to hand. It
>> has to invent one by whatever means (maybe it   found one in
>> the To: header, maybe it had a table of mappings).
>
> yes.

The more of this sort of thing we can shift into the downgrading 
document, the better off we will be.   It has to cover the 
material, covering it here as well is just an invitation to 
inconsistency.   See if it is possible to define the syntax and 
semantics of the parameters but indicate that the information is 
not needed unless downgrading is necessary; if downgrading is 
necessary, the downgrade document specifies how they are used. 
Does that make sense?

>> There is a further complication. Suppose a message arrives at
>> some   intermediate server with one ALT-ADDRESS in its RCPT
>> TO: command, and a   different one in its To: header. I think
>> the only sensible thing to do is   to use the one in the RCPT
>> TO: command in the ongoing (downgraded) RCPT   TO:, and the
>> one from the To: header in the ongoing (downgraded) To:
>> header. Note that this anomaly is perfectly possible if, for
>> example, this   is a message being propagated by some mailing
>> list expander.
>
> can we do some specification (address synchronize) in the
> utf-8-headers document to avoid such anomaly in the "RCPT To"
> and "To: header"? for example, if ALT-ADDRESS is used in "RCPT
> To", "To: header" will also use ALT-ADDRESS.

Probably not, since the addresses in the RCPT commands need not 
appear at all in the headers.   The usual general guideline 
probably applies here -- to the extent possible, MTAs pay 
attention to the  envelope only.  That makes any ALT-ADDRESS in 
the headers mostly useful for forming replies at the receiving 
MUA.

>...
>> However, you cannot specify Stringprep without stating what
>> 'profile' of   Stringprep you require. Maybe you can find an
>> existing one that suits your   need (but not Nameprep,
>> because that removes and upper/lowercase   distinctions). You
>> could easily define your own profile by indicating   which of
>> the various tables from RFC 3454 were to be applied. I agree
>> that   local-parts should have _some_ suitable Stringprep
>> applied to them. They   are essentially 'identifiers' in
>> nature, and should at least adhere to   WYSIWYG (but note in
>> passing that RFC 2822 local-parts already fail   Stringprep
>> by permitting control characters).

WYSIWYG is more or less equivalent to the fax constraint Charles 
suggested earlier.  Because it is in the eye of the beholder, it 
is basically impossible with Unicode.
>
> Yes, we should have our own Stringprep for the email local
> part.

We need to be careful here.  First, stringprep, as usually used, 
specifies treatment of strings, potentially unnormalized ones, 
so they can be compared.  I think we are going to get into bad 
shape if we permit unnormalized/ uncanonicalized strings at all. 
See the discussion in draft-klensin-net-utf8, but note that the 
conventions recommended there are probably not appropriate for 
these addresses.  My guess is that we should stick fairly 
closely to NFKC, but others probably want to comment on that. 
And, of course, doing anything with NFKC or NFC gets us into the 
Unicode versioning mess.

I thought we had agreed to prohibit control characters in 
non-ASCII local parts -- in the past, they have been more 
trouble than they are worth.   If we haven't, we need to discuss 
it quickly.  If we have, then this document must be explicit 
about it.
>> As an aside, I just noticed that John KLensin's utf8 draft
>> cannot be   expressed as a profile of Stringprep (which is
>> actually as it should be,   because it is not trying to
>> restrict itself to 'identifier's).

That is correct and was intentional.
>...
>>     ... If the
>>     ALT-ADDRESS value is not set by the sender but the value
>>     of ATOMIC is 'y', the sender SMTP server should apply
>>     some algorithmic transformation such as punycode to the
>>     entire local part of IMA; ...
>>
>> No, from the POV of the server that might be going to
>> downgrade the   address, the "sender" is actually an SMTP
>> "client" (which might be another   server, or might be an
>> MUA).
>
> so we change the word "sender" to "email user", is it ok?

Please don't try to invent more terminology unless absolutely 
necessary.  I would read "email user" as the human being who 
interacts with the message-creating MUA, and that is certainly 
not what is intended in all of the uses in that paragraph.  2821 
uses "SMTP client" and "SMTP server" where feasible and reverts 
to the 821 terminology of "SMTP sender" and "SMTP receiver" when 
needed for clarity.  "sender SMTP server" is pretty close to a 
contradiction.   See above about getting as much of this into 
the downgrade document as possible.

>...

     john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 01 Jun 2006 06:03:40 -0700

John C Klensin wrote:

>
> Several comments below.  Let's make an agenda item for Beijing on 
> terminology to be used.  The decisions are more or less arbitrary, but 
> we need to be consistent.  And I, at least, have achieved confusion.
>
>
> Charles's comment, partially because there is no client practice yet.  
> My assumption is that clients or servers supporting EAI MUST be 
> fully-conformant with 8BITMIME.  I assume that makes the answer to 
> Charles's question not "supposed to" but "MUST".  As far as the 
> transport is concerned, the 2821 headers are just message content so, 
> if they are extended and transmitted in UTF-8 then 8BITMIME must be 
> present.   We could, in principle, define these extensions as 
> subsuming 8BITMIME, but it seems to me that would be error-prone and 
> of no particular advantage. Others may disagree; if so, we need to 
> discuss.

The relevant quotation from RFC 1652 is:

>    The value associated with the BODY parameter indicates whether the
>    content body which will be passed using the DATA command consists of
>    a MIME message containing some arbitrary octet-aligned material
>    ("8BITMIME") or is encoded entirely in accordance with [1] ("7BIT").
>
>    A server which supports the 8-bit MIME transport service extension
>    shall preserve all bits in each octet passed using the DATA command.
>
RFC 1652 makes no reference to the concept of "header".

---

From: John C Klensin <[email-masked]>
Date: Thu, 01 Jun 2006 09:23:52 -0400

--On Thursday, 01 June, 2006 06:03 -0700 Harald Alvestrand
<[email-masked]> wrote:

> John C Klensin wrote:
> 
>> 
>> Several comments below.  Let's make an agenda item for
>> Beijing on  terminology to be used.  The decisions are more
>> or less arbitrary, but  we need to be consistent.  And I, at
>> least, have achieved confusion.
>> 
>> 
>>> still use MIME in the email message or content. we only
>>> update the fields containing the UET-8 email address to UTF8.
>> 
>> 
>> I don't understand this response, but I'm not sure I
>> understand  Charles's comment, partially because there is no
>> client practice yet.   My assumption is that clients or
>> servers supporting EAI MUST be  fully-conformant with
>> 8BITMIME.  I assume that makes the answer to  Charles's
>> question not "supposed to" but "MUST".  As far as the 
>> transport is concerned, the 2821 headers are just message

s/2821/2822/   # writing in the wee hours of the morning. sorry

>> content so,  if they are extended and transmitted in UTF-8
>> then 8BITMIME must be  present.   We could, in principle,
>> define these extensions as  subsuming 8BITMIME, but it seems
>> to me that would be error-prone and  of no particular
>> advantage. Others may disagree; if so, we need to  discuss.
> 
> 
>>    The value associated with the BODY parameter indicates
>>    whether the content body which will be passed using the
>>    DATA command consists of a MIME message containing some
>>    arbitrary octet-aligned material ("8BITMIME") or is
>>    encoded entirely in accordance with [1] ("7BIT").
>> 
>>    A server which supports the 8-bit MIME transport service
>>    extension shall preserve all bits in each octet passed
>>    using the DATA command.
>> 

Understood, but that was one reason why Yao's comment confused
me.  However, if the downgrading procedure of 8BITMIME is
invoked, then the structure of the message is changed, insert
CTEs on body parts where appropriate.  The ability to do that is
why the extension is 8BITMIME, rather than 8BITSTUFF -- the
extension is not defined for non-MIME messages.   And the design
of that downgrade procedure doesn't anticipate non-ASCII header
material (counting encoded words as ASCII).  But, again, I think
all of that should be addressed in the downgrade document -- the
discussion in the smtp extension document should be
straightforward.

    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 02 Jun 2006 11:08:10 +0100

> Charles's question not "supposed to" but "MUST".

Sure, I had just been reading RFC 1652 and came across that BODY  
parameter. I was asking whether implementations actually used it in the  
real world (sure they "MUST", but since when did that stop implementors  
 from taking shortcuts :-( ).

Essentially, you check whether the next server does IEmail. If not, then  
you either bounce or downgrade using whatever ALT-ADDRESS and/or ATOMIC  
information is available. After that, you check whether it does 8BITMIME  
and again either bounce or change the CTE according to whatever BODY=  
information is available. Essentially, that BODY= parameter fulfils the  
same role as our proposed "this is an IEmail" header - it saves you the  
bother of checking through the whole thing looking for a bit-8. So it is a  
MUST as you say. But if might be useful to point out that after you have  
downgraded in accordance with our smtp draft, you may still need to do a  
further downgrade as per RFC 1652.

>>> However, there is another possible 8BITMIME problem......
>
> The downgrade document is the place to do it.

Yes, the downgrade document is the place for most of the "messy" stuff. I  
just wanted to make sure that Content-* headers introduced into the body  
by multiparts and message/rfc822 did not get overlooked when downgrading,  
since they too might contain utf-8 and changing the CTE as per RFC 1652  
will not fix that.

> Still not quite right, because either "message" or "mail body" also  
> contains content.  E.g., we have

Yes, my main concern was that the mention of "mail body" was misleading.

> Probably not, since the addresses in the RCPT commands need not appear  
> at all in the headers.   The usual general guideline probably applies  
> here -- to the extent possible, MTAs pay attention to the  envelope  
> only.  That makes any ALT-ADDRESS in the headers mostly useful for  
> forming replies at the receiving MUA.

I think we have to assume that the To/From addresses in the headers might  
differ from those in the envelope, even in situations where we might have  
expected them to be the same. So you downgrade them independently of each  
other, using whatever ALT-ADDRESS or ATOMIC each provides
+
> specifies treatment of strings, potentially unnormalized ones, so they  
> can be compared.  I think we are going to get into bad shape if we  
> permit unnormalized/ uncanonicalized strings at all. See the discussion  
> in draft-klensin-net-utf8, but note that the conventions recommended  
> there are probably not appropriate for these addresses.  My guess is  
> that we should stick fairly closely to NFKC, but others probably want to  
> comment on that. And, of course, doing anything with NFKC or NFC gets us  
> into the Unicode versioning mess.

Yes, but when a local-part arrives at its final destination, comparison  
with some locally stored list of things is the likeliest thing to happen  
to it. If it is not just a simple comparison, then it may well be examined  
by some regular expression (or other parsing device). But either way, the  
task will be much simplified if it can be assumed to be NFKC-compliant,  
and the way to ensure that is to insist that it conformed to a suitable  
profile of Stringprep at the place where it was generated.
>
> I thought we had agreed to prohibit control characters in non-ASCII  
> local parts -- in the past, they have been more trouble than they are  
> worth.

I am all for prohibiting them. The trouble is that we cannot prohibit them  
in ASCII local-parts, where they are already allowed. One of the things we  
had to do in USEFOR was to forbid them in message-ids in Netnews, even  
though RFC 2822 allows them in Email.

> That is correct and was intentional.

Yes, I understand that.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5