Subject: Re: [EAI] Comments on SMTP extension

From: "YAO Jiankang" <[email-masked]>
Date: Tue, 11 Apr 2006 09:53:24 +0800

----- Original Message ----- 
From: "Marcos Sanz/Denic" <[email-masked]>
To: <[email-masked]>
Sent: Friday, April 07, 2006 10:59 PM
Subject: [EAI] Comments on SMTP extension

> All,
> 
> these are my comments on draft-yao-ima-smtpext-02. I haven't been 
> following the last discussions in the list. If some of these issues (or 
> any of those in my following mails) have already been dealt with, then 
> just give me a pointer.
> 

thanks a lot for your detailed comments.

> * Section 1.1: RFC 1869 is obsolete, current is 2821

will update it according to your comments.

> * Section 1.2: "Domain part of the email address may be internationalized 
> through IDNA". It should rather be "..has been internationalized..".

in our initial version, we use  "..has been internationalized..". but when we consider
that domain part of email is different from the normal domain name, we may use different internationalized method for domain part of email address. so we use "Domain part of the email address may be internationalized through IDNA". after we finalize the IMA method, we may use this kind of sentense : "Domain part of the email address has been internationalized through IDNA".

> * Section 1.2: "Since older SMTP servers and the mail-reading clients 
> [...] may not be prepared to handle these extended addresses, an SMTP 
> extension is specified". This could be nitpicking, but I don't think that 
> we are protecting those mail-reading clients with this SMTP extension. I 
> would cross them out of that sentence.

will update it according to your comments

> * Section 2.2: "DNS MUST be processed into punycode form as specified in 
> IDNA". I don't think that punycode (concrete ACE encoding) is relevant at 
> all here. Punycode is part of the internals of IDNA. I suggest modifying 
> the sentence to "DNS MUST be processed as specified in IDNA by means of 
> the ToASCII() operation".

will update it according to your comments.

> * Section 2.2: "and then MUST be compared as specified in IDNA". My 
> experience is that people overlook this comparison definition, thus it 
> would be practical to make an exact reference: Section 3.1.4 of RFC 3490

will update it according to your comments.

> * Section 2.2: s/either punycode/either ACE/

will update it according to your comments.

> * Section 2.2: "the original SMTP client SHOULD first verify that the 
> string is valid for a domain name according to IDNA rules". Why and for 
> which fields? And what if that fails? Reaction should probably depend on 
> the exact field of failure appearance. That must be defined.

it checks the validity of email address. if the client found that domain name is not valid, it means that the email address is not valid. it the address is not valid, it will refuse to send the email message.

> * Section 2.2: s/it MUST not attempt to parse/it MUST NOT attempt to 
> parse/

will update it according to your comments.

> * Section 2.2: s/Client MUST not transmit/client MUST NOT transmit/

will update it according to your comments.

> * Section 2.3: "Change the definition of 'Atom' to permit either the 
> definition above or a UTF-8 string". This complete liberalization of the 
> local part makes me feel very uneasy, not all characters are appropriate 
> to become part of an identifier (cf the Unicode Standard Annex 31, 
> Identifier and Pattern Syntax).

yes, we have noticed this problem. we will refine the definition.

> * Section 2.3: RFC2234 is obsolete, current is 4234

will update it according to your comments.

> * Section 2.3: Nitpicking "sub-domain = Let-dig [Ldh-str] / <any 
> internationalized domain label specified by IDNA>" The former are a subset 
> of the latter, thus "sub-domain = <any internationalized domain label 
> specified by IDNA>" is enough for that informal purpose.

will update it according to your comments.

> * Section 2.4: s/may set by sender/may be set by the sender/

will update it according to your comments.

> * Section 2.4: "are data or instructions embedded in the address that the 
> ACE process would hide". That should be ellaborated a bit more.

will update it according to your comments.

> * Section 2.4: s/ATMOIC/ATOMIC/g

will update it according to your comments.

> * Section 2.4: "the sender SMTP server can apply some algorithmic 
> transformation such as punycode". "Can"? "Should"? "SHOULD"? "MUST"? I am 
> for the latter in this point. In anycase the sentence would better be 
> "..apply some ASCII compatible encoding transformation".

before IETF65 meeting, we still not sure which algorithmic 
transformation will be applied. now it seems that most of us agree to use punycode. so we will update it according to your comments.

> * Section 2.4: "IDNA may also be applied to the domain part". "may"? or 
> "MUST"?

now we will use "MUST"

> * Section 2.4: I couldn't find an explanation in the algorithm for the 
> case in which neither ALT-ADDRESS nor ATOMIC are present. I guess 
> rejection is the correct thing, but then, why is ATOMIC optional at all?

it is up to the  email sender to decide whether ATOMIC is set. some of them may prefer the bounce if IMA capability is not supported.

> * Section 2.5: "the name should be in punycode form if its raw form is 
> non-ASCII". "should" or "MUST"? Cf "may" two bullets ago. In any case, and 
> since ToASCII(ascii)=ascii, the sentence should read "...always in ASCII 
> encoded form".

will update it according to your comments.

> * Section 2.5: "8BITMIME should be advertised". If 8BITMIME is a 
> requirement for IMA, as I think it is, then it should be "8BITMIME must be 
> advertised".

will update it according to your comments.

> * Section 2.5.1: s/punycode form/ACE form/

will update it according to your comments.

> * Section 2.5.2: "Internationalized domain names in Received fields should 
> be transmitted in punycode form". "should"? "SHOULD"? Wouldn't it be 
> better "MUST"?

ok, "MUST" will be used.

> * Section 2.5.4: It is moot since the inclusion of the "i18n-mail" header

ok, I will remove it.

Thanks a lot again for your very detailed comments.

Best Regards

YAO Jiankang
CNNIC

> 
> Best regards,
> Marcos
> 
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ima
> 

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Tue, 11 Apr 2006 03:30:40 -0400

Yao,

You will note below that there are several cases below in which
I have either disagreed with Marcos or agreed about the problem
but disagreed about the proposed fix.  My beliefs are not
inherently more valid than his: my only possible claim to
authority is that I've been thinking about some of these issues
for longer than either of you, and that is not a very strong
claim.  But, where there is controversy, you, Yao, need to go to
the WG list with specific questions and then make changes in
response to whatever consensus emerges, not to simply change
things based on the remarks of the last person who sent you
comments.

If you can't get resolution and consensus before you are ready
to produce -03, it is perfectly reasonable to insert notes in
the draft that indicate the section or paragraph is still under
discussion and what the issues are.  Then it becomes Harald's
problem to try to pose questions and get consensus.

Much more below.

 
>...

Please be careful about context here.  My recollection is that
1869 was referenced in the document you drew from in creating
the SMTP extension draft because it contains a certain amount of
rationale and explanation that is not present in 2821.   Knowing
that 2821 obsoletes 1869 for standards-reference purposes does
not imply that 2821 contains all of the same text and
explanations that 1869 does.   Please note that I am _not_
asking you to avoid making this change, only that you do the
work to figure out what the reference is pointing to and, on
that basis, decide whether the change is appropriate. My guess,
without rereading the two, is that it might be better to
reference both documents rather than simply dropping the 1869
reference in favor of the 2821 one.

>> * Section 1.2: "Domain part of the email address may be
>> internationalized  through IDNA". It should rather be "..has
>> been internationalized..".
> 
> in our initial version, we use  "..has been
> internationalized..". but when we consider that domain part of
> email is different from the normal domain name, we may use
> different internationalized method for domain part of email
> address. so we use "Domain part of the email address may be
> internationalized through IDNA". after we finalize the IMA
> method, we may use this kind of sentense : "Domain part of the
> email address has been internationalized through IDNA".

As far as I can tell, those two sentences are identical.   More
important, the domain part of email is a normal domain name --
nothing I know of makes it any different.    Please explain.

>> * Section 1.2: "Since older SMTP servers and the mail-reading
>> clients  [...] may not be prepared to handle these extended
>> addresses, an SMTP  extension is specified". This could be
>> nitpicking, but I don't think that  we are protecting those
>> mail-reading clients with this SMTP extension. I  would cross
>> them out of that sentence.
> 

Please do not do so without further discussion.  I have
explained, in an earlier note, why the existing claim may be
appropriate.

>...

While I agree on avoiding specific references to punycode,
rather than IDNA processing, where feasible (see above), it is
not clear to me that there is any advantage to substituting the
generic "ACE" for the specific "punycode" and I can see some
arguments for not doing so, e.g., appearing to permit local
variations on the encoding used.  Marcos, could you explain
better why you think this change would be an advantage?

>> * Section 2.2: "the original SMTP client SHOULD first verify
>> that the  string is valid for a domain name according to IDNA
>> rules". Why and for  which fields? And what if that fails?
>> Reaction should probably depend on  the exact field of
>> failure appearance. That must be defined.
> 
> it checks the validity of email address. if the client found
> that domain name is not valid, it means that the email address
> is not valid. it the address is not valid, it will refuse to
> send the email message.

First, I don't think you are responding to Marcos's comment.
The response would be, I think, to specify that this test is
being applied to the domain-part, not the entire email address.
Second, there is the appearance of an extra step here, and it is
not clear to me that step is necessary.   The proper procedure
is to apply IDNA (ToASCII()  or ToASCII(ToUnicode()) ) as
appropriate and then look the thing up in the DNS.   If it isn't
found, it isn't found and that is the end of processing since
without a valid domain, you can't open an SMTP connection.

>...
>> * Section 2.3: "Change the definition of 'Atom' to permit
>> either the  definition above or a UTF-8 string". This
>> complete liberalization of the  local part makes me feel very
>> uneasy, not all characters are appropriate  to become part of
>> an identifier (cf the Unicode Standard Annex 31,  Identifier
>> and Pattern Syntax).
> 

Be very careful here.  The IETF has not signed up for Unicode
Standard Annex 31 (that would take an update to RFC 2277) and it
is not clear that it should apply to either email addresses or
to all of the ABNF "Atom"s that might appear in the relevant
contexts.

>...
>> * Section 2.3: Nitpicking "sub-domain = Let-dig [Ldh-str] /
>> <any  internationalized domain label specified by IDNA>" The
>> former are a subset  of the latter, thus "sub-domain = <any
>> internationalized domain label  specified by IDNA>" is enough
>> for that informal purpose.
> 

Doesn't this imply a restriction on non-i18nAddresses?  If so,
is that restriction appropriate?  For example "abc--def" is a
valid 2821 and 2822 domain label, but is certainly not an
IDNA-compliant internationalized label.

>...
>> * Section 2.4: "are data or instructions embedded in the
>> address that the  ACE process would hide". That should be
>> ellaborated a bit more.
> 

This should really be left as much as possible to the framework
document, IMO.  Send text.

>...
>> * Section 2.4: "the sender SMTP server can apply some
>> algorithmic  transformation such as punycode". "Can"?
>> "Should"? "SHOULD"? "MUST"? I am  for the latter in this
>> point. In anycase the sentence would better be  "..apply some
>> ASCII compatible encoding transformation".
> 
> transformation will be applied. now it seems that most of us
> agree to use punycode. so we will update it according to your
> comments.

Huh?  I seem to have not been there when punycode was agreed to.
There are two difficulties.  First, while Adam has made some
interesting suggestions about a case-preserving version of
punycode, the only punycode we have today is the one defined in
RFC  3492.  That one is closely tried to IDNA.  It assumes NFKC
and, while I don't know if any of NFKCs other mappings are
important to it, the fact that it is not case-preserving --and
there are non-Roman scripts that do have case distinctions-- is
a disaster for email use.  It seems to me that would be a
showstopper for the use of punycode.  Second, IDNA imposes a
number of other restrictions that are inappropriate for email
local-parts.  It is not clear, as a result, that punycode can be
used to code the entire range of local parts without
modification. 

In any event,  please just refer to the downgrade document for
what is done with ATOMIC... and then make sure that, as a member
of the WG, it is satisfactorily addressed there.

>> * Section 2.4: "IDNA may also be applied to the domain part".
>> "may"? or  "MUST"?
> 
> now we will use "MUST"

See above.  Just reference the downgrade document for the
details of how ATOMIC is used.  We really don't want this type
of information in two places.  Having it in two places is an
invitation to getting things seriously wrong.

>> * Section 2.4: I couldn't find an explanation in the
>> algorithm for the  case in which neither ALT-ADDRESS nor
>> ATOMIC are present. I guess  rejection is the correct thing,
>> but then, why is ATOMIC optional at all?
> 
> it is up to the  email sender to decide whether ATOMIC is set.
> some of them may prefer the bounce if IMA capability is not
> supported.

This should probably be covered better in the framework document
as well.

>> * Section 2.5: "the name should be in punycode form if its
>> raw form is  non-ASCII". "should" or "MUST"? Cf "may" two
>> bullets ago. In any case, and  since ToASCII(ascii)=ascii,
>> the sentence should read "...always in ASCII  encoded form".
> 

No.  The sentence is an explanation of how the rules are to
developed, not a normative statement.  Note that it starts "In
general, the rule is that,..." and the "should" is intentionally
in lower case.  The next paragraph starts out "the following
subsections list and discuss all of the relevant cases".  It is
in those subsections that the normative statements occur and
they must be very clear about what is expected.

>> * Section 2.5: "8BITMIME should be advertised". If 8BITMIME
>> is a  requirement for IMA, as I think it is, then it should
>> be "8BITMIME must be  advertised".

> will update it according to your comments.

Actually, it should be "8BITMIME MUST be advertised"

> will update it according to your comments.

See above about the difference between the generic "ACE" and the
specific "punycode".  Please do not change this unless you have
a satisfactory explanation, which you are prepared to explain to
the rest of the WG, as to why the generic (and less specific)
form would be better.  The "must" in the last two lines of that
paragraph should be "MUST".

>...

Please mark it as requiring ongoing review and keep it for the
time being.  I think we need to move significantly further with
analysis of the role of that header to know whether this
discussion can be completely removed.  On the other hand, even
if it is retained, it probably should be placed in the framework
document, not here.

  regards,
      john

---

From: Marcos Sanz/Denic <[email-masked]>
Date: Wed, 12 Apr 2006 14:57:22 +0200

John,

> rationale and explanation that is not present in 2821.

It is certainly true that the fact that 2821 obsoletes 1869 for 
standards-reference purposes does not mean that the text of 1869 is a 
subset of the text of 2821. But Yao's and Lee's document goes:

"This document specifies an element of that work, specifically the 
definition of an SMTP extension [RFC1869] for IMA transport delivery"

This looks for me like a pretty banal (no pun intended) reference in the 
introductory part of the document, without any deeper meaning based on a 
certain rationale present in 1869 and not present in 2821. If it were the 
case that this specific, relevant rationale in 1869 exists, it would be 
helpful to make a more concrete reference (section X, paragraph Y of 1869) 
to be able to understand it in the context. Otherwise -and I don't want to 
give this more importance than it deserves- it would be better to 
reference the current specification that defines SMTP extensions, that is 
2821.

> nothing I know of makes it any different.    Please explain.

Certainly, I cannot follow Yao's comment here either.

> explained, in an earlier note, why the existing claim may be

Would you mind sending a pointer to that earlier note?

> better why you think this change would be an advantage?

Sure: I hate to say this, but it's about separation of concerns. The exact 
method of ASCII compatible encoding should be described somewhere else 
(downgrading-draft? separate draft?), and for the efects of the SMTP 
extension specification, the concrete method (here punycode) is 
irrelevant. I am not claiming we should or shouldn't use punycode (that 
should be a different thread), I am only claiming it's not relevant at 
this exact place.

> being applied to the domain-part, not the entire email address.

Agreed.

> without a valid domain, you can't open an SMTP connection.

This dovetails with my sentence "reaction should probably depend on the 
exact field of appearance". If the normal transport processing does not 
require making a DNS lookup on the, e.g. MAIL FROM mailbox domain part, 
then we don't *need* to artificially introduce a new DNS lookup or a new 
test "ToASCII(mailbox) doesn't fail" (ToUnicode() is defined to never 
fail, so it's not necessary here). OTOH, the most inmediate way of 
checking the valid syntax of that domain part is applying ToASCII()...

> Standard Annex 31 

I haven't claimed that we've signed up. I am only pointing out that:
a) As of now we have a complete liberal definition of the local part
b) Not all characters are appropriate to be in an identifier or in a 
certain position within an identifier
c) We are using Unicode
d) There is a part of the Unicode standard that deals with that issue

If Yao claims that they are refining the definition, I'd like to know on 
the basis of what.

> IDNA-compliant internationalized label.

It's not a restriciton. You are addressing domain names with a hyphen in 
third and fourth position (e.g. "ab--cdef"), is that right? Those ASCII 
domain labels are still valid after the advent of IDNA and are valid 
*internationalized* domain labels. ToASCII("ab--cdef") = "ab--cdef" and 
ToUnicode("ab--cdef") = "ab--cdef". ASCII domain labels are (part of) 
valid IDNs.

I agree that "<any internationalized domain label  specified by IDNA>" is 
not a formal definition anyway, but the ABNF in the current form is 
redundant at best, misleading at worst.

> document, IMO.  Send text.

I cannot send text because I don't have enough background on the problem. 
It has been addressed in the list before, but all I could read where vague 
references like "netnews experience", "bang-paths", "%-hacks", "imbedded 
commands, subaddresses, and so on". I think it could be helpful for the 
reader to document the problem, or alternatively, include a reference to a 
place where the problem has been documented. To place that 
explanation/reference in the framework document is fine by me.

> Huh?  I seem to have not been there when punycode was agreed to.

Neither me.

> RFC  3492.  That one is closely tried to IDNA.  It assumes NFKC

Punycode is not NFKC/Nameprep dependent and does not assume it.

> important to it, the fact that it is not case-preserving

Punycode does preserve case.

> what is done with ATOMIC...

Correct. It is exactly my suggestion that the concrete ACE encoding is not 
relevant in the SMTP document and must be addressed separately.

> invitation to getting things seriously wrong.

I thought you always have to write all things in all possible places where 
the reader could expect to find them :-) *scnr*

> This should probably be covered better in the framework document

Disagree. The SMTP algorithm must be completely defined in the SMTP 
extension document and cover the case there in which neither ALT-ADDRESS 
nor ATOMIC are present. 

> they must be very clear about what is expected.

It's not about 2119 language here, but about internal coherence. The 
sentence at stake is:

"the name should be in punycode form if its raw form is non-ASCII"

In its current form..
a) ..it collides with the sentence "IDNA may also be applied to the domain 
part" in the Section 2.4. "may" or "should"? I am not defending either 
option now (though I like "must" best), I am only pointing out an 
incoherence.
b) ..it mentions punycode, which is wrong, since punycode is only a part 
of IDNA, which does not include Nameprep and does not include ACE prefix 
prepending.
c) ..it includes a misleading condition ("if its raw form is 
non-ASCII..."), since ToASCII("ASCii")="ASCii" 

Again, my suggestion for the reformulation is:

"the name should/may/must be ACE encoded as of IDNA [RFC3490]"

> Actually, it should be "8BITMIME MUST be advertised"

Agreed.

> a satisfactory explanation,

I have (hopefully) given that, see above. But for me, it looks like you 
are contradicting yourself.

> time being.

That is fine by me, too. I just wanted to have the documents in sync.

Regards,
Marcos