Subject: Re: [EAI] text for ascii@non-ascii issues

From: "Jiankang YAO" <[email-masked]>
Date: Tue, 17 Aug 2010 17:16:07 +0800

based on the comments from john and shawn(thanks), updatd to the following texts:

   If a server advertises UTF8SMTPbis and the client does not recognize the
   extension, the client should send a regular message only using the legacy SMTP defintions [RFC5321] and
   [RFC5322]. 

A UTF8SMTPbis aware MUA/MSA sending to a legacy SMTP server [RFC5321] and
   [RFC5322]  may convert  the ASCII@non-ASCII addresse into the format of ASCII@A-label
 where A-label is defined in IDNA definitions [RFC5890] if the email address is in the format of ASCII@non-ASCII.

----- Original Message ----- 
From: "Shawn Steele" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, August 17, 2010 3:52 AM
Subject: Re: [EAI] text for ascii@non-ascii issues

>I agree with John.  I also don't think the last sentence is necessary.  RFC5322 already says what legacy servers do.  If we want to say that UTF8SMTP aware clients MUST NOT send such messages to legacy servers, that's different, but it's pointless to say in here what an RFC5322 server can/cannot do.
> 
>>    [RFC5322],  the legacy SMTP servers MUST reject the message
>> with the    ASCII@non-ASCII if the non-ASCII domain part is
>> not transformed into    the format of A-label by the MUAs/MSAs."
> 
> An attempt at making it more readable:  I don't claim much, I didn't spend a lot of time on it.
> 
>> If a server advertises UTF8SMTP and the client does not
>> recognize the extension, the client may still send a 
>> message using legacy SMTP [LegacySMTP].  
>> If such a Legacy SMTP client uses the IDNA Definitions [IDNA] and 
>> an ascii@a-label address, then the
>> UTF8SMTPbis server MUST
>>    recognize and decode any A-Label(s) as appropriate.
>>
>>  A UTF8SMTP aware MUA/MSA sending to a legacy SMTP server
>> should convert ASCII@non-ASCII addresses into IDNA A-labels.
> 
> Then add [legacySMTP] and [IDNA] to the references, using 5321/5322 & 5890.
> 
> -Shawn
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 16 Aug 2010 11:43:59 -0400
> From: John C Klensin <[email-masked]>
> Subject: Re: [EAI] text for ascii@non-ascii issues
> To: YAO Jiankang <[email-masked]>, [email-masked]
> Message-ID: <[email-masked]>
> Content-Type: text/plain; charset=us-ascii
> 
> 
> 
> --On Monday, August 16, 2010 17:42 +0800 YAO Jiankang <[email-masked]> wrote:
> 
>> 
>> 
>> the last paragragh of section 3.2 of rfc5336bis is supposed to the 
>> following text:
>> 
>> 
>> 
>>    If a server advertises UTF8SMTP and the client does not
>> recognize the    extension, the client may send a regular
>> message only using the legacy SMTP defintions [RFC5321] and [RFC5322].  
>> In this case, the MUAs/MSAs  may continue to use
>> the    IDNA Definitions document[RFC5890] to transform the
>> domain portion of an address to A-label    [RFC5890] if the
>> email address is in the format of ASCII@non-ASCII,   then the
>> UTF8SMTPbis server MUST
>>    recognize and decode the ACE label(s) as appropriate.
>>    According to the legacy SMTP specifications [RFC5321] and
>>    [RFC5322],  the legacy SMTP servers MUST reject the message
>> with the    ASCII@non-ASCII if the non-ASCII domain part is
>> not transformed into    the format of A-label by the MUAs/MSAs.
> 
> I think this is consistent with what the WG agreed to in Maastricht (and to which there has been no dissent on the mailing list), but it is in serious need of editing.  The text is hard to follow and does not appear to me to be acceptable for something to be submitted to the IESG for IETF Last Call.
> 
>     john
> 
> 
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Mon, 16 Aug 2010 12:00:02 -0700 (PDT)
> From: [email-masked]
> Subject: [EAI] I-D Action:draft-ietf-eai-frmwrk-4952bis-03.txt
> To: [email-masked]
> Cc: [email-masked]
> Message-ID: <[email-masked]>
> Content-Type: text/plain; charset="us-ascii"
> 
> 
> 
> ------------------------------
> 
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
> 
> 
> End of IMA Digest, Vol 61, Issue 10
> ***********************************
> 
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Tue, 17 Aug 2010 14:32:45 -0400

> based on the comments from john and shawn(thanks), updatd to

This is a lot closer to something that I can parse.  Thanks.  I
suggest two additional changes (without going back and looking
at the draft itself, so this might not quite fit):

(1) Write an introductory sentence, if there isn't one there
already that says something like:

"There are two cases of interest when UTF8SMTPbis-enabled and
legacy systems interact:"

Then turn the two cases into bullet points or even subsections,
e.g.,

"o Server support and advertisement for UTF8SMTPbis but no
client support.

	The client should send a regular message only using the
	legacy SMTP [RFC5321] and Internet Message Header
	[RFC5322] definitions, since it will have no idea how to
	do anything else.

o No advertised server support for UTF8SMTPbis, but a client
(MUA/MSA combination) that is UTF8SMTPbis-capable.

	The client MUST assume that the server is capable of
	handling only traditional Internet transport [RFC5321]
	and messages [RFC5322].  If the relevant addresses are
	in ASCII@non-ASCII form, i.e., the domain-part in
	non-ASCII but the local-part is ASCII, it MAY convert
	U-labels (see IDNA2008 definitions [RFC5890]) in the
	domain-part to A-labels and convert any non-ASCII header
	fields to encoded names (see RFC-whatever-that-is) and
	transmit the resulting legacy-format message."

I think that might be followed by a statement about what happens
if either the local-part is non-ASCII, the Client decides to not
make the conversions, or the headers fields contain some
non-ASCII that cannot be converted to encoded words.

Note that this issue about the headers is identical to the one
that Fujiwara-san raised vis-a-vis Framework.  We can have
messages that require the UTF8SMTP[bis] extensions without
having any non-ASCII addresses present.

If you decide to use any of the text above, please be sure that
terms like "local-part" and "domain-part" are used and spelled
exactly as they are in RFC 5321: I haven't checked and relying
on my memory is a really bad idea.

Is it clear why the above style of presentation is easier to
understand?

best,
    john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 17 Aug 2010 19:42:49 +0000

Being very nit-picky:

> "There are two cases of interest when UTF8SMTPbis-enabled and legacy systems interact:"

There're maybe other cases?  So maybe?

	Two cases of special interest when UTF8SMTPbis-enabled and legacy systems interact:

We keep scaling back, then extending the "legacy" definition.  Does this 2nd case have to be so long?  This seems sufficient?

	The client MUST assume that the server is not capable of
	handling UTF8SMTPbis mail.  Such legacy connections are
	covered by the appropriate standards and are out of the
	scope of this RFC.  In such cases, messages with non-ASCII
	content may not be allowed to be sent.

IMO: this clearly points out that "newer" clients might still be restricted by the existing standards.  

> I think that might be followed by a statement about what happens if either the local-part is non-ASCII, the Client decides to not make the conversions, or the headers fields contain some non-ASCII that cannot be converted to encoded words.

I think this is covered by above.

I'm also happy with Yao's earlier suggestion, though 2 bullets may make it more readable?

Again, I think the key point is "If the client and server don't both support UTF8SMTP, then these standards don't cover the transmission, and it may not be possible to send mail."  If we say this document doesn't cover the case, I'm not sure we have to spell out how to use legacy smtp.  Legacy smtp and IDNA exists independently of this RFC.  In particular, I do NOT think we have to say "use 5321, which needs 5322, and could use 1652 (and a lot of other RFCs), and 589x might be helpful,.... ".

Clearly if you're using UTF8SMTPbis, then you have to obey it, if not, then use something else.  The ONE case that impacts UTF8SMTP and might be worth mentioning is that if the UTF8SMTP server gets a punycode domain name, it should convert that back to UTF-8.  But that's saying what you do if you're a UTF8SMTP server, not what you don't do if you don't support the standard.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Tue, 17 Aug 2010 16:07:03 -0400

> 	Two cases of special interest when UTF8SMTPbis-enabled and

That works for me.  On the other hand, note that there are,
logically, only four cases

	(i) Both client and server support EAI
	(ii) Server does, client doesn't
	(iii) Client does, server doesn't
	(iv) Neither do.

Unless I've forgotten my mathematical logic, that describes the
universe; there are no other cases.

> We keep scaling back, then extending the "legacy" definition.
> Does this 2nd case have to be so long?  This seems sufficient?
> 

Well, I've been trying to puzzle this out and admit I've
probably been moving back and forth.  The true legacy case is
(iv) above and we have no possible interest or anything to say
about it.  However, in case (ii), we can say some things if we
want to about how the server is to behave.   I don't think there
is much we can say that would be useful because the SMTP-sender
(client) has to send a legacy (5321/5322-conforming) message
and, given such a message, there isn't anything the
SMTP-receiver (server) can do other than treat that exactly the
way a non-EAI-compliant server would (we have never
specified/supported in-transit upgrade).   If we want to, we
could specify conversion of A-labels back to U-labels, however
(see below).

Case (iii) above, and my second bullet, are different.  We _can_
require that all EAI-compliant SMTP clients be IDNA-aware and,
indeed, if they are going to look up domain names to resolve MX
records, they pretty much have to be.  Assuming one believes the
traditional IDNA model, which neither of us really do, that
means they have the code needed to do a U-label to A-label
conversion.  We really cannot prevent them from doing that
without violating the IDNA specs, so the only question is
whether we encourage it (e.g., by making a comment with MAY) or
discourage or prohibit it (because the EAI specs are
deliberately unfriendly to addresses with A-labels in the domain
part).   

The same comment applies to the subcase with non-ASCII headers.
Such headers cannot be transmitted at all without 8BITMIME and
we require that anything that supports UTF8SMTP[bis] also
support 8BITMIME.  But the 8BITMIME spec doesn't require the
ability to downgrade and just about nothing _requires_ the
ability to convert UTF-8 strings to encoded words.  So we can't
possibly require that particular downgrade, but we cannot
reasonably prohibit it either since the magic that turns
messages with UTF-8 headers or addresses into legacy ones is
behind a curtain that we cannot penetrate even though what
should happen if anything happens is absolutely obvious because
we do have clear specs for U-label <-> A-label conversions and
for UTF-8-String <-> encoded word ones (note the difference
between those cases and the situation of trying to "convert" a
local-part); the question is whether to say nothing, to make a
recommendation, or to at least make the possibility explicit. 

I guess I'm recommending against "say nothing" because it is
just an invitation to confused users. 

> be restricted by the existing standards.  

Sure.  And by what options of those existing standards they have
chosen to support.

>> I think that might be followed by a statement about what
>> happens if either the local-part is non-ASCII, the Client
>> decides to not make the conversions, or the headers fields
>> contain some non-ASCII that cannot be converted to encoded
>> words.
> 
> Again, I think the key point is "If the client and server
> don't both support UTF8SMTP, then these standards don't cover
> the transmission, and it may not be possible to send mail."
> If we say this document doesn't cover the case, I'm not sure
> we have to spell out how to use legacy smtp.  Legacy smtp and
> IDNA exists independently of this RFC.  In particular, I do
> NOT think we have to say "use 5321, which needs 5322, and
> could use 1652 (and a lot of other RFCs), and 589x might be
> helpful,.... ".

I agree with this last.

> Clearly if you're using UTF8SMTPbis, then you have to obey it,
> if not, then use something else.  The ONE case that impacts
> UTF8SMTP and might be worth mentioning is that if the UTF8SMTP
> server gets a punycode domain name, it should convert that
> back to UTF-8.  But that's saying what you do if you're a
> UTF8SMTP server, not what you don't do if you don't support
> the standard.

But that is my case (ii) above.  

    john

---

From: Joseph Yee <[email-masked]>
Date: Tue, 17 Aug 2010 16:13:29 -0400

> I think this is covered by above.

Shawn,

Referring to just RFC 5321, 5322, 5980 is more specific and accurate than "messages with non-ASCII content may not be allowed to be sent".

This is me relying on memory too, recalling RFC 5321 had no mention of any conversion of domain-part.  The options are not clear for mismatched client/server capabilities.

Joseph

---

From: Shawn Steele <[email-masked]>
Date: Tue, 17 Aug 2010 20:52:48 +0000

For case #iii I don't think it's quite so simple (maybe I'm wrong).  We can suggest using IDNA (if we want), but that's not really going to guarantee that it behaves.  (It "should", but are we going to go into failure cases?  2003 vs 2008? ).  I still think (iii) is a lot like (iv).  If my existing mail client right now gets non-ASCII data it needs to do something.  Its options look a lot like (almost exactly like?) a case (iii) client's options.  So I'd rather say that if you're in bucket iii, pretend you're in bucket iv and leave it at that.  I suppose we could forbid punycoding the domain, but I think any such recommendation would just be ignored, at least in the near term.  (Even if we said "MUST", they'd just say they were a legacy SMTP client that supported EAI instead of an EAI client that fell back to legacy SMTP).

Clients may have other options?  If I try to send EAI mail with a pretty header that says "X-SendingClient my favorite Japanese client" (in Japanese), but the message is entirely ascii, then I could just resend without the X-SendingClient, or using a different string for it.  

I suppose another axis would be "client ONLY supports EAI".  So any downlevel message would fail.  (Presumably we wouldn't see these very soon, unless maybe it's very narrow like custom clients for a specific service provider (phone talking to telco)).

Anyway, I'm not sure what any of this has to do with EAI.  If the client can't send with EAI do we really care about the details?  I can think of "informational" type things to suggest (hey, if you don't support EAI, try sending without EAI, or, like, ask the sender if they have an ASCII address), but nothing that's necessary for the EAI protocol itself.  (Excepting if we want to talk about the (ii) case where I get a punycode address sent to my server)

I also don't think this topic is worth blocking the spec.  The client's option(s) are reasonably clear and out of our control, so whatever language someone feels is important to have I can probably live with.  I'd prefer being very brief to make it easier for the reader to see what they have to add for new "EAI" behavior, instead of restating what they already know (or should know, or think they know) about legacy SMTP.

Another way to think about it:  You "own" an SMTP client (or server).  Your manager comes in and says "we have to add support for EAI, make it happen."  At that time, you're going to be "adding" support for EAI.  You won't touch any legacy stuff because your legacy customers still need to use your product and you don't want to break them.  So you'll have an "EAI" path and a "no-EAI" path.  Presumably you'll try the 1st and fallback or something.  There's not much point in us overspecifying this stuff.

- Shawn

 
http://blogs.msdn.com/shawnste
(Selfhost 7799)

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Tuesday, August 17, 2010 1:07 PM
To: Shawn Steele; Jiankang YAO; [email-masked]
Subject: RE: [EAI] text for ascii@non-ascii issues

> 	Two cases of special interest when UTF8SMTPbis-enabled and legacy 

That works for me.  On the other hand, note that there are, logically, only four cases

Unless I've forgotten my mathematical logic, that describes the universe; there are no other cases.

> 	content may not be allowed to be sent.

Well, I've been trying to puzzle this out and admit I've probably been moving back and forth.  The true legacy case is
(iv) above and we have no possible interest or anything to say about it.  However, in case (ii), we can say some things if we
want to about how the server is to behave.   I don't think there
is much we can say that would be useful because the SMTP-sender
(client) has to send a legacy (5321/5322-conforming) message and, given such a message, there isn't anything the SMTP-receiver (server) can do other than treat that exactly the way a non-EAI-compliant server would (we have never
specified/supported in-transit upgrade).   If we want to, we
could specify conversion of A-labels back to U-labels, however (see below).

Case (iii) above, and my second bullet, are different.  We _can_ require that all EAI-compliant SMTP clients be IDNA-aware and, indeed, if they are going to look up domain names to resolve MX records, they pretty much have to be.  Assuming one believes the traditional IDNA model, which neither of us really do, that means they have the code needed to do a U-label to A-label conversion.  We really cannot prevent them from doing that without violating the IDNA specs, so the only question is whether we encourage it (e.g., by making a comment with MAY) or discourage or prohibit it (because the EAI specs are deliberately unfriendly to addresses with A-labels in the domain
part).   

The same comment applies to the subcase with non-ASCII headers.
Such headers cannot be transmitted at all without 8BITMIME and we require that anything that supports UTF8SMTP[bis] also support 8BITMIME.  But the 8BITMIME spec doesn't require the ability to downgrade and just about nothing _requires_ the ability to convert UTF-8 strings to encoded words.  So we can't possibly require that particular downgrade, but we cannot reasonably prohibit it either since the magic that turns messages with UTF-8 headers or addresses into legacy ones is behind a curtain that we cannot penetrate even though what should happen if anything happens is absolutely obvious because we do have clear specs for U-label <-> A-label conversions and for UTF-8-String <-> encoded word ones (note the difference between those cases and the situation of trying to "convert" a local-part); the question is whether to say nothing, to make a recommendation, or to at least make the possibility explicit. 

I guess I'm recommending against "say nothing" because it is just an invitation to confused users. 

> restricted by the existing standards.

Sure.  And by what options of those existing standards they have chosen to support.

>> I think that might be followed by a statement about what happens if 
>> either the local-part is non-ASCII, the Client decides to not make 
>> the conversions, or the headers fields contain some non-ASCII that 
>> cannot be converted to encoded words.
> 
> Again, I think the key point is "If the client and server don't both 
> support UTF8SMTP, then these standards don't cover the transmission, 
> and it may not be possible to send mail."
> If we say this document doesn't cover the case, I'm not sure we have 
> to spell out how to use legacy smtp.  Legacy smtp and IDNA exists 
> independently of this RFC.  In particular, I do NOT think we have to 
> say "use 5321, which needs 5322, and could use 1652 (and a lot of 
> other RFCs), and 589x might be helpful,.... ".

I agree with this last.

> Clearly if you're using UTF8SMTPbis, then you have to obey it, if not, 
> then use something else.  The ONE case that impacts UTF8SMTP and might 
> be worth mentioning is that if the UTF8SMTP server gets a punycode 
> domain name, it should convert that back to UTF-8.  But that's saying 
> what you do if you're a UTF8SMTP server, not what you don't do if you 
> don't support the standard.

    john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 17 Aug 2010 21:06:24 +0000

> Referring to just RFC 5321, 5322, 5980 is more specific and accurate than "messages with non-ASCII content may not be allowed to be sent".

So, if I'm going to strictly follow that statement, then my application is forbidden from using 8BITMIME and UTF-8 for the already-encoded body, even if I can punycode a domain name and otherwise resolve the non-ASCII conflict.  So, rather obviously, clients are going to pull in other RFCs that are helpful to make the mail get sent.

I don't really care if we're specific or not, I just think it makes it harder to read and a little more confusing.  I'd rather say "If the server doesn't send UTF8SMTPbis, then you can't use it," and leave it to them, or an informational RFC, as how they might do fallback.

Off the top of my head, reasons why it might not work in non-EAI:

* ascii@non-ascii - they could resolve this by using punycode, if they want, but it's not completely guaranteed to work.
* non-ascii@non-ascii - they're stuck, but they might be able to resolve it to all-ascii if they have an alternate address for the recipient/sender (after all, this is the client, and they have the user's address book, &/or might be able to ask the user for more information).
* Some header they want isn't ASCII - they might be able to MIME encode it, or maybe even drop it if not required.  (Again this is the originator, the "message" may not even exist yet, and there is opportunity to change some things if necessary).
*non-ASCII body - hopefully MIME would help them, but if the server doesn't support that either they're kinda stuck.
* default - presumably the user could resend the message if necessary, or even a different message.  They could even post a file to a web site and send a very boring ASCII message with a link to the file if MIME really doesn't work.  They could even use a different client if necessary.  (This isn't really what we're talking about, but worth remembering).

Anyway, if you guys think that we need to discuss these here, I won't stop you.  I'd just rather seeing it in the informational how-to, or at least "easy to read".

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Tue, 17 Aug 2010 18:38:43 -0400

--On Tuesday, August 17, 2010 20:52 +0000 Shawn Steele
<[email-masked]> wrote:

> For case #iii I don't think it's quite so simple (maybe I'm
> wrong).  We can suggest using IDNA (if we want), but that's
> not really going to guarantee that it behaves.  (It "should",
> but are we going to go into failure cases?  2003 vs 2008? ).

That is a distraction, I think.  First, we reference 2008 and
only 2008 and rely on the (entirely realistic, IMO) assumptions
that all implementations of EAI are going to be either derived
from the experimental implementations or new since IDNA2008, so
no transition problems there.  Perhaps as important in practice,
remember that there are only two areas of practical
incompatibility: one character each for two language groups that
haven't exactly been beating a path to the EAI door and the
joiner characters without which a number of languages were
claimed to be largely unusable under IDNA2003 (i.e., when they
beat a path to the door, it will be with the IDNA2008
interpretation).

> I still think (iii) is a lot like (iv).  If my existing mail
> client right now gets non-ASCII data it needs to do something.
> Its options look a lot like (almost exactly like?) a case
> (iii) client's options.  So I'd rather say that if you're in
> bucket iii, pretend you're in bucket iv and leave it at that.
> I suppose we could forbid punycoding the domain, but I think
> any such recommendation would just be ignored, at least in the
> near term.  (Even if we said "MUST", they'd just say they were
> a legacy SMTP client that supported EAI instead of an EAI
> client that fell back to legacy SMTP).

The problem is that nothing --especially nothing that we can
do-- can prohibit a small variation on your last case -- a
legacy 5321/5322 MUA (or Submission server) that supports IDNA
and follows the rules that IDNA (both versions) say it is
supposed to follow: converting U-labels to A-labels before
passing them into IDNA-unaware slots.   And an IDNA-unaware slot
is exactly what the domain part of an email address is --
IDNA2008 (I think Rationale but it could be in Defs), even calls
that out as a specific example.
 
> Clients may have other options?  If I try to send EAI mail
> with a pretty header that says "X-SendingClient my favorite
> Japanese client" (in Japanese), but the message is entirely
> ascii, then I could just resend without the X-SendingClient,
> or using a different string for it. 

Sure.  But, if the relevant header material were the Subject
header field or a display name, your simply discarding it would
be likely to make your users _very_ unhappy (especially since
some anti-spam mechanisms give "probably spam" scores to
messages with no Subject header fields).    So, while you are
correct, I think you are describing an unrealistic edge case in
which that type of header contains the only UTF-8 material in
the headers.

> I suppose another axis would be "client ONLY supports EAI".
> So any downlevel message would fail.  (Presumably we wouldn't
> see these very soon, unless maybe it's very narrow like custom
> clients for a specific service provider (phone talking to
> telco)).

I am not certain what "ONLY supports EAI" is expected to mean.
One cannot support EAI at all without supporting classic (even
non-MIME) 5321/5322 messages.

> Anyway, I'm not sure what any of this has to do with EAI.  If
> the client can't send with EAI do we really care about the
> details?  I can think of "informational" type things to
> suggest (hey, if you don't support EAI, try sending without
> EAI, or, like, ask the sender if they have an ASCII address),
> but nothing that's necessary for the EAI protocol itself.
> (Excepting if we want to talk about the (ii) case where I get
> a punycode address sent to my server)

At one level, I don't know the answer to that question (you need
to convince the WG, not me).  At another, I think it would be
perfectly rational to include a statement in the introductory
material to 5336bis (possibly with parallel language in 4952bis
(Framework) that said something like "this specification applies
only when a UTF8SMTPbis-aware client is trying to send a message
that requires the UTF8SMTPbis extensions to a server that
advertises the ability to handle those extensions as specified
here.  For all other cases, and for addresses and messages that
do not require the extension, SMTP clients and servers are
expected to behave exactly as specified in RFC 5321 and its
successors." and then drop all of this material.

I've been trying to get the text we have been discussing
clarified only because the Editor still believes, in good faith,
that the WG wants these cases to be explicitly covered in some
way.

> I also don't think this topic is worth blocking the spec.  The
> client's option(s) are reasonably clear and out of our
> control, so whatever language someone feels is important to
> have I can probably live with.

I suggest that the discussions of the last 24 hours, including
your notes, indicates that the clients options are not quite
that clear.  YMMD.

>  I'd prefer being very brief to
> make it easier for the reader to see what they have to add for
> new "EAI" behavior, instead of restating what they already
> know (or should know, or think they know) about legacy SMTP.

That is a reasonable; see above.

> Another way to think about it:  You "own" an SMTP client (or
> server).  Your manager comes in and says "we have to add
> support for EAI, make it happen."  At that time, you're going
> to be "adding" support for EAI.  You won't touch any legacy
> stuff because your legacy customers still need to use your
> product and you don't want to break them.  So you'll have an
> "EAI" path and a "no-EAI" path.  Presumably you'll try the 1st
> and fallback or something.  There's not much point in us
> overspecifying this stuff.

Granted.  But suppose you don't own an SMTP client and your
manager comes in and says "we've decided that we need a mail
client in our portfolio and you get to implement a new one that
offers all modern features, including non-ASCII addresses and
UTF-8 headers, and that provides the best user experience
possible".   The behavior that follows is probably a little
different than that for the scenario you describe above.  And we
have to deal with both audiences.

--On Tuesday, August 17, 2010 21:06 +0000 Shawn Steele
<[email-masked]> wrote:

>> Referring to just RFC 5321, 5322, 5980 is more specific and
>> accurate than "messages with non-ASCII content may not be
>> allowed to be sent".
> 
> So, if I'm going to strictly follow that statement, then my
> application is forbidden from using 8BITMIME and UTF-8 for the
> already-encoded body, even if I can punycode a domain name and
> otherwise resolve the non-ASCII conflict.  So, rather
> obviously, clients are going to pull in other RFCs that are
> helpful to make the mail get sent.
> 
> I don't really care if we're specific or not, I just think it
> makes it harder to read and a little more confusing.  I'd
> rather say "If the server doesn't send UTF8SMTPbis, then you
> can't use it," and leave it to them, or an informational RFC,
> as how they might do fallback.

Please understand that I'm trying to explain a perceived
miscommunication, rather than taking sides, but...

 * The example about 8BITMIME is probably a poor one because
UTF8SMTP and UTF8SMTPbis require support for 8BITMIME.  But we
cite key base specifications all the time without, e.g., citing
TCP because SMTP is now expected to run over it.  On the other
hand, there are mail transports floating around that aren't SMTP
(much less extended SMTP) and mail header formats that are not
strictly conformant to 5322, so citing those specs rather than
making citation-less references to, e.g., "email" seems
plausible even if one can argue that it shouldn't, strictly
speaking, be necessary.  So I don't see where you are going.
 
> * ascii@non-ascii - they could resolve this by using punycode,
> if they want, but it's not completely guaranteed to work.

See above.  They are actually require to do this if they are
IDNA-compliant (either version) and I think IDNA is, in
practice, a normative requirement for UTF8SMTP.  So "not
completely guaranteed to work" requires an appeal to either edge
cases or incorrect implementations.  There is no defense against
the latter, but it doesn't, and should not, prevent our writing
standards-track specs.

> *
> non-ascii@non-ascii - they're stuck, but they might be able to
> resolve it to all-ascii if they have an alternate address for
> the recipient/sender (after all, this is the client, and they
> have the user's address book, &/or might be able to ask the
> user for more information).

Yes, but that is part of the scope for a document that the WG
has not yet agreed to write.  If something is behaving like a
legacy system (with or without IDNA support), this is simply an
invalid address syntax.

 * Some header they want isn't
> ASCII - they might be able to MIME encode it, or maybe even
> drop it if not required.  (Again this is the originator, the
> "message" may not even exist yet, and there is opportunity to
> change some things if necessary).

Right.  But this is covered by the existing language, so your
point is?

> *non-ASCII body - hopefully
> MIME would help them, but if the server doesn't support that
> either they're kinda stuck. 

If the non-ASCII body is properly MIME encoded and identified,
then it can be transmitted only by encoding it or using
8BITMIME.  The latter is extremely specific about its particular
downgrading mechanisms _and_ about rejecting the message if the
downgrade is not possible.  No issue here that I can see.

* default - presumably the user
> could resend the message if necessary, or even a different
> message.  They could even post a file to a web site and send a
> very boring ASCII message with a link to the file if MIME
> really doesn't work.  They could even use a different client
> if necessary.  (This isn't really what we're talking about,
> but worth remembering).

In addition, nothing qualifies it as the default.

> Anyway, if you guys think that we need to discuss these here,
> I won't stop you.  I'd just rather seeing it in the
> informational how-to, or at least "easy to read".

Ack.  And, again, I'm not taking a position on that.

    john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 17 Aug 2010 23:14:31 +0000

> Sure.  But, if the relevant header material were the Subject header field or a display name, your simply discarding it would be li kely to make your users _very_ unhappy (especially since some anti-spam mechanisms give "probably spam" scores to
messages with no Subject header fields). 

I didn't say "the client should drop Subjects".  In practice THE SUBJECT: DOESN'T EXIST YET.  The client doesn't have to build a header, add a body, and then shove it into the pipe.  A client may open a connection without a message, then generate the information to populate Subject:, etc. on the fly.  In that case it's meaningless to say what you can or cannot convert.

Anyway this misses my point, which was that clients may (or may not) be able to do things that we cannot anticipate. IF (and I grant that it's a big if).  If they only had some droppable header that was non-ASCII, then they could do that.  If they wanted to use "best fit" (shudder) behavior to convert extended Latin to ASCII in the body (since they're the client), they could do that too.

A client can if it never talks to non-UTF8SMTP servers.  If Nokia builds a cell phone for a Japanese telco and always talks to the telco's servers, then everything is controlled and there's never a non-UTF8SMTP path.  I don't think we say that clients have to support no-UTF8SMTP cases (and I'm not sure they'd listen in this scenario if we did.)

> "this specification applies only when a UTF8SMTPbis-aware client is trying to send a message that requires the UTF8SMTPbis extensions to a server that advertises the ability to handle those extensions as specified here.  For all other cases, and for addresses and messages that do not require the extension, SMTP clients and servers are expected to behave exactly as specified in RFC 5321 and its successors." and then drop all of this material.

That would be fine.

> I've been trying to get the text we have been discussing clarified only because the Editor still believes, in good faith, that the WG wants these cases to be explicitly covered in some way.

That would be fine too :)

> Granted.  But suppose you don't own an SMTP client and your manager comes in and says "we've decided that we need a mail client in our portfolio and you get to implement a new one that offers all modern features, including non-ASCII addresses and
UTF-8 headers, and that provides the best user experience
possible".   The behavior that follows is probably a little
different than that for the scenario you describe above.  And we have to deal with both audiences.

Presumably they'd still want to be able to drop-in replace legacy systems and would have some compatibility requirements.  Or maybe not (Nokia/telco example).  I just think if we start running into things that impact business decisions made by app developers then they need to add demonstratable value to EAI.  New or old, app developers will build what they think the market needs, and think of edge cases we'll miss.

FWIW: I think the "working group's" desire to include this language is circular.  I think it was added because someone thought someone else thought it would be a good idea.  I'm not hearing a lot of people saying that they MUST have language describing the legacy interaction included in UTF8SMTP.  I like your idea of an generic "this isn't the legacy spec" statement.

- Shawn