Subject: [EAI] My view of a "downgraded" address.

From: Shawn Steele <[email-masked]>
Date: Thu, 20 Aug 2009 21:50:17 +0000

My view of EAI and the current standards is kinda simple:

* Email uses UTF-8 instead of ASCII.  That shouldn't be too hard and should pretty much work everywhere.
* Legacy software doesn't use UTF-8, so some sort of handshaking is required to say "I know UTF-8."
* Also I still need to email older systems that need an ASCII address.  So my UTF-8 mailbox needs an ASCII alias.
  * If my mail system handled multiple addresses/aliases for ASCII, then when it's updated to UTF-8 all the same mechanisms should still work.
  * A little different because all mail accounts would pretty much have aliases.
  * Account management tools might need updated to make the pairing easier to manage.
* Something interesting should happen when the "I know UTF-8" handshaking fails.

The "only" changes needed to support that model are 1) Use UTF-8 instead of ASCII, and 2) send/recognize the UTF-8 handshaking.

Probably no change is really necessary to handle ASCII aliases for UTF-8 mailboxes, sendmail, Exchange, etc already handle multiple aliases.  I could see improvements in tools, but I don't think it's required.

When handshaking fails.

The most interesting part of the problem is when "I know UTF-8" handshaking fails.  IMO the sender can then just use the ASCII addresses, convert to ASCII, and get replies in ASCII.  That's probably trivial when the client/server of the sender are well integrated.  It's probably a bit harder if there's some sort of gateway involved.  I would prefer that in "UTF-8" aware mode, simple ASCII information was included in case it had to be converted to ASCII-only mode.  Specifically the <Unicode<ASCII>> type syntax might be useful.

I don't see any big problems/disagreement with the part before handshaking fails.  The only real difference between no-downgrade and minimal downgrade is whether an ASCII form of the address is necessary, and that could be handled by <Unicode<ASCII>>, so I'm not sure there's anything blocking the standards that don't handle handshaking.

I would prefer that the standards allow the concept of aliasing rather than requiring an independent architecture where a server depends on an independent pair of addresses.  That seems like an architectural decision for an implementation of how to handle aliases rather than a standards requirement.

Thanks,

- Shawn

 
http://blogs.msdn.com/shawnste

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 21 Aug 2009 15:38:57 +0100

> When handshaking fails.
>
> The most interesting part of the problem is when "I know UTF-8"  
> handshaking fails.  IMO the sender can then just use the ASCII  
> addresses, convert to ASCII, and get replies in ASCII.  That's probably  
> trivial when the client/server of the sender are well integrated.  It's  
> probably a bit harder if there's some sort of gateway involved.  I would  
> prefer that in "UTF-8" aware mode, simple ASCII information was included  
> in case it had to be converted to ASCII-only mode.  Specifically the  
> <Unicode<ASCII>> type syntax might be useful.

Yes, but you are looking at the wrong case.

A UTF-8 aware has no business sending UTF-8  or headers containing  
<unicode<ascii>> to a submission agent that does not advertise UTF8SMTP,  
and a UTF-8 server has no business sending such messages onwards to a  
further server that does not advertise UTF8SMTP. The protocols we have  
written simply do not permit that to happen, and if some supposedly  
UTF-8-aware system allows it to happen, then it is not correctly  
implemented according to what we have written. We simply have to assume  
that new software is properly written, otherwise we might as well all go  
home and forget the whole thing.

What we are supposed to be looking at in this thread is whether there  
exists some devious and roundabout route whereby some legacy server might  
nevertheless be given a header containing <unicode<ascii>>, and such  
routes will certainly not involve the straightforward sending of normal  
messages through the network, even when some agents en-route do not  
advertise UTF8SMTP, nor will they involve bounce messages that might try  
to find their way home via a different route than the one by which they  
originally came. We looked carefully at all those cases while we were  
designing the new protocols.

What we DO need to worry about is devious and roundabout routes involving  
ancillary bits of software such as mailing list expanders, mailto URIs,  
address books, ad-hoc scripts, and whatever else you might think of, and  
which our protocols have no control over, and which UTT-8-aware people  
might unknowlingly try to use, and which might cause some legacy server to  
get sight of one of those <unicode<ascii>> headers. And the ONLY way you  
can prevent that with 100% certainty is by not allowing those headers ever  
to exist at all, not even in UTFF-8-aware systems.

So the only matter at issue is whether the risk of introducing such  
headers is worthwhile (for what is, at the moment, an experimental  
protocol, in which you are allowed to try out such experiments), or  
whether the risk is too great even to try it at all. And that is why we  
are exploring other possibilities, such as reinterpreting the meaning of  
multiple addresses in existing headers, or bringing back some variant of  
the group sysntax (which has seemingly disappeared from the later RFCs,  
but which might still have a better chance of passing through legacy  
servers unscathed.

The whole point of an experimental protocol is that, if the experiment  
fails, then you are still allowed to change it (e.g. by removing the  
broken feature) before you offer it up to the standard-track process. And  
the only reason we are discussing this now is because it is reported that  
a Chinese standard is about to be written based on our experimental  
protocol. So really the FIRST question to ask is how would the Chinese be  
effected if we decided to change out experimental protocol before it goes  
to IETF standards track? And if the answer to that is that it would cause  
a great mess, then indeed we should be reviewing this feature NOW.

But if not, then I would prefer to leave it alone and let the experiment  
proceed.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Aug 2009 14:06:17 -0400

--On Thursday, August 20, 2009 21:50 +0000 Shawn Steele
<[email-masked]> wrote:

> * Email uses UTF-8 instead of ASCII.  That shouldn't be too
> hard and should pretty much work everywhere. 
> * Legacy software
> doesn't use UTF-8, so some sort of handshaking is required to
> say "I know UTF-8." 

Ok so far.

> * Also I still need to email older systems that need 
> an ASCII address. So my UTF-8 mailbox needs an
> ASCII alias.

You need to recognize that some others may not have that
requirement.  I don't see that changing your analysis in any
important way.  Especially if one pursues the "alias" model, if
my native language and script are Lower Slobbovian and I really
don't care whether I can communicate with non-Lower Slobbovians
or them with me or not, I don't need an ASCII alias.  I might
reasonably even believe that not having one would offer some
anti-spam protection.

>   * If my mail system handled multiple
> addresses/aliases for ASCII, then when it's updated to UTF-8
> all the same mechanisms should still work.
>   * A little
> different because all mail accounts would pretty much have
> aliases.

Or not.  See above.  I assume there would be aliases where
people think they want or need aliases and not otherwise.  

> * Something interesting
> should happen when the "I know UTF-8" handshaking fails.

If your model is aliasing, then, if I send a message to an alias
(or any other address --as the sender, I can't tell the
difference) that does not exist or is otherwise not deliverable,
the message gets rejected.  That is how we've done things for
twenty or thirty years and turns out to be exactly what the
effect of "no forward downgrading" would be as far as the user
is concerned.
 
> The "only" changes needed to support that model are 1) Use
> UTF-8 instead of ASCII, and 2) send/recognize the UTF-8
> handshaking.

Except that isn't the model.  In addition to non-ASCII
addresses, EAI permits a number of header fields to be
represented directly in UTF-8 rather than encoded.   The email
traditions to which you appeal involve delivering messages to
the relevant final delivery server if the envelope information
is plausible and deliverable.  If we stick with that model,
consider the following case:

I've got a message all of whose envelope addresses (forward and
backward pointing) are in ASCII.  There are header fields in
UTF-8, so what you describe as "I need UTF-8" has to get a
positive response for the message to be forwarded.  If the
answer is "no", there is still a plausible downgrade action
--one that is much more similar to the 8BITMIME situation than
the address cases are-- but one either need to do that
particular downgrading/recoding or the message must be rejected.

> Probably no change is really necessary to handle ASCII aliases
> for UTF-8 mailboxes, sendmail, Exchange, etc already handle
> multiple aliases.  I could see improvements in tools, but I
> don't think it's required.

Agreed.
 
> When handshaking fails.
> 
> The most interesting part of the problem is when "I know
> UTF-8" handshaking fails.  IMO the sender can then just use
> the ASCII addresses, convert to ASCII, and get replies in
> ASCII.  That's probably trivial when the client/server of the
> sender are well integrated.  It's probably a bit harder if
> there's some sort of gateway involved.  I would prefer that in
> "UTF-8" aware mode, simple ASCII information was included in
> case it had to be converted to ASCII-only mode.  Specifically
> the <Unicode<ASCII>> type syntax might be useful.

Please remember that the issue isn't either a pure client/server
model or even client-> SubmissionServer-> SomeSortOfGateway->
DeliveryServer.   We have to either have something that works
well in the presence of relays or we have to start consistently
saying "configuration error and you get what you deserve".

But, if you derive all of this from an alias argument, then

	<Unicode <ASCII>>
(noting that we don't even have a proposal for the "no space"
case) becomes exactly equivalent to
   <MyAddress <Address-to-try-if-that-doesn't-work>>
The latter has been a failure in every mail system that has
tried it, causes all of the security problems that have been
discussed earlier, etc.  It isn't impossible, but the price is
fairly high.

> I don't see any big problems/disagreement with the part before
> handshaking fails.  The only real difference between
> no-downgrade and minimal downgrade is whether an ASCII form of
> the address is necessary, and that could be handled by
> <Unicode<ASCII>>, so I'm not sure there's anything blocking
> the standards that don't handle handshaking.

I don't know what you mean by "could be handled".  We know
addresses leak.  We know that "<Unicode<ASCII>>" is going to
cause parsing problems in some cases, although probably not as
many parsing problems as "<Unicode <ASCII>>".   And one cannot
avoid the handshaking because of the UTF-8-headers cases, so I
don't know what the "standards that don't handle handshaking"
are.

> I would prefer that the standards allow the concept of
> aliasing rather than requiring an independent architecture
> where a server depends on an independent pair of addresses.
> That seems like an architectural decision for an
> implementation of how to handle aliases rather than a
> standards requirement.

As I said above, I see little practical difference between your
alias model and the "no forward downgrading" one, at least until
one violates the traditional alias model by incorporating "if
this address doesn't work, try that one" into the protocol
exchange rather than, e.g., in a message footer or signature
block.

best,
   john

---

From: "YAO Jiankang" <[email-masked]>
Date: Sat, 22 Aug 2009 16:30:27 +0800

----- Original Message ----- 
From: "Charles Lindsey" <[email-masked]>
To: "IMA" <[email-masked]>
Sent: Friday, August 21, 2009 10:38 PM
Subject: Re: [EAI] My view of a "downgraded" address.

.....snipped....
>ails, then you are still allowed to change it (e.g. by removing the  
>to IETF standards track?

Chinese standard will follow the new IETF standard or RFC. If necessay, we will update it.
I am involved in the Chinse standard process.
so do not care about this.

Yao Jiankang
CNNIC  

>And if the answer to that is that it would cause  

-- 
>PGP: 2C15F1A9 Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: Shawn Steele <[email-masked]>
Date: Mon, 24 Aug 2009 22:09:59 +0000

>requirement.

Of course if you don't need ascii.

>people think they want or need aliases and not otherwise.

I'm assuming the case where someone did care.

>represented directly in UTF-8 rather than encoded.

As I said, use UTF-8.  I meant for the entire message.

>particular downgrading/recoding or the message must be rejected.

I also meant to convert the entire message from UTF-8 to ASCII/MIME encodings.  I don't think we disagree much if at all.

> saying "configuration error and you get what you deserve".

In practice most people can't send through relays that don't trust them.  In other words you're account is tied to the relay in some fashion.  (eg: its a relay @ work, etc, gateway between intranet & extranet, etc.)  I'm sort of assuming that if someone wants to adopt EAI in their enterprise, they'd update the relays as well.  Of course I coudl be wrong...

>discussed earlier, etc.  It isn't impossible, but the price is

I'm assuming the failures were from non-EAI machines.  If I convert <Unicode<ASCII>> to just ASCII on downgrade, then it doesn't look any different to the existing machines.  I'm assuming the EAI machines need updated anyway to support UTF-8, and that the extra address parsing wouldn't be too bad.

> exchange rather than, e.g., in a message footer or signature

I hadn't considered a signature block as a solution.  I'm not sure if that's practical. (Does it work if you fill out a sweepstakes card at the mall?)

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Mon, 24 Aug 2009 18:59:47 -0400

(eliding things about which we seem to be in complete agreement)

--On Monday, August 24, 2009 22:09 +0000 Shawn Steele
<[email-masked]> wrote:

>...

"Care" is different from perceived need, but we are probably in
agreement.
 
>...

> In practice most people can't send through relays that don't
> trust them.  In other words you're account is tied to the
> relay in some fashion.  (eg: its a relay @ work, etc, gateway
> between intranet & extranet, etc.)  I'm sort of assuming that
> if someone wants to adopt EAI in their enterprise, they'd
> update the relays as well.  Of course I coudl be wrong...

Failure for the sender properly match functionality to that of
the submission server or of the receiver (party advertising
non-ASCII addresses) to arrange things so that all MX servers
have EAI capability both fall into the category that some of us
have been describing as "configuration errors".
 
> I'm assuming the failures were from non-EAI machines.  If I
> convert <Unicode<ASCII>> to just ASCII on downgrade, then it
> doesn't look any different to the existing machines.  I'm
> assuming the EAI machines need updated anyway to support
> UTF-8, and that the extra address parsing wouldn't be too bad.

The extra address parsing is part of what I'm concerned about,
especially in the context of "addresses leak".  Of course,
"discard the Unicode address" is rather different from the
current downgrade spec, which attempts to preserve it to avoid
loss of information.

>> As I said above, I see little practical difference between
>> your alias model and the "no forward downgrading" one, at
>> least until one violates the traditional alias model by
>> incorporating "if this address doesn't work, try that one"
>> into the protocol exchange rather than, e.g., in a message
>> footer or signature block.
> 
> I hadn't considered a signature block as a solution.  I'm not
> sure if that's practical. (Does it work if you fill out a
> sweepstakes card at the mall?)

Signature blocks are our traditional mechanism for expressing
"if this address doesn't work for you, try that one".   As far
as the sweepstakes card at the mall is concerned, if my systems
and their MX backups are properly configured for EAI, then they
are not the issue (I think we agree about that).  So the
question is whether the systems the mall uses to send mail are
able to handle an EAI address.  If I have any doubts about that,
I'd better give them an ASCII address.  Again, this is nothing
new -- most of the storefronts (online or traditional) who can't
handle "[email-masked]" probably won't be able to cope
with "<Unicode<ASCII>>" either.  And you probably know how long
the former has been valid under the standards.

    john

---

From: Shawn Steele <[email-masked]>
Date: Mon, 24 Aug 2009 23:22:44 +0000

You're swaying me in the direction of letting the EAI address "just be an alias" if a user has both UTF-8 and ASCII addresses, and letting the user figure out the difference, like with an address block.

It seems like there's going to be a user education curve.  Certainly users don't even bother with signature boxes, and regardless of any fallback, users probably would only write a single address on the sweepstakes entry form.

So then maybe the most interesting scenario is when I send email to you, and you are limited to ASCII.  It should be reasonably trivial for my server to switch to my ASCII reply address, however an EAI aware relay in the middle could confuse things if there wasn't a way to forward the ASCII reply address to that point.

-Shawn

________________________________________
From: John C Klensin [[email-masked]]
Sent: Monday, August 24, 2009 3:59 PM
To: Shawn Steele; [email-masked]
Subject: RE: [EAI] My view of a "downgraded" address.

    john

---

From: John C Klensin <[email-masked]>
Date: Mon, 24 Aug 2009 19:56:03 -0400

--On Monday, August 24, 2009 23:22 +0000 Shawn Steele
<[email-masked]> wrote:

> You're swaying me in the direction of letting the EAI address
> "just be an alias" if a user has both UTF-8 and ASCII
> addresses, and letting the user figure out the difference,
> like with an address block.
> 
> It seems like there's going to be a user education curve.
> Certainly users don't even bother with signature boxes, and
> regardless of any fallback, users probably would only write a
> single address on the sweepstakes entry form.

I think any use of EAI facilities is going to require some user
education except when they are used strictly within a community
all of whose systems have been upgraded.  For the
(self-identified) communities for which this is really
important, I expect that transition to happen rather quickly
(YMMD on that subject).  It will be slower in the more marginal
ones (e.g., those that use Latin script but need a few decorated
characters -- but those are the ones for which user-driven,
rather than protocol-driven, ASCII work-arounds are going to be
easiest. 

> So then maybe the most interesting scenario is when I send
> email to you, and you are limited to ASCII.  It should be
> reasonably trivial for my server to switch to my ASCII reply
> address, however an EAI aware relay in the middle could
> confuse things if there wasn't a way to forward the ASCII
> reply address to that point.

I'd appreciate it is you would work carefully through the use
cases, because I may have missed something.  But...

* My free advice to your mail client is that, if the recipient
address is ASCII, you use an ASCII reply address unless you are
fairly sure that UTF-8 is supported.

* Assuming that we keep the SMTP option (which I see no way to
avoid) but remove in-transit downgrading, and that relay is
conformant, then it is going to have to reject or bounce the
message if it encounters a non-EAI-aware next-hop.  That isn't
confusion - it is a predictable example of the "can't forward
this because the next-hop server doesn't support the extension
and I don't know what else to do" mechanism that we've lived
with since the SMTP extension model started being deployed.

    john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 25 Aug 2009 00:27:58 +0000

> fairly sure that UTF-8 is supported.

That's reasonable, but then they remain in the dark about my EAI address :(.  Also I'd like the mail headers/body to stay in UTF-8 itself, but that is easily converted to ASCII anywhere along the line, so that's not an issue.

>message if it encounters a non-EAI-aware next-hop.

Unless the address is ASCII and it can convert the body/headers.  I really want UTF-8 in the mail :)

-Shawn

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 25 Aug 2009 10:52:05 +0100

> so do not care about this.

Then, in that case, I would suggest that we let the Experiment proceed as
it stands, see what the Chinese and others do with it, observe what works
and doesn't, and then review what changes need to be made to put it on the
Standards Track in maybe one or two years time.

And note that "observing" means watching out for all the devious and
roundabout ways that we never thought of which cause difficulties.

---

From: Shawn Steele <[email-masked]>
Date: Tue, 25 Aug 2009 17:47:43 +0000

So if we're headed in the direction of "use Unicode addresses when it works" and "use ASCII when it doesn't work," then I had a random thought, kinda devil's advocaty.

If my mailer/server/yours used UTF-8 and yours did, then Unicode works.  If not, then it doesn't work.  If we're relying on users to know when the Unicode address (eg UTF-8) worked and when they didn't, couldn't we just say "use utf-8 with the existing standards", and not worry about the UTF8SMTP extensions?  It seems that they'd fail anyway and you'd end up in the same place....

Just a stray thought, I'm not really advocating that, and it wouldn't help body-only UTF-8.

-Shawn

Sent: Monday, August 24, 2009 4:56 PM

    john

---

From: "YAO Jiankang" <[email-masked]>
Date: Thu, 27 Aug 2009 11:29:10 +0800

----- Original Message ----- 
From: "Shawn Steele" <[email-masked]>
To: "John C Klensin" <[email-masked]>; <[email-masked]>
Sent: Wednesday, August 26, 2009 1:47 AM
Subject: Re: [EAI] My view of a "downgraded" address.

> If my mailer/server/yours used UTF-8 and yours did, then Unicode works.  If not, then it doesn't work.  If we're relying on users to know when the Unicode >address (eg UTF-8) worked and when they didn't, couldn't we just say "use utf-8 with the existing standards", and not worry about the UTF8SMTP extensions?

the problem is that the user can not decide whether the email system can support EAI or not via the email address format.

even in the situation where EAI sends the message to EAI, there is a still possibility that the relay server does not support EAI.
if you send the utf-8 header message directly to non-EAI server, that serve may collapse.
UTF8SMTP extensions protection for that.

Yao Jiankang
CNNIC

>  It seems that they'd fail anyway and you'd end up in the same place....
>

---

From: Shawn Steele <[email-masked]>
Date: Fri, 28 Aug 2009 05:38:16 +0000

> the problem is that the user can not decide whether the email system can
> support EAI or not via the email address format.

If it bounces as undeliverable they know :)

> even in the situation where EAI sends the message to EAI, there is a still possibility
> that the relay server does not support EAI.

It should bounce then, and the user can resend with an ASCII address.

I'm not sure it's a great solution, but, frankly, if someone gives me a Unicode email address, I'll likely try that.  It's unlikely I'm going to enter 2 addresses just to send a mail to a new contact.  Similarly if I give out addresses, I'll likely give out the one I think will work in the context, I'm unlikely to hand out 2 addresses for every transaction.  That's particularly true for automated systems that reject any fancy form like <Unicode<ASCII>>

So I think it's likely users will continue to use ASCII when they don't know what works, and to use Unicode when they're pretty confident it'll work.  I think that presenting both addresses is rare.  If that's true, then full downgrade/upgrade isn't adding a lot of value.

- Shawn

---

From: "YAO Jiankang" <[email-masked]>
Date: Fri, 28 Aug 2009 16:54:51 +0800

----- Original Message ----- 
From: "Shawn Steele" <[email-masked]>
To: "YAO Jiankang" <[email-masked]>; "John C Klensin" <[email-masked]>; <[email-masked]>
Sent: Friday, August 28, 2009 1:38 PM
Subject: RE: [EAI] My view of a "downgraded" address.

>If it bounces as undeliverable they know :)

yes.

but some servers may  drop it directly as the spam.

even in the situation they bounce as undeliverable, some may take more time such as 5 minutes or half an hour to response.
Another problem is  whether the sender can tolerate 5 minutes or more to resend the message with the ASCII address.

Yao Jiankang