Subject: [EAI] Unicode / ASCII fallback mappings?

From: Shawn Steele <[email-masked]>
Date: Fri, 26 Jun 2009 23:31:57 +0000

I realize that a UTF-8 email address and the ASCII fallback for that address are left to the server to figure out, but I was wondering if anyone had any preconceptions?

I can think of 3 basic ways to solve generating ASCII addresses: 1) Some sort of encoding mechanism that is algorithmic and likely makes an unreadable ASCII address, 2) aliasing, where aliasing happens automagically, like perhaps Cyrillic to ASCII transliteration, and 3) aliasing where the ASCII alias doesn't necessarily correlate with the native-script form.

My preconception is that the last is the most useful, unless the native script has well-defined transiliteration to Latin rules.

Furthermore I'd expect that when I created a new account on something like live mail/gmail/yahoo mail, that it'd be nice to get prompted for both the Unicode and ASCII aliases, possibly with a generate ASCII alias button for users that aren't as familier with that form.

Lastly, I'd assume that "enabling" EAI on such a system, such as with a Chinese ISP, would mean that when I log in with an existing ASCII address, I could be prompted with a "Get your chinese alias now!" button, so that I wouldn't have to make a new account or lose service/migrate messages, etc. from my old account.

It occurred to me that what I think is obvious may not be what everyone/anyone else is thinking.  Does anyone care to share any other options or whether they agree?  I know that earlier Mark had suggested a punycode form, but I'd assumed that most end-users would prefer a more friendly alias.  One that would make sense on a double-sided business card for example.

-Shawn

---

From: Yangwoo Ko <[email-masked]>
Date: Sat, 27 Jun 2009 10:09:04 +0900

On Sat, Jun 27, 2009 at 8:31 AM, Shawn Steele<[email-masked]> wrote:
> I realize that a UTF-8 email address and the ASCII fallback for that address
> are left to the server to figure out, but I was wondering if anyone had any
> preconceptions?
>
> I can think of 3 basic ways to solve generating ASCII addresses: 1) Some
> sort of encoding mechanism that is algorithmic and likely makes an
> unreadable ASCII address, 2) aliasing, where aliasing happens automagically,
> like perhaps Cyrillic to ASCII transliteration, and 3) aliasing where the
> ASCII alias doesn't necessarily correlate with the native-script form.

First and second options are unacceptable in many ways including
technically and culturally.

> My preconception is that the last is the most useful, unless the native
> script has well-defined transiliteration to Latin rules.
>
> Furthermore I'd expect that when I created a new account on something like
> live mail/gmail/yahoo mail, that it'd be nice to get prompted for both the
> Unicode and ASCII aliases, possibly with a generate ASCII alias button for
> users that aren't as familier with that form.

Well it seems to depend on cases. For instance, if such a mail service
provider only provided (or were only known for its) ASCII domain name,
it would be far less useful to support registration of unicode local part
resulting in [email-masked].

>
> Lastly, I'd assume that "enabling" EAI on such a system, such as with a
> Chinese ISP, would mean that when I log in with an existing ASCII address, I
> could be prompted with a "Get your chinese alias now!" button, so that I
> wouldn't have to make a new account or lose service/migrate messages,
> etc. from my old account.

As a marketing term, "alias" can be used. But, technically, we should not
call them as alias because we are not providing a technical binding between
them except delivering the fact that user of those mail addresses bound them
together.

>
> It occurred to me that what I think is obvious may not be what
> everyone/anyone else is thinking.  Does anyone care to share any other
> options or whether they agree?  I know that earlier Mark had suggested a
> punycode form, but I'd assumed that most end-users would prefer a more
> friendly alias.  One that would make sense on a double-sided business card
> for example.

I agree with the double sided business card analogy in a sense that it requires
human & manual intervention to bind/match two different names and that we
usually don't need to look at both sides.

>
> -Shawn
>
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>
>

-- 
a human / twitter as yangwooko / msn as [email-masked] / nate
as [email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 29 Jun 2009 15:27:59 +0900

Hello Shawn,

I guess you got it about right. Definitely alternative 3) is more 
desirable. What might happen, though, especially for some well-known 
webmail providers, is that the ASCII space is already pretty well used, 
and new users who need to get a fallback ASCII address with their 
non-ASCII mail address might have some difficulties finding something 
they think is appropriate. Not sure much can be done about this.

Regards,    Martin.

On 2009/06/27 8:31, Shawn Steele wrote:
> ------------------------------------------------------------------------
>

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Mon, 29 Jun 2009 13:12:49 -0400

--On Friday, June 26, 2009 23:31 +0000 Shawn Steele
<[email-masked]> wrote:

>...
> everyone/anyone else is thinking.  Does anyone care to share
> any other options or whether they agree?  I know that earlier
> Mark had suggested a punycode form, but I'd assumed that most
> end-users would prefer a more friendly alias.  One that would
> make sense on a double-sided business card for example.

The WG (and some pre-WG effort) discussed this issue in some
depth early on and more or less reached the conclusion that you
have, with very similar reasoning.  The "friendly alias" isn't
the only problem with a punycode (or other) encoded form, but we
have already had DNS (and other) experience that users who
wanted to use native (non-ASCII) characters don't think one has
done them any favors by migrating them from a lousy ASCII-based
transliteration or mnemonic to an encoded from that isn't even
memorable.

    john

---

From: Shawn Steele <[email-masked]>
Date: Mon, 29 Jun 2009 18:34:02 +0000

Thanks for the replies, I guess I may have missed earlier discussion.  I'd realized that I was working on what I thought was "an obvious assumption" and didn't recall much discussion about the issue.

Other than not requiring a mapping I don't see mention of this in the RFCs.  Did I miss it, or do we want to make the recommendation in one of the RFCs that users may want ASCII address to be in a friendly form (rather than encoded?)

I don't think this impacts the standard directly, but rather it may impact the thinking about the overall architecture of a system that works with such addresses.  Eg: if you design a single-sign-on system expecting Unicode addresses with encoded ASCII you may make different decisions that if you anticipated friendly ASCII forms.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Mon, 29 Jun 2009 14:55:12 -0400

--On Monday, June 29, 2009 18:34 +0000 Shawn Steele
<[email-masked]> wrote:

> Thanks for the replies, I guess I may have missed earlier
> discussion.  I'd realized that I was working on what I thought
> was "an obvious assumption" and didn't recall much discussion
> about the issue.
> 
> Other than not requiring a mapping I don't see mention of this
> in the RFCs.  Did I miss it, or do we want to make the
> recommendation in one of the RFCs that users may want ASCII
> address to be in a friendly form (rather than encoded?)

We could certainly drop that into Framework, should we revise
it.  It really isn't a protocol issue because the decision is
ultimately up to the user and whatever server he or she is
working with.  I'm not sure whether it works as an analogy, but
I note that there is still no requirement for case-insensitive
local-parts in RFC 5321.  What it says is that the strings are
case sensitive in the protocol but that one will surprise a lot
of people, and probably lose some mail, unless at least the
obvious case variations work.  (FWIW, I've seen server
implementations that I consider both conforming and sensible
that equate "Shawn", "SHAWN", and "shawn" but that would reject
"ShAwN" as not matching the others.)  

ASCII alternate addresses are different from that because there
are no generally-applicable algorithms (even if some systems
might choose to use them, contrary to the recommendations we
might make), but it is still ultimately up to the server.

> I don't think this impacts the standard directly, but rather
> it may impact the thinking about the overall architecture of a
> system that works with such addresses.  Eg: if you design a
> single-sign-on system expecting Unicode addresses with encoded
> ASCII you may make different decisions that if you anticipated
> friendly ASCII forms.

For many reasons, I would hope that no one would do that at all
and that, except in the cases where the alternative address is
routed to separate systems for translation if needed, alternate
addresses would be supported by mailbox aliases rather than
separate mailboxes with potentially separate sign-on.   If I
recall, Framework does discuss that issue to some small extent
but it has been a long time since I looked at it. 

Think about it this way:  the alternate address is expected to
be just that -- an alternate address, not an alternate identity.
It is going to be meaningful only in the email environment and,
one hopes, only during transition.  You want the user to
establish and use one key identity, not a selection of
alternatives.  If your identity system is Unicode compatible,
then that key identity --especially for users of non-Latin
keyboards-- will be Unicode and whatever makes sense to the user
and system.   If the user doesn't have a keyboard that
accommodates the relevant character set, an escaping system is
going to be painful and one more complex than either U+nnnn or
\nnnn notation is going to be error-prone and _very_ confusing
to all concerned.

    john

---

From: Shawn Steele <[email-masked]>
Date: Mon, 29 Jun 2009 19:10:49 +0000

> be just that -- an alternate address, not an alternate identity.

That's what I meant for SSO.  SSO would have to "know" that Unicode@xxx and Ascii@xxx weren't different identities, but rather the same identity.  (Assuming that some SSO clients wouldn't be able to support Unicode immediately).

I could further imagine a SSO UI that displayed the ASCII form (YouUnicode@xxx aka YouAscii@xxx), but maybe they'd do something different if they thought YouAscii was going to be gibberish?

I agree it's not really a protocol thing, and maybe more of a "guidelines" or "best practices" concept that isn't maybe even something the working group need spell out...

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Mon, 29 Jun 2009 21:30:19 -0400

--On Monday, June 29, 2009 19:10 +0000 Shawn Steele
<[email-masked]> wrote:

> That's what I meant for SSO.  SSO would have to "know" that
> Unicode@xxx and Ascii@xxx weren't different identities, but
> rather the same identity.  (Assuming that some SSO clients
> wouldn't be able to support Unicode immediately).

But this is email, so "alternate address" would usually imply
"alias or forward into  the same mailbox".  There are, of
course, different models, which is what makes it hard to give
any sort of definitive advice.

As far as SSO clients that can't support Unicode immediately,
maybe I have a different picture in mind than you do, but what I
keep being told is that, in modern operating systems, everything
is sufficiently clean that there is no problem passing a UTF-8
string off to whatever authenticates the user.  IMO, permitting
non-ASCII addresses and/or mailbox owner names before the
authenticator is ready to deal with those would be pretty close
to nuts.  

Put differently, I'd want to design my SSO systems, etc., on the
assumption that alternate addresses didn't exist and that they
were, at best, a short-term transitional convenience.  If my
receiving mail systems and the user access to them weren't fully
Unicode-identifier and Unicode-authenticator compatible, support
for alternate addresses would just be an invitation for
increased support costs and/or irritated users.  I'd much rather
listen to "why haven't you implemented this yet" rather than
"well, you implemented it, you did a half-a**ed job, and you
botched it completely and made it unusable".  I presume, so
would you.

So, let's take Windows Live Mail / Hotmail as an example.  I'd
think the first deployment step would be to permit the SSO
system to handle non-ASCII user IDs and related credentials,
even if those credentials accessed only those mailboxes with
ASCII local-parts (I don't know, but I suspect you are there
already).  _Then_ I'd permit mailboxes with non-ASCII local
parts, probably as more of a replacement for whatever ASCII ones
the user had earlier rather than as a complement.  And, if I saw
the demand, and I probably would, I'd offer users of those new
mailboxes ASCII aliases for the mailboxes --aliases that could
be used as alternate addresses in EAI downgrade but, far more
important, could be put on business cards and the like for
senders who had no internationalized address handling capability
at all.  

Note that, if the MUAs and Submission servers are
5321/5322/4409-conformant they can't use either non-ASCII
addresses or Alternate-addresses at all.  Assuming no one is
doing anything really stupid, there are only two case in which
Downgrading is really useful.   One involves an originating
system that is EAI-conformant _and_ a receiving one that is
EAI-conformant enough to support non-ASCII addresses but where
there is a relay in the middle that is not EAI-conformant.  I
think that case is almost always configuration error (having an
EAI-conformant receiving system whose users advertise non-ASCII
addresses configured with MX records pointing to a
non-EAI-conformant machines is not the smartest idea), but it
falls short of "really stupid".  The other case has mostly to do
with backward-pointing addresses, i.e., your sending mail using
a non-ASCII address from your EAI-conformant system to my
perfectly good ASCII address (with no non-ASCII address or
alternate for me).  

For all of the other cases I've been able to imagine the
"downgrading" is going to have to be done by the user at the
sending end or some agent close enough to her that the only
things that go on the wire are ASCII addresses.

> I could further imagine a SSO UI that displayed the ASCII form
> (YouUnicode@xxx aka YouAscii@xxx), but maybe they'd do
> something different if they thought YouAscii was going to be
> gibberish?

Maybe.  But I'm still not sure that it would show the alternate
address any more than you see [email-masked] aka
[email-masked] (or, if you have been there long
enough to have used what I gather is the older convention, "aka
[email-masked]").   One identity, zero or more aliases for
email and perhaps zero or more aliases for other purposes.

> I agree it's not really a protocol thing, and maybe more of a
> "guidelines" or "best practices" concept that isn't maybe even
> something the working group need spell out...

Yep.  And it obviously looks a bit different depending on the
direction from which one looks.  

   john

---

From: SM <[email-masked]>
Date: Mon, 29 Jun 2009 22:51:29 -0700

Hi Shawn,
At 12:10 29-06-2009, Shawn Steele wrote:
>Unicode@xxx and Ascii@xxx weren't different identities, but rather 
>the same identity.  (Assuming that some SSO clients wouldn't be able 
>to support Unicode immediately).

That sounds like identity linking.  In the world of email, it would 
be more of an alias if the delivery goes to the same mailbox.  It may 
be easier for a user to manage that if the fallback behavior is 
intuitive.  Let's consider a user in an "EAI" region.  The user will 
probably have an EAI aware interface for email.  The only time when 
the user needs to deal with the Ascii@example email address is when 
he/she has to communicate the email address to someone using a 
non-EAI service.  If the Unicode@example email address is considered 
as an add-on feature, it may have an adverse effect on EAI in the long term.

>(YouUnicode@xxx aka YouAscii@xxx), but maybe they'd do something 
>different if they thought YouAscii was going to be gibberish?

It may appear as gibberish for a native.

>something the working group need spell out...

It would be good to have a common concept for "selling" EAI.  You 
also have to convince the "legacy" world to support it.

Regards,
-sm

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 30 Jun 2009 12:41:08 +0100

> senders who had no internationalized address handling capability

The sort of scenario I would expect is a 'sign up' window (in Chinese, of
course) that invites you to to fill in some 'User ID' box (in Chinese, and
probably with a little '*' beside to show it was an obligatory field).

Then there would be another (non-obligatory) box for 'ASCII User ID',
perhaps with a little "click here for explanation" (in Chinese) beside it;
and if you clicked that the popup would explain that if it was likely you
would need to exchange email messages with non-Chinese correspondents
using non-Chinese characters, it wold be a Good Idea to fill in some ASCII
ID here; but otherwise to leave it blank.

So a Chinese person who had no expectation of sending such emails, or did
not know what Latin/ASCII chacaters looked like and/or how to make his
keyboard produce them would not bother. But a Chinese person who had a
slight smattering of English, and maybe already had some anglicised
transliteration of his name, might Have a Go (and I guess that there is
quite a significant proportion of Chinese people who would indeed do so).

Of course, both of his IDs would have to be Unique amongst all IDs known
to the Hotmail/Whatever system, but that is a problem we already have, and
which already leads to typically obscure and un-memorable Hotmail/Whatever
addresses :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Shawn Steele <[email-masked]>
Date: Tue, 30 Jun 2009 17:10:38 +0000

My presumption is that the ASCII & Unicode aliases are indeed going to the same mailbox and that ASCII is ONLY used for downgrading when the Unicode fails.  I expect that users do NOT want 2 mail accounts just so they can interact with non-EAI users.  Instead I expect most systems to support two addresses (one IMA Unicode, one downgrade ASCII) for the same mailbox.  I suppose you could view this as an "add-on", however I'd make the EAI address be the primary address.  I'm not sure it's an add-on if the add-on takes over :)

This also impacts address books, etc.

Were I to own something like Live Mail or gMail or an ISP.  (I'm in Windows, I only provide guidance to the Live teams,) then I'd enable an alias.  If I was a Chinese ISP, then in early 2010, after the Nov 2009 Chinese standards come out, I could put up a banner saying "pick your Chinese email address now", which would enable an additional alias for the existing user account.  Then I'd get nearly 100% adoption of EAI names for my service in markets like China.  If I make users get new accounts, adoption rates would be low.

But that's just me :)  My initial question was to try to get a feel for how other developers might approach the problem.

-Shawn

________________________________________
From: SM [[email-masked]]
Sent: Monday, June 29, 2009 10:51 PM
To: Shawn Steele
Cc: [email-masked]
Subject: Re: [EAI] Unicode / ASCII fallback mappings?

Hi Shawn,

Regards,
-sm

---

From: SM <[email-masked]>
Date: Tue, 30 Jun 2009 12:35:28 -0700

Hi Shawn,
At 10:10 30-06-2009, Shawn Steele wrote:
>My presumption is that the ASCII & Unicode aliases are indeed going 
>to the same mailbox and that ASCII is ONLY used for downgrading when 
>the Unicode fails.  I expect that users do NOT want 2 mail accounts 
>just so they can interact with non-EAI users.  Instead I expect most 
>systems to support two addresses (one IMA Unicode, one downgrade 
>ASCII) for the same mailbox.  I suppose you could view this as an 
>"add-on", however I'd make the EAI address be the primary 
>address.  I'm not sure it's an add-on if the add-on takes over :)

If we are thinking in technical terms, we are going to draw up 
constraints based on existing systems.  I don't think that the model 
should even emphasize the downgrade ASCII address as it's only there 
to support legacy systems.  The premise is that the EAI address is 
the primary address and not an add-on.  It may be tempting to graft 
EAI on an existing infrastructure. But then we'll end up with hacks 
for the next 30 years. :-)

>This also impacts address books, etc.

There are several other areas where email addresses are used.  I 
vaguely recall seeing an Internet-Draft or a RFC about the subject.

Should we store the EAI address in the address book or the ASCII 
address if the English user has a system that is EAI aware?  If we 
choose the ASCII address, we are not really internationalizing.  As 
it's not a technical issue, I don't know where such questions can be discussed.

>Were I to own something like Live Mail or gMail or an ISP.  (I'm in 
>Windows, I only provide guidance to the Live teams,) then I'd enable 
>an alias.  If I was a Chinese ISP, then in early 2010, after the Nov 
>2009 Chinese standards come out, I could put up a banner saying 
>"pick your Chinese email address now", which would enable an 
>additional alias for the existing user account.  Then I'd get nearly 
>100% adoption of EAI names for my service in markets like China.  If 
>I make users get new accounts, adoption rates would be low.

If you were running Live Mail (used as an example only) you would 
have some EAI users outside China.  Let's assume that you are in the 
EU as it's one market with over approximately 23 languages.  If you 
offer EAI names, you can cover all these languages in one go.  The 
Chinese ISP is an extreme case as you have a large internal market 
there.  By having a unified system, you create a mindset.  If you 
think in terms of adoption rates, you will end up with complex 
systems with different behaviors.  The glue is how you use the EAI 
specifications to achieve you objectives.

Regards,
-sm

---

From: Shawn Steele <[email-masked]>
Date: Tue, 30 Jun 2009 20:36:59 +0000

> for the next 30 years. :-)

I don't think I explained myself correctly.  I'm not suggesting that EAI be hacked on to the existing system, and think it's doing well for a clean approach.  Regardless there will be a transition period.  All I'm saying is that it's not hard to migrate a current ASCII system to an EAI system.  It's also not that different than a clean EAI system.

Brand new EAI system:
* Needs a Unicode enabled email address
* Needs an alias (alt-address)
* Should use the Unicode email address in EAI contexts.

(This is a lot like how [email-masked] and [email-masked] both get mail to me.  One is used by my mailer, and the alt-address isn't typically advertised, but it could be.)

The new account experience in such a system:
1. Sign up for an account
        a) User is prompted for a primary account name, could be non-ASCII (or ASCII if they don't need Unicode)
        b) If non-ASCII, the user is prompted for an optional alt-address, with the caveat that if they don't pick one it might not work everywhere.
        c) The system stores both addresses.  One is primary, the other's used for alt-address in non-EAI aware contexts.

An EAI system upgraded from non-EAI:
* Already has an address that fits the requirements for alt-address
* Needs a Unicode enabled email alias.
* The new Unicode email address should be "primary" and used in EAI contexts.

This is pretty much the same as the "brand-new" EAI system.

The upgrading account experience in such a system:
1. Logon to existing account
        a) User is prompted for the Unicode EAI account name (if desired).
        b) Already has an ASCII address, so prompt not necessary for that.
        c) The system stores the new Unicode EAI account name.  This will be used as the primary account name now and in EAI contexts.  The "old name" will be used for alt-addresses.  If necessary it could even flip-flop the records to look like the "new eai" system, but I doubt it matters.

So I don't get how this contaminates the EAI standards?  The architecture is effectively identical in the brand-new EAI system and the upgraded system.  It's not a hack, but a migration path.  The nice thing is that in the upgraded system a user gets to keep their old mailbox, and doesn't have to remember to update their address with all their friends and banks.  New contacts can get the new EAI name, but there's no pain for the user.

I'd be really surprised if most mail providers weren't considering such an upgrade strategy.

-Shawn

---

From: SM <[email-masked]>
Date: Tue, 30 Jun 2009 15:45:11 -0700

Hi Shawn,
At 13:36 30-06-2009, Shawn Steele wrote:
>I don't think I explained myself correctly.  I'm not suggesting that 
>EAI be hacked on to

I think that it is I who did not explain myself correctly.  I do not 
consider what you suggested as hacking on EAI.

>  the existing system, and think it's doing well for a clean 
> approach.  Regardless there will be a transition period.  All I'm 
> saying is that it's not hard to migrate a current ASCII system to 
> an EAI system.  It's also not that different than a clean EAI system.

Agreed.

>(This is a lot like how [email-masked] and 
>[email-masked] both get mail to me.  One is used by my 
>mailer, and the alt-address isn't typically advertised, but it could be.)

If I recall correctly, the initial discussion was about generating 
the alt-address.  I don't know yet whether it should be automated or 
not for this case.  The users I have come across tend to have 
multiple mailboxes as they do not know about aliases.  I have a 
preference for aliases as it's more convenient.  The above should work.

>         a) User is prompted for a primary account name, could be 
> non-ASCII (or ASCII if they don't need Unicode)
>         b) If non-ASCII, the user is prompted for an optional 
> alt-address, with the caveat that if they don't pick one it might 
> not work everywhere.

If the user is not familiar with email, we have to figure out how to 
explain alt-address to them.  Note that current aliases are familiar 
to us as we know the characters.

>         c) The system stores both addresses.  One is primary, the 
> other's used for alt-address in non-EAI aware contexts.
>

Agreed.

>         c) The system stores the new Unicode EAI account 
> name.  This will be used as the primary account name now and in EAI 
> contexts.  The "old name" will be used for alt-addresses.  If 
> necessary it could even flip-flop the records to look like the "new 
> eai" system, but I doubt it matters.

Agreed.

>So I don't get how this contaminates the EAI standards?  The 
>architecture is effectively

It does not contaminate the EAI specifications.  My argument was to 
avoid using the new EAI address as an alternative address.

Regards,
-sm

---

From: Shawn Steele <[email-masked]>
Date: Tue, 30 Jun 2009 23:21:22 +0000

Apparently we were saying the same thing :)

> to us as we know the characters.

I assume ISPs will find some nice way of explaining your "Chinese" address and your "Pinyin" address that may still be needed for older systems.  (Presumably the terminology will change in different markets.)

> avoid using the new EAI address as an alternative address.

Sorry I wasn't clear, yes that'd be bad.

-Shawn