Subject: [idn] A UTF-8 test for mail clients

From: "D. J. Bernstein" <"djb-pibounce-�"@cr.yp.to>
Date: Wed, 08 Aug 2001 10:35:50 -0700

1. This is a MIME message with charset=utf-8. Your mail client should
display the two-byte UTF-8 sequence \317\200 in the message body as the
Greek letter pi. Here's a pi between brackets: [π]

2. This message also includes a pi in the From line, right before the @:

   From: "D. J. Bernstein" <[email-masked]>

Of course, there's no character-set declaration for that address, but
your client should display it as UTF-8 anyway. The most common mistake
will be to treat it as 8859-1: a double-dotted I and a block.

3. Try replying to this message. Your mail client and mail server should
be able to handle the [email-masked] address. You will receive
a bounce from muncher.math.uic.edu showing the exact contents of your
mail as received by my mail server.

Note that there's no IDN here. The non-ASCII characters in the address
are in the box part, not the domain part. But that difference doesn't
matter for typical mail clients. This test avoids triggering DNS client
bugs, such as the BIND gethostbyname() bug.

The address will be corrupted if your message passes through Sendmail:
the second pi byte, \200, will be stripped. You will still receive a
bounce from muncher in this case.

Finally, send a message to [email-masked] telling me what happened.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Wed, 08 Aug 2001 13:48:29 -0400

> Greek letter pi. 

no it shouldn't, as nothing in the MIME standard requires support for UTF-8.
 
>    From: "D. J. Bernstein" <djb-pibounce-Ï<80>@cr.yp.to>
> 
> your client should display it as UTF-8 anyway. 

No it shouldn't, since addresses are defined to be ASCII in both RFC 822
and RFC 2822.

> be able to handle the djb-pibounce-Ï<80>@cr.yp.to address. 

My mail client treats the CF and 80 as invalid characters when composing
the reply.  This is reasonable behavior given that they are illegal.

Keith

---

From: "A. Vine" <[email-masked]>
Date: Wed, 08 Aug 2001 10:51:36 -0700

"D. J. Bernstein" wrote:
> Greek letter pi. Here's a pi between brackets: [Ï€]
> 
>    From: "D. J. Bernstein" <djb-pibounce-Ï€@cr.yp.to>
> 

Or Windows CP1252 including euro, which causes a euro symbol to be displayed
next to the capital I with diaeresis.

> be able to handle the djb-pibounce-Ï€@cr.yp.to address. You will receive

---

From: "D. J. Bernstein" <[email-masked]>
Date: 9 Aug 2001 20:26:35 -0000

Keith Moore writes:
> addresses are defined to be ASCII in both RFC 822 and RFC 2822

Wake up, Keith! We're changing this definition. The whole point of IDN
is to allow _non-ASCII_ characters, such as the Greek pi, in addresses.

Which character encoding are we going to use instead of ASCII? Maybe
UTF-8. Maybe ACE-FUBAR-8675309. Either way, the interpretation of
address bytes as ASCII will be obsolete.

> the reply.

Thanks for the data point. Are you still using MH 6.8.4? Have you tried
the new MH?

---Dan

---

From: Keith Moore <[email-masked]>
Date: Fri, 10 Aug 2001 12:31:41 -0400

> Wake up, Keith! We're changing this definition. 

get back on your meds, Dan.  IDN is *not* changing the format of addresses 
used in any protocol, with the possible exception of DNS.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 10 Aug 2001 17:19:35 -0000

> used in any protocol, with the possible exception of DNS.

On the contrary. IDN is replacing ASCII with another character encoding
in every Internet protocol that uses domain names.

The ASCII standard specifies, for example, that the byte string bq--blah
(i.e., \142\161\055\055\142\154\141\150) represents characters bq--blah.
Under a typical ACE IDN proposal, however, that byte string represents a
different sequence of characters, and there's another byte string that
represents the characters bq--blah.

Sure, ASCII and ACE both use 7-bit bytes; this doesn't mean that ASCII
and ACE are the same! A character encoding is not a set of byte strings;
it is a _mapping_ between character strings and byte strings.

---Dan

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 10 Aug 2001 22:43:19 +0000

"D. J. Bernstein" <[email-masked]> wrote:

> The ASCII standard specifies, for example, that the byte string
> bq--blah (i.e., \142\161\055\055\142\154\141\150) represents
> characters bq--blah.  Under a typical ACE IDN proposal, however, that
> byte string represents a different sequence of characters, and there's
> another byte string that represents the characters bq--blah.

Strings beginning with the ACE prefix are never ACE-encoded.

Here's the way I think of it now:  In the current system, FOO and
foo and Foo and fOo etc. are all equivalent.  We don't say that FOO
represents foo, or that foo represents FOO.  They are distinct ASCII
strings, but they are equivalent labels.

The new system merely expands the set of strings and expands the notion
of equivalence.  Now <lowercase-pi> and <uppercase-pi> and bq--blah
and BQ--BLAH etc. are all equivalent.  Sometimes we like to think that
bq--blah represents <lowercase-pi>, and sometimes we like to think
that <lowercase-pi> represents bq--blah, but actually it's a symmetric
equivalence relationship.  They are distinct Unicode strings, but they
are equivalent labels, just like FOO and foo above.

Having each equivalence class include at least one element containing
only ASCII characters is convenient for interoperating with legacy
software/protocols/interfaces.

AMC

---

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Aug 2001 00:14:31 -0000

Adam M. Costello writes:
> Now <lowercase-pi> and <uppercase-pi> and bq--blah and BQ--BLAH etc.
> are all equivalent.

That's ridiculous. The user wants to see lowercase pi. He might tolerate
uppercase pi, but he will kick and scream if he sees bq--blah.

This is typical of your arguments: you show us the road to failure, you
define failure as being equivalent to success, and you claim success.
Sorry, Adam, but that isn't what the users want to see.

---Dan

---

From: "Marco d'Itri" <[email-masked]>
Date: Sat, 11 Aug 2001 11:37:39 +0200

>uppercase pi, but he will kick and scream if he sees bq--blah.
Did you do some kind of usability survey among users or are you just
speculating?

-- 
ciao,
Marco

---

From: Magnus Bodin <[email-masked]>
Date: Sat, 11 Aug 2001 12:40:52 +0200

> Did you do some kind of usability survey among users or are you just

Thats quite obvious anyway, but if ACE is gonna happen, then I really hope
that the users won't see bq-blah in the URL-line. 

That would be a sure way to kill IDN before it goes popular.
 
/magnus

-- 
This message secured by Rot-52 encryption technology.
Unauthorized decryption prohibited under the DMCA.
Violators will be prosecuted to the full extent of the law.

---

From: "James Seng/Personal" <[email-masked]>
Date: Sun, 12 Aug 2001 01:37:00 +0800

> uppercase pi, but he will kick and scream if he sees bq--blah.

Agree. ACE leaks horribly. When ACE leaks, users will see bq--gibberish
and they would be scratching their head.

But when UTF-8 leaks, which is common, users will see some 8-bit
gibberish and they would be scratching their head too.

Bottomline: Both ACE and UTF-8 leaks. No one can make sure it wont
without fixing the UI/Application.

-James Seng

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 13 Aug 2001 15:56:38 +0200 (CEST)

> in every Internet protocol that uses domain names.

Dan,

I don't know whether the above use of "IDN" refers to this working group
or some larger problem statement. I'm assuming you mean the WG.

If updating RFC 822, 2822 for smtp as well as all other application protocols
was in scope for the IDN working group, they would have been explicitly
listed in the WG charter.

They are not.

Having said that, when thinking about different solutions to the 
Internationalized Domain Name problem it is critical to understand the
impact on the application protocols (and other "users" of domain names)
in particular as it relates to the transition.

   Erik