Subject: [idn] Why we cannot go directly to UTF-8

From: Keith Moore <[email-masked]>
Date: Thu, 24 May 2001 00:09:37 -0400

It's really quite simple.

If we use UTF-8 names, each component in a signal path that handles
an IDN has to be upgraded before the application will work with IDNs.

For email, this means every UA, MTA, message store, mail filter,
mailing list, etc that uses the addresses in the header or
envelope of a message.  For the web, this means every web browser,
proxy, cache, and origin server that makes use of domain names
in the request or response (header or payload).  For both cases,
it means that every DNS query library, resolver, cache, and server
involved in the lookups supports UTF-8 also (unless you believe
that the existing ones will already support UTF-8 without protocol
extensions, which is far from a given).  There's little incentive
to upgrade because so many other components need to be upgraded
before you can get reliable operation.

If we use ASCII compatible names, each component in a signal path
that handles a domain name can upgrade independently, and things
will keep working - they just won't display the name as nicely if 
they're not updated.    And only the components that interface with 
users need to be upgraded before the users see a benefit.

It's easier to get real IDN support into the various components 
using ASCII compatible names because fewer components need to be
upgraded.  And the incentives for adoption are greater with ASCII 
names because the benefit of upgrading will be seen sooner.

Users won't care about whether the applications protocols represent 
IDNs in ACE or UTF-8.  But they will care about whether their
applications support IDNs.  ACE lets them do so far more quickly.

Keith

---

From: Martin Duerst <[email-masked]>
Date: Fri, 25 May 2001 10:50:22 +0900

>an IDN has to be upgraded before the application will work with IDNs.

Each component *may* have to be upgraded. Some may not need an
upgrade. For many, 8-bit transparency will do just fine.

>in the request or response (header or payload).

'every' is too general. It's just those in the relevant paths.
For the average web case, it's just the browser and the server.
Proxies can be changed if necessary.

>For both cases,

I very much doubt your last sentence. Companies that are interested
in being found with an idn will obviously upgrade their DNS and
web servers (if necessary). Users interested in using idns will
upgrade their browsers (if necessary). If people will be using
idns as frequently as we all think, the missing bits will be
filled in quite quickly.

>users need to be upgraded before the users see a benefit.

This is an end-to-end problem, and the end is the user, not some
system. A system that gives (people like you) the impression that
it works, but displays ASCII garbage, is a total failure.

*Nobody* who has a choice between let's say 'toshiba.co.jp' and
some garbage like 'xyz--ttnhpur83g4prhoaunh3.co.jp' (rather
than something like TOUSIBA.co.jp, imagine upper-case as kanji)
will ever want to use the later. It's completely useless for
humans.

>names because the benefit of upgrading will be seen sooner.

No, what will happen is that the problems will be seen sooner,
and people will complain. Using UTF-8 will help to make sure
that clients are upgraded before idns are used, and that's
crucial for making sure that things work reasonably well
(for the humans who are the purpose of this exercise, not
only for some machines) from the start.

>applications support IDNs.  ACE lets them do so far more quickly.

Users care very much whether they see things in their script or they
see some ACE garbage. A half-way solution is not a solution, and ACE
will expose a half-way solution for a long time. Very ugly.

Of course, most probably to you ASCII garbage looks better than
Arabic, Chinese, Japanese, Korean, Hebrew, or other scripts.
Given that you have spent decades reading English, and maybe
only seconds or minutes looking at the other scripts, this is
nothing to be blamed for, and it's something that is in many
ways very difficult to become aware off, to get atop of it
and be able to judge from an independent viewpoint. I know
these kinds of things very well, because still after learning
and reading Japanese for close to 20 years, if I see an
announcement in both Japanese and English, my eyes move
to the English first even if it's printed very small,
and the Japanese is much bigger. Reading is an activity
that we spend more time on learning and doing than anything
else, and so it's no wonder that it happens all automatically
without us being aware of what happens.

So again, I would like to remind you: If users see ACE garbage,
this is a protocol failure in an end-to-end system, not something
we can gloss over lightly.

Regards,   Martin.

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 24 May 2001 20:38:07 -0700

> 'every' is too general. It's just those in the relevant paths.

I agree that "every" is too general, but so is "just those in the relevant
paths".

The number of systems which will break with UTF-8 is greater than the
number of systems which break when they encounter an underscore in the
domain name. Moreover, the "path" includes protocols such as BOOTP/DHCP
clients and servers, X clients and servers, and a bunch of other stuff.

Moreover, some of these systems are firmare-based boxes that are no longer
maintained. Think of all the terminal servers and printer servers that
were acquired, orphaned or died.

If we were to go to UTF-8, we would pretty much have to cleanly break with
history in order to ensure interoperability.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Keith Moore <[email-masked]>
Date: Thu, 24 May 2001 23:52:34 -0400

> upgrade. For many, 8-bit transparency will do just fine.

the reason I say that each component has to be upgraded is that for
every component at least one of the following is likely to be true:

- it needs to be upgraded to keep from breaking the next component 
  by sending it utf-8 when it's not ready to deal with it

- it needs to be upgraded to do nameprep before making a query
  so that the existing DNS servers will work in the face of various
  representations of the IDN

- it needs to be upgraded to be able to correctly compare utf-8
  names in the protocol  with names in a configuration file

- it needs to be upgraded to treat UTF-8 as UTF-8 and not as some
  other locale-specific character set.

> 'every' is too general. It's just those in the relevant paths.

that's why I said "every component in the signal path".
of course, an IDN solution that works only for a few signal
paths isn't terribly useful.

> Proxies can be changed if necessary.

the browser and every server that the user wants to contact which
uses IDNs, and possibly the web proxies. and the dns servers and 
their proxies also.

> upgrade their browsers (if necessary). 

or the user will upgrade his browser and not see any improvement,
because few companies support it, and because his service provider
or LAN has a proxy that prevents it from working.  the service 
might even be worse than before because the user will see IDNs
in print and other media and try to type them in, and they will fail
because one compoent or another hasn't been upgraded. and the user 
will say "why bother"?  

and the  web is easier to ugprade than most applications.

the user will upgrade his email client and try to use IDNs in his
address and find that lots of folks cannot reply to them, and that
sometimes his mail doesn't get delivered or gets damaged in transit.
and his recipients postmasters will rigthly blame the sender of mail 
containing IDNs because that mail violates the protocol specifications. 
and once again the user will say "why bother?"  

> If people will be using

OTOH, if people find that IDNs don't work reliably, and if they cause 
things to break, IDNs will get a bad reputation and users will
learn to avoid them.

> it works, but displays ASCII garbage, is a total failure.

if you believe that then there is no possible successful solution.

an IDN will display as garbage on a system that doesn't support IDNs.
it doesn't matter whether it's in ACE or UTF-8, it still looks like 
garbage.

but ACE garbage doesn't mess up my display formatting; UTF-8 garbage
will.  ACE garbage doesn't cause parse errors in existing software;
UTF-8 garbage will.  ACE garbage can be looked up in DNS even if
the software can't display it; UTF-8 garbage is less likely to work.

> will ever want to use the later. It's completely useless for

probably true.  I suspect that at first people will use IDNs 
sparingly at first, regardless of what representation is chosen. 
 
> and people will complain. 

perhaps.  but the logical extension of this argument is that we
should avoid deploying IDNs as long as possible since that way
we can put off seeing the problems for as long as possible.

> Using UTF-8 will help to make sure
> that clients are upgraded before idns are used

how do you figure that?  the clients have to be upgraded in either
case before the user sees a benefit.  and there's no incentive
to upgrade the clients to support IDNs before they can be used.
I suppose the vendors might just update their clients to support
IDNs in the next release cycle,  and eventually most computers
will be running a relase that supports IDNs.  But that could
take a decade or more.

meanwhile, we don't want use of IDNs to cause things to break.
nor do we want to get a bunch of software widely deployed
after many years' time and only then realise that it has
serious bugs or incompatibilities.  (some vendors might try to
do this, believing it would give them a competitive advantage)

> will expose a half-way solution for a long time. Very ugly.

so will UTF-8, and it's also very ugly.  

humans will see the ugliness either way.  you are correct that
IDNs are being done for the benefit of humans. but the difference 
between the ACE approach and the UTF-8 approach is that the ACE 
approach allows existing software to continue to work reliably
(even if it looks ugly), while the UTF-8 approach casues existing 
software to fail randomly (while also being ugly).

> Arabic, Chinese, Japanese, Korean, Hebrew, or other scripts.

it probably does.  but for lots of people, the UTF-8 
version won't display in Arabic, Chinese, Japanese, etc. either -
it will display as unprintable garbage rather than the ACE
version which will display as printable garbage.

> we can gloss over lightly.

fine, I'll accept that definition for the purpose of argument.
but that means that if users see UTF-8 as garbage, that's an even 
worse protocol failure - since the UTF-8 is less likely than ASCII
to be displayable at all.  (which follows because the ASCII 
characters used in ACE are a strict subset of UTF-8)

you have now defined terms in such a way that success is impossible.

you've also defined success criteria in such a way that having
the names look right is more important than having them work 
reliably when used.  which seems pretty strange to me.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 25 May 2001 03:59:48 -0000

Eric A. Hall writes:
> The number of systems which will break with UTF-8

Details, please.

My UNIX list at the bottom of http://cr.yp.to/proto/idn.html includes
gethostbyname() (some versions of which have already been fixed), mutt,
pine, and sendmail. Are you claiming that something else needs to be
upgraded? If so, what? We want verifiable facts.

---Dan

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 24 May 2001 21:36:45 -0700

> upgraded? If so, what? We want verifiable facts.

If I had kept a list of every device and application that broke when a
customer of mine thought "_" was legal, I'd gladly give it to you.

Those systems enforce LDH, and will reject non-LDH sets.

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 24 May 2001 21:49:14 -0700

> Those systems enforce LDH, and will reject non-LDH sets.

If we're into lists, here's one I'd like to see: all of the systems which
provide user input of domain names, and which are also aware of the
variable-octets underlying UTF-8 domain names. These systems need to know
that the domain name will have a maximum label length of 63 octets and
prevent additional input when the UTF-8 octet limit is exceeded, which
will be dramatically different than the displayed character count.

The list of systems which do support this will certainly be shorter than
the list of systems which don't, so that is an action item for you I would
guess. :/

Now look, seriously, there is *no* *freakin* *way* that we can go to
just-do-UTF-8. At the very least resolvers need to know about the
difference between wide() and normal names, systems need $HOSTNAME calls
that support UTF-8 encodings, etc. This is a massive undertaking.

I am clearly on record in the archives as supporting a clean break from
the past, but it must be a clean break. Fudgery just will not work.

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Fri, 25 May 2001 09:00:52 -0400

| Now look, seriously, there is *no* *freakin* *way* that we can go to
| just-do-UTF-8. At the very least resolvers need to know about the
| difference between wide() and normal names, systems need $HOSTNAME calls
| that support UTF-8 encodings, etc. This is a massive undertaking.

  The 'just send UTF-8' approach also ignores the normalization issues
addressed by nameprep, unless you think, as Bill Semich has suggested,
that a protocol based on nameprep only somehow makes sense and is 
actually deployable.

  -bws

---

From: "D. J. Bernstein" <[email-masked]>
Date: 25 May 2001 14:29:12 -0000

Eric A. Hall writes:
> This is a massive undertaking.

Every IDN proposal involves massive costs---in particular, upgrading
sendmail. The point of my IDN web page is to collect information on
those costs:

   http://cr.yp.to/proto/idn.html

ACE with slow nameprep requires redeployment of apache, fetchmail,
gethostbyname, lynx, mutt, named, netscape, pine, publicfile, qmail,
sendmail, sshd, tcpclient, tcprules, tinydns, w3m, and many similar
pieces of code.

UTF-8 with fast nameprep requires redeployment of gethostbyname (except
that some versions have been fixed already), mutt, pine, and sendmail.
If anything else needs to be upgraded, please let me know.

> These systems need to know
> that the domain name will have a maximum label length of 63 octets

Wrong again. The 63-byte limit is isolated inside one library routine
that converts textual domain names to DNS packet format. That routine
already works perfectly with UTF-8. I don't know where you got the idea
that applications have to stop users from typing more than 63 bytes.

> customer of mine thought "_" was legal, I'd gladly give it to you.

In other words, you don't have a single shred of verifiable evidence.   
Your memories are so vague that you can't even tell us what programs you
were using---but you're confident that those programs would have trouble
with UTF-8, and you're making major decisions on this basis.

---Dan

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 25 May 2001 10:10:17 -0700

"D. J. Bernstein" wrote:

> Your memories are so vague that you can't even tell us what programs
> you were using---but you're confident that those programs would have
> trouble with UTF-8, and you're making major decisions on this basis.

Feel free to rename your domain to _test_.cr.yp.to. and report back with
your success. Since I already know what will happen, I won't bother.

---

From: "James Seng/Personal" <[email-masked]>
Date: Sat, 26 May 2001 11:30:39 +0800

If you need a UTF-8 example, please try this: 实验.cr.yp.to

-James Seng

---

From: "James Seng/Personal" <[email-masked]>
Date: Sat, 26 May 2001 11:41:08 +0800

Sometimes, people think I spoken broken English with my bad grammers and
spelling which I only caught after I send out. The fact is that it
happens with me in Chinese too!

s/实验/试验/

-James Seng