Subject: [idn] Let's go forward with IDNA and UTF-8

From: Dan <[email-masked]>
Date: Sat, 26 May 2001 12:09:31 +0200 (MEST)

From all the discussion on the list I can see two major directions
for non-ASCII names in DNS: use ACE (in application) and use UTF-8.

The people saying ACE/IDNA say it is quickest and least disruptive
to non-IDN aware software. Those wanting UTF-8 thinks this format
is best for internationalisation and fear that starting with IDNA
will result in ACE forever.

I have for a long time wanted UTF-8 but realise that we probably
must use ACE for backward compatiblity.
Before I go on with what I think we should do, I will give some comments
on UTF-8.

Today we have many character sets in use. In protocols we also have many
and we have ACE like encoding, the best example is quoted-printable
in MIME. I have used these encodings and character sets as a user,
a system administrator and as a software developer. It is a terrible
mess! As a user I quite often get to see junk because of things
like quoted-printable in e-mail and as a software programmer I
need very complex software to decode all different encodings used and
to handle all the character sets. To decode an e-mail is difficult,
in it are character sets, encodings and normal text mixed forcing me
to complex parsing. If everything had been in one character set the
world had been much simpler. I would be much more willing to support
languages different from my own. If normalised UCS had been used
everywhere it would be wonderful.
While I know that my system that have everything encoded in ISO 8859-1
cannot change overnight to UCS, if would work much better if all
interoperability protocols used one encoding of all character data.
While UTF-8 is far from perfect, it is the best choice available.
I can today see no other format than UCS normalised using Unicode form C
and then encoded using UTF-8 to be the best to use for interoperability.
That is my reason for wanting UTF-8/UCS/normalisation form C.

Back to business.

I can see we have three things to decide:
1) Internationalsation of the DNS protocol and handling
2) Selecting ACE
3) How to match names (nameprep).

1) Internationalsation of the DNS protocol and handling
Here people want IDNA for quick employment.
ACE for backward compatibility.
UCS/UTF-8 as standard encoding instead of one more encode in ASCII format.

Could we not then do this?
   The DNS protocol/handling selected is:
   - ACE for backward compatibility
   - IDNA for quick employment
   - UCS/UTF-8 as standard encoding combined with EDNS for longer labels.
Of what I can see this can be done. Combining the IDNA draft with the
UDNS draft will give most of it. And both IDNA and UCS/UTF-8 can be
used at the same time.

Would not this satisfy most people?

2) Selecting ACE.
This need to be done quickly. Especially as Verisign plans to use RACE
even though it may not be the best. We cannot accept that one company
selects the ACE to use. The ACE selected must be done after comparing
all ACE formats. We should give the main criteria for selecting it.
I suggest that the ACE working group inform the rest of us what they
are thinking and what they have done so far.

3) How to match names (nameprep).
We also need to define how names are to be matched.
Today we have one draft for that (nameprep) but that one is
written to fit with IDNA and may have to be changed to fit general
matching rules.
The most important matter to take up here, which have been pointed
to by other (like Dan Bernstein), is: shall two characters
that look identical to typical users match as equal?
They do not in the current version of nameprep.

Could we not go forward with something like this?
Or do those favoring IDNA not think UTF-8 is at all acceptable?
Or do those favoring UTF-8 not accept ACE to be used for backward
compatibility?

If we do a compromise and allow for both preferred formats,
IDN can be working fairely soon. Otherwise I suspect IDN handling
in DNS (and other protocols) will be a mess for a long time.

   Dan

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 26 May 2001 14:36:09 +0200

> Or do those favoring UTF-8 not accept ACE to be used for backward

Whether UTF-8 can be used in protocols or not have very much to do with the
ability for a client and server to be able to signal the use of this
character set instead of what is in use today (explicitly or implicitly
specified).

This depends on each and every protocol if this is possible or not, and if
the signalling fails (i.e. client announces UTF-8 and server says "foo?")
what then should the client do?

We have in the SMTP world some experience with Quoted Printable as you say
Dan, where the ESMTP extension for 8bitmime is used, and as most (I claim)
servers can handle 8bit message bodies today AND that signalling works we
don't see as much Quoted-Printable now as we did some 5 years ago. And IF
the quoted printable reach the client, most clients know how to decode it.

Yes, I am one of the persons which think we definitly need an ACE encoding
for the reasons above. We can not add negotiations in all protocols, and we
can not change the middleboxes (application layer firwalls and proxies)
which many times are upgraded the least amount of times, and server
software which the enduser doesn't have control over.

So, an encoding which is non-destructable which makes it clear that all
information which the sender send comes to the recipient regardless of
protocol is a good thing.

        paf

---

From: "D. J. Bernstein" <[email-masked]>
Date: 26 May 2001 17:16:23 -0000

> will result in ACE forever.

I have a much more serious concern: the short-term costs of deploying
ACE appear to be much larger than the total costs of moving directly to
UTF-8. See http://cr.yp.to/proto/idn.html for details.

---Dan

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 27 May 2001 09:48:44 +0200

> UTF-8. See http://cr.yp.to/proto/idn.html for details.

I see two differences between UTF-8 and ACE which was what I used as
arguments when I made up my mind:

(1) Risk of loss of information

ACE approach is NOT destroying any information when data is sent from
sender to receiver in any of the protocols we have today. It is 100%
backward compatible.

Use of UTF-8 in any place in any protocol where we have a domainname today
MIGHT (I don't use any stronger term, but I don't see I have to either)
destroy some data during transport.

(2) How to fix broken domainnames

In an ACE approach, the enduser himself can change his own applications to
get rid of the ACE encoding and instead see the IDN as it was expected.

In an UTF-8 approach the user MIGHT have to change his applications so they
can display UTF-8 characters and (here comes the important part) someone
else than the enduser MUST upgrade the intermediary boxes (HTTP proxies,
SMTP gateways, SMTP MTA's, POP servers, IMAP servers etc) which is not
under his control.

I.e. if we compare with when we launched MIME, many people have told me
that "but the goal is that the user should not see the ACE, and geee what
Quoted Printable was ugly, let's not use the same mechanism again".

You all should know that I do not listen to people saying that which do not
need Quoted Printable to display their own name.

We need something which can be moved from A to B with existing protocols
and services which are deployed, and then the ability for A and B (alone,
not their network managers, or anyone with a root password) to use IDN's.

    Patrik H:son Fältström