Subject: RE: [idn] UNIX moving to UTF-8

From: Karlsson Kent - keka <[email-masked]>
Date: Sat, 27 Jan 2001 12:57:25 +0100

Keith,

(regarding your reply to Ken:)
I would suggest you read Ken's e-mail again, slowly.

It's not that "ACE" didn't start out at ISO that is of concern.
It's the fact that the "ACE"s have an extremenly narrow field
if applicability (IDNs) that is of concern.  UTF-{8,16,32} are
general purpose encodings, that do and will find support
in many, many places.  "ACE"s can only be used for IDNs,
and will never find widespread support for anything else.

(regarding your reply to Mark:)
There may be other reasons for not using MS Word for editing config
files; but MS Word can indeed edit plain text files, in UTF-8 as well
as many other encodings.

However, to be able to edit config files with "ACE"s in them, there
is no, and will never be any general purpose editor that supports that.
You will need an editing tool (or auxilliary tool) SPECIFIC for IDN
containing config file editing to be able to edit such a config file if
you want to be able to read what is really encoded there in the 
"ACE"d IDNs.

		/kent k

----------------------------------------------------------------------------
-----

> -----Original Message-----
> From: Keith Moore [mailto:[email-masked]]
> Sent: Friday, January 26, 2001 7:01 AM
> To: Mark Davis
> Cc: James Seng/Personal; D. J. Bernstein; [email-masked]
> Subject: Re: [idn] UNIX moving to UTF-8 
> 
> 
> > A huge number of people in non-English speaking countries 
> are using Unicode,
> > *and don't know it*. Anyone using Windows NT/2000, or 
> Microsoft Office is,
> > as well as many other products. Many websites use Unicode 
> internally, and
> > just convert to the user's codepage, etc.
> 
> sure.  but just because someone is using microsoft office that happens
> to use some encoding of Unicode doesn't mean they can use it to edit 
> a "plain text" config file that expects IDNs to be in UTF-8 format.

----------------------------------------------------------------------------
-----
> -----Original Message-----
> From: Keith Moore [mailto:[email-masked]]
> Sent: Friday, January 26, 2001 9:23 PM
> To: Kenneth Whistler
> Cc: [email-masked]; [email-masked]; [email-masked]; 
> [email-masked]
> Subject: Re: [idn] UNIX moving to UTF-8 
> 
> 
> > > > People are clearly moving to Unicode. Exactly which UTF 
> they choose (8, 16,
> > > > 32) is not as important, since they all can be 
> converted to each other very
> > > > efficiently and without loss.
> > > 
> > > And since an ACE is just another encoding of Unicode, you 
> can add ACE to
> > > that set.
> > 
> > This was malarkey the first time it was stated, and it is 
> still malarkey.
> > 
> > UTF-8, UTF-16, and UTF-32 are official, standard encoding 
> forms of the
> > Unicode character encoding.
> > 
> > ACE is a transformation encoding syntax (TES) -- *not* an 
> encoding form.
> 
> I didn't use the precise term "encoding form" I used the 
> imprecise term
> "encoding".  
> 
> ACE most certainly is an encoding of Unicode.  If adopted by 
> IETF, ACE 
> will also be an "official" "standard" - though not an ISO standard.  
> If memory serves, neither Unicode nor UTF-8 started out in 
> ISO either, 
> so the fact that ACE didn't start there shouldn't be cause 
> for concern.
> 
> > I'm not opposed to the ACE proposals for solving the short-term IDN
> > issue, but...
> > 
> > UTF-8 is a general-purpose encoding form for open interchange of
> > Unicode data is all kinds of contexts. It is also used as a 
> general-purpose
> > processing form in many systems.
> 
> UTF-8 certainly has wider applicability than ACE.  However ACE
> is not designed to meet the constraints of *a* particular protocol,
> it is designed to meet the constraints of *several* protocols that
> use ASCII DNS names as protocol elements.
> 
> Keith
>

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 27 Jan 2001 17:10:23 +0100

>you want to be able to read what is really encoded there in the

You don't only need the ACE encoding but also the nameprep algorithm. 
So, specific tools _will_ be needed, just as special applications 
which use IDN.

And, these special tools will be needed regardless of ACE or UTF-8. I 
think that that is the point Keith (and myself) tries to make.

With the introduction of IDN, we can no longer "just" edit the config 
files in whatever applications.

So, existing applications can not be used. We need new tools.

    paf

---

From: "D. J. Bernstein" <[email-masked]>
Date: 27 Jan 2001 19:57:56 -0000

> files in whatever applications.

The worst case---for places like Japan, where we've heard that you can't
even type a dot without special effort---is that the operating system
needs ONE domain-name-typing tool as part of the keyboard interface.

I'm explicitly assuming that we _won't_ allow names needing nameprep in
configuration files. Bad names disappear as soon as they leave the
keyboard. That's fast nameprep. The changes are isolated inside the
keyboard interface.

You're implicitly assuming that we _will_ allow names needing nameprep
in configuration files. That's slow nameprep. It's horribly expensive.
Hundreds of programs will need to learn about nameprep.

---Dan

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 27 Jan 2001 21:22:48 +0100

>Hundreds of programs will need to learn about nameprep.

No, you misunderstand me. We talk about exactly the same thing. We 
need something between the keyboard and the config file which do the 
nameprep, so never non-nameprepped things end up in configuration 
files.

That's the whole idea with IDNA. Do nameprep as "close" to the 
keyboard as possible (and the reverse when displaying).

   paf

---

From: Marc Blanchet <[email-masked]>
Date: Sat, 27 Jan 2001 18:05:56 -0500

At/À 21:22 2001-01-27 +0100, Patrik Fältström you wrote/vous écriviez:
>No, you misunderstand me. We talk about exactly the same thing. We need 
>something between the keyboard and the config file which do the nameprep, 
>so never non-nameprepped things end up in configuration files.

well, we hope so, but reality will probably be not that perfect.

Marc.

>That's the whole idea with IDNA. Do nameprep as "close" to the keyboard as 
>possible (and the reverse when displaying).
>
>   paf
>

Marc Blanchet
Viagénie inc.
tel: 418-656-9254
http://www.viagenie.qc.ca

----------------------------------------------------------
Normos (http://www.normos.org): Internet standards portal:
IETF RFC, drafts, IANA, W3C, ATMForum, ISO, ... all in one place.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 28 Jan 2001 04:12:37 -0000

Patrik writes:
> We need something between the keyboard and the config file which do the
> nameprep, so never non-nameprepped things end up in configuration files.

I'm glad you agree. So why do you keep claiming that (for example) BIND
and djbdns and Sendmail and qmail need to learn about nameprep?

---Dan

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 28 Jan 2001 10:25:32 +0100

>and djbdns and Sendmail and qmail need to learn about nameprep?

I don't.

I have been talking about ACE v.s. UTF-8 (which is completely 
independent on where nameprep is done), and claim that IF we do 
nameprep and ACE in whatever tools that exists between the user and 
the software which handle the application protocols, then no changes 
are needed in the software you mention.

See the idna spec which myself and Paul has written.

The main idea in that spec is that you do nameprep and encoding as 
"high" up in the protocol stack as possible, as the nameprepped 
encoded domainname can be easilly reused in all protocols without any 
changes what so ever in qmail, sendmail, bind, djbdns etc.

It's even the case that the existing webpages which some people have 
created (or separate software) which converts between whatever 
charset one use natively and encoded nameprep is used, so the user 
know the ace encoded nameprepped string, then that string can be used 
in all existing software. Ugly, but doable.

I said one day on this list:

At 09.24 +0100 01-01-06, Patrik Fältström wrote:
>I.e. I could register one day maybe:
>
>   fältström.com
>
>...but could also on my buissness card have (if the ACE ends up being RACE):
>
>   bq--abtoi3duon2hf5tn .com
>
>Yes, it looks ugly, it explicitly is leakage which we all hate, but, 
>it  makes it possible for people with non-IDN software to access the 
>domain. Of course, I can in parallell have a domain which is 
>"faltstrom.com" or whatever, but I might think that my customers if 
>I help them that way never update their software. I.e. I _WANT_ 
>people to be able to use "fältström.com" probably, so I would only 
>help them to some degree, and that might be to give them the ace 
>encoding in paralel with the real name.

    paf

---

From: Keith Moore <[email-masked]>
Date: Sun, 28 Jan 2001 23:49:44 -0500

> if applicability (IDNs) that is of concern.  

agreed that it's a narrow field of applicability, but it's an
important one for what we're trying to solve.  UTF-8 might be
quite flexible, but it doesn't let us transmit domain names
in the ASCII-only fields that most existing protocols and
applications were written to expect.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 Jan 2001 09:22:12 -0000

There are five possible approaches under discussion:

   (1) UTF-8 with fast nameprep. Names in configuration files are good:
       KC-normalized, no confusing characters, no uppercase characters.
       (Except uppercase ASCII, for compatibility.)

       The keyboard interface has to help users type good names. No
       changes are needed in qmail, djbdns, and many other programs. A
       few programs with 8-bit bugs, notably Sendmail, need to be fixed.

       Patrik and Keith refuse to acknowledge this approach.

   (2) UTF-8 with slow nameprep. Names in configuration files may be
       bad.

       This lets us leave the keyboard interface alone, at the expense
       of putting nameprep into a huge number of programs. Yuck.

       When Patrik said that qmail and djbdns and so on ``have to be
       changed because of the nameprep stage,'' he was talking about #2.

   (3) ACE with fast nameprep and slow encoding. Names in configuration
       files are good UTF-8 names.

       The keyboard interface has to help users type good names.
       Furthermore, a huge number of programs need to learn about ACE.
       Yuck.

       When Patrik recently agreed that qmail and djbdns and so on
       _don't_ have to learn about nameprep, he was talking about #3 and
       #4. He still refuses to acknowledge #1.

   (4) ACE with fast nameprep and fast encoding. Names in configuration
       files are good ACE names.

       The keyboard interface has to help users type good names and
       convert them to ACE. Furthermore, users need to use an extra
       filter to read the ACE names every time they access configuration
       files with tools such as vi, grep, etc. Yuck.

       The big difference between #1 and #4 is that the world is moving
       to UTF-8, not ACE, as the default extension of ASCII. UTF-8
       already works with vi and grep on my system; ACE never will.

   (5) ACE with slow nameprep. Names in configuration files are UTF-8,
       and may be bad.

       This lets us leave the keyboard interface alone. But a huge
       number of programs need to learn about nameprep and ACE. Yuck.

I hope this list is helpful. I wish Patrik and Keith would stop denying
the existence of #1.

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 Jan 2001 09:56:15 -0000

Keith says that ``most'' applications expect ``ASCII-only fields.''

Details, please. Exactly which software has to be upgraded for UTF-8?

My current list includes the gethostbyname() library, mutt, pine, and
sendmail. Note that all of these would also need changes for ACE with
slow nameprep, and for ACE with fast nameprep and slow encoding.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Tue, 30 Jan 2001 12:33:02 -0500

> Details, please. Exactly which software has to be upgraded for UTF-8?

Essentially everything that accepts input containing DNS names.

Or do you claim that you've exhaustively analyzed every program
that handles domain names on every platform and that none of them 
(other than pine, mutt, sendmail, and than those which dynamically 
load gethostby* and for which the problems can be remedied by 
replacing that library?) exhibit any kind of failure when presented 
with UTF-8 domain names - including munging those names or a failure 
to recognize two different representations of the same name as equivalent?

I thought not.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 30 Jan 2001 12:25:17 -0500

this is indeed a helpful list, as it helps identify at least part of 
the difference in assumptions.

my assumption is that for most languages and most people's platforms 
the "keyboard interface" for entry of DNS names does not exist - 
i.e. it's no different from the "keyboard interface" used for ordinary 
text.  and nameprep is still needed for many languages for which
the input environments don't need a special keyboard interface to
allow users to type in DNS names.

thus, we don't have the luxury of isolating nameprep with the keyboard
interface.  most of the programs that accept entry of DNS names will
do so using the normal text interface, which doesn't do any kind of 
nameprep. and in many cases (email messages, zone files, etc.) IDNs will 
be mixed with normal text.  so they have to separate IDNs from other 
text and perform nameprep (and perhaps ACE) on those IDNs.

thus I don't think the approach you listed as #1 is feasible.

Keith