Subject: Re: [idn] An experiment with UTF-8 domain names

From: Keith Moore <[email-masked]>
Date: Fri, 05 Jan 2001 09:39:28 -0500

Dan,

you're testing the wrong things.  we already know that the DNS protocol
and common DNS implementations will handle arbitrary 8bit character
strings, and those can be UTF-8 if we want them to.  we also know that 
the vast majority of applications cannot properly input or display
UTF-8 strings, and that some applications will either break or
improperly handle UTF-8 strings when they appear in domain names.

> This sent a message to [email-masked]. I read the message using
> the UTF-8 version of less, and it was displayed comprehensibly, with the
> To line shown as follows:
> 
> To: postmaster@"S".cr.yp.to
> 
> qmail-inject doesn't mind weird characters, such as control characters
> and 8-bit characters, in atoms. It converts the atoms to quoted strings.
> Of course, RFC 822 doesn't allow quoted strings, never mind 8-bit
> characters, in domain names, but these are easy protocol extensions.
...
> What's wrong with handling S this way? The answer seems to be that some
> other programs don't work. What are those programs? What exactly do they
> do wrong? How hard is it to fix them? Why should we believe that the
> other IDN proposals will require less effort?
> 
> Keith Moore writes:
> > all apps will have to be modified (many will require significant
> > modification) if they want to deal meaningfully with IDNs.
> 
> False. As demonstrated above, qmail and djbdns already work with UTF-8
> domain names. They're both widely deployed. Apparently Microsoft also
> has some clients and servers that work with UTF-8 domain names.

no, you've just demonstrated that qmail does not work with UTF-8 domain
names, because it generates an illegal message header.  other mail
parsers will choke on the message header that qmail generates because
it violates RFC822 syntax.  other programs may (in some sense) also 
treat the same input "reasonably" but will behave differently than
qmail under such conditions.  from experience, some of them will
generate RFC 2047 encoded-words (different ones in different ways) - 
which is also a violation of the specifications, but a different 
violation of the specifications.  replies to such messages will fail.

> Changing those programs has a cost. What is the benefit?

interoperability.

Brian W. Spolarich writes:
> > Using 8-bit data will break some applications.
> > Using 7-bit data (presumably) will not.
> 
> False. If the user's S.cr.yp.to has to be encoded inside DNS and mail
> messages as ace-blah.cr.yp.to, then qmail will be faced with S.cr.yp.to
> in (e.g.) /var/qmail/control/virtualdomains, and ace-blah.cr.yp.to in
> SMTP. This simply won't work unless the software is changed.

that's a good point.  but until qmail is updated to accept UTF-8 
in virtualdomains and other places, users can at least get IDNs
to work by cutting-and-pasting the ACE form of the name.  presumably
we would want similar support for DNS zone files.

but it's even worse than that, because most UNIX users today are not
using UTF-8 as a local charset.  so they'll quite naturally edit the
virtualdomains file using iso-8859-1, or iso-2022-jp, or whatever is
the default for their environment.  either tools that read IDNs will
have to know how to translate from local chraset to UTF-8, or users
will have to learn to use different commands to edit files containing
IDNs than they use to edit "normal" (for them) files.  (I think the
former is more likely; users are loathe to give up their familiar
editors)

(the problem is slightly different on windows and mac platforms, but
exists nonetheless)

actually the problem is much the same in both cases, because in 
both cases the sysadmin has to type in a form of the domain name
which is not the same as is used natively - unless the local
charset just happens to be UTF-8.

> If, on the other hand, S.cr.yp.to is used as is, then the software will
> work fine.

no, it just breaks a different set of things.  and (at least in this case)
it moves the points at which the failures are noticed away from the systems 
that are controlled by the folks who are setting up the IDN, to remote 
systems that are used by folks who are less likely to be able to identify,
much less fix, the problem.

> James Seng/Personal writes:
> > Patching sendmail might be trival for a good programmer like yourself.
> > How fast do you think you can get everyone to use your patch and would
> > unpatch software fallback safely?
> 
> All the proposals require a sendmail patch. To tell sendmail to accept
> mail for S.foo.dom, the user adds S.foo.dom to a file with his UTF-8
> editor; sendmail mishandles the \210 if it isn't patched.
> 
> The patch required for direct use of UTF-8 is by far the simplest. No,
> deployment isn't free, but the other proposals don't change this fact.

true, deployment isn't free for any of these proposals, nor can it be.
but your scenario doesn't even begin to address the impact on MUAs
which cannot properly parse or generate replies for messages with UTF-8
in the message header.  nor does it consider what happens with different
legacy mail software handle the UTF-8 IDN in different ways.

and of course email is just one application; lots of applications use DNS names.

                                  --

you're a smart guy, so I suspect that you've realized most or all of 
this already.  I'm wondering whether the real difference of opinion 
might be about how much disruption to the installed base is acceptable
in the name of producing an apparently "cleaner" result in the long run.

taking email as an example, what percentage of each of the following would
be acceptable (relative to the number of messages sent), in your opinion:

- catastrophic MTA failures (crashes)
- catastrophic UA failures (crashes)
- unreported delivery failures
- reported delivery failures
- messages delivered but undisplayable due to parse errors etc.
- messages delivered but unreplyable due to lack of UA support
  (would work if a different or upgraded UA were used) 
- messages delivered but unreplyable due to being modified in transit

Keith

p.s.

> P.S. I'm a subscriber to this mailing list. I don't want to receive
> extra copies of messages sent to the list. I've set Mail-Followup-To
> accordingly.

and since Mail-Followup-To is a nonstandard extension, I assume you're
willing to accept the lack of reliability that naturally comes from
use of nonstandard extensions.  :)
 
> P.P.S. You may have noticed an unusual From line on this message. The
> problem is that the software running this mailing list can't deal with
> the concept of sublist subscribers; it forwards my messages to Seng.
> Seng eventually approved two of my messages, editing the Date field and
> removing the Received lines to hide the delay. He has refused to approve
> this one unless I take special actions to fool the list software. So I'm
> using his address in From, and my address in Reply-To.

Hmm.  I use subaddresses also, and at first I had this problem also.
If memory serves it was worked around by adding my address to a
list of non-subscribers who are authorized to post.  Not an ideal
solution to be sure, but the problem of mailing lists that don't
allow postings from non-subscribers isn't likely to get fixed here.

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 5 Jan 2001 07:15:38 -0800

> So if the applications use the correct normalisation of UTF-8 today they
need not be changed.

There are many ways to normalize data. NamePrep is designed to provide a
single, absolutely determinate process for this IDN normalization. The
chances that the average application would, by chance, duplicate the same
normalization is minimal.

Mark
----- Original Message -----
From: "Dan" <[email-masked]>
To: <[email-masked]>; <[email-masked]>
Sent: Friday, January 05, 2001 02:07
Subject: Re: [idn] An experiment with UTF-8 domain names

Patrik Fältström wrote:
>At 07.55 +0000 01-01-05, D. J. Bernstein wrote:
>>Keith Moore writes:
>They have to be changed because of the nameprep stage, which will
>have to be done regardless of whether you use UTF-8 or ACE or
>whatever encoding of Unicode.

That depends on how the applications encode UTF-8 today and how we use
UTF-8 in DNS. All that is needed is that the UTF-8 is normalised to the
form selected for use in DNS. So if the applications use the correct
normalisation of UTF-8 today they need not be changed.
The rest of the "nameprep" handling can be done in the DNS servers.

  Dan

---

From: "D. J. Bernstein c/o James Seng" <[email-masked]>
Date: 5 Jan 2001 18:41:41 -0000

paf writes:
> They have to be changed because of the nameprep stage,

Ah, yes, the stage where af.mi1 is converted to af.mil.

I know this will come as a shock to you, but I've never implemented that
stage. The registries don't accept af.mi1 as an alias for af.mil;
neither do I.

Users who see af.mil on a business card have to type af.mil, not af.mi1,
into the computer. If they don't know how to do that, too bad.

I'm going to continue allowing DNS administrators to set up ``lDNs''
even though I know that some users will have trouble typing them.

Keith Moore writes:
> you're testing the wrong things.

Setting up domain names in DNS, arranging for them to receive mail,
sending mail to them, and reading the mail, are the ``wrong things''?
Do you also think that putting up web pages and reading them are the
``wrong things''?

> we also know that the vast majority of applications cannot properly
> input or display UTF-8 strings, and that some applications will either
> break or improperly handle UTF-8 strings when they appear in domain names.

Several of us keep asking for details of what exactly will break. The
anti-UTF-8 people keep refusing to provide those details. Why? Could it
be because the failures aren't as widespread as they claim?

Yes, sendmail corrupts UTF-8 IDNs. Perhaps other programs have trouble
too. How difficult will it be to fix them? Why should we believe that
painful MIME-style IDNs will be less expensive?

All I'm asking for is an accurate evaluation of the costs and benefits
of each IDN proposal.

> most UNIX users today are not using UTF-8 as a local charset.

XFree86 4.0 uses the UTF-8 version of xterm. It's easy to configure
Emacs and vim and less to treat text as UTF-8. This is going to replace
8859-1 as the default; UNIX users are tired of dealing with deficient
character sets.

Is this upgrade free? No. People will be stuck with 8859-1 text files
for years, and will have to go through extra effort to view them until
they're converted to UTF-8 on disk. But choosing another IDN proposal
isn't going to reduce this cost.

> illegal message header

Irrelevant. The format was extended. Calling a successful experiment
unsuccessful because it wasn't IETF-sanctioned is silly.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Fri, 05 Jan 2001 15:21:24 -0500

> into the computer. If they don't know how to do that, too bad.

that's not the problem.  the problem is that there are multiple, legal,
on-the-wire representations for af.mil which produce the same glyphs,
and different systems will generate different representations when
the same keys are pressed.
 
> sending mail to them, and reading the mail, are the ``wrong things''?

yes.  you're testing the things which were most likely to work, and
ignoring the things which are least likely to work.   there's nothing 
wrong with testing everything, but claims of victory that are based 
on partial success of the tests that are chosen to be most likely 
to work are meaningless...sort of like testing the resiliance of 
O rings in warm temperatures, observing that they fail some of the
time, and then expecting them to work just as well when it's cold.  

> Do you also think that putting up web pages and reading them are the

yes. try putting up web pages with links to URLs containing IDNs
which are encoded in UTF-8 (using various URL prefixes and various
protocols with servers on a variety of platforms) and seeing whether 
those links work in various browsers.  Then try putting URLs 
containing IDNs into text files, mailing them around, and using
cut-and-paste to enter them into a broswer's "get URL" dialog box.
Then try printing URLs containing IDNs on business cards, and typing
them in to browsers dialog boxes.

now devise similar tests for all other applications that use DNS
names.

of course ACE approaches will fail some of these tests also, but
the tests have to be realistic if you want meaningful results.

> be because the failures aren't as widespread as they claim?

even with a small sample size, when all of your trials produce
failures, there's a good chance that it won't work on a large scale.
and especially when the cost of failure is a serious disruption of
critical services, it makes sense to be conservative.

(and your trial produced a failure, you just refuse to recognize it)

> painful MIME-style IDNs will be less expensive?

the real cost of upgrades isn't the effort to fix the code for
current releases, it's:

- identifying the nature of the problems caused and tracing them
  to their source
- getting millions of individuals to upgrade their software
- supporting older systems and old releases of software that are still 
  in use [*]
- dealing with the disruption until everything gets upgraded

[*] upgrades tend to have a domino effect.  to upgrade an application you 
might well need a newer version of the OS, which might in turn require 
more memory, which might justify the cost of entirely new hardware, which 
then requires that all of the other tools on that system be re-installed,
which tends to imply that other apps get updated also... so the barrier
to upgrading even a single component can be significant

a strategy for minimal disruption is: 

- affect as few components as possible  (since the effort required to
  deal with breakage is on a per-component basis rather than a 
  per-line-of-code basis)

- put most of the burden of upgrading on those who benefit most

it appears that ACE schemes fare better at this strategy than UTF-8 schemes.

> All I'm asking for is an accurate evaluation of the costs and benefits

I'm sure the list would appreciate it if you did more experiments
and reported the results, as long as you were looking at realistic
scenarios.  but even with such results about which cases work and
which ones fail,  "costs and benefits" are highly subjective.

bottom line is that each WG participant will have to evaluate the 
costs and benefits on his or her own, and each participant will
have to invest the amount of effort that he or she believes is
sufficient to reach a conclusion.

we have to reach rough consensus on the design decisions, we don't 
have to reach rough consensus on the motivations.  

> they're converted to UTF-8 on disk. 

I think it's the other way around.  people will not give up their
favorite tools en masse in favor of unfamiliar tools that support UTF-8,
especially when their existing files are in other formats.  And changing 
to a new xterm won't automatically make the old tools work (actually
it will probably break some tools that expect each character takes one
octet).

and this process won't be driven by UNIX users anyway.  

> But choosing another IDN proposal isn't going to reduce this cost.

you haven't said anything to support this argument.

Keith

> unsuccessful because it wasn't IETF-sanctioned is silly.

perhaps.  but calling an experiement successful when the proferred
solution (a) didn't get deployed and (b) didn't solve the problem, 
is even sillier.  fortuantely, your ersatz header doesn't actually
break things that ignore it - unlike MUAs that generate UTF-8 in
message headers.

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 5 Jan 2001 20:37:06 +0100

>Ah, yes, the stage where af.mi1 is converted to af.mil.

Not at all. There are other cases which doesn't look as stupid as 
this one which you just suggested.

Several examples have been brought up on this mailing list about a 
year ago, so I will not repeat. Se Unicode specification.

   paf

---

From: [email-masked]
Date: Fri, 5 Jan 2001 19:49:36 +0000 (GMT)

<002901c0772a$5c2eb2f0$6e5b8e18@mark>
<[email-masked]>
<[email-masked]>
<p05100629b67b3b4a592e@[192.168.124.81]>
<[email-masked]>
X-Mailer: VM 6.43 under 20.4 "Emerald" XEmacs  Lucid
Message-ID: <[email-masked]>
From: Martin Oldfield <[email-masked]>
Sender: [email-masked]
Precedence: bulk

>>>>> "D" == D J Bernstein c/o James Seng <[email-masked]> writes:

--> snip <--

    D> Yes, sendmail corrupts UTF-8 IDNs. Perhaps other programs have
    D> trouble too. How difficult will it be to fix them? Why should
    D> we believe that painful MIME-style IDNs will be less expensive?

--> snip <--
Please can I ask a dumb question ?

Suppose Boris and Ivan want to communicate by email.

Boris has a wonderfully efficient ISP who is good at upgrading to the
latest software, and lives in a region where IDNs will make a really
big difference. Accordingly he acquires a swanky new domain and email
address very quickly. By contrast. Ivan's ISP is lazy and lives in a
country where IDNs probably aren't so big a deal. Perhaps Ivan's an
emigre.

Am I right in thinking that UTF-8 style IDNs are used then Boris and
Ivan can't communicate unless Ivan can persuade his ISP to upgrade his
sendmail, but an ACE would only need Ivan to upgrade his mail client ?
In fact if Boris starts the exchange and although it would look odd,
wouldn't Ivan be able to just hit `Reply' on his existing email client ?

Cheers,
-- 
Martin Oldfield,
AdamsNames Ltd.

---

From: Keith Moore <[email-masked]>
Date: Fri, 05 Jan 2001 21:03:02 -0500

> sendmail, but an ACE would only need Ivan to upgrade his mail client ?

probably so.  the chance is good that a UTF-8 return address will
either be mangled or be unusuable for some other reason.

> wouldn't Ivan be able to just hit `Reply' on his existing email client ?

probably so.  though if experience with RFC 2047 is any guide, there is 
some chance that some MTA or gateway will prematurely "decode" the IDN
and and thereby make it unusable.  we could try to write the specifications
clearly enough that this wouldn't happen very often.  but people 
often mis-implement much more subtle things than this.

Keith

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 6 Jan 2001 09:24:32 +0100

>sendmail, but an ACE would only need Ivan to upgrade his mail client ?

Correct.

>wouldn't Ivan be able to just hit `Reply' on his existing email client ?

Even this is true. It is even the case that Boris can give his 
ACE-version of his real email address to people in a form as "if you 
can not enter cyrillic characters in my email address, please enter 
this instead".

I.e. I could register one day maybe:

   fältström.com

...but could also on my buissness card have (if the ACE ends up being RACE):

   bq--abtoi3duon2hf5tn .com

Yes, it looks ugly, it explicitly is leakage which we all hate, but, 
it  makes it possible for people with non-IDN software to access the 
domain. Of course, I can in parallell have a domain which is 
"faltstrom.com" or whatever, but I might think that my customers if I 
help them that way never update their software. I.e. I _WANT_ people 
to be able to use "fältström.com" probably, so I would only help them 
to some degree, and that might be to give them the ace encoding in 
paralel with the real name.

    paf

---

From: "Martin J. Duerst" <[email-masked]>
Date: Sat, 06 Jan 2001 18:42:22 +0900

At 01/01/05 20:37 +0100, Patrik F$BgM(Btstr$B‹N(B wrote:
>Not at all. There are other cases which doesn't look as stupid as this one 
>which you just suggested.
>
>Several examples have been brought up on this mailing list about a year 
>ago, so I will not repeat. Se Unicode specification.

I think we should be realistic on both sides here.
We don't have conversions for ASCII, but we have
prohibited characters. Dan, what would happen if
you try to register e.g. +-+.cr.yp.to ?
Also, we have case mapping, currently done on the
server but for many reasons better moved to the
client for UCS.

Also, for UCS, there are quite a few cases where the degree
of determinism and success can be significantly improved
for the end user by additional foldings. The best example
probably is the full-width/half-width folding for Japanese
and some other contexts.

However, we should also be clear about the following:

- Even with the best name preparation, some ambiguities,
   hopefully not worse than mil vs. mi1, will remain.

- Trying to be too inclusive in folding will lead to
   work on cases that may occur once in a million or a
   billion. A rather typical example is the 'fi' ligature.
   [An additional argument for not folding that into 'f','i'
    is that it would change the behaviour for ASCII-only
    names, which is probably not something we want.]

- Expecting that every single software component that
   will deal with internationalized domain names will
   do name preparation on every processing step is not
   realistic (Patrick, I'm not saying that you have such
   expectations, I just want to make sure others don't).

Regards,   Martin.

---

From: "Martin J. Duerst" <[email-masked]>
Date: Sat, 06 Jan 2001 19:06:46 +0900

>them in to browsers dialog boxes.

Keith, are you assuming that we are doing all this work so
that people can put 'ace--blah' into their URIs, mail them
around, put them in their 'location:' fields, and so on?
It's clear that that would work with ACE, but that's not
the point of our work, is it? If that's an argument for
using ACE instead of UTF-8, then that's the wrong one.

Please note that mailing truely internationalized URIs
or domain names around does work quite nicely now, there
is absolutely no need for any standards upgrades, although
some implementations still leave room for improvement.
The same applies for printing and typing. The only
point where work is needed is where the domain name or
URI is taken as characters (independent of how they are
encoded, just the way the user sees them on the screen)
and then resolved.

>   deal with breakage is on a per-component basis rather than a

If it's by component, I doubt that ACE is better than UTF-8.
ACE needs a lot of special considerations, scripts that have
to be fixed, and so on, in order to work.

>- put most of the burden of upgrading on those who benefit most

I'm not sure I agree with this. First of all, it implies that
internationalized domain names have to be usable everywhere,
immediately. I don't think that's really the demand we are
facing.

>favorite tools en masse in favor of unfamiliar tools that support UTF-8,

They don't have to. On unix, the general thing you will have to
do is to create aliases or wrapper shell scripts for your editor
to either set the locale to work with UTF-8 or to convert the
file to the encoding your editor can handle and back.

Do you expect editors that can handle ACE transparently to become prominent?
Do you think it is sufficient to handle ACE as ASCII?

>it will probably break some tools that expect each character takes one

First, UTF-8 is *extremely* well designed to work in most cases even
for such tools (shell scripts, filter programs,...). If you can give
a concrete example of such a breakage in such kinds of programs,
I would appreciate it.

Regards,   Martin.

---

From: "Martin J. Duerst" <[email-masked]>
Date: Sat, 06 Jan 2001 18:28:37 +0900

At 01/01/06 09:24 +0100, Patrik F$BgM(Btstr$B‹N(B wrote:
>Even this is true.

Yes. But if Ivan writes back to Boris and asks why he is suddenly using
such a strange mail address, then Boris will be surprised, even
embarrassed, and will not know what happened and why.

>It is even the case that Boris can give his ACE-version of his real email 
>address to people in a form as "if you can not enter cyrillic characters 
>in my email address, please enter this instead".
>
>   f$BgM(Btstr$B‹N(B.com

Sorry, but I very strongly doubt that this should be done, and that
very many people will want to embarrass themselves that way.

>"faltstrom.com" or whatever,

Yes, that's what will happen. People want names, not garbage.
We all know that very well.

>but I might think that my customers if I help them that way never update 
>their software. I.e. I _WANT_ people to be able to use "f$BgM(Btstr$B‹N(B.com" 
>probably, so I would only help them to some degree, and that might be to 
>give them the ace encoding in paralel with the real name.

This sounds like a very secondary argument. If you want them to upgrade,
you will point them to your nice address and tell them what they have
to do to use it. Annoying your friends or business partners to get them
to upgrade is a really strange idea.

Regards,   Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 6 Jan 2001 13:27:08 +0100

>   expectations, I just want to make sure others don't).

I agree completely with this.

My point in all of this discussion is exactly that, that we _have_ to 
change software regardless of what we decide, to get full IDN 
functionality -- and because of this, we will live in a world where 
people have not upgraded their software yet, so backward 
compatibility is really important.

Some examples I use involve my own name which is

   Patrik H:son Fältström

If you look closely, you will see that "H:son" might be problematic 
to have as a domain part because of the colon, and the 'ä' can be 
written in two ways, which are equal according to the normalization 
forms defined by the Unicode Consortium. One of the ways will 
probably be used in Sweden more (where the 'ä' is a special 
character) and another outside of Sweden (where the 'ä' is an 
accented 'a').

Martin gives another good example which is the full-width and 
half-width characters.

This is not easy, and claiming that "this works already if we choose 
UTF-8 encoded Unicode is too naive.

What I said some mail ago was that "what the encoding of Unicode is, 
UTF-8 or ACE, is the simple part of this puzzle -- and the big 
difference is that an ACE encoding guarantees that the encoded words 
work in the application protocols we have today". I still claim that 
is the case.

    paf

---

From: "J. William Semich" <[email-masked]>
Date: Sat, 06 Jan 2001 08:48:30 -0500

At 09:24 AM 1/6/01 +0100, Patrik Fältström wrote:

<<snip>

>...but could also on my business card have (if the ACE ends up being
RACE):

>Yes, it looks ugly, ...

Not to mention it would be hard for most people to remember.

It looks a lot simpler to put "patrik@<fontfamily><param>MS Sans
Serif</param><smaller>208.184.130.48</smaller></fontfamily>" on your
business card as the "ASCII alternative", and probably is much easier to
remember. Exposing ACE to users begins to defeat the whole idea of using
meaningful "domain names", ASCII or UNICODE, to simplify identification
of hosts because "you can't remember all those numbers."

Numbers suddenly look a lot easier to remember than the ACE equivalences,
I expect, ignoring the "portability" issue.

Bill

Bill Semich

President and Founder

WorldNames, Inc.

http://www.worldnames.net

[email-masked]

---

From: Keith Moore <[email-masked]>
Date: Sat, 06 Jan 2001 09:14:48 -0500

> around, put them in their 'location:' fields, and so on?

no.  you're correct in saying that - at least in email - all of the
necessary standards are in place.  whether the deployed implementations
support those standards is a completely different question.  

my point is that IDNs (and URIs containing IDNs) will be
passed around from one program to another via a variety of means -
including protocols, cut-and-paste, and human transcription.
due to (in some cases) lack of protocol support (in others)
different things being upgraded at different times, 
some of those programs will be able to input, or display IDNs 
natively, and some will not.  

the primary reason for using ACE is within protocols  - i.e.
in contexts that most humans do not see.  but we of course
realize that ACE-encoded IDNs will leak into contexts that
humans do see, just as native IDNs will leak into protocols
that cannot deal with UTF8.  and those IDNs will get passed
to other protocols.  

the point I was making in my message to Dan is that the success 
or failure of whatever scheme we choose for IDNs will be determined 
by the degree to which things fail when IDNs get passed around 
in a wide variety of ways.  we therefore cannot predict success 
or failure by only looking at one or two test cases with one or 
two implementations of a single protocol.

> to be fixed, and so on, in order to work.

we need to look at this more carefully then, and choose an ACE 
which is unlikely to break things that currently deal 
successfully with domain names.  this is of paramount concern.

> immediately. I don't think that's really the demand we are

depends on what you mean by 'usable'.  I think IDNs need to 
provide minimum disruption everwhere, immediately.  but my 
point is that those who benefit most from being able to input 
or display IDNs are the ones who are most likely to be willing
to upgrade their applications.  if we adopted a scheme that
required *everyone* to upgrade to new applications - regardless
of whether they saw any benefit from IDNs - it would be more 
likely to result in pushback against IDNs.

> file to the encoding your editor can handle and back.

I suspect that most UNIX users are unwilling to do even this much - 
they'd have to remember to run a different command when editing 
a UTF-8 file than when editing a normal file.  though perhaps emacs 
would acquire a UTF-8 mode that would automagically detect UTF-8 -
that's quite doable.

> Do you think it is sufficient to handle ACE as ASCII?

Depends on what you mean by 'sufficient'.  But a lot of people, if faced
with the choice between using a new tool (or a new script) and handling
IDNs in ACE format, would choose the latter.
 
> a concrete example of such a breakage in such kinds of programs,

I agree that UTF-8 is quite well designed, but it does break the common
assumption that all characters are a single octet.  anything that uses 
'.' in a regular expression to match a single character will fail given 
UTF-8 as input, as will anything that tries to parse input by columns.  

Keith

---

From: "Marco d'Itri" <[email-masked]>
Date: Sat, 6 Jan 2001 15:33:03 +0100

On Jan 06, Patrik Fältström <[email-masked]> wrote:

 >>   realistic
Maybe this is a good argument for doing nameprep in the resolver?

-- 
ciao,
Marco

---

From: "Martin J. Duerst" <[email-masked]>
Date: Sun, 07 Jan 2001 00:56:10 +0900

>support those standards is a completely different question.

They support them to the degree that native users can
send around text in their own language. That support
is very solid, otherwise email wouldn't exist in some
places at all. What is missing is Japanese mailers
being able to do Arabic,..., but that's secondary.

>that cannot deal with UTF8.  and those IDNs will get passed

It's not necessarily needed that protocols deal with UTF-8.
Mail and Web pages are typical cases. If I send a mail in
iso-2022-jp or put up a web page in iso-8859-1, the IDNs
will be in these encodings, not in UTF-8. Everything else,
including ACE, would be wrong. And that already works
(Again with the problem that you can't put an ARABIC
domain name into an iso-2022-jp mail. But you can put
it in a web page using the &#dddd; syntax.)

>successfully with domain names.  this is of paramount concern.

It's not really a question of which ACE to pick. Scripts
are in many ways at the boundary of the protocols and the
user. That means that a lot of conversion between ACE and
something readable will have to go on. For ACE, all of
this will have to be hand-knitted, because you will
continuously have to parse the data to separate domain
names from other stuff. In many ways, we will get into
a very NAT-like situation. NAT can be dealt with by
knowing exactly what's going on on the wire, for each
and every protocol, in particular to know all the places
where internet addresses turn up. But it has turned out
that this isn't feasible. ACE depends on knowing exactly
where domain names turn up. The only difference is that
the problem is not on the wire, it's at each end, but
it's nevertheless a mess.

For UTF-8, things just go through. If you want to use
another encoding, you don't have to know where the
domain names are, you can convert everything, including
comments,...

>a UTF-8 file than when editing a normal file.

The alternative is to have the files in the encoding of their
preference, and insert an additional step in situations such
as when loading a zone file.

The important point is that users and administrators will
with close to absolute certainty prefer to edit text they
can read and understand rather than ACEed stuff. And for
this, UTF-8 is much less painful than ACE.

>though perhaps emacs

That has already been done. See e.g.
http://packages.debian.org/stable/editors/mule-ucs.html

>IDNs in ACE format, would choose the latter.

Not in Japan, not in any other place of the world where people
actually can read and understand these characters. Or would
you prefer to edit configuration files in nonsense sequences
of Greek characters rather than define and use an alias or two?
And if you think you would prefer it that way, do you think
that that's what people would prefer in general? Very hard
for me to believe!

>assumption that all characters are a single octet.

Which is anyway wrong in large parts of the world.

>anything that uses
>UTF-8 as input,

Yes. But how many such regular expressions do you expect
for handling domain names? Can you give a reasonable
example?

>as will anything that tries to parse input by columns.

Good point. But not exactly. The problems will appear
if applications that assume one octet=one column position
and applications that assume one character=one column position
get combined.

Regards,   Martin.

---

From: "Martin J. Duerst" <[email-masked]>
Date: Sun, 07 Jan 2001 00:24:32 +0900

At 01/01/06 13:27 +0100, Patrik F$BgM(Btstr$B‹N(B wrote:

>functionality -- and because of this, we will live in a world where people 
>have not upgraded their software yet, so backward compatibility is really 
>important.
>
>   Patrik H:son F$BgM(Btstr$B‹N(B
>
>If you look closely, you will see that "H:son" might be problematic to 
>have as a domain part because of the colon, and the '$Bg(B can be written in 
>two ways, which are equal according to the normalization forms defined by 
>the Unicode Consortium. One of the ways will probably be used in Sweden 
>more (where the '$Bg(B is a special character) and another outside of Sweden 
>(where the '$Bg(B is an accented 'a').

No, the data will almost always contain the precombined version as it is
used in Sweden. This is independent of what people think about it or
how they may input it, where you can indeed see quite some difference.
So for this case, normalization is mainly needed to avoid surprises
in very rare and strange cases, not to bring too variants together.

>Martin gives another good example which is the full-width and half-width 
>characters.

This is much more of a problem, because both exist side-by-side,
at least in Japan.

>This is not easy, and claiming that "this works already if we choose UTF-8 
>encoded Unicode is too naive.
>
>What I said some mail ago was that "what the encoding of Unicode is, UTF-8 
>or ACE, is the simple part of this puzzle -- and the big difference is 
>that an ACE encoding guarantees that the encoded words work in the 
>application protocols we have today". I still claim that is the case.

Well, let's try with some picture. ACE assures that all cars can pass
all tunnels by squeezing them together in a way that makes them look
like bikes. They pass, but they are not cars anymore, and you have
to be really lucky if somebody can help you to make a car again
out of a bike that came through the tunnel. And even if the tunnels
get wider, we will still squeeze all cars in the future.

Of course, that's only half of the story; it may be worth noting
that in principle, all tunnels nowadays are wide enough for cars,
indeed it's easier to build wide tunnels, but there are some
extremely old tunnels still remaining, and there are still
some engineers that, by accident or whatever, that don't clean
up their tunnels.

Regards,   Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 6 Jan 2001 17:36:01 +0100

>Not to mention it would be hard for most people to remember.

That was my point when I said that people would try to use the 
original IDN version.

>It looks a lot simpler to put "[email-masked]" on your 
>business card as the "ASCII alternative", and probably is much 
>easier to remember.

But, that would not work if not the top level domain "48" exists. If 
it was an IPv4 address literal you wanted to write it is 
syntactically wrong (you forgot the '[' and ']') and even if that was 
what you wanted that would not work either as the IP address change 
quite often for my mail host.

   paf

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 6 Jan 2001 17:38:29 +0100

At 15.33 +0100 01-01-06, Marco d'Itri wrote:
>On Jan 06, Patrik F”ltstr–m <[email-masked]> wrote:
>

Definitly _NO_ as the domain part is used not only in the DNS 
protocol but also in the various application protocols that exist, 
such as SMTP and HTTP. The nameprep have to be done on a higher level 
than in the resolver.

If your point was that the operating system (standard libraries or 
wherever) should have such a conversion routine which should be 
called by any program when using IDN domain names wherever, yes, I 
agree with you.

    paf

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Sat, 6 Jan 2001 17:55:17 +0000 (GMT)

On Sat, 6 Jan 2001, Martin J. Duerst wrote:

| Do you expect editors that can handle ACE transparently to become
| prominent?
|
| Do you think it is sufficient to handle ACE as ASCII?

  I'm not sure that this behaviour is a requirement.  Applications should
be able to deal with IDNs in native scripts (codepage or hopefully
Unicode).  Applications and/or resolvers will need to do the job of IDN
(to Unicode) to nameprepped Unicode to ACE conversion, ensuring that what
goes on the wire is ASCII.

  It sounds like there is a diversity of opinion as to where the IDN
processing will happen, and how far up and down the stack the ACE handling
will go. 

  -bws

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Sat, 6 Jan 2001 18:01:17 +0000 (GMT)

On Sat, 6 Jan 2001, J. William Semich wrote:

| >...but could also on my business card have (if the ACE ends up being
| RACE):
| >   bq--abtoi3duon2hf5tn .com
| 
| >Yes, it looks ugly, ...
| 
| Not to mention it would be hard for most people to remember.

  I don't think this is in reality a requirement.  If you decide to put an
IDN on your business card, some people may have problems using it, but I
would suspect rather quickly that most users, particularly non-English
speakers, will become aware that they will need to update their software
to use these names.

  Its sort of like the situation you had a few years ago when URLs and
e-mail addresses started appearing on business cards.  Many people who
received those cards weren't able to use those new identifiers as they
didn't have e-mail or web access.  This I think helped encourage more
rapid adoption of those technologies.  Something similar will probably
happen with IDN.

  Probably a common scenario will be a non-technical person receiving a
business card with an IDN on it.  They'll try to type the name into their
browser, and it will fail to resolve.  They'll contact their IT support
staff, who will tell them what they need to do to use the IDN (and educate
them a little bit on this 'new thing'), they'll update their software as
required, and they'll move on.

  Its not clear to me that you really need to expose ACE in a context like
this.

  -bws

---

From: "Martin J. Duerst" <[email-masked]>
Date: Sun, 07 Jan 2001 03:00:53 +0900

Thanks Ken for explaining this clearly. A bit of follow up:

At 01/01/04 11:14 -0800, Kenneth Whistler wrote:
>A terminological quibble here:
>
> >   I guess I still don't get why some people are so focused on UTF-8.
> > UTF-8 is an 8-bit encoding of the UCS.  ACE (whatever flavor) is a 7-bit
> > encoding of the UCS.

ACE is not a 7-bit encoding of the UCS. It's a two-step encoding,
first from the UCS to the legacy host name repertoire, and then
from there to 7-bit octets (using US-ASCII). Of course, for most
people on this list, the second step is just ignored, because
US-ASCII is taken for granted anyway.

To get to truly solid internationalization, it is highly desirable
that there is no difference between the handling of US-ASCII and
non-US-ASCII characters, and in particular that the later are
represented directly, and not via the former.

Also, because ACE maps to ASCII, it is only a very restricted
encoding of the UCS. Can you imagine a terminal emulator for ACE?
An XML page starting with <?xml version='1.0' encoding='ACE'?> ?
An editor that reads in a file in ACE, allows to edit it, and
write it back out again? It's possible to create such things,
but they won't be useful at all for IDN, because kind of
data has a special pattern of where domain names appear.

Regards,   Martin.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sat, 6 Jan 2001 10:58:00 -0800

>data has a special pattern of where domain names appear.

ACE was never meant for anything other than host name parts. Saying 
"it doesn't work well for things that it was not designed for" is 
both true and wholly irrelevant.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Jan 2001 15:18:09 -0500

--On Saturday, 06 January, 2001 18:01 +0000 "Brian W. Spolarich"
<[email-masked]> wrote:

> | >...but could also on my business card have (if the ACE ends
> up being | RACE):
> |...
>   I don't think this is in reality a requirement.  If you
>   decide to put an IDN on your business card, some people may
> have problems using it, but I would suspect rather quickly
> that most users, particularly non-English speakers, will
> become aware that they will need to update their software to
> use these names.

Brian, the problem really isn't updating software but updating
people -- an activity that we know, experimentally, tends to
take a few generations at least.  

Let me use myself as an example (this one is inherently
European-language-oriented, but I assume it has Asian and
African analogues that are as bad or worse).  If I'm handed a
business card with an IDN on it, in the "native" character glyph
set for that name, the next encoding step depends on my
knowledge and pattern-recognition and discrimination skills, and
not on any computer issues.  I need to be able to take that
collection of glyphs and somehow transcribe and enter them into
a computer.

Now, if the characters used are Latin-based, I can probably cope
(and I do today).  I'm going to recognize the basic structures.
Even if I have to consult a table, I can match what I see on the
card with what I see on the screen.  In the worst case --in
which I have no idea how to "type" the character-- I can then
enter the character position from the table into a software
program.  Interestingly, for that purpose, a simple hexadecimal
encoding of UCS-4 (or UCS-2) is "easier" than UTF-8 which is a
good deal "easier" than ACE.  While tedious, 0x000000E4 is quite
easy to figure out and write.  And selection of characters from
tablets is not exactly a new technology; I'd expect user
interface software to show up quickly and be widely deployed
that uses table-selection technologies to build up strings that
could then be pasted.

But that is the easy part and it is conditioned on my being able
to recognize Latin-based characters well enough to tell them
apart.  Extrapolating from it as an example is dangerous:

First, I can't tell European characters apart to the degree
needed by IDN.  If you had me a card with a native-glyph IDN on
it, I need to know the language (or a surrogate for the
language) to pick the right glyphs from a table.   I simply
cannot tell U+0041 from U+0391 from U+0410 by looking at a glyph
written on a page (and those are, of course, not the only
examples).  I have to know the language context, or apply
heuristics, to pick one.  And, if I can't be guaranteed to
consistently get it right, we are going to need
character-matching rules much more subtle than anything yet
seriously contemplated.

But that isn't the serious problem either.  It turns out that I
can do just about the same pattern-matching job --looking at a
glyph, deducing its essential characteristics because I'm
familiar with the repertorie from which it comes, finding it in
a table and picking it or entering it by table position-- if
that card contains Greek, Cyrillic, or Hebrew glyphs.  That puts
me a bit ahead of most of my American and Western European
colleagues, for which those sets of glyphs may look like
indistinguishable chicken-scratches.   I've even been around
this work long enough that I've learned to distinguish enough
kana that I might be able to successfully look them up in a
table (but not pronounce them), although it would take me a long
time since I don't know the table order.   But most Han, Kanji,
and Hangul are hopeless for me: there are lots of them, I can't
(in general) tell them apart, and I just don't have the pattern
recognition skills (or training in distinguishing radicals) to
be able to look them up, especially if the font-rendering on the
business card is different from that in my table.

That is not, of course, an alphabetic versus ideographic issue:
I'm equally hopeless with Arabic, and Devanagari, and Bengali,
and Thai, and dozens (if not hundreds) of languages and
character collections, most of which I've never even seen in
print.  
Anything I can't recognize well enough to match with a table is
something I can't copy from a business card.  Whether the DNS
interface is UTF-8, or an ACE, or UCS-4, or some other encoding
is irrelevant, and software isn't going to fix that.

So, for the business card, we need, I think, to start asking
different questions.  If want an address on my business card to
be recognized and usable, after the translator leaves, by a Thai
native who lacks familiarity with Latin alphabets, I had best
figure out how to get a Thai rendering (or something else
recognizable) of my address on that card.   And, operationally,
I had better fix it so that rendering does the right thing when
looked up in the DNS.  Note that this is my problem and not
hers: she has a problem only if I don't do something reasonable
and she decides to try to communicate with my anyway.

Another important inference from this story is almost certainly
that a solution that works for "ASCII" and "other language" is,
at least in the long term, inadequate.  The same issues exist
between "other language 1" and "other language 2", and the
virtual business card will need to be many-sided.  And that may
imply that I may need many DNS identities, associated with
different character sets, to be accessible from all over the
world.

Or we can try to turn it into an ACE-like question, where what
goes on the business card is the "native" set of glyphs plus
some Latin-based encoding.  I'm not sure, as argued above, that
there is a "native" set of glyphs for the general case.  But,
more important, if one is going to put an encoding on a card
from which one can get back and forth from 10646/Unicode, it is
not clear to me that ACE is superior to hexified UTF-8.  And
neither may be as convenient as 
hexified UCS-4 (or UCS-2) or a PGP-like biometric word list
encoding.  Any of these are going to require software to get
from the business card form to the DNS one; the correct answer
for encoding them may be different from the correct answer for
what should actually go into the DNS.

>   Probably a common scenario will be a non-technical person
>   receiving a business card with an IDN on it.  They'll try to
> type the name into their browser, and it will fail to resolve.
> They'll contact their IT support staff, who will tell them
> what they need to do to use the IDN (and educate them a little
> bit on this 'new thing'), they'll update their software as

	Sigh.
	
	JU:  Hello, IT support staff, this is Joe User.  I've
	got a business card in front of me with a URL on it, and
	I don't know how to type it in.
	
	ITSS: Just copy the letters from the card into the "Go
	to" box on the browser.  Haven't you learned that yet?
	
	JU: But I don't recognize any of the characters.. they
	aren't on my keyboard.
	
	ITSS: We can send you out a keyboard for Latin-1, which
	covers Swedish, German, French,... and instructions for
	installing it.
	
	(A week later)
	
	JU: Hello, IT Support Staff, your technician put in the
	nice new keyboard and software.  Some of my applications
	now say "Guten Tag", rather than "Hello", but that is
	ok.  I still can't type that URL in, the characters
	still aren't on the keyboard.
	
	ITSS:  Huh?  What language did you say it was in?
	
	JU: Beats me.   Doesn't look much like either English or
	Klingon.
	
	ITSS:  (The next two or three exchanges are left as an
	exercise for the reader.  You may assume, if convenient,
	that the IT support staff speaks Klingon and has been
	studying the use of the Batlef (sp?))

Sorry Brian, I don't believe "a little education, update the
software, and move on" is going to do it.

     john

---

From: Jonathan Rosenne <[email-masked]>
Date: Sat, 06 Jan 2001 23:37:20 +0200

I would like to note that my numeric IP address (for www.qsm.co.il) is shared
with other companies hosted by my ISP and they use the domain name to sort
things out. A numeric IP on my business card would not necessarily be equivalent
to an ACE or IDN.

Jony

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf
> Of Patrik Fältström
> Sent: Saturday, January 06, 2001 6:36 PM
> To: J. William Semich; [email-masked]

---

From: [email-masked]
Date: Sat, 06 Jan 2001 18:33:11 -0700 (PDT)

> some engineers that, by accident or whatever, that don't clean

That may be your belief, but saying it is so doesn't make it true. My
experiencce with this sort of stuff is extensive, and it is exactly the
opposite of what you claim.

					Ned

---

From: "Adam M. Costello" <[email-masked]>
Date: Sun, 7 Jan 2001 10:23:30 +0000

"Martin J. Duerst" <[email-masked]> wrote:

> If I send a mail in iso-2022-jp or put up a web page in iso-8859-1,
> the IDNs will be in these encodings, not in UTF-8.  Everything else,
> including ACE, would be wrong.

That's an interesting problem.  People exchanging Japanese email do
indeed use iso-2022-jp, not utf-8.  If one of them wants to include a
Japanese domain name in the body of a message (as part of a URL, for
example), it would be a shame to use ACE, because it will be visible to
both sender and recipient, unless the MUAs do some kludgy conversions on
message bodies.  On the other hand, suppose the URL in question refers
to an image, and the sender wants his non-Japanese friend to see the
image.  If we don't use ACE, then there may not exist a charset that the
sender and recipient are both able to use that's capable of representing
the domain name.

This suggests that users should have some way of representing any domain
name using ASCII (as a last resort), regardless of what goes on the
wire.

This scenario raises a big question about URIs.  According to RFC
2396, a URI is a sequence of characters (not bytes) from a very small
repertoire (smaller than printable ASCII); this sequence of characters
can be used to represent a sequence of bytes (using ASCII and %hh),
which in turn could be used to represent characters from a larger
repertoire, but that last mapping is not standardized.  In any case,
when a URI is presented to the user, the characters making up the URI
itself are supposed to be shown, not the characters represented by the
octets represented by the URI.

If we want URIs to appear to contain non-ASCII characters, we need to
alter the URI model somehow.  Here are three possibilities (there may be
others):

1. Use ACE, and add a rule saying that user agents should convert
to/from ACE when interacting with the user.  This has the advantage of
changing nothing at the lower layers, but doesn't immediately answer
the next obvious question: how to allow non-ASCII characters to appear
in the rest of the URI.  Perhaps ACE should be generalized to be usable
in other parts of URIs in addition to the domain names.  This approach
exposes the ACE to people editing files containing URIs.

2. Change the rule about how URIs are presented to the user, so that %hh
sequences (except those for excluded ASCII characters) are unescaped and
interpreted as UTF-8.  When URIs are obtained from the user, they must
be converted to UTF-8 and escaped before being dereferenced.  This has
the same advantages and disadvantages as the previous approach.

3. Allow a much larger set of characters in URIs.  This approach suffers
from the same problem as in the non-Japanese friend scenario above.
It would also require changing any protocol that embeds URIs without
specifying a sufficiently expressive encoding, most notably HTTP.  HTTP
does not say what encoding is used for URIs; it is implicitly ASCII
or an extension of ASCII.  HTTP assumes ISO-8859-1 for comments and
unstructured text in message headers, so it would not go without saying
that 8-bit URIs are in UTF-8; that would have to be added to the spec,
and who knows how existing implementations would react to 8-bit URIs.

Allowing URIs to appear to contain non-ASCII characters (using any
approach) is counter to the URI design goal of transcribability (the
business card scenario), but if we value transcribability more than i18n
then we should forget about IDN altogether.

> first from the UCS to the legacy host name repertoire, and then from
> there to 7-bit octets (using US-ASCII).

Thank you!  I had missed that subtlety, but you're right, host names
are not required to be represented in ASCII; they are just sequences
of characters.  An EBCDIC system would apply an EBCDIC-to-ASCII
transformation to any outgoing host names; if ACE mapped Unicode to
bytes rather than characters, the ACE host names leaving the EBCDIC
system would get scrambled.

John C Klensin <[email-masked]> wrote:

[business-card scenario]

Ouch.  IDN is so difficult, maybe it's hopeless.

The unaccented Latin alphabet and Arabic numerals (and a few punctuation
marks) make an excellent set of characters for universal names, because
it's a small set that is recognized by more humans and more machines
than any other.  That's surely a big part of why this set was chosen
for domain names and URIs, which are intended to be usable by anyone,
anywhere.

A giant character set is by nature ill-suited to the task of universal
naming.  Perhaps we have no choice but to view IDNs as deliberately
non-interoperable things, optimized for particular communities.

As for the business cards, maybe people will just register two names,
one native and one romanized (perhaps translated), with one being a
CNAME for the other.  I can't immediately think of anything easier for
the people trying to use the names on the card.

AMC

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Sun, 7 Jan 2001 13:41:43 +0000 (GMT)

[thanks for thinking to change the subject]

On Sat, 6 Jan 2001, John C Klensin wrote:

  [ much Good Stuff snipped ]

| Sorry Brian, I don't believe "a little education, update the software,
| and move on" is going to do it.

  You've of course uncovered the hidden assumption in my thinking, which
is that the recipient of the business card can operate in the language(s)
displayed on the card.

  This is, as you point out, a people problem, and not a software one.  
Its also not clear if this problem is really ours to solve, or is
solveable by us.  My working problem definition for IDN is to enable
rational support (and deployment of support) for UCS in the DNS.  While
usability, particularly in the context of a diverse and global audience,
has to be carefully considered, I don't think we need to address the whole
range of communication problems imposed by an increasingly globalized and
interconnected world.

  I recall when I visited Hong Kong last year my Japanese and Chinese
colleagues at UUNET gave me business cards with English on one side,
Japanese or Chinese on the other.  This seemed like a reasonable
compromise.  Its likely that this is the only version that they had
available, but I could imagine a very careful global enterprise having
multiple sets of cards printed with the 'flip side' language being Arabic,
English, Russian, etc. depending on who the recipient was.  

  As an aside, two-sided business cards are kind of annoying.  How often
do you write notes on the back of someone's card?

  The point here I think is that, when two parties are going to
communicate, it is incumbent upon them to negotiate a common language or
communications protocol.  If they cannot, then communication will be
extremely difficult.  This is true whether we're talking about domain
names/URLs on business cards, or more prosaic things like addresses,
telephone numbers, etc.

  Being the enterprising creatures we are, I can see any number of
solutions to this kind of problem.  So-called 'smart paper' will be
available economically within a few years.  The static business card may
be replaced with a 'smart' version that has multiple translations the user
can switch between (or even a wireless network connection).  Simpler
solutions could be extensions of the practice the VIDN draft author
identifies, such as having an alternative form of a domain name available
in transliterated Latin characters. 

  This begs the question of whether the Latin alphabet is the appropriate
'interchange' alphabet (and all of the cultural hegemony issues, and much
history).  Over a billion Chinese speakers would probably disagree.  But
again, is this our problem to solve?

  -bws

---

From: "D. J. Bernstein c/o James Seng" <[email-masked]>
Date: 7 Jan 2001 20:44:33 -0000

Operating systems are all going to support direct input of Unicode
characters by number. The ISO standard method is Shift-Ctrl-222E for
character 222E.

People who put hard-to-recognize characters onto their business cards
are going to add a line giving the Unicode numbers, perhaps in boxes:

   [email-masked]            (with a contour-integral sign)

              +----+
   postmaster@|222E|.cr.yp.to       (in smaller type)
              +----+

The second line won't distract the people in the intended audience who
recognize the contour-integral sign and have, say, Alt-S configured to
produce it.

End of problem. Notice that this doesn't require any extra DNS names.
The conversion of Shift-Ctrl-222E to \342\210\256 is isolated inside the
keyboard interface; other software doesn't have to worry about it.

---Dan

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 7 Jan 2001 13:31:46 -0800

>are going to add a line giving the Unicode numbers, perhaps in boxes:

*Are* going to?! It is incredibly presumptuous of anyone, even Dan 
Bernstein, to say what people all over the world are going to do on 
their business cards.

>    postmaster@|222E|.cr.yp.to       (in smaller type)

That almost works for host names that contain a single non-ASCII 
character. A typical company name in Arabic, for example, might have 
ten non-ASCII characters, and the TLD might contain another five 
(yes, we should be assuming longer TLDs that contain country names 
that use non-ASCII characters!). So, the proposition is that the 
translation from 15 characters in the "real" address will expand to 
60 characters, plus "perhaps boxes" around each set of four 
characters.

This might barely work for Europeans; it is unworkable for the 
majority of the world.

>End of problem.

For your host name; others might have a few more problems than you. 
Maybe the IETF should work a bit harder on a solution that meets the 
needs of more than a tiny audience.

--Paul Hoffman, Director

---

From: "D. J. Bernstein c/o James Seng" <[email-masked]>
Date: 7 Jan 2001 22:29:17 -0000

Why do we need name preparation in applications?

I'm not saying the nameprep work is useless. We should provide a program
to detect bad names: names with confusing characters, or with uppercase
characters, or that aren't KC-normalized. Registries will then prohibit
bad names.

But what will go wrong if these rules are hidden from applications? Why
should a typical application worry about bad names? Good names will be
accurately copied by cut-and-paste. When will bad names show up?

Mark Davis writes:
> The chances that the average application would, by chance, duplicate
> the same normalization is minimal.

You're saying that a user will see a good name on paper, type it into
his computer in his favorite way, and end up with a bad name that looks
the same.

Can you please give some real examples? What were the good names? What
keyboard-interface software was involved? What did the user type? Why
didn't the user end up with good names? Was it a bug in the software?

Perhaps there's a need for a tool that fixes bad Chinese domain names.
However, I don't see why this conversion should be stuck into every
piece of software that reads domain names from configuration files.

---Dan

P.S. RFC 1034 says ``When you receive a domain name or label, you should
preserve its case.'' Are we going to scrap this part of RFC 1034, and
encourage clients to fold case for ASCII characters? Or are we going to
say that the case of ASCII characters should be preserved?

Of course, for interoperability, DNS software and mail software and so
on will continue _comparing_ ASCII characters without regard to case.

---

From: "D. J. Bernstein c/o James Seng" <[email-masked]>
Date: 7 Jan 2001 23:38:18 -0000

Martin Oldfield writes:
> although it would look odd,

If you think that incorrectly displayed IDNs are acceptable, why are you
here? You can move from mail.tc to fifth-word-is-ace-xyzzy-mail.tc, and
we'll pretend that xyzzy has some accents in it. They aren't displayed
as accents, but that doesn't matter to you!

> Ivan can't communicate

What you mean here is that Ivan can't send mail to Boris's IDN.

> unless Ivan can persuade his ISP to upgrade his sendmail,

Maybe, maybe not.

The specific problem is that sendmail discards bytes \200 through \237.
This doesn't affect the UTF-8 encodings of the lowercase characters that
you can find in 8859-1. For example, a-circumflex is \303\242.

But I don't find this situation acceptable. sendmail has to be fixed or
replaced. Note that USENET messages now use UTF-8 in the header and are
often gatewayed verbatim to mail, so sendmail has to be fixed no matter
what happens with IDNs.

> but an ACE would only need Ivan to upgrade his mail client ?

By saying ``only,'' you are implying that UTF-8 IDNs require a client
upgrade. In fact, there are already clients that handle UTF-8 IDNs.

Certainly ACE IDNs require a mail client upgrade. When the mail client
sees a From line with an ACE IDN, it has to display some international
characters for the user.

Now, your question is whether ACE IDNs require a sendmail upgrade, if
the system administrator doesn't need IDNs in his configuration files,
and if addresses in outgoing messages are converted to ACE by the MUA.
The answer is still yes, because sendmail often copies domain names into
text that users are supposed to read.

---Dan

---

From: "D. J. Bernstein c/o James Seng" <[email-masked]>
Date: 8 Jan 2001 00:28:55 -0000

UNIX is switching to UTF-8. I said a bit about the cost of this
conversion. I commented that ACE IDNs wouldn't reduce this cost.

``You haven't said anything to support this argument,'' Keith wrote.

Do you believe that standardizing ACE IDNs will stop UNIX from switching
to UTF-8, Keith? Will it also return us to the age of 16-bit processors,
and give us tidy houses with porches and yards and white picket fences?

Or do you believe that standardizing ACE will somehow reduce the cost of
the switch? If so, how? Exactly what cost will we avoid? Do you believe
that we won't have to install X Unicode fonts, for example, because ACE
IDNs will shield us from the plague of un-American characters?

> (and your trial produced a failure, you just refuse to recognize it)

Keith, when you refer to a successful experiment as a ``failure'' merely
because it used protocol extensions that haven't been approved by the
IESG, could you please surround your comments by <religious-fanaticism>
and </religious-fanaticism>? Thanks.

> "costs and benefits" are highly subjective

Tell that to the users who have suffered needlessly because of your
previous protocol design screwups, Keith.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Sun, 07 Jan 2001 21:39:01 -0500

Dan,

Even if "UNIX" switches to UTF-8 there will still be legacy systems,
legacy application codes, and legacy text files, in use for many years.  
It's not as if everybody will upgrade en masse.

Keith

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 8 Jan 2001 03:14:47 +0000

"Brian W. Spolarich" <[email-masked]> wrote:

> This begs the question of whether the Latin alphabet is the
> appropriate 'interchange' alphabet...  Over a billion Chinese speakers
> would probably disagree.

Why would they?  Chinese speakers would like to use Chinese characters
when communicating with other Chinese speakers, but when they're
communicating with non-Chinese speakers, Chinese characters are pretty
useless.  What symbols would they choose to use then?  Do you think
there are some symbols other than the Latin letters and Arabic numerals
that would be easier for them?

AMC

---

From: Jonathan Rosenne <[email-masked]>
Date: Mon, 08 Jan 2001 09:02:01 +0200

For example, when I type a name, or even cut and paste, I have no control over
the encoding. Since the UCS offers alternative equivalent encodings, they must
be normalized before they are used.

Jony

> Of D. J. Bernstein c/o James Seng
> Sent: Monday, January 08, 2001 12:29 AM
> To: [email-masked]
> Subject: [idn] Where will we see bad domain names?
>
>

---

From: "Martin J. Duerst" <[email-masked]>
Date: Tue, 09 Jan 2001 21:08:51 +0900

> > the same normalization is minimal.

I think what Mark meant is that the chance that an average implementation
trying to do something in this area as virtually zero chance to by
chance get everything the same as another one or a well-defined standard.
I agree with this.

This is different from the chance that an average user typing
in an average internationalized domain name will get different
character codes without nameprep than with nameprep from his
keyboard software are in most cases extremely minimal, with a
few very notable exceptions.

>didn't the user end up with good names? Was it a bug in the software?

The classical case is half-width and full-width alphabetics and
Katakana in Japan. It's not a bug in the software, but a configuration
option. Many people are not aware of the fact that full-width
Latin letters are different character codes than half-width (ASCII),
and their keyboard may come set to full-width, or they may set it
that way.

>Perhaps there's a need for a tool that fixes bad Chinese domain names.

There are quite some things that may go wrong with Chinese domain
names, but there are so many characters, with so many subtly or totally
different relations, that it is impossible to use a uniform (e.g.
table-based) approach. The solution here is to register more than
one form if it's needed; the number necessary is usually very
small (not as in case folding).

>on will continue _comparing_ ASCII characters without regard to case.

The idea is to:

- Have different case versions behave as equivalents.
- Move case-folding away from the servers because of performance
   and updating issues (and potentially because of cultural differences
   that cannot be handled on the server side).

Whether ASCII case folding is treated as up to now or will be
integrated with the rest isn't clear to me yet.

Regards,   Martin.

---

From: "Martin J. Duerst" <[email-masked]>
Date: Tue, 09 Jan 2001 21:52:31 +0900

Hello John,

I think your considerations are important, but I think
you went a bit overboard on some issues.

>collection of glyphs and somehow transcribe and enter them into

There is one very clear reason that this problem will in practice
be more serious for what you call European languages (really,
the Latin script) than for other scripts.

It's interestingly enough the fact that Latin is easier for many.
French or Swedish business cards are not usually double-sided,
because they are recognizable even if there are some accents.
If you use them to send a letter, the letter will arrive even
without the accents. So there is quite some danger that people
happily create domain names with accents or other diacritics,
and put them on their business card.

On the other hand, for Arabic or Japanese (or even just Greek
or Cyrillic), there is no expectation nor that your international
business partners will somewhat be able to send you snail mail
with just the 'local' side of the business card, or even for
that matter have any idea of whom they met when they look at
the card a few days later. Therefore I think we will quickly
see things develop to have dual domain names, one for each
side of the business card.

As Paul has noted, in the European context (with few non-
ASCII characters), an approach such as Dan's might even
work, although I don't think it will become widespread,
for various reasons including the fact that people don't
know how to use the hex input feature.

>that uses table-selection technologies to build up strings that

Already available, in any Japanese input method (not really
for Latin) and in software such as MS Word, and probably
quite a few others.

>apart.  Extrapolating from it as an example is dangerous:

Yes, it's dangerous, because things are not linear.

>heuristics, to pick one.

Well, there are situations where this can go wrong. But
as I said above, for Cyrillic and Greek, we'll probably
see a 'two-sided business card' approach. Also, please
note that if there are ambiguities, the main people who
will suffer from them are the locals, so they will try
to avoid them. Although for them less things may be ambiguous
than for you, and it may be easier for you to start inputing
an ascii A and then somehow continue on the rest with tables
(rather than switch the keyboard at the start, as they will
do), the cases where this won't work won't be that frequent.

>business card is different from that in my table.

You are explorating this too far. Just because you personally
can handle (with a lot of pain) some part of this, don't
expect that this is what your business partners will expect
from you, or what everybody will have to do, and so on.

>recognizable) of my address on that card.

Yes. Visitors to such countries interested in business contacts
with such people regularly prepare double-sided namecards.
But it's usually just the name of the person, and maybe the
name of the institution, that is transcribed/translated.
I would not expect the street address or the email address
to be translated, because this may raise expectations that
you can be sent Thai email. For the Web address, that's
different, if your company has a Thai Web site with a
Thai address, it should go on there.

>different character sets, to be accessible from all over the

For Web sites for a multinational, clearly yes. For email
addresses, clearly no. If somebody can't type an ASCII
address, how are they going to send you email you can
read?

>what should actually go into the DNS.

People want names, words, whatever you call it, for their
Web and email addresses. Trying to provide a fallback
for a business card won't work. And translating or transliterating
names is not something that can be automated, so this has
to be solved by double registrations.

>         JU: Beats me.   Doesn't look much like either English or

This is the point where your example becomes extremely
unrealistic. Do you really think somebody from Japan will
hand you a Japanese-only business card and tell you:
"Here is my business card, I don't bother whether it's
of any use to you."? People are a bit more intelligent
than that.

The case of languages written with the Latin script
really leaves me much more to worry, because people are
accustomed to the fact that the accents can be dropped
(e.g. in snail mail, or for the pronunciation of the
name, where the average foreigner will anyway not get
it right).

I'm not saying that we should ignore Latin accents, not
at all. But it may take people much more time to understand
that putting an accented email address on their name
card will not work for every recipient, than it will
take somebody in Russia, Egypt,... to understand that
a Latin equivalent is needed.

Regards,   Martin.

---

From: [email-masked]
Date: Tue, 09 Jan 2001 09:48:54 -0700 (PDT)

> few very notable exceptions.

I think you grossly underestimate the liklihood of this happening.

Here's one example. Suppose I'm sitting in front of a mail client that uses
ISO-8859-whatever and which is attached to both an Internet and  X.400 mail
system. (Although I wish it were otherwise, such setups are quite common.) I
send a message whose body contains a domain that in turn contains accented
characters which, for whatever reason, takes two paths, one through X.400 and
one not.

Now, while X.400 has support for ISO-8859-whatever as part of generaltext,
support for generaltext is so spotty in practice that it is best avoided. So
many (even most) X.400 systems are configured to fall back to the T.61
character set. Unlike ISO-8559-whatever, T.61 represents accented characters as
composed sequences. (The composition order is the opposite of Unicode, but no
matter.)

So we have one path that converts ISO-8859-whatever to Unicode directly and
another that converts ISO-8859-whatever to T.61 and then to Unicode. But given
that all ISO-8859-1 characters are directly representable in Unicode, the
obvious thing to do is use the composed forms. And given that T.61 combining
accents are all in Unicode as well, the obvious thing to do is use decomposed
forms. The result is that the Unicode sequence depends on the path the message
takes.

Is this a contrived example? Hardly -- the reason it occurred to me is that I
saw it happen. Such cases may be hard to think up in the abstact, but in
practice software tends to do things that even its designers and implementors
could not have imagined.

This is why interoperability is a mandatory thing for us, not an optional
one. When we find a standard where conforming behavior could result in
an interoperability, we call that a bug and we fix the standard, regardless
of whether or not we think the problem is likely to occur.

One final note. The recent flap about UTF-8 security issues and the resulting
prohibition of recognition non-shortest length forms illustrates that name
comparison can have security implications. Given that applications frequently
compare domain names as part of security checks this makes this a potential
security problem as well as an interoperability problem.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Jan 2001 13:55:33 -0500

> you went a bit overboard on some issues.

Given that I was trying to identify the extreme cases, that would
not be a surprise.

>> business card with an IDN on it, in the "native" character
>>...

> There is one very clear reason that this problem will in
> practice be more serious for what you call European languages
> (really, the Latin script) than for other scripts.

Actually, I really intended European languages, although that
term too is a little sloppy.   A "western-trending subset of the
character sets derived from Old North Semitic" would have been
better, but that isn't precise either, and fewer people would
have understood it.  Explanation below.

> It's interestingly enough the fact that Latin is easier for
> many. French or Swedish business cards are not usually
> double-sided, because they are recognizable even if there are
> some accents. If you use them to send a letter, the letter will
> arrive even without the accents. So there is quite some danger
> that people happily create domain names with accents or other
> diacritics, and put them on their business card.

I can't speak with any confidence about French or Swedish, but it
is certainly possible --as has been pointed out on this list by
others more expert than I-- to create some rather serious
ambiguities in German by simply dropping diacriticals (not,
strictly, accents) without substituting some in-stream
convention.  Whether the results are still deliverable in the
postal system may depend more on the ability of humans to do
fuzzy matching of an envelope with a known number of possible
targets than with any characteristics that would do us good in
the DNS.
> or Cyrillic), there is no expectation nor that your
> international business partners will somewhat be able to send
> you snail mail with just the 'local' side of the business card,
> or even for that matter have any idea of whom they met when
> they look at the card a few days later. Therefore I think we
> will quickly see things develop to have dual domain names, one
> for each side of the business card.

And here I would argue that you are correct but, unless you are
willing to walk the slippery slope that overhangs the "it would
just be easier if everyone learned English" argument, you have it
backwards.  Your presumption seems to be that my Russian business
partners will need to put English on their business cards and,
for the DNS, dual-register.  I don't see why they shouldn't take
the position that it is my responsibility to put Cyrillic on my
business cards (presumably the front side, with English on the
back).  And why they should use cards with English/ASCII on them
at all when communicating with, say, their Korean business
partners escapes me.

Just in passing, there are interesting historical artifacts in
some of this, but we should not confuse them with general
principles.  For example, most Greek characters bear more than a
passing resemblance to Roman ones (no surprise, see comment about
North Semitic above) and most of the rest are known to those who
have studied mathematics.  That makes Greek recognizable to a
significant fraction of the educated US and Western European
population.   Does it mean single-sided cards and
single-registrations?  Well, if I were Greek and wanted to carry
out commercial relationships with English-speakers, I wouldn't
risk it.  But I wouldn't risk it if I were Swedish, either.

Similarly, these postal analogies are really very misleading.
What is officially deliverable internationally is specified to a
collection of treaties which, if my memory is correct, have their
origins in the Napoleonic period and require that anything
internationally-significant on envelopes moving between countries
be in French.  So things are deliverable, inter-country, either
on the basis of tolerance and fuzzy matching (as mentioned above)
or because they are in a Roman-based language.  The ideological
portion of the IDN discussion involves not imposing another few
hundred years of French (or English) on people who aren't native
users or those, or closely-related languages... and that position
seems to me to be reasonable.

> know how to use the hex input feature.

I agree, although a reasonable person might suggest that hex
input is significantly easier for the typical person than
figuring out the algorithms to construct and write an ACE from
scratch.
 
>> Now, if the characters used are Latin-based, I can probably
>> cope (and I do today).  I'm going to recognize the basic
>> structures. Even if I have to consult a table, I can match
>> what I see on the card with what I see on the screen.  In the
>> worst case --in which I have no idea how to "type" the
>> character-- I can then enter the character position from the
>> table into a software program.  Interestingly, for that
>>...

Available for Roman-based characters longer than there has been
an Internet-- such tablets are widely used to enable writing (and
sometime "speaking") by those who are impaired manually and/or
orally.

>> cannot tell U+0041 from U+0391 from U+0410 by looking at a
>> glyph written on a page (and those are, of course, not the only

I'm sure it isn't what you intend, but I can't parse the above
without a "Roman-based characters are the center of the universe"
assumption.
 
>> that card contains Greek, Cyrillic, or Hebrew glyphs.  That
>>...

That is exactly my point.  I will rush right out and order some
fifteen-sided business cards... as soon as a discover a
sixteen-dimensional space in which to hand them out.

>> be recognized and usable, after the translator leaves, by a
>> Thai native who lacks familiarity with Latin alphabets, I had
>>...

But, if a Thai user has a keyboard without ASCII characters on
it, and I expect him to communicate with me, I'd better be able
to render an address into Thai.  Yes?

> address, how are they going to send you email you can

I don't mean to be faceteous here, but, if they wanted to send a
picture, or a musical piece, there really would not be an issue,
would there?  And my examples don't imply that I can't read Thai
(I can't, but...) or go find a translator and bring a diskette
with me, only that, if my email address is only renderable in
ASCII, there may be transmission problems if we are in a world in
which such address-exclusivity is not considered problematic.

> for a business card won't work. And translating or
> transliterating names is not something that can be automated,
> so this has to be solved by double registrations.

With the qualification that I think "double" is going to need to
be a much larger number, we are probably in agreement about this.
But think about what it means: As a database with synchronize and
commit primitives, the DNS just stinks.  CNAMEs are extremely
limited in capability and often surprise people even within that
range.  

So, "multiple registrations", if done in the DNS, will tend to
imply multiple names (in different languages) mapping onto the
same object/target (RHS of A, MX, or NS RR).  But, when changes
need to be made in, e.g., the address of some host, zone
differences in propagation are going to get us badly, to say
nothing of the odds of human errors (e.g., forgetting one or
two).    But that implies that all of the "multilingual" forms
ought to map onto --perhaps with CNAMEs or some new RR or from
outside the DNS-- a single ASCII-renderable name which is then
the source for, e.g., name to address translation.  And, whether
it takes us toward a directory or a two-stage intra-DNS lookup,
that is a rather different model from the current directions of
the IDN WG.
>> >   receiving a business card with an IDN on it.  They'll try
>> >   to type the name into their browser, and it will fail to
>> >...

Not my example, just my (slightly sarcastic) comments on another
note/ point of view.  But, usually because I don't have time to
prepare properly, I carry English-only cards into places where
they may be of no direct (i.e., translator-free) use to the
native population fairly regularly.  I shouldn't, and I'm
sensitive enough to feel bad about it, but I've done it too many
times.

And I don't know how anyone not intimately familiar with a
language (and knowing the language, not just the script in which
it is written, can distinguish between an "accent" and a
diacritical symbol that is actually part of the character-symbol.

     john

---

From: "D. J. Bernstein" <[email-masked]>
Date: 10 Jan 2001 02:40:03 -0000

Martin J. Duerst writes:
> Katakana in Japan.

But we've already learned that nameprep doesn't solve the Japanese
name-typing problem. People in Japan aren't getting the right dots.

The obvious solution is to improve the address-typing support in the
operating system's Japanese keyboard interface. That's the only piece of
software that has to worry about bad domain names. It will provide a
friendly Japanese-specific environment for typing good domain names. It
will deal with the dot problem.

Done.

In contrast, Patrik and Bill want us to add bad->good conversion code
(and Japanese dot handling) to Outlook, Messenger, Eudora, Mutt, Pine,
Sendmail (which will have to check whether incoming IDNs match UTF-8
domain names in user-typed configuration files), qmail (same reason),
Exchange, IMail, Exim, Post.Office, Explorer, Communicator, Opera, Lynx,
w3m, Apache, IIS, Enterprise, WebLogic, Zeus, BIND, djbdns, Webmin, etc.

That is not good software engineering.

Of course, if every one of those programs has to be changed, then every
one of them has to be redeployed. Patrik claims that this is necessary
for full IDN support. But my solution will allow some of these programs
to be left alone: they can already handle UTF-8 IDNs without trouble.

---Dan

---

From: William Morris <[email-masked]>
Date: Tue, 9 Jan 2001 23:01:28 -0500 (EST)

> will deal with the dot problem.

I am not sure I am following you. Are you suggesting that the
Input Method Editor be rewritten to do the right thing? or the
domain name text widget?

> w3m, Apache, IIS, Enterprise, WebLogic, Zeus, BIND, djbdns, Webmin, etc.

Something like nameprep is already in place.
U+0061 == U+0041 aka  A == a
U+0041 LATIN CAPITAL LETTER A
U+0061 LATIN SMALL LETTER A

How do we handle the same characters at seperate code points?
U+FF21 FULLWIDTH LATIN CAPITAL LETTER A
U+FF41 FULLWIDTH LATIN SMALL LETTER A

Should code point location be preserved or mapped?
U+0061 == U+0041 == U+FF21 == U+FF41
 or
(U+FF21 -> U+0041) == (U+FF41 -> U+0061)

Then how should this be handled? 
U+30F2 KATAKANA LETTER WO
U+FF66 HALFWIDTH KATAKANA LETTER WO

> to be left alone: they can already handle UTF-8 IDNs without trouble.

My cynical view is that it is better to rewrite the applications and
protocols to support UTF-8. Until then every other language will be
second to thouse that map cleanly into ASCII. I dont see things changing
after they get implemented.

-Bill

---

From: GIM Gyeongseog-KIM Kyongsok <[email-masked]>
Date: Tue, 16 Jan 2001 16:01:31 +0900 (KST)

On Sat, 6 Jan 2001, John C Klensin wrote:

> But most Han, Kanji,
> recognition skills...

> something I can't copy from a business card.  ...

  as i understand it, IDNs in Hangeul, for example, are for
people who understand Hangeul, not for non-Hangeul-aware people.  
  if some company's main customers are people using Hangeul, 
then they can use IDN in Hangeul.  if that company also
wants to attract people using English, then they should use
English domain name too (of course, the contents 
of web pages must be in English as well).

  am i missing something here?

  KIM Kyongsok/GIM Gyeongseog, Busan National Univ.
  [email-masked], http://HANGEUL.cs.pusan.ac.kr/hangeul/
  Ph: (051) 510-2292, Fax: (051) 515-2208

---

From: John C Klensin <[email-masked]>
Date: Tue, 16 Jan 2001 03:16:26 -0500

--On Tuesday, 16 January, 2001 16:01 +0900 GIM Gyeongseog-KIM
Kyongsok <[email-masked]> wrote:

>> and Hangul are hopeless for me: there are lots of them, I
>> can't (in general) tell them apart, and I just don't have the
>> pattern recognition skills...
> 
>> Anything I can't recognize well enough to match with a table
>> is something I can't copy from a business card.  ...
> 
> people who understand Hangeul, not for non-Hangeul-aware
> people.   if some company's main customers are people using
>   Hangeul,  then they can use IDN in Hangeul.  if that company
> also wants to attract people using English, then they should
> use English domain name too (of course, the contents 

No, except that the DNS supports synchronized changes to
coordinated entries very poorly.  For both, e.g., web and email
addresses, one would normally like

   http://<hangeul-domain-name>/...   and
   http://<english-name-for-same-enterprise>/...
to point to the same site.  Likewise
   <user-name-in-hangeul>@<hangul-domain-name> and
   <user-name-in-english>@<english-name-...>
to point to the same mailbox.

But, if, e.g., one implements this as (using IPv4 as an example)
as

  <hangeul-domain-name>. IN A 10.1.2.3
  <english-name-for-same-enterprise>.  IN A 10.1.2.3

And then tries to make a change, such as to re-home the host,
the two names may, due to differences in usage, take
significantly different times to propagagate.  And, since there
is some history of taking the first matching entry only and
ignoring the others, it may make some significant difference
whether one lists

   3.2.1.10.in-addr.arpa. IN PTR <hangeul-domain-name>.
   3.2.1.10.in-addr.arpa. IN PTR <english-name-...>.

or

   3.2.1.10.in-addr.arpa. IN PTR <english-name-...>.
   3.2.1.10.in-addr.arpa. IN PTR <hangeul-domain-name>.

trying to do the matching with CNAMEs, e.g.,
  <hangeul-domain-name>. IN A 10.1.2.3
  <english-name-...>.  IN CNAME <hangeul-domain-name>.

has most of the same synchronization problems, plus all of the
usual problems with CNAMEs, including, normally, no appearance
of the English name in reverse-mapping records at all.  And the
other variations (Hangeul-> English -> address and both Hangeul
and English names labeling CNAMEs that point to some contrived
intermediary which then labels a single A RR) help with some
problems and introduce others.  And note that several of these
(as well as use of two PTR records for the target address in the
first, "both use A RRs" model) are, for me, the main argument
for ACE-like solutions (or something more radical than embedding
this information in Class=IN), which I otherwise dislike
intensely. They will certainly "leak" strings representative of
the Hangeul into legacy applications.  If those strings are in
anything but the protocol-element restricted ASCII those
applications traditionally expect, we will have trouble, whether
the problem is a leading hyphen or an octet with the high-order
bit turned on, or an octet whose bit value is zero.

Note that this gets worse if there is some coordinated
collection of servers with related domain names (fairly common
for some arrangements today), if the servers are multihomed (and
hence have multiple A RRs for a single FQDN), or if you are
supporting not only English and Hangul names, but names that use
several other languages (and hence scripts) as well.

In looking at "multilingual" names, solutions that work 80% of
the time, or cover 80% of the cases, are actually fairly easy.
Any of the example scenarios above would "pretty much" work, and
synchronized changes may not be very important.   I suppose that
is why we've had so many organizations spring up with claims of
easy and unique solutions to the requirements as they see them.
But the DNS is fairly basic infrastructure for users of the
Internet, and 80% solutions, or even 99% solutions, or solutions
that assume concurrent upgrading everywhere, are almost
certainly not good enough.

    john

---

From: John C Klensin <[email-masked]>
Date: Tue, 16 Jan 2001 03:43:53 -0500

Sorry, I left one important case out that deserves comment
(probably not the only one, but I'll leave others I've missed to
others)...

> of web pages must be in English as well).

Please also keep in mind that you may, for many years to come,
have Hanguel-using customers within sites or environments which
have not upgraded their software.  Even if the ability to view
and enter Hangeul DNS names is an application issue only, it may
not be possible for them to upgrade their applications without,
e.g., action on the part of a slow-moving ISP or IT department.
That may argue strongly that you need an "English" set of domain
names for your Hangeul-based web pages (and email servers, etc)
as well as for your English-language ones.

   john

---

From: GIM Gyeongseog-KIM Kyongsok <[email-masked]>
Date: Tue, 16 Jan 2001 17:47:08 +0900 (KST)

> > On Sat, 6 Jan 2001, John C Klensin wrote:

  thanks for your exmpales.

> coordinated entries very poorly.

  ...

> significantly different times to propagagate.  

  i am somewhat confused here.  this situation can happen
with ordinary (i.e., non-IDN) domain names.  
  i would like to know whether you claim the above problem is 
the DNS problem in general or a problem specific with IDNs?

>     john

김 경석, 부산대 정보 컴퓨터 공학부;

---

From: GIM Gyeongseog-KIM Kyongsok <[email-masked]>
Date: Tue, 16 Jan 2001 21:39:24 +0900 (KST)

On Tue, 16 Jan 2001, John C Klensin wrote:

> have not upgraded their software.

...

> as well as for your English-language ones.

agreed.

김 경석, 부산대 정보 컴퓨터 공학부;

---

From: John C Klensin <[email-masked]>
Date: Tue, 16 Jan 2001 08:07:07 -0500

--On Tuesday, January 16, 2001 5:47 PM +0900 GIM Gyeongseog-KIM

> the DNS problem in general or a problem specific with IDNs?

The widespread introduction of IDNs is likely to put considerable
multipliers on these problems.  We have learned to avoid or work
around them at relatively small scale, but they may not be
acceptable at larger scale.  As a single, trivial, example, most
web sites today that utilize many names pointing to the site
typically list only one name in the reverse-mapping.  In an IDN
context, there would be considerable (consideraby more) incentive
to try to reverse-map to both "English" and "national" names.
Note, for example, that, for routers and similar resources, IA4
names are required by an ITU recommendation that many ISPs feel
obligated to honor and that integrity of reverse mappings for
these devices is often considered quite important.

    john

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Tue, 16 Jan 2001 10:24:09 -0500

| In an IDN context, there would be considerable (consideraby more) 
| incentive to try to reverse-map to both "English" and "national" 
| names. Note, for example, that, for routers and similar resources, 
| IA4 names are required by an ITU recommendation that many ISPs feel
| obligated to honor and that integrity of reverse mappings for
| these devices is often considered quite important.

  This is a good point, and perhaps a justification for something like
IPTR (which I've initially thought of as perhaps something that was 
more of an interest to CJK speakers).  ISPs definitely rely on the
IPV4 address to name mappings to determine information about the 
network, including device identity and function and administrative
domain.  Usually there's a geographical and functional heirarchy expressed
by the name components (I know this was the case on the ANS and UUNET
backbones).  This is probably another example of functional 'overloading'
in the DNS, and perhaps a better way to do this is via SNMP.

  This would imply that as a consequence of IDN-related activities
the device management-oriented WGs may need to consider defining MIB
extensions to assist network operators in discovering needed information
about the network.  Easier said than done, I'm sure.

  -bws