Subject: [idn] UTF-8 / RACE

From: "Sherin Alsoudani" <[email-masked]>
Date: Sat, 26 May 2001 18:21:35 -0700

Dear members of IDN working group,

As an IT representative from the Middle East, I have been following the 
discussion with some interest but I have not felt qualified to comment.  I 
do have a question however after reading the emails being exchanged back and 
forth.

It appears that switching to UTF-8 will cause much existing software to 
break due to the large number of applications that are only 7-bit clean.  
This will necesitate much work to replace that software with updated 
versions that can process UTF-8.

My question is this: will RACE have a similar problem?  Even if RACE 
converts multilingal domain names to English, will we not have to update our 
applications to convert the domain names via RACE?

For example, if my email program needs to be updated to support UTF-8, does 
it not also need to be updated to handle RACE?  Just different places of the 
code - but nonetheless an update either way?

Thank you very much.  I have many other questions, but please educate me 
first by helping me understand the answer to this first one.

Sherin

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 27 May 2001 09:54:40 +0200

--On 01-05-26 18.21 -0700 Sherin Alsoudani <[email-masked]>
wrote:

> converts multilingal domain names to English, will we not have to update
> our  applications to convert the domain names via RACE?
> 
> For example, if my email program needs to be updated to support UTF-8,
> does  it not also need to be updated to handle RACE?  Just different
> places of the  code - but nonetheless an update either way?

See separate message sent by me.

You have two different problems here:

(1) If we start using UTF-8, the octets which are sent over the wire will
in the cases the data passes some software which is not upgraded destroy
the data. This is why we need to upgrade software for UTF-8. The software
can be the email client you use, the SMTP server, the DNS forwarding
server, the HTTP proxy, the firewall application, or just whatever between
the service you use and yourself.

(2) If you use an ACE approach (RACE will most certainly NOT be the
encoding used), the data will be possible to be sent from A to B, just
because it is encoded "in english" or rather, "with characters/octets which
we already use today". You need to upgrade your email client, your web
broswer and other client applications to be able to see the orginal
characters in the script it was written in by the sender of the data.

Difference is that in UTF-8 you need to replace all intermediate servers,
while with ACE, the enduser can upgrade his applications, and only that, to
be able to see IDN's. I.e. an enduser can start using IDN's even though his
systems manager have not upgraded the system software on the servers (for
DNS, HTTP and SMTP etc) he use.

Two different reasons for upgrading, and in my view, going with ACE gives
the enduser a choice, and that is far more and better than not giving him a
choice. Further, I claim that created the success MIME ended up being for
non-ascii characters.

    paf

---

From: "D. J. Bernstein" <[email-masked]>
Date: 27 May 2001 16:59:03 -0000

Patrik writes:
  [ about 8-bit bytes ]
> software which is not upgraded destroy the data

False. For example, qmail and tinydns handle 8-bit data without trouble.
In message <[email-masked]> I gave a complete
example of setting up a UTF-8 IDN and sending mail to it, with no
software upgrades. The UTF-8 IDN was displayed properly by xterm.

In contrast, ACE guarantees that the data is unreadable for users of
today's non-ACE software. When you claim that an ACE IDN is transmitted
perfectly, you are ignoring the final step: getting the IDN into the
user's head!

Yes, there are a few pieces of software that have to be upgraded and
redeployed to make UTF-8 IDNs work. But there are many more pieces of
software that have to be upgraded and redeployed to make ACE IDNs work.
See http://cr.yp.to/proto/idn.html for details.

---Dan

---

From: Dave Crocker <[email-masked]>
Date: Sun, 27 May 2001 10:20:43 -0700

>False. For example, qmail and tinydns handle 8-bit data without trouble.

What a shame that giving an example of some software that handles things 
one way, says nothing at all about other software, which can of course 
behave entirely differently.

d/

----------
Dave Crocker   <mailto:[email-masked]>
Brandenburg InternetWorking   <http://www.brandenburg.com>
tel: +1.408.246.8253;   fax: +1.408.273.6464

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 27 May 2001 11:48:53 -0700

Patrik Fältström wrote:

> Difference is that in UTF-8 you need to replace all intermediate
> servers, while with ACE, the enduser can upgrade his applications,
> and only that, to be able to see IDN's. I.e. an enduser can start
> using IDN's even though his systems manager have not upgraded the
> system software on the servers (for DNS, HTTP and SMTP etc) he use.

Let's keep this in perspective. With either ACE or UTF8, the end-user
applications have to be upgraded before the targeted functionality (local
representation of IDNs) is fully working. We are already talking about a
forklift upgrade of the entire Internet before email addresses, URLs and
other IDN occurances work right. In that context, upgrading the remaining
boxes is not exactly a show-stopper.

Also, if upgrading the servers means that the admin gets native language
support across their entire network instead of only getting it at the
end-user presentation time, I feel certain they would choose that
solution. Given the choice between encoded IDNs which were even more
user-hostile than ASCII domain names, or direct native language support
for slightly more cost, which would you choose?

Moreover, with ACE-only, they do not get this choice. They *cannot* get
native language support across all of their applications. In fact, asking
end-users to upgrade once for ACE and upgrade again later for UTF8 is
essentially asking for two forklift upgrades, which is the most expensive
solution of all.

So options and cost-of-deployments are:

   do nothing:     low-cost     low-functionality
   ACE only:       high-cost    low-functionality
   UTF8/ACE:       higher-cost  high-functionality
   ACE then UTF8:  2X-cost      high-functionality

Furthermore, given the overlap in functionality and deployment costs, I
think that it would be easiest on the development community to do UTF-8
and ACE at the same time, since they have the same basic core requirements
WRT nameprep and such. It would be much more expensive to do one upgrade
for ACE and then come back and repeat the development project for UTF-8,
since it would be two complete development cycles.

So cost-of-deployment for the user is not significantly greater to do
UTF8/ACE, but cost-of-development for the industry *IS* higher to do ACE
and UTF8 on separate development tracks.

One other point: Going to UTF8/ACE means that we can upgrade applications
as/when the WGs want to do it. ACE-only means the WGs cannot use rich IDNs
even if they wanted to.

I will also point out that there is one potential downside to doing ACE
and UTF8 together, and that is the issue of managing duplicate encodings
within the namespace (the ACE and raw domain names both have to be
provided at different times, so they both have to be maintained). That may
be resolvable through protocol, however, although I did want to bring it
up as a potential issue.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 27 May 2001 21:04:29 +0200

> perfectly, you are ignoring the final step: getting the IDN into the

The point of my message was to show that we talk about two different
problems, two different kind of errors.

Not to continue to argue about whether the whole world is using the
software you list as "ok" or not.

   paf

---

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 28 May 2001 03:35:48 +0800

Hi Dan,

Let me take a stab at some of the comments you have at
http://cr.yp.to/proto/idn.html
Making it work. apache needs to be upgraded. Domain names in
configuration files need to be converted from possibly bad to good, and
from UTF-8 to 7-bit. 7-bit domain names need to be converted to UTF-8
for logs.

Question: Why would apache needs to be upgraded if ACE+Nameprep been
used? The administrator should already have the ACE for his domain and
that could goes directly into the config file. [1]

fetchmail needs to be upgraded. Domain names embedded in POP usernames
need to be converted from possibly bad to good, and from UTF-8 to 7-bit.

Question: Why is this so? See reason [1].

gethostbyname needs to be upgraded. The input domain name needs to be
converted from possibly bad to good, and from UTF-8 to 7-bit, before it
is sent as a DNS query. The output domain names in h_name and h_aliases
need to be converted from 7-bit to UTF-8.

Question: Why would gethostbyname needs to be upgrade? The specification
in IDNA is quite clear that it does not do anything to gethostbyname.
Thus, if you *do* change gethostbyname, you are violating IDNA, no?

lynx needs to be upgraded. Domain names in configuration files need to
be converted from possibly bad to good, and from UTF-8 to 7-bit. 7-bit
domain names need to be converted to UTF-8 for internationalized URL
displays.

Question: Why is this so? See reason [1].

mutt needs to be upgraded. 7-bit domain names need to be converted to
UTF-8 when messages are displayed. UTF-8 needs to be spaced properly.
Addresses in configuration files need to be converted from possibly bad
to good, and from UTF-8 to 7-bit.

Agree.

named needs to be upgraded. Domain names in configuration files such as
/etc/named.boot need to be converted from possibly bad to good, and from
UTF-8 to 7-bit. 7-bit domain names in queries need to be converted to
UTF-8 for logs.

Question: Why is this so? See reason [1].

netscape needs to be upgraded. 7-bit domain names need to be displayed
as Unicode glyphs. Domain names in configuration files need to be
converted from possibly bad to good, and from UTF-8 to 7-bit.

Agree. That is what IDNA is suppose to do. But in realty, actually it is
much more complex since browser actually works with URL at the UI level,
not just domain names.

pine needs to be upgraded. 7-bit domain names need to be converted to
UTF-8 when messages are displayed. UTF-8 needs to be spaced properly.
Addresses in configuration files need to be converted from possibly bad
to good, and from UTF-8 to 7-bit.

Agree. That is what you mean by IDNA.

publicfile needs to be upgraded. Domain names in the /public/file
directory need to be converted from possibly bad to good, and from UTF-8
to 7-bit. 7-bit domain names need to be converted to UTF-8 for logs.

Question: Why is this so? See reason [1].

qmail-send needs to be upgraded. Domain names in configuration files
such as /var/qmail/control/virtualdomains need to be converted from
possibly bad to good, and from UTF-8 to 7-bit. 7-bit domain names need
to be converted to UTF-8 for logs.

Question: Why is this so? See reason [1].

sendmail needs to be upgraded. Domain names in configuration files such
as /etc/sendmail.cw need to be converted from possibly bad to good, and
from UTF-8 to 7-bit. 7-bit domain names need to be converted to UTF-8
for logs.

Question: Why is this so? See reason [1].

sshd needs to be upgraded. Domain names in configuration files such as
/etc/shosts.equiv need to be converted from possibly bad to good, and
from UTF-8 to 7-bit. 7-bit domain names need to be converted to UTF-8
for logs.

Question: Why is this so? See reason [1].

tcpclient needs to be upgraded. The input domain name needs to be
converted from possibly bad to good, and from UTF-8 to 7-bit, before it
is sent as a DNS query.

Question: Why is this so? See reason [1].

tcprules needs to be upgraded. Domain names in configuration files need
to be converted from possibly bad to good, and from UTF-8 to 7-bit.
7-bit domain names need to be converted to UTF-8 for logs.

Question: Why is this so? See reason [1].

tinydns needs to be upgraded. Domain names in configuration files such
as /service/tinydns/root/data need to be converted from possibly bad to
good, and from UTF-8 to 7-bit. 7-bit domain names in queries need to be
converted to UTF-8 for logs.

Question: Why is this so? See reason [1].

w3m needs to be upgraded. Domain names in configuration files need to be
converted from possibly bad to good, and from UTF-8 to 7-bit. 7-bit
domain names need to be converted to UTF-8 for internationalized URL
displays.

Question: Why is this so? See reason [1].

-James Seng

---

From: "D. J. Bernstein" <[email-masked]>
Date: 27 May 2001 20:26:13 -0000

My web page carefully specifies details of the ACE solution: where ACE
is used, where it is not used, and what has to be changed to make that
work. The reason that gethostbyname() has to be upgraded, for example,
is that ACE is not used in ``command lines for telnet, ssh, etc.'' but
ACE is used in ``DNS queries and responses.''

My web page then carefully specifies details of the UTF-8 solution. See
http://cr.yp.to/proto/idn.html.

I'm willing to analyze the costs of other IDN solutions. But I'm not
interested in proposals that don't actually provide internationalized
domain names. It's idiotic to claim that we have Greek domain names if
Greek users aren't seeing Greek alphas and betas on the screen. (Does
anyone dispute this?)

James Seng/Personal writes:
> The administrator should already have the ACE for his domain and
> that could goes directly into the config file.

No. That would not be a working IDN solution. A Greek administrator
would see unreadable ACE gobbledygook instead of a readable Greek alpha.

---Dan

---

From: "Adam M. Costello" <[email-masked]>
Date: Sun, 27 May 2001 21:30:52 +0000

Some people seem to be arguing that using ACE requires no less (or even
more) upgrading of software than using UTF-8 without ACE.  While it may
be true that ACE-fully-working-everywhere requires as much upgrading as
UTF-8-working-fully-everywhere, that comparison overlooks an important
point.

ACE affords incremental deployment much better than no-ACE.  Suppose I
am considering getting an IDN for my domain.  With ACE, this will make
things better for some users (who have upgraded their clients to decode
the ACE) and worse for others (who have old clients and will see ugly
ACEs) but nothing will actually break (mail will get through, web pages
will load, etc).

But without ACE, if I get an IDN for my domain, this will make things
better for some users (who have upgraded their clients, or who are lucky
enough to already be using UTF-8 clients) and will *completely* *break*
things for other users (mail will not get through, web pages will not
load, etc).  There may be nothing those users can do to fix it, because
the breakage might be happening in their provider's software.  The
provider might be very slow to upgrade, because 99% of their customers
might be English speakers, and the other 1% are just screwed.

With ACE, people might have to put visible ACEs in some config
files, which is annoying, but at least it will work.  Eventually
the application might get upgraded to support native characters in
the config files and then things will get easier.  Without ACE, the
application will simply be unusable with IDNs until it is upgraded.

Conclusion: There should be ACE.

Next question:  Given that there will be ACE, should DNS support 8-bit
queries in addition to ACE queries?  I don't know.  No matter what
is recommended or discouraged, some DNS servers will probably try to
guess the encoding of 8-bit queries.  This will help old clients, but
increases the risk of spoofing, and allows some applications to be
lazier about upgrading.  I haven't yet formed an opinion on this.

Next question:  Should ACE ever be phased out?  I don't think so.  Very
few systems will ever support all Unicode characters, so applications
will sometimes try to display IDNs containing unsupported characters.
What should they display?  They should display the ACE, because that's
no uglier than anything else they might display, and has the added bonus
that you can copy it into any other application and it will still work.

Furthmore, domain names are first and foremost *global* identifiers
intended to be used by *humans* *anywhere* to refer to network objects.
For people who know the Unicode characters in the domain name, the
native representation is easiest, but for the billions of people who
don't know those characters, the ACE is much easier to type, write,
speak, and visually compare.  (The Roman alphabet is one of the smallest
alphabets in existence, and is already widely recognized.  If you have
to chose a fallback character set for everyone to learn, that's the best
choice.)

I'm not saying that various protocols shouldn't allow 8-bit encoding in
addition to ACE (I have no opinion on that yet), I'm just saying that
ACE will always serve a useful function, and should never be deprecated.

AMC

---

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 28 May 2001 05:29:21 +0800

Lets go on this track a bit further since you touch the topic of "seeing
on the screen".

On an OS/Application, we can consider an I18N UI to have three layer,
altho it is not obvious sometimes:

Layer 1: Rendering of Glyphs/Fonts
Layer 2: Application Encoding
Layer 3: Machine Encoding

Layer 3 is what the OS use internally for representation of codepoints.
Some OS uses Unicode (UTF-16), very few uses UTF-8, and a lot of legacy
one uses  their own native encodings. In Windows example, this would be
either UTF-16 (>Win98) or native encodings (<Win98SE).

Layer 2 is what the application uses internally. Similarly at OS,
different apps choose different encodings. For example, Microsoft IE
uses native encodings (vary occuring to locale OS) at its interface
layer but you can switch it to support mulitple encodings including
UTF-8.

Layer 1 is where Layer 2 encodings are rendered into proper characters
so it can be display. This is by far the most troublesome since we have
to assumed the users have the right fonts and rendering engine to
display it.

Layer 1 cause the problem of 'seeing on the screen'. Supposing I were to
attach some chinese characters in this email using UTF-8 but you are not
equip with the chinese fonts, all you end up seeing is some gibberish.
This is a problem which cannot be solve using UTF-8 (Layer 2/3), but
rather an rendering and font problem (Layer 1).

AFAICS, some 8-bit gibberish is no different some seeing some ACE
gibberish if I am not ready to handle your scripts. A gibberish is a
gibberish if I cant see it even if it is encoded in UTF-8. Hence, the
end-user have to upgrade/update the client, in some ways.

Thus, "seeing on the screen" is probably not so a simple problem of
'using UTF-8 and we are done'.

Aside, I am a bit confuse by your defination of "UTF-8 with Fast
Nameprep". When you mean "Fast Nameprep", do you mean we don't do
Nameprep at all? I would be glad to give comments on the second part of
your website once this is clear up.

-James Seng

---

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 28 May 2001 05:51:08 +0800

> ACE is used in ``DNS queries and responses.''

If we were to consider telnet/ssh etc as "Applications" and then look at
IDNA:

*If* we adopt IDNA, then telnet/ssh etc have to be upgraded to be IDNA
compatible. So we will end up with IDNA-enabled telnet/ssh or
non-IDNA-enabled telnet/ssh. Sad but true. On the bright-side, only
users would needs IDNA-enabled telnet would need to upgrade their
telnet.

Upgrading gethostbyname() is not in-line with IDNA. In fact, by
upgrading gethostbyname(), you are doing IDNRA not IDNA. Correct me if I
am wrong, but IDNRA has already been given up by the authors.

Therefore, I do not think gethostbyname() should be upgraded (again, on
the assumption that IDNA is been adopted).

-James Seng

---

From: "Adam M. Costello" <[email-masked]>
Date: Sun, 27 May 2001 22:05:31 +0000

James Seng/Personal <[email-masked]> wrote:

> compatible.

Just to be perfectly clear:  If we adopt IDNA, then telnet/ssh etc.
have to be upgraded to be IDNA *friendly*, but they are already IDNA
*compatible*.  While people are waiting for those upgrades, they are not
completely out of luck.  They can do something like:

telnet `ace NATIVECHARACTERS`

where ace is a single program that can be used in this way with telnet,
ssh, ftp, finger, etc.

AMC

---

From: "Sherin Alsoudani" <[email-masked]>
Date: Sun, 27 May 2001 15:25:08 -0700

Dear Adam,

I have some very strong reservations about some of the points you have 
raised.  Though I do not consider myself a DNS expert by any means, I 
believe my experience as a software developer gives me enough insight to 
hopefully understand most of what you wrote.

My comments are interspersed below.

>From: "Adam M. Costello" <[email-masked]>
>To: [email-masked]
>Date: Sun, 27 May 2001 21:30:52 +0000
>

This is completely wrong because it disregards the human face of the 
Internet.  This working group should not just focus on "clients", 
"applications", and "protocols".  We should keep in mind the bottom line: 
the Internet is basically a medium for communication.  If by converting to 
ACE, it makes it easier for the client/application/protocol developers, but 
makes it harder for the *average* person to get on the Internet, then I have 
strong reservations against ACE.  In my part of the world, the Middle East, 
the hugely overwhelming majority of the population is not English-fluent at 
all.  To say that "nothing will actually break" is wrong - this is only at 
the protocol etc. level.  However, at the human level, if the IDN is an ACE, 
and my application does not support the proper display of that IDN, then 
something most definitely WILL break: the human user of that application 
will simply not type in that IDN.

To me, the final result is the same: even if ACE is inherently friendlier to 
the existing infrastructure, most people will not be able to utilize IDN's 
because they will not understand the ACE encoded name whatsoever.  People 
will have to upgrade their applications.

>might be English speakers, and the other 1% are just screwed.

This is not true.  In the Middle East, there are at least two or three 
providers of IDN's currently: Walid, i-DNS, and Nativ Name.  As far as I 
understand, Native Name provides a UTF-8 solution, I do not know about the 
others.  The response has been very strong from what I know: providers ARE 
switching to UTF-8; there is a good chance that my ISP will upgrade to a 
UTF-8 compatible software assuming they have not done so already.  I believe 
it is a later version of BIND that is necessary.

I also read on this list that CNNIC has chosen UTF-8 as their current 
encoding.  This is another indication that providers ARE switching.

Another thing: assuming 99% English and 1% non-English in the Middle East is 
completely wrong.  I, and many other IT professionals here, are committed to 
bringing our people online.  Most of our people have little understanding of 
the written Arabic language, let alone English.

>Conclusion: There should be ACE.

This conclusion is premature to say the least.

>to chose a fallback character set for everyone to learn, that's the best

If you wish to try to come to Egypt or any other place in the Middle East, 
and convince people here that English is a good choice even as a "fallback 
character set for everyone to learn", I do not envy your task as you will 
not be taken seriously.  I do not understand your comment of "billions of 
people" who do not know "those characters", where those characters are the 
native language of the people.  This is so patently wrong to me that I 
believe I must have misunderstood you.  I personally am not interested in 
seeing English propagated as the default language, as a half solution for 
multilingal domains.  It is impossible to ask us to learn the Roman language 
simply to satisfy the protocol writers.  Instead we are happy to go with a 
tougher solution that will give more access to the average, non-English 
speaker ultimately.  If we have to wait a little or a lot longer for that to 
occur, that is worth it compared to a short-term solution that requires me 
to send my mother to English spelling school.

Sherin

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 27 May 2001 15:33:27 -0700

> Some people seem to be arguing that using ACE requires no less (or
> even more) upgrading of software than using UTF-8 without ACE.

Some of us are arguing the cost of UTF8-with-ACE is only slightly greater
than the cost of ACE-only.

Some of us are also arguing that the cost of ACE-now-UTF8-later is 2x the
cost of UTF8-with-ACE.

> ACE affords incremental deployment much better than no-ACE.

ACE is analogous to dipping your toe into the kiddie pool. Even if you
find out that the water is comfortable, you won't be able to thoroughly
enjoy it.

The international user community doesn't end at email and web. Think about
the end-user networks out there with admins who are struggling with
ASCII-based DHCP, lpd, and just about everything else that ACE doesn't
help with. Worse, it makes embracing IDNS more difficult for them, since
they have to go from using ASCII to a highly cryptic naming service that
is outright user-hostile. It is highly unlikely that those services will
ever be fixed until a UTF8 form is developed by us, embraced by the WG,
and coded by the vendor community.

ACE is transitional, not a destination. The cost and time of getting to
the real destination can be shortened by embracing it early.

---

From: Keith Moore <[email-masked]>
Date: Sun, 27 May 2001 21:06:45 -0400

> today's non-ACE software. 

so what?  UTF-8 guarantees that the data is unreadable for users of
today's non UTF-8 software.  and even the software that today can
display UTF-8 and accept it as input, cannot reliably compare two 
UTF-8 representations of an IDN and see if they are equivalent.

> When you claim that an ACE IDN is transmitted

with ACE, we have the case that zero existing software can display them
properly or accept them as input.  this software will have to be upgraded
for those who wish to use IDNs, but at least it will be obvious that this
software has faults that need to be corrected.  in the meantime, uses of
IDNs that do not require  input or display (say, replying to an email 
message, relaying mail from one MTA to another, or clicking on a URL 
link) will continue to work.

with UTF-8, we have the case that some existing software can display them
properly and/or accept them as input.  however, this software will not
nameprep IDNs accepted as input, and thus, for some languages, the 
software will not reliably lookup IDNs even though it will accept them as
input.  these faults will not be so obvious to the user as those with ACE.

summary: in either case software has to be upgraded.  ACE is less likely
to disrupt operations; its flaws are more likely to be noticed and therefore
more likely to be fixed, and ACE is more amenable to incremental upgrade.

> See http://cr.yp.to/proto/idn.html for details.

this is a mischaracterization, based on numerous erroneous assumptions.

Keith

---

From: Keith Moore <[email-masked]>
Date: Sun, 27 May 2001 21:14:36 -0400

> work. 

yes, but those details are chosen to favor the argument you want to make.
it's unlikely that a real ACE solution would work that way.

> The reason that gethostbyname() has to be upgraded, for example,

wrong.  gethostbyname should not be upgraded, because this would cause
harm to applications that expect ASCII domain names.   it's pure fantasy
to think that you can simply change gethostbyname and expect most
applications to start support UTF-8.  some will work, some won't,
and some of those that appear to work will work unreliably.

changing an existing interface underneath an application is a good
way to make that application fail in subtle ways.

> domain names. 

nor do you appear to be interested in analyzing the difficulties 
associated with incremental transitions of UTF-8, since you 
consistently fail to consider them.

> It's idiotic to claim that we have Greek domain names if

it's idiotic to claim to perform an analysis when you refuse to consider
significant and obvious costs and failure modes.

---

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 28 May 2001 09:52:32 +0800

Sherin,

I agree with most of your comment here on user aspect and friendliness.
It is certainly one of the consideration the Working Group need to
balance with interoperability and usefulness.

However, I am curious about one point: You made an assumption that ACE
would be unfriendly to end-users. Why is that so?

And how would a UTF-8 solution more friendly given that most of the
Arabic Internet is using Sakhr encoding or CP1256?

-James Seng

----- Original Message -----
From: "Sherin Alsoudani" <[email-masked]>
To: <[email-masked]>; <[email-masked]>
Sent: Monday, May 28, 2001 6:25 AM
Subject: Re: [idn] UTF-8 / RACE

> >Some people seem to be arguing that using ACE requires no less (or
even

---

From: Keith Moore <[email-masked]>
Date: Sun, 27 May 2001 22:00:18 -0400

> Conclusion: There should be ACE.

concur.  it's the only way to get incremental deployment of IDNs
without massive operational disruption.

> queries in addition to ACE queries? 

the value in doing so is mostly psychological.  applications that are
modified to deal with UTF-8 natively will call a different routine
than gethostbyname() to look up IDNs - this routine can easily  convert 
nameprepped UTF-8 IDNs to ACE format before the query goes on the wire.
similarly, gethostbyaddr() equivalents can convert ACE results from
a DNS query to UTF- results.  and the conversion routines will also
be available to those applications that need them.

still, I don't see any harm in having a native UTF-8 DNS query
interface if we can achieve the following:

- delegation paths for UTF-8 are the same as for ACE
  (queries for equivalent names go to the same servers)
- results of a query are consistent between the two encoding schemes
  (a query of one type made to a server produces equivalent results
  - modulo encoding - as a query of the other type)
- success/failure semantics are the same for the two
  (if a query using one encoding succeeds or fails, the same 
  should be true for the same query using the other encoding)
- delay characteristics are similar for the two

however, that might be tall order, given (a) the need to upgrade
DNS servers to support UTF-8 comparisons (between query strings
and zone files where the names aren't nameprepped by the editor),
and (b) problems with sending EDNS queries through old DNS servers.
even if it works, it could take some time to work out the details.

> Next question:  Should ACE ever be phased out?  I don't think so. 

nor do I.  consider that if ACE is used in email, folks will have stored
email dating back for many years.  if ACE were phased out it could 
needlessly make those email addresses invalid.

never is a long time.  but I don't think we can reliably predict when 
ACE will no longer be needed.  we might discover many years from now 
that this is the case, but we should not try to establish a deadline.

Keith

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 28 May 2001 02:15:58 +0000

This message contains two replies, one to Eric Hall and one to Sherin
Alsoudani.

"Eric A. Hall" <[email-masked]> wrote:

> Some of us are arguing the cost of UTF8-with-ACE is only slightly
> greater than the cost of ACE-only.

I don't claim otherwise.

> Some of us are also arguing that the cost of ACE-now-UTF8-later is 2x
> the cost of UTF8-with-ACE.

I don't claim otherwise.  I made no argument about ACE-only versus
ACE-with-UTF8.  I merely argued that ACE is good and should exist.

> Think about the end-user networks out there with admins who are
> struggling with ASCII-based DHCP, lpd, and just about everything else
> that ACE doesn't help with.

ACE helps with all of them by allowing administrators to put IDNs in the
config files now, before the software has been upgraded, if they have an
urgent need.  There is nothing stopping any software from adding support
for UTF-8 (or any other encoding) in their config files, and I doubt
anyone here would want to discourage that.

As for how those protocols represent domain names on the wire, that's
another question, I don't know, but I think that has less of an impact
on the admins.

> It is highly unlikely that those services will ever be fixed until a
> UTF8 form is developed by us...

As soon as we have fully specified the ACE form, the UTF-8 form is
implicitly defined.

Sherin Alsoudani <[email-masked]> wrote:

> To say that "nothing will actually break" is wrong - this is only at
> the protocol etc. level.  However, at the human level, if the IDN is
> an ACE, and my application does not support the proper display of that
> IDN, then something most definitely WILL break: the human user of that
> application will simply not type in that IDN.

Some (maybe most) people will not type in the ACE, and that can be
considered broken.  But someone who really needs to access that domain
name before their application has been upgraded has the option of typing
in the ACE.  And besides, typing domain names is not the only way to use
them.  Replying to mail will work, clicking on hyperlinks will work,
checking SSL certificates will work, etc.  Without ACE a lot of those
operations would break for a lot of users, but with ACE they completely
work for all users.

> People will have to upgrade their applications.

Yes, with or without ACE, people will have to upgrade their
applications.  But with ACE, upgrading their own applications is
sufficient.  Without ACE, they need to not only upgrade their
applications, but also wait for other people to upgrade the
infrastructure, which is out of their control.

> > The provider might be very slow to upgrade, because 99% of their
> > customers might be English speakers, and the other 1% are just
> > screwed.
>
> This is not true...  The response has been very strong from what I
> know: providers ARE switching to UTF-8...

I didn't claim that all providers would be slow to upgrade, I'm saying
that some providers might be slow to upgrade, and without ACE, users
will not be in control, they'll need to wait for the infrastructure to
catch up with them.

> Another thing: assuming 99% English and 1% non-English in the Middle
> East is completely wrong.

I said nothing about the Middle-East.  What about Arabic speakers in the
rest of the world, where they are a small minority?  I bet they'd like
to access the Arabic domain names when they appear.  What about people
who don't even speak Arabic but like to listen to Arabic music?  The
internet is supposed to be for everyone; it's supposed to bring people
together, not put up walls between them.  Without ACE, a domain that
uses an IDN would basically be saying "foreigners keep out".

> I do not understand your comment of "billions of people" who do
> not know "those characters", where those characters are the native
> language of the people.  This is so patently wrong to me that I
> believe I must have misunderstood you.

I believe you must have.  There are billions of people in the world who
do not know Arabic characters, and whose computers do not support Arabic
characters, but sometimes those people will want to use an Arabic domain
name.  Maybe to hear some music, maybe to send a reply to all recipients
of an email message, one of whom is bilingual and has an Arabic address.

> I personally am not interested in seeing English propagated as the
> default language, as a half solution for multilingal domains.  It is
> impossible to ask us to learn the Roman language simply to satisfy the
> protocol writers.

I am not asking anyone to learn any language.  I am suggesting that
occasionally an Arabic speaker will want to access a Chinese domain,
and before they get around to learning several thousand Han characters,
and before they get around to upgrading their computer to support those
characters, they might find it more convenient to learn 52 Roman letters
(38 if you consider upper/lowercase pairs that look the same) and use
the ACE.

> If we have to wait a little or a lot longer for that to occur, that is
> worth it compared to a short-term solution that requires me to send my
> mother to English spelling school.

I'm not asking your mother to use Roman letters to access Arabic
domains.  You should upgrade her applications so she doesn't see the ACE
when accessing Arabic domains.

AMC

---

From: Keith Moore <[email-masked]>
Date: Sun, 27 May 2001 22:27:32 -0400

> than the cost of ACE-only.

this is probably true.  but the biggest part of the 'cost' in adding
UTF8 support to an ACE solution is probably not the implementation 
effort but the additional delay required to figure out how to make
the two work reliably and consistently together given the limitations
of presently-deployed DNS servers.

if getting the UTF8 part right would delay the adoption of a standard
ACE enough that there were significant deployment of nonstandard
IDN solutions, this might make it far more difficult to ever get a
standard solution (be it ACE, UTf-8 or whatever) in place.

this is why I favor moving forward with an ACE solution now.

> Some of us are also arguing that the cost of ACE-now-UTF8-later is 2x the

I think this presumes that UTF8-with-ACE can be deployed in the same 
timeframe as ACE-now-UTF8-later, and I don't think that's the case.
My guess is six months to a year additional standards work just to figure 
out how to make the the ACE and UTF-8 queries work consistently with
one another.

> find out that the water is comfortable, you won't be able to thoroughly

someone recently told me to be careful when using analogies, because it 
tempts people to argue about whether then analogies are valid rather than
about the matter at hand.

I don't think it's fair to characterize ACE as a halfway solution.
Most applications that get upgraded to use ACE will get every bit as much 
of a facelift as they would if they were upgraded to use UTF-8.    The only
difference is that they will encode IDNs using ACE "on the wire" when
exchanging IDNs with applications that might not have been upgraded.

The real issue is not the difficulty in upgrading individual applications, 
but the difficulty in upgrading all implementations of a given application.   
And the latter is far easier if you can upgrade each component incrementally 
without disrupting service.

> help with. 

I think that's a mischaracterization.   The format that is used for 
on-the-wire DNS queries is almost completely irrelevant to applications -
since the vast majority of applications will be using standard APIs
which convert between native character set and encoding and those
used on the wire.  For this purpose, the application doesn't care
whether the format is UTF-8 or ACE.  But the applications that currently
use DNS names are expecting ASCII names, ACE does help ease 
the transition with these, and these applications do include DHCP and lpd 
and anything else that uses DNS names. 

> Worse, it makes embracing IDNS more difficult for them, since

think of it another way - there's no way that the IDN WG can "fix" these
services, because each service has its own transition issues.  each 
protocol must be "fixed" by people who understand that protocol.  the
IDN WG cannot simply say "all protocols will use UTF-8 IDNs".  a statement
like that would be tantamount to saying that we cannot move to UTF-8 
IDNs until we understand how to fix every major application - and that 
would take decades.

> the real destination can be shortened by embracing it early.

The Internet itself is transitional.  It continues to evolve, and in the
past we have not demonstrated a good ability to predict how it will evolve.
However we do have some experience that indicates that solutions which
require massive infrastructure upgrades are far less likely to succeed
than solutions which were designed to allow incremental upgrading.

Keith

---

From: Keith Moore <[email-masked]>
Date: Sun, 27 May 2001 22:39:25 -0400

> Internet. 

I don't see how you get that idea.  Both ACE and UTF-8 solutions would
accept and display IDNs in native form for upgraded applciations.
Human users of upgraded applications would see no difference in the
input or display of IDNs.  

The only difference that they would see is that:

- the ACE solution gets deployed more quickly
- the ACE solution is more reliable in the near term
- the ACE solution works better with interoperating with non-upgraded clients

you are correct that the Internet is a medium for human communications.
but humans need for those communications to be reliable, and going directly
to UTF-8 disrupts that reliability.

here is one way of comparing the two:

- if we use ACE, human users will get annoyed when those ACE names are
  displayed by non-upgraded tools, and when they leak
  (as for example when an email header address is copied into the message 
  text)

- if we use UTF-8, human users will get annoyed when those UTF-8 names
  are displayed by tools that don't support UTF-8, and when they
  leak to environments that don't display UTF-8, and when they do not
  work as reliably as before (because the UTF-8 is causing sutble
  bugs in software that was written to expect ASCII), and/or they cannot
  interoperate with tools that are not upgraded.

people will complain no matter which path we take.  but the former approach
works better and produces fewer problems.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 28 May 2001 03:25:18 -0000

It's very helpful to have these reports from countries that don't use
Latin-based character sets. The users would benefit tremendously from a
working IDN solution; I hope the IDN WG doesn't ignore their needs.

Sherin Alsoudani writes:
> The response has been very strong from what I know: providers ARE 
> switching to UTF-8; there is a good chance that my ISP will upgrade to
> a UTF-8 compatible software assuming they have not done so already.  I
> believe it is a later version of BIND that is necessary.

Could you ask your ISP for more details about what changes it is making?

The only BIND problem with UTF-8 is the gethostbyname() library, which
rejects 8-bit characters in violation of RFC 2181. Current versions of
BIND support no-check-names in /etc/resolv.conf to fix this behavior.

---Dan

---

From: [email-masked]
Date: Mon, 28 May 2001 13:55:44 +1000

> rejects 8-bit characters in violation of RFC 2181.

	gethostbyname() is a client of the DNS and as such is not
	covered by RFC 2181.  gethostbyname() is part of a
	implementation of RFC 952.

	Yes we are talking here about extending RFC 952, however
	that does not imply that it is necessarily safe to extend
	gethostbyname().  The safest thing to do is to specify a
	new API which is defined to accept IHN and return IHN's.
	That way you don't break all the legacy applications when
	you upgrade your shared library.

	If you can extend a existing API, well and good, but it should
	require some action on the part of the application developer to
	get at the extended parts.

> Current versions of BIND support no-check-names in /etc/resolv.conf to
> fix this behavior.
	
	And when you do this you still get a ASCII string back.  Most
	applications don't look at it but it is there.

	Mark
--
Mark Andrews, Nominum Inc.
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: "D. J. Bernstein" <[email-masked]>
Date: 28 May 2001 04:36:39 -0000

> telnet `ace NATIVECHARACTERS`

Even if that were a complete solution---what do you do about telnet's
open command? or its status display? or the ssh password prompt? etc.---
it would have no advantages over simply fixing gethostbyname().

When I was working out the details of the ACE solution, I was careful to
minimize the number of pieces of code that would have to be upgraded and
redeployed, subject to the constraint of names being encoded as ACE over
the wire. Using ACE on the telnet command line and the ssh command line
and so on would be, to put it bluntly, bad programming.

> with or without ACE, people will have to upgrade their applications

Except, of course, for all the programs that _already_ work with UTF-8.
See http://cr.yp.to/proto/idn.html.

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 28 May 2001 05:09:43 -0000

Keith Moore writes:
> gethostbyname should not be upgraded, because this would cause
> harm to applications that expect ASCII domain names.

Sorry, Keith, but the real world doesn't care about your delusions. The
current gethostbyname() release already works with UTF-8 domain names.

> UTF-8 is causing sutble bugs in software that was written to expect ASCII

Obviously those programs should be fixed as soon as possible. I've
already proposed that the WG publish a document requiring this. The only
known example is sendmail, and the fix in that case is straightforward.

> each protocol must be "fixed" by people who understand that protocol.

You're absolutely right, Keith. This WG can't handle a job as tricky as
changing ``sequence of bytes 1-127'' to ``sequence of bytes 1-255.''
Those Internet text protocols are amazingly subtle.

> UTF-8 guarantees that the data is unreadable for users of
> today's non UTF-8 software.

Programs that don't support UTF-8 are being fixed. We have, for example,
terminals and editors that work with UTF-8 text. UTF-8 is much bigger
than IDN; it is going to be available everywhere no matter what this WG
does. See http://www.cl.cam.ac.uk/~mgk25/unicode.html.

In contrast, ACE is not a general-purpose text format. Nobody is writing
an ACE version of xterm or vi. There is no movement to convert all
textual data to ACE; the idea is absurd. ACE is never going to be
readable the way that UTF-8 is.

> reliably compare two UTF-8 representations of an IDN

There is only one good (``canonical'') UTF-8 representation of an IDN.
With fast nameprep, the keyboard interface helps the user type this good
name, so applications don't have to worry about bad names.

This is just like the current handling of dots. Yes, there are bad dots,
but the keyboard interface helps the user type domain names with the
ASCII dot, so applications don't have to worry about bad dots. Are you
going to demand that we change and redeploy thousands of programs to
accept non-canonical dots in domain names?

---Dan

---

From: Keith Moore <[email-masked]>
Date: Mon, 28 May 2001 01:38:05 -0400

> Except, of course, for all the programs that _already_ work with UTF-8.

which is the null set.

Keith

---

From: Keith Moore <[email-masked]>
Date: Mon, 28 May 2001 01:56:09 -0400

> Sorry, Keith, but the real world doesn't care about your delusions. 

And it regards your delusions as the irrefutable truth?

I'm not interested in whose delusions the world cares about, I'm
interested in what works well.

> The current gethostbyname() release already works with UTF-8 domain names.

delusion #1.

In the current DNS world, neither the client nor the server implements 
any kind of name canonicalization before comparing two names.     thus
there is no way for any current implement gethostbyname () to "work"
for any useful definition of "work".

> Obviously those programs should be fixed as soon as possible. 

obviously.  and everyone should upgrade all of their software immediately,
regardless of whether the people involved are actually likely to use
IDNs, because Dan said so.   (delusion #2 is that this is going to happen)

> I've already proposed that the WG publish a document requiring this. 

as if that were in this WG's purview (delusion #3)

> The only known example is sendmail, 

(delusion #4)

and the fix in that case is straightforward.

(delusion #5)

> changing ``sequence of bytes 1-127'' to ``sequence of bytes 1-255.''

(delusion #6 is that the change is as simple as making the tool 8-bit
transparent.)

> Programs that don't support UTF-8 are being fixed. 

delusion #7.  yes, more and more programs support UTF-8, but that doesn't
support a generalization that "programs...are being fixed".  the programs
that are being upgraded might or might not be the ones that people
want to use.    After all, people are still using VMS MAIL and ucbmail.

> We have, for example,
> does. 

wow...you actually made a defensible statement.  I was starting to worry.

yes, UTF-8 is a very useful encoding that will eventually be supported 
on all platforms.  that does not mean it will replace all other encodings,
nor that all applications will support it.

> name, so applications don't have to worry about bad names.

let's see - so you reject the approach that the IDN portions of a 
plain UTF-8 text file can be encoded differently than the rest of
the text file (as if plain text files were a dominant text format
anymore), but you somehow think that everybody's keyboard interface
should have a separate IDN input mode. 

nothing in the ACE approach suggests that IDNs in plain text
should be in ACE format.  the only constraint is that they be
nameprepped and converted to ACE format before they are passed
as protocol elements.  

> accept non-canonical dots in domain names?

no.  I'm just going to demand that we don't break existing applications
by sending them UTF-8 domains when they quite reasonably expect ASCII.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 27 May 2001 23:35:15 -0700

> - delay characteristics are similar for the two

This is doable with the following scenario:

 - UTF8 is only sent in new EDNS message.

 - ACE is only sent in legacy message.

 - Server returns answer data based on query message format.

In that model, the answers that the client gets back depends on the query
that they send. Both sets of answers are sourced from the same server so
there is no divergence. Zone data would be managed in the UTF8 form, and
ACE conversion equivalents would be generated by the servers, thereby
providing consistency at the source.

There are a few other issues here:

 - There needs to be a wide() version of gethostbyname and the
   like so that clients can make the right calls depending on
   their application support.

 - Queries have to go to the UTF8 server or its cache, which
   means that the authoritative servers for the zone have to
   be upgraded. As to the size of this as a problem, I put it
   in the same camp as IMAP-v-POP: if you want IMAP on your POP
   server, upgrade.

   Note that this does not affect queries for remote zones,
   since ignorant queries would be in legacy form, and would
   be therefore answered in ACE form. Only the authoritative
   servers for the zone need upgrading.

 - Caches will not always be completely synchronized, since an
   ACE query may arrive two hours before a UTF8 query, and the
   data may have changed in the meantime. I don't consider
   this to be a propblem since cache integrity is not assured
   currently. But it may be excessively confusing to have two
   users issue the same query and get different answers.

---

From: Keith Moore <[email-masked]>
Date: Mon, 28 May 2001 02:44:03 -0400

>  - Server returns answer data based on query message format.

that's pretty much what I imagined.  however my understanding is that many 
existing DNS caches/proxies cannot forward EDNS queries, which implies
that the ACE queries will be more reliable than the EDNS ones.

if you adopt a strategy of "try EDNS, and if that fails, try ACE"
then the EDNS queries take longer on average than the ACE ones.

it's difficult to imagine why your host library wouldn't always convert
the query to ACE format, since that would be both faster and more reliable.

>    their application support.

we would need this in any case, regardless of the format used
for IDN queries on the wire.

>    users issue the same query and get different answers.

another good point I hadn't thought of.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 27 May 2001 23:51:03 -0700

> I don't think it's fair to characterize ACE as a halfway solution.

It requires significant additional computations and presentation
processing for it to be anywhere near seamless for the user. UTF8 encoded
data does not require multiple translations. Maybe "halfway" isn't proper
descriptor for ACE (did I use it? no matter), but it's not a complete
solution either.

> whether the format is UTF-8 or ACE.

My guess is that over the next 10 years, the majority of the systems will
be either be native UTF8 or UTF32.

> > It is highly unlikely that those services will ever be fixed until
> > a UTF8 form is developed by us, embraced by the WG, and coded by
> > the vendor community.
> 
> think of it another way - there's no way that the IDN WG can "fix"
> these services

I made a typo, "WG" should have been plural, as in "the appropriate IETF
working groups for each of the protocols".

---

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 28 May 2001 15:09:54 +0800

Is this your defination for Fast Nameprep?

If that is the case, are you suggesting we change the keyboard interface
to do "Nameprep" so there is no 'bad' dot? Please explain further how
you proposed to interact with keyboard interface...

Incidently for those who cares, on a Chinese/Japanese IME, a dot can
either be U+3002 or U+002E depending if it is full/half width and in
Korean IME, a dot can either be U+FF9E or U+002E.

-James Seng

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 28 May 2001 00:12:10 -0700

> that's pretty much what I imagined.  however my understanding is that
> many existing DNS caches/proxies cannot forward EDNS queries, which
> implies that the ACE queries will be more reliable than the EDNS ones.

Mark Andrews, could you comment on this issue?

There are other concerns. Delegation can only be made to a server that is
capable of handling the zone data, for example, which may present certain
operational challenges. Clients have to be able to deal with rich data is
another (what format should URLs be in, for example?), although most of
that can be worked out in specific WGs).

---

From: Alan Barrett <[email-masked]>
Date: Mon, 28 May 2001 12:25:23 +0200 (SAST)

On Mon, 28 May 2001, James Seng/Personal wrote:
> Nameprep at all?

Dan Bernstein's "fast nameprep" means "early nameprep", where nameprep
happens once, as early as possible when the user types a domain name,
and everything outside the keyboard interface can assume that nameprep
has already been done.  Dan's "slow nameprep" means "late nameprep",
where just about anything that wants to do an IDN lookup needs to do
nameprep.

I don't believe that doing early nameprep removes the need for late
nameprep, because any number of intermediate transformations
(involving different encodings of UCS, different normalisations, or
even completely different character sets) might have happened between
the keyboard and the lookup.

--apb (Alan Barrett)

---

From: Keith Moore <[email-masked]>
Date: Mon, 28 May 2001 08:59:02 -0400

> processing for it to be anywhere near seamless for the user. 

Somehow I don't think I would characterize the additional computation
as 'significant' - it's a fairly trivial format conversion.  The
presentation processing is only a significant issue for those applications 
that don't already have to parse the input to distinguish components 
that contain domain names from those that don't.  Most appliations
that I can think off offhand already need to do this.  And even with
UTF-8, any application that accepts input of domain names would also
need to do this, becaue it would need to handle nameprep.

> UTF8 encoded data does not require multiple translations. 

That depends on whether you consider nameprep a translation.  

> Maybe "halfway" isn't proper

Again, I don't see it.  In either the ACE case or the UTF8 case you have
to separate domain names from other text for the purpose of input.
In the case of ACE you have to separate them for the purpose of presentation
also, but often you would have had to do it anyway.  So there's a chance
of needing some additional complexity on the part of ACE for the purpose
of presentation but this is more than offset by not needing the complexity
of negotiation between components before exchanging domains in the
UTF8 case.  

> be either be native UTF8 or UTF32.

I'm not sure what you mean by "systems".  Operating systems will certainly
support UTF8, but may use other encodings also.  Applications will use
a variety of encodings internally.  New applications will probably use
a combination of UTF-8 and 32 bit representations.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 28 May 2001 08:30:06 -0700

> Again, I don't see it.  In either the ACE case or the UTF8 case you
> have to separate domain names from other text for the purpose of
> input. In the case of ACE you have to separate them for the purpose
> of presentation also, but often you would have had to do it anyway.

Domain names show up all over the place. lpq listings, netstat, etc. Even
in a GUI environment where font management was simpler, not much of this
stuff would convert ACE for output purposes. Opinion.

---

From: Keith Moore <[email-masked]>
Date: Mon, 28 May 2001 11:50:32 -0400

> stuff would convert ACE for output purposes. Opinion.

I agree.  even with an ACE scheme, most of the time when a domain appeared
in text it would appear in native format (i.e. whatever character set and
encoding were used by that text format) rather than ACE.

this is fine so long as:

- things that look up domains first nameprep and convert them to ACE
- things that compare domains know how to handle domains in both 
native and ACE formats

The worst problem with ACE is that legacy software will sometimes leak the
ACE form of a domain into ordinary text.   But I don't think that this 
will happen too much with ugpraded software (which will hide ACEs from the user). 

Keith

---

From: C C Magnus Gustavsson <[email-masked]>
Date: Mon, 28 May 2001 19:45:27 +0200 (MET DST)

> that you can copy it into any other application and it will still work.

Yes, it is uglier than something else. How much uglier depends on what
ACE is chosen. In the worst case everything is garbage while most of
the characters could have been displayed correctly. In the best case
it's only unnecessary ugly because it's displaying an internal prefix
to the user and some of the characters in an uncommon way.

Furthermore, a computer can't always recognize what is a domain name
and what is not. I think it's pretty darn ugly that the same domain
e g in the header of a mail message and in the body of the same
message should be displayed in two completely different ways.

The problem of characters which can't be displayed should be solved in
exactly the same way it is today. Which is: Not standardized. Some
systems or programs can use a default character. Others, like Emacs,
could use a backslash ('\') and a coding point number. And so on.
Since the coding point is stored internally even though it can't be
displayed, there should be no problems with copying and pasting it.

ACE has it's advantages but the display problem is not a reason to
choose ACE. I think it's obvious that it should eventually be phased
out. Once again, in the long perspective we need a common encoding
like ASCII: The Unified Encoding that is used for every single piece
of text that is transmitted on the Internet.

> (The Roman alphabet is one of the smallest

Since you're not talking about the Roman alphabet here, I suggest you
use the term "English alphabet" instead. Even though the first is
based on the latter, they are not the same. Being correct doesn't hurt.

/Magnus

---

From: Keith Moore <[email-masked]>
Date: Mon, 28 May 2001 15:15:05 -0400

> message should be displayed in two completely different ways.

I agree that it's ugly.  However this is a natural result of the fact
that header information is structured according to one standard
(RFC 822, now RFC 2822), and textual message bodies are encoding
with any of a variety of character encoding standards.

Currently, message headers are specified to be ASCII (with non-ASCII
portions encoded per RFC 2047), though in reality a variety of
character sets are used without labelling.  Textual body parts may be in 
any of several different character encodings, and these are sometimes
labelled correctly.    UTF-8 is allowed as one of the encodings for 
textual message bodies , but it is not widely used.

In order to reduce this "ugliness" of having different encodings
for message header and message body it would be necessary not only
to extend message headers to allow UTF-8 but also for UTF-8 to become
a popular method of encoding textual message bodies.  The latter 
seems unlikely to happen for a great many years, because even if many
user agents support the ability to display UTF-8, the number of user 
agents that can display other common character sets will still be greater.
Thus, for many languages, it will be safer to send text in some other
character set besides UTF-8. 
 
> of text that is transmitted on the Internet.

I am not sure that we will find that Holy Grail anytime soon.
Even if we adopt UTF-8 we will still have to deal with various ways
of encoding "rich" text.

But this is all beside the point.  The IDN WG cannot legislate the 
encodings that are used by other applications; and it cannot legislate
that existing applications change their behavior.  It can only recommend
how to solve the I18N problem for domain names.  If the solution that
IDN recommends doesn't work well for some applications, it will not get
adopted for those applications - even if the recommended solution gets
approved as a standard.

It's very important to be realistic about what can be acheived.

I support the effort to craft a compromise around the idea of
ACE now - UTF-8 eventually.  But we need to realize that we cannot
mandate UTF-8 for existing applications within any particular
time frame.  

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 28 May 2001 21:18:11 -0000

> Korean IME, a dot can either be U+FF9E or U+002E.

This is part of what I was referring to. Why aren't Keith and Patrik
screaming that we need dotprep in thousands of programs? Don't they care
about millions of confused users who have typed the wrong dot?

Reports from Japan suggest that, in fact, keyboard interfaces already
have adequate support for typing domain names. Users type the good dot,
the canonical dot, the ASCII dot. A domain name with a bad dot simply
doesn't work; the user fixes it.

Maybe users will keep trying to put bad dots into domain names for some
reason. Maybe we _do_ want to put dotprep into thousands of programs.
Okay; we'll do that if and when the need for it is clear. However, in
the meantime, we can still use dots!

We can handle UTF-8 IDNs the same way:

   * People will register good UTF-8 IDNs. Lowercase, ASCII dots, no
     confusing characters. We can distribute name-checking software so
     that DNS administrators can make sure their IDNs are good.

   * It's the responsibility of the keyboard interface to help users
     type good IDNs. Bad IDNs simply won't work.

   * Maybe users will keep putting bad characters into domain names for
     some reason. Maybe we really _do_ want slow nameprep, with
     thousands of programs converting bad IDNs to good IDNs. Okay; we'll
     do that if and when the need for it is clear. However, in the
     meantime, we can still use IDNs!

This way we get useful IDNs as soon as possible. We take advantage of
all the existing UTF-8 support. If it turns out that the user interface
isn't good enough, that we need to suffer the massive costs of upgrading
and redeploying thousands of programs, then we'll do that. The existing
IDNs will continue to work.

There are two obstacles to having these IDNs work instantaneously. One
is that many versions of gethostbyname() reject 8-bit characters. The
other is that sendmail destroys bytes 128-159 (which don't show up in
lowercase European characters in UTF-8, but which do show up in other
characters). Both of these problems should be fixed as soon as possible.

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 28 May 2001 22:16:45 -0000

> new API which is defined to accept IHN and return IHN's.

Perhaps you have forgotten that RFC 952 is obsolete. RFC 1123 extended
the host name syntax twelve years ago.

The gethostbyname() maintainers could have insisted that gethostbyname()
be used only for the old syntax. They could have introduced an idiotic
new gethostbyextendedname() function. They could have forced programs to
switch from gethostbyname() to gethostbyextendedname() to support the
new syntax.

In short, they could have changed a simple syntax extension into a
compatibility disaster. But they didn't.

---Dan

---

From: C C Magnus Gustavsson <[email-masked]>
Date: Tue, 29 May 2001 00:53:33 +0200 (MET DST)

> textual message bodies , but it is not widely used.

No, it's a natural result of mr Costellos suggestion that domains with
characters that can't be displayed should be displayed as ACE. I fail
to see that your comment is relevant to what I wrote. Perhaps you
should read what I wrote and replied to one more time?

Let's say John Luser sends the following mail to you (where <oe> is o
with umlaut).

  From: John Luser <luser@f<oe>reningen.org>
  To: Keith Moore <[email-masked]>
  Subject: Hello
  Content-Type: TEXT/PLAIN; charset=ISO-8859-1
  Content-Transfer-Encoding: 8BIT
  
  Hello Keith,
  
  Why don't you take a look at our new website on f<oe>reningen.org?
  
  Have a nice day!
  
  John

Now, let's say your local system knows about ISO 8859-1 and is able to
convert it to the internal encoding but unable to display <oe>. Thus
it displays it as [] (square box - your systems default character).

With mr Costellos suggestion this would be displayed to you as:

  From: John Luser <[email-masked]>
  To: Keith Moore <[email-masked]>
  Subject: Hello
  
  Hello Keith,
  
  Why don't you take a look at our new website on f[]reningen.org?
  
  Have a nice day!
  
  John

(Or whatever the chosen ACE encoding turns it into.)

It's hard to recognize px--sdn3fnfuwy4rn5wutn.org as the same domain
as f[]reningen.org. It's a lot uglier as well.

A less ugly way of displaying it would be:

  From: John Luser <luser@f[]reningen.org>
  To: Keith Moore <[email-masked]>
  Subject: Hello
  
  Hello Keith,
  
  Why don't you take a look at our new website on f[]reningen.org?
  
  Have a nice day!
  
  John

You don't know what character [] is but at least it's displayed in the
same way. Since there's no way for your computer to know that
f[]reningen.org in the body is a domain name instead of a number, a
missed space or whatever it can't turn it into ACE.

It doesn't matter that the header domain is encoded in ACE/UTF-8 and
the body in ISO 8859-1 since your system can convert them both to it's
internal format. The set of characters available in your systems
font(s) is still the same no matter what encoding is used.

ACE is always a worse and an uglier thing to display when a system
encounters unknown characters. Thus it's hardly an argument for using
ACE. I can't agree that displaying the domain in the header as ACE,
i e as "px--f+xg-reningen.org", "px--sdn3fnfuwy4rn5wutn.org" or
whatever, is "no uglier than anything else they might display".

The same goes for everything else. Let's say we have a web page where
location is "http://px--sdn3fnfuwy4rn5wutn.org" and the text on the
page says "Welcome to f[]reningen.org!". And so on...

Do you understand what I'm saying now?

> character set besides UTF-8. 

See above. That is not the ugliness I was refering to. What I was
refering to was the suggestion that applications wanting "to display
IDNs containing unsupported characters [...] should display the ACE".

> Even if we adopt UTF-8 we will still have to deal with various ways

True, but we have to start somewhere. Every journey begins with a
single step. (Although, this journey started with BCP 18. This is just
a very large and important step on the way.)

> It's very important to be realistic about what can be acheived.

Sure, but it is hardly a reason or excuse for this WG to suggest a bad
solution. And I fail to see what it has to do with displaying domains
as ACE when the application encounters unknown characters.

/Magnus

---

From: [email-masked]
Date: Tue, 29 May 2001 11:05:43 +1000

> the host name syntax twelve years ago.

	RFC 1123 did not obsolete RFC 952, it updated it.

> switch from gethostbyname() to gethostbyextendedname() to support the

	I'm sure they thought about it.  Relaxing the requirements
	to allow a digit as the first character of a label broke
	some applications as did the extention from 63 to 255
	characters.  These were trivial changes compared to what
	is being proposed here.

	It took years for some mailers to be able to mail to
	[email-masked].

> compatibility disaster. But they didn't.

	This is not a simple syntax extension.  It a radical change.

	Mark

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 29 May 2001 03:30:41 +0000

This message contains replies to D. J. Bernstein, C C Magnus Gustavsson,
and Eric A. Hall.

"D. J. Bernstein" <[email-masked]> wrote:

> Even if that were a complete solution

It's not.  It's a workaround that people could use while they're waiting
for stuff to get upgraded.

> what do you do about telnet's open command?

Nothing.  I'd quit telnet and start it again as above.

> or its status display?

Ignore it.  This is just a workaround.  The point is that with ACE
workarounds are possible.  Without ACE, when things choke on UTF-8
names, you're just out of luck until stuff gets upgraded, and some of
that stuff might not be under your control.

> it would have no advantages over simply fixing gethostbyname().

It would have a few minor advantages:  Installing a standalone program
is easier than upgrading libc.  The upgraded gethostbyname() will work
only if my local DNS server doesn't choke on UTF-8 names, whereas the
ace program doesn't depend on anything outside my system.

But I'll admit, for telnet, the advantages are minor.  Let's try mail.
Upgrading gethostbyname() is not going to allow "mail [email-masked]
[email-masked]" to work, because the mail program still has to get
IDN.org into the header somehow.  Putting UTF-8 directly into the header
violates RFC 822 and RFC 821, and even if it somehow gets to user1
intact, it might not get to user2 intact, and even if it does, user2's
mail program might choke on it.  Without ACE, it's simply not going
to work until I upgrade my mail program, and user2 upgrades his mail
program, and all the intervening SMTP servers get upgraded.  But with
ACE, I have a workaround that doesn't depend on anyone else upgrading
anything: mail user1@`ace IDN.org` [email-masked]

> Except, of course, for all the programs that _already_ work with

Yes, the advantage of UTF-8 is that some programs already use it
natively, and they might be able to use IDNs by accident.  But there are
also many programs that don't use UTF-8.  I use a terminal emulator and
a text editor and a web browser all written to support Japanese (kterm,
nvi-m17n, w3m), and none of them supports UTF-8.  They could all benefit
immediately from the sort of ACE workarounds described above.

Remember, I'm not arguing against UTF-8; I'm merely arguing in favor of
ACE.  I'm certainly willing to consider an architecture where both are
used.

C C Magnus Gustavsson <[email-masked]> wrote:

> The problem of characters which can't be displayed should be solved
> in exactly the same way it is today. Which is: Not standardized. Some

Okay, I'll back off a little on my claim that the ACE should be
displayed when the name contains unsupported characters.  Ideally it
should be a user-configurable option.  I personally would choose the
ACE, so that I could cut and paste it.

> displayed, there should be no problems with copying and pasting it.

I agree that there "should" be no problems, but there will be.  In my
experience, copying text with the mouse usually gives you exactly what
is displayed; very rarely does the application intercept the operation,
because it's usually being handled entirely by the GUI library.

> I think it's obvious that [ACE] should eventually be phased out.

It's not obvious to me.  Someone (Keith?) mentioned archived mail.
Should those addresses arbitrarily stop working someday?

> use the term "English alphabet" instead.

<offtopic> I rarely hear the phrase "English alphabet".  When I do, I
think to myself, "There is no English alphabet.  English uses the Roman
alphabet."  The phrase "Roman alphabet" is the more common name for the
letters A-Z.  "English alphabet" sounds arrogant to me, since England
merely inherited the alphabet from the Roman empire.  The most formally
correct phrase might be "modern Roman alphabet". </offtopic>

"Eric A. Hall" <[email-masked]> wrote:

> Domain names show up all over the place. lpq listings, netstat,
> etc. Even in a GUI environment where font management was simpler, not
> much of this stuff would convert ACE for output purposes. Opinion.

Agreed.  If netstat gets back UTF-8 from gethostbyaddr() and blindly
outputs it, it will look nice if it happens to be running in a UTF-8
environment, and will produce useless garbage otherwise.  If it gets
back ACE, it will produce garbage in all environments, but potentially
useful garbage (because it can be copied into other applications).
There will be problems of one sort or another until netstat is upgraded
to call a function that returns the domain name in the local charset.

AMC

---

From: "D. J. Bernstein" <[email-masked]>
Date: 29 May 2001 04:47:34 -0000

Did the RFC 1123 authors worry about some badly written programs having
trouble with names like 3com.com? Yes, they did. Did that stop them from
extending the host name syntax? No, it didn't.

Those programs were declared buggy, and they were fixed, and now we can
use 3com.com without trouble. Programmers don't have to encode 3com.com
as ACE-GARBAGE-DIGIT-THREE-LETTER-C-LETTER-O-LETTER-M.com.

Now, do we want to follow the model of RFC 1123's easy syntax extension?
Or do we want to follow the model of RFC 1342, which is continuing to
annoy users and programmers a decade after its introduction, and which
promises to continue annoying us for decades to come?

[email-masked] writes:
> Relaxing the requirements to allow a digit as the first character of a
> label broke some applications

Extending gethostbyname() didn't break anything. In fact, extending
gethostbyname() made it vastly _easier_ for applications to support the
new syntax. Most applications worked perfectly, no effort required.

In contrast, if gethostbyname() had enforced the old syntax, and if an
idiotic new gethostbyextendedname() function had been introduced for the
new syntax, nothing would have worked. Every application would have had
to switch from gethostbyname() to gethostbyextendedname(). Ridiculous!

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 29 May 2001 04:56:56 -0000

Apparently Adam has lost track of this thread.

The question was why, in http://cr.yp.to/proto/idn.html, I listed
gethostbyname() as something to be upgraded as part of the ACE solution.
This has nothing to do with the UTF-8 solution!

The answer is that ACE conversion has to happen somewhere between the
user's command lines and the DNS packets. Changing the gethostbyname()
code is easier than changing every program that uses gethostbyname().

---Dan

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 01:06:34 -0400

> as f[]reningen.org. It's a lot uglier as well.

okay, I see your point.  I can see why a user might want either form -
the "ugly" format so that he/she could transcribe the ACE format, and
the "pretty" format so that it wouldn't be so distracting.  If I were
writing an MUA I'd probably display the pretty one by default but 
have an option to display the ugly form (just as MUAs have options to
display certain message headers that aren't displayed by default).

I don't think I'd want the protocol standards to dictate how 
non-presentable characters are displayed, except perhaps to say 
that they shouldn't mislead the recipient.  (i.e don't display
o with umlaut as o or oe and make the recipient think that it's
a different address than it really is.)

> whatever, is "no uglier than anything else they might display".

nor can I.  and the experience with RFC 2047 encodings would also 
seem to support your argument (and those were designed to be at least
somewhat readable for supersets of ASCII).  people found display of
2047 encodings annoying even when they could manage to read them.
to a lesser extent this was also true of quoted-printable in message 
bodies.
 
> Do you understand what I'm saying now?

I think so (though perhaps I should not be so presumptious as to
claim that I do for sure?)  thanks for the clarification.
 
> a very large and important step on the way.)

True.  But  (as long as we are being philosophical) just because you start 
on a journey does not mean you know where it will take you, or how you 
will get there. :)

> as ACE when the application encounters unknown characters.

I think that ACE is necessary for other reasons, but I agree that this 
particular argument does not lend much support to ACE.

Keith

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 29 May 2001 05:23:22 +0000

> gethostbyname() as something to be upgraded as part of the ACE solution.

I don't think it was me who asked that question.

> code is easier than changing every program that uses gethostbyname().

I am willing to entertain the notion that changing gethostbyname() may
be a good idea.  It has been argued that existing programs do not expect
to get non-ASCII text back from gethostbyname(), and I buy that argument
to some extent:  If a program passes an ASCII name to gethostbyname(),
then gethostbyname() should not return any non-ASCII text (not even
if the ASCII name turns out to be an alias for an IDN), unless the
program has somehow explicitly invited non-ASCII text.  But if a program
passes non-ASCII text to gethostbyname(), and is surprised to get
non-ASCII text back, it's arguably the program's fault.  Perhaps passing
non-ASCII text to gethostbyname() should be considered permission for
gethostbyname() to return non-ASCII text.

If gethostbyname() receives 8-bit text, should it assume that it's
UTF-8, or should it assume that it's in the encoding of the current
locale?  My gut feeling is the latter.  And of course gethostbyname()
should use the same encoding for both the argument and the return value.

AMC

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 29 May 2001 07:31:14 +0200

> particular argument does not lend much support to ACE.

You will get exactly the same kind of problem as soon as you have a charset
parameter for a MIME bodypart which have a different value than the
charset/encoding we will use for a domainname.

Reason for this is that the transformation in the header:

<sender local script> -> <whatever the IDN group selects> -> <receiver
local script>

and the transformation in the body:

<sender local script> -> <whatever the email client selects as charset> ->
<receiver local script>

most certainly will produce different results in receiver local script
regardless of what the IDN group selects.

This especially in clients where the client is not "IDN aware".

And, this regardless of whether UTF-8 or ACE encoding will be selected (or
anything else).

I think John has already pointed this out a number of times...

    paf

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 29 May 2001 07:47:34 +0200

> should use the same encoding for both the argument and the return value.

I start feeling like a parrot...repeating myself over and over and over
again.

The problem is NOT the DNS protocol or gethostbyname() or such, the problem
are all the applications while using some application layer protocol, and
what charset they are going to use when passing around what they belive is
a domainname.

Not all operating systems and applications (far from the majority) use
Unicode of some encoding locally, so a conversion need always to happen.
All applications have to be changed.

   paf

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 01:45:41 -0400

> about millions of confused users who have typed the wrong dot?

My guess is that this can be done in nameprep.  Perhaps it's not defined
in the current version of nameprep, but this could be fixed.  I'm not
screaming about this partially because I see us as currently arguing about 
more fundamental architectural issues, and because I have confidence 
that we will be willing to address this issue once we agree on how to
proceed at an architectural level.

That is, I don't expect that every detail of the protocol needs to be
worked out before we settle on an architecture.  

> doesn't work; the user fixes it.

As far as I'm concerned that's a perfectly reasonable solution if it works.
But there are some caveats.  For instance, it is one thing if typing the 
"bad" dots causes the lookup to fail; quite another if typing the "bad" 
dots causes a lookup for a completely different domain; and still another 
if the "bad" dots work differently in different clients.

If we don't specify how the "bad" dots are handled as part of the IDN 
protocol, we are essentially defering this to the registries.  Given their 
history and the competitive environment in which they exist, it's not clear 
whether they will strive to minimize bad effects.   At the same time, some 
of this will be left to the registies no matter what - we clearly cannot 
fix all potential IDN ambiguities in the protocol.

> Okay; we'll do that if and when the need for it is clear. 

I think the question (for this and many other issues) is - to what extent 
we can risk requiring hosts or applications or servers to be upgraded 
when we discover some failure in our design?  The flip side of this 
question is - to what extent we can get this right the first time, and 
how long should we delay the standard trying to get it right?  I think that 
nameprep goes a long way toward solving the problem, and that it's a
useful approach, but it's not quite there yet.

> However, in

if we can precisely define what is "good" beforehand (well enough to 
write the name-checking software), couldn't we just as easily define 
nameprep in such a way that it would either convert "bad" names to 
"good" names (for obvious translations) or reject "bad" names that
couldn't be reliably translated to good ones?

>      type good IDNs. Bad IDNs simply won't work.

quite honestly, I don't think that this will fly.  it strikes me as far
more difficult to get platforms to create a special input mode for
IDNs, and to get applications to use that input mode, than it does to
get applications to accept IDNs using a normal text interface.
it's especially difficult to retro-fit existing applications to use
new input modes.  and as somebody pointed out, we will also need to 
check names that originate from other sources besides the keyboard.

We could quite reasonably define a name-check API routine that would 
return good/bad for a given domain name, and that would be far easier 
than defining a new input mode.  But this could also just be
done by the nameprep routine.

>      meantime, we can still use IDNs!

this is going to be a massive upgrade.  we would like to avoid having
millions of programs upgraded only to find out that their IDN support is 
buggy and that (for instance) people can't reliably use certain IDNs 
with certain clients, and that there will be yet another massive upgrade
required to fix this.   good design is generally cheaper than good maintenance.

> IDNs will continue to work.

we already know of instances where the UTF-8 support isn't good enough.
surely you would not have us ignore what we've learned from experience 
with these prototypes?

> characters). Both of these problems should be fixed as soon as possible.

I strongly disagree with this analysis, and it seems that many others do 
also.  But we've been through this argument before, and I don't see the 
point of rehashing it.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 02:00:48 -0400

In fact, when gethostbyname() was extended to use DNS (and YP, later NIS)
rather than simply looking up names in /etc/hosts, it broke numerous applications.
This happened because gethostbyname() had no way to return temporary failure 
(e.g. DNS server unreachable) as opposed to "this domain does not exist".
This caused a lot of bounced mail.

More generally, experience suggests that any time you change a programming
interface in an incompatible way then you need to make that change explicit,
rather than to change the behavior that existing applications expect.

Perhaps for some programs the change really will be as trivial as 
s/gethostbyname/gethostbyidn/ or whatnot.  But I doubt this will be true 
in general.  For instance, given that UNIX systems deployed today use a wide 
variety of locales and character sets, if I were updating a UNIX program that 
was in widespread use I wouldn't want that program to assume that its output 
device displays UTF-8.   Even if UNIX vendors are now supporting UTF-8,
not everyone will upgrade their systems immediately, and some will continue
to use old locales.  So I'd probably want a separate routine to convert 
an IDN from internal format to the format required by the display.
It might be a NOOP for the case when the display supported UTF-8, but it
would be needed in other cases.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 02:29:52 -0400

> trouble with names like 3com.com? 

perhaps.  but they did not specify that the APIs had to stay the same,
or for that matter, that they had to change.  rather they assumed that
API writers would do something reasonable.  

also, RFC 1123 didn't change the interpretation of the octets used
in a domain name. both before and after RFC 1123 those octets were 
understood to be from the printable subset of ASCII.  There was 
no question as to whether a program printing the name obtained from 
gethostbyaddr() before RFC 1123 would be able to print a name obtained
from that routine after a post-1123 upgrade, and the ambiguities 
that crop up when using ASCII (eg 0 vs O, l vs 1, - vs _) were 
reasonably well-understood.

These questions do crop up when considering whether to use native UTF-8 
in IDNs.

Also, the 'net was *much* smaller and in many ways less diverse when 
1123 was written. It was much easier to understand the likely impact of 
such changes, and there were fewer constituencies that had to be satisfied.

Keith

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 29 May 2001 14:31:02 +0800

> about millions of confused users who have typed the wrong dot?

Actually this was brought up in the Nameprep design team.

> doesn't work; the user fixes it.

I see what you mean now. Hmm..it is a thought: to leave the
normalization problem to the users and let them sort learn what is the
right thing to type.

However, I see some problems too. Normalization of diacritics. If my
keyboard only produce decomposed diacritics but the name is in compsed
diacritics, irregardless how much I try, I wont be able to get it in.

And also, I remember Sherin bought up yesterday on a topic of
userfriendiness and I am not sure if this fits it well.

>      that DNS administrators can make sure their IDNs are good.

Agree. I think this should be done anyway, whether it is UTF-8, ACE or
whatever else we have (or decided). It is a good practice to do so.

>      type good IDNs. Bad IDNs simply won't work.

I don't agree with this or even think it is possible to be done.
Keyboard interface deals with generic input. IDN is only one specific
use for the keyboard. I don't think keyboard producer would have any
incentives to change what they doing for one specific IDN.

There are very likely good reasons someone wants to key in a full-width
dot (U+3002). Expecting the keyboard to produce only ASCII dot is not an
option. You see, that is how Chinese and Japanese write a full-stop ie a
hollow circle. Pick up a Chinese/Japanese book and take a look.

>      meantime, we can still use IDNs!

I am not sure I agree with the last statement. Experience has shown that
we can start using IDN in a just-send-utf8 method in some particular
setup with some particular software.

> IDNs will continue to work.

I could build upon this argument and suggest we all use GBK immediately.
In fact, GBK has better support than UTF-8 since it is mandated by the
Standard Burea. GBK is supported on all Chinese Windows and IE. Users
can already render/display and input GBK without problems and a lot of
applications already support GBK. If we were to use GBK on the wire now,
we can put GBK in the zone files, works with IE and Chinese users would
not have any problem input and using these GBK domain names.

I could go down this analogy in greater scary details...but lets stop
here for now.

What I am trying to say is this argument of 'immediate works' can have
devastating consequences if you are not careful. It is should not be an
argument for supporting some standard.

-James Seng

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 29 May 2001 06:50:27 +0000

Patrik Fältström <[email-masked]> wrote:

> The problem is NOT the DNS protocol or gethostbyname() or such, the
> problem are all the applications while using some application layer
> protocol, and what charset they are going to use when passing around
> what they belive is a domainname.

Oh yes, I understand this.

> All applications have to be changed.

There are some small applications that never pass domain names around
except to/from the user and to/from DNS (like ping and netstat, and
telnet I think).  For these applications, changing gethostbyname() to
support the encoding of the current locale might be enough (if they are
already 8-bit clean).

But application-layer protocols do indeed need to define a common
representation of domain names and convert between that and the local
charset.

AMC

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 29 May 2001 09:57:09 +0200

> about millions of confused users who have typed the wrong dot?

The issues with individual characters (especially the dot in Japanese and
Chinese input mechanisms) has been discussed extensively in the group which
have been talking about nameprep.

See the nameprep draft section 5.10.

  paf

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 29 May 2001 18:17:28 +0800

As I say I will try to give some comments on UTF-8 with fast nameprep
after I get an understanding of what you mean by that.
Costs of UTF-8 with fast nameprep
What it means. Domain names are encoded as UTF-8 strings in all of the
above contexts.
Bad names are not allowed to appear. (Exception: Users can send bad
names in DNS registration forms; the registrar will send back a
rejection notice showing the closest good name.)

JS: What is considered a "bad names"? From some of your note, you
suggested only lower-case characters allowed in DNS. Does this two match
"françious" vs "FRANÇIOUS"? (By your fast-nameprep definition, they wont
match unless the users key in exactly in the same way the administrator
puts it in).

Making it work. gethostbyname needs to be upgraded. Many current
installations, in violation of RFC 2181, reject DNS answers that contain
unusual characters. (However, some versions will work correctly with
options allow_special all or options no-check-names in
/etc/resolv.conf.)

JS: Agree with this altho it is probably a simplistic summary of the
pain.

mutt needs to be upgraded. UTF-8 needs to be spaced properly.

pine needs to be upgraded. UTF-8 needs to be spaced properly.

JS: Lets expand this to any non-UTF-8 applications which uses Domain
Names.

sendmail needs to be upgraded. Current versions are not 8-bit clean:
they discard bytes \200 through \237 in mail message headers, because
those bytes are used for other purposes inside sendmail's
string-handling routines. The relevant section of code is in collect.c:
...

JS: It is too early to suspect how I18N Email will look like to decide
what changes would be required in sendmail but I suspect making it 8-bit
clean as you suggested here is probably needed if UTF-8 are use in IDN.

There's one report that an obsolete version of the Netscape mailer
crashes under Solaris when it reads UTF-8 messages. I need verifiable
details.

JS: Lets go thru the list you made for IDNA and I see most of them would
need to be patch or made 8-bit clean to work (including bind).
Applications which does not support UTF-8 has to be upgraded to support
UTF-8.

But most important of all, you left out the needs to 'patch/replace' the
keyboard so that 'bad names' cannot be entered.

-James Seng

---

From: C C Magnus Gustavsson <[email-masked]>
Date: Tue, 29 May 2001 16:08:31 +0200 (MET DST)

> because it's usually being handled entirely by the GUI library.

It's not the graphics but the encoded characters that are copied. If a
characters is available in the fonts does not make a difference.

> Should those addresses arbitrarily stop working someday?

They should not be archived as ACE in the first place. If they are,
they should be converted whenever possible.

> correct phrase might be "modern Roman alphabet". </offtopic>

I'm very sorry for keeping this off topic a little while longer but I
feel that statements as wrong as this should be corrected.

No, English does not use the Roman (or Latin) alphabet. English uses
an alphabet based on the Latin alphabet but so does numerous other
languages, each with a different set of characters. The Latin alphabet
contains 14 consonants (bcdfghlmnpqrst) and 5 vowels (aeiou). (It
might be argued that it's actually 16 consonants if we choose to count
j and v as separate letters from i and u, which the Romans did not.)

It might be that the phrase "Roman alphabet" is commonly used for the 
letters A-Z in the USA, but if so it's historically incorrect since
that is not the alphabet the Romans used. England did not only inherit
it's alphabet. It inherited it's language as well (although not from
Latin) and it evolved with time. Yet we call it the English language
to separate it from e g the related French and German languages. 

If anything is arrogant, it's using the term "modern Roman alphabet"
for the English alphabet. What makes it more modern or more Roman than
any of the other alphabets based on the Latin (e g the German)?

I suppose you don't believe me so to make sure I quote from my copy of
the Oxford Latin Dictionary (page v): "The Latin alphabet contains
fifteen consonants" and "It contains five vowels".

(They count v but not j as a separate character. I quote (page iii):
"While the letter 'i' is both a vowel and a consonant and thus there
is no 'j' in Latin, it unfortunately still appears necessary to make a
distinction between 'u' as a vowel (as in English) and as a consonant,
the latter being given as 'v'. This is unhistorical and philologically
unsound and it is hoped that before too long the letter 'v' can be
eradicated from this dictionary.")

Nosce te Latinum.

Now, can we please leave this topic?

/Magnus

---

From: "D. J. Bernstein" <[email-masked]>
Date: 29 May 2001 18:59:11 -0000

Adam M. Costello writes:
> If gethostbyname() receives 8-bit text, should it assume that it's UTF-8,

The current gethostbyname() release (if you put no-check-names into
/etc/resolv.conf) simply copies the 8-bit bytes to the DNS packet. It
doesn't care about the interpretation of the bytes as characters.

If we make the obvious specification of 8-bit bytes in DNS as UTF-8, the
current gethostbyname() semantics will be consistent with two coherent
programming models:

   (1) The local character encoding might not be UTF-8. Higher-level
       routines are responsible for converting from the local character
       encoding to UTF-8 before calling gethostbyname(). It would be
       convenient to have a central routine that does this.

   (2) The local character encoding is UTF-8. No conversion is required
       in this case. This is simpler than #1, and it's wildly popular
       among programmers---dealing with multiple character encodings is
       a royal pain. UNIX is rapidly moving from #1 to #2.

There are other programming models that aren't consistent with the
current gethostbyname() semantics. This might be a sufficient reason to
berak compatibility if those models were as nice as #2, but they aren't.

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 29 May 2001 19:45:40 -0000

Keith Moore writes:
> For instance, it is one thing if typing the "bad" dots causes the
> lookup to fail; quite another if typing the "bad" dots causes a lookup
> for a completely different domain; and still another if the "bad" dots
> work differently in different clients.

I agree that having two different names, one using a good dot and one
using a bad dot, would be confusing for users. It would also mean that,
if we ever decided to move to ``bad dots are converted to good,'' we'd
break some working names. Disaster!

The current situation, however, is that the bad dot isn't used. So
neither problem happens. There's only one name. The worst case is a
failure, forcing the user to change the bad dot to a good dot.

My proposal is to do the same for IDNs. We can allow good UTF-8 IDNs,
and prohibit bad UTF-8 IDNs. Everything necessary to make this work is
something that we want to do anyway. Nothing has been lost if it turns
out that the users also need bad->good conversion.

> we would like to avoid having millions of programs upgraded only to
> find out that their IDN support is buggy and that (for instance)
> people can't reliably use certain IDNs with certain clients, and that
> there will be yet another massive upgrade

Aha---you're missing a basic point. My plan is compatible with bad->good
conversion: bad IDNs will be prohibited on the wire. If a programmer has
time to do all the work necessary for bad->good conversion, then he can
do that and deploy it. Nothing in my plan slows him down.

What I'm trying to do is make IDNs work as soon as possible. We already
have widespread support for UTF-8 in existing programs. We can take
advantage of this---as many people already have---to get UTF-8 IDNs.

Yes, it's possible that simple UTF-8 IDNs won't be as good as UTF-8 IDNs
with bad->good conversion in thosuands of programs. But they're clearly
much better than no IDNs at all. We can still add bad->good conversion;
we simply have to make sure to prohibit bad IDNs on the wire.

> it strikes me as far more difficult to get platforms to create a
> special input mode for IDNs,

The worst case is exactly what Adam has been proposing for ACE: a
separate little IDN tool, added on to the OS, that reads input from the
user and prints a good IDN.

> we already know of instances where the UTF-8 support isn't good enough.

What are you talking about?

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 29 May 2001 20:15:56 -0000

James Seng/Personal writes:
> What is considered a "bad names"?

Read the page: ``An uppercase ... is guaranteed to cause confusion: when
the uppercase Alpha is printed properly, it looks just like an uppercase
A. Some other strings are also guaranteed to cause confusion. There's a
complicated definition of good names among all Unicode strings. Good
names won't be confused with each other. Registrars won't allow
registration of bad names.''

Good names are the same concept as the output of nameprep, except that
the current nameprep specification is flawed---it doesn't handle Alpha,
for example.

> Lets expand this to any non-UTF-8 applications which uses Domain Names.

No. The issue is specific to text-mode programs that need to know the
width of their output, such as full-screen tty MUAs. A few of these
programs don't support UTF-8 yet.

> It is too early to suspect how I18N Email will look like to decide

You are confused. The changes I specified are necessary and sufficient.

> keyboard

No, I didn't leave it out. Read the page.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 21:28:26 -0400

> neither problem happens. There's only one name. 

Right.  So one way to prevent this ambiguity is to encourage applications 
to refuse to accept domain names containing "bad" dots.  That way, there
won't be any market for them. (Another way is to encourage DNS servers
to complain when they find bad dots in their zone files.)

> out that the users also need bad->good conversion.

I think that's right - the crucial question being: what is the mechanism
for "prohibiting" bad IDNs?

(note that this is completely orthogonal to whether UTF-8 or ACE is used)
 
> do that and deploy it. Nothing in my plan slows him down.

I think the difficulty is ensuring that we've identified all or most
"bad" IDNs before we deploy the code that prohibits them.  Again, we
have this difficulty regardless of the encoding that is used on the wire.

> What I'm trying to do is make IDNs work as soon as possible. 

I think we share that goal, but we favor different strategies for getting 
there.  One difference may be in how we define what it means for IDNs
to "work".  Another difference may be in our assumptions about when and to 
what extent people will be willing to upgrade their existing software.

> We already

But there are risks and costs associated with going down this path -
if the results are inconsistent (either from one application to another
or from one operating environment to another) or unpredictable, if they 
won't work well for some important languages, of if they preclude or 
delay deployment of a better solution that does work well for most languages.

> much better than no IDNs at all. 

I'm not sure that this is "clearly" the case.  When we know that there 
are significant problems with the simple approach for some languages, or
if we're pretty sure that the simple approach won't work well for very
long, it's sort of irresponsible for us to encourage people to follow
that approach - even if a few vendors have (irresponsibly?) shipped code
that assumes that approach.

Put another way, an IETF WG's job is not to endorse quick fixes, but to 
figure out what will work well in the medium to long term.  That doesn't 
mean that vendors cannot deploy short-term fixes, but we believe that
for a solution to be worth the massive deployment effort, it needs to
work well for a long time and for the vast majority of users.  We cannot
accept limitations in an IETF solution that a vendor might accept in a
short-term solution.  If we cannot do better than the short-term approach, 
we  shouldn't be wasting our time - the vendor community doesn't need 
IETF to bless something that already works well.  It's precisely because 
the quick and easy solutions do not work well that we are engaged in this 
effort.

> user and prints a good IDN.

I didn't see Adam's proposal as anything more than a workaround for expert
users of UNIX.  The vast majority of users out there would not use such
a tool regardless of whether it emitted ACE or UTF-8.
 
> What are you talking about?

We already know that different systems generate different representations
of the same characters, even when the users try to generate the same 
characters,  and that this isn't fixed by the currently-deployed software 
that just sends UTF-8. 

Keith

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 30 May 2001 09:47:01 +0800

> registration of bad names.''

So you mean I could only have "françious" but could never "FRANÇIOUS"?
What if due to my personal preference/trademark, I want to have
uppercase char? To draw an english analogy, try "IBM" or "eBay" or "AOL"
or ...

> the current nameprep specification is flawed---it doesn't handle
Alpha,

In other words, there would be a routine to create 'good' names. It may
be less complex than the current Nameprep but there will be.

> You are confused. The changes I specified are necessary and
sufficient.

I dont think I am confused. I am just not ready to do I18N Email right
now.
Your current implict proposal is to do it by pushing UTF-8 thru SMTP and
I am not ready to conclude this is right way.

> No, I didn't leave it out. Read the page.

Okay. My mistake. IMHO, this would be the greatest pain of all to go
thru...

-James Seng

---

From: [email-masked]
Date: Wed, 30 May 2001 03:27:03 -0400 (EDT)

> Okay. My mistake. IMHO, this would be the greatest pain of all to go

I belive the hardest problem is upgrading the end users. Please try and
keep in mind that whatever the solution, the greatest pain will probably
be compatibility issues between the computer and the user.

The sysadmins and support techs will be the ones dealing with these
compatability issues. Dont make our lives harder.

Please.

-Bill

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 30 May 2001 17:24:43 +0800

> > Okay. My mistake. IMHO, this would be the greatest pain of all to go

Actually "all" refers to the list of things Dan listed in his website
which needs to be change to handle Fast-Nameprep-UTF8.

> compatability issues. Dont make our lives harder.

Any suggestions or ideas how we can make it easiler? I am curious to
know...

If I am a tech support guy in Europe, and some users calls me up over
the phone asking me how to access his Korean domain names, I probably
freak out anyway. (Yes, it does not matter how slowly he speaks in
korean :-).

-James Seng

---

From: John C Klensin <[email-masked]>
Date: Wed, 30 May 2001 09:13:05 -0400

--On Wednesday, 30 May, 2001 17:24 +0800 "James Seng/Personal"
<[email-masked]> wrote:

>> I belive the hardest problem is upgrading the end users.
>> Please try
> and
>> keep in mind that whatever the solution, the greatest pain
>> will
> probably
>> The sysadmins and support techs will be the ones dealing with
>> these compatability issues. Dont make our lives harder.
>> 
>> Please.
> 
> Any suggestions or ideas how we can make it easiler? I am
> curious to know...
> 
> If I am a tech support guy in Europe, and some users calls me
> up over the phone asking me how to access his Korean domain
> names, I probably freak out anyway. (Yes, it does not matter
> how slowly he speaks in korean :-).

James,

I think this is illustrative of one of the key problems, and is
tied to the reason why I no longer believe in a "class=IN"
DNS-based solution.  I believe that the only thing that helps
that tech support person is for the call to never occur, and
that means we have to engineer protocols and user interfaces to
do the right thing in the marginal cases.  I was taught many
years ago that the key to software that is perceived by users as
effective, easy-to-use, and reliable is to figure out all of the
failure cases (actual errors and having the computer do
something different from what the user expected) and then design
backward from those cases to make sure that reasonable things
happened.

The case (to use your example) of a Korean-speaking user,
sitting in Korea and connected to a Korean ISP, in front of a
Korean workstation, accessing a site in the KR domain, and
(obviously) with Korean-speaking tech support, is always going
to be the easiest one (although a DNS solution may still not be
adequate if fuzzy search or keywords/hint are needed). There are
a whole range of approaches that will "work" or appear to work
(even though some of them are Internet- fragmenting, etc.).

The hard problems all arise when some of those assumptions are
not met.  The person speaking a particular language in a country
where another one dominates, accessing a largely
internationalized domain from a different one, using client
software that does not provide for the relevant language, etc.,
is in more or less big trouble, and "call tech support" may just
result in additional frustration.  

Clearly, seeing the Korean characters through a user agent that
understands how to handle them is the best solution, especially
if they are supplemented by language-switchable help files and
hint mechanisms.  If that is not possible, it is an open
question as to whether seeing an untranslated ACE or a
collection of "undisplayable character" glyphs is likely to
cause the greater frustration.  There is even a strong case that
the best answer is to get a clear "undisplayable international
domain name -- upgrade your user agent software or install a
shim" message rather than any failing approximations at the
display.  And, of course, such mechanisms are aided by a clear
separation between "legacy" and "internationalized" systems,
rather that tricking the existing DNS (and class) into carrying
non-LDH names or names with non-LDH interpretations.

    john

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 May 2001 17:49:31 -0000

Keith again complains that UTF-8 might not ``work well for some
important languages.'' I think it would be a complete waste of time to
discuss this; the world has settled on UTF-8. I again ask the WG chairs
to run a poll to see whether we have consensus that the long-term IDN
solution will encode Unicode characters as UTF-8 on the wire.

Keith Moore writes:
> So one way to prevent this ambiguity is to encourage applications 
> to refuse to accept domain names containing "bad" dots.  That way,
> there won't be any market for them.

Probably bad->good conversion is just as easy as rejecting bad names, if
the application is going to be upgraded anyway. The mere threat should
be enough to dissuade people from trying to use bad names.

> One difference may be in how we define what it means for IDNs to "work".

http://cr.yp.to/proto/idn.html: ``We want to let people use domain names
like ... (`alpha beta gamma dot com'). They should be able to register
the name, set up computers under the name, connect to those computers by
name, set up web pages under the name, set up links to those web pages,
browse those web pages given the name or a link, send email from an
address under the name, receive email at that address, etc.''

> Another difference may be in our assumptions about when and to 

The crucial advantage of the plan I've described is that we _don't_ have
to wait for massive software upgrades before deploying IDNs. We can take
advantage of widespread UTF-8 support in existing software.

If it turns out that users won't be completely happy without the slow,
painful nameprep-in-every-program updates, then nothing has been lost.
There won't be any transition problems, because bad IDNs won't exist.

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 May 2001 17:58:53 -0000

James Seng/Personal writes:
> uppercase char?

Existing DNS servers, SMTP servers, HTTP servers, etc. don't treat an
uppercase C-cedilla the same way as a lowercase c-cedilla.

This is why, in both the ACE approach and the UTF-8 approach, C-cedilla
is prohibited on the wire.

If you want users to be able to type C-cedilla and access your c-cedilla
domain name, you'll have to wait for thousands of programs to be changed
and redeployed. I don't want to suffer these costs, and I certainly
don't want to wait that long for working IDNs.

---Dan

P.S. The UTF-8 encoding of c-cedilla passes safely through sendmail.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 May 2001 18:08:28 -0000

> be compatibility issues between the computer and the user.

This problem is exactly what ISO 14755 is all about.

Users should be able to type Shift-Ctrl-222E for Unicode character 222E.
Users should be able to move the mouse on top of that character and see
222E CONTOUR INTEGRAL. For printouts on business cards I'd expect a
small boxed 222E on the next line.

---Dan

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 30 May 2001 12:37:31 -0700

"D. J. Bernstein" wrote:

> I again ask the WG chairs to run a poll to see whether we have
> consensus that the long-term IDN solution will encode Unicode
> characters as UTF-8 on the wire.

So that we know exactly what it is that we'd be voting for or against,
could you put together a draft which clearly documents how your
just-send-utf8 proposal would work with the legacy protocol and
implementations in the following scenarios:

 a) Where does bad-v-good filtering occur? For example, allowing
    current apps and resolvers to just-send-utf8 means that
    Z-with-caron will have different code values based on the
    local charset (windows-1252 is different from 8859-15 is
    different from 10646/unicode). Where and how is the
    normalization handled in your scheme?

 b) How do you prevent utf8 chars from breaking legacy devices?
    EG, for an older (and orphaned) X terminal that legitimately
    rejects underscore in hostnames, how do you keep utf8 from
    jamming up bootp/tftp on that host? Similarly, how do you
    prevent utf8 in CNAME responses from jamming up all of the
    other hosts that legitimately enforce existing standards?

 c) Also, how do you prevent utf8 in PTR RRs from jamming up
    syslog daemons, received headers, netstat output and the
    hundreds if not thousands of other applications that
    currently process PTR data for hostnames and which reject
    data that does not comply with LDH rules?

I don't think you can do it with the current protocol and implementation
limitations. Your hand-waving is unconvincing. Prove me wrong. Consider
yourself double-dog-dared if you must, but please either poop or get off
the pot. Unless you can clearly describe how this will work, please let us
get on with real progress.

I repeat, this needs to be in an ID so that we know for certain what you
really mean. Your web site does not serve that purpose.

---

From: [email-masked]
Date: Wed, 30 May 2001 15:43:03 -0400 (EDT)

> small boxed 222E on the next line.

No offense but, what planet do these users come from? and are they
hiring sysadmins on this planet? Seriously, enough users have problems
double clicking that Microsoft dropped it as a required skill for
using Windows.

I dont belive in users that can handle typing Shift-Ctrl-anything,
and I talk to them all day. I have more faith in Santa, and he is alot
easier to talk to.

-Bill

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 May 2001 20:18:52 -0000

Eric A. Hall writes:
> So that we know exactly what it is

``The long-term IDN solution will encode Unicode characters as UTF-8 on
the wire.'' I suspect that we have consensus on this point, even though
we don't have consensus on the short-term approach.

> that we'd be voting for or against

I'm sure several people will pop up to inform you that IETF has ``straw
polls'' but doesn't have ``votes.'' Votes are things done by legitimate
standards organizations that obey United States antitrust law; the IETF
is not one of those organizations.

> How do you prevent utf8 chars from breaking legacy devices?

Fix the legacy devices. This is obviously something that's going to
happen anyway.

> an older (and orphaned) X terminal that legitimately rejects
> underscore in hostnames

Are you fantasizing, or can you identify an actual X terminal that has
to be fixed?

Same question about your ``hundreds if not thousands of other
applications that ... reject data that does not comply with LDH rules.''
Do you have any evidence that the fixes listed on my web page---in
particular, enabling no-check-names in /etc/resolv.conf, with the
current gethostbyname() library---are not sufficient?

> I don't think you can do it with the current protocol

You are speaking nonsense. The proposal changes the protocols.

> and implementation limitations.

Implementation problems are a serious issue. That's why I've been going
to so much effort to identify the implementation work that actually has
to be done. See http://cr.yp.to/proto/idn.html.

---Dan

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 30 May 2001 14:52:18 -0700

> Fix the legacy devices. This is obviously something that's going to

Sooner or later, but your approach breaks them until they are replaced.
Your refusal to concede this point does not change the fact.

> Are you fantasizing, or can you identify an actual X terminal that has

I don't remember the exact make and model, but I encountered this
particular problem at AVIS 10 or 15 years ago. Probably NCD, but it is
irrelevant really.

Your list of broken apps proves that there is more than one. Surely you
know that your list is incomplete, statistically and quantitatively. Your
position that any list of would be finite is completely untenable.

> You are speaking nonsense. The proposal changes the protocols.

What proposal?

Also, you ignored item (a) in your response. Please tell us how you plan
to restrict the eight-bit data that enters the DNS service space.

---

From: Keith Moore <[email-masked]>
Date: Wed, 30 May 2001 18:11:18 -0400

> important languages.'' 

once again, misrepresenting my statements.

you're absolutely right - there's no point in us arguing about such things.

Keith

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 1 Jun 2001 17:34:29 +0800

If this is the vote for the strawpoll, it is too vague to move forward
with. UTF-8 on the wire ... how? tag? untag? negiotated? retrofit into
the packet? what needs to be change? what will break? what wouldnt? what
difficulties would we face.

We could move forward with a strawpoll if only you have a concrete
proposal (or one which you can point to). Currently, all we have is what
you have on your website and honestly, I think it is not a complete
picture nor a detail description of who needs to do what and what have
to be taken care of. And given our discussion, it is apparently your
Fast-Nameprep (on Keyboard) idea is far too different compared to some
other proposal we have.

In order to move this forward, could I ask you document your idea into
an I-D first? Thanks!

ps: If I were to vote now, I would vote for UTF-64 (if there is one). My
guess is 10-20 years later, we going to see 64bits hardware, with 64bits
OS computing in 64bits wide characters. Thus UTF-64 makes more sense for
a long term solution. We wont want our future engineers to scream at us
for limiting to them only 8-bits when they have 64-bits to play with.

-James Seng

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 1 Jun 2001 02:18:21 +0000

> doesn't care about the interpretation of the bytes as characters.

What about in the normal case?  Is gethostbyname() copying the bytes
because the interface takes bytes, or because the interface takes ASCII
and DNS also takes ASCII?  I've tried to discover the answer, to no
avail; I think the semantics are ambiguous.

Putting no-check-names into /etc/resolv.conf is obviously a workaround,
a way to tell the library "get out of my way, I'm doing something
experimental, and I'll take responsibility for it", so I don't think we
can infer anything about the intended semantics from the behavior under
those circumstances.

It would be instructive to see how gethostbyname() behaves in a
locale for which the LDH characters do not occupy their ASCII code
positions.  But I know of no such locale on any system that implements
gethostbyname().

My gut feeling is still that if gethostbyname() is going to start
accepting 8-bit names, it should interpret them according to the charset
of the current locale (unless "no-check-names" is in effect, which
disables any interpretation and lets the caller specify the raw bytes).
If application programmers want it to assume UTF-8, they merely need to
set the locale to one that uses UTF-8.  And if you're predictions are
correct, that will be the default anyway.

AMC