Subject: [idn] Reality Check

From: Dan Oscarsson <[email-masked]>
Date: Tue, 10 Jul 2001 13:04:08 +0200 (MEST)

Has this working group lost the contact with the real world?

All discussion is about ACE/nameprep/IDNA/backward compatibility!

What about the current real needs and the future?
What about DNS?

It looks like many think the only important thing is
backward compatibility without changing the DNS protocol/server.
Why is people talking about DNSSEC, SRV records, DNAME,..., if
nobody is going to update their DNS servers?
Why not go for a solution that updates DNS and requires upgrade
of the DNS protocol and servers?
Why should it be quicker to implement DNS in the appications and
get them to be updated faster? MIME was introduced very long time
ago and still there are a lot of mail programs that do not handle it.

What does the normal human beings of the real world expect from DNS?
The semantics of current DNS says:
- a name can have mixed upper and lower case letters
- names will be matched case insensitively
IDNA says:
- a name can only be in lower case
- names will be matched exactely
So IDNA breaks current semantics of DNS. Why not go for a solution
that works as people expect? And matches the current semantics of DNS?

I have tried to think of how I as a programmer, developer, editor and
user would handle DNS names.
An ACE-solution will result in e-mails containing lines like:
To: =?quoted-printable-encoded?= <[email-masked]>
A line with a lot of different mixed encoded forms. To handle this
as a programmer you have to idetify the different parts of the line and then
for each part identify what encoding is used, and the decode each before the
line can be displayed to a user or handled in a program.
As of today I have fewer programs than I want that handles e-mail - this
is because the difficulty in programmatically handling MIME.
With ACE in domain names and something like it in user names it will be
even worse. Why could we not use something simple like having the
entire line encoded in UTF-8? Instead of having a difficult time to
parse data and a lot of different decoders I could use decode UTF-8 into my
local character set.
Many edits their html files with a text editor, or writes documents
with embedded DNS names and URLs. The only way you can expect people
to enter DNS names and URLs in those files is by using the same character
set as the rest of the text and they will not convert them into
ACE, %-encoding or other unnatural form. If we use different encodings
for all things, all software handling text files have to handle the
convertion to/from the character set of the file into the special
encoding needed for protocols. I can see lots of failures giving the
user failures they cannot understand and a lot of encoded forms getting
into the users text files, and a lot of non-encoded forms getting into
the DNS protocol. At my site clearly the simplest and best working
solution is to let my DNS server do most of the work.
While many programmes working for interoperability have code to handle
UCS to local character set conversions and have case insensitive matching,
they do not have code for things like ACE or quoted-printable.
A solution based on using UCS as transport format and matching based
on the normal "case insensitive" matching rules would make many programs
much easier to adopt to non-ASCII DNS names.
And if we used UCS only and did not use ACE as a backward compatibility
encoding, would it really be that bad? In my system the function that
might have most problems is e-mail, but that is also one of the systems
that probably can be handled easiest - I can fix my MTA and I can also
avoid using non-ascii in my mail domain to begin with. or ACE could be used
in the few places where it cannot be avoided for a transition period.

Does the IDNA (ACE in application) solution that appears be the only
focus of this working group match the real needs of people in the
current and future world?

As it feels for me, this working group is going for a solution that
is far from the simple clean handling of non-ASCII in DNS that I have
always wanted and worked for.

    Dan

---

From: Martin Duerst <[email-masked]>
Date: Wed, 11 Jul 2001 00:23:48 +0900

Hello Dan, others,

>To: =?quoted-printable-encoded?= 
><[email-masked]>

This is a subject I have partially discussed with Keith earlier.
I think it's indeed very serious. There are thousands of different
file formats and occasions where domain names turn up. If ACE leaks
into only a small percentage of them, a lot of people have a lot
of problems. They may complain about ACE more than Keith is
complaining about NAT.

Practical books about a programming language often contain examples
of how to process your mail, and so on. All these examples are based
on the fact that the characters can be parsed directly. Grepping
for a word in the subject headers in a mail spool file is easy as long
as the subject is in ASCII. For anything beyond ASCII, it just fails.

I remember Ulrich Drepper, responsible for gclib, saying at a dinner
here in Japan that UTF-8 was the right way to go, because it would
allow people like him to provide the base for internationalization
(rather than just doing nothing at all), and would allow others,
more familiar with their own language, to build on it. In some sense,
RFC 2277 is based on a similar assumption. ACE severely breaks this
assumption.

Internationalization doesn't come for free. Most people who think
ACE is a good idea just don't see that work. In the short term,
having to update a DNS server (even if it's just with an 8-bit
clean version) seems a lot of work. But it's actually extremely
easy, compared to updating applications. Also, passing ACE in
application protocols seems extremely easy. But it just means
that these application protocols aren't really internationalized
yet, and that a lot of work is waiting out there.

>ACE, %-encoding or other unnatural form.

Yes indeed. Please see
http://www.ietf.org/internet-drafts/draft-masinter-url-i18n-07.txt
for URIs (working on updating it) and
http://www.ietf.org/internet-drafts/draft-ietf-idn-uri-00.txt
for the DNS part in URIs (will resubmit it to keep it alive;
I don't expect this to be discussed in London, but I haven't
seen any other proposals for how to solve this part of the
problem, and I guess the WG should deal with it once the
important issues are dealt with, and it would look silly
to resubmit as a personal contribution and later again
as a WG draft [sorry for this lengthy sentence]).

>current and future world?

No. People will suffer from ACE for years to come. When MIME
was created, Unicode barely existed, so there was some excuse.
But there is no excuse for ACE. I wish I had patented it in
December 1996; all bad ideas should be patented ;-(.

Regards,   Martin.

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Tue, 10 Jul 2001 14:28:47 -0400

| What does the normal human beings of the real world expect from DNS?
| The semantics of current DNS says:
| - a name can have mixed upper and lower case letters
| - names will be matched case insensitively
| IDNA says:
| - a name can only be in lower case
| - names will be matched exactely
| So IDNA breaks current semantics of DNS.

  This isn't an entirely correct statement.  Names after nameprep are folded
to lower case for purposes of comparison to ensure that the
case-insensitivity requirement is preserved.  Users can enter names in mixed
case, and nameprep+a case preserving ACE could preserve this in the ACEd
name.  These semantics are essentially preserved in the current
IDNA+nameprep+DUDE recommendation, and could be enhanced to preserve case in
the ACE->IDN conversion step.

  Dan, you've mentioned elsewhere in this message that the IDNA approach is
flawed because it assumes that no one wants to update their DNS servers.

  IDN is not a DNS problem.  It is an applications-that-use-the-DNS problem.

  Given that you're going to have to update many applications to properly
handle IDNs, why not only update the applications?

  I think one area where the IDNA document currently falls down is in not
providing enough discussion on where IDNs will be in native form and where
they will be in ACE form.  I think much of the confusion in this discussion
arises from that point.

  -bws

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 10 Jul 2001 19:18:56 +0000

> What about DNS?

What about DNS?  How DNS works just doesn't matter very much, because
it's invisible to everything except the resolver, which can offer any
combination of interfaces to applications (including ASCII, ACE, UTF-8,
and the local charset) regardless of what actually appears on the wire
between the resolver and the DNS server.  We can define a way for DNS to
use UTF-8 now, or later, or never, and I really don't think it will make
much difference to anything.

What matters much more than DNS are all the other protocols that
embed domain names in messages (RFC 2822, SMTP, URIs, HTTP cookies,
SSL certificates, SSH keys, PGP keys, etc).  Maybe these should all
be upgraded to support UTF-8 domain names, and maybe they all will be
eventually, but that could take a long time.  We need ACE to be an
option (in addition to UTF-8), at least for a while.

> Why should it be quicker to implement DNS in the applications and get
> them to be updated faster?

Huh?  I don't think anyone is suggesting this.  DNS should be
implemented in the resolver, as it always has been.  Applications are
going to need to be updated to support nameprep if they want to compare
names, and they are going to need to be updated to support UTF-8 if they
want to display IDNs returned from a UTF-8 resolver or pass user-entered
names to a UTF-8 resolver.  This is true regardless of whether the
protocols (email etc.) use ACE or UTF-8.

> MIME was introduced very long time ago and still there are a lot of
> mail programs that do not handle it.

Good point.  Do you think an extension to RFC 2822 to allow UTF-8 domain
names is going to be supported much more quickly?  I wouldn't count on
it.

> - names will be matched exactly
> So IDNA breaks current semantics of DNS.

No, IDNA does not say that.  Application protocols can transmit
names in non-nameprep'd mixed-case form.  Applications can store
and display names in non-nameprep'd mixed-case form.  The only
time it is necessary to apply nameprep is just before comparing
two names.  Nameprep is mainly just a way to do case-insensitive
alternate-represenation-insensitive comparisons.

In DNS, in order to perform case-insensitive name comparisons, the
server must perform case-folding, or it must rely on the client to do
it.  It makes no semantic difference whether it's done on the server or
the client, but doing it on the client is better for load balancing,
since case-folding expensive and there are more clients than servers.

As for the names returned from the DNS server, there are two ways to get
the full mixed-case name, either of which will work:  First, DNS could
return both the nameprep'd ACE name and the non-nameprep'd UTF-8 name,
and the application could use whichever one it prefers (IDN-unaware
applications would automatically get the nameprep'd ACE name) (maybe
there could be an optional flag in the request to tell the server not
to bother with the ACE name, as an optimization).  Second, all the ACEs
currently under consideration can represent mixed-case Unicode strings
as mixed-case ACE strings.

> Why could we not use something simple like having the entire line
> encoded in UTF-8?

We could, eventually.  But if you want to send mail to your friend with
an IDN address and also to your other friend whose ISP/employer/school
does not yet use a UTF-8-aware MTA, then you need ACE.

Martin Duerst <[email-masked]> wrote:

> on the fact that the characters can be parsed directly.  Grepping for
> a word in the subject headers in a mail spool file is easy as long as
> the subject is in ASCII.  For anything beyond ASCII, it just fails.

Yes, and this argues for some other working group to define an extension
to RFC 2822 that uses UTF-8 instead of ACE and MIME encoded-strings.
And one of the first places this extension would get used would be, not
on the wire, but in mailbox files, because the interoperability issues
are less severe there.  Eventually this format might get deployed for
on-the-wire use.  This extension is already motivated by Subject fields,
but the demand hasn't been enough for the extension to appear.  Maybe
after IDNs become common the demand for this RFC 2822 extension will
become great enough, but I don't see how IDNs can get off the ground
without ACE.

In summary, I think applications should use UTF-8 when possible, but
ACE must be an option to allow interoperability with existing protocols
that require LDH characters.  It hardly matters what DNS uses.  Nameprep
should be thought of as the mechanism for comparing IDNs, analagous to
the case-insensitive ASCII comparison currently used for LDH names.

AMC

---

From: "Edmon" <[email-masked]>
Date: Tue, 10 Jul 2001 16:14:07 -0400

From: "Adam M. Costello" <[email-masked]>
> that require LDH characters.

Currently, IDNA does not provide the "option" for UTF-8 that you are
suggesting.  Do you think it is appropriate therefore that there be a
mechanism whereby an application can choose to use either the UTF8 or ACE
form of an IDN depending on the compliancy of the destination server?  This
ensures that once the other components or protocols become UTF8 compliant,
the DNS will not be stuck with ACE.

Edmon

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 10 Jul 2001 20:46:04 +0000

Edmon <[email-masked]> wrote:

> Do you think it is appropriate therefore that there be a mechanism
> whereby an application can choose to use either the UTF8 or ACE form
> of an IDN depending on the compliancy of the destination server?

The application does not talk to the DNS server.  The application talks
to the resolver, and the resolver talks to the DNS server.  The DNS
protocol matters to servers and resolvers, but not applications.  (Am I
wrong about this?  Are there applications that speak DNS directly?  More
than a negligible number of them?)

> This ensures that once the other components or protocols become UTF8
> compliant, the DNS will not be stuck with ACE.

I have no objection to defining a UTF-8 extension for DNS, but I don't
see how it will have much impact on applications, and I certainly
don't see any urgency.  IDN deployment can commence as soon as IDNA is
finalized and approved, while the DNS extension can be rolled out soon
after, or much later, with the same chance of success either way.

Perhaps the people focused on the DNS protocol should redirect their
attention to the resolver API, because that's where you can help
applications.  This API is already being changed to support IPv6 (see
RFC 2553), so I'd suggest coordinating with that effort.

AMC

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Tue, 10 Jul 2001 17:31:37 -0400

| The application does not talk to the DNS server.  The application talks
| to the resolver, and the resolver talks to the DNS server.  The DNS
| protocol matters to servers and resolvers, but not applications.  (Am I
| wrong about this?  Are there applications that speak DNS directly?  More
| than a negligible number of them?)

  MTAs (sendmail, etc.) generate MX lookups directly instead of through the
resolver.

  DNS tools such as dnsquery and dig (and the venerable but deprecated
nslookup) generate their DNS packets directly, but aren't really much of an
issue (and can be easily wrapped).

  Anything that uses DNS data but doesn't do get{host|node}by{name|addr}
doesn't use the system resolver.

  -bws

---

From: "Edmon" <[email-masked]>
Date: Tue, 10 Jul 2001 17:39:23 -0400

> wrong about this?

What I meant was the application using the DNS information then
communicating to a destination server such as an http server.  That is,
after they obtain an RR from the DNS, how they use the information to then
connect with the destination.  Should we not allow this to be in UTF8 in the
future?

> after, or much later, with the same chance of success either way.

I think it is important that a UTF-8 extension is available either in
parallel or soon after, or there is a greater risk that we wont see an
upgrade anytime soon.

> RFC 2553), so I'd suggest coordinating with that effort.

Are you suggesting then that the discussion on using a UTF8 extension for
IDN in the DNS would be outside the scope of the IDN wg?

Edmon

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 10 Jul 2001 22:03:10 +0000

"Brian W. Spolarich" <[email-masked]> wrote:

> MTAs (sendmail, etc.) generate MX lookups directly instead of through
> the resolver.
>

Okay, thanks.  It sounds like there are indeed some significant
applications that speak DNS directly, but going through the resolver
is by far the common case.  (Hmmm, now that I think about it, I bet
the inability of the resolver to provide access to arbitrary resource
records is the main reason why so few resource records have been
deployed.)

Edmon <[email-masked]> wrote:

> communicating to a destination server such as an http server.  That
> is, after they obtain an RR from the DNS, how they use the information
> to then connect with the destination.  Should we not allow this to be
> in UTF8 in the future?

I don't follow.  The application gets the domain name from the user or
an href attribute or wherever (not from DNS).  The application hands
the domain name to the resolver, and gets back an IP address.  The
application then opens a TCP connection to that IP address and sends the
HTTP request, which includes the original domain name but not anything
that came back from DNS.

In some other scenarios, applications might indeed receive domain
names from DNS and copy them into application messages.  But the
application is almost always getting the name from the resolver, not
from DNS directly.  Therefore, if you want to make things easier for
the application by giving it UTF-8 domain names, you need a resolver
interface that can return UTF-8 when invited to do so by the application
(but the resolver should return UTF-8 only if explicitly invited by the
application, otherwise it will break existing applications).  Once you
have a UTF-8 resolver interface, it doesn't matter what protocol is
running between the resolver and the DNS server, because the application
can get UTF-8 from the resolver regardless.

> Are you suggesting then that the discussion on using a UTF8 extension
> for IDN in the DNS would be outside the scope of the IDN wg?

Not at all.  I am suggesting that if your goal is to make life easier
for application programmers by allowing them to use UTF-8 internally
and never use ACE, then the way to accomplish that goal is to extend
the resolver interface.  You can also change DNS, but that is less
helpful to applications (it mostly affects resolvers and DNS servers)
and therefore less urgent.

AMC

---

From: Mats Dufberg <[email-masked]>
Date: Wed, 11 Jul 2001 00:12:11 +0200 (CEST)

> records is the main reason why so few resource records have been

The destinction between resolver and (authoritative) nameserver is not as
sharp as might seems. In fact, the responses from both are the same,
except for the aa flag. Often, an nameserver is used for both types of
services, typically authoritative for the local domain and resolver for
the local net. If ACE is used for IDN, then both resolvers and
authoritative servers must give respones with the same encoding.

Mats

-----------------------------------------------------------------
Mats Dufberg                                     +46-8-545 857 06
[email-masked]                           fax: +46-8-545 857 29

---

From: "Edmon" <[email-masked]>
Date: Tue, 10 Jul 2001 18:46:05 -0400

> HTTP request, which includes the original domain name but not anything

Yes, I meant that when a dns application connects to the destination server,
it includes the "original domain name".  Do you at this point use ACE or
UTF8 or some other encoding?  IDNA mandates the use of ACE.  I think we
should have an "option" where if the destination HTTP server is IDN/UTF8
aware, then the UTF8 could be used.  This would give the incentive for
related servers and protocols such as HTTP servers to update to IDN/UTF8
extension.

Edmon

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 10 Jul 2001 23:41:05 +0000

[A response to Mats Dufberg appears at the bottom.]

Edmon <[email-masked]> wrote:

> Yes, I meant that when a dns application connects to the destination
> server, it includes the "original domain name".  Do you at this point
> use ACE or UTF8 or some other encoding?  IDNA mandates the use of
> ACE.  I think we should have an "option" where if the destination HTTP
> server is IDN/UTF8 aware, then the UTF8 could be used.

But "DNS applications" are the exception.  In the common case,
applications use the resolver, in which case the language spoken between
the resolver and DNS server is independent of the language spoken
between the application and the resolver.  (If a resolver provides a
UTF-8 interface but the DNS server does not support UTF-8, then the
resolver should ACE-encode the queries and ACE-decode the responses for
applications using the UTF-8 interface.)

Does IDNA really forbid DNS extensions for UTF-8?  Or does it merely
specify how IDNs can be used without extending DNS, while remaining
silent on the issue of DNS extensions?  I see no reason to forbid a DNS
extension for UTF-8, but nor do I see a pressing need for one.

> This would give the incentive for related servers and protocols such
> as HTTP servers to update to IDN/UTF8 extension.

I don't see how the incentive for other protocols to use UTF-8 is any
greater if DNS uses UTF-8.  DNS responses are seldom directly seen by
applications, and domain names obtained from DNS are seldom copied
into application messages.  The only time UTF-8 support in DNS would
encourage UTF-8 support in application protocols is when domain names
are copied directly from DNS responses into application messages, and
the frequency of that is seldom * seldom = approximately never.

A UTF-8 resolver interface, on the other hand, would indeed encourage
other protocols to support UTF-8, because then the applications could
copy domain names directly from application messages to the resolver and
vice-versa, without ever needing to do ACE encoding or decoding.

Mats Dufberg <[email-masked]> wrote:

> The destinction between resolver and (authoritative) nameserver is not
> as sharp as might seems.

I wonder if you misunderstand what I mean by resolver.  The resolver is
a library on the client machine, linked into the application process.
It is not a DNS server of any kind--it is a DNS client.

AMC

---

From: Keith Moore <[email-masked]>
Date: Tue, 10 Jul 2001 20:55:47 -0400

> suggesting.

sure it does.  the application can use whatever encoding it wants to.
the protocol used by DNS on-the-wire is irrelevant.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 11 Jul 2001 17:12:54 +0900

>the protocol used by DNS on-the-wire is irrelevant.

It would be extremely helpful if IDNA would make it clear that
applications can and should do something else than ACE.

The current IDNA draft (02), in the the unnumbered diagram,
says 'Application-specific protocol: predefined by the
protocol or defaults to nameprepped ACE'. This should be
changed.

Some other comments on the IDNA draft:

The abstract and intro give the impression that only updating
applications results in a significant advantage. Readers of
the doc may get the impression that servers/resolvers are
50% and applications 50%. As far as I know, it's more like
10%/90%. The draft should be changed to not give such wrong
impressions.

2.1.1: "Applications MAY allow ACE input and output, but are
not encouraged to do so except as an interface for advanced
users, possibly for debugging.": Please remove the 'advanced
user'. Giving people who want to look at the names the way they
are written the impression that they are retarded is not a
good idea.

2.1.1: "Since ACE can be rendered either as the encoded ASCII
glyphs or the proper decoded characters, the rendering engine
for an application SHOULD have an option for the user to select
the preferred display.

- The word 'ACE' is used here completely differently from above
   (including input, output, and process, rather than only one
    form).
- 'Rendering engine' is usually in the OS or window system/
   toolkit, not in the application. It would be strange to require
   such an engine to be able to decode ACE.
- The 'SHOULD' has to be conditioned on the MAY above.

My rewording would look like this:

In case Applications allow ACE input and output, they SHOULD
provide an option for the user to select ACE, which SHOULD
be off by default.

2.1.1:
In protocols and document formats that define how to handle
specification or negotiation of charsets, IDN host name parts can be
given in any charset allowed by the protocol or document format. If a
protocol or document format only allows one charset, IDN host name parts
must be given in that charset.

Preferably change 'charset' to 'character encoding' throughout
the draft. 'charset' is the name of a parameter, not a concept.
Also, in addition to character encodings, escapes are also possible
(I'm thinking of HTML &eacute;,...).

Also 'given in any' is much too general. And negotiation is irrelevant.
A single encoding is also just a special case, and so it should be
either left out, or called out ("In particular,..."). Also
On the other hand, protocols often use different character encodings
in different places. So I would write:

In any place where a protocol or document format allows to transmit
the characters in IDN host name parts, IDN host name parts MUST/SHOULD
be transmitted using whatever character encoding and escape mechanism
that the protocol or document format uses at that place.

2.1.2:
"that is resolving a non-internationalized host name parts MUST NOT do"
singular-plural mismatch.

2.1.4:
----
If an application decodes an ACE name but cannot show all of the
characters in the decoded name, such as if the name contains characters
that the output system cannot display, the application SHOULD show the
name in ACE format instead of displaying the name with the replacement
character (U+FFFD). This is to make it easier for the user to transfer
the name correctly to other programs using copy-and-paste techniques.
Programs that by default show the ACE form when they cannot show all the
characters in a name part SHOULD also have a mechanism to show the name
with as many characters as possible and replacement characters in the
positions where characters cannot be displayed.
----

Copy-and-paste often works even if display doesn't, and on the other
hand display may work but copy-and-paste may fail. So I would suggest
to just write:

This is to make it easier for the user to transfer
the name correctly to other programs.

Also, I would note that there are other ways to deal with this,
namely to show some hex code or use some popup tooltip,...
And also in many cases, the application doesn't know exactly what
the underlying rendering engine can display or not.

2.1.5

----
An application which receives a host name SHOULD verify whether or not
the host name is in ACE. This is possible by verifying the prefix in
each of the labels, and seeing whether or not the label is in ACE. This
MUST be done regardless of whether or not the communication channel used
(such as keyboard input, cut and paste, application protocol,
application payload, and so on) has negotiated ACE.
----

'negotiating ACE' sounds quite strange, in particular for keyboards.
It is not completely impossible to e.g. go to the X Consortium and
register something like "ACE_STRING" in addition to "COMPOUND_TEXT"
and "UTF8_STRING",... and then have application A fill in all of
these and application B pick ACE_STRING if it doesn't understand
the others,... It is also in theory concievable to create an input
method that directly produces ACE from some keyboard layout.
But there are considerable problems with this, I'm sure it's not
done for similar things such as Mail mime headers, and I think
it's a bad idea that the draft mention such things because it
may lead implementers into the woods.

Another problem is that the MUST seems to overrule the SHOULD.
Which one do we take?

----
The reason for this requirement is that many applications are not
ACE-aware. Applications that are not ACE-aware will send host names in
ACE but mark the charset as being US-ASCII or some other charset which
has the characters that are valid in [STD13] as a subset.
----

I thought that the general idea was to register a 'charset' parameter
value for the finally chosen ACE so that it could be used by transcoding
libraries, but to strongly discourage the use of this label in any
other context. Also, in a large number of cases, the 'charset' parameter
cannot be changed for every single bit of text (think email bodies).

2.1.6 Bidirectional text
----
In IDNA, bidirectional text is entered and displayed exactly as it is
specified in ISO/IEC 10646. Both ISO/IEC 10646 and the Unicode standard
have extensive discussion of how to deal with bidirectional text. Any
input mechanism and display mechanism that handles characters from
bidirectional scripts should already conform to those specifications.
Note that the formatting characters that manually change the direction
of display are prohibited by nameprep, thus making the task for input
and display mechanisms easier.
----

Sorry, but this is completely bogus. Some examples, even though
logically clear, would look totally mutilated.
Paul told me that there is an alternative proposal that requires
bidirectional text to be entered and stored in visual order.
This would have several very severe problems, namely:
- People would have to type things in backwards (i.e. typing
   FTEI to get IETF)
- Cut/paste from general text would not work at all.
- Sorting of domain names in Arabic or Hebrew would work the
   wrong way (first all words ending in 'a', then all ending in
   'b' and so,...)
- Text-to-speech applications will read the text the wrong way
   round (ftei for IETF again).

An alternate proposal has been around for a very long time, please
see http://www.ietf.org/internet-drafts/draft-masinter-url-i18n-07.txt
section 3.2. This proposal, originally by Francois Yergeau
(main contributor to RFC 2070 and to Tango, first web browser
that worked with Arabic as far as I know) works by taking advantage
of the Unicode bidirectional algorithm (thus not requiring special
code in particular for display) and can easily be integrated into
the nameprep algorithm (mainly removing specific bidi control
characters under specific conditions).

It may look more complicated than other (non-)proposals, but
bidirectionality IS a very thorny issue without easy ways out.

3. Name Server Considerations

----
The host name data in zone files (as specified by section 5 of RFC 1035)
MUST be both nameprepped and ACE encoded.
----

I thought that the zone files described here refer to some format
as exchanged over the network. But reading sec. 5 of 1035, it
speaks about 'loading a zone'. Obviously, there may be different
formats and ways to load a zone, so I have no idea why we would
have to require that ACE is used. I can very well imagine enhanced
dns software that is able to read in data in UTF-8 (or some local
encoding) and do nameprep and ACE internally. Actually, it would
seem that this would be a lot more convenient for a lot of users.

So either you have to explain the requirement, and restrict it
to the cases where it's really needed, or you remove it.
(or even better, say that dns software should provide facilities
for input from something readable).

Regards,    Martin.

---

From: Keith Moore <[email-masked]>
Date: Wed, 11 Jul 2001 09:03:14 -0400

> applications can and should do something else than ACE.

Transition issues for existing applications are sufficiently
thorny, and differ enough from one application to another, that
it would be difficult to word an appropriate blanket 
recommendation to use "something else".  Even for new applications,
a blanket recommendation may not be appropriate.

The trend of using UTF-8 in new applications is already 
well-established even without encouragement from IDN; the
fact that IDN names are based on the 10646 character 
repertoire will only serve to further encourage the practice.
It will be natural for many of these applications 
to use the UTF-8 form of IDNs also.  Still, this choice needs
to be made on a per-application basis.  For example, should
IMAP server accept search queries for domains in UTF-8 even
though the stored messages that they're searching have those
domains expressed in DUDE (or whatever)?  We might have a guess
about the answer (and your guess might even agree with mine), 
but this is a question that should be answered by MTA and UA 
implementors (who have a better understanding of the issues,
and who will have to deal with their choices) rather than by 
the IDN WG.
 
Keith

---

From: "Edmon" <[email-masked]>
Date: Wed, 11 Jul 2001 10:11:12 -0400

> vice-versa, without ever needing to do ACE encoding or decoding.

If you think that a UTF8 resolver interface is helpful, why then do we not
go one step further to layout a plan for UTF8 to be used on the dns as well?
How would the wait benefit us?  We are in a DNS WG, we are talking about IDN
and we are supposed to explore protocol extensions too.  I do not disagree
that IDNA is not a good start, but should we not go a few steps more to
layout a plan that would take us through IDNA(ACE) and into UTF8 eventually?

Edmon

---

From: Keith Moore <[email-masked]>
Date: Wed, 11 Jul 2001 15:02:56 -0400

> I do not disagree

first we need to get an ASCII-based solution out the door.

perhaps by then someone will have come up with a convincing argument as to
why having the ability to use UTF-8 on the wire in the DNS protocol 
(and producing consistent results with ASCII queries) actually produces 
a tangible benefit that is worth the incremental cost. 

so far, nobody has managed to do that.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 11 Jul 2001 14:30:54 -0500

> first we need to get an ASCII-based solution out the door.

Not "out the door" so much, but one definitely needs to exists as a legacy
mode. IOW, ACE must exist in any event, but it does not need to exist as
"the solution to the problem".

> perhaps by then someone will have come up with a convincing argument
> as to why having the ability to use UTF-8 on the wire in the DNS
> protocol (and producing consistent results with ASCII queries)
> actually produces a tangible benefit that is worth the incremental
> cost.
> 

Nobody has proposed a protocol for it. I will do this soon.

The costs for embedding and managing encrypted domain names in protocols
is obviously greater than the costs for embedding and managing UTF8 domain
names. This should be so obvious that no "arguments" are needed.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Keith Moore <[email-masked]>
Date: Wed, 11 Jul 2001 15:35:43 -0400

> names. This should be so obvious that no "arguments" are needed.

if you think the cost savings are obvious, you haven't done the analysis.

hint: the cost of dealing with two different name representations is greater
than the cost of dealing with a single one.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 11 Jul 2001 14:47:17 -0500

> but due to backward compatibility and deployment issues, the ACE
> representation is the essential one.  The UTF8 one is not needed.

It is only essential to the past twenty years of activity, while UTF8 is
essential to the next twenty years.

Managing a single representation of data is going to be the most important
aspect going forward. You know this deep inside.

> as I've said before, the time to move to a UTF8 representation in the
> DNS protocol is when we upgrade to something better than DNS.

Not necessary.

---

From: Keith Moore <[email-masked]>
Date: Wed, 11 Jul 2001 15:40:55 -0400

> > hint: the cost of dealing with two different name representations is
> > greater than the cost of dealing with a single one.
> 
> No kidding. With UTF8 they only do it once. With ACE they do it twice.

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 11 Jul 2001 14:38:01 -0500

---

From: Keith Moore <[email-masked]>
Date: Wed, 11 Jul 2001 16:04:08 -0400

> essential to the next twenty years.

for some applications, ACE will be needed for the next twenty years.
we can't expect people to upgrade their applications en masse.
nor would this make good operational sense.
 
> aspect going forward. You know this deep inside.

Gut feelings are often wrong, especially when they favor widespread and
disruptive change.  I place far more confidence in cost-vs-benefit analysis.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 11 Jul 2001 15:13:33 -0500

> for some applications, ACE will be needed for the next twenty years.

Yes, this is why it is necessary for legacy support. It is not necessary
for moving forward, and will actively interfere with progress.

> nor would this make good operational sense.

We are expecting them to do it for ACE (so that it is usable anyway). We
might as well let them move to recent state-of-the-art when they do it.

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 11 Jul 2001 21:22:44 +0000

Edmon <[email-masked]> wrote:

> If you think that a UTF8 resolver interface is helpful, why then do we
> not go one step further to layout a plan for UTF8 to be used on the
> dns as well?  How would the wait benefit us?

We cannot develop standards at infinite speed.  If we worry about the
less urgent problems now, we distract ourselves away from the more
urgent problems, and thus delay the more urgent solutions.

The priorities are (in my estimation):

ACE & nameprep: essential
UTF-8 resolver: helpful
UTF-8 in DNS:   slightly helpful, maybe

Applications can start using IDNs as soon as ACE and nameprep are
finalized, and not a moment sooner.  A UTF-8 resolver interface might
encourage more applications to start supporting IDNs, because it will
make it easier.  A UTF-8 DNS extension will do very little, if anything,
for applications; it will make DNS servers and resolvers more complex
in the short term (when they support both UTF-8 and ACE) and simpler in
the long term (if ACE can someday be deprecated).  This is why I see no
urgency in developing a DNS extension.

"Eric A. Hall" <[email-masked]> wrote:

> The costs for embedding and managing encrypted domain names in
> protocols is obviously greater than the costs for embedding and
> managing UTF8 domain names.

If a protocol has the option of using only UTF-8 and not ACE (a new
application protocol would have this option, for example), then yes
that is obviously simpler.  But DNS does not have this option--it must
support ACE, at least in the short term.  It therefore has a choice
between ACE-only or ACE-plus-UTF-8.  Obviously, ACE-only is simpler
(but ACE-plus-UTF-8 is still an option that could be chosen for other
reasons).

> [ACE] is necessary for legacy support.  It is not necessary for moving
> forward, and will actively interfere with progress.

Huh?  Backward compatibility is most needed at the beginning, and the
need for it only decreases over time.  It is indeed necessary for moving
forward.  People are rarely willing to embrace a new thing if it causes
their existing stuff to stop working.

AMC