Subject: [idn] proposals and deadlines

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 12 Jul 2001 23:58:53 -0500

I'd like to clarify that he ACE-UTF8 coexistence protocol I am working on
will *not* be in tomorrow's pool. First of all, it requires the presence
of an ACE for backwards compatibility, and I want to wait for a decision
to be made on that first, without distracting from that process.
Secondarily, this is looking to be an extremely large and complex document
involving multiple delicate factors, and the science simply is not cooked
all of the way through as of yet. Thirdly, I just moved to the other side
of the country and have been without stable connectivity or systems for a
month, and its nowhere near completion. For all of those reasons, my
ACE-UTF8 coexistence protocol will not be in tomorrow's pool.

For those who care, here's an overview of the protocol as planned:

 1) Master files become UTF8.

 2) Servers convert and store ACE and UTF8 versions of IDNs together.

 3) Resolvers present two separate APIs: one for legacy names, one
    for IDNs.

 4) When an application calls the resolver, it uses the appropriate
    API as determined by the application protocol. EG, if an href=
    was encountered that had been encoded in ACE -- and if there
    were no external protocol demands from HTTP or the W3C or
    whoever -- then the legacy APIs would be used. Conversely,
    if the user entered an IDN into the URL input field and this
    were deemed legal through some other protocol that the browser
    was aware of (possibly an HTTP extension), then the extended
    API would be used.

 5) The UTF8 IDN resolver calls would generate messages with an
    EDNS extended label type. Legacy apps or resolvers would
    continue to use the legacy APIs and therefore legacy labels.

 6) Servers would answer EDNS-labelled queries with the raw UTF8
    data, and would answer the legacy queries with the ACE data.
    Any CNAME or PTR handling or any other labels would use this
    same rule (this means that it is important for clients to
    always use the extended API whenever they are allowed to do
    so according to protocol mandate).

 7) In those cases where the extended lookup failed (possibly due
    to a non-compliant server returning FORMERR or NOTIMP, or due
    to the resolver not supporting the extended API), the client
    would have to convert the UTF8 IDN to ACE, according to the
    mandate of the higher-layer protocol (EG, whatever the HTTP
    spec said to do whenever an IDN URL failed).

Those are the seven major elements to the protocol.

While this is not a proposal, a few rebuttals to "obvious" problems:

 A) Caches should not be a problem, as clients will have to be
    upgraded to adjust for application-specific UTF8 protocols,
    while nobody will be able to deploy UTF8 IDNs without using
    compatible servers and caches. IOW, as UTF8 IDNs are deployed
    the relevant infrastructure is upgraded along with it. This
    is a marginal cost increase over the upgrade burden for ACE
    usability (client transposition).

 B) The root server only needs to understand the extended label
    type, and will not need to store/use UTF8 until ICANN starts
    to assign IDN TLDs. The TLD servers will need to be upgraded
    to support IDN 3LDs, but this should happen enthusiastically
    in the relevant markets.

 C) The UTF8-to-ACE client fallback conversion should happen
    very infrequently. Protocols will have to support UTF8 IDNs
    themselves before *any* UTF8 IDNs are passed to resolvers,
    and applications will have to be upgraded for those protocols
    to work, and there will most likely have to be a way for a
    server to signify that it supports UTF8 explicitly (ESMTP
    extension, HTTP extension, etc.). For the immediate future,
    all of the queries will be ACE. Eventually, almost all of
    the queries will be UTF8. Fallback should be very rare.

 D) Label lengths are not increased. The hard limit of 255
    octets in the question section is unavoidable. Although it
    would be possible to lengthen the individual labels, it
    is not possible to change the maximum domain name length
    without changing the structure of the DNS message itself.
    This is possible (question section is null, or some other
    hack) but it would be disruptive.

    In fact, the use of EDNS extended label types reduces the
    IDN label length to a maximum of 62 characters.

 E) The coexistence "load" on the servers is unfortunate but it
    should not be preventative. For one thing, it will only 
    require 2x entries for every IDN, not for every domain name
    in every zone. Furthermore, the sooner this happens the
    better, since there will obviously be a smaller total number
    of entries this year than 10 years from now. By then, it
    should be possible to have made most systems speak UTF
    directly, such that support for ACE can be deprecated.

I've started scratching together the document, and am grateful for the
assistance of any interested parties.

Thanks

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 13 Jul 2001 16:58:12 -0500

"Adam M. Costello" wrote:
> 
> "Eric A. Hall" <[email-masked]> wrote:
> 
> >  3) Resolvers present two separate APIs: one for legacy names, one for
> >     IDNs.
> >
> >  5) The UTF8 IDN resolver calls would generate messages with an EDNS
> >     extended label type.
> >
> >  7) In those cases where the extended lookup failed, the client would
> >     have to convert the UTF8 IDN to ACE
> 
> It would not only ACE-encode IDNs in DNS queries, but also ACE-decode
> IDNs in DNS responses (because the application is using the UTF-8 API
> and expects a UTF-8 answer), right?
> 
> This looks like it would work, but if I'm writing a resolver, and I'm
> supposed to implement ACE-encode, ACE-decode, and EDNS, I'm probably
> going to think to myself, "Hmmm, if I just pretend that the EDNS lookup
> always fails, then I won't have to bother implementing EDNS, and the
> application will never know the difference."

Couple of things:

a) the resolver doesn't encode anything, it just passes whatever the
client application passed to it via one of the APIs

b) the fall-back mechanisms for UTF8-to-ACE would need to be specified for
each application. How ESMTP chooses to deal with a failed UTF8 lookup will
be different from how HTTP chooses to deal with it, and so forth.

c) I am going to assume that there will be libraries for most OSes that
provide ACE-encode and ACE-decode functions, since those are already
required for ACE presentation services

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 13 Jul 2001 21:06:19 +0000

AMC

---

From: "Edmon" <[email-masked]>
Date: Fri, 13 Jul 2001 18:03:16 -0400

From: "Adam M. Costello" <[email-masked]>
> application will never know the difference."

But if I am writing a DNS server, I'm probably going to think to myself,
"hmmm, if I pretend that ACE names will be answered by LDH records anyway
(which will be entered into by the zone administrator), then I won't have to
bother implementing ACE conversion and applications will never know the
difference.  I only need to make sure master files could be in UTF8 and I
can answer IDN/EDNS"

And if I am writing an http server (given that http will later support IDN
based on some of the works of this WG), I'm probably going to think to
myself... "hmmm, if I pretend that I dont know ACE and let the information
(ACE form) leak to the users and just implement UTF8, then I wont have to
bother implementing ACE conversion, and the application will never know the
difference."

These in turn (especially ACE leakage) would encourage the
application/resolver developer to eventually implement EDNS.

Edmon

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 13 Jul 2001 23:40:21 +0000

> application/resolver developer to eventually implement EDNS.

You'll have to walk me through the argument, because I just don't get
it.  DNS servers will be motivated to answer ACE queries correctly
because a lot of queries will be in ACE format, at least at the
beginning.  It doesn't matter whether they perform ACE conversion
on-the-fly, or at zone-loading time, or if they rely on administrators
to use preprocessing tools on the zone files, DNS servers will be
motivated to handle ACE queries somehow or other.

Therefore, resolvers and/or clients will have a choice between
attempting to use a DNS extension and falling back to ACE when it fails,
or just always using ACE.  By the previous argument, the latter option
will work anyway, and is easier, so it will probably be used in many
cases.  And that in turn will keep alive the motivation for DNS servers
to handle ACE queries correctly.

I'm not saying the DNS extension is bad, I'm just explaining why I
wouldn't expect it to become ubiquitous quickly.

> client application passed to it via one of the APIs

Okay, I thought you were saying that if the EDNS query fails, then the
resolver would perform ACE transformations on behalf of the application.
Now I guess you're saying that the resolver would simply return an
error, and the application would try again using the old resolver API
and the ACE transformations.

This just moves my argument up a layer.  Now the application has a
choice between trying the new API and falling back to ACE, or just
always using ACE with the old API.  The latter is simpler and works just
as well (and is more portable).

If you want to entice me to use the new resolver API, you're going to
have to offer more incentive.  Having the resolver perform the ACE
conversions for me would do it.

AMC

---

From: Martin Duerst <[email-masked]>
Date: Sat, 14 Jul 2001 11:35:06 +0900

At 21:11 01/07/13 -0500, Eric A. Hall wrote:
>You are certainly free to try embedding ACE into your data, displaying it
>properly in all situations, managing clipboard representations of it (did
>they want the visual representation or the mailto link?), educating users
>on its syntax, and so forth. Not having to do any of that should be
>compelling enough.

Very good point. Glad to see that more people see that ACE will
bring us more problems than it will solve.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Sat, 14 Jul 2001 11:26:44 +0900

> >  3) Resolvers present two separate APIs: one for legacy names, one for

I think this should read: one for legacy names and one for
IDNs (including legacy names).

Regards,   Martin.

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 13 Jul 2001 21:11:06 -0500

> always using ACE with the old API.  The latter is simpler and works
> just as well (and is more portable).

You avoid transliteration when using UTF8. What you see is what you get,
rather than a codified representation of it. For email messages, text
files, and just about every other form of user-manipulatable storage, this
is extremely important, and will become moreso over time.

Users don't want to write "ask jane@aq--gobbledygook for info". Users
don't want to see it in netstat output. Etc.

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 14 Jul 2001 20:50:57 +0000

"Eric A. Hall" <[email-masked]> wrote:

> > If you want to entice me to use the new resolver API, you're going
> > to have to offer more incentive. Having the resolver perform the ACE
> You are certainly free to try embedding ACE into your data, displaying
> it properly in all situations, managing clipboard representations of
> it (did they want the visual representation or the mailto link?),
> educating users on its syntax, and so forth. Not having to do any of
> that should be compelling enough.

I must have completely failed to make my argument understood, because
your response is unrelated to the point I was trying to make.  Let me
try again.

You said that in your proposed architecture, if the EDNS/UTF-8
lookup fails, then there is a fallback to DNS/ACE (at least in some
situations).  At first, I assumed that the fallback would happen inside
the resolver, but then you said it wouldn't, so that means the fallback
must happen inside the application, right?  In other words, you expect
these applications to include code that looks like this:

    start with a UTF-8 query
    look up using newAPI
    if that failed then begin
      ACE-encode the query
      look up using oldAPI
      ACE-decode the response
    end
    finish with a UTF-8 response

But they could instead use code that looks like this:

    start with a UTF-8 query
    ACE-encode the query
    look up using oldAPI
    ACE-decode the response
    finish with a UTF-8 response

The latter starts and ends with UTF-8 just like the former, but
is smaller and simpler.  You were recommending a fallback to be
used sometimes by some applications, and my point is that for those
applications it's easier to just use the "fallback" all the time.

If you want to help these applications to forget about ACE, you can do
that by hiding the fallback mechanism inside the resolver.  Then the
application can simply do:

    start with a UTF-8 query
    look up using newAPI
    finish with a UTF-8 response

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 14 Jul 2001 23:23:23 -0500

> must happen inside the application, right?

It has to, since each application (or more accurately, each usage) will
have its own behavioral considerations.

For example, a mail client may allow users to specify the SMTP and POP3
servers by using UTF8 IDNs. This doesn't affect the protocol operations,
but instead only affects the lookup of the server's IP address. A fallback
to ACE can be expected for this lookup operation, but this doesn't affect
the protocol, and the resulting ACE IDN would not need to be rendered by
the client anywhere.

Conversely, the same mailer may keep an address book containing UTF8 IDNs.
Without some sort of extension to SMTP, those addresses will have to be
converted to ACE whenever they are written into the envelope or whenever
they are written into the message headers. If a UTF8-envelope extension
mechanism is created, but it fails to negotiate, then presumably LDH rules
will still apply, so conversion will have to take place in the
applicaiton. In this example, the option negotiation has failed, but no
lookups have failed.

Assuming that the option negotiation succeeds, the SMTP server will
eventually need to log the connection for a Received header. Obviously it
speaks UTF8 if the option was successfully negotiated, so it could try to
use UTF8 for the PTR lookup, although the default behavior should be to
use LDH (ACE), using the legacy API calls. We can also assume that if the
EDNS PTR lookup fails, then it would have to fallback to ACE.

Once mail arrives, the recipient system may need to convert UTF8 envelopes
back to ACE, or if the local message routing maps are using UTF8 IDNs,
vice versa.

That is four conversion scenarios: 

   o A RR lookup failure
   o extended-envelope-option negotiation failure
   o PTR RR lookup failure
   o message routing lookup

As can be seen from the above, each usage will need its own rules, only
two of which use DNS, and two of them have to be defined by an external
protocol (UTF8 in Received headers). As such, the applications have to
manage the conversion process.

> In other words, you expect these applications to include code that
> looks like this:
> 
>     start with a UTF-8 query
>     look up using newAPI

No, I am thinking:

      start with a UTF-8 query
      look up using newAPI
      if that failed then do whatever they are supposed to do next

> applications it's easier to just use the "fallback" all the time.

I think this is a really creative idea, but as illustrated, the onus of
conversion is on the applications and their protocols just as much as it
is on DNS. Even though we could do this for the DNS parts, the
applications will still have to perform conversions. This would be
somewhat easier on the server infrastructure, but it wouldn't save the
applications any labor, and would simply move the server's burden to the
resolver's side of the fence.

There is also a technical consideration here, in that this approach does
not provide a way to distinguish between spurious 8bit data and IDNs. An
older application that encounters an 8bit domain name may pass it into the
stream when it shouldn't (there is at least one application out there now
that passes UTF8 into the DNS). With the dual-message model, at least we
can tell that this is an old client and the malformed data can be compared
to LDH and rejected, while EDNS messages should only be coming from
modernized applications that (hopefully) follow the normalization rules.
This flag is required for Internet stability, IMO.

More importantly (to me anyway), this approach does not provide a
migration path to a UTF8-aware DNS infrastructure. We would still have to
store the raw zone data in ACE format, we would have to invent ACE
editors, we would have to educate the administrative masses on ACE
encoding syntax, and there would be less impetus to move to UTF8
end-to-end in higher-layer protocols. We should be helping to make the
network truly internationalized with this work, IMHO.

>     finish with a UTF-8 response

The unique message is the flag; the API just provides a separate access
point for the message.

---

From: "Adam M. Costello" <[email-masked]>
Date: Sun, 15 Jul 2001 06:00:09 +0000

"Eric A. Hall" <[email-masked]> wrote:

> Even though we could do this [put ACE conversions in the resolver]
> for the DNS parts, the applications will still have to perform
> conversions.

Are you resigned to have applications contain ACE conversion code
forever?  If so, why should it bother you if DNS likewise uses ACE
forever?  On the other hand, if you'd like to see applications evolve
away from ACE (by upgrading old protocol formats, or replacing old
protocols by new protocols), then you need to give them a way to avoid
ACE in DNS lookups too, and that means giving them a UTF-8 resolver
interface that makes a best effort, not one that gives up too easily and
expects the application to do the fallback to ACE.

> This would be somewhat easier on the server infrastructure, but it
> wouldn't save the applications any labor...

Yes it would.  There will definitely be some applications that want a
UTF-8 response to a UTF-8 query.  If newAPI tries EDNS/UTF-8 and gives
up on failure, then these applications can do their lookup in either of
two ways:

    look up using newAPI
      ACE-encode query
      look up using oldAPI
      ACE-decode response
    end

or:

    ACE-encode query
    look up using oldAPI
    ACE-decode response

The latter is less code to write and maintain.  I can't see a reason
why any programmer would opt for the more complex method given those
two choices.  I also can't think of any scenario where an EDNS/UTF-8
lookup failure should not be followed by an attempt with DNS/ACE.  Which
means I can't think of a single scenario where a programmer would want
to use newAPI at all.  That's my point.  I don't see an incentive for a
programmer to call newAPI if newAPI only makes a half-hearted attempt.
If it makes a best-effort attempt, then it becomes attractive.

> this approach does not provide a migration path to a UTF8-aware DNS
> infrastructure.  We would still have to store the raw zone data in ACE
> format, we would have to invent ACE editors, we would have to educate
> the administrative masses on ACE encoding syntax, and there would be
> less impetus to move to UTF8 end-to-end in higher-layer protocols.

Zone files can be stored in the local charset, or UTF-8, and the DNS
server can do ACE encoding when it starts up and reads the file.  No
need for ACE editors, or for administrators to even see ACE.

If the UTF-8 resolver interface is best-effort, then there is greater
impetus to move to UTF-8 in higher layers, not less, because then
applications can be completely free of ACE.  If the resolver interface
is half-hearted, then applications will always have to do their own
fallback, and will therefore never be free of ACE, so why should they
bother migrating the upper layers away from ACE?

> There is also a technical consideration here, in that this approach
> does not provide a way to distinguish between spurious 8bit data and
> IDNs.  An older application that encounters an 8bit domain name may
> pass it into the stream when it shouldn't (there is at least one
> application out there now that passes UTF8 into the DNS).  With the
> dual-message model, at least we can tell that this is an old client
> and the malformed data can be compared to LDH and rejected, while
> EDNS messages should only be coming from modernized applications that
> (hopefully) follow the normalization rules.

I don't see how this conflicts with anything I've proposed.  I have not
proposed putting 8-bit data into old-style DNS messages.

AMC

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 15 Jul 2001 09:13:26 -0700

At 10:59 AM -0500 7/15/01, Eric A. Hall wrote:
>Paul Hoffman / IMC wrote:
>>
>>  At 10:09 AM -0500 7/15/01, Eric A. Hall wrote:
>>  >No, only until ACE can be deprecated. This will happen on a per-app
>>  >basis where it happens,
>>
>>  In the IETF, we are not in the habit of saying "features of an
>>  earlier version of this protocol that worked fine will be broken in
>>  this new version". Can you list the advantages to the end users that
>>  deprecating ACE will have?
>
>For specific applications, there are natural benefits. Perhaps HTML v99
>will say "all URLs must be UTF8" just so state doesn't have to be
>maintained between browsers and servers.

Sorry, but that doesn't answer the question, and it instead raises a 
straw man that is untrue. What would be the advantage to end users of 
HTML v99 not interoperating with HTML v98? Also, IDNA does not 
require any state to be maintained between browsers and servers.

>  > >  or with entirely new applications that avoid using ACE
>>  >altogether.
>>
>>  That's like saying "entirely new applications that avoid using the
>  > DNS altogether". Again, please list the advantages to the end user of
>  > creating a protocol that doesn't use the same DNS as the rest of the
>  > IETF protocols.
>
>Not all applications come out of the IETF.

Applications don't come out of the IETF: protocols do. None of this 
discussion is about creating new applications.

>Whose to say that something
>won't use UTF8 solely? Maybe winsock v99 will only provide a wide API.

In the IETF's IDN Working Group, we are discussing IETF protocols. 
Anyone else is free to make whatever other protocols they want, and 
those protocols might or might not use some of the IETF's work.

So, to be clearer: please list the advantages to the end user of the 
IETF creating a protocol that doesn't use the same DNS as the rest of 
the IETF protocols.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 15 Jul 2001 08:44:24 -0700

At 10:09 AM -0500 7/15/01, Eric A. Hall wrote:
>No, only until ACE can be deprecated. This will happen on a per-app basis
>where it happens,

--Paul Hoffman, Director

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 10:09:10 -0500

"Adam M. Costello" wrote:
> forever?

No, only until ACE can be deprecated. This will happen on a per-app basis
where it happens, or with entirely new applications that avoid using ACE
altogether. But when it happens, the app should end up with a native UTF8
environment, rather than being stuck with ACE.

> On the other hand, if you'd like to see applications evolve away
> from ACE (by upgrading old protocol formats, or replacing old
> interface that makes a best effort, not one that gives up too easily
> and expects the application to do the fallback to ACE.

The application *MUST* do the conversion, since protocol operations may
require it.

The example I gave with UTF8 envelopes shows this. If the envelope is
allowing UTF8 IDNs, then it also makes sense that the left-hand-side of
the address (user@) would be encoded in UTF8 rather than using MIME
charset tags. If the option-negotiation succeeded, the mailer would send
UTF8@UTF8, but if not then it would have to send MIME@ACE. Simply put,
some extensions will affect multiple elements and not just DNS, and the
application is the only thing that will know what to do.

This is also the case with DNS lookups. If the POP3/SMTP mailer is
configured with an IDN for the mail server -- and if that lookup fails
(resolver pukes on API, server returns FORMERR, or whatever) -- then the
client knows that it does not have access to an UTF8 environment, and it
can use this information to make subsequent decisions for the remainder of
that session. It will know that it should not attempt to negotiate
UTF8-envelope, for example.

Another example would be X.509 certificates; the resolver could break
things if it returned a UTF8 IDN that didn't match the domain name
provided with the certificate. This would also be important for HTTP where
negotiation would affect URL inputs and redirect outputs; the server must
not send UTF8 IDNs to a client that is not capable of dealing with them if
the client is stuck with an old resolver.

In any event, the application *MUST* perform the conversion, since it is
the only thing that knows all of the ramifications from failure.

You are also working on the assumption that fallback will happen
frequently. I posit that this is not true, for two basic reasons:

  1) If an initial IDN lookup (or test binding) fails, the app
     finds out immediately, and can use ACE for the duration
     of the session. There would be no need for the app to
     negotiate over extended features, no need for the app to
     try issuing any additional UTF8 IDN queries, etc.

  2) Server failure is not to be expected. Nobody should be
     providing IDNs that are not accessible via the IDN
     protocol. What would be the point? Maybe some domain name
     speculators who have not upgraded their servers will
     cause some failures, but legitimate orgs who are trying
     to embrace IDNs will upgrade their infrastructure
     accordingly. Cumulatively, there should be very few
     server-side failures. [Further, where this does happen,
     the client will find out about it and can stop issuing
     UTF8 IDN queries for that session.]

Taken together, very few DNS-level fallback queries should occur. There is
no great penalty. The only measurable penalty in this model is that zone
and cache sizes increase, and this probably isn't much of one (my research
for another effort indicates that memory consumption is not an issue for
the vast majority of servers and/or caches).

> proposed putting 8-bit data into old-style DNS messages.

DNS allows any 8-bit data. There are already implementations that pass
8-bit data in domain names, and the only reason they don't do it
frequently is because IDNs aren't commonly found in the wild yet. Once
IDNs start showing up, these legacy apps will pass 8-bit names more
frequently. It can be expected that at least a few legacy apps will pass
IDNs as rendered (ISO-8859-nn, Windows-nnnn, MacRoman, whatever) instead
of converting them to UTF8 first. At the very least, they will *not* run
them thruogh nameprep first.

The use of an alternative API provides a way to segregate the queries at
the source (the calling application). If 8-bit data comes into the system
using the legacy APIs, it will also use the legacy message, and can be
compared at the zone authoritatively. Conversely, if it comes into the
system using the new APIs and EDNS message, then we are safe in assuming
that the data is UTF8, and that it has been generated according to
application-specific rules (rather than coming from spurious inputs), and
that it has been run through nameprep. In order for this to work, the
messages must somehow be flagged at the server.

Furthermore, this flag must also exist in order for the server to
intelligently handle IDNs in ASCII names. EG, an organization that has
obtained an IDN may want to setup a CNAME for the old ASCII name that
points to an IDN, and the server needs a way to know that it is safe to
return the IDN in the CNAME response. A flag of some kind is required for
this. Reconverting ACE at the client side will not work for all RRs.

The use of a new API allows IDNs to be flagged, but it is the use of a new
and different message that is what's really important. Your proposal does
not provide this differentiator, and allows corrupt data to enter the
query chain on the same footing as legitimate data.

Furthermore, as pointed out earlier, there is no great "I'm always
failing!" penalty, since applications discover failure quickly, and use
this information to make subsequent decisions. If they know they don't
have access to UTF8 IDNs, they will not negotiate the use of UTF8
protocol-specific extensions, and will not generate UTF8 queries.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 11:36:50 -0500

> maintained between browsers and servers.

Let me expand on this by saying that any application which supports
version negotiation is a candidate for UTF8 migration. NFS v99 for example
may benefit from an UTF8-only approach, and since NFS supports version
negotiation, this is feasible to implement (relying on version numbers for
fallback negotiation). LDAP v99 would be another.

The IETF is unfortunately not in the habit of mandating versioning in
their application protocols. But wherever it exists, migration is feasible
and even simple, and will offer natural benefits. Where users need ACE,
they can stay with the earlier versions.

For protocols that do not support in-line versioning, once a critical mass
is achieved and sustained (say 10 or even 20 years from now), I expect
that many implementations will default to UTF8, even if the protocol
doesn't. Look at TELNET: there aren't very many implmentations that
support Go-Ahead anymore (even though it is mandated by the protocol),
simply because there isn't much of a need for it anymore. Why would UTF8
be any different? If every SMTP client and server on the planet supports
UTF8, new implementations may implement just that, and freak out when
negotiation fails (like most of the TELNET implementations do when SGA
fails).

> IETF creating a protocol that doesn't use the same DNS as the rest of

It's not a different DNS. I don't know why you're saying that. The same
authoritative servers and the same zone data is being used. What is
different about it?

The advantages may or may not be compelling at this point. It is most
likely to simply be a matter of pragmatism when it happens (not "if" but
"when"). New protocols are already required to use ISO-10646 for charsets,
so how long will it be before the IAB simply decrees that UTF8 is the
preferred encoding as well? The ISO-10646 decree certainly wasn't driven
by user benefit, why would this one?

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 10:59:02 -0500

> creating a protocol that doesn't use the same DNS as the rest of the

Not all applications come out of the IETF. Whose to say that something
won't use UTF8 solely? Maybe winsock v99 will only provide a wide API.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 15 Jul 2001 10:31:48 -0700

At 11:36 AM -0500 7/15/01, Eric A. Hall wrote:
>version negotiation is a candidate for UTF8 migration.

Not unless there is a possible need for that migration. And, again, 
you have not shown even a single example of that possible need.

>doesn't.

Ignoring backwards compatibility for no advantage is not a common 
trait among application writers.

>authoritative servers and the same zone data is being used. What is

It uses a different part of the DNS protocol, namely the marking of 
the UTF-8 requests and responses. We are talking about *protocols* 
here, not file formats.

>The advantages may or may not be compelling at this point.

Well, you haven't given a single instance of one that might be 
compelling, so it is more accurate to simply say "there are no known 
advantages now".

>  It is most
>"when").

So, although there is no need now, you firmly predict that there will 
be and therefore we have to act now with a protocol. How do you know 
that your protocol will in fact meet the future need? There are lots 
of different proposals for how to do marked UTF-8 in DNS; why are you 
so sure that your marking plan will work for the as-yet-unknown needs?

>  New protocols are already required to use ISO-10646 for charsets,

That statement is incorrect. BCP 18 says that protocols that use 
character data are required to at mark the character data with 
charsets, and at least be able to use the UTF-8 charset, but are 
explicitly allowed to use other charsets. IDNA does not use character 
data: it encodes labels.

>preferred encoding as well?

There are IAB members who are following this list; maybe you should 
let them answer the question instead of answering it for them.

>  The ISO-10646 decree certainly wasn't driven

BCP 18 most certainly was driven by user benefit; to say otherwise is 
fairly insulting to the many people who worked on it. Mandating that 
any protocol that can accept charsets MUST accept UTF-8 greatly 
increases interoperability (which benefits end users) *and* makes 
sure that all applications use at least one charset that uses all of 
the scripts in the world (which benefits end users outside 
English-speaking countries).

Please re-read BCP 18 before you use it in the context of this discussion.

--Paul Hoffman, Director

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 12:56:34 -0500

> you have not shown even a single example of that possible need.

I do not assume that I can predict all possible uses of inline UTF8 data.
What I do know is that people will not want ACE in their soup.

If you are only wanting one possible example, think of a protocol for
replicating mail-routing maps. Wouldn't it be beneficial for the admins of
the affected mail servers to have a common format for that data? Wouldn't
it make sense to use UTF8 for the IDNs in the router maps, given its
usability with multiple languages? Furthermore, wouldn't it be beneficial
if the data and the IDN protocol both used the same encodings? How would
it benefit anybody to manage ACE-encoded lists of mail domains? How would
it benefit anybody to replicate a UTF8 list that had to be converted into
ACE for pre-processing? If UTF8 were available for the list contents and
the pre-processing, wouldn't it that be beneficial to the users?

Clearly there are natural benefits to using UTF8 everywhere. Providing
access to UTF8 (whether with this mechanism or some other) allows for
greater levels of functionality, servicability and usability.

> >It's not a different DNS. I don't know why you're saying that. The
> >same authoritative servers and the same zone data is being used.
> >What is different about it?
> 

By this argument, every RR represents a different DNS.

> That statement is incorrect. BCP 18 says ...

I was referring to RFC2130.

As to the other comments following this, all I will say is that I was
referring to the commentary provided in RFC2130. I was not intending to
insult anybody, and if anybody was offended by any misrepresentation or
misintepretation, I apologize for the brush.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 15 Jul 2001 12:14:52 -0700

>I do not assume that I can predict all possible uses of inline UTF8 data.

Nor were you ever asked to.

>replicating mail-routing maps.

That's rather obscure, but let's go with it.

>  Wouldn't it be beneficial for the admins of
>the affected mail servers to have a common format for that data?

Yes.

>  Wouldn't
>usability with multiple languages?

Not necessarily. The format of the router maps could be plain text 
with UTF-8, plain text with ACE, ASN.1, and so on. You are mixing up 
document formats with DNS protocol.

>  Furthermore, wouldn't it be beneficial
>if the data and the IDN protocol both used the same encodings?

Probably, and that would be ACE.

>  How would
>it benefit anybody to manage ACE-encoded lists of mail domains?

Because ACE is being used as the host names that are being routed. 
You can't say "UTF-8 is needed for this protocol" unless UTF-8 is 
being used in the DNS.

>  How would
>ACE for pre-processing?

The list does not have to be UTF-8: it could be in a wide variety of formats.

>  If UTF8 were available for the list contents and

No more so than ACE, or ASN.1, or whatever.

>Clearly there are natural benefits to using UTF8 everywhere.

Sure, if you ignore most existing protocols and the severe problems 
during the wholesale upgrade of DNS servers. Have you now shifted 
from "ACE then UTF-8" to "never ACE"?

>  Providing

You still haven't shown that. You gave one example in which a 
document format might be in UTF-8, but could just as easily been in 
any other format.

Do you have other, hopefully better, examples?
>I was referring to RFC2130.

RFC 2130 is not a "decree": it is an Informational RFC that is simply 
a report from a workshop. It does not "require" anything, and it is 
incorrect to say that it does. BCP 18, which came out after RFC 2130, 
is a synthesis of the ideas from the workshop. BCP 18 has 
requirements, but they do not apply to IDNA because IDNA does not 
deal with text content.

--Paul Hoffman, Director

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 14:49:42 -0500

> document formats with DNS protocol.

Yes I am, because in this case, the document will be edited and its
contents reused later in SMTP and DNS. If we are encouraged to use UTF8
for the document contents by various OS and protocol issues, then we
should also be using UTF8 for the DNS parts.

> Probably, and that would be ACE.

Once a UTF8 format exists, why not leverage it? It would not be unheard of
for a new protocol to make such a demand. HTML 4.0 has different technical
demands than HTML 3.2, for example.

> during the wholesale upgrade of DNS servers.

What severe problems?

Is the need to upgrade ACE clients (and servers, unless you consider
manually editing ACE gobbledygook in zone files to be a wonderful feature)
also a severe problem?

> Have you now shifted

For new applications, there is absolutely the possibility that they will
be able to only specify the use of UTF8. This is a good thing, as new
applications can be written for international markets from the beginning,
and UTF8 is becoming very common. UTF8-in-DNS is an enabler. Furthermore I
would argue that it is absolutely critical to the next twenty years of
protocol work.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 15 Jul 2001 14:05:38 -0700

>What severe problems?

If any DNS server in the search path for a name has not been updated, 
or if the user's resolver has not been updated, the query will fail. 
That seems pretty bad, yes?

>manually editing ACE gobbledygook in zone files to be a wonderful feature)

No. Applications will need to be upgraded no matter what. Further, 
DNS servers do *not* need to be upgraded: the editing interface to 
the host file does. That is pretty trivial: a single script that 
converts the native-charset file (which does not have to be UTF-8: it 
can be any native charset) to the host file. It took me about 5 
minutes to write mine in Perl (using an existing ACE<=>UTF-8 routine).

>be able to only specify the use of UTF8.

One more time: we are talking about protocols, not applications. And, 
even then "able to" is completely different than "needs to". Where is 
the need here? How will balkanizing the DNS help end users?

>  This is a good thing, as new

Because...?

You still haven't given any examples of why it is needed, only one 
where it would make writing an application only slightly easier, but 
only if we already had UTF-8 names. Please reconsider your arguments 
about an upgrade path.

--Paul Hoffman, Director

---

From: "Adam M. Costello" <[email-masked]>
Date: Sun, 15 Jul 2001 23:25:00 +0000

"Eric A. Hall" <[email-masked]> wrote:

> basis where it happens, or with entirely new applications that avoid
> using ACE altogether.  But when it happens, the app should end up with
> a native UTF8 environment, rather than being stuck with ACE.

Okay, now let's think about this hypothetical new application that
uses UTF-8 natively.  It uses UTF-8 to talk to its peers, and UTF-8 to
interact with the user, data files, etc.  Absolutely everything it ever
does with domain names uses UTF-8.  Except one thing: DNS lookups, which
sometimes use UTF-8 and sometimes use ACE.  Why?  Because the UTF-8
resolver interface you have postulated gives up when EDNS fails.  Our
application has no control over whether the local DNS server supports
EDNS, or whether any particular caching DNS server between the local
server and the target supports EDNS.  Therefore, the application will
need to do its own fallback to the old resolver interface, which means
the application will need to be able to perform ACE encoding, so it is
not completely free of ACE.

But your stated goal is to avoid having the application stuck with ACE.
If there were a UTF-8 resolver interface that would make a best effort,
instead of giving up when EDNS is unavailable, then the application
could truly eliminate all traces of ACE within itself.  Isn't that what
you want?

> In any event, the application *MUST* perform the conversion, since it
> is the only thing that knows all of the ramifications from failure.

I'm not trying to assert that providing a best-effort UTF-8 resolver
interface will magically free applications from doing ACE conversions.
Of course it won't, because there are many reasons unrelated to DNS that
applications might need to use ACE.  But even after applications manage
to eliminate all those other reasons, they still cannot completely free
themselves of ACE unless this best-effort UTF-8 resolver interface
exists.

> frequently.

No I am not.  I am working under the assumption that, for an application
programmer, it doesn't matter whether the fallback happens frequently
or rarely, either way the programmer still needs to write code for
doing it.  The only way to avoid that code in the application is if the
fallback happens *never* (that is, never in the application, but instead
in the resolver).

> Your proposal does not provide this differentiator, and allows corrupt
> data to enter the query chain on the same footing as legitimate data.

What do you think my proposal is?  I have not suggested that calls to
the old resolver interface should ever perform EDNS queries or ACE
transformations; the old interface should continue to do exactly what it
does today.  I have not suggested that DNS servers should ever put 8-bit
data into old-style responses.  I have not suggested that resolvers
falling back from EDNS to DNS should ever put 8-bit data into old-style
queries.  I don't see how I'm allowing corrupt data to enter anything.
If you still think I am, you'll have to walk me through a concrete
scenario.

> If [applications] know they don't have access to UTF8 IDNs, they will
> not negotiate the use of UTF8 protocol-specific extensions, and will
> not generate UTF8 queries.

Just because my local DNS server doesn't support EDNS, that doesn't mean
I should speak ACE to my application-layer peer.  If that peer and I
both support UTF-8, then we should speak UTF-8 to each other, regardless
of the capabilities of my name server, in order to reduce the risk of
ACE leakage.

AMC

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 16 Jul 2001 00:32:14 +0000

Martin Duerst <[email-masked]> wrote:

> In HTML 4.0, there is a provision that says that URIs that contain
> non-ASCII characters should be interpreted based on conversion to
> UTF-8
>
> The Web and the W3C took RFC 2277 seriously.  We are ready for UTF-8,
> we don't need ACE at all.

Not quite.  HTML 4.01 still says that URIs are defined by RFC 2396.  So
consider href="http://IDN.com/foo/".  The provision you mention will
convert it to http://%HH%HH%HH.com/foo/, but that's not a syntactically
valid URI according to RFC 2396, which says the host part can contain
only letters, digits, hyphens, and dots, not percent signs.  And nowhere
does the HTML spec say that the %HH encoding should not be used on the
host part (or should be undone before the name is looked up).

AMC

---

From: Martin Duerst <[email-masked]>
Date: Mon, 16 Jul 2001 09:11:58 +0900

>Sorry, but that doesn't answer the question, and it instead raises a straw 
>man that is untrue. What would be the advantage to end users of HTML v99 
>not interoperating with HTML v98? Also, IDNA does not require any state to 
>be maintained between browsers and servers.

Talking about HTML v99 or HTML v98 is a complete strawman indeed.
But let's talk about HTML 4.0. In HTML 4.0, there is a provision that
says that URIs that contain non-ASCII characters should be interpreted
based on conversion to UTF-8 (which is a slight, but very important
difference from 'all URIs must be UTF-8). And there is no old HTML
that contains IDNs.

And the newest versions of the main browsers support this convention.
And there is no need for state to be maintained between browsers and
servers; that's not what the HTTP protocol is about.

The Web and the W3C took RFC 2277 seriously. We are ready for UTF-8,
we don't need ACE at all. For HTML and the Web (also including XML,
XML Schema, XLink,...), ACE is useless baggage.

Regards,   Martin.

#-#-#  Martin J. Du"rst, I18N Activity Lead, World Wide Web Consortium
#-#-#  mailto:[email-masked]   http://www.w3.org/People/D%C3%BCrst

---

From: Martin Duerst <[email-masked]>
Date: Mon, 16 Jul 2001 09:28:49 +0900

At 14:05 01/07/15 -0700, Paul Hoffman / IMC wrote:

>[T]he editing interface to the host file does. That is pretty trivial: a 
>single script that converts the native-charset file (which does not have 
>to be UTF-8: it can be any native charset) to the host file. It took me 
>about 5 minutes to write mine in Perl (using an existing ACE<=>UTF-8 routine).

That's five minutes for each different format. And then that's a few seconds
every time a conversion is needed. Nobody in the US would want to do that
for US-ASCII. For US-ASCII, the widest variety of Internet protocols is
designed so that no such additional conversion step is necessary.
And for those protocols and formats where it's not the case, everybody
complains about, and for good reason. Why is it so difficult for somebody
like you to understand that the rest of the world would also benefit
greatly from this, and that it's a bad idea to deny them such benefits?

>>>  from "ACE then UTF-8" to "never ACE"?

I for one always claimed that ACE is a big step backwards.

Regards,   Martin.

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 16 Jul 2001 00:39:00 +0000

At 14:05 01/07/15 -0700, Paul Hoffman / IMC wrote:

> [T]he editing interface to the host file does.  That is pretty
> trivial: a single script that converts the native-charset file (which
> does not have to be UTF-8: it can be any native charset) to the host
> file.  It took me about 5 minutes to write mine in Perl (using an
> existing ACE<=>UTF-8 routine).

Martin Duerst <[email-masked]> replied:

> That's five minutes for each different format.  And then that's a few
> seconds every time a conversion is needed.

Okay, anyone who doesn't like that method can get a DNS server that
reads UTF-8 zone files directly (I have no doubt that such servers will
appear) and use their favorite UTF-8 capable editor to edit it.  The
encoding used by the server on the wire is orthogonal to the zone file
encoding.

AMC

---

From: Martin Duerst <[email-masked]>
Date: Mon, 16 Jul 2001 10:37:52 +0900

At 00:39 01/07/16 +0000, Adam M. Costello wrote:

>appear)

I also have little doubt this should and will happen. But the current
IDNA draft explicitly specifies that zone files must be in ACE, and
therefore prohibits such implementations.

>and use their favorite UTF-8 capable editor to edit it.  The

Yes, it is. But zone files are one of the easiest places to address
in that respect. Domain names turn up in many more places. How do you
solve the problem for these?

Regards,  Martin.

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 16 Jul 2001 01:50:00 +0000

Martin Duerst <[email-masked]> wrote:

> But the current IDNA draft explicitly specifies that zone files must
> be in ACE, and therefore prohibits such implementations.

Well that's just silly.  Paul and Patrick, the IDNA spec should not say
that master files must use ACE, it should say that DNS servers must
accept master files that use ACE and may also accept master files that
use other encodings, like UTF-8.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 20:39:54 -0500

> That seems pretty bad, yes?

If that were actually the case, yes. The dual-mode proposal allows for the
use of ACE wherever it is required. It will work just as well as the
ACE-only approach.

The difference comes from what the user gets when they commit to the
technology and upgrade. In the ACE-only camp, they get transliteration. In
the dual-mode camp, they get transliteration and they also get direct
access to UTF8. The ACE approach restricts *ALL FUTURE ACTIVITY* to the
boat anchor that is ASCII, while the dual-mode approach lays a foundation
for a new breed of internationalized applications.

How you can oppose the obvious choice here is completely beyond me. What
exactly is the problem that this WG is trying to solve? Aren't we trying
to internationalize domain names so that apps can enjoy them? And yet you
want us to make it even *HARDER* for developers by mandating that they may
only use encrypted strings that show up nowhere else in the entire
computing universe. Outrageous.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 21:10:03 -0500

> use other encodings, like UTF-8.

The format of a master file needs to be agreed on. This is required for
DNS to operate properly.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 21:22:11 -0500

"Adam M. Costello" wrote:
> > using ACE altogether.  But when it happens, the app should end up
> > with a native UTF8 environment, rather than being stuck with ACE.
> 
> does with domain names uses UTF-8.  Except one thing: DNS lookups,
> which sometimes use UTF-8 and sometimes use ACE.

This hypothetical application would be written to only use UTF8. After a
certain point, it wouldn't make sense to continue writing apps for an
outdated encoding. This wouldn't happen immediately, but once there was a
sufficient installed base, why not? EG, how many new apps are being
written for 7-E-1 parity, or for punchcard storage? After some point, we
should assume that the mere presence of UTF8-encoding will result in some
new application using UTF8 alone, and not supporting a fallback to ACE.

> > Your proposal does not provide this differentiator, and allows
> > corrupt data to enter the query chain on the same footing as
> > legitimate data.
> 
> What do you think my proposal is?

  | Subject: Re: [idn] proposals and deadlines
  |    Date: Sun, 15 Jul 2001 06:00:09 +0000
  |    From: "Adam M. Costello" <[email-masked]>
  |      To: IDN <[email-masked]>

  |     ACE-encode query
  |     look up using oldAPI
  |     ACE-decode response
  | 
  | The latter is less code to write and maintain. I can't see a
  | reason why any programmer would opt for the more complex method
  | given those two choices.  I also can't think of any scenario
  | where an EDNS/UTF-8 lookup failure should not be followed by an
  | attempt with DNS/ACE.  Which means I can't think of a single
  | scenario where a programmer would want to use newAPI at all. 

When you put conversion in the resolver, the resolver will attempt to
convert data that should not be converted.

For example, there are currently applications on the market that will try
to process 8-bit data that is given to them. Because these applications
were written before nameprep, before the IDNS draft, and the other new
rules (which may or may not come about), these applications are guaranteed
to make wrong assumptions.

If this data is passed to oldAPI, and then the resolver says "oh goodie,
UTF8!", converts it to ACE and then pushes it into the query chain,
unexpected results will absolutely occur. Detrimental effects (such as
positive matches for wrong data) are likely to occur. This is particularly
true with applications that try to work with charset data from
locale-based encodings (like 8859-15) or OS-specific encodings (like
Windows-1252 or MacRoman).

Furthermore, the resolver must not perform the conversion because it
doesn't know what the application is going to do with the data. If a
resolver answered in the affirmative to a probe-test, then the application
may wrongly perform a subsequent task (such as an authentication request).

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 16 Jul 2001 02:29:42 +0000

"Eric A. Hall" <[email-masked]> wrote:

> The format of a master file needs to be agreed on.  This is required
> for DNS to operate properly.

Why?  Zone transfers use the regular binary DNS message format, not the
text master file format.  As far as I know, having a standard master
file format is not necessary for DNS to operate properly, but is a
convenience for DNS administrators because it allows them to switch to
a different server implementation without having to translate their
zone files.  Some server implementations (like BIND) support extensions
to the master file format (like $GENERATE), and DNS administrators who
use such extensions assume the risk that if they switch to a different
server implementation, those extensions might not work.  The same would
be true of an extension for supporting UTF-8.  The extension for UTF-8
might be sufficiently useful that it becomes a standard extension, like
$TTL added by RFC 2308.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 15 Jul 2001 22:04:37 -0500

"Adam M. Costello" wrote:
> zone files.

The master file is separate from the database. While the use of a common
master file format does make switching convenient, the primary purpose is
to ensure that a zone *can* be moved. People who are moving servers or
recovering from a crash or whatever will email the master file to some
other admin or server, and will expect it to function.

RFC 1035 discusses the master file in several contexts -- not just as the
database -- meaning that it's an integrated part of the spec.

Finally, RFC 1034 says that zone transfers are the "preferred" method but
not the only method to move zones around:

  | 4.3.5. Zone maintenance and transfers
  | 
  | Part of the job of a zone administrator is to maintain the zones
  | at all of the name servers which are authoritative for the zone.
  | When the inevitable changes are made, they must be distributed
  | to all of the name servers.  While this distribution can be
  | accomplished using FTP or some other ad hoc procedure, the
  | preferred method is the zone transfer part of the DNS protocol.

It is legal to use FTP or rdist or anything else to move master files
around. Their format must be known for this to work.

My thinking is that master files for IDNs must be in UTF-8. This ensures
that an authoritative server cannot be upgraded without all of the servers
being upgraded simultaneously. Since all of the servers have to speak
UTF-8, they all need to be upgraded at roughly the same time.

Note that this MUST would be hollow; all of the servers could still run in
cripple-mode by using ACE with the legacy APIs.

But I am sure to get flamed for this anyway.

---

From: Martin Duerst <[email-masked]>
Date: Mon, 16 Jul 2001 12:30:29 +0900

At 00:32 01/07/16 +0000, Adam M. Costello wrote:
>only letters, digits, hyphens, and dots, not percent signs.

Yes. I already have written a draft to do the necessary update.
Please see http://search.ietf.org/internet-drafts/draft-ietf-idn-uri-00.txt.
And the philosophy of URIs is that you don't check more than necessary,
and many applications do that.

>And nowhere
>host part

Yes, on purpose. The U in URI stands for Uniform (among else).

>(or should be undone before the name is looked up).

That's not the business of the HTML spec, obviously.

Regards,   Martin.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 15 Jul 2001 20:24:49 -0700

>The format of a master file needs to be agreed on. This is required for

There is a *large* amount of disagreement here. The fact that the 
zone file was made standard by RFC 1034/1035 does not mean it is 
needed. Nothing exchanges zone files, so there is no interoperational 
impact.

Further, even if you wanted to have a standard zone file format, you 
can create precursors to the zone file in any format. It could be a 
text file in UTF-8, a text file in GB, a bunch of dialog boxes, an 
SQL database, and so on. (All of those are already in use today.) 
Quite frankly, I doubt that many non-BIND DNS systems create 
compliant zone files from their much-friendlier input formats unless 
you ask them to.

--Paul Hoffman, Director

---

From: "D. J. Bernstein" <[email-masked]>
Date: 16 Jul 2001 03:32:23 -0000

Paul Hoffman / IMC writes:
> Can you list the advantages to the end users that deprecating ACE will have?

I already answered this in May:

: Consider a programmer writing network software five years from now, or
: ten years from now.
:
: He'll be working with text in UTF-8. But wait, you say, he has to
: convert to some screwy ACE format on the wire. When he asks why all the
: Internet protocols have this strange complication, you mumble something
: about ancient versions of Sendmail that didn't support 8-bit text.
:
: So he'll have to deal with ACE, taking time away from implementing new
: features that users actually want.
:
: You should be ashamed of yourself.

---Dan

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 16 Jul 2001 18:27:08 +0200

> use other encodings, like UTF-8.

If that is in the text (I don't have it in front of me) yes, that is silly
and should of course be changed. What charset the user uses when entering
data into his favourite DNS server is up to the implementation.

What Paul and I want to say is that the zone data (in some meta-way) is ACE
encoded in the master server, and the master server operates on that data.
How the data is entered, converted or whatever is out of scope. The same
would be true for UTF-8 by the way given the ability for users to have
editors and operating systems which doesn't use _NAMEPREPPED_ UTF-8
natively in files which later are read by the DNS server.

And, regarding UTF-8 or ACE, the most important thing is the nameprep and
not what encoding the labels uses. I am sorry to say that I am extremely
worried that if we go with an encoding that is implemented (like UTF-8)
people will not do nameprep.

   paf

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 16 Jul 2001 18:28:03 +0200

> The format of a master file needs to be agreed on. This is required for

We should differ between "the holy master file format" and what users _can_
use for entering data into their DNS server.

I think we are in agreement but have problems stating what it is.

  paf

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 16 Jul 2001 12:43:27 -0500

> And, regarding UTF-8 or ACE, the most important thing is the nameprep
> and not what encoding the labels uses. I am sorry to say that I am
> extremely worried that if we go with an encoding that is implemented
> (like UTF-8) people will not do nameprep.

Any application which is modern enough to use the new API should also be
applying the nameprep rules. If it doesn't, it is broken.

Just-use-ACE is actually the most succeptible to this particular problem,
since old and new applications pass data into the system through the old
APIs equally, without any distinctive markings. Was it nameprepped? nobody
knows... At least with the new API approach we know that a particular IDN
was most likely nameprepped.

---

From: Keith Moore <[email-masked]>
Date: Mon, 16 Jul 2001 14:02:19 -0400

> knows...

ACE is sufficiently ugly that it's difficult to imagine that it will be produced 
other than by calling the APIs that produce it - and those APIs will support 
nameprep.

---

From: Keith Moore <[email-masked]>
Date: Mon, 16 Jul 2001 13:50:42 -0400

> I think we are in agreement but have problems stating what it is.

perhaps we should state things in terms of "exchange format" and "local format".

"exchange format" is the format used to exchange zones in a flat file;
DNS implementations should be able to dump and accept zones in that format.
for the time being, it should use ACE, for compatibility with old DNS 
implementations.

"local format" can be anything the DNS implementation wants (web form input,
XML, S-expressions, Morse code, whatever) and the standard shouldn't place any 
restrictions on it.

Keith

---

From: Bright Fulton <[email-masked]>
Date: Mon, 16 Jul 2001 11:10:43 -0700 (PDT)

> was most likely nameprepped.

Implementing ACE without Nameprep wouldn't be logical, and we need not
consider it any more than we consider any other mis-implementation of
IDNA.

Bright

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 16 Jul 2001 21:30:09 +0000

Patrik Fältström <[email-masked]> wrote:

> What Paul and I want to say is that the zone data (in some meta-way)
> is ACE encoded in the master server, and the master server operates on
> that data.

I don't recommend trying to talk about how the names are represented
internally.  Perhaps the idea you want to express is this:  An ASCII
query Q matches a name N iff Q matches ace(nameprep(N)) using a
case-insensitive ASCII comparison.  One obvious (but not required)
implementation strategy is for the server to precompute ace(nameprep(N))
for each name N when it starts up, and then act just like an old
ASCII-only DNS server.  Another strategy is to use a separate
preprocessor to perform ace(nameprep(N)) on the master file, and
actually use an old ASCII-only DNS server.

Keith Moore <[email-masked]> wrote:

> DNS implementations should be able to dump and accept zones in that
> format. for the time being, it should use ACE, for compatibility with
> old DNS implementations.
>
> "local format" can be anything the DNS implementation wants (web form
> input, XML, S-expressions, Morse code, whatever) and the standard
> shouldn't place any restrictions on it.

I like this.

AMC

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 17 Jul 2001 08:05:12 +0900

We can verify whether a ACE label is NAMEPREPed one or not
by re-applying NAMEPREP process to ACE-decoding-produced
code points and see if any changes.

Soobok lee
----- Original Message -----
From: "Keith Moore" <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
Cc: "Patrik F?tstr?" <[email-masked]>; "Adam M. Costello"
<[email-masked]>; "idn working group" <[email-masked]>
Sent: Tuesday, July 17, 2001 3:02 AM
Subject: Re: [idn] Why follow IDNA with UTF-8?