Subject: [Ietf-not43] Domain Names, IDNs and CRISP

From: leslie at thinkingcat.com (Leslie Daigle)
Date: Thu Sep 4 18:13:21 2003

Feeling I wasn't being clear in my points on domain names
and IDNs last week, I paused, reread the IDN-related RFCs,
and am attempting to write a short letter.

As I said last week, I do not believe it is appropriate to conflate
the handling of domain names and IDNs.

Syntax:

As Eric has noted, valid domain (not host) names, even pre-IDN-era,
can include binary characters (bytes with the 8th bit set).
It is inappropriate to apply the IDN algorithms to those names --
not because they may contain bytes that are invalid in IDNs, but
rather because they may not.  The IDN algorithms include normalization
and mapping rules that will *change* the byte values.  If the
original domain label was not ever intended as an IDN, it is
quite possible that two different domain names would yield the
same IDN representation in UTF-8.  For example, if they differed
in only one byte that happened to map to nothing in the NAMEPREP
process, the distinction would be lost.  

Furthermore, although the IDN spec says that nothing other than
an IDNA encoded IDN SHOULD be registered with the "xn--" prefix,
it does not say (and cannot enforce) that they MUST not.  This
means that there could be valid domain names starting with "xn--"
that do not "un-IDNA" successfully.

In short -- (some representations of) IDNs share the same
space of bytes as domain names, but they are still not the same
thing. 

Therefore:  for the purposes of a protocol that is capable
of handling queries for *all* valid domain names, it MUST be 
possible to express domain names in a fashion that will be
unmolested by the IDN processing algorithms.

Semantics:

I believe a "match" for an IDN has different semantics than
for a domain name.  Whether it is "just" applying the
IDN normalization and mapping rules, or allowing 
other approximative matches based on locale knowledge, IDNs
match differently than byte-by-byte matching of unadulterated
input of domains.

Overhead:

As Andy has already noted, allowing a distinction of the expression
of IDNs (queries, results) from domain names does NOT necessarily
imply doubling the data store.  It is a mechanism for the protocol
to express intention.

Therefore, I reiterate that I believe it is important that the protocol
have separate queries and response elements for domain name based 
queries and IDN ones.  To do otherwise is not, in my opinion, preparing
for the future,  or allowing flexibility, so much as it is
creating a system that doesn't handle its full range of inputs properly,
or provide the users the ability to properly express what they mean.

Leslie.

-- 

-------------------------------------------------------------------
"Reality:
      Yours to discover."
                                 -- ThinkingCat 

Leslie Daigle
[email-masked]
-------------------------------------------------------------------

---

From: ehall at ehsco.com (Eric A. Hall)
Date: Thu Sep 4 23:34:47 2003

> and IDNs last week, I paused, reread the IDN-related RFCs,

you have entered a maze of dark and twisty passages

> the handling of domain names and IDNs.

You are confusing encoding formats with domains. Issues like IDNA and
UTF-8 are arguments about encodings, not about the canonical domain names
that they ultimately represent. The IDNA specs don't make this clear
enough, granted.

> can include binary characters (bytes with the 8th bit set).

*especially* pre-IDN; their handling is fairly undefined with IDNA, but
they are clearly documented for use in the pure DNS specs.

> and mapping rules that will *change* the byte values.

That's what my escape syntax is for, to protect those characters. BTW,
that syntax is derived from the escape syntax in RFC1035 section 5.1 (but
which is also widely used outside of the "zone file" context of that
section), and is also similar to the escape syntax defined in RFC2252
section 4.3, so it's not exactly foreign to the people who are already
comfortable with extended string data in DNS or LDAP.

Whether your camp is up for this kind of work is a question for you to
answer. I didn't list it in my list of architectural issues with IRIS
because I don't expect your service proposal to go beyond hostname
syntaxes, and so it probably won't be needed there. For a generalized
system, however, its absolutely mandatory towards future usages. I've
already decided to face this music and have done so with an escape syntax.

> that do not "un-IDNA" successfully.

I guess there are a couple of issues here.

First of all, the insistence on round-trip validation checking is to
ensure that the domain name is a valid IDN. If an xn--blah domain name
doesn't decode to a valid UTC sequence which correctly re-encodes, then
leave it as an ASCII domain name. This is actually a stricter test than
necessary, given the usability factors (see next item). Anyway, the
reliability problem is solved for me, and solved using the same logic
every other robust application will eventually have to use.

Secondarily, anybody who registers a legitimate xn--blah domain name but
who doesn't want the domain name to be rendered in UCS characters would
find out that pretty quick that they don't have a vote. Email clients and
web browsers and whatnot will render the domain name without the domain
operator's permission all the time, every time. If the operator wants to
live in oblivion wrt that specific domain name, they can use whatever
products that suit their needs, and the same is true for this as well.

In any event, both of these are client-side transformation issues that
have very little to do with protocol encoding choices.

> space of bytes as domain names, but they are still not the same

That's incorrect. Rather, the user has explicitly chosen to work with
codec output instead of using the canonical domain name associated with
that encoded sequence (assuming it's valid). Otherwise, its just another
ASCII domain name (if it's invalid),

> match differently than byte-by-byte matching of unadulterated

Provide a demonstration please.

> imply doubling the data store.  It is a mechanism for the protocol

At the last checkpoint on this topic, Andy was refusing to require that
the entries provide the same answer data, never mind anything along the
lines of operational requirements (eg, dual-indexes against a common
store). Absent a requirement along those lines, dual data-stores are
guaranteed to exist, and are likely to exist in any event.

Frankly, given the work here, your camp would be better off just
canonicalizing the query input to IDNA and paying the performance cost of
having servers do conversion for comparison purposes. Lots more procesing
overhead but lots less synchronization and query maintenance.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/