Subject: [idn] Time to reconsider

From: John C Klensin <[email-masked]>
Date: Thu, 24 May 2001 09:24:56 -0400

I started to write the attached as a response to a note in a
side-conversation.  I think it belongs on the list.  Since I
haven't asked the author of the quoted material/ questions, I've
anonymized those remarks (variations on which, actually, could
have been made by several people in the threads of the last few
days although I particularly like this formulation).

Quick summary: I believe that we should not agree to either
Nameprep or the Requirements document as they stand because the
underlying assumptions behind what we are doing have been shown,
by the WG's work, to be fundamentally flawed.  I try to explain
"why" below, followed by a quick summary of what we should be
doing instead.  More on the latter within the next day or two.

(referring to the former (as of draft -06) requirement [30] and
other "localization" ideas)...
> Is scope a requirement? Is scopelessness a requirement?

My personal response would be that this is more evidence that
trying to do these things in the DNS is a mistake.  The issue
isn't that of one coding versus another.  It is that we are
expecting a lookup-and-exact-match system to do the impossible
-- fuzzy matching that doesn't astonish a user whose cultural
and linguistic assumptions cannot be predicted or identified
given the tools at hand.

> There was substantive discussion involving CNNIC, and the
> question of TC and SC equivalence. [...]  at Minneapolis [...]
> I'd prefer to see technical
> consensus.

My personal belief is that you will never get it.  The problem
is that interpretations of "matching" vary locally.  That means
there will be cases of ambiguity -- things that do, or do not,
match depending on context or the user's assumptions.
Simplified and traditional Chinese are either matching issues or
language translation, depending on those assumptions: the
requirement is clear for those who need the matching, but
agreement that it is universally necessary or desirable is
problematic.  

And that may be one of the cases with an answer on which global
consensus can be reached, but the more general problem is
certainly not unique to one language or script.  As we keep
looking, it seems that we have at least a few examples per
script.  Even ASCII (although not the "hostname" subset) has
them if one goes back far enough or starts looking carefully.
E.g., are "tilde" (Ux007e) and "not sign" (Ux00AC) the same
character?   Well, several programming languages and database
systems made them the same (the glyphs were treated as alternate
stylizations of the same character) back in what could be
described as the "unified Roman" days of ISO 646.  For some,
"macron" (Ux00AF)  If, as some registries are proposing (or
actually registering), we can have dingbats in the DNS, there is
certainly no reason to ban these characters.  There are several
single-line optical scanner products on the market that proport
to read URLs and enter them directly into browsers.  Do these
code points match?    How about "vertical line" (Ux007C) and
"broken bar" (Ux00A6)?  That pair and the tilde/not-sign one are
equivalenced in character set translations between ASCII and
another _very_ popular coded character set.   To a person not
trained or socialized in Roman alphabets, the pairs may "look"
more similar than two Han characters, or two characters from
different Han-based language scripts look to an untrained person
such as myself.

But they are separated in Unicode because there are good reasons
to separate them, especially if one is worried about typography.
Should they be matched (treated as equivalent)?  Well,
sometimes: no "yes" or "no" answer is going to satisfy everyone
and, worse, the same answer may be unsatisfactory to the same
person on different occasions.  One needs context --cultural or
use-- and the DNS cannot supply or support that context in an
unambiguous way, at least without kludges which are much more
horrible than any we have contemplated so far.

Consequently, there is no way to get technical agreement without
favoring the social/ political/ cultural preferences of one
group over another.  I believe that the WG has managed to prove
a negative by exhaustion, and that we need to stop before we
wreck the DNS only to get an unsatisfactory and incomplete
result.

So, my formulation to respond to Marc's "time to move" question
is that we should:

(i) Carefully review the requirements document again.  The
issues over [30] (which I saw as a "localization" -- every zone
can have its own matching rules-- hook) may exist in other
places.  The resulting document should either focus on solutions
that are not limited to changes to the DNS or what is permitted
in it or how its names are interpreted (I consider IDNA, etc.,
to be cases of such changes, disclaimers that the interpretation
is elsewhere notwithstanding), or should clearly identify where
important requirements cannot be met in the DNS alone.

(ii) We should make another sweep over "nameprep", to change the
matching and mapping lists from "yes" or "no" to "yes"/ "no"/
and "sometimes" (or an equivalent three-way choice).  The
latter, ambiguous, category, should include cases like those
above as well as the "is 'A' in Roman, Cyrillic, or Greek?"
issue we have kicked around before, and all of their relatives
and parallels in other scripts.

(iii) And then we should concentrate IETF's efforts on searching
and matching systems that permit context and user choice over
the ambiguous cases, rather than hoping that "sneak this into
the DNS by clever coding and hope the problems will somehow
solve themselves elsewhere --perhaps by the world adapting its
natural languages and cultures to our conventions" will be
adequate.  It won't.

     john

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Thu, 24 May 2001 13:29:45 -0400

> Since I
> anonymized those remarks

John, I suggested that we take the scope issue back to the main list.
Quote me till the cows come home. We also now know the origin of [30].

Semantic scope isn't encoding selection, encoding is just one possible
policy that could have scoped semantics, or could be scopeless. In the
context I mentioned in our side-bar, now back in the main list, rules
for equivalency over the same encoding (utf8) were the subject of
discussion. In the origin-of-[30] note, rules for equivalency over the
same encoding (ASCII) were the subject of discussion.

> by the WG's work, to be fundamentally flawed. 

This far I agree. Where I expect we disagree is what the underlying
assumption(s) are which appear (to each of us) to be fundamentally
flawed, which I'll restate as affirmative goals.

	JK:	extend "meaning" (languages, scripts, and fuzzy matching)
	EB-W:	extend "labels" (0x2d, 0x30-0x39, 0x41-0x5a, 0x61-0x7z)

Is consensus possible?

In practical terms, "apparently not":

	VGS is going live with its RACE, the mechanism for non-ASCII
	in the .com namespace.

	CN went live with UTF8, the mechanism for ASCII and non-ASCII
	in the .cn namespace.

There are a lot of other choices to pick from, but of these two, I'm
going to deploy UTF8, subject to possible ICANN and possible Registry
Trade Association regulatory oversight and technical coordination.

In working group terms, the answer isn't much better.

There are 6,182,148 (Walid), and 5,410,306 (Ydisg), encumberments, and we
have simply not had the bandwidth to conclude independently as VGS did that
neither applies to IDNA, a lack of knowledge I happen to know is shared by
at least two other namespace vendors.

We've a nameprep/idna poll (my reply was "no" and "rfc2044") outstanding.

All this, and about 20 drafts in various stages of aging are before this
working group, along with the prior organizational art, rfc2044 and rfc2130,
and rfc1034/1035.

It will take a wiser head than mine to sort out the WG roadmap to rough
consensus in finite time, and it appears that time is very finite indeed.

Eric

---

From: Keith Moore <[email-masked]>
Date: Thu, 24 May 2001 15:21:16 -0400

John,

> and linguistic assumptions cannot be predicted or identified

Here's why I'm skeptical about this argument.  Most of the 
computer operating systems I'm aware of use more-or-less
the same matching algorithm for filenames (including filenames
typed by humans) as that used by DNS - exact comparison modulo
case folding.  Yes, point-and-click interfaces exist, but they 
don't scale.

Now it might turn out that that kind of filename matching is 
completely impractical for some language because they have
too many different ways of spelling words.  But I suspect 
that, to the extent possible, folks who speak languages
that have multiple ways of spelling words can develop
conventions for spelling domain names in that language - just 
as we have found some restrictions on English-language domain 
names (e.g. the LDH rule) useful in minimizing the differences 
in how domain names can be spelled.

I do think we should realize the limitations that are inherent
in the approach of extending DNS to non-ASCII characters.  
However, I don't think we should stop developing an IDN solution
simply because it's not the perfect solution to all name lookup
problems.

The other bit is that it's very hard to build a federated name
space once you get away from canonicalization followed by exact 
matching.  Which means that in such a world you either have a 
single point of control for ALL names (not just the TLDs), or 
you don't get consistent results from one lookup service to 
another.  Neither one of these sounds particularly desirable as
a replacement for domain name lookup.  The struggle for control
of the DNS root and TLDs has been too painful already, and the
major contributors to that war have shown no sign of willingness
to exercise restraint or minimize the collateral damage.
 
The last thing we need is to have a fight over control of all names.
Because people will demand fuzzy-matching all the way to the root,
with all of the chaos and inefficiency that this implies.

So I think that IDN is still needed, even though we may also 
recognize that a fuzzy-matching lookup service might also be
needed.

Keith