Subject: [idn] Is space allowed in a hostname?

From: Dan Oscarsson <[email-masked]>
Date: Sun, 07 Jul 2002 12:15:41 +0200

I have looked att parsing of hostname or domain name, and there are some
areas I think may give problems.

Today a hostname can be made of A-Z,-,0-9 and a domain name A-Z,-,0-9
and .
When we go over to UCS we will have many more characters.

Looking at the handling of combining code points in UCS, Unicode do not
handle them in a way that will be easy to hande for many programmers.
For example: SPACE which is not allowed in a ASCII hostname and should
probably
not be allowed in a UCS hostname, can easily be checked and parsed as a
separator in ASCII. But in UCS it is possible to represent spacing
accents
as SPACE + combining accent. This means that the UTF-8 form may contain
the SPACE code point which do not represent the SPACE character. That
will
make parsing much more difficult. Looking at the Unicode normalisation
forms
and trying IBM's ICU, NFC do not normalise SPACE + combining accent into
the "spacing accent" code point. NFKC does decompose instead of compose,
spacing accents.

To make things a little bit easier for software handling hostnames we
could
forbidd all accents, or we could allow spacing accents but not
"SPACE+combining
accent". In short: the SPACE code point is only allowed when not
followed
by a combining character.

While NFKC may be a good idea for matching names, it is not a good idea
for normalised form of a name. NFKC removes duplicate forms of single
characters (like wide A and circled A) which is good. But it also
replaces code points representing many characters by many characters.
In some cases that may be well (makes no semantic difference) but in
others
the resulting name is not the same (for example: superscript 2 (U+00B2)
is replaced by character 2 (U+0032)).

From what I have found out, the best normalised form of a
domain name is to use NFC with the alternative code points for those
LETTERS
that have more than one code point forbidden, and all code point
sequences
that can be represented by a single code point be combined into that
code point.

(Note: the IDNA nampreprepped name is a form used for domain name
matching.
It is not the same as the normalised form above.)

  Dan

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 7 Jul 2002 10:11:07 -0700

Dan Oscarsson <Dan dot Oscarsson at trab dot se> wrote:

> While NFKC may be a good idea for matching names, it is not a good
> idea for normalised form of a name. NFKC removes duplicate forms of
> single characters (like wide A and circled A) which is good. But it
> also replaces code points representing many characters by many
> characters. In some cases that may be well (makes no semantic
> difference) but in others the resulting name is not the same (for
> example: superscript 2 (U+00B2) is replaced by character 2 (U+0032)).

Does the semantic distinction between 2 and ² really need to be
preserved in domain names?  Suppose you have a company called Foo²
("foo-squared").  How much damage is caused if your Web site can be
accessed not only through www.foo².com but also through www.foo2.com?

Can it really be possible that this whole NFKC issue was not adequately
considered and we need to stop everything to revisit it?

-Doug Ewell
 Fullerton, California

---

From: John C Klensin <[email-masked]>
Date: Sun, 07 Jul 2002 17:44:20 -0400

--On Sunday, 07 July, 2002 10:11 -0700 Doug Ewell
<[email-masked]> wrote:

> Does the semantic distinction between 2 and Â² really need to
> be preserved in domain names?  Suppose you have a company
> called FooÂ² ("foo-squared").  How much damage is caused if
> your Web site can be accessed not only through www.fooÂ².com
> but also through www.foo2.com?

Doug, I don't know the answer to your question, but it takes us
onto a slippery slope.  If that site is accessible in either of
those ways, does the organization whose name is really "Foo2"
have any special claims?  Is it best to keep them separate so
that both have a shot at their names as they really write them?
And, if the answer is "no, they should be combined", how is that
position ultimately different from arguing that Foo and Föo are
really the same name?  Does this cause a whole new variety of
name dispute and dispute resolution problems?

> Can it really be possible that this whole NFKC issue was not
> adequately considered and we need to stop everything to
> revisit it?

Doug, this is more a response to Dan, since I think we may
agree.  But, since I think the question is too important to be
brushed off...

I think there is a case to be made that only a very small
percentage of the WG (and still less of the IETF) has actually
understood all (or most) of that decision and its consequences.
The percentage is clearly rising at this late date and I always
get concerned when the whole IETF seems to be taking the word of
a very small number of people that a decision that may have
far-reaching impact is correct.

But, is it worth revisiting the decision?   Much as I am
concerned about the potential for damage to the Internet
resulting from putting this stuff into the DNS, I haven't seen a
lot of justification for going around this particular loop
again.  I would feel more positive about doing so had matching
and mapping issues with pairs of characters not come up before,
but they have and people who didn't read themselves in have to
bear some of the blame.  

And, worse, we know enough about them to know that there aren't
any fixes: if superscript "2" and "2" match, then the people who
thinks they should be differentiated will scream that one can't
use one's proper name.  If they don't match, then people will
worry, with some justification, about consumer confusion and
similar problems.  There is really no correct choice, or,
ultimately, a non-arbitrary one, unless one concludes that _DNS_
internationalization is a bad idea and that it needs to be
accomplished elsewhere.

In that context, using NFKC has a huge advantage, which is that
all of the essentially-arbitrary decisions are made at once; the
main alternative seems to be for us to try to make our own
decisions about every pair (or small group) of characters in the
Unicode set.  And one of the few things I'm certain about in
this situation is that going down that path leads to madness.

     john

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 08 Jul 2002 00:53:06 +0200

> name dispute and dispute resolution problems?

All these questions can be asked given the current framework too.
Consider a company name containing the word Maße and another
containing Masse (which are distinct words with distinct meanings).
These words are the same after IDN processing.  So the scheme Doug is
proposing is already implemented, although ² happens to not be
canonicalized into 2.

> but they have and people who didn't read themselves in have to

I agree, the current solution is not well understood, and there seems
to be no time to revisit all issues and wait for people to understand
the consequences.

---

From: "Doug Ewell" <[email-masked]>
Date: Sun, 7 Jul 2002 20:07:08 -0700

> to be no time to revisit all issues and wait for people to understand

This must be a new sense of the phrase "I agree" which I was previously
unaware of.

John's point was that the NKFC topic had already been discussed and
decisions made, and that bringing those people up to speed who had not
been part of the original discussions would be unlikely to shed new
light on the situation, but would simply cause old issues and questions
to be needlessly replayed.

This is very different from saying that the decisions were made without
sufficient input or knowledge, and that people are being kept in the
dark because of an inflexible and overly ambitious WG schedule.

-Doug Ewell
 Fullerton, California

---

From: John C Klensin <[email-masked]>
Date: Mon, 08 Jul 2002 00:19:09 -0400

--On Sunday, 07 July, 2002 20:07 -0700 Doug Ewell
<[email-masked]> wrote:

>> resulting from putting this stuff into the DNS, I haven't
>> seen a lot of justification for going around this particular
>> loop again.  I would feel more positive about doing so had
>> matching and mapping issues with pairs of characters not come
>> up before, but they have and people who didn't read
>> themselves in have to bear some of the blame.
> 
>> I agree, the current solution is not well understood, and
>> there seems to be no time to revisit all issues and wait for
>> people to understand the consequences.
> 
> This must be a new sense of the phrase "I agree" which I was
> previously unaware of.

Actually, probably not.

> John's point was that the NKFC topic had already been
> discussed and decisions made, and that bringing those people
> up to speed who had not been part of the original discussions
> would be unlikely to shed new light on the situation, but
> would simply cause old issues and questions to be needlessly
> replayed.

Yes, but, this is partially due to my not being able to muster
as much sympathy for those who were "not part of the original
discussions" because they sat them out and who are now raising
old issues as I might have for someone who arrived now and
started raising new issues that challenged the conclusions.
And, more important,...

> This is very different from saying that the decisions were
> made without sufficient input or knowledge, and that people
> are being kept in the dark because of an inflexible and overly
> ambitious WG schedule.

Ah.  But I think the decisions _were_ made without sufficient
input, knowledge, and understanding from sufficiently many
people to constitute a legitimate _working group_ decision.  I
don't see that as desirable, and I think an accurate report of
the number of people in the WG who constitute the
personally-informed consensus on this issue --either as an
absolute number or as a percentage of active participants in the
WG-- would be an embarrassment.

That said, I claim that I do understand the issues and that
several other people in the WG also understand them and have
understood them more or less all along.  And, based on that
understanding, I don't believe that revisiting NFKC is going to
get us anywhere.  That isn't because NFKC is right or wrong, but
because its selection involves tradeoffs, that _no_ alternative
is without similar tradeoffs, and that there are no rational
criteria for selecting one set of tradeoffs over another.  

NFKC is, IMO, guilty of one severe sin, and it is the one I
believe Simon was pointing out (but it isn't a new observation
either): it is not consistent when examined on a language by
language and code-point by code-point basis.  If a single
meta-rule could be made up that would identify the "right"
choice for IETF and DNS purposes, it would handle some sets of
characters consistently with the rule and some inconsistently.
But, again, no better solution is on the table, and, in all
likelihood, no objectively better solution is possible: no
single, simple, statement can be made that accurately predicts
whether characters are duplicated or unified in the
Unicode/10646 code set itself.  The allocation decisions may
well have been made rationally, but enough different rules were
applied, and generic enough rules were applied, that it would be
unreasonable to expect complete consistency with  any simple
rule, much less an IETF-optimized one.

It is all going to come down to arbitrary choices at some stage.
Some of those arbitrary choices are going to irritate some
groups of people and seem harmless to others.  If they were made
a different way, different people would be irritated.  If we
were to apply the test of whether NFKC (and many of the other
stringprep/ nameprep decisions) made everyone who had studied it
carefully about equally unhappy, it would almost certainly pass
that test.

So I don't think this one is worth revisiting, not because the
consensus is solid, or because I am feeling serious constraints
from a rigid schedule, but because I see absolutely no reason to
believe that going around the loop again would produce, a
different decision that was objectively better when viewed
globally.

I do believe that there is, plausibly, a different conclusion
that can be reached from this, but reexamining NFKC isn't going
to help with it one way or the other.  That conclusion is,
essentially, the null hypothesis, i.e., that true, global,
internationalization that is language-sensitive and sensitive to
the semantics commonly attached to various characters is
incompatible with the DNS and the way it is organized and does
lookups.   If one reaches that conclusion, there are then, I
believe, only two choices: 

	* one is to decide, explicitly, that some languages (or
	scripts, or sets of choices about normalization and
	canonicalization) are "more equal than others" and that
	the others just lose.
	
	* and the other is to decide that internationalization
	using the DNS is technically infeasible given all of the
	constraints.

I believe the first of these choices is an inappropriate and
impossible one for the IETF to make (whether explicitly or
implicitly).  And I believe that the second has been
insufficiently discussed in the WG, only partially because some
WG members have taken the position that the WG's returning an
answer of "don't do it", rather than a protocol, is so untenable
that it cannot be discussed.

      john

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 09 Jul 2002 17:04:53 +0200

[snip]
> a different way, different people would be irritated.
[snip]

This was a good summary.

I think the reason that I and others are bringing up old questions is
that several of us did participate when this working group started,
raising these questions.  We are now expecting that the documents
answer the questions, or where they cannot be answered to everyone's
agreement, at least say what inherent problems and risks are caused by
not solving the problems completely.  Since internationalized strings
is a tricky subject, it is IMHO as important to carefully describe
what problems are _not_ solved as to describe what problems actually
are solved.  Admittedly, the latest document revision fixed several of
my pet problems (thanks!), but being new to the topic and not having
read the specifications all that carefully makes me worry about there
being other inherent problems that aren't documented or discussed.

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Jul 2002 11:40:02 -0400

(I've added the IESG back into this note, as I think we finally
have a problem statement worth bothering them with)

--On Tuesday, 09 July, 2002 17:04 +0200 Simon Josefsson
<[email-masked]> wrote:

>> well have been made rationally, but enough different rules
>> were applied, and generic enough rules were applied, that it
>> would be unreasonable to expect complete consistency with
>> any simple rule, much less an IETF-optimized one.
>> 
>> It is all going to come down to arbitrary choices at some
>> stage. Some of those arbitrary choices are going to irritate
>> some groups of people and seem harmless to others.  If they
>> were made a different way, different people would be
>> irritated.
> [snip]
> 
> This was a good summary.
> 
> I think the reason that I and others are bringing up old
> questions is that several of us did participate when this
> working group started, raising these questions.  We are now
> expecting that the documents answer the questions, or where
> they cannot be answered to everyone's agreement, at least say
> what inherent problems and risks are caused by not solving the
> problems completely.  Since internationalized strings is a
> tricky subject, it is IMHO as important to carefully describe
> what problems are _not_ solved as to describe what problems
> actually are solved.  Admittedly, the latest document revision
> fixed several of my pet problems (thanks!), but being new to
> the topic and not having read the specifications all that
> carefully makes me worry about there being other inherent
> problems that aren't documented or discussed.

We are in violent agreement.

My understanding, when we gave up on the "Requirements" document
(for what I still believe were good reasons), was that it would
be replaced by an exact statement of the problem the WG thought
it was solving and, ideally, a review of problems that seemed to
fall within the IDN space that it was not trying to solve.
Implicit in that was the assumption that the WG's outputs would
be narrowed and focused to that statement and tested, where
appropriate, against it.

Instead, we are missing that statement and, without it, seem to
have a collection of specifications that are underspecified in
some areas and that overreach in others.  Given that, as you
say, internationalized strings are a tricky subject, that isn't
a good situation.  

And it leads me to conclude that, even if I believed that IDNA,
as now defined from a protocol standpoint, were a perfect match
for the critical problem being solved, it would still be vitally
important to carefully describe that problem and then to trim,
narrow, and focus the relevant documents.

      john