Subject: [idn] IDNA problem statement

From: Erik Nordmark <[email-masked]>
Date: Tue, 15 Oct 2002 14:39:31 +0200 (CEST)

Back in May when I sent AD review comments on the IDNA specification
I raised a desire to make the problem IDNA claims to solve more clear
and some relatively recent discussion on the IDN mailing list
has re-emphasized the need for this. I think the text below is
not very controversial in that it tries to summarize what IDNA
does and what it does not. If you see a particular problem with the
text please suggest alternative text.

The intent is to add this text to draft-ietf-idn-idna in section 1.

   Erik

1.1 Problem Statement

The IDNA specification solves the problem of extending the repertoire 
of characters that can be used in domain names to include the Unicode 
repertoire.

IDNA does not extend the service offered by DNS to the applications. Instead,
the applications (and, by implication, the users) continue to see an
exact-match lookup service. Either there is a single exactly-matching name
or there is no match. This model has served the existing applications well,
but it requires that users know the exact spelling of the domain names
that the users type into applications such as web browsers and mail user 
agents. The introduction of the larger repertoire of characters potentially
makes the set of misspellings larger, especially given that in some
cases the same appearance, for example on a business card, might visually match
several Unicode code points or several sequences of code points.

IDNA allows the graceful introduction of IDNs not only by avoiding
upgrades to existing DNS infrastructure (servers, caches, stub resolvers),
but also by allowing some rudimentary use of IDNs in applications by
using the ASCII representation of the non-ASCII name labels.
While such names are very user-unfriendly to read and type, and hence are
not suitable for user input, they allow (for instance) replying to email
and clicking on URLs even though the domain name displayed
is incomprehensible to the user. In order to allow user-friendly
input and output of the IDNs, the applications need to be modified to
conform to this specification.

IDNA uses the Unicode character repertoire which avoids the significant
delays that that would be inherent in waiting for a different 
and specific character set be defined for IDN purposes by some other
standards developing organization.

1.2 Limitations of IDNA

<EXISTING section 6.6 moved to here>
The IDNA protocol does not solve all linguistic issues with users
inputting names in different scripts. Many important language-based and
script-based mappings are not covered in IDNA and need to be handled
outside the protocol. For example, names that are entered in a mix of
traditional and simplified Chinese characters will not be mapped to a
single canonical name. Another example is Scandinavian names that are
entered with U+00F6 (LATIN SMALL LETTER O WITH DIAERESIS) will not be
mapped to U+00F8 (LATIN SMALL LETTER O WITH STROKE).

<ADDED>
An example of an important issue that is not considered in 
detail in IDNA is how to provide a high probability that a user who 
is entering a domain name based on visual information (such as from a 
business card or billboard) or aural information (such as from a 
telephone or radio) would correctly enter the IDN.
This a complex issue relating to languages, input methods on
computers, and so on.  Furthermore, the kind of matching and
searching necessary for a high probability of success would not fit
the role of the DNS and its exact matching function.

---

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 15 Oct 2002 15:13:45 +0200

> of characters that can be used in domain names to include the Unicode 

Your text is very useful.  I have only one minor comment on that first
sentence -- it might be good to make it clear that not the entire
Unicode repertoire is included, but a Unicode repertoire with some
restrictions.  Having this in mind early on makes it easier to
comprehend what is going on later when some strings trigger errors.

"... to include the Unicode repertoire with some restrictions."

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Tue, 15 Oct 2002 16:48:06 +0200

> comprehend what is going on later when some strings trigger errors.

I would still like it to be called the ISO 10646 repertoire.

Kind regards
Keld Simonsen

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 15 Oct 2002 17:51:47 +0200 (CEST)

> "... to include the Unicode repertoire with some restrictions."

While I agree with the intent, I wonder if the text should
be more specific that just saying "some".
But it is hard to do this without having a forward reference
to either sections in the document, or referring to nameprep.

Ideas?

  Erik

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 15 Oct 2002 12:09:54 -0400

> to either sections in the document, or referring to nameprep.

A reference to nameprep would work:

"... to include the Unicode repertoire with restrictions described in
[NAMEPREP]."

-Scott-

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 15 Oct 2002 18:06:26 +0200 (CEST)

> I would still like it to be called the ISO 10646 repertoire.

Doing that change to this added text would be very odd given
that that the rest of the document has no mention of ISO 10646.
Thus I think consistency is the overriding concern here.

  Erik

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Tue, 15 Oct 2002 18:37:11 +0200

> Thus I think consistency is the overriding concern here.

Is that not a major change from IETF policy? We should mention
ISO standards when they are applicable.

Kind regards
keld

---

From: John C Klensin <[email-masked]>
Date: Tue, 15 Oct 2002 15:10:11 -0400

--On Tuesday, 15 October, 2002 18:37 +0200 Keld Jørn Simonsen
<[email-masked]> wrote:

>> that that the rest of the document has no mention of ISO
>> 10646. Thus I think consistency is the overriding concern
>> here.
> 

Keld, this is an old argument in the IDN WG, and I think ISO
basically decided to lose it a year or two ago.  While, I, too,
prefer to reference ISO standards where possible, the situation
here is that the IDN effort needed both a character and code
point repertoire and a collection of norms about how those code
points were to be used, compared, etc.  My preference, and I
think that of the IETF generally, would have been to reference
ISO Standards for all of this but, as you know, the
complementary "usage" standards did not follow the code point
ones.  Even where TRs exist, ISO generally doesn't like having
its TRs referenced normatively. 

We approached ISO about the problem of the missing standards at
the JTC1 level and stressed that, if they couldn't respond
usefully and fairly quickly, we would have to rely on UTC.  We
didn't get a response for a long time, and then, in my opinion,
were brushed off.  And you are all-too-aware what happened when
we tried to work something with SC22... from my perspective, not
only did we not get active cooperation, we were deliberately
insulted by their "agreeing" to something we had previously told
them (formally and informally) we would not accept.  While this
was going on, the Unicode folks were actively working with the
WG, inviting IETF participation in their meetings and in
liaisons with on their Board, and trying to be responsive to our
needs in their ongoing work.  I think they are entitled to
recognition for those efforts, including having their preferred
name for the CCS and associated materials used.  And, if JTC1
wants to isolate themselves from the Internet in this area, and
to hint that they are doing so because the IETF is just not
important enough to deal with on a peer basis, I don't see any
reason to respond by advertising the relevant ISO Standard in
more than a footnote.

Just my opinion, of course.  But, if my analysis is correct or
rings true, the problem you are addressing needs to be raised
within JTC1, not in this working group or over these documents.

    john

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 15 Oct 2002 13:11:33 -0700

>"... to include the Unicode repertoire with restrictions described in

Although this is true, it feels like it is too early in the IDNA 
document to say this. Having the problem statement contain no 
references makes it much easier to read. "with some restrictions" 
tells the reader enough to know what Simon wanted, which is that it 
isn't the entire repertoire.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Tue, 15 Oct 2002 23:40:33 +0200

At 22:11 15/10/02, Paul Hoffman / IMC wrote:
>Although this is true, it feels like it is too early in the IDNA document 
>to say this. Having the problem statement contain no references makes it 
>much easier to read. "with some restrictions" tells the reader enough to 
>know what Simon wanted, which is that it isn't the entire repertoire.

My Frenglish? "some restriction" sounds unprecise to me.
What about "limited restrictions" which implies the author did his best to 
limit the size/number of restrictions (thefored well defined?)jfc

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 16 Oct 2002 00:08:32 +0000

Erik Nordmark <[email-masked]> wrote:

> While I agree with the intent, I wonder if the text should be more
> specific that just saying "some".

This is just the introduction, so it's not expected to fill in all the
details.  I think the reader will naturally assume that the specific
restrictions will be given later in the document.

"JFC (Jefsey) Morfin" <[email-masked]> wrote:

> "some restriction" sounds unprecise to me.

Because it is.  For the purposes of the problem statement, the essential
fact is that the repertoire is expanded from a small one to a large one.
Mentioning Unicode is helpful because many readers will immediately
recall the two primary characteristics of Unicode (it's big, and it aims
to include all characters from all other repertoires).  But additional
details regarding how much of Unicode is or is not allowed would just
distract from the point of this section, which is to describe the
problem being addressed.

In fact, I'd even put "with some restrictions" in parentheses, to make
it clearer that the restrictions are unimportant to understanding the
problem statement.

AMC

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 16 Oct 2002 03:36:19 +0200

"JFC (Jefsey) Morfin" <[email-masked]> writes:

> What about "limited restrictions" which implies the author did his
> best to limit the size/number of restrictions (thefored well
> defined?)jfc

Perhaps the text can be improved, not by clarifying how large the
restriction is, but rather with the reason why there is a restriction.

"... to include the Unicode repertoire restricted in a way that is
intended to reduce ambiguities when strings are compared."

I'm not entirely happy with the result...

Btw, I don't like the phrases (in nameprep and stringprep) that says
IDN related standards provided the "highest chance of getting content
correct" when entering strings (what does that MEAN?  Getting content
correct?  Maybe the RFC editor should implement the algorithm).  It
seems clear that it only reduces some limited and character-by-
character specific concerns. The text is confusing after reading the
following paragraph in your introduction, which I think describes the
situation more accurately:

,----
| An example of an important issue that is not considered in 
| detail in IDNA is how to provide a high probability that a user who 
| is entering a domain name based on visual information (such as from a 
| business card or billboard) or aural information (such as from a 
| telephone or radio) would correctly enter the IDN.
| This a complex issue relating to languages, input methods on
| computers, and so on.  Furthermore, the kind of matching and
| searching necessary for a high probability of success would not fit
| the role of the DNS and its exact matching function.
`----

I don't dislike it strong enough to provide better text though...

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Wed, 16 Oct 2002 14:06:03 +0200

> reason to respond by advertising the relevant ISO Standard in

Could you give me a reference to the approach from IETF to JTC1?
As the liaison officer from JTC1/SC2 with IETF I cannot remember
such a request, so I would like to see it.

The problem with SC22 is that Unicode/L2 has infiltrated it, and tries to
sabotage the cooperation between ISO and IETF, by not having any
cooperation being done, like delaying liaison or making it in a way that
is unacceptable. And also by having ISO standards in the area
voted down, delayed, or turned into TRs. And then working smoothly with IETF
directly. Their policy sems to be efficient, and this is the behaviour
that you are rewarding, IMHO.

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 Oct 2002 08:36:59 -0400

--On Wednesday, 16 October, 2002 14:06 +0200 Keld Jørn Simonsen
<[email-masked]> wrote:

>> basically decided to lose it a year or two ago.  While, I,
>> too, prefer to reference ISO standards where possible, the
>> situation here is that the IDN effort needed both a character
>> and code point repertoire and a collection of norms about how
>> those code points were to be used, compared, etc.  My
>> preference, and I think that of the IETF generally, would
>> have been to reference ISO Standards for all of this but, as
>> you know, the complementary "usage" standards did not follow
>> the code point ones.  Even where TRs exist, ISO generally
>> doesn't like having its TRs referenced normatively. 
>> 
>> We approached ISO about the problem of the missing standards
>> at the JTC1 level and stressed that, if they couldn't respond
>> didn't get a response for a long time, and then, in my
>> opinion, were brushed off.  And you are all-too-aware what
>> happened when we tried to work something with SC22... from my
>> perspective, not only did we not get active cooperation, we
>> were deliberately insulted by their "agreeing" to something
>> we had previously told them (formally and informally) we
>> would not accept.  While this was going on, the Unicode folks
>> were actively working with the WG, inviting IETF
>> participation in their meetings and in liaisons with on their
>> Board, and trying to be responsive to our needs in their
>> ongoing work.  I think they are entitled to recognition for
>> those efforts, including having their preferred name for the
>> CCS and associated materials used.  And, if JTC1 wants to
>> isolate themselves from the Internet in this area, and to
>> hint that they are doing so because the IETF is just not
> Could you give me a reference to the approach from IETF to
> JTC1? As the liaison officer from JTC1/SC2 with IETF I cannot
> remember such a request, so I would like to see it.

As I said, this was not to SC2, but to JTC1 and occurred as a
direct inquiry/request from me (as IAB Chair at the time) to Tom
Frost (as JTC1 Chair at the time).  Since Internationalization
work was being done in several SCs of JTC1 at the time, part of
the question was "where can we really get something done".  And,
since JTC1 has refused to consider a liaison with IETF at that
level, there is no mechanism for causing a formal, documented,
set of contacts.

> The problem with SC22 is that Unicode/L2 has infiltrated it,
> and tries to sabotage the cooperation between ISO and IETF, by
> not having any cooperation being done, like delaying liaison
> or making it in a way that is unacceptable. And also by having
> ISO standards in the area voted down, delayed, or turned into
> TRs. And then working smoothly with IETF directly. Their
> policy sems to be efficient, and this is the behaviour that
> you are rewarding, IMHO.

I'll let someone else comment on most of this description.  But
it seems to me that you are describing a problem within JTC1 and
that it is up to JTC1 to fix it if they, or ISO, care.   If the
P-members of SC22, or of JTC1, believe there is a problem that
needs fixing, they clearly have the power to make other
selections of conveners, to modify work items, or to give clear
instructions to Secretariats about ballots.  And, if I were to
see the problem the way you do, I'd see this documentation
decision on IETF's part as part of the justification for making
some changes, on a "if we proceed this way, that happens, and
ISO becomes ever more irrelevant" basis, rather than just as
rewarding behavior which you find unattractive.

    john

    john

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 16 Oct 2002 23:23:54 -0700

> sabotage the cooperation between ISO and IETF, by not having any

This is misrepresenting the situation. Certainly SC22 has not been
"infiltrated", nor has there been "sabotage". There are some people
from on a particular working group, WG20, that are in support of
Unicode, and there are others who are not.

But completely independent of Unicode, the real problem is that most
of the documents that were being proposed *to* the WG20 group (such as
the internationalization API, or 14652) for standardization were --
how can I put this most charitably -- of extremely low quality. Look
at the negative comments in
http://std.dkuug.dk/JTC1/SC22/WG20/docs/n857-n3261.pdf for examples.
TR 14652 ended up being a Type 1 TR: "when the required support cannot
be obtained for the publication of an International Standard, despite
repeated effort". The only reason, I believe, that it passed -- even
as a type-1 TR (not a standard), even with huge sections marked as
controversial -- was that a number of countries tend to simply vote
without bothering to actually read the text. The proposed
internationalization API was even worse: if a college student sent us
that as a sample of his work we wouldn't even bother interviewing him.

The one exception is ISO 14651. With a lot of hard work by people on
the WG20 committee, both those who support Unicode and those who
don't, together with the UTC, the collation standard has turned out
well. But even there, at this point the work would be better
transferred to SC2/WG2, which is where the linguistic expertise is in
ISO.

Mark

The above are my own opinions, and do not represent those of any
organization I am affiliated with.
__________
http://www.macchiato.com
◄  “Eppur si muove” ►

----- Original Message -----
From: "Keld Jørn Simonsen" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: "Keld Jørn Simonsen" <[email-masked]>; "Erik Nordmark"
<[email-masked]>; "Simon Josefsson" <[email-masked]>;
<[email-masked]>
Sent: Wednesday, October 16, 2002 05:06
Subject: Re: [idn] Re: IDNA problem statement

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 17 Oct 2002 16:35:21 -0500

> of characters that can be used in domain names to include the Unicode

This is broader than the actual solution. IDNA allows applications to make
use of characters from the Unicode repertoire by providing a standardized
encoding of i18n domain name which can be used to represent and interpret
extended characters consistently. It does not extend the repertoire of
characters which are usable in domain names, nor does it provide
applications with access to these characters in their unencoded form.

> IDNA does not extend the service offered by DNS to the applications.

Correct

> Instead, the applications (and, by implication, the users) continue to
> see an exact-match lookup service. Either there is a single
> exactly-matching name or there is no match. This model has served the
> existing applications well, but it requires that users know the exact
> spelling of the domain names that the users type into applications such
> as web browsers and mail user agents.

This is irrelevant. It's interesting to about five people here, but it a
complete side-track to the functionality offered by IDNA. Suggest you drop
this entirely.

> The introduction of the larger
> repertoire of characters potentially makes the set of misspellings
> larger, especially given that in some cases the same appearance, for
> example on a business card, might visually match several Unicode code
> points or several sequences of code points.

Move that to the security section, or to the 1.2 section below

> upgrades to existing DNS infrastructure (servers, caches, stub

Put it another way. IDNA doesn't "break" ANY node in the infrastructure,
including DNS servers/resolvers/blah, but ALSO including protocols and
their applications. You are limiting the discussion to certain elements of
the infrastructure and casually omitting the other 99% of the
infrastructure, implying that the 99% doesn't matter. You are talking
about the 1% getting a free-ride like it is a big deal.

Look, I have never understood the fascination with the 1% that doesn't
have to pay. This continued focus on "we saved the 1%" is intellectually
dishonest all the way around. Not only is it a miniscule portion of the
infrastructure that actually USES domain names, but it completely ignores
the fact that 99% DOES HAVE TO PAY.

All I'm asking here is that you provide the full analysis on this point,
rather than telling the 1% story again. If we are so embarrassed by the
truth, then we should rethink it. If we are comfortable with the truth,
then let's tell it.

> but also by allowing some rudimentary use of IDNs in
> applications by using the ASCII representation of the non-ASCII name
> labels.

Yeah, that's it

> While such names are very user-unfriendly to read and type, and
> hence are not suitable for user input, they allow (for instance)
> replying to email and clicking on URLs even though the domain name
> displayed is incomprehensible to the user. In order to allow
> user-friendly input and output of the IDNs, the applications need to be
> modified to conform to this specification.

^every application that wants to display them

> delays that that would be inherent in waiting for a different and
> specific character set be defined for IDN purposes by some other

No justification is required here, IMO.

> 1.2 Limitations of IDNA
>
> <EXISTING section 6.6 moved to here> The IDNA protocol does not solve
> all linguistic issues with users inputting names in different scripts.
> Many important language-based and script-based mappings are not covered
> in IDNA and need to be handled outside the protocol. For example, names
> that are entered in a mix of traditional and simplified Chinese
> characters will not be mapped to a single canonical name. Another
> example is Scandinavian names that are entered with U+00F6 (LATIN SMALL
> LETTER O WITH DIAERESIS) will not be mapped to U+00F8 (LATIN SMALL
> LETTER O WITH STROKE).
>
> <ADDED> An example of an important issue that is not considered in
> detail in IDNA is how to provide a high probability that a user who is
> entering a domain name based on visual information (such as from a
> telephone or radio) would correctly enter the IDN. This a complex issue
> relating to languages, input methods on computers, and so on.
> Furthermore, the kind of matching and searching necessary for a high
> probability of success would not fit the role of the DNS and its exact
> matching function.

there are other hot topics which might be well-served here, specifically
including the issues with local functions (clipboards, piping, etc),
dangers from auto-conversion (message-id, searching), etc.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/