Subject: [idn] rederivation of an IDN architecture

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 9 Aug 2001 02:59:38 +0000

idn-arch-deriv 0.0.0 (2001-Aug-09-Thu)
Adam M. Costello <[email-masked]>

This is a derivation of an Internationalized Domain Name architecture
that is extremely similar (if not perfectly identical) to IDNA.  My
purpose in composing this derivation is to expose dependencies between
requirements, goals, and assumptions, in order to facilitate fruitful
discussions.  For example, if a certain requirement is necessary to
achieve a certain goal, it is pointless to attack the requirement when
arguing with someone who believes in the goal; ones needs to argue
against the goal itself, and perhaps against the assumptions underlying
the goal.

The bulk of this derivation is definitions.  Readers may wish to skip
the definitions the first time through, and refer back to them when
necessary.

Since there is a preexisting standard for domain name syntax, it is
useful to have some definitions for talking about that syntax:

    basic character:  Any of the 128 characters in the ASCII repertoire
    (not to be confused with the integers or octets mapped to those
    characters in the ASCII coded character set).

    basic string:  A sequence of basic characters.

    tolower():  A function from a basic string to a basic string.  It
    converts A-Z to a-z respectively, and leaves the other characters
    unchanged.  [Observation: tolower() is idempotent, that is, for any
    basic string B, tolower(B) = tolower(tolower(B)).]

    equivalent basic strings:  Basic strings for which tolower()
    gives identical results.  [Observation:  Because tolower() is
    idempotent, a basic string and its tolower() image are equivalent
    basic strings.]

    basic label:  A basic string containing from 1 to 63 characters.
    [This is the preexisting standard for textual domain labels.]

    host restrictions (applicable to any sequence of characters):
    Contains no basic characters other than the letters, digits, and
    hyphen-minus; and neither begins nor ends with hyphen-minus.

    basic host label:  A basic label that satisfies the host
    restrictions.  [This is the preexisting standard for host labels.]

    equivalent basic labels:  Basic labels that are equivalent basic
    strings.  [This is the preexisting standard for equivalence of
    textual domain labels.]

    basic domain name:  A sequence of basic labels (typically separated
    by full-stop characters).  [This is the preexisting standard for
    textual domain names.]

    basic host name:  A basic domain name for which every label is a
    basic host label.  [This is the preexisting standard for internet
    host names.]

    equivalent basic domain names:  Basic domain names that have the
    same number of labels and for which corresponding labels are
    equivalent basic labels.  [This is the preexisting standard for
    equivalence of textual domain names.]

    domain name slot:  A field or sub-field in a protocol message, or a
    function argument in an interface, etc, explicitly designated for
    carrying a domain name.

Fact 1:  Many existing protocols, interfaces, etc. have domain name
slots and require the names in those slots to conform to the syntax and
equivalence rules for basic host names (or at least basic domain names).

Fact 2:  Even when existing protocols, interfaces, etc. do not
require domain names to conform to the preexisting syntax, different
implementations of the same protocol/interface will sometimes behave
differently if presented with domain names that do not conform, because
there has been no preexisting standard for how such domain names should
be handled.  (For example, some implementations will reject the names,
some will alter the names, and some will pass them through unchanged.
There is similar uncertainty when names are compared.)

Fact 3:  Many people would like to use domain names containing
additional characters not allowed by the preexisting syntax.

Conjecture 1:  Many people will be reluctant to create domains using
additional characters if those domains would be inaccessible to
preexisting protocols, interfaces, software, etc.

Goal 1 (motivated by facts 1-3 and conjecture 1):  Allow domain names to
contain additional characters, while still allowing all domains to be
referred to by all preexisting protocols, interfaces, software, etc.

Requirement 1 (follows from goal 1):  Part of the preexisting namespace
must be used to represent names containing additional characters,
which requires altering the semantics of that part of the preexisting
namespace.

Goal 2:  The part of the preexisting namespace that is repurposed should
be a part that is not currently in use, or at least is used to an
extremely small extent, so that disruption is minimized.

Goal 3:  The part of the preexisting namespace that is repurposed
should be easily identifiable and easily avoidable (both by humans and
machines).

Fact 4:  Many protocols and standards split domain names into labels,
re-join them, count the number of labels, and test whether two domain
names agree on individual labels.

Goal 4 (motivated by fact 4):  The way that preexisting names are used
to represent names containing additional characters should not disrupt
operations that treat labels individually.

Requirement 2 (follows from goals 2-4):  The representation of
previously disallowed names by previously allowed names must be done
independently per label, and labels using this representation must have
a well-known prefix or suffix that was almost never in use.

Fact 5:  Internet protocols usually support US-ASCII, and future
internet protocols are encouraged to support UTF-8.

Fact 6:  UTF-8 is an extension of US-ASCII, that is, characters in the
ASCII repertoire are represented by the same bytes in UTF-8 as in ASCII.

Goal 5 (motivated by facts 5 and 6):  Given a protocol that uses ASCII
to encode domain names, it should be easy to design an updated version
of the protocol that uses UTF-8 to encode domain names.

Requirement 3 (follows from goal 5):  The preexisting restrictions on
what ASCII characters are allowed in host names, and in what positions,
must be retained in the expanded namespace.

Requirements 1, 2, and 3 motivate the following definitions for the
expanded namespace:

    international character:  A character in the Unicode repertoire (not
    to be confused with the integers or byte sequences used to represent
    those characters in the various encodings defined by the Unicode
    standard).

    international string:  A sequence of international characters.
    [Observation:  Every basic string is an international string.]

    canon():  A function (defined elsewhere) from an international
    string to an international string.  It is an extension of tolower(),
    that is, for any basic string B, canon(B) = tolower(B).  Like
    tolower(), canon() is defined everywhere and is idempotent (canon(S)
    = canon(canon(S)) for any international string S), but unlike
    tolower(), canon(S) and S might have different lengths.

    equivalent international strings:  International strings for which
    canon() gives identical results.  [This is consistent with the
    definition of equivalence of basic strings, because canon() behaves
    like tolower() for basic strings.  Observation:  Because canon()
    is idempotent, an international string and its canon() image are
    equivalent international strings.]

    degenerate international string:  An international string that
    is equivalent to a basic string.  [The definitions of equivalent
    international strings and canon() imply that an international string
    S is degenerate iff canon(S) is a basic string.]

    ACE prefix:  A particular sequence of basic characters (defined
    elsewhere).  [A suffix would work as well as a prefix.  The phrase
    "begins with the ACE prefix" should be understood to mean "ends with
    the ACE suffix" if a suffix is used instead.]

    compat():  A function (defined elsewhere) from an international
    string to a basic label.  For some international strings S,
    compat(S) does not exist, either for policy reasons, or because
    of the length restriction on basic labels.  The compat() function
    satisfies the following properties:

        Stability:  For any basic string B, compat(B) = B.

        Preservation of equivalence:  For any equivalent international
        strings S1 and S2, if compat(S1) exists, then compat(S2) exists,
        and compat(S1) and compat(S2) are equivalent basic labels.

        Preservation of non-equivalence:  For any nondegenerate
        international strings S1 and S2 for which compat(S1) and
        compat(S2) exist, if S1 and S2 are not equivalent international
        strings, then compat(S1) and compat(S2) are not equivalent basic
        labels.

        Preservation of host restrictions:  For any international string
        S, if S satisfies the host restrictions and compat(S) exists,
        then compat(S) satisfies the host restrictions.

        Tagging:  For any nondegenerate international string S, if
        compat(S) exists then it begins with the ACE prefix.

        Non-recursion of tagging:  For any nondegenerate international
        string S that begins with the ACE prefix, compat(S) does not
        exist.

    [Observation:  The preservation of equivalence and non-equivalence
    implies that the definition of compat() will include the definition
    of canon().]

    international label:  An international string S for which compat(S)
    exists.  [Observation:  Every basic label is an international
    label.]

    equivalent international labels:  International labels for which
    compat() yields equivalent basic labels.  [This is consistent with
    the definition of equivalent basic labels, because compat(B) = B for
    any basic label B.  Observation:  For any international labels L1
    and L2 for which canon(L1) and canon(L2) do not begin with the ACE
    prefix, L1 and L2 are equivalent international labels iff they are
    equivalent international strings.]

    international domain name:  A sequence of international labels
    (typically separated by full-stop characters).  [Observation:  Every
    basic domain name is an international domain name.]

    equivalent international domain names:  International domain names
    that have the same number of labels and for which corresponding
    labels are equivalent international labels.  [This is consistent
    with the definition of equivalent basic domain names, because
    equivalence of international labels is consistent with equivalence
    of basic labels.  Observation:  For any international domain name
    there exists a basic domain name equivalent to it, which can be
    obtained by applying compat() to each label.]

    decompat():  A function from basic labels to international labels.
    For any basic label B, if there exists a nondegenerate international
    string S such that compat(S) and B are equivalent basic labels, then
    S and decompat(B) are equivalent international labels, otherwise
    decompat(B) = B.

    simple label:  A basic label that does not begin with the ACE
    prefix.  [Observation:  Simple labels have the same semantics as in
    the preexisting standard.]

    ACE label:  A basic label that is altered by decompat().
    [Observation:  All ACE labels begin with the ACE prefix.  ACE labels
    have new semantics not present in the preexisting standard.]

    invalid label:  A basic label that begins with the ACE prefix but is
    not altered by decompat().  [Observation:  Every basic label that is
    neither a simple label nor an ACE label is an invalid label.  These
    labels carry a semantic dubiousness that was not present in the
    preexisting standard.]

Fact 7:  Software that was not deliberately written to handle
international domain names cannot properly compare two international
domain names.

Fact 8:  Even though many software entities do not compare domain
names, it is common for them to pass domain names on to other software
entities, which might need to compare them.

Goal 6 (motivated by facts 2, 7, and 8):  Whenever possible, keep
non-basic international domain names out of the hands of software that
was not deliberately written to handle them.

The following definition is motivated by goal 6:

    international domain name slot:  A domain name slot explicitly
    designated to carry an international domain name.  The designation
    may be static (for example, in the specification of the protocol or
    interface) or dynamic (for example, as a result of negotiation in an
    interactive session).

Requirement 4 (follows from goal 6):  Non-basic international domain
names must not be put into non-international domain name slots.  (But of
course the equivalent basic domain name, obtainable via compat(), can
be put into the slot instead.)

Observation 1:  Obviously, requirement 4 can apply only to IDN-aware
entities.  A non-IDN-aware entity is not normally supposed to have any
non-basic international domain names in the first place, but if it
somehow obtains one, perhaps via manual entry, then of course it cannot
be expected to obey a requirement that it knows nothing about.  It will
obey whatever requirements it was written to obey.

Observation 2:  DNS is a protocol with non-international domain name
slots (in both directions).  Therefore requirement 4 applies to those
slots.  Extensions to DNS could of course define new fields that are
international domain name slots.

Observation 3:  Existing resolver interfaces have non-international
domain name slots (in both directions).  Therefore requirement 4 applies
to those slots (regardless of whether the resolver is IDN-aware, and
regardless of whether it makes use of any DNS extensions).  New resolver
interfaces could of course define new arguments that are international
domain name slots.

Goal 7:  Whenever possible, show users the meaningful form of an
international domain name rather than the ACE form (unless they request
otherwise).

Requirement 5 (follows from goal 7):  IDN-aware entities should by
default apply decompat() to all domain labels before displaying them
to the user, and before putting the domain name into an international
domain name slot likely to be seen by a user.

End of idn-arch-deriv.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 9 Aug 2001 06:38:36 -0000

> preexisting protocols, interfaces, software, etc.

No. All that people care about is whether the software works. Protocols,
interfaces, and so on are relevant only to the extent that they help the
software work. Also, people don't care about ``preexisting'' software
unless that software is still in use.

> referred to by all preexisting protocols, interfaces, software, etc.

Here you've lost sight of the goal of actually DISPLAYING THE NAMES
PROPERLY. This is the whole point of internationalized domain names!

ACE IDNs don't work today, and they aren't going to work in the
foreseeable future. They can, however, be ``referred to.'' Given these
facts, and given your confusion over the basic goal, it's entirely
unsurprising that you end up advocating ACE IDNs.

---Dan

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 09 Aug 2001 08:39:57 +0100

> unless that software is still in use.

Everyone want functionality, but we seem to disagree on how to reach it.

People come to the IETF because they see a standard is needed to ensure
functionality. If a standard was not needed (and you claim that a standard
is not needed, as software already "work" (we have all seen the results of
the tests on this list that we obviously disagree on what "work" means)).

If you don't want standards, or don't think they are needed, please don't
disrupt the people which are discussing new standards in the IETF.

The IETF is not forcing, and can not force people using it's standards.
People can either follow them or not. It is the choice of the implementors.

   paf

---

From: "Barr Hibbs" <[email-masked]>
Date: Thu, 9 Aug 2001 06:41:34 -0700

Adam--

thank you for your well-considered work!  I have a few minor comments...

>     characters in the ASCII coded character set).
...ignoring the ability of DNS servers to transparently handle binary
labels, I think you really mean that a basic character is any one of the
characters which (a) has an associated glyph, (b) is the "non-printing"
space character, or (c) is a diacritical or punctuation symbol.  These
limitations, which essentially are required for characters to be
"human-readable," result in the set of characters being 31-127 of the whole
set.

>     hyphen-minus; and neither begins nor ends with hyphen-minus.
...this is not quite correct:  DNS does not make this restriction, although
I believe it is correct for the current standard for host names...

...also, I would observe that certain other characters, such as the
underscore ("_") are commonly permitted in URLs, which many non-experts find
indistinguishable from host names...

>     by full-stop characters).  [This is the preexisting standard for
...one should be very clear that the label element separator (full-stop) is
the x'2E' character (".") and not something similar

> which requires altering the semantics of that part of the preexisting

--Barr

---

From: "D. J. Bernstein" <[email-masked]>
Date: 9 Aug 2001 23:10:52 -0000

Patrik writes:
> you claim that a standard is not needed, as software already "work"

Liar. I have consistently stated that some software needs to be upgraded
for UTF-8. The reason I support this approach is that it will give us
working IDNs much sooner and at much lower cost than the ACE approach.

---Dan

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 10 Aug 2001 00:49:06 +0000

"D. J. Bernstein" <[email-masked]> wrote:

> > Goal 1
> PROPERLY.

That's goal 7.

Barr Hibbs <[email-masked]> wrote:

> I think you really mean that a basic character is any one of
> the characters which (a) has an associated glyph, (b) is the
> "non-printing" space character, or (c) is a diacritical or punctuation
> symbol.

In this section I was merely trying to write formal definitions of
existing standards.  The existing domain name standards talk about ASCII
characters and the more limited set of LDH characters, but as far as I
know they do not mention the set of printable ASCII characters.

> ...this is not quite correct:  DNS does not make this restriction,
> although I believe it is correct for the current standard for host
> names...

I never claimed that DNS made this restriction.  I just wanted to have
the term "host restrictions" available for defining "basic host label"
and later for defining one of the properties of the compat() function.

Thanks for the feedback,
AMC