Subject: [idn] IDNA questions

From: Erik Nordmark <[email-masked]>
Date: Mon, 6 May 2002 17:56:11 +0200 (CEST)

Folks,

I've been trying to understand certain aspects around the handling of
ASCII vs. non-ASCII code points in IDNA, nameprep, and stringprep
but I don't yet have good arguments for all of them.

Before I start the last call I'd like to at a minimum have good arguments
for why things are done the way they are.

The ASCII world
---------------

This is the way things look like in the non-IDN world today.
We have two types of domain name labels which I'll call
host name labels (HL) and non-host name labels (NHL) in this writeup.
Most actual usage are HLs but e.g. SRV records are a limited extension
to NHL (SRV records are named by two labels which include a leading U+005F).

The syntax for NHL allows any byte value i.e. 00..FF.
The syntax for HL is letter+digit+hyphen but a label can not
have a leading or trailing hyphen.
As I understand both HL and NHL are compared using case-insensitive
matching. And case-insensitive matching is not well defined outside
of 00..7F. Thus well-defined NHL seems to in practise be limited
to 00..7F.

The HL are compared with case-insensitive matching on DNS servers, but
there might be some applications or DNS client software that modify
their case which still works.

If an application wants to compare two domain names it can use
a fairly efficient piece of code (e.g. strcasecmp offers such a thing
on certain Unix platforms).

IDNA specifications
-------------------

The state of the IDNA specifications is as follows.
We have two new types of domain name labels which I'll 
call IHL and INHL in this writeup. Both IHL and INHL can be ACE encoded
with the resulting label being a HL in most cases. (An INHL with ASCII 
characters outside of LDH will get encoded as an ACE that is a NHL; see 
example (S) in punycode).

The difference between the allowed code points in IHL and INHL are
not that great. The only difference is whether ASCII code points outside
of LDH are allowed; they are allowed in INHL but not in IHL.
Both IHL and INHL prohibit the same set of code points outside of the
ASCII range (they are all specified in stringprep).

Thus e.g. space and control characters are not allowed in INHLs.
And punctuation, symbols, etc are allowed in IHLs.

The handling of case is as follows:
If a label consists of all ASCII code points the case is not modified -
the label is not subject to nameprep.
If the label consists of some ASCII code points and some non-ASCII,
then all code points are case folded - including the ASCII code points.
This prevents ToASCII from changing the case of an all-ASCII label.

The IDNA specification says
    This document does not attempt to define an "internationalized host
    name".  It is expected that protocols and name-handling bodies
    will want to limit the characters allowed in IDNs further than
    what is specified in this document, such as to prohibit additional
    characters that they feel are unneeded or harmful in registered
    domain names.

The "protocol" part could be interpreted to mean (at least) 
two slightly different things:
 - That a future IETF standard will define internationalized host names
   and in the process exclude some code points from the set allowed in 
   IHL.
 - That a future application protocol might decide that certain code points
   should not be allowed to specify a host name as used by that application
   protocol.

If an application wants to compare two domain names it needs to perform
ToASCII on those strings and then compare them using case-insensitive
comparison. (See section 3 in the idna spec.)
If the application is going to do repeated comparisons (e.g. against
a list of domain names) in order for it to perform well the application
is likely to end up storing the result of ToASCII for the domain names
it compares against.

My questions
------------

1. Allowed characters in INHL.

For some code points (like unassigned, private use, non-character, and 
surrogate code points) it seems to make sense to prohibit them in both 
IHL and IHNL. In fact, unassigned code points MUST be prohibited in 
both for proper handling as they get assigned. [The fact that this only 
applies to stored strings is critical in general, but not important in 
this discussion.]

But why not take the same approach as for ASCII and allow space characters,
control characters, etc in INHL?
It might not matter much since protocols will (mostly) use IHLs but it
would make things follow the same philosophy as in the ASCII case.

The argument for doing things the way they are seems to be that
 - The goal is to enable the richer set of code points to be used by 
   applications
 - The existing of of non-LDH domain names in ASCII are SRV and the data
   part of the SOA record. Those records work fine in this approach.
 - While one could worry more about non-host name syntax there aren't
   clear arguments for either approach (allowing as many code points
   as possible vs. preventing the ones that obviously don't make sense
   in names), so just pick the simplest approach.
   The current approach is simple in that nameprep/stringprep is the
   same for host names and non-host names.

Are there other arguments for the current approach?

2. Initially allowed characters in IHL.

The above quote from the IDNA specification with the possible interpretations
of "protocol" doesn't provide much guidance to registries on what code points
can be safely registered i.e. does not risk to be 
declared "not a host name" in some later IETF standard.

I think there can be two different intents here:
1. The intent is that all restrictions be solely by registries ("name-
   handling bodies") based on their local policies of what to register.
   No application protocols will restrict the set of allowed code points
   beyond what is currently specific in IDNA+nameprep+stringprep.
2. The intent is at a later date the IETF will produce a specification
   for host name syntax (either for specific application protocols
   or in general) that will place some additional restrictions.

If #1 is the case then as far as I can tell "protocol" should be removed
from the sentence.

If #2 is the case then I think the document needs to provide some
guidance on what set of code points users and registries should
use in order to minimize the risk that their domain names be
declared invalid host names at a future date.

Which is the intent here?

If #2 then why can't the document explicitly state that the intent is 
that no IETF standard restrict what can be used as an internationalized 
host name by prohibiting <list of code points>?
Presumably <list of code points> could be a reference to a set of
general categories in the Unicode standard.

This would not prevent a future IETF standard to declare that a larger
set of code points can be used in internationalized host names.
And it would not prevent a particular registry to have a policy that
only allows a smaller set of code points.

3. Downcasing of ASCII

Avoiding downcasing of ASCII code points by ToASCII in an all-ASCII label
seems a bit odd. I understand that this makes an IDN-aware application
generate the same DNS queries on the wire as the application before it
was made IDN-aware (i.e. if the application looked up "Example.COM"
it would appear identically in the DNS packet after the application
was made IDN-aware). But is this important?

The reasons I've seen seem to be that some applications would take
the result of ToASCII and use it in protocol fields such as
the From: in SMTP. The effect of having ToASCII downcase an all-ASCII
label is that users will see different case characters when they
upgrade to IDNA-aware applications even though they do not use
an IDN themselves. Thus there might be some noticeable short-term effect
depending on exactly how the IDNA-aware application is coded.

The reason I'm asking is because when application protocols start carrying
IDNs in a non-ACE encoding (people seem to think that this will
happen sooner or later - perhaps for brand new application protocols and
perhaps by making some existing application protocols negotiate the 
encoding)  there will still be a need to nameprep those domain names in 
order for  comparisons to work. However, the ACE encode/decode isn't needed 
except to deal with peers that do not handle the new on-the-wire encoding.
[NOTE: I'm not talking about the DNS as a potential application protocol
here. But perhaps SMTP could be handled this way.]

This can be simply handled when that day comes by splitting
ToASCII into two parts: approximately step 1-3 to prepare in order for
comparisons to work and approximately step 4-7 for ACE conversion.

When that day comes the comparisons of the prepared labels still needs
to use case-insensitive matching for 00..7F code points just because
the preparation didn't downcase the ASCII code points. But the non-ASCII code
points don't need this i.e. applying all of the Unicode Fold() isn't
necessary.

Thus the current approach seems a bit short-sighted unless there
is a good reason that I don't understand for not case folding ASCII-only
labels.

4. Comparison

Section 3 item 3 seems to implicitly prevent Unicode-aware applications
from comparing two nameprepped strings without first converting them to ACE.
Why can't it be made explicit that, as a result of the design of the ACE
algorithm being 1-1, two Unicode labels compare as equal if and only if
the corresponding ACE labels compare as equal?

This would mean that an application, as long as it has applied
ToUnicode on the strings when received from an input method, the
protocol, or some other source, it could do a comparison without
ACE conversion.
And if case folding was applied to ASCII-only labels just like
all other labels, then the resulting comparison would just need
to be a code point by code point comparison.

---

  Erik

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 7 May 2002 04:33:51 +0000

Erik Nordmark <[email-masked]> wrote:

> IHL and IHNL.

Agreed.

> But why not take the same approach as for ASCII and allow space
> characters, control characters, etc in INHL?

Good question.  I wouldn't mind seeing those characters allowed in
non-host IDNs.

> The above quote from the IDNA specification with the possible
> interpretations of "protocol" doesn't provide much guidance to
> registries on what code points can be safely registered i.e. does not
> risk to be declared "not a host name" in some later IETF standard.
> 

The reason the intent is unclear is that the IDNA authors never agreed
on the intent, so we agreed on a document that abstained from expressing
the intent.

I personally would like to see #2.  I think the two-syntax model has
served us well.  Today's DNS supports an anything-goes syntax (arbitrary
ASCII) and also defines a restricted well-behaved syntax (LDH host
names).  Having an anything-goes syntax means arbitrary protocols with
bizarre names can still use the DNS.  Having a restricted well-behaved
syntax means people can register names that are almost certain to work
in any protocols that use domain names, and protocols can use almost any
characters they like to delimit names in larger structures.

Notice that the additional host syntax restrictions are usually not
enforced by applications as names are being passed around and compared.
They are usually enforced only when names are created.

> If #2 then why can't the document explicitly state that the
> intent is that no IETF standard restrict what can be used as an
> internationalized host name by prohibiting <list of code points>?

That would be a good idea if we settled on #2.

> set of code points can be used in internationalized host names.  And
> it would not prevent a particular registry to have a policy that only
> allows a smaller set of code points.

Right.

> Avoiding downcasing of ASCII code points by ToASCII in an all-ASCII
> label seems a bit odd.
>
> the result of ToASCII and use it in protocol fields such as the
> From: in SMTP. The effect of having ToASCII downcase an all-ASCII
> upgrade to IDNA-aware applications even though they do not use an IDN
> themselves. Thus there might be some noticeable short-term effect

Yes, that is my main concern.  And also the general principle that
extending a standard should not alter the behavior of things that were
already covered in the old standard.

> The reason I'm asking is because when application protocols start
> carrying IDNs in a non-ACE encoding there will still be a need
> to nameprep those domain names in order for comparisons to work.
>
> the preparation didn't downcase the ASCII code points.

The IDNA spec does not require applications to actually perform ToASCII
and actually perform a case-insensitive ASCII comparison when comparing
labels.  It merely requires applications to use a procedure that yields
the same answer as that procedure.  One procedure that meets the
requirement is to perform a modified ToASCII that always outputs lower
case everything, and then perform an exact comparison.

Would you like us to mention that optimization in the spec?  We have
generally decided not to describe optimizations in the spec, but rather
let the spec define correct externally-visible behavior and not concern
itself with internal implementation issues.

> Thus the current approach seems a bit short-sighted unless there is a
> good reason that I don't understand for not case folding ASCII-only
> labels.

I think maintaining the current behavior for existing names is a good
reason.

> Section 3 item 3 seems to implicitly prevent Unicode-aware
> applications from comparing two nameprepped strings without first
> converting them to ACE.  Why can't it be made explicit that, as a
> result of the design of the ACE algorithm being 1-1, two Unicode
> labels compare as equal if and only if the corresponding ACE labels
> compare as equal?

Correction:  Two non-ACE labels compare as equal if and only if the
corresponding ACE labels compare as equal.  It's a subtle but important
distinction.

> protocol, or some other source, it could do a comparison without ACE
> conversion.
>
> And if case folding was applied to ASCII-only labels just like all
> other labels, then the resulting comparison would just need to be a
> code point by code point comparison.

Be careful.  Comparing ToUnicode(X) with ToUnicode(Y) is not a
valid comparison of X and Y, because the output of ToUnicode is not
necessarily nameprepped.  An exact comparison of Nameprep(ToUnicode(X))
and Nameprep(ToUnicode(Y)) is the same as a case-insensitive ASCII
comparison of ToASCII(X) and ToASCII(Y) when X and Y are valid labels.
But when X and Y are invalid, the former comparison might return a match
when the latter comparison would not (because ToASCII would fail).  If
you don't care about the result of comparing two invalid labels (and I
suppose you often wouldn't), then you could use the former comparison.

Would you like this alternate comparison procedure described in the
spec?  Some might see it as unnecessary bloat.  I know a faster but more
complex method of performing ToUnicode, and I once suggested including
both versions in the spec, but we ultimately decided to keep the spec
short and leave optimizations to the implementors.

Maybe an informational RFC full of optimizations could be drafted after
the standard is out of the way.

AMC

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 7 May 2002 10:17:39 +0200 (CEST)

> itself with internal implementation issues.

In general I don't think the spec needs to list all possible optimizations.
But the fact that the spec says 
	3) Whenever two labels are compared, they MUST be considered to
	match if and only if their ASCII forms (obtained by applying ToASCII)
	match using a case-insensitive ASCII comparison.
it would be helpful to point out that this doesn't require applying
ToASCII for every comparison.

> comparison of ToASCII(X) and ToASCII(Y) when X and Y are valid labels.

I think this makes sense pointing out in the above text.

  Erik

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 07 May 2002 22:52:03 -0500

A couple of comments on this discussion.

> Good question.  I wouldn't mind seeing those characters allowed in

Let me also add my support for this. Protocols should be able to use the
full US-ASCII range of 0x00-0x7F if they really have a need to do so. This
is already supported in STD13, so not allowing them is guaranteeing
eventual problems.

> > upgrade to IDNA-aware applications even though they do not use an
> > IDN themselves. Thus there might be some noticeable short-term
> > effect depending on exactly how the IDNA-aware application is coded.
> 

Dan O also brought up the issue wrt PTR labels providing domain names as
RR data which are sometimes used with case-sensitive access maps. Probably
not the best programming in the world but it's out there and also
supported by STD13.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/