Subject: [idn] proposed i18n naming rules

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 21 Nov 2001 13:04:13 -0600

Does this meet the consensus:

Internationalized domain names have the following attributes:

  sequence of labels:

      labels contain any valid UCS character code, regardless of
        whether or not a character representation is assinged to that
        character code

      character codes are explicit, not required to be normalized in
        any way

      minimum label length of 1 UCS character code

      maximum label length of 63 UCS character codes

  maximum FQDN length of 255 UCS character codes (including the
    label separators when the IDN is written out)

Internationalized host identifiers are a subset of internationalized
domain names, and have the following attributes:

  sequence of labels:

      only those UCS characters from the "safe-set"

         [version of] http://www.unicode.org/Public/UNIDATA/Scripts.txt

         U+002D (HYPHEN-MINUS)

      MUST be normalized through nameprep prior to processing

      leading and trailing characters MUST NOT be HYPHEN-MINUS

      minimum label length of 2 UCS characters

      maximum label length of 63 UCS characters

  maximum FQDN length of 255 UCS character codes (including the
    separator characters when the IHI is written out) 

  U+002E (FULL STOP) is ONLY valid for use as a separator when
    the IHI is written out

The suggestion would be that the IHI rules would apply to all encoding and
decoding systems which knew that an IDN specifically referenced a host.
This would mean applications that were issuing a lookup for the purpose of
establishing a connection, protocol operations that would use a domain
name for a host-specific function (such as message routing, HTTP HOST
headers, NS RRs in DNS where the NS explicitly referenced a host, and so
forth), and processes which decode structured data (such as email
addresses and URLs) for display.

All other IDNs would fall under the IDN rules. This includes SRV entries
in the DNS, email addresses stored in SOA and RP RRs (which can contain
FULL STOP, among others), and so forth.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Wed, 21 Nov 2001 14:39:06 -0500

Eric,

> maximum label length of 63 UCS character codes

q: are you confusing octets and characters?

>      [version of] http://www.unicode.org/Public/UNIDATA/Scripts.txt

q: is there mechanism for extension? is it in the ietf, or in the utc?

>      minimum label length of 2 UCS characters

q: is two really better than one?

>      maximum label length of 63 UCS characters

q: are you confusing octets and characters?

>  U+002E (FULL STOP) is ONLY valid for use as a separator when

i don't know what this means.

that said, it beats the heck out of what the chair(s) have to offer.

eric

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 21 Nov 2001 13:45:35 -0600

> q: are you confusing octets and characters?

No. The characters/codes are the canonical representation of the IDN, not
the characters which are encoded and encapsulated in a data stream. The
encoded values will have their own length restrictions when they are
encapsulated. EG, ACE will provide a maximum of 59 octets when the
four-octet prefix is subtracted from the maximum encapsulation length of
63 octets, but this is a secondary restriction apart from the canonical
representation restrictions.

> q: is there mechanism for extension? is it in the ietf, or in the utc?

Undefined in the rules. Got any ideas here?

> q: is two really better than one?

Not a question I can answer, alas.

> i don't know what this means.

It means that "." is legal as a label separator, but not as label data.

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Wed, 21 Nov 2001 15:01:33 -0500

> Undefined in the rules. Got any ideas here?

so suppose nii quanor and co come to the ietf and the utc and say ...

	"we've got our character repitoire" (unified central african syllabics)

do we tell him "go see the utc", or do we tell him "we've got a rule, and it
is slightly more complex than `go see the utc'"?

i prefer (b), and i know that many here prefer (a).

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 21 Nov 2001 14:01:16 -0600

> domain names, and have the following attributes:

add[?]:

    the FQDN MUST contain at least one character that is not a
      Latin number (U+0030...39)

---

From: "ben" <[email-masked]>
Date: Wed, 21 Nov 2001 15:17:46 -0500

> FULL STOP, among others), and so forth.

Since we are on the subject of "full stop", I remember that it was
brought up some time ago that the "Chinese full stop" should serve the
same purpose as the "English full stop"- from what I can tell, it
currently appears to be still an illegal character.  This certainly is
indeed an important issue for users of CDNs and hopefully, this
problem can be address at the same time.

For example, CDN users can easily type in <whatever>。TLD (instead of
<whatever>.TLD) and have no clue why he gets an error.

Thanks,
Ben Chan

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Wed, 21 Nov 2001 15:34:37 -0500

Eric,

>      Latin number (U+0030...39)

1591 isn _not_ controlling, and even section 3 of that document has seen
better days. the authors of 2929, were unable to find technical necessity
to constrain the values in the right-most label to contain one or more
non-digits.

one can, as john has, argue for taste, or invoke jon postel's practice,
but those are not "international" nor are they part of the protocol.

this bit, if it needs to exist, is something to toss _now_ over the fence
to dnsop, for a bcp on bozo-nonos, like "don't create dns names that look
a heck of a lot like a dotted quad (or a v6 addr either)"

is this really necessary? imo, no.

eric

---

From: [email-masked]
Date: Thu, 22 Nov 2001 11:47:54 +1100

>   U+002E (FULL STOP) is ONLY valid for use as a separator when

	Does any one see that this *will just not work*.  This has
	unnormalised sequences of labels in the query trying to be
	matched against normalized labels in the resulting from
	delegation process.

> FULL STOP, among others), and so forth.

	Mailboxes after the first label are subject to the same contraints
	as hostnames.  I would also argue that SRV ownernames are also
	subject to the same constraints as hostnames once you remove the
	first two labels.  In otherwords you can't assume that one set
	of rules about what is legal applies to all labels in a domainname.

	Mark
--
Mark Andrews, Internet Software Consortium
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 21 Nov 2001 19:15:02 -0600

>         matched against normalized labels in the resulting from

The syntax rules above say nothing about queries versus responses. The
clarification at the bottom should make it clear that systems which are
issuing lookups for the purpose of connecting to a host would be using the
hostname ruleset, which requires normalization.

Are you referring to lookups of IDNs (as opposed to IHIs) that would not
be normalized?

>         of rules about what is legal applies to all labels in a domainname.

Yeah, good point.

---

From: [email-masked]
Date: Thu, 22 Nov 2001 13:49:36 +1100

> Are you referring to lookups of IDNs (as opposed to IHIs) that would not

	I'll put it another way.  Your plan requires that registries
	delegate both normalised and (potentially multiple)
	unnormalised forms of every delegation.  If they just
	delegate the normalised form the unnormalised request may
	not match against normalised delegation.

	You are applying two *different* sets of transformations
	to generate the wire form depending the query type being
	looked up.  Delegations are independent of the query type
	being looked up.

	Mark

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 21 Nov 2001 20:59:07 -0600

> of rules about what is legal applies to all labels in a domainname.

Some points on this after having thought about it some. Overall, I think
the issues you raise have to do with the use of IDNs as protocol data and
are not necessarily problems with the syntax. As such, the proper place to
fix these problems would be in the protocols.

First, there is nothing in RFC 2782 that says SRV lookups "should ensure
the rest of the FQDN is legitimate." But this text (and probably some
specific guidance) SHOULD be there. In that regard, a malformed lookup for
an SRV entry in an IDN hierarchy will work exactly the same as a malformed
lookup for an SRV entry in an STD13 hierarchy: they will both fail.

The fix to this problem (and others like it) is to fix the source: "if you
want SRV lookups to succeed (regardless of whether they are IDNs or STD13
DNs), make sure the rest of the FQDN is legitimate." A more specific
solution would be to suggest that clients treat labels to the right of the
protocol elements as hostname labels, and to reject those which do not
conform to [LDH|IHI] rules.

As for email addresses in SOA and RP RRs, those are provided as part of
the RR data and are not specified in queries; queries for them cannot be
formed so queries for them cannot break. The proper place to deal with
IDNs as RR data is when the domain name is added to the zone.

For all other protocol uses of IDNs, a word of advice on this point is in
order, but the protocols should be fixed.

I guess I'm saying that I don't see the issues you raise as inherent
problems with the syntax structures, since the conflict is not the syntax
but in the protocol operations that utilize the syntax. The protocol
operations is the appropriate point of focus.

[OT for this particular discussion, but I will add the above changes to
the next rev of the dual-mode draft when I add the new syntax rules.]

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 21 Nov 2001 20:59:02 -0600

> not match against normalised delegation.

Why do illegal delegations have to be provided? Illegal queries do not
have to be matched. The onus is on the client to ensure they generate
valid lookups, not the server.

---

From: [email-masked]
Date: Thu, 22 Nov 2001 14:34:47 +1100

> valid lookups, not the server.

	Shall I put it in ASCII terms. 

	The DNS is now case sensitive.
	When looking up HOSTNAMES you will down case all entries.
	When looking up non hostnames preserve the case.

	Now lookup "foo.ExAmPlE.com" for both TXT and A records.

	Mark

---

From: [email-masked]
Date: Thu, 22 Nov 2001 14:58:31 +1100

> conform to [LDH|IHI] rules.

	That's why I said that I would *argue* that it should be that way.

> formed so queries for them cannot break.

	They have been in the past (see MB).  And could be in the future.

	Mailboxes are how you reference people, rather than hosts,
	by name in the DNS.  They arn't expected to have address
	records or MX records but are expected to have other types.

> The proper place to deal with IDNs as RR data is when the domain name is
> added to the zone.
> 

	You can't have labels undergoing different transformations
	depending upon what is being looked up.

	The only thing that can change between a IHN and a IDN is
	that IDNs that are not IHNs should be rejected by the
	lookup tools.  Similarly the zone maintence tools should reject
	(by default) IDNs that are not IHNs when used in a context that
	requires a IHN.

	Mark

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 21 Nov 2001 22:34:31 -0600

>         Shall I put it in ASCII terms.

Whatever works

>         Now lookup "foo.ExAmPlE.com" for both TXT and A records.

I assume you mean "using dig or some other application which doesn't have
a specific <protocol> usage for the entry it's requesting."

> depending upon what is being looked up.

My argument at this point is that the transformations can/should occur
based on what is doing the lookup. I agree from your point above that this
isn't always viable either.

> (by default) IDNs that are not IHNs when used in a context that

It may be that the only way to completely and totally solve this problem
is to put case-neutral comparison back into the servers. Dan Oscarson has
been arguing for this and you may have made his case for him. This is
pretty complex due to several reasons but it may be the only way to
ultimately resolve this.

Another possibility is to fix the protocol usages like I've been arguing,
and do something with the generic tools such as dig. Note that the legacy
tools aren't affected because those tools don't have access to native IDNs
and case-neutral comparisons have to be performed for ASCII alpha in IDNs,
so a "fix" for those tools would only apply to the i18n compliant
versions. This could be something stupid (I beat you to it) like a "-ce"
switch for "case-exact" and downcase everything else.

As for rejecting entries in the zone, has this proven to work? Is there
sufficient experience with this to say with high certitude that RR
collisions are rare and avoidable? EG, nobody tries to add an A RR to an
existing TXT RR, so this would work? Also, has the userbase rebelled on
this feature to any great extent?

---

From: [email-masked]
Date: Thu, 22 Nov 2001 16:36:09 +1100

> a specific <protocol> usage for the entry it's requesting."

	In your world only "example.com" would be delegated.  Every one
	that wanted to lookup foo.ExAmPlE.com TXT would have to *manually*
	normalise ExAmPlE.com -> example.com, then lookup foo.example.com.
	Either that or you have to delegate/DNAME ExAmPlE.com in addition
	to delegating example.com.

> isn't always viable either.

	It's *never* viable.

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 22 Nov 2001 16:40:04 +0100 (CET)

>       MUST be normalized through nameprep prior to processing

Looking for clue. Why aren't IDNs normalized through nameprep just like IHNs?
What are the benefits of this?

  Erik

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 22 Nov 2001 09:52:17 -0600

Erik Nordmark wrote:
> Looking for clue. Why aren't IDNs normalized through nameprep just
> like IHNs? What are the benefits of this?

There are no benefits per se, but penalties which are avoided. Nameprep
forbids characters which are commonly used in labels that are not valid in
hostnames (including "_" for SRV, and chars valid for LHS of email
addresses in SOA/RP), so running all IDNs through nameprep would prohibit
IDNs which are not IHIs.

http://search.ietf.org/internet-drafts/draft-ietf-idn-nameprep-06.txt

| 5. Prohibited Output

| 5.1 Currently-prohibited ASCII characters
| 
| Some of the ASCII characters that are currently prohibited in host
| names by [STD13] are also used in protocol elements such as URIs [URI].
| The other characters in the range U+0000 to U+007F that are not
| currently allowed are also prohibited in host name parts to reserve
| them for future use in protocol elements.
| 
| 0000-002C; [ASCII CONTROL CHARACTERS and SPACE through ,]
| 002E-002F; [ASCII . through /]
| 003A-0040; [ASCII : through @]
| 005B-0060; [ASCII [ through `]
| 007B-007F; [ASCII { through DEL]

I'm putting together a summary of the issues and will post it soon.

---

From: David Hopwood <[email-masked]>
Date: Thu, 22 Nov 2001 19:29:18 +0000

-----BEGIN PGP SIGNED MESSAGE-----

> addresses in SOA/RP), so running all IDNs through nameprep would prohibit

So remove the prohibitions from nameprep and put them somewhere else.
This is much simpler than having two distinct notions of equivalence for
IDNs and IHIs.

Note that just prohibiting a set of characters is in any case not sufficient
to express the following:

 - restriction of ASCII host labels to the RFC 1123 syntax,
 - prohibiting top-level labels that are all ASCII digits,
 - ensuring that the BiDi algorithm is one-to-one on valid host strings
   (the problem is with labels that arbitrarily mix RTL characters and
   numbers - although it is possible to allow numbers at the end of an
   RTL label in logical order),
 - prohibiting mixed TC/SC strings, if we want to do that [*],
 - prohibiting illegal sequences of Hangul Jamo (i.e. sequences that do
   not follow the syntax in the Unicode Standard),
 - prohibiting irregular sequences of Hangul Jamo (i.e. that could be
   equivalently expressed using a cluster Jamo, or that do not use the
   standard form for fillers),
 - prohibiting labels that start with a combining character.

[*] This may not be appropriate as a restriction on the hostname syntax,
    because mixed TC/SC is used in Hong Kong, for example. It can still
    be applied by some registrars as a matter of policy, though.

IMHO all of these restrictions, except possibly mixed TC/SC, should be
applied to registered IHIs. (There's no need for clients to enforce
them, since it is not a security problem if they aren't enforced by
clients, AFAICS.)

The draft I'm preparing will include a formal syntax for IHIs along
these lines, that also prohibits symbols and punctuation characters
not used as diacritics (Unicode has a convenient 'Diacritic' property
that covers the latter). I agree with the people who have said that it
is easier to start with a restrictive syntax and then relax it if
problems are found, than to do the opposite.

[Incidentally, I made a mistake in a previous post that gave an ABNF
definition of the RFC 1123 hostname syntax. It should be:

  RFC1123HostName = *(HostLabel ".") TopLevelHostLabel
  HostLabel = (LETTER / DIGIT) [0*61(LETTER / DIGIT / "-") (LETTER / DIGIT)]
  TopLevelHostLabel = HostLabel \ *DIGIT

i.e. 0*61 instead of 61*.]

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 22 Nov 2001 14:34:17 -0600

The fundamental conflict here is that domain names are different from host
names, with different rules having been applied.

STD13 host names are case-neutral US-ASCII alpha, numbers and hyphen,
where hyphen is prohibited from appearing on either end of a label. STD13
domain names MAY contain case-neutral US-ASCII alpha, MAY contain
eight-bit values with no character assignments (and thus no case), and MAY
contain any seven-bit US-ASCII character codes outside of LDH in any
position in the label.

There are very, very few *formal* uses for STD13 domain names, but where
they are used they are very important. One example is SRV entries which
use "_" in the label. Another example is email addresses, where the
left-hand side ("local-part") of the address is stored in a label which is
encapsulated as RR data in SOA and RP RRs.

LHS of email addresses is a great example of the domain name rules. Under
RFC [2]822, the local-part of an email address can contain:

| atext           =       ALPHA / DIGIT / ; Any character except controls,
|                         "!" / "#" /     ;  SP, and specials.
|                         "$" / "%" /     ;  Used for atoms
|                         "&" / "'" /
|                         "*" / "+" /
|                         "-" / "/" /
|                         "=" / "?" /
|                         "^" / "_" /
|                         "`" / "{" /
|                         "|" / "}" /
|                         "~"
| 
| atom            =       [CFWS] 1*atext [CFWS]
| 
| dot-atom        =       [CFWS] dot-atom-text [CFWS]
| 
| dot-atom-text   =       1*atext *("." 1*atext)

The local-part of the email address is also case-sensitive.

Cumulatively, the local-part of an email address can be something as
bizarre as "-Eric A. Hall-", and this would be stored in an STD13 domain
name as "-Eric A. Hall-.ehsco.com.", which would be deconstructed by
compliant systems as "-Eric A. [email-masked]" (with the case preserved).

Obviously, such usage conflicts with the STD13 hostname rules, and would
also conflict with an IHN ruleset that maintained compatible with the
STD13 hostname rules.

Another condition which has to be watched for is that STD13 allows
eight-bit values to be specified as exact octet sequences. This isn't
specifically a problem with the examples given to date, and there are no
protocol mechanisms which use the eight-bit values (to my knowledge), but
it something to keep in mind.

Nameprep provides several functions which conflict with the usage of STD13
domain names (and thus also conflict with IDNs). The two of these which do
the most harm are US-ASCII character prohibitions (only allowing LDH), and
mandatory case-conversion.

There are several options here.

1) Expand the IHN rules to accomodate a larger ("reasonable") set of
   characters, such as "_" for SRV and some common mail characters.

   This breaks backwards compatibility, and is not really much of an
   option. For example, ACE would not encode the label if it it only
   contained characters from the ASCII range 0x00-7F. A label such
   as e_x_a_m_p_l_e.com. would be legal under the liberal IHI rules,
   would be passed through unencoded to STD13 systems, and then it
   would break at those systems.

   Furthermore, mandatory case-conversion in the IHI nameprep would
   still break email addresses.

2) Define a stringprep profile for IDNs explicitly.

   One of the hard part of this approach is deciding on case-
   conversion. Life is easier if everything is lowercased but as
   shown, this breaks local-parts of email addresses.
   
   Also, there is a potential future problem with normalization of
   characters, in that some future version of RFC 2822 may allow for
   non-normalized characters in local-parts. Requiring mandatory
   normalization would prohibit this usage, or at the least, would
   make such usage more complex.

   Finally, there is a problem with the eight-bit codes which are
   currently unspecified but legal in DNS. These codes map to the
   Latin-1 subset, so if they are always mandatorily interpreted and
   managed as such and are always normalized and case converted,
   then the wrong data will be returned.

3) Redefine the allowable data in DNS.

   If we globally say "no more domain names that are not hostnames"
   we break SRV and all other entries which can use STD13 domain
   names legitimately.

   This approach also introduces a requirement for authoritative
   servers, including replication masters and DDNS listeners. It
   also affects legacy systems that follow the old rules in good
   faith (such as DDNS clients), and where the sudden failure
   cannot be adequately represented by an RCODE. In short, such a
   mandate will require at least some updates to legacy systems.

4) Clarify the protocol usage rules. EG, clarify SRV to be
   [service].[transport].<IHI>. This is similar to #3 above but
   rather than defining global rules, clarify the handling rules
   for each RR type independently.

   This is feasible for the situations where this is a known problem
   with protocol operations. The problem is that there are a bunch
   of them. SOA is used umpteen different ways so it has to be
   clarified upteen different times (or at least considered upmteen
   different ways before a single clarification is useful).

   This is the most work but it generates the smallest wake,
   assuming that the handling rules can be clarified so as they do
   not break legacy systems. EG, we specifically clarify that the
   email addresses in SOA/RP are <IDN>.<IHI>, which doesn't break
   anything but it is considerable effort to go through each and
   every usage scenario.

I think #4 is the option we should pursue, or that DNSEXT should pursue,
as part of an effort to internationalize the DNS in general. DNS is the
same as every other service in that it has to be updated to fully utilize
internationalized domain names and host names, so we should not be
surprised that some work will be required for it to function properly.

---

From: David Hopwood <[email-masked]>
Date: Thu, 22 Nov 2001 23:31:24 +0000

-----BEGIN PGP SIGNED MESSAGE-----

"Eric A. Hall" wrote:
>    option.

Agreed.

>    shown, this breaks local-parts of email addresses.

It breaks e-mail addresses only if a local part encoded as a label
in an RR is required to be normalised. Why would it need to be normalised?

The same name equivalence relation should be used for all names in
queries, and names in RRs should preserve case (just as it is now for
ASCII).

>    Latin-1 subset,

No, they don't. IDNS interprets them as Latin-1, but I don't understand
why it does that; it's completely unnecessary, and incompatible with
the specification in section 3.1 RFC 1035 that octets >= 0x80 are
caseless ("Non-alphabetic codes must match exactly.", where "alphabetic"
means ASCII a-z and A-Z).

>    so if they are always mandatorily interpreted and

Not an option.

>    [service].[transport].<IHI>.

There's no need to do that if a single equivalence relation is used.

- --

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 22 Nov 2001 19:36:56 -0600

> in an RR is required to be normalised. Why would it need to be

I guess I mis-phrased the option. The value of a profile would be to
normalize IDNs closely similar to nameprep (with different exception
lists) so that Mark's particular concerns would be addressed.

I think we are roughly in agreement on the point, judging from your
response to my poorly worded response. The local part is a special label
that gets special treatment. We should not apply global rules to an FQDN
unless it is an IHN; for IDNs each label has to be treated specifically.

> queries, and names in RRs should preserve case (just as it is now for

Can you expand on that?

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 26 Nov 2001 09:54:40 +0100 (CET)

> addresses in SOA/RP), so running all IDNs through nameprep would prohibit

OK, so clearly all of nameprep doesn't make sense.

But the unicode normalization and presumably other parts of nameprep
would make sense for IDNs as well as IHNs. Perhaps it is only section 5.1
and 5.10 in nameprep that need to be different for IDNs.

> I'm putting together a summary of the issues and will post it soon.

Good.

  Erik