Subject: [idn] WG last call documents

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 09 Feb 2002 14:04:58 -0600

This is commentary on

  draft-hoffman-stringprep-00.txt
  draft-ietf-idn-nameprep-07.txt
  draft-ietf-idn-idna-06.txt

draft-hoffman-stringprep-00.txt:

 | 4. Normalization

 | [[[[[ Patrik asks: Is not addition of unassigned codepoints regarded
 | an update of UAX15? This is relevant here. ]]]]]

embedded comments need to be addressed or removed

Overall, I think that stringprep needs to define a data structure for
profiles and also needs to discuss version negotiation, both of which are
needed to facilitate programmatic upgrades. This means defining a
standardized character notation scheme to be used by all profiles, a file
structure (version tags, comment notation, line syntax, EOL markers, etc),
and so forth.

This also means describing suggested common mechanisms for retrieving
profiles. One obvious method would include "will be made available via FTP
by IANA", for example. Encouraging vendors to "provide updates in routine
patch kits" is another. Protocols can extend on that as needed as well,
(it is feasible that DNS could even use zone files and the zone transfer
services to retrieve version 1.1 of the subordinate stringpreps). The
point here would be to encourage rapid adoption of updated profiles,
through as many mechanisms as possible. No method will work 100% of the
time, so find five mechanisms that each work 60% of the time.

The hole I am concerned with here is with future mappings, where 'A' is
currently assigned but it is not currently mapped to 'a', and where a
future version creates such a mapping. In that case, the client will
generate 'A' as unmapped, the server will respond with an error or with
'a', either of which would trigger a failure on the client. Consider
ISO/IEC 8859-1 and 8859-15, where the former had lowercase y with
diaeresis and where the latter defines an uppercase y with diaeresis. It
seems likely that this kind of disjointed mapping will happen again, so we
need to be thinking about ways to minimize the disjointedness.

Secondarily, the IETF has had the lesson of versioning beaten into it over
the last few years, and it seems odd not to address this issue head-on in
the controlling spec.

draft-ietf-idn-nameprep-07.txt:

 | Stringprep Profile for Internationalized Host Names

Since this profile is specifically for host names, all of the rules that
apply to hostnames should be placed here. Consider that there are apps and
protocols which may use these rules to build internationalized host names
but will not use IDNA (either because they do not need IDNA conversion for
their data-path or they use something other than IDNA to encode the
names). Those services will need access to the complete set of rules.
Putting some of the rules here and some of the rules in IDNA essentially
means that every application that wants to build an i18n hostname has to
use both specifications, even though the "profile" is the only document
that a reasonable person should be expected to know about.

As it stands, this profile is incomplete without IDNA, but it does not
reference IDNA as a mandatory component (nor should it).

Furthermore, any subsequent work in this field will almost certainly be
required to obsolete this profile in order to clarify these issues, which
reinforces the need for versionin in stringprep.

draft-ietf-idn-idna-06.txt:

 | Abstract

 | This document defines internationalized domain names (IDNs)

That definition should be in a separate document, specifically either in
nameprep or in a stringprep profile that defines i18n "domain names"
rather than hostnames.

IDNA should "define a mechansim for encoding internationalized domain name
labels into STD13 labels".

 | 2 Terminology

 | The "ACE prefix" is defined in this document to be a string of ASCII
 | characters that appears at the beginning of every ACE label. It is
 | specified in section 5.

I thought the idea behind adopting "punycode" was to avoid the use of the
generic "ACE" term?

 | 3. Requirements

 | IDNA conformance means adherence of the following three rules:

 | 1) Whenever a domain name is put into a generic domain name slot,
 | every label MUST contain only ASCII characters. 

"generic domain name slot" should be substituted with "STD13 slot". There
is no reason whatsoever that application protocols cannot exchange IDNs or
IHNs in whatever form they wish.

 | 2) ACE labels SHOULD be hidden from users whenever possible. 

I have argued against this before and I will do so again now.

Should WHOIS servers perform this conversion on output (email addresses
and nameservers) since it is likely to be displayed to a user? Should an
email client perform conversion on a message? Every protocol and
data-format has unique considerations. Making blanket statements like this
is harmful.

CONVERSION MUST ONLY OCCUR WHERE A PROTOCOL OR DATA-FORMAT EXPLICITLY
DEFINES THE BEHAVIOR. MANDATORY CONVERSION WILL BREAK THINGS IF THERE IS
NOT AN AGREED-UPON METHOD FOR CONVERSION SPECIFIC TO THE APPLICATION.

| 4.1 ToASCII
| 4.2 ToUnicode

Add a disclaimer to the effect of ~"these routines MUST NOT be used except
where the data is known to contain an i18n domain name label". Trying to
pipe all Unicode data through ToASCII will be harmful, to say the least.
Trying to pipe an STD13 binary domain name through ToUnicode may result in
some false-positives.

 | B. Design philosophy

What value does this section provide? This text is religion not science,
seeing as how it omits key counter-arguments, namely that there are
hundreds if not thousands of applications for every DNS server, so
upgrading servers in addition to applications represents an incremental
cost (rather than the "The cost of implementing IDN may thus be much
lower, and the speed of implementation could be much higher" erroneous
opinion). Other aspects (such as ~"no change to protocols") are just as
wrong (protocols MUST BE CHANGED FOR CONVERSION TO BE PROPERLY MANAGED),
if not argumentative. This section is non-contributory to say the least,
and possibly even a hindrance to future developments, and has no place in
a technical specification such as this.

General comments:

In general, I think that there is a tremendous amount of architectural
work that needs to be done here. We need to strongly and clearly define
the different domain name data-types that are in use, make up stringprep
profiles for them, impose versioning controls on stringprep profiles, and
tighten up the IDNA spec. I think at this point that IDNA is pretty close
to where it needs to be, but it is still far too grandiose. It should only
define an encoding for specific data-types, and it should not encourage
any interpretative behavior on the applications.

One specific item, there does not seem to be a prohibition against certain
characters in the first position of a domain name label. There is a minor
prohibition in IDNA, but there needs to be an exception list in nameprep
that covers the characters we discussed in the WG earlier, which are
punctuation, diacritical marks, or dashes. We discussed this at length on
the mailing list.

"Adam M. Costello" wrote in <[email-masked]>:

> There are some code points that are prohibited in host names, but not
> in all textual domain names.  The underscore is the best example.
> These prohibitions belong in ToASCII.

I disagree with this for a couple of reasons. First of all, these
exceptions are associated with protocol identifier labels, which are not
hostnames. Nameprep defines i18n hostnames, so these characters belong in
the nameprep prohibition. Secondarily, "protocol identifiers" are
generally considered as protocol operators which do not need to be
internationalized (therefore never needing anything but ASCII). This is
why I defined them as a separate data-type (and why I think they need
their own stringprep profile). See BCP18:

 | 2.  Where to do internationalization
 | 
 | Internationalization is for humans. This means that protocols are not
 | subject to internationalization; text strings are. Where protocol
 | elements look like text tokens, such as in many IETF application
 | layer protocols, protocols MUST specify which parts are protocol and
 | which are text. [WR 2.2.1.1]

SRV owner labels are protocol elements, not text. They should never go
through any conversion. A dedicated profile for this would help.

I agree with many of the other comments which have been made. I think that
a parent i18n "domain name" stringprep profile can describe certain
generic characteristics (such as mapping of full-stop characters) that the
label-centric profiles cannot.

DJ Bernstein is right-on when he says that certs don't solve the spoofing
problem. There is nothing preventing the wiley hacker from getting a cert
for [email-masked], and there would be nothing about that cert which
would alert anybody that spoofing had occurred. In general terms, this
means that I agree with DJB that the initial scope should be tightly
controlled and slowly incremented. I think that this opinion was also
expressed by J Klensin's solicitation for a "safe-set" and D Crocker's
endorsement of a safe-set. This doesn't mean we all agree with DJB, but I
think that in this case, he is probably correct. A stronger versioning
system in Stringprep would make this a feasible approach, I think.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Dan Oscarsson <[email-masked]>
Date: Sun, 10 Feb 2002 17:52:33 +0100

>From: "Eric A. Hall" <[email-masked]>

>apply to hostnames should be placed here. 

From what I could read the title is wrong.
It should be: Stringprep Profile for Domain Names.

To make this work, we need ONE stringprep plus ACE.
One that can be used on ALL labels in DNS.
To have different profiles for different labels would be
a mess. Well, you could mark each profile+ACE with a
different ACE prefix to indicate profile used but having
lots of different encodings/profiles have so far proven to be
a big difficulty for programmers to handle.

A part from the stringprep plus ACE we need rules on what
characters should (or must) be used on different things.
For example a host name should be printable letters
with intermixed printable separators (and no repeated ones).
An other rule is applied on e-mail names.
The above rules on what character are allowed does
only belong in applications, not at DNS protocol level.

>IHNs in whatever form they wish.

I agree. Also the term "generic" means anything and that would mean
non-ASCII characters.

>data-format has unique considerations. Making blanket statements like this

I agree in that the WHOIS server should output names using the character
context of it's protocol. We need a WHOIS protocol that uses UTF-8 and
when it does, ACE labels may not be transmitted over that protocol.

But a WHOIS client should always decode ACE data and display them using
the
local character set.

There should also be a requirement that a protocol using UTF-8 should
never send data using ACE so the receiver does not have to handle
the extra work of checking for embedded ACE.

>opinion).

I agree. Remove this section. It contains too much religion.

>hostnames. Secondarily, "protocol identifiers" are
>internationalized (therefore never needing anything but ASCII). 

Depend on who is expected to view protocol identifiers and what
programs using the protocol expects.
Protocol identifiers can have non-ASCII characters.

>through any conversion.

Why not. There can be non-ASCII SRV owner labels. As prople are afraid
the non-ASCII host names (which is also a type of protocol element)
can break applications they should be afraid that non-ASCII SRV owner
labels could break applications.

It is easiest if at the DNS protocol level ONE type of ACE (and
stringprep)
is used on all labels. The the lowest level can all use the same
code. Higher levels in applications can put additional constraints
on the content of labels. By having the lowest level as broad as
possible and as few as possible constraints we have a good building
ground for the future. Otherwise we might just like to day with
this ugly "use only ASCII constraint" have to invent a new layer
on top of the old one to bypass the constraints.

>think that in this case, he is probably correct.

Most of this burden is on the domain owners. Registration of bad type
of domains below .com should not be allowed. This includes names
like micr0soft.com and micorsoft.com which clearly never should
have been allowed.

-
One other thing. As the IDNA supporters think this idea will work fine
and the users will soon not have to see or write ACE names, there
should be a date on which this should be examined. If at that date
not most important software and protocols have been updated, IDNA
should be removed and replaced with something making people update
their software. UTF-8 would probably do that.
For example, at that date I expect to be able to write URLs in HTML
documents using the documents native character set.
The date should be fairely soon as the supporters of IDNA have said
this is quick. 2003-12-31 is acceptable.

   Dan

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 10 Feb 2002 12:03:53 -0600

> One that can be used on ALL labels in DNS.

We have been through this subject many times. STD13 binary domain names
and i18n host names are fundamentally incompatible. One specification will
not suffice.

> a big difficulty for programmers to handle.

I disagree. IDNA only needs to perform encoding and decoding, it doesn't
need to validate the domain names. Validation has ALWAYS been performed by
the participating applications, and has NEVER been performed by the
resolver. There is no reason why a codec should perform this validation.
The profiles are/should-be for the benefit of the applications, the
protocol designers and the data-format designers. The codec should just
convert them.

> I agree in that the WHOIS server should output names using the
> character context of it's protocol.

Are you sure the server should do this and not the client? Well, what if
the "client" is TELNET, should TELNET always do it?

The point here is that EVERY network service has its own considerations.
Demanding that all services implement a user-level feature is absurd. It's
as ridiculous as saying that all text MUST be white-on-black from this
moment forward.

The outcome of this mandate is that every existing specification is made
obsolete and non-compliant. BY A DRAFT! It's arrogant and ignorant of the
ramifications that every network service has different issues that have to
be worked through. Completely absurd.

> Protocol identifiers can have non-ASCII characters.

/etc/services cannot be expected to hold UTF-8 on all Internet systems

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 11 Feb 2002 04:16:01 +0000

[A response to a remark by Dan Oscarsson appears at the end of this
message.]

"Eric A. Hall" <[email-masked]> wrote:

> Since this profile is specifically for host names, all of the rules
> that apply to hostnames should be placed here.

I think the intended title of this document is "Stringprep Profile
for Internationalized Domain Names".  The previous IDNA draft was
"Internationalizing Host Names in Applications", but the latest IDNA
draft is "Internationalizing Domain Names in Applications".  I think
when that change was made, the same change should have been to the title
of the nameprep draft, but was not due to a simple oversight.  Can the
nameprep authors confirm that?

In my view, (and in the view of the IDNA draft) nameprep is *not*
specifically for host names, but is for all internationalized textual
domain names.  That is how IDNA uses it.

> Consider that there are apps and protocols which may use [nameprep]
> to build internationalized host names but will not use IDNA (either
> because they do not need IDNA conversion for their data-path or they
> use something other than IDNA to encode the names).

Those apps still need the IDNA spec, because the IDNA spec defines
what is a valid internationalized domain label.  Nameprep alone cannot
possibly tell you whether the label is too long; you can know that only
after applying Punycode.

> That definition should be in a separate document

Here is the definition in question:

    An "internationalized domain name" (IDN) is a domain name for which
    the ToASCII operation (see section 4) can be applied to each label
    without failing.

Clearly the IDNA spec is the logical place for that definition.  Maybe
you'd like to change the definition, but that's a separate issue.

> I thought the idea behind adopting "punycode" was to avoid the use of
> the generic "ACE" term?

Nope.  Currently, the relationship between the terms "ACE" and
"Punycode" is as follows:  An ACE label is an ASCII label that is
equivalent to a non-ASCII label.  ToASCII is the operation that converts
labels to equivalent ASCII labels.  Punycode is one of several steps of
ToASCII (other steps include nameprep and checking for various other
restrictions, like length).

The term "ACE" has been used to refer to several subtley different
concepts over the past few years, but no formal definition was ever
given until the recent IDNA drafts.

> "generic domain name slot" should be substituted with "STD13 slot".
> There is no reason whatsoever that application protocols cannot
> exchange IDNs or IHNs in whatever form they wish.

The second sentence is true, but I don't see how the first sentence
follows from it.

IDNA defines "generic domain name slot" as follows:

    An "internationalized domain name slot" is defined in this document
    to be a domain name slot explicitly designated for carrying an
    internationalized domain name as defined in this document.  The
    designation may be static (for example, in the specification of
    the protocol or interface) or dynamic (for example, as a result of
    negotiation in an interactive session).

    A "generic domain name slot" is defined in this document to be any
    domain name slot that is not an internationalized domain name slot.
    Obviously, this includes any domain name slot whose specification
    predates IDNA.

Regardless of what we call it, that is the concept that is needed in
rule 1.  Perhaps you think the term "generic domain name slot" is
counter-intuitive and should be renamed (but not redefined)?

I don't think the term "STD13 slot" is appropriate.  I would expect a
term called "STD13 slot" to be defined in terms of STD13 only, but the
concept we need here is defined in terms of the IDNA spec.

> Should WHOIS servers perform this conversion on output (email
> addresses and nameservers) since it is likely to be displayed to a
> user?

That rule also says "When requirements 1 and 2 both apply, requirement
1 takes precedence."  If the output of whois is considered to be
machine-readable, then the email address domains and nameservers occupy
generic domain name slots and rule 1 requires the ASCII forms to be
used.  If the output is considered to be plain text for humans, and if
the charset supports non-ASCII, then ToUnicode should be applied to the
domain labels when the output is composed.

> Should an email client perform conversion on a message?

On the body, no.  But it should perform conversion on the domain names
in the header.

Hmmm, maybe we should insert "obtained from domain name slots" after
"ACE labels" in rule 2.

Notice that it's a SHOULD and not a MUST.  That is deliberate, because I
don't think any simple rule can describe exactly the circumstances where
you want to perform ToUnicode while excluding all the circumstances
where you wouldn't want to.  Neglecting to apply ToUnicode will at worst
cause users to see garbage, but otherwise won't break anything, so we
leave room for the application programmer to decide that rule 2 is
better left unheeded in special situations.

> DEFINES THE BEHAVIOR.

That would defeat the whole point of IDNA, which is to allow
applications to use internationalized domain names with legacy protocols
and interfaces, without having to revise those protocols and interfaces.

> | 4.1 ToASCII
> | 4.2 ToUnicode
> 
> Add a disclaimer to the effect of ~"these routines MUST NOT be used
> except where the data is known to contain an i18n domain name label".

They are designed to work for all domain labels, and the rules in
section 3 require them to be applied to all domain labels.

The IDNA draft nowhere suggests using them for anything but domain
labels.  But forbidding their use on other things would be out of scope.
IDNA can assert restrictions related to domain names, but the one you
suggest would be a requirement that applies to everything *except*
domain names.

> Trying to pipe an STD13 binary domain name through ToUnicode may
> result in some false-positives.

If a domain label gets altered by ToUnicode, then it *is* an ACE label,
whether by designed or by accident.  We've known from the start that
the possibility of accidental ACEs is unavoidable in the IDNA approach,
which takes an existing corner of the namespace and repurposes it.  (But
accidents will be quite rare:  The label must contain only ASCII code
points, and begin with the ACE prefix, and the remainder must be a valid
Punycode encoding, and the decoding of that must remain unchanged by
nameprep).

You don't need to invoke "binary" domain names to illustrate this
problem, since it is equally present for ASCII domain names.  In either
case, an accidental ACE label might get displayed (or even transported)
in an untintended way, but no information will be lost, and the original
form will be restored before any program attempts to interpret the
name as intended.  (Why?  Because a program that knows the intended
interpretation will be receiving it through a slot that isn't an
internationalized domain name slot, and hence the name must have been
converted back (by ToASCII) before being put into that slot.)

>  | B. Design philosophy
> 
> What value does this section provide?

It documents the rationale behind the design decisions.  I think it's
often helpful and/or instructive for people who later read the spec and
wonder "Why did they do it that way?"

I think its generally good to have a rationale appendix in any technical
spec.  (Punycode lacks one only because I never got around to writing
one.)

> One specific item, there does not seem to be a prohibition against
> certain characters in the first position of a domain name label.

Has the current prohibition of leading & trailing hyphens been terribly
helpful?  I think it probably would have been better if the host
label syntax were simpler--just a set of allowed characters, with no
restrictions on position.  So I think we should keep any new positional
restrictions to a minimum.

But there is one class of characters that might indeed be dreadful at
the beginning: combining characters.  I recently refered to labels
that begin with combining characters as invalid Unicode strings, but
they're not, are they?  They just behave in surprising ways when abutted
with something else.  Maybe nameprep should prohibit initial combining
characters.

> > There are some code points that are prohibited in host names,
> > but not in all textual domain names.  The underscore is the best
> > example.  These prohibitions belong in ToASCII.
>
> exceptions are associated with protocol identifier labels, which are
> not hostnames.

Right, they're not host names.  That's the whole point.  There exist
domain names that are not host names.

> Nameprep defines i18n hostnames, so these characters belong in the
> nameprep prohibition.

See above.  I think nameprep applies to domain names, not just host
names.

> SRV owner labels are protocol elements, not text.  They should never
> go through any conversion.  A dedicated profile for this would help.

Consider the following UTF-8 zone file:

$ORIGIN THAI1.net.
_ldap._tcp CNAME _ldap._tcp.THAI2.net

It would be ludicrous to expect the preprocessor that converts this to
a STD-13 zone file to do a lookup on _ldap._tcp.THAI2.net (which is
not defined in this zone file) and discover that it has a SRV record
and hence avoid applying conversions to _ldap._tcp.  It should be able
to simply apply ToASCII to all domain labels.  As IDNA is currently
specified, it can.

> The point here is that EVERY network service has its own
> considerations.  Demanding that all services implement a user-level
> feature is absurd.

We recommend, we don't demand.  And we don't try to apply the user-level
feature in absurd places, we say "before a domain name is displayed
to a user or is output into a context likely to be viewed by users".
Maybe you think that description is too broad.  Suggest some alternative
wording.  But we need to give some sort of encouragement for ToUnicode
to be applied in some situations.

> The outcome of this mandate is that every existing specification is
> made obsolete and non-compliant.

I don't know what it means for a specification to comply or not comply
with IDNA.  Applications can comply with IDNA, or not.

Every existing application is non-compliant with IDNA.  Obviously we
cannot expect any existing application to comply with a new spec written
after the application.

Existing applications will never apply ToASCII or ToUnicode, because
they don't know about them.  IDNA-compliant applications will apply
ToASCII and ToUnicode in various situations according to rules 1-3.

I don't see how existing specifications are made obsolete.

Dan Oscarsson <[email-masked]> wrote:

> never send data using ACE so the receiver does not have to handle the
> extra work of checking for embedded ACE.

That would be out of scope.  IDNA specifies a method for applications
to use IDNs with non-IDN-aware protocols and interfaces.  IDNA has no
business telling new IDN-aware protocols what representations they
should allow or forbid.

Besides, I think it's a good idea to do the check anyway, whenever you
display a domain name, regardless of where it came from.  ACEs might
have been leaked into the system by a gateway or cut/paste operation.

AMC

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Mon, 11 Feb 2002 08:00:37 -0800

>nameprep authors confirm that?

Yep, that is correct. I worked so hard on getting it right in IDNA 
that I forgot to be careful in nameprep. This will be fixed.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 11 Feb 2002 15:25:34 -0600

[this is part 1 of a multi-part response]

> We recommend, we don't demand.

Just to be clear, is your argument that the use of SHOULD does not impose
mandatory behavior? See 2119:

 | may exist valid reasons in particular circumstances to ignore a
 | particular item, but the full implications must be understood and
 | carefully weighed before choosing a different course.
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

SHOULD defines *default* behavior, it is mandatory UNLESS there is a
reason NOT to do it.

The word you are looking for is MAY, but I would argue with that as a
default position as well.

> And we don't try to apply the user-level feature in absurd places,
> we say "before a domain name is displayed to a user or is output into
> a context likely to be viewed by users".

The problem I have here is that you apply to ALL places, some of which are
absurd, some of which are not. You need to choose your defaults with
greater position. The default you have chosen results in tremendous
absurdities. Consider the following DEFAULT mandatory interpretation:

  From: <support>
  To: <customer>

  Please add zz--gobbledygook to your sendmail.cf file

which is converted BY DEFAULT into

  Please add <IDN> to your sendmail.cf file

Or, consider a web-based tutorial explaining how IDNA works:

  Ignore all of the munchkins behind the curtain, the basic principle
  is as follows:

    a domain name is encoded per punycode
    a prefix of zz-- is appended to the encoded output
    thus the domain name of <IDN> is encoded as zz--gobbledygook

which is converted BY DEFAULT into

    thus the domain name of <IDN> is encoded as <IDN>

Congratulations on producing a specification that cannot be used to
document itself. Everytime an example is given, the example is
transliterated. :/

> Maybe you think that description is too broad.  Suggest
> some alternative wording.  But we need to give some sort of
> encouragement for ToUnicode to be applied in some situations.

  1) Connection identifiers (as used for forward lookups by
     application clients, or as generated by reverse lookups)
     MAY be transliterated for display purposes, although
     careful consideration is to be given to any effects such a
     conversion may have on any secondary applications.  For
     example, if a utility or library is commonly used by
     multiple applications as part of a network service, a
     separate function call or command-line switch may be needed
     to generate internationalized domain names as output.

  2) Domain names which are provided as protocol data MUST NOT
     be transliterated, except where the governing
     specification  explicitly permits and describes such usage.

  3) Domain names which are provided as structured data (such
     as email addresses, URLs, and other common data-formats)
     MUST NOT be transliterated, except where the governing
     specification explicitly permits and describes such usage.

  4) Domain names which are provided as unstructured data in
     application output MUST NOT be transliterated, except where
     the governing  specification explicitly permits and
     describes such usage.

I think we all agree on 2 and 3. I don't think you have given 1 much
consideration, but think about things like tcpdump or netstat as examples
of where this needs to be treated gingerly. You seem to be suggesting that
an application should be able to intuit where 4 should or should not be
applied. As the two examples above illustrate, rule #4 is also needed if
there is to be any consistency in implementations.

> applications to use internationalized domain names with legacy
> protocols and interfaces, without having to revise those protocols
> and interfaces.

I'm sorry that you are operating under that belief. As I have pointed out
multiple times on the mailing list, IDNA serves a single objective of a
backwards-compatible encoding format. It does not bring i18n domain names
to legacy applications, protocols or data-formats by itself.

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 11 Feb 2002 16:14:17 -0600

[this is part 2 of a multi-part response]

"Adam M. Costello" wrote:

> for Internationalized Domain Names".

Based on a cursory analysis, this document is suitable as a standalone
description of i18n domain names, although I would reiterate my belief
that there is room to provide dot-separator character mapping on full
domain names (possibly belonging in a separate super-profile).

However, we still need a separate profile that describes the hostname
rules, in particular including a prohibition against certain leading and
trailing characters, as was discussed at length on the mailing list on
multiple occassions.

> possibly tell you whether the label is too long; you can know that only

This co-dependency needs to be called out at the top of the i18n domain
name profile.

> concept we need here is defined in terms of the IDNA spec.

The use of "generic" is what bothers me, because it implies that the slot
(another word I dislike) can hold any domain name. I would call it STD13
because that is the governing specification (actually, "RFC1035 slot"
would be better since RFC1035 is likely to be supplanted from STD13 at
least a couple of times in our lifetimes).

> in an untintended way, but no information will be lost, and the
> original form will be restored before any program attempts to
> interpret the name as intended.

The situation I am thinking about is where one of the NUL replacements
takes place. I admit that I haven't fully explored the possibility of
weaknesses in this area, but deleting input is always risky so I am
keeping it in my mind until I do get around to it.

> often helpful and/or instructive for people who later read the spec
> and wonder "Why did they do it that way?"

So you have an example of another WG specification where only half the
opinions are provided, and that they provide value towards understanding
the protocol or service? Citations Please

The fact of the matter is that this section is one side of a holy war,
argumentative, does not provide any scientific value, and certainly does
not represent WG consensus. It is also factually in error.

> they're not, are they?  They just behave in surprising ways when
> abutted with something else.  Maybe nameprep should prohibit initial
> combining characters.

Note that these are rules for hostnames in particular.

A prohibition against combining characters is not feasible. See
<[email-masked]> from K Whistler.

This was resolved without disconsent in <[email-masked]>:

 | First and last characters in the label MUST NOT be a diacritical
 | mark or hyphen-minus.

[Several other issues with stringprep were not yet addressed, and I still
consider them as unresolved and/or open.]

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 12 Feb 2002 02:00:50 +0000

> [this is part 2 of a multi-part response]

Do I have the last part yet?  I don't know, I'll just respond now...

> SHOULD defines *default* behavior, it is mandatory UNLESS there is a

Yes, I understand.  We don't demand that implementors apply ToUnicode,
we recommend it.  Or seen another way, we demand that they either apply
ToUnicode or think of a good reason not to.

I can see your point that maybe even that is a little too strong.

> Consider the following DEFAULT mandatory interpretation:
> 
>   From: <support>
>   To: <customer>
> 

It was never our intention that applications should go digging into text
directed at humans (as opposed to protocol messages, which are directed
at the application itself) and convert things that they guess might be
domain names.

This is probably not clear from the current wording.

>      conversion may have on any secondary applications.
> 
> I think we all agree on 2 and 3.

Assuming "transliterated" means "converted from ACE to Unicode" (as
opposed to the other way around), then yes I agree on points 2 and 3,
and in fact the IDNA spec already implies points 2 and 3 in requirement
1 of section 3 (domain labels inserted into generic domain name slots
must contain only ASCII).

Point 4 could be taken care of by limiting requirement 2 to ACE labels
obtained from domain name slots (as opposed to ACE labels guessed at in
plain text).  Point 1 could be taken care of by explicitly allowing for
exceptions.

What do you think of this wording:

    2) ACE labels obtained from domain name slots SHOULD be hidden
    from users except when the use of the non-ASCII form would cause
    problems or when the ACE form is explicitly requested.  Given an
    internationalized domain name, an equivalent domain name containing
    no ACE labels can be obtained by applying the ToUnicode operation
    (see section 4) to each label.  When requirements 1 and 2 both
    apply, requirement 1 takes precedence.

I could live with that.  I don't know about Patrik and Paul.

> I'm sorry that you are operating under that belief.  As I have pointed
> out multiple times on the mailing list, IDNA serves a single objective
> of a backwards-compatible encoding format.  It does not bring i18n
> domain names to legacy applications, protocols or data-formats by
> itself.

It does not bring i18n to legacy applications, but it *does* bring i18n
to legacy protocols and data formats.  One of the main examples driving
the design of IDNA was this one:  We should be able to update mail
user agents (like Pine) and immediately be able to use IDNs in email
addresses without having to touch mail transfer agents (like sendmail)
or DNS servers (like bind), and without having to wait for any updates
to protocol specifications (like the DNS spec and the SMTP spec) or data
format specifications (like RFC 2822).

> > ...the IDNA spec defines what is a valid internationalized domain
> > label.  Nameprep alone cannot possibly tell you whether the label is
> > too long; you can know that only after applying Punycode.
>
> This co-dependency needs to be called out at the top of the i18n
> domain name profile.

The last sentence of the nameprep abstract is:

    This profile of the stringprep protocol is used as part of a suite
    of on-the-wire protocols for internationalizing the DNS.

I agree that there should be a citation to IDNA at the end of the
abstract.  Nameprep is useless without IDNA.  I'd also like to see "the
DNS" changed to "domain names", since we're not actually doing anything
to the DNS.

> > Perhaps you think the term "generic domain name slot" is
> The use of "generic" is what bothers me, because it implies that the
> slot (another word I dislike) can hold any domain name.

Some domain name slots are explicitly internationalized and are thus
special; all other domain name slots, being non-special, are "generic".
That's the way I intended the word to be understood.  Other words that
could serve the same purpose are "vanilla" and "ordinary".  Would either
of those be better?  Or maybe "non-internationalized".

"RFC-1035 slot" is not inclusive enough.  If a protocol/interface
spec says that a field/argument is a domain name, and says nothing
else (makes no mention of RFC 1035 or any other standard for domain
names), then that is a generic domain name slot.  Or if it cites various
standards that govern the slot, but IDNA is not one of them, then it's a
generic domain name slot.

As for the word "slot", would you like to suggest some other word?  Here
is the concept that the term "domain name thingamabob" needs to stand
for: "a protocol element or a function argument or a return value (and
so on) explicitly designated for carrying a domain name."

> So you have an example of another WG specification where only half
> the opinions are provided, and that they provide value towards
> understanding the protocol or service? Citations Please

Not an IETF spec, but the PNG spec includes a rationale appendix, and
we've received comments saying that it was helpful and it would be nice
if other specs did the same.  I don't think we ever received comments
saying it should not have been included.

> It is also factually in error.

If that's true, it should be fixed.

> <[email-masked]> from K Whistler.

He said that we could not simply prohibit combining characters, and
I agree.  I was suggesting that we might want to prohibit *initial*
combining characters.  Combining characters combine with the character
preceeding them, so a label beginning with a combining character is a
strange beast, and might cause bizarre and suprising behavior.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 11 Feb 2002 22:00:46 -0600

>   apply, requirement 1 takes precedence.

Was there a problem with the explicit text I provided? The text presented
here still encourages a default behavior of transliterating anything that
looks like a domain name, when it should be the opposite.

We seem to have a disconnect here. You're concerned with the symptoms, I'm
talking about the root cause. Every protocol and data-format has its own
specific considerations, and it is not possible to mandate (or strongly
suggest, if you prefer) that every Internet service change its behavior to
suit IDNA. It's absurd. The only thing you can do here is say "don't do
anything until the masters of the spec say how to do it".

Again, consider WHOIS output. Should a WHOIS server transliterate output?
Should it be performed at the client so that negotiation is not required?
And you have described the protocol negotiation or client behavior where?

Or consider HTML, which has a limited notion of PDU structures. Should
domain names enclosed in HREF tags be transliterated? or should IDNA never
be transliterated, and if a user wants an i18n domain name to be displayed
then they should use the charset capabilities for this? And you have
documented this behavior for the W3C developer community where?

Even if you want to limit this behavior to a very specific community,
something small and syntactically precise such as email header fields,
which header fields should do this? Should this be limited to the
originator and recipient header fields only? And you have documented this
extension to RFC 2822 where? I mean, even RFC 2047 is restricted to
specific data-types.

You only have one choice here: nobody transliterates nothing until the
governing specifications for the network service in question clearly
describe how it should be done.

The exception is local presentation of connection identifers. There is no
reason that ping shouldn't be able to handle IDN<->IDNA conversion,
although even that has to be handled with care. Look, I can't even tell
one of my newsreaders to use an i18n domain name since the .newsrc files
are shared by multiple clients.

> to protocol specifications (like the DNS spec and the SMTP spec) or
> data format specifications (like RFC 2822).

Not possible. See above.

> Other words that could serve the same purpose are "vanilla" and
> "ordinary".  Would either of those be better?  Or maybe
> "non-internationalized".

"LDH slot"?

> saying it should not have been included.

If it wasn't lop-sided opinion (aka religion), you probably wouldn't have
gotten the two complaints you got here either.

> If that's true, it should be fixed.

 | Many proposals for IDN protocols have required that DNS servers be
 | updated to handle internationalized domain names.

Do you honestly seriously believe that IDNA will not require upgraded
servers to handle transliteration on input? That it *CAN* work without an
upgrade does not mean that it *WILL* work without an upgrade. Which of
those words is a property of "required"? Both.

 | Because of this, a person who wanted to use an internationalized
 | domain name would have to be sure that their request went to a DNS
 | server that had been updated for IDN.

The two dual-mode proposals before this WG have both allowed legacy
servers to provide answers. False.

Etc. The logic-path is entirely corrupt and based on nothing more than an
opinion that an encoding which appears nowhere else in the known universe
is somehow better than embracing the pain of an incremental upgrade.

> strange beast, and might cause bizarre and suprising behavior.

When rendered, it will combine with the dot separator. However, that's
only half the complaint. Also consider that leading punctuation is often
used as a command-line qualifer (-verbose). The prohibition needs to be
against all punctuation. When I looked (very briefly, that was composed at
a hotel room), the characteristic that seemed most useful was a property
of diacritical mark or dashes. I don't think there is a property of
punctuation. I should look.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 12 Feb 2002 06:10:20 +0000

"Eric A. Hall" <[email-masked]> replied:

> Was there a problem with the explicit text I provided?

I prefer to be more concise.  I tried to incorporate all the points I
agreed with.  The above wording is significantly less strong that the
wording in the current draft.

> The text presented here still encourages a default behavior of
> transliterating anything that looks like a domain name,

Not things that look like domain names, things that are known to be
domain names.  That's what the "obtained from a domain name slot" is
for.  And I added "except when the use of the non-ASCII form would cause
problems".  But yes, I still want users to be shown the non-ASCII form
except when there's a reason for them to be shown the ACE form.

> when it should be the opposite.

I'm afraid I disagree.  People have said in the past that IDNA is
pointless if users always see the ACE form.  The world you seem to be
envisioning is one in which users would be seeing the ACE form an awful
lot.

> > We should be able to update mail user agents (like Pine) and
> > immediately be able to use IDNs in email addresses without having
> > to touch mail transfer agents (like sendmail) or DNS servers (like
> > bind), and without having to wait for any updates to protocol
> > specifications (like the DNS spec and the SMTP spec) or data format
> > specifications (like RFC 2822).
>
> Not possible.
>
> We seem to have a disconnect here.

Apparently.

> "LDH slot"?

Many generic domain name slots allow all ASCII characters, not just LDH
characters.  But "ASCII slot" is not appropriate either, because the
essence of the concept is not what the slot allows or what standard it
cites; the essence of the concept is what standard the slot does *not*
cite, namely the IDNA spec.  So the term needs to convey non-specialness
or non-i18n-ness.  If you don't like "generic" or "vanilla" or
"ordinary" or "non-internationalized" or anything along those lines,
then we're probably not going to find common ground.

> The prohibition needs to be against all punctuation.

For host names, right, not domain names in general?  I think I like this
model:  Just as ASCII host names exclude many of the characters allowed
in ASCII domain names, internationalized host names would exclude
many of the characters allowed in internationalized domain names (the
same sorts of characters: punctuation and symbols).  There would be
two stringprep profiles, nameprep with a small prohibition table, and
hostprep with a large prohibition table.

However, I think it's probably too late for this.  I very much doubt
that most people would be interested in making such drastic changes at
this late date.  Non-ASCII punctuation is not threatening enough to
warrant derailing the process.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 12 Feb 2002 00:39:55 -0600

"Adam M. Costello" wrote:

> pointless if users always see the ACE form.

I'm not saying that the ACE form is the only form that should EVER be
used, what I'm saying is that the default mandate is dysfunctional.

The only thing that IDNA provides is a backwards-compatible encoding. The
benefit will come when the protocols and data-formats are extended to
support it. I know this violates the religious doctrine section that
promises milk honey and sunshine for everybody immediately. Sorry.

But hey, prove me wrong. I've given you plenty of examples, so you should
be able to come up with one. Please list the RFC822 header fields that
IDNA should be transliterated versus the header fields that should not be
transliterated. Any such list will by nature not conform with the default
mandate. The mandate is dysfunctional. You cannot prove otherwise.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 12 Feb 2002 07:18:11 +0000

"Eric A. Hall" <[email-masked]> wrote:

> Please list the RFC822 header fields that IDNA should be
> transliterated versus the header fields that should not be
> transliterated.

When an application is displaying header fields to the user, it should
apply ToUnicode to domain labels found in domain name slots.  RFC 822
defines the syntax of every header field, and some of those fields have
<domain> in their definition.  Those <domain>s are the domain name
slots.  Anything else in the header is not a domain name slot.

AMC

---

From: "D. J. Bernstein" <[email-masked]>
Date: 12 Feb 2002 09:09:00 -0000

Adam M. Costello writes:
> When an application is displaying header fields to the user,

Define your terminology. How does a UNIX program figure out whether it
is displaying something ``to the user''? (Anyone who doesn't understand
this question should read ``The UNIX Philosophy,'' by Mike Gancarz.)

One of many examples: Surely you are familiar with the ``dig'' program.
The output of ``dig soa berkeley.edu'' includes all the SOA information
for berkeley.edu. This output might be

   * piped to the ``less'' program, a generic text viewer that (usually)
     displays the results to the user; or

   * piped to a script that extracts the hostmaster address and invokes
     the ``sendmail'' program to send email; or

   * saved to a file, later viewed with ``less''; or

   * saved to a file, later piped to that script.

Yes or no: Should ``dig'' convert its results from your 7-bit encoding
to the local character set? (Assume the LANG=en_US.UTF-8 locale, so that
the conversion is at least theoretically doable.)

An answer of ``no'' means that you're failing to display the desired
non-ASCII glyphs to the user. In other words, you're missing the whole
point of IDNs.

An answer of ``yes'' creates an interoperability failure, just like the
copy-and-paste interoperability failure that you've been ignoring for
months. The mail will bounce.

If you try to dodge this question by drawing some silly distinction
between ``dig'' and other programs, I'll start asking about those other
programs. The nmh mail-handling system leaps to mind.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 12 Feb 2002 13:00:42 +0100

>  | First and last characters in the label MUST NOT be a diacritical

I agree that having a combining character as the first character
in a hostname part (label) is a bad idea (nominally it combines with what
is before it and can in rare cases be "absorbed" into syntax before
the acutal string (like in >/, for xhtml, where / is combining long solidus;
after NFC (applied blindly to the entire text) there will be a precomposed
negated greater than character there...)

But I see absolutely no reason to prohibit combining characters
at the end of a label (unless applied to a hyphen-like character...).

		Kind regards
		/kent k

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 12 Feb 2002 10:51:11 -0600

> slots.  Anything else in the header is not a domain name slot.

Are you SURE you want to do this with Received? or rather, are you SURE
you want to mandate this behavior with Received on behalf of the other
communities, even though copy-and-paste with transliterated Received
headers will get ugly.

Does your list include all of those data-types which have domain names as
a subordinate element? I will assume that your list also includes "slots"
that hold email addresses since that is one of your other religious
doctrines, and since those hold domain names they are candidates.

Are URLs in header fields therefore also candidates? Are you SURE you want
to do this with List-* and Content-Location header fields?

Are message identifiers also candidates, since they are structured the
same as email addresses? Are you SURE you want to do this with Message-ID?

Sweeping rules that affect every instance of every domain name across
every network service in use on the Internet are absurd.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 12 Feb 2002 21:20:16 +0000

[This message replies to both D. J. Bernstein and Eric A. Hall regarding
similar issues.]

"D. J. Bernstein" <[email-masked]> wrote:

> is displaying something ``to the user''?

(To anyone who doesn't use UNIX:  We're talking about a program that
outputs text to standard output.  Standard output usually gets displayed
on a terminal emulator unless the user has redirected it to a file or
to the input of another program.  Some programs are intended to be
redirected and have human-unfriendly machine-friendly output.  Some are
intended to be viewed and have human-friendly machine-unfriendly output.
Some are somewhere in between, which motivates the question here.)

> Yes or no:  Should ``dig'' convert its results from your 7-bit
> encoding to the local character set? (Assume the LANG=en_US.UTF-8
> locale, so that the conversion is at least theoretically doable.)

Yes.  The second sentence of the man page for dig is "It performs DNS
lookups and displays the answers..."  Furthermore, the man page does
not specify the output format of dig, so the output does not contain
any domain name slots.  Clearly it's primary purpose is to display
information to humans, and any program that attempts to parse the output
of dig is assuming the risk that someday this undocumented output format
might change.

Any user clever enough to use dig and to use programs that
parse its output is probably also clever enough to prepend "env
LANG=en_US.US-ASCII" to the command line if necessary, or to use the
--no-idna option of dig that would almost surely have been added at the
same time IDNA support was added.

Another option for any application would be to have IDNA support
completely disabled by default, and enabled only at the user's request.

"Eric A. Hall" <[email-masked]> wrote:

> Are you SURE you want to do this with Received?

Yes.

> or rather, are you SURE you want to mandate this behavior
> with Received on behalf of the other communities, even though
> copy-and-paste with transliterated Received headers will get ugly.

Novice users never even see Received headers.  Users who know enough to
request to see Received headers will also know enough to request ASCII
if they need it.

As for "mandate", keep in mind this sentence from the IDNA introduction:

    This document does not require any applications to conform to IDNA,
    but applications can elect to use IDNA in order to support IDN while
    maintaining interoperability with existing infrastructure.

If IDNA is more trouble than it's worth for particular applications,
those applications can just ignore it.

> Does your list include all of those data-types which have domain names
> as a subordinate element?

Yes.

> I will assume that your list also includes "slots" that hold email
> addresses

The list you requested was a list of header fields.  The list includes
header fields that contain <mailbox>, because <mailbox> ultimately
contains a <domain>.  The <domain> is the domain name slot.

> Are URLs in header fields therefore also candidates?

If a header field is defined to contain a URL, and the URL scheme
indicates that it contains a domain name, then that is a domain name
slot.

> Are you SURE you want to do this with List-* and Content-Location
> header fields?

Content-Location is an HTTP header field, and HTTP responses are not
normally displayed, but if an application wanted to display an HTTP
response, then it should display the non-ASCII form of the domain name
if it can (unless the user requests otherwise or special circumstances
make this problematic).

I don't know what List-* fields are, but if they are defined to contain
domain names, then it's the same story as Content-Location.

> Are message identifiers also candidates, since they are structured
> the same as email addresses?  Are you SURE you want to do this with
> Message-ID?

Same story as Received.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 12 Feb 2002 16:19:23 -0600

> request to see Received headers will also know enough to request ASCII

Spam complaints are the common example of normal users seeing received
header fields, when they are copied and pasted into complaint messages,
the encoded representation (your default) will be what's sent.
Administrators blocking sender domains are the common example of advanced
users seeing domains. In many cases (such as ISP mail systems) the only
view to this information will be through a web UI.

Nowhere in your specification does it state that encoded and non-encoded
views of a domain name should be provided. Do you plan on adding this,
since you won't back away from the claim that it is safe to mandate the
automatic transliteration of ACE sequences?

> domain names, then it's the same story as Content-Location.

Content-Location is a MIME header field (see RFC 2557). The List-* header
fields defined in RFC 2369 provide URLs for common list-management
functions. The point of this question is to see if you would dive in and
embrace transliteration of URLs as well, which you seem to think is also a
good idea. Have you documented the use of i18n domain names in URLs
somewhere? I mean, as hard as some of your peers argued against this very
issue, I'm surprised that you're inviting it without any sort of
documentation for review.

> > the same as email addresses?  Are you SURE you want to do this with

Message-ID values are frequently referenced in multiple locations, not all
of which are in structured fields. By allowing them to be transliterated
for display, you are inviting them to be converted. This is, in turn,
implicitly expanding the structured header fields that will allow them.

For example, I have cited previous messages by Message-ID throughout this
very thread, and your default interpretation would have made it so that
those values were only seen (and therefore copied and pasted) in their
encoded form. In conjunction with a SeeAlso header field or a RFC822-style
References header field, this would not only mean that I was now
generating illegal header fields but that I was poisoning the entire
thread and every copy of those messages (including archives), AND THIS
WOULD BE ENTIRELY DUE TO YOUR DEFAULT MANDATE.

In short, mandates such as this one encourage bleed-over inside of
structured data-formats, ACCELERATE ENTROPY ON THE INTERNET. That this is
all being done without any documentation on the affected elements, without
any viable justification whatsoever (supporting the promise of your
religious text is not viable justification), and without any authority
over the affected network services is the very paragon of opinioneering
and absurdity.

> those applications can just ignore it.

Then why are you defending it? Declare IDNA as a method for encoding i18n
domain names into an ASCII-compatible encoding, step away from the foolery
of mandating its use with everything that looks like a domain name, and
leave the integration work to the governing bodies that actually have the
authority, expertise and bandwidth to deal with the associated issues.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 12 Feb 2002 23:48:56 +0000

"Eric A. Hall" <[email-masked]> wrote:

> Nowhere in your specification does it state that encoded and
> non-encoded views of a domain name should be provided.

IDNA section 6.1 says:

    Because name labels encoded as ACE name labels can be rendered
    either as the encoded ASCII characters or the proper decoded
    characters, the application MAY have an option for the user to
    select the preferred method of display;

If there is a demand for this ability, applications will tend to provide
it.

> Have you documented the use of i18n domain names in URLs somewhere?

IDNA.  It documents how to use IDNs anywhere you can use regular domain
names.

> In conjunction with a SeeAlso header field or a RFC822-style
> generating illegal header fields

I don't know what a SeeAlso header field is.  A References header field
contains a list of <word>s and <msg-id>s, which can be distinguished
syntactically because the <msg-id>s begin and end with angle brackets
while the <word>s do not.  If the mail program is IDNA-compliant, then
when it constructs the References field (or when it checks the field
that the user has suggested) it obeys IDNA section 3 requirement 1 and
makes sure that the <domain>s inside the <msg-id>s are ASCII-only (it
can apply ToASCII if needed).

AMC

---

From: "D. J. Bernstein" <[email-masked]>
Date: 12 Feb 2002 23:43:59 -0000

> > locale, so that the conversion is at least theoretically doable.)

That will cause interoperability failures. It will break other programs,
even though those programs are fully compliant with today's standards.
Mail will bounce!

Costello persists in claiming that IDNA will allow IDNs to be deployed
right now _without_ any failures other than incorrect displays. That
claim is simply not true. Mail will bounce. Web links will fail.

If IDNA's special-purpose 7-bit encoding is supported by _no_ networking
programs, or (in a fully Unicode world) by _all_ networking programs,
then mail won't bounce. In the intermediate stage, when the encoding is
supported by _some_ networking programs, mail will bounce.

> LANG=en_US.US-ASCII" to the command line if necessary,

Costello is weaseling.

He is saying that mail won't bounce _if_ the program invoking dig (in
this case, a shell script) is changed _before_ the dig program is
changed. But the IDNA specification does not impose any such ordering.
Costello wants the dig authors to change their code right now.

More importantly, there _does not exist_ an ordering compatible with all
of today's cross-program data transfers. Addresses are copied from mail
programs to browsers; addresses are also copied from browsers to mail
programs.

> the man page does not specify the output format of dig

Actually, the dig manual page clearly states that dig prints answers
from name servers. There is only one printed DNS record format in BIND,
namely BIND's zone-file format, which is documented in detail and meant
to be machine-parsed.

The format is very badly engineered---it requires far too much effort to
parse completely---but that's a side issue. The point is that programs
_do_ read it.

In fact, the current dig man page explicitly mentions at one point that,
to simplify machine parsing, dig avoids one feature of the format by
default. Costello is simply wrong when he claims that the dig output is
meant only for display and that programs shouldn't be looking at it.

> Some programs are intended to be
> Some are somewhere in between, which motivates the question here.

The line that Costello is attempting to draw is directly contrary to the
UNIX philosophy. UNIX programs are _designed_ to be ``in between.''

This is explained in detail in Gancarz's book. Here, for example, is an
excerpt from the ``Make Every Program A Filter'' section:

   When you assume that the receptacle of a program's data flow might be
   another program instead of a human being, you eliminate those biases
   we all have in trying to make an application user friendly. You stop
   thinking in terms of menu choices and start looking at the possible
   places your data may eventually wind up. Try not to focus inward on
   what your program can do. Look instead at where your program may go.
   You'll then begin to see the much larger picture of which your
   program is a part.

These ideas are most fully developed in UNIX, but they can also be seen
in other operating systems, in tools ranging from copy-and-paste to
object-linking frameworks. Some of these tools have a side channel for
character-set information, but most of them are designed for a world
with a unified character encoding---ASCII yesterday, UTF-8 tomorrow.

The bottom line is that the ``dig'' problem is shared by _thousands_ of
UNIX programs. They deliberately provide output in a format that can be
viewed by the user, or sent to another program, or both.

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 12 Feb 2002 19:06:57 -0600

> IDNA.  It documents how to use IDNs anywhere you can use regular domain

With all of the authority and justification afforded to the authors by the
IETF and W3C, no doubt.

> while the <word>s do not.

Ok, let's go there (sticking with 822 for context):

 | msg-id      =  "<" addr-spec ">"
 | addr-spec   =  local-part "@" domain
 | domain      =  sub-domain *("." sub-domain)
 | sub-domain  =  domain-ref / domain-literal
 | domain-ref  =  atom
 | atom        =  1*<any CHAR except specials, SPACE and CTLs>
 | CHAR        =  <any ASCII character>        ; (  0-177,  0.-127.)

As we can see, a Message-ID header field that includes decoded output from
an ACE-encoded IDN is not a valid Message-ID.

Although you may believe that this data-type only has relevant definition
when it is encapsulated within an email message, I can assure you that the
data-type is used in several locations, and that the values will come and
go in and out of messages. Yes, a messaging application should check
before it accepts the value, but it should also check before it
TRANSLITERATES THE ENCODED VALUE. The DEFAULT behavior should be to NEVER
GENERATE THE DECODED OUTPUT.

IDNA provides a encoding of an i18n domain name which is safe for transfer
within the Message-ID header field. That is all. It is not safe for
anything to transliterate the encoded sequence. You must not encourage
such foolishness with a "requirement".

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 13 Feb 2002 02:32:27 +0000

[This message contains responses to D. J. Bernstein and Eric A. Hall
regarding related issues.]

"D. J. Bernstein" <[email-masked]> wrote:

> from name servers.  There is only one printed DNS record format in
> BIND, namely BIND's zone-file format, which is documented in detail
> and meant to be machine-parsed.

I don't see any mention of RFC 1035 or "master file" or "zone file" in
the dig man page, but let's go ahead and suppose that the output format
of dig is indeed the RFC 1035 master file format.

In that case, IDNA section 3 requirement 1 *prohibits* dig from
outputing the non-ASCII form of domain names.  So there's no problem,
mail does not bounce, the sky does not fall.

"Eric A. Hall" <[email-masked]> wrote:

> As we can see, a Message-ID header field that includes decoded output
> from an ACE-encoded IDN is not a valid Message-ID.

Exactly right.  Therefore RFC-822 compliant mail programs will not put
non-ASCII domain names in a Message-ID field.  An RFC-822 compliant
program faced with the task of putting a non-ASCII domain name into a
Message-ID field will call ToASCII if it's IDNA-aware, otherwise it will
have no option left but to return an error to whatever handed it the
non-ASCII domain name.  It was the calling program's error for passing
a non-ASCII domain name to a non-IDNA-aware program, or it was a human
error for piping data into a program that doesn't understand the data
format.

As in the dig example, any program with a well-defined pre-existing
output format that includes domain names is prohibited by IDNA from
outputting the non-ASCII form (unless it redefines the output format).
But if the program does not define its output format, then making
assumptions about the output format (like assuming that it won't change
in the next version) is an inherently risky business, and people who
write scripts that pipe such output into other programs are deliberately
choosing to take that risk.  (I'm not saying they shouldn't write
such scripts, just that they should accept responsibility for the
consequences of their unfounded assumptions.)

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 12 Feb 2002 21:54:41 -0600

> regarding related issues.]

Please stop doing that, we are on about two totally different things.

> non-ASCII domain names in a Message-ID field.

You have completely missed the point. Message-ID is unique in that it
provides a serial number for a message. The decoded output for a
Message-ID header field MUST NEVER be presented, since it would be a
functionally different identifier whenever it was used. For example,
searching Google or an IMAP server for all of the messages associated with
the Message-ID of nn@zz--gobbledygook is completely different than
searching for a Message-ID of nn@<IDN>. Shall all such services be
upgraded to suit IDNA, even though many of these operations take place
outside of RFC [2]822 data-space?

The problem of pollution will also happen despite any mandate to the
contrary. This will most noticably occur with email addresses, where an
address is copied from say Pine to Hotmail; note that 2047 pollution also
occurs in this way, but since 2047 doesn't pollute the actual address, it
isn't actually fatal. You can say that applications shouldn't do that and
I'd agree with you. The secondary point is that we shouldn't be knocking
these applications over in the first place.

Regardless, I'm tired of chasing down your escape clauses. The idea that
you can somehow dictate a "requirement" (your word) that applications
SHOULD (your word) rewrite domain names regardless of functionality or
form is absurd. You have only ever had one option here, which is to defer
any and all such processing to those groups which actually define and
control those specifications. As much as you would like to believe
otherwise, you have zero authority to impose such behavior on anybody or
anything. Encouraging it is completely irresponsible. Until you remove
that requirement, this spec is broken. Period.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 13 Feb 2002 03:55:31 -0000

> outputing the non-ASCII form of domain names. So there's no problem,

The problem, of course, is that your so-called ``internationalized
domain names'' are being presented to the user as gobbledygook.

Let's try another example: the ``show'' program in the UNIX MH/NMH
mail-handling system. Yes or no: Should this program convert domain
names from your special-purpose 7-bit character set to the local
character set? Again assume LANG=en_US.UTF-8, so this can be done.

Your newly invented ``well-defined pre-existing output format''
criterion says that ``show'' is not allowed to convert. But this means
that MH/NMH users, such as Keith Moore, will see gobbledygook instead of
IDNs.

How is Keith Moore supposed to see IDNs displayed properly in his email?
Let's assume that he has the current version of X, with Unicode fonts
and a UTF-8 xterm. Where exactly are your PunyCode IDNs converted to
UTF-8? Which programs do the conversion?

You keep dodging these questions because you know that any conversion
will trigger interoperability failures. The output that you think is
going to a user may actually be going to another program that does a
domain-name lookup without knowing anything about IDNA. Kaboom.

(To forestall any ``check for a tty'' weaseling: Programs such as Emacs
and Expect will give you a tty anyway; and tools such as copy-and-paste
are on the other side of xterm's tty.)

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 13 Feb 2002 05:06:51 +0000

"Eric A. Hall" <[email-masked]> wrote:

> The decoded output for a Message-ID header field MUST NEVER be
> presented, since it would be a functionally different identifier
> whenever it was used.  For example, searching Google or an IMAP
> server for all of the messages associated with the Message-ID of
> nn@zz--gobbledygook is completely different than searching for a
> Message-ID of nn@<IDN>.

I might also search for a From: field, or a To: field.  In fact, I can
search Google for pretty much anything.  So therefore all ACE labels
MUST NEVER be decoded.  Wonderful.

> you have zero authority to impose such behavior on anybody or
> anything.

Right, as IDNA section 1 says, "This document does not require any
applications to conform to IDNA..."

I think this thread has probably outlasted its usefulness, but you are
welcome to have the last word.

AMC

---

From: Martin Duerst <[email-masked]>
Date: Wed, 13 Feb 2002 14:53:42 +0900

>IDNA.  It documents how to use IDNs anywhere you can use regular domain

Upgrading URIs consistent with URI guidelines and internationalization
is described in:

http://www.ietf.org/internet-drafts/draft-ietf-idn-uri-01.txt

Regards,     Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 13 Feb 2002 14:48:37 +0900

>With all of the authority and justification afforded to the authors by the

Disclaimer, just to make sure:

The IDN WGs' work, including IDNA, is done in the IETF. W3C does not
'afford' any authority or justification to this work.

Regards,    Martin.

#-#-#  Martin J. Du"rst, I18N Activity Lead, World Wide Web Consortium
#-#-#  mailto:[email-masked]   http://www.w3.org/People/D%C3%BCrst

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 13 Feb 2002 01:20:14 -0600

"Adam M. Costello" wrote:

> therefore all ACE labels MUST NEVER be decoded.  Wonderful.

At last you see the light. IDNA has one single function, which is that it
provides a backwards compatible encoding. That's it.

Supporting internationalized domain names requires that the applications,
protocols and formats, like, ADD SUPPORT FOR internationalized domain
names. A real shocker, I know!

If you want UTF-8 output treated as normal, guess what you need to use for
the normalized input form. C'mon, it should be easy.

---

From: "Marco d'Itri" <[email-masked]>
Date: Wed, 13 Feb 2002 15:41:12 +0100

>character set? Again assume LANG=en_US.UTF-8, so this can be done.
This assumption is wrong. Most people whose primary language is not
english do not use Unicode and UTF-8 as their native character set but
some national charset like ISO-8859-1 or KOI8-R.
You may not like it, but the native character set for most people is not
UTF-8 and they will not switch to UTF-8 just to use IDN because they
would not be able to see accented letters in their plain text documents,
in email or usenet articles generated by broken microsoft software and
everywhere else a not identified charset is used.

I just looked at /usr/share/doc/locales/SUPPORTED.gz in my Debian
GNU/Linux workstation and the locales for which UTF-8 is not a second
choice are:

ar_IN
en_IN
fa_IR.UTF-8
hi_IN.UTF-8
mr_IN.UTF-8
ta_IN
te_IN
ur_PK 

md@wonderland:pts/0:~$zcat /usr/share/doc/locales/SUPPORTED.gz | sed -e 's/[\. @].*//' | sort | uniq | wc -l
    132
md@wonderland:pts/0:~$

Some locales may be missing, but 8 of 132 locales are not enough to
justify your assertion that everybody should just use UTF-8.

-- 
ciao,
Marco

---

From: "D. J. Bernstein" <[email-masked]>
Date: 13 Feb 2002 17:06:04 -0000

> > Again assume LANG=en_US.UTF-8, so this can be done.
> This assumption is wrong.

Please look up the word ``assumption'' at m-w.com. I did not say that
everyone uses LANG=en_US.UTF-8. I explicitly restricted attention to
that locale, because it's a locale where the programs under discussion
_can_ display IDNs properly.

The point of this thread is that the IDNA deployment plan will cause
interoperability failures: in particular, mail will bounce. Until the
IDNA proponents can explain how to fix these problems in _one_ modern
Unicode-aware locale, there's no point in considering other locales.

> some national charset like ISO-8859-1 or KOI8-R

Converting UTF-8 to KOI8-R is unreliable, obviously. The reason we want
the world to settle on Unicode is to eliminate such problems. Has it
never occurred to you that some Russian users want to see non-KOI8-R
characters? Or that some Europeans want to see non-8859-1 characters?

Maybe you're happy never seeing anything outside 8859-1. But people
writing software still have to take time to deal with the character-set
mess---time taken away from giving you features that you want. Try not
to be so shortsighted.

---

From: "Marco d'Itri" <[email-masked]>
Date: Thu, 14 Feb 2002 15:00:18 +0100

>_can_ display IDNs properly.
And my point is that assuming wide deployment of UTF-8 (like you do in
http://cr.yp.to/proto/idn.html) is pointless, because such a thing does
not exist and will not exist in the near future, at least in most of
Europe and in the UNIX community of Japan (I do not know about windows
users).

 >Unicode-aware locale, there's no point in considering other locales.
This has already been discussed to death on this mailing list and I feel
I do not have anything new to add.

 >the world to settle on Unicode is to eliminate such problems. Has it
Maybe you want this, but most of the world does not feel the need to use
unicode instead of their usual national charsets and does not want to
experience the interoperability problems of such a change.

 >characters? Or that some Europeans want to see non-8859-1 characters?
Yes, I've met on the network a few people with this need, but they are
not at all representative of the users of their country.

-- 
ciao,
Marco

---

From: "D. J. Bernstein" <[email-masked]>
Date: 14 Feb 2002 18:54:15 -0000

I'm asking how IDNA is supposed to work on a modern UNIX system with
full support for Unicode and UTF-8.

Programs that convert PunyCode to UTF-8 are creating interoperability
problems. Programs that display PunyCode as is are presenting
gobbledygook to the users.

IDNA appears to be based on the delusion that there are always separate
channels for programs to talk to users and to other programs. I've given
several counterexamples, and explained why many systems are deliberately
engineered to have a unified communications channel.

Anyway, Marco is saying that many UNIX systems don't even have full
Unicode support. This is true. I don't know why he thinks it's relevant.

---

From: "J. William Semich" <[email-masked]>
Date: Thu, 14 Feb 2002 15:06:09 -0500

To 99% of Internet users, it's not.

Bill Semich

---

From: "Marco d'Itri" <[email-masked]>
Date: Thu, 14 Feb 2002 22:26:55 +0100

>Unicode support. This is true. I don't know why he thinks it's relevant.
You are misrepresenting my opinion. I'm saying that many people do not
use unicode as their native character set and have no plan to do so.
This means that "just using UTF-8" will not magically work as you
suggest.

-- 
ciao,
Marco

---

From: "D. J. Bernstein" <[email-masked]>
Date: 14 Feb 2002 21:53:59 -0000

Marco d'Itri writes:
> "just using UTF-8" will not magically work

Nobody has said it will magically work. Supporting Unicode and UTF-8 has
taken quite a bit of effort. Some programs still need to be fixed.

But none of this is relevant to the current discussion. It is simply not
rational to respond to ``IDNA is extremely harmful and should not be
deployed'' by making claims about the merits of another proposal.

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Feb 2002 13:21:27 -0600

"Eric A. Hall" wrote:

> description of i18n domain names

Looking at this in more detail, I take that back. The i18n "domain name"
specification needs to allow any valid UCS character code, and must not be
restricted to hostname characters. As has been discussed multiple times,
there are some domain names which are not hostnames, and this model will
likely be valid in the future as well.

The obvious example of this is mailbox names. Some future version of an
email specification may allow (for example) the use of non-combining
characters in mailbox names. However, mailbox names are often stored in
DNS labels, and under the current arrangement, those mailbox names would
be normalized and lowercased as part of this process. In essence, the
rules which are currently provided impose restrictions on any future
mailbox name designs which the representative working group may come up
with in the future.

That is the obvious example. There are less-obvious examples of the same
concern. However, in general, there should be a different stringprep
profile for i18n "domain names" than for "host names", and IDNA (among
other encoding or tagging sequences) must allow for all of these
variations, rather than imposing artificial restrictions in this area.

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 15 Feb 2002 22:15:12 +0000

"Eric A. Hall" <[email-masked]> wrote:

> The i18n "domain name" specification needs to allow any valid UCS
> character code,

You might be able to argue that it should allow all valid code points,
but I don't see why it's *necessary*.  Since IDNA will be the first
specification to say anything about Unicode characters in domain names,
later specifications will know exactly which code points they must live
without.  It's not as if IDNA's prohibitions will cause incompatibilites
with existing IDN-related standards.  New IDN-related standards will
cause incompatibilities if they attempt to use code points prohibited by
IDNA.

> and must not be restricted to hostname characters.

Domain names in IDNA are not restricted to host name characters.  Only
host names are.  See step 3 of ToASCII.

(If you're concerned about the ASCII control characters and space, which
indeed are prohibited in the current Nameprep draft, then relax, we've
seen the flaw, and those prohibitions are being removed.)

> However, mailbox names are often stored in DNS labels, and under the
> current arrangement, those mailbox names would be normalized and
> lowercased as part of this process.

This problem already exists to some extent with ASCII mailbox names.
Sticking an ASCII mailbox name (which is allowed by RFC 822 to be
case-sensitive) in a domain label (which is defined by RFC 1035 to be
case-insensitive) is already a dubious proposition.

> In essence, the rules which are currently provided impose restrictions
> on any future mailbox name designs which the representative working
> group may come up with in the future.

The email working group has at least two options:  They can either
take IDNA (which was designed for domain labels) and apply it as-is to
mailbox names (even though it wasn't designed for that) and live with
the implications, or they can define their own way of representing
internationalized mailbox names as ASCII strings (which might bear some
resemblance to IDNA and might reuse components).  Either way, they will
interoperate with IDNA just as well (or as badly) as ASCII mailbox names
interoperate with ASCII domain names today.

> there should be a different stringprep profile for i18n "domain names"
> than for "host names"

There effectively is.  Nameprep is the profile for domain names, and
ToASCII adds a few more restrictions for host names.

The case-folding of all domain names (not just host names) is consistent
with the existing DNS, which uses case-insensitive matching for all
domain names (not just host names).

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Feb 2002 17:54:06 -0600

"Adam M. Costello" wrote:
> later specifications will know exactly which code points they must
> live without.

Why must anything live without some reasonable characters just because
IDNA didn't see the reason for it at this particular moment? There is no
reason to impose any such restriction, so I fail to see why it should be
done in the first place. Perhaps you should argue why this restriction
needs to be in place?

> internationalized mailbox names as ASCII strings (which might bear
> some resemblance to IDNA and might reuse components).  Either way,
> they will interoperate with IDNA just as well (or as badly) as ASCII
> mailbox names interoperate with ASCII domain names today.

Why can't they be accomodated now? Will this flexibility break something
that I don't know about? I mean, why come up with YET ANOTHER ACE ENCODING
just for email addresses or some other data-type which happens to get
stored in DNS sooner or later, when we can design for flexibility right
now? Why?

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 16 Feb 2002 00:48:26 +0000

"Eric A. Hall" <[email-masked]> wrote:

> IDNA didn't see the reason for it at this particular moment?

Exactly which "reasonable" characters are you talking about?  The
prohibited list in nameprep is pretty small, and they're all characters
that you'd have to be somewhat crazy to want to use in domain names.

> I mean, why come up with YET ANOTHER ACE ENCODING just for email
> addresses or some other data-type which happens to get stored in DNS
> sooner or later, when we can design for flexibility right now?  Why?

I asked long ago, shouldn't we try to design a solution that handles
both domain labels and email address local-parts?  I was told no, that's
out of scope, a different working group will take care of email address
local parts.  I share your concern, but I got the impression that this
was not open for debate.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Feb 2002 19:49:42 -0600

"Adam M. Costello" wrote:

> Exactly which "reasonable" characters are you talking about? 

All UCS character codes are valid for something, and anything can be
stored in DNS as an owner domain name (think TXT) or as data (SOA, RP, or
as-yet-invented). The problem with the current spec is that case-folding
and normalization will impose restrictions on these assignments. This is
inappropriate for a generic i18n domain name label syntax.

On a secondary note, normalization and lowercasing are problems having to
do with hostnames, and those should be enforced by delegation and
application rules. Note well: the existing DNS does not enforce hostname
rules anywhere else, we should be mimic'ing this behavior. The hostname
rules should only define, the codecs should only encode/decode, and the
apps/protos/formats should enforce.

> The prohibited list in nameprep is pretty small, and they're all
> characters that you'd have to be somewhat crazy to want to use in
> domain names.

I think you meant hostname here. Remember that hostname rules apply to
delegations. Domain name rules are a broader category which also include
things like SRV, email addresses, and so forth.

> both domain labels and email address local-parts?  I was told no,
> that's out of scope, a different working group will take care of
> email address local parts.  I share your concern, but I got the
> impression that this was not open for debate.

The email people will come up with their own character restrictions, which
is exactly the point. The namespace should ALLOW them to come up with
mapping rules, case-folding, normalization and whatever other rules they
wish. I mean, if they want to have a perverse mailbox sequence, why
shouldn't they? The codecs should only deal with inputs and outputs. I'm
pretty certain that's what you were told.

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Feb 2002 19:57:23 -0600

See attached.

This WG should define the global rules as in section 3[.0] (plus a
dot-mapping syntax), the generic domain name rules as in section 3.1 as a
minimum definition for other data-types to be built on, and the hostname
rules as in section 3.2. Everything else can go into an update of STD13.

Subject: [idn] naming syntax rules
From: "Eric A. Hall" <[email-masked]>
To: IDN <[email-masked]>
Date: Wed, 05 Dec 2001 02:08:48 -0600

James has been needling me to put together a second summary of the naming
rules in time for the IETF meeting. However, I have been extremely busy
lately (it is 3 am in a strange hotel room) but I wanted to at least
scratch together enough material for the concepts to be tangible.

The following text is by no means complete. It's hardly just begun.
However, it illustrates what the scope will be, exposes some of the open
issues, and may be usable as a touchstone to see if this whole IDN thing
is going to work or not.

What I mean by that is, if systems are going to work with IDNs in their
raw form (not encoded form, but raw form) these are the rules they will
have to work with. If these rules are too complex, the whole approach has
to be reconsidered. That will affect a lot of other things, including ACE.

The basic idea here is to declare formal data-types for labels, and to
incorporate the data-types into syntaxes for applications and protocols to
use when they need to interact with domain names.

1.	Summary

This memo describes two sets of definitions which are necessary for the
consistent and reliable use of internationalized domain names across the
Internet. First and foremost, this memo specifies the rules which govern
the structure and syntax of internationalized domain names in various
scenarios, and also describes their legitimate characters and any
normalization which may be required. Secondarily, this memo also clarifies
and extends usage rules of common resource records so that
internationalized domain names can be stored and exchanged (either as
resource record owner domain names or as resource record data) in a form
which is consistent across all usage environments.

2.	Introduction

There are many issues which affect the characters that are desirable for
use in DNS domain names. Among these considerations are obvious aspects
such as breadth, as well as less-obvious aspects such as normalized forms
of particular character sequences, comparison efficiencies, and more.

The general consensus of the IDN working group is that domain names should
use a mildly-restricted subset of the character codes and arrangement
sequences which are documented in the UCS for use with languages, as this
subset excludes non-verbal symbols and spurious punctuation which are
likely to be problematic, while still allowing international domain names
to be created. Furthermore, the consensus is that these character
sequences should be normalized and converted to lowercase [in that order?]
wherever this is possible, since this will provide the tightest
syntactical representation of the supported characters with the least
amount of ambiguity.

While both of those objectives are highly desirable (and are met in most
of the scenarios), there are many instances where these objectives are
incompatible with existing practice. For example, existing
(STD13-compliant) DNS implementations are allowed to use domain names
which contain any eight-bit character code (0x00 through 0xFF), while
there are some protocol models which specifically require the use of
punctuation (SRV requires underscore, for example), while some resource
records can contain domain names that combine both of these elements (SOA
and RP both provide email addresses as domain name labels that can
contain, and those can use punctuation or case-specific US-ASCII letters).

In order to facilitate these divergent requirements, this memo describes
multiple types of domain name labels, including their valid characters,
any case-conversions and/or normalizations which may be required, and so
forth.

Furthermore, in order to ensure that these rules are consistently
implemented (and to minimize damage when they are not), this memo also
states which label data-types are valid for use with many of the common
resource records.

Cumulatively, this means that a system which attempts to use an
internationalized domain name for a specific purpose will have to be aware
of the rules which govern the resource record which provides that service,
and will have to be aware of the rules which govern the domain name
data-types which are valid for that resource record. For example, if an
application knows that an internationalized domain name will be used for a
forward lookup, it will have to be aware of the label data-types that are
usable with A (or AAAA) resource records, and must ensure that the domain
name is processed (normalized and lower-cased, in this example) before it
is used.

NOTE: Legacy systems which use a backwards-compatible encoding scheme for
access to resources with internationalized domain names will not be
required to perform any of these tests. However, systems which embrace
internationalized domain names as specific data (EG, any system which
encodes or decodes an internationalized domain name as explicit data) will
need to be aware of these issues and will likely be required to enforce
their usage.

3.	Domain Names and Label Data-Types

An internationalized domain name is a sequence of labels which are
encapsulated in a message. The message may provide the labels as separate
units of data (as is the case with DNS), or may provide them as a series
of dot-separated textual strings (as is the case when domain names are
"written-out" in protocol or application data streams).

In global terms, an internationalized domain name has the following
characteristics:

*	Series of labels (1*label)

*	Maximum cumulative length of 255 UCS character codes (not necessarily
codes with matching characters, and most definitely not octets or any
encoded representation). This limit includes any separators which may be
provided (such as the full-stop character commonly used as a separator
when the domain name is written), and also includes one character for the
root domain (the trailing dot).

The labels that make up a domain name will vary according to the
contextual use of the domain name.

3.1.	Opaque Labels

Some functions can use domain names which consist of unstructured or
unknown labels. For example, a TXT resource record can describe anything,
and as such, it can use any sequence of UCS characters for its owner
domain name.

Opaque labels require no processing on the part of the application which
is using the domain name. It is the responsibility of the user to provide
the domain name to the application in its correct case and/or
normalization form.

Opaque labels have the following characteristics:

*	Any valid UCS character code (not necessarily a valid UCS character).

*	Minimum length of one UCS character code.

*	Maximum length of 63 UCS character codes.

NOTE: Even though a domain name may sometimes consist of a variable number
of opaque labels, most domain names will also contain at least some host
labels. In those cases, the entire domain name should be provided as a
series of opaque labels, and the host labels should be determined
beforehand. For example, a CNAME resource record can reference anything,
including an A RR that consists entirely of host labels, or a TXT RR that
consists of a mixture of opaque and host labels. As such, it will depend
on the formats in use by the alias target, and will inherit those
attributes.

3.2.	Host Identifier Labels

Most functions will use domain names to identify a host, either directly
or indirectly. For example, a host may be identified by a relative domain
name which consists of only a local label, or by an FQDN which contains a
series of host labels. Since all forms must be supportable, all namespace
delegation functions also use the host label syntax.

The UCS characters provided in host labels are required to be converted to
lowercase and normalized according to the rules in [nameprep] before they
are processed. Servers are likely to treat such labels as exact matches of
the encoded data, so it is imperative that applications perform this work
before they encode the label into a DNS query.

Host labels are used for any lookups, protocol actions, or message formats
which specifically make use of internationalized domain names for host
identification purposes.

Host labels have the following characteristics:

*	UCS characters from the following ranges:

"letters" [need a property]

characters with number property [?]

characters with diacritical mark property [?]

hyphen-minus (U+002D)

*	MUST be converted to lowercase according to [nameprep].

*	MUST be normalized according to [nameprep].

*	First and last characters in the label MUST NOT be a diacritical mark or
hyphen-minus.

*	Minimum length of two characters.

*	Maximum length of 63 characters.

3.3.	ASCII Labels

Some functions require labels that contain extended punctuation, but which
also require case-neutral comparisons. The most readily apparent of these
usages is the SRV resource record, which makes use of the underscore
character (U+005F) and case-neutral US-ASCII in the owner labels.

ASCII labels have the following characteristics:

*	Any printable character from US-ASCII (0x21 through 0x7E, inclusive).

*	SHOULD be converted to lowercase as specified in [nameprep] (note that
servers are required to perform case-neutral comparisons, but certain
tools will likely prefer to generate and use lower-case wherever possible,
so lowercase is the preferred form). All comparison operations on these
domain names MUST be performed in a case-neutral form.

*	Minimum length of one character.

*	Maximum length of 63 characters.

NOTE: some resource records may define tighter restrictions.

NOTE: Even though a domain name may sometimes consist of a variable number
of ASCII labels, most domain names will also contain at least some host
labels. In those cases, the entire domain name should be provided as a
series of opaque labels, and the ASCII and host labels should be
determined beforehand.

3.4.	Mailbox Labels

Some functions provide SMTP mailboxes as labels within domain names. For
example, the SOA and RP resource records both provide email addresses,
with the first label providing a mailbox (local-part) of the address, and
with the remainder of the labels providing the delivery domain of the
address.

In order for these resources to be accessible, applications must process
labels which are known to contain email addresses through these rules.
This means that data must be provided in a non-normalized, non-lowercased
form, and must be restricted to the range of characters which are valid,
as specified in section XX of RFC 2822. Until RFC 2822 is deprecated or
until such a time as UCS characters can be stored in the mailbox portion
of Internet standard email addresses, the mailbox label is to processed
according to the rules set forth in RFC 2822.

There are two additional rules which govern this data-type:

*	Minimum length of one character.

*	Maximum length of 63 characters.

NOTE: mailbox labels can contain a large number of special characters such
as spaces or full-stop. These characters may require escaping as described
in section XX of this document.

NOTE: Mailbox labels are NOT a subset of the ASCII labels. Mailbox labels
are case-sensitive, while ASCII labels are case-neutral.

4.	Resource Records

The following structure is used to describe resource records and their
usage of internationalized domain names and labels.

<owner domain name labels> <mnemonic> <[data] [data] [...]>

A, always provides a host identifier

<1*host> <A> <[IPv4 address]>

AAAA, always provides a host identifier

<1*host> <AAAA> <[IPv6 address]>

CNAME, can reference anything, can target anything

<1*opaque> <CNAME> <[1*opaque]>

NS, references a host, provides a host identifier

<1*host> <NS> <[1*host]>

SOA, references a host (delegation), provides host identifier, email
address, and custom data

<1*host> <SOA> <[1*host] [1mailbox (*host)] [serial] [refresh] [retry]
[expire] [ttl]>

WKS, always provides a host identifier

<1*host> <WKS> <[XX] [XX]>

PTR, can reference anything, must inherit target attributes

<1*opaque> <PTR> <[1*opaque]>

HINFO, references a host, provides RR-specific data

<1*host> <HINFO> <[hardware] [opsys]>

MX, references a host, provides a preference and a host identifer

<1*host> <MX> <[preference] [1*host]>

TXT, can reference anything, provides free-text data

<1*host> <TXT> <[text]>

RP, can reference anything, provides email address and a pointer to a TXT
RR

<1*opaque> <RP> <[1mailbox (*host)]> <1*opaque>

SRV, references a protocol (which is specified using the ASCII data-type),
provides preference values and a host identifier

<1*ASCII> <SRV> <[priority] [weight] [port] [1*host]>

[NOTE: cannot define <2ASCII *HOST> because not all SRV protocol labels
are just _service._transport]

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Feb 2002 20:09:04 -0600

> *       Minimum length of two characters.

ps--that requirement was dropped in subsequent discussion, since minimum
length is a function of the delegation authority's policy (.com has
minimum length of 2 chars, .info has minimum length of 3, internal
delegations have minimum length of 1, etc).

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 16 Feb 2002 03:50:25 +0000

"Eric A. Hall" <[email-masked]> wrote:

> The problem with the current spec is that case-folding and
> normalization will impose restrictions on these assignments.  This
> is inappropriate for a generic i18n domain name label syntax.  On a
> secondary note, normalization and lowercasing are problems having to
> do with hostnames

The existing DNS requires *all* domain labels to be case-insensitive,
not just host labels.  See RFC 1035.  So domain names are already
inappropriate for any data types that are case-sensitive.

> things like SRV, email addresses, and so forth.

I understand that not all domain names are host names.  I meant domain
names.  Have you actually looked at the prohibited list in Nameprep?  I
think it will be extremely rare that someone wishes they could use one
of those characters in a domain name (of any type).

> The email people will come up with their own character restrictions,
> which is exactly the point.  The namespace should ALLOW them to come
> up with mapping rules, case-folding, normalization and whatever other
> rules they wish.  I mean, if they want to have a perverse mailbox
> sequence, why shouldn't they?  The codecs should only deal with inputs
> and outputs.  I'm pretty certain that's what you were told.

I was basically told not to discuss email local-parts on this mailing
list.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Feb 2002 22:33:31 -0600

> inappropriate for any data types that are case-sensitive.

The rules that govern STD13 labels only apply to STD13 labels. This is
somewhat irrelevant to the subject at hand.

The rules for i18n domain names can say that a domain name contains
explicit character codes. If the app/user is in the ToASCII step then the
label is not an STD13 label. Different rules can apply for those labels.
Note that I am not saying that the rules should be enforced by ToASCII
(they should be enforced by the applications that generate the domain
names), but that if the label needs to go through ToASCII then it is an
i18n domain name, and not an STD13 domain name. Conversely, if the label
does not need to go through ToASCII or if it falls out of ToASCII
unmolested, then it is an STD13 domain name.

> > I think you meant hostname here.  Remember that hostname rules apply
> > to delegations.  Domain name rules are a broader category which also
> > include things like SRV, email addresses, and so forth.
> 

It's the case-folding and normalization that I'm worried about. We should
not be enforcing this kind of reduction on every label that can be stored
in the namespace. And if we can make that distinction, why worry about
prohibitions? Even the issue with leading combining characters is
something that can be foisted onto the hostname subset rules, and simply
noted in the domain name rules as an issue that additional subset
specifications should be aware of.

If there are some codes that flat-out break IDNA which can be REASONABLY
withheld from the generic i18n domain name rules, then by all means, those
should be prohibited. But for the most part, everything that has
historically been described in nameprep should be in the i18n hostname
rules and enforced by the applications, protocols and data-formats. The
generic i18n domain name rules should be as flexible as possible, and IDNA
should only encode/decode the generic i18n domain names.

This model will allow additional subordinate profiles to be defined by any
other WG as needed (EG, mailbox names), and IDNA will still function as
expected. Too many restrictions and we will end up with label- and
data-specific codecs, which is absolutely not what we want.

> I was basically told not to discuss email local-parts on this mailing

We have talked about email local-parts several times. I'm not sure what
you are referring to here.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 16 Feb 2002 04:35:20 -0000

> inappropriate for any data types that are case-sensitive.

False. Let's look at what the specifications actually say:

   * RFC 1034 says that ``all present domain functions'' are
     case-insensitive ``assuming an ASCII character set.'' (For
     non-English speakers: ``present'' means ``now existing.'')

   * RFC 1035 says that case-insensitivity is used throughout the domain
     system ``at present.'' (``At present'' means ``at this time.'')

   * RFC 1035, section 2.3.3, quite explicitly allows administrators to
     store case-sensitive data in the domain name system, contrary to
     your claims. Have you ever read the RFC?

Both documents demand that case be preserved. Consequently, if you
destroy the case of (for example) a mailbox name inside an SOA record,
you are violating the DNS specifications.

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 16 Feb 2002 05:01:45 +0000

"D. J. Bernstein" <[email-masked]> wrote:

>    * RFC 1035, section 2.3.3, quite explicitly allows administrators
>    to store case-sensitive data in the domain name system

You can't have foo.com and Foo.com refer to different information.
If you have case-sensitive names that you're trying to map onto
domain names, you're going to run into that limitation.  Domain names
may be case-preserving to some extent, but they do not accomodate
case-sensitive names.

AMC

---

From: "D. J. Bernstein" <[email-masked]>
Date: 16 Feb 2002 05:22:12 -0000

Adam M. Costello writes:
> domain names, you're going to run into that limitation.

False. I have already given an example: the mailbox name stored in an
SOA record.

One SOA record can say Johnny.fbi.gov, for example, while another one
says johnny.fbi.gov. Mail in the first case must go to [email-masked].
Mail in the second case must go to [email-masked].

There is no guarantee that these are the same address. If you screw up
the case, you are violating RFC 1034 and RFC 1035.

Your ignorance of this use of DNS is typical of the carelessness that
pervades the IDNA design.

> Domain names may be case-preserving to some extent, but they do not
> accomodate case-sensitive names.

RFC 1035, section 2.3.3, final paragraph, clearly and explicitly allows
case-sensitive names in DNS. It even uses the words ``case-sensitive.''
Are you completely unable to read?

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Feb 2002 23:47:52 -0600

> You can't have foo.com and Foo.com refer to different information.

As owner names, those labels will be treated the same for comparison
purposes. As owner names of RRs which are provided as supplemental data
(such as CNAME completion, NS RRs in Authority, etc) the capitalization
will be preserved. As RR data, their capitalization will be preserved.

> may be case-preserving to some extent, but they do not accomodate

The case-sensitive data-types which are normally found in DNS do not
normally exist as owner domain names. For example, there are no RRs that
use mailboxes as owner domain names. However, as M Andrews pointed out in
<[email-masked]>, this has not always been the
situation and it isn't guaranteed to always be the situation in the future
either. Mandatory lowercasing on all i18n domain names would be
counter-productive in that scenario as well.

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 16 Feb 2002 06:06:31 +0000

> False.  I have already given an example: the mailbox name stored in an

Okay, there you can get away with it because that domain name is never
the name of a record in the DNS, it is only part of the returned
data.  Names of records (that is, names that can be queried) are
case-insensitive.

> RFC 1035, section 2.3.3, final paragraph, clearly and explicitly
> allows case-sensitive names in DNS.  It even uses the words
> ``case-sensitive.''  Are you completely unable to read?

Are you?  Here's the paragraph you refer to:

    Systems administrators who enter data into the domain database
    should take care to represent the data they supply to the
    domain system in a case-consistent manner if their system is
    case-sensitive.  The data distribution system in the domain system
    will ensure that consistent representations are preserved.

Nowhere does that paragraph apply the term "case-sensitive" to the
names.  It applies to term "case-sensitive" to the system on which the
database resides.  It's warning administrators not to accidentally enter
both "foo" and "Foo" into the database if their database would blindly
do that, because the DNS database is not allowed to have both "foo" and
"Foo" in the same zone.

It says that the DNS will preserve the case of whatever was entered,
but as I explained above, that doesn't mean the DNS can accomodate
case-sensitive names of records.  If I have a collection of
case-sensitive names, it might contain names that differ only in their
case, and DNS will not make the distinction.

AMC