Subject: Re: [idn] I-D ACTION:draft-ietf-idn-idna-08.txt

From: Dan Oscarsson <[email-masked]>
Date: Mon, 27 May 2002 17:08:48 +0200

Comments on draft-ietf-idn-idna-08.txt

-
Section 1: Introduction

Here it is stated that IDNA does not require changes in any
infrastructure and that the existing ASCII name service of DNS
is sufficient.

This is wrong.

Regarding the ASCII names of DNS is sufficient you probably mean
that it is sufficient for IDNA. It is not generally sufficient.

As DNS allowes non-ASCII octet values in names, IDNA require
changes in the infrastructure as IDNA prohibits non-ASCII
octet values which in turn breaks backward compatibility.

Also, it is stated that resolvers do not need to be changed.
But it is probably the resolver libraries that should be
changed. By changing the resolver libraries on my system
so that applications are not aware of ACE-encoded names,
there is a good possibility for me to use domain names
containg non-ASCII very quickly as all applications will
get it at the same time. If it is implemented using new
application code I am not sure I can use non-ASCII in
my names, even 10 years in the future.

-
The definition of what IDN means is complex.
It is defined to be a domain name on which ToASCII can
be applied without failing. And then in the text IDN
is sometimes used to mean a ACE-encoded name and
sometimes to mean a UCS-encoded name.

This can easily confuse the reader.

The basic definition for domain names should be
something like:
A domain name is a sequence of hierachicaly orderd
labels, each label being a sequence of characters.

Note: characters, not octets.

A domain name must always use the character set of
its context!

Using the above two rules you get domain names working as
most people expect them.

Section 3 says in requirement 1) that several
different "dots" are allowed between labels when written
that way. There are probably more "dots" in UCS that
could work. I see no reason to forbid them, unless
all are forbidden exepct U+002E.
Only one type of dots simplifies comparing of names.
Only one type of "dot" should be used in each context.
In a Latin based context only U+002E should be used.

Most of Section 3 is directly given by the two rules above.

2) If a name is put into an ASCII context, it will have
to be converterd into ACE using ToASCII.

3) If a name is put an a non_ASCII context it must be
changed to the character set of that context (ToUnicode
is not enough, all context are Unicode).

4) When two names are compared they must both be
in the same context and be normalised.

IDNA mandates (probably) that a domain name used
in a link in a html document must be ACE encoded.
This is not what is natural for people to use.
People will expect a domain name in a html document
to use the same character set as the rest of the text.
Expect conflicts here. People will not ACE-encode names.
It is important for everybody to immediately
accept non-ASCII characters in domain names, even though
standards are not fully up to date.

-
Section 6.3 and Section 6.6

In the beginning of the document it is stated that IDNA
do not change the DNS system. But in Section 6.3 and 6.6
this draft states a change to DNS conflicting with the
existing standard.

It says: domain names conating non-ASCII characters must
use ACE encoded labels.

This is in conflict with current DNS RFCs which allowes
domain names containing non-ASCII. It also breaks
backward compatibility as that type of domain names are
in use today.

The difficulty of the current DNS standard is that it does not
define what octet values should be used for characters
outside the ASCII range. As ISO 01646 is the most recommended
form of code point values, the current DNS standard should
be updated defining that octet values outside the ASCII
range are UTF-8 or ISO 8859-1 (as both are ASCII compatible).
Best would be if DNS labels were updated to use UTF-8 or
ISO 8859-1 (both can be used as they can easily be
identified and handled by the same decoder code).

IDNA must NOT restrict current usage and DNS standards
making it impossible to update the DNS standard later
with a suitable defined handling of non-ASCII octet
values!

IDNA is not an update of the DNS standard. IDNA is a
way to put one more layer on top of DNS by encoding
names into ASCII code values. DNS itself must
be allowed to evolve in a simple easy manner.

   Dan

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 27 May 2002 18:18:56 +0200

> octet values which in turn breaks backward compatibility.

No, it doesn't prohibit non-ASCII in DNS. It only say that labels which
pass ToUnicode function are legal IDN names and that other labels which
start with the ACE prefix SHOULD be avoided.

It is very explicitly stated this way so (a) IDN version 2 (whoever dare to
talk about it) can use a different ACE prefix and (b) some different binary
encoding of whatever charset can be used in labels.

IDNA does NOT change the definition of DNS labels. It only says that labels
with a specific prefix are said to be "IDN labels".

   paf

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 29 May 2002 01:48:28 +0000

Dan Oscarsson brings up an interesting point.  The introduction section
of IDNA says:

    IDNA does not require changes to any infrastructure.  In particular,
    IDNA does not require any changes to DNS servers, resolvers, or
    protocol elements, because the ASCII name service provided by the
    existing DNS is entirely sufficient.

Later, section 6.3 says:

    All IDNs served by DNS servers MUST contain only ASCII characters.

Clearly IDNA does "require" a change to the DNS infrastructure, because
it introduces some new requirements, some of which apply to DNS servers.

This apparent contradiction comes from multiple senses of the word
"require".  When I wrote that paragraph in the introduction, I meant
"require" in the sense of "depend on", not "demand".

I suggest clarifying this by changing "require" to "depend on" in the
two instances quoted above, and making similar changes in the last
paragraph before 1.1:

  - Proposals that were not chosen by the IDN Working Group would
  - require that user applications, resolvers, and DNS servers be
  - updated in order for a user to use an internationalized domain name.
  + Proposals that were not chosen by the IDN Working Group would depend
  + on user applications, resolvers, and DNS servers being updated in
  + order for a user to use an internationalized domain name.

  - Rather than require widespread updating of all components, IDNA
  - requires only user applications to be updated;
  + Rather than rely on widespread updating of all components, IDNA
  + depends on updates to user applications only;

Even though IDNA does make some demands on DNS servers, IDNA will still
work with DNS servers that predate IDNA, as long as the new zone files
are created in an IDNA-conformant way (with IDNs stored in their ASCII
forms).  That's why we can say that IDNA does not depend on changes to
the infrastructure.

Dan Oscarsson <[email-masked]> wrote:

> that it is sufficient for IDNA.

Yes, exactly.

> It is not generally sufficient.

I think the meaning is clear enough, but I wouldn't object if "for IDNA"
were added at the end of that paragraph (again, the first paragraph of
the introduction).  I'll leave the decision to Paul and Patrick.

> it is stated that resolvers do not need to be changed.

Indeed, they don't need to be changed.

> But it is probably the resolver libraries that should be changed.

The spec doesn't say they shouldn't be changed, it just says they don't
need to be.  Whether they should be changed is something that OS makers
can decide for themselves.

> The definition of what IDN means is complex.  It is defined to be a
> domain name on which ToASCII can be applied without failing.

The internals of ToASCII are complex, but once you encapsulate that
concept, the definition of IDN is quite simple, as you have just
illustrated.

> And then in the text IDN is sometimes used to mean a ACE-encoded name
> and sometimes to mean a UCS-encoded name.

When the text says IDN, it means IDN as defined is section 2.  If there
is someplace where IDN is used to mean something else, please point it
out, because it would be an error.

An IDN can contain ACE labels and non-ACE ASCII labels and non-ASCII
labels.  As long as every label of a domain name passes ToASCII, it's an
IDN.

> The basic definition for domain names should be

IDNA does not define domain names or labels.  That's not its job.  IDNA
defines internationalized domain names and internationalized labels in
terms of the preexisting concepts of domain names and labels.

> Section 3 says in requirement 1) that several different "dots" are
> allowed between labels when written that way.  There are probably more
> "dots" in UCS that could work.

There are.

> I see no reason to forbid them, unless all are forbidden except
> U+002E.

We don't forbid them.  The draft nowhere prohibits applications from
accepting additional characters as dots.

But only full stop and ideographic full stop (and their
fullwidth/halfwidth versions) are required to be accepted by all
applications.  Ideographic full stop was singled out because it's the
only one that anyone ever asked for.  I think that's probably because
it's the only one that belongs to a script large enough to use a modal
input method.  CJK users need to switch modes in order to type the ASCII
full stop, whereas for other users the ASCII full stop is probably
within easier reach.

> Only one type of dots simplifies comparing of names.

True, but comparing names is already pretty complex (it involves Unicode
normalization, and sometimes also transcoding to Unicode).

> In a Latin based context only U+002E should be used.

Names can be copied from one context into a different context.  I'd
rather both dots work in both contexts.  I can't think of any way users
would benefit if Latin contexts refused to accept ideographic full stop.

> changed to the character set of that context

We distinguish between structured data (domain name slots, which contain
domain names intended to be used by programs) and unstructured data
(text intended primarily to be viewed by humans).

If an IDN is put into an IDN-unaware domain name slot, then it must
be converted to the ASCII form regardless of whether that context
supports non-ASCII characters.  That's because even if the application
receiving the name knows how to represent non-ASCII strings, it doesn't
necessarily know how to compare IDNs.  We must play it safe and use the
ASCII form unless we know the receiver knows how to compare IDNs.

If an IDN is put into unstructured text, the usual rules for putting any
string into that context apply; IDNA adds no special requirements for
IDNs in this situation.

> 4) When two names are compared they must both be in the same context
> and be normalised.

IDNA defines comparison of names very precisely.

> IDNA mandates (probably) that a domain name used in a link in a html
> document must be ACE encoded.  This is not what is natural for people
> to use.

It's up to the HTML folks to update the HTML spec to support IDNs
natively.  I think they're already working on this.  In the meantime,
authors of HTML documents can either enter the ACE themselves or use an
HTML editor/preprocessor that understands IDNA.

> IDNA must NOT restrict current usage and DNS standards making it
> impossible to update the DNS standard later with a suitable defined
> handling of non-ASCII octet values!

The door is left open for updates to DNS:

    If a signaling system which makes negotiation possible between old
    and new DNS clients and servers is standardized in the future, the
    encoding of the query in the DNS protocol itself can be changed from
    ACE to something else, such as UTF-8.

What we don't want to allow is untagged non-ASCII text in DNS messages,
because there is no standard for what it means or how it should be
compared.

> IDNA is not an update of the DNS standard.

IDNA is mainly orthogonal to the DNS standard, but it could be
considered an update in one small respect.  RFCs 1034 and 1035 say:

    domain name comparisons for all present domain functions are done in
    a case-insensitive manner, assuming an ASCII character set, and a
    high order zero bit.

    we may someday need to add full binary domain names for new
    services;

    future additions beyond current usage may need to use the full
    binary octet capabilities in names

    Name servers and resolvers must compare labels in a case-insensitive
    manner (i.e., A=a), assuming ASCII with zero parity.

It seems pretty clear that the 8th bit is reserved for future use.
As far as I know, no use for the 8th bit has ever been standardized.
IDNA effectively says that the 8th bit is not to be used for
internationalization, at least not for the time being.  It's still
available for other purposes (like non-textual domain names).

Patrik Fältström <[email-masked]> wrote:

> IDNA does NOT change the definition of DNS labels.

Right.

> It only says that labels with a specific prefix are said to be "IDN
> labels".

There is no such term as "IDN label".  Labels that get modified by
ToUnicode are said to be "ACE labels".

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 28 May 2002 21:17:47 -0500

> two instances quoted above, and making similar changes in the last

Please delete the reference. It is non-contributory to implementation of
the specification and is somewhat argumentative. Cumulatively, it is a
distraction to the protocol.

> As far as I know, no use for the 8th bit has ever been standardized.

The standardized usage is local interpretation. Any usage is compliant.

> available for other purposes (like non-textual domain names).

These can still be supported, but they can't be run through the converter.
They have to be treated as exceptions. This is in DM-IDNS-01, now that it
is going to a strong data-typing model.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 29 May 2002 03:00:26 +0000

> The standardized usage is local interpretation.  Any usage is

I'll agree that any usage is compliant, but by definition "local
interpretation" is not "standardized".  By the way, using the 8th bit is
risky, because the DNS spec is very unclear (if not downright silent)
about how DNS servers compare octets with the 8th bit set.

> These can still be supported, but they can't be run through the

Right.  IDNA requirements do not apply to non-text labels, because:

    Throughout this document the term "label" is shorthand for "text
    label", and "every label" means "every text label".

(Section 2.)

AMC

---

From: [email-masked]
Date: Wed, 29 May 2002 13:44:30 +1000

> about how DNS servers compare octets with the 8th bit set.

	I don't know what's "unclear" or "silent" about the following
	from RFC 1035.

Although labels can contain any 8 bit values in octets that make up a
label, it is strongly recommended that labels follow the preferred
syntax described elsewhere in this memo, which is compatible with
existing host naming conventions.  Name servers and resolvers must
compare labels in a case-insensitive manner (i.e., A=a), assuming ASCII
with zero parity.  Non-alphabetic codes must match exactly.

--
Mark Andrews, Internet Software Consortium
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 29 May 2002 04:23:40 +0000

[email-masked] wrote:

> Although labels can contain any 8 bit values in octets that make up
> a label, it is strongly recommended that labels follow the preferred
> compare labels in a case-insensitive manner (i.e., A=a), assuming
> ASCII with zero parity.  Non-alphabetic codes must match exactly.

How do I "compare labels assuming ASCII with zero parity" when the input
strings are not ASCII or don't have zero parity?  If I can really assume
zero parity, then I can ignore the 8th bit completely and compare only
the bottom 7 bits.  On the other hand, "match exactly" suggests that I
should require the 8th bit to match too.  But even then, among the codes
128..255, which are non-alphabetic?  I don't know, but I'm required to
do a case-insensitive comparison, so I'm screwed.

Is that unclear enough for you?  :)

AMC

---

From: [email-masked]
Date: Wed, 29 May 2002 15:10:49 +1000

> Is that unclear enough for you?  :)

	The alphabetic characters in this context are 0x41-0x5a and 0x61-0x7a
	(zero parity ASCII A-Z, a-z).  All others are compared exactly.

	Mark

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 29 May 2002 00:23:07 -0500

>Is that unclear enough for you?  :)

You are the only one that is unclear on it. 0x80-0xFF must match
exactly. Any usage of any of these character codes is allowed and legal,
but the meaning of those codes only has context within the application
that uses the data. As far as STD13 is concerned, these aren't
necessarily even characters, they are just eight-bit codes.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Wed, 29 May 2002 07:52:11 -0700

At 12:23 AM -0500 5/29/02, Eric A. Hall wrote:
>exactly.

Funny, that doesn't match what a DNS expert posted a little before 
you did; maybe there is more unclarity than you think.

Further, neither your rules nor Mark's rules take into account the 
many encodings that don't reserve the octets used for ASCII. Think 
UTF-16. Or ISO-2022-JP. Or numerous others.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 29 May 2002 11:54:38 -0500

> you did; maybe there is more unclarity than you think.

There is no conflict between the answers. I responded to the earlier query
regarding "using the 8th bit" while Mark responded to the "alphabetic
characters" part of the followup query.

The only unclarity remains to be in the IDNA camp, where it took seven
drafts to figure out there was a difference between domain names and
hostnames, and where there is continued misunderstanding on the basic
premises underlying the use of domain names in general.

> UTF-16. Or ISO-2022-JP. Or numerous others.

What's your point? The DNS rules are pretty simple.

---

From: [email-masked]
Date: Thu, 30 May 2002 08:28:45 +1000

> --Internet Mail Consortium

	The rules are how you compare labels as "they appear on
	the wire".

	If you are using a local encoding that does not reserve
	the those octets to ASCII (like one of the EBCIDCs) you
	map to ASCII then compare.  IBM has been doing that for
	years for their main frames.

	Mark

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 29 May 2002 23:59:47 +0000

> (zero parity ASCII A-Z, a-z).  All others are compared exactly.

That may be what the popular DNS implementations do, but I don't see RFC
1035 saying how to compare 0x80-0xFF.

> You are the only one that is unclear on it.  0x80-0xFF must match

I'm not the only one.  Here's an excerpt from a message by John Klensin:

http://www.ietf.org/mail-archive/ietf/Current/msg15742.html

RFC1035> For all parts of the DNS that are part of the official
RFC1035> protocol, all comparisons between character strings (e.g.,
RFC1035> labels, domain names, etc.) are done in a case-insensitive
RFC1035> manner.  At present, this rule is in force throughout the
RFC1035> domain system without exception.

John Klensin> To emphasize, that is "all parts" and "all comparisons",
John Klensin> not "unless you happen to find the high bit turned on".

John Klensin> An existing and conforming implementation has no way to do
John Klensin> those required case-insensitive comparisons outside the
John Klensin> ASCII range.

Robert Elz> No, nor is it required to.

John Klensin> There we probably disagree -- I suggest that the text is
John Klensin> at least ambiguous and might require it.

AMC

---

From: Dan Oscarsson <[email-masked]>
Date: Thu, 30 May 2002 08:27:47 +0200

>because there is no standard for what it means or how it should be

It is simple to standardise it. We just publish a short RFC
defining how octet values with the 8th bit should be handled.

There are already DNS servers supporting names in UTF-8 and other
encodings. Why not standardise what encoding should be used?

>    manner (i.e., A=a), assuming ASCII with zero parity.

Yes ASCII characters must have zero parity! This is because DNS uses 8
bit
octets, not 7bit ASCII.

>As far as I know, no use for the 8th bit has ever been standardized.

Then it is high time it was standardised. Let's standardise it to
be UTF-8 as that is highly compatible with current usage.

>available for other purposes (like non-textual domain names).

That is bad, the 8th bit is already used for internationalisation!

   Dan

---

From: John C Klensin <[email-masked]>
Date: Thu, 30 May 2002 06:21:25 -0400

The disagreement between kre and myself about how to read  the
spec is, indeed, I think the key to this issue.  It should be
noted that we disagree about two things, which are separate:

(i) Whether the correct reading on the DNS specs is that "ASCII"
versus "binary" is a function of RR type, or a function of the
bits in the octets.  I take the text to suggest that _only_
ASCII (note, ASCII, not LDH) is permitted in RR types defined in
or by 1034/1035, but that "future" RR types (and Classes) can be
defined to have binary labels.  Robert takes it as permitted
characters outside the ASCII range, with case handling undefined
for those characters.

(ii) Whether the documents are clear or not, at least on this
matter.  Robert believes that they are.  Perhaps we are in
agreement -- I think they are fairly clear too, but in the other
direction-- but that is at least the basis for a claim of
ambiguity (or that I am an idiot, which is also a reasonable
possibility).

      john

--On Wednesday, May 29, 2002 11:59 PM +0000 "Adam M. Costello"
<[email-masked]> wrote:

>> The alphabetic characters in this context are 0x41-0x5a and
>> 0x61-0x7a (zero parity ASCII A-Z, a-z).  All others are
>> compared exactly.
> 
> That may be what the popular DNS implementations do, but I
> don't see RFC 1035 saying how to compare 0x80-0xFF.
> 
>> You are the only one that is unclear on it.  0x80-0xFF must
>> match exactly.
> 
> RFC1035> protocol, all comparisons between character strings
> (e.g., RFC1035> labels, domain names, etc.) are done in a
> case-insensitive RFC1035> manner.  At present, this rule is in
> force throughout the RFC1035> domain system without exception.
> 
> John Klensin> To emphasize, that is "all parts" and "all
> comparisons", John Klensin> not "unless you happen to find the
> high bit turned on".
> 
> John Klensin> An existing and conforming implementation has no
> way to do John Klensin> those required case-insensitive
> comparisons outside the John Klensin> ASCII range.
> 
> John Klensin> There we probably disagree -- I suggest that the
> text is John Klensin> at least ambiguous and might require it.
> 
> AMC
> 
> 
>

---

From: vint cerf <[email-masked]>
Date: Thu, 30 May 2002 07:33:15 -0400

what has been the practice in implementations of the various
referenced RFCs? John's interpretation or Robert's?

or has the apparent ambiguity also shown up in working
implementations?

vint

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 30 May 2002 11:01:43 -0500

> characters outside the ASCII range, with case handling undefined

You are both correct, but Robert is more correct. Note that Robert didn't
disagree with you "future" use. Essentially this boils down to having
biblical scholars arguing over how MUCH they should love their neighbor.

There are three elements here:

  1) The lower seven-bit range is definitely interpreted as ASCII.

  2) The upper eight-bit range is definitely undefined for those RRs
     and classes which do not have an interpretation. While some new
     RR and/or class could define an interpretation for the eight-bit
     range, the existing RRs in the IN class treat this range as
     undefined values.

  3) STD13 is quite clear that queries may contain eight-bit codes.

So, taking all of the above into consideration, how do you treat the
eight-bit codes for existing RRs in the IN class, where an interpretation
has not been defined? You have to treat them as undefined. Since they are
undefined, they are not alphabetic, and the non-alphabetic comparison rule
kicks in.

Whether or not any of these RRs allow eight-bit codes to be defined is a
totally different question, but one which was answered in the affirmative
by RFC 2181.

So, unless a new RR or class (or EDNS tag, or whatever) defines some
specific interpretation, the eight-bit codes are to be treated as
undefined codes, and must match exactly.

> agreement -- I think they are fairly clear too, but in the other

You are both wrong on that one. This is something that I hope to help fix
at some point.

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 30 May 2002 11:04:42 -0500

> or has the apparent ambiguity also shown up in working

Whenever any ambiguity appears, it is at the UI level, and as a violation
of RFC 2181 (not allowing the use of eight-bit codes when creating an RR).

All of the servers I have looked at treat eight-bit codes as opaque data,
in accordance with the spec.

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 30 May 2002 11:56:28 -0500

> encodings. Why not standardise what encoding should be used?

The reason why this won't work is that there is no way to determine if the
remote end-point is compliant with the updated specification. If you want
to enforce an interpretation of the eight-bit range, you have to use new
RRs, a new class, an EDNS identifier, or something, in order to
distinguish between the legacy and modern systems. Otherwise, you send
UTF-8 to the remote system, it treats the data as MacRoman, and the
subsequent usages are working with invalid data. There must be an
identifier of some kind.

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 30 May 2002 21:52:39 +0000

Funny, I find myself almost entirely agreeing with Eric Hall on this
topic.  :)

>   1) The lower seven-bit range is definitely interpreted as ASCII.

For all text strings, yes, which at least includes the names and values
of all resource records defined in 1035.

>      range, the existing RRs in the IN class treat this range as

Agreed.

>   3) STD13 is quite clear that queries may contain eight-bit codes.

Agreed.

> So, taking all of the above into consideration, how do you treat
> the eight-bit codes for existing RRs in the IN class, where an
> interpretation has not been defined?

Good question!

> Since they are undefined, they are not alphabetic

That's the one point where I don't follow you.  If they are undefined,
then their alphabeticness is undefined, and hence the proper way to
compare them is undefined.  RFC 1035 says that comparisons of text
strings must be case-insensitive "without exception".  Doing an exact
match of 0x80-0xFF is guaranteed to violate that requirement for every
charset I know of that uses the 8th bit.

If you send a query containing an 8-bit name, I think you do so at your
own risk, because RFC 1035 doesn't say what the server will do.

> If you want to enforce an interpretation of the eight-bit range, you
> have to use new RRs, a new class, an EDNS identifier, or something, in
> order to distinguish between the legacy and modern systems.

Agreed.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 30 May 2002 18:10:32 -0500

"Adam M. Costello" wrote:

> then their alphabeticness is undefined

Being undefined, they are just opaque codes. Nothing more, nothing less.
They may be represented as characters within the scope of a local system
who only has a charset for output purposes, but that doesn't mean the data
contains characters. The data contains opaque codes. Nothing more.

> > order to distinguish between the legacy and modern systems.

Then change IDNA. Mandatory transliteration introduces the same kind of
absurd recklessness.

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 31 May 2002 00:31:57 +0000

"Eric A. Hall" <[email-masked]> wrote:

> Being undefined, they [80..FF] are just opaque codes.  Nothing more,
> nothing less.  They may be represented as characters within the scope
> of a local system who only has a charset for output purposes, but that
> doesn't mean the data contains characters.  The data contains opaque
> codes.  Nothing more.

So you're saying that a label containing some octets in the range 0..7F
and some in the range 80..FF is neither text nor binary, but actually
a mixture of the two.  That's a clever motivation for the comparison
algorithm that you endorse, but I don't see that model described by RFC
1035.  If anything, RFC 1035 appears to take the view that labels are
text, at least until further notice.

    <domain-name>s make up a large share of the data in the master file.
    The labels in the domain name are expressed as character strings and
    separated by dots.

    \DDD where each D is a digit is the octet corresponding to the
    decimal number described by DDD.  The resulting octet is assumed to
    be text and is not checked for special meaning.

    For all parts of the DNS that are part of the official protocol, all
    comparisons between character strings (e.g., labels, domain names,
    etc.) are done in a case-insensitive manner.  At present, this
    rule is in force throughout the domain system without exception.
    However, future additions beyond current usage may need to use the
    full binary octet capabilities in names...

Taken together, these statements strongly suggest that labels are
text until further notice, and therefore label comparisons must be
case-insensitive for the entire labels.

I doubt we're going to agree on this, but the fact that we could keep
arguing about it indicates that RFC 1035 is not clear about how to
compare 80..FF.

> > > If you want to enforce an interpretation of the eight-bit range,
> > > you have to use new RRs, a new class, an EDNS identifier, or
> > > something, in order to distinguish between the legacy and modern
> > > systems.
> >
> > Agreed.
>
> Then change IDNA.  Mandatory transliteration introduces the same kind
> of absurd recklessness.

You say it's dangerous to put non-ASCII data into a protocol message
where the semantics of non-ASCII data are undefined, and I agree.
That's why IDNA prohibits putting non-ASCII domain names into protocol
messages (and function arguments, structured data files, etc) unless the
protocol/interface/format explicitly invites IDNs.

I assume it's the ASCII --> non-ASCII conversion that worries you, but
that happens only when displaying names to users (the ASCII form would
be user-hostile), and only when the aforementioned prohibition does not
apply, so the risk is usually small, and implementors are invited to
forgo this conversion when they judge the risk to be significant.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 30 May 2002 19:55:05 -0500

"Adam M. Costello" wrote:

> a mixture of the two.

That's one way to read it. Another way to read it is that all of the
codepoints are opaque, except that ASCII ranges A-Z and a-z are to be
treated as case-neutral for the purposes of comparison ONLY. Which is, in
fact, exactly what RFC 1035 says.

> be user-hostile)

The results are identical to what would happen if we reinvented the
existing RRs. There is no practical difference to the user community. The
only safe way to handle either of these ridiculous scenarios is to let the
governing bodies decide how the data should be handled.

---

From: John C Klensin <[email-masked]>
Date: Fri, 31 May 2002 12:55:53 -0400

--On Thursday, 30 May, 2002 19:55 -0500 "Eric A. Hall"
<[email-masked]> wrote:

>...
> That's one way to read it. Another way to read it is that all
> of the codepoints are opaque, except that ASCII ranges A-Z and
> a-z are to be treated as case-neutral for the purposes of
> comparison ONLY. Which is, in fact, exactly what RFC 1035 says.

Eric,

In the interest of making progress in this discussion (if that
is possible at all), can I ask that you, and others, try to
avoid statements like "exactly what RFC 1035 says".  Many of us,
even among those who believe that RFC 2181 settled the issue
definitively, believe that 1035 is a bit unclear.  And, even
when I suspend disbelief and accept that 1035 intended "binary
labels" to be accepted in any RR, I'd suggest that your
interpretation is not obvious from 1035.

I would suggest, instead, that you would need to define three
types of labels:

	* ASCII-only, interpreted with case-matching
	
	* Character, but not necessarily ASCII, interpreted with
	case-matching in the 0x00 to 0x7F range and as absolute
	matches outside that range.
	
	* Binary, which had best not be subject to case-matching
	comparisons because they might be, e.g., unsigned
	numbers.  I think we can agree that 1035 gives little or
	no advice about how to interpret such labels, although
	it is clear that they can exist, at least for
	as-yet-undefined RR types or Classes.

My problem is that I don't know how to tell the three cases
apart (except by looking at RR type or Class definitions).  If
you believe that only the first two cases exist (which is what
your comments seem to imply), then the first is a special case
of the second and all is well... except that 1034 and 1035
clearly (at least to me) provide for binary content, which would
not get case-dependent interpretation.

If you believe, as I do, that only the first and third cases
exist, then there is still a problem, since a binary label could
reasonably have every 8th bit turned off, making it
indistinguishable from the ASCII ones.

And that is, it seems to me, the solution to the dilemna that
Dan poses.  Regardless of whether it would be a good idea, it
would be plausible for IETF to come along at this point and say
something much more definitive than 2181 does.  One form of that
might be:

	Heretofore, as defined by at least one reading of RFC
	1034/ 1035, the labels associated with A, MX, TXT, SRV,
	etc., RRs in Class IN consisted of ASCII characters,
	with labels containing octets outside the ASCII range
	being treated as undefined (note, the label, not the
	octet) and their use being non-standard.   Henceforth,
	octets with values greater than 0x7F are permitted in
	labels for those RR types.  Octets with values from 0x00
	to 0x7F are to be compared as specified in RFC 1035,
	i.e., case-independently for ASCII graphic characters.
	Octets with values from 0x80 to 0xFF are to be compared
	directly, i.e., without any mapping or translation.
	Such labels, and the associated RRs, will be described,
	for convenience, as "character labels" and "character
	RRs".
	
	Future RRs and Classes may be defined so that their
	labels are binary, i.e., that they are to be interpreted
	without consideration of the content or possible
	mappings of any octet.  New RRs MUST indicate whether
	their labels are "character" or "binary"; new Classes
	MUST specify some appropriate rule, i.e., either
	specification on a per-label basis or a single rule for 
	the Class.

Personally, I don't particularly like the above rule, since it
would give us case-independence for ASCII, but case-dependency
for all other scripts and languages that make distinctions.
That strikes me as a really bad idea, one that would cause far
more trouble in the future.  But it is a decision and change the
IETF could logically make.

More generally, if we are trying to _change_ a rule, or specify
a rule where none existed, I think we need to do two things.
The first is to be sure that our change is compatible and
acceptable, either by identifying the new case as different from
the old (such as with a label) or by assuring ourselves that the
previous behavior was undefined and non-standard and that we can
therefore safely specify a behavior.   The latter probably
requires looking at usage, not just text of documents, but it
should still be possible and, as Dan points out, we have done it
in the past.

Second, we should be very clear that we are making a change.  We
should avoid replacing ambiguous or contradictory text with new
text that claims to clarify or not change anything but that,
intentionally or not, involves tricky reading that later permits
people to come back and say "well, we got this into Proposed
Standard by claiming it wasn't a change, but it now specifies
what was previously unspecified".  If nothing else, the latter
approach tends to lead to a very poor level of specification,
such as whether a label that contains octets in both the
0x00-0x7F and 0x80-0xFF ranges is to be interpreted as "binary"
or with different comparision rules on a per-octet basis.
Similarly, we should avoid pretending that cases that were never
anticipated, and that have odd effects at the margins, were
really what was intended all along.

FWIW, there is also a small performance issue here, which people
seem to be ignoring.  If I know a label is all ASCII graphics
and is to be compared on a case-matching basis, I can mask off
the high bits (on an atomic full-string basis on some hardware)
and then do a full-string compare (again, in hardware on much
hardware).  If I know it is "binary", then I can skip the
masking and do a full-string compare.   But, if some octets are
to be treated a ASCII, and some as binary, I need to construct a
loop and examine each octet in turn to figure out how to compare
it.  Not a good idea if it can be avoided.

    john

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 31 May 2002 12:29:14 -0500

John C Klensin wrote:

> you believe [...]

What I believe is that DNS systems store and exchange raw octet values.

Secondary to that point, the only time that ASCII enters into the equation
is when the octet values need to be interpreted, as in: (1) the 7-bit
values are rendered for display, and (2) the 7-bit values in the A-Z and
a-z ranges from ASCII are compared. In all other cases, the data is
treated as simple octet values. This includes ASCII character values (NB
the case-specific storage rules), the non-character ASCII values, and the
eight-bit non-interpreted codes.

Tertiary to that point, the rules for hostnames are recommendations which
act as a higher-layer octet-range mask, but this does not change the
storage and transfer formats, nor does it change the interpretation rules.
Instead it only defines a reasonable subset of characters with high
probabilities of successful interpretation.

In short, all code values in the DNS database and message are treated as
octets. If a domain name contains a seven-bit code, that value must be
treated as a precise ASCII character. If a domain contains an eight-bit
code, then that code must be treated as undefined data. If the domain name
needs to be compared with another domain name, all of the codes must be
treated as eight-bit values, with the exception of the A-Z and a-z ranges.

I also believe that it is entirely possible for people to think way too
much about this.

Finally, I also believe that any subsequent discussion on this matter
should be directed to namedroppers. This WG is not the proper forum for
any advancement of this discussion.