Subject: Re: [idn] Re: 7 bits forever!

From: Gunnar Lindberg <[email-masked]>
Date: Thu, 4 Apr 2002 17:32:24 +0200 (MET DST)

This discussion resembles much of the arguments in the pre-MIME days:
"8-bit clean MTAs is enough, fix that and we're done". I didn't agree.

Now, I can't resist a followup on the "MALAYALAM" example... In
    http://www.unicode.org/charts/charindex.html	{,2,3}
I looked for
    cat charindex*.html | egrep '>2E|2E<'

Unless I'm misstaken 0x2E == US ASCII '.'. Of course, this character
never occurs inside the DNS itself, since it's the separator and is
coded as such in all DNS transactions (implicitly coded in the length
fields, rather).

But, how are "8-bit clean" applications supposed to handle domain
names where these symbols occur mixed with "normal" US ASCII domain
symbols & separators - think of

    www."PEACE SYMBOL".org
    www 0x2E 0x26 0x2E 0x2E org

(this discussion could have made use of some of that :-). So, there
is a need to encode anyway. My 10 öre: Let's stick with an encoding
that is "as little destructive as possible", i.e. 7 bit US ASCII, and
leave all the fancy improvements for the future.

	Gunnar Lindberg

ALL AROUND-PROFILE              232E
BELOW, COMBINING BREVE          032E
BREVE BELOW, COMBINING          032E
CIRCLE, PLUS SIGN IN RIGHT HALF 2A2E
CJK Phonetics and Symbols Area  2E00
CJK Radicals Supplement         2E80
COMBINING BREVE BELOW           032E
CONTOUR INTEGRAL                222E
decimal point                   002E
dot                             002E
ESTIMATED SYMBOL                212E
FULL STOP                       002E
HALF CIRCLE, PLUS SIGN IN RIGHT 2A2E
INTEGRAL, CONTOUR               222E
OVERRIDE, RIGHT-TO-LEFT         202E
PEACE SYMBOL                    262E
period                          002E
Phonetics and Symbols Area, CJK 2E00
PLUS SIGN IN RIGHT HALF CIRCLE  2A2E
PROFILE, ALL AROUND-            232E
Radicals Supplement, CJK        2E80
RIGHT-TO-LEFT OVERRIDE          202E
rlo                             202E
Symbols Area, CJK Phonetics and 2E00

---

From: Robert Elz <[email-masked]>
Date: Thu, 04 Apr 2002 23:18:09 +0700

Date:        Thu, 4 Apr 2002 17:32:24 +0200 (MET DST)
    From:        Gunnar Lindberg <[email-masked]>
    Message-ID:  <[email-masked]>

  | Unless I'm misstaken 0x2E == US ASCII '.'

Yes.

  | Of course, this character never occurs inside the DNS itself,

Nonsense.   Rarely occurs perhaps, but absolutely any 8 bit value
can occur in a domain name in the DNS.   *ANY*.

The DNS has never had any kind of problem with dealing with 8 bit
data, all it has ever lacked was rules for interpretation.  Simply
saying "It is UTF-8" would fix that part.  But so would "It is 8859-1"
except that would not be a wonderful choice.   "It is UCS-2" (or 4) might
perhaps be OK, but would be horribly in-compatible with what is deployed.

  | But, how are "8-bit clean" applications supposed to handle domain
  | names where these symbols occur mixed with "normal" US ASCII domain
  | symbols & separators - think of
  | 
  |     www."PEACE SYMBOL".org
  |     www 0x2E 0x26 0x2E 0x2E org

Trivial.

	len 3, value "www"
	len 2, value 0x26 0x2E
	len 3, value "org"
	len 0

(though I suspect the UTF-8 encoded value of 0x262e would be 3
bytes long, and would not include any US-ASCII lookalikes - that is
one of the features of UTF-8)

  | (this discussion could have made use of some of that :-). So, there
  | is a need to encode anyway.

Nonsense.

The only need for encoding appears in the other applications whose rules for
dealing with domain names are more restricted.

If people arguing about this stuff actually had half a clue about how
the DNS actually works, things would be much easier.

kre

---

From: Gunnar Lindberg <[email-masked]>
Date: Thu, 4 Apr 2002 18:39:37 +0200 (MET DST)

>From [email-masked]  Thu Apr  4 18:16:41 2002
>From: Robert Elz <[email-masked]>
>To: Gunnar Lindberg <[email-masked]>
>cc: [email-masked]
>Subject: Re: [idn] Re: 7 bits forever!
>In-Reply-To: <[email-masked]>
>References: <[email-masked]>
>Date: Thu, 04 Apr 2002 23:18:09 +0700
>Message-ID: <[email-masked]>

>	len 0

Please excuse the mix of text and 0x and remove spaces

    gethostbyname("www 0x2E 0x26 0x2E 0x2E org");

Where did your "len" come from?

>the DNS actually works, things would be much easier.

Agreed gethostbyname() is not part of the DNS per se, but rather few
applications will work well without it. Nonsense? Trivial?

	Gunnar Lindberg

>kre

PS
    gethostbyname("www.&..org");
								    DS

---

From: "Gary E. Miller" <[email-masked]>
Date: Thu, 4 Apr 2002 09:50:01 -0800 (PST)

Yo Robert!

> can occur in a domain name in the DNS.   *ANY*.

Maybe it can, but that does not make it right.

RFC 1035 "DOMAIN NAMES - IMPLEMENTATION AND SPECIFICATION"

2.3.1

[...]

"The labels must follow the rules for ARPANET host names.  They must
start with a letter, end with a letter or digit, and have as interior
characters only letters, digits, and hyphen.  There are also some
restrictions on the length.  Labels must be 63 characters or less."

That has been updated by RFC 1123 to allow a number as the first char
of a host name.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 20340 Empire Blvd, Suite E-3, Bend, OR 97701
	[email-masked]  Tel:+1(541)382-8588 Fax: +1(541)382-8676

---

From: Robert Elz <[email-masked]>
Date: Fri, 05 Apr 2002 22:53:16 +0700

Date:        Thu, 4 Apr 2002 09:50:01 -0800 (PST)
    From:        "Gary E. Miller" <[email-masked]>
    Message-ID:  <[email-masked]>

  | Maybe it can, but that does not make it right.
  | 
  | RFC 1035 "DOMAIN NAMES - IMPLEMENTATION AND SPECIFICATION"
  | 
  | 2.3.1

If you actually go read that section, carefully, instead of just
quoting the part from it that everyone notices first, you will see
that it says something quite different from what you think it does.

You need to read the part of the section that appears on the preceding
page of the formatted RFC...

Or see (part of) rfc2181 for a longer verison of this.

kre

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Apr 2002 14:41:53 -0500

--On Friday, 05 April, 2002 22:53 +0700 Robert Elz
<[email-masked]> wrote:

>     Message-ID:
> <[email-masked]>
> 
> If you actually go read that section, carefully, instead of
> just quoting the part from it that everyone notices first, you
> will see that it says something quite different from what you
> think it does.
> 
> You need to read the part of the section that appears on the
> preceding page of the formatted RFC...
> 

Actually, having read that section, and several other sections,
_very_ carefully in recent months, I think 2181 is contradictory
at best, and possibly seriously wrong, on this point.

As I read them, what 1034 and 1035 say is that the DNS can
accomodate any octets, but that [at least then]
currently-specified RRs are restricted to ASCII.  The LDH rule
is a good ("best"?) practices one.  It is the LDH rule that RFC
1123 modified slightly.  And it is quite correct to assert that
the LDH rule is not a _DNS_ requirement.

But the ASCII rule is a firm requirement.  For evidence of this,
temporarily ignore the text (although, personally, I think it is
clear -- especially in 2.3.3-- if read carefully) and examine
the requirement that, for the defined RRs, labels and queries be
compared in a case-insensitive way.  For ASCII, that is a
well-defined operation, one that can be performed by doing the
comparison under a bit mask.  For other scripts, as the IDN WG
discovered, "case insensitive comparison" is typically not
completely well-defined, often involves complex tables and/or
knowledge of local context, and is sometimes quite controversial
as to what is intended.

So I believe that the "future RRs" language with regard to
binary labels in 1034 and 1035 must be taken seriously and as
normative text: if new RRs (or new classes) are defined, they
can be defined as binary and, hence, as not requiring
case-insensitive comparisons.  Conversely, within the current
set (or at least the historical set at the time of 1034/1035),
case-insensitive comparison is required and hence binary must
not be permitted.

Any other reading, I believe, leads immediately either to
contradictions or to undefined states within the protocol.

As an aside, it appears to me that this requirement for
case-insensitive comparison is the real problem with "just put
UTF-8 in the DNS" approaches.  An existing and conforming
implementation has no way to do those required case-insensitive
comparisons outside the ASCII range.  Worse, if it does those
comparisons by bit-masking (which would be conforming today),
there is a risk of its getting rather bizarre errors (of either
matching or not matching) on characters outside the ASCII range.
One supposes that we could modify the protocol to specify that
case-insensitive comparisions be made only for octets in the
ASCII range, but, unless that were done through an EDNS option,
it would be a potentially fairly significant retroactive change.

    john

---

From: Gunnar Lindberg <[email-masked]>
Date: Sat, 6 Apr 2002 11:55:50 +0200 (MET DST)

Case insensitive lookup is likely to be a similar problem regardless
of whether you use "raw" UTF-8 or you encode things in 7-bit ASCII.

To me the real issue, however, seems to be in the applications, i.e.
in the resolvers (libresolv.*/gethostbyname()/gethostbyaddr()) and
all the places where the resulting names are to be processed
    netdb.h:
    char *h_name;
    char **h_aliases";

It would be, er, hm, unwise to assume that all applications would do
The Right Thing (TM) when the "char *" starts to carry UTF-8 data
(which is 8-bit, unless ASCII equivalent). And, do remember that
these strings can come to you without you asking for them, it's NOT
www URLs only - just think of spam "From [email-masked]"
(SE 8859-1, although nicely encoded in UTF-8, 8-bit).

Now, the IT industri has been declining the last last months, but I
doubt it would do good to create such massive amount of extra work.

MIME QP - agreed it's a kludge, agreed it's uggly - did save us from
a lot of problems (i.e. no Flag Day needed). Don't forget that.

	Gunnar Lindberg

>From [email-masked]  Fri Apr  5 22:21:17 2002
>Date: Fri, 05 Apr 2002 14:41:53 -0500
>From: John C Klensin <[email-masked]>
>To: Robert Elz <[email-masked]>
>cc: [email-masked]
>Message-ID: <9863660.1018017713@localhost>
>In-Reply-To: <[email-masked]>
>References:  <[email-masked]>

---

From: Robert Elz <[email-masked]>
Date: Sat, 06 Apr 2002 18:44:56 +0700

I really hoped to be able to avoid having to do this, yet again...

  | As I read them, what 1034 and 1035 say is that the DNS can
  | accomodate any octets, but that [at least then]
  | currently-specified RRs are restricted to ASCII.

Sorry John, I can't fathom how you could possibly reach that conclusion
from what is in 1034 & 1035.

Eg: from 1035 (section 3.1) ...

	Although labels can contain any 8 bit values in octets that make up a
	label, it is strongly recommended that labels follow the preferred
	syntax described elsewhere in this memo, which is compatible with
	existing host naming conventions.

How much clearer do you want it?

1034 is less clear, but (section 3.5 --- note its title)

  3.5. Preferred name syntax

  The DNS specifications attempt to be as general as possible in the rules
  for constructing domain names.  The idea is that the name of any
  existing object can be expressed as a domain name with minimal changes.
  However, when assigning a domain name for an object, the prudent user
  will select a name which satisfies both the rules of the domain system
  and any existing rules for the object, whether these rules are published
  or implied by existing programs.

"The prudent user will select" ...  that is, this is a damn good idea,
but you don't have to do it if you know what you're doing.

  | The LDH rule is a good ("best"?) practices one.

It is required (as updated) if the domain name is to be used in an e-mail
header (which back then, was almost the only other formalised place that
domain names appeared - other than that was all OS specific command/arg stuff).

  | But the ASCII rule is a firm requirement.

No it isn't, there is nothing at all which says that.

  | For evidence of this, temporarily ignore the text

Hmm - ignore what is written, and attempt to infer from something else...

  | (although, personally, I think it is clear -- especially in 2.3.3--
  | if read carefully)

2.3.3 is about character case, and I agree, that is a very messy area indeed.

  | and examine
  | the requirement that, for the defined RRs, labels and queries be
  | compared in a case-insensitive way.

Not quite.   What it says that ascii labels (ones with the top bit clear)
must be handled that way, it carefully refrains from saying what should
be done in other cases - leaving that for future definition (which is kind
of what this recent work has all been about).   However, it clearly allows
non-ascii labels - it just doesn't specify what they mean, or how to
interpret them.   That's what needed to be done to allow non-ascii names
to have some kind of meaning.

  | So I believe that the "future RRs" language with regard to
  | binary labels in 1034 and 1035 must be taken seriously and as
  | normative text: if new RRs (or new classes) are defined, they
  | can be defined as binary and,

Have you actually thought about what you have just said?   That is,
the rules for naming the DNS tree depend upon the data that is stored
there?

Do you seriously mean that?

Classes are a whole other mess, that no-one really seems to understand,
one of those "this might be a good idea" frills, that is completely undefined.
It isn't clear whether different classes share the same namespace or
not (just they they share a few RR type definitions).   Classes are
essentially extinct.

  | hence, as not requiring
  | case-insensitive comparisons.  Conversely, within the current
  | set (or at least the historical set at the time of 1034/1035),
  | case-insensitive comparison is required and hence binary must
  | not be permitted.

Case insensitive comparison of ascii is required, what is done with the
rest is undefined.   To make it meaningful it needs to be defined, that
I agree with.

One easy (though perhaps not desirable, I don't know) solution would be
to simply restrict the case insensitive part, as far as the DNS is
concerned, to ascii only, so that A==a but Á!=á.   Eventually doing away
with case insensitive for all labels seems like a good idea to me.

  | Any other reading, I believe, leads immediately either to
  | contradictions or to undefined states within the protocol.

Undefined, yes.   That's not unusual, lots of protocols have undefined states.

  | As an aside, it appears to me that this requirement for
  | case-insensitive comparison is the real problem with "just put
  | UTF-8 in the DNS" approaches.

Not really - what causes the problem is putting more than ascii there.
As soon as you permit that, you have to deal with all of the issues.
The way the bytes are encoded is irrelevant.   One way out of this is
to require that the DNS always use the "lower" case (whatever that happens
to be in any particular instance - that is, whenever multiple characters are
generally assumed to mean the same, pick one as the one that must always be
used within the DNS) and have the resolver enforce it.   Whether the
data once chosen is encoded in UTF-8 or some other way is irrelevant.

The problem with doing this is that it requires every resolver to be able
to handle every possible case mapping (for any domain that it may ever
encounter - which is all of them, or course).   On the other hand, doing it
in the server only requires the server to understand the case folding rules
for the actual domain names it serves, not necessarily anyone else's
(back end caches have a problem either way of course)

In any case, these are the issues that a WG that was tasked with defining
how the DNS should treat non ascii labels should be dealing with.
Currently, there's none of that happening - idn simply decided not to
bother, and make everything inside the DNS remain ascii forever.
(Recently I have seen some ramblings about long term conversion from ACE to
UTF-8 inside the DNS - that's a ludicrous prospect that can never happen).

  | An existing and conforming
  | implementation has no way to do those required case-insensitive
  | comparisons outside the ASCII range.

No, nor is it required to.

  | One supposes that we could modify the protocol to specify that
  | case-insensitive comparisions be made only for octets in the
  | ASCII range, but, unless that were done through an EDNS option,
  | it would be a potentially fairly significant retroactive change.

That's not actually a modification, that's what is currently required.

kre

---

From: Robert Elz <[email-masked]>
Date: Sat, 06 Apr 2002 18:55:42 +0700

Date:        Sat, 6 Apr 2002 11:55:50 +0200 (MET DST)
    From:        Gunnar Lindberg <[email-masked]>
    Message-ID:  <[email-masked]>

  | To me the real issue, however, seems to be in the applications, i.e.
  | in the resolvers

If applications have problems with this, they almost certainly need to
be fixed anyway - regardless of what the standards say should appear,
the application has to deal with what actually exists, which need not
have a lot in common with what is specified.

I actually suspect that you'll find that far fewer applications have
problems dealing with anything other than labels containing 0x00 or 0x2e
the former of which is quite likely to be a major problem for most
applications, and the latter for a large number.  UTF-8 usage would
not trigger either of those however.

kre

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Apr 2002 11:21:59 -0500

> I really hoped to be able to avoid having to do this, yet

And I apologize for taking a shortcut because I didn't want to
take the time to pull out the text again.  
> Sorry John, I can't fathom how you could possibly reach that
> conclusion from what is in 1034 & 1035.

See below.
> 	Although labels can contain any 8 bit values in octets that
> make up a 	label, it is strongly recommended that labels
> follow the preferred 	syntax described elsewhere in this memo,
> which is compatible with 	existing host naming conventions.
> 

Yes, that is one of the "LDH rules are a good practice" sections
I was referring to.  And

>   The DNS specifications attempt to be as general as possible
> in the rules   for constructing domain names.  The idea is
> that the name of any   existing object can be expressed as a
> domain name with minimal changes.   However, when assigning a
> domain name for an object, the prudent user   will select a
> name which satisfies both the rules of the domain system   and
> any existing rules for the object, whether these rules are
> published   or implied by existing programs.
> 
> "The prudent user will select" ...  that is, this is a damn
> good idea, but you don't have to do it if you know what you're
> doing.

And that is the other one.  We are in complete agreement about
this.

> It is required (as updated) if the domain name is to be used
> in an e-mail header (which back then, was almost the only
> other formalised place that domain names appeared - other than
> that was all OS specific command/arg stuff).

Hmm.  I'd argue that the other existing protocols count[ed] --
FTP and Telnet connection specifications, two-level specs in
Finger and Whois, etc., but I think this isn't important.

> No it isn't, there is nothing at all which says that.

See below.
 
> Hmm - ignore what is written, and attempt to infer from

As I said, trying to take a shortcut and to provide some
additional evidence of intent/logical consistency in case the
sections that I think are important are read as contradicting
those that you cite above.  I don't read the spec as
contradictory, but as specifying three sets of rules: LDH
(recommended), ASCII (required for "existing" RRs), binary
(possible future extensions in new RRs).
 
>   | (although, personally, I think it is clear -- especially
> in 2.3.3--   | if read carefully)
> 
>   | the requirement that, for the defined RRs, labels and
> queries be   | compared in a case-insensitive way.
> 
> Not quite.   What it says that ascii labels (ones with the top
> bit clear) must be handled that way, it carefully refrains
> from saying what should be done in other cases - leaving that
> for future definition (which is kind of what this recent work
> has all been about).   However, it clearly allows non-ascii
> labels - it just doesn't specify what they mean, or how to
> interpret them.   That's what needed to be done to allow
> non-ascii names to have some kind of meaning.

More on this below (I'm going to paste in an earlier analysis,
with the text cited, rather than screwing it up by trying to
reconstruct) but I don't see anything in the text that says "if
you see the high bit set, you can assume it is binary and other
rules apply; if the high bit is zero, then it is ASCII and needs
case-independent comparison".  It seems to me that a statement
of that general nature would be needed to justify your assertion
above.  I note with interest that even 2181 doesn't seem to
include such a statement as a clarification of what is an "ascii
label" and what is a "binary label".  What it says instead
(section 11) is

		Those restrictions aside, any binary string whatever can
		be used as the label of any resource record.  Similarly,
		any binary string can serve as the value of any record
		that includes a domain name as some or all of its value
		(SOA, NS, MX, PTR, CNAME, and any others that may be
		added).

and, from the abstract, where "the last two" refer to the
canonical name issue and the valid contents of a labeL:

		The other two are already adequately specified, however
		the specifications seem to be sometimes ignored.  We
		seek to reinforce the existing specifications.

>From which I assume that 2181 did not intend to change anything
about 1034/1035 in this area and that its approval by the IESG
was based on that assumption.

>   | binary labels in 1034 and 1035 must be taken seriously and
> as   | normative text: if new RRs (or new classes) are
> defined, they   | can be defined as binary and,
> 
> Have you actually thought about what you have just said?
> That is, the rules for naming the DNS tree depend upon the
> data that is stored there?
> 

I think what I'm suggesting is that the valid content of a given
label depends on the RR type (and Class) with which it is
associated.  One can question the wisdom of that in retrospect,
but that it what the specification says.

> Classes are a whole other mess, that no-one really seems to
> understand, one of those "this might be a good idea" frills,
> that is completely undefined. It isn't clear whether different
> classes share the same namespace or not (just they they share
> a few RR type definitions).   Classes are essentially extinct.

We could debate that too, but I agree that it does not seem
important at this stage, except, perhaps, to understanding where
binary labels might be used.

>   | case-insensitive comparisons.  Conversely, within the
> current   | set (or at least the historical set at the time of
> 1034/1035),   | case-insensitive comparison is required and
> hence binary must   | not be permitted.
> 
> Case insensitive comparison of ascii is required, what is done
> with the rest is undefined.   To make it meaningful it needs
> to be defined, that I agree with.
> 
> One easy (though perhaps not desirable, I don't know) solution
> would be to simply restrict the case insensitive part, as far
> as the DNS is concerned, to ascii only, so that A==a but Á!=á.
> Eventually doing away with case insensitive for all labels
> seems like a good idea to me.

Of course, to support "case insensitivity for ASCII only", it
would be nice to have an algorithmic rule for identifying ASCII.
But binary labels can, in principle, have octets with the high
bit clear, or even all octets with the high bit clear.  And one
does not want to apply case-insensitivity matching to binary
labels, not matter how they are structured.  So, I believe, in
logic, that one needs to know, on a per-RR type (or, in
principle, per-query-type or other per-query) basis, whether the
comparison involves character comparison (hence case insensitive
over at least some of the octets) or binary comparison
(comparison of bits, no fussing).

I can't comment on whether doing away with case insensitivity is
a good idea, since I can argue either for of against it in new
applications.  But the transition from case insensitive
comparison to case sensitive (or binary) comparison would be a
very interesting exercise.

> Undefined, yes.   That's not unusual, lots of protocols have

See below.

>   | case-insensitive comparison is the real problem with "just
> put   | UTF-8 in the DNS" approaches.
> 
> Not really - what causes the problem is putting more than
> ascii there. As soon as you permit that, you have to deal with
> all of the issues. The way the bytes are encoded is
> irrelevant.   One way out of this is to require that the DNS
> always use the "lower" case (whatever that happens to be in
> any particular instance - that is, whenever multiple
> characters are generally assumed to mean the same, pick one as
> the one that must always be used within the DNS) and have the
> resolver enforce it.   Whether the data once chosen is encoded
> in UTF-8 or some other way is irrelevant.

Except that some of those "other ways" may result in octets with
the high bit clear that do not represent ASCII characters
(assuming, as you do, that 1034/1035 require case insensitive
comparison for ASCII only).  In other words, the DNS needs to
know something about the encoding in order to know when to apply
case insensitive comparison (and, potentially, how to do it).

> The problem with doing this is that it requires every resolver
> to be able to handle every possible case mapping (for any
> domain that it may ever encounter - which is all of them, or
> course).   On the other hand, doing it in the server only
> requires the server to understand the case folding rules for
> the actual domain names it serves, not necessarily anyone
> else's (back end caches have a problem either way of course)

I think this is correct.   While I haven't done the analysis, my
intuition tells me that, if we are going to go down this path on
the server side, we may have big problems potentially-recursive
RRs like DNAME and NAPTR, but that is a separate problem.  I
hope.
 
> In any case, these are the issues that a WG that was tasked
> with defining how the DNS should treat non ascii labels should
> be dealing with. Currently, there's none of that happening -
> idn simply decided not to bother, and make everything inside
> the DNS remain ascii forever. (Recently I have seen some
> ramblings about long term conversion from ACE to UTF-8 inside
> the DNS - that's a ludicrous prospect that can never happen).

Yes.
 
>   | implementation has no way to do those required
> case-insensitive   | comparisons outside the ASCII range.
> 

There we probably disagree -- I suggest that the text is at
least ambiguous and might require it.  But, at some level, it
isn't important, because the text clearly prohibits non-ASCII
labels in "existing" RRs.  See below.

>   | One supposes that we could modify the protocol to specify
> that   | case-insensitive comparisions be made only for octets
> in the   | ASCII range, but, unless that were done through an
> EDNS option,   | it would be a potentially fairly significant
> retroactive change.
> 

Not my reading of sections you didn't cite.  See below.

That earlier analysis (slightly updated) and the text
citations...

[...]

... and that has led me to carefully re-read old text.
That, in turn, leads to a question: it is very clear that
nothing in the DNS spec requires the LDH rule, even though it
appears as "prudent user" guidance in section 2.3.1 of RFC 1035
(and elsewhere). But it appears to me that binary labels are not
permitted on the common RR types, for at least one
technically-rational reason, and that 2181 glosses this over a
bit.

Specifically...

>From RFC1034, section 3.1

		By convention, domain names can be stored with arbitrary
		case, but domain name comparisons for all present domain
		functions are done in a case-insensitive manner,
		assuming an ASCII character set, and a high order zero
		bit.  This means that you are free to create a node with
		label "A" or a node with label "a", but not both as
		brothers; you could refer to either using "a" or "A".
		When you receive a domain name or label, you should
		preserve its case.  The rationale for this choice is
		that we may someday need to add full binary domain names
		for new services; existing services would not be
		changed.

That statement is presumably part of your justification for
assuming that all bets are off if the high order bit is on.
Whether that is important depends on what "existing services"
refers to, plus the problem of binary labels that don't happen
to contain octets with the high bit set and how they are to be
recognized and thence compared.

and RFC1035:

		2.3.3. Character Case
		
		For all parts of the DNS that are part of the official
		protocol, all comparisons between character strings
		(e.g., labels, domain names, etc.) are done in a
		case-insensitive manner.  At present, this rule is in
		force throughout the domain system without exception.
		However, future additions beyond current usage may need
		to use the full binary octet capabilities in names, so
		attempts to store domain names in 7-bit ASCII or use of
		special bytes to terminate labels, etc., should be
		avoided.

I'm inclined to read "additions beyond current usage" as
implying new RRs or new Classes, you are inclined to read it as
having octets with the high bit on appear in existing RRs.  It
seems to me that this is at least a bit ambiguous, rather than
crystal-clear in the latter direction.  More important, it
appears to me to make a clear (and necessary) distinction
between "character strings" and "full binary octet capabilities"
in the DNS, to require case-insensitive comparison only for the
former, and hence to require that one be able to tell the
difference unambiguously.

But the first part of this does say "For all parts of the DNS
that are part of the official protocol, all comparisons between
character strings ...  are done in a case-insensitive manner."
To emphasize, that is "all parts" and "all comparisons", not
"unless you happen to find the high bit turned on".  So, in the
absence of some standards-track document that changes the
comparison rule -- either for new RRs or retroactively for
existing ones -- it seems to me that we are stuck with it.  And
that "However" sentence seems to apply to storage forms in
implementations, not to what is permitted in labels or queries.

[...]
The requirement to do case-mapping is, I think, ultimately a
restriction on the labels.  It makes it hard for me to think
about the interpretation of a binary label unless the label is
specified as "binary" as part of the description of the
associated RR.  Indeed, given understanding we have gained with
the IDN WG (which PVM probably didn't have when 1034/1035 and
their predecessors were written), it makes it hard for me to
think about anything but ASCII for anything but new RRs (or,
potentially, classes).  Moreover, the text of 1034/1035 appears
to me to require ASCII labels for all RR types specified in
those documents, and maybe even for all new RR types that don't
explicitly specify binary labels.

And, if after going through this, you find that we are still
reading the text differently, I suggest that 2181 probably need
updating to clarify how one of those "any binary string" labels
are to be interpreted when they appear in queries that require
case-insensitive matching.  Otherwise, we have what appears to
be a very strong statement about what is permitted with no
specification at all about how it is handled if one appears.
That doesn't seem to me to be the path to interoperability.

     john

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Apr 2002 11:21:59 -0500

See below.

See below.

 

See below.

 

Yes.

[...]

Specifically...

and RFC1035:

		2.3.3. Character Case

[...]

     john

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Apr 2002 11:21:59 -0500

See below.

See below.

 

See below.

 

Yes.

[...]

Specifically...

and RFC1035:

		2.3.3. Character Case

[...]

     john

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Apr 2002 11:21:59 -0500

See below.

See below.

 

See below.

 

Yes.

[...]

Specifically...

and RFC1035:

		2.3.3. Character Case

[...]

     john

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 06 Apr 2002 15:20:10 -0600

> have a lot in common with what is specified.

There are a couple of intertwined problems here. First off, the octet
values are not specified with an interpretation, so what is the
application supposed to use when it has to render the values? Conversely,
if the user specifies some non-ASCII characters, will these be the same
characters that are displayed by the remote end-point? The problem isn't
transferring the octets, it is converting the octets from display to
transfer, or vice versa. In that regard, the applications which multilate
eight-bit names are somewhat broken (hyper-technical perspective), but to
be fair, it's not solely their fault. This is the exact same problem as
IDNA transliteration -- there's no telling what happens to the data after
it has been mutated for rendering, and vice versa.

This is where isolation becomes important. Keep the old apps out of the
new *interpreted* eight-bit namespace, using new APIs, new messages,
whatever it takes. If they screw up due to an interaction problem with
charsets, give them an error instead of giving them the wrong information
or allowing the wrong information to pollute other apps. I agree that it
would be nice if the old apps could be fixed (then we could go with
'just-use-UTF8'), but the truth of the matter is that there isn't any way
to tell if an old app *HAS* been fixed, other than to use some new
mechanism, as in the above.

So, long story short, there's no way to just lay this on the apps.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 06 Apr 2002 15:50:10 -0600

> Do you seriously mean that?

I had the same initial reaction a while ago. However, once you think about
it some, its not really such a bad idea. I mean, DNS is the only major
protocol which does not strictly define the allowable syntax for each
data-type. The result is that we get things like user_name.example.com in
homepage URLs, which further results in things like BIND 9's warning
mechanisms. We can rant about idiots ignoring the rules all day long, but
it seems to me that clearly defining the data-types in a way that software
can enforce the allowable syntax is the only way out of this mess in the
long term.

BTW, I don't agree with John's insinuation that binary is an exception.
Clearly binary labels are fully legitimate today (albeit very
problematic). What this means however is that all RR types will have to be
explicitly defined (TXT = anything, A = safe-set, CNAME = aliased-data,
and so forth).

There's a related problem here, which is that DNS specifies a way to
escape characters locally (that "\65" equates to 0x41 and not "A"), but it
doesn't specify a way to transfer this marking in the message. This is the
counter-problem to the display problem mentioned in the other message, and
it has equal ramifications. Defining data-types is one way to solve this
problem, as John suggests. Extending the message structure to allow these
markings to survive from end-to-end is another. I would prefer the former
given that there are other issues which would be solved by that approach
already (plus the fact that there's no reason why 0x41 should need to be
used in A RRs).

> with case insensitive for all labels seems like a good idea to me.

That's the direction things are heading.

---

From: Robert Elz <[email-masked]>
Date: Mon, 08 Apr 2002 16:42:05 +0700

Date:        Sat, 06 Apr 2002 11:21:59 -0500
    From:        John C Klensin <[email-masked]>
    Message-ID:  <84281257.1018092119@localhost>

  | Hmm.  I'd argue that the other existing protocols count[ed] --
  | FTP and Telnet connection specifications,

Neither telnet nor FTP transmit domain names (other than as uninterpretable
data, as in "Welcome to hostname") - the only relevance of domain name
rules there is to the OS interface (which is perhaps what you mean by
connection specifications) which isn't part of the spec.   It is entirely
possible for any OS to allow just about anything as the name spec, just
as long as it eventually gets mapped to the correct IP address.   Eg: when
I want to fetch an RFC, I type "ftp rfc" and a local mapping mechanism
converts "rfc" into a suitable IP address.

  | two-level specs in
  | Finger and Whois, etc., but I think this isn't important.

While the protocol is ancient, the first finger spec is in 1194 (it
post-dates 1034...).  But yes, finger would have been one use beyond
e-mail - if it had been specified just what was legal and what wasn't.
I kind of suspect that it is more "anything after @ to end of line is passed
to the hostname lookup function" than anything more precise in reality.

Whois is a database lookup protocol, and looks up whatever is stored in the
database, it has nothing to do with the DNS at all (it happens that the
most popular database is that used to generate DNS zone files - but whois
queries are database keys (or searches), not domain names).

  | As I said, trying to take a shortcut and to provide some
  | additional evidence of intent/logical consistency in case the
  | sections that I think are important are read as contradicting
  | those that you cite above.

I don't, the sections quoted are absolutely unambiguous.  Anything
that could contradict them would have to be equally as clear, and there is 
nothing.

  | I don't read the spec as
  | contradictory, but as specifying three sets of rules: LDH
  | (recommended), ASCII (required for "existing" RRs), binary
  | (possible future extensions in new RRs).

But the LDH isn't a rule, it is very clear that it is a suggestion (as
far as the DNS is concerned).   It is a rule for other protocols that
use domain names (particularly 821/822 and their replacements).

  | More on this below (I'm going to paste in an earlier analysis,
  | with the text cited, rather than screwing it up by trying to
  | reconstruct) but I don't see anything in the text that says "if
  | you see the high bit set, you can assume it is binary and other
  | rules apply; if the high bit is zero, then it is ASCII and needs
  | case-independent comparison".

Except there aren't other rules that apply, so it doesn't say exactly that.
What it says is that if it is an ascii char, case independence applies,
otherwise it doesn't, and (it actually says) ascii chars have the top bit 0
(written as "zero parity" or something like that...)

  | It seems to me that a statement
  | of that general nature would be needed to justify your assertion
  | above.  I note with interest that even 2181 doesn't seem to
  | include such a statement as a clarification of what is an "ascii
  | label" and what is a "binary label".

No, aside from the (much newer) binary labels proposal, which is an
entirely different thing, there is no difference, and cannot be.  There
is exactly one kind of label.   Some of the octets in the label may be
ascii chars, in which case the case independence applies.  Others not.

  | From which I assume that 2181 did not intend to change anything
  | about 1034/1035 in this area and that its approval by the IESG
  | was based on that assumption.

No, it didn't intend to change anything about 1034/5 in this area.
That is, 1034/5 were already perfectly clear that any 8 bit value is
permitted in a DNS label, and all that was ever needed was to reinforce
that.   That's what 2181 does.

[Aside: The environment for this part of 2181 was more that one well
known implementation of the DNS had taken to either prohibiting, or
issuing warnings about (depending on config options, which defaulted to
error for primary, and warning for secondary) names that weren't strictly
LDH.  That is, other prefectly natural ascii chars (like _ and %) were
being rejected.   Actually doing something when the top bit of an octet
in the label is set wasn't the issue - but reiterating that it was
permitted (and hence, DNS implementations must not complain about it)
was important.   Whether anyone sane would use such a thing is a
whole different issue.]

  | I think what I'm suggesting is that the valid content of a given
  | label depends on the RR type (and Class) with which it is
  | associated.  One can question the wisdom of that in retrospect,
  | but that it what the specification says.

No, that isn't what it says or means.   You're not reading it rationally.

Well, that may be for classes, no-one really knows what different classes
mean, anything at all might be possible, so let's skip that part for now.

But if what is permitted as the label of a domain name depends upon what
RR type is stored, how could that rationally work?  Unless you were to
restrict that to only the leaf label, which wouldn't be very useful to
allow the kind of arbitrary data that 1034/5 clearly envisage some day being 
stored in the DNS.  Without that restriction nodes in the DNS that have
NS records, and SOA records, would need to allow your "binary" labels,
as well as others.   And if they do, why not just everything?

No, what 1034/5 are saying is that users need to choose domain names that
meet the requirements of the applications storing the data.  At the time,
essentially everything wanted LDH, as that's all that was permitted in
the hosts file that was being replaced.   However, future applications
manipulating other kinds of names and data were clearly seen as something
for the future, and for them, the domain name to choose might not need
to be restricted.

But in any case, the DNS was already designed to cope (or at least it seemed
that way) - any random 8 bit data can be a label.

  | Of course, to support "case insensitivity for ASCII only", it
  | would be nice to have an algorithmic rule for identifying ASCII.
  | But binary labels can, in principle, have octets with the high
  | bit clear, or even all octets with the high bit clear.

No, you're trying to create two different kinds of labels.  There's no
way to do that, there's just one.   All labels (again leaving aside the
binary labels that were created later, which were entirely different)
use the same syntax - 6 bits of length, and 1 to 63 arbitrary octets.
There aren't ascii labels and binary labels, just labels that sometimes
contain some ascii characters (or often contain all ascii characters).

  | And one
  | does not want to apply case-insensitivity matching to binary
  | labels, not matter how they are structured.

It might not be what one wants, but it is what the DNS does.  If
the value of an octet is between 0x41 and 0x5a then it compares equal
to the corresponding octet with value between 0x61 and 0x7a.
Otherwise all octets are distinct.   And there's just one label type.

  | So, I believe, in logic, that one needs to know,

If one were designing this now, and attempting to make a rational design,
one may indeed want to know that.   But that isn't what was done, and you
simply cannot attempt to retrofit desires from today onto the specification
of yesterday.   It is what it says it is, no more and no less.

  | But the transition from case insensitive
  | comparison to case sensitive (or binary) comparison would be a
  | very interesting exercise.

yes...   But perhaps never attempting to create it for anything other
than ascii would give insights onto what is needed to allow it to be
deleted in ascii as well - someday, maybe.

  | Except that some of those "other ways" may result in octets with
  | the high bit clear that do not represent ASCII characters
  | (assuming, as you do, that 1034/1035 require case insensitive
  | comparison for ASCII only).

That simply isn't possible as the DNS is spec'd.  If the high bit is
clear it *is* to be interpreted as ascii.   There is no option, however
much you might want to have 0x41 and 0x61 as distinct "binary" labels,
you simply cannot.  And it doesn't matter what the RR type is, you cannot.
(Perhaps in some other class, but even that is "probably not possible").

  | In other words, the DNS needs to
  | know something about the encoding in order to know when to apply
  | case insensitive comparison (and, potentially, how to do it).

No, you're inventing what you think it should need - it doesn't need
anything of the kind, it simply does what the spec says it does.

  | I think this is correct.   While I haven't done the analysis, my
  | intuition tells me that, if we are going to go down this path on
  | the server side, we may have big problems potentially-recursive
  | RRs like DNAME and NAPTR, but that is a separate problem.  I
  | hope.

Perhaps - though if anyone took the DNS seriously, about only ever storing
names in the DNS in their one true case, it wouldn't be an issue at all.

That is, in the root zone, it is COM that exists, not com - so everywhere
else in the DNS, when the (TLD) COM is meant, it should be entered as COM,
never as com (nor Com nor any other variant).   If someone does a lookup
they can give any case, and the matching rules find it - but everyone is
supposed to enter data using the one defined case for the name, as created
by the owner of the name (so it is Berkeley.EDU or should be, not
BERKELEY.EDU and not berkeley.edu ...)

Of course, that rules was ignored form day 1 (the EDU zone file persists
in storing BERKELEY.EDU I believe).   (However, you may note that you
essentially always see my e-mail address as "@munnari.OZ.AU" because that's
its representation in the DNS, except in places I don't control).

  | There we probably disagree -- I suggest that the text is at
  | least ambiguous and might require it.  But, at some level, it
  | isn't important, because the text clearly prohibits non-ASCII
  | labels in "existing" RRs.  See below.

No, it doesn't prohibit them at all, it says that they probably should
not be used.

  | >From RFC1034, section 3.1

  | That statement is presumably part of your justification for
  | assuming that all bets are off if the high order bit is on.

Yes.  The "someday we may need to add" is the only text in either 1034 or 5
which really supports your suggested text, and I claim it means "that
someday people might need to actually use", and that "full binary labels"
just means "labels using the full range of characters" not some new
kind of label.

  | I'm inclined to read "additions beyond current usage" as
  | implying new RRs or new Classes, you are inclined to read it as
  | having octets with the high bit on appear in existing RRs.

I read it as whenever an application want to be able to use more than
just ascii labels, whatever the RR type.   Eg: if someone had decided
that X.400 should replace 821/822, but run over the internet, using domain
names, then A records (and presumably MX records) would still be needed,
but there would be no LDH rule to comply with.  Names used for such an
application would be whatever it specified as legal.   The DNS is unchanged.

  | It seems to me that this is at least a bit ambiguous, rather than
  | crystal-clear in the latter direction.

Even if so, the other words that permit any 8 bit value in a label
are crystal clear - nothing that is "a bit ambiguous" can possibly
be treated as overriding what is very clearly stated.

  | More important, it
  | appears to me to make a clear (and necessary) distinction
  | between "character strings" and "full binary octet capabilities"
  | in the DNS, to require case-insensitive comparison only for the
  | former, and hence to require that one be able to tell the
  | difference unambiguously.

If there was such a distinction, then yes, but there isn't.  That's
not the way the spec was written, and not the way it is implemented.
There are just labels, the case insensitive rule is applied individually
to each octet, not to whole labels.

  | But the first part of this does say "For all parts of the DNS
  | that are part of the official protocol, all comparisons between
  | character strings ...  are done in a case-insensitive manner."
  | To emphasize, that is "all parts" and "all comparisons", not
  | "unless you happen to find the high bit turned on".

Sure, no problem with that.   But what's there when the high bit is on
isn't ascii, and so has no "case" to be sensitive to.

  | So, in the
  | absence of some standards-track document that changes the
  | comparison rule -- either for new RRs or retroactively for
  | existing ones -- it seems to me that we are stuck with it.

Yes, but we're stuck with what it actually says, not what you're prefer
that it had said.

  | And, if after going through this, you find that we are still
  | reading the text differently, I suggest that 2181 probably need
  | updating to clarify how one of those "any binary string" labels
  | are to be interpreted when they appear in queries that require
  | case-insensitive matching.

We could update 2181 again, but this part was really though almost 
unnecessary even when 2181 was written, if that well known implementation
hadn't started doing absurd things, this part of 2181 would never have been
there, as this part of 1034/5 was never really considered to be unclear
in the slightest.   And note: the implementation didn't change because of
any lack of clarity about the DNS spec, but as a means of protecting hosts
from lots of broken implementations that did "bad" things if presented
with suitably crafted DNS responses (partitularly to PTR lookups).
That is buggy implementations made bad assumptions, and that was "fixed"
by having the DNS filter the "bad" data for the application.

There was never a serious argument that the DNS actually specified this.

kre

ps: I have no real issues with what Eric Hall wrote in his replies, but
I need to read them again, but have no time right now.

---

From: Robert Elz <[email-masked]>
Date: Mon, 08 Apr 2002 16:42:05 +0700

| two-level specs in

  | And one

  | More important, it

  | So, in the

kre

---

From: Robert Elz <[email-masked]>
Date: Mon, 08 Apr 2002 16:42:05 +0700

| two-level specs in

  | And one

  | More important, it

  | So, in the

kre

---

From: Robert Elz <[email-masked]>
Date: Mon, 08 Apr 2002 16:42:05 +0700

| two-level specs in

  | And one

  | More important, it

  | So, in the

kre