Subject: [idn] length restrictions on IDN label

From: Soobok Lee <[email-masked]>
Date: Sun, 13 Oct 2002 15:56:56 +0900

[ When i read IDNA draft today, I still can't find
  the answer from it for the following question about IDN label length.
 If the following issue is already addressed in the draft, please correct me. ]

 I have a punycode label of length 63 octets:
  L1: zq--o39AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
 
 L2=ToUnicode(L1) produces: U+AC00 x 56 times ( Hangul "KA" repeated 56 times)

  L2:
U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 
U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 
U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 
U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 
U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 
U+AC00 U+AC00 U+AC00 U+AC00 U+AC00 U+AC00

 But this L2 can be encoded in various unicode/legacy encodings into
  various lengths of octets:

  UTF8 : 3 x 56 = 168 octets
  UCS2 : 2 x 56 = 112 octets
  UCS4 : 4 x 56 = 224 octets
  KSX1001/EUC-KR : 2 x 56 = 112 octets 
 
 These encodings produce labels longer than  63 octets 
 
 Moreover, each ACE label of valid (<256 octets) ACE-form FQDN IDN  may be
  converted into below-63-octets valid UTF8 labels, while the cumulative sum
  of the length of each UTF8 label of the FQDN IDN may exceed 256 octets 
  limits.
  
 Many internet applications impose/assumes  the 63-octets-limit of label lengths.
 IF this assumption is violated, the label will be regarded as invalid
 labels, and produce unpredictable errors by some implementations.
 
 From implementators' point of view, more precise specificiation is needed
 about whether IDN label/FQDN has *NEW* length restrictions in various char encodings,
 if IDNA tries to extend the character repertoires of allowable characters.

 The above case is very rare, but in any cases, the implementors have practical
 security-related need to impose some limits on the iDN lables in non-ACE encodings.
 (for example, to avoid buffer overflow errors due to expanded ToUnicode labels)

 Cheers,

Soobok Lee

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 00:30:34 +0900

L1 is a utf8-encoded IDN label.

 L2=nameprep(L1), L2 is still in utf8.

 A1=ToASCII(L2)

 Function length(X) returns the octet length of the string X
 in unknown encoding and charset.

 My question in previous questions was:
   in the case of  length(A1)<=63 and length(L2)>63,
    is  L2  a valid iDN label conforming to DNS label length
     restrictions ?

 My second question is:

   if length(L2) <= 63 < length(L1),
     is L1 is valid iDN label ?

   if length(L1) <= 63 < length(L2),
     is L2 is valid iDN label ?

 I can repeat similar questions  by changing utf8 to
   other charset encodings, which will further complicate
   the problem.

 Some applications may exchange non-ASCII-form IDN instead 
 of ACE-form IDN  after negotiations. Non-ASCII-form IDN
 may be used and stored and transfered and processed as 
 its equivalent ACE version  by the new application protocols.

 If some of above labels should be declared as invalid,
  there may be needed more complex definitions of
   what labels are invalid and therefore related protocol actions
   and exception handlings of such too-long IDN labels.

 This problem has security implications from the implementors'
 viewpoints.
 
 If IDNA  draft already
 contains any clues about  iDN label length restrictions,
 suggest me the section please, so that i remain comfortable with
 this issue.

 Regards,

Soobok Lee

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 13 Oct 2002 09:39:42 -0700

>  If the following issue is already addressed in the draft, please 

It is indeed covered in the draft. The input to IDNA is code points, 
not encoded characters. As you point out, different encodings give 
different lengths for the same string. The only lengths that matter 
are those that are already in STD 13.

>  labels, and produce unpredictable errors by some implementations.

Which Internet applications are you speaking of? Which encodings are 
they using? As you pointed out, different encodings give different 
lengths. Thus, no sensible application could assume a 63-octet length 
if it deals with different encodings.

>  if IDNA tries to extend the character repertoires of allowable characters.

It seems likely that most implementers can understand that they must 
continue to follow the same rules that they always have for the 
length of domain names and labels.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 10:54:32 +0900

> if it deals with different encodings.

UTF8,EUC-KR etc are all ASCII compatible encoding/charset.
Applications don't need to give up/modify old 63-octets restrictions for
 LDH labels even in utf8 or euc-kr, because those encodings produce
the same octets string  as  pure  ASCII encoding does. That is,
in those ASCII compatible encoding of LDH chars, the number of codepoints and
the number of octets are equal, while they are not equal in encoding of
non-LDH chars like Hangul, CJK letters (the octet length is doubled or tripled).

> length of domain names and labels.

The unit of length restriction matters: # of code points or # of octets ?
That should be made clearer. RFC1035 uses "octets", not a character/code point.

I enclose related RFC1035 (STD13) sections here.

Mockapetris                                                     [Page 9]

RFC 1035        Domain Implementation and Specification    November 1987

(snip)

2.3.4. Size limits

Various objects and parameters in the DNS have size limits.  They are
listed below.  Some could be easily changed, others are more
fundamental.

labels          63 octets or less

names           255 octets or less

TTL             positive values of a signed 32 bit number.

UDP messages    512 octets or less

3. DOMAIN NAME SPACE AND RR DEFINITIONS

3.1. Name space definitions

Domain names in messages are expressed in terms of a sequence of labels.
Each label is represented as a one octet length field followed by that
number of octets.  Since every domain name ends with the null label of
the root, a domain name is terminated by a length byte of zero.  The
high order two bits of every length octet must be zero, and the
remaining six bits of the length field limit the label to 63 octets or
less.

To simplify implementations, the total length of a domain name (i.e.,
label octets and label length octets) is restricted to 255 octets or
less.

Although labels can contain any 8 bit values in octets that make up a
label, it is strongly recommended that labels follow the preferred
syntax described elsewhere in this memo, which is compatible with
existing host naming conventions.  Name servers and resolvers must
compare labels in a case-insensitive manner (i.e., A=a), assuming ASCII

[Soobok Lee]

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 13 Oct 2002 19:16:02 -0700

>That should be made clearer. RFC1035 uses "octets", not a 

Exactly right. It seems like you have answered your own question!

In case I'm missing something, where in any of the IDN documents does 
it indicate that the length restrictions are in number of code points 
instead of number of octets?

--Paul Hoffman, Director

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 11:43:08 +0900

Then,
  U+AC00 x 56 times (in my previous posting)  is a valid label  
conforming to RFC1035 ?
   and its equivalent ACE label (of  63 octets ) is a valid label ?

   UTF8-encoded IDN labels are not governed by RFC1035 length restrictions ?
   IDNA contains  brand new length restrictions for 8bit labels   which 
obsoletes RFC1035 ?

   Soobok Lee

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 13 Oct 2002 19:47:51 -0700

>   and its equivalent ACE label (of  63 octets ) is a valid label ?

If it follows the rules for ToASCII, yes.

>   UTF8-encoded IDN labels are not governed by RFC1035 length restrictions ?

There is no such thing. IDN labels are always encoded in ASCII 
following the rules of STD 13, just as it says in the draft.

>   IDNA contains  brand new length restrictions for 8bit labels 
>which obsoletes RFC1035 ?

No. Where in the draft do you see such "brand new length restrictions"?

The current goal of the WG is to fix any unclear statements in the 
document. We can't do that if you don't say exactly what text you 
find unclear.

--Paul Hoffman, Director

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 12:05:42 +0900

> following the rules of STD 13, just as it says in the draft.
That is true only in  protocols predating IDNA draft.
IDN labels can be typed in/ displayed/ copy&pasted/ or exchanged in 
 UTF8 (or other) encoding
in now and future applications or protocols slots as described in IDNA 
draft itself.
See enclosed excerpts from IDNA draft  ( "SEE HERE").

I think some length restriction in code points is needed, rather than in 
octets ....
IDNA is the right place to put such things..

Soobok Lee

6.3 DNS servers

Domain names stored in zones follow the rules for "stored strings" from
[STRINGPREP].

For internationalized labels that cannot be represented directly in
ASCII, DNS servers MUST use the ACE form produced by the ToASCII
operation. All IDNs served by DNS servers MUST contain only ASCII
characters.

If a signaling system which makes negotiation possible between old and
new DNS clients and servers is standardized in the future, the encoding
of the query in the DNS protocol itself can be changed from ACE to
something else, such as UTF-8. The question whether or not this should (SEE HERE)
be used is, however, a separate problem and is not discussed in this
memo.

6.1 Entry and display in applications

(snip)

In protocols and document formats that define how to handle
specification or negotiation of charsets, labels can be encoded in any
charset allowed by the protocol or document format. If a protocol or
document format only allows one charset, the labels MUST be given in
that charset.

In any place where a protocol or document format allows transmission of
the characters in internationalized labels, internationalized labels
SHOULD be transmitted using whatever character encoding and escape   ( SEE HERE )
mechanism that the protocol or document format uses at that place.

All protocols that use domain name slots already have the capacity for
handling domain names in the ASCII charset. Thus, ACE labels
(internationalized labels that have been processed with the ToASCII
operation) can inherently be handled by those protocols.

6. Implications for typical applications using DNS

In IDNA, applications perform the processing needed to input
internationalized domain names from users, display internationalized
domain names to users, and process the inputs and outputs from DNS and
other protocols that carry domain names.

The components and interfaces between them can be represented
pictorially as:

                     +------+
                     | User |
                     +------+
                        ^
                        | Input and display: local interface methods
                        | (pen, keyboard, glowing phosphorus, ...)
    +-------------------|-------------------------------+
    |                   v                               |
    |          +-----------------------------+          |
    |          |        Application          |          |
    |          |   (ToASCII and ToUnicode    |          |
    |          |      operations may be      |          |
    |          |        called here)         |          |
    |          +-----------------------------+          |
    |                   ^        ^                      | End system
    |                   |        |                      |
    | Call to resolver: |        | Application-specific |
    |              ACE  |        | protocol:            |
    |                   v        | ACE unless the       |
    |           +----------+     | protocol is updated  |
    |           | Resolver |     | to handle other      |
    |           +----------+     | encodings            |   (SEE HERE)
    |                 ^          |                      |
    +-----------------|----------|----------------------+
        DNS protocol: |          |
                  ACE |          |
                      v          v
           +-------------+    +---------------------+
           | DNS servers |    | Application servers |
           +-------------+    +---------------------+

---

From: "James Seng" <[email-masked]>
Date: Mon, 14 Oct 2002 11:45:30 +0800

There is already restriction in the number of codepoint or octets in UTF-8
(or other) encoding.

The restriction is defined currently as octet(ToASCII(X)) < 63.

-James Seng

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 14 Oct 2002 03:51:58 +0000

> Many internet applications impose/assumes the 63-octets-limit of

IDN-unaware applications use this simple 63-octet limit.  These
applications also assume that the domain label is ASCII.  IDN-aware
applications will be careful to use the ASCII form when talking
to IDN-unaware applications.  Applications that use non-ASCII
representations will know the more complex syntax rule for non-ASCII
labels (namely, that the label is valid if and only if ToASCII can be
applied to it without failing).

> From implementators' point of view, more precise specificiation is
> needed about whether IDN label/FQDN has *NEW* length restrictions in
> various char encodings

Section 2 defines "internationalized label" as a label to which the
ToASCII operation can be applied without failing.  There is no other
restriction on IDN label syntax.

> the implementors have practical security-related need to impose some
> limits on the iDN lables in non-ACE encodings. (for example, to avoid
> buffer overflow errors due to expanded ToUnicode labels)

That's true.  A cursory examination of the Punycode algorithm reveals
that each ASCII character can represent at most one code point;
therefore an internationalized label can represent at most 63 code
points, whether it's ACE or not.  A given encoding uses a bounded number
of octets per code point, so you can allocate your buffers based on
that.

> The unit of length restriction matters: # of code points or # of
> octets ? That should be made clearer. RFC1035 uses "octets", not a
> character/code point.

RFC 1035 limits domain labels to 63 octets, but RFC 1035 predates IDNA,
and it speaks under the explicit assumption that text is ASCII.  Because
DNS is IDN-unaware, all internationalized labels in DNS are in their
ASCII forms.  For these reasons, the 63-octet limit applies only to the
ASCII forms of internationalized labels.

IDNA does not introduce any new length restrictions.  The 63-octet limit
on ASCII labels is the only length restriction on internationalized
labels.

> Then, U+AC00 x 56 times (in my previous posting) is a valid label
> conforming to RFC1035 ?

No, it's not, and that's why IDNA requires that it be converted to its
ASCII form before being passed into an IDN-unaware protocol like DNS.

> UTF8-encoded IDN labels are not governed by RFC1035 length

Not directly.  The 63-octet limit applies to the ASCII form, not the
UTF-8 form.  It would be absurd to apply the 63-octet limit to every
possible encoding form.  You'd have to transcode a label into every
possible encoding just to check whether it's valid.

> IDNA contains brand new length restrictions for 8bit labels which

No, it contains no new length restrictions.  The RFC 1035 restriction
on the ASCII form is still the only restriction on the length of
internationalized labels.

AMC

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 13:12:36 +0900

>The restriction is defined currently as octet(ToASCII(X)) < 63.
I already mentioned this in my first posting.
What I request is confirmation and clarification about that restriction.

If UTF8-encoded, that valid 8bit label will exceed 63 octets limits (up 
to 168 octets or more)
 which is imposed by RFC1035 even upon non-ASCII 8bit labels .
 IDNA section 6.3 does not rule out that  utf8 encoded labels may be used
 in DNS wire protocols in the future. And that will affect UDP based 
 DNS protocols
 which suffers from lack of space in the UDP packet length limits (512).
 Packet truncation or protcol errors are inevitable.

That is why i suggest that some separate length restriction on utf8 
(other encoding) IDN labels
 be needed. Or clarifications about those problems, at least.

Soobok Lee

>>something else, such as UTF-8. The question whether or not this should
>>    
>>
>(SEE HERE)
>  
>
>>SHOULD be transmitted using whatever character encoding and escape   ( SEE
>>    
>>
>HERE )
>  
>

---

From: "James Seng" <[email-masked]>
Date: Mon, 14 Oct 2002 12:18:00 +0800

Let me repeat again,

the restriction is octet(ToASCII(X)) <= 63 (sorry, forget the =).

The length restriction of a domain name (incidently, UTF-8 encoded string
which looks like domain name is not a domain name) in a DNS UDP packet is
something beyond this working group. Please bring it to the DNSEXT working
group.

-James Seng

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 14 Oct 2002 04:47:58 +0000

Soobok Lee <[email-masked]> wrote:

> If UTF8-encoded, that valid 8bit label will exceed 63 octets limits
> (up to 168 octets or more)

True.

> which is imposed by RFC1035 even upon non-ASCII 8bit labels .

Yes, but labels in DNS containing octets >= 128 are not
internationalized labels, because internationalized labels use only
octets <= 127 in DNS.  Labels in DNS containing octets >= 128 are
mysterious creatures that have no standard interpretation as text
(because ASCII is the only text encoding used by the DNS standard).

> IDNA section 6.3 does not rule out that utf8 encoded labels may be
> used in DNS wire protocols in the future.

In which case the specification of those future wire protocols will need
to deal with the fact that UTF-8 forms of internationalized labels can
have more than 63 octets.  (255 is an upper bound, though not the least
upper bound.)

AMC

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 13:56:31 +0900

James Seng wrote:

>something beyond this working group. 
>
The restriction orginated from RFC1035 affects all  label 
creation/validations.
And "octet(ToASCII(X)) <= 63" seems to loosen the restrictions about
8bit labels. That is why i think RFC1035 restriction is about to be 
obsoleted.

>Please bring it to the DNSEXT working  group.
>
As a novice IETF pariticpant, I have little experience with DNSEXT WG.
Would  Area Directors make comment on this issue, Erik ?

Soobok Lee

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 14:06:57 +0900

Soobok Lee wrote:

>> which looks like domain name is not a domain name)
>
I cant' agree on your parenthesized claims.  Please look into IDNA draft 
sections
which mention utf8 encoding as alternative encoding  for labels in 
applications
and future protocols and even in future DNS protocols.

Soobok Lee

>> in a DNS UDP packet is

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 14:21:07 +0900

I  read your previous long answer. Thanks, Adams.
My comments begins...

Adam M. Costello wrote:

>octets <= 127 in DNS. 
Really ?   then, please goto to the next comment below and compare your  
claim with IDNA's utf8 position.

Length restriction itself in RFC1035 seems to have nothing to do with
ASCII and non-ASCII distinctions, from the contexts. UDP packets
length limit or practical label needs  consideration seem to be behind that.

> Labels in DNS containing octets >= 128 are
"UTF-8 forms of internationalized labels"  are not "internationalized 
labels"  ?
 if not, we should call them just " strings" as someone said ?  
I can't understand why utf8-form of a label is not a label.

Soobok Lee

---

From: [email-masked]
Date: Mon, 14 Oct 2002 15:27:49 +1000

> 8bit labels. That is why i think RFC1035 restriction is about to be 

	There is nothing new with the number of octets in presentation
	form being greater those presented on the wire.  In RFC
	1034 0x00 is presented as "\000", 0x5c as "\\" or  "\093",
	0x2e as "\." or "\046".

	Mark

--
Mark Andrews, Internet Software Consortium
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 14:40:23 +0900

>	1034 0x00 is presented as "\000", 0x5c as "\\" or  "\093",
UTF8-form of a label may be used as protocol elements , in addition to
presentation forms.
In the latter case, there will be no problem as you said above: just
display it.

But, in the former case of being used as protocols elements, utf8-form
label length limit is
of our concern. IDNA drafts does not rule out that utf-8 form of labels
may be used as protocol
elements. This will clairfy my point.

Thanks.

Soobok Lee

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 14 Oct 2002 05:52:50 +0000

Soobok Lee <[email-masked]> wrote:

> Really ?

Really.

> "UTF-8 forms of internationalized labels" are not "internationalized

The UTF-8 form of an internationalized label is an internationalized
label.  But that's irrelevant, because labels in DNS containing octets
>= 128 are not UTF-8.  The only text encoding used by DNS is ASCII
(according to the current DNS standard).  The octets >= 128 in DNS are
non-ASCII, but that doesn't mean they are UTF-8.  We don't know what
they are, except octets.

The UTF-8 form of an internationalized label is an internationalized
label, but a sequence of octets with no charset tag is not an
internationalized label (it's not even text).

So I stand by my position quoted at the top of this message.

AMC

---

From: [email-masked]
Date: Mon, 14 Oct 2002 16:09:52 +1000

> elements. This will clairfy my point.

	The restrictions have always derived from the DNS wire format.
	Each individual presentation format will have its own maximum
	number of octets however just because a string fits within
	that number of octets doesn't mean that it will be valid.

	Invalid 
	0123456789012345678901234567890123456789012345678901234567891234

	Valid
	\048\049\050\051\052\053\054\055\056\057\048\049\050\051\052\053

	Both of the above presentation strings are 64 octets long.  Only
	one is a legal label, both are < 252 octets that you would have
	to allow to convert a arbitary label from wire format to
	to RFC 1034 presentation format.  Similar things will happen
	with UTF-8 as a presentation format.  Only after you have attempted
	to covert to wire format can you determine if a arbitarty string
	of UTF-8 characters that is less than the maximum number of octets
	long will fit.

	Mark

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 15:29:41 +0900

>non-ASCII, but that doesn't mean they are UTF-8.  We don't know what
But,  UTF-8 forms make subset of the entire set of non-ASCII forms. Thus,
the utf8-compliant subset has been under the oversall length restriction 
imposed
by RFC1035 on the entire set.

>internationalized label (it's not even text).
UTF8-form of labels carry no language/encoding tag in them (unlike MIME 
does.)
They can be only  assumed or negotiated through other channel in protocols.

Soobok Lee

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 15:55:43 +0900

About presentation formats, you argument stands.
UTF8 form of labels further can be URLEncoded with %xx sequences into
tripled length of character streams.
That means "56 x U+AC00 " can be converted into
56 x 3 (utf8) x 3 (URLencode) = 504 octets string.
RFC1035 length restriction does not apply to presentation forms. This is
your point. right?

But, my question is about utf8 form as protocol elements (wire format
for future
application or DNS protocols), not as presentation forms.
You can find that in IDNA section 6.3.
Would you make another comments on that section ?

THanks.

Soobok Lee

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 14 Oct 2002 07:55:02 +0000

Soobok Lee <[email-masked]> wrote:

> UTF-8 forms make subset of the entire set of non-ASCII forms.
> Thus, the utf8-compliant subset has been under the overall length
> restriction imposed by RFC1035 on the entire set.

UTF-8 data stored directly in 8-bit DNS labels would be subject to
the 63-octet limit.  This is irrelevant to IDNs, because IDNs do not
store UTF-8 data directly in 8-bit DNS labels.  IDNA requires that
internationalized labels use their 7-bit ASCII form in DNS.

If someday you want to use UTF-8 forms of internationalized labels
directly in newDNS, you will need to make sure that newDNS allows more
than 63 octets per label.  Or you could use the UTF-8 form when it fits,
and fall back to the ASCII form when UTF-8 doesn't fit.

(Or you could decide it's easier to stick with ASCII in the DNS
protocol, and create the illusion of UTF-8 using a new resolver on the
client.)

Your argument seems to be:

1. An internationalized label in UTF-8 form is a sequence of octets.

2. RFC 1035 limits labels to 63 octets.

3. Therefore internationalized labels must have no more than 63 octets in
   UTF-8 form.

But you could try the same argument for UTF-16, and EUC-KR, and
iso-2022-jp, and BIG5, etc.  Do we conclude that any string that uses
more than 63 octets in any encoding cannot be an internationalized
label?

That would be absurd.

Perhaps the key to understanding this is to recognize that 8-bit DNS
labels are not internationalized labels.  IDNA makes no use of them.
Neither IDNA nor DNS defines any textual interpretation for them.  They
are just opaque binary data (except for the values <= 127, which are
ASCII characters).  We have no way of deciding whether 8-bit labels are
UTF-8 or ISO-8859-1 or EUC-JP, etc.  Until the DNS standard is updated
to assign some semantics, they are none of the above.

IDNA created some brand new kinds of labels that had never existed
before: non-ASCII textual labels.  They have never appeared in DNS,
cannot appear in DNS, and will not be able to appear in DNS unless DNS
is updated to support them (because the only text supported by today's
DNS is ASCII).  These new non-ASCII textual labels are outside the
universe of labels defined by RFC 1035, and therefore the RFC 1035
length restriction does not apply to them (not directly, although it
applies to their corresponding ASCII forms).

AMC

---

From: [email-masked]
Date: Mon, 14 Oct 2002 17:52:10 +1000

> RFC1035 length restriction does not apply to presentation forms. This is

	Yes.
 
> Would you make another comments on that section ?

	Well if you use raw utf-8 today int the DNS you are limited to
	63 octets per label.  255 octets for a domain name.  As long as
	you continue to use the same label encoding you are limited to
	63 octets.  YMMV at levels other than the DNS.

	However

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 17:25:55 +0900

>applies to their corresponding ASCII forms).
Okay.
non-ACE textual labels can be used and rendered in document format.

But, IDNA section 6.1 goes further than that
 by allowing _protocols_ to use non-ACE labels  which are
not presentation forms nor textual labels, but protocol elements.
What if future ESMTP allows utf8 encodings in RCPT: headers ?

Would you comment on this section?

(snip)

In any place where a protocol or document format allows transmission of
the characters in internationalized labels, internationalized labels
SHOULD be transmitted using whatever character encoding and escape
mechanism that the protocol or document format uses at that place.

---

From: "James Seng" <[email-masked]>
Date: Mon, 14 Oct 2002 17:13:11 +0800

> >>(incidently, UTF-8 encoded string which looks like domain name is
> >>not a domain name)
>
> sections which mention utf8 encoding as alternative encoding  for labels
in
> applications and future protocols and even in future DNS protocols.

Adam has already answered you on this.

"Yes, but labels in DNS containing octets >= 128 are not
internationalized labels, because internationalized labels use only
octets <= 127 in DNS.  Labels in DNS containing octets >= 128 are
mysterious creatures that have no standard interpretation as text
(because ASCII is the only text encoding used by the DNS standard)."

Therefore, a UTF-8 string which was somehow squeeze into the DNS packet has
no meaning right now.

IDNA do not ignore the possibility of using UTF-8 in DNS Packet. But the use
is yet to be defined. Until it is defined, it is useless to discuss that
possibility.

> Would  Area Directors make comment on this issue, Erik ?

I think even a novice participiant like yourself would know better to lookup
the the DNSEXT yourself then to bother ADs with such little nitpicks.

http://www.ietf.org/html.charters/dnsext-charter.html

"Issues surrounding the operation of DNS, recommendations concerning the
configuration of DNS servers, and other issues with the use of the protocol
are out of this Working Group's charter. These issues are considered in
other venues, such as operational issues in the DNS Operations Working
Group."

So bring your problem of label length (which I am not very sure what you
arguing...expand? reduce?) and propose a solution for that in DNSEXT.

Alternatively, you can look at some of the past proposal which utilized
UTF-8 and EDNS. One of them (the one written by Marc) allows about 255 UTF-8
octet and considering getting the authors resubmit them again to the DNSEXT
WG.

Of course, you may continue to talk about the confusion of the label length
limit here. But it will do you no good especially it seem that you are the
only one who is confused.

-James Seng

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 18:47:11 +0900

>is yet to be defined. Until it is defined, it is useless to discuss that
Your argument stands only in DNS on-the-wire context. IN all other 
protocol/display
 contexts, utf8 labels are legitimate domain labels. That is why zone 
admins may input
their labels in utf8 into their zone files and end users recoginze them 
as  domain names.

Moreover,
I didn't limit my question into  DNS and its protocols (just 'protocol', 
DNS is not the  'all')
Rather, that may span into all application protocols that use domain names,
like future ESMTP.

There will be many application protocols (IETF's or home-made) that
may exchange utf8-form labels as protocols elements (not for human eyes)
according to IDNA section 6.1-3. IDNA draft granted such use of utf8
labels as legitimate one.

Length restriction should be clarified before  IDN  deployments.
Currently valid IDN label should not be invalidated in the future due to
today's misspecifications.

To avoid  buffer overflow errors and security breaches, label length issues
are practically important to implementors who want to estimate enough 
buffer
spaces for IDN labels.

Soobok Lee

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 18:55:14 +0900

Soobok Lee wrote:

> their labels in utf8 into their zone files and

of course, the dns server may read them and convert into ACE forms on 
the fly in reading
zone files. Just to prevent misunderstating the sentence. Soobok Lee

---

From: "James Seng" <[email-masked]>
Date: Mon, 14 Oct 2002 18:18:53 +0800

> their labels in utf8 into their zone files and end users recoginze them

utf8 labels have no meaning as domain name labels, as currently defined in
RFC 1034/1035, whether on wire or otherwise. anything above 127 is a
no-man-zone-use-it-at-your-own-risk.

If you mean IDN labels encoded in UTF-8, then the restriction is fairly
simply, one already explained to you multiple times, octet(ToASCII(X))<=63.

> Rather, that may span into all application protocols that use domain
names,

Then it is worst then you think. This working group is definately not going
to be able to address all the concerns of all application protocols that
uses domain names, much less a future work in ESMTP. That work should be
address in other specific applications protocol and DEFINATELY not in this
working group.

I still hope to conclude this working group within the next few months and
get on with my life, dealing with other IDN related issues.

> labels as legitimate one.

Then shouldn't that application protocol be updated?

> today's misspecifications.

Yes, it should be clarify if there is any confusion.

But there is no confusion as far as I know. The length restriction is quite
clear, octet(ToASCII(X))<=63. Or are you still confused?

> are practically important to implementors who want to estimate enough

Since when implementation issue is part of the working group charter?

Please stop....Some might be scared by your FUD.

-James Seng

---

From: "James Seng" <[email-masked]>
Date: Mon, 14 Oct 2002 18:52:00 +0800

> Then, you should not convert ACE-form iDN int  UTF8 and render it for
users.
> Doesn't this Looks absurd ?  You claims seem against IDNA drafts.
> My current argument is based on IDNA draft section 6.

Section 6.1 describe how "Entry and display in application". It makes no
mention of UTF-8 as can read. Which paragraph are you referring to?

> As one security programmer, i think the above issue is very important.
> James, you begin to be nervous, typical of you. Be comfortable and easy.
> The audiences are knowledgable,experienced and prudent enough.

What makes you think I am nervous? Obviously you didnt catch the sacarism in
FUD...

-James Seng

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 19:49:55 +0900

>no-man-zone-use-it-at-your-own-risk.
Then, you should not convert ACE-form iDN int  UTF8 and render it for users.
Doesn't this Looks absurd ?  You claims seem against IDNA drafts.
My current argument is based on IDNA draft section 6.

>>Rather, that may span into all application protocols that use domain
>>    
>>
>names,
>  
>
>>like future ESMTP.
>>    
>>
>
Home-made or legacy protocols outnumbers IETF protocols.
Moreoever, when even IETFers begin to update some protocols to use
utf8 labels under IETF umbrella, they will be stuck at this stone:  utf8 
label length.
They will find an utf8 label may have 168 octets, contrary to RFC1035.
When IDNA draft granted utf8 label use in application protocols, it is 
natural that
it should have also specified utf8 label length restrictions.

>>To avoid  buffer overflow errors and security breaches, label length
>>    
>>
>issues
>  
>

Soobok Lee

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 20:03:16 +0900

>mention of UTF-8 as can read. Which paragraph are you referring to?
Look into Section 6.1 and the IDNA architecture figure, please.
The authors themselves may clarifiy the reading of the section,
better than you James and me.

============================

(snip)

In any place where a protocol or document format allows transmission of
the characters in internationalized labels, internationalized labels
SHOULD be transmitted using whatever character encoding and escape
mechanism that the protocol or document format uses at that place

---

From: "James Seng" <[email-masked]>
Date: Mon, 14 Oct 2002 19:00:03 +0800

The length restriction in IDN label is confusing comes from a confused
individual. The specification is quite clear that the only restriction is
based on the octet length of the ToASCII string.

So unless there is others who wish to continue this topic, lets put this
thread to a rest.

-James Seng

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 22:10:08 +0900

>	63 octets.  YMMV at levels other than the DNS.
Sure.
If the limit may vary at levels (applications)other than the DNS, the
utf8 labels may exceed 63 octets
in appliction protocol formats (not for display) and the implementors
should reserve enough buffer
spaces for ToUnicoded(ACE) utf8 labels. This really matters because many
programmers favor
utf8 as internal representation format of unicode strings for its ascii
compatibility.

Most applications programmer have been reserving 256 bytes for any LDH
FQDN buffer space .
But that convention should be changed to cover the cases of long utf8
IDN FQDN which may be
3 or 4 times longer than 256 octets. So, 1024 or 768 bytes are good. But
those utf8 FQDN cannot
be put into single UDP packet of DNS response/query. This will constrain
future DNS protocol
update efforts around utf8 supports in wire format. TOday's long iDNs
may be one of the obstacles
in the way to the effort.

If this warning is neglected by application programmers,
some remote malicious crackers will send to users' applications long ACE
IDNs manufactured to
cause buffer overflow errors when toUnicoded and seaze control of the
machine.

Soobok Lee

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Mon, 14 Oct 2002 06:47:33 -0700

>Most applications programmer have been reserving 256 bytes for any LDH

It is amazingly arrogant for anyone to make statements about "most 
applications programmer".

>3 or 4 times longer than 256 octets.

Why just UTF8? Why not UTF16? Or GB? Or ... ?

>cause buffer overflow errors when toUnicoded and seaze control of the

Oh, come on. Step 6 of ToUnicode is exactly two words long. Which one 
of those two words do you think that other applications programmers 
will not understand?

--Paul Hoffman, Director

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 14 Oct 2002 15:49:20 +0200 (CEST)

> of octets per code point, so you can allocate your buffers based on

63 code points is presumably a conservative number. Given the 4 octet ACE 
prefix you can only fit a 59 octets worth of punycode output per label, hence
presumably 59 code points is a tighter limit for non-ASCII internationalized 
labels while 63 code points is the limit for ASCII labels.

  Erik

---

From: Soobok Lee <[email-masked]>
Date: Mon, 14 Oct 2002 23:02:44 +0900

> applications programmer".

I accepted. :-)

> Why just UTF8? Why not UTF16? Or GB? Or ... ?

I already mentioned other encodings in the early postings.

> of those two words do you think that other applications programmers 

"6. Apply ToASCII  "
is   for verification purpose and it won't change that situation.
Step 8 of ToUnicode function which may have enough buffer space  itself
will return  long unicode string result to  cal;ling applications that 
may have shorter buffer space.

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 14 Oct 2002 16:29:11 +0200 (CEST)

> Would  Area Directors make comment on this issue, Erik ?

When and if there is a need to carry IDNs in the DNS protocol using octet
values above 127 the DNSEXT WG would be the place to discuss this.
However, I still haven't seen a fully worked out approach for doing this
that takes DNSSEC into account. (draft-hall-dm-idns is the clostest I've seen,
but it doesn't get into DNSSEC issues).

When and if this is approach, there are presumably multiple encodings
that can be used. One is UTF-8; another possibility would be to
define a bootstring algorithm from Unicode to octets in the range 0-255.
The latter would carry the bootstring compression features forward into
these encodings.

Taking length limits into account would be one part of that work.

> elements. This will clairfy my point.

See above.

But perhaps IDNA should be more clear that there is a length issue for
such future work. For instance, would adding a sentence at the end of
this paragraph be helpful?
	If a signaling system which makes negotiation possible between old and
	new DNS clients and servers is standardized in the future, the encoding
	of the query in the DNS protocol itself can be changed from ACE to
	something else, such as UTF-8. The question whether or not this should
	be used is, however, a separate problem and is not discussed in this
	memo.
E.g. "Such work would have to take into account the length limit for IDNs
as specified in this document".

It might also be useful to make the IDNA specification more clear about
the length limit and its implications somewhere in the document.
Strawman:
	The length limit for internationalized labels is the limit
	of at most 63 octets of output from ToASCII. The design
	of the Punycode encoding is such that some sequences of up to 63 Unicode
	code points result in ToASCII output of 63 or less octets, but
	all possible Unicode strings of 64 code points result in ToASCII
	output of 64 or more octets.

	Implementation Note:
		Depending on the local encoding the software uses for
		Unicode this can correspond to a lot more than 63 octets.
		Similarely, the total length of a IDN expressed in the local
		encoding can significantly exceed 255 octets.

  Erik

---

From: [email-masked]
Date: Tue, 15 Oct 2002 07:35:38 +1000

> Most applications programmer have been reserving 256 bytes for any LDH

	Most applications will use NS_MAXDNAME or MAXDNAME to store
	a domain name today as that is enough to cover all the
	potential expansions from wire to RFC 1034 presentation
	format.  Have a look at NS_MAXDNAME in <arpa/nameser.h>.
	Unless you have a really old OS it will be at least (250*4
	+ 5) (minimum number of label that can appear in a full
	length domain name is 4 labels + root, all 4 periods +
	null).

	As I said what you are talking about is NOT new.  These
	issues have existed for years.

> in the way to the effort.

	That was one of the reasons why raw utf8 was rejected for the wire,
 
> cause buffer overflow errors when toUnicoded and seaze control of the

	FUD.

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 14 Oct 2002 22:22:17 +0000

Erik Nordmark <[email-masked]> wrote:

> > an internationalized label can represent at most 63 code points,
> > whether it's ACE or not.  A given encoding uses a bounded number of
> > octets per code point, so you can allocate your buffers based on
> > that.
>
> 63 code points is presumably a conservative number.  Given the 4 octet
> ACE prefix you can only fit a 59 octets worth of punycode output
> per label, hence presumably 59 code points is a tighter limit for
> non-ASCII internationalized labels while 63 code points is the limit
> for ASCII labels.

True, but which limit you care about depends on the encoding.  For
example, if you're using UTF-32, then a regular ASCII label can have 63
code points each occupying 4 octets.

Soobok Lee <[email-masked]> wrote:

> IDNA section 6.1 goes further than that by allowing _protocols_ to use
> non-ACE labels which are not presentation forms nor textual labels,
> but protocol elements.  What if future ESMTP allows utf8 encodings in
> RCPT: headers ?

Then applications that implement future ESMTP will need to be prepared
for UTF-8 labels to contain more than 63 octets.  This is not a
problem, because any application that can even think about using
non-ASCII labels is aware of IDNA, and therefore knows the definition of
internationalized label, and therefore knows that the maximum possible
label length depends on the encoding used.

> They will find an utf8 label may have 168 octets, contrary to RFC1035.

There is no contradiction.  RFC 1035 says nothing about UTF-8 labels.
The RFC 1035 limit of 63 octets per label applies to the universe of
labels that RFC 1035 defined.  IDNA defines some new labels outside that
universe (each of which is equivalent to a label inside that universe,
for backward compatibility).  If you want to know the maximum possible
length of these new labels that were created by IDNA, don't bother
looking at RFC 1035, because it can't possibly tell you, because it
doesn't even know about the new labels.  Look at IDNA, which contains
the complete definition of internationalized label.

> When IDNA draft granted utf8 label use in application protocols,
> it is natural that it should have also specified utf8 label length
> restrictions.

It did, by defining internationalized label as anything that ToASCII
can be applied to without failing.  From this you can easily conclude
that internationalized labels, when encoded in UTF-8, can exceed 63
octets, but cannot exceed 63*4 octets.  A tight upper bound is trickier
to figure out, but you don't need it in practice.

> So, 1024 or 768 bytes are good.  But those utf8 FQDN cannot be put
> into single UDP packet of DNS response/query.  This will constrain
> future DNS protocol update efforts around utf8 supports in wire
> format.  Today's long iDNs may be one of the obstacles in the way to
> the effort.

That will indeed be an issue that any UTF-8 DNS protocol update will
need to address.

AMC

---

From: Soobok Lee <[email-masked]>
Date: Tue, 15 Oct 2002 10:07:25 +0900

>	length domain name is 4 labels + root, all 4 periods +
If you have a linux (Redhat) box, "chdir /usr/include; grep MAXHOST */*".
Then you will get the enclosed header lists.
Many applications headers use 64/65/257 except for resolver/network
libraries headers which
use 1024/1025 for FQDN. If you have openssh 3.4p1 sources, you can find
that it uses MAXHOSTNAMELEN (64) in
some place in addition to 1025 in other places. of course, 64 is for a
hostname label length.
See rpc/* headers.

asm/param.h:#define MAXHOSTNAMELEN 64 /* max length of hostname */
imap/mail.h:#define NETMAXHOST 65
imap/mail.h: char host[NETMAXHOST]; /* host name (may be canonicalized) */
imap/mail.h: char orighost[NETMAXHOST]; /* host name before
canonicalization */
lwres/netdb.h:#undef NI_MAXHOST
lwres/netdb.h:#define NI_MAXHOST 1025
mozilla/plresolv.h:#define PL_RESOLVE_MAXHOSTENTBUF 1024
protocols/timed.h: char tsp_name[MAXHOSTNAMELEN];
rpcsvc/rusers.h:#define RUSERS_MAXHOSTLEN 257
rpcsvc/rusers.x:const RUSERS_MAXHOSTLEN = 257;
rpcsvc/rusers.x: string ut_host<RUSERS_MAXHOSTLEN>; /* host user logged
on from */
rpc/types.h:#ifndef MAXHOSTNAMELEN
rpc/types.h:#define MAXHOSTNAMELEN 64
w3c-libwww/wwwsys.h:#ifndef MAXHOSTNAMELEN
w3c-libwww/wwwsys.h:#define MAXHOSTNAMELEN 64 /* Any better ideas? */
X11/Xos_r.h: char h_name[MAXHOSTNAMELEN];

>	issues have existed for years.
yes, for presentation formats, it is not new. But for protocol elements,
it is new. isn't it ?

Soobok Lee

---

From: [email-masked]
Date: Tue, 15 Oct 2002 12:23:21 +1000

> some place in addition to 1025 in other places. of course, 64 is for a

	Repeat after me: "HOSTNAME != DOMAIN NAME".

	Hostnames are a subset of domain names (ignoring hostnames
	that are larger that 253 and hence not supported by the
	DNS).  If you are only dealing with hostnames then you
	should be rejecting domain names that are not legal hostnames.
	Anything reading unsanitized domainnames has to expect a
	strings bigger than 255 when converted to RFC 1034 presentation
	format.

	Some implementations of gethostbyaddr() do this sanitization
	for you.  This was also one of the reasons IDNs are converted
	to and from ACSII.  ToUnicode should be perform outside of
	gethostbyaddr().  Moving it inside of gethostbyaddr() is a API
	change.

	Mark

---

From: Soobok Lee <[email-masked]>
Date: Tue, 15 Oct 2002 12:18:09 +0900

>	Repeat after me: "HOSTNAME != DOMAIN NAME".
Everyone knows that here. :-)
Applications , in general, deal with hostnames (smtp?), not abstact
domain names (nslookup,dig?).
It has been clear that i am saying in the context of appliations's
hostname label buffer space problems.

Yes.

>	strings bigger than 255 when converted to RFC 1034 presentation
You repeated this twice. And i had agreed.

True.

Soobok Lee

---

From: Soobok Lee <[email-masked]>
Date: 15 Oct 2002 14:44:22 +0900

> 		encoding can significantly exceed 255 octets.

This clarification will really help.  Thanks.

Soobok Lee

---

From: Soobok Lee <[email-masked]>
Date: 15 Oct 2002 15:10:45 +0900

> code points each occupying 4 octets.

YES.

> for UTF-8 labels to contain more than 63 octets.  This is not a

My focus was on whether the "UTF-8 labels in ESMTP sessions" are
legitimate internationalized hostnames (labels) or not  at the future
and at the present time, since IDNA section 6.1 allows utf8 encodings
of transmitted labels. Does this section seems to propose changes in
hostname rules ? But, you and other authors denied that. Authors's real
intent alway wins over other's interpretations (mine) in the draft. ;-)
Don't you think you authors be more clear about that section?

> label length depends on the encoding used.

But label length restrictions are not added per application prptocol
basis, rather they should be added at lower level like the DNS (and so
IDNA )
as has been done at RFC1035.  How do you think about that?

> the complete definition of internationalized label.

See My first above paragraph.

> to figure out, but you don't need it in practice.

I prefer Erik's suggestion for explicit clarifications, rather than
implicit ones.

Sure. 

Regards,

Soobok Lee

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 15 Oct 2002 23:51:54 +0000

Soobok Lee <[email-masked]> wrote:

> hostname rules ?

Nothing in section 6.1 reduces the requirement of section 3 item 2:
Whenever a domain name is put into an IDN-unaware domain name slot, it
MUST contain only ASCII characters.

At the present time, all domain name slots in ESMTP are IDN-unaware
(because they predate IDNA).  Therefore, all domain names put into
those slots must contain only ASCII characters.

Section 6.1 is not aimed primarily at this situation, but it does
have implications for this situation.  If ESMTP were to use EBCDIC or
UTF-16 for some slots (which isn't very likely), then domain names
being put into those slots would contain only ASCII characters (that
is, characters from the ASCII repertoire) as required by section 3
item 2, but would need to be encoded using EBCDIC or UTF-16 rather than
the ASCII encoding.  (The name ASCII is old, from the days when people
weren't so careful to distinguish between repertoires and encodings).

If a future revision/extension of ESMTP defines some IDN-aware slots,
then the implications of section 6.1 become more interesting.  If those
slots use, say, UTF-8, then IDNs containing non-ASCII characters could
be put into those slots, and they would need to be encoded using UTF-8.

> But label length restrictions are not added per application protocol
> IDNA ) as has been done at RFC1035.  How do you think about that?

RFC 1035 defined the length restriction in terms of the form of the
label that is used in DNS protocol messages.  That form must be no more
than 63 octets.

In addition to the DNS protocol, RFC 1035 also defined the master file
format.  The domain names in master files are protocol elements, because
the master file format is explicitly intended to be machine readable.
Labels in master files can be up to 63*4 octets long.  This fact is
never mentioned in RFC 1035, it is left as an exercise for the reader.

In a completely analogous fashion, IDNA defines the length restriction
of internationalized labels in terms of the ASCII form.  That form must
contain no more than 63 code points.  The length of labels in other
forms is currently left as an exercise for the reader, though perhaps
we will want to add a hint that there can never be more than 63 Unicode
code points in an internationalized label.  IDNA cannot specify the
length restriction in terms of octets, because that depends on the
encoding used, and IDNA does not favor any encoding over any other
encoding.

AMC