Subject: RE: [idn] I-D ACTION:draft-ietf-idn-nameprep-07.txt

From: Yves Arrouye <[email-masked]>
Date: Wed, 6 Feb 2002 17:07:07 -0800

Is it just me (having just browsed through that), or is the new prohibited
table looking strange? It starts with:

0000-0020
007F
0080-009F

so a bunch of ASCII characters are not prohibited anymore. For instance, the
name $$$.com is now allowed. It may be that one expects IDN applications to
do more checks than just Nameprep (for example, check for non-ASCII
characters first), but then why bother with the 0000-0020 and 007F, right?
Anyway, with this, a simple implementation of IDN that relies on the
Nameprep tables only to determine whether a label can be encoded or not,
will allow $$$.com, which isn't changed at all by Punycode.

YA

gabier% ./npdiff -u --name nameprep-06 --name nameprep-07
--- /tmp/npdiff.1716.1  Wed Feb  6 13:37:54 2002
+++ /tmp/npdiff.1716.2  Wed Feb  6 13:37:55 2002
@@ -1367,11 +1367,8 @@
 ----- End Mapping Table -----
 
 ----- Start Prohibited Table -----
-0000-002C
-002E-002F
-003A-0040
-005B-0060
-007B-00A0
+0000-0020
+007F-00A0
 070F
 1680
 180E
gabier% 

In the following, all the characters starting with a - have been removed
from the prohibited table, and are thus allowed by Nameprep.

gabier% ./npdiff -u --name nameprep-06 --name nameprep-07 --char-names
--- /tmp/npdiff.1784.1  Wed Feb  6 13:39:12 2002
+++ /tmp/npdiff.1784.2  Wed Feb  6 13:39:23 2002
@@ -1400,37 +1400,6 @@
 001E 
 001F 
 0020 SPACE
-0021 EXCLAMATION MARK
-0022 QUOTATION MARK
-0023 NUMBER SIGN
-0024 DOLLAR SIGN
-0025 PERCENT SIGN
-0026 AMPERSAND
-0027 APOSTROPHE
-0028 LEFT PARENTHESIS
-0029 RIGHT PARENTHESIS
-002A ASTERISK
-002B PLUS SIGN
-002C COMMA
-002E FULL STOP
-002F SOLIDUS
-003A COLON
-003B SEMICOLON
-003C LESS-THAN SIGN
-003D EQUALS SIGN
-003E GREATER-THAN SIGN
-003F QUESTION MARK
-0040 COMMERCIAL AT
-005B LEFT SQUARE BRACKET
-005C REVERSE SOLIDUS
-005D RIGHT SQUARE BRACKET
-005E CIRCUMFLEX ACCENT
-005F LOW LINE
-0060 GRAVE ACCENT
-007B LEFT CURLY BRACKET
-007C VERTICAL LINE
-007D RIGHT CURLY BRACKET
-007E TILDE
 007F 
 0080 
 0081 
gabier%

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Wed, 6 Feb 2002 17:30:10 -0800

At 5:07 PM -0800 2/6/02, Yves Arrouye wrote:
>so a bunch of ASCII characters are not prohibited anymore.

It's not just you. They are prohibited in IDNA for other reasons, so 
we took them out of nameprep because the prohibition was redundant.

>  For instance, the
>name $$$.com is now allowed.

No, nor has it ever been. nameprep is called by IDNA. If you see a 
way to get "$$$" though IDNA, but all means let us know, because it 
should not.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 7 Feb 2002 03:01:26 +0000

Yves Arrouye <[email-masked]> wrote:

> a bunch of ASCII characters are not prohibited anymore.

Yep.

> It may be that one expects IDN applications to do more checks than
> just Nameprep

Indeed.  Please look at the IDNA draft.  Nameprep is just one of a whole
list of steps to be performed.

> why bother with the 0000-0020 and 007F

I've wondered about that myself.  The nameprep draft gives the reasons
for all the prohibited code points.  I'm not sure how convincing all the
reasons are, but you can look at them and judge for yourself.

The reason some printable ASCII characters were prohibited in earlier
versions of nameprep is that they are prohibited in host names, but
that was not a good reason, because we want nameprep to be usable not
only for host names, but for all domain names, including names like
_ldap._tcp.foo.net (see RFC 2782 about SRV records).

The special restrictions on host names (not only the LDH restriction,
but also the leading/trailing hyphen restriction) are now checked in
ToASCII in the IDNA spec, not in nameprep.

For convenience, here are steps 2 and 3 of ToASCII:

    2. Perform the steps specified in [NAMEPREP].

    3. Host-specific restrictions: If the label is part of a host name
       (or is subject to host name syntax rules) then perform these
       checks:

         * Verify the absence of non-LDH ASCII code points; that is, the
           absence of 0..2C, 2E..2F, 3A..40, 5B..60, and 7B..7F.

         * Verify the absence of leading and trailing hyphen-minus; that
           is, the absence of U+002D at the beginning and end of the
           sequence.

Paul Hoffman / IMC <[email-masked]> wrote:

> If you see a way to get "$$$" though IDNA, but all means let us know,
> because it should not.

If an application is using the domain name $$$.com and knows that
it's not a host name and not subject to the host name syntax rules (I
can think of no example scenarios today, but maybe someday this will
happen), then $$$ will indeed pass through ToASCII successfully (and
unchanged), as it should.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 06 Feb 2002 21:46:06 -0600

> reasons are, but you can look at them and judge for yourself.

I haven't yet had a chance to go through the latest drafts (other
deadlines) so this is coming from a slightly ignorant perspective, but I
would ask that the IDNA team re-evaluate their position on this matter.

First of all, those code-points should not be universally prohibited,
since they are absolutely legitimate in STD13 domain names. ANY octet
value is allowed. If you are going to be moving SOME of the prohibited
characters from nameprep to the IDNA hostname processing, then you need to
move ALL of them at that stage.

However, this is not the approach I would recommend. Instead, I would
recommend that a series of nameprep-like documents be prepared under the
stringprep umbrella, with one document for each of the known data-types.
For example, there should be a document for i18n hostnames, there should
be another document for STD13 hostnames (to act as an IDNA output filter),
another document for current RFC 2822 email addresses, and so forth. Each
of these documents can then be used as data-type inputs. Or to put it in
another perspective, each of them can be treated somewhat like Unicode
"property" tables, with certain types of output only being allowed to
contain characters from the relevant property.

I'm sure that somebody will say something about this being too late, but
there you have it. At the very least, you should consolidate the
prohibited characters into IDNA, as the prohibited characters which appear
to be in nameprep are in fact valid for STD13 domain names.

I will try to read the last-call drafts and make more informed commentary
tomorrow or Friday.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 7 Feb 2002 04:37:07 +0000

"Eric A. Hall" <[email-masked]> wrote:

> If you are going to be moving SOME of the prohibited characters from
> nameprep to the IDNA hostname processing, then you need to move ALL of
> them at that stage.

There are some code points that should be prohibited in all
internationalized textual domain names.  The private use code points,
noncharacter code points, surrogate codes, and left-to-right mark are
some of the best examples of such code points.  These prohibitions do
indeed belong in nameprep.

There are some code points that are prohibited in host names, but not in
all textual domain names.  The underscore is the best example.  These
prohibitions belong in ToASCII.

It's not always clear which side of the line particular code points fall
on.  The least clear-cut are the ASCII prohibitions: 0..20 and 7F.  Feel
free to offer some arguments.

> At the very least, you should consolidate the prohibited characters
> into IDNA, as the prohibited characters which appear to be in nameprep
> are in fact valid for STD13 domain names.

In the broadest sense a STD13 domain name can contain arbitrary binary
data.  Nameprep is not intended for domain names in this broadest sense.
It is intended for domain names composed of internationalized *text*.
It is appropriate for nameprep to prohibit things that are difficult to
interpret as text.

Therefore, the prohibition of ASCII control characters doesn't worry me
much.  The prohibition of ASCII space worries me a little.  Notice that
nameprep doesn't prohibit dots in domain labels, even though dots are
usually used to delimit labels, are are therefore tricky to put into
labels.  In fact RFC 1035 shows how to get dots into labels for the
purpose of representing email addresses that contain dots in the local
part.  If nameprep doesn't prohibit dots, why should it prohibit spaces,
which are also allowed in email address local parts?

I support the prohibition of all other whitespace characters, because
it would be nasty to distinguish between different kinds of whitespace,
but I'm not so confident about the prohibition of ASCII space.  That one
could stand some more scrutiny.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 7 Feb 2002 00:27:42 -0600

"Adam M. Costello" wrote:

> There are some code points that are prohibited in host names, but not
> in all textual domain names.

These include the octet values to which I was referring. Jut as a domain
name can include underscore and period, it can also include the
codepoint values for NUL, BS, CRLF, or any other value in the range of
0x00-0xFF.

> could stand some more scrutiny.

Unless you are planning to update STD13, all octet values need to be
passed
for non-hostname domain names.

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 7 Feb 2002 22:27:03 +0000

"Eric A. Hall" <[email-masked]> replied:

> These include the octet values to which I was referring.  Just as a
> domain name can include underscore and period, it can also include the
> passed for non-hostname domain names.

Let's not confuse code points with octets.  Nameprep does not input or
output octets.  It inputs code points and outputs code points.

STD13 says that domain labels, in the broadest sense, are sequences
of octets, each of which can have any of the 256 possible values for
octets.  Domain labels in this broadest sense may or may not be text;
the octets can, in theory, be used to represent anything.

Depending on what the domain labels are being used for, some of those
256 possible octet values might not get used, even though STD13 allows
them.  For example, when domain labels are used as host labels, only 63
of the possible 256 values get used (the ones corresponding to the LDH
characters).

Similarly, when domain labels are used as IDNA internationalized text
labels, not all 256 octet values get used.  Only the values that can
be output by ToASCII get used.  Clearly, the values 80..FF can never
be output by ToASCII.  Furthermore, because of the prohibitions in
nameprep, the values 7F and 0..20 also cannot be output by ToASCII.

Even though ToASCII never outputs the values 80..FF which are allowed
by STD13, that obviously doesn't mean that ToASCII is broken.  Avoiding
80..FF is the entire point of ToASCII!  Similarly, even though ToASCII
never outputs the values 7F or 0..20 which are allowed by STD13, that
doesn't mean ToASCII (or nameprep) is broken.  IDNA simply doesn't need
those values in order to represent the things it's designed to represent
(internationalized text labels).

If you want to argue that nameprep shouldn't prohibit some of the
code points it's currently prohibiting, you'll need to come up with a
different argument.  Invoking STD13 doesn't help.

AMC