Subject: [idn] comments on IDNA-04

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 20 Nov 2001 14:33:39 +0800

Sorry for duplicate: Previous one got send out accidently while still
editing.

-James Seng

---
The ToASCII operation takes a sequence of Unicode code points and
transforms it into a sequence of code points in the ASCII range (0..7F).
The original sequence and the resulting sequence are equivalent host
labels.

The ToUnicode operation takes a sequence of Unicode code points and
returns a sequence of Unicode code points. If the input sequence is a
host label in ACE form, then the result is an equivalent host label
that is not in ACE form, otherwise the original sequence is returned
unaltered.

JS>> Suggest to define ASCII Compatible Encoding (ACE) before using it.
then s/ToASCII/ToACE/.

ToACE operation takes a sequence of Unicode code points and transform it
into an ACE.

ToUnicode operation takes an ACE and return a sequence of Unicode code
points.

---
ToASCII consists of the following steps:

    1. If all code points in the sequence are in the ASCII range (0..7F)
       then skip to step 3.

JS>> Step 1 seem to be optimization, but not a required step.

---
The ACE prefix, used in the conversion operations (section 4), will
be specified in a future revision of this document. It will be two
alphanumeric ASCII characters followed by two hyphen-minuses. It MUST
be recognized in a case-insensitive manner.

JS>> Suggest s/ACE prefix/ACE tag/. An "ACE tag" could be a uniquely
defined prefix and/or suffix defined by IANA and not neccessary in the
form of xx--.

---
Internationalized host name data in zone files (as specified by section
5 of RFC 1035) MUST be processed with ToASCII before it is entered in
the zone files.

JS>> Dont think we should make a mistake to define zone file format. No
reasons Zone file cant be in UTF-8 so long ToASCII/ToACE is applied
before using it.

---
It is imperative that there be only one ASCII encoding for a particular
host name. ACE is an encoding for host name labels that use non-ASCII
characters. Thus, a primary master name server MUST NOT contain an
ACE-encoded label that decodes to an ASCII label. The ToASCII operation
assures that no such names are ever output from the operation.

JS>> An ACE label that decodes to ASCII label should be defination an
invalid ACE. You dont want this to happen in any place where ACE is
used, not just primary master name server.

---

From: David Hopwood <[email-masked]>
Date: Tue, 20 Nov 2001 11:01:42 +0000

-----BEGIN PGP SIGNED MESSAGE-----

[quoting fixed]

James Seng/Personal wrote:
> draft-ietf-idn-idna-04.txt:
>
> # The ToASCII operation takes a sequence of Unicode code points and
> # transforms it into a sequence of code points in the ASCII range (0..7F).
> # The original sequence and the resulting sequence are equivalent host
> # labels.
> 
> # The ToUnicode operation takes a sequence of Unicode code points and
> # returns a sequence of Unicode code points. If the input sequence is a
> # host label in ACE form, then the result is an equivalent host label
> # that is not in ACE form, otherwise the original sequence is returned
> # unaltered.
> 
> Suggest to define ASCII Compatible Encoding (ACE) before using it.
> then s/ToASCII/ToACE/.

ToASCII doesn't convert to ACE. It converts to ASCII (in particular,
if the string contains only ASCII characters, the result will not be ACE).

> # ToASCII consists of the following steps:
> #
> #    1. If all code points in the sequence are in the ASCII range (0..7F)
> #       then skip to step 3.
> 
> Step 1 seem to be optimization, but not a required step.

It is required.

> defined prefix and/or suffix defined by IANA and not neccessary in the

Is there any particular reason to reserve the possibility of this being
a suffix? The space of names with "xx--" prefixes has already been reserved,
in the sense of warning registrars that it may be used for IDN.

(Incidentally, anyone know what happened to the process of reporting on
whether any such names had already been registered, as required by
<http://www.rfc-editor.org/internet-drafts/draft-ietf-idn-aceid-02.txt>?)

The only minor issue I can think of relating to this choice, is that
using a prefix means that an ACE label can end with a digit, and using
a suffix means that it can begin with a digit. That might be relevant
for top-level IDN labels, since some existing software might only check
whether the last character is a digit in order to distinguish a domain name
from a dotted-decimal IPv4 address. (The correct test, implied but not
explicitly stated in RFC 1035, is to check whether the top label consists
*only* of ASCII digits.)

> # Internationalized host name data in zone files (as specified by section
> # 5 of RFC 1035) MUST be processed with ToASCII before it is entered in
> # the zone files.
> 
> Dont think we should make a mistake to define zone file format. No

I don't think this is a good way of handling zone files, but I do think
that the zone file format is entirely within the scope of an IDN proposal
(as a format for zone transfers and for interoperability between different
server implementations, not as the required representation of a zone).
This is how IDNA-04 handles it - by requiring ACE in zone files.

> # It is imperative that there be only one ASCII encoding for a particular
> # host name. ACE is an encoding for host name labels that use non-ASCII
> # characters. Thus, a primary master name server MUST NOT contain an
> # ACE-encoded label that decodes to an ASCII label. The ToASCII operation
> # assures that no such names are ever output from the operation.
> 
> An ACE label that decodes to ASCII label should be defination an

I agree.

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: David Hopwood <[email-masked]>
Date: Tue, 20 Nov 2001 11:24:21 +0000

-----BEGIN PGP SIGNED MESSAGE-----

> explicitly stated in RFC 1035, is to check whether the top label consists

Actually it is stated in RFC 1123, section 2.1.

- --

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 21 Nov 2001 01:24:05 +0800

> if the string contains only ASCII characters, the result will not be
ACE).

It depends how you define ACE right? :-) (I think that was not defined
anywhere yet ever since the term was coined by Dan Occarson.)

I could defined ASCII string as a subset of ACE, therefore, ASCII is
also an ACE.
e.g. An ACE without ACE tag is a valid ACE which decodes to ASCII range

And subsequently, an ACE with an ACE tag that decodes to ASCII is
invalid ACE.

> > Step 1 seem to be optimization, but not a required step.

Why is it required? Step 4 repeat its again. Step 2 [NAMEPREP] only
effect on ASCII input is case folding.

> in the sense of warning registrars that it may be used for IDN.

Is there a reason that it *must* be in the form of prefix "xx--"?

Why not suffix --xx or a "x-" prefix followed by "-x" suffix?

What if we cant find an common "xx--" prefix according to aceid process?

I feel that the selection of a final ACE tag is an IANA decision.
Technically, prefix or suffix, it works as well.

> This is how IDNA-04 handles it - by requiring ACE in zone files.

It is a mistake to standardized zone file format in the first place.

What if I am using a DNS server which do not have a zonefile but it is
stored in database?

-James Seng

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 20 Nov 2001 21:26:17 +0000

First let me point out a typo in IDNA-05: "resolv" should be "resolve".

James Seng/Personal <[email-masked]> wrote:

> An ACE label that decodes to ASCII label should be by definition an
> invalid ACE.

Indeed it is.  In fact, I wouldn't even call it an ACE label, I'd call
it an ASCII label that begins with the ACE prefix but is not an ACE
label.  Another example would be a label that decodes to a Unicode
string that is not nameprepped.

Any existing labels that coincidentally begin with the ACE prefix are
likely to fall into this category (though they might also turn out to be
valid ACE labels).

IDNA-05 says:

> It is imperative that there be only one ASCII encoding for a
> particular host name.  ACE is an encoding for host name labels that
> use non-ASCII characters.  Thus, a primary master name server MUST NOT
> contain an ACE-encoded label that decodes to an ASCII label.

Hmmm, that requirement could (with small probability) cause an existing
all-ASCII name that happen to begin with the ACE prefix to become
forbidden.

This requirement does not follow from the rules in section 3, and I
don't see why it's necessary.  If a master server contains a label X
beginning with the ACE prefix that decodes to an all-ASCII string Y,
then X is not an ACE label.  ToUnicode will not alter X, and ToASCII
will not alter Y, and X and Y will not compare equal.  They are simply
two distinct all-ASCII labels, neither of which is an ACE label (despite
the fact that one of them happens to begin with the ACE prefix).

IDNA-05 goes on to say:

> The ToASCII operation assures that no such names are ever output from
> the operation.

Oops, that is simply not true.  The ToASCII operation can indeed output
such a label; simply feed it such a label as input, and it will return
it unchanged.

Paul, we need to fix this.  I think a fixed version would say something
like this:

    It is imperative that there be only one ASCII encoding for a
    particular host name.  ACE is an encoding for host name labels that
    use non-ASCII characters.  An ACE-encoded label X that decodes to an
    all-ASCII label Y is not an alternate form of Y; in fact, X is not
    the ACE form of anything, and applying ToUnicode to X will not alter
    it.  ToASCII will leave both X and Y unchanged, and hence X and Y
    are not matching labels (by rule 3).

But this revised paragraph no longer contains any requirements, nor
does it particularly related to DNS servers, so maybe it should just be
removed.

James said:

> Don't think we should make a mistake to define zone file format.  No

That's true for private zone files read by DNS server implementations.
But for master files used as an interchange format between DNS servers
there needs to be a standard format, and that's why RFC 1035 defines
one.  These master files are in fact protocol messages, containing
generic domain name slots, and so rule 1 of section 3 applies (ToASCII
must be applied before insertion).

Paul, by the way, IDNA-05 says "zone files (as specified by section 5 of
RFC 1035)", but actually section 5 of RFC 1035 defines "master files",
not "zone files".

> Suggest to define ASCII Compatible Encoding (ACE) before using it.

The first appearance of "ACE" in the draft is the definition of "ACE
label" in the Terminology section.

> I could define ASCII string as a subset of ACE, therefore, ASCII is
> also an ACE. e.g. An ACE without ACE tag is a valid ACE which decodes
> to ASCII range

That's one way to define it, but not the way the IDNA draft defines it.
I think it's more useful to have a term referring to the new non-WYSIWYG
labels being introduced by IDNA, so we use the term "ACE label" for
that.

> s/ToASCII/ToACE/.

As David said, the output of ToASCII is not necessarily an ACE label.
For example, if the input is entirely ASCII, it will be unchanged by
ToASCII.

> JS>> Step 1 seem to be optimization, but not a required step.

It avoids doing nameprep on all-ASCII labels.  This is not only a
performance win, but it also prevents case-folding of all-ASCII
labels.  Existing standards already recommend that case be preserved
for all-ASCII labels.  This recommendation is not being extended to
internationalized labels (because nameprep does case-folding), but if
step 1 were omitted, IDNA-conformant applications would be required
to alter the way they treat all-ASCII labels, contrary to existing
recommendations.

> Suggest s/ACE prefix/ACE tag/.  An "ACE tag" could be a uniquely

It needs to be a prefix, not a suffix, otherwise an ACE label (using
AMC-ACE-Z) could begin with a hyphen.  (The AMC-ACE-Z draft contains an
incorrect statement on this subject, which will be fixed in the next
version.)

I thought there was already consensus on the form xx--, but we could
revisit that question.

David Hopwood <[email-masked]> wrote:

> using a prefix means that an ACE label can end with a digit

AMC-ACE-Z always ends with a letter.  (It could also end with a hyphen
if the input were all-ASCII, but IDNA forbids that.)

AMC

---

From: David Hopwood <[email-masked]>
Date: Tue, 20 Nov 2001 20:07:08 +0000

-----BEGIN PGP SIGNED MESSAGE-----

> > if the string contains only ASCII characters, the result will not be

I should have said: will contain no ACE labels.

> It depends how you define ACE right? :-)

An ACE label means a type 00 domain label that consists of the ACE tag and
an output of the ACE encoding algorithm.

[...]
> > > #    1. If all code points in the sequence are in the ASCII range
> > > #       (0..7F) then skip to step 3.
> > >

Not so: it also disallows some code points (including non-LDH ASCII).
Step 1 definitely changes the algorithm, so it is not an optimisation.

In fact the ToASCII algorithm appears to be incorrect, i.e. not what
was intended, for two reasons:
 - by calling nameprep, it disallows general domain names (i.e. not
   hostnames) that contain both octets >= 0x80 and octets that
   represent non-LDH ASCII.
 - it removes case information before applying ACE encoding - so case
   annotation cannot be done. The only way to specify case annotation
   rigourously is for the case bit of each character to be an output of
   nameprep.

> > Is there any particular reason to reserve the possibility of this
> > being a suffix? The space of names with "xx--" prefixes has already
> > been reserved, in the sense of warning registrars that it may be
> > used for IDN.
> 

No. However, since it is almost completely arbitrary, and several drafts
have used "xx--", why change to something else?

> What if we cant find an common "xx--" prefix according to aceid process?

I think that's unlikely.

> I feel that the selection of a final ACE tag is an IANA decision.

It's pointless for IANA to do this, because all of the current proposals
call for only one ACE. IANA's role is to handle registration where there
is an ongoing requirement to assign "numbers"/identifiers from a defined
set of alternatives. It doesn't choose what the set of alternatives is,
and there is no need to involve it in one-off decisions that can just
as easily be taken by working groups.

> > I don't think this is a good way of handling zone files, but I do
> > think that the zone file format is entirely within the scope of an IDN
> > proposal (as a format for zone transfers and for interoperability
> > between different server implementations, not as the required
> > representation of a zone). This is how IDNA-04 handles it - by
> > requiring ACE in zone files.
> 

I disagree. The actual format is perhaps a little ugly, but there is
nothing wrong with the idea of a standardised text-based format.

> What if I am using a DNS server which do not have a zonefile but it is

That works fine; the zone file format is not a required representation,
and no-one is suggesting that it should be. However, suppose you want to
migrate a zone to a different server implementation: if there is a
standardised zone file format, then you can export your existing database
to that format, then import it to the new database. If there is no
standardised format, then you have an n:n database conversion problem.

- --

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 21 Nov 2001 00:48:59 +0000

David Hopwood <[email-masked]> wrote:

> An ACE label means a type 00 domain label that consists of the ACE tag
> and an output of the ACE encoding algorithm.

Not quite.  The simplest fully-precise definition is:  An ACE label is a
label that gets altered when ToUnicode is applied to it.

The definition in the IDNA draft preceeds the definition of ToUnicode,
so it's a little more vague:  An ACE label is a label that contains only
ASCII characters but represents a label containing non-ASCII characters.

Actually, that definition is a tad sloppy regarding equivalence of
Unicode strings.  For example, if X is an all-ASCII ACE label, then the
fullwidth equivalent of X is also an ACE label, even though it's not
all-ASCII.  If Y is a simple all-ASCII label (without the ACE prefix),
and Z is the fullwidth equivalent of Y, then Y might appear to satisfy
the definition (Y is an all-ASCII label representing Z, which contains
non-ASCII characters), but Y is not an ACE label.

So what the definition is trying to say is something like:  An ACE label
is an ASCII (or ASCII-equivalent) label that represents a non-ASCII (and
non-ASCII-equivalent) label.  But ASCII-equivalence would be defined in
terms of nameprep, so we can't very well put that in the Terminology
section.

>    hostnames) that contain both octets >= 0x80 and octets that

Currently, nameprep includes one of the host restrictions (the
prohibition of non-LDH ASCII characters) but not the other one (the
prohibition of leading/trailing hyphens).  I have been arguing for the
removal of the ASCII characters from nameprep's prohibition table.  The
way it is now, if you want to use nameprep with the host restrictions,
it doesn't get the job done, and if you want to use nameprep without
the host restrictions, you can't.  ToASCII provides a place (step 3,
immediately following nameprep) where the full host restrictions can be
applied (if applicable).

> The only way to specify case annotation rigourously is for the case
> bit of each character to be an output of nameprep.

The only way to *implement* mixed-case annotation so that it records
the case of the original string is for the case flags to be output from
the nameprep implementation.  Yes indeed.  There's no way around that,
because nameprep can alter the length of the string.  (Of course an
implementation of nameprep can compute and output extra information
while still conforming to the nameprep spec.)

However, I think mixed-case annotation is *specified* rigorously in the
AMC-ACE-Z spec.  It tells exactly what the annotations mean, in terms
of what they are asking the decoder to do.  There are no requirements
for creating the annotations; they do not necessarily record the
original case; they record recommendations for how the string should be
displayed.  Presumably, in most cases, encoders will want to recommend
that the string to be displayed in a way that looks identical to the
original string.  But that is not required, and no particular method for
creating annotations to accomplish that goal is required.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 20 Nov 2001 19:37:46 -0600

> label that gets altered when ToUnicode is applied to it.

In terms of the specific context of domain names embedded in data streams,
wouldn't the simplest definition be: any label that begins with the ACE
prefix? This definition doesn't care if the label is provided in
application data or in a DNS message or anything else, but it clearly
delineates the label as an ACE encoded label and also contributes to
clearly marking the hostile case of ACE labels that only contain ASCII.

The question of hostile case came up earlier. The problem scenario is
having an app decode and display the name, but where DNS does not. This
allows for a hostile party to provide a link to www.zz--amazon.com which
decodes for display as www.amazon.com on the compliant browser, but where
DNS is sending the victim party to www.zz--amazon.com. The obvious method
for preventing this is to forbid delegations with the prefix. But not all
cases are isolated to public delegations, so all encoding/decoding systems
must reject them equally if the protection is to exist globally. As a
point of reference, my draft went so far (too far?) as to prohibit DNS
errors from being returned with these names.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 21 Nov 2001 01:50:26 +0000

"Eric A. Hall" <[email-masked]> wrote:

> > The simplest fully-precise definition is:  An ACE label is a label
> > that gets altered when ToUnicode is applied to it.
>
> In terms of the specific context of domain names embedded in data
> streams, wouldn't the simplest definition be: any label that begins
> with the ACE prefix?

Sorry, I meant the simplest definition consistent with the usage in
the IDNA draft.  Your proposed definition is simpler, but it defines
a different concept from the one the IDNA draft is trying to define.
I wanted a term for non-WYSIWYG label, that is, a label that doesn't
directly say what it means, and is therefore displayed differently in
IDNA-aware applications versus IDN-unaware applications.  Obviously, the
definition "label that gets altered by ToUnicode" corresponds exactly to
this concept.

> The problem scenario is having an app decode and display the name, but
> where DNS does not.  This allows for a hostile party to provide a link
> to www.zz--amazon.com which decodes for display as www.amazon.com on
> the compliant browser, but where DNS is sending the victim party to
> www.zz--amazon.com.

I don't think there is a problem.  The ToUnicode operation will never
return an all-ASCII label that differs from the input.  If the input
is zz--amazon then the result cannot be amazon, because amazon is
all-ASCII.  ToUnicode never alters invalid ACE labels.  If zz--amazon
is an invalid ACE label, then ToUnicode will leave it unchanged, and
therefore the IDNA-compliant application will display it as zz--amazon.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 20 Nov 2001 20:44:41 -0600

> definition "label that gets altered by ToUnicode" corresponds exactly to

Do you mean the IDN label that results from decoding, versus the STD13
label with a prefix?

> therefore the IDNA-compliant application will display it as zz--amazon.

Okay, so this would only fail if the prefix was stripped before the label
was decoded. I was/am under the impression that general purpose encoding
and decoding libraries would exist to support the use of AMC-Z for non-DNS
purposes since it seems like a good general purpose 7-bit TES, and in
those environments prefix processing would occur separately. If prefix
processing is an embedded part of the conversion process as you describe
it in ToUnicode, the problem is avoided.

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 21 Nov 2001 16:57:01 +0800

> as easily be taken by working groups.

Note:

"A prefix for AMC-Z will be assigned using the normal IESG/IANA/
RFC-editor procedure for coordinating assignment of protocol
parameters during IETF last call and publication as an RFC."

-James Seng

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 21 Nov 2001 20:49:39 +0000

"Eric A. Hall" <[email-masked]> wrote:

> > directly say what it means, and is therefore displayed differently
> > in IDNA-aware applications versus IDN-unaware applications.
> > Obviously, the definition "label that gets altered by ToUnicode"
> > corresponds exactly to this concept.
>

I mean X is an ACE label iff ToUnicode(X) != X.  This implies that for
any ACE label Y, nameprep(Y) is an all-ASCII string beginning with the
ACE prefix.  But not every all-ASCII string beginning with the ACE
prefix is an ACE label.

> I was/am under the impression that general purpose encoding and
> decoding libraries would exist to support the use of AMC-Z for non-DNS
> purposes

That may be true, but regardless of whether an application calls a
single ToUnicode function or calls smaller functions separately, if it
is IDNA-compliant it will either do nothing to the label or it will
perform ToUnicode or it will perform ToASCII, and all of those options
are safe from spoofing problems.

AMC