Subject: [idn] ToUnicode output can be longer than input

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 24 Apr 2003 20:45:53 +0000

The IDNA spec contains an incidental statement that was intended to be
helpful, in section 4.2:

    The ToUnicode output never contains more code points than its input.

Oops, that's not true, because Nameprep can cause strings to expand.
For example, consider the input:

x n - - fi fi - a ffl u e n t - s o u ffl - viii - u i c

The spaces are not really there, they just indicate the clusters, which
represent single code points (ligatures and roman numerals: U+FB01,
U+FB04, U+2177).  That's 24 code points.

ToUnicode would apply Nameprep (which expands the ligatures and roman
numerals to their ASCII equivalents), then apply the Punycode decoder,
yielding:

fifi-affluent-soufflé-viii

(For the Latin-1 impaired, the non-ASCII character is e with an acute
accent.)  That's 26 code points.  26 > 24.

So the statement needs to be removed or altered if/when the RFC is
revised.  It would be correct to say that the Punycode decoder cannot
output more code points than it inputs, but Nameprep can, and therefore
ToUnicode can.

AMC

---

From: "Edmon Chung" <[email-masked]>
Date: Fri, 25 Apr 2003 09:20:43 -0400

Hi Adam,

----- Original Message -----
From: "Adam M. Costello" <[email-masked]>
> U+FB04, U+2177).  That's 24 code points.

If I counted it correctly, there are 33 "codepoints" in the above ACE
string. (I agree with your assessment however, please see below, but the
example doesnt seem to illustrate your point...)

> Oops, that's not true, because Nameprep can cause strings to expand.

I can understand this possibility.
Basically, if the length of the Unicode composition for one or more
characters in the string is longer than the ACE composition and the total
excess for all the characters within the string is more than 4 (compensating
the "xn--"), then the ToUnicode output will be longer than the input.

> output more code points than it inputs, but Nameprep can, and therefore

Seems reasonable.

Edmon

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 25 Apr 2003 19:30:41 +0000

Edmon Chung <[email-masked]> wrote:

> string.

fi represents one code point (U+FB01), ffl represents one code point
(U+FB04), and viii represents one code point (U+2177).  Now if you count
again, you should count 24.  I'm trying to describe a non-ASCII ACE
string containing 24 code points, some of which are ASCII and some of
which are compatibility characters.

AMC

---

From: Erik Nordmark <[email-masked]>
Date: Sun, 27 Apr 2003 01:21:25 +0200 (CEST)

> output more code points than it inputs, but Nameprep can, and therefore

The RFC editor maintains an errata page. It would make sense to send them
email asking them to add this to their page.

  Erik