Subject: Re: [idn] Debunking the ACE myth

From: Dan Oscarsson <[email-masked]>
Date: Tue, 24 Jul 2001 08:39:06 +0200 (MEST)

Adam M. Costello wrote:

>> 1) All names are transmitted in a form/semantic insensitive format
>> (that is what nameprep is for) so that names can be compared using
>> binary matching.

>> 1) is the IDNA case and also in Microsofts draft-skwan-utf8-dns.  This
>> results in that for example upper case characters cannot be used in
>> responses from DNS.
>
>For the UTF-8 case, as I've pointed out before, it is not necessary
>for DNS queries and DNS responses to use the same rules.  It would
>be perfectly reasonable to use nameprep'd UTF-8 in queries and
>non-nameprep'd UTF-8 in responses.
>
>For the ACE case, as I've pointed out before, mixed-case ACE can easily
>be used to represent mixed-case Unicode.  You would lose things like
>circled-digit-one (which becomes a regular digit-one), but mixed case
>can be conveyed in ACE.
>
>By the way, it would be pointless to combine ACE with option 2 (no
>nameprep).

1) above is used to allow only binary matching to be done on a DNS server.
This means that the client is required to send nameprepped names to
the server, if they are going to match. If we allow a response to
have non-namprepped names (ACE or UTF-8), only a IDN aware client can
use that name to send new queries because only those will do nameprep
before sending the query. So all other clients may fail when using
a name returned from DNS.

You can allow non-nameprepped answers if you require the DNS server
to do nameprep on names in queries (if needed). For those who
think it is too expensive to do nameprep in the server (it is not for
areas like latin, greek or cyrillic) to have most names in queries
being nameprepped might reduce the burden on the server. That depend
on if a already nameprepped name can quickly be checked.

   Dan

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 24 Jul 2001 06:58:13 +0000

Dan Oscarsson <[email-masked]> wrote:

> If we allow a response to have non-namprepped names (ACE or UTF-8),
> only a IDN aware client can use that name to send new queries because
> only those will do nameprep before sending the query. So all other
> clients may fail when using a name returned from DNS.

Non-nameprep'd ACE is pointless, so forget about that.  You're right
that it would be bad to return non-nameprep'd UTF-8 to IDN-unaware
clients, but I think it would be bad to return any non-ASCII names to
IDN-unaware clients.

> For those who think it is too expensive to do nameprep in the server
> (it is not for areas like latin, greek or cyrillic) to have most names
> in queries being nameprepped might reduce the burden on the server.
> That depend on if a already nameprepped name can quickly be checked.

I suspect that the cost of doing nameprep depends on the length of the
string, but not much on its contents.  I doubt that checking whether a
name is nameprep'd is much cheaper than performing nameprep on it.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 24 Jul 2001 10:28:43 -0500

> on if a already nameprepped name can quickly be checked.

One solution is to have the authoritative servers nameprep on load, such
that they check when they read the database, A/IXFR, DDNS update, etc.

We should also separate the reasons for having nameprep in two places, and
why they are both needed.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: David Hopwood <[email-masked]>
Date: Tue, 24 Jul 2001 19:55:38 +0100

-----BEGIN PGP SIGNED MESSAGE-----

"Adam M. Costello" wrote:
> I doubt that checking whether a name is nameprep'd is much cheaper than
> performing nameprep on it.

It is, actually: see UAX #15 Annexes 3 and 8, at
http://www.unicode.org/unicode/reports/tr15/

(The code in Annex 8 would need to be modified to also check for lowercase.
It also has to be checked that the name is a valid UTF-8 encoding.)

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: David Hopwood <[email-masked]>
Date: Fri, 27 Jul 2001 02:06:35 +0100

-----BEGIN PGP SIGNED MESSAGE-----

> (The code in Annex 8 would need to be modified to also check for lowercase.

Correction: it is slightly more complicated than just checking for lowercase
(but this doesn't change the efficiency).

Let getCanonicalClassX(ch) be the Canonical Combining Class of ch, except that
it returns a negative value if ch \in X. The set X is defined as the union of:

 - characters that are mapped to something else (or mapped out) by the nameprep
   mapping step,
 - characters for which the NFC flag in DerivedNormalizationProperties.txt
   is either NFC_NO or NFC_MAYBE,
 - characters that are prohibited in the output of nameprep (i.e. category D).

[X is a superset of MN union D. It would be interesting to see what kind
of characters, if any, are in X \ (MN union D); I'll try to work that out.]

Then the Java-like pseudocode would look like this:

public String nameprep(String source) {
    short lastCanonicalClass = 0;

    for (int i = 0; i < source.length(); ++i) {
        char ch = source.charAt(i);
        short canonicalClass = getCanonicalClassX(ch);
        if (lastCanonicalClass > canonicalClass && canonicalClass != 0) {
            return fullNameprep(source, i);
        }
        lastCanonicalClass = canonicalClass;
    }
    // The string was already nameprepped, and has no disallowed characters.
    return string;
}

public String fullNameprep(String source, int i) {
    // Full nameprep algorithm (including checking for disallowed characters),
    // where the first character that may need mapping is at index i.
}

> It also has to be checked that the name is a valid UTF-8 encoding.)

This can be done in the same loop.

- --

---

From: David Hopwood <[email-masked]>
Date: Fri, 27 Jul 2001 03:13:00 +0100

-----BEGIN PGP SIGNED MESSAGE-----

>    is either NFC_NO or NFC_MAYBE,

This should be:
   - characters for which the NFKC_QuickCheck flag in
     DerivedNormalizationProperties-3_1_0.txt is either NFKC_NO or NFKC_MAYBE.

- --

---

From: David Hopwood <[email-masked]>
Date: Fri, 27 Jul 2001 06:42:41 +0100

-----BEGIN PGP SIGNED MESSAGE-----

>  - characters that are mapped to something else (or mapped out) by the nameprep
[  - characters for which the NFKC_QuickCheck flag in
     DerivedNormalizationProperties-3.1.0.txt is either NFKC_NO or NFKC_MAYBE.]
>  - characters that are prohibited in the output of nameprep (i.e. category D).

To answer my question about what X \ (MN union D) is:

Let M be the set of characters that are mapped to something else or mapped out.
Then we have the following:
  X = M union NFKC_NO union NFKC_MAYBE union D.
  MN = (M union NFKC_NO) \ D.

Therefore X \ (MN union D) = X \ (M union NFKC_NO union D)
                           = NFKC_MAYBE \ (M union NFKC_NO union D)
                           = NFKC_MAYBE.

NFKC_MAYBE is:

 - a subset of the Combining Diacritical Marks block (0300..0345).
 - some Arabic combining marks (0653..0655).
 - vowel signs and length marks for Devanagari, Bengali, Oriya, Tamil,
   Telugu, Kannada, Malayam, Sinhala, and Myanmar.
 - the combining Katakana-Hiragana voiced and semi-voiced sound marks.
 - a subset of the Hangul Jamo medial vowels (1161..1175).
 - a subset of the Hangul Jamo final consonants (11A8..11C2).

These are the characters that can occur in a nameprepped name, but will
inhibit the optimization if nameprep is applied again. All of them are
combining characters of one type or another, which makes sense because
it is complicated to check that there is no corresponding precomposed
form.

Should the common case optimization be described in the next version of
the nameprep draft?

- --

---

From: Roozbeh Pournader <[email-masked]>
Date: Fri, 27 Jul 2001 17:56:11 +0430 (IRST)

>      DerivedNormalizationProperties-3_1_0.txt is either NFKC_NO or NFKC_MAYBE.

I think refering to Unicode data files other than the originals is a bad
thing. The format will change, the level of information they provide will
change, ...

For example, take a look at what happened to 'PropList.txt' between 3.0.1:

	http://www.unicode.org/Public/3.0-Update1/PropList-3.0.1.txt

and 3.1.0:

	http://www.unicode.org/Public/3.1-Update/PropList-3.1.0.txt

roozbeh

---

From: David Hopwood <[email-masked]>
Date: Sat, 28 Jul 2001 04:13:05 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Roozbeh Pournader wrote:
> thing.

It would not be a normative reference: the information would be used to
calculate the set X, and that set would be given in the nameprep spec.
This causes no problems because each version of nameprep corresponds to a
specific version of Unicode.

> The format will change, the level of information they provide will change,
> ...
> 

The latter is the standard machine readable format used by all the Unicode
files; the former isn't, so it's obvious why the format changed in this case.

- --