Subject: Re: [idn] call for comments for REORDERING

From: Kenneth Whistler <[email-masked]>
Date: Thu, 18 Oct 2001 20:28:24 -0700 (PDT)

James Seng said:

> Third, I would really prefer to reference a work from established expert
> group if possible. For example, ISO/IEC JTC1/SC22/WG20 publishes ISO
> 14651 on weighted sorting. I am not sure how ISO 14651 would perform for
> the IDN purpose but I thought it might be worthwhile to examine.

As one of the principal authors of ISO 14651, who has also implemented
the synchronized Unicode Technical Standard #10, the Unicode Collation
Algorithm, I can attest that this is a very tricky and complicated
area, and the algorithms to do all this correctly are not the kind
you can write on the back of a cocktail napkin. It is very complex
to get all the details right and to get good-performing algorithms
(in speed and in resource usage). It is also very difficult for
independent implementations to get themselves all exactly lined
up, and even more difficult for independent implementations to *prove*
that they are getting the same results for all data (as opposed to
a particular result for one set of data -- which is pretty easy).

IDN doesn't need to add this kind of headache to the already
complex enough issues of nameprep.

> 
> Whatever the case, we should make a decision quickly on this. Lets not
> drag this further if possible.
> 
> -James Seng
> 
> ----- Original Message -----
> From: "Martin Duerst" <[email-masked]>

> > So this is a solution in search of a real problem,
> > not worth bothering the whole world with additional
> > complexity.

I heartily concur with Martin's assessment.

--Ken Whistler

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 18 Oct 2001 21:04:45 -0700

Ken is right; collation is *quite* complex. Anyone wanting to see some of
what is involved can look at the ICU implementation (which is UCA and ISO
14651 compliant, and open-source):

User Guide:
http://www-124.ibm.com/icu/userguide/Collate_Intro.html

Internal Design:
http://oss.software.ibm.com/cvs/icu/~checkout~/icuhtml/design/collation/ICU_
collation_design.htm
(source files are linked from there)

ICU home:
http://oss.software.ibm.com/icu/

While the perfomance is good, the code is many, many orders of magnitude
more complicated than the current nameprep. It is not appropriate for IDN.

Mark

—————

Δός μοι ποῦ στῶ, καὶ κινῶ τὴν γῆν — Ἀρχιμήδης
[http://www.macchiato.com]

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 19 Oct 2001 12:20:24 +0800

Thanks Ken & Mark for note. I defer to your opinion in this.

-James Seng

----- Original Message -----
From: "Mark Davis" <[email-masked]>
To: "Kenneth Whistler" <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, October 19, 2001 12:04 PM
Subject: Re: [idn] call for comments for REORDERING

> http://www-124.ibm.com/icu/userguide/Collate_Intro.html
http://oss.software.ibm.com/cvs/icu/~checkout~/icuhtml/design/collation/
ICU_

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 19 Oct 2001 04:33:55 +0000

Mark Davis <[email-masked]> wrote:

> Ken is right; collation is *quite* complex.

I don't doubt that, but I don't see the relevance to the reordering
proposal, which would map each code point to another code point (in a
reversible way) just before ACE encoding, and would apply the reverse
mapping just after ACE decoding.  Complexity is indeed a concern, but
this is nowhere near as complex as collation, because it operates on
individual code points, not strings.

Do the NICs have some numbers for us?  Soobok Lee has used the names in
the VGRS testbed both to train his reordering tables and to evaluate
their effectiveness.  It would be very interesting to see how effective
the latest tables are for names from other registries, like the Japan,
China, and Taiwan NICs.

AMC