Subject: Re: [idn] call for comments for REORDERING

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 19 Oct 2001 12:24:57 +0800

Erm, I mean I concur with your opinion. (s/defer/concur/) Sorry if
anyone misunderstood me.

-James Seng

----- Original Message -----
From: "James Seng/Personal" <[email-masked]>
To: "Mark Davis" <[email-masked]>; "Kenneth Whistler"
<[email-masked]>
Cc: <[email-masked]>
Sent: Friday, October 19, 2001 12:20 PM
Subject: Re: [idn] call for comments for REORDERING

> Thanks Ken & Mark for note. I defer to your opinion in this.
>
> -James Seng
>
> ----- Original Message -----
> From: "Mark Davis" <[email-masked]>
> To: "Kenneth Whistler" <[email-masked]>; <[email-masked]>
> Cc: <[email-masked]>
> Sent: Friday, October 19, 2001 12:04 PM
> Subject: Re: [idn] call for comments for REORDERING
>
>
> > Ken is right; collation is *quite* complex. Anyone wanting to see
some
> of
> > what is involved can look at the ICU implementation (which is UCA
and
> ISO
> > 14651 compliant, and open-source):
> >
> > User Guide:
> > http://www-124.ibm.com/icu/userguide/Collate_Intro.html
> >
> > Internal Design:
> >
>
http://oss.software.ibm.com/cvs/icu/~checkout~/icuhtml/design/collation/
> ICU_
> > collation_design.htm
> > (source files are linked from there)
> >
> > ICU home:
> > http://oss.software.ibm.com/icu/
> >
> > While the perfomance is good, the code is many, many orders of
> magnitude
> > more complicated than the current nameprep. It is not appropriate
for
> IDN.
> >
> > Mark
> >
> > —————
> >
> > Δός μοι ποῦ στῶ, καὶ κινῶ τὴν γῆν — Ἀρχιμήδης
> > [http://www.macchiato.com]
> >
> > ----- Original Message -----
> > From: "Kenneth Whistler" <[email-masked]>
> > To: <[email-masked]>
> > Cc: <[email-masked]>
> > Sent: Thursday, October 18, 2001 8:28 PM
> > Subject: Re: [idn] call for comments for REORDERING
> >
> >
> > > James Seng said:
> > >
> > > > Third, I would really prefer to reference a work from
established
> expert
> > > > group if possible. For example, ISO/IEC JTC1/SC22/WG20 publishes
> ISO
> > > > 14651 on weighted sorting. I am not sure how ISO 14651 would
> perform for
> > > > the IDN purpose but I thought it might be worthwhile to examine.
> > >
> > > As one of the principal authors of ISO 14651, who has also
> implemented
> > > the synchronized Unicode Technical Standard #10, the Unicode
> Collation
> > > Algorithm, I can attest that this is a very tricky and complicated
> > > area, and the algorithms to do all this correctly are not the kind
> > > you can write on the back of a cocktail napkin. It is very complex
> > > to get all the details right and to get good-performing algorithms
> > > (in speed and in resource usage). It is also very difficult for
> > > independent implementations to get themselves all exactly lined
> > > up, and even more difficult for independent implementations to
> *prove*
> > > that they are getting the same results for all data (as opposed to
> > > a particular result for one set of data -- which is pretty easy).
> > >
> > > IDN doesn't need to add this kind of headache to the already
> > > complex enough issues of nameprep.
> > >
> > > >
> > > > Whatever the case, we should make a decision quickly on this.
Lets
> not
> > > > drag this further if possible.
> > > >
> > > > -James Seng
> > > >
> > > > ----- Original Message -----
> > > > From: "Martin Duerst" <[email-masked]>
> > >
> > > > > So this is a solution in search of a real problem,
> > > > > not worth bothering the whole world with additional
> > > > > complexity.
> > >
> > > I heartily concur with Martin's assessment.
> > >
> > > --Ken Whistler
> > >
> > >
> > >
> >
>

---

From: [email-masked]
Date: Fri, 19 Oct 2001 12:09:33 EDT

In various messages, [email-masked] writes:

> The sizes of Latin alphabets or variants do not exceed 30.

Russian Cyrillic has 33.  Many Central European implementations of the Latin 
script, with carons and acute accents and such, use quite a bit more than 30. 
 But this is nit-picking, and Soobok's point about the difference between 
Latin and Hangul is well taken.

>  If we should change the frequency table, new versioning
>  prefix should be introduced to avoid conflicts in the same
>  time when there is need to make major modifications to 
>  NFC/NFKC due to erros in them. 
>  If version 1 has "dq--" prefix, version 2 should have
>  other ones like "xq--". Reordering tables could be
>  improved at that time if needed.

I think Martin has already adequately explained the problematic nature of 
creating different versions of the whole IDN scheme.  Users, of course, will 
not generally understand that there is a nameprep part and a reordering* part 
and an ACE part and that only the reordering part has changed; they will 
simply see that IDN "works" or "doesn't work," and of course they will be 
right.  (*Note: the ordinary English word "reordering" is spelled here in 
normal lowercase letters.)

Requiring everyone to update browsers and server-side software to accommodate 
IDN will be an understandable costs for many.  Creating a new and 
incompatible version of IDN to add reordering for Tagalog -- a really minor 
optimization for a small alphabet, by Soobok's own admission -- will not.

>  > The idea behind this is that if e.g. Tagalog gets added to
>  > Unicode, and the IETF decides to add it to the allowed set
>  > of characters for domain names, then the registries that
>  > want to accept Tagalog have to update their software
>  > immediately (no big deal for them), but deployed software
>  > can use Tagalog without having to change nameprep/ACE
>  > (unless they use characters which have to be normalized,
>  > which may happen but will be rare). So existing clients
>  > will already ACE the Tagalog codepoints without reordering,
>
>  maybe problematic and unsafe. 
>  What if future NFC/NFKC maps them into other code points ?
>  There will be a mess, too.

You don't have to worry about that because the ISO 10646 and Unicode 
committees are firmly committed not to add any new compatibility characters 
that would break normalization in this way.  They can (and will) add Tagalog 
and other scripts, as well as adding new characters for existing scripts, but 
you will never see a precomposed Tagalog ligature that would cause NFC or 
NFKC to have to be updated.  To do so would break existing implementations 
and cause the same kind of havoc that "Reordering v2.0" would cause.

-Doug Ewell
 Fullerton, California

---

From: Karlsson Kent - keka <[email-masked]>
Date: Mon, 22 Oct 2001 23:05:54 +0200

Please distinguish NFKC and NFC.  They are not the same, and have
different properties.  In several places below you have confused them,
and how they operate.

For your first problem it just shows that NFKD and NFKC should be
used with caution. They cannot be applied blindly, especially not
to text in general.  However, I don't see that this problem
(which is quite apparent to anyone reading UAX 15) hinders the use
of NFKC for for the particular application of IDNs.  Your first
problem does not affect NFD or NFC at all.

Your second problem does affect NFC (and thus also NFKC), but does
not affect NFD (fully decomposed).  This problem could, in principle,
have been avoided if there had been no Hangul syllable characters...
So for historic texts (using historic Jamo, or historic (or uncommon)
consonant (or vowel) clusters of otherwise modern Jamo) NFD would probably
be the cleanest approach, especially for rendering such substrings. As
I have read this, it just means that the conjoining Jamo can interact
with the Hangul syllables too (I'm not sure if they actually do in
any (rendering) implementation, nor if that was intended). However,
again I don't see that this would be a problem for the particular
application of IDNs, though I do see a problem for rendering engines.
(Sorting, by the way, when Jamo are supported, should support Hangul
syllables through canonical decomposition.)

The entire intent, IFAIK, with the Hangul syllables was as an
expedient for implementations that do not support Hangul Jamo at all.
For those that do support Hangul Jamo, Hangul syllable characters are
more of a pain than a help, and should decompose them for rendering
and sorting (at least), but need not do so for storage, or for IDNs...

Sorry for not following the IDN debate for a while, but I kind of
gave up when the focus turned solely onto "ACE"s, making the entire
exersice into something that may likely result in something that I
don't wish to see deployed, having very negative experiences with
"Quoted-Printable". And now there is suggestion for 'reordering' too!
This does not look good.  We don't need to have convoluted
'solutions' that are applicable only for IDNs.

		Kind regards
		/kent k

> -----Original Message-----
> From: Soobok Lee [mailto:[email-masked]]
...
> two problems:
> 
>  1) NFKC maps   compatibility hangul jamo (U+32??) into
>                 conjoining hangul jamo without fillers.
>     That converts compat hangul jamo sequences into conjoining
>       jamo sequences and they are combined partially in some cases
>         by the NFC following  compatibilty decomposition.
>     Modern hangul informal texts contains compatibiilty jamos 
> very often.
>     If the texts contains choseong + jungseong sequence in 
> compat jamo,
>        they will be affected by this fault.
>     This fault was found by Soobok Lee, too lately.
> 
>  2) NFC  does not combine  archaic(old) hangul trailing jamo into the
>      preceding half-combined syllable (from choseong jamo + 
> jungseong jamo),
>       but leave it alone as an isolated conjoining jamo.
>     That means archail hangul syllable which has old hangul 
> trailing jamo 
>       cannot be preserved through out NFKC.
>     NFKC cannot be used for scholarly texts which contains 
> archail hangul 
>       syllables.  This fault was found by Martin, too lately.
>     
> Mark and Martin knew and told me why this happened.
> Martin, would you comment on this?
> 
> Soobok Lee
> 
> 
> > 
> > > I hope UTC cures this BUGGY NFC/NFKC jamo handling as an 
> > > exceptional case while it is in  strict adherence to 
> UTR's backward 
> > > compatiblity policy & promise.
> > 
> > The normal forms for strings of already allocated characters
> > will not be changed any more.
> > 
> > Kind regards
> > /Kent K
> > 
>