Subject: [idn] Re: Fwd: Unicode letter ballot

From: Simon Josefsson <[email-masked]>
Date: Mon, 25 Nov 2002 13:10:39 +0100

This message seems important for IDN.

Note that if option A win the vote, it seems that stringprep/nameprep
will have to fork the Unicode standard to remain backwards compatible
when stringprep/nameprep updates its Unicode reference.

I have added the characters below to the stringprep test vectors
document.

Patrik Fältström <[email-masked]> writes:

> As liason with the Unicode Consortium, I report the following.
>
>      Regards, Patrik
>
> Begin forwarded message:
>
>> From: Rick McGowan <[email-masked]>
>> Date: lör nov 23, 2002  21:12:10 Europe/Stockholm
>> To: [email-masked]
>> Subject: Unicode letter ballot
>>
>> To members of liaisons organizations:
>>
>> Clerical errors were found in canonical mappings for 5 Han
>> compatibility chars in plane 2... There is a letter ballot being
>> issued to
>> Unicode members, attached below.
>>
>> For further information on possible actions resulting from this ballot,
>> please see the following two pages:
>> 	http://www.unicode.org/unicode/uni2errata/UnicodeErrata.html
>> 	http://www.unicode.org/Public/UNIDATA/NormalizationCorrections.txt
>>
>> Regards,
>> 	Rick McGowan
>> 	Unicode, Inc.
>>
>> ---------------------------------------
>>>
>>> BALLOT to decide the correction for five canonical mapping errors
>>> in the
>>> Unicode Character Database.
>>>
>>>  APPROVE Option A: __
>>>
>>> OR
>>>
>>>  APPROVE Option B: __
>>>
>>> ___________________________________________
>>>
>>> SUMMARY
>>>
>>> There are five clear errors have been discovered in the canonical
>>> mappings
>>> for Plane 2 CJK compatibility characters, verified independently by
>>> multiple experts. Those errors are for U+2F868, U+2F874, U+2F91F,
>>> U+2F95F,
>>> and U+2F9BF.
>>>
>>> An additional five characters have been determined to have certain
>>> visual
>>> or clerical errors, but these will be handled through annotations
>>> to the
>>> names list.
>>>
>>> At UTC 93, the committee approved that a letter ballot be issued to
>>> choose
>>> the appropriate corrective action, either Option A or Option B:
>>>
>>> A) Fix the canonical mappings and issue another normalization
>>> corrigendum
>>> with corrected mappings in the Unicode 4.0 time frame, as follows:
>>>
>>>    Make the following corrections in UnicodeData.txt for 4.0:
>>>    Correct canonical mapping for 2F868 from 2136A to 36FC.
>>>    Correct canonical mapping for 2F874 from 5F33 to 5F53.
>>>    Correct canonical mapping for 2F91F from 43AB to 243AB.
>>>    Correct canonical mapping for 2F95F from 7AAE to 7AEE.
>>>    Correct canonical mapping for 2F9BF from 4D57 to 45D7.
>>>    Add the following entries to NormalizationCorrections.txt:
>>>    2F868;2136A;36FC;4.0.0 # Corrigendum 4
>>>    2F874;5F33;5F53;4.0.0 # Corrigendum 4
>>>    2F91F;43AB;243AB;4.0.0 # Corrigendum 4
>>>    2F95F;7AAE;7AEE;4.0.0 # Corrigendum 4
>>>    2F9BF;4D57;45D7;4.0.0 # Corrigendum 4
>>>
>>> B) Add five new characters with the correct canonical mappings and
>>> deprecate the existing characters with the incorrect mappings, as
>>> follows:
>>>
>>>    Deprecate 2F868, 2F874, 2F91F, 2F95F, 2F9BF.
>>>    Encode 5 new CJK compatibility characters as follows:
>>>    2FA1E CJK COMPATIBILITY CHARACTER-2FA1E --> 36FC
>>>    2FA1F CJK COMPATIBILITY CHARACTER-2FA1F --> 5F53
>>>    2FA20 CJK COMPATIBILITY CHARACTER-2FA20 --> 243AB
>>>    2FA21 CJK COMPATIBILITY CHARACTER-2FA21 --> 7AEE
>>>    2FA22 CJK COMPATIBILITY CHARACTER-2FA22 --> 45D7
>>> ___________________________________________
>>>
>>> IMPLICATIONS OF THE TWO OPTIONS:
>>>
>>> Implications of Option A.
>>>    a. Can be accomplished in Unicode 4.0 timeframe.
>>>    b. Requires issuing technical corrigendum 4 for
>>>       Unicode and a technical corrigendum for 10646.
>>>    c. Impacts normalization stability.
>>>    d. Follows precedent of technical corrigendum 3.
>>>
>>> Implications of Option B.
>>>    a. Deprecates 5 characters and creates 5 duplicates
>>>       of them to map correctly.
>>>    b. May be problematical to complete in Unicode 4.0
>>>       if not levered into current FPDAM for 10646.
>>>    c. Preserves normalization stability.
>>>    d. Sets new precedent for correction of CJK
>>>       compatibility mapping errors.
>>> ___________________________________________
>>

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 27 Nov 2002 14:50:29 +0100

> when stringprep/nameprep updates its Unicode reference.

sigh.... all 5 are beyond-BMP characters added recently.
if we could go back in time, we could have implemented a policy of not 
accepting characters as stable before 2 unicode versions had gone by....

proofreading takes time.

                       Harald

---

From: Simon Josefsson <[email-masked]>
Date: Thu, 28 Nov 2002 01:20:33 +0100

> accepting characters as stable before 2 unicode versions had gone

Even two Unicode releases doesn't guarantee that the tables are
correct.  The only proper solution I can see is to stop modifying
published decomposition tables.  When mistakes are discovered, new
character codes with proper decompositions should be added and the old
character codes declared obsolete -- which is option B in the vote,
but unfortunately neither IDN nor IETF has any voting powers (which
suggest a methodological problem).

---

From: Soobok Lee <[email-masked]>
Date: Thu, 28 Nov 2002 10:26:49 +0900

> suggest a methodological problem).

If option B  wins, a few issues may be raised easily:
 1) added characters does not preserve the original/ideal lexicographic
    radical-based order of those CJK glyphs.
 2) How does any obsoletion of characters work ? 
    compatiblity CJK characters are NFC-ed into other CJK characters
    in stringprep and so required obsoletion(mapping to newly added ones)
    should be done _before_ the stringprep operation is taken.
    
If option A wins, I see a chaos:
 1) NFKC_3.2_nameprep(NFKC_4.0_calling_application(IDN))  
       !=  NFKC_3.2_nameprep(IDN)  
    What if future HTML/canonical XML standards adopt NFKC_4.0 ?

Soobok Lee

---

From: [email-masked] (Martin v. Löwis)
Date: 28 Nov 2002 10:39:57 +0100

> If option A wins, I see a chaos:

I don't see chaos. Since these are compatibility characters, they
won't be part of any registered domain. Users of domains that contain
the compatibility equivalents of these characters should simply use
those equivalents in their HTML pages.

Notice that this "problem" already exists for Unicode 3.2.0, and
U+F951:

Regards,
Martin

---

From: Soobok Lee <[email-masked]>
Date: Thu, 28 Nov 2002 20:43:23 +0900

> those equivalents in their HTML pages.

Right, if the html page use NFCed UTF8 as basic encoding.

But, Most homepages in east asia use local charset /encoding. And
that requires localcharset<->unicode conversion that produces
compatibility CJKs which will be further fed into nameprep.
Korean IME produces compatiblity-CJK-equivalent KSX characters in
interactive input session on the url bar in browsers.

Soobok Lee

> Notice that this "problem" already exists for Unicode 3.2.0, and

Yes. This issue is not new problem. We had already discussed . And
concluded it is not a big problem. but, this instance of 5 chars
seems to need serious decision that we had not expected, iMO.
the pure possiblity become the reality.

Soobok lee

---

From: [email-masked] (Martin v. Löwis)
Date: 28 Nov 2002 13:26:16 +0100

> > > If option A wins, I see a chaos:
[...]
> interactive input session on the url bar in browsers.

Whether this would cause chaos is still questionable. First, I
understand that these are Chinese compatibility characters, so your
experience with Korean IME might not be relevant. The question is
rather whether users that use IMEs which produce CNS 11643 could enter
these characters, and whether they could also enter the
compatibility-normalized character instead.

[It appears that it will be hard to find an IME which supports the
full CNS 11643 repertoire in the first place].

In any case, the idea that incorrectly-normalized characters could
cause chaos seems unlikely.

> the pure possiblity become the reality.

What is the change here? Why is U+2F868 more real than U+F951?

Regards,
Martin

---

From: Soobok Lee <[email-masked]>
Date: Thu, 28 Nov 2002 22:33:44 +0900

On Thu, Nov 28, 2002 at 01:26:16PM +0100, Martin v. L?wis wrote:
> experience with Korean IME might not be relevant. 

I agree. Korean compatbility  CJKs are in every day CJK use, but CNS11643 
compatibility CJK  may or may not! I guess they will be rarely used.

> The question is

Whether or not  full CNS11643 support is rare does nothing to do with
the current issue. In any cases, _if_ someone enters those characters,
he will get errornous results surely! That is the concern.

> cause chaos seems unlikely.

"Chaos" does not mean stringprep/nameprep is a total failure. For example,
stringprep/nameprep is useful and satisfactory for everyday hangul syllable use!
My concern is that even such partial failures would make chaos _in some
cases_. We can't ignore this. We should do "something", but i am not
sure what to do yet.

> What is the change here? Why is U+2F868 more real than U+F951?

As i stated above, whether they are rare/real is not the issue.
Do you agree?

Soobok Lee

---

From: Soobok Lee <[email-masked]>
Date: Fri, 29 Nov 2002 00:43:03 +0900

> stringprep/nameprep is useful and satisfactory for everyday hangul syllable use!

I will add this: Precomposed hangul syllables (in every day use) remain unchanged  
through NFC.  But other ancient-syllable-forming jamo sequences may not be treated 
as it should be due to NFC's errors which is dated to the early stage of 
ISO10646/NFC deveopments.

If option A wins in the coming ballot, then I will push forward "correct NFC's hangul
jamo handling errors!" in Unicode 4.0, because NFC backward compatibility promises would
be regarded as being broken already in those cases of 5 CJK characters. I think "backward
compatiblity promise of NFC" is useful and neccesary but proved to be premature at least 
in current version 3.2.  It is proved that proofreading takes much time.

If option B wins, careful zone admin guidelines can help to alleviate the problems 
into less painful levels, as though they could not elliminate the problem completely.
Of course, option A causes some transitional (3.2->4.0) problems , which will however 
disappear sometime.