Subject: [idn] call for comments for REORDERING

From: "Soobok Lee" <[email-masked]>
Date: Wed, 10 Oct 2001 13:11:23 +0900

Hi, all

I am ready to receive any criticisms on REORDERING.
Any suggestions of improvements or downsides of REORDERING are all welcome.

I expect many feedbacks from non-US and non-European participants and observers.

Thanks.

Soobok Lee

----- Original Message ----- 
From: "Soobok Lee" <[email-masked]>
To: <[email-masked]>
Sent: Sunday, September 30, 2001 8:12 PM
Subject: Re: [idn] I-D ACTION:draft-ietf-idn-lsb-ace-02.txt

> Hi, all
> 
> This table summarizes the # of max chars in labels in each script block.
> This will be included the next revision of reordering I-D.
> For han/hangul block, REORDERING+ACE-Z outperforms DUDE by nearly 200%.
> If you have suggestions or other label samples to test with, 
> please leave me a note.
>  
> Thanks.
> 
> Soobok Lee,   [email-masked]
> 
> 
> +------------------------------------------------------------+
> |Maximum number of characters in a label in each script block|
> |                                                            |
> |                max char = (63 - 4) / ratio                 |
> |   ACE length = 4 + ratio * N ( N : # of chars in label)    |
> +---------------+-------------------+-------------------+----+
> | Script Block  |     AMC-ACE-Z     |   Reordering +    |Gain|
> |               |                   |     AMC-ACE-Z     |    |
> +---------+-----+---------+---------+---------+---------+    |
> |  Name   |Size |  ratio  |max char |  ratio  |max char |    |
> +---------+-----+---------+---------+---------+---------+----+
> |   Han   |20992|  2.98   |   19    |  2.21   |   26    | +7 |
> +---------+-----+---------+---------+---------+---------+----+
> | Hangeul |11172|  2.97   |   19    |  2.04   |   28    | +9 |
> +---------+-----+---------+---------+---------+---------+----+
> |  Greek  | 144 |  1.29   |   45    |  1.14   |   51    | +6 |
> +---------+-----+---------+---------+---------+---------+----+
> | Arabic  | 256 |  1.34   |   44    |  1.28   |   46    | +2 |
> +---------+-----+---------+---------+---------+---------+----+
> |Cyrillic | 304 |  1.23   |   47    |  1.15   |   51    | +4 |
> +---------+-----+---------+---------+---------+---------+----+
> | Hebrew  | 102 |  1.29   |   45    |  1.18   |   50    | +5 |
> +---------+-----+---------+---------+---------+---------+----+
> |  Hindi  | 128 |  1.31   |   45    |  1.17   |   50    | +5 |
> +---------+-----+---------+---------+---------+---------+----+
> |Hiragana | 96  |  1.50   |   39    |  1.45   |   40    | +1 |
> +---------+-----+---------+---------+---------+---------+----+
> |Katakana | 96  |  1.44   |   40    |  1.33   |   44    | +4 |
> +---------+-----+---------+---------+---------+---------+----+
> |  Tamil  | 128 |  1.62   |   36    |  1.14   |   51    |+15 |
> +---------+-----+---------+---------+---------+---------+----+
> |  Thai   | 128 |  1.39   |   42    |  1.13   |   52    |+10 |
> +---------+-----+---------+---------+---------+---------+----+
> |Ethiopic | 416 |  2.10   |   28    |  1.60   |   36    | +8 |
> +---------+-----+---------+---------+---------+---------+----+
> 
> 
> ----- Original Message ----- 
> From: <[email-masked]>
> To: <IETF-Announce:>
> Cc: <[email-masked]>
> Sent: Tuesday, September 18, 2001 10:33 PM
> Subject: [idn] I-D ACTION:draft-ietf-idn-lsb-ace-02.txt
> 
> 
> > A New Internet-Draft is available from the on-line Internet-Drafts directories.
> > This draft is a work item of the Internationalized Domain Name Working Group of the IETF.
> > 
> > Title : Improving ACE using code point reordering v2.0
> > Author(s) : S. Lee
> > Filename : draft-ietf-idn-lsb-ace-02.txt
> > Pages : 
> > Date : 17-Sep-01
> > 
> > This document describes a method to improve ACE label efficiency by
> > frequency-based temporal reordering of code points in ACE encoding
> > and decoding processes in order to relocate scattered frequent 
> > characters into much more compact area in reordered code space.
> > 
> > A URL for this Internet-Draft is:
> > http://www.ietf.org/internet-drafts/draft-ietf-idn-lsb-ace-02.txt
> > 
> > To remove yourself from the IETF Announcement list, send a message to 
> > ietf-announce-request with the word unsubscribe in the body of the message.
> > 
> > Internet-Drafts are also available by anonymous FTP. Login with the username
> > "anonymous" and a password of your e-mail address. After logging in,
> > type "cd internet-drafts" and then
> > "get draft-ietf-idn-lsb-ace-02.txt".
> > 
> > A list of Internet-Drafts directories can be found in
> > http://www.ietf.org/shadow.html 
> > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
> > 
> > 
> > Internet-Drafts can also be obtained by e-mail.
> > 
> > Send a message to:
> > [email-masked].
> > In the body type:
> > "FILE /internet-drafts/draft-ietf-idn-lsb-ace-02.txt".
> > 
> > NOTE: The mail server at ietf.org can return the document in
> > MIME-encoded form by using the "mpack" utility.  To use this
> > feature, insert the command "ENCODING mime" before the "FILE"
> > command.  To decode the response(s), you will need "munpack" or
> > a MIME-compliant mail reader.  Different MIME-compliant mail readers
> > exhibit different behavior, especially when dealing with
> > "multipart" MIME messages (i.e. documents which have been split
> > up into multiple messages), so check your local documentation on
> > how to manipulate these messages.
> > 
> > 
> > Below is the data which will enable a MIME compliant mail reader
> > implementation to automatically retrieve the ASCII version of the
> > Internet-Draft.
> > 
> 
>

---

From: Martin Duerst <[email-masked]>
Date: Wed, 10 Oct 2001 14:16:21 +0900

Hello Soobok,

Here are my comments on reordering. Apologies if they are
too clear.

The additional complexity introduced by reordering is a very
serious problem. It is true that this complexity is somewhat
similar to the complexity of e.g. nameprep or conversion from
a legacy encoding. However, on many platforms, both conversion
from a legacy encoding as well as many aspects of nameprep
are available as libraries, and are used for other purposes.
In particular on constrained devices (mobile phones,...),
most of nameprep can be simplified a lot if one knows what
kinds of characters can be input.

On the other hand, the benefits for the users are actually
very small. Nobody wants to input domain names with 15 or
more Hanzi or Hangul. Nobody will be able to remember them.
Writing them down on a napkin will take a long time.
Every company or organization that has such a long label
in their domain name, and no shorter alternative, will
simply not get any contacts directly to their web site.
If they have a short alternative, why do they need a
long version? (please note that there is no danger of
spoofing by somebody else getting the long version :-).

So this is a solution in search of a real problem,
not worth bothering the whole world with additional
complexity.

Regards,   Martin.

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 10 Oct 2001 15:43:41 +0900

thanks for you comment. Here goes my answer.

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Wednesday, October 10, 2001 2:16 PM
Subject: Re: [idn] call for comments for REORDERING

> are available as libraries, and are used for other purposes.

Current Windows 98,2K,XP and Linux  contain   NFKC codes ?

> kinds of characters can be input.

In this case, the reordering table _also_ can be simplified
only to that input script blocks .

As for arabic/hindi/kata/hira/tamil/greek/hebrew, 
it adds only +10~+20 lines of simple character mapping array
for each script.
those additional lines of data are less than the # of  comment
lines in the ACE source code.

> spoofing by somebody else getting the long version :-).

lets' think about the shorter ACE label produced for the native label of 
mean _average_ length. 
The next table for hangul script block says:
  For N=6, 3.12*6 - 2.28*6 = 18 - 13 = 5 characters are saved.

The main benefits of REODERING is not only for very long domains,
but for average ones. It also helps administation,transcription and eye-
comparison on the ACE labels.
  
Moreover, as IDNA I-D recommends, 
ACE labels should be rendered "as it is",
when the decoded native-label contains characters which 
the rendering engine cannot display. For example,
if you have _no_ huge han script font set in your mobile phone (or PC),
the ACE labels for han/hangul email addresses of your friends should
be displayed "as it is". But, it may be often too long to be displayed 
in your narrow phone LCD lines of width 16~20, without REORDERING.

Moreover, If we get ACEed i18n email addresses in the future, 
its native form will look like [email-masked]. In the case of han/hangul script , 
the sum of lengths of two non-latin strings (XXXX,YYYYYY) may 
exceed 10 very often. 
For such cases, the reordering would help  to save up to 8~9
characters for ACEed email addresess. Big saves..

8. hangul-1024

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    1953|      1953|      7812(4.00)|      7812(4.00)| 0.00|
|  2|   17149|     34298|    124782(3.64)|    106238(3.10)|14.86|
|  3|   39643|    118929|    403205(3.39)|    323801(2.72)|19.69|
|  4|   62285|    249140|    816093(3.28)|    622067(2.50)|23.77|
|  5|   39675|    198375|    636102(3.21)|    470174(2.37)|26.09|
|  6|   23891|    143346|    452483(3.16)|    326242(2.28)|27.90|
|  7|   12448|     87136|    271953(3.12)|    192139(2.21)|29.35|
|  8|    5441|     43528|    134600(3.09)|     94322(2.17)|29.92|
|  9|    2264|     20376|     62405(3.06)|     43266(2.12)|30.67|
| 10|     895|      8950|     27223(3.04)|     18764(2.10)|31.07|
| 11|     373|      4103|     12420(3.03)|      8511(2.07)|31.47|
| 12|     141|      1692|      5080(3.00)|      3505(2.07)|31.00|
| 13|      77|      1001|      2986(2.98)|      2039(2.04)|31.71|
| 14|      32|       448|      1331(2.97)|       911(2.03)|31.56|
| 15|      20|       300|       884(2.95)|       603(2.01)|31.79|
| 16|      10|       160|       460(2.88)|       337(2.11)|26.74|
| 17|       7|       119|       354(2.97)|       243(2.04)|31.36|

|All|  206304|    913854|   2960173(3.24)|   2220974(2.43)|24.97|

non-CJK script often has long labels that exceeds 15.

1. arabic

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|      42|        42|       126(3.00)|       126(3.00)| 0.00|
|  2|      59|       118|       258(2.19)|       249(2.11)| 3.49|
|  3|     363|      1089|      2121(1.95)|      1992(1.83)| 6.08|
|  4|     888|      3552|      6359(1.79)|      5811(1.64)| 8.62|
|  5|    1122|      5610|      9550(1.70)|      8529(1.52)|10.69|
|  6|    1009|      6054|      9890(1.63)|      8620(1.42)|12.84|
|  7|     845|      5915|      9309(1.57)|      8134(1.38)|12.62|
|  8|     378|      3024|      4590(1.52)|      3992(1.32)|13.03|
|  9|     263|      2367|      3523(1.49)|      3063(1.29)|13.06|
| 10|     152|      1520|      2230(1.47)|      1941(1.28)|12.96|
| 11|     130|      1430|      2058(1.44)|      1787(1.25)|13.17|
| 12|     110|      1320|      1873(1.42)|      1614(1.22)|13.83|
| 13|      67|       871|      1230(1.41)|      1040(1.19)|15.45|
| 14|      61|       854|      1211(1.42)|      1015(1.19)|16.18|
| 15|      52|       780|      1085(1.39)|       924(1.18)|14.84|
| 16|      34|       544|       743(1.37)|       630(1.16)|15.21|
| 17|      11|       187|       256(1.37)|       218(1.17)|14.84|
| 18|      19|       342|       465(1.36)|       392(1.15)|15.70|
| 19|       8|       152|       201(1.32)|       175(1.15)|12.94|
| 20|      10|       200|       268(1.34)|       235(1.18)|12.31|
| 21|       3|        63|        85(1.35)|        75(1.19)|11.76|
| 22|       4|        88|       116(1.32)|        99(1.12)|14.66|
| 23|       3|        69|        89(1.29)|        76(1.10)|14.61|
| 24|       2|        48|        62(1.29)|        55(1.15)|11.29|
| 25|       5|       125|       165(1.32)|       143(1.14)|13.33|
| 26|       2|        52|        67(1.29)|        56(1.08)|16.42|
| 27|       2|        54|        73(1.35)|        61(1.13)|16.44|
| 33|       1|        33|        41(1.24)|        37(1.12)| 9.76|
| 34|       1|        34|        45(1.32)|        36(1.06)|20.00|

|All|    5646|     36537|     58089(1.59)|     51125(1.40)|11.99|

Regards,

Soobok Lee

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 19 Oct 2001 09:09:03 +0800

I would like to add my comments to this:

Reordering increases efficient of the encoding allowing longer labels.
This allows us to go sometimes as much as 25% more. OTOH, as IDN as a
form of identifiers, then practically, shorter labels are better. Aside,
re-ordering also increases the complexity of the algorithm. This is an
engineering trade-off we have to made should we consider re-ordering.

The bigger concern I have with re-ordering remains in the fact that
tables mappings proves efficient with existing IDN names in some
registries *BUT* it  does not indicate what performance it would be like
in the future. We do not know what happened when the names space get
saturated and would other names which would have been useable without
lsb become un-usuable due to lsb.

And with the existing draft, it does not explain how it going to deal
with new codepoints in ISO10646 in the future, nor does it explain the
process to implementing them. The critia here is stablity - if new code
is added and tables for re-ordering expand, then the algorithm should
not make existing names invalided.

Third, I would really prefer to reference a work from established expert
group if possible. For example, ISO/IEC JTC1/SC22/WG20 publishes ISO
14651 on weighted sorting. I am not sure how ISO 14651 would perform for
the IDN purpose but I thought it might be worthwhile to examine.

Whatever the case, we should make a decision quickly on this. Lets not
drag this further if possible.

-James Seng

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 12:13:01 +0900

HI, James

I will answer one by one for your  questions.

----- Original Message ----- 
From: "James Seng/Personal" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>; "Martin Duerst" <[email-masked]>
Sent: Friday, October 19, 2001 10:09 AM
Subject: Re: [idn] call for comments for REORDERING

> not make existing names invalided.

REORDERING's mapping occurs within each script block, NOT across script blocks.
Therefore, new additions of script block won't invalidate or collide
with existing names. Even additions of new rarely used characters in 
existing script block won't affect the performance of REORDERING.
In this respect, REORDERING maintains stability over time.

Current IDNA/nameprep does not prohibit, but discourage including unassigned code points in legal IDN labels, because new normalization/case mappings
would be defined on them in the future. some ACE labels including unsigned code block (tagalog?) might be proven invalid in the future. Nameprep/NFKC Versioning tag schems using new ACE prefix will be needed in the future, i guess.

Therefore, REORDERING  follows the same IDNA/nameprep recommendations on the issues
of new SCTIPT blocks/unsigned code points. 

This statements will be included in the final version of REORDERING I-D.

Thanks.

Soobok Lee

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 19 Oct 2001 12:14:37 +0800

Your answer does not answer how we going to update re-ordering in future
with new codepoints. Do you expect to continue to work with this and to
have update RFC for it? And we going to update RFC again and again? The
typical solution IETF use for this is IANA action and that involves a
well-defined policy which IANA could reused for future addition. But if
we do so, we also need to consider if we are imposing codepoints issues
which is best dealt with other organisation.

As I repeated, I really like to point to another organisation for these
tables if possible. If ISO 14651 is not suitable (as Ken & Mark said),
then we look at some others. If there is none, then maybe we should
consider NOT to do it at all.

-James Seng

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "James Seng/Personal" <[email-masked]>; <[email-masked]>;
"Martin Duerst" <[email-masked]>
Sent: Friday, October 19, 2001 11:13 AM
Subject: Re: [idn] call for comments for REORDERING

HI, James

----- Original Message -----
From: "James Seng/Personal" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>; "Martin Duerst"
<[email-masked]>

> not make existing names invalided.

REORDERING's mapping occurs within each script block, NOT across script
blocks.

Current IDNA/nameprep does not prohibit, but discourage including
unassigned code points in legal IDN labels, because new
normalization/case mappings
would be defined on them in the future. some ACE labels including
unsigned code block (tagalog?) might be proven invalid in the future.
Nameprep/NFKC Versioning tag schems using new ACE prefix will be needed
in the future, i guess.

Therefore, REORDERING  follows the same IDNA/nameprep recommendations on
the issues
of new SCTIPT blocks/unsigned code points.

Thanks.

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 13:11:55 +0900

Another answer for your concern.

----- Original Message ----- 
> lsb become un-usuable due to lsb.
 
1) saturations in TLD namespaces would require longer names for which
    REORDERING is designed to give greater benefits/compression ratio.

2) future variations on character usage frequency in each script

    2.0) the character frequency table are constructed from
         Verisign GRS' ML.com testbeds.
         Even for chinese han script, their
          registrations came from China/TAIWAN/JAPAN/KOREA and other
           non-asian squatters.
         Each country of the 4 have their own different han character
           usage patterns. The reordering table for han , therefore,
          cannot  for the worst case, the mutual difference in improvement ratios
          did not exceed  +- 2% around 20%.
           
    2.1) this issue is already answered by latest REORDERING I-D 2.0
         see the enclosed excerpts from it. The influence of this
         frequency variations is marginal.

3)

My REORDERING I-D contains experiments with various lengths of han
frequency tables : 1024,2048,3072,4096. In order to look into 
the influence of INs and OUTs of han characters from the most frequent 
2048 han characters.

For the two cases of table size 2048 and 4096, there was merely +- 1% of
differences in achieved improvements. I also reversed the order of the table itself,but it produced nearly the same result. 
The partial change in the order of the reordering table does not make big differences. some will lose and some will win. the net effect is near to zero.

I believe most of SJIS and KSC5601 han characters are  included in the
most frequent 4096 han character tables, because their governments bodies
selected   a few thousands of most frequent subset of entire han characters.

i believe that frequency fluctuation of han characters over time  is 
WITHIN the frequent set. INs and OUTs from 4096 ones are rare and does not invalidate the validity of most frequent 1024 and 2048 ones.
Moreover, TC/SC/KC characters are put side-by-side to avoid countriy-specific biases in han reordering table.

non-CJK scripts often haver small set of basic alphabets, and their
character usage patterns are more stable than those for han/hangeul.

REORDERING does not recommends reordering on shares latin scripts,
because latin characters are already encoded as it is (in literal mode,
the most efficient form ). latin script for europeans (0.6 billions) are the most favored one in ACE-Z. There shoulbe be some conpensations for
non europeans. Han script: 2 billions, Arabic: 0.7 billion, Hindi: 0.5 billion

This new frequency-based reordering is always more efficient than 
original lexicographical ordering in UCS 
even with some fluctuation in future script usage patterns.

We are not pursuing elusive "perfection and optimal" solution.
REORDERING tables cannot be modified if it is once freezed as standards.
Therefore,REORDERING is a sub-optimal solution in its nature but will be remain
as a valid and effective solution for a long time .

Soobok
 
---------------------------

Unified Han and Hangeul

    11172 Hangul syllables and 20912 CJK Unified Han ideographs occupy 
    roughly two thirds of current assigned unicode code points.
    Their lexicographical ordering makes various ACE compression 
    algorithm work poorly for them, because they are spread evenly 
    through out those wide code blocks.

    According to one usage frequency statistics on hangeul syllables
    in general hangeul texts, the most frequent 256 Hangul syllables 
    have the cumulative frequency sum of 88.2% and for the case of top 
    512 ones, it reaches 99.9%. That means the maximum variation of 
    code point values(11172) can be shrinked into 512 in reordered
    hangeul block with a probability of 99.9%.

    Likewise, the most frequent 256 Han letters have the cumulative 
    frequency sum of 58.2% and for the cases of top 512,1024,2048 and 
    4096 ones, it reaches 72.8%,85.9%,95.4% and 99.4%, respectively.
    That means the maximum variation of code point values (20912) can 
    be shrinked into 2048 with a probability of 95.4%.

    The han/hangul frequency mapping tables are constructed from 
    nameprepped ML.com domains from VGRS MultiLingual testbeds.
    The frequenet characters in the tables are organized by their
    increasing frequency order to minimize the AMC-ACE-Z bootstring 
    delta values which can be lowered when bigger code distances are 
    from the lower positions of the sorted labels in AMC-ACE-Z step 2.

    In general,character frequency distributions in any script block 
    may undergo some shifts within the frequent set by the passage of 
    time, but the in and out of some characters from the frequent set
    are very rare. So, their impacts may be as marginal and negligable 
    as the following comparison of experiment results shows.

    Reorering tables based on most frequent 1024,2048,3072 and 4096 han
    and hangul letters in increasing frequency order, produced marginal 
    differences in improvements:

      N is the length of sample labels and other
      decimal values (in percentage) are the improvement ratios for 
      all the combinations of all N and 4 reordering tables.

      |  N|  HAN-4096|  HAN-3072|  HAN-2048|  HAN-1024|
      |  1|     7.07 |     5.49 |     3.58 |      1.64|
      |  2|    13.61 |    13.22 |    11.57 |      8.06|
      |  3|    16.26 |    16.05 |    15.10 |     12.26|
      |  4|    20.80 |    20.71 |    20.19 |     18.11|
      |  5|    22.17 |    22.03 |    21.47 |     19.41|
      |  6|    24.85 |    24.77 |    24.41 |     22.48|
      |  7|    25.52 |    25.40 |    24.99 |     23.17|
      |  8|    26.47 |    26.36 |    26.00 |     24.15|
      |  9|    26.54 |    26.46 |    26.04 |     24.26|
      | 10|    27.47 |    27.40 |    27.01 |     25.09|
      | 11|    27.30 |    27.26 |    26.85 |     25.12|
      | 12|    27.74 |    27.64 |    27.41 |     25.60|
      | 13|    27.27 |    27.17 |    26.78 |     25.28|
      | 14|    27.48 |    27.35 |    27.08 |     24.94|
      | 15|    28.60 |    28.43 |    28.56 |     26.54|
      | 16|    27.70 |    27.84 |    27.70 |     25.51|
      | 17|    25.68 |    25.68 |    25.43 |     23.70| 
      |ALL|    20.30 |    20.14 |    19.43 |     17.09|

    Experiments with two reorering tables in increasing and descreasing 
    orders for most frequent 2048,4096 han letters,also produced 
    marginal differences in improvements:

     (4096D means: the ordering table is in decreasing frequency order) 

      |  N|  HAN-4096| HAN-4096D|  HAN-2048| HAN-2048D|
      |  1|     7.07 |     7.01 |     3.58 |     3.51 |
      |  2|    13.61 |    13.44 |    11.57 |    11.27 |
      |  3|    16.26 |    16.35 |    15.10 |    14.93 |
      |  4|    20.80 |    20.56 |    20.19 |    19.90 |
      |  5|    22.17 |    21.80 |    21.47 |    21.12 |
      |  6|    24.85 |    24.21 |    24.41 |    23.82 |
      |  7|    25.52 |    24.59 |    24.99 |    24.14 |
      |  8|    26.47 |    25.68 |    26.00 |    25.36 |
      |  9|    26.54 |    25.55 |    26.04 |    25.18 |
      | 10|    27.47 |    26.79 |    27.01 |    26.42 |
      | 11|    27.30 |    26.82 |    26.85 |    26.36 |
      | 12|    27.74 |    27.46 |    27.41 |    27.13 |
      | 13|    27.27 |    26.97 |    26.78 |    26.59 |
      | 14|    27.48 |    27.31 |    27.08 |    26.99 |
      | 15|    28.60 |    28.60 |    28.56 |    28.56 |
      | 16|    27.70 |    27.55 |    27.70 |    27.20 |
      | 17|    25.68 |    25.93 |    25.43 |    25.68 | 
      |ALL|    20.30 |    20.00 |    19.43 |    19.07 |

    These experiments show that the influences of some fluctations in 
    character frequency distributions in the frequent set of a script 
    would not be so great that could invalidate or outdate this 
    reordering approach in the forseeable future.

    But,to be as neutral and fair as possible in dealing with the cases 
    with different usage patterns in China,Japan,Korea and Taiwan, here 
    are provided some provisions for grouping country-specific variants
    of certain han letters. Especially, a group of simplified chainese 
    letter (SC) and traditional chinese letter (TC) and Kanji-specific
    letter (KC) are ranked by the sum of their frequecies and placed 
    side-by-side in the reordering table for Unified Han block. 
    For example, the reordering table looks like:
     (TC1) (TC2 SC2) (TC3 KC3) (TC4) (TC5 SC5 KC5) (TC6) .....
    This grouping will serve to prevent the frequency orders from being
    skewed toward one of those country-specific usage patterns.
    The experiments results 27 and 28 in [A3] shows that this reordering
    scheme improve 21.95% and 18.50% for SC and TC labels,respectively.

    According to experiments with huge han/hangeul domain samples,
    as for 15 or more letters of  han/hangeul domains, AMC-ACE-Z with 
    reordering produced the shortest ACE labels which length approximate
    to 2.0*n~2.2*n (n= number of han/hangul code points in a label), 
    33.3% more efficient than bare AMC-ACE-Z without the reordering.
    This efficiency is close to that of UCS-2 ( 2.0 * n) and much better
    than that of UTF8 ( 3.0*n ).

    The appendix [A3] also contains some tuning experiments on ACE-Z's
    skew and damp parameters. With skew==48 and damp==75, +1.3% in 
    compression ratio was achieved for han domains with some marginal 
    loss of efficiency in non-CJK scripts.

----------------
19. unihan-1024

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    4427|      4427|     14957(3.38)|     14711(3.32)| 1.64|
|  2|   57418|    114836|    384468(3.35)|    353466(3.08)| 8.06|
|  3|   41335|    124005|    401283(3.24)|    352095(2.84)|12.26|
|  4|   89296|    357184|   1139404(3.19)|    933070(2.61)|18.11|
|  5|   21091|    105455|    332420(3.15)|    267893(2.54)|19.41|
|  6|   15128|     90768|    284134(3.13)|    220263(2.43)|22.48|
|  7|    5181|     36267|    112576(3.10)|     86487(2.38)|23.17|
|  8|    3082|     24656|     76272(3.09)|     57854(2.35)|24.15|
|  9|    1417|     12753|     39319(3.08)|     29779(2.34)|24.26|
| 10|    1203|     12030|     37136(3.09)|     27817(2.31)|25.09|
| 11|     474|      5214|     16072(3.08)|     12035(2.31)|25.12|
| 12|     398|      4776|     14714(3.08)|     10947(2.29)|25.60|
| 13|     164|      2132|      6532(3.06)|      4881(2.29)|25.28|
| 14|     122|      1708|      5232(3.06)|      3927(2.30)|24.94|
| 15|      50|       750|      2283(3.04)|      1677(2.24)|26.54|
| 16|      29|       464|      1419(3.06)|      1057(2.28)|25.51|
| 17|       8|       136|       405(2.98)|       309(2.27)|23.70|

|All|  240823|    897561|   2868626(3.20)|   2378268(2.65)|17.09|

20. unihan-2048

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    4427|      4427|     14957(3.38)|     14422(3.26)| 3.58|
|  2|   57418|    114836|    384468(3.35)|    339996(2.96)|11.57|
|  3|   41335|    124005|    401283(3.24)|    340675(2.75)|15.10|
|  4|   89296|    357184|   1139404(3.19)|    909323(2.55)|20.19|
|  5|   21091|    105455|    332420(3.15)|    261039(2.48)|21.47|
|  6|   15128|     90768|    284134(3.13)|    214781(2.37)|24.41|
|  7|    5181|     36267|    112576(3.10)|     84440(2.33)|24.99|
|  8|    3082|     24656|     76272(3.09)|     56439(2.29)|26.00|
|  9|    1417|     12753|     39319(3.08)|     29082(2.28)|26.04|
| 10|    1203|     12030|     37136(3.09)|     27106(2.25)|27.01|
| 11|     474|      5214|     16072(3.08)|     11756(2.25)|26.85|
| 12|     398|      4776|     14714(3.08)|     10681(2.24)|27.41|
| 13|     164|      2132|      6532(3.06)|      4783(2.24)|26.78|
| 14|     122|      1708|      5232(3.06)|      3815(2.23)|27.08|
| 15|      50|       750|      2283(3.04)|      1631(2.17)|28.56|
| 16|      29|       464|      1419(3.06)|      1026(2.21)|27.70|
| 17|       8|       136|       405(2.98)|       302(2.22)|25.43|

|All|  240823|    897561|   2868626(3.20)|   2311297(2.58)|19.43|

21. unihan-2048-D ( the reordering in decreasing frequency order)

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    4427|      4427|     14957(3.38)|     14432(3.26)| 3.51|
|  2|   57418|    114836|    384468(3.35)|    341134(2.97)|11.27|
|  3|   41335|    124005|    401283(3.24)|    341362(2.75)|14.93|
|  4|   89296|    357184|   1139404(3.19)|    912694(2.56)|19.90|
|  5|   21091|    105455|    332420(3.15)|    262224(2.49)|21.12|
|  6|   15128|     90768|    284134(3.13)|    216465(2.38)|23.82|
|  7|    5181|     36267|    112576(3.10)|     85401(2.35)|24.14|
|  8|    3082|     24656|     76272(3.09)|     56931(2.31)|25.36|
|  9|    1417|     12753|     39319(3.08)|     29420(2.31)|25.18|
| 10|    1203|     12030|     37136(3.09)|     27324(2.27)|26.42|
| 11|     474|      5214|     16072(3.08)|     11835(2.27)|26.36|
| 12|     398|      4776|     14714(3.08)|     10722(2.24)|27.13|
| 13|     164|      2132|      6532(3.06)|      4795(2.25)|26.59|
| 14|     122|      1708|      5232(3.06)|      3820(2.24)|26.99|
| 15|      50|       750|      2283(3.04)|      1631(2.17)|28.56|
| 16|      29|       464|      1419(3.06)|      1033(2.23)|27.20|
| 17|       8|       136|       405(2.98)|       301(2.21)|25.68|

|All|  240823|    897561|   2868626(3.20)|   2321524(2.59)|19.07|

22. unihan-3072

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    4427|      4427|     14957(3.38)|     14136(3.19)| 5.49|
|  2|   57418|    114836|    384468(3.35)|    333660(2.91)|13.22|
|  3|   41335|    124005|    401283(3.24)|    336865(2.72)|16.05|
|  4|   89296|    357184|   1139404(3.19)|    903458(2.53)|20.71|
|  5|   21091|    105455|    332420(3.15)|    259189(2.46)|22.03|
|  6|   15128|     90768|    284134(3.13)|    213746(2.35)|24.77|
|  7|    5181|     36267|    112576(3.10)|     83977(2.32)|25.40|
|  8|    3082|     24656|     76272(3.09)|     56168(2.28)|26.36|
|  9|    1417|     12753|     39319(3.08)|     28917(2.27)|26.46|
| 10|    1203|     12030|     37136(3.09)|     26962(2.24)|27.40|
| 11|     474|      5214|     16072(3.08)|     11690(2.24)|27.26|
| 12|     398|      4776|     14714(3.08)|     10647(2.23)|27.64|
| 13|     164|      2132|      6532(3.06)|      4757(2.23)|27.17|
| 14|     122|      1708|      5232(3.06)|      3801(2.23)|27.35|
| 15|      50|       750|      2283(3.04)|      1634(2.18)|28.43|
| 16|      29|       464|      1419(3.06)|      1024(2.21)|27.84|
| 17|       8|       136|       405(2.98)|       301(2.21)|25.68|

|All|  240823|    897561|   2868626(3.20)|   2290932(2.55)|20.14|

23. unihan-4096

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    4427|      4427|     14957(3.38)|     13899(3.14)| 7.07|
|  2|   57418|    114836|    384468(3.35)|    332156(2.89)|13.61|
|  3|   41335|    124005|    401283(3.24)|    336045(2.71)|16.26|
|  4|   89296|    357184|   1139404(3.19)|    902406(2.53)|20.80|
|  5|   21091|    105455|    332420(3.15)|    258709(2.45)|22.17|
|  6|   15128|     90768|    284134(3.13)|    213522(2.35)|24.85|
|  7|    5181|     36267|    112576(3.10)|     83844(2.31)|25.52|
|  8|    3082|     24656|     76272(3.09)|     56083(2.27)|26.47|
|  9|    1417|     12753|     39319(3.08)|     28883(2.26)|26.54|
| 10|    1203|     12030|     37136(3.09)|     26935(2.24)|27.47|
| 11|     474|      5214|     16072(3.08)|     11684(2.24)|27.30|
| 12|     398|      4776|     14714(3.08)|     10632(2.23)|27.74|
| 13|     164|      2132|      6532(3.06)|      4751(2.23)|27.27|
| 14|     122|      1708|      5232(3.06)|      3794(2.22)|27.48|
| 15|      50|       750|      2283(3.04)|      1630(2.17)|28.60|
| 16|      29|       464|      1419(3.06)|      1026(2.21)|27.70|
| 17|       8|       136|       405(2.98)|       301(2.21)|25.68|

|All|  240823|    897561|   2868626(3.20)|   2286300(2.55)|20.30|

24. unihan-4096-D

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    4427|      4427|     14957(3.38)|     13909(3.14)| 7.01|
|  2|   57418|    114836|    384468(3.35)|    332799(2.90)|13.44|
|  3|   41335|    124005|    401283(3.24)|    335682(2.71)|16.35|
|  4|   89296|    357184|   1139404(3.19)|    905086(2.53)|20.56|
|  5|   21091|    105455|    332420(3.15)|    259944(2.46)|21.80|
|  6|   15128|     90768|    284134(3.13)|    215353(2.37)|24.21|
|  7|    5181|     36267|    112576(3.10)|     84893(2.34)|24.59|
|  8|    3082|     24656|     76272(3.09)|     56682(2.30)|25.68|
|  9|    1417|     12753|     39319(3.08)|     29273(2.30)|25.55|
| 10|    1203|     12030|     37136(3.09)|     27189(2.26)|26.79|
| 11|     474|      5214|     16072(3.08)|     11762(2.26)|26.82|
| 12|     398|      4776|     14714(3.08)|     10674(2.23)|27.46|
| 13|     164|      2132|      6532(3.06)|      4770(2.24)|26.97|
| 14|     122|      1708|      5232(3.06)|      3803(2.23)|27.31|
| 15|      50|       750|      2283(3.04)|      1630(2.17)|28.60|
| 16|      29|       464|      1419(3.06)|      1028(2.22)|27.55|
| 17|       8|       136|       405(2.98)|       300(2.21)|25.93|

|All|  240823|    897561|   2868626(3.20)|   2294777(2.56)|20.00|

25. unihan-4096-DAMP075-SKEW48

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    4427|      4427|     14957(3.38)|     13899(3.14)| 7.07|
|  2|   57418|    114836|    375901(3.27)|    324587(2.83)|13.65|
|  3|   41335|    124005|    394416(3.18)|    330550(2.67)|16.19|
|  4|   89296|    357184|   1126357(3.15)|    890277(2.49)|20.96|
|  5|   21091|    105455|    329783(3.13)|    255913(2.43)|22.40|
|  6|   15128|     90768|    282751(3.12)|    211339(2.33)|25.26|
|  7|    5181|     36267|    112181(3.09)|     83126(2.29)|25.90|
|  8|    3082|     24656|     76111(3.09)|     55712(2.26)|26.80|
|  9|    1417|     12753|     39285(3.08)|     28699(2.25)|26.95|
| 10|    1203|     12030|     37150(3.09)|     26767(2.23)|27.95|
| 11|     474|      5214|     16028(3.07)|     11603(2.23)|27.61|
| 12|     398|      4776|     14712(3.08)|     10567(2.21)|28.17|
| 13|     164|      2132|      6528(3.06)|      4735(2.22)|27.47|
| 14|     122|      1708|      5248(3.07)|      3762(2.20)|28.32|
| 15|      50|       750|      2281(3.04)|      1628(2.17)|28.63|
| 16|      29|       464|      1425(3.07)|      1017(2.19)|28.63|
| 17|       8|       136|       404(2.97)|       301(2.21)|25.50|

|All|  240823|    897561|   2835518(3.16)|   2254482(2.51)|20.49|

26. unihan-4096-DUDE

|  N|    FREQ|    N*FREQ|  SUM OF DUDE(X)| SUM OF LDUDE(Y)| COMP|
|  1|    4427|      4427|     17708(4.00)|     17708(4.00)| 0.00|
|  2|   57418|    114836|    443874(3.87)|    409657(3.57)| 7.71|
|  3|   41335|    124005|    474117(3.82)|    408039(3.29)|13.94|
|  4|   89296|    357184|   1361917(3.81)|   1074237(3.01)|21.12|
|  5|   21091|    105455|    401146(3.80)|    308378(2.92)|23.13|
|  6|   15128|     90768|    344208(3.79)|    250925(2.76)|27.10|
|  7|    5181|     36267|    137275(3.79)|     99475(2.74)|27.54|
|  8|    3082|     24656|     93013(3.77)|     65889(2.67)|29.16|
|  9|    1417|     12753|     48000(3.76)|     34230(2.68)|28.69|
| 10|    1203|     12030|     45427(3.78)|     31663(2.63)|30.30|
| 11|     474|      5214|     19564(3.75)|     13708(2.63)|29.93|
| 12|     398|      4776|     18013(3.77)|     12468(2.61)|30.78|
| 13|     164|      2132|      7969(3.74)|      5590(2.62)|29.85|
| 14|     122|      1708|      6377(3.73)|      4476(2.62)|29.81|
| 15|      50|       750|      2811(3.75)|      1926(2.57)|31.48|
| 16|      29|       464|      1749(3.77)|      1213(2.61)|30.65|
| 17|       8|       136|       508(3.74)|       355(2.61)|30.12|

|All|  240823|    897561|   3423676(3.81)|   2739937(3.05)|19.97|

27. unihan-SC-4096 ( SC only or SC+TC mixed )

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|     769|       769|      2717(3.53)|      2378(3.09)|12.48|
|  2|   16065|     32130|    108598(3.38)|     92597(2.88)|14.73|
|  3|   14315|     42945|    139693(3.25)|    116054(2.70)|16.92|
|  4|   48871|    195484|    623650(3.19)|    491073(2.51)|21.26|
|  5|   12135|     60675|    190928(3.15)|    147721(2.43)|22.63|
|  6|   10463|     62778|    196038(3.12)|    146516(2.33)|25.26|
|  7|    3594|     25158|     77931(3.10)|     57412(2.28)|26.33|
|  8|    2373|     18984|     58686(3.09)|     42907(2.26)|26.89|
|  9|    1078|      9702|     29875(3.08)|     21736(2.24)|27.24|
| 10|     934|      9340|     28786(3.08)|     20855(2.23)|27.55|
| 11|     392|      4312|     13279(3.08)|      9612(2.23)|27.62|
| 12|     314|      3768|     11579(3.07)|      8376(2.22)|27.66|
| 13|     144|      1872|      5724(3.06)|      4158(2.22)|27.36|
| 14|     104|      1456|      4455(3.06)|      3226(2.22)|27.59|
| 15|      41|       615|      1868(3.04)|      1348(2.19)|27.84|
| 16|      25|       400|      1219(3.05)|       887(2.22)|27.24|
| 17|       7|       119|       353(2.97)|       264(2.22)|25.21|

|All|  111624|    470507|   1495379(3.18)|   1167120(2.48)|21.95|

28. unihan-TC-4096 ( TC only )

|  N|    FREQ|    N*FREQ|  SUM OF AMCZ(X)| SUM OF LAMCZ(Y)| COMP|
|  1|    3658|      3658|     12240(3.35)|     11521(3.15)| 5.87|
|  2|   41353|     82706|    275870(3.34)|    239559(2.90)|13.16|
|  3|   27020|     81060|    261590(3.23)|    219991(2.71)|15.90|
|  4|   40425|    161700|    515754(3.19)|    411333(2.54)|20.25|
|  5|    8956|     44780|    141492(3.16)|    110988(2.48)|21.56|
|  6|    4665|     27990|     88096(3.15)|     67006(2.39)|23.94|
|  7|    1587|     11109|     34645(3.12)|     26432(2.38)|23.71|
|  8|     709|      5672|     17586(3.10)|     13176(2.32)|25.08|
|  9|     339|      3051|      9444(3.10)|      7147(2.34)|24.32|
| 10|     269|      2690|      8350(3.10)|      6080(2.26)|27.19|
| 11|      82|       902|      2793(3.10)|      2072(2.30)|25.81|
| 12|      84|      1008|      3135(3.11)|      2256(2.24)|28.04|
| 13|      20|       260|       808(3.11)|       593(2.28)|26.61|
| 14|      18|       252|       777(3.08)|       568(2.25)|26.90|
| 15|       9|       135|       415(3.07)|       282(2.09)|32.05|
| 16|       4|        64|       200(3.12)|       139(2.17)|30.50|
| 17|       1|        17|        52(3.06)|        37(2.18)|28.85|

|All|  129199|    427054|   1373247(3.22)|   1119180(2.62)|18.50|

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 19 Oct 2001 12:17:46 +0800

You dont get my point.

Reordering achieve shorter by putting more oftenly used characters in
one block and others at the back. BUT this also means less oftenly used
characters would result in a *LONGER* label then usual. And who are we
to say these less oftenly used characters are less important or worst,
become invalid (too long to fit) because of this reordering?

-James Seng

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "James Seng/Personal" <[email-masked]>; <[email-masked]>;
"Martin Duerst" <[email-masked]>
Sent: Friday, October 19, 2001 12:11 PM
Subject: Re: [idn] call for comments for REORDERING

> differences in achieved improvements. I also reversed the order of the
table itself,but it produced nearly the same result.
> The partial change in the order of the reordering table does not make
big differences. some will lose and some will win. the net effect is
near to zero.
> WITHIN the frequent set. INs and OUTs from 4096 ones are rare and does
not invalidate the validity of most frequent 1024 and 2048 ones.
> Moreover, TC/SC/KC characters are put side-by-side to avoid
countriy-specific biases in han reordering table.
> the most efficient form ). latin script for europeans (0.6 billions)
are the most favored one in ACE-Z. There shoulbe be some conpensations
for

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 13:52:28 +0900

----- Original Message ----- 
From: "James Seng/Personal" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>; "Martin Duerst" <[email-masked]>
Sent: Friday, October 19, 2001 1:14 PM
Subject: Re: [idn] call for comments for REORDERING

> consider NOT to do it at all.

James,
I full-heartedly welcome any change to REORDERING tables from authoritative
organizations. 

But i dont' agree on using ISO 14651  like Mark and Ken.

 REORDERING has no intersection with ISO 14651 (string comparison service), which is about collation of unicode points, how to sort unicode strings.
ISO 14651 has nothing to do with what REORDERING would provide with.

REORDERING exists for reducing successive code distances in ACE encodings.
REOREDERING is based on RENUMBERING code points values by character frequency, 
but ISO14651 takes some variants of lexicographical ordering for various needs
and exists for comparisons , not for supporing subtraction  between
any two code points values.

ISO 14651 is totally irrelvant to REORDERING.

REORDERING tables are hidden in the ACE routines, not for reference 
material for the general public. 
The reordering mappings are a kind of huge tuning paramters set geared for 
ACE.

Tuning parameters need authorities ?  NOT bad idea, but also not neccesary
because frequency based tables are always suboptimal in its nature  however
authoratitive organizations approve and issuse them.

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 14:21:27 +0900

----- Original Message ----- 
From: "James Seng/Personal" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Friday, October 19, 2001 1:17 PM
Subject: Re: [idn] call for comments for REORDERING

> characters would result in a *LONGER* label then usual. 

No. Even For GROUPS of those rare cases, we get Always SHORTER labels than usual.

For example, 
if REORDERING uses a frequent characters set of length 4096 in entire 20992-letter
han ideographic script block, rarely used han letter will be put
into the subblock of length 20992-4096 = 16896 which is 20% shorter than 20992.

let's assume we have with a label consisting of only those characters from 16896 block. without reordering, they are randomly distributed in 20992 letter block.
with reordering, they are in shorter 16896-letter block.
In the latter case, we got shorter successive code distances and shorter
ACE labels than in the former one.

 Even without REORDERING, ACE has favored some of them, and disfavored
others. But, with REORDERING, BOTH groups get shorter ACE labels.
That is the virtue of REORDERING.

Soobok

> And who are we

---

From: Martin Duerst <[email-masked]>
Date: Fri, 19 Oct 2001 15:46:26 +0900

> > not make existing names invalided.

I think this is an extremely important concern.

>In this respect, REORDERING maintains stability over time.

For additions of rare characters, in most cases yes.
But let's assume that these are characters that are not
used in one language (A), but quite frequent in another (B).
People using language (B) might then come forward and ask
for a different reordering to fit their language better.

In general, it's easily possible that the statistics you
currently have favor e.g. the major language that uses
that script, but disfavor another language. It could e.g.
be that the reordering for Arabic makes Arabic names shorter,
but makes Farsi or Urdu names longer. Even if there may
be many more Arabic than Farsi or Urdu names, this would
be quite a bit unfair. And we just don't have enough data
at the moment to be able to say this is not the case.

>unassigned code points in legal IDN labels, because new normalization/case 
>mappings
>would be defined on them in the future. some ACE labels including unsigned 
>code block (tagalog?) might be proven invalid in the future. Nameprep/NFKC 
>Versioning tag schems using new ACE prefix will be needed in the future, i 
>guess.

Yes. But for the majority of really useful characters, in old
and new scripts, it's rather obvious that they will be allowed.
On the other hand, it's totally unclear how to reorder them.

Also, in case of some implementation mistake in Nameprep/NFKC,
in most cases, it will just make a few names unusable, but
not affect the rest. For reordering, a bug will completely
confuse a whole script.

Also, now we have a testbed, and you just think that the testbed
is representative. But once IDN is running, to run a testbed
for a new script will be difficult, because we need the testbed
data for the reordering statistics, but we need the reordering
for the testbed.

>This statements will be included in the final version of REORDERING I-D.

Well, in theory, this works. But I think it's just too unstable
to work in practice.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 19 Oct 2001 17:26:42 +0900

>Current Windows 98,2K,XP and Linux  contain   NFKC codes ?

No. My personal preference would be for NFC, anyway, because
most of the additional mappings in NFKC are not really useful,
because they are from characters that are quite difficult
to type in. So these characters could just be forbidden rather
than mapped. The only exception I know is full-width ASCII,
which could either be included in the nameprep mapping step,
or could be done as a front-end 'quality of implementation'
step (similar to how the ideographic fullstop is currently
treated).

Anyway, it's not so much a question of whether current versions
contain that code (there are libraries that do), but how much
other software will use it. There will definitely be other
software that uses NFC, and quite probably also NFKC.
This is not at all the case for reordering.

>only to that input script blocks .

Yes. But NFC or NFKC boils down to nothing for many subsets,
while reordering doesn't.

>lines in the ACE source code.

The comment lines obviously don't get compiled and
put on a small device.

>comparison on the ACE labels.

It doesn't really help. Transcription and eye-comparison of
ACE labels is a pain. If they are a bit shorter, it still
just hurts.

>in your narrow phone LCD lines of width 16~20, without REORDERING.

Displaying ACE when nothing else is available is acceptable
as a 'best-effort' solution. But it is very far from a good
solution, and designing the rest of IDN for this doesn't
make sense.

I have no problem with IDNA (if it stays at a SHOULD level),
but on a mobile phone, I would probably just use a few question
marks to make clear that there is something that cannot be
displayed.

>characters for ACEed email addresess. Big saves..

For what? For transcribing ACE? Who wants to do that?
Is that what we are designing IDN for? I very much hope not.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 19 Oct 2001 17:34:33 +0900

Hello James,

I agree with most of what you said.

>the IDN purpose but I thought it might be worthwhile to examine.

14651 just doesn't help at all. I think you already said so in
another mail. Sorting and compression are totally different
problems. [Well, except for the Burrows-Wheeler-Transform,
which uses sorting for compression, but doesn't care about
the sorting order used and only compresses long files,
see http://dogma.net/markn/articles/bwt/bwt.htm.] Also,
the algorithm contains a lot of details to work with
various alphabetic sort orders, but leaves Hangul
(which is already in the right order) and Han
(where there are many different orderings) just the
way they are.

Regards,   Martin.

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 17:39:39 +0900

----- Original Message ----- 
> No. What James referred to is that saturation tends to fill up the
> short name slots, and thus flatten the probability distribution.
> I.e. if somebody doesn't get the name they wanted, the chance is
> that they go for something like xq.com, because it's easy to
> remember because it's short. Neither x nor q are very frequent
> letters.

Han/hangeul characters carries meanings while latin alphabets
denote phonemes. Therefore your analogy between latin and han domains
may be false. Chinese people would rather choose to register 
digit-added variants of  alreagy taken desired domains in saturated ML.com,
instead of choosing non-sense irrelevant rare han characters.

Later time, I will provide some proofs that SC and TC only have
small partial set of frequent characters. That's already clear in
SJIS and KSC5601 han characters set which size is less than 5000.

> >INs and OUTs from 4096 ones are rare and does not invalidate the validity 
> >of most frequent 1024 and 2048 ones.
> >Moreover, TC/SC/KC characters are put side-by-side
> 
> Can you explain that better? What about Japanese cases?

KC means "Kanji-specific". ALready addressed. Korea share  TC with TAIWAN.
KC/TC/SC are put side by side in reordering table.

> 
> 
> >to avoid countriy-specific biases in han reordering table.
> >
> No, many other scripts are used for many more languages, with
> quite different usage patterns. (A lot of Han usage in Japan,
> and most of it in Korea, is due to loanwords from Chinese.)
> 

But, even without Urdu consideration in 
arabic reordering, the efficiency of reordering is always  better than
without it, because the lexicographic ordering in un-reordered 
arabic script block can be regarded as *RANDOM* ordering 
in frequency measure (maximum entropy).
Partial reordering (without Urdu consideration) is always better than
no reordering. 
If Urdu text samples  are available, my arabic reordering table may be
improved to reflect them, though.

SOobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 17:48:43 +0900

----- Original Message ----- 
> not affect the rest. For reordering, a bug will completely

Never.
REoredering does not touch on unassigned code points at all.
REORDERING is idempotent  for unassigned code points.

> data for the reordering statistics, but we need the reordering
 
Optimal frequency table does not exist now and future and forever.
sub-optimal solutions are often very useful and accepted and 
REORDERING is just one of them.

Soobook Lee

---

From: Martin Duerst <[email-masked]>
Date: Fri, 19 Oct 2001 17:15:35 +0900

>          frequency variations is marginal.

Well, there are always some variations with marginal influence,
and there are others with much larger influence.

>WITHIN the frequent set.

Well, it's extremely difficult to predict the future. You may be
right, or you may be wrong.

>the most efficient form ). latin script for europeans (0.6 billions) are 
>the most favored one in ACE-Z. There shoulbe be some conpensations for

The main advantage for these scripts is the use of ACE-Z,
when e.g. compared to UTF-8. Reordering is rather minor.

>REORDERING tables cannot be modified if it is once freezed as standards.

Very true.

>as a valid and effective solution for a long time .

Valid and effective for a long time is just something you have
to believe in, there is no way to prove it.

Regards,   Martin.

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 18:24:21 +0900

> while reordering doesn't.

Never. REORDERING tables are modular and groupd by script blocks.
If you never receive and send han domains, 
you can drop in your implementation.

> The comment lines obviously don't get compiled and

You are right. :-)))

Thank you very much  for your interests and comments.

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 18:21:32 +0900

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Friday, October 19, 2001 5:26 PM
Subject: Re: [idn] call for comments for REORDERING

> step (similar to how the ideographic fullstop is currently

My personal preference is NFKC. Because it solves the mappings between
the extended han ideographic block and han ideo block.

> This is not at all the case for reordering.
 
I expect REODERING+ACE may be used also in mailbox name part
in IDN email addressess.
What if the reordering table will find  many other internet
applications in the future like NFC/NFKC. No one knows.

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 18:29:50 +0900

> ACE labels is a pain. If they are a bit shorter, it still

The pain will be everywhere in mobile devices.
That is why The pain-killing drug are in the world.

> marks to make clear that there is something that cannot be
CJK peolple have problems without REORDERING.
Other european folks but You martin (han experts),
won't install han/hangeul fonts even in his/her PCs, i guess.
Most europeans will see leaked ACE labels sometime in the future
when IDN host/email addresses are everywhere and interchanged globally
like LDH ones.  surely,
IDNA WITHOUT REORDERING also become pains for non-CJK folks.

Thanks,

Soobok Lee

---

From: Martin Duerst <[email-masked]>
Date: Fri, 19 Oct 2001 17:58:28 +0900

>instead of choosing non-sense irrelevant rare han characters.

Some really rare and irrelevant han characters may indeed never
be chosen. But still if you want to name a company, there are
many different possibilities, and people will look for short,
not yet used possibilities (which still make some sense)
rather than use longer and longer names.

>SJIS and KSC5601 han characters set which size is less than 5000.

Yes, this is true.

>in frequency measure (maximum entropy).

It's probably not, because most alphabets contain a few
'late additions'. And just using first order frequency
to bring the most frequent characters to the front may
not be the most efficient way for compression.

>Partial reordering (without Urdu consideration) is always better than

I don't deny that you may be able to squeeze out a few bits.
But I don't think that should be the aim of this exercise.

>improved to reflect them, though.

Which might then make it less efficient for Arabic.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 19 Oct 2001 18:23:15 +0900

At 17:48 01/10/19 +0900, Soobok Lee wrote:

> unsigned

Actually, because we do on purpose not prohibit unassigned codepoints
on the client side, it is completely impossible to introduce reordering
for any script that is added later.

The idea behind this is that if e.g. Tagalog gets added to
Unicode, and the IETF decides to add it to the allowed set
of characters for domain names, then the registries that
want to accept Tagalog have to update their software
immediately (no big deal for them), but deployed software
can use Tagalog without having to change nameprep/ACE
(unless they use characters which have to be normalized,
which may happen but will be rare). So existing clients
will already ACE the Tagalog codepoints without reordering,
and if we introduce reordering for Tagalog, there will
be a mess.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 19 Oct 2001 18:09:35 +0900

>REORDERING is idempotent  for unassigned code points.

I didn't talk about unassigned code points.
What I mean is this: If you have two frequency tables,
and they differ by e.g. one forgetting a particular
character, then virtually all your compressed results
are different. If you have a wrong mapping in a
NFC or NFKC algorithm, then only that character is
affected.

>REORDERING is just one of them.

ACE-Z without reordering is also a sub-optimal solution.
I propose that we just use it.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 19 Oct 2001 18:15:30 +0900

>No. Even For GROUPS of those rare cases, we get Always SHORTER labels than 

Sorry, but this is nonsense. You cannot compress all labels
by reordering. It just doesn't work. The argument below is
about as foolproof as a perpetuum mobile design.

In reordering, you will always bring some codepoints closer
together (and names from these will then compress better)
at the expense of moving some farther appart (and thus
reducing compression).

You just have to think about a mixed label, with some
frequent and some not so frequent characters.

Regards,   Martin.

>let's assume we have with a label consisting of only those characters from 
>16896 block. without reordering, they are randomly distributed in 20992 
>letter block.

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 19:23:54 +0900

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "James Seng/Personal" <[email-masked]>; <[email-masked]>
Sent: Friday, October 19, 2001 6:09 PM
Subject: Re: [idn] call for comments for REORDERING

> NFC or NFKC algorithm, then only that character is

Never. The REORDERING table for a script has *NO*
missing characters because it applies idempotent
mappings for even unassigned code points.

For one ACE prefix, one fixed reordering table!

If we should change the frequency table, new versioning
prefix should be introduced to avoid conflicts in the same
time when there is need to make major modifications to 
NFC/NFKC due to erros in them. 
If version 1 has "dq--" prefix, version 2 should have
other ones like "xq--". Reordering tables could be
improved at that time if needed.

SOobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 19:30:00 +0900

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "James Seng/Personal" <[email-masked]>; <[email-masked]>
Sent: Friday, October 19, 2001 6:23 PM
Subject: Re: [idn] call for comments for REORDERING

> will already ACE the Tagalog codepoints without reordering,

maybe problematic and unsafe. 
What if future NFC/NFKC maps them into other code points ?
There will be a mess, too.

> and if we introduce reordering for Tagalog, there will

Current REORDERING does nothing with TAGALOG and adds
no new problem into ACE. the problem is alerady inherent
in ACE and nameprep version scheme, and is not due to
REORDERING.

Prefix-based version scheme will solve those problems.

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 20:05:58 +0900

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "James Seng/Personal" <[email-masked]>; <[email-masked]>
Sent: Friday, October 19, 2001 6:15 PM
Subject: Re: [idn] call for comments for REORDERING

> about as foolproof as a perpetuum mobile design.

Never non-sense. You have misunderstandings.

See the word "GROUPS" in the above statement. 
I didn't assert them on  "every individual labels".

> at the expense of moving some farther appart (and thus

That's true for rare individual cases, but not in overall results, because
The expense in latter part is much less than the gain in former part.
The net gain is always positive for even rare groups of characters.

> frequent and some not so frequent characters.

Sure. REORDERING also works for these kinds of lables.

Frequent charc: f1 f2 f3 f4 ...
Rare  chars   : r1 r2 r3 r4
We have a mixed reordered label:  f1 r1 f2 r2 f3 r4 . com
AMC-Z sort it into :  f1 f2 f3 f4 r1 r2 r3 r4 . com
AMC-Z encode them  :  label <f1 f2 f3 f4>  + junction overhead <f4 r1>+ label <r1 r2 r3 r4>

You know reordering works well for label <f1 f2 f3 f4 > and label <r1 r2 r3 r4>.

Soobok lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 19 Oct 2001 22:05:36 +0900

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "James Seng/Personal" <[email-masked]>; <[email-masked]>
Sent: Friday, October 19, 2001 5:58 PM
Subject: Re: [idn] call for comments for REORDERING

> rather than use longer and longer names.
In most cases, they add latin digits. CJK people would know what i am saying.

> 'late additions'.
 
If and only if the reordering table for a script needs
modifications for added characters, it can be done in the
next version of nameprep/ACE with new ACE prefix.

> And just using first order frequency

Do you a good idea that can replace current first-order frequency reordering ?
Welcome any changes to that.

If someone devise new ordering scheme in the future, that may substitute 
current reordering scheme in the next namepre/ACE version with
new ACE prefix.

> Which might then make it less efficient for Arabic.

Yes, but marginally.

---

From: "James Seng/Personal" <[email-masked]>
Date: Sat, 20 Oct 2001 01:31:29 +0800

> There will be a mess, too.

read the design principle of NFC or NFKC. Future addition of scripts and
addition normalization will *NOT* cause existing normalized string to
changed.

do you have the same stability principle or ensure that to be the same
for re-ordering? or one which can be sure?

> in ACE and nameprep version scheme, and is not due to

But Tagalog may then come back to IDN to claim unfair treatment!
Are you going to deny them that?

> Prefix-based version scheme will solve those problems.

No no no. Lets not get into other scheme which involves playing with
multiples prefix or suffix as "version tag". Try not to make things more
complex here.

The key word here is stability. You have not address it yet.

And no, there is no such thing as "only compress" algorithm. It is
mathemtically impossible.

-James Seng

---

From: "James Seng/Personal" <[email-masked]>
Date: Sat, 20 Oct 2001 02:34:46 +0800

> CJK peolple have problems without REORDERING.

I like to see some quantify prove for this.

as a chinese i am quite happy without reordering. (in fact, i am already
happy with utf5 15 label limtation but thats another story). nor have i
heard verisign (or other registries) registration stop due to lack of
reordering for CJK.

> IDNA WITHOUT REORDERING also become pains for non-CJK folks.

so u mean a "zq--dskjssdljdssdds32" is better than
"zq--jdzpzxcklxx9jspsddsdsd" to naked eye? I cant believe this
statement.

-James Seng

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Fri, 19 Oct 2001 14:51:30 -0700

Apologies for taking so long to comment on this thread. However, 
given the comments from the last 48 hours, it is clear that comments 
from more people are needed.

The reordering proposal and its proponent are based on the faulty 
premise that there will be a need for very long domain names for the 
scripts that have been reordered. The fact that there are very long 
names in the Asian languages is irrelevant because there is no 
assumption that anyone would use those long names in domain names. 
The author seems to think that domain names have the same properties 
and regular names, which is of course wrong.

It is very troubling that the proposal's author pretty much ignores 
requests for clarifications with statements like "if you were a CJK 
person, you would know this". He has said that to people who are, in 
fact, well-known CJK experts. This is not the way that good standards 
are created.

So far, there has been no significant support for the proposal, and a 
lot of people objecting. I am in the latter category for the simple 
reason that the proposal adds complexity and overhead to fix a 
non-existent problem.

The WG should ignore this proposal unless the author can respond to 
questions and concerns from the rest of the WG.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 19 Oct 2001 23:23:52 +0000

Soobok Lee <[email-masked]> wrote:

> Even For GROUPS of those rare cases, we get Always SHORTER labels than
> usual.

As has already been pointed out, this is impossible.  Here is one class
of labels that are made longer by reordering:  Labels that use lots of
"uncommon" code points from the bottom 2K (or whatever) of the block.
Without reordering, these code points are already close together, but
with reordering they will be scattered all over the block, resulting in
a longer ACE.  Of course such labels are supposed to be very rare.

All compression schemes must make some inputs shorter and some longer.
The goal is to makes the more common ones a lot shorter, and the very
rare ones only slightly longer.

> That ACE encoding overhead is so big that it cannot be compensated by
> the superiority in the information capacity of a han letter compared
> to a latin letter.

That's not obvious.  Here are the AMC-ACE-Z encodings of the same
sentence in English and Chinese (from the Examples section of the
AMC-ACE-Z spec):

              English: WhyCantTheyJustSpeakChinese
 Chinese (simplified): ihqwcrb4cv8a8dqg056pqjye
Chinese (traditional): ihqwctvzc91f659drss3x8bo0yb

Of course this is only one data point, but my hunch is that it's not
atypical.

Paul Hoffman / IMC <[email-masked]> wrote:

> scripts that have been reordered.

Actually, he has said many times that his goal is not only to allow long
domain names, but also to make average domain names easier for humans to
cope with when they are forced to see the ACE.  (And others have argued
that neither goal is important enough to justify the complexity.  I'd
like to know if the CJK NICs have opinions on the matter.)

AMC

---

From: "Kenny Huang" <[email-masked]>
Date: Sat, 20 Oct 2001 09:44:52 +0800

as a chinese i am quite happy with 8-bit clean DNS, nor have i heard TWNIC or
CNNIC registration stop due to BIG5/GB stored in the resource records. Besides,
both NIC provide resolution service, one step ahead Verisign service. 
If the discussion is based on this kind of scenario, that will be an endless loop.

> "zq--jdzpzxcklxx9jspsddsdsd" to naked eye? I cant believe this

These encoding are transparent to USER (NAKED EYE). Of course you should know it.

Kenny Huang

---

From: David Hopwood <[email-masked]>
Date: Sat, 20 Oct 2001 02:30:37 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Soobok Lee wrote:
> "Martin Duerst" <[email-masked]> wrote:
> > Soobok Lee wrote:

There are no compatibility mappings between any Han ideographs, either
in Extensions A and B or in the original U+4E00 block. If you're referring
to the CJK Compatibility Ideographs block, those are canonical mappings.

(There are some compatibility mappings from CJK *radicals* to
CJK ideographs. Those will be discussed in my post on NFC vs. NFKC.)

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: David Hopwood <[email-masked]>
Date: Sat, 20 Oct 2001 01:16:23 +0100

-----BEGIN PGP SIGNED MESSAGE-----

> step (similar to how the ideographic fullstop is currently

I absolutely agree with this - I've been preparing a detailed
rationale for using NFC in preference to NFKC, that considers all
the categories of compatibility mappings. I'll post it tomorrow.
An important consideration is that by disallowing a larger set of
characters, we can avoid the need for "additional foldings" - i.e.
it is possible to just apply NFC and the standard Unicode case
folding, in either order, which strengthens your argument above
about being able to use existing libraries.

> solution, and designing the rest of IDN for this doesn't

A better solution for cases where a name can't be displayed in
its original script, is to use a transliteration (which unlike an
ACE encoding, has a chance of actually meaning something to end-
users). I have a proposal for a new RR that allows translations
and transliterations of a name to be looked up, and provides a
way to encode the language and script of each name. This is a new
feature that is independent of the rest of IDN; it doesn't break
the basic model of DNS as an untagged namespace.

- --

---

From: David Hopwood <[email-masked]>
Date: Sat, 20 Oct 2001 02:41:54 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Soobok Lee wrote:
> denote phonemes.

?? Unless I'm very confused about Hangul, it is at least as much
phonetically-based as Latin. Hangul Jamo are letters of an alphabet,
which happen to be arranged in square cells corresponding to syllables,
instead of linearly.

Moreover, each Hangul syllable (encoded as a single character when
NFC/NFKC-normalised), normally represents 3 Jamo. That should be taken
into account when assessing whether Hangul is encoded compactly enough.

> Therefore your analogy between latin and han domains
> digit-added variants of already taken desired domains in saturated
> ML.com, instead of choosing non-sense irrelevant rare han characters.
> 
> small partial set of frequent characters.

That's not in dispute. The argument is about whether the complexity of
reordering is worth the additional compression. IMHO it isn't -
AMC-Z (or UTF-8) encodings are sufficiently compact that the 63-octet
and 255-octet limits are not a serious problem for any language or script,
and the savings for average names are marginal.

- --

---

From: "Soobok Lee" <[email-masked]>
Date: Sat, 20 Oct 2001 21:41:13 +0900

----- Original Message ----- 
From: "Adam M. Costello" <[email-masked]>
 
> rare ones only slightly longer.
 
Exactly right! 
That is what i meant and already answered so in my previous article
you may have missed. Look at uppercased "GROUPS". "rare" characters
are ALL the ones not in the frequent set.

To make it more concrete, I took an experiment.
For 240823 han ML.com labels, I got the following statistics.

From now one, I will focus on only one question at a time
for productive discussions. 
I won't miss other pending questions. Please be patient.

Please comment on the next result.

Soobok Lee
====================================================================

AMCZ < LAMCZ :   4764    (Reordering got longer labels for 1.9% han ML.com )
AMCZ = LAMCZ :  24553
AMCZ > LAMCZ : 211506
Total:         240823

AMCZ < LAMCZ : # of overhead chars for each label length N

  N   labels   +overhead

  1      141   +1.00
  2     2840   +1.13
  3      948   +1.15
  4      675   +1.19
  5      107   +1.18
  6       32   +1.53
  7       12   +1.33
  8        8   +1.38
  9        1   +2.00

AMCZ > LAMCZ : # of saved chars for each label length N

  N   labels   -saved

  1     1199   -1.00
  2    42119   -1.32
  3    35280   -1.88
  4    85233   -2.79
  5    20589   -3.59
  6    15009   -4.71
  7     5151   -5.58
  8     3069   -6.58
  9     1412   -7.39
 10     1201   -8.49
 11      474   -9.26
 12      397   -10.28
 13      164   -10.86
 14      122   -11.79
 15       50   -13.06
 16       29   -13.55
 17        8   -13.00

For N>9, we  always get shorter ACE labels.
For each 1=<N<=9, the net gain is always positive.

27. unihan-4096

The next part is hangul ML.com experiment.

AMCZ < LAMCZ :    697  ( 0.3 % )
AMCZ = LAMCZ :   6933
AMCZ > LAMCZ : 198674
Total:         206304

AMCZ < LAMCZ :

  N   labels   +overhead

  2      463   +1.09
  3      175   +1.07
  4       52   +1.25
  5        5   +1.00
  6        2   +2.00

AMCZ > LAMCZ :

  N   labels   -saved

  2    13491   -1.41
  3    38086   -2.09
  4    61900   -3.14
  5    39612   -4.19
  6    23880   -5.29
  7    12447   -6.41
  8     5441   -7.40
  9     2262   -8.46
 10      895   -9.45
 11      373   -10.48
 12      141   -11.17
 13       77   -12.30
 14       32   -13.12
 15       20   -14.05
 16       10   -12.30
 17        7   -15.86

9. hangul-1024

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sat, 20 Oct 2001 11:42:34 -0700

At 11:23 PM +0000 10/19/01, Adam M. Costello wrote:
>cope with when they are forced to see the ACE.

You are right, he has said that, but it is so absurd that I ignored 
it. A ten-character string of ASCII gibberish is as hard to "cope 
with" as a twelve-character string of ASCII gibberish. (These are 
approximately the lengths you would get for a name part that consists 
of four Han characters.)

--Paul Hoffman, Director

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 21 Oct 2001 10:43:40 +0900

----- Original Message ----- 
From: "David Hopwood" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Saturday, October 20, 2001 10:41 AM
Subject: Re: [idn] call for comments for REORDERING

> which happen to be arranged in square cells corresponding to syllables,

You are only partly correct in that Hangul is phonetic.

If you ever read a hangul-to-hangul dictionary, you can find easily that
over 70~80% of modern hangul vocabularies came from 1:1 mappings of
Chinese words like  most english & french words  came from latin ones.
Therefore, one hangul character carries similar amount of information 
with its chinese character counterparts.

hangul/han both carries as much information  as  about 2 latins characters.

> -----BEGIN PGP SIGNATURE-----
> Version: 2.6.3i
> Charset: noconv
> 
> iQEVAwUBO9DO2zkCAxeYt5gVAQF/7AgAzp3KB/kPA2XAxb43hCSbrLBOxavd4WSq
> DYfvw2UuwloLkEZB+tkkoOPucW/ElLmaYjuYMKt6nea2LZthLpTWDc8a8ENXqM34
> Z+aP8nqN9XzeMTPisebpCcTE7PZYWdi87a0grmL0KFBzYG0PsxAB905Yvf12oU4U
> u3da6Ku37YJeYK0jNi4/qhoAUZ8gyz+gW4MWWxCmuAIrvmIkaf/d4lX4Tu+75mg2
> VcS3ezCGbOt3Wf0GIfUl869BBRbPB7bScBX0EjP/C+sQpCVR6gVs6SKDS9zY/W6k
> XImrf7IuLg57za70dy5YiCgNBYOvlNa4Xgi3d+DFoW7jntmj4MEUYw==
> =4Lmr
> -----END PGP SIGNATURE-----
> 
>

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 21 Oct 2001 12:34:25 +0900

----- Original Message ----- 
> hangul/han both carries as much information  as  about 2 latins characters.

This estimation came from the fact :
  han labels of length N (==8)  have  reordered ACE lables of length 2.27*N
  and hangul labels have  2.17*N , while with bare AMC-ACE-Z , they have 3.1 * N.
  Reordered Greek  labels have 1.2*N for N>16.

hangul : han: Greek   = 2.17 : 2.27 : 1.2

 The amount of information in han/hangul  labels is proportional to the number of corresponding LDH characters in ACE labels?  Maybe false measure from
misconceptions because most han/hangul labels from  nouns or composite nouns.
 I won't use the term  "amount of carried information " from now on..

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 21 Oct 2001 22:46:29 +0900

----- Original Message ----- 
From: "James Seng/Personal" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>; "Martin Duerst" <[email-masked]>
Sent: Saturday, October 20, 2001 2:31 AM
Subject: Re: [idn] call for comments for REORDERING

> addition normalization will *NOT* cause existing normalized string to

My  question was that:
  1) newly-approved TAGALOG characters X,Y  have   NFC X -> Y,
  2) old Nameprep/ACE encodes X.com and Y.com as two distinct ACE labels,
      because it don't know about NFC X -> Y
  3) new Nameprep/ACE encodes X.com as the the same ACE label as Y.com's 
      with NFC  X -> Y.

  If X.com and Y.com are taken by two distinct registrants, there will
    be a mess. of course, it should have be blocked by careful registries.
  Even if new ACE libaries are distributed, some old ACE tools will still
    try to send X.com dns queries which may fail.

> for re-ordering? or one which can be sure?

yes. Like old nameprep bypasses unassigned code points for future TAGALOG script,
REORDERING bypasses,too. No ACE improvement for taglog now.
See below.

> Are you going to deny them that?

Are you saying ?:
   Tagalog folks may come back  with complaints about old nameprep 
   which has no NFC X->Y, saying "unfair treatment!" ??

Before TAGALOG is added, current nameprep/REORDERING have 
no reponsibility for inevitable missing supports of them.

Now I will propose two solutions.

1) last resort: somewhat tricky

Future TAGALOG may provides two sets of TAGALOG basic alphabets.
One set A in official lexicographical ordering and the other set B 
is in frequecy ordering (sub-optimal one OKAY) with 1:1 NFKC defined 
from A onto B.
Then all tagalog basic alphabets in Set A will be "reordered by NFKC"
in nameprep , not in ACE. A and B share the same font.
Then valid ACE labels of TAGALOG script only contains characters from B.
This may have some problems in comparisons which i have no full analysis.

2) UTC solution

IF UTC accepts REORDERING as an official normalization form like 
NF-REORDERING , then we need no such tricks like above, and
TAGALOG support can be done within that NF in the new 
NAMEPREP steps:  mapping/NFKC/PROHIBIT and then NF-REORDERING .

Frequency tables are always sub-optimal in its nature, and 
marginal frequency fluctuation will occurs to make marginal 
efficieny changes, but in most cases, i am sure, it will benefit
most of TAGALOG labels, and that's why i push REORDERING into UTC. 

In conclusion, 2) is the my preferenece.
It's the best way for acquiring stability and authority and applicability
of REORDERING.

> The key word here is stability. You have not address it yet.

Answered. with the above solutions we need no version tag..

> mathemtically impossible.

Already answered in the separate thread "[idn] REORDERING : makes labels shorter or longer  ?".

Soobok Lee

---

From: "James Seng/Personal" <[email-masked]>
Date: Sun, 21 Oct 2001 22:22:55 +0800

Firstly, lets not try to twist the whole argument to Normalization,
which you are doing here. We are talking about Reordering so lets focus
on it.

>   3) new Nameprep/ACE encodes X.com as the the same ACE label as
Y.com's

My shortest answer is for you to read UTR#15. It is already addressed
there.

>     try to send X.com dns queries which may fail.

No it wont. Patrik Faltstorm have already discuss this possiblities and
the work around this. It was one of the version of Nameprep but I
realised it is no longer in -06.

>    which has no NFC X->Y, saying "unfair treatment!" ??

No, I am saying REORDERING. Lets keep the topic on REORDERING.

> This may have some problems in comparisons which i have no full
analysis.

Okay. This should go into ISO10646 SC2/WG2 then.

> It's the best way for acquiring stability and authority and
applicability

Okay. This should go into UTC.

Both your solution unfortunately cannot be done in this WG.

-James Seng

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 21 Oct 2001 23:34:10 +0900

----- Original Message ----- 
From: "James Seng/Personal" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>; "Martin Duerst" <[email-masked]>
Sent: Sunday, October 21, 2001 11:22 PM
Subject: Re: REORDERING: stability issues and UTC solutions

> My shortest answer is for you to read UTR#15. It is already addressed

I read http://www.unicode.org/unicode/standard/policies.html and
http://www.unicode.org/unicode/reports/tr15/#3.1_Corrigendum.

Would anyone suggest more ?

Soobok Lee

> Okay. This should go into ISO10646 SC2/WG2 then.

YEs. When New script is added.

Soobok lee

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 21 Oct 2001 08:03:32 -0700

>realised it is no longer in -06.

It is in stringprep-00, which is where it should be.

--Paul Hoffman, Director

---

From: "Soobok Lee" <[email-masked]>
Date: Mon, 22 Oct 2001 10:24:17 +0900

We have no TAGALOG characters X,Y now. 
After we finalized nameprep/ACE,
TAGALOG script may be added to Unicode.
Then,
 
>   2) old Nameprep/ACE encodes X.com and Y.com as two distinct ACE labels,

unassingned code points are allowed in query from old version nameprep.
X.com Y.com are passed through into DNS servers which have no stored tagalog
labels before TAGALOG is added, but begin to accept it after it's approved & added.

>   3) new Nameprep/ACE encodes X.com as the the same ACE label as
>       Y.com's       with NFC  X -> Y.
> 
this case is described in stringprep-00, section 6.3 paragraph 3.
that query should fail. Old nameprep/ACE applications should send
Y.com instead of X.com even when it gets native X.com from html page
(human input should contain only Y.com, not X.com). 
See the *HAS TO* sentence below.
In such case, end users should upgrade their old applications.
Until upgradea are finishied, someone would have complaints 
for such inconveniences.

****
Newer stored string -- Suppose that an requesting application is using
oldVersion and the stored string was created using a profile that uses
newVersion. Because the requesting application passed through any
unassigned code points, the user can query on stored strings that use
code points in newVersion. No stored strings can have code points that
are unassigned in newVersion, since that is illegal. In this case, the
querying application *HAS TO* enter the unassigned code points in the
correct order, and *HAS TO* to use unassigned code points that would make it
through both the mapping and the normalization steps.
*****
 
Soobok Lee
==============================================

http://www.ietf.org/internet-drafts/draft-hoffman-stringprep-00.txt

6.2 Reasons for difference between stored strings and requests

Different software using different versions of a stringprep profile need
to interoperate with maximal compatibility. The scheme described in this
section (stored strings MUST NOT contain unassigned code points,
requests MAY include unassigned code points) allows that compatibility
without introducing any known security or interoperability issues.

The list below shows what happens if a request contains a code point
from category U that is allowed in a newer version of a profile. The
request either matches the string that was intended, or matches no
string at all. In this list, the request comes from an application using
version "oldVersion" of a profile, the stored string was created using
version "newVersion" of the same profile, and the code point X was in
category U in oldVersion, and has changed category to AO, MN, or D.
There are 3 possible scenarios:

1. X is assigned to AO -- In newVersion, X is in category AO. Because
the application passed X through, it gets back a correct match with the
stored string. There is one exceptional case, where X is a combining
mark.

The order of combining marks is normalized, so if another combining mark
Y has a lower combining class than X then XY will be put in the
canonical order YX. (Unassigned code points are never reordered, so this
doesn't happen in oldVersion). If the request contains YX, the request
will get correct match with the stored string. However, no string can be
stored with XY, so a request with XY will correctly get a negative
answer to the test for matching.

2. X is assigned to MN -- In newVersion, X is normalized to code point
"nX" and therefore X is now put in category MN. This cannot exist in any
stored string, so any request containing X will correctly get a negative
answer to the test for matching. Note, however, if the request had
contained the letter nX, it would have correctly matched.

3. X is assigned to D -- In newVersion, X is in category D. This cannot
exist in any stored string, so any request containing X will correctly
get a negative answer to the test for matching.

In none of the cases does the request get data for a stored string other
than the one it actually tried to match against.

The processing in this document is always stable. If a string S is the
result of processing on newVersion, then it will remain the same when
processed on oldVersion.

6.3 Versions of applications and stored strings

Another way to see that this versioning system works is to compare what
happens when an application uses a newer or older version of this
document.

Newer query application -- Suppose that a requesting application is
using version newVersion and the stored string was created using version
oldVersion. This case is simple: there will be no characters in the
stored string that cannot be requested by the application because the
new profile uses a superset of the code points used for making the
stored string.

Newer stored string -- Suppose that an requesting application is using
oldVersion and the stored string was created using a profile that uses
newVersion. Because the requesting application passed through any
unassigned code points, the user can query on stored strings that use
code points in newVersion. No stored strings can have code points that
are unassigned in newVersion, since that is illegal. In this case, the
querying application has to enter the unassigned code points in the
correct order, and has to use unassigned code points that would make it
through both the mapping and the normalization steps.

7. Security Considerations

The Unicode and ISO/IEC 10646 repertoires have many characters that look
similar. In many cases, users of security protocols might do visual
matching, such as when comparing the names of trusted third parties.
Stringprep does nothing to map similar-looking characters together.

---

From: Martin Duerst <[email-masked]>
Date: Mon, 22 Oct 2001 11:45:32 +0900

>labels before TAGALOG is added, but begin to accept it after it's approved 

That makes a lot of sense. Or do you want to say it doesn't.

>(human input should contain only Y.com, not X.com).

Well, it shouldn't get X.com from the webpage, because the
web page should also be normalized. Please see
http://www.w3.org/TR/charmod :-).

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Mon, 22 Oct 2001 12:11:56 +0900

>with its chinese character counterparts.

First, the mappings are not 1:1. There are many Hanja that map to
the same Hangul (same pronunciation), and there are some Hanja
that have two or more pronunciations.

Second, the fact that much of Korean vocabulary (same for
Japanese) came from Chinese doesn't disprove that Hangul is
phonetic, in the same way as the fact that Much of English
or French came from the Latin language doesn't disprove that
the Latin script is phonetic.

>hangul/han both carries as much information  as  about 2 latins characters.

For han, Adam has shown that this easily can go up to 4.
By your arguments, this should be the same for Chinese-
origin Hangul. I would agree, and say that for names,
I expect a higher percentage of Chinese-origin Hangul
than for Korean running text.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Mon, 22 Oct 2001 11:49:49 +0900

>mappings for even unassigned code points.

What I meant is that e.g. somebody goes ahead and
implements reordering based on one of your tables.
Then e.g. the IETF includes reordering in IDN
(Don't misunderstand me, this is only an example,
of course I think this should never happen.), but
with a different table based on newer statistics.
The original implementer doesn't catch that he has
a different table, and as a result, we have total
chaos, nothing matches.

In comparison, if somebody by chance gets an old
NFC/KC table, a few labels won't match, but it won't
be a whole script or more that gets ruined.

Regards,   Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 22 Oct 2001 07:29:57 +0200

--On 2001-10-22 11.45 +0900 Martin Duerst <[email-masked]> wrote:

>> X.com Y.com are passed through into DNS servers which have no stored
>> tagalog labels before TAGALOG is added, but begin to accept it after
>> it's approved  & added.
> 

I would appreciate if discussions on versioning uses the terminology etc
which is already established in earlier versions of the nameprep document,
and now moved to section 6 of draft-hoffman-stringprep-00.txt.

I especially see an embryo of reinvention of an already in this wg
established way of handling unassigned codepoints.

That this is handled the correct way is especially important as that
section of the stringprep/nameprep algorithm is in accordance with
agreements between IETF and UTC on how additional codepoints are assigned.

  paf

---

From: Martin Duerst <[email-masked]>
Date: Mon, 22 Oct 2001 12:35:52 +0900

>Welcome any changes to that.

Well, just for the record, if a script is used with two languages,
there are some characters used in both, others used only in one
language, and still others only used in the other, then it may
be better to arrange them to first have those characters used
in one language only, then those used in both, and then those
used only in the other language.

Although 'language' is not the right term, that would very clearly
apply to TC/SC.

Regards,   Martin.

---

From: "Soobok Lee" <[email-masked]>
Date: Mon, 22 Oct 2001 19:53:03 +0900

----- Original Message ----- 
> a different table, and as a result, we have total

But newer reordering table with newer statistics
cannot be deployed without new ACE PREFIX, as i said many times.

But current nameprep rules out the possibility of
newer PREFIX. Therefore, newer reordering table would not
occur until this objection to future new prefix is
un-freezed and lifted up by IETF.

With new prefix, there won't occur any confusion problems you raised
regarding to REORDERING.

Soobok Lee

---

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 22 Oct 2001 21:39:50 +0800

Wait wait..new prefix??

You are going down a path whereby a *single* domain name may have *X*
number of ACE, each one with a different prefix, depending on the
version number of re-ordering & nameprep. Great, please explain how the
client going to do matching with different version, or client & server
without some form of communication?

No, going to multiple prefix is a BAD *BAD* **BAD** *B*A*D* idea. Lets
not even go there.

-James Seng

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: <[email-masked]>; "Martin Duerst" <[email-masked]>
Sent: Monday, October 22, 2001 6:53 PM
Subject: Re: [idn] call for comments for REORDERING

----- Original Message -----

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 23 Oct 2001 03:10:15 +0900

----- Original Message ----- 

> number of ACE, 

Yes,it may happen, but  only if we do it loosely.

To treat unassigned code points more consistenly and safely 
through out  all nameprep upgrade paths with/without reordering,
i propose the next new prefixing scheme with two prefices: uq-- and zq--.

Let's assume we adopt "zq--" prefix for valid ACE labels.
I suggest we use another "uq--" prefix for invalid ACE labels which contains at least one unassigned code point at one time. When the unassigned code points become *defined* later, new nameprep/ACE libraries may encode them using "zq--", no more with "uq--". Then we could devise some fallback mechanims on all "uq--????" labels coming from old nameprep/ACE   in application sides  receiving the uq-- labels.

all 'zq--????' labels can be trusted if registries
are careful enough to enforce the integrity.

All "uq--????" labels should not be trusted and all application/dns servers should
have some fallback/warning mechanisms form them. 

Case 1) a newly assigned code point don't have any NFC/NFKC/Case mapping:
Then, the native label containing it would be ACE-encoded into :
"uq--abcd" from old nameprep/ACE, and 
"zq--efgh" from new nameprep/ACE.
then  the authoritative dns server  for the native label,
of course, sets "uq--abcd"  to be an alias to the other "zq--efgh".
 
Case 2) a newly assigned code point is mapped into other code point by one
of NFC/NFKC/CaseMapping ( X->Y ):
Then the native label would have related three different ACE labels:
"uq--abcd" from old nameprep/ACE for X and 
"uq--ijkl" from old nameprep/ACE for Y and 
"zq--mnop" from new nameprep/ACE for Y.
then  the authoritative dns server  for the native label, should
set "uq--abcd" and "uq--ijkl" to be  aliases to the right "zq--mnop".
The number of such aliased labels would be combinatorically multiplied 
if the native label contains more than 1 mapped characters.

no dns server upgrade is needed. additional zone adminstration is enough.

But, the applications should issue warnings for all occurrences of "uq--xxxx" for security reasons, because all old(out-dated) online/offline-mode applications may fail to match "uq--abcd" and "uq--ijkl" and "zq--mnop" in Case 2) which should be regards as equal.
 
certificate authories should not issue website/email
certificates applied with identity information containing "uq--abcd".
Applicants should be warned to retry with upgraded software to 
generate their new email/website ACE addresses with "zq--" prefix.

If old email clients receives senders' addresses with "uq--",
then they should give warnings to end users that something dangerous
string may be in it. If the email clients is upgraded to know both old and new nameprep, the uq-- label is decoded into native label and then re-encoded into zq--
label with new nameprep/ACE and can try successful comparisons!!!
This may be the unique feature that  cannot be provided by current nameprep
versioning scheme.

Without this  "two prefices scheme", online/offline applications ( dns servers  are
adminstrated by intelligent and careful zone masters) may not know
whether "zq--????" labels are from dangerous old nameprep or new nameprep, and
don't know whether "zq--????" is equal to "uq--????" from old nameprep/ACE.

These new prefixing scheme ,of course, cannot be used to correct false NFC/NFKC.
But it provide more security.

Moreover,it may used to incorporate new reordering tables for newly approved scripts/newly assigned code points  even with single "zq--" prefix, whenever the native labels contain at least one of previously-unassigned-but-now-added code block for which we can make new-statitiscs-backed new reordering tables that cover only the code ranges consisting of the new added code points. Old reordering tables don't touch the unassigned code points. There  will be no collision because new zq-- labels will be unique in that old zq-- labels don't contain new code points. "uq--" has no reordering support for unassigned code points, only "zq--" does. "uq--" labels are always unique, too.

halfbaked idea. welcome any corrections and suggestions.

Soobok Lee

> each one with a different prefix, depending on the

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 23 Oct 2001 08:56:18 +0800

Lee,

Other may be too polite to say this but this proposal is so absurb that
I don't even know where to start.

[snip section on two prefix]
> halfbaked idea. welcome any corrections and suggestions.

You got this part right tho.

Suggest you go work on it further. Do not think of this as a problem of
"how can I get reordering into IDN" and forget the other issues of
general use of domain names, behavior and stability.

-James Sen

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 22 Oct 2001 20:20:49 -0700

My only comments are:

1. Like others, I believe that it is unnecessary.
2. If it were adopted, to ensure stability of IDN over time, the reordering
should absolutely be permanent -- never changed in the future, even if new
characters are added. This is not an onerous restriction, since reordering
is only a permutation of code points for the purpose of compression, and
neither adds nor removes information.

Mark

—————

Δός μοι ποῦ στῶ, καὶ κινῶ τὴν γῆν — Ἀρχιμήδης
[http://www.macchiato.com]

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 09, 2001 21:11
Subject: [idn] call for comments for REORDERING

Hi, all

I expect many feedbacks from non-US and non-European participants and
observers.

Thanks.

Soobok Lee

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Tue, 23 Oct 2001 00:14:18 -0400

> I don't even know where to start.

It is the job of a chair of an IETF WG have access to sufficient expertise
to "know where to start", and usually it is a plus if the chair is civil.

> You got this part right tho.

When a contributor to the IETF sollicits "corrections and suggestions",
he or she is working in the spirit of the IETF, which fundamentally is
a body which coordinates and harmonizes "corrections and suggestions"
(aka "interoperation") between network infrastructure implementors through
a formal specification process we refer to as "drafting protocols".

Anyone who fails to understand specification of at least two independent
interoperable implementations from different code bases, with sufficient
successful operational experience, is the core criteria for an IETF memo
to be a draft standard, fails also to understand that only constructive 
engagement between implementors leads to an interoperable outcome. There
is no place for non-constructive "wit", particularly in a chair.

> general use of domain names, behavior and stability.

Exactly how does transformations on code-blocks at some point within the
proposed framework (e.g., unicode, nameprep, reordering (tc,sc,jc,kc), 
amc-z|utf-8) have a non-interoperable manifestation as "general use of
domain names, behavior and stability"?

Eric

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Tue, 23 Oct 2001 00:42:51 -0400

James,

I read Soobok's proposal last week.

Eric

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 23 Oct 2001 12:51:22 +0800

Read his proposal dated today (23th Oct) please.

http://www.imc.org/idn/mail-archive/msg04332.html

-James Seng

----- Original Message -----
From: "Eric Brunner-Williams in Portland Maine" <[email-masked]>
To: "James Seng/Personal" <[email-masked]>
Cc: "Eric Brunner-Williams in Portland Maine" <[email-masked]>;
"Soobok Lee" <[email-masked]>; <[email-masked]>;
<[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2001 12:42 PM
Subject: Re: [idn] Re: suggestion: two prefices scheme for unassigned
code points treatments.

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Tue, 23 Oct 2001 00:46:33 -0400

Mark,

...
> characters are added. ...

This is a different issue than the one raised by Erik Nordmark, which has to
do with WG process (exit criteria), and iterations of proposals to consider
reordering.

Does the same stability of IDN over time requirement lead to the same
conclusion for other transformations of blocks of code-points?

What is the "stability of IDN over time" requirement more specifically, and
in particular to reordering(s) that are not "absolutely [] permanent"? If
this question has already been answered, feel free to point to the item in
the archive where the answer can be found.

Eric

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 23 Oct 2001 12:39:55 +0800

Normally I wont bother with a reply on this, but:

1. When I speak as a co-chair, I will normally do so. Until then, I am
   a wg member like everyone else, entitled with my opinion. Do not
   confuse my various hats. I have too many of them, enough for you
   put my words into other context which is inappropriate.

2. Suggest you read the original proposal on two prefix *first* then
   maybe you could post something more technical constructive. Obviously
   from your questions, you have not even readLee's original proposal on
   two prefix.

-James Seng

----- Original Message -----
From: "Eric Brunner-Williams in Portland Maine" <[email-masked]>
To: "James Seng/Personal" <[email-masked]>
Cc: "Soobok Lee" <[email-masked]>; <[email-masked]>;
<[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2001 12:14 PM
Subject: Re: [idn] Re: suggestion: two prefices scheme for unassigned
code points treatments.

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Tue, 23 Oct 2001 01:08:34 -0400

James,

You are correct, you have a right to participate as an individual contributor,
not only as a co-chair.

That said, your reply to Soobok had no technical content and is wasted bits
in everyone's mail queue.

Eric

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 23 Oct 2001 13:13:30 +0800

I accept that if that is your opinion of my comments.

-James Seng

---

From: Martin Duerst <[email-masked]>
Date: Tue, 23 Oct 2001 12:04:16 +0900

Hello Soobok,

I'm speaking about a BUG, without changing the prefix.
Bugs happen, unfortunately.

Regards,   Martin.

---

From: David Hopwood <[email-masked]>
Date: Tue, 23 Oct 2001 08:17:15 +0100

-----BEGIN PGP SIGNED MESSAGE-----

David Hopwood wrote:
> ... I've been preparing a detailed rationale for using NFC in
> preference to NFKC, that considers all the categories of compatibility
> mappings. I'll post it tomorrow.

Took me a bit longer than I thought to finish, but here it is:

- -----
This post categorises and describes all of the compatibility
mappings in Unicode 3.1. The intention is to show that these
mappings are of little value for name preparation, because in
almost all cases, one or more of the following applies:

DISALLOWED: the source characters are already disallowed by
   nameprep-06, so removing the mapping has no effect.
NOT-USEFUL: the source characters are not useful in domain names.
PUNCT-SYMBOL: the source characters are punctuation or symbols,
   similar to characters that are disallowed for ASCII.
NOT-EQUIV: the source characters are not semantically equivalent
   to the target characters - i.e. folding them would be more
   confusing than not folding them (if, for the sake of argument,
   they were allowed).
LEGACY: the source characters are *only* intended to be used for
   round-trip mappings from legacy charsets.
DEPRECATED: the source characters are formally deprecated by
   Unicode 3.1.
OBSCURE: the source characters are very difficult to type, or
   produce using an input method, so not folding them is unlikely
   to cause any practical difficulty to users.
INTERNAL: the source characters are normally only intended for
   internal use within an application or rendering engine.
INCONSISTENT: a category of mappings is arbitrary and inconsistent,
   with only some of the potential mappings in that category being
   defined as compatibility equivalences.

There are a very small number of exceptions:

EXCEPTION: the mapping could be quite useful, and there is no
   reason to exclude it on any of the grounds above.

I claim that there are few enough exceptions to show that NFKC
is not the right mapping to use: *if* some of these [EXCEPTION]
mappings are wanted, they should be handled as an additional step
to NFC normalisation (similar to the mapped-out characters).

Here are the mapping categories. They're in no particular order,
and some characters appear in more than one category.

<nobreak>
  The source character is a non-breaking version of the target
  character:
    00A0 NO-BREAK SPACE                       [DISALLOWED]
    0F0C TIBETAN MARK DELIMITER TSHEG BSTAR   [NOT-USEFUL; see below]
    2007 FIGURE SPACE                         [DISALLOWED]
    2011 NON-BREAKING HYPHEN                  [EXCEPTION]
    202F NARROW NO-BREAK SPACE                [DISALLOWED]

  Tibetan script consists of morphemes separated by tseks (also
  transliterated as "tsheg"); see section 9.13 of the Unicode
  standard. U+0F0C is a non-breaking tsek (the character name is
  a mistake). In domain names, either an ordinary tsek (U+0F0B),
  or a hyphen should be used instead.

  It's possible that a document might use a non-breaking hyphen
  to prevent a domain name or URI being split over lines, and it
  could be useful to map it to a hyphen when cutting and pasting,
  so this is an [EXCEPTION].

<super> and <sub>
  The source character is a superscripted or subscripted version of
  the target character. These can be further categorised as:

    letters (ordinal indicators, modifier letters, and 'n')
       [NOT-EQUIV]

    digits [NOT-USEFUL, OBSCURE, NOT-EQUIV]

    symbols (including superscript SM and TM)
       [NOT-USEFUL, OBSCURE, NOT-EQUIV]

    Kanbun (annotation of classical Chinese in Japanese texts)
       [NOT-USEFUL, OBSCURE, NOT-EQUIV]

  Whether the letters are useful is arguable, but if they are, they
  should not be folded (since they are definitely not semantically
  equivalent to the target character).

<fraction>
  Various legacy character sets have characters for fractions, e.g.
  1/2, etc. [LEGACY, OBSCURE, NOT-USEFUL].

<circle> and <square>,
also <compat> 3036 CIRCLED POSTAL MARK

  Circled and squared variants. [NOT-USEFUL, NOT-EQUIV]

  (Note that the decomposition is to the uncircled/unsquared character
  on its own, without a U+20DD COMBINING ENCLOSING CIRCLE or
  U+20DE COMBINING ENCLOSING SQUARE. So the effect of this folding
  is that names with and without the circle/square are equivalent,
  despite being visually distinct.)

<wide> Full-width variants
  FF01..FF0C Full-width symbols/punctuation        [DISALLOWED, PUNCT-SYMBOL]
  FF0E       FULLWIDTH FULL STOP                   [DISALLOWED]
  FF0F       FULLWIDTH SOLIDUS                     [DISALLOWED, PUNCT-SYMBOL]
  FF1A..FF20 Full-width symbols/punctuation        [DISALLOWED, PUNCT-SYMBOL]
  FF3B..FF40 Full-width symbols/punctuation        [DISALLOWED, PUNCT-SYMBOL]
  FFE0..FFE6 Full-width symbols                    [PUNCT-SYMBOL]

  FF0D       FULLWIDTH HYPHEN-MINUS                [EXCEPTION]
  FF10..FF19 FULLWIDTH DIGIT ZERO..NINE            [EXCEPTION]
  FF21..FF3A FULLWIDTH LATIN CAPITAL LETTER A..Z   [EXCEPTION]
  FF41..FF5A FULLWIDTH LATIN SMALL LETTER A..Z     [EXCEPTION]

  CJK input methods can sometimes produce full-width characters,
  and it may be useful to map these to half-width (normal) LDH
  ASCII characters.

  However, nameprep is probably not the best place to do that.
  Doing it there would mean that it is valid for full-width ASCII
  to appear in an encoded name [*]. This will display as replacement
  boxes when viewed on a system without CJK fonts. It would be far
  preferable to make sure that encoded names always use normal
  ASCII. That suggests doing this folding in name input widgets,
  and/or defining a way to tell input methods when a domain name (or
  similar identifier) is being entered.

  Note that CJK users already have to set input methods to produce
  half-width ASCII characters, in order to type existing LDH ASCII
  domain names. So although this folding may improve usability, it
  isn't essential.

  [*] I'm making the assumption that whatever IDN solution is chosen
      will allow names to be encoded transparently in at least some
      cases, i.e. it won't force ACE to be used everywhere.

<narrow>
  FF61..FF64 Half-width punctuation          [PUNCT-SYMBOL]
  FF65..FF9F Half-width Katakana             [EXCEPTION]
  FFE8..FFEE Half-width symbols              [PUNCT-SYMBOL]

  The same comments apply to half-width Katakana as to the <wide>
  mappings above.

<compat>, <narrow> Hangul Compatibility Jamo
  3131..318E Full-width Compatibility Jamo
  FFA0..FFDC Half-width Compatibility Jamo

  The normal set of Jamo encoded at 1100..1100 is conjoining, that
  is, sequences of Jamo are displayed as, and are NFC-equivalent
  to, the corresponding syllables. The Compatibility Jamo (both
  full-width and half-width) are non-conjoining, i.e. they each
  take up a character cell; that is the only reason why they were
  encoded separately.

  So, the effect of using NFKC is that a domain name could be
  displayed with Jamo in separate character cells, but would
  actually be equivalent to the corresponding name displayed as
  syllables. I can't see any reason why that would be desirable.

  Also, section 10.4 of [Unicode3.0] says, "These characters are
  provided solely for compatibility with the KS C 5601 standard."
  [LEGACY, NOT-EQUIV].

<small>
  These are all in the CJK Compatibility Forms block (FE30..FE44).
  They were only encoded for compatibility with CNS 11643.

  Most are [DISALLOWED] because the corresponding ASCII symbol
  is disallowed; the following are not:
    FE51 SMALL IDEOGRAPHIC COMMA              [LEGACY, PUNCT-SYMBOL]
    FE58 SMALL EM DASH                        [LEGACY, PUNCT-SYMBOL]
    FE5D SMALL LEFT TORTOISE SHELL BRACKET    [LEGACY, PUNCT-SYMBOL]
    FE5E SMALL RIGHT TORTOISE SHELL BRACKET   [LEGACY, PUNCT-SYMBOL]
    FE63 SMALL HYPHEN-MINUS                   [LEGACY]

  Note that it would probably be more useful for a converter from
  CNS 11643 to map to the ordinary variants of these characters,
  anyway, rather than the small variants, which no-one uses.

<compat> Overline variants
  FE49..FE4C
  [NOT-USEFUL, PUNCT-SYMBOL]

<compat> Spaces
  (mapping is U+0020 SPACE; also <wide> 3000 IDEOGRAPHIC SPACE).
  [DISALLOWED]

<compat> Spacing marks
  (mapping starts with U+0020 SPACE)
  These are mappings from a spacing diacritical mark, to <space> +
  the corresponding combining mark. They are [DISALLOWED] because
  the <space> is disallowed.

<compat> Maps to disallowed ASCII (other than space)
  2024 ONE DOT LEADER
  2025 TWO DOT LEADER
  2026 HORIZONTAL ELLIPSIS
  203C DOUBLE EXCLAMATION MARK
  2048 QUESTION EXCLAMATION MARK
  2049 EXCLAMATION QUESTION MARK
  2474..2487 PARENTHESIZED DIGIT/NUMBER ONE..TWENTY
  2480..249B DIGIT/NUMBER ONE..TWENTY FULL STOP
  249C..245B PARENTHESIZED LATIN SMALL LETTER A..Z
  3200..321C PARENTHESIZED HANGUL *
  3220..3243 PARENTHESIZED IDEOGRAPH *
  FE4D DASHED LOW LINE
  FE4E CENTRELINE LOW LINE
  FE4F WAVY LOW LINE

  [DISALLOWED].

<compat> Hangzhou numerals
  3038 HANGZHOU NUMERAL TEN
  3039 HANGZHOU NUMERAL TWENTY
  303A HANGZHOU NUMERAL THIRTY

  These map to the ideographs U+5341 meaning ten (or complete or
  perfect), U+5344 meaning twenty, and U+5345 meaning thirty.
  I suspect that input methods will normally produce those
  ideographs, not the numeral characters (i.e. these characters
  are [OBSCURE]) - can anyone confirm that?

<compat> Ideographic telegraph symbols for months, hours, and days
  32C0..32CB
  3358..3370
  33E0..33FE

  These map to a decimal ASCII number, followed by the ideograph
  U+6708 (for months) or U+70B9 (for hours) or U+65E5 (for days).
  Again, I suspect that input methods will produce those sequences
  rather than the symbols. [OBSCURE].

<compat> CJK Radicals
  2E9F CJK RADICAL MOTHER
  2EF3 CJK RADICAL C-SIMPLIFIED TURTLE
  2F00..2FD5 KangXi radicals block

  See section 10.1 of [Unicode3.0] for a discussion of radicals. Their
  main uses are:
    - to categorize or collate ideographs (e.g. in an index)
    - to describe new ideographs, especially using the "ideographic
      description sequence" convention.

  The first of these isn't applicable to domain names, and nameprep
  already disallows ideographic description characters. Therefore,
  the simplest approach would be to disallow all radicals.

  Note that even if mapping from radicals to ideographs were a good
  idea, the selection of such mappings defined by NFKC is highly
  inconsistent - e.g. the following radicals from the CJK Radicals
  Supplement block correspond to unified ideographs:

    2E83 -> 4E5A  2E85 -> 4EB8  2E8E -> 5140  2E8F -> 5C23  2E90 -> 5C22
    2E92 -> 5DF3  2E96 -> 5FC4  2E98 -> 624C  2E9F -> 6BCD  2EC0 -> 535D
    2EA1 -> 6C35  2EA3 -> 706C  2EA8 -> 72AD  2EAD -> 793B  2EAF -> 7CF9
    2EB0 -> 7E9F  2EB1 -> 7F53  2EB2 -> 7F52  2EBD -> 81FC? 2EBE -> 8279
    2EC1 -> 864E  2EC2 -> 8864  2EC3 -> 8980  2EC8 -> 8BA0  2ECC -> 8FB6
    2ED0 -> 9485  2ED1 -> 9577  2ED2 -> 9578  2ED3 -> 957F  2ED4 -> 95E8
    2ED6 -> 961D  2ED8 -> 9752  2ED9 -> 97E6  2EDB -> 98CE  2EDC -> 98DE
    2EDD -> 98DF  2EDF -> 98E0  2EE0 -> 9963  2EE2 -> 9A6C  2EE3 -> 9AA8
    2EE5 -> 9C7C  2EE6 -> 9E1F  2EEA -> 9EFE  2EEC -> 9F50  2EEE -> 9F7F
    2EF0 -> 9F99  2EF1 -> 9F9C  2EF3 -> 9F9F

  but only two of these are compatibility mappings (2E9F and 2EF3).
  [NOT-USEFUL, NOT-EQUIV, INCONSISTENT]

  The Yi radicals should probably also be disallowed because they
  are not useful in domain names, even though they don't have any
  compatibility mappings.

<vertical>
  Presentation forms of symbols for use in vertical (top-to-bottom)
  layout. These are all in the Small Form Variants block (FE50..FE6B).
  These mappings are not useful because:

  - the corresponding left-to-right symbols are not normally
    used in domain names (most of them are brackets).
  - domain names are not normally laid out vertically (it would
    be better to use a left-to-right footnote in most cases).

  [NOT-USEFUL, PUNCT-SYMBOL]

<font>, <compat>, <initial>, <medial>, <final>, <isolated>
Presentation forms and some ligatures:

  Latin:    0132..0133, FB00..FB06
  Armenian: 0587, FB13..FB17
  Arabic:   0675..0678
  Lao:      0EDC..0EDD
  Hebrew:   FB20..FB29, FB4F
  Arabic:   <initial>, <medial>, <final>, <isolated>

  Presentation ligatures/forms are rendering variants, so these
  characters should not normally appear in external representations
  of text (they are often used internally as part of a rendering
  implementation, but that isn't relevant for nameprep).

  Note that the word "ligature" is overloaded: some ligatures behave
  like presentation forms (e.g. ff, fi, ffi, ij in Latin scripts),
  while others (e.g. oe and ae) are part of the spelling of words,
  such as "arch<ae>ology" (British English spelling). The argument
  above does not apply to the second type of ligature, but those
  don't have compatibility mappings.

  An input method/keyboard driver should never generate a ligature
  or presentation form, and lots of existing software would break
  if it did. (In general, language-specific rules are necessary to
  properly ligaturize text - e.g. in English the "ff" in "shelfful"
  should not ligaturize because the two "f"s are in different
  syllables.) Even if a user copies text containing presentation
  ligatures from a word processor, they will be decomposed on the
  clipboard, unless the word processor is completely broken in this
  respect.
  [INTERNAL, OBSCURE]

<compat> Deprecated characters
  0F77 TIBETAN VOWEL SIGN VOCALIC RR
  0F79 TIBETAN VOWEL SIGN VOCALIC LL

  The character descriptions say that "use of this character is
  strongly discouraged". (ISTR some text in the standard explaining
  why, but I can't find it now.)
  [DEPRECATED]

<compat> Combinations of spacing characters:
  013F LATIN CAPITAL LETTER L WITH MIDDLE DOT
  0140 LATIN SMALL LETTER L WITH MIDDLE DOT
  0149 LATIN SMALL LETTER N PRECEDED BY APOSTROPHE
  0E33 THAI CHARACTER SARA AM
  0EB3 LAO VOWEL SIGN AM
  1E9A LATIN SMALL LETTER A WITH RIGHT HALF RING

  These are combinations of characters that were encoded as a single
  character in other standards. The only reason why they aren't
  canonical equivalences, is that the decomposition is to two spacing
  characters, rather than a spacing character and a combining mark.

  These mappings could be treated as [EXCEPTION]s, although the
  combined characters are rare enough that it probably isn't worth
  the hassle to do that. I don't know whether they are produced by
  keyboard drivers.

<compat> Croatian digraphs:
  01C4..01CC
  01F1..01F3

  Chapter 7 of [Unicode3.0] says:

    Croatian Digraphs Matching Serbian Cyrillic Lettters.

    Serbo-Croatian is a single language with paired alphabets: a
    Latin script (Croatian) and a Cyrillic script (Serbian). A set
    of compatibility digraph codes is provided for one-to-one
    transliteration.

  IOW, these digraphs should occur only in text that has been
  automatically transliterated from Serbian to Croatian. Normally the
  digraph would be typed as two separate characters, so there is no
  need for a nameprep mapping. [OBSCURE]

<compat> Roman numerals
  2160..217F
  [NOT-USEFUL, OBSCURE]

<font>, <compat> Latin and Greek letter-like characters
  Most of the Letter-like Symbols block   [OBSCURE, NOT-USEFUL, NOT-EQUIV]
  00B5 MICRO SIGN                         [OBSCURE, NOT-USEFUL, NOT-EQUIV]
  20A8 RUPEE SIGN                    [INCONSISTENT, NOT-USEFUL, NOT-EQUIV]

  Various symbols that look like stylized letters, sometimes with
  mathematical meanings.

  (It's not clear why the Rupee sign should have a compatibility
  mapping to "Rs", when the same doesn't apply to other currency
  symbols - e.g. the Pesata sign does not have a compatibility
  mapping to "Pts". In any case, that doesn't really matter, since
  currency symbols are not useful in domain names.)

  Greek keyboard drivers will produce the "proper" lowercase mu
  character (U+03BC), not U+00B5.

  Note that the following are canonical equivalents, so they should
  not be disallowed (in order to satisfy the Unicode requirement of
  treating canonical equivalents identically):
    U+2126 OHM SIGN -> Omega
    U+212A KELVIN SIGN -> K
    U+212B ANGSTROM SIGN -> A with ring above

  All of the remaining Letter-like Symbols should be disallowed.

<font> Mathematical Alphanumeric Symbols block
  1D400..1D7FF

  These characters are for specialised use in mathematical text.
  (In fact the whole point of encoding them was that they are
  not semantically equivalent to the corresponding plain letters
  and digits - so folding them would be pointless.)

  [NOT-USEFUL, OBSCURE, NOT-EQUIV].

<compat> Greek symbols:
  03D0..03D6
  03F0..03F2
  03F4..03F5
  These are technical symbols, not normal Greek text.
  [NOT-USEFUL, OBSCURE, NOT-EQUIV]

<compat> Miscellaneous
  U+017F LATIN SMALL LETTER LONG S

  This is really a glyph variant of 's'. It is rarely used, so it
  doesn't really matter if it is not mapped to 's'. [OBSCURE].

<compat> Repeated characters
  2033 DOUBLE PRIME
  2034 TRIPLE PRIME
  2036 REVERSED DOUBLE PRIME
  2037 REVERSED TRIPLE PRIME
  203C DOUBLE EXCLAMATION MARK
  222C DOUBLE INTEGRAL
  222D TRIPLE INTEGRAL
  222F SURFACE INTEGRAL
  2230 VOLUME INTEGRAL

  [OBSCURE, PUNCT-SYMBOL].

- --

---

From: Martin Duerst <[email-masked]>
Date: Tue, 23 Oct 2001 17:43:01 +0900

This is very interesting material. Thanks a lot for all this work.

The conclusions coincide mostly with mine, although I haven't
done such an explicit analysis.

Some comments below:

>to NFC normalisation (similar to the mapped-out characters).

I fully agree.

>   so this is an [EXCEPTION].

It seems inconsistent to treat a non-breaking hyphen as an
[EXCEPTION], but a non-breaking tsheg as [NOT-USEFUL].
The non-breaking tsheg will be as useful to Tibetan domain
names as the non-breaking hyphen to domain names that use it,
or actually more so because there is a tsheg between any
two syllables in Tibetan, but hyphens appear less often.

>   1/2, etc. [LEGACY, OBSCURE, NOT-USEFUL].

This is a typical example of where there is no problem for
IDN, because the '/' produced is also disallowed, but where
problems will appear if NFKC is applied for something similar.

>   ASCII. That suggests doing this folding in name input widgets,

This could be handled together with the 'ideographic fullstop'
issue, for which special treatment is neccessary anyway.

>   domain names. So although this folding may improve usability, it

I disagree quite a bit. For current all-ASCII domain names, it's
not a big issue, but for mixed Kanji/Latin domain names, it's
much more of an issue. Users in particular in Japan are used to
switch to a different mode for ASCII-only work, but there are
a lot of input methods that produce full-width Latin.

>   The same comments apply to half-width Katakana as to the <wide>

I have to disagree. Half-width Katakana is much less of a problem.
Some systems (e.g. Unix, Japanese Email) don't handle half-width
Katakana. And they look ugly and clearly 'not right' to the average
Japanese. To input them, you have to switch to a special mode.
(I wrote a longer piece about full-width Latin and half-width
Katakana a while ago for the nameprep design team; I'll
post it separately).

>   provided solely for compatibility with the KS C 5601 standard."

The half-width ones should be prohibited similar to the half-width
Katakana. There may be some point in allowing the full-width ones,
but the discussion on this in the nameprep team wasn't conclusive.
Anyway, as already discussed in the nameprep team, not folding
them into the basic Jamo (in the U+1100 block) would address
a problem that Soobok mentioned again very recently.

>   are [OBSCURE]) - can anyone confirm that?

I can't confirm, but I would strongly suspect that to be true.

><compat> Ideographic telegraph symbols for months, hours, and days

Same comment as above.

>   [NOT-USEFUL, PUNCT-SYMBOL]

I slightly disagree with your second point. The current domain
names already appear in vertical print (each letter is turned 90
degrees clockwise) in Japanese newspapers and magazines.

But this doesn't change your classification, because for
punctuation, the correct glyph should be chosen automatically
based on the writing direction. That's what the present-day
system I know do.

>   Arabic:   0675..0678

I would be very careful with these.

>   clipboard, unless the word processor is completely broken in this

I wouldn't be completely sure about the last sentence. But anyway,
the dns has worked for ages without mapping ff, fi, ffi,..., and so
it's rather safe to assume that it will continue without these kinds
of mappings.

>   characters, rather than a spacing character and a combining mark.

I'm not sure. There are some canonical decompositions into
two spacing characters. The algorithm in UTR 15 is carefully
designed to put them back again.

>   the hassle to do that. I don't know whether they are produced by

This probably should be checked on a one-by-one base.

>     U+212B ANGSTROM SIGN -> A with ring above

I don't understand this. Prohibition happens after normalization,
so there is no need to discuss whether they should be prohibited
or not, they will just be normalized away.

>   doesn't really matter if it is not mapped to 's'. [OBSCURE].

For some people, it might matter (Irish?).

>David Hopwood <[email-masked]>

Thanks again for your thorough work!

Regards,   Martin.

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 23 Oct 2001 08:20:53 -0700

I disagree with your assessment. For example, I believe that it is a good
thing that the non-breaking feature is suppressed -- that is irrelevant to
IDNs.

However, it would make your paper much easier to examine if you removed all
the characters that end up getting disallowed -- they are not
counter-examples. Of the characters that are left that you feel are
problematic, there are three possibilities that the committee has to judge:

A. they don't matter if they are converted
B. they do matter, but they can be remapped, or prohibited*
C. they do matter, and enough to make us abandon NFKC

The downside of abandoning NFKC is that we lose the equivalence between
thousands of characters that do represent the same fundamental abstract
characters, such as width variants, Arabic ligatures, etc.

Note: a character can be deleted by the remapping phase. It can also be
effectively prohibited *before* NFKC by simply mapping it to an invalid
character, like space, that is not affected by normalization and ends up
being prohibited.

BTW I have very little hope that this committee will ever produce a result
if issues keep getting re-raised time and time and time again. For the
committee to ever reach some kind of resolution, people have to ask
themselves, *not* whether they think the current situation is absolutely
optimal (in their view) -- since it *never* will be -- but instead, whether
they can live with the result; whether there are really any results that
will cause significant problems in practice.

Mark
—————

----- Original Message -----
From: "David Hopwood" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, October 23, 2001 00:17
Subject: [idn] NFC vs NFKC

> iQEVAwUBO9UXrDkCAxeYt5gVAQGspwf8DcHtURizKZj5I5mN/oE4krd7WfIXNwNj
> F7KIdavHPkNrL9JUt9j1vBr8iJ7eaYaTZ0zns0l3kL9m9QUWpmCiuqyWsdRKPRJQ
> w3mwDbartNV/en+OFp2qY8uHC1WAlcwZwcgS+RmSzfuSDdiYZ2gvXbySZjVNTAk1
> +LjaGuoBu8bL+0YDNClWpwQha5uPUkYvw2WvKUr5+F0ASLwoMmSqnHSIlvHVX0rd
> mOphfQfgo6k/4yG6YZKmp3F+8Onfs/IC2jZeorCWBMmre9uWO49Cf+WfO0C8CzOe
> o17SWXla+oNqo5dasb9ewSAlehdGxMi5Lx4HDZoqshJe4Fh2P8Ea2A==
> =iMh9

---

From: John C Klensin <[email-masked]>
Date: Tue, 23 Oct 2001 09:21:58 -0400

David,

I found your note thought-provoking in several respects, although
I don't think along quite the dimensions you intended.  As many
people in the WG know, I'm extremely nervous about permitting
_any_ non-numeric symbol in domain names, where "non-numeric
symbol" can be crudely defined as something that does not
normally appear in the text of sentences (or the equivalent) of
in any language.  I'm especially concerned about symbols that
belong either to the typographical "dingbat" collection or those
that can be (are designed to be?) used to draw pictures.

Whether to have such a prohibition or not is ultimately a policy
problem, not a technical one --symbols pose no special technical
problems as long as we don't try to map similar looking ones--
but we had best figure out how to get an answer or at least some
opinions from people or groups with different perspectives.

But, in the context of those concerns, you wrote, as part of a
technical discussion of mappings, utility, and appropriatenes of
various Unicode characters:

>   mapping to "Pts". In any case, that doesn't really matter,
>   since currency symbols are not useful in domain names.)

"Not useful", it seems to me, is somewhat in the eye of the
beholder.   For example, the term "Micro$oft" appears fairly
often in satirical (or sometimes just snide) articles, email, and
so forth.  I assume that, in the same spirit, if the dollar sign
character were available, there would be many applicants for
micro$oft.com as a domain name, and a fairly frightening
controversy as to whether it constituted legitimate satire (i.e.,
"free speech") or a trademark infringement.  I would assume that
there are similar potential puns and other amusing substitutions
of currency symbols in other languages or mixtures of them.

Again, this has nothing to do with your compatibility mapping
argument with which, from my very limited knowledge, I'm inclined
to agree, but with the "usefulness" of the currency symbols
themselves.

    john

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 24 Oct 2001 10:09:45 +0200

David,
trying to map your categories onto "what would happen at the DNS side".
I think Occam's razor on this problem is roughly:

- If there are 2 names that map to the same sequence of characters
  under NFKC, but not under NFC, and we think they should be DIFFERENT,
  NFKC is bad, and NFC should be used.
- If there are 2 names that map to the same characters under NFKC, but
  not under NFC, and we think the standard should force them to be ALWAYS 
THE
  SAME, NFKC is good.

Characters that do not produce valid domain names are Not A Problem.

>    nameprep-06, so removing the mapping has no effect.

Not A Problem.

> NOT-USEFUL: the source characters are not useful in domain names.

"Useful" is a value judgment, so we need to be careful.
Do you suggest outlawing this category?

>    similar to characters that are disallowed for ASCII.

Do you suggest outlawing this category?

>    confusing than not folding them (if, for the sake of argument,

This is the category that would argue strongly for NFC rather than NFKC.

>    round-trip mappings from legacy charsets.

I do not understand what you want to do with those.
Do you suggest outlawing this category?

> DEPRECATED: the source characters are formally deprecated by

Do you suggest outlawing this category?

>    to cause any practical difficulty to users.

Do you suggest outlawing this category?

>    internal use within an application or rendering engine.

Do you suggest outlawing this category?

>    defined as compatibility equivalences.

I don't understand what you want to do with those.

>    reason to exclude it on any of the grounds above.

This is the category that would argue in favour of NFKC and against NFC.

Below, I assume that you are proposing to do the following changes to
nameprep:

- Make NFC rather than NFKC the normalization function of nameprep
- Add NOT-USEFUL, FUNCT-SYMBOL, LEGACY, DEPRECATED, OBSCURE, INTERNAL
to the set of DISALLOWED characters
- Add the EXCEPTION rules to the nameprep ruleset

So the important categories are EXCEPTION (argue in favour of NFKC), 
NOT-EQUIV (argue strongly in favour of NFC) and INCONSISTENT.

> to NFC normalisation (similar to the mapped-out characters).

Here is the list I worry about.
I have followed your recommendation on everything you regard as OBSCURE or 
NOT-USEFUL.

>  the hassle to do that. I don't know whether they are produced by

The INCONSISTENT label is used with the Radicals block and the Rupee sign, 
which is also labelled "NOT-USEFUL". Probably right.

*ALL* of the NOT-EQUIV cases are also labelled as NOT-USEFUL, OBSCURE or 
another marker I have interpreted as "don't want to use these".

So far, if I have interpreted your missive correctly, you argue for:

- Outlawing a lot of characters (I wouldn't mind that :-)
- Adding the rules from NFKC for the remaining problematic characters
  to Nameprep

Either I am missing something, or there isn't a single domain name that
would be legal under your proposed change to Nameprep where you could tell 
from the output of Nameprep whether NFKC or NFC was applied.

Did I understand you correctly?

                      Harald

Nit (mandatory lingua-political hobby-horse :-)

>  don't have compatibility mappings.

As a Scandinavian, I have to point out that æ (ae) is not a ligature, it is 
a letter. Others seemingly (ab)use it as a ligature, but it isn't :-)

---

From: Martin Duerst <[email-masked]>
Date: Thu, 25 Oct 2001 15:23:14 +0900

At 08:20 01/10/23 -0700, Mark Davis wrote:
(answering David Hopwood)

>thing that the non-breaking feature is suppressed -- that is irrelevant to

This is a detail that can go one way or another.
The fact that the current DNS has survived a few years
with a hyphen but without the non-breaking hyphen being
accepted suggests that not including it won't hurt too much.
But including it won't hurt too much, either.

>counter-examples.

I think they are very important. In terms of numbers, there
are I think 3165 compatibility mappings in
http://www.unicode.org/Public/UNIDATA/UnicodeData.txt.
According to David's analysis, which closely matches with
my understanding, something of the order of 50 of these
really need to be mapped. This shows that using NFKC is
serious overkill.

Also, there 1866 NFC mappings. Reducing the number of mappings
from 5031 to 1866 is a saving of over 60%.

>Of the characters that are left that you feel are

Yes, most of them will show up so rarely that it indeed
doesn't matter whether they get converted or prohibited.

>characters, such as width variants, Arabic ligatures, etc.

Some of the width variants have been identified as exceptions.
How many of the Arabic ligatures will be entered by users
when they e.g. type in an arabic domain name on a keyboard?

Also, please note that the concept of 'fundamental abstract
character' is not something the IDN WG has been really
concerned about. As David has shown, many of the compatibility
(i.e. NFKC) equivalences are disputable. On the other hand,
for many characters not covered by NFKC you would easily
find some people claiming that some of them represent the
same 'fundamental abstract character'. As an example, many
people on this list might argue that there should be a
SC/TC mapping because the simplified and the traditional
variants represent one and the same 'fundamental abstract
character'.

What this WG has been concerned about is to avoid excluding
variants that could easily be input by the user in place
of some equivalent character (to distinguish this from
a lookalike that represents something completely different).

David's analysis, which coincides with my findings, has show
that about 50 out of about 3000 compatibility (i.e. NFKC)
equivalents are relevant for user input. Nobody has claimed
anything to the contrary, or brought up any evidence to the
contrary. If anybody has some such evidence, please send it in.

>Note: a character can be deleted by the remapping phase.

Which is not a good idea except for characters where deletion
doesn't make much of a difference (e.g. Arabic Tatweel,
non-spacing marks,...). If somebody would type in
fooBARbaz, and would get to foobaz, because B, A, and R
are mapped out, it would realy be quite bad.

>It can also be

Yes. If there were just *a few* characters that we wanted to prohibit
before doing NFKC, while we want to keep most of the NFKC mappings,
then that would be a reasonable idea. But as it turns out, we want
to ignore/prohibit most of the characters mapped by NFKC, while
only mapping very few of them.

>if issues keep getting re-raised time and time and time again.

How many times has this issue (NFC vs NFCK) been raised, on this list?
How much background material was provided in these discussions?

>For the

It is the first time anybody has done such a careful analysis of
NFC vs. NFKC. The choice between NFC and NFKC is rather fundamental,
no the least because IDN also most probably will serve as an example
to other similar problems. The conclusions from this analysis are
quite clear, at least to me.

I agree that asking whether the solution is absolutely optimal
in somebody's personal view is not a question that leads to consensus.
But the question of whether some change is an overall improvement
that helps many while not causing problems for anybody is a very
relevant question.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Thu, 25 Oct 2001 14:32:04 +0900

Hello Harald,

I can't speak for David, but some answers below anyway.

At 10:09 01/10/24 +0200, Harald Alvestrand wrote:
>  not under NFC, and we think the standard should force them to be ALWAYS THE
>  SAME, NFKC is good.
>

>Do you suggest outlawing this category?

Yes.

>Do you suggest outlawing this category?

Yes.

>This is the category that would argue strongly for NFC rather than NFKC.

Yes indeed.

>Do you suggest outlawing this category?

Yes. IDN is a new system, with the goal of being able to
represent labels in a wide range of scripts. There is no
need to allow artefacts of legacy character encodings,
because IDN isn't designed to map existing names back
and forth from other naming systems.

>Do you suggest outlawing this category?

Yes.

>Do you suggest outlawing this category?

Yes indeed. The argument for using NFKC that I have heard
most often is that we want to avoid the situation that
somebody types in a domain name from something s/he has
seen on a billboard,..., but the name is not found because
some compatibility equivalents instead of the actual
characters were input. This, and many of the other categories,
shows that (with exception of [EXCEPTION], of course) there
is really no such problem.

>Do you suggest outlawing this category?

Yes.

>I don't understand what you want to do with those.

Outlawing them is the best way to go.

Please note that most characters and character classes in David's
analysis carry more than one label, and that in many if not most
such cases, all of the labels suggest outlawing.

>This is the category that would argue in favour of NFKC and against NFC.

Yes, or in favor of including it in the IDN-specific mapping
before the normalization step, or in delegating this to some
earlier process in the same way we have delegated the
ideographic fullstop to preprocessing.

>- Make NFC rather than NFKC the normalization function of nameprep

Yes.

>to the set of DISALLOWED characters

Yes.

>- Add the EXCEPTION rules to the nameprep ruleset

Yes, add the relevant mappings to the mapping stage before
normalization.

>NOT-EQUIV (argue strongly in favour of NFC) and INCONSISTENT.

Yes. But the other categories are also important. They show
that NFKC maps around a lot of dubious, very rarely used, or
otherwise legacy characters. This means that NFKC does a lot
of work that isn't needed at all.

Changing from NFKC to NFC will simplify things, in particular
for standalone implementations that don't have access to
normalization data. The number of normalization mappings is
very significantly reduced. The number of pre-normalization
mappings is just slightly increased. The 'prohibited' data
is exactly the same.

>>    2011 NON-BREAKING HYPHEN                  [EXCEPTION]

This is open for discussion. We could go either way.

>>  FF21..FF3A FULLWIDTH LATIN CAPITAL LETTER A..Z   [EXCEPTION]

These have to be dealt with in case mapping anyway.
We can just directly map them to halfwidt lowercase.

>>  FF41..FF5A FULLWIDTH LATIN SMALL LETTER A..Z     [EXCEPTION]

Yes.

>>  FF65..FF9F Half-width Katakana             [EXCEPTION]

As I have explained in two earlier postings, these are not
something we have to worry about. Just excluding them should
be fine.

>- Adding the rules from NFKC for the remaining problematic characters

Yes.

>Did I understand you correctly?

Yes. The set of legal domain labels would stay exactly the same.

Regards,   Martin.