Subject: [idn] Reordering evaluation result by existing Japanese JP domain names

From: Yoshiro YONEYA <[email-masked]>
Date: Tue, 13 Nov 2001 14:18:47 +0900 (JST)

Following table shows Reordering evaluation result by existing
approximately 70000 Japanese JP domain names.

Here,
- Japanese JP domain names consists from one or more JP-char(s) and zero
  or more LDH-char(s)
  + Hyphen cannot be a beginning or ending character
  + Length of Japanese JP domain name is 1 through 15
- JP-char is Kanji or Hiragana or Katakana or one of several CJK marks
  (Specific table of JP-char can be obtained from draft-ietf-idn-jpchar-01)
- LDH-char is one of Alphabets or Numerics or Hyphen
- In this evaluation,
  + Kanji is in the range of U+3005, U+4E00-U+9FA0
  + Kana is in the range of U+3041-U+309E, U+30A1-U+30FE

[Kanji Only]
|  N| FREQ|N*FREQ|SUM OF AMCZ(X)|SUM OF LAMCZ(Y)|  COMP||   X>Y|   X=Y|   X<Y|
|  1|  650|   650|    2169(3.34)|     1978(3.04)| 8.81%||30.00%|69.38%| 0.62%|
|  2| 5745| 11490|   38368(3.34)|    32602(2.84)|15.03%||76.83%|19.32%| 3.85%|
|  3| 5733| 17199|   55389(3.22)|    46704(2.72)|15.68%||85.63%|12.58%| 1.80%|
|  4| 6178| 24712|   78764(3.19)|    63066(2.55)|19.93%||95.19%| 4.21%| 0.60%|
|  5| 1946|  9730|   30815(3.17)|    24316(2.50)|21.09%||97.94%| 1.90%| 0.15%|
|  6| 1879| 11274|   35453(3.14)|    27144(2.41)|23.44%||99.47%| 0.53%| 0.00%|
|  7| 1202|  8414|   26118(3.10)|    20109(2.39)|23.01%||99.83%| 0.08%| 0.08%|
|  8| 1437| 11496|   35793(3.11)|    26785(2.33)|25.17%||99.72%| 0.14%| 0.14%|
|  9|  745|  6705|   20745(3.09)|    15674(2.34)|24.44%||100.0%| 0.00%| 0.00%|
| 10|  693|  6930|   21337(3.08)|    16096(2.32)|24.56%||100.0%| 0.00%| 0.00%|
| 11|  391|  4301|   13235(3.08)|    10113(2.35)|23.59%||99.74%| 0.26%| 0.00%|
| 12|  302|  3624|   11168(3.08)|     8583(2.37)|23.15%||100.0%| 0.00%| 0.00%|
| 13|  145|  1885|    5737(3.04)|     4438(2.35)|22.64%||100.0%| 0.00%| 0.00%|
| 14|  105|  1470|    4412(3.00)|     3483(2.37)|21.06%||100.0%| 0.00%| 0.00%|
| 15|   71|  1065|    3234(3.04)|     2561(2.40)|20.81%||100.0%| 0.00%| 0.00%|
|ALL|27222|120945|  382737(3.16)|   303652(2.51)|20.66%||89.11%| 9.53%| 1.36%|

[Kana Only]
|  N| FREQ|N*FREQ|SUM OF AMCZ(X)|SUM OF LAMCZ(Y)|  COMP||   X>Y|   X=Y|   X<Y|
|  1|  176|   176|     528(3.00)|      528(3.00)| 0.00%|| 0.00%|100.0%| 0.00%|
|  2|  537|  1074|    2588(2.41)|     2550(2.37)| 1.47%||18.25%|70.58%|11.17%|
|  3| 2830|  8490|   18683(2.20)|    17797(2.10)| 4.74%||39.65%|44.88%|15.48%|
|  4| 4672| 18688|   38753(2.07)|    36123(1.93)| 6.79%||51.86%|34.25%|13.89%|
|  5| 3768| 18840|   37744(2.00)|    34517(1.83)| 8.55%||62.15%|25.13%|12.71%|
|  6| 3769| 22614|   43653(1.93)|    39392(1.74)| 9.76%||66.91%|22.50%|10.59%|
|  7| 3317| 23219|   43721(1.88)|    39350(1.69)|10.00%||70.64%|18.36%|11.00%|
|  8| 2235| 17880|   32870(1.84)|    29433(1.65)|10.46%||74.23%|16.60%| 9.17%|
|  9| 1418| 12762|   23061(1.81)|    20651(1.62)|10.45%||78.07%|12.83%| 9.10%|
| 10| 1008| 10080|   17890(1.77)|    15942(1.58)|10.89%||80.46%|12.20%| 7.34%|
| 11|  689|  7579|   13247(1.75)|    11802(1.56)|10.91%||82.29%| 9.29%| 8.42%|
| 12|  486|  5832|   10029(1.72)|     8980(1.54)|10.46%||79.84%|11.52%| 8.64%|
| 13|  312|  4056|    6903(1.70)|     6141(1.51)|11.04%||83.33%|10.90%| 5.77%|
| 14|  220|  3080|    5157(1.67)|     4644(1.51)| 9.95%||80.00%|12.73%| 7.27%|
| 15|  174|  2610|    4361(1.67)|     3904(1.50)|10.48%||86.21%| 8.62%| 5.17%|
|ALL|25611|156980|  299188(1.91)|   271754(1.73)| 9.17%||62.35%|26.17%|11.48%|

[Kanji + Kana]
|  N| FREQ|N*FREQ|SUM OF AMCZ(X)|SUM OF LAMCZ(Y)|  COMP||   X>Y|   X=Y|   X<Y|
|  1|    0|     0|       0(----)|        0(----)| ---- || ---- | ---- | ---- |
|  2|   99|   198|     705(3.56)|      693(3.50)| 1.70%||12.12%|87.88%| 0.00%|
|  3|  687|  2061|    6721(3.26)|     6558(3.18)| 2.43%||27.80%|67.83%| 4.37%|
|  4| 1088|  4352|   13364(3.07)|    12852(2.95)| 3.83%||49.91%|38.60%|11.49%|
|  5| 1950|  9750|   28359(2.91)|    27165(2.79)| 4.21%||53.79%|30.56%|15.64%|
|  6| 1898| 11388|   31835(2.80)|    29785(2.62)| 6.44%||66.86%|23.71%| 9.43%|
|  7| 1476| 10332|   28609(2.77)|    26241(2.54)| 8.28%||80.08%|14.84%| 5.08%|
|  8| 1448| 11584|   31888(2.75)|    28620(2.47)|10.25%||87.64%| 8.70%| 3.66%|
|  9| 1129| 10161|   27421(2.70)|    24142(2.38)|11.96%||90.61%| 6.64%| 2.75%|
| 10|  987|  9870|   26260(2.66)|    22844(2.31)|13.01%||95.04%| 2.84%| 2.13%|
| 11|  672|  7392|   19464(2.63)|    16715(2.26)|14.12%||95.98%| 2.83%| 1.19%|
| 12|  652|  7824|   20760(2.65)|    17642(2.25)|15.02%||96.32%| 2.91%| 0.77%|
| 13|  423|  5499|   14205(2.58)|    12154(2.21)|14.44%||95.04%| 4.73%| 0.24%|
| 14|  317|  4438|   11434(2.58)|     9591(2.16)|16.12%||97.16%| 1.89%| 0.95%|
| 15|  222|  3330|    8165(2.45)|     6914(2.08)|15.32%||96.40%| 3.60%| 0.00%|
|ALL|13048| 98179|  269190(2.74)|   241916(2.46)|10.13%||74.13%|19.46%| 6.41%|

[Whole Japanese JP domain names]
|  N| FREQ|N*FREQ|SUM OF AMCZ(X)|SUM OF LAMCZ(Y)|  COMP||   X>Y|   X=Y|   X<Y|
|  1|  826|   826|    2697(3.27)|     2506(3.03)| 7.08%||23.61%|75.91%| 0.48%|
|  2| 6431| 12862|   41960(3.26)|    36144(2.81)|13.86%||70.35%|25.28%| 4.37%|
|  3| 9391| 28173|   81984(2.91)|    72165(2.56)|11.98%||66.93%|26.95%| 6.12%|
|  4|12254| 49016|  134055(2.73)|   114975(2.35)|14.23%||73.74%|19.58%| 6.68%|
|  5| 8154| 40770|  102544(2.52)|    91261(2.24)|11.00%||68.08%|21.98%| 9.95%|
|  6| 7996| 47976|  116989(2.44)|   101931(2.12)|12.87%||74.10%|18.38%| 7.52%|
|  7| 6401| 44807|  104677(2.34)|    91494(2.04)|12.59%||77.39%|15.34%| 7.26%|
|  8| 5471| 43768|  106625(2.44)|    90444(2.07)|15.18%||83.81%|11.02%| 5.17%|
|  9| 3553| 31977|   76160(2.38)|    65007(2.03)|14.64%||85.96%| 9.18%| 4.87%|
| 10| 2925| 29250|   70335(2.40)|    59344(2.03)|15.63%||89.03%| 7.11%| 3.86%|
| 11| 1911| 21021|   49513(2.36)|    41912(1.99)|15.35%||89.59%| 6.59%| 3.82%|
| 12| 1603| 19236|   46006(2.39)|    38905(2.02)|15.43%||89.77%| 6.55%| 3.68%|
| 13|  981| 12753|   29483(2.31)|    25142(1.97)|14.72%||90.52%| 7.24%| 2.24%|
| 14|  708|  9912|   22828(2.30)|    19401(1.96)|15.01%||89.83%| 6.64%| 3.53%|
| 15|  510|  7650|   17123(2.24)|    14606(1.91)|14.70%||92.35%| 5.69%| 1.96%|
|ALL|69115|399997| 1002979(2.51)|   865237(2.16)|13.73%||75.03%|18.73%| 6.24%|

-- 
Yoshiro YONEYA <[email-masked]>
           aka <[email-masked]>

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 13 Nov 2001 08:21:52 -0800

Yoneya-san, thank you for giving real information in this discussion. 
Your table are great, and they show that the reordering for Japanese 
names will shorten typical Japanese ACE names by only a tiny fraction.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 13 Nov 2001 21:42:13 +0000

> names will shorten typical Japanese ACE names by only a tiny fraction.

20% is a tiny fraction?  (For all-kanji labels, except the short ones,
where compression doesn't matter anyway.)

It looks to me like the gain tends to be too small to care much about
for small scripts (around 10%), but is significant for large scripts
(20% for Han and maybe 30% for Hangul).  Given that Chinese is spoken by
about 1/5 of the world's population, I don't think it would be ludicrous
to include functionality that mostly benefits the large scripts.  (I'm
not saying I favor it, but I don't think it would be ludicrous.)  Does
the CJK community think it would be worth the extra complexity?  If they
don't, then the rest of us can forget about it.  If they do, then we
have something to think about.

AMC

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 13 Nov 2001 14:59:53 -0800

>where compression doesn't matter anyway.)

Yep. If you look at Yoneya-san's chart, you will see that the large 
majority of the names are 8 characters or less. In fact, since you 
want to narrow it to all-kanji names, the large majority are 4 
characters or less. So, a 20% length reduction means ACE strings that 
are about 2 characters shorter for typical all-Kanji Japanese names.

I truly don't believe that we will see end users typing in ACE names, 
but even if you do, is it really easier to type in h4xe90wsie3b than 
h4xe90wsie3b5a? This is worth the hair and incompleteness of the 
reordering draft?

>   Does

That's what the straw poll is about. The complexity comes at a high 
cost. For instance, the current draft  bases its frequency tables on 
the current VGRS testbed, which is unlikely to really reflect typical 
Asian names. Adopting that strategy will be based on a very myopic 
view of Asian names.

--Paul Hoffman, Director

---

From: "Bruce Thomson" <[email-masked]>
Date: Wed, 14 Nov 2001 08:45:58 +0900

It would be interesting to hear Yoneya-san's views on the benefits to
Japanese users of the 20% improvement. Until we hear from him, let
me offer my own opinion, as an ICANN-accredited registrar whose
principal business is selling Japanese idns in the Verisign testbed.

We have not seen any demand at all for long names, either for ASCII
ones or Japanese ones. Demand for idns themselves is very high,
and our mail server broke down under the unexpected load when
testbed names went on sale last November.

Customer concern about names is on a much more practical level:
when can I use them? Will they work with e-mail, or only for the
web? What application software will work and what will not?

Anyway, I don't see the need for this improvement at all, and I
think our energy should be applied elsewhere.

Bruce

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 14 Nov 2001 10:14:17 +0900

----- Original Message ----- 
From: "Paul Hoffman / IMC" <[email-masked]>
> Asian names. Adopting that strategy will be based on a very myopic 
 
There are plenty of NIC provided han/hangeul samples from different cultures
and countries, waiting for reordering table training.
So far, the errors (deviation) of improvement ratios are less than 1% for JPNIC
kanji cases compared to that of ML.com CJK-wide han samples.

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 14 Nov 2001 10:18:15 +0900

----- Original Message ----- 
From: "Paul Hoffman / IMC" <[email-masked]>
  all-Kanji Japanese names.
> h4xe90wsie3b5a? This is worth the hair and incompleteness of the 

The More frequent names, more compression  even in the same length!  
You may not have evaluated  the full merit of REORDERING.

INPUT -> 東京大學
U+6771 U+4EAC U+5927 U+5B78

AMCZ     : 1lQ68W3xBh4U
           12
LAMCZ    : nrT3CHX     
           7

INPUT -> 富士銀行
U+5BCC U+58EB U+9280 U+884C

AMCZ     : zqSv0Er63DrsI
           13
LAMCZ    : vmT0D0FrL
           9

INPUT -> 中國東方航空公司
U+4E2D U+570B U+6771 U+65B9 U+822A U+7A7A U+516C U+53F8

AMCZ     : fiQy2F5vC5vE46YolCwy4C23M
           25
LAMCZ    : nkT8IiFwQiB8ExDP      
           16

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 14 Nov 2001 10:40:49 +0900

----- Original Message ----- 
From: "Soobok Lee" <[email-masked]>
To: "IETF idn working group" <[email-masked]>; "Paul Hoffman / IMC" <[email-masked]>
Sent: Wednesday, November 14, 2001 10:18 AM
Subject: Re: [idn] Reordering evaluation result by existing Japanese JP domain names

> The More frequent names, more compression  even in the same length!  

s/frequent names/frequent characters/;
Corrrecting  my wording mistake. 
Soobok Lee

---

From: MIYAYAMA Takayuki <[email-masked]>
Date: Thu, 15 Nov 2001 16:19:21 +0900

Dear All,

# This is on behalf of Mr. Yoneya as he is not available now.

On Tue, 13 Nov 2001 14:59:53 -0800

> h4xe90wsie3b5a? This is worth the hair and incompleteness of the 

The following are the opinions of those in research of reordering for
Japanese JP domain names.

- It may have some benefit (max: 25% compression) for all-kanji labels
  (39% of the total of JP domain names in Japanese). However, the
  outcome for kanji-kana and kanji-ascii labels which is the rest 61%
  of the total, would amount to max: 16% compression only.

- Within the range of 4 to 5 characters, which is the majority of JP
  domain names in Japanese, even done in all-kanji labels, the benifit
  would remain to be only 2-4 octets.

Considering the overhead of reordering, we don't think it is the
advantage for Japanese Domain Names.

---
MIYAYAMA Takayuki ([email-masked])