Subject: [idn] I-D ACTION:draft-chung-idn-ace37-00.txt

From: [email-masked]
Date: Fri, 06 Jul 2001 06:50:28 -0400

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title		: ACE Utilizing All 37 Alphanumeric Characters (ACE37)
	Author(s)	: E. Chung, D. Leung
	Filename	: draft-chung-idn-ace37-00.txt
	Pages		: 17
	Date		: 05-Jul-01
	
ACE37 is a combination of DUDE-02, AMC-W/V and LACE.  ACE37 utilizes 
the simple one pass algorithm of DUDE, the character block 
considerations of AMC-W/V and the Base-32 compression of LACE.  It 
also fully utilizes entire LDH set currently allowed in the DNS (A-
z, 0-9 and '-') within its character repertoire to optimize 
performance and compression.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-chung-idn-ace37-00.txt

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-chung-idn-ace37-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-chung-idn-ace37-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---

From: "Soobok Lee" <[email-masked]>
Date: Sat, 7 Jul 2001 02:21:51 +0900

I compared ACE37 with my LDUDE (reordered DUDE).

LDUDE's reordering mapping tables  are tuned to produce
as many single-quintet-XOR- diff values as possible
based on character & adjacency statistics.

If we apply reordering to ACE37, we can improve ACE37.

However, if reordering applied to ACE37 become MOST successful,
ACE37 should support  single quintet XOR difference when encoding,
that ACE37 does not support now.

Reordering  already statistically  guarantees XOR diff values less than
0x1000 for Han, by mapping most frequent 4096 han letters into 4096
codes block starting from 0x5000. Most freq 4096 han letters
( in about 23000 han letters  )  have cumulative frequency sum
of 99.9% in typical chinese texts and business names.
(for top 256, 58% )

LDUDE compresses the next 25 letters of Japanese strings down to 47
chars of ACE labels, 10 chars shorter than that of AMCW and ACE37.

JP1) Japanese String 1: ( 25 letters )
      U+793E U+56E3 U+6CD5 U+4EBA U+65E5 U+672C U+30CD U+30C3 U+30C8
      U+30EF U+30FC U+30AF U+30A4 U+30F3 U+30D5 U+30A9 U+30E1 U+30FC
      U+30B7 U+30E7 U+30F3 U+30BB U+30F3 U+30BF U+30FC

      DUDE-02 : z3xQu97Pv4vGuuyRu5xRu6Jxz8BQMuHtDxDMxHuGzNwItPwMxAtE\
                wIwIwNwD  (60 chars)
      LDUDE   : xs8Nu2Cu4RvMGBysxGyCKtHtQCPFtAyPyKtPBGPyAyAyFyR
                ( 47 chars, 21.6% shorter)
      AMCW    : z3vQ28DDyxs5KB9fCjnvs6P6DI8R9N4RE9D7F4J8B9N5H8H9D5M9\
                D5R9N ( 57 chars )
      LAMCW   : xs2NwsQu4B3KNPvs6M4JD5E4KIFA5A7P5H4KMPA6A4A6F4K
                ( 47 chars )

      ACE37: i9urut6hm8jfaqv0m9dv1wewbx7wjyjwbynx6zsy8wtybygwky8y8ycy3
                (57 char)

     JP3) Japanese String 3: ( 17 letters )
      U+6771 U+4EAC U+90FD U+60C5 U+5831 U+30B5 U+30FC U+30D3 U+30B9
      U+7523 U+696D U+5065 U+5EB7 U+4FDD U+967A U+7D44 U+5408

      DUDE-02 : yztBu37P78xB9svIv29Ey22EwJuRyKwx3Kt6wQv3sI87CttyK734\
                H85vQu3wN (61 chars)
      LDUDE   : xttHxPvtFu9CDyssAyEyHyRys9PxQ4KHGEu4CuwJ
                ( 40 chars, 34.4% shorter)
      AMCW    : z3vQ28DDyxs5KB9fCjnvs6P6DI8R9N4RE9D7F4J8B9N5H8H9D5M9\
                D5R9N ( 57 chars )
      LAMCW   : xs2NwsQu4B3KNPvs6M4JD5E4KIFA5A7P5H4KMPA6A4A6F4K
                ( 47 chars )

      ACE37: drhaetvihk1o67ka44y9xfzahcqv2e6883micbaud7apuqac (48 char)

Soobok Lee, [email-masked]

---

From: Keith Moore <[email-masked]>
Date: Fri, 06 Jul 2001 13:53:05 -0400

I would *really* like to avoid complex reordering of codepoints, because
it would require fairly large tables (that would have to be implemented
on some machines of modest capability) and would be error-prone.  

I also fear that this kind of mapping would encourage greater use
of proxies over end-to-end implementations, with all of the implications
for reliability, control, delay, etc. that would result from this design 
choice.  I don't think we want to encourage a walled garden.

I could see having a very simple reordering of codepoints in order
to increase the efficiency of DUDE for ideographic languages.  But my
(admittedly biased) sense is that we're fast approaching the point
of diminishing returns.  Just how important is it to provide for names 
that are one or two ideographs longer?

Keith

---

From: "Edmon" <[email-masked]>
Date: Fri, 6 Jul 2001 16:59:49 -0400

> on some machines of modest capability) and would be error-prone.
I agree.  The code block shifting mechanism in ACE37 however is very simple
and yeilds great benefit for CJK ideographs.

> that are one or two ideographs longer?

It is more than one or two... in the worst case scenario for CJK, ACE37 out
performs DUDE by 5 characters!
And there is an undeniable demand from the CJK community to up the 14
character limit to at least, I guess close to the original 23 character
limit.

> Keith

Edmon

---

From: "Soobok Lee" <[email-masked]>
Date: Sat, 7 Jul 2001 08:14:37 +0900

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>

> on some machines of modest capability) and would be error-prone.  

It's not complex.
Its table is not so large compared to that of KC norm and
NAMEPREP. ACE & NAMEPREP will be packaged into one library
as MDNkit of JPNIC. Look into KC norm and NAMEPREP tables and
think about the reordering table, more simpler. 

Reordering reduce the length typical ACE label about 40% ~ 50%.
Please throughly test my example sources with average length of
han/hangul domains. 

> choice.  I don't think we want to encourage a walled garden.

No proxies needed. No walls. 
All reordering is hidden and done in ACE & de-ACE libraries transparently.

No delay at all. Because what is needed is "n=maptable[n]" operation.
Single machine instruction away!!!
What i implemented in my example source is using mapping functions
with tables. IN the final version, we can use only tables.

> that are one or two ideographs longer?

With DUDE+REORDERING , we get 7~8 more Han ideographs. 
If you have enough time, test with many han examples.

code-block shifting does not help to reduce average XOR distance
in general cases ( good only for the first 4096 code block ).

What is important is  to reduce ACE label length of AVERAGED 
han/hangul domains. THis working group should concentrage
attentions  not to the max length issue but to  issues of
reducing ACE label length of average hangul/han domains.

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Sat, 7 Jul 2001 08:50:38 +0900

If you tested with my old reordering I-D version 0.9:
 You'd better try with my newly submitted I-D version 1.0 at
  http://www.postel.co.kr/lsb-ace-01.txt

 it contains many examples for CJK.
 40~50% shorter ACE labels are produced!

 It does better compression for European
 domains, too.

---

From: "Soobok Lee" <[email-masked]>
Date: Sat, 7 Jul 2001 09:07:27 +0900

----- Original Message -----
From: "Edmon" <[email-masked]>
To: "Keith Moore" <[email-masked]>; "Soobok Lee" <[email-masked]>

> and yeilds great benefit for CJK ideographs.

Code block shiffting  (  0x4E00:8FFFF ----> 0x0000: 0x52FF)
does reduce XOR diff value  for small subset of the code points
from   first 0x4E00 ~ 0x5E00 block.

For example, 0x4e00 and 0x5000 (xor diff is 0x1e00)
is mapped to 0x0000, 0x0200 (xir diff value 0x200 ).

But, For 0x4e00 and 0x6e00, xor diff values are  0x2000  for either
cases !

> performs DUDE by 5 characters!

For 99% CJK domains, LDUDE allows 30~58% longer CJK domains than DUDE .

In other words, LDUDE compress 30~50% better than DUDE for average length
domains. This is more important point.
I will report when I complete testing with ACE37 + REORDERING later.

---

From: "Soobok Lee" <[email-masked]>
Date: Sat, 7 Jul 2001 10:54:29 +0900

----- Original Message -----
From: "Edmon" <[email-masked]>
To: "Keith Moore" <[email-masked]>; "Soobok Lee" <[email-masked]>
Cc: <[email-masked]>
Sent: Saturday, July 07, 2001 5:59 AM
Subject: Re: [idn] ACE37,AMCW and LDUDE

> performs DUDE by 5 characters!

No. It may be so for the worst case.
For nominal case for hangul/han domains, max length reaches 28~29 syllables.

moreover, ACE37 with REORDERING will do much more compression, i guess.
REORDERING is another compression layer applicabe to all ACEs, not for
DUDE.