Subject: [idn] reordering strawpoll

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 12 Nov 2001 18:35:49 +0800

The working group co-chairs would like to conduct a strawpoll to
guage the consensus of the working group on reordering
http://www.ietf.org/internet-drafts/draft-ietf-idn-lsb-ace-02.txt

Instructions:
- Send an email to [email-masked]
- In your subject, put a single word "YES", "NO", or "ABSTAIN"
- Leave your body empty. Any text in body will be ignored.

The strawpoll will conclude on 16th Nov 23:59 GMT. Counting will be
done by wg members, Yoshiro Yoneya, Yves Arrouye, Dave Crocker and
the co-chairs, Marc Blanchet & James Seng. The result of the poll will
be made public after we tally the count.

--- STRAWPOLL QUESTION ---
Do you agree to include reordering (draft-ietf-idn-lsb-ace) into
NAMEPREP-ACE architecture?

---- YES -----: mailto:[email-masked]?Subject=YES
If you say 'yes', then you agree to include reordering into
NAMEPREP-ACE architecture.

The implication of 'yes' result is that draft-ietf-idn-lsb-ace will be
added into NAMEPREP-ACE architecture

----- NO -----: mailto:[email-masked]?Subject=NO
If you say 'no', then you disagree to include reordering into
NAMEPREP-ACE architecture.

The implication of 'no' result is that draft-ietf-idn-lsb-ace
will be dropped from the wg core interest.

--- ABSTAIN ---: mailto:[email-masked]?Subject=ABSTAIN
If you say 'abstain', then you are happy with either 'yes' or 'no',
following the majority opinion.

--- SILENT ----: Of course, you may choose to remain silent.

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 12 Nov 2001 15:55:49 +0100

...

From http://www.ietf.org/internet-drafts/draft-ietf-idn-lsb-ace-02.txt:

>    As such examples shows, most ACE algorithms are designed to favor 
>    latin and small script blocks over very large blocks like han and 
>    hangeul. For CJK people (in China,Hongkong,Macao,Japan,South/North
>    Korea and Taiwan), that disadvantage results in longer ACE labels
>    and less room for free-form long names. It is clear that there 
>    must be some improvements to ACEs to compensate this unfair 
>    disadvantage.

It's a bit strange that this comes from quarters where there is already quite
a lot of "compaction" in the representations of text.  A single Han ideograph
expresses "more" than a single letter in other scripts.  And a single Hangul
syllable character expresses from 2 to 6 letters in one single character.
Hangul is fundamentally is an alphabetic script, with 17 consonant letters
and 11 vowel letters, plus some variant (and historic) letters.

<ironic-mode>

I think it's an urgent matter to follow the example of Hangul for all alphabetic
scripts and encode tens of thousands of syllables in the Latin, Greek, etc.
scripts, so that we can compensate this unfair disadvantage that Latin, etc.
have compared to Hangul (and Han, though that is not alphabetic).

And then, of course, we have to have a reordering stage, where the
most common syllables are ordered so that also an ACE encoded
string gets as short as possible.

</ironic-mode>

Of course, which languages should be considered for selection for
encoding of Latin or Cyrillic syllables, and which languages's statistics
should be used for the reordering will be hotly debated!  ;-)

            On the ironic side
            /kent k

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 12 Nov 2001 16:01:25 +0100

Regardless of reordering, there is an actual problem for Hangul, which
I don't think has been addressed. I have lately, and with the help of a
Korean colleague, been looking fairly deeply into the problem of
collating (ordering) Hangul strings properly.  So even though I cannot
understand Korean, and only begin to be able to read the letters (I look
more at code point numbers than glyphs), I've looked quite a lot into this.
See also page 53 of The Unicode Standard 3.0, which deals with
Hangul syllables. Let me just pick an example. The number of instances
are in the thousands, but the basic problem is the same.

The precomposed Hangul syllable U+AE4C (GGA) is canonically equivalent
with <U+1101, U+1161> (GG, A), through algorithmic decomposition. That is fine
so far. But <U+1101, U+1161> is in turn equivalent to <U+1100, U+1100, U+1161>
(G, G, A), but this equivalence is neither a canonical equivalence, as it
should have been, nor a compatibility equivalence. Still, the latter letter
sequence represents EXACTLY the same syllable as the two earlier character
sequences, and a proper rendering engine (of which there are already some,
I'm told) would correctly render the three sequences in the same way.
But for historical reasons, there is now neither a canonical, nor a compatibility
equivalence there.  Just an equivalence, in the same script, in syllabic meaning
and (when properly implemented) in display. (Yes, G and GG are pronounced
differently, but this is about spelling.)

This is something that 'nameprep' should handle, since it is unfortunately not
handled by NFKC.  The logical steps would be to 1) algorithmically decompose
Hangul syllables, 2) map cluster Jamos to the basic letter sequences each
represent. Then either (design decision) invoke NFKC or NFKC augmented
to compose also "modern" cluster Jamo's before the part of NFKC formation
that does algorithmic composition of Hangul syllables (the historic cluster
Jamos can (design decision) stay decomposed).  Or, indeed, do the
decomposition into basic (i.e. non-cluster) Hangul Jamo letters, after
conversion to NFKC form, leaving Hangul "subnames" as sequences of
letter characters, just like for other alphabetic scripts (I don't know how this
would effect the length of ACE encoded IDN names).  (Some thought
needs to go into how ((Halfwidth)) Compatibility Hangul Letters are to be
handled. The compatibility mapping are, ahem, not fully appropriate... 
The Hangul "filler" characters are also a problem, which needs to be
considered.)

          Kind regards
          /kent k

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 13 Nov 2001 06:55:09 +0900

Hi, Kent
Welcome to hangul problems.
You may have missed "hangulchar" I-D in WG pool.
Your analysis is thorough, but seems to be already addressed by the I-D.
And there have been more discussions offline more about other hangul nameprep
issues.That will be included new hangulchar 2.0 I-D soon.

Regards,

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 13 Nov 2001 07:16:59 +0900

----- Original Message ----- 
> syllable character expresses from 2 to 6 letters in one single character.

In hangul natural sentence,it may reach 2.6~7.
In hangul business names, the average number is  2.2 or so, i guess.

> and 11 vowel letters, plus some variant (and historic) letters.

If you encode each Hangul syllabic in 3 jamos in utf8,
it need 3 octets * 3 = 9 octets, while 3 basic latin letter need 3 octets in utf8.
3 times more space!  if there were any real "compaction" on hangul
syllable code points, that may be just the bare minimum.

From What i get from reorering experiments, It became clear that 
long han/hangul code points sequence of length N can be represented 
by 2.0~2.2 * N  latin letters. Without reordering, it would be 3.0~3.1.
33% improvement is possible! Why should we go without reordering
which merely require simple mapping tables with so many benefits?

Soobok Lee

---

From: "Bruce Thomson" <[email-masked]>
Date: Tue, 13 Nov 2001 11:16:04 +0900

> syllable code points, that may be just the bare minimum.

This statement seems quite misleading. 3 jamos is not
the average Hangul syllable, it is definitely high-end.
3 letters is quite short for an English syllable. If you
want a more impressive syllable example, try "glyphs".

The best way to solve the question of information content
per character is to use statistics from actual text examples.
You recall that this has already been done on the list with
the Book of Genesis, which gives a completely different
result from what you are saying here. Do you have another
statistical example that supports your opinion?

From: "Adam M. Costello" 

> Here are the counts for Genesis chapter 1:
>
> King James:     3167 letters
> Basic English:  3088 letters
> Chinese Union:   778 ideographs
> Korean Revised: 1201 Hangul

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 13 Nov 2001 11:25:24 +0900

----- Original Message ----- 
From: "Bruce Thomson" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Tuesday, November 13, 2001 11:16 AM
Subject: Re: [idn] reordering strawpoll

> statistical example that supports your opinion?

You may have missed my another answer to Kent which provides 2.6 as the average
researched by Adam. My above examples argument stands even with  2.6 jamos. 
2.6*3octets vs 2.6 latins == 3time more space needed. That was my point.
3 jamos syllable is just for an example. Okay? :-)

Soobok Lee

---

From: David Hopwood <[email-masked]>
Date: Tue, 13 Nov 2001 00:05:20 +0000

-----BEGIN PGP SIGNED MESSAGE-----

> Your analysis is thorough, but seems to be already addressed by the I-D.

AFAICS, draft-ietf-idn-hangeulchar-00.txt doesn't talk about this issue
(equivalence of Jamo clusters) at all. Have I missed something?

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 13 Nov 2001 12:06:46 +0900

----- Original Message ----- 
From: "David Hopwood" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Tuesday, November 13, 2001 9:05 AM
Subject: Re: Hangul and IDN (was Re: [idn] reordering strawpoll)

> (equivalence of Jamo clusters) at all. Have I missed something?

one of the jamo sequence he raised was illegal one ( choseong+jungseong+filler is illegal,
choseong+jungseong is enough). stand-alone filler (looks like space) is illegal use.

NFC already  map modern jamo sequences into modern hangul syllables. But,
Some illegal jamo/filler sequence are left unfiltered/unchanged and that cencern was raised
in the I-D.  I had proposed  such illegal/abnormal jamo/filler sequence should be
prohibited by nameprep. But, some nameprep authors are reluctant  to 
hard code such cases, because current UTC has no standard code for such case.
Instead, filtering out by registrations was proposed by them. 
currently not resolved issues.

Soobok Lee

> -----BEGIN PGP SIGNATURE-----
> Version: 2.6.3i
> Charset: noconv
> 
> iQEVAwUBO/BjojkCAxeYt5gVAQHlWwf/X8oWkgtNXJhaP19we8WrCepf5Iwtb3Cp
> BiZtpd5NYDSKePMWjnibv6t8Dp+JxHxhY3Ea6RmHE8kF5+alYDiHjrNyvaBR9TqH
> Kf1+TwZurn39lsQiRoUydO5g5HJBg2GyXq8opbwqxsDvrGXmrEYGVuBP3aV8s3OG
> tBjQRbhDTTqNARjOdNRU/MWJFoqj+5faBhduAWZXD0NMvu5b3Gi+Eczd7nnADgdZ
> kboN2mTiHvXFRZHHmBOyDglF/GrqOH/mYdLuWDsqo4opwAq+o7kXGlGWnrRIRq4q
> hSHPSoQZ8h5o1zJFv5NDSDV45sQ8K6FqTjEUrXdIBuxg/h168WNVXA==
> =PD5A
> -----END PGP SIGNATURE-----
>

---

From: "Bruce Thomson" <[email-masked]>
Date: Tue, 13 Nov 2001 15:52:56 +0900

> 3 jamos syllable is just for an example. Okay? :-)

I do see what Adam researched, and it was that there were 2.6 English letters/Hangul
in terms of information content. (He is comparing the length of the Hangul translation
of the bible with the English.) This is a completely different number from the number
of jamos/syllable. This is getting more confusing that it needs to be but it would help
if you quote a bit of the original research you refer to. Like this:

Written by Adam:

> references:
> http://www.ccim.org/bible/
> http://bible.wisenet.co.kr/
>
> So it's about 4.0 English letters per Chinese ideograph, and about 2.6
> English letters per Korean Hangul.
>
> Each Korean Hangul takes about 2.9 octets in AMC-ACE-Z, which means a
> maximal Korean domain label (20 hangul) holds about as much information
> as a 52-letter English string, which about 17% less information than
> a maximal English domain label (63 letters), and about 38% less
> information than a maximal Chinese domain label (19 ideographs).
>
> I now retract this statement:
> 
>> Of all the languages I've looked at, Korean is by far the least dense
>> when encoded using AMC-ACE-Z.
>
> In light of the new data, I doubt that Korean is the least dense.
>
> AMC

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 13 Nov 2001 11:10:26 +0100

> choseong+jungseong is enough). 

I had no such example. Look again!  The "FILLER"s are problematic, but was
not the main point in my previous e-mail.

> NFC already  map modern jamo sequences into modern hangul syllables.

Not for the example I picked (in it's full three-character form)!  It was not a random
example, but there are thousands of similar instances.

        Kind regards
        /kent k

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 13 Nov 2001 20:48:01 +0900

----- Original Message ----- 
> (G, G, A), 

> but this equivalence is neither a canonical equivalence, as it

Are you raising this possiblity:    U+1101 <---> U+1100 U+1000  (GG <-> G G) ?
Design of conjoining (cluster) jamos treat two choseong sequence 
U+1100 U+1000  as illegal sequence (syllable break condition). That is described somewhere in Unicode 3, chapter 3,section 11. That will help you.

if some rendering engine display the two as the same syllable,
I suspect the product is buggy or beyond the standard. :-)
Would you tell me the version of the rendering engine ?

> The Hangul "filler" characters are also a problem, which needs to be

After reading unicode 3 3.11 section, would you rephrase this question
again in this list? maybe you will have different idea on the real hangul
jamo processing. 

Sorry for misreading your posting. I have been hurrying on... :-)

Cheers,

Thanks.

Soobok lee

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 13 Nov 2001 14:04:46 +0100

Hi!

>U+1100 U+1000  as illegal sequence (syllable break condition).

(You mean <U+1100, U+1100>.) No, that is not a syllable break condition. 
See tables 3-4 and 3-5 on page 53 in TUS 3.0 (link below).

(B.t.w. apart from algorithmic decomposition (per se) of Hangul syllable characters,
the cluster Jamos are not needed, and should ideally not be used. That is not
spelled out in TUS 3.0, though...)

> That is described somewhere in Unicode 3, >chapter 3,section 11. That will help you.

Please reread that section carefully! (Link below.) In particular page 53.

>I suspect the product is buggy or beyond the standard. :-)

No, it's not beyond the Unicode standard at all.  See page 53 of TUS 3.0,
http://www.unicode.org/unicode/uni2book/ch03.pdf which says (my emphasis):
"A standard syllable block is composed of a *sequence* of choseong followed
by a *sequence* of jungseong and optionally a *sequence* of jongseong."

That that description is only about NFD form is not spelled out, nor
is the fact that combining characters, in particular a Hangul tone mark,
may follow (logically they apply to the entire syllable!).  Whether to consider
the combining characters as part of the syllable or not, I think is a matter
of taste. If we also take combining characters into account, but still NFD,
the syntax for a Hangul syllable is:

        Hangul-syllable-NFD  ::=   C+  V+  F*  T*

where C is a choseong, a V is a jungseong (vowel), F is a jongseong, and T is a
combining character (like a Hangul tone mark).  (I'll ignore the FILLER issues
for the moment, including their automatic insertion.)

This is needed to be able to spell historic (and future) Hangul texts that may use
consonant or vowel clusters that are not given a character of their own.

Taking into account what NFC may cause, the full Hangul syllable syntax is:

Hangul-syllable  ::= 
                 C*  CVsyllable  V*  F*  T* 
            |    C*  CVFsyllable  F*  T*  
            |    C+  V+  F*  T*

where CVsyllable is a consonants-vowels syllable character, and CVFsyllable is a
consonants-vowels-consonants syllable character.

>Would you tell me the version of the rendering engine ?

I'm told(!) this is (properly!) implemented at least in Windows XP and
IE 6 (maybe also IE 5.5).  Perhaps also elsewhere (I'm not keeping track).

        Kind regards
        /kent k

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 14 Nov 2001 00:06:10 +0900

Hi, Kent and Shin

Thanks for your valuable contributions. I promise to investigate it thoroughly.
First, please test the enclosed link in IE 5.0, 5.5, 6.0  in Win 2K and Win XP.
That contains some javascript codes to make on-the-fly html containing
the 3 sequences you give me.
In my win2k and IE 6.0 (english version) rendering, 
syllable breaks are assumed in the choseong sequence.
I will test this link in other platforms.

Regards. Soobok lee

http://164.124.123.207/etc/f8.html

%u1100%u1100%u1161 ,  NFKC  may try  partial combination:  %u1100%u1161 --> %AC00 (below)
ᄀ가 

%u1101%uAC00  
ᄀ가 

%u1101%u1161                NFKC may try to unfiy this into  : %AE4C (below)
까 

%uAE4C 
까 
---------------------------------

This is RACE conversion using NFKC:

1100 1100 1161
bq--3aiqblaa

1100 AC00
bq--3aiqblaa

#####  the above two sequence are unified.

1101 1161
bq--vzga

AE4C
bq--vzga

---

From: Jungshik Shin <[email-masked]>
Date: Tue, 13 Nov 2001 21:08:57 -0800

On Wed, 14 Nov 2001, Soobok Lee wrote:

 Hi,

> First, please test the enclosed link in IE 5.0, 5.5, 6.0  in Win 2K

  I'm sure MS IE 5.0/5.5 doesn't work.  As I wrote earlier,
even MS IE 6.0 under Korean MS Office XP wouldn't work for sample
cases in your test page (MS hasn't yet added support of generic Hangul
syllable formation stipulated in TUS 3.0 for modern Korean - which
can be represented by precomposed Hangul syllables at U+AC00 block -
although it supports that for Middle Korean.) However, two test cases
in my test page at <http://jshin.net/i18n/uyeo.html> are likely to work
in MS IE 6.0 + Korean MS Office XP/Korean MS Windows XP.

> http://164.124.123.207/etc/f8.html

  BTW, I'm afraid your javascript(ECMAscript) is not compliant to
ECMA-262
(ftp://ftp.ecma.ch/ecma-st/Ecma-262.pdf) and  ISO/IEC 16262.  unescape()
is not a part of the standard (see section B.2 of ECMA-262) so that
unescaping "%uhhhh" is not supported by Netscape   although it's
supported
by MS IE.  You can simply use "\uhhhh" to include Unicode characters in
a string (see section 7.8.4 of ECMA-262)

  Regards,

  Jungshik Shin

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 16 Nov 2001 21:27:59 +0900

Hi, Shin and Kent

hangul jamo sequences like LLV(choseong1+choseong2+jungseong) are already
normalized by NFC partial jamo combination into (L)(LV).
If we do not modify current version of NFC jamo handling behavior,
we should add some (L)(LV) <==> LLV equivalence for correct rendering
implementation.

Another partial jamo combination is also found in
NFC's handling of (choseong+jongseong+ ancient jongseong) sequence.
: LVFa ---> (LV)(Fa)
Martin Duerst had said the latter normaliation may not be a fault, but
(LV)(Fa) is just an intermediate form of hangul syllable.
(Martin, correct me if i am misrepresent you ).

Martin and Mark,would you comment on these partial jamo combination ?

Regards,

Soobok Lee

----- Original Message -----
From: "Jungshik Shin"
To: "Soobok Lee"
Cc: "Kent Karlsson" ;
Sent: Wednesday, November 14, 2001 2:08 PM
Subject: [idn] Re: new hangul nameprep issue

> in my test page at are likely to work

---

From: "Kent Karlsson" <[email-masked]>
Date: Fri, 16 Nov 2001 13:48:36 +0100

hangul jamo sequences like  LLV(choseong1+choseong2+jungseong) are already
   normalized by NFC partial jamo combination  into (L)(LV).
Yes.
If we do not modify current version of NFC jamo handling behavior,
   we should add some (L)(LV) <==> LLV equivalence for correct rendering
   implementation.
Well, it's too late to change NFC.  For a rendering engine, the bestapproach may to map the input to a sequence of basic
(non-cluster) Jamos,and then let the font and font support mechanisms deal with contectual(sub)glyph selection, ligature (sub)glyph
selection and (sub)glyphpositioning.  Note that a handfull of glyph variants are used for eachbasic Hangul Jamo letter.  I don't
know to what extent ligatures maybe needed.  Both of these are font matters, not encoding matters.For efficiency, in a rendering
engine, syllable characters that donot have adjacent Jamos may use precomposed glyphs.
Another partial jamo combination is also found in
   NFC's handling of (choseong+jongseong+ ancient jongseong) sequence.
   : LVFa ---> (LV)(Fa)
Martin Duerst had said the latter normaliation may not be a fault, but
(LV)(Fa) is just an intermediate form of hangul syllable.
Not quite. Its a (single) complete syllable, represented using twocharacters, one of which is a (conjoining!!) syllable
character.(As Unicode stands, there is nothing intermediate about it.)

        Kind regards
        /kent k

Regards,

Soobok Lee

---

From: Yoshiro YONEYA <[email-masked]>
Date: Tue, 20 Nov 2001 18:18:01 +0900 (JST)

On behalf of counters, I report the summary of reordering strawpoll
done during Nov 12-16.

There were 55 mail in total during the period, and 2 were from the
same, so 53 were valid.  Following list shows senders' domain names.

NO: 30 (there were 32 NO but 2 were duplicated)
aist.go.jp
alldomains.com
bic.nus.edu.sg
cequrux.com
cs.com
dsl.gr.jp
enic.cc
extundo.com
fm-net.ne.jp
goto.info.waseda.ac.jp
imc.org
ItsYourDomain.Com
jprs.jp
jprs.jp
linux.or.jp
macchiato.com
md.chalmers.se
microsoft.com
nic.ad.jp
nominum.com
po.ntts.co.jp
pobox.org.sg
pobox.org.sg
realnames.com
sra.co.jp
verisign.com
verisign.com
w3.org
zetnet.co.uk
sharif.edu

YES: 21
cc.ncu.edu.tw
cnnic.net.cn
cnnic.net.cn
cnnic.net.cn
gate.sinica.edu.tw
iis.sinica.edu.tw
iis.sinica.edu.tw
mail.moe.gov.tw
ms1.url.com.tw
nic-naa.net
nic.or.kr
ports.se
postel.co.kr
sinica.edu.tw
spsoft.co.kr
twnic.net.tw
twnic.net.tw
twnic.net.tw
twnic.net.tw
whale.cnnic.net.cn
whale.cnnic.net.cn

ABSTAIN: 2
lysator.liu.se
switch.ch

-- 
Yoshiro YONEYA <[email-masked]>
           aka <[email-masked]>

----
From: "James Seng/Personal" <[email-masked]>
Subject: [idn] reordering strawpoll
Date: Mon, 12 Nov 2001 18:35:49 +0800

---

From: "Kenny Huang" <[email-masked]>
Date: Tue, 20 Nov 2001 17:32:49 +0800

What does the statistics mean ? 

Kenny Huang

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On 
> Behalf Of Yoshiro YONEYA
> Sent: Tuesday, November 20, 2001 5:18 PM
> To: [email-masked]
> Subject: [idn] SUMMARY: reordering strawpoll
> 
>

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 21 Nov 2001 01:08:48 +0800

Yoneya-san,

Thanks for posting the strawpoll summaries.

Remember, strawpoll is not a vote. the number in itself (21 or 30) only
means something depending on our interpretion.

So Kenny, let me ask the question back to you :-) What do you think this
statistic means?

-James Seng

----- Original Message -----
From: "Kenny Huang" <[email-masked]>
To: "Yoshiro YONEYA" <[email-masked]>; <[email-masked]>
Sent: Tuesday, November 20, 2001 5:32 PM
Subject: RE: [idn] SUMMARY: reordering strawpoll

Kenny Huang