Subject: [idn] stringprep comment 5: hangul conjoining sequence

From: "Soobok Lee" <[email-masked]>
Date: Tue, 12 Feb 2002 01:12:16 +0900

Two issues briefed in hangulchars drafts and in this mailling list, but not mentions in
  the latest nameprep and stringprep drafts:

 1. When  old trailing hangul jamos are included in conjoining jamo sequences, UAX15(NFC)
     performs  partial combinations to produce  "a modern hangul syllable(LV) + a standalone
      old hangul jamo(oT)" and that form satify a ridiculous syllable break condition (X.T) .

 2. UAX15 also tries ridiculous partial combinations when it met a combining sequence if
      two or more leading hangul jamos followed by hangul vowel jamos, and produce
      a syllable break condition (L.LV)

 3. Compatibility hangul jamos are mapped into conjoining jamos without any fillers.

 Errors briefed in 1,2 and 3 came from  past poor cooperations between UAX15 authors and Korean UTC participants, and NFC routines
was already widely distributed.

 Now I propose UTC make new normalizations (call it NFN)to correct such errors and faults and
    let stringprep include it after casefolding : that is  NFKC(NFN(casefold(x)).

 We dont need to modify NFC and NFKC. Just add new correcting normalizations as soon as possible.

 Stringprep efforts should be in sync with further UTC's works. No hastes!

 Soobok Lee

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 11 Feb 2002 08:44:03 -0800

Syllable breaks are different than character boundaries. Please see
http://www.unicode.org/versions/beta.html, where that is clarified.

Now, of course, it would be possible to choose a different
normalization. That has, however, been debated at great length on this
list; see the archives. I see no pressing need for change.

Mark
—————

Πόλλ’ ἠπίστατο ἔργα, κακῶς δ’ ἠπίστατο πάντα — Ὁμήρου Μαργίτῃ
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: <[email-masked]>
Cc: "Martin Duerst" <[email-masked]>; "Mark Davis"
<[email-masked]>
Sent: Monday, February 11, 2002 08:12
Subject: stringprep comment 5: hangul conjoining sequence

>  Errors briefed in 1,2 and 3 came from  past poor cooperations
between UAX15 authors and Korean UTC participants, and NFC routines

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 12 Feb 2002 01:47:38 +0900

----- Original Message -----
From: "Mark Davis" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Cc: "Martin Duerst" <[email-masked]>; "Mark Davis" <[email-masked]>
Sent: Tuesday, February 12, 2002 1:44 AM
Subject: Re: stringprep comment 5: hangul conjoining sequence

> list; see the archives. I see no pressing need for change.

That is a fundamental issue to us Koreans, not merely a practical or engineering
issues. The language and scripts are like  parts of our bodies and minds and spirits.

All IETF participants!
Be loyal to and pay greatest repects for local language communities, please.

Soobok Lee

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 11 Feb 2002 18:32:15 +0100

> -----Original Message-----
> From: Soobok Lee
...
> syllable break condition (X.T) .

There is no syllable break there.  There may be a sequence of L before
a Hangul syllable character, and a sequence of T after it, without any syllable
break inbetween.  For some of the Hangul syllable characters, there may even
be both Vs and Ts after, without there being a syllable break.

>       a syllable break condition (L.LV)

There is no syllable break there.  [Skipping syllable characters for the moment]
Note that a Hangul syllable consists of a non-empty **sequence** of L, followed
by a non-empty **sequence** of V, followed by a (possibly empty) **sequence** of T.
Note that in many cases compatibility equivalents with regard to these were
(erroneously) made non-equivalent between Unicode 2.1 and Unicode 3.0. 

> jamos without any fillers.

Compatibility (non-conjoining) Hangul letters are best prohibited.  Doing
the correct mapping is not expressible in via nameprep without adding
a new, special for Hangul, mechanism.  Would there be any major problems
just prohibiting them? (Allowing only the conjoining jamo and syllable characters.)

...
>     let stringprep include it after casefolding : that is  

I have suggested a solution that involves only additions to the tables in
"nameprep" (these include prohibiting non-conjoining compatibility Jamo
as well as the Hangul filler characters).  Table available upon request.

I agree that it is very unfortunate that "letter sequence" equivalent strings,
like [gg] (SSANGKIYEOK) and [g][g] (<KIYEOK, KIYEOK>) are not
formally equivalent in Unicode; indeed these should have been canonically
equivalent; but the normal forms are by now frozen, and I don't think anyone
wants to have yet another Unicode formal equivalence or normal form.

		Kind regars
		/kent k

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 12 Feb 2002 03:14:35 +0900

Hi, Kent

----- Original Message ----- 
From: "Kent Karlsson" <[email-masked]>
 
> just prohibiting them? (Allowing only the conjoining jamo and syllable characters.)

That's impossible. Unicode-issued KSC5601 ==> UNicode mapping tables
 provides "hangul legacy code jamos --> Unicode u+3??? compability jamos".
 When ordinary Korean users enter hangul jamo sequence in 
all version of Windows or LInux which adop the table above, 
those legacy-coded jamo sequences will be eventually converted into 
Unicode compatibility jamos sequence. There is no way to input directly  conjoining jamo sequences
including necessary fillers and that will be painfully complex to end users.

And Even windows and linux does not render correctly the conjoining jamo sequence including fillers.
The compability jamos are the only viable alternative for correct jamo sequence rendering in
commerical windows and linux platforms.

Soobok

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 11 Feb 2002 20:03:42 +0100

> From: Soobok Lee [mailto:[email-masked]]

>  provides "hangul legacy code jamos --> Unicode u+3??? 

Yes. 

> including necessary fillers and that will be painfully 

Keyboard drivers+IME handling for Hangul often take Hangul compatibility
letters and (when the IME deems appropriate) **replaces** subsequences
of (maybe a Hangul syllable character and) Hangul compatiblity letters with
a Hangul syllable character.

It is true that many IMEs may not be able to produce all and any Hangul syllable,
in the logical sense, not even all and any Hangul syllable Johab character.  But
that is beside the point here.

> And Even windows and linux does not render correctly the 
> conjoining jamo sequence including fillers.

True, but that is a lack of support issue with those systems, and should be
taken up in some other forum.

> commerical windows and linux platforms.

The compatibility letters render as spacing characters; they do not normally
conjoin into syllable blocks.  Doing that analysis (with a filler to tell that a
"compatibility 'maybe conjoning'" sequence begins) of the input and proper
conversion to Hangul conjoining jamo is still beyond what nameprep can do
with just simple additions to the tables. Adding a proper, and Hangul specific,
conversion step *to nameprep* seems inappropriate.  This is an input issue.
But yes, that conversion may be useful, but I would not push it into nameprep.

A "better keyboard" could generate the conjoining single-letter jamos directy,
without using any IME (17 consonants (twice, via shift: lead, trail) and 11 vowels
plus some very few historic variants fit easily on a keyboard).  Indeed, there would
then be no problem in typing all and any Hangul syllable (in the logical sense), as
long as the result could be fitted into a rendered syllable block. But such a keyboard
would be appropriate only after the display issue became obsolescent; which it
isn't yet.  However, it may someday relatively soon be an obsolescent rendering
problem.  (Netscape 6 is close to rendering them correctly already:
http://jshin.net/i18n/jamo.html)

		Kind regards
		/kent k

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 11 Feb 2002 21:48:32 +0000

Kent Karlsson <[email-masked]> wrote:

> Compatibility (non-conjoining) Hangul letters are best prohibited.

Stringprep does normalization before prohibition, so prohibiting
compatibility characters would have no effect.  After the normalization
step, there are no compatibility characters in the string.

A trick you could play, however, would be to map the characters in
question to an arbitrary prohibited character that survives the
normalization step (like U+0000).

AMC

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 12 Feb 2002 12:41:39 +0100

> From: [email-masked] 
> [mailto:[email-masked]]On Behalf Of
> Adam M. Costello
> Sent: den 11 februari 2002 22:49
> To: [email-masked]
> Subject: Re: [idn] stringprep comment 5: hangul conjoining sequence
> 
> 

Ouch!  That leads to results that are completely off the board:
NFKC or NFKD on Hangul compatibility letter sequences leads
to completely wrong results; in particular the distinction between
lead and trail consonants becomes just plain wrong.

> normalization step (like U+0000).

Ok, consider that my revised suggestion.

		Kind regards
		/kent k