Subject: RE: [idn] call for comments for REORDERING

From: Karlsson Kent - keka <[email-masked]>
Date: Mon, 22 Oct 2001 19:05:56 +0200

> -----Original Message-----
> From: Soobok Lee [mailto:[email-masked]]
...
> But including 11172 hangul made Unicode more useful and 
> widely acceptable to korean community, because existing 
> KS5601's 2350 or more hangul has been criticized for not being
> able to serve many scholarly purpose or book publishings.
> 
> The pain for inclufing 11172 hangul  was REWARDED!

And still it is not sufficient, since historic Hangul
Jamo aren't covered by the precomposed Hangul syllable
characters. I have also indications that many useful
consonant (and vowel?) clusters aren't covered as separate
(Jamo) characters, but have to be constructed from multiple
'more basic' (non-cluster) Hangu Jamo characters. (The
cluster ones are the ones that had compatibility
decompositions in Unicode 2.0, but don't have such
decompositions in Unicode 3.0.)

> I know current NFC/NFKC has *severely flawed* hangul jamo 

Now that's a new complaint.  At least to me.  Not even the
delegations from DPRK complain about that. (And they have
all sorts of complaints.)  Do you care to say a little bit
more about what you mean?  Or are we just to take your word
for it?  There was a problem in earlier drafts of UAX 15
concerning Hangul and NFKC (Hangul syllables were not preserved),
but that problem was removed before that UTR became a UAX.

> I hope UTC cures this BUGGY NFC/NFKC jamo handling as an 
> exceptional case while it is in  strict adherence to UTR's backward 
> compatiblity policy & promise.

The normal forms for strings of already allocated characters
will not be changed any more.

		Kind regards
		/Kent K

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 23 Oct 2001 03:40:39 +0900

----- Original Message ----- 
From: "Karlsson Kent - keka" <[email-masked]>
> but that problem was removed before that UTR became a UAX.

two problems:

 1) NFKC maps   compatibility hangul jamo (U+32??) into
                conjoining hangul jamo without fillers.
    That converts compat hangul jamo sequences into conjoining
      jamo sequences and they are combined partially in some cases
        by the NFC following  compatibilty decomposition.
    Modern hangul informal texts contains compatibiilty jamos very often.
    If the texts contains choseong + jungseong sequence in compat jamo,
       they will be affected by this fault.
    This fault was found by Soobok Lee, too lately.

 2) NFC  does not combine  archaic(old) hangul trailing jamo into the
     preceding half-combined syllable (from choseong jamo + jungseong jamo),
      but leave it alone as an isolated conjoining jamo.
    That means archail hangul syllable which has old hangul trailing jamo 
      cannot be preserved through out NFKC.
    NFKC cannot be used for scholarly texts which contains archail hangul 
      syllables.  This fault was found by Martin, too lately.
    
Mark and Martin knew and told me why this happened.
Martin, would you comment on this?

Soobok Lee

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 22 Oct 2001 20:39:02 -0700

1. The compatibility hangul jamo are not the preferred way to express
isolated jamo in Unicode. That would be by the use of conjoining jamo with
fillers. If the preferred forms are used, there are no problems.

2. NFC does not combine ancient Hangul into syllables because there are no
syllables that have ancient Hangul in them. It does combine as many modern
Jamo as possible into syllables. That does not -- at all -- prevent the use
of NFC with ancient Hangul.

Mark
—————

Δός μοι ποῦ στῶ, καὶ κινῶ τὴν γῆν — Ἀρχιμήδης
[http://www.macchiato.com]

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "Karlsson Kent - keka" <[email-masked]>; <[email-masked]>
Cc: "Martin Duerst" <[email-masked]>; <[email-masked]>
Sent: Monday, October 22, 2001 11:40
Subject: Re: [idn] call for comments for REORDERING

----- Original Message -----

two problems:

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 23 Oct 2001 13:06:38 +0900

----- Original Message ----- 
From: "Mark Davis" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "Karlsson Kent - keka" <[email-masked]>; <[email-masked]>
Cc: "Martin Duerst" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2001 12:39 PM
Subject: Re: [idn] call for comments for REORDERING

> fillers. If the preferred forms are used, there are no problems.
But, KSC5601-->Unicode mapping tables  maps  KSC5601 isolated hangul jamo
into compat hangul jamo. The mapping table was issued by UTC itself and
placed into WIN98,2K,XP and most unix and linux platforms.
Inconsistency!

> of NFC with ancient Hangul.

Rendering problem remains unsolved. 
"hangul  syllable ga + old jamo bansioth " != "old hangul syllable ga+bansioth".
There is defined no such false equivalence  in UTR15.
 Why rendering engine vendors should  make such false workarounds not
recommended in UTC standards?
NFC is  dangerous for ancient hangul.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 23 Oct 2001 12:36:55 +0900

>                 conjoining hangul jamo without fillers.

Well, NFKC is just a combination of many kinds of mappings,
and in most cases should be applied piece-by-piece, not
wholesale.

>       but leave it alone as an isolated conjoining jamo.

Yes. To give a bit more of a background to those not too familiar
with the topic, Hangul Jamo can be categorized along two axes:

Position:
- Initial consonants (or consonant clusters)
- Vowels (or vowel clusters)
- Final consonants (or consonant clusters)

History:
- Used in modern times
- Historical use only

This gives six categories: Im, Ih, Vm, Vh, Fm, Fh

The huge block of 11172 is the result of forming all
the following combinations:

- Im Vm      (for later reference, let's denote that by IV)
- Im Vm Fm   (for later reference, let's denote that by IVF)

NFC (and therefore NFKC, by the fact that composition in NFKC is
the same as in NFC) whenever possible tries to use the precomposed
IV or IVF, rather than the individual Im, Vm, and Fm.

There was a design choice to make for the case where historical
components (Jamos) get involved. In all the following cases:

- Im Vh
- Im Vh Fm
- Im Vh Fh
- Ih Vm
- Ih Vm Fm
- Ih Vm Fh
- Ih Vh
- Ih Vh Fm
- Ih Vh Fh

It is not possible to do any composition, because the combinations
are not available. However, in the case of

- Im Vm Fh

i.e. a modern initial, a modern vowel (cluster), followed by a
historical final, the question was whether this should result in

a) IV Fh
b) Im Vm Fh

The arguments for the first were to precompose as much as possible
and to avoid to have to do lookahead. The arguments for the second
might have been that historical texts would be treated more uniformly
(which doesn't apply because syllables such as Im Vm Fm -> IVF
occur in historical texts and by NFC are definitely recomposed),
or that historical syllables might be treated more uniformly
(a syllable would be either completely decomposed or completely
precomposed). The second argument has some point, but neither
linguistic analysis (for which NFD is best) nor rendering
(which has to deal with lots of combinations anyway) are
severely affected.

>       cannot be preserved through out NFKC.

First, NFKC isn't an issue here. It's the same for NFC. Second, the actual
syllable is always preserved, whether as IV Fh or whether as Im Vm Fh.
These represent one and the same thing (and that's why we need some kind
of normalization in the first place).

>       syllables.

Again, it's an issue that applies to both NFKC and NFC.
And both can be used for archaic hangul syllables without
any problems.

>This fault was found by Martin, too lately.

It's not a fault, it's by design. And it wasn't found lately, either.
Actually, I remember that it was explicitly discussed at an Unicode
Technical Committee meeting in which I participated by phone.

>Martin, would you comment on this?

As done above. Hope this helps.

Regards,   Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 23 Oct 2001 14:10:20 +0900

>Rendering problem remains unsolved.

What do you mean? Not all current implementations may be able to
render IV Fh correctly, but it's very easy to change the implementation
to do it correctly (assuming the application already renders
Im Vm Fh).

>"hangul  syllable ga + old jamo bansioth " != "old hangul syllable 

What kind of evidence do you have for the 'not equal'?

>There is defined no such false equivalence  in UTR15.

There is no character for "old hangul syllable ga+bansioth",
so in that sense indeed UTR 15 doesn't define such an equivalence.
What UTR 15 defines is the following equivalence

    Im Vm Fh <==> IV Fh
      (NFD)       (NFC)

This equivalence is clearly defined in UTR 15, in the same way
the following equivalences are defined:

    Im Vm Fm <==>   IV Fm   <==> IVF
     (NFD)     (intermediate)   (NFC)

UTR 15 doesn't have the concept of false equivalence.

>recommended in UTC standards?

There is nothing false with making sure IV Fh is correctly
rendered. For some implementations, it may be a bit of a
workaround, but it's a very easy one.

Also UTR 15 (also called UAX 15) is an integral part of the Unicode
Standard. Please see http://www.unicode.org/unicode/reports/:

A Unicode Standard Annex (UAX) forms an integral part of the Unicode
Standard, carrying the same version number as the standard, but is
published as a separate document. Note that conformance to a version
of the Unicode Standard includes conformance to its Unicode Standard
Annexes.

It may be true that there is no part of The Unicode Standard that
explicitly says that IV Fh should be rendered as a single syllable
block, but there are many, many other details about rendering that
The Unicode Standard doesn't say explicitly.

>NFC is  dangerous for ancient hangul.

I wouldn't know why. If you know, please explain.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 23 Oct 2001 12:16:44 +0900

At 19:05 01/10/22 +0200, Karlsson Kent - keka wrote:

>'more basic' (non-cluster) Hangu Jamo characters.

Now that's new to me. Can you provide some evidence
(e.g. some scans)?

I use to write my last name as a Hangul syllable as a joke,
but as the 'rst' combination doesn't turn up in as a final
cluster, I have to do this using graphics. So I'm familliar
with the problem. But I assumed that all actual combinations
that have been used historically are actually available
in the Unicode Jamo block.

Regards,  Martin.

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 23 Oct 2001 12:57:26 +0200

----- Original Message ----- 
From: "Martin Duerst" <[email-masked]>
...
> Now that's new to me. Can you provide some evidence

I don't have any scans, but a quote from the public Unicode mailing list.
In http://www.unicode.org/mail-arch/unicode-ml/y2001-m10/0253.html:
:Unicode Hangul Jamo block has about 220 Hangul Jamos used in modern Korean 
:and middle Korean. Why are there so many? Because instead of encoding only 
:17 consonants and 11 vowels (one can go further down the 'reductionist' 
:road and say that only several base consonants and a few base vowels 
:along with 'diacritical' elements are all necessary to represent/encode 
:Hangul. See p. 143 of Sampson), Unicode encodes all forms of 'composite 
:Jamos' (consonant clusters, 'double' consonants to represent tense 
:consonants, complex diphtongs, etc). However, this is not sufficient 
:and Microsoft Word 2002/Windows XP/MS IE 6.0 comes with truetype fonts 
:that can be used by 'Uniscribe' to combine dynamically Hangul Jamos 
:beyond a simple I.C + M. V. (+ F.C.) model (I.C. = initial consonant, 
:M.V. = medial vowel, F.C= final consonant) as described in Unicode 3.0 
:.......... 
:
:  Jungshik Shin 

I assume that this functionality is not included just for fun...

            Kind regards
            /kent k

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 23 Oct 2001 10:15:58 -0700

1. Yes, the mapping tables for (1) were singular mappings, so it was not
possible to add the fillers. I agree that ideally the decomposition could
have included fillers -- and that might have been a better choice in
retrospect-- but the compatibility mappings are intended to preserve
fundamental character identity, not formatting -- including whether the jamo
are grouped into a single box in display or not.

Some time ago I asked whether there were any cases where it was absolutely
necessary to have two different IDNs, one with a syllable and the other with
two separate Jamo. Nobody responded with any examples. If you have some,
that would be productive.

2. Martin wrote a very nice response on this, so I won't add anything.

Mark
—————

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "Mark Davis" <[email-masked]>; "Karlsson Kent - keka" <[email-masked]>;
<[email-masked]>
Cc: "Martin Duerst" <[email-masked]>; <[email-masked]>
Sent: Monday, October 22, 2001 21:06
Subject: Re: [idn] call for comments for REORDERING

----- Original Message -----
From: "Mark Davis" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; "Karlsson Kent - keka" <[email-masked]>;
<[email-masked]>

"hangul  syllable ga + old jamo bansioth " != "old hangul syllable
ga+bansioth".