Subject: Re: [idn] An ignorant question about TC<-> SC

From: liana Ye <[email-masked]>
Date: Tue, 23 Oct 2001 13:02:23 -0700

From screen display point view, TC/SC are different glyph 
 sets(who defines the sets? How is it used by 1/5 of the world 
population? Is Uicode group the only authoritive one? In 
China there are over 600 recorded views on this).

 From linguistic on phoneme analysis,  TC/SC are identical by 
all authoritive dictionaries and standards with exceptions 
listed (for dialective and historical use).

 From a higher level anaysis, the semantics are the same by 
all authoritive dictionaries and standards with exceptions listed
and historically differenciated (in different dictionaries) and 
frequence of usage in current sectors (by software implementors). 

If for Latin only with 26 letters and 10 numbers, there are upper
i and lower L looks alike, 'I' and 'l', and 'O' '0' looks alike, then 
for 4,000 to 20,000 symbols the ratio is 4:36, a 1/9 not one-one
mapping case.  This is a much higher ratio than standard TC/SC 
mapping.  So yes, all language symbols share the same types 
of characteristics but with defferent magnitude. 

To treat UCS code points on the same base, the 4,000 to 20,000
number needs to be doubled, and treat TC/SC in a general way 
is a test for correctly treat 8,000 to 56,000 symbols for the long run. 
Since the 2,000 has pair-mapping( for sloppy discussion), but not 
all one-to-one (for another 17), the others are not have case 
mapping on another 2000, so do Indian languages.  The 4000 are 
high frequency symbols, similarly for Kanji and Hanja. These
symbols have highest chance to collide in IDN names, and critical 
to map them in registration and IDN usage. 

With the 2000 pair-mapping taken care of as case mapping, the 
above 4000 increases to 6,000, a great drop in processing 
complexity.  If the other symbols follow the same equivalent 
mapping, then the number getting better and manageable. 

Liana

On Tue, 23 Oct 2001 08:59:23 -0400 John C Klensin <[email-masked]>
writes:
> While reading David's NFC versus NFKC note, I had an odd thought.
> I've been dissatisfied, as have many others, with the notion that
> TC <-> SC mapping is analogous to case mapping in Roman-derived
> alphabets.   Arguments about whether that analogy applies have
> helped to make the discussion of what is, to me, a very difficult
> topic even more obscure.
> 
> To quote the Unicode standard, "Serbo-Croatian is a single
> language with paired alphabets".  This is a definition with which
> native speakers of the language agree (although, when tensions in
> the Balkans are high, I assume some of them are not completely
> happy about it).  Would it be constructive to think about Chinese
> as "one language, two alphabets"?  If it is, then nameprep or a
> related process ought to be able to map back and forth between
> the Roman-based characters usually used in Croatian contexts and
> the Cyrillic characters usually used in Serbian ones (people do
> this all the time, and certainly expect the two to match).
> 
> Of course, the analogy is not exact (these things never are):
> perhaps partially because there are just fewer characters to deal
> with, there are no cases in which there are potential ambiguities
> in the mappings.  On the other hand, one problem is more severe
> than in the Chinese case: in the general case, a Serbo-Croatian
> string written in Cyrillic cannot be distinguished, on a
> character string basis, from uses of Cyrillic for other languages
> (e.g., Russian), which should not be mapped and, similarly, a
> string written in Roman-based characters cannot be distinguished,
> on a character string basis, from the Roman-based characters of
> another language (English?) which, again, cannot be mapped.
> 
> In either case, the mapping becomes readily plausible if the
> language, in addition to the content of the character string, is
> known, but is hard to think about without causing side-effects in
> other languages if not.
> 
> Is that helpful?
>      john
> 
>

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 24 Oct 2001 09:30:07 +0800

> listed (for dialective and historical use).

It should be "TC/SC Phrase" to be more exact.

There is also a problem with "identical by all authoritive
dictionaries". All dictionaries have (slight/some) differences in what
they considered identical. The devil is in the level of details.

> is a test for correctly treat 8,000 to 56,000 symbols for the long
run.

There are 70,000+ han ideograph in ISO10646:2001.

-James Seng

---

From: Martin Duerst <[email-masked]>
Date: Thu, 25 Oct 2001 16:27:10 +0900

>China there are over 600 recorded views on this).

The Han ideographs in Unicode/ISO 10646 are defined by the
IRG (Ideographic Raporteur Group). This group reports to
ISO/IEC SC2 WG2, the ISO WG responsible for ISO 10646.
It is composed of representatives from all the countries
or similar entities interested in Han ideographs. That
includes China, Japan, Korea (both South and North), Taiwan,
Hong Kong, Singapore, and the US (I hope I didn't forget
anybody, and please excuse the maybe politically uncorrect
shortcuts). The US is the only country represented without
a tradition of using Han ideographs, but usually only
sends a small delegation and mainly helps with wording.
Many other countries may send rather large delegations
(given the number of characters, which means a lot of
work, this is no surprise). The Unicode consortium
participates in the IRG with an observer status only.

The IRG has published guidelines for deciding when to
unify two occurences and when not. Because of the very
huge number of characters, there is in some cases indeed
a thin line as to whether something should be unified or
not. And in these cases, the IRG just has to make a decision.

Overall, the guidelines are somewhat difficult to understand
at first, but they are designed mainly with a 'least surprise
to the average user' in mind, and I think they have achieved
this goal very well. The guidelines are based on earlier
ones used for the Japanese standard.

The core of the guidelines says that if two characters look
significantly different, then they are encoded with two codes
even if they e.g. are one-to-one SC/TC equivalents. This is
to avoid suddenly changing the appearance of letters for a
user who may not be familiar with the significantly differently
looking shape. On the other hand, cases where there is only
a small difference in shape are unified (i.e. only one code)
unless this small difference in shape makes an actual
difference in meaning.

Overall, the results are so that if you present a text
where you change the glyph shapes within the range that
is unified, people who have done basic education but don't
know about different shapes (e.g. people in Taiwan or
Hong Kong who only know about TC, or people in China or
Singapore who only know about SC, or people in Japan who
only know about the forms used in Japan) will read over
these changes without problems, and might at some points
say 'this looks a bit strange', but will still identify
the character.

There are some exceptions to these rule related to backwards-
compatible roundtripping (source separation rule).

Hope this helps,      Martin.