Subject: Re: [idn] Re: Chinese Domain Name Consortium (CDNC) Declaration

From: Kenneth Whistler <[email-masked]>
Date: Wed, 6 Feb 2002 20:13:22 -0800 (PST)

Janming said:

> Two comments:
> 
> 1. The TC/SC problem or variant problem comes from Unicode 

It does not.

"The TC/SC problem" derives from 50 years of political history in
China, where the PRC engaged in an orthographic revision of
the writing system that Taiwan (and Hong Kong) did not, during
the same time span that information technology was developed and
rolled out on both sides of that political chasm. The result was
the development of incompatible character encodings and
input method editors tailored either for the "simplified" Chinese
of the PRC or the "traditional" Chinese of Taiwan and Hong Kong.

This was not a "problem" to end users before the Internet, because
never the twain did meet, and computer technology generally
didn't have to support both, but only one choice at any one
site. Furthermore, traditional IME's didn't support entry of
simplified characters, even if the traditional character encodings
(like CNS) nominally had them in the list; and vice versa.

Now it *is* a problem to end users, because the Internet and
the web, and changed political conditions, and globalization
mean that everybody has to deal with all of the characters in
one context or another.

I would accept a reformulation that:

  A. The availability of Unicode-enabled systems, such as
     Windows 2000, with IME's that can support either traditional
     or simplified characters, and fonts that display both,
     has made the preexisting TC/SC problem acutely visible
     to end users on their computers.

  B. The move to Unicode implementations means that mingling
     of traditional and simplified orthographies is easier.
     In effect, users now have the rope to hang themselves,
     if they so desire. Whereas, before, the constraints of
     the deployment of IME's and fonts generally meant that
     you couldn't easily mix SC/TC, even when the code page 
     nominally supported it.

> and the
> decision
>    to adopt Unicode as the basis to develop IDN technology.

And the alternative is ... ?

What character encoding alternative at this point in history
would not have the exact same problem? Certainly no choice
of a current national character encoding out of China itself
would avoid the exact same problems you attribute to Unicode.

But you are going to have to pick *some* encoding for the
characters you represent in domain names. If not Unicode,
then what?

> If the
> working
>    group chooses to use Unicode as basis, then it becomes a partial
> solution 
>    to the "global" IDN problem because of this side-effect.
> 2. We do not want to delay your process either. Please refer to the
> draft
>    "Phased Implementation of Internationalized Domain Names in
> Applications" 
>    for more details.

I have looked at that draft in detail, and see no solution
offered -- but only denial and vain hope. In short, a proposal
to simply exclude *all* the Chinese characters from IDN, with
a vague promise that someday someone will figure it all out
and decide which of 20,902 characters are o.k. to let back in
and which are only variants of the "good" ones. That promise
will be dashed on the rocks of the reality of complexity in
Chinese character variation -- and no code point based solution,
Unicode or otherwise, is going to fix it.

And in the meantime, the Phased Implementation proposal merely 
exports the Chinese SC/TC "problem" in a way that also disables 
IDN for Japan. I don't think that is acceptable.

--Ken

---

From: hoho <[email-masked]>
Date: Thu, 07 Feb 2002 14:06:20 +0800

> of the PRC or the "traditional" Chinese of Taiwan and Hong Kong.

This is true for TC/SC problem. But, the variant problem roots in the
long history of Han characters. You might be interested in taking a
peek at the problem at one of the Chinese or Japanese dictionaries on
Han variants.

> mean that everybody has to deal with all of the characters in

Try Windows 2000 by yourself. You could use their IME to input
TC, SC and even Japanese Han characters. Thanks to Unicode 2.0.

>      you couldn't easily mix SC/TC, even when the code page

I would accept A with minor modification as follows. B is not true.

     Windows 2000, with IME's that can support BOTH traditional

> characters you represent in domain names. If not Unicode,

My personal opinion after consulting several experts in Han characters
is to find an international organization, e.g., Unicode Consortium, to
host the standardization of variants. They are also willing to
 collaborate
with  CJK experts from other countries and regions. Some of their
suggestions are described in the "phased implementation" draft.

> Unicode or otherwise, is going to fix it.

Without a internationalized dictionary for Han variants, the current
IDN proposals are bringing side-effects to users and holders of
domain names of Han characters.

> IDN for Japan. I don't think that is acceptable.

I appreciate your loving for Japanese. But, if you do care about
Japanese, why did you ignore Chinese and Taiwanese, and perhaps
even more silent Japanese and Korean out there?

---

From: "John H. Jenkins" <[email-masked]>
Date: Thu, 7 Feb 2002 10:28:18 -0700

> peek at the problem at one of the Chinese or Japanese dictionaries on

As Ken well knows.  It's a nasty, nasty problem in general, and one
which
Unicode is trying to address.  At its last meeting the UTC agreed to
request that the IRG start to work on definitive variant data for the
CJK
repertoire of Unicode/10646.

If there is any body in the IT standardization community which should
own
the problem, it's the IRG.  It's international, multilingual, and has
the
longest and best experience with the characters involved.

> I would accept A with minor modification as follows. B is not true.

I would disagree here.  As you point out, Windows 2000 has the ability
to
mingle traditional and simplified orthographies because of Unicode 2.0.
Mac OS X now has the problem, too, because of Unicode 3.1.  In the past
on
the Mac, if you wanted to do Japanese you used the Japanese "code page,"
if you wanted to do traditional Chinese, you did the traditional Chinese
"code page," and if you wanted to do simplified Chinese, you used the
simplified Chinese "code page."  This inherent link between code page
and
nuanced version of a script was something Unicode intended to break.  In
the past, one had to deliberately work to mix the two, and the mixture
was
generally obvious.  This isn't true now.

> suggestions are described in the "phased implementation" draft.

Again, this is really the IRG's job.  The main problem with handing it
off
to the IRG is that they are not the fastest-moving standards body in
history.

Another part of the problem is that the data *is* out there.  Ken
mentions
the Sanseido dictionary.  I've got a CD of data from Taiwan on Chinese
variants, and the HKSAR government has a Web page.  I know that MS has
data they're sitting on which is used in Office.  Unicode's data has
been
derived in the past by character set analysis, but we are currently
working on incorporating data from a commercial product, Wenlin, which
has
been donated to us.

Unfortunately, most of the people who have spent the time and energy to
develop this data don't want to contribute it to the public gratis,
which
is what would happen if it went into Unicode.  Even if we simply said
that
we wanted to take over the data from an authoritative source (Sanseido,
the Hanyu Da Zidian), we'd probably have to get legal permission to do
so.

And there's the additional problem that nobody has a clear model, at
least
that I've seen, that captures the full plethora of variant "types" with
a
reasonable taxonomy and sufficient clarity that it can be used in
simple,
algorithm-generated, lexical-analysis-free situations.

> domain names of Han characters.

I agree with Ken that it's not clear how *anything* could both include
hanzi in IDN *and* be free from unpleasant side effects.

> even more silent Japanese and Korean out there?

I'm sure it wasn't intentional.  I believe that Ken's point is that you
can't do a Chinese-only solution here.  (And I'll naturally bring up the
Cantonese speaking population of the world, as just adding to the mess.)

==========
John H. Jenkins
[email-masked]
[email-masked]
http://homepage.mac.com/jenkins/

---

From: hoho <[email-masked]>
Date: Mon, 11 Feb 2002 01:45:41 +0800

> longest and best experience with the characters involved.

Hope that the Han variant problem can be solved in IRG. The people I
consulted are Prof. C.C. Hsieh of our institute and Chritian Wittern of
Kyoto University. They are both experts in Han variants. There is also
a online variant dictionary published by Ministry of Education, Taiwan.
In the past, the CCCII group of people also contributed enormous amount
of efforts on Han variants.

> generally obvious.  This isn't true now.

Thanks, I accept it. I am sorry that I didn't quite understand some English
sentences in statement B previously.

> hanzi in IDN *and* be free from unpleasant side effects.

This is hard indeed. But, by decomposing name equivalence matching
function into the IDN protocol, registration policy and the recently
proposed IRNSS protocol, it is likely to minimized its unpleasant side
effects.

> Cantonese speaking population of the world, as just adding to the mess.)

This is fine. But, I think you also understand. If having some people to
wait could help some other people from being damage, then we choose
to help, don't we?