Subject: [idn] Summary of TS-SC discussion

From: Paul Hoffman / IMC <[email-masked]>
Date: Sun, 2 Sep 2001 21:08:28 -0700

We have now rehashed every part of the question many times; maybe it 
is a good time to summarize.

1) Doing traditional-to-simplifed conversion must be done from a 
table; it cannot be done by algorithm.

2) No widely-accepted official standard table yet exists. Multiple 
different tables exist.

3) The same code points that would be converted in some domain name 
parts would not be converted in other domain name parts because the 
conversion is only appropriate for Chinese, not Korean or Japanese, 
which use the same code points. Thus, the language (not the script) 
must be known in order to do the conversion correctly.

4) There is no way to flag the language in name parts in a consistent 
way that end users would understand. Heuristics such as the TLD could 
be used, but doing so would cause some names to not be converted when 
they should be (such as a Chinese name part under .com) and would 
cause other names to be converted when they should not be (such as a 
Korean name part under a Chinese SLD). Hueristics such as language 
tagging would require that the end user tag each Chinese name part, 
and would not work at all with names that were cut and paste.

5) The problem can be definitively solved in zone files with multiple 
records. Some claim that this takes 2^length records, while others 
claim this takes 2 records.

6) Until the system is deployed, we cannot tell how well the users 
will adapt. We (the "experts" on domain names) can make predictions 
about what typical users will and won't be able to do, but we simply 
don't know, and our past track record at predictions is not all that 
good.

My conclusion from this is that we cannot standardize on 
traditional-to-simplified at this time with the protocols that are 
under discussion in the WG. We might be able to add this later after 
both a widely-accepted official standard table exists and a language 
tagging mechanism that makes sense to users is implemented.

Neither IDNA nor nameprep preclude this potential future change, so 
we can move forward with them now, and leave the door open to this 
change, and other similar changes, in the future.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Dave Crocker <[email-masked]>
Date: Sun, 02 Sep 2001 22:18:57 -0700

At 09:08 PM 9/2/2001, Paul Hoffman / IMC wrote:
>Neither IDNA nor nameprep preclude this potential future change, so we can 
>move forward with them now, and leave the door open to this change, and 
>other similar changes, in the future.

An excellent summary -- especially point # 3.

however your final point is iarticularly important, and I believe has not 
been stated previously.  That is, we need to understand that we are not 
required to solve this problem now.  We can develop (or, more likely, 
adopt) an enhancement later.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.273.6464

---

From: Eric Brunner <[email-masked]>
Date: Mon, 03 Sep 2001 11:04:26 -0400

> 2) No widely-accepted official standard table yet exists. Multiple 

Is non-uniqueness the problem that bars scoping SC/TC equivialency rules at
global via nameprep?

Is uniformity of use among two or more different tables the problem that bars
scoping SC/TC equivalency rules at global via nameprep?

How would we know when one of the existing multiple different tables became
the widely-accepted prevalent table?

If our issue is qualitative, then it may not be traditional engineering,
which contains cost/benefit evaluation and optimization.

If our issue is quantitative, then it may be traditional engineering, and
a "best (present and/or projected) fit" of cost/benefit choices made.

I want to know what data exists, not just what some preferences are.

Eric

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 04 Sep 2001 11:17:47 +0200

> How would we know when one of the existing multiple different tables
> became the widely-accepted prevalent table?

When ISO or the Unicode Consortium (preferably both) declare this to be 
"the standard", based on advice from the CJK IRG, this would be a pretty 
clear hint.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 04 Sep 2001 22:35:55 +0900

Hello Paul,

I mostly agree with your summary. However, there is one important
point that is not explicit enough.

At 21:08 01/09/02 -0700, Paul Hoffman / IMC wrote:
>We have now rehashed every part of the question many times; maybe it is a 
>good time to summarize.
>
>1) Doing traditional-to-simplifed conversion must be done from a table; it 
>cannot be done by algorithm.
>

This can easily be misunderstood. The issue is not that different
(standard) bodies or different individuals disagree about the mappings.

The problem is that the structure of the mappings is complex.
Even if Korean and Japanese are completely ignored, there are
some traditional characters that can be mapped to more than one
simplified character, and some simplified characters that can
be mapped to more than one traditional character.
(one-to-many mappings, for examples see table 1 in
Jack Halpern's paper at http://cjk.org/cjk/c2c/c2cbasis.htm).

To repeat, these one-to-many mappings are not the result of disagreements
between organizations or individuals, but they are agreed by everybody.
Experts both from China (including of course CNNIC), the UTC, as well
as many others can easily agree on which characters can be mapped
one-to-one, which characters don't need mappings (because they are
the same in both traditional and simplified Chinese), and for which
characters there are one-to-many mappings, and where these mappings
go to.

In order to do the conversion correctly for all cases, you need
some*body* who knows Chinese in both writing variants. There will
be few if any disagreement among such people. Trying to
have a machine do the conversion will in many cases be incomplete.

Both the Unicode Technical Committee and draft-ietf-idn-tsconv-00.txt
clearly agree on this point (see in particular chapter 4 of tsconv).
I haven't looked in detail at the data in tsconv, and I don't think
the UTC has looked at it in detail. And if we decide to actually
use such a list, we definitely have to make quite a few more checks
to make sure there are no accidental errors. But as far as the job
is 'list all the one-to-one equivalences between the simplified
Chinese characters (as covered by GB2312, I assume) and the traditional
Chinese characters (as covered by Big5, I assume)', the variation
between any two experts, or bodies of experts, will be negligibly
small, and solvable.

The main point of disagreement (as long as we stay with Chinese only)
is what to do to address the one-to-many problem. Is it better to
address the one-to-one problem in nameprep, or in the DNS itself,
and only leave the one-to-many problem to registration policies and
actual registrations (as tsconv proposes, as far as I understand),
or is it better to handle everything on the registration side
(as, for example, the UTC proposes).

Regards,   Martin.

>3) The same code points that would be converted in some domain name parts 
>would not be converted in other domain name parts because the conversion 
>is only appropriate for Chinese, not Korean or Japanese, which use the 
>same code points. Thus, the language (not the script) must be known in 
>order to do the conversion correctly.
>
>4) There is no way to flag the language in name parts in a consistent way 
>that end users would understand. Heuristics such as the TLD could be used, 
>but doing so would cause some names to not be converted when they should 
>be (such as a Chinese name part under .com) and would cause other names to 
>be converted when they should not be (such as a Korean name part under a 
>Chinese SLD). Hueristics such as language tagging would require that the 
>end user tag each Chinese name part, and would not work at all with names 
>that were cut and paste.
>
>records. Some claim that this takes 2^length records, while others claim 
>this takes 2 records.
>
>6) Until the system is deployed, we cannot tell how well the users will 
>adapt. We (the "experts" on domain names) can make predictions about what 
>typical users will and won't be able to do, but we simply don't know, and 
>our past track record at predictions is not all that good.
>
>traditional-to-simplified at this time with the protocols that are under 
>discussion in the WG. We might be able to add this later after both a 
>widely-accepted official standard table exists and a language tagging 
>mechanism that makes sense to users is implemented.
>

---

From: [email-masked] <[email-masked]>
Date: Tue, 4 Sep 2001 21:54:26 +0800

> table; it cannot be done by algorithm.
            I  have  question .   Does  the ACE translation ( ACE-Z or RACE
..) can be done without  any UNICODE table ?
> 2) No widely-accepted official standard table yet exists. Multiple
           Does all table in proposed nasmeprep are all widely-accepted
official standard table ?
> must be known in order to do the conversion correctly.
          You have get the results from any Korean , Japan official
announcement ?  The language term is not treated in this WG , why you
annouce "all" scripts are "all language dependent" ?  If it is so, then all
scripts of UNICODE should not be treated in IDN.

> and would not work at all with names that were cut and paste.
         Why the tag in MIME mail header and subject can be recognize the
language character typed-in and display them correctly ?  Does user of
E-mail tag each part of CJK characters ?
          Another question , why the IDN  domain name must be  IDN.COM , why
not it is ACE(ML-name.ML-company) ?

> claim this takes 2 records.
         I  like to know  "Taiwan University"   台灣大學
                                  "Tsinghua University"  清華大學
can be solved by  2 records  in DNS zone table with TC/SC mixing ?

> --Internet Mail Consortium
       Very sorry , I have too much questions in your conclusions.

L.M.Tseng,

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 4 Sep 2001 10:02:16 -0700

>..) can be done without  any UNICODE table ?

Correct: they do not require tables.

>official standard table ?

Yes.

>announcement ?

Sorry, but I cannot understand the question.

>   The language term is not treated in this WG , why you
>annouce "all" scripts are "all language dependent" ?

I never said any such thing. Scripts are independent of language, but 
scipt usage can be language-dependent.

>   If it is so, then all

It is not so.

>language character typed-in and display them correctly ?

MIME body part headers allow MIME language tagging; mail subject 
lines do not allow MIME language tagging, but do allow Unicode 3.1 
language tagging. Neither has had much success in implementation, and 
inconsistent use causes poor display.

>   Does user of

No, they don't. MIME body part headers cover the entire message, not 
individual characters. Unicode 3.1 language tagging is applied to 
groups of characters, but so far, there have been no common 
implementations. This is probably because it is tedious for a user to 
have to insert tags into simple text saying "these characters are 
from this language, but these other characters are from this other 
language".

>not it is ACE(ML-name.ML-company) ?

Because that is not how the DNS works. What you propose makes perfect 
sense for a directory system, and is being actively pursued in many 
places. In fact, this one of the reasons that people want to abandon 
using the DNS for naming things like companies: it does a very poor 
job of it. The IDN WG work will make that poor job a little better 
(by allowing more characters), but it will still be fairly poor.

>          I  like to know  "Taiwan University"   ????
>                                   "Tsinghua University"  ????

One record would be all-traditional, one would be all-simplified.

>        Very sorry , I have too much questions in your conclusions.

I hope I have answered them all!

--Paul Hoffman, Director

---

From: [email-masked] <[email-masked]>
Date: Wed, 5 Sep 2001 02:37:52 +0800

> Correct: they do not require tables.
         A user type-in a Mutilingual Name (For ex. JIS  code) ,  From  here
to ACE code,
there are a translation table from JIS to UNICODE ,  then based UNICODE data
resultas, you have ACE code .
         To do nameprep , character mapping and forbidden table can not
aviod, If there are no nameprep , there will be no succeed ACE conversion.
the pre-condition is there need table to make sure the input data is in
correct data range.
> >official standard table ?
      I think many people like to know all the official name and source of
starndard.
> Sorry, but I cannot understand the question.
My question is TC/SC conversion is only applied to Chinese character, no
Japanese or Korean can be applied ?
> >scripts of UNICODE should not be treated in IDN.
          So, there are some small part is  scripts only and independent of
language using , like the whole case Latin characters in Japan, Chinese, and
Korean language, they should be treated like the ASCII character case
mapping (For ex.  .COM).
> from this language, but these other characters are from this other
Actually , E-Mail system use language tag , right ?
> >          I  like to know  "Taiwan University"   臺灣大學
    Sorry,  it is not true , " Taiwan " means 2 chinese characters, in these
two character , the first one has 2 scripts , all are TC , but one of them
are only used in PRC-SC. The all-traditional need at least 2 record in
"Taiwan University" , this is a example of  1(SC) --> 2 (TC).  1-n mapping
are forced to selected one in registraion is based on the assumption the 1-1
mapping had treated in nameprep. You can not cut them to simplify it and
forget they may be 2^^n combinations  without SC/TC 1-1 mapping to reduce
them.

L.M.Tseng

---

From: [email-masked]
Date: Tue, 04 Sep 2001 18:02:49 -0700 (PDT)

> inconsistent use causes poor display.

Actually, mail subject lines do allow for language tagging. See RFC 2231 for
details. (Warning: It isn't pretty.)

I've seen implementations of body part language tagging and of
mail subject line language tagging, but AFAIK they are rarely if ever
used. I've never seen an implementation that supports the Unicode 3.1
language tagging stuff.

Bottom line: I agree with your assessment that this has not worked well.

				Ned

---

From: "James Seng/Personal" <[email-masked]>
Date: Wed, 5 Sep 2001 13:08:59 +0800

> forget they may be 2^^n combinations  without SC/TC 1-1 mapping to
reduce

Prof Tseng,

In Latin, if we have A/a B/b C/c, it is reasonable for someone to type
in "ABC", "abc" or "Abc" and so on...this gives us 2^n possibility.

Now is A/a is TC-SC, similarly for B/b and C/c, it is usual for one to
do variant like "Abc" or "ABc"...It is like to have "ABC" or "abc" only
which gives us n*2 combination.

But because TC-SC also have is Traditional-Traditional,
Traditional-Simplified, Simplified-Traditional and Simplified-Simplified
variant, this gives us a max of n*4 combination, one for each. ("Taiwan"
in your example falls into the T-T and T-S variant).

This is why TC/SC is not really n^2 combination. Opertionally, it is
closely to n*2 to n*4 from experience.

As Mark said, we should compare this 'confusion' to Latin A (U+0041) and
Greek A (U+0391). This is not normalized in Nameprep either and if AOL
wish to have both Latin and Greek, they have to put it in the zonefile
twice.

-James Seng

---

From: Jack Halpern <[email-masked]>
Date: Wed, 05 Sep 2001 23:12:37 -0500

Greetings

I
 >closely to n*2 to n*4 from experience.
 
This way of thinking is based on a misunderstanding of the exact meanings 
what SC and TC. The simplified form of TAI in Taiwan is indeed an SC form, 
but **it is also a TC form** in the sense that it is used in Taiwan -- in fact, 
it is the preferred form. Also note, as well known, that form identical in SC 
and TC are very common. In the case of TAI, we are talking about TC 
variants (see details in my paper on TC vartiants at
http://www.cjk.org/cjk/reference/tcvar.htm ).

Comparing "ABC" vs. "abc" to SC vs TC is a flase analogy at best and does not take
the true nature of the Chinese script into account.

 >-James Seng

Regards, Jack Halpern
	President, The CJK Dictionary Institute, Inc. 
        http://www.cjk.org Phone: +81-48-473-3508

---

From: "James Seng/Personal" <[email-masked]>
Date: Thu, 6 Sep 2001 11:34:31 +0800

Jack,

I am not saying TC-SC is like upper/lower case. Please read my mail
again.

I am saying if "A" is TC and "a" is SC then ... (I should have use a
real chinese characters here but I thought "A" and "a" would be easiler
to explain)

And I stand correct in that TC-SC is a n*2 to n*4 problem, not a 2^n
problem. This is something which we discussed before privately and both
agreed.

-James Seng

----- Original Message -----
From: "Jack Halpern" <[email-masked]>
To: "James Seng/Personal" <[email-masked]>
Cc: "tsenglm@è¨^ç¶²ä¸­å¿f.ä¸­å¤§.tw" <[email-masked]>;
<[email-masked]>; "Paul Hoffman / IMC" <[email-masked]>
Sent: Thursday, September 06, 2001 12:12 PM
Subject: Re: [idn] Summary of TS-SC discussion

---

From: "Soobok Lee" <[email-masked]>
Date: Thu, 6 Sep 2001 16:23:57 +0900

> problem.  

That may be true  even  among latin,cyrillic and cherokee labels.
For all-latin "cisco", there are 3^5=273 confusibles in the combinations of
latin,cherokee and cyrillic characters , since  those three script blocks
have similarly looking characters for c,i,s,o.

But zone masters may give  all the 3 domains  ( all-latin , all-cyrillic and 
all-cherokee versions) to one registrant  and decide to prohibit all other 270  
mixed labels ( 273 - 3= 270 ) from begin registered.

These amibiguities are not a "rathole", but a "thief hole" that every serious
 zone masters should be aware of.

Soobok