Subject: Re: [idn] opting out of SC/TC equivalence

From: [email-masked]
Date: Wed, 29 Aug 2001 18:17:09 -0700

That is not the case.  If IETF does not want to put TC/SC
folding in [nameprep], then it has no good reason to
agree a versioning table to include GB, Big5, KSC, JIS to
 transliterated ACE map.  In that case, I am no motivation
to push for Unicode to accept the long list of radicals.  
I can sit back and see how long this will go, as I have been
assumed that by now the TC/SC should have been in there 
long time ago, which has been proved by James that I
was wrong. 

As this point, I think a little more Han radical classification
background infor is helpful.  The Han radical has about
a dozen dictionary editor's classifications, which is the radicals
included in UCS.  There are about a multitude of hundreds 
of programmers' viewpoints on this.  The one I am in favor of
 and have been implemented by quite a few good encoding 
software are all AI and phonetic based approach, which is used 
by common people when they communicate specific characters 
mostly used in names.   Similar in the way, English speakers
say "D as it's in Dog."  Studys in the late 80's have been shown 
there are about 1000 -1200 such radicals conventionally not 
considered as radicals.  Since RADICAL in Chinese often 
means the indexing group in a dictionary, and only takes
one radical per character.  So radical set in this sense is not 
the complete listing of radicals in Han.  So it is not complete list 
for SC radicals, neither the UCS has a complete list for the 
same reason.

If IETF has no architecture to accomodate these types 
of script requirement, and is not planning to use a complete 
list of radicals,  please give me a reason for me to push it for 
Unicode standard.   

Another option is that IETF still can go ahead giving the world a 
simple listing of 4128 TC/SC equivalent listing some where 
else, catching up with your product delivering schedule, waiving
 hands and say: take care of it, but out of my sight.  The end result
 is just like Unicode imposing a misconception of  TC/SC are
 two different languges.  I hope my explaination can shed a little
 light on CNNIC's feeling about TC/SC arguements.  You also
can tell me that my input is out of the scope of this group, and
I am ready to leave too. 

Liana

On Wed, 29 Aug 2001 22:58:08 +0200 Harald Tveit Alvestrand
<[email-masked]> writes:
> 
> 
> --On 29. august 2001 10:59 -0700 [email-masked] wrote:
> 
> >
> > That is the reason I propose to increase radical section in
> > Unicode to cover 1000 radicals, and for WG to agree a versioning
> > table to include GB, Big5, KSC, JIS to transliterated ACE map.
> 
> OK, so this proposal cannot be acted on by the IETF until the 
> proposal for 
> additional radicals in Unicode has been accepted.
> 
> Then I think it is not a possible solution at the timescale when 
> this group 
> is planning to deliver a product.
> 
> Thank you for the clarification.
> 
>           Harald
> 
>

---

From: "xiaodong lee" <[email-masked]>
Date: Thu, 30 Aug 2001 10:00:41 +0800

Hi Liana

----- Original Message ----- 
From: <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>; <[email-masked]>
Sent: Thursday, August 30, 2001 9:17 AM
Subject: Re: [idn] opting out of SC/TC equivalence

> list of radicals,  please give me a reason for me to push it for 

I remember that someone in IETF suggest to provide an architecture 
to accomodate these requirement. to handle codepoint is not the task 
and superority of IETF, but we in IETF should define the architecture 
and leave the interface for other consortium to deal with that. If 
all consortium just push it to others without enough efforts, there 
may be deadlock to implement it.

> can tell me that my input is out of the scope of this group, and

I agree with you. :)

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 30 Aug 2001 08:41:52 +0200

apologies if this was not clearly communicated....
I speak here as an (opinionated) participant, not as "the IETF".

> long time ago, which has been proved by James that I

The group decided a long time ago to go for a single character set,
and chose UNICODE/ISO 10646 for that single character set.
So any place where GB, Big5 and so on touches the system MUST be
outside the parts that are standardized by this group.

..... informative and interesting stuff deleted ...

> list of radicals,  please give me a reason for me to push it for

The IETF is the creation of its members.
We have created an architecture that is dependent on outside parties for 
character set and linguistic expertise as captured in character set 
standards.
When those standards are not available, we cannot use them.

>  two different languges.

Delivering broken solutions is abhorrent to me.
So this option is not an option.

>  I hope my explaination can shed a little

I think the group will have a better chance of delivering output that is 
useful to both you and me if we declare that the TC/SC issue is not going 
to be solved inside the IDN work.

I believe this problem is hard enough that it has to be solved at another 
level.
Sorry about that.

                Harald

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 30 Aug 2001 09:18:24 +0200

--On 2001-08-30 08.41 +0200 Harald Tveit Alvestrand <[email-masked]>
wrote:

> level. Sorry about that.

Just because nameprep and what the IDN group is doing is only mechanical
operations on individual characters in a single character set, my view is
that SC<->TC conversion can only be done in a keyword like system before
IDN is even started to be used.

Reason for this is because of the different kind of equality which James
described about a year ago and also described in this document. Especially
see section 2.4.3 about the need for user interaction.

http://www.basistech.com/articles/C2C.html

So, outside of IDN, but most certainly part of a system which is described
in the individual drafts by Dr. Klensin and Mr. Mealling.

Further, as James pointed out, if a registry want to set a policy on what
subset of the characters which IDN group is allowing is to be allowed
(only), that is perfectly alright, but also not part of what the IDN group
is doing.

    paf

---

From: [email-masked] <[email-masked]>
Date: Thu, 30 Aug 2001 15:48:16 +0800

> I believe this problem is hard enough that it has to be solved at another
           The major confusing of CJK  characters can not be solved
especially in gTLD but not in ccTLD. May we suggest the time limitation and
incompleteness of IDN work , so Multilingual.com should not be delivered. It
is too difficult to solve the confliction without local policy and language
related ccTLD as the implicit tag.
           Althout this is not a good method now, but it will not produce
more troubles. The postponding or not do wrong thing is important.

L.M.Tseng

---

From: Martin Duerst <[email-masked]>
Date: Thu, 30 Aug 2001 17:13:57 +0900

>long time ago, which has been proved by James that I

What you are implying here, and you have said elsewhere, is
that the TC/SC problem can be made easier by decomposing
Chinese characters into components (usually called radicals).

It is true that for the bulk of the straightforward TC/SC
equivalences, it is sufficient to just change a single radical,
and that in some other cases, it is sufficient to just change
more than one radical.

However, considering the operational complexities of:

A) Defining and maintaining decompositions of Chinese
    characters into radicals for use in TC/SC mapping.
B) Defining and maintaining TC/SC mappings directly.

are about the same. The size of the tables is the same
order. And in any case, there are exceptions that cannot
be covered by radical decomposition (in particular, there
are exceptions where radical decomposition would give
the WRONG result), so just doing B) is more straightforward.

**But all this is largely irrelevant**, as you say. There are
two problems that are orthogonal to the above choice:

1) The problem of 1-to-many, many-to-1, and many-to-many
    mappings. We have not yet seen a working proposal that
    addresses these.

2) The fact that we have to handle not only Chinese, but
    also Japanese and Korean.

So I agree with you that there is no reason to press the
Unicode consortium to add more radicals, for multiple reasons.

Regards,   Martin.

P.S.: Completely unrelated to IDN:
       I have spent about two years of my life doing research
       on decomposition of Kanji and related subjects, and
       published a few papers.
       If you are interested, please send me private mail,
       and I can send you a .ps version.

---

From: [email-masked] <[email-masked]>
Date: Fri, 31 Aug 2001 18:40:18 +0800

Hi !  Fältström :
        I had ask you a question in our discussion why not also put alphabet
case holding in keyword system ?
I do not like to push things in this way,  that must hurt everybody, but I
must point out how many cobinations of name in  case mixing of  Alphabet
will be produced in name registration ?
        Even in CJK area , we all have no complete total solution to reduce
the complexity in this open set characteristics, but it not means we do not
care it.  The confusing must be reduced as more as possible, no to produce
them more by keyword system. I think it is unfair to let keyword system
become a trouble sources of CJK characters.

> http://www.basistech.com/articles/C2C.html

I must  say that most of them are language related , not related to
easy-to-confuse scipts or characters.

L.M.Tseng

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 31 Aug 2001 13:07:13 +0200

--On 2001-08-31 18.40 +0800
"=?utf-8?B?dHNlbmdsbUDoqIjntrLkuK3lv4Mu5Lit5aSnLnR3?="
<[email-masked]> wrote:

>         I had ask you a question in our discussion why not also put
> alphabet case holding in keyword system ?

The simple answer is that the Unicode Consortium have defined "case
folding", and other kind of equivalence tables. All of them are mechanical
mappings, and we have in the IDN group chosen to include them in the
nameprep algorithm.

If some of them should be removed, or if some should be added, that is a
valid discussion.

If a table existed which handled mappings between TC and SC, we could of
course use it.

The problem is that such a table doesn't seem to exist. I have seen in one
draft (as Harald points out) a table with approximately 2000 characters out
of the 20-40000 which exists. Harald have asked valid questions about what
to do with the other characters.

Further, you should not compare SC-TC equivalence with case folding inside
one script, for example Latin. Instead you should compare with the fact
that the same, and similar, characters exists in Greek, Latin and Cyrillic
scripts. For example the character which looks like 'A'.

We do _NOT_ have solutions to that kind of equivalence between different
scripts for non-chinese characters.

Because of that, I feel that if you don't have a table, then you ask for a
solution for chinese characters which we don't solve for other scripts in
the world.

Further, you say that the number of permutations will be high. Linguistics
I have talked with say that the number of permutations will be low, i.e.
that one label is in most cases either in SC or TC.

I don't know, and don't care, what the correct asnswer is. But what
troubles me is that people which really do know chinese characters and
scripts say so different things. I don't hear one story. That makes
discussions very problematic.

    Regards, Patrik

---

From: [email-masked] <[email-masked]>
Date: Fri, 31 Aug 2001 20:46:53 +0800

> If some of them should be removed, or if some should be added, that is a
       Thanks  your careful concern , I  expect that mechanism can be used
to all scripts to reduce the confusing. The work can not be completed at one
night ! But by this way , that can be improved step by step.
> to do with the other characters.
      If you ask an expert of chinese language to give you this "completed
table", it will be never delivered , because the characters are in an open
set. Even we use chinese, we just use not more than 8000 frequently used
characters in our newspaper . So no one will answer this impossible
problems.  As I have mensioned  most PRC-SC are frequently used and some of
them are quick-written font that are very similar to formal-written font ,
so the problem of easy-to-confuse is serious, especially in Hong-Kong, the
mixing of  SC/TC  will produce more confusing domain name.  We can not
solved all of them , but we can reduce the troubles by doing some character
mapping .
> scripts for non-chinese characters.
    I  hope  IDN WG can inform the related NIC to propose the mapping table
in UNICODE  to reduce the confusing in their scripts.
> that one label is in most cases either in SC or TC.
        That is true in GB and BIG5 , so UNAME draft show the example of
registration in full SC (pure GB not  GBK) and full TC  not the mixing ,
but the GBK used in Hong-Kong including part of  frequently used TC
characters are  mixing  of  SC and TC .  The  chinese.com  does not limit
the mixing of  SC/TC in UNICODE form.

> discussions very problematic.
        The fact is that there are many part and many case applied in
different situations. The culture of chinese characters is the
characteristics of an open set. That is very different with western model
based on simple basic elements.
        Just  like we present mapping by table first and you may like to
present it by transformation formula first. That is all based on historic
experiences.

Thanks your help to solve our problems.

L.M.Tseng

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 31 Aug 2001 14:56:38 +0200

--On 31. august 2001 20:46 +0800 
"=?utf-8?B?dHNlbmdsbUDoqIjntrLkuK3lv4Mu5Lit5aSnLnR3?=" 
<[email-masked]> wrote:

>> The problem is that such a table doesn't seem to exist. I have seen in
>> one draft (as Harald points out) a table with approximately 2000
>> characters
> out
>> of the 20-40000 which exists. Harald have asked valid questions about
>> what to do with the other characters.
>>
> problems.

Please then answer a simpler one:

When the matching algorithm you envision encounters a character where the 
tables do not tell what to do about it, what will the algorithm do to it?
Pass it through unmodified? Reject it? Does something else which you have 
not defined yet?

If you tell us this, and claim that this is acceptable treatment of the 
20-40.000 characters not in the tables, we can at least discuss this 
proposal (whether such a partial TC/SC mapping is acceptable), instead of 
asking again and again what to do about them.

But it is impossible to implement an algorithm which says "this part will 
be worked out later".

                  Harald

---

From: Dave Crocker <[email-masked]>
Date: Fri, 31 Aug 2001 07:55:18 -0700

Folks,

At 04:07 AM 8/31/2001, Patrik Fältström wrote:
>...

The second statement is the more important point:  It is the reason that 
this entire topic is entirely OUT OF SCOPE for this working group.  It is 
therefore the reason that the topic needs to be moved off of this list, so 
that the working group can work on finishing tasks that are WITHIN the charter.

There is a rumor that some people actually want to use the results of this 
working group, and that they want to use the results AS SOON AS POSSIBLE.

Having discussions that are outside the scope of the working group hurts 
that goal.

It does not matter that this discussion is both important and 
interesting.  What DOES matter is that it is causing further delay, for a 
working group effort that has already gone on much longer than it should have.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.273.6464

---

From: [email-masked] <[email-masked]>
Date: Sat, 1 Sep 2001 00:44:05 +0800

> Pass it through unmodified? Reject it? Does something else which you have
           The 1-1 mapping table first treated the frequently used
characters in TC and SC . I  have suggest CJK  people to review the level of
easy-to-confuse in font scripts in draft of  tsconv and to remove some pair
that is concerned  in different language using.  I think  no  CJK people
like to use some characters to confuse with each others.  So , this is a
simple character mapping table in UNICODE TR 20 that can be  major concerned
to easy-to-confuse only.   A matching algorithm can be  table driven in
treat her input and stored data to check it is  in the equivalent set  and
convert it to the defined standard scripts, if it is a character not in the
defined equivalent set why it should be convertyed ?

> asking again and again what to do about them.
           As Patrik FÃ¤ltstrÃ¶m  described , if it is a mechanism in
UNICODE consosium , then they have the version update procedures and IDNA
also treat  the version update in UNICODE , why it can not be updated step
by step  to  increse it ?  The most important thing is tell user in
registration time what character set is treated as equivalence and what is
not  and what is the current rage of total characters can be used .  The
procedure and announcement is important and not the boundary of range is
important. If the range is monotonically increasing and new added will be
fully cross checked , it is in the correct way .
            The most important thing in CJK area is that the allowed
characters set must be announced by official organization of goverment .  So
, each  ccNIC  need time to exchange informations.

> But it is impossible to implement an algorithm which says "this part will
   The algorithm has no change , it just increase the data scale . If the
range of data is still in this type of algorithm defined  , why it can not
work ?
   The  problem is in version control if you change the function part. If
function is still applied to the new data , there are no problems in
algorithm.

L.M.Tseng

---

From: Keith Moore <[email-masked]>
Date: Fri, 31 Aug 2001 14:22:16 -0400

> this entire topic is entirely OUT OF SCOPE for this working group.
...
> working group, and that they want to use the results AS SOON AS POSSIBLE.

...regardless of whether or not those results work well.

yes, I am quite sure that you are correct.

it's also clear that any DNS-compatible solution will have significant 
problems with transcription (either written to typed or audible to typed)
in some languages.

addressing this may be out of scope for this WG, but that doesn't mean
that IDNs can be made to work well without addressing it.

and yet this WG clearly lacks sufficient expertise to solve this problem 
no matter what the WG charter says.

it appears that the best this WG can do is 

	- identify known limitations with its approach

	- state any known workarounds for those limitations

	- define or suggest some mechanism (if any can be found)
	  for the IDN resolution infrastructure to be upgraded
	  at a later date to better solve the transcription problems.

Keith

---

From: David Hopwood <[email-masked]>
Date: Thu, 30 Aug 2001 16:44:22 +0100

-----BEGIN PGP SIGNED MESSAGE-----

[email-masked] wrote:
> Patrick Fältström wrote:
> > Further, you say that the number of permutations will be high. [Linguists]

Provided that input mechanisms can be set to produce all SC or all TC,
does that matter? In practice, would a Chinese input mechanism, as it
would be set up for a user in Hong Kong, say, typically produce an
arbitrary mix of SC and TC characters?

Note that there are other scripts which have special requirements for typing
domain names, URIs and email addresses - for example Arabic and Hebrew. So
in general, fully satisfactory user input of these identifiers will depend
on support from input mechanisms (and in some cases the user telling the
input mechanism that an identifier is being entered, if it appears in free
text). Chinese is just another case where this support would be useful.

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: David Hopwood <[email-masked]>
Date: Thu, 30 Aug 2001 17:20:12 +0100

-----BEGIN PGP SIGNED MESSAGE-----

[email-masked] wrote:
> Patrik Fältström wrote:
> [pass it through unmodified]
> 
>            As Patrik Fältström described, if it is a mechanism in
> UNICODE consortium, then they have the version update procedures and IDNA
> also treat the version update in UNICODE, why it can not be updated step
> by step to increase it?

Because that could cause existing domain names to break (i.e. fail to resolve
when processed by a new version of nameprep). This would happen when a
name contains a source character for one of the new mappings. In order to
avoid that problem, all authoritative servers with names in the affected
scripts would have to be upgraded before any resolvers or applications. That
is at least undesirable, and may be impractical.

Note that there are rules for updating nameprep and NFKC that ensure the
same problem doesn't happen for canonicalization or case mappings (roughly
speaking, no new mapping can be added from a previously assigned character;
see UAX #15 for a more precise statement of the update policy for NFKC).

- --

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 31 Aug 2001 22:40:00 +0200

--On 2001-09-01 00.44 +0800
"=?utf-8?B?dHNlbmdsbUDoqIjntrLkuK3lv4Mu5Lit5aSnLnR3?="
<[email-masked]> wrote:

>            As Patrik FÃ?Â€ltstrÃ?Â¶m  described , if it is a mechanism in
> by step  to  increse it ?

Very simple answer.

Let's assume that the table day one include the following characters:

A
B
C
D
E
F

We agree that the simple equivalence rules for 1-1 mapping maps A->B.

This means that the following characters are available for domain name
registration:

B
C
D
E
F

People register domainames such as:

B.example.com.
C.example.com.
D.example.com.
E.example.com.
F.example.com.

Five different people register these five domains, which are different.

If one type in A.example.com in the browser, nameprep will map that name to
B.example.com. So, A.example.com and B.example.com are treated as "the same
domain".

Now, let's say that we increase the size of that table, and decide that D
and E are equivalent, so D->E mapping should be added.

This means that only the following characters are allowed:

B
C
E
F

The domains D.example.com and E.example.com are equivalent.

Now the questions, and I want you to answer them. Short answers are
required:

(1) Should the holder of D.example.com or the holder of E.example.com be
forced to give back his domain?

(2) Who is making that descision?

(3) Who will tell the domain name holder that he is not allowed to have his
domain anymore?

     paf

---

From: David Hopwood <[email-masked]>
Date: Thu, 30 Aug 2001 21:49:45 +0100

-----BEGIN PGP SIGNED MESSAGE-----

I wrote:
> scripts would have to be upgraded before any resolvers or applications.

In fact it's worse than that, as Patrick Fältström pointed out: the source
and destination names may already be owned by different people/organizations.
So adding new mappings for already-assigned Unicode characters must be
prohibited.

- --

---

From: "Soobok Lee" <[email-masked]>
Date: Sat, 1 Sep 2001 10:32:57 +0900

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Dave Crocker" <[email-masked]>
Cc: <[email-masked]>
Sent: Saturday, September 01, 2001 3:22 AM
Subject: Re: [idn] Working group focus -- and progress 

> no matter what the WG charter says.

I agree. Moreover,  even if we have the expertise,
most problems cannot be solved by mechanical processing 
like nameprep which is designed to be a "utility" for
input convenience and matching of IDN labels, 
_NOT_ to be the complete solution to ambiguities 
introduced by i18n characters from various scripts.

>   at a later date to better solve the transcription problems.

I agree. I am preparing the first  (maybe incomplete) BCP for IDN. 
I hope someone would make GPLed zone checking tools
based on that.. :-)

soobok