Subject: Re: [idn] opting out of SC/TC equivalence

From: [email-masked]
Date: Wed, 29 Aug 2001 03:24:48 -0700

Hi, Harald

You are quite right regarding how the Chinese linguistics
works, there are never complete!  Right now, there are
formally classified characters already exceed 100,000.
I am not advocate include the whole set in [nameprep]
at all.  Neither do I advocate to exclude any characters, since
it is what the user wants.  But I do recommend to include
the characters included in Big5 and GB standard, that is
about 23,658 code points in Unicode, while 2238 are TC/SC 
equivalence and 14 radical equivalence we have been
discussing.   

There are two questions, 1) how to we implement TC/SC
in Unicode standard, 2) how do we implement CJK in 
[nameprep].  

For 1), I said 1100 ( I don't have the table at this time) 
half sized new Unicode points is to put radicals into the Unicode 
standard, just as other scripts did for diacritics.  This gives a
 base for decomposing a character for IDN identifiers 
(but not good enough for user input interface). 
 Within these radicals, 1886  TC/SC equivalence can be 
addressed.  The  352 TC/SC equivalence and 14 radical 
equivalence can be addressed in a supplement document, 
which shall state how to treat the other 1886  TC/SC 
equivalence based on their radicals.  The radical class has
been quite different from early history to recent era.  The class
of radical definition I am proposing is for a Han speaker and 
parting away from a computer programmer or a dictionary 
editor:

CJK Radical set has two sections: the first section is traditional
dictionary radical set, such as "Kangxi" and "Cihai" radical
set.  The second set is any characters which have been used
as radicals for other characters are radical set members.
The first  set is about 200, depending which dictionary we are
based on, the second set is about 1000.  Since the two sets 
overlap, my estimate is 1100 code points. 

2) I assume (since I did not check)  that  about 23,658 code 
points in Unicode 3.0 has included Hanja and Kanji.  
The other codepoints in Unicode and future new comers, 
can be treated on needed base.  This means, only when
someone has used in a name at registration time, and 
supplied the name with a codepoint in Unicode, then the
codepoint is added to zonefile.  (Not in [nameprep]?)
 If such a character is not in Unicode, then a bit map of the 
new character has to be provided in the zonefile.  This is 
the reason, I propose a "Request for Reference to be sent" 
protocol to be drafted.  

Liana

On Wed, 29 Aug 2001 10:20:08 +0200 Harald Tveit Alvestrand
<[email-masked]> writes:
> 
> 
> --On 28. august 2001 13:40 -0700 [email-masked] wrote:
> 
> > Hi, James and Chinese experts:
> >   You are right on the TC/SC equivalence not in Unicode.
> > I know they wanted to put it in long time ago, so I assumed
> > it is reflected in there some how.   I have just read a reason
> > that it is not in there, because they think it is too difficult to
> > put it in.   I happend to have an idea that 1100 half size
> > code point may solve part of the problem and another 200
> > TC/SC listing completes it.  This can be used in [nameprep].
> > What do you think?
> 
> I would be happy to see a complete proposal.
> 
> draft-ietf-idn-tsconv-00 describes a TC/SC mapping for 2064 
> traditional/
> simplified pairs, saying that other tables are needed for 
> single/many
> and many/single mappings.
> 
> This means that we have a documented proposal on what to do with 
> 4128 
> characters.
> In Unicoode 3.0, there are 23.658 *more* characters classified as 
> "Han"; 
> Unicode 3.1 adds 42.711 more, and it has been noted here that 
> because of 
> the way Chinese linguistics work, it is almost 100% certain that 
> there will 
> be more added.
> I assume (foolishly) that for some large class of these characters, 
> the 
> answer is "don't touch them" when mapping TC/SC - but I have no way 
> of 
> telling which characters belong in that class.
> 
> If you can come up with a proposal that describes what to do about 
> ALL the 
> Han characters in Unicode, I will be very happy to hear it.
> 
> Until then, I have to say that I have not seen any complete 
> proposal.
> Remember - the implementations of the algorithm for the non-Chinese 
> part of 
> the world will mainly be done by non-Chinese-speaking programmers; 
> it's got 
> to be simple & complete enough that even I can get it right...
> 
>              Harald
> 
> 
> 
> 
> 
> > .
> > Liana
> >
> > On Mon, 20 Aug 2001 13:53:38 +0800 "James Seng/Personal" 
> <[email-masked]>
> > writes:
> >> > The SC character set has been used for decades and has went
> >> > through extensive nationwide testing in China.  SC is stable 
> and
> >> they
> >> are
> >> > properly reflected in Unicode standard.   The question is a
> >> definition:
> >> > is TC/SC a case folding?  It seems that in this WG, there has 
> no
> >> > consensus on this definition yet.
> >>
> >> I am not sure what you mean by "properly reflected" in Unicode
> >> Standard.
> >> If you mean it is in ISO10646 codepoints, then yes, both TC/SC 
> are
> >> in
> >> the code points. But if you saying Unicode Consortium have proper
> >> definition of TC/SC, then I afraid to say there is none.
> >>
> >
> >
> 
>

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 29 Aug 2001 13:24:45 +0200

> overlap, my estimate is 1100 code points.

The Unicode 3.0 standard contains 2 blocks of radicals - the 214 
traditional KangXi radicals encoded in U+2F00 through U+2FD5, and the CJK 
Radicals Extension block U+2E80 through U+2EF3.

There is some language on using those characters to construct unencoded Han 
characters in section 10.1 of Unicode 3.0.

Is this the same kind of thing you are talking about?
If changes to Unicode are needed, this has to go to the Unicode Consortium 
and ISO; this group can't do much about it.....

> the reason, I propose a "Request for Reference to be sent"

Better check....

So you foresee a system where

- User upgrades his data entry system
- User types a new ideograph into his systemm
- The client software calls out to some global repository for the
  new canonical dideograph decomposition of the new ideograph
- The DNS system looks up the decomposition, not the original codepoint
- The server knows enough to canonically decompose the zonefile's ideograph
- All this works correctly for software written by Indian programmers for
  American companies?

Seems complicated to me....

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 29 Aug 2001 17:23:28 +0200

--On 2001-08-29 13.24 +0200 Harald Tveit Alvestrand <[email-masked]>
wrote:

> - The server knows enough to canonically decompose the zonefile's
> ideograph - All this works correctly for software written by Indian
> programmers for   American companies?
> 

I rather see people come with comments on current text about versioning in
the nameprep document than starting all over again with new ideas.
Versioning is complicated, but Mark and myself though quite a lot on the
text which describes how clients and servers are to handle "unassigned"
codepoints.

  paf

---

From: Dave Crocker <[email-masked]>
Date: Wed, 29 Aug 2001 08:40:40 -0700

>the nameprep document than starting all over again with new ideas.

"versioning"?  please explain the problem.

thanks.

d/

----------

Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.273.6464