Subject: Re: [idn] what are the IDN identifiers?

From: Kenneth Whistler <[email-masked]>
Date: Wed, 28 Nov 2001 16:00:29 -0800 (PST)

Liana ruminated:

>   We have [STD13] defines that LDH are the DNS identifiers, 
> then what are the IDN identifiers?  UCS is too big and contains 
> many semantically equivalent characters for IDN.  Should we 
> ask for a table of semantically equivalent character sets 
> definition table from Unicode Consortium?

In my opinion, no. This concept of "semantically equivalent
character sets" is way too imprecisely defined to make sense.

What the Unicode Consortium provides is a large number of
precisely defined data tables, giving various properties
of the entire set of characters in the UCS. It is then
up to a group such as this, in the context of their particular
requirements (as for IDN identifiers) to make use of those
property tables to pick and choose among the characters
as appropriate to their application(s). (As has been done
for nameprep.)

> 
> If we are agree on the first RFC in Dan's list,
> I suggest to ask Unicode group to provide a table of
> "Semantically equivalent chatacters of UCS", where
> we can define which characters are used for 
> 1) label separators, ie puncturations and formating marks
> 2) structured data indicators, ie. $/%/& ...
> 3) unstructured data identifiers, ie. alphabet, CJKs, 
>  sound marks...
> "IDN identifiers" should be subset of such a table,
> to determine IDN nomalization protocol in the RFC.

The Unicode Consortium's take on identifiers is already
published in section 5.16 of the Unicode Standard, Version
3.0. Updated summary table information, covering the
repertoire of Unicode 3.1.1 can be found in:

ftp://www.unicode.org/Public/UNIDATA/DerivedCoreProperties.txt

and look for the ID_Start and ID_Continue properties.

By the way, "format[t]ing marks" are typically not label
separators. They are mostly ignorable for identifier
formation -- and can either be omitted from your identifier
repertoire or can be ignored (included silently) if not
omitted from your identifier repertoire. In either case they
do not *delimit* identifiers.

> "Semantically equivalent chatacters of UCS" means
> characters are equivalent to be used as an IDN identifier 
> when they are 
> 1)case insensitive, 
> 2)size or width insensitive,
> 3)font insensitive (include majority of TC/SC)
> 4)language insensitive (include CJK), 
> 5)combination insensitive(regardless NFC or KNFC). 
> 
>   Case, size, font insensitive is easy to understand,
> and have been addressed. 

What you seem to be aiming at here is a collection of various
kinds of character foldings. Character foldings -- even case
foldings -- are a rather murky area. The UTC position on
case folding is summarized in:

ftp://www.unicode.org/Public/UNIDATA/CaseFolding.txt

Regarding other kinds of foldings, the UTC is currently working
on a Unicode Technical Report on the subject. Nameprep involves
a number of foldings -- but these issues are not, in fact all
that easy to understand.

> TC/SC shall be under font 
> category, which is not addressed in Unicode. 

That you characterize TC/SC as a font folding illustrates part
of the problem. It is not a font folding, and cannot be handled
that way, except in the grossest manner.

> But 
> language and combination insensitive are the ones I'd 
> like to explain.
> 
>   Language insensitive: ie. circled numbers, circled
> Han numerals, Dingbats, subset of CJKs.  But other
> subset of CJK will be different semantically for each 
> languages, then we have to have separated tables to 
> work with for each or them.

Even with your examples, it isn't clear what you are talking
about here with the term "language insensitive".

> I think we are designing future IDN, we 
> assume all IDN has to be loaded somehow.   If Japanese
> agree on the semantic equivalence on the symbol to be 
> used in IDN, then we can ask  if  the current <business2> 
> handled by existing JIS local system can stay local without 
>  leaking into new IDN, and let <business2> be in 
> the semantically equivalent set for globle communication. 
>  Unicode group has to make such a choice for IETF.

Why? Language use and country conventions are not areas
that the Unicode Consortium holds expertise nor wishes
to establish standards in. 

> Case study 3):
>   Armenian samll n should be in with Latin n or not 
> is depending on the users' decision, that is we 
> take Unicode group advice on this, since they are the 
> language usage experts to make such a decision. 

Well, we aren't language use experts. But if you want
an expert opinion on *character* identity, ARMENIAN SMALL
LETTER NOW is not and never will be grouped with,
confused with, interchanged with LATIN SMALL LETTER N, any
more than it would be with U+30F3 KATAKANA LETTER N or
for that matter the Han character U+53C3 ni2, used in
Chinese transliteration of Nepal, Nero, Nile, nylon,
nicotine, Nicaragua, Nietzsche, Nice, and Nixon.

> If 
> the Armenian samll n is in with Latin, then we have 
> another case similar with CJK unification case 1).
> If they are not in with Latin, then we have another 
> case of Bengali and the alikes.

?? I presume this is an allusion to your concern that
U+09EA BENGALI DIGIT FOUR is confusable (out of context)
with the appearance of U+0038 DIGIT EIGHT. But Armenian
doesn't look the slightest bit like Latin, so it isn't
clear what you are on about here.

>   
> Combination insensitive: <i><acute on top>,<i><acute>
> <acute on top><i> shall be the same,

Well, these aren't all the same. This is a fundamental
misunderstanding of how combining characters work in
Unicode.

> all in Set
> <i+acute on top>.  This is the base for normalizing 
> from either a table (TC/SC like) or by a procedure 
> (NFC or KNFC like).

The Unicode Normalization Algorithm (which defines NFC
and NFKC  -- not "KNFC") is based on tables, too.

And it is not done by some vague notion of assembling all
the sets that we think ought to be "the same". It is
done by applying the algorithm rigorously to the defining
tables (of decompositions and of composition exclusions).

> 
> So the format is something like:
> <i>:           <I>,<tilt i>,<fat i>,<Greek i>,<Greek I>,...

Wrong from the start. Trying to mix scripts together like
that is completely unextensible.

> <i with acute>:<I with acute>,<i><acute>,<i><acute on top>,<I><acute>...
>  
> For reasonable request, I suggest we limit our scope
> to UCS Plain 0 characters. And we will end up with a 
> nicely display on the Web for us to read and for the 
> public to judge, instead of ieft draft with all the 
> U+E456.. which is meant for forting data and spotting
> checks.

The relevant data tables regarding Unicode normalization are
all already posted and are public to judge.

Beyond that, this effort to get someone (who?) to define all
the "semantically equivalent characters" just seems like
an ill-destined detour from actually getting the work on
IDN accomplished.

--Ken

---

From: liana Ye <[email-masked]>
Date: Wed, 28 Nov 2001 18:40:47 -0600

> as appropriate to their application(s). (As has been done

So that now, I think we need something less precise based
on their precise definitions, to group a larger equivalent set for 
IDN comparison purpose. 

> and look for the ID_Start and ID_Continue properties.

Thanks for the extracted pointer.  And 
these sections shows the problem:

3400..4DB5    ; ID_Start # Lo [6582] CJK UNIFIED IDEOGRAPH-3400..CJK
UNIFIED IDEOGRAPH-4DB5
4E00..9FA5    ; ID_Start # Lo [20902] CJK UNIFIED IDEOGRAPH-4E00..CJK
UNIFIED IDEOGRAPH-9FA5

> a number of foldings -- but these issues are not, in fact all

We have been discussed the narrownese on this document.
But it is great that the problem is been working on, and 
my suggestion is too late to be any useful here.  Can you 
provide some inside to the work?  

> that way, except in the grossest manner.

You can complain about how grossest the term I use, but 
I am talking about code points to be used in IDN as equivalent
to another code point when we are doing IDN matching. 
For example TC/SC, in addition to Kanji TC, Kanji SC, Hanja 
TC.  I am NOT talking about folding, sorry. 

> to establish standards in. 

Are you push the codepoint issue back to JET?  And aren't  we 
only dealing with codepoints comes from UTC, remember?

> nicotine, Nicaragua, Nietzsche, Nice, and Nixon.

So, according to your assertion ( I'd like UTC's confirmation 
on this of course, can you give me a pointer?) this is not 
a question any more.  Then how do you propose to deal
with an n from Armenian than the n from Latin in a label?

> clear what you are on about here.

It seems you have a way to handle context.  Then please 
describe how is your context is saved from input and passed
onto IDN and goes to DNS, and come out at another end 
correctly. 

> misunderstanding of how combining characters work in

Okay, give me another pointer so that I can learn.

> tables (of decompositions and of composition exclusions).

What is the purpose of these decompositions and 
compositions?  For input matching? for DNS matching?
for IDN folding?  I am looking for IDN matching.  Are we
talking about the same thing?

> that is completely unextensible.

You are right, this is an example of possible not look-alike 
cases from different scripts but for the same purpose to be 
used in IDN and to be treated equivalent, for example, Chinese 
period and an ASCII period.

> an ill-destined detour from actually getting the work on

I wander, what the work on IDN are you referring to? 

Liana

---

From: liana Ye <[email-masked]>
Date: Fri, 30 Nov 2001 01:24:59 -0600

Humm, I guess you mean I have raised CJK 
problem?

Liana

On Thu, 29 Nov 2001 14:35:47 +0800 "James Seng/Personal"
<[email-masked]> writes:
> > First, I have never used the term "multilingual",
> > second, I have always speak about IDN identifiers.
> 
> Once you touch "dealing with language" topic, you are talking about
> "multilingual domain names" and away from IDN identifiers.
> 
> -James Seng
>