Subject: Re: Combining characters (was: Re: [idn] hostname historyhell)

From: liana Ye <[email-masked]>
Date: Sun, 25 Nov 2001 00:16:00 -0600

I think  [nameprep] handle them inconsistently, that is 

038F; 03CE; Case map 

and your latter case  <A><acute> ==> > <a><acute> is 
not considered. But: 

0390; 03B9 0308 0301; Case map
0587; 0565 0582; Case map

do use character combination to normalize the symbols.

Liana

On Sun, 25 Nov 2001 13:59:39 +0900 "Soobok Lee" <[email-masked]>
writes:
> (resent after correcing some formatting error)
> 
> Does a combining sequence inherit the bicameral  characteristic of 
> the leading letter ?   
> 
> For example,  
> 
> <A acute> can be lowercased into <a acute> ?  Then, <A><acute> ==> 
> <a><acute> ?
> <A><acute><diaeresis>... ==> <a><acute><diaeresis>... ?  I am not 
> sure whether it can be.
> 
> Can <I><dot-above>   be lowercased into <i><dot-above> or not?
> <I dot-above> has no combined form <i dot-above>, but has <i> as its 
> lowercase form 
> due to its language context in turkish and azerbaijani.
> Is this case just an exception ?
> 
> Soobok Lee
> 
>   ----- Original Message ----- 
>   From: Soobok Lee 
>   To: [email-masked] 
>   Sent: Sunday, November 25, 2001 1:32 PM
>   Subject: Re: Combining characters (was: Re: [idn] hostname 
> historyhell)
> 
> 
> Does a combining sequence inherit the bicameral  characteristic of 
> the leading letter ?   
> 
> For example,  
> 
>  can be lowercased into  ?  Then,  ==>  ?
> ... ==> ... ?  I am not sure whether it can be.
> 
> Can    be lowercased into  or not?
>  has no combined form , but has  as its lowercase form 
> due to its language context in turkish and azerbaijani.
> Is this case just an exception ?
> 
> Soobok Lee
> 
> 
> ----- Original Message ----- 
> From: "Eric A. Hall" 
> To: "Kenneth Whistler" ; ; 
> Sent: Friday, November 23, 2001 1:33 AM
> Subject: Re: Combining characters (was: Re: [idn] hostname 
> historyhell)
> 
> 
> > 
> > "Eric A. Hall" wrote:
> > 
> > > The specific concern I had is already addressed in nameprep so 
> there
> > > is no need for the exception.
> > 
> > What I am concerned with is the use of domain name which consist 
> entirely
> > of punctuation, or in this case, which consist entirely of 
> combining
> > marks. These represent a rules problem with the principles behind 
> the
> > "safe-set". EG, we forbid domain names like ````````.com using 
> U+0060, but
> > ````````.com may be legal if the combining character U+0300 is 
> used
> > instead.
> > 
> > I get lost in the shrubbery of specs, so can somebody tell me 
> whether or
> > not this is a valid concern? I can't tell for certain what happens 
> at the
> > end of nameprep with such a label.
> > 
> > If this is a problem, a solution might be to change the 
> prohibition
> > against leading hyphen to a prohibition against leading hyphen and
> > combining characters.
> > 
> > -- 
> > Eric A. Hall                                        
> http://www.ehsco.com/
> > Internet Core Protocols          
> http://www.oreilly.com/catalog/coreprot/
> > 
> 
> 
>

---

From: liana Ye <[email-masked]>
Date: Sun, 25 Nov 2001 14:55:37 -0600

Sorry, I fogot about checking cc field.  Thank you for catching it,
and I'd better clean up this message a little for a easier reading.

> > > liana Ye wrote:
> > >
> > > > I'd like to propose a more specific layering of IDN symbols:
> > > >
> > > >  From the top where the user input buffer offers:
> > > >
> > > >  Layer 3:  label seperators and label order normalizing
> > > >
> > > >  Layer 2: Bidi label normalizing (or verticle label >
normalizing)
> > >
> > > What is the current display order for unstructured and  structured
> > > data in right-to-left display systems? Does unstructured data "the 
> files are on server1") typically flow RTL, while URLs and other
structured 
> data display as LTR?

 As far as I know, unstructured data mostly based on LTR  internally,
and displayed as RTL or swaped to topdown display. But I don't
 know how people handles Mongolian and Man text specifically,
they are also the cases I have in mind when I have mentioned Layer 3
 and Layer 2.

> > > It seems that these questions are for the structured data groups to
> > > handle when they decide on an output presentation mechanism. 
        But if  URLs and
> > > other structured types will display RTL then that may affect us 
> as well (your Layer 3 label ordering in particular).

Yes.  These are issues have been raised before by Ben.  For
example, Chinese addresses from general to specifics:
 China, province, city, district, street, building#, appartment#

While English is the reverse but only partially:
  Building#, street, appartment#, city, state, country

From processing view point, the Chinese way produces less
confusion, while our TDL has been following the English way.
For IDN, if we are speaking of backward compatibility with
 the current DNS, we do need to allow different local label 
ordering. Which group handling this part?

 As this group concerns, I think we need come up with
1) a list of equivalent symbols as label separators;
2) a list of special character processing protocol, which 
mostly have been in use from the first day of ASCII 
standard, for example:   $/%/?/*, which may indicate 
how to handle special characters whthin a label.  
I think this the base for us to work on bidi and vertical
 issues in Layer 2 nomalization.

 After we get the two lists out of our way, and we can have 
a table of prohibition and a sensible equivalent list to 
work with before we broadly exclude all of them in
 [nameprep].

 Liana

> > > >  Layer 1.5: diacritic marks and combining symbol normalizing
> > > >
> > > >  Layer 1:  IDN identifier matching or whatever comes out
> > > >                of [nameprep].
> > > >
> > > > The reason for Layer 1.5 is that these symbols can be
> > > > treated in a similar way with Han characters depending on
> > > > what architecture we end up with, and what ACE will be
> > > > our focus.
> > >
> > > --
> > > Eric A. Hall
> > > http://www.ehsco.com/
> > > Internet Core Protocols

---

From: liana Ye <[email-masked]>
Date: Sun, 25 Nov 2001 15:19:09 -0600

I can see web pages to accomodate all of this, but does 
this belong to IDN? What is IDN's duty area from symbol
set perspective?  I have listed three layers of symbols. 
I hope we can propose an RFC to get label separator and 
bidi/vertical or whatever text related issue out of our way
so that we can deal with IDN identifier symbols only. 

Liana

On Sun, 25 Nov 2001 22:56:02 +0200 <[email-masked]> writes:
> From a bidi perspective, URLs are not structured. A present day all
> ASCII URL will be presented left-to-right, because all the 
> non-letters
> contained within it, such as periods and slashes, are neutral and 
> the
> directionality of the surrounding letters, i.e. LTR. Trailing digits
> work out correctly too.
> 
> When the URL, in the future when it is possible, will be entirely in
> Hebrew, it will display correctly with components going from right 
> to
> left.
> 
> Mixed language URLs may cause some unexpected results.
> 
> See http://www.qsm.co.il/Hebrew/HebrewTest/url.htm
> 
> Jony Rosenne
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]] 
> > On Behalf Of Eric A. Hall
> > Sent: Sunday, November 25, 2001 7:43 PM
> > To: liana Ye
> > Cc: [email-masked]
> > >  Layer 2: Bidi label normalizing (or verticle label normalizing)
> > 
> > What is the current display order for unstructured and 
> > structured data in right-to-left display systems? Does 
> > unstructured data ("the files are on
> > server1") typically flow RTL, while URLs and other structured 
> > data display as LTR?
> > 
> > It seems that these questions are for the structured data 
> > groups to handle when they decide on an output presentation 
> > mechanism. But if URLs and other structured types will 
> > display RTL then that may affect us as well (your Layer 3 
> > label ordering in particular).
> > 
> > > treated in a similar way with Han characters depending on what 
> > > architecture we end up with, and what ACE will be our focus.
> > 
> > -- 
> > Eric A. Hall                                        
> http://www.ehsco.com/
> Internet Core Protocols

---

From: liana Ye <[email-masked]>
Date: Mon, 26 Nov 2001 00:17:51 -0600

Have you thought about " Mixed language URLs "
with language tags, for example:

www.zh-china/mo-mogolia/zh-county/mybusiness.com

shall be able to work?

Liana

---

From: [email-masked]
Date: Mon, 26 Nov 2001 11:40:52 EST

> www.zh-china/mo-mogolia/zh-county/mybusiness.com

I thought one of the fundamental characteristics of domain names, host names, 
URLs, etc. is that they were identifiers, not true names, and hence they were 
not intended to be language-tagged.

Just as an example, two popular search engines are teoma.com and 
altavista.com.  What language is "Teoma"?  Is "Alta Vista" supposed to be 
Spanish or Italian?  Should we care?

-Doug Ewell
 Fullerton, California

---

From: liana Ye <[email-masked]>
Date: Mon, 26 Nov 2001 09:59:50 -0600

Hi, Doug:

Due to many different use of scripts, and look-alike symbols,
some sort of classification is unavoidable. 

> Spanish or Italian?  Should we care?

No, we don't care, because Latin script has taken 
care of the some of the spoken language difference for 
us already.  So does Arabic, Cyrillic and Chinese scripts.  

However, the problem is a lot more complex for those 
mixed scripts, such as Japanese and Vietnamese.
Japanese is not only phonetically different from Chinese,
it is grammatically completely different from Chinese too.
Even we are dealing with structured data, the difference 
so great, that we have to consider to treat them differently, 
thus the use of "language tags" or "script tags".  
The Mogolian and Han mixed use is only another 
example for such a case, which the Chinese group 
has not raised before this group - They have too much 
on their hands already :-(

I have used "language tag" instead of "script tag", because
1) different languages using the same script, as the CJK 
cases.  2) "language tag" has been in [ISO639] already, 
we don't need argue if Cantonese needs a tag or not. 
That is an issue has been solved by [ISO639]. 3) We 
can use the tags already defined, but IETF doesn't need
to implement every language tag defined in [ISO639],
it is up to engineering consideration to cover all the cases
well enough to facilitate communications on DNS.

If this tag issue is raised in IDN down the line, for 
example, regarding different  uses of diacritic marks 
between French and Dutch, then it is a challenge to the 
design of DNS tag coverage we are doing now, 
since we should have taken care of this issue among 
Latin users when DNS tag coverage is discussed.

So for language tags we have to solve at this stage,
I would suggest:

CJK
Latin
Cyrillic
Arabic
Bengali
Greek

Although Greek does not necessarily cover a lot native 
users, it is familar to many Latin users, and serve a 
good case study for discussion. 

Liana

---

From: liana Ye <[email-masked]>
Date: Mon, 26 Nov 2001 11:28:07 -0600

Hi, Doug:

Due to large character set like UCS, mixed use of scripts
and look-alikes cases are so prominent, some sort
of classification is unavoidable.

Should it be

> Spanish or Italian?  Should we care?

No, we don't care about these cases, since they 
are Latin users.  Latin has taken care of many different 
spoken languages already, so do Cyrillic, Arabic and
Chinese. 

> Spanish or Italian?  Should we care?

However,  Mixed script is a lot more complex.  For 
example, Japanese uses Kanji, but it is not only phoneticaly  
different from Chinese, its grammar is completely different
from Chinese.  The difference is so great that we have to 
reflect them separeatly in structured data too.  My Chinese
address label in previous message is an example of 
such a difference.  My Mogolia and Chinese example is 
another example of mixed used of structured labels, though
the Chinese group has not raise this in front of this group. 
They have too much on their hands already :-(

The classification, language tags or script tags,  must be 
used sometime in URL to deal with these issues.  I have
used "language tag" instead of "script tag", since
1) Different languages uses the same script, such as 
  CJK.
2) Language tag has been defined in [iso639], and 
  some of the issues have been solved already. For 
 example:  Does Cantonese have a language tag or not?
3) From engineering point of view, IETF has a list of language
 reqiurements to consider.  That is, can we come up a 
solution to cover these cases in DNS?

If in the future, down the line in IDN, someone challenge
us regarding diacrtic marks between French and Dutch, 
for example, then we have to be able to say this case is  
covered with a Latin tag.  If someone wants more localized
features with a French tag, then we may question if such a 
feature can be accomodated with existing methods or 
not before we lunch into another tag. 

I would suggest the following language tags to be 
considered first:

CJK
Latin
Cyrillic
Arabic
Bengali
Greek

Although Greek has smaller group of native users, it is
familar to many Latin users, and can serve as a study
case in discussion. 

Liana

---

From: liana Ye <[email-masked]>
Date: Mon, 26 Nov 2001 11:45:51 -0600

Sorry, I did not know the previous copy was sent out
correctly, since I had a dead lock case and have killed 
the buffer.  

Liana

---

From: liana Ye <[email-masked]>
Date: Mon, 26 Nov 2001 13:42:36 -0600

> 
> David Hopwood wrote:
> > Soobok Lee wrote:
> > > Now that <I><dot-above> is downcased to <i> as an exceptional 
> case,
> > > Then, we have an interesting question:
> > > which direction should we  lowercase   <I><dot-above><acute>   
> into ?
> > 
> > To <i acute>. That is, the equivalence class is:
> > 
> >   <I><dot-above><acute>                         U+0049 U+0307 
> U+0301
> >   <I dot-above><acute>                          U+0130 U+0301
> >   <I><acute>                                    U+0049 U+0301
> >   <I acute>                                     U+00CD
> >   <i><acute>                                    U+0069 U+0301
> >   <i acute>                                     U+00ED
> >   <dotless i><acute>                            U+0131 U+0301
> >   <fullwidth I><acute>                          U+FF29 U+0301
> >   <fullwidth I><dot-above><acute>               U+FF29 U+0307 
> U+0301
> >   <fullwidth i><acute>                          U+FF49 U+0301
> > 
> > and if NFKC is used, also: [snip]
> > 
> > <i acute> U+00ED is the normalised representative for all of 
> these.
> > 
> > <i><dot-above><acute> is in a different equivalence class (AFAIK, 
> no
> > language uses it, so this doesn't matter).
> 
> My mistake; it is used in Lithuanian. The Lithuanian usage would 
> argue
> for <i><dot-above><acute> being in the same equivalence class (since 
> its
> Lithuanian uppercase form is <I acute>). So, another solution that
> should be considered is to use NFC o fold as in the current version 
> of
> stringprep, but map out U+0307 whenever it is attached to a 
> character
> based on 'i' or 'I'. That wouldn't cause any problems for Turkish or
> Azeri. I'll list all the options in another post.
> 

This is a good demostration case, that when we deal 
with symbol usages cross different locality, a procedure
like NFC will overlook something.  It is better to list 
all the input and output characters in a table for 
easy checking, and easy to understand what are in 
an equivalent set. 

However, something like NFC is necessary as a tool 
to check consistence and as a guide to form such
a table.

Liana