Subject: Re: Combining characters (was: Re: [idn] hostname history hell)

From: Kenneth Whistler <[email-masked]>
Date: Wed, 21 Nov 2001 14:58:48 -0800 (PST)

John Klensin said:

> > Note that many scripts inherently include combining
> > characters. I absolutely agree with Kent that a blanket
> > prohibition of combining characters is unacceptable. In a
> > discussion dominated by English, Chinese, and Korean
> > speaker/writers, it might seem o.k., but I assure you that if
> > there were as many Arabic, Urdu, Hindi, and Bengali
> > speaker/writers participating, it would *not* seem o.k.
> 
> Ken, I may not have been reading closely enough, but I don't
> believe this discussion has included a proposal to ban combining
> characters.  I do have an issue with them, but I think it is
> separable (see below).

Kent and I were responding to Eric Hall's first proposal for 
a "safe set":

> How about this as a believable compromise: We start with a "safe set" of
> alphanumeric characters and specifically exclude punctuation, spacing,
> symbols, and combining characters.

John continued:

> The combining character problem (if it is a problem) is that, so
> far, we have no proposals on the table that would require that a
> DNS label be a valid name in any particular language, or even
> that it be drawn, homogeneously, from any particular script.
> Until and unless one of those rules is made (my guess is that it
> would be nearly impossible to do so, but this is not my area of
> expertise), 

It is impossible to do so, and hopeless to attempt. The concept of
what is "a valid name in any particular language" is hopelessly
compromised by borrowing of terminology and names back and forth. Is
"bach.org" a German name or an English name? How about "alliance.com"
versus "alliance.fr" (both exist) -- one is English and one is
French. Cross-script names are less common, but you do run into 
orthographies that borrow one or more letters from another script 
into their own alphabets, and historically these things change status 
over time.

> we are thrown back on the traditional DNS rule that,
> subject to the hyphen-placement rule, any valid character of the
> chosen CCS can appear in any relationship to any other valid
> character of the CCS.  In particular, there is no way to require
> or assume script-homogeniety.

Correct. Nor would you want to.

And this would be a problem even if you limited the IDN to,
for example, 8859-5 (Latin/Cyrillic) -- an 8-bit CCS with no
combining characters. This isn't a problem introduced by
Unicode -- it is just more obvious now because Unicode has so
*many* scripts in it.

And I don't think that combining characters have anything to
do with it.

> 
> If, to use your example, we have a selection of Arabic, Urdu,
> Hindi, and Bengali, which characters from each script designed
> by its first character,  
>    AUHBBHUA
> ought to be a valid label.  While we know how to construct AAAA,
> UUUU, HHHH, and BBBB, regardless of whether a given character is
> combining or non-combining, I worry about interpretation and
> ambiguity if combining characters (or partial breaks, etc.) are
> taken from one of these scripts and surrounded by characters
> from an unrelated script.  Maybe it is not a problem, but I'd
> like someone to assure me that is the case.

It is not a problem.

If you have AUHBBHUA which would, by the way, actually be

            AADBBDAA (where A=Arabic script, used for both Arabic and Urdu)
            12345678
and which would (assuming right-to-left context) display as:
            AADBBDAA
            <<>>>><<
            87345621

the presence of any combining characters amongst the various characters
in the string could influence the resolved direction of some characters,
and hence their display, but would not result in any ambiguities in
character *identity* or the string itself. The label is simply defined
by the underlying sequence of characters in their logical order.

Bidi can get you into visual contortions, but that is the result of
the bidi itself, and not of the presence of combining characters amidst
the bidi. You'd have the same problem with just Hebrew (no points) and
Latin (no accents).

--Ken

>      john

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 21 Nov 2001 19:22:48 -0600

Kenneth Whistler wrote:

> > How about this as a believable compromise: We start with a "safe set"
> > of alphanumeric characters and specifically exclude punctuation,
> > spacing, symbols, and combining characters.

The specific concern I had is already addressed in nameprep so there is no
need for the exception.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 22 Nov 2001 10:33:22 -0600

"Eric A. Hall" wrote:

> The specific concern I had is already addressed in nameprep so there
> is no need for the exception.

What I am concerned with is the use of domain name which consist entirely
of punctuation, or in this case, which consist entirely of combining
marks. These represent a rules problem with the principles behind the
"safe-set". EG, we forbid domain names like ````````.com using U+0060, but
````````.com may be legal if the combining character U+0300 is used
instead.

I get lost in the shrubbery of specs, so can somebody tell me whether or
not this is a valid concern? I can't tell for certain what happens at the
end of nameprep with such a label.

If this is a problem, a solution might be to change the prohibition
against leading hyphen to a prohibition against leading hyphen and
combining characters.

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 22 Nov 2001 22:53:20 +0100

> ````````.com may be legal if the combining character U+0300 is used

If that long sequence were displayable properly (it's a bit long for that), the
accents would stack up on top of each other, not be displayed as you indicate.

> against leading hyphen to a prohibition against leading hyphen and

I think that would be a good idea for the subnames* (otherwise the 
combining character applies to something before the name, or to
a period in the name). Ideally, a hyphen-minus should also never
be followed by a combining character in a domain/host** name.

However, no technical restriction to at least two characters in a
domain/host name should be accepted.

        Kind regards
        /kent k

* subnames (here): the name parts that are delimited by full stops.

** I will not try to keep track of which is what.  That issue appears totally confused.

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 25 Nov 2001 13:32:41 +0900

Does a combining sequence inherit the bicameral characteristic of the leading letter ?

For example,

can be lowercased into ? Then, ==> ?
... ==> ... ? I am not sure whether it can be.

Can be lowercased into or not?
has no combined form , but has as its lowercase form
due to its language context in turkish and azerbaijani.
Is this case just an exception ?

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 25 Nov 2001 13:59:39 +0900

(resent after correcing some formatting error)

For example,  

<A acute> can be lowercased into <a acute> ?  Then, <A><acute> ==> <a><acute> ?
<A><acute><diaeresis>... ==> <a><acute><diaeresis>... ?  I am not sure whether it can be.

Can <I><dot-above>   be lowercased into <i><dot-above> or not?
<I dot-above> has no combined form <i dot-above>, but has <i> as its lowercase form 
due to its language context in turkish and azerbaijani.
Is this case just an exception ?

Soobok Lee

  ----- Original Message ----- 
  From: Soobok Lee 
  To: [email-masked] 
  Sent: Sunday, November 25, 2001 1:32 PM
  Subject: Re: Combining characters (was: Re: [idn] hostname historyhell)

For example,  

Soobok Lee

---

From: David Hopwood <[email-masked]>
Date: Sun, 25 Nov 2001 18:46:30 +0000

-----BEGIN PGP SIGNED MESSAGE-----

> Does a combining sequence inherit the bicameral  characteristic of the

It doesn't matter: folding the base character of the combining sequence
is sufficient (usually; <I><dot-above> and sequences that include
Greek ypogegrammeni are exceptions).

> For example,  
>
> <A acute> can be lowercased into <a acute> ?
> Then, <A><acute> ==> <a><acute> ? <A><acute><diaeresis>... ==>
> <a><acute><diaeresis>... ?

Yes.

> Can <I><dot-above> be lowercased into <i><dot-above> or not?

The case-folded form [*] of <I><dot-above> is <i>, not <i><dot-above>.
This only works if the string is NFC-normalised before applying case
folding.

[*] Remember that we are applying case folding, not lowercase mapping.

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: "Soobok Lee" <[email-masked]>
Date: Mon, 26 Nov 2001 09:18:05 +0900

Now that <I><dot-above> is downcased to <i> as an exceptional case,
Then,  we have an interesting question:
which direction should we  lowercase   <I><dot-above><acute>   into ?

  1.    <I dot-above><acute>     ===>     <i><acute>    :  CaseMap(NFKC(x))
  2.    <i><dot-above><acute>                                       :  NFKC(CaseMap(x))

Does the whole combining sequence inherit the bicameral property of
the base character  or the partial combining sequence  minus the last mark ?
 
Soobok Lee
  
----- Original Message ----- 
From: "David Hopwood" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Monday, November 26, 2001 3:46 AM
Subject: Re: Combining characters (was: Re: [idn] hostname historyhell)

> -----BEGIN PGP SIGNATURE-----
> Version: 2.6.3i
> Charset: noconv
> 
> iQEVAwUBPAE8azkCAxeYt5gVAQFcVQf9Em4jK7x+IjPf/ZKtWPxKS80tmOJIVH2C
> 11sNkonVdILkcskWHQn/z6nkrXDh/l212AdJN2pMWKEVkbayOu3pB3HDYzo2Inve
> Oay4hPIX25jOBqAiaw7QYnHpECMjIQmRft27rK6CrRcF2dtD6cgqByBt5WV4IQH7
> S2V5EXD1JBOjCZGKUVioO7z+kZmyS+QvQngbd31D7xiFhEud3kvLK8XX6B6GYQR4
> 6l+aUHj1TVoxeR5hcExtKia5zC4b/Qa9i0U4hZraS01H0ATiiSavJUtv971cjhFU
> fmrg1LSTu+dkPS1yw/AwO82do2Vnf/SLeujLrPqNGgIn2FZEAT2thA==
> =2NF+
> -----END PGP SIGNATURE-----
>

---

From: David Hopwood <[email-masked]>
Date: Mon, 26 Nov 2001 01:12:57 +0000

-----BEGIN PGP SIGNED MESSAGE-----

> which direction should we  lowercase   <I><dot-above><acute>   into ?

To <i acute>. That is, the equivalence class is:

  <I><dot-above><acute>                         U+0049 U+0307 U+0301
  <I dot-above><acute>                          U+0130 U+0301
  <I><acute>                                    U+0049 U+0301
  <I acute>                                     U+00CD
  <i><acute>                                    U+0069 U+0301
  <i acute>                                     U+00ED
  <dotless i><acute>                            U+0131 U+0301
  <fullwidth I><acute>                          U+FF29 U+0301
  <fullwidth I><dot-above><acute>               U+FF29 U+0307 U+0301
  <fullwidth i><acute>                          U+FF49 U+0301

and if NFKC is used, also:

  <information source><acute>                   U+2139 U+0301
  <roman numeral one><acute>                    U+2160 U+0301
  <roman numeral one><dot-above><acute>         U+2160 U+0307 U+0301
  <small roman numeral one><acute>              U+2170 U+0301
  <circled I><acute>                            U+24BE U+0301
  <circled I><dot-above><acute>                 U+24BE U+0307 U+0301
  <circled i><acute>                            U+24D8 U+0301
  <bold I><acute>                               U+1D408 U+0301
  <bold I><dot-above><acute>                    U+1D408 U+0307 U+0301
  <bold i><acute>                               U+1D422 U+0301
  <italic I><acute>                             U+1D43C U+0301
  <italic I><dot-above><acute>                  U+1D43C U+0307 U+0301
  <italic i><acute>                             U+1D456 U+0301
  <bold italic I><acute>                        U+1D470 U+0301
  <bold italic I><dot-above><acute>             U+1D470 U+0307 U+0301
  <bold italic i><acute>                        U+1D48A U+0301
  <script I><acute>                             U+2110 U+0301
  <script I><dot-above><acute>                  U+2110 U+0307 U+0301
  <script i><acute>                             U+1D4BE U+0301
  <bold script I><acute>                        U+1D4D8 U+0301
  <bold script I><dot-above><acute>             U+1D4D8 U+0307 U+0301
  <bold script i><acute>                        U+1D4F2 U+0301
  <fraktur I><acute>                            U+2111 U+0301
  <fraktur I><dot-above><acute>                 U+2111 U+0307 U+0301
  <fraktur i><acute>                            U+1D526 U+0301
  <double-struck I><acute>                      U+1D540 U+0301
  <double-struck I><dot-above><acute>           U+1D540 U+0307 U+0301
  <double-struck i><acute>                      U+1D55A U+0301
  <bold fraktur I><acute>                       U+1D574 U+0301
  <bold fraktur I><dot-above><acute>            U+1D574 U+0307 U+0301
  <bold fraktur i><acute>                       U+1D58E U+0301
  <sans-serif I><acute>                         U+1D5A8 U+0301
  <sans-serif I><dot-above><acute>              U+1D5A8 U+0307 U+0301
  <sans-serif i><acute>                         U+1D5C2 U+0301
  <sans-serif bold I><acute>                    U+1D5DC U+0301
  <sans-serif bold I><dot-above><acute>         U+1D5DC U+0307 U+0301
  <sans-serif bold i><acute>                    U+1D5F6 U+0301
  <sans-serif italic I><acute>                  U+1D610 U+0301
  <sans-serif italic I><dot-above><acute>       U+1D610 U+0307 U+0301
  <sans-serif italic i><acute>                  U+1D62A U+0301
  <sans-serif bold italic I><acute>             U+1D644 U+0301
  <sans-serif bold italic I><dot-above><acute>  U+1D644 U+0307 U+0301
  <sans-serif bold italic i><acute>             U+1D65E U+0301
  <monospace I><acute>                          U+1D678 U+0301
  <monospace I><dot-above><acute>               U+1D678 U+0307 U+0301
  <monospace i><acute>                          U+1D692 U+0301

<i acute> U+00ED is the normalised representative for all of these.

<i><dot-above><acute> is in a different equivalence class (AFAIK, no
language uses it, so this doesn't matter).

>   2.    <i><dot-above><acute>                           :  NFKC(CaseMap(x))

Again, we're doing a variant of case folding, not case mapping. UTR #21
describes the difference.

Note that neither fold(NFKC(x)) nor NFKC(fold(x)) is sufficient in general,
because fold(NFKC(x)) is not guaranteed to be NFC-normalised, and this
function is not idempotent.

My preference to fix the problems with dotless-i and ypogegrammeni would
be NFC(fold(NFC(x))).

(This can be optimised to NFC-compose(fold(NFD(x))), or alternatively
the folding can detect cases where the second NFC application is needed -
usually it is not.)

> the base character or the partial combining sequence minus the last mark ?

Combining sequences technically don't have properties; only characters have
properties. However, for most purposes a combining sequence should be treated
as having the case of its base character.

- --

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 26 Nov 2001 08:21:32 +0100

> the folding can detect cases where the second NFC application is needed -

See section 3 of draft-hoffman-stringprep-00.txt.

   paf

---

From: David Hopwood <[email-masked]>
Date: Mon, 26 Nov 2001 18:41:54 +0000

-----BEGIN PGP SIGNED MESSAGE-----

David Hopwood wrote:
> and if NFKC is used, also: [snip]
> 

My mistake; it is used in Lithuanian. The Lithuanian usage would argue
for <i><dot-above><acute> being in the same equivalence class (since its
Lithuanian uppercase form is <I acute>). So, another solution that
should be considered is to use NFC o fold as in the current version of
stringprep, but map out U+0307 whenever it is attached to a character
based on 'i' or 'I'. That wouldn't cause any problems for Turkish or
Azeri. I'll list all the options in another post.

- --

---

From: David Hopwood <[email-masked]>
Date: Mon, 26 Nov 2001 21:15:32 +0000

-----BEGIN PGP SIGNED MESSAGE-----

['o' means functional composition. Everything below applies equally to
NFKC and NFC, but I'll say "NFC".]

Patrik Fältström <[email-masked]> wrote:
> On 01-11-26 01.12 +0000 David Hopwood <[email-masked]> wrote:
> 

That section doesn't address the problems we're talking about.
There are four problems with the interaction between case folding and
NFC that need to be considered:

1. For a string that includes a character with Greek ypogegrammeni/
   prosgegrammeni (U+0345) in its decomposition, folding only works
   correctly if the input string is normalised (i.e. either U+0345
   must be last in the combining sequence, or the character must be
   fully composed).

2. A string that includes decomposed <I><dot-above> will be case-folded
   to <i><dot-above>, which is inconsistent with the folding of composed
   <I dot-above> to <i>. Any solution to this must also handle the case
   where there are other combining characters between the <I> and
   <dot-above>, or composed with the <I>.

3. If a character X folds to a sequence of two characters Y Z, then
   Z may have precomposed forms that do not correspond to precomposed
   forms of X.

   For example, <sharp-s> U+00DF does not precompose with anything,
   but <s> U+0074 does, so <sharp-s><acute> will fold to <s><s><acute>,
   instead of the NFC-normalised form <s><s acute>.
   (In practice this will not occur in real-world names, so strings
   like <sharp-s><acute> *could* be disallowed.)

4. There are a small number of lowercase precomposed characters that do
   not have uppercase precomposed equivalents; they are listed in
   the SpecialCasing-5.txt file:

     U+0390 GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS
     U+03B0 GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS
     U+01F0 LATIN SMALL LETTER J WITH CARON
     U+1E96 LATIN SMALL LETTER H WITH LINE BELOW
     U+1E97 LATIN SMALL LETTER T WITH DIAERESIS
     U+1E98 LATIN SMALL LETTER W WITH RING ABOVE
     U+1E99 LATIN SMALL LETTER Y WITH RING ABOVE
     U+1F50 GREEK SMALL LETTER UPSILON WITH PSILI
     U+1F52 GREEK SMALL LETTER UPSILON WITH PSILI AND VARIA
     U+1F54 GREEK SMALL LETTER UPSILON WITH PSILI AND OXIA
     U+1F56 GREEK SMALL LETTER UPSILON WITH PSILI AND PERISPOMENI
     U+1FB6 GREEK SMALL LETTER ALPHA WITH PERISPOMENI
     U+1FC6 GREEK SMALL LETTER ETA WITH PERISPOMENI
     U+1FD2 GREEK SMALL LETTER IOTA WITH DIALYTIKA AND VARIA
     U+1FD3 GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA
     U+1FD6 GREEK SMALL LETTER IOTA WITH PERISPOMENI
     U+1FD7 GREEK SMALL LETTER IOTA WITH DIALYTIKA AND PERISPOMENI
     U+1FE2 GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND VARIA
     U+1FE3 GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA
     U+1FE4 GREEK SMALL LETTER RHO WITH PSILI
     U+1FE6 GREEK SMALL LETTER UPSILON WITH PERISPOMENI
     U+1FE7 GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND PERISPOMENI
     U+1FF6 GREEK SMALL LETTER OMEGA WITH PERISPOMENI

   Folding an NFC-normalised string containing the decomposed uppercase
   forms of these characters, will result in a string that is not
   NFC-normalised.

   (The file also lists

     U+0149 LATIN SMALL LETTER N PRECEDED BY APOSTROPHE
     U+1E9A LATIN SMALL LETTER A WITH RIGHT HALF RING

   but U+0149 causes only problem 3 above, and U+1E9A does not cause
   any problems for normalisation.)

Problems 1 and 2 mean that "NFC o fold" doesn't work, unless the
foldings that map ypogegrammeni/prosgegrammeni to iota and dotted-I
to i are omitted.

Problems 3 and 4 mean that "fold o NFC" doesn't work, regardless
of which foldings are used.

A pathological example that shows all four problems is:

  <alpha ypogegrammeni><varia><I><dot-below><dot-above><sharp-s><acute><J><caron>
  "\u1FB3\u0300\u0049\u0323\u0307\u00DF\u0301\u004A\u030C"

There are several ways of solving all the problems simultaneously:

 a) NFC o fold o NFC.

 b) foldafter o NFC o foldbefore, where foldafter handles only <dotted-I>
    and <ypogegrammeni> and foldbefore handles the rest.

 c) foldafter' o NFC o foldbefore', where foldbefore' handles the characters
    that only have lowercase precomposed variants, and foldafter' handles
    the rest. If a character folds to two characters, then it must not be
    immediately followed by a combining mark [*].

 d) NFC o fold', where fold' maps out <dot-above> when it applies to
    <i> or <I> or their fullwidth variants, and includes only the
    "simple" foldings for characters containing ypogegrammeni/prosgegrammeni.
    (Unlike the standard Unicode case folding algorithm and options a)-c)
    or e), this handles Lithuanian correctly.)

 e) NFC o fold'', where fold'' does not include a case folding for
    <dotted-I>, and includes only the "simple" foldings for characters
    containing ypogegrammeni/prosgegrammeni. This means that names are
    not fully case-insensitive for Turkish, Azeri or Lithuanian, but it
    is simpler than a)-d).

 f) foldascii, where encoded strings are required to already be
    NFC-normalised, and cannot contain uppercase or titlecase non-ASCII
    characters. This approach does not support case preservation for
    non-ASCII strings; it also requires that any process that destroys
    NFC normalisation must re-normalise the name. OTOH, it means that
    systems that only resolve names and do not generate them, need not
    implement NFC. If I understand correctly, this is what Dan Bernstein
    is arguing for.

I'm now leaning towards d) or f), but any of these approaches would work.

[*] The definition of a combining mark here is a character with canonical
    combining class > 0.

- --

---

From: David Hopwood <[email-masked]>
Date: Tue, 27 Nov 2001 10:48:08 +0000

-----BEGIN PGP SIGNED MESSAGE-----

> There are several ways of solving all the problems simultaneously:

I should clarify that the functions I'm referring to here in a)-f) are
alternatives for the function that is used by applications to test
equivalence of valid encoded names. In the case of g), unnormalised names
are not valid, so it is sufficient to use ASCII case folding to test
equivalence.

>     and <ypogegrammeni> and foldbefore handles the rest.

Actually a) and b) define the same function, so they should be considered
as a single alternative; call it ab).

- --

---

From: David Hopwood <[email-masked]>
Date: Tue, 27 Nov 2001 11:14:56 +0000

-----BEGIN PGP SIGNED MESSAGE-----

David Hopwood wrote:
> equivalence of valid encoded names. In the case of g),

That should be "in the case of f)".

> unnormalised names are not valid, so it is sufficient to use ASCII case
> folding to test equivalence.
> 

- --