Subject: [idn] IDNA comment 1 : applications' own normalization vs stringprep

From: "Soobok Lee" <[email-masked]>
Date: Tue, 12 Feb 2002 01:39:48 +0900

In the IDNA draft version6,  section 6.1, paragraph 4 and 5,

"In protocols and document formats that define how to handle
specification or negotiation of charsets, labels can be encoded in any
charset allowed by the protocol or document format. If a protocol or
document format only allows one charset, the labels MUST be given in
that charset."

"In any place where a protocol or document format allows transmission of
the characters in internationalized labels, internationalized labels
SHOULD be transmitted using whatever character encoding and escape
mechanism that the protocol or document format uses at that place."

Here,we need more security warnings in this section regarding to applications'
own normalizations that may collided with STRINGPREP's NFC/NFKC.
 
Like some XML and HTML  standards, there may be applications that performs
NFC,NFKC or other normalizations on their data/text contents that 
may contain IDN labels that will be stringprepped and the ACEed later time.

But, even in the case NFC, stringprep(NFC(x)) == stringprep(x) is not always 
guaranteed, especially in the case of <I dot above> and <I><dot above> ( + <acute>).
That will cause silent failures in applications.

Current premature UAX15 used in stringprep and other compensating or superior normalizations that may
be from the same UTC or other organizations  may collide to each other in
a sequence of normalizatiom processes eventually ending in stringprep  
in many mission critical applications.

We need more researches and inspections for the possiblities of normalization vs normalization
conflicts and include the warning or recommendations in the IDNA specficiations.

Soobok Lee

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 11 Feb 2002 08:59:40 -0800

>stringprep(NFC(x)) == stringprep(x)

This was brought up early in the Unicode 3.2 development. We have
programmatically checked, and I with dot is the only case that causes
a problem. It will be discussed at the UTC meeting this week. I have
no doubt that it will be resolved for U3.2, and even if StringPrep
doesn't pick up U3.2, it could add a mapping to that one case.

Mark
—————

Πόλλ’ ἠπίστατο ἔργα, κακῶς δ’ ἠπίστατο πάντα — Ὁμήρου Μαργίτῃ
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: <[email-masked]>
Sent: Monday, February 11, 2002 08:39
Subject: [idn] IDNA comment 1 : applications' own normalization vs
stringprep

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 12 Feb 2002 02:03:43 +0900

----- Original Message -----
From: "Mark Davis" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>
Sent: Tuesday, February 12, 2002 1:59 AM
Subject: Re: [idn] IDNA comment 1 : applications' own normalization vs stringprep

> doesn't pick up U3.2, it could add a mapping to that one case.

Okay!     I am glad to make useful contributions to UNICODE.

The honor is mine!

But, still we may have many compensation UTC or non-UTC normalizations
especially for hangul or other scripts, and that will make similar stringprep
faiilures..

That is , the possibility of     stringpre(newnormlization(x) != stringpre(x).

Soobok Lee

---

From: David Hopwood <[email-masked]>
Date: Tue, 12 Feb 2002 08:52:19 +0000

-----BEGIN PGP SIGNED MESSAGE-----

[Cross-posted from the IDN list; reply-to set to [email-masked].
Change it back for replies that relate specifically to IDN.]

Mark Davis wrote:
> >stringprep(NFC(x)) == stringprep(x) [does not always hold]
> 
> a problem.

No it isn't:

    NFD(toCasefold("\u1FB2"))
        = NFD("\u1F70\u03B9")       = "\u03B1\u0300\u03B9" (alpha-varia, iota)
    NFD(toCasefold("\u1FB3\u0300"))
        = NFD("\u03B1\u03B9\u0300") = "\u03B1\u03B9\u0300" (alpha, iota-varia)

(Which NF is used after toCasefold isn't important.)

What algorithm did you use to check this? In any case, I'll discuss case
folding in part 3 of my Unicode 3.2 comments, later today.

> I have no doubt that it will be resolved for U3.2, and even if StringPrep

How would you add a mapping from a sequence of two characters (e.g.
"I\u0307"), without changing the stringprep/nameprep algorithm, rather
than the tables?

(Actually there is an indirect way to do it: map out U+0307, and map all
composed characters with dot-above to the forms without dot-above. However,
that won't work for the standard case folding algorithm, because it would
break consistency with earlier versions, and it doesn't fix the problem
with Greek ypogegrammeni/prosgegrammeni, anyway.)

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip