Subject: [idn] 1st stringprep issue: not answered and ignored

From: "Soobok Lee" <[email-masked]>
Date: Sat, 4 May 2002 12:59:27 +0900

Is the following issue    addressed or mentioned in new stringprep/nameprep draft?

<I dot above> and <I><dot above>  represent the same character. that is, they are NFC-equivalent.
But, UAX21 (casefolding in stringprep) maps the latter one errornously into <i><dot above> 
(very different than the correct one <i>). 
In lowercaseing, <I><dot above> should be converted as a whole, instead of by char-by-char basis.
 
Stringprep should have contained warnings about this or included new revised casefolding operatins/tables,
but it didn't . the authors and co-chairs have ignored this repeated warnings 
without any responses in this list. This is not a theoretical issue, 
but a practical issue because the sequences are used by modern east-europeans.

Therefore,stringprep(NFC(x)) == stringprep(x)  is not guaranteed. This will cause unnoticeable failures.

It's ridiculous that repeated warnings are ignored/unanswered and the drafts went to IESG.
This will result in downplay on overall IETF processes and its reputations.

Soobok Lee

----- Original Message ----- 
From: "Soobok Lee" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, February 12, 2002 1:39 AM
Subject: [idn] IDNA comment 1 : applications' own normalization vs stringprep

> 
> In the IDNA draft version6,  section 6.1, paragraph 4 and 5,
> 
> "In protocols and document formats that define how to handle
> specification or negotiation of charsets, labels can be encoded in any
> charset allowed by the protocol or document format. If a protocol or
> document format only allows one charset, the labels MUST be given in
> that charset."
> 
> "In any place where a protocol or document format allows transmission of
> the characters in internationalized labels, internationalized labels
> SHOULD be transmitted using whatever character encoding and escape
> mechanism that the protocol or document format uses at that place."
> 
> Here,we need more security warnings in this section regarding to applications'
> own normalizations that may collided with STRINGPREP's NFC/NFKC.
>  
> Like some XML and HTML  standards, there may be applications that performs
> NFC,NFKC or other normalizations on their data/text contents that 
> may contain IDN labels that will be stringprepped and the ACEed later time.
> 
> But, even in the case NFC, stringprep(NFC(x)) == stringprep(x) is not always 
> guaranteed, especially in the case of <I dot above> and <I><dot above> ( + <acute>).
> That will cause silent failures in applications.
> 
> Current premature UAX15 used in stringprep and other compensating or superior normalizations that may
> be from the same UTC or other organizations  may collide to each other in
> a sequence of normalizatiom processes eventually ending in stringprep  
> in many mission critical applications.
> 
> We need more researches and inspections for the possiblities of normalization vs normalization
> conflicts and include the warning or recommendations in the IDNA specficiations.
> 
> Soobok Lee
> 
> 
> 
> 
>

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sat, 4 May 2002 09:03:17 -0700

At 12:59 PM +0900 5/4/02, Soobok Lee wrote:
>but it didn't .

Correct. This has been covered many times on the mailing list. You 
want stringprep to change the tables issued by the Unicode Consortium 
so that stringprep does not do NFKC the same way all other software 
that does NFKC does. There is little support for that proposition in 
the WG.

>  the authors and co-chairs have ignored this repeated warnings
>without any responses in this list.

There is a big difference between "ignoring" and "disagreeing with". 
We hear you asking over and over for something for which there is 
little support. Are you saying that we should include the concerns of 
every single person, even if the rest of the group disagrees with 
them?

>This will cause unnoticeable failures.

Your equation is completely wrong. NFKC is part of stringprep.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 5 May 2002 03:15:02 +0900

----- Original Message ----- 
From: "Paul Hoffman / IMC" <[email-masked]>
To: <[email-masked]>
Sent: Sunday, May 05, 2002 1:03 AM
Subject: Re: [idn] 1st stringprep issue: not answered and ignored

> that does NFKC does. There is little support for that proposition in 

I am not against NFKC.  I am not proposing for modification to NFKC,
but to UAX21 casefoldings. I just pointed our that The UAX21 casefolding is 
wrong at the example.
Who had answered at my questions before your this article? 
How to measure the consensus? I have never seen this WG had discussed this issue in depth.

> Your equation is completely wrong. NFKC is part of stringprep.

Alas. You still don't get my point.

The NFC(x) above is , for example, from IRI preprocessing or HTML authoring,
which often require NFC normalization on their inputs. 
To rephrase the above equation,

"NFKC(UAX21_casefold(NFC(x))) ==  NFKC(UAX21_casefold(x)) is not guaranteed."

If you preprocess IDN with NFC, you will get different namepreped ACE labels
than before in the IDN samples including <I><dot-above>.

Soobok Lee

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Sat, 4 May 2002 12:05:24 -0700

At 3:15 AM +0900 5/5/02, Soobok Lee wrote:
>but to UAX21 casefoldings.

My apology for mixing up your previous arguments for changing NFKC 
with your current argument for changing UAX21. The result is, of 
course, the same: you are asking us to change an external standard so 
that our protocol will not work the same as other software that 
follows UAX21.

>  I just pointed our that The UAX21 casefolding is
>wrong at the example.

That's not true: you did much more than "just" point that out. You 
asked for us to change the mapping from the Unicode Consortium, and 
you accused the authors of the drafts of ignoring you.

>Who had answered at my questions before your this article?

You brought it up on the list in February; only one person supported 
your request, and he suggested that you take this to the Unicode 
Consortium.

There are numerous threads on this mailing list about why we should 
use the Unicode Consortium tables, even if we we think there are some 
problems with them.

>discussed this issue in depth.

Have you participated in any other IETF Working Groups? If so, how 
did they measure consensus? If not, do you think it is appropriate 
for every working group to spend time discussing this? If you have 
not done so already, you should probably read 
<http://www.ietf.org/tao.html> and the other documents describing how 
the IETF works.

>"NFKC(UAX21_casefold(NFC(x))) ==  NFKC(UAX21_casefold(x)) is not guaranteed."

Quite true. I think the following simpler statement is also true:
   "UAX21_casefold(NFC(x))) ==  UAX21_casefold(x)) is not guaranteed."

>than before in the IDN samples including <I><dot-above>.

True, but irrelevant. Nameprep is called from IDNA. IDNA does not 
say, and has never said, "you can do NFC before processing the name". 
The fact that you might have done some processing on a name *before 
you processed IDNA*, and that pre-processing may cause you some 
surprises, is not an IDNA problem.

--Paul Hoffman, Director

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 5 May 2002 04:30:59 +0900

----- Original Message ----- 
> surprises, is not an IDNA problem.
 
<I dotabove>==<I><dotabove> : these two represent the same character.
Likewise, <A acute>==<A><acute>: so do these two .
But, for the latter case, stringprep treat them as equal ones,but
for the former case, it doesn't. it is inconsistent, and moreover
errornous,because LOWERCASE(<I><dotabove>) != <i><dotabove>.

IDNA inherits the error of UAX21. Therefore , this is  IDNA problem,
because IDNA adopted the problematic UAX21 without patches, even though
IETF is not directly responsible for the specific error in UAX21.

---

From: "Soobok Lee" <[email-masked]>
Date: Sun, 5 May 2002 12:42:05 +0900

----- Original Message ----- 
 ?
> use the Unicode Consortium tables, even if we we think there are some 
 
UTC casefolding (UAX21) is made for char-by-char casefolding,  not for
combining sequences, but stringprep blindly applies UAX21 into them.
That is not the problem of UAX21, rather  of the stringprep.

NFCing before casefolding solves this problem, but this suggestion
was also ignored or not discussed in depth.

Without any modificationa to UAX21 and NFKC and NFC, we could cure
this <I><dot above> stringprep errors, simply by adding NFC in
step zero in stringprep.

Soobok Lee

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 6 May 2002 22:40:00 +0200 (CEST)

> Stringprep should have contained warnings about this or included new revised
> casefolding operatins/tables, but it didn't . the authors and co-chairs have
> ignored this repeated warnings  without any responses in this list. This is
> not a theoretical issue,  but a practical issue because the sequences are
> used by modern east-europeans.

What type of warning would make sense in stringprep?

Is there some warning that makes sense in IDNA as well?

Do you have suggested text that folks can review?

  Erik