Subject: [idn] Re: Fwd: Unicode letter ballot

From: Kenneth Whistler <[email-masked]>
Date: Wed, 27 Nov 2002 17:04:27 -0800 (PST)

Simon Josefsson said:

> > sigh.... all 5 are beyond-BMP characters added recently.
> > if we could go back in time, we could have implemented a policy of not
> > accepting characters as stable before 2 unicode versions had gone
> > by....
> >
> > proofreading takes time.
> 
> Even two Unicode releases doesn't guarantee that the tables are
> correct.  

Of course. But in this case, because of the sensitivity of this
problem, 5 independent audits of the Plane 2 CJK compatibility
characters by CJK experts have converged on the same answer.
There are 5 mistakes (4 clerical and one visual) in the current
mappings for that set of recently added 542 supplementary characters
on Plane 2.

[There are also other known issues in the mappings, whereby
one Han variant or another might be a "better" mapping, but there
is also consensus among the experts that none of those issues
rise to the level of blatant *mistakes* that must be corrected in
the current tables -- which is why none of those is being
ballotted or will be in the future.]

The U.S. national body and the UTC are both also pushing
back hard on the proposed addition of another 122 CJK compatibility
characters because the CJK mapping experts have discovered errors
in *that* table as well. In this case, having learned our lesson,
the UTC is trying to be proactive and ensure that all errors are
removed from the table *before* such an addition is standardized.

> The only proper solution I can see is to stop modifying
> published decomposition tables.  When mistakes are discovered, new
> character codes with proper decompositions should be added and the old
> character codes declared obsolete -- which is option B in the vote,

This will lead to other interoperability problems. The 542
supplementary characters in question (and all of the ones involving
the errors) are CNS compatibility characters. They are there to
provide round-trip mappings to the CNS 11643 standard. If you
"obsolete" 5 code points and then add 5 new ones, then it is
inevitable that CNS mapping tables will get updated to use the
new code points instead of the old ones (and there will be some
inconsistency in the mappings, because of the duplications, during
this transition) -- because the old code points get normalized
away to nonsense characters. This will undoubtedly lead to further
problems, including for IDNA string matching, as one of the
duplicated pair normalizes one way, and the other -- apparently
identical -- normalizes another way. And you can't escape the
problem by just adding the 5 obsolete code points to the
stringprep prohibited list, because that, *too*, would have
destabilized your specification: a string that was valid before
you did that would be invalid after you did so.

> but unfortunately neither IDN nor IETF has any voting powers (which
> suggest a methodological problem).

Why would IDN have voting powers here? You don't expect the
UTC or SC2/WG2 to have voting powers in an IDN working group, do you?

As for the IETF, the UTC and the IETF have a *liaison* relationship.
The UTC immediately informed the IETF liaison about this ballot, because it
knew this was an important issue that IETF participants are
concerned about. That is why this discussion has migrated over
to interested parties on the IDN list who have worked on IDNA
and stringprep.

But the buck has to stop somewhere. Ultimately the UTC and WG2
are responsible for the CJK compatibility character mapping
tables. So those committees have to take the relevant votes,
and if they end up standardizing errors, also have to take
the relevant knocks when they go to fix the errors.

To influence the actual *voting* on this (or other issues),
one works through the UTC voting member representatives in
the UTC case, or one works through the national bodies
participating in SC2 in the SC2/WG2 case.

--Ken

---

From: Simon Josefsson <[email-masked]>
Date: Thu, 28 Nov 2002 03:43:07 +0100

Kenneth Whistler <[email-masked]> writes:

> away to nonsense characters.

Assuming CNS 11643 discusses decomposition, you will have this problem
with either option A or B anyway.

Naïvely the conclusion then is that the entire 542 character block was
in error, and should be replaced by a new 542 character block that
provide the correct round-trip mapping with CNS 11643.

> This will undoubtedly lead to further problems, including for IDNA
> string matching, as one of the duplicated pair normalizes one way,
> and the other -- apparently identical -- normalizes another way. And
> you can't escape the problem by just adding the 5 obsolete code
> points to the stringprep prohibited list, because that, *too*, would
> have destabilized your specification: a string that was valid before

It is not the stability I'm worried about here, but security issues.
I'm not convinced the approach you propose here have the same security
problems as having one Unicode string translate into different bit
strings dependening on which Unicode version stringprep uses.  In
fact, I believe your proposed solution guarantees security at the
expense of breaking backwards compatibility, which sounds like good
engineering to me.

> participating in SC2 in the SC2/WG2 case.

My point was that if IDN wanted to influence the character set, and
decomposition tables in particular, it should not have referenced UTC
standards; it was a remark targetted against IDN, not UTC.  But that
is an entirely different discussion.  UTC and SC2/WG2 do have the same
voting powers as all other IDN WG participants have, btw.

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 28 Nov 2002 03:12:04 +0000

Kenneth Whistler <[email-masked]> wrote:

> And you can't escape the problem by just adding the 5 obsolete code
> points to the stringprep prohibited list,

True, but...

> because that, *too*, would have destabilized your specification: a
> string that was valid before you did that would be invalid after you
> did so.

Actually, adding compatibility characters to the Stringprep prohibited
list would have absolutely no effect, because Stringprep performs
normalization before prohibition.

Let's consider the possible scenarios:

1. The decomposition mappings are changed.

  1a. Stringprep/Nameprep track the update, breaking their promise of
      backward compatibility.

      If someone registers a name using a CNS 11643 string in
      combination with the old Nameprep, and later someone tries to look
      up the name using the very same CNS 11643 string in combination
      with the new Nameprep, it won't match (if it contains any of the
      five characters in question).  As more clients upgrade to the new
      Nameprep, the name will become less and less accessible.

      But the old-Nameprepped form of the name (the one that actually
      got stored in the database) will look visibly wrong, won't it?  If
      the registrant had been shown the Nameprepped form and asked for
      confirmation, the registration would probably have been aborted.
      So maybe this lookup failure would turn out to never happen in
      practice.

      On the other hand, if someone registers a name using a CNS 11643
      string in combination with the new Nameprep, and later someone
      tries to look up the name using the very same CNS 11643 string in
      combination with the old Nameprep, it won't match.  But this time
      the Nameprepped form shown to the registrant will look correct, so
      names causing this failure might be more likely to be registered
      than names causing the previous failure.  For this failure, as
      more clients get upgraded, the name will become more and more
      accessible.

  1b. Stringprep/Nameprep do not track Unicode updates; they remain
      frozen at a version containing the old mappings.

      If someone registers a name using a CNS 11643 string, and later
      someone tries to look up the name using the very same CNS 11643
      string, it will match.  But if a modern normalization operation
      gets inserted somewhere (for the heck of it, or because some
      other protocol that carries the domain name requires text to be
      normalized), it won't match.

      Names using the broken compatibility characters might not be
      registered in practice, because the Nameprepped form will look
      wrong.

      Since Nameprep never changes again, there is no transition as
      software gets upgraded.  Names involving the five characters in
      question remain just as broken and undesirable in the far future
      as in the near future.  It is likely that these five characters
      would never be used in domain names in practice, forever.

  1c. Stringprep/Nameprep track Unicode updates, but require the use of
      NormalizationCorrections.txt to undo any changes to decompositions
      since Unicode 3.2.

      This is exactly like case 1b except that IDNA can take advantage of
      other updates to Unicode, like new characters.

      As software gets upgraded, the mappings of the five characters
      in question remain the same (broken), so it is likely that these
      five characters would never be used in domain names in practice,
      forever, same as case 1b.

2. The decomposition mappings are not changed; the characters are
   deprecated, and new characters added with the correct decompositions.

  2a. Stringprep/Nameprep eventually allow the use of a version of
      Unicode that that includes the added characters.

      If someone registers a name using a CNS 11643 string in
      combination with the old CNS-to-Unicode tables, and later someone
      tries to look up the name using the very same CNS 11643 string in
      combination with the new CNS-to-Unicode tables, it won't match.
      As more clients upgrade to the new CNS-to-Unicode tables, the name
      will become less and less accessible.

      If the registrant is shown the Nameprepped form of the name, it
      will look visibly wrong, so the name will probably not end up
      getting registered, so this lookup failure might never happen in
      practice.

      If someone registers a name using a CNS 11643 string in
      combination with the new CNS-to-Unicode tables, and later someone
      tries to look up the name using the very same CNS 11643 string in
      combination with the old CNS-to-Unicode tables, it won't match,
      but this time the Nameprepped form shown to the registrant will
      look correct, so names causing this failure might be more likely
      to be registered than names causing the previous failure.  For
      this failure, as more clients get upgraded, the name will become
      more and more accessible.

  2b. Stringprep/Nameprep freeze on a version of Unicode that does not
      include the added characters, never to be updated again.

      If someone registers a name using a CNS 11643 string in
      combination with the old CNS-to-Unicode tables, and later someone
      tries to look up the name using the very same CNS 11643 string in
      combination with the new CNS-to-Unicode tables, it won't match.
      As more clients upgrade to the new CNS-to-Unicode tables, the name
      will become less and less accessible.

      If the registrant is shown the Nameprepped form of the name, it
      will look visibly wrong, so the name will probably not end up
      getting registered, so this lookup failure might never happen in
      practice.

      If someone tries to register a name using a CNS 11643 string in
      combination with the new CNS-to-Unicode tables, it will fail,
      because the new CNS-to-Unicode tables will use the new characters,
      which are unassigned in the version of Unicode used by Nameprep,
      and therefore disallowed in stored strings.

      It is likely that these five characters would never be used in
      domain names in practice, forever.

It's interesting that cases 1a and 2a are basically identical, and cases
1b and 2b are extremely similar, although they correspond to opposite
outcomes of the vote.

The biggest difference I can see between case 1 and case 2 is that if
the deprecate/add approach is taken (case 2), IDNA's only choices are
to track Unicode updates (2a) or not track them (2b), all or nothing.
But if the fix-decompositions approach is taken (case 1), IDNA has a
third option (1c) of tracking all Unicode updates except changes to
decompositions, via the NormalizationCorrections.txt file.

I find this result to be counter-intuitive.  My initial gut feeling was
that the deprecate/add approach was more conservative, and safer.  But
apparently not.

AMC

---

From: Soobok Lee <[email-masked]>
Date: Thu, 28 Nov 2002 13:01:48 +0900

>       registered in practice, because the Nameprepped form will look

Maybe. But, i think this is still over-simplified. I will add more sub-cases:

The compatiblity form has three equivalent forms.
0) the compatiblity form itself
1) the (falsely) nameprepped form  from  Unicode 3.2 false-NFC
2) the desired-and-correct-but-abandoned equivalent form from correct-NFC

Now,
0) can't be registered as itself in any case, 
   because it is always mapped to something other.
1) may be abandoned as you described above because it renders differently
   than expected when registration time.
1) may be, however, registered as un-nameprepped form itself 
   by other applicants who may occasionally
   have this 1) name as their biz name.
2) may be registered as un-nameprepped form itself 
   by other applicants who may occasionally
   have this 2) name as their biz name.

Adam, Could you continue to take your thorough analysis on these sub-cases?

Soobok Lee

---

From: Simon Josefsson <[email-masked]>
Date: Thu, 28 Nov 2002 05:53:08 +0100

> Let's consider the possible scenarios:

This was a useful summary.

>       Nameprep, the name will become less and less accessible.

During the time frame when clients upgrade to the new stringprep
(which will be a year or more big) you will also be able to fool
security software that matches stringprep processed strings for
equality, because the security software may use different
decomposition tables than user applications.

> decompositions, via the NormalizationCorrections.txt file.

1c is a compelling option, but it essentially means forking the
Unicode NFKC tables.  Forking non-IETF standards in IETF has not been
a good idea historically.  But since NormalizationCorrections.txt will
be short and documented, and the rationale clear, I kind of like it.

> that the deprecate/add approach was more conservative, and safer.  But

I seem to disagree.  To me, case 1a seems to be a disaster as far as
security is concerned.  All other cases are better.  I prefer 2a,
followed closely by 1c.

(Note that I intentionally talk about stringprep and not nameprep, as
the same issues apply to non-IDN applications using stringprep too.
There are several security protocols considering using stringprep.)

---

From: Soobok Lee <[email-masked]>
Date: Thu, 28 Nov 2002 14:41:52 +0900

>    have this 2) name as their biz name.

I forgot to add this:

     0) and 2) strings share the same character glyphs (by correct NFC!), 
     but 1)  string does not. Then, when  2) is registerd and end users 
     enter 0) for 2), this will lead to a failure or a wrong site '1)'
     since 1) != 2).

Soobok Lee

---

From: [email-masked] (Martin v. Löwis)
Date: 28 Nov 2002 10:45:35 +0100

> It is not the stability I'm worried about here, but security issues.

Can you please elaborate on the security issues? I believe they are
irrelevant in presence of compatibility characters, since
compatibility characters are never part of a IDN label (i.e. the
ToUnicode operation never produces them).

Regards,
Martin

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 30 Nov 2002 09:02:27 +0000

Simon Josefsson <[email-masked]> wrote:

> To me, case 1a seems to be a disaster as far as security is concerned.
> All other cases are better.  I prefer 2a, followed closely by 1c.

This is very curious.  You say 2a is your favorite, and 1a is a
disaster, but to me they appear to have the very same characteristics.
Could you give a scenario that illustrates why 1a is so much worse than
2a?

It seems to me that for any security hole in 1a involving the comparison
of Unicode strings, the same security hole exists in 2a involving the
comparison of CNS 11643 strings.

(Reminder: 1a means the decomposition mappings are changed and
Stringprep tracks the update.  2a means characters are deprecated and
added, and Stringprep tracks the update.)

AMC

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 30 Nov 2002 15:36:05 +0100

> Could you give a scenario that illustrates why 1a is so much worse than

One class of scenarios are software checking ACLs, such as an
application authentication proxy (application level firewall, if you
wish).  To take the example of SASL (which doesn't have a stringprep
profile, but might get one), connecting in with administrator
credentials to an employee account on an IMAP server with SASL.

Detailed example:

Authentication identity "admin", authorization identity U+4711,
password X.  For the argument, let's say U+4711 decomposes into U+1234
in Unicode 3.2 but is later changed to U+4321.

The SASL library, acting as a proxy in front of the application
software, implements the current libstringprep correctly.  It checks
that admin's password is X and that he is authorized to log in as
U+1234 (which is the result after stringprep of U+4711, which was sent
because the client hadn't been updated to use stringprep, which should
cause no problem) and says OK to the application.

Now, in 1a the application is using updated tables from a more recent
stringprep that incorporates the fixed decomposition mapping, causing
it to admit the user to an account U+4321.  This is bad.

In 2a, the application sees that the characters are deprecated due to
its decomposition mapping changed, and rejects the user.  This is
good.

This example might seem contrived, but this kind of proxying and
username forwarding handles alot in practice if you think about it, it
does not have to be as explicit as an admin-logon.

> added, and Stringprep tracks the update.)

[I used the word "disaster" carelessly, btw. This problem only
concerns a small number of characters.  It is a problem, but it does
not have disastrous proportions.]

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 30 Nov 2002 15:40:47 +0100

[email-masked] (Martin v. Löwis) writes:

> Can you please elaborate on the security issues? 

See my message to Adam M. Costello.

> I believe they are irrelevant in presence of compatibility
> characters, since compatibility characters are never part of a IDN
> label (i.e. the ToUnicode operation never produces them).

Yes, you are right.  I was thinking of when changes like the one
proposed here applies to any character, that doesn't happen to be a
compatibility character.  Hm.  Perhaps I was unclear about that, if so
I'm sorry for my confusion.

---

From: "Adam M. Costello" <[email-masked]>
Date: Sun, 1 Dec 2002 00:13:50 +0000

Simon Josefsson <[email-masked]> wrote:

> In 2a, the application sees that the characters are deprecated due
> to its decomposition mapping changed, and rejects the user.  This is
> good.

It looks like the security hole in 1a stems from the existence of two
Unicode strings X and Y such that now Stringprep(X) != Stringprep(Y)
(so that two distinct accounts for X and Y can be created), but later
(after the update of the decomposition mappings) Stringprep(X) ==
Stringprep(Y), so the two accounts will get confused.

But I think the same phenomenon can happen with 2a.  There are CNS
11643 strings A and B such that now Stringprep(CNS11643toUnicode(A))
!= Stringprep(CNS11643toUnicode(B)) (so that two distinct accounts for
A and B can be created), but later (after the deprecation and addition
of Unicode characters, and the subsequent update of CNS11643toUnicode
to use the new Unicode characters instead of the deprecated ones)
Stringprep(CNS11643toUnicode(A)) == Stringprep(CNS11643toUnicode(B)),
so again the two accounts get confused.  No deprecated characters are
going to be seen and rejected, because no CNS 11643 characters are
deprecated, and the deprecated Unicode characters do not appear in the
new CNS11643toUnicode table.

An approach that would really avoid this pitfall would be to deprecate
these characters not only in Unicode, but also in CNS 11643 and any
other character sets that contain them, and create new characters
in all these character sets, and leave all the mappings of the old
deprecated characters unchanged in both the Unicode database and the
WhateverToUnicode tables.

AMC