Subject: Re: [idn] something a little lighter for the weekend

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 1 Mar 2005 07:02:48 +0000

Doug Ewell <[email-masked]> wrote:

> There may be 52 scripts currently encoded in Unicode, but I am sure
> Unicode does not claim that is the total number of scripts in the
> world.  Others can and will be encoded.

Michel Suignard <[email-masked]> wrote:

> The 2 current amendments of ISO/IEC 10646 (The ISO sibling of Unicode)
> being processed are adding about 10 new scripts.  And a new amendment
> will be initiated in September with few more scripts.  Any scheme
> based on a finite number of scripts is doomed.

Of course the number 52 would not be hard-coded into the policy.  It
would be expressed as "the number of Unicode scripts supported by the
latest version of IDNA".

Unicode tends to grow linearly, not exponentially, so that shouldn't
present a scaling problem.

AMC

---

From: Gervase Markham <[email-masked]>
Date: Wed, 02 Mar 2005 11:50:19 +0000

Erik van der Poel wrote:
> Here I agree with you. I'm not going to try to come up with the wording 
> for that, but this morning I started to think that the right-to-left DNS 
> and IDN spoofing problems *could* be addressed at the UI level by 
> providing a *tool* that security-conscious users could *choose* to use.

While security-conscious users are always less at risk than ordinary 
users, thinking in terms of a tool is IMO wrong.

> I'm thinking of a tool that might be implemented as an extension for 
> Mozilla, for example. It would offer to display domain names in the safe 
> order, i.e. left-to-right for users whose main language is 
> left-to-right. I have not heard of any UIs that offer top-to-bottom in 
> their menus, dialogs, etc, so I would guess that this would be omitted 
> in the extension too, though right-to-left might be offered for 
> right-to-left users (many of which are in the Middle East -- Hebrew and 
> Arabic).

The problem this is supposed to mitigate is mitigated in Firefox by the 
domain-only indicator in the status bar.

> In addition, such a tool would offer to display domain names in a clear 
> font, unlike the sans-serif that is commonly used today. This would make 
> the distinction between lowercase l and digit 1 clearer. And it would 
> separate the domain name from its context, e.g. using color.

Assuming we could determine such a font, why would we not always use it? 
Why wait for a tool to be deployed?

Gerv

---

From: Gervase Markham <[email-masked]>
Date: Wed, 02 Mar 2005 11:55:12 +0000

Erik van der Poel wrote:
> 1. Is this the right time to start working on Internet Drafts leading up 
> to new version(s) of the IDNA RFC(s)? If not, when?

IMO, no. Nothing like consensus has yet emerged. However, I feel that 
the way forward will become clear eventually - we aren't going round in 
circles. It's just a big issue.

Gerv

---

From: Gervase Markham <[email-masked]>
Date: Wed, 02 Mar 2005 12:03:55 +0000

Paul Hoffman wrote:
> Yes. It turned out to be a bad idea that got more complicated and less 
> justifiable (that is, worse) with each rev, so I let it die. Others 
> (notably JET) tried different things.

Paul,

Could you tell us more about the problems you found with the ideas of 
bundling and blocking?

Gerv

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 1 Mar 2005 16:16:07 -0800

At 12:03 PM +0000 3/2/05, Gervase Markham wrote:
>Could you tell us more about the problems you found with the ideas 
>of bundling and blocking?

It was impossible to come up with a bundling scheme that kept 
everyone happy. The needs of the Chinese language communities for 
bundling were different than the needs of the Scandinavian language 
communities, which in turn were different than the needs of the Indic 
language communities, which were different than the needs of the 
Arabic language communities, and so on. Then toss in the communities 
that truly want multiple scripts but want to avoid homograph attacks 
(yes, we really did think about that years ago...), and your brain 
starts dripping from your ears.

Other folks with more brains or who are less prone to dripping are 
welcome to try to fix this for the world, or at least for one 
community as the JET folks did.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 01 Mar 2005 17:05:29 -0800

> users, thinking in terms of a tool is IMO wrong.

Perhaps I was wrong to use the word "tool". There is a fundamental 
tension between security and user-friendliness. Some applications and 
vendors have a history of making their user interfaces *too* friendly, 
thereby neglecting to warn users of potential security risks. Other 
vendors have tried hard to strike a balance between security and 
seamlessness. I believe Netscape and Mozilla have been in this camp 
since Day One.

I hope that mozilla.org will deploy a better solution than the TLD and 
domain black/whitelists that have been discussed.

>> It would offer to display domain names in the 
>> safe order, i.e. left-to-right for users whose main language is 
>> left-to-right.
> 

I just double-checked Firefox 1.0.1, and it just says "Done" at the 
lower left. Then I tried a secure (https) site, and, lo and behold, I 
saw the "domain-only" indicator at the lower right, next to the padlock 
icon. This is very good news (to me). And thank you for educating this 
particular user (me) about this security issue. As I have often said, 
education is key.

A couple of questions/comments: It might be nice to have this 
domain-only display even for non-secure sites (http). Also, do you know 
what happens if the domain name is very long? Finally, do you have any 
thoughts about the slash homograph problem? Thanks.

>> In addition, such a tool would offer to display domain names in a 
>> clear font, unlike the sans-serif that is commonly used today. This 
>> would make the distinction between lowercase l and digit 1 clearer. 
> 

Indeed, why wait? I filed a bug a while ago:

https://bugzilla.mozilla.org/show_bug.cgi?id=282079

My feeling is that a sans-serif font (such as Arial) places the 
characters too close to each other and does not have the serifs that 
often serve to distinguish the characters better. How about a fixed 
width font with serifs, such as Courier New?

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 01 Mar 2005 17:48:48 -0800

Paul Hoffman wrote:
> It was impossible to come up with a bundling scheme that kept everyone 
> happy. The needs of the Chinese language communities for bundling were 
> different than the needs of the Scandinavian language communities, which 
> in turn were different than the needs of the Indic language communities, 
> which were different than the needs of the Arabic language communities, 
> and so on. Then toss in the communities that truly want multiple scripts 
> but want to avoid homograph attacks (yes, we really did think about that 
> years ago...), and your brain starts dripping from your ears.

Yes, as a long-time internationalization engineer, I can imagine that it 
was difficult to come up with a single set of guidelines for all of the 
world's registries. (In addition to language differences, some comments 
on this list have led me to believe that there are also protocol 
differences between the registries, i.e. VeriSign's multiple versions of 
RRP vs the EPP that Edmon Chung seems to have been working on vs fax and 
sneaker net vs any others?)

However, I note that this particular conversation is between a browser 
developer (Gervase) and one of the IDNA authors (Paul), neither of which 
is a registry representative, so why exactly are you 2 having this 
conversation? :-)

Sorry, I'm half joking. Half, because you two have every right to 
discuss whatever you wish. The other half because I believe browser 
developers can afford to focus more on their end of things. Allow me to 
insert an excerpt from a previous email I wrote up:

-----------------

It is pretty clear that none of the organizations can completely solve
the problem on its own. Unicode can warn about these issues, but that is
all they can do. They cannot remove characters. The IETF is currently
discussing the prohibition of certain characters or character types.
Even if the IETF publishes updated versions of the specs, there will
still be the problem of certain characters being unfamiliar to many
users (simply because they do not know all the legitimate characters in
the world), thereby leaving them exposed to the phishers. The registries
can enforce rules at their level, but nobody has yet shown that they can
truly enforce any rules at other levels. So, the browser developers must
address that problem.

There are several issues here. One is that domain names are typically
displayed inside something else, e.g. a URI. This, in itself, gives the
phishers something to work with. So the browser developers must think
about other ways to display domain names. This is not very easy. People
exchange URIs via email and other means all the time. Apps turn those
URIs into clickable links, as a service to users. If not, they can copy
and paste the URI into the URI field. Both of these methods could be
improved to highlight the domain name in the interests of security.

Another problem is that humans are only familiar with a small set of
characters. Some humans know *many* characters (i.e. the East Asians),
but most know a lot less than that. Now, within the set of characters
that each user is familiar with, there are no homograph problems (or
just a few). However, as soon as you stray outside any single user's
familiar set, there are many homographs, near-homographs and unfamiliar
symbols. When a typical computer user is faced with something
unfamiliar, they are quite likely to shrug it off and assume it's just
one of those "computer" things that they cannot understand. This is
something that IDN phishers could take advantage of, if the browsers do
not take steps to highlight the unfamiliar characters (via HTTP
Accept-Language and browser localization as I suggested). Of course,
highlighting is not sufficient. Education is also very important.

So, instead of wasting time talking about a non-solution (white/black
lists), it would be nice to see these parties spending their valuable
time on real solutions. The registries could be working on the
guidelines, to address the concerns about language tagging, variants and
so on. They could also get in touch with the IETF, to let them know
which Unicode characters and character types they wish to use, so that
the IETF can consider how to publish new specs that might prohibit other
characters. Browser developers could start working on ways to display
domain names in ways that give the phishers less to work with.

---------------------

In other words, I do not think browser developers need to be overly 
concerned with the particular bundling/blocking schemes that the 
registries might be using. Instead, I wish the browser developers would 
focus more on the *user*, who may be "surfing" from one site to the 
next, spanning the globe, and crossing language boundaries. In order to 
protect such a user, the browser should focus on the core set of 
characters that s/he is familiar with, and provide some sort of 
indication when unfamiliar characters appear, so that the 
security-conscious, educated user may know when to be careful. I.e. the 
language of the *user* is important, not the language of the domain name.

I am *not* saying that this would be easy to implement. I am not at all 
surprised that Mozilla and Opera have chosen an easy stopgap, hopefully 
only for the interim. It's great to see Mozilla and Opera lead the way 
as they have been!

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 01 Mar 2005 20:47:24 -0800

> developers can afford to focus more on their end of things.

Sorry, I've been told that this half-joking thing was confusing, and I 
now believe I shouldn't have tried to be so cute.

All I'm trying to say to *Gervase* is that it doesn't really matter 
*what* characters are allowed to be registered in a registry, as long as 
the browser takes steps to warn the user when something phishy might be 
going on, e.g. a slash homograph, or a Cyrillic small 'a' when the user 
was probably expecting a Latin small 'a'. As I have pointed out, the 
registry does *not* have control over higher-numbered level domains. 
E.g. .de controls the 2nd level domain (2LD), but not the 3LD, 4LD and 
so on. That is where the slash homograph problem *really* matters.

> Instead, I wish the browser developers would 
> next, spanning the globe, and crossing language boundaries.

Sorry, this may not have been the best logic to use in my argument. It 
would have been better to talk about phishers, who often spam users with 
email containing URIs that *could* contain IDN labels with dangerous 
homographs at any level of the name, 2LD, 3LD, or whatever.

(Most users *don't* surf around the world, since many are monolingual or 
maybe bilingual.)

Anyway, help me out, guys and gals. Pull my logic through the wringer, 
and comb it with the finest comb you have at your disposal. This way, we 
can collectively improve our understanding of the IDN phishing problem 
and ways to address it.

Erik

---

From: Gervase Markham <[email-masked]>
Date: Wed, 02 Mar 2005 08:56:05 +0000

Erik van der Poel wrote:
> tension between security and user-friendliness. 

Well, maybe. I'm not convinced the tension is absolute, but I agree you 
need to work very hard indeed to get both.

> domain-only display even for non-secure sites (http).

We are probably going to change this for 1.1. It takes some careful 
thought so as not to confuse people.

> Also, do you know 
> what happens if the domain name is very long? 

It just gets very long, currently.

> Finally, do you have any 

Well, the current domain indicator will show the domain, slash 
homographs and all. We're still developing our response, but it's likely 
that we'll have to blacklist this character. Opera's new beta already 
has a small set of characters it doesn't allow.

Ideally, we wouldn't be acting unilaterally on this one, and would be 
doing the restrictions based on consensus. But before we can go there, 
we need to figure out what we think is needed first. That process is 
still going on.

> https://bugzilla.mozilla.org/show_bug.cgi?id=282079

Thanks :-)

> width font with serifs, such as Courier New?

The issue, of course, is that the font designation we use has to produce 
a good font on all platforms. This isn't fundamentally impossible, it 
just requires work and testing.

Gerv

---

From: John C Klensin <[email-masked]>
Date: Wed, 02 Mar 2005 10:43:27 -0500

Erik,

A few observations...

	(1) First, a registry does have the right to require
	that registrants observe particular rules and conditions
	in subdomains they delegate and to pass those rules down
	the tree.  Whether that is wise or sensible is another
	issue, and enforceability is yet another question.
	But, unless national law prevents it, RFC 1591, to which
	all TLD registries more or less agreed, rather
	explicitly provides for passing the responsibilities to
	the community down the tree.  Even ignoring troublesome
	concepts like "require" and "enforce", certainly nothing
	prevents registries from educating and persuading
	registrants about how they should behave.
	
	(2) In my regular role as a luser, I really like fast,
	easily-used, small-footprint browsers.  I'm more
	security-conscious and suspicious than the user average,
	and therefore also like handy tools to help me dissect
	and verify things that might look suspicious.  Tying up
	a browser with heuristics, such as mixed-script
	detectors, that may not work well and have a large
	footprint, doesn't impress me as a good tradeoff.  For
	better or worse, the assumption of a decade ago that
	most criminals, especially most electronic criminals,
	were stupid is no longer applicable, if ever it was.
	That implies, I think, that if we design a simple test
	that blocks some look-alike cases but permits other,
	more subtle, ones, we will simply drive the phishers to
	better understand and use the subtle stuff: not a good
	tradeoff.

	(3) As far as surfing around the world is concerned,
	we've got a situation today in which the domain name
	associated with a particular URL does not really predict
	the content to be found on that page.  That will
	undoubtedly get worse, as more folks discover that the
	intersection of domain and host administration with web
	site organization often makes it much easier to maintain
	versions of pages in multiple languages in the same,
	rather than different, DNS trees.  So, since I don't
	read Chinese, I'm unlikely to frequently seek out pages
	whose content is in Chinese.  But I frequently find
	pages I can read via URLs that contain elements written
	in pinyin.  I fully expect those elements, and some of
	the subdomain names, will shift to Chinese characters as
	IDNs and IRIs are more widely available.   I also expect
	that transition will make things more comfortable for
	someone who reads Chinese and would prefer to not deal
	with Latin characters and harder for me, but that is a
	reasonable tradeoff over which none of us will have much
	influence.
	
	(4) We need to get unstuck from thinking about this
	purely as a browser problem.   The usual phishing attack
	involves an email message containing a link.  For those
	email clients that don't immediately invoke a full
	browser as soon as a link appears --and many of those
	links occur in plain-text, not HTML, email-- they are
	invoking the browser when the link is clicked on.  The
	situation in the browser is then different, since none
	of the "hover over link", "look at status bar", etc.,
	tools are going to apply, or, at least, are not going to
	work in the ways that some of these discussions suggest
	for links that appear on web pages that are already open
	in the browser.  Now, we have given MUA writers no
	advice about what they should pass to the browser if
	they see an IRI or otherwise-encoded string that
	contains an IDN.  If they pass the IRI/
	native-script-form IDN, they risk passing it to a
	browser version that doesn't have a clue.  So maybe they
	force the thing into URI/ punycode form and pass that.
	Now, do you really want the browser to look at the
	thing, perform ToUnicode on the name (which, of course,
	may yield something other than what the user saw),
	perform some tests, and then pop up a "you just passed
	me an IDN that looks suspicious, do you really want to
	open that page?" box.  I think probably not.   Moreover,
	I think that, if you do, there would quickly be a
	sufficient number of false positives (positive for bad
	stuff) to get users really used to clicking "yes"
	without thinking... and cursing the browser implementer
	for bothering them with a pointless warning.

So my conclusion is that we need a mixed
protocol-registry-browser strategy.  That strategy, IMO, should
shifted the processing burdens as much as possible to the first
two.  And I think that notions that the problem can or should be
solved in any of those three places alone are probably misguided.

     john

--On Tuesday, 01 March, 2005 20:47 -0800 Erik van der Poel
<[email-masked]> wrote:

>> However, I note that this particular conversation is between
>> a browser  developer (Gervase) and one of the IDNA authors
>> (Paul), neither of which  is a registry representative, so
>> why exactly are you 2 having this  conversation? :-)
>> 
>> Sorry, I'm half joking. Half, because you two have every
>> right to  discuss whatever you wish. The other half because I
>> believe browser  developers can afford to focus more on their
>> end of things.
> 
> Sorry, I've been told that this half-joking thing was
> confusing, and I now believe I shouldn't have tried to be so
> cute.
> 
> All I'm trying to say to *Gervase* is that it doesn't really
> matter *what* characters are allowed to be registered in a
> registry, as long as the browser takes steps to warn the user
> when something phishy might be going on, e.g. a slash
> homograph, or a Cyrillic small 'a' when the user was probably
> expecting a Latin small 'a'. As I have pointed out, the
> registry does *not* have control over higher-numbered level
> domains. E.g. .de controls the 2nd level domain (2LD), but not
> the 3LD, 4LD and so on. That is where the slash homograph
> problem *really* matters.
> 
>> focus more on the *user*, who may be "surfing" from one site
>> to the  next, spanning the globe, and crossing language
>> boundaries.
> 
> Sorry, this may not have been the best logic to use in my
> argument. It would have been better to talk about phishers,
> who often spam users with email containing URIs that *could*
> contain IDN labels with dangerous homographs at any level of
> the name, 2LD, 3LD, or whatever.
> 
> (Most users *don't* surf around the world, since many are
> monolingual or maybe bilingual.)
> 
> Anyway, help me out, guys and gals. Pull my logic through the
> wringer, and comb it with the finest comb you have at your
> disposal. This way, we can collectively improve our
> understanding of the IDN phishing problem and ways to address
> it.
> 
> Erik

---

From: Cary Karp <[email-masked]>
Date: Wed, 2 Mar 2005 17:06:44 +0100 (CET)

Quoting Erik van der Poel:

> I note that this particular conversation is between a browser 
> developer (Gervase) and one of the IDNA authors (Paul), neither of 
> which is a registry representative, so why exactly are you 2 
> having this conversation? :-)
>
> discuss whatever you wish. The other half because I believe 
> browser developers can afford to focus more on their end of 
> things.

Under the assumption that the discussion might be furthered by a 
registry rep describing that end of things --

I'm responsible for the development and maintenance of the policies 
for .museum. This is a sponsored gTLD (sTLD), which means that there 
are eligibility requirements for name holders, and restrictions on 
the way names may be structured and used. The policies specific to 
IDN are stated in detail at http://about.museum/idn/idnpolicy.html, 
which includes links to both the general policy statement and the 
listing of permitted characters and code points. Although 
prospective name holders are unlikely to read the fine print, the 
detailed statement provides an unequivocal reference in the 
case-to-case discussions about policy instantiation that we 
frequently conduct with individual museums. That dialogue provides 
an effective barrier to deliberate abuse of the .museum namespace 
(which we are contractually obligated to prevent) and is also a 
safeguard against inadvertent homographic confusion.

There are significant differences in the operation of an 
unrestricted gTLD (uTLD), where there may be a far greater volume of 
registration traffic, and direct contact between the registry and 
registrant is not a part of the registration process. In this 
situation, such things as restriction on IDN registration need to be 
fully automated, with sentient review being a remedial device when 
it is noted that something the algorithmic filter is intended to 
stop has nonetheless passed through.

The gTLD registries are as concerned by and with the present 
homograph alert as is any other group attempting to curb the risk 
for such damage. In addition to the range of operating conditions 
implied above, there are differences in the way the registries have 
interpreted and implemented ICANN's IDN Guidelines. That document 
states, "As the deployment of IDNs proceeds, ICANN and the IDN 
registries will review these Guidelines at regular intervals, and 
revise them as necessary based on experience." We are now clearly at 
such a juncture, and the revision process is already being 
initiated. Reducing the latitude for interpretation of the 
Guidelines will bring the registries one important step closer to 
being able to establish a best practice that can bring an end to the 
concern about point-of-registration control that is currently being 
expressed.

The development of these best practices would be further abetted by 
the candidate IDN repertoire being purged of the graphic symbols and 
other signs that are not needed in a naming system (using that term 
in a sense that a linguist might regard as reasonable) but clearly 
exacerbate the risk for malicious exploitation. There is similar 
utility in freeing IDNA from its current lock to Unicode 3.2. Since 
both of these concerns can only be addressed through revision of 
nameprep and stringprep, that action is of priority concern to the 
registries.

There is more to be said about gTLD perspectives on the role IDN 
plays in the communities that we serve, but I'll save that for 
separate communication.

/Cary

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Wed, 02 Mar 2005 17:37:51 +0100

On 16:43 02/03/2005, John C Klensin said:
>I fully expect those elements, and some of the subdomain names, will shift 
>to Chinese characters as IDNs and IRIs are more widely available.

The problem is the two necessary keyboards.
Please stop considering the solution to your own problems. Or let turn the 
Internet entirely in Chinese by default, and let help the IETF to document 
the best way to access pages written in English.
jfc

---

From: Erik van der Poel <[email-masked]>
Date: Wed, 02 Mar 2005 09:18:49 -0800

Gervase Markham wrote:
> Erik van der Poel wrote:
> > 
> > Also, do you know what happens if the domain name is very long? 
> 

I hope you will think about what to do when the name is really long, 
keeping in mind that the most important part of the name is the end (the 
right side).

> we need to figure out what we think is needed first. That process is 

It's really great to hear this! I fully agree with you that consensus 
would be ideal. I'm taking the liberty of Cc'ing one of the Opera 
developers, in the hope that they might be willing to discuss their 
particular choices for the character blacklist. If various people can 
start to agree on a blacklist, this info can be fed into any RFC 
revision process (which I hope will begin).

> a good font on all platforms.

Yes, a good font on *all* platforms, and also for *all* parts of the 
world. I am quite aware of those issues. See the bottom of:

http://www.mozilla.org/projects/intl/fonts.html

One of the great things about open source is that lots of people from 
all over the world come to help you, with localizations, default font 
choices, etc.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Wed, 02 Mar 2005 11:02:55 -0800

John,

> solved in any of those three places alone are probably misguided.

I strongly agree with everything you said. I am sorry if I gave the 
impression that the browser implementors are the *only* people that can 
and should address this IDN phishing problem. I don't think I said that, 
but maybe I'm just not very good at email and/or expressing myself. I 
also agree that the burden should be shifted as much as possible to the 
first two. It would be bad if the many implementations all did it 
differently.

I would only add one component to your strategy. It should be a 
Unicode-protocol-registry-browser strategy. Unicode has already started 
working on their part:

http://www.unicode.org/reports/tr36/

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Wed, 02 Mar 2005 11:57:54 -0800

Cary Karp wrote:
> In addition to the range of operating conditions implied above, there 
> are differences in the way the registries have interpreted and 
> implemented ICANN's IDN Guidelines. That document states, "As the 
> deployment of IDNs proceeds, ICANN and the IDN registries will review 
> these Guidelines at regular intervals, and revise them as necessary 
> based on experience." We are now clearly at such a juncture, and the 
> revision process is already being initiated.

Hi Cary,

It is really nice to hear from you. I have an idea for the Guidelines. 
As Paul has indicated, the various communities around the world have 
different needs, and some have already started writing down the rules 
that they are following in their registries. The JET community comes to 
mind:

http://www.ietf.org/rfc/rfc3743.txt

Other communities have other needs. I've been told that some communities 
use a set of letters that are currently encoded in two different ranges 
of the Unicode space (e.g. Latin and Cyrillic). Today, my idea is that 
these communities can "occupy" their "own" part of the DNS space, for 
example a .tld or a .2ld.tld. They can publish the rules that they 
enforce in their registries, and then the browsers can either allow any 
character sequence in those labels or check them to see if the rules 
were indeed followed.

Of course, it is much harder to come up with and enforce rules in a 
"global" TLD like .com. As a result, the browsers may simply blacklist 
.com in its entirety. Or maybe .com will eventually figure out some 
rules and actually enforce them in the 2LDs, so that the browsers don't 
have to check the 2LDs. Indeed, in a perfect world, .com would even 
enforce rules in 3LDs, 4LDs, etc, so that browsers would not have to 
check those either. We shall see what .com does.

But in the meantime, how about my idea for the Guidelines?

Erik

---

From: Gervase Markham <[email-masked]>
Date: Wed, 02 Mar 2005 23:46:58 +0000

> solved in any of those three places alone are probably misguided.

Absolutely. A strategy which involves changes all the way down the line 
is going to have the best chance of quashing the problem.

Gerv

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 3 Mar 2005 06:32:58 +0000

Erik van der Poel <[email-masked]> wrote:

> Other communities have other needs. I've been told that some
> communities use a set of letters that are currently encoded in two
> different ranges of the Unicode space (e.g. Latin and Cyrillic).
> Today, my idea is that these communities can "occupy" their "own" part
> of the DNS space, for example a .tld or a .2ld.tld.  They can publish
> the rules that they enforce in their registries, and then the browsers
> can either allow any character sequence in those labels or check them
> to see if the rules were indeed followed.

I've also thought along these lines, but I rejected this approach.  The
domain hierarchy is ultimately based on delegation of naming authority,
and trying to use it for any other purpose will run into conflicting
constraints.  Suppose country X wants to support language Y, which is
used in many countries around the world.  Who would be the registry for
the Y domain, and how would you get worldwide agreement on that?  Would
country X be delegated a subdomain of Y?  Would registrants accept X.Y
as a legitimate country X domain, or would they demand to be in an X
top-level domain?  Would users of language Y not get annoyed at seeing Y
at the end of almost every domain name they use?  It's bad enough that
so many domains end in .com, imagine if they all ended in .com.lat (for
"Latin").

I still like the idea of allowing every TLD to have one synonym-TLD per
script, although we might need to recognize some scripts in addition to
the Unicode scripts, for example, the subset-of-(Latin-plus-Cyrillic)
script that you allude to.

AMC

---

From: "YAO Jiankang" <[email-masked]>
Date: Thu, 3 Mar 2005 15:13:00 +0800

----- Original Message ----- 
From: "Adam M. Costello" <[email-masked]>
To: <[email-masked]>
> script that you allude to.

   I appreciate this idea of  allowing every TLD to have one synonym-TLD per
 script  in general. however, in China, Simple Chinese Characters and Traditonal Chinese Characters are equality. So the TLDs which have both Simple Chinese Characters and Traditonal Chinese Characters should be authorized to the same organization.

YAO Jiankang

---

From: William Tan <[email-masked]>
Date: Thu, 03 Mar 2005 17:32:31 +1100

> of the DNS space, for example a .tld or a .2ld.tld. 

If by "community" you mean users of a certain language / culture group 
within a geographic region, yes. In fact, they already do. The Japanese 
already "occupy" .jp, Korean .kr, Chinese in PRC .cn, Chinese in Taiwan 
.tw, Chinese in Singapore .sg, etc.

I'm not sure what you are proposing here - are you saying to allocate 
new TLDs for each "community"?

> They can publish the rules that they enforce in their registries, ...

They already do. The rules are just not in a machine readable format, 
and John has already made the case against standardizing language 
tables, let alone other rules that may not be character-based (imagine 
the .th registry saying, allow both Thai and Latin digits, but not both 
in the same label/domain).

> and then the browsers can either allow any character sequence in those 
> labels or check them to see if the rules were indeed followed.

I'd vote against browsers trying to enforce rules set by various 
registries. This is the sort of thing you'd build into a specialized 
tool (as you mentioned) but not in a general application.

> "global" TLD like .com.

Don't forget countries who choose to honour multiple cultures within 
their society - .PL allows many different tables, including Cyrillic 
(but does not allow mixing of Cyrillic and Latin scripts, see Andrzej 
Bartosiewicz's draft.

> As a result, the browsers may simply blacklist .com in its entirety.

It looks like a reasonable interim solution, but I'm worried about 
whether .com can actually get off the list. Unlike DNSBL, if the list is 
hardcoded or statically included in the installation package, it's going 
to be difficult to get off that list.

Come to think of it, as an off-IETF solution, maintaining an IDNBL of 
sorts might be a good idea. I know, it didn't really work for mail 
abuses, the landscape is quite different for the problem at hand though. 
The IDNBL can ban an entire zone based on the reasoning that the zone 
administrator has shown to be negligent (".com"), or ban individual 
domains of known phishers, and can even be used to implement character 
blacklists instead of having them hard wired in the browser.

> Or maybe .com will eventually figure out some rules and actually 
> enforce them in the 2LDs, so that the browsers don't have to check the 
> 2LDs. 

I'm hopeful that this will happen.

> Indeed, in a perfect world, .com would even enforce rules in 3LDs, 
> 4LDs, etc, so that browsers would not have to check those either. 

It's not enforceable in 3LD and beyond. period.

wil.

---

From: Cary Karp <[email-masked]>
Date: Thu, 3 Mar 2005 10:03:34 +0100 (CET)

Quoting Erik van der Poel:

> I've been told that some communities use a set of letters that are 
> currently encode in two different ranges of the Unicode space 
> (e.g. Latin and Cyrillic). Today, my idea is that these 
> communities can "occupy" their "own" part of the DNS space, for 
> example a .tld or a .2ld.tld.

The community occupying 2ld.tld doesn't write the rules that 
determine the character repertoire available for use in .tld, and 
can therefore not necessarily represent even its own name as it 
might ideally prefer. The 2ld.tld folks do get to make the 
corresponding decision for 3ld.2ld.tld (if permitted in .tld 
policy). In the reasonably commonplace situation where all 
subdomains under 2ld.tld are operated by a single entity, coherent 
rules can be applied throughout. This situation is, however, by no 
means the only one that pertains, and it certainly does not apply to 
the point of delegation under a TLD.

Most people would probably use the term 'language' to designate the 
attribute of community identity that is expressed by its use of a 
certain set of letters. A community wishing to project that identity 
into the domain namespace will therefore need either to locate a 
parent domain that accepts the registration of names including the 
needed characters, or convince what would otherwise be the most 
desirable parent to implement that support. Languages are, however, 
frequently shared by numerous communities without any other aspect 
of shared identity, and identical sets of letters often appear in 
more than one language. (This is one of the reasons why gTLDs so 
prominently appear in the present discussion.)

> I have an idea for the Guidelines. As Paul has indicated, the 
> various communities around the world have different needs, and 
> some have already started writing down the rules that they are 
> following in their registries. The JET community comes to mind:

Both the JET and the ICANN guidelines are intended to assist TLD 
operators in establishing safe and responsible IDN policies that 
will prove useful to the broadest number of nameholder communities. 
The JET action addresses the needs of three languages but I doubt 
that the people who use those languages perceive the slightest 
additional sense of 'JET community'. And, as an uncomfortable matter 
of fact, the extent to which the genuinely excellent JET Guidelines 
will be accepted by the target language groups remains to be 
determined.

> Of course, it is much harder to come up with and enforce rules in 
> a "global" TLD like .com

One might think the situation to be straightforward in a ccTLD, but 
there are clear current trends in ccTLD policy development toward 
removing the entry-level requirement for national nexus, and 
permitting the use of more extensive character repertoires than 
would normally be associated with the nominal TLD designation. It is 
by no means uncommon for a country to have more than one official 
language and an even larger number of officially recognized minority 
languages. It is also common for countries to belong to 
multinational alliances, with member states recognizing all of the 
languages used within that union. All this needs to be reflected in 
a ccTLD's IDN policies, which will often require every bit as artful 
juggling of a range of scripts and languages as would be needed in a 
gTLD, with a heavy further amount of political intricacy that a gTLD 
might be able to avoid.

It is true, nonetheless, that a ccTLD operator will generally be in 
a better position to produce an authoritative statement of the 
character repertoire necessary for the IDN representation of one of 
'its' languages, than would the operator of a gTLD serving the same 
language community. For this reason, many gTLDs introduce IDN 
support for a given language first subsequent to a ccTLD clearly 
associated with that language having described its requirements, or 
when a similar statement has been produced by some other obviously 
authoritative group.

Please also keep in mind that the geographic permimeters within 
which many languages are used do not coincide with national 
boundaries, and that many communities do not associate their 
language identities with the national identities of the countries in 
which they reside. IDN provides unprecedented means for such things 
as allowing a diaspora to maintain it sense of cultural cohesion, or 
furthering the cause of a group struggling to have its language 
officially recognized or attempting to reverse threats to its 
survival. In such contexts the national implication of a cc label 
may be undesired, which is another of the reasons why gTLDs so 
prominently appear in the present discussion.

> They can publish the rules that they enforce in their registries, 
> and then the browsers can either allow any character sequence in 
> those labels or check them to see if the rules were indeed 
> followed.

I am grateful that my only headache in this regard is anticipating 
the policy and technical requirements for supporting the thousand or 
so languages that some segment of the museum community may sooner or 
later express interest in representing via IDN in .museum. The same 
repertoire may also appear elsewhere in the TLD space and I 
certainly don't envy the people who intend to devise and implement 
the algorithmic underpinnings for the automation of that process or
the validation of its results :-)

/Cary

---

From: Erik van der Poel <[email-masked]>
Date: Fri, 11 Mar 2005 15:42:26 -0800

John C Klensin wrote:
> the view that, at the time, the Unicode
> classifications of characters were considered a little soft

FYI, I asked about Unicode category stability on the Unicode list, and 
received the following info:

From: "Andrew C. West" <[email-masked]>

According to my calculations, the number of characters which changed 
their General Category from one version of Unicode to the next is :

1.1.5 -> 2.0.14 = 474 (1.384%)
2.0.14 -> 2.1.2 = 1 (0.0025%)
2.1.2 -> 2.1.5 = 16 (0.0410%)
2.1.5 -> 2.1.8 = 18 (0.0462%)
2.1.8 -> 2.1.9 = 3 (0.0077%)
2.1.9 -> 3.0.0 = 85 (0.2182%)
3.0.0 -> 3.0.1 = 0 (0%)
3.0.1 -> 3.1.0 = 3 (0.0061%)
3.1.0 -> 3.2.0 = 7 (0.0074%)
3.2.0 -> 4.0.0 = 16 (0.0168%)
4.0.0 -> 4.0.1 = 1 (0.0010%)
4.0.1 -> 4.1.0 = 12 (0.0124%)

I don't know what this tells you about the stability of the UCD data though.

---

From: Erik van der Poel <[email-masked]>
Date: Sat, 12 Mar 2005 01:14:37 -0800

All,

Please do not draw any conclusions from the raw Unicode category 
stability data that I sent earlier. Ken Whistler, a Technical Director 
at the Unicode Consortium, was so kind to provide further information to 
put the data into their proper perspective. See below.

Sorry about that,

Erik

-------------------------------------------

Date: Fri, 11 Mar 2005 18:23:51 -0800 (PST)
From: Kenneth Whistler <[email-masked]>
Subject: Re: UCD stability
To: [email-masked]
Cc: [email-masked], [email-masked]

Erik,

If you are going to do things like pass these raw calculations
along to the IDN list, ostensibly as some measure of stability
of the UCD data, then you should take into consideration another
metric.

The raw number of characters changing is less reflective of
stability than considering how many *decisions* to change
a property (of one or more characters) were taken.

I intersperse some notes to Andrew West's calculated numbers
below, to help put this in context.

 > Andrew C. West wrote:
 > > According to my calculations, the number of characters which 
changed their
 > > General Category from one version of Unicode to the next is :
 > >

Many, many, changes, since 1.1.5 was developed in house,
without general public review, and since 2.0.14 (the
data version corresponding to Unicode 2.0) was the first
public release of the data files.

1 decision

2 decisions: addition of Pi/Pf subcategories, and 1 fix for 8 Tibetan
characters

1 decision: changes to converge identifier definitions

2 decisions: fix for Greek numeral signs; fix for halfwidth forms light
vertical

I'd have to dig further for this, but these were likely mostly
changes involved in nailing down normalization for Unicode 3.0.

1 decision: 3 Runic golden numbers

5 decisions: 2 fixes for Khmer signs, 1 for Tamil aytham, 1 for
Arabic end of ayah (architectural), 1 for the 3 Mongolian free
variation selectors

2 decisions: 1 fix for 12 modifier letters, 1 fix for decimal digit
alignment

1 decision: fix for ZWSP

3 decisions: 1 fix for Ethiopic digits, 1 for 2 Katakana middle dots,
1 for Yi syllable wu

 > > I don't know what this tells you about the stability of the UCD 
data though.

The significant point of instability in General Category
assignments was in establishing Unicode 2.0 data files
(now more than 8 years in the past).

There was a significant hiccup for Unicode 3.0, at the point
when it became clear that normalization stability was going
to be a major issue, and when the data was culled for
consistency under canonical and compatibility equivalence.

Since that time, the UTC has been very conservative, indeed,
in approving any General Category change for an existing
character. The types of changes have been limited to:

   A. Clarification regarding obscure characters for which
      insufficient information was available earlier.

   B. Establishment of further data consistency constraints
      (this impacted some numeric categories, and also
      explains the change for the Katakana middle dot)

   C. Implementation issues with a few format characters
      (ZWSP, Arabic end of ayah, Mongolian free variation selectors)

Since the publication of Unicode 3.0 in 2000, the only
significantly common-use characters that had any General
Category change were:

    U+0B83 TAMIL SIGN VISARGA (=aytham, Tamil data)
    U+200B ZERO WIDTH SPACE  (mostly relevant to Thai data)
    U+30FB KATAKANA MIDDLE DOT  (Japanese)

Of those 3, only U+30FB would exist in any commonly
interchanged character set other than Unicode, and
*that* change was merely to
change a punctuation subclass (gc=Pc --> gc=Po) -- and
was additionally a *reversion* to the General Category
assignment that U+30FB had in 2.1.5 and earlier.

--Ken

---

From: Erik van der Poel <[email-masked]>
Date: Sat, 12 Mar 2005 01:45:58 -0800

All,

This is probably well known to most of you, but the General Category 
Value in the Unicode Character Database and the stability of that value 
are not very relevant to IDNA, which does not depend on the Unicode 
Categories.

IDNA depends on the Unicode Normalization Form KC table, and there have 
been very few changes indeed in this table:

http://www.unicode.org/Public/UNIDATA/NormalizationCorrections.txt

Also, IDNA apps depend on tables for converting from various non-Unicode 
encodings to Unicode. This is another place where instability could 
affect lookups, potentially even in dangerous ways. Stringprep and IDNA 
already mention this issue in their Security Considerations sections.

Erik

---

From: "Martin v. Löwis" <[email-masked]>
Date: Sat, 12 Mar 2005 13:09:18 +0100

> already mention this issue in their Security Considerations sections.

I think an evaluation of whether something is potentially dangerous
should take linguistic research into account.

As a layman, I recognize that these are all compatibility characters
(and all for CJK encodings). For example, F951 is, according to my
Linux /usr/share/i18n/charmaps, part of CP 949, EUC-KR, and JOHAB.
Looking at the character tables privided by the Unicode consortium,
it seems obvious that it is correctly mapped to 964B and not 96FB
(but it could be that the font tables were explicitly designed to
show the same glyph).

According to Unicode corrigendum 3,

http://www.unicode.org/versions/corrigendum3.html

this character is only used for Korean encodings, to support
a different pronounciation (so the character was duplicated
in the Korean standard also). The corrigendum also asserts
that the character will not readily be recognized by most Korean
speakers (Chinese speakers recognize the character, but they
won't use the compatibility character).

So in practical life, it appears that this normalization change
is unlikely to occur, so it is hard to see how it could cause
danger. I believe the same is true for all the other cases where
the normalization has changed.

The real danger is to judge other people's work inadequately,
and to believe one can do better than them. The people who
created these tables are experts in their field (linguistics
and scripts), if they make mistakes, then likely because the
subject matter is so obscure that "normal people" never
encounter the issue at hand.

I'm not concerned about stability of IDN.

Regards,
Martin

---

From: John C Klensin <[email-masked]>
Date: Sat, 12 Mar 2005 11:40:26 -0500

--On Saturday, 12 March, 2005 01:14 -0800 Erik van der Poel
<[email-masked]> wrote:

> All,
> 
> Please do not draw any conclusions from the raw Unicode
> category stability data that I sent earlier. Ken Whistler, a
> Technical Director at the Unicode Consortium, was so kind to
> provide further information to put the data into their proper
> perspective. See below.
>...

Erik, Ken, and others,

The difficulty here is not, IMO, the specific numbers or
percentages.  It is an important difference in perspective.
>From the standpoint of UTC, these changes are few, minor, and
corrections to obscure errors.  That is a perfectly sensible
position.  

>From the standpoint of the IETF, or anyone else worried about a
piece of protocol that must support many applications, the
problem is a little different.  Some of the recent developments
in automatic updating tools notwithstanding, IDNA (and its
supporting tables) are designed to be embedded in and used from
clients.  Many of those clients, and the associated operating
systems, have been historically updated only when the machine in
which they run is replaced.   That argues for an extremely
conservative view of protocol design and compatibility, with
very high thresholds for justifying incompatible changes of any
sort.  From that viewpoint, the differences between 0.01%
changes and 5% changes is like measures of being partially
pregnant: perhaps helpful in some types of risk assessment, but
less so in making the next design decision.  

      john

---

From: "Martin v. Löwis" <[email-masked]>
Date: Sat, 12 Mar 2005 19:14:02 +0100

> less so in making the next design decision.  

While the facts are all true (clients only updated rarely, and
protocol stability being important), I somewhat disagree with the
conclusion. Design decisions need to take all these things into
account, but on a factual basis, not a moral one. E.g. if a protocol
change is known to potentially break existing clients, but the
number of actual clients being broken is also known to be small,
the protocol change might still be acceptable. Likewise if the
impact of "breakage" would be "small".

In the context of IDNA, I would conclude that upgrading to a
newer Unicode version in IDNA would do no significant harm, even
if it is, strictly speaking, an incompatible protocol change.
Existing clients must be considered, but in doing so, the effects
of a proposed change to such clients must also be considered -
not in a theoretical way, but trying the changes on a real
existing client. For IDNA, it seems likely that the existing
clients would not change in any user-visible way, and that
the behaviour change in artificial cases should be considered
acceptable.

That said, I also believe that upgrading to a newer Unicode
version would do little good. Additional characters are likely
of little use to the broad masses, as font support etc. is still
lacking. If certain new characters are of high interest to IDNA
users, I expect registrars to weaken the "AllowUnassigned"
setting of "false" to, say, "AllowUnassignedAsPerLatestUnicodeSpec" -
independent of what any RFC says.

Regards,
Martin

---

From: "Mark Davis" <[email-masked]>
Date: Sat, 12 Mar 2005 10:15:18 -0800

We do not make any absolute guarantees of stability for the general category
and many other properties, because a miscategorized character would cause
incorrect behavior in computers for the languages that use it. And as new,
increasingly obscure characters are added to the standard, it may take some
time to get really accurate information.

However, I do want to call attention to certain properties of the Unicode
Standard that may be relevant -- characterizing identifer and pattern syntax
characters -- which do have strict requirements on stability. There is a
draft of the newest specification on
http://www.unicode.org/reports/tr31/tr31-5.html. Programming identifiers
have a bit different requirements from IDN, but they are related enough that
the information may be useful.

The data for Pattern_Syntax and Pattern_White_Space is in
http://www.unicode.org/Public/4.1.0/ucd/PropList-4.1.0d12.txt. For XID_Start
and XID_Continue, it is in
http://www.unicode.org/Public/4.1.0/ucd/DerivedCoreProperties-4.1.0d12.txt

All of these will be finalized and released as part of Unicode 4.1.0.

‎Mark

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 12 Mar 2005 12:04:36 +0100

> http://www.unicode.org/Public/UNIDATA/NormalizationCorrections.txt

Don't forget the normalization flaw in Unicode 3.2 NFKC discussed in:

http://www.unicode.org/review/pr-29.html

Apparently the recommendation will be applied to future Unicode
versions.

PR-29 doesn't merely affect a small set of code points, but rather a
class of strings.  The special strings are all unstable under NFKC3.2.

I think PR-29 is a useful example to consider when deciding how much
trust you should place in the UTC's stability guarantees.  The UTC's
track record in this area suggest to me that the guarantee is
worthless in practice.  I haven't seen an evaluation of alternative
solutions to the PR-29 problem.  Not even signs that alternative
approaches were considered.  I would have expected both.

> already mention this issue in their Security Considerations sections.

Right.

Thanks,
Simon

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Mon, 14 Mar 2005 03:31:44 +0100

On 12:04 12/03/2005, Simon Josefsson said:
>trust you should place in the UTC's stability guarantees.

> > already mention this issue in their Security Considerations sections.

I try to understand the best way for ccTLDs to approach the variations in 
Unicode tables and a possible punycode update. I also try take into account 
the impact of the revision of BCP 047 (RFC 3066) engaged by the WG-ltru 
(which seems mainly motivated by W3C concerns).

1. the IDN Tables are in fact the list of Unicode points accepted by a 
Registry manager to form domain names in the "virtual zone" related to that 
table. This table may or may not be related to a language in the way RFC 
3066 means it. Let for example take ASCII 256, it does not include every 
French characters, but includes characters used in many European countries. 
I don't think any European ccTLD Registry Manager would object to support 
all the ASCII 256 characters and to add the missing ones which would not 
conflict - but would be faced with the problem of their ASCII equivalent 
(French "oe" as 2 characters instead of one, for example). The reason why 
is that there are/will be more and more words of European languages being 
accepted in other languages. Specially in commercial, cultural, societal 
areas which are a good source of domain name. Just let think about sport 
champions, singers, etc.

This means that the list of accepted characters would not relate to a 
determined language, while the IANA calls for its registration in 
documenting it with the language/script attributes which are those 
considered by the WG-ltru draft.

We therefore need another tag system for the IDN tables. This tag is 
attached to many management aspects, registrant support, homograph control, 
etc. It must also be clearly documented to the SLD zone managers if one 
want to push them to respect the same rules for 3+LD labels.

This could lead to a virtual zone management BCP (preferable?), or to an 
IDN Table support part in the WG-ltru Draft (my first idea, but I am less 
inclined to that now, unless we propose that a IDN Table is made of the 
addition of the language tables accepted by the TLD Registry Manager, but 
obviously IESG has not followed my suggestion of a more complete tag review 
permitting to document tag operations, like tag(eu)=sigma european language 
tags).

I think this does not create a problem for the European scripts. But I have 
no idea about the non Latin scripts?

2. the IDN Tables adopted by a ccTLD Manager (either code by code, or as 
the addition of IDN language tables) are contractual annexes to the IDN 
registration contracts. The ccTLD Manager should word in its terms and 
conditions what happens if the code supporting a character is changed, or 
if punycode is updated resulting in the same change. My understanding is 
that such a change will result in having the registered IDN to transcode in 
another ACE label.

- if this label does not exist the two versions should be registered for free.
- if the new label conflicts with an existing label, we have a problem - 
but I understand this is unlikely?
- I understand we should not have the case of a stringprep failure where 
there was none before? Or that this should then result in an update?

Then the ccTLD will have to document its new IDN Table. This means that she 
will add the new code to the table. But she will not be able to remove the 
former code since it is still used, but she should be able to mention that 
it cannot be used in requests of names - only in free CNAMEs.

Also a program should help the ccTLD Registry to parse her table to make 
sure all the listed codes are correct (last date of change).

Am I correct with this?

jfc

=====================================================
For your convenience:
Draft: 		http://www.inter-locale.com/ID/draft-ietf-ltru-registry-00.txt
Charter: 	http://ietf.org/html.charters/ltru-charter.html
gmane: 	http://dir.gmane.org/gmane.ietf.ltru
If you were Bcced for information and not familliar with the IETF process:
http://www.ietf.org/rfc/rfc2026.txt
http://www.ietf.org/rfc/rfc2418.txt
http://www.ietf.org/rfc/rfc3934.txt
http://www.ietf.org/rfc/rfc3669.txt
http://www.ietf.org/rfc/rfc3160.txt
http://www.ietf.org/internet-drafts/draft-hoffman-taobis-02.txt
=====================================================
Jon Postel (RFC 1591): "The IANA is not in the business of deciding
what is and what is not a country. The selection of the ISO 3166 list
  as a basis for country code top-level domain names was made with
  the knowledge that ISO has a procedure for determining which
entities should be and should not be on that list."
=====================================================
Brian Carpenter (RFC 1958/3.2): "If there are several ways of doing the
same thing, choose one. If a previous design, in the Internet context
or elsewhere, has successfully solved the same problem, choose the
same solution unless there is a good technical reason not to.
Duplication of the same protocol functionality should be avoided as far as
possible, without of course using this argument to reject improvements."
=====================================================
It seems that what works for countries and ISO 3166 since 1978 should
apply to languages and to ISO 693.
=====================================================

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 14 Mar 2005 07:15:04 -0800

You keep harping on that, but we really had no choice in that matter. The
definition of normalization in UAX #15 was internally inconsistent. Certain
implementations of the UAX algorithm would exhibit unacceptably aberrant
behavior, although only in a small number of degenerate cases, none of which
occurring in ordinary text. The problems are:

1. Broken Idempotency. A non-idempotent implementation by its very nature
cannot be stable, because repeated application of a non-idempotent
normalization could produce different results.The application of the
inconsistent interpretation therefore causes fundamental problems for
implementations as further outlined in PRI#29; briefly, these are
comparable to using a comparison function that isn't transitive when
sorting.

2. Broken Canonical Equivalence. The inconsistent interpretation of the
old UAX version could "normalize" some text to something that is not
canonically equivalent to the input -- it changes some text to some
completely different text.

3. Broken Canonical Order. Application of NFC[old UAX] or NFKC[old UAX]
produces output that is not only different text (not canonically
equivalent) but also not in canonical order. As a result, something
returned from a normalization function may not even pass the normalization
quick check: NFC_quick_check(NFC(string))=NO.

After carefully evaluating the nature and effects of this inconsistency
the UTC reached a decision to address these problems as follows:

The current version of UAX #15 in Unicode 4.1.0 addresses the internal
inconsistency. The changes do not affect any versions of UAX #15 prior to
Unicode 4.1.0 and therefore do not affect stringprep or IDN. No
backwards-compatibility problems will be introduced as a result of the
changes.

Stringprep and IDN rely on Unicode 3.2 version of UAX #15, which is:

http://www.unicode.org/unicode/reports/tr15/tr15-22.html

Implementations that claim conformance to Unicode 3.2 normalization may
not produce identical results in all cases, and may not produce *correct*
normalizations, because versions of UAX #15 prior to 4.1.0 have been
internally inconsistent. While normalization problems only happen in
degenerate cases, the inconsistency in the definition is significant enough
that UTC felt compelled to make the change. During deliberations, UTC did
discuss stability policies in the standard, and concluded that this
inconsistency itself is unstable; it led to demonstrably divergent
implementations, and could not stand without correction.

In addition to the new 4.1.0 version of UAX #15, the UTC decided to issue
a corrigendum which can be applied to other versions of Unicode. None of
the prior versions of the Unicode Standard or its annexes will be changed
in any way. Any implementation that claims conformance to Unicode 3.2 can
stay precisely the same. Only if an implementation claims conformance to
3.2 plus the new corrigendum, or to version 4.1.0 or later of Unicode,
would it change. So the current stringprep and IDN are not affected.

When it comes time to update stringprep to a new version of Unicode, such
as 4.1.0, there are two paths that IETF can take:

(a) simply update to the newer version, or
(b) specify a method which takes the previous algorithm and applies it to
the new Unicode data.

Option (a) sacrifices some compatibility, although (1) strings that have
already been stringprepped *once* with the old version will have the same
results under either version, and (2) the UTC does not expect any real data
to contain the degenerate cases that trigger the problem.

The UTC strongly recommends against Option (b). While it maintains
backwards compatibility It does not fix the underlying problems: two
successive applications of stringprep can still result in different
strings.

And if you look carefully at the stability requirements, you see "If a
string contains only characters from a given version of the Unicode Standard
(e.g., Unicode 3.1.1), and it is put into a normalized form in accordance
with that version of Unicode, then it will be in normalized form according
to any past or future versions of Unicode. " Which is true, even after
applying PRI #29.

It would also be interesting to me to see the level of stability that is
guaranteed by the other organizations. I know that there are W3C
Recommendations that do not maintain perfect stability. How about the IETF?
Is there a policy that any RFC that obsoletes another RFC is required to be
absolutely -- bug-for-bug -- backwards compatible?

‎Mark

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 14 Mar 2005 16:37:18 +0100

"Mark Davis" <[email-masked]> writes:

> internally inconsistent.

We seem to disagree on this.  I believe Unicode 3.2 was consistent.
Only the non-normative sections was in conflict with the normative
text.  I admit an implementation would not meet some normalization
invariants discussed in the document.  But I don't believe the
invariants were discussed as requirements on the implementation.

> absolutely -- bug-for-bug -- backwards compatible?

For the IETF, my understanding is that the policy is to make whatever
changes works best for people.  Including breaking backwards
compatibility when appropriate.  The stability guarantee place the UTC
in a different seat, though.

Thanks,
Simon

---

From: Erik van der Poel <[email-masked]>
Date: Mon, 14 Mar 2005 22:06:27 -0800

> invariants were discussed as requirements on the implementation.

I read UAX #15 and PRI #29. It's quite unfortunate that such a mistake 
was made in the spec, and that several implementations have implemented 
that mistake so faithfully. Although I would normally feel that the IETF 
should just stick with the original normalization table and rules (to 
avoid DNS lookup failures or, heaven forbid, security breaches), in this 
case, it may be wiser to adopt the new UAX #15 rules, since the 
invariants are important to IDNA also. The idempotence invariant seems 
especially important.

I feel that we are still at the very beginning of the adoption of the 
particular Unicodes affected by this mistake. Most of them are for South 
Asian languages. Hangul is much further along, but not the particular 
Unicodes that are affected here (i.e. the Jamo). More importantly, this 
mistake only affects highly unusual, malformed data. I think that if 
IDNA decides not to follow Unicode's recommendation now or in the next 
couple of years, 10 or 20 years from now we would look back in time and 
regret this decision. If there is a time to break compatibility for 
something, it is now, for this.

The Korean IDN table at IANA does not contain the Jamo that are affected 
by this mistake. (They use the precomposed syllables, rather than the 
individual pieces.) I don't know anything about IDN in South Asia, but I 
doubt that any labels have been registered with this particular type of 
malformed data.

It is interesting that, in this case, Unicode seems to have implemented 
first and written the spec later, which is the way the IETF is supposed 
to do things too. It's just unfortunate that the Unicode spec was 
transcribed incorrectly from the implementation(s). On the other hand, 
IDNA seems to have done it in the opposite order. First, the spec was 
written, and now that we have deployed some implementations, we are 
finding serious problems with punctuation marks and symbols.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 15 Mar 2005 11:04:13 -0800

It's very quiet on this mailing list...

> The idempotence invariant seems 
> especially important.

Keep in mind that IDNA requires Nameprep to be applied a 2nd time in 
order to determine whether a Punycode label may be displayed in its 
Unicode form.

> If there is a time to break compatibility for 

Note that we wouldn't actually be "breaking compatibility" since it is 
highly unlikely that anyone would have created a domain label with such 
strange combinations of characters. Also, even though there are 
implementations out there that implement UAX #15 the "wrong" way, keep 
in mind that there are also implementations that implement it the other 
way. This, in itself, is sufficient grounds for IETF to not only show 
concern, but also take action.

> On the other hand, 

Nobody has bothered to point out that this remark was unfair, so I'll 
just say it myself. IDNA is at the Proposed Standard Maturity Level, so 
we still have the opportunity to improve the specs for the Draft 
Standard and Internet Standard Maturity Levels. This is not only 
possible, it is expected.

Erik

---

From: "Martin v. Löwis" <[email-masked]>
Date: Tue, 15 Mar 2005 21:26:50 +0100

Erik van der Poel wrote:
> that mistake so faithfully.

It's also quite understandable. It is not at all obvious that the
correction is necessary; even know that I read it, and even though
I have implemented the algorithm myself (for Python), I found it very
difficult to understand the issue. Here is the problem:

In NFD, combining characters are sorted according to their combining
class, in increasing order. So you always have

starter small_combiner_A large_combiner_B next_start ...
(with A <= B)

The old text says that a combiner is blocked if it has the same
combining class, so

starter combiner_A other_combiner_B (with A==B; if starter
cannot be combined with combiner_A, then combiner_A blocks
combiner_B)

Now, the correction says that you should consider also the case

starter combiner_A combiner_B; with A > B ?!

How can that be? NFD should have sorted them so that combiner_B
comes *before* combiner_A, so it would not be blocked. Think about it.

The answer is this: This is *only* possible if combiner_B is a
starter, i.e. B==0. But if so, why could you possibly combine it
with the starter? Can you ever combine two starters? Think about it.

The answer is yes: for Hangul Jamo. They all have combining class 0,
yet they can be combined. There are also a few other characters which
have combining class 0 and still can be combined. However, it is not
at all obvious.

For the specific case of Python, it turns out that I special-cased
Hangul composition, so it won't apply the standard algorithm (of
looking for blockers); this means that all the examples in PR#29
apparently work "correctly" with Python. However, for the non-Hangul
cases, it is possible to produce the "bad" behaviour with Python 2.4.

> Unicodes that are affected here (i.e. the Jamo).

It's not that easy. When you use the old algorithm, you get normal
Hangul syllables, which would be allowed in IDNA. It's only that the
sequence *before* the normalization should not be allowed.

> More importantly, this 
> regret this decision. 

I don't think so. "We" could still change the decision in 20 years, and
not a single registration would be affected. The sequences causing the
behaviour change are *really* unusual - I don't know if software can
visually render them in a meaningful way, and I guess a native speaker
would just consider them moji-bake. So it is unlikely that anybody would
try to use them as input to IDNA in the next 20 years in a reasonable
application.

> finding serious problems with punctuation marks and symbols.

That's why IDNA is still a Proposed Standard Protocol (not even
a Draft Standard Protocol); see STD 1. It will advance to Draft
Standard if two independent and interoperable implementations
from different code bases have been developed, and sufficient
successful operational experience has been gained; see BCP 9.

It also *not* the case that it was specified first and implemented then.
All along the process, people have been implementing bits and pieces of
it, test beds have been run, and so on. You might not have been around,
but some people still remember.

Regards,
Martin

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 15 Mar 2005 13:34:43 -0800

Martin v. Löwis wrote:
>> particular Unicodes affected by this mistake. Most of them are for 
>> South Asian languages. Hangul is much further along, but not the 
>> particular Unicodes that are affected here (i.e. the Jamo).
> 

No, these strange sequences should not be disallowed. The specs should 
be corrected so that the implementations can all treat these strange 
sequences the same way.

>> More importantly, this mistake only affects highly unusual, malformed 
>> data. I think that if IDNA decides not to follow Unicode's 
>> recommendation now or in the next couple of years, 10 or 20 years from 
>> now we would look back in time and regret this decision. 
> 

If we do not correct the specs, more and more implementations will be 
created and deployed, some implementing it one way, the others the other 
way. It is hard to change something when a lot of implementations have 
been deployed. This is why we have to act now (or soon). We have to nip 
it in the bud.

Erik

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 15 Mar 2005 22:53:43 +0100

Erik van der Poel <[email-masked]> writes:

>>> More importantly, this mistake only affects highly unusual,
>>> malformed data. I think that if IDNA decides not to follow
>>> Unicode's recommendation now or in the next couple of years, 10 or
>>> 20 years from now we would look back in time and regret this
>>> decision. 

I think it may be too late to do a simple flag day and fix things.

I believe it would be useful to start thinking of the problem in terms
of a transition plan from what we have today and what we would like to
have tomorrow.  It is not clear to me exactly what we would like to
have tomorrow, so settling that would have to be part of the plan as
well.

Thanks,
Simon

---

From: "Martin v. Löwis" <[email-masked]>
Date: Tue, 15 Mar 2005 23:04:53 +0100

> been deployed. This is why we have to act now (or soon). We have to nip 

Not sure what aspect you are referring to, however, I doubt that the
majority of the implementations cares too much about what the RFC says.
So whether the RFC is changed or not in a minor aspect has little
impact on reality.

In the specific case of normalization, I think most IDNA implementations
will rely on a normalization implementation created elsewhere - few
IDNA implementations will come with their own normalization routines.
This is because normalization is difficult to implement, so you rather
reuse than reinvent. Then, when the underlying normalization routine
is changed, the IDNA implementation also changes - most likely without
the author of the IDNA implementation (let alone its user) even knowing.
In that scenario, the implementor of IDNA has little control over what
specific incarnation of normalization is used.

Regards,
Martin

---

From: "Martin v. Löwis" <[email-masked]>
Date: Tue, 15 Mar 2005 23:07:26 +0100

> have tomorrow, so settling that would have to be part of the plan as

Indeed. For IDNA, I think the current specification serves fine for
today and tomorrow, so nothing needs to change at all.

What is much more relevant is how further constraints in the registry
(beyond those imposed by IDNA) get implemented. Only when that is
sufficiently settled and deployed, considering *updates* to IDNA
should start.

Regards,
Martin

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 15 Mar 2005 16:50:56 -0800

> I think it may be too late to do a simple flag day and fix things.

Why? Do you believe that there may already be domain labels involving 
the relevant types of strange character sequences?

If we don't fix the specs and the implementations, then we may end up 
with interoperability problems in the future if some strange sequences 
start to appear or new characters are added to Unicode. PRI #29 already 
indicates that VeriSign's implementation does not need a change, while 
idnkit does. I believe Mozilla uses idnkit.

> have tomorrow, so settling that would have to be part of the plan as

There may be other things to change in the IDNA RFCs, but as far as this 
normalization problem is concerned, it is pretty clear to me that we 
need to get the RFCs to point to a newer version of UAX #15, i.e. 
tracking number 24 or higher.

http://www.unicode.org/reports/tr15/tr15-24.html

Note that this spec does not require an implementation to update the 
normalization table itself to a newer version. If we believe there are 
good reasons for IDNA to keep the old table entries that changed in 
newer versions of Unicode, then we may be able to do so. Is that right, 
Mark? I.e. could IDNA upgrade to Unicode 4.1.0 while keeping the 
individual mappings indicated below at the older Unicode version?

(Whether we would want to do this is a separate question.)

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 15 Mar 2005 17:19:38 -0800

Martin v. Löwis wrote:
>> created and deployed, some implementing it one way, the others the 
>> other way. It is hard to change something when a lot of 
>> implementations have been deployed. This is why we have to act now (or 
>> soon). We have to nip it in the bud.
> 
> Not sure what aspect you are referring to,

I was referring to the RFC that has a pointer to UAX #15, i.e. RFC 3454 
(Stringprep). The pointer should be updated to tracking number 24 or 
higher. Since this issue is hard to understand, it might also be a good 
idea to add an explanation in the new versions of Stringprep and the 
IDNA family.

> however, I doubt that the

There may be some organizations that ignore some or all of the changes 
we make to the RFCs, but others are paying attention, and I'm pretty 
sure they will follow at least some parts of the new RFCs. Mozilla and 
Opera have already indicated that they believe the punctuation mark and 
symbol problem is a big problem, and that they would/will address this 
problem whether or not the RFCs do.

> specific incarnation of normalization is used.

This specific issue is already mentioned in RFC 3490 (IDNA) in the 
Security Considerations section. Of course, it is up to the implementors 
whether they will adhere to the letter of the specs, but the IETF tries 
to achieve a "rough consensus" among the parties discussing the matter, 
and the authors try to write clear specs that foster interoperability.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 15 Mar 2005 17:53:01 -0800

> sufficiently settled and deployed, considering *updates* to IDNA

I disagree. The IETF should not wait for any of the registries to do 
anything before publishing new drafts or RFCs. The registries are not 
the only other players here. We have application developers and zone 
administrators depending on our work too.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 15 Mar 2005 13:34:43 -0800

Erik

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 16 Mar 2005 09:11:27 +0000

Erik van der Poel <[email-masked]> wrote:

> I was referring to the RFC that has a pointer to UAX #15, i.e. RFC
> 3454 (Stringprep).  The pointer should be updated to tracking number
> 24 or higher.
>

I agree.  Right now there does not exist a correct implementation of
Unicode normalization anywhere.  Some implementations are wrong because
they deviate from the spec, and other implementations are wrong because
they adhere to the spec which itself is wrong (because it violates all
the fundamental properties that anything called a canonical form is
assumed to have).  It's an unfortunate situation, but the best thing to
do now is fix the spec and encourage all implementations to converge on
the fixed spec, which is what the Unicode Consortium is doing.  Our part
in this is to update our pointer, so that it points at a correct spec
rather than a wrong spec.

> have tomorrow, so settling that would have to be part of the plan as

It's clear to me what we ought to have tomorrow: a canonical form
(that is, a function that selects a unique representative from every
equivalence class).  The Unicode Consortium is taking care of defining
that.

Do you have any ideas for a transition to that?

AMC

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 16 Mar 2005 10:38:07 +0100

> the relevant types of strange character sequences?

No, but I also don't think that should be the metric.  I believe the
RFCs are clear that Unicode 3.2, without the NFKC fix, is what is to
be used today...

> http://www.unicode.org/review/pr-29.html

...so you can't just "fix" the implementations.  If you do, they are
not complying with the RFC's.  If VeriSign implement PR-29, I believe
it doesn't conform to the RFC's.

I think the "needs change" statements on the PR29 page are misleading,
and based on UTC's subjective position on the matter.

See the second mail in the following section for my response:

http://www.gnu.org/software/libidn/manual/libidn.html#PR29-discussion

The issue should be handled in an update of the StringPrep.  If the
PR-29 fix is incorporated, there must a good transition discussion in
the document.  The problem cannot only be discussed in the realm of
IDNA, since StringPrep is used for other purposes as well.  For me,
the most important use is SASLprep, because it is used to prepare
username and passwords.  Hence, it is used in security critical
application, where the requirements are different than from IDNA.  The
discussion must be wider than for only IDNA.

Thanks,
Simon

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 16 Mar 2005 11:03:38 +0100

> rather than a wrong spec.

I'm less certain that is the best thing.  I'd like to see alternatives
explored before making such confident statement.

There appears to me be a lot of decisions made out of subjective
opinions on how normalization "should" behave, or is "assumed" to
behave.  I'd like to see things focused on what the changes actually
will mean, technically, in practice.  Or perhaps I'm simply not
clued-in on the evaluations of different approaches.

> Do you have any ideas for a transition to that?

I think reaching that goal can be done in different ways.

One way is to incorporate the PR-29 fix, declare the earlier attempt
as buggy, and re-cycle at PROPOSED.  I suspect you prefer that way?  I
am hesitant about that approach, because we have already deployed the
old RFC and it is not clear what problems there will be in mixing the
old and the new code.  Both Kerberos and SASL appears to be going to
use the old StringPrep as well, so we will be seeing security critical
infrastructure based on the old interpretation.

Another way is to carry on with the Unicode 3.2 NFKC even though it
breaks some human's assumptions on what "normalization" means in a
theoretic setting.  Only a select few weird cases (the PR-29 problem
sequences) will be affected.  Machines will cope, they compute an
algorithm, they don't care if the output meet some unstated invariant
or not.

A third way, which is what I am deploying, is to use the Unicode 3.2
NFKC together with a filter to reject the PR-29 problem sequences.
This is in line with the RFC's, it solves problems related to PR-29
problem sequences, and is simple to implement.

Thanks,
Simon

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Wed, 16 Mar 2005 13:28:22 +0100

>have tomorrow, so settling that would have to be part of the plan as

That part seems quite easy to address. We want to be PAD compatible, so the 
users have a single system. This means that we may have three systems:

Unicode to IP   tables
Unicode to DN tables
Unicode to lingual SLD to display as a lingual TLD tables

using NameScript tables listing the Unicode codes permitted throught out 
the FQMLDN to prevent homograph problems. Every label being an ACE label in 
an MLDN, ASCII in an ascii label.

The general construct being labels.table-indicator.tld. The table indicator 
being by default the table indicator of the TLD. This way existing ascii 
TLDs are label.ascii-indicator.tld (the ascii indicator is not necessary). 
The same for a Chinese TLD, chinese.chinese-indicator.tld: the chinese 
indicator is not necessary. Every label part of a MLDN is subject to a 
punycode translation and a nameprep removing the codes not compatible with 
its indicator.

This fully permits atypical domain names to be registered in using a 
no-filtered indicator, easy to underline in an application display.

This also permits table-indicator.TLD to be converted in the proper MLTLD 
display or from the MLTLD entry.

This is obvsiously fully compatible with the DNS and ask no complex special 
program, procedures, contract by Registries.  No need for lingual users to 
switch to Handles. It can even support phonetic, menu server, icon based 
vernacular names. This is also what is already in use.

This is what the initial demand was for.
jfc

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Wed, 16 Mar 2005 13:04:49 +0100

At 02:53 16/03/2005, Erik van der Poel wrote:
>anything before publishing new drafts or RFCs. The registries are not the 
>only other players here. We have application developers and zone 

Fully true. But we are in a real world. If you propose anything again 
without the support of the Registries you will have a lack of 
understanding, adherence and support. Also what you will propose will be 
less reviewed from different point of view and will have more risks to have 
its own flaws. You will not be able to tell the Registries they shared in 
the mistake they have to share in the fix. A second blunder and the 
multilingual internet will disinterest itself from the IETF and work out 
its own solutions - I think we all know. Which means a split between ASCII 
Internet and Multilingual Internet. What we all want to avoid, I hope.

The first step is to permit the Registries to operate in this still debated 
environement. I have asked responses about that and got no answer. I can 
only infer from your position and of this lack of concern, that this debate 
is purely theoretical. IETF does not rule but advises.

What are the objections (and BTW were to find described the consquences for 
a Registry) to Adam and Simon positions?
jfc

---

From: James Seng <[email-masked]>
Date: Thu, 17 Mar 2005 01:16:20 +0800

Sorry, I wasn't able to decipher what JFC is trying to say with all the 
strange terminologies.

It is an important topic and I better don't pretend to know what he is 
talking about. Can someone help to translate this to English to me?

-James Seng

On 16-Mar-05, at PM 08:28, JFC (Jefsey) Morfin wrote:

> That part seems quite easy to address. We want to be PAD compatible, 
> so the users have a single system. This means that we may have three 
> systems:
>
> Unicode to IP   tables
> Unicode to DN tables
> using NameScript tables listing the Unicode codes permitted throught 
> out the FQMLDN to prevent homograph problems. Every label being an ACE 
> label in an MLDN, ASCII in an ascii label.
>
> The general construct being labels.table-indicator.tld. The table 
> indicator being by default the table indicator of the TLD. This way 
> existing ascii TLDs are label.ascii-indicator.tld (the ascii indicator 
> is not necessary). The same for a Chinese TLD, 
> chinese.chinese-indicator.tld: the chinese indicator is not necessary. 
> Every label part of a MLDN is subject to a punycode translation and a 
> nameprep removing the codes not compatible with its indicator.
>
> This also permits table-indicator.TLD to be converted in the proper 
> MLTLD display or from the MLTLD entry.
>
> This is obvsiously fully compatible with the DNS and ask no complex 
> special program, procedures, contract by Registries.  No need for 
> lingual users to switch to Handles. It can even support phonetic, menu 
> server, icon based vernacular names. This is also what is already in 
> use.
>

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 Mar 2005 11:37:52 -0500

--On Tuesday, March 15, 2005 5:53 PM -0800 Erik van der Poel 
<[email-masked]> wrote:

> Martin v. Löwis wrote:
>> What is much more relevant is how further constraints in the
>> registry (beyond those imposed by IDNA) get implemented. Only
>> when that is sufficiently settled and deployed, considering
>> *updates* to IDNA should start.
>
> I disagree. The IETF should not wait for any of the registries
> to do anything before publishing new drafts or RFCs. The
> registries are not the only other players here. We have
> application developers and zone administrators depending on
> our work too.

Yes.

And let me add one observation.  At the moment, "the registries" 
(remember that there are around 275 of them) are going in every 
direction possible.  ICANN's guidance, based to some extent on 
the advice that IESG gave them, is so vague as to permit almost 
any interpretation, clearly doesn't work for some important 
cases (encouraging even more interpretations), and is being 
outright ignored by some ccTLDs.   If we want the registries to 
do something, we are going to need to provide some more specific 
guidance and we are going to need to document why it is a good 
idea.  The same observation applies to application writers: 
unless we actually change the standard (the algorithms or the 
tables), all we are doing is offering advice.  That advice will 
probably be taken if it is clear and well-reasoned and the 
reasoning is well- explained justified.  And, in both cases, it 
will be easier to persuade people to make changes before they 
have an installed base that has to be forced to adjust to new 
ways of doing things.  The more deployment occurs before we make 
suggestions or change things, the more likely whatever we do 
will be ignored.

   john

---

From: Erik van der Poel <[email-masked]>
Date: Wed, 16 Mar 2005 10:11:30 -0800

JFC (Jefsey) Morfin wrote:
> less reviewed from different point of view and will have more risks to 
> have its own flaws. You will not be able to tell the Registries they 
> shared in the mistake they have to share in the fix. 

I agree, and I can see why you react in this way to my email, but please 
look again at Martin's email. If we followed Martin's advice, we would 
wait quite a long time before even *considering* updates to IDNA. I 
think it's reasonable to *consider* changes and even to write drafts 
*before* the registries deploy further constraints.

Of course, we must involve the registries in the draft review process.

> The first step is to permit the Registries to operate in this still 
> debated environement.

I'm afraid I don't understand. The registry operators *are* already 
permitted to operate their registries.

> I have asked responses about that and got no 
> answer.

I find it very difficult to respond to your emails because you often 
bring up topics that I don't understand and seem somewhat unrelated, 
such as "PAD". Also, some of your emails seem to talk about a 
revolution, which is very difficult to bring about, and hence unlikely. 
I have made such proposals myself (e.g. displaying the domain name from 
left to right instead of right to left), but others have pointed out 
that such ideas are more in the theoretical realm than the practical 
one, and so I have stopped talking about them.

If you want to achieve consensus, you have to come closer to the rest of 
the group.

Of course, you may claim that this group is itself far removed from the 
real world, but I don't see much evidence of that.

> What are the objections (and BTW were to find described the consquences 
> for a Registry) to Adam and Simon positions?

It is too early to write a description of the consequences for the 
registries, since we haven't even reached a rough consensus on the 
changes to the specs.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Wed, 16 Mar 2005 10:40:31 -0800

> for a Registry) to Adam and Simon positions?

In order to reach a rough consensus on any changes we make to the specs 
to address PRI #29, we should consider the consequences for the 
registries. PRI #29 claims that only a handful of characters are 
involved, but we may wish to confirm that. It may also be a good idea to 
write a program that the registries could use to see whether any labels 
containing such character sequences might have been Nameprepped and 
registered.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Wed, 16 Mar 2005 11:22:45 -0800

> infrastructure based on the old interpretation.

Even if Stringprep is not updated to address the PRI #29 problem, we 
must at least let the other Stringprep profile authors know about the 
situation, so that they can make their own decisions for their areas. 
The IDN mailing list seems to be dedicated to IDNA issues (including 
Nameprep). Is there a Stringprep mailing list?

http://www.unicode.org/review/pr-29.html
http://www.imc.org/idn/mail-archive/maillist.html

The IANA Stringprep registry has not been updated to point to the new 
SASLprep RFC:

http://www.iana.org/assignments/stringprep-profiles
ftp://ftp.rfc-editor.org/in-notes/rfc4013.txt

I am Cc'ing the author of RFC 4013 (Kurt Zeilenga) on this email.

I am also Cc'ing the author of the expired Internet Draft of the 
Kerberos profile of Stringprep (Jeffrey Altman). Kurt and Jeffrey may 
wish to take a look at the various things I've included URIs for in this 
email.

http://josefsson.org/cgi-bin/rfcmarkup?url=http://josefsson.org/cgi-bin/viewcvs.cgi/*checkout*/libidn/doc/specifications/draft-ietf-krb-wg-utf8-profile-01.txt

Erik

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 16 Mar 2005 23:02:26 +0100

> Nameprep). Is there a Stringprep mailing list?

There was one ("ietf-stringprep"), but I believe it has been closed.

I suspect none of this is news to any author of a StringPrep profile.
RFC 4013 acknowledge the PR29 issue in the security consideration.

> http://josefsson.org/cgi-bin/rfcmarkup?url=http://josefsson.org/cgi-bin/viewcvs.cgi/*checkout*/libidn/doc/specifications/draft-ietf-krb-wg-utf8-profile-01.txt

The Kerberos WG is using SASLprep, IIUC.

Thanks,
Simon

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Thu, 17 Mar 2005 04:16:08 +0100

At 19:11 16/03/2005, Erik van der Poel wrote:
>wait quite a long time before even *considering* updates to IDNA. I think 
>it's reasonable to *consider* changes and even to write drafts *before* 
>the registries deploy further constraints.

Dear Erik,
John documented what he sees of the Registries. This is already a mess he 
documents well. What he does not see is that the users are not renewing. 
They are not interested. Except possibly - but their ccTLDs do not talk too 
much of IDNs -  in Latin countries. The issue is easy to understand you 
need two keyboards to enter the IDNs and two languages to spell them on the 
phone - except when the ascii TLD can be typed.

So there is no real need to get excited about IDNA. Phishing was 
interesting. I will write the Draft on that.

>permitted to operate their registries.

John described that. Registries understand nothing to the complex signals 
(or lack of) from the ICANN, IETF, market, etc. I am on the WG-IDN of the 
ccTLDs as a lingual organization (Eurolinc). I suppose there is one year I 
received the last mail. 2 mails in one year++ on ICANN IDN list. They even 
told me James Seng does not attend all the ICANN IDN meetings (James?) 
anymore.

James sold us something brillant in MdR2000. But the IETF delivered 
something else.

>>I have asked responses about that and got no answer.
>
>I find it very difficult to respond to your emails because you often bring 
>up topics that I don't understand and seem somewhat unrelated, such as "PAD".

PADs are obviously the core of the problem and a revolution. Handles could 
also be used.
In case you forgot about PAD I documented several times, it stands for 
Private Alias Directory. This is the Directory of the ML keywords or DN you 
can register in your own system to call an external host. This is a Quick 
and Dirty excellent solution when an IP address is available. What will be 
the case with IPv6 and that main sites can obviously already support.

>  Also, some of your emails seem to talk about a revolution, which is very 
> difficult to bring about, and hence unlikely.

There are several revolutions right now. Just look at the network, the 
behaviors and the people. A revolution does not mean an immediate tide. It 
means that something we did one way can be done another way, that it has 
started being used and that it will only grow. IDNA is not a revolution.

> left to right instead of right to left), but others have pointed out that 
> such ideas are more in the theoretical realm than the practical one, and 
> so I have stopped talking about them.

This is not a revolution. This is a strange idea :-). Naming was left to 
right first. ARPANET naming was flat and for practical reason became right 
to left. It stayed that way. Your proposal works. Left to right and / 
separators. I understand there are already 15 millions of them. Named handles.

A revolution is that less than 50% of the Internet connections still use 
the DNS, that handles develop, that PADs permits private keywords and 
ultimately billions of roots. And that IAB can issue another document for 
information on the matter or not, the trend is that way and we have to 
organize it or it will be a mess.

>real world, but I don't see much evidence of that.

This is may be that you should go into the real world :-)  I think that in 
this group I am the only one who happens to be there. This is why I keep 
poping in here, in the hope that some field information can go through and 
some great guy like you, Adam, Simon, Brian, John can give a good idea.

All "the rest of the group" are in IETF or SGs. You know, one think the 
things go slowly. That changes will never happen. And all the sudden one 
discovers that they did happen and we did not saw them. Reality is simple: 
IDNA did not really convinced. Some techies had fun implementing them, some 
Registrars tried to make money in calling and registering TMs and some 
cybersquatters lost some money there.

Just a question: when was that the last time you supported a registrant for 
his server to be accessed with the DN he just registered?
This do help to understand what the people want and do.

jfc

---

From: Erik van der Poel <[email-masked]>
Date: Wed, 16 Mar 2005 20:51:27 -0800

JFC (Jefsey) Morfin wrote:
> The issue is easy to 
> understand you need two keyboards to enter the IDNs and two languages to 
> spell them on the phone - except when the ascii TLD can be typed.

Some countries have keyboards with both ASCII and local characters. 
Which countries are you talking about? In any case, we can't just jump 
from the current DNS to one that does not require typing some ASCII 
characters in one go. I think these things must evolve, over time.

> Registries understand nothing to the complex 
> signals (or lack of) from the ICANN, IETF, market, etc. I am on the 
> WG-IDN of the ccTLDs as a lingual organization (Eurolinc). I suppose 
> there is one year I received the last mail. 2 mails in one year++ on 
> ICANN IDN list. They even told me James Seng does not attend all the 
> ICANN IDN meetings (James?) anymore.

The ICANN guidelines for IDNs probably should be updated. People are 
talking about this. Hopefully it will happen quite soon.

> James sold us something brillant in MdR2000. But the IETF delivered 

Are you talking about the following?

http://www.dnso.org/dnso/notes/20011112.ICANN-mdr-IDN-JamesSeng.ppt

> PADs are obviously the core of the problem and a revolution. Handles 
> could also be used.
> Private Alias Directory. This is the Directory of the ML keywords or DN 
> you can register in your own system to call an external host. This is a 
> Quick and Dirty excellent solution when an IP address is available. What 
> will be the case with IPv6 and that main sites can obviously already 
> support.

Maybe people will manually create a few private aliases, but I doubt 
they would bother to do so for the many, many sites on the Internet. 
People visit many sites, one after the other, in quick succession. Maybe 
I'm misunderstanding you...

> behaviors and the people. A revolution does not mean an immediate tide. 
> It means that something we did one way can be done another way, that it 
> has started being used and that it will only grow. IDNA is not a 
> revolution.

I don't see very many signs of a PAD revolution. Am I in the wrong 
country (USA)? The wrong mailing list?

> Naming was left to 
> right first. ARPANET naming was flat and for practical reason became 
> right to left. It stayed that way. Your proposal works. Left to right 
> and / separators. I understand there are already 15 millions of them. 
> Named handles.

I don't see them. Do people exchange handles via email? Please show me a 
handle.

> the DNS,

Are you saying that less than 50% of the TCP connections on the 
"Internet" (open Internet or also behind the firewall?) are immediately 
preceded by a DNS lookup? Where did you get this number from? Please 
state your source.

> that handles develop, that PADs permits private keywords and 
> ultimately billions of roots.

Are there any specs for these handles and PADs? Are there any working 
groups?

> Just a question: when was that the last time you supported a registrant 
> for his server to be accessed with the DN he just registered?

Do your registrants use PADs? Are they happy with them?

Erik

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Thu, 17 Mar 2005 08:19:38 +0100

At 05:51 17/03/2005, Erik van der Poel wrote:
>>The issue is easy to understand you need two keyboards to enter the IDNs 
>>and two languages to spell them on the phone - except when the ascii TLD 
>>can be typed.
>
>Some countries have keyboards with both ASCII and local characters. Which 
>countries are you talking about? In any case, we can't just jump from the 
>current DNS to one that does not require typing some ASCII characters in 
>one go. I think these things must evolve, over time.

Dear Erik,
the "two keyboard" formula is a common nickname of the IDNs in non ASCII 
countries.

>>Registries understand nothing to the complex signals (or lack of) from 
>>the ICANN, IETF, market, etc. I am on the WG-IDN of the ccTLDs as a 
>>lingual organization (Eurolinc). I suppose there is one year I received 
>>the last mail. 2 mails in one year++ on ICANN IDN list. They even told me 
>>James Seng does not attend all the ICANN IDN meetings (James?) anymore.
>

This looks 2001. I mean 2000. It was a great time when many things came to 
a decision.

>>will be the case with IPv6 and that main sites can obviously already support.
>
>Maybe people will manually create a few private aliases, but I doubt they 
>would bother to do so for the many, many sites on the Internet. People 
>visit many sites, one after the other, in quick succession. Maybe I'm 
>misunderstanding you...

I am afraid you missed the point. I work on PADs, for security reason. Even 
if there is a DNS dysfunction you keep going through. For the time being 
this is used for security. I documented that at length in Nov. 2003 when 
preparing meetings on nations vulnerability to the Internet. When our tools 
are ready and tested we will have to release them. But I am sure nice kids 
will make them running before. Anyway, this will be like RSS tags on web 
pages. You click and you have the keyword entered in your PAD. This gives 
the site an edge on others: a free keyword in 7260 language, 20000 dialects 
and an infinity of nicknames. Far cheaper than Real Names.

The main problem is to keep them someway consistent with the other name 
space systems.

>>has started being used and that it will only grow. IDNA is not a revolution.
>
>I don't see very many signs of a PAD revolution. Am I in the wrong country 
>(USA)? The wrong mailing list?

You could come on our mailing list. It is in French. But we will sell 
expertise and may be a product to pay for the study. ISP will offer the 
service. I suppose we will publish lists of PAD infos. But this comes to be 
interesting with IPv6. And competition about IP numbers. NRO will love it. 
NRO first provider of ML keywords :-)

>>Naming was left to right first. ARPANET naming was flat and for practical 
>>reason became right to left. It stayed that way. Your proposal works. 
>>Left to right and / separators. I understand there are already 15 
>>millions of them. Named handles.
>

Ask Library of Congress etc. This is Bob Khan's system.
RFC 3650, 3651, 3652. IMHO you are going to love it!

>Are you saying that less than 50% of the TCP connections on the "Internet" 
>(open Internet or also behind the firewall?) are immediately preceded by a 
>DNS lookup? Where did you get this number from? Please state your source.

Sorry come in the real network land, Europe :-)
This information is probably available everywhere. The authoritative source 
I have is the Chair of AFNIC using notes prepared by Stephane Bortzmeyer. I 
use to trust Stephane.

>>that handles develop, that PADs permits private keywords and ultimately 
>>billions of roots.
>
>Are there any specs for these handles and PADs? Are there any working groups?

Handles is CNRI. IESG expressed a comment which says "not IETF consensus, 
yet it works". Same for PAD, I will not bother writting a Draft, I will 
leave the market using it. Look what they did with our RFC 920 MoU :-)

>Do your registrants use PADs? Are they happy with them?

You kidding? PADs are free.
This is the problem for a Registry!

But this is also something users tend to like. Also that it works. Aslo 
that is simple. Even if Microsoft does not help the programing. But old 
Hosts.txt works well as a starter.
jfc

---

From: Erik van der Poel <[email-masked]>
Date: Fri, 18 Mar 2005 11:33:27 -0800

> discussion must be wider than for only IDNA.

I downloaded all the RFCs and Internet Drafts and grepped for 
Stringprep. Some of the profiles specified in RFCs are not registered at 
IANA. There are 3 profiles in an NFS4 spec (RFC 3530) that have not been 
registered. It is not clear to me that profiles must be registered. I 
have started a table of profiles that shows which parts of Stringprep 
are being used. So far, I have only listed profiles found in RFCs (not 
Internet Drafts). This table was far easier to create in an HTML editor, 
and I didn't want to send HTML to this list, so it's on my Web site:

http://nameprep.org/stringprep.html

It's still rough and doesn't indicate all the details, but it's a start. 
As I read the profiles and text in their vicinity, it seems as though 
some may not be fully aware of some of the complexities of string 
processing in Unicode. (This is not a flaw in Unicode. There are various 
written languages in the world, and some of them have complex rules.)

For example, RFC 3530 says:

    If the case_preserving attribute is present and set to false, then
    the NFS version 4 server MUST use table B.2 to map case when
    processing utf8str_cs strings.  Whether the server maps from lower to
    upper case or the upper to lower case is an implementation
    dependency.

Stringprep's case mapping table is intended to be used only to map upper 
case to lower case. For example, here is a special case:

    00DF; 0073 0073; Case map

You can't really use this entry to go "backwards".

There are other RFCs and Internet Drafts with various issues like this, 
or that are so vague as to make you wonder whether interoperability is 
truly assured. The IETF community is of course full of networking 
experts, and the Unicode community is full of internationalization 
experts. Stringprep finds itself in the rather special position of being 
wedged between networking (specifically, string input and matching) and 
Unicode. It might be a good idea to collect the issues and try to 
increase understanding of international string processing among 
networking community members.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Sat, 19 Mar 2005 18:03:08 -0800

Simon Josefsson wrote:
> behave.

I don't think it's subjective. The concept of normalization requires 
that it be idempotent.

> old and the new code.

We already have the situation where some implementations do it one way, 
and some do it the other way. It is quite clear what will happen when 
somebody uses a character sequence that is interpreted differently by 
these implementations. Keep in mind that Unicode may add new characters 
in the future that may also be affected.

> Both Kerberos and SASL appears to be going to

You write "the old interpretation" as if there is only one 
interpretation of the old spec. That's not true. As we have seen, there 
are implementations that do it one way, and those that do it the other way.

> theoretic setting.

It's not just an "assumption", and it's not merely "theoretical". This 
is a very basic requirement for the normalization process.

> Machines will cope, they compute an

IDNA specifies that a Punycode label must be decoded and then 
Nameprepped and Punycoded again to make sure you get the same string 
back in order to decide what to display (Unicode vs Punycode). This, in 
itself, should make you realize that the process is supposed to be 
idempotent. So we *do* care how the machines compute this algorithm.

> problem sequences, and is simple to implement.

I don't think this is in line with the RFCs. You are rejecting sequences 
that are not rejected by the RFCs.

More importantly, when you continue to ship your implementation as is, 
more and more installations of your popular library will occur, making 
it more difficult for the world to adjust if and when the affected types 
of character sequences are introduced, either with the current 
characters or new characters.

You are in a position to make a difference. You already have. Please 
reconsider.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Sat, 19 Mar 2005 22:02:46 -0800

> Keep in mind that Unicode may add new characters 

New characters are not recognized by old implementations, so they are 
less relevant. Sorry about that part. This relatively small mistake in 
my email should not diminish the overall argument, however.

>> infrastructure based on the old interpretation.

SASLprep is for user names and passwords. People sometimes put strange 
character sequences into passwords to make them harder to guess, but I 
don't know if apps make it easy to enter the strange sequences affected 
by PRI #29. Just a thought -- not particularly important.

Erik

---

From: Simon Josefsson <[email-masked]>
Date: Sun, 20 Mar 2005 08:51:16 +0100

> that are not rejected by the RFCs.

I don't see how RFCs can force an implementation to accept strings
that may cause security problems.  My SASL implementation reject the
PR-29 sequences, as will my Kerberos implementation, and I suggest
that everyone implement similar precautions.

Until this problem is addressed in an update of RFC 3454, rejecting
the problem sequences appear to be the most conservative approach to
me.  What would you propose to do instead?

If the situation is as you claim, that there are multiple StringPrep
implementations out there that implement NFKC in different ways, it
seems hazardous to permit the strings through when you don't know how
other components will behave.  This is especially true if you perform
authentication or authorization on an internationalized authentication
or authorization identity, and then pass it on to another component
that may run another NFKC implementation on the string, before it is
used by a component that assume the identity has been properly
authenticated or authorized earlier.

> characters or new characters.

The problem sequences doesn't normally occur in the wild, at least
that is what PR-29 says, so it is unlikely to ever be a practical
problem.  It remains a security concern though, and that concern won't
go away until the spec's are updated, regardless of what I do in my
implementation.

> You are in a position to make a difference. You already have. Please 

This isn't up to individual implementors.  RFC 3454 need to be updated
to address the problem.  Everyone can submit their own update of RFC
3454 to the IETF and advocate for their proposal.  I don't care
strongly which solution is chosen, if there is a good migration plan
to the new idea.  Meanwhile, implementors will route around the damage
and pick their own solutions.

Regards,
Simon

---

From: Erik van der Poel <[email-masked]>
Date: Sun, 20 Mar 2005 09:53:07 -0800

Hi Simon,

Thanks for taking the time to reply to my email.

Simon Josefsson wrote:

> that may cause security problems.

I agree. If an implementor discovers a problem in a spec, they may wish 
to address that problem in their implementation.

> My SASL implementation reject the

I haven't been involved with this issue for very long. Would you say 
that a lot of the implementors have already heard of your approach, and 
that they are either using the same approach or considering it?

> me.  What would you propose to do instead?

Well, my feeling is that it will take too long to publish a new version 
of the Stringprep RFC, and that implementors should get together to try 
to reach a rough consensus and make changes before the new RFC is 
published. PRI #29 has been published, with a fair amount of info for 
the implementors to make a decision.

> authenticated or authorized earlier.

I can see that your approach is ultra-conservative, given that there is 
some evidence that the affected character sequences are rare.

> go away until the spec's are updated, regardless of what I do in my

Why would a mere spec update make the concern go away? Didn't you say 
that the differing implementations were the concern?

> and pick their own solutions.

OK, let's suppose just for the moment, that we decide to have Stringprep 
point to version 24 or higher of UAX #15. Can you think of a migration 
plan that would satisfy ultra-conservative implementors like yourself?

Or, if you wish, suppose that we do _not_ have Stringprep point to a 
newer version of UAX #15. Can you think of a good migration plan for 
that? (Given that some implementations implement UAX #15 differently.)

Erik

---

From: Simon Josefsson <[email-masked]>
Date: Sun, 20 Mar 2005 22:12:45 +0100

> that they are either using the same approach or considering it?

I haven't seen much evidence of that.  The SASL and Kerberos specs
that use StringPrep haven't been published yet.  Few, if any,
implementors are speaking about this issue in those WG's.  Presumably
the specs are pushed out before the ideas have been implemented and
tested in practice.

> the implementors to make a decision.

I think it may be better to try and reach that consensus within the
IETF.  The expertise on this are supposed to be here.

> that the differing implementations were the concern?

Right.  I was assuming implementors will implement a sanely updated
spec, and that this in combination would solve the problem.

> plan that would satisfy ultra-conservative implementors like yourself?

The spec could suggest that all problem sequences are to be rejected.

> that? (Given that some implementations implement UAX #15 differently.)

Here too, the spec could suggest to reject all problem sequences.  The
NFKC implementations could continue to work as the Unicode 3.2
algorithm description suggest, but since the problem sequences are
rejected, you wouldn't notice the difference in practice.

Assuming PR#29 is correct on that these strings doesn't occur
naturally, rejecting them doesn't appear to have any interoperability
consequences.

Regards,
Simon

---

From: Erik van der Poel <[email-masked]>
Date: Mon, 21 Mar 2005 10:30:01 -0800

>>>Everyone can submit their own update of RFC

One way is to have more than one Internet Draft from individuals, and 
another way is to have a single draft that lists all the alternatives 
and discusses pros and cons of each. I guess the latter approach is 
sometimes used by working groups, but we don't currently have a 
Stringprep working group. Thoughts?

Erik

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 21 Mar 2005 19:38:10 +0100

Erik van der Poel <[email-masked]> writes:

>>> OK, let's suppose just for the moment, that we decide to have
>>> Stringprep point to version 24 or higher of UAX #15. Can you think
>>> of a migration plan that would satisfy ultra-conservative
>>> implementors like yourself?

I don't think a working group is required to do this work.  I believe
updates of older standards are sometimes done outside a working group.

Having a draft outlining the alternatives would be a useful
contribution.  Could be a first step towards producing a single draft
incorporating a solution.

Thanks,
Simon

---

From: "Adam M. Costello" <[email-masked]>
Date: Sun, 27 Mar 2005 21:29:49 +0000

Simon Josefsson <[email-masked]> wrote:

> contribution.

For an implementation, I see the following alternatives:

1) Reject the problem sequences (and no others) before normalizing (add
   a step 0 to the stringprep algorithm).

2) Reject some superset of the problem sequences that is easier to
   compute and still rejects only implausible strings.

3) Reject nothing, but make sure the normalization algorithm follows the
   proposed fixed normalization spec (and the original sample code).

4) Reject nothing, but make sure the normalization algorithm follows the
   current broken normalization spec (not the original sample code).

5) Reject nothing, and follow whichever normalization spec is already
   implemented (that is, ignore the problem).

I have listed these in order of decreasing quality (though the order of
2 and 3 is debatable), and in order of decreasing complexity (except
that 3 and 4 are equally complex).

I don't expect anyone to defend 5.

Given that there already exist implementations of both kinds of
normalization, and neither is more right or more wrong than the other
(in light of the internally inconsistent spec), it would be silly for
stringprep to standardize on the broken one, so we can discard 4.

I can imagine an implementor making a case for any of 1-3.  The next
question is how much latitude the stringprep spec should give the
implementor.

We might consider allowing implementations to choose from among more
than one approved alternative.  I lean against that idea, because
stringprep is used to define valid strings in protocols, and it's hard
to analyze what can happen when implementations disagree about the
validity of a string.  (There is already one way this can happen in
stringprep: unassigned code points become assigned.  The analysis was
very tricky, and I'd prefer to keep that sort of thing to a minimum.)

If stringprep were to pick alternative 3 as the one and only allowed
behavior, some implementations would consider that an unacceptable
security risk and implement 1 or 2 anyway, so in practice we'd still
have implementations choosing from multiple alternatives.

This argues for the stringprep spec to require the rejection of either
(1) the problem sequences or (2) some particular superset of the problem
sequences that is easier to compute.  But the implementation simplicity
of (3) is very appealing, and part of me hopes to be persuaded to go
with that in spite of the analytical complexity.

If we choose 1 or 2, we still need to decide how to define the sequences
that must be rejected.  For example, if we choose 1, I suggest something
like this:

    Definition:  A string must be rejected if and only if oldNormalize
    and newNormalize return different results.

    Theorem:  The rejected strings are exactly the ones containing
    code points with such-and-such properties in such-and-such
    combinations...

    Theorem:  In Unicode version x.y, the previous theorem amounts to
    the following table of specific code point combinations...

Notice that if an implementation has access to both oldNormalize and
newNormalize, it can implement the definition directly without the
complexity of applying either theorem.  It could also do a simple check
for the presence of any code points from the table, and optimize away
the second normalization if none are present.

AMC