Subject: [newprep] intrim experience report

From: Mark Lentczner <[email-masked]>
Date: Fri, 9 Apr 2010 15:08:26 -0700

I thought you all might be interested in a report on my current work in this area:

Both as part of my work in the VWRAP working group, and for an internal design problem at the company I work for (Linden Lab), I have been deeply exploring the issues of defining acceptable identifiers from from the range of Unicode strings. Alas, I find myself in a similar boat as the protocols discussed at the BoF: There is no longer definitive spec to base on!

I started by finding guidance from three major sources:
	StringPrep (RFC 3454)
	IDNAbis 2008 (specifically the tables document)
	Unicode 5.2 UAX #15 & UAX #31 (normalization and identifier syntax)

From each of these, I extracted the core algorithm for validating characters in identifiers. In the case of UAX #31, and StringPrep, I had to develop profiles of each to match my intended use (user names). I tried to make the simplest choices here and follow the guidance that each document had.

For StringPrep I also had to "rebase" the work: It is defined in terms of Unicode 3.2. What i did is take the guidance from the text as well as investigating the Unicode Database (UCD) and developed what I think the intent of StringPrep is, but now defined in terms of properties from the UCD rather than just lists of code points.

In the cases of IDNAbis 2008 and UAX #31, I also, for now, ignored the contextual checks that a very few characters in those specs require.

I rendered each as an function in Python that took a character and returned one of: ALLOWED, DISALLOWED, STRIPPED, UNASSIGNED. Since StringPrep and UAX #31 don't distinguish UNASSIGNED from DISALLOWED, I added that discrimination logic into each of those for purposes of comparison. I recognize that these functions do not represent a precise implementation of the relevant specs, but they are close enough for the purpose of trying gain an insight into how these specs are designed and how I might proceed with building the specs I will need.

Now what I did is run every Unicode code point from U+0000 through U+10FFFF through these three functions and generate lists of blocks of code points where these differ.  There are surprisingly fewer blocks than I imagined, only 282 blocks that break down as:

   1 IDNA:  ALLOWED      ||  PREP:  DISALLOWED   ||  UAX32:  ALLOWED
  50 IDNA:  DISALLOWED   ||  PREP:  ALLOWED      ||  UAX32:  ALLOWED
 230 IDNA:  DISALLOWED   ||  PREP:  ALLOWED      ||  UAX32:  DISALLOWED
  11 IDNA:  DISALLOWED   ||  PREP:  STRIPPED     ||  UAX32:  DISALLOWED

Looking at the details, these four categories of blocks are:
	1) U+0340 & U+0341, combining tone marks
	2) Various punctuation, inter-character fillers, letter-like numbers, symbol combining forms, and variation selectors
	3) Various symbols, and punctuation (mostly script specific punctuation)
	4) Non-semantic zero-width, deprecated, language tags

It seems clear to me that the StripPrep differs primarily in being more liberal with symbols, punctuation and numbers. I will choose the more conservative path. StringPrep also strips some things that the others disallow. This could be considered part of a "pre-sanitization" step (like normalization), and so, is essentially an independent design choice.

The remaining differences between IDNAbis 2008 and UAX #31 are in that the later is more liberal with punctuation, numbers and few other things. Since UAX #31 is expected to be "augmented" by adjusting it's base character classes based on what is legal punctuation in a given scenario, these differences may not, ultimately be that different. For my projects at hand, I will lean toward IDNAbis' more restrictive set.

Hence, my current design for work will be essentially IDNAbis 2008 with a pre-sanitization step involving some stripping and NFKC normalization.  However, implementing IDNAbis 2008 directly, and practically has some significant problems:

Since I want the design to be implementable in readily available environments, I need to ensure that the tests required will be possible. As a strawman, I took the Python 2.4.4 as a baseline. This presents problems for IDNAbis 2008. Many of its tests are based on character properties in the Unicode Character Database. However, Python's unicodedata module offers only a few of the properties from the UCD. This leaves the implementor either the choice of hard coding lists of characters, or writing something that preprocesses the UCD. This isn't very satisfying. Further, IDNAbis 2008's registry of contextual exceptions (not unlike UAX #31's list of restricted contexts), is a poser: Do we define something like identifiers that relies on an external registry of algorithmic fragments? That feels awkward for developers and for code that may be deployed with a rather long shelf life.

Next week I'll be designing a proposal (and Python implementation) for our internal use, working around these issues. I'm happy to share that work here if the group finds it useful. I hope that my experience with this will be able to inform the work here, and the work  with identifiers in the VWRAP working group.  Also, if anyone wants more details of the work listed above (including the Python code), just ask.

	- Mark

Mark Lentczner
Sr. Systems Architect
Technology Integration
Linden Lab

[email-masked]

Zero Linden
[email-masked]

---

From: Marc Blanchet <[email-masked]>
Date: Thu, 15 Apr 2010 15:00:42 -0400

mark,
  that is super valuable and very interesting feedback. please continue 
to update us with your work and findings.

Marc.

Mark Lentczner a écrit :
> Both as part of my work in the VWRAP working group, and for an internal 
> design problem at the company I work for (Linden Lab), I have been 
> deeply exploring the issues of defining acceptable identifiers from from 
> the range of Unicode strings. Alas, I find myself in a similar boat as 
> the protocols discussed at the BoF: There is no longer definitive spec 
> to base on!
> 
>  From each of these, I extracted the core algorithm for validating 
> characters in identifiers. In the case of UAX #31, and StringPrep, I had 
> to develop profiles of each to match my intended use (user names). I 
> tried to make the simplest choices here and follow the guidance that 
> each document had.
> 
> For StringPrep I also had to "rebase" the work: It is defined in terms 
> of Unicode 3.2. What i did is take the guidance from the text as well as 
> investigating the Unicode Database (UCD) and developed what I think the 
> intent of StringPrep is, but now defined in terms of properties from the 
> UCD rather than just lists of code points.
> 
> In the cases of IDNAbis 2008 and UAX #31, I also, for now, ignored the 
> contextual checks that a very few characters in those specs require.
> 
> I rendered each as an function in Python that took a character and 
> returned one of: ALLOWED, DISALLOWED, STRIPPED, UNASSIGNED. Since 
> StringPrep and UAX #31 don't distinguish UNASSIGNED from DISALLOWED, I 
> added that discrimination logic into each of those for purposes of 
> comparison. I recognize that these functions do not represent a precise 
> implementation of the relevant specs, but they are close enough for the 
> purpose of trying gain an insight into how these specs are designed and 
> how I might proceed with building the specs I will need.
> 
> Now what I did is run every Unicode code point from U+0000 through 
> U+10FFFF through these three functions and generate lists of blocks of 
> code points where these differ.  There are surprisingly fewer blocks 
> than I imagined, only 282 blocks that break down as:
> 
> 2) Various punctuation, inter-character fillers, letter-like numbers, 
> symbol combining forms, and variation selectors
> It seems clear to me that the StripPrep differs primarily in being more 
> liberal with symbols, punctuation and numbers. I will choose the more 
> conservative path. StringPrep also strips some things that the others 
> disallow. This could be considered part of a "pre-sanitization" step 
> (like normalization), and so, is essentially an independent design choice.
> 
> The remaining differences between IDNAbis 2008 and UAX #31 are in that 
> the later is more liberal with punctuation, numbers and few other 
> things. Since UAX #31 is expected to be "augmented" by adjusting it's 
> base character classes based on what is legal punctuation in a given 
> scenario, these differences may not, ultimately be that different. For 
> my projects at hand, I will lean toward IDNAbis' more restrictive set.
> 
> Hence, my current design for work will be essentially IDNAbis 2008 with 
> a pre-sanitization step involving some stripping and NFKC normalization. 
>  However, implementing IDNAbis 2008 directly, and practically has some 
> significant problems:
> 
> Since I want the design to be implementable in readily available 
> environments, I need to ensure that the tests required will be possible. 
> As a strawman, I took the Python 2.4.4 as a baseline. This presents 
> problems for IDNAbis 2008. Many of its tests are based on character 
> properties in the Unicode Character Database. However, Python's 
> unicodedata module offers only a few of the properties from the UCD. 
> This leaves the implementor either the choice of hard coding lists of 
> characters, or writing something that preprocesses the UCD. This isn't 
> very satisfying. Further, IDNAbis 2008's registry of contextual 
> exceptions (not unlike UAX #31's list of restricted contexts), is a 
> poser: Do we define something like identifiers that relies on an 
> external registry of algorithmic fragments? That feels awkward for 
> developers and for code that may be deployed with a rather long shelf life.
> 
> Next week I'll be designing a proposal (and Python implementation) for 
> our internal use, working around these issues. I'm happy to share that 
> work here if the group finds it useful. I hope that my experience with 
> this will be able to inform the work here, and the work  with 
> identifiers in the VWRAP working group.  Also, if anyone wants more 
> details of the work listed above (including the Python code), just ask.
> 
> - Mark
> 
> 
> Mark Lentczner
> Sr. Systems Architect
> Technology Integration
> Linden Lab
> 
> [email-masked] <mailto:[email-masked]>
> 
> Zero Linden
> [email-masked] <mailto:[email-masked]>
> 
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> newprep mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/newprep

-- 
=========
IPv6 book: Migrating to IPv6, Wiley. http://www.ipv6book.ca
Stun/Turn server for VoIP NAT-FW traversal: http://numb.viagenie.ca
DTN news service: http://reeves.viagenie.ca
NAT64-DNS64 Opensource: http://ecdysis.viagenie.ca