Subject: [urn] International URNs

From: Freek Dijkstra <[email-masked]>
Date: Thu, 05 May 2011 13:20:50 +0200

Hi,

Does anyone in this group have experience with allow Unicode code points
in URNs, in particular what code points to allow?

I am writing a URN spec, and was wondering to allow percentage-escaped
characters or not. Currently, there is no need by the community, but I
feel that allowing ASCII-only is a bit too restrictive these days.

The basic requirements of my URN namespace are:
- percentage-escaped characters must be valid UTF-8 encoded code points
- lexical equivalence: two URNs are equivalent if and only if the two
  URNs are byte-by-byte equivalent after case folding.
  (Thus no decoding should be necessary to compare URNs)

Any more complex rules will not be accepted by the community.

Now my current draft says that everything confirming to the above should
be accepted as a valid URN. At the same time, the draft has more strict
rules in what may be assigned. That allows some room for future
assignments of identifiers with non-latin characters.

For now, I like to limit the code points in assigned URNs.
My question, what would you do?

 1- not allow percentage-escaped characters in the first place
 2- only allow code points defined by RFC 5892
 3- only allow NFKC-normalised unicode strings
 4- only allow NFKD-normalised unicode strings
 5- Only allow NFKC strings with RFC 5892 code points
 6- Only allow NFKD strings with RFC 5892 code points

Pro 1: simple, but not very "international" :)
Pro 2: No problems with case normalization (RFC 5892 does not allow
uppercase characters, and forbids a whole slew of control characters; it
is in use for international domain names, and seems to replace the older
stringprep stuff.)
Con 2: Disallows upper case. It seems to allow both precomposed and
decomposed character combination (so ü, u-umlaut, can be both "u%cc%88"
= code points 0075+0308  or "%c3%bc" = code point 00FC)
Pro 3: Nearly all input systems use NFC (The Mac file system is the only
thing that uses NFD as far as I know), so no problems with copy & paste.
Con 4: Rarely used, might give problems with copy & paste.
Con 3,4: Allows a slew of control characters.

It seems useful to disallow may of the control characters, so 5 & 6
seems best:

5: No problems with copy & paste, allows lowercase accented characters
and non-latin characters (but no uppercase accented characters)
6: Might have copy & past problems, allow accented lower AND uppercase
characters, and non-latin characters.

NKFD gives decomposed diacriticals. E.g. U-umlaut is decomposed in U +
combining diaeresis -- and because the U is not percentage-escaped, it
is even lexical equivalent to lower case u-umlaut. Awesome.

3 and 4 may give problems with the control characters later. 2 is
ambiguous. So what is the best choice?
- 1 (ASCII only),
- 5 (no upper case, but no copy & paste problems) or
- 6 (upper case allowed, but may have copy & paste problems)?

Regards,
Freek

(PS: sorry for any possible delay in replying on-list; my experience
with urn-nid is that it takes a couple of days before my posts are approved)

---

From: Ted Hardie <[email-masked]>
Date: Thu, 5 May 2011 10:11:40 -0700

>  (Thus no decoding should be necessary to compare URNs)
So far, this seem to me to follow RFC 2141 pretty well.  The only thing that
it seems to me
that you need to say is that all %-escaped characters will be in UTF-8
encoded, as per RFC 3986.

>  1- not allow percentage-escaped characters in the first place

I don't get the use case for this, but you can limit the characters in the
NSS at registration time, and I suspect you could simply say "No character
*requiring* %-escaping will be present in this NSS".  Then you would add a
normalization step that said any %-escaped characters should be restored and
case-folded before comparison.

I would reference
http://www.iana.org/assignments/idnabis-tables/idnabis-tables.xml,
along with the characteristic.  E.g. only characters marked "PVALID" in that
table may occur.

>  6- Only allow NFKD strings with RFC 5892 code points
I don't think referencing strings is the right way to do it; I think you
have to do it using code points.
That would mean you'd either have to reference a table with the valid code
points or present a way to derive them.  I suppose that you could use ABNF
that permitted PVALID characters but note in the text that only
NFKC-normalized strings would be created, pointing to
http://unicode.org/reports/tr15/. But writing ABNF that handles something
like the composition exclusion table or the non-Starter rule seems like it
is going to be hard to do.

regards,

Ted

> _______________________________________________
> urn mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/urn
>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 06 May 2011 00:56:07 +0200

>feel that allowing ASCII-only is a bit too restrictive these days.

Note that per RFC 3986 for non-reserved characters the literal character
is the same as its %hh-encoded equivalent, so allowing percent-encoding
is not an option you have (there was a discussion in the IRI WG about it
that I have not fully kept track of, but I would assume this to be true
at least for characters in the ASCII range nonetheless).

> 6- Only allow NFKD strings with RFC 5892 code points

I think you need very good reasons to specify more than "UTF-8" if you
decide to use UTF-8 (you could also choose, say, Base64 encoded UTF-8,
or whatever else may suit you). Personally, I think if it is important
for, say, resolution or for establishing identity, which of these you
use, then it's best to perform late normalization and ignore variance
during exchange of identifiers.

>It seems useful to disallow may of the control characters, so 5 & 6

Generally speaking, where control characters like U+0000 or line feeds
do not make sense as part of the protocol slash identifier slash what-
ever, it does not really matter much whether you allow or disallow the
use of them, what's more important is that the security implications
are properly understood (for instance, if you get an identifier with,
say, %00 in it, do you reject it outright).
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Kocer, Kadir Karaca" <[email-masked]>
Date: Fri, 6 May 2011 16:10:49 +0200

>feel that allowing ASCII-only is a bit too restrictive these days.

...
>Pro 1: simple, but not very "international" :)

Dear Freek,

first of all I definitely appreciate your "Political Correctness" ;-)

But what you are trying to do will end up with URNs that have parts with "meaning-for-humans" and you should seriously discuss this with your colleagues if it is the right way for your purposes.

My personal experience (over 5 years URN:NBN with 4.600.000 URNs) is:
if you allow people to assign URNs as they prefer, they always tend to "invent" some semantic rules.

At the end you have your database full of Identifiers like:
[institution-name]-[division-name]-[collection-name]-[date]-[item-number]

This goes fine many years.
Till the day the collections are renamed or two divisions fusion under another name or renaming of the institution or ...

Experiences like those are the reason why many colleagues with Long Term Archiving background propagate Identifiers without semantics --meaningless strings just for machines.

IMHO that is the problem and not if those meaningful-names written in Kanji, Chinese, Arabic or Krill.

Seriously I would allow ONLY numbers and 3-4 separator chars if i could define a new generation ID system. That is NOT "restrictive" but functional and problem free.

Best Regards
Karaca Koçer (Yes, my name has also a Non-ASCII character!)

---

From: Freek Dijkstra <[email-masked]>
Date: Fri, 06 May 2011 17:19:26 +0200

Bjoern,

Thank you for you helpful email.

> at least for characters in the ASCII range nonetheless).

Thanks, I missed that. Note that RFC 3986 section 2.3 recommends to not
create URIs with %-encodings for unreserved characters, and at the same
time suggests receivers should decode them.

I just looked through the 40 RFCs and 1 I-D listed at
http://www.iana.org/assignments/urn-namespaces/, but to my surprised
none of the specifications takes this into consideration.

Two specs (iso and nzl) explicitly mention that the percentage-encoded
characters should be decoded to a code point. (Even though iso does not
allow them in the syntax). None specifies if and what type of
normalization should with the Unicode code points.

For the record, here is my quick breakdown (see attachment):
* 2 specs (above) explicitly describe the equivalence relation
* 19 specs explicitly disallow percentage encoding, and one (publicid)
limits it to 8 allowed code points.
* 2 specs allow percentage encoding, and specify "case insensitive" for
lexical equivalence
* 5 specs allow percentage encoding, and specify "case sensitive" for
lexical equivalence (which violates the equivalence rules of RFC 2141)
* 5 specs delegate syntax and equivalence to subregistries or other
documents
* 6 specs do not specify syntax, and specify "case sensitive" for
lexical equivalence
* 2 specs do not specify syntax, and specify "case insensitive" for
lexical equivalence

>> [...]

> decide to use UTF-8 [...]. Personally, I think if it is important

What do you mean with "late normalization"?

My preference would be not to normalize during exchange of identifiers
at any time, but either forward/store/use a received URN as-is, or
reject it outright because of bad syntax (just like any application
should now reject a URI with a caret (^) character because that's not by
RFC 3986.) Is that what you mean with "ignore variance during exchange
of identifiers" -- do not normalise?

> [...] what's more important is that the security implications

There is  currently limited need for people to copy & paste identifiers
in this namespace. That may be a security risk if copy & paste happens
with the decoded form of the URN (instead of the percentage-decoded
form) inadvertently changes the normalization (e.g. translation to NFC).
This could potentially lead to malversed URIs (e.g. www.g00gle.com)

I see four basic solution to this potential risk:
1. do not allow non-ASCII code points
2. only display/input encoded URNs
3. allow all code point and require the use of a validation service
4. only allow code points that are known not to cause this risk

With "allow" in 4, I mean:
- A created URN must contain only allowed code points
- A received URN that is not a valid UTF-8 encoded string must be
rejected. Other URNs should be accepted.
- A received URN that contains illegal code points must not be displayed
in decoded form (but only in encoded form).

Is (4) a good strategy? Or do you recommend I change my draft to
something else?

Thanks for the feedback so far,
Freek Dijkstra

---

From: Freek Dijkstra <[email-masked]>
Date: Fri, 06 May 2011 20:47:33 +0200

Kocer, Kadir Karaca wrote:

> Till the day the collections are renamed or two divisions fusion
> under another name or renaming of the institution or ...

Thanks. The draft warned users of these risks. However, your note
convinced me to disallow percentage encoded characters altogether.
Thanks for the nudge.

Freek