Subject: [urn] Unicode, UTF-8 and persistent identifiers

From: "Hakala, Juha E" <[email-masked]>
Date: Fri, 4 Dec 2015 07:07:28 +0000

Hello, 

identifier standards always specify the characters / character set allowed. They may also specify encoding. 

Encoding is usually UTF-8, but characters / character sets vary. Old domain specific standards like ISSN and ISBN used just numbers from 0 to 9 (and ISBN also X).  Generic identifier systems like ARK and Handle are more liberal, but in different ways. ARK allows just printable ASCII characters, whereas Handle character set includes all printable Unicode characters. The way this is specified in RFC 3650 is a bit careless: 

"Handles may consist of any printable characters from the Universal
   Character Set (UCS-2) of ISO/IEC 10646, which is the exact character
   set defined by Unicode.  The UCS-2 character set
   encompasses most characters used in every major language written
   today.  To allow compatibility with most of the existing systems and
   to prevent ambiguity among different encodings, the Handle System
   protocol mandates UTF-8 to be the only encoding used for handles."

UCS-2 is not a character set, but an obsolete encoding which became UTF-16 when a mechanism to support planes beyond Unicode BMP was added. UCS-2 and UTF-8 are not compatible, so methinks RFC 3650 was not prepared as carefully as rfc2141bis ;-). As an aside, Handle community has not updated the character set related documentation; the latest version of the Handle technical manual (from 2010) says 

"Handles may consist of any printable characters from the Universal Character Set, two-octet form (UCS-2) of ISO/IEC
10646, which is the exact character set defined by Unicode v2.0."

It is not obvious for me what "printable character" means in the context of Unicode. Definitely exclusion of all Unicode control characters, but what about for instance emojis? There were none in Unicode 3.0, but Unicode 8.0 has about 1200, and IMO identifiers containing emojis should not be allowed at all. One reason for such exclusion is that different operating systems and applications may have different symbols for the same emoji. For instance, heavy black heart is actually often red: 

http://apps.timwhitlock.info/emoji/tables/unicode    

Other issues include diacritics without their host characters. And should only precomposed versions of characters like a mit umlaut be allowed, or is it OK to have them both in decomposed and precomposed forms? Generally, what can and should be said in the Handle/URN/DOI syntax? 

The lesson to learn from RFC3650? Although this part of the Handle documentation is both a bit confusing and out of date, Handle is the most widely used persistent identifier system out there. Millions and millions of Handles have been assigned, they serve the users just fine, and I have not seen any complaints about the Handle character set & encoding specification being both erroneous and out of date. So I think URNBIS should not spend plenty of time and effort in trying to make character set / character encoding related parts of rfc2141bis perfect; good will do. Of course, it would be nice if Handle documentation were up to date since eventually some people may start having questions about what is OK and what isn't. 

AFAIK the biggest issue with allowing (printable) Unicode characters in URNs is security. According to Unicode confusables utility, simple identifier like "abc" has 9072 different variants when printed: 

http://unicode.org/cldr/utility/confusables.jsp?a=abc&r=None 

and "123" has more than 10.000 variants. Computers will be able to tell them apart (unless they rely on OCR), but humans may have problems. 

It is easy to check if standard identifier like ISBN is correct. If a single number "3" in an ISBN is replaced with Cyrillic capital letter ZE which in some fonts looks just the same, ISBN parser knows that this "ISBN" is not correct and a link based on might lead users astray. With Handle system, replacing number 3 with ZE is fine according to the Handle specification, so no generic Handle parser is able to turn such identifier down. In practice, some Handle prefixes may have their own, much limited character set specifications, and they may parse their Handles according to these specs. 

I am not sure if we should say more about security in rfc2141bis. We already have a reference to RFC6943 which may be sufficient (see especially chapter 4.2 in that RFC). Another link might be added to Unicode confusables page. And it might be useful to say that as a rule security issues increase when character set allowed grows since then the confusables such as Cyrillic ZE replacing "3" are harder to detect.      
 
All the best, 
   
Juha

> -----Original Message-----
> From: urn [mailto:[email-masked]] On Behalf Of Sean Leonard
> Sent: 3. joulukuuta 2015 20:08
> To: [email-masked]
> Subject: Re: [urn] UTF-8 ABNF Re: What Happened to UTF-8?
> 
> Good news!
> 
> I reviewed every URN registration, formal and informal, in the IANA URN
> Namespaces registry [1]. Every single one was registered under the rules of
> RFC 2141 / RFC 3406, after those were standardized. So that includes
> UTF-8 encoding.
> 
> RFC 2141 is the operating standard and has the IETF consensus; all
> namespace registrants are bound to follow it, including UTF-8 encoding.
> There is no evidence that any namespace registrants are bound to prior or
> different rules. If you want to change the UTF-8 part of RFC 2141 [bis], it
> needs a different IETF consensus. Otherwise this whole UTF-8 issue is just an
> editorial matter.
> 
> Should anyone have a different view, documented evidence is necessary to
> construct a credible argument.
> 
> Implementers can rely on the premise that the NSS percent-encoded
> characters are UTF-8 encoded. (Conveniently that makes IRIs directly
> transposable, at least for the NSS component. The other proposed
> components like f-, q-, and r-, not necessarily, but that's
> out-of-scope.) URNs are supposed to be persistent, and they can't be
> persistent if meddlers at IETF change the rules, even subtly, for no good
> reason.
> 
> Regards,
> 
> Sean
> 
> [1] http://www.iana.org/assignments/urn-namespaces/urn-
> namespaces.xhtml
>