Subject: Re: [urn] Finding the end of the URN string

From: "Svensson, Lars" <[email-masked]>
Date: Tue, 6 Sep 2011 20:07:40 +0200

Juha, all,

I'm not quite sure I understand exactly where the problem is, but I have
a feeling that it is based in a confusion between the urn and the
identifier embedded in the urn string (the NSS).

> RFC2141 has this to say about parsing URN strings embedded in plain
> text:
> 
> "In textual context, a URN ends when an octet/character from the
> excluded character set (<excluded>) is encountered.  The character
> from the excluded character set is NOT part of the URN."
> 
> Although URNs will usually be incorporated in Dublin Core and other
> kind of metadata records where the encoding makes the data machine
> understandable, it may be useful or even necessary to discuss the
> issue of URN parsing in plain text in more detail in RFC 2141bis. And
> unlike back in 1997 we know a lot how URNs are used in textual
context.
> 
> The problem with the current statement is that there is no guarantee
> that the first character after the URN string is from the excluded
> character set. For instance, International Standard Name Identifier
> 
> urn:isni:1111222233334444
> 
> could be followed by the telephone number of the person / organization
> to whom the ISNI belongs to. This information would not be part of the
> ISNI, but RFC 2141 would fail in making this clear.

Would I in that case add the telephone number directly to the urn:isni,
making it urn:isni:1234567890123456-49-69-123456789 (assuming that my
phone number is +49-69-123456789, which it isn't)? Can you elaborate
more on this?

> In order to be able to parse URNs in plain text in reliable manner,
> applications must be aware of the syntax of the identifier system. For
> instance, there are 16 digits in ISNI, which makes it easy to find the
> end of the NSS.
> 
> ISBNs assigned since 2007 consist of 13 digits, like this:
> 
> urn:isbn:978-0-07-166346-5
> 
> (As an aside, Principles of Digital Audio, 6th ed., to which this ISBN
> belongs to, contains a chapter which explains the structure of ISBN. I
> was a little bit surprised.)
> 
> Applications parsing urn:isbn's based on the new ISBN standard must
> know that the NSS should consist of 13 digits. Therefore, if somebody
> builds URNs like these:
> 
> urn:isbn:<isbn-string>.<foo> (where "." is a local fragment separator)
> 
> or
> 
> urn:isbn:<isbn-string>#<foo>
> 
> or
> 
> urn:isbn:<isbn-string>?<foo>

But RFC 2141 is not about where the NSS ends, it's about where the URN
itself ends: "In textual context, a _URN_ ends" (emphasis mine). And in
urn:isbn:<isbn-string>#<foo> the _URN_ ends after <foo>, just as
urn:isbn:<isbn-string>#<bar> ends after <bar>.
 
> the ISBN parsing algorithms should either ignore everything that comes
> after the 13th digit, or conclude that the NSS string in this
> particular URN is incorrect. A sophisticated algorithm should always
> do the former when the thing added to the URN is <query>. To my
> knowledge, no such algorithms exist yet.

If I want to find the NSS in a generic URN, shouldn't I be able to do
that by decomposing the URN using the ABNF?

> There are namespaces such as urn:nbn where no generic rules for the
> NSS parsing exist. It may be that such parsing rules can be developed
> sub-sections of these namespaces (for instance: to the NBNs generated
> in the National Library of Finland). In these cases, a last resort
> parsing rule can be used, which is that the first character which is
> not allowed in that particular namespace indicates the end of the URN.
> And if the namespace does not have any rules that go beyond the URN
> syntax specification as regards the allowed characters, RFC 2141 can
> be used as the basis for parsing the URN.
> 
> To sum up, the short text section quoted above could be edited to this
> form in the next version of RFC2141bis in order to reflect the reality
> more closely:
> 
> URNs will often be embedded in metadata records or in structured text.
> There the encoding will make it clear where the NSS ends, and
> therefore this method of incorporating URNs is recommended. A special
> case of this is the use of URNs as HTTP URIs to make them resolvable
> in the present Internet.
> 
> When URNs are incorporated in plain text, reliable parsing of
> namespace specific strings is possible only if the identifier system
> has a well formed syntax (which specifies e.g. the length of the
> string) and the parser is familiar with it. For instance, ISBN numbers
> are either 10 or 13 digits long depending on the version of the ISBN
> standard, and the parser must be able to recognize which type of ISBN
> is being processed, and behave accordingly.

Again, shouldn't this be done by using the ABNF?

> If NSS contains a <query>, parsing algorithms MUST ignore it. If NSS
> contains a <fragment> or a local add-on,

Would that local add-on be part of the URN? Does that have to be
specified in the syntax?

> parsing algorithms SHOULD
> behave in different ways depending on the namespace. In the ISBN
> namespace the algorithms SHALL ignore anything that comes after the
> ISBN itself, and MAY label the ISBN as erroneous. Some other
> namespaces (and possibly even future versions of ISBN) may allow the
> user of <fragment> or local add-ons for identification of fragments of
> resources, in which case the parsing algorithms will know how to deal
> with them.

I would not have anything specific to a particular namespace in a
document concerned with a generic syntax (in this case for URNs).

> If the identifier does not have a specified syntax, but its character
> set is more limited than that allowed by the URN syntax, a URN has
> certainly ended when an octet/character from the character set
> excluded in that namespace is encountered. This method is not
> reliable; NSS may have ended earlier if the characters following the
> NSS and not excluded.

Again, in my understanding it's not just about finding the NSS, it's
currently about finding the end of the URN in text. 

> If the namespace does not have any limitations concerning the
> character set, a URN has certainly ended when an octet/character from
> the excluded character set (<excluded>) is encountered.This method is
> not reliable: NSS may have ended earlier.
> 
> Please note that the character from the excluded character set is NOT
> part of the URN.

As I don't know enough about those things, can someone with more
knowledge of URI parsing algorithms help us out?

Thanks and all the best,

Lars

---

From: Juha Hakala <[email-masked]>
Date: Wed, 07 Sep 2011 13:55:28 +0300

Hello Lars; all,

The discussion started from the problem that RFC 2141 specifies only 
this method for finding the end of the NSS:

> An URN ends when an octet/character from the excluded character set
>    (<excluded>) is encountered. 

This method fails when the first character after the NSS is not an 
excluded one. Therefore, when the identifier standards specify the 
syntax of the identifier, including its length, parsers should use this 
information for more reliable parsing results. Syntax may even allow the 
applications to check if the NSS string is correct (several identifiers 
contain a check digit).

RFC2141bis must still include the current method for a) backward 
compatibility, and b) in order to accommodate namespaces where the 
length of the identifier string is not specified.

> phone number is +49-69-123456789, which it isn't)? Can you elaborate

In the (admittedly unlikely) situation shown above, parsing according to 
RFC2141 would go wrong; the end of the NSS would be the space following 
the last digit (9). In contrast, a parser using the ISNI syntax would 
know that ISNI has just 16 digits and thus the last digit of the NSS 
becomes the 6 before the first hyphen.

There is also a related issue that (ISO) identifier standards often 
specify how to express the identifier in human readable form. This 
involves adding non-semantic characters such as spaces and hyphens. For 
instance, 9780071663465 becomes ISBN 978-0-07-166346-5 and 
1111222233334444 ISNI 1111 2222 3333 4444. The ISO standards provide no 
guidelines as regards how to express these identifiers as URIs / URNs. 
This may be just a blessing, because it is easy to provide the general 
guidelines in RFC2141bis and namespace specific details in namespace 
registrations.

So, 2141bis should say something to the effect that

A. if the NSS itself contains  excluded characters, they must be percent 
encoded. An example of this are the square brackets in ISCI.

B. if the identifier in human readable form contains non-semantic 
characters which are acceptable, these may be used in NSS, but lexical 
equivalence must be specified. For instance,urn:isbn:9780071663465 is 
the same as urn:isbn:978-0-07-166346-5. Any non-semantic characters must 
be ignored when e.g. the length of the NSS is calculated.

C. if the identifier in human readable form contains non-semantic 
characters which are excluded, they should not be used in NSS, because 
in percent encoded form they no longer improve readability. For 
instance, string like 1111%202222%203333%204444 would not help anybody.

> urn:isbn:<isbn-string>#<bar> ends after <bar>.

No, according to the rules of the ISBN namespace as specified in RFC 
3187, you will not be allowed to use fragment, and if you do, then the 
parsing algorithm shall / should ignore it. RFC 2141bis already says 
that namespace registrations must specify if fragments are allowed, and 
in the ISBN namespace they will not be.

Please note that for instance this URN

urn:nbn:<isbn-string>#<foo>

is well formed and does end only after <foo>, and the parsing rules, if 
any, must take that into account.
> that by decomposing the URN using the ABNF?

That is the idea underlying the current RFC2141. But in the plain text 
context we cannot be sure that the NSS will always be followed by an 
excluded character such as space, although that will often be the case.

A lot of attention has been put to being able to find URNs from text 
(the "urn:" in the beginning of the string is only there for this 
purpose). But we have not really worked out properly how to find the end 
of the NSS.

> Again, shouldn't this be done by using the ABNF?

See above.
> Would that local add-on be part of the URN? Does that have to be

Local add-ons used for identification of e.g. logical fragments will be 
part of NSS and URN. As "local" implies, namespace registration cannot 
be aware of them. URN resolution process must know what to do with them, 
though. That is, a URN with local add-on must resolve to URI (URL).
> document concerned with a generic syntax (in this case for URNs).

RFC2141bis can (and I think should) use e.g. ISCI, ISBN and ISNI as 
examples in order to clarify how to implement the generic guidelines 
specified in the RFC.
> knowledge of URI parsing algorithms help us out?

Parsing URNs is one step further from parsing URIs, especially if URN 
parsers should have namespace specific syntactical knowledge.

Since we are only working on the principles the algorithms are not there 
yet. But since e.g. ISO identifiers have simple syntax, it is not 
exactly rocket science to develop these applications. And to be honest, 
we have done quite well for 10 years without having any kind of 
sophistication in place as regards URN parsing. That does not mean that 
such functionality will not be of use in the future.

Best regards,

Juha
> 
> Thanks and all the best,
> 
> Lars
> _______________________________________________
> urn mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/urn
> 

-- 

  Juha Hakala
  Senior advisor, standardisation and IT

  The National Library of Finland
  P.O.Box 15 (Unioninkatu 36, room 503), FIN-00014 Helsinki University
  Email [email-masked], tel +358 50 382 7678