Subject: [urn] 2141bis equivalence rules

From: Andrew Newton <[email-masked]>
Date: Mon, 17 Nov 2014 09:28:23 -1000

All,

I'm inviting feedback on the equivalence rules of section 3.3 and
section 4 of 2141bis. It states that p-components are to be used for
equivalence testing where as q- and f- components are not.
Additionally, the NSS may also have separate rules per NID with
regards to equivalency. Are there any questions, comments, or
objections to these rules? If so, why?

-andy,
co-chair

---

From: Juha Hakala <[email-masked]>
Date: Tue, 18 Nov 2014 08:16:59 +0200

Hello,

A few comments.

In 4.1, instead of saying:
> URN namespaces MAY define additional rules for equivalence, such as
>     case-insensitivity of the NSS (or parts thereof).
we might provide a stricter formulation like this:

If required by the syntax and semantics of the identifier, a URN namespace MUST
define additional rules for equivalence, such as
    case-insensitivity of the NSS (or parts thereof), or listing of characters which
should be ignored when equivalence is determined (like hyphens in ISBN strings).

There should be step 5 in the determination of equivalence:

5. namespace specific rules for equivalence, if any

Some namespaces provide means (check sums) which may be used for
determining if the identifier string is valid or not.

Checking the NSS validity should be included in the equivalence analysis if possible.

In 4.2, examples with f-, q- and p-components could be useful.

Juha

On 17.11.2014 21:28, Andrew Newton wrote:
> _______________________________________________
> urn mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/urn

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 18 Nov 2014 18:27:52 +0900

Hello Juha, others,

> should be ignored when equivalence is determined (like hyphens in ISBN

The alternative would be to define the syntax of an NSS so that such 
equivalences are eliminated by definition. For ISBNs, this could e.g. be 
done by just disallowing the hyphens, or in general defining that the 
NSS is all lower-case instead of requiring case-insensitivity.

I think we should at least leave this choice to the creators of URN 
namespaces, and we should probably recommend to eliminate equivalences 
by restricting syntax rather than mandate that each scheme MUST define 
equivalences.

> 5. namespace specific rules for equivalence, if any

This would exclude implementations that know about URNs in general, but 
not about (all) specific URN namespaces. Creating an implementation that 
knows about all specific URN namespaces is very difficult, if not to say 
impossible: New namespaces get defined, and the equivalences are defined 
in prose, not in an executable way.

> Checking the NSS validity should be included in the equivalence analysis

How exactly should this work?

We can distinguish three cases. Comparing valid URNs with valid ones, 
valid ones with invalid ones, and invalid ones with invalid ones.

As for comparing valid URNs among themselves, checking for invalidity 
shouldn't affect the result.

When comparing a valid and an invalid URN, I'd assume that they would 
turn out to be non-equivalent without needing to check for validity, 
which would be the result I'd expect anyway. Or would you want that they 
turn out to be equivalent if one of them e.g. just has a wrong checksum, 
but is otherwise the same?

That would leave two non-equivalent identifiers. Comparing two identical 
but non-equivalent identifiers will result in them being judged 
equivalent even without validity checks. That's just GIGO (garbage in, 
garbage out) and shouldn't concern us too much.

Given the above considerations, my guess would be that we can live very 
well without involving validity checks in equivalence analysis. But 
maybe if you can explain how you thought it should work, that might help.

Regards,   Martin.

---

From: Juha Hakala <[email-masked]>
Date: Tue, 18 Nov 2014 13:14:55 +0200

Hello Martin,

On 18.11.2014 11:27, "Martin J. Dürst" wrote:
> equivalences are eliminated by definition. For ISBNs, this could e.g. 
> be done by just disallowing the hyphens, or in general defining that 
> the NSS is all lower-case instead of requiring case-insensitivity.

The current ISBN namespace registration allows the use of hyphens, and 
there are a lot of URN:ISBNs out there which already have them. 
Disallowing hyphens in a revised registration request would cause 
problems for current implementers.

ISBNs are commonly expressed both with and without hyphens. URN system 
has to accommodate that, and that can be done by creating namespace 
specific rules for lexical equivalence which says that hyphens must be 
ignored.

In the case of ISBN, the main challenge in determination of lexical 
equivalence is that publishers, when reprinting a book (which does not 
require a new ISBN), often replace ISBN-10 with lexically equivalent 
ISBN-13.

> by restricting syntax rather than mandate that each scheme MUST define 

IMO the best approach is the one that creates the smallest amount of 
turbulence. In the ISBN case it is the one which allows two ways of 
expressing ISBN strings (with or without hyphens).
> This would exclude implementations that know about URNs in general, 
> but not about (all) specific URN namespaces. Creating an 
> implementation that knows about all specific URN namespaces is very 
> difficult, if not to say impossible: New namespaces get defined, and 
> the equivalences are defined in prose, not in an executable way.

It is unlikely that any application out there needs to deal with all URN 
namespaces with this level of specificity. A library system must know 
how to deal with URN:ISBNs and other bibliographic identifiers, but that 
is trivial since these applications already know how to deal with ISBNs 
etc.

Even new URN namespaces will often be based on established identifiers, 
so there will be applications which are able to process these 
identifiers.  Usually it is not difficult to determine equivalence 
especially in the URN context where the identifier strings are not 
heavily modified. With other persistent identifiers things may not be as 
straightforward - for instance, DOI system requires non-trivial changes 
to be made to the ISBN strings.

> valid ones with invalid ones, and invalid ones with invalid ones.

The case I had in mind is the comparison between the valid (corrected) 
and invalid (original) ISBNs. Library metadata usually incorporates both 
variants. It is important to show that they are equivalent (identify the 
same resource) since both versions of the URN:ISBN may be actionable in 
the Web. For instance, the publisher may prefer to use the invalid 
identifier and libraries the correct one.

> shouldn't affect the result.

Yes.
> which would be the result I'd expect anyway. Or would you want that 
> they turn out to be equivalent if one of them e.g. just has a wrong 
> checksum, but is otherwise the same?

If the only difference is that one identifier has a wrong check sum and 
another the correct one, we may assume that  the two are in fact 
equivalent and there is just a typo or miscalculated check sum in 
another one.

If there are other differences between valid and invalid identifier, 
they cannot be regarded to be lexically equivalent. Library system would 
be aware of both ISBN versions since they are always included in the 
metadata describing the resource, tagged in appropriate way. Providing 
both (all) ISBN alternatives is important since both URN:ISBNs may be 
actionable, for instance the invalid one in the publisher's system and 
the valid one in the national library's repository.

>
> That would leave two non-equivalent identifiers. Comparing two 
> identical but non-equivalent identifiers will result in them being 
> judged equivalent even without validity checks. That's just GIGO 
> (garbage in, garbage out) and shouldn't concern us too much.

Yes.
>
> Given the above considerations, my guess would be that we can live 
> very well without involving validity checks in equivalence analysis. 
> But maybe if you can explain how you thought it should work, that 
> might help.

This is not a question of life and death for me :-).

It is also possible to determine (using the namespace specific rules) if 
the NSS is valid, and - if it is not - cancel the rest of the lexical 
equivalence checking process because the result would not be reliable. 
I'm afraid this is unavoidable if there is no way of knowing / making an 
educated guess what the correct NSS would have been.

All the best,

Juha

---

From: Andrew Newton <[email-masked]>
Date: Tue, 18 Nov 2014 08:50:14 -0500

On Tue, Nov 18, 2014 at 4:27 AM, "Martin J. Dürst"
<[email-masked]> wrote:
> done by just disallowing the hyphens, or in general defining that the NSS is
> all lower-case instead of requiring case-insensitivity.
>
> namespaces, and we should probably recommend to eliminate equivalences by
> restricting syntax rather than mandate that each scheme MUST define
> equivalences.

Wouldn't this have backwards compatibility issues?

> This would exclude implementations that know about URNs in general, but not
> about (all) specific URN namespaces. Creating an implementation that knows
> about all specific URN namespaces is very difficult, if not to say
> impossible: New namespaces get defined, and the equivalences are defined in
> prose, not in an executable way.

I agree. It seems impossible to create an implementation that is aware
of all URN namespaces.

Furthermore, do we care that two URNs from two different namespaces
are equivalent? They may resolve to the same location, but they are
two different names. So they aren't equivalent, right?

-andy

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 19 Nov 2014 19:27:39 +0900

Hello Juha,

> ISBNs are commonly expressed both with and without hyphens.

Of course.

> URN system has to accommodate that,

urn:isbn: already exists, and I definitely don't want to change its 
definition, especially not in ways that are not backward-compatible 
(i.e. declaring some ISBN URN that was valid before suddenly invalid).

But if we take ISBNs just as an example for a future registration, then 
I think it's a very valid idea to e.g. say that ISBNs can contain 
hyphens, but these are removed when constructing an urn:isbn: URN. It 
may not always be the best solution, but it would be strange to reject 
such as solution from the get-go.

> and that can be done by creating namespace

Yes, it can. But we should write our document so that the other choice 
(eliminating gratuitous equivalences by restricting the permitted 
syntax) is at least also permitted, if not preferred.

> require a new ISBN), often replace ISBN-10 with lexically equivalent

One solution would be to just keep urn:isbn: syntax at 10 digits. The 
remaining 3 digits are the same all the time anyway. I'm not saying this 
is the best solution, but it is one choice with some clear advantages.

> expressing ISBN strings (with or without hyphens).

'turbulence' is a very vague concept. In the case of ISBN URNs, my 
interpretation of

    An ISBN is a ten-digit number (actually, the last digit can be the
    letter "X" as well, as described below) which is divided into four
    variable length parts usually separated by hyphens when printed.

(from https://tools.ietf.org/html/rfc3187) is that hyphens are already 
allowed at specific places, and therefore in this specific case, it's a 
bad idea to try to backtrack and disallow them.

> is trivial since these applications already know how to deal with ISBNs

Then your above proposal of

<<<<
<<<<

should be changed to something like

5. namespace specific rules for equivalence, for the (possibly empty) 
set of namespaces that an application chooses to apply such rules

The general "if any" to me at least means that any (i.e. all) rules that 
exist should be applied.

> to be made to the ISBN strings.

I'm not saying that any specific equivalence rules for any specific 
namespaces are complicated. I'm saying that it's difficult to make sure 
they are applied across the board, including new schemes. Even if an 
application already has some equivalence rules somewhere, digging these 
up, checking that they can be used for the URN syntax (which may be 
different from the syntax of a bare identifier of the same kind), and so 
on may involve quite some work.

> variants.

What do you mean by "incorporates"? My hope is that library metadata as 
much as possible contains only valid data, and that software used for 
inputting it checks inputs for validity. But maybe I'm having too high 
expectations for library metadata, or I don't understand what you mean 
by "incorporates".

> It is important to show that they are equivalent (identify the

I understand that mistakes happen, but why would a publisher prefer to 
use something invalid?

> equivalent and there is just a typo or miscalculated check sum in

So if I have an ISBN of (987-)1-449-32188-9, who is going to tell me 
what this is supposed to be, given that I have a choice of 14 
alternatives at 
http://www.isbn-check.de/checkisbn.pl?isbn=1-449-32188-9&submit=test&lang=en 
?

After all, we all know that checksums such as this one are a way to 
catch some errors, but in no way allow us to fix the errors. In order to 
be able to fix a single-digit error, we would need at least a two-digit 
checksum.

Or do you expect to also involve an actual book catalog, which might 
help to reduce the choices somewhat?

[Hint: If you check all the alternatives, rule out the ones that are not 
actually assigned, and then look at the topics of the resulting books, 
you might find out which book I referred to; it's somewhat related to 
the topic at hand :-).]

>> judged equivalent even without validity checks.

[sorry, that should have read "Comparing two identical but non-valid 
identifiers".]

>> That's just GIGO

My impression is that at least for ISBNs, knowing is impossible. Making 
an educated guess may be possible for humans (see my example above), but 
it should not be part of URNs per se. I'm not sure about others, but at 
least in my book, the concept of permanent, carefully assigned 
identifier and the concept of "educated guess" don't go very well together.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 19 Nov 2014 19:32:42 +0900

> Wouldn't this have backwards compatibility issues?

I was assuming that this discussion is mostly about new namespaces. I 
think we should make it clear that whatever we write in our document, it 
applies to existing namespaces only as far as it doesn't make backwards 
incompatible changes.

> two different names. So they aren't equivalent, right?

Yes. As identifiers, they are inherently not equivalent. I don't think 
we need to put this in words; I think the text in section 6.1 of RFC 
3986 should be enough.

Regards,   Martin.

---

From: Juha Hakala <[email-masked]>
Date: Wed, 19 Nov 2014 13:35:25 +0200

Hello Martin,

> (i.e. declaring some ISBN URN that was valid before suddenly invalid).

Good.
>
> But if we take ISBNs just as an example for a future registration, 
> then I think it's a very valid idea to e.g. say that ISBNs can contain 

IMO this is something which should be considered whenever a URN 
namespace is registered. The ISBN community could afford to be liberal 
since all applications which deal with ISBNs know that hyphens are to be 
ignored when ISBNs are processed.  But some other communities may have 
valid reasons for providing additional guidelines for how to express 
their identifiers as URNs (that is, something that goes beyond the 
simple percent encoding).
> syntax) is at least also permitted, if not preferred.

OK.
> remaining 3 digits are the same all the time anyway. I'm not saying 
> this is the best solution, but it is one choice with some clear 
> advantages.

Alas, it is not as simple as that. ISBN-10 and ISBN-13 use different 
algorithms for check sum calculation. ISBN-13 is compatible with EAN 
system, so the check digit had to be a number. ISBN-10 allows X as well. 
If a check digit of ISBN-10 is not recalculated, it is likely that the 
ISBN-13 will not be valid.
> allowed at specific places, and therefore in this specific case, it's 
> a bad idea to try to backtrack and disallow them.

+ 1.
> set of namespaces that an application chooses to apply such rules

OK.

> What do you mean by "incorporates"? My hope is that library metadata 
> as much as possible contains only valid data, and that software used 
> for inputting it checks inputs for validity. But maybe I'm having too 
> high expectations for library metadata, or I don't understand what you 
> mean by "incorporates".

You underestimate the complexity of library metadata ;-).

It is important to provide canceled ISBN since that ISBN is often the 
one printed in the book.

Valid and invalid / canceled ISBNs are tagged differently in MARC 21 
records. In this example

> *020* 	*##**$z*0835200028 :*$c*$10.00
> *020* 	*##**$a*0835200019*$q*pbk.*$c*$2.95/
> /
>

valid ISBN is the one in the second row, and the invalid one is the one 
on the top.  Library systems can tell these ISBN variants apart.

> I understand that mistakes happen, but why would a publisher prefer to 

Because the invalid ISBN may be the one printed in the book.

Unlike some systems in book shops, library systems usually check the 
validity of the ISBN and do not approve a number which is invalid.

> http://www.isbn-check.de/checkisbn.pl?isbn=1-449-32188-9&submit=test&lang=en 

The algorithm used for calculating ISBN check sums is simple. The 
correct check sum can be calculated even manually pretty fast, both for 
ISBN-10 and ISBN-13.

National ISBN centers (often located in national libraries) have other 
means for spotting errors. Large publishers usually get chunks of ISBN 
numbers which they assign for books by themselves. When something goes 
wrong, the ISBN center can check what has happened.

In your example there is nothing wrong, according to the catalogue of 
Der Deutschen Nationalbibliothek. The book just happens to have both 
ISBN-10 and ISBN-13, which have different check sum calculation 
algorithms. In the record retrieved by

https://portal.dnb.de/opac.htm?method=simpleSearch&query=978-1-449-32188-9

neither ISBN has been tagged as invalid: they are in separate tag 020 $a 
occurrences in the metadata record. And according to the rules of the 
ISBN community, it is OK to give a book both ISBN-13 and ISBN-10 (or 
generate automatically one from the another). This is necessary during a 
transitory period since some organizations may still have outdated 
systems which cannot deal with ISBN-13.

> catch some errors, but in no way allow us to fix the errors. In order 
> to be able to fix a single-digit error, we would need at least a 
> two-digit checksum.

OK.
> [Hint: If you check all the alternatives, rule out the ones that are 
> not actually assigned, and then look at the topics of the resulting 
> books, you might find out which book I referred to; it's somewhat 
> related to the topic at hand :-).]

Well, I found only one book: Bad data handbook by Reilly :-).

> My impression is that at least for ISBNs, knowing is impossible. 
> Making an educated guess may be possible for humans (see my example 
> above), but it should not be part of URNs per se. I'm not sure about 
> others, but at least in my book, the concept of permanent, carefully 
> assigned identifier and the concept of "educated guess" don't go very 
> well together.

So you agree that if the identifier is invalid (according to the check 
sum), it is usually OK to stop analysing the lexical equivalence, 
because the result would be meaningless in any case?

The example you chose also indicates that it is important to know 
exactly which identifier assignment rules are being used by a community 
in question. Otherwise it is possible to make wrong conclusions.

ISBN and other semantic identifiers make it easy for humans to spot 
certain kinds of errors (for instance, it is easy to see if there is a 
typo in the publisher code). Machines are rarely capable of this kind of 
analysis because they do not dig deep enough (for instance, they never 
compare the publisher name and the publisher code in ISBN).

All the best,

Juha

-- 

  Juha Hakala
  Senior advisor

  The National Library of Finland
  Library Network Services
  P.O.Box 26 (Teollisuuskatu 23)
  FIN-00014 Helsinki University
  Tel. +358 9 191 44293
  Mobile +358 50 3827678