Subject: Re: [urn] Suggested PWID URN for Persistent Web IDentifiers - version 3

From: Eld Zierau <[email-masked]>
Date: Fri, 10 Aug 2018 10:16:56 +0000

Let me start by apologizing for the late response, which is due to the holiday season.

I would like to make some precisions and explanations in this response, since I think most of the critique is based on misunderstandings - probably because I have not been clear enough. I will of course gladly incorporate in a new version if that can help. 

Below I have included the different responses with my comments in (on line starting with "--"):
1) to Dale R. Worley's mail
2) to Henry S. Thompson's mail
3) to Juha's response to Henry S. Thompsons mail

Best regards, Eld

-----  Comments to Dale R. Worley's mail -----
1) 
Dale R. Worley [email-masked]
Re: [urn] Suggested PWID URN for Persistent Web IDentifiers - version 3
21-07-2018

Comments on this draft:

I am by no means an archivist, so my comments will be largely technical.

The draft seems to me to be prolix, giving a lot of discussion of details that are not reflected in the URN proposal itself.  A careful editing will probably fix this.

A major component of pwid-urn is archive-id.  My assumption is that the archive-id is the top-level component and defines what abstract "archive" the URN is a reference into.  And that the "archive" defines the exact interpretation of archival-tine, coverage-spec, and archived-item.  However, in order for PWID URNs to be unambiguous, it must be unambiguous what "archive" a given archive-id refers to.  The draft suggests "it is recommended to use the web domain as the identifier for the web archive".

What if an institution decides to create PWID URNs that use an archive-id which is a domain name that the institution does not own?  As written, the draft does not forbid *me* from creating PWID URNs with archive-id "netarkivet.dk".
-- Eld: I think there is a misunderstanding here - it is not the archive that chooses the archive name, it is the creator of the reference, - and that is why it is recommended to use the domain name. It would be best to have a formal registry, but I have to start somewhere, - and with the domain name it is possible to find a reference to any existing web archive (restricted access or not). Within the next 5-10 years, I think there will be evidence enough (e.g. from web archives) to reconstruct a registry in case there are web archives changing domains within that period.  I will definitely work on having such a registry formerly established.
-- It could be that it should be stronger here and instead say: 'must use' - instead of 'recommend to use'

In regard to archival-time, the draft states "The 'archival-time' [...] can therefore be specified at any of the levels of granularity as described in [W3CDTF]."  However, W3CDTF gives a number of formats for time designations, and calls those formats "granularities", whereas this draft allows only one format.  I suspect that a more careful description of the meaning of the quoted sentence would fix this problem.
-- Eld: absolutely

The syntax assumes that knowing archive-id, archival-time, coverage-spec, and archived-item (the archived URI) will always be sufficient to specify exactly the resource in question.  However, it seems likely that special situations will arise when two distinct resources will have attributes that are similar enough that the distinction between them is not easily expressed in terms of archive-id, archival-time, coverage-spec, or archived URI.  In this case, additional information is needed to create an exact reference, and the URN syntax needs to have an expansion facility to allow this.
-- Eld: The very nature of web archives is based on information archived URI and archival time - per archive - if there are such cases the web archive would have a problem with their indexes, as these are based exactly on this information. This is also why it is this information that is chosen and why it is important to specify the archive as well. The content-coverage is another story, which I will go more into in my response to Henry S. Thompson's mail

archived-item can be a URI, and a URI can contain the characters '?', '#', '[', and ']', among others.  However, those 4 characters may not appear in the NSS part of a URN.  Note that the first 2 of these characters are used in URIs only to introduce the "query" and "fragment"
parts, but as the draft is written, the archived-item URI is not restricted to not having a query or fragment part.
-- Eld: that is a problem if the archived URI does have fragments

Dale

-----  Comments to Henry S. Thompson's mail -----
2)
Henry S. Thompson <[email-masked]>
Re: [urn] Suggested PWID URN for Persistent Web IDentifiers - version 3
27-07-2018

Eld Zierau writes:

> Please note that I have updated the suggested PWID URN for Persistent 
> Web Identifiers with the following changes:
>
> ...
>
> It is available as version 3 from
> https://datatracker.ietf.org/doc/draft-pwid-urn-specification/, and it 
> is attached to this email as a pdf.

[Before my comments, please note that I don't believe it's best practice to attach drafts in mail to this list -- your original was classified as spam as a result of the attachment, and I only found out about it as a result of Juha's mail.]
-- Eld: Got that

Much of what follows is a critique of the requirements analysis and the presumed use cases, rather than the specifics of the namespace itself, and the overall conclusion is negative.

This is a bit unusual, in that I wouldn't normally bother with an extended critique whose TL;DR would be "It doesn't seem likely to me that this will solve anyone's problems".  After all, if it doesn't solve anyone's problems, no-one will use it, let the 'market' decide, right?

This might be so, for a narrowly focused namespace targeting a specialist community.  But this proposal has _very_ large-scale use in view.
-- Eld: Yes it is aimed big, and there is already a big interest for it - and there are several reasons for this.
-- Let me start by pointing out that I have a background as a computer scientist (from 1989) with a PhD in digital preservation from 2011. With this background, four years ago, I would probably raise some of the same questions.
-- The background for the PWID is a research project, which I undertook with a historian and a literary scientist, and they have much harder requirements for references than what I have been used to in my field. It is this difference that has made it clear to me that there is a lack of ways to reference sources in web archives for digital humanist working with web archives. Please note that the only way to make such a reference today is via URLs, which only works for online web archives. A lot of European web archives are not online, and we are unable to employ such references. At some point, the Danish web archive recommended the 'URL' used in a protected environment addressing the files directly. These references are now useless after we have compressed all the data. Please also note that there is nothing sustainable about a web archive URL - as noted in the draft, the paths are in many cases tools dependent and are therefore very likely to change in 5-10 years.
-- Especially for the humanities there is great focus on the sustainability of references. It must be evident that all referenced sources can be retrieved later on in order to re-evaluate the findings (for more than a 5-10 year horizon). As many studies shows (referenced in the draft) there is a great link rot just over a short number of years. My personal experience is that 15 % of the referenced links in my PhD (from 2011) are now dead. What we found in our studies was that within the literary field, there were quite a few works that did not reference the web at all - we suspect that this could point at a tendency that they do not want to use the insecure web references - which in practice can mean that the internet might not be the subject of relevant research.
-- A present need for the URN PWID is for definition of web corpora. A definition of a corpora (web collection) is for example the European initiative to select a part of the archived web for each year - cleaned for duplicates etc. - in order to have a basis for statistical measurements. In order to be able to do recalculation of the measurements and look into the basis for the measurements, it will be required that you somehow preserve a representation of it. It will be too expensive to preserve the data separately because of the volume, therefore it must be references that is saved. There have been ideas about saving CDX indexes, but these are not standard either. Therefore, specification of PWID URNs as references is a good choice.
-- This leads to the use of PWID as a general form, if you have your resource archived or find the resource in an archive: Either by getting a possibility to refer to the large volumes of material in European web archives (with restricted access) or by getting a sustainable technology agnostic way to address resources that have been archived for the purpose of creating a persistent reference.
-- I hope I with this have made it clear that this is NOT just narrowly focused namespace, but will be a huge help especially for any researcher that wants to make acceptable research in web archive materials.

_And_, the legitimacy of discourse around various aspects of 'persistent' reference has been so compromised by flawed analyses and failed solutions that there is actually real reputational risk for the urn infrastructure itself from yet another example, which I'm afraid I think this is, particularly given its very high aims.
-- Eld: I hope that the above gives some clarification on this, - and about the infrastructure: this only relies on the definition, - anyone can build resolvers at any point of time (as long as we have the history of domains, which I will come back to). The reason is that the PWID is based on the very basic elements from registering a "located" thing in an archive: the archive, the location, and the time of archiving. Therefore, with the right indexes and grants, a resource can be found based on this information.
-- I suspect that I may have confused things by both talking about the textual and the technical URN PWID. It was meant as a way to explain this, but that can be changed if it is confusing things instead.

So, I'm sorry but for these reasons, and the many specific problems discussed below, I don't personally see a future for this document.

============

Specific comments, by section

*Purpose*

As noted above, the goal stated in the open paragraph is very ambitious.

The only parts of the argument that follows that I can make sense of, as to why the proposed scheme is needed, consist of two mistaken and/or unsupported high-level claims:

 1) Citations of web-hosted resources should refer to archives, not to
    the resources themselves;
-- Eld: I have included several references to conference papers in the draft to support this claim, - I will of course gladly extend this if it is needed

 2) Citation of archived material needs help to be sufficiently 'precise'.
-- Eld: There are also references to support this - mainly from the research group that I refer to above - but in these papers there are also other references - I will of course also gladly extend this if it is needed

The first is at best contentious as a general claim, and even in cases where an archive reference is preferred, why packaging it in a PWID urn will add value is not explained.  After all, if I know that what I want to cite is available in some archive, it's because I've found it there, in which case I already _know_ the URI I can use to retrieve it.  More on this when below when we come to the question of uniqueness.
-- Eld: This is due to the fact that an URI to a web archive is not persistent - and only existing for online web archives. I thought I made it clear, but I can of course elaborate on this as well.

The second surely applies to any kind of citation at all, and seems to depend on a confusion between web identity, media type and the ontology of citable things and loci associated with them -- all of FRBR lurks just around the corner here.
-- Eld: The precision is needed because references to web materials are very imprecise. Especially within the humanities, you need to specify what you refer to and thereby also what the author has cheked that the  specific reference points to. There is a big difference between referencing a web page or a website or just the html for the web page. The reason that this is part of the PWID URN is to have all the information there and to make it possible (later on) for the web archives to supply services that correspond to the reference

Just because some representation is hosted on the Web does not make it easier (or harder) than it ever was to make clear in a citation what part/aspect/property of what you _name_ in a citation is what you actual are _referring to_.  More on this below when we come to the 'coverage-spec'.
------------
One more specific issue with this section

  "The precision regards both regards precise reference where there
   can be no doubt about that you have the correct web material as
   well as precision about what is actually referred by the reference
   (e.g. is it the page or the whole website)"

There's nothing in the proposal which follows to back up the "you have the correct web material" claim, and the subsequent inventory of 8 'type[s] of archived item[s]' is clearly not sufficient to unambiguously determine "what is actually referred [to] by the reference".
-- Eld: I just realize that there is a context missing and precision is needed. It relates to the way Memento argues for URL and date being sufficient information in order to identify a web resource, i.e. that it does not matter which archive you get the resource from as long as it is approximately from that time. That should definitely be more clear. I have numerous examples of differences in archived material of the same URL around the same time, where a reference can make sense in one case but not the other (typically from often updated news sites, or due to bad harvest settings or warnings requiring additional information before entering the site). A way to make the precision could be to say that it is precise about which archive it was evaluated in.
-- I agree that you cannot be sure of your reference as long as it is not entirely based on parts, since it will be up to e.g. Wayback software to determine each of the parts (usually the closest in time, which may change if included materials from other archives or harvest parts that did not originally exist in the archive). I gladly include description of this as well.

*Syntax*

I see no problems in the ABNF.
------------
The definitions of the 'coverage-spec' values are hopelessly underspecified, and heterogenous.  The 'part'/'page' distinction is particularly unclear/non-obvious/ambiguous/medai-type dependent.
-- Eld: hopefully it is more clear from the above. - the point is to make it possible to present the resource as reference - or for a user of the reference to make the right choices in order to extract the referred information. This means in case of 'part' for html, asp, jpg . it will be the actual source of the archived file, in case of web page it will be the "Wayback" software version (and actually the same for subsite and site). Some archives have possibility for choosing between snapshots or harvested web files, which is where the snapshot comes in. A comment from the iPRES 2013 conference was that it was missing web recordings, which is why recording is in, and finally there is a need for addressing collection (corpuses) also in order to make a hierarchi of collections (e.g. each year of the Danish example corpuses, another containing those years, a third containing the Danish years and some years from UK web archive etc.). if it is useful I gladly include a modified version of this explanation also.

*Assignment*

The initial claim here and its subsequent gloss taken together don't hold up:

 "The PWID URNs does not have to be assigned by an authority, as they
  are based on the information created at the time of archiving:"

 "In other words: the PWID URNs are created independently, but
  following an algorithm that itself guarantees uniqueness."

On a strict reading of "uniqueness" (i.e. a one-to-one relation between items and PWIDs), this amounts to a claim that _any_ 3rd party considering _any_ item in _any_ archive will always construct the _same_ PWID.  This reading is obviously false: the presence of the two "+( unreserved )" expansions in the ABNF amount to a _guarantee_ that there will be multiple distinct PWIDs for the same item.
-- Eld: I never meant to say that it was a two-ways uniqueness, and I can see it can be read as that, so that should be clarified. For example, if internet archive exchange their domain name with ai.org then future PWIDs with ai.org as archive identifier will point to the same as archive.org for the same archived items (which covers the first "+( unreserved )". There may also be rare cases with the other one, which represents a generalization (to include e.g. collections specified by an identifier) - and yes in some cases an archive have more identifiers for the same thing. But as you say there should be no problem with that.
-- If the assignment is confusing I can also expand that, to clarify that it is the basic metadata for archiving (archive, got from location, at time) combined with precision of what is referenced.

RFC8141 does not require one-to-one relations between URNs, even URNs within the same namespace.  But it does require, in the case of "URNs . . . created independently" that they be created "following an algorithm that itself guarantees uniqueness".  The underspecification of the 'coverage-spec' already alluded to above makes independent _and_ inconsistently understood coinages of identical PWIDs not just possible but likely.  Consider for example

urn:pwid:web.archive.org:2018-01-01T17:03:53Z:part:http://www.ltg.ed.ac.uk/~ht/

One person might coin this to mean the text/html character sequence which was served from http://www.ltg.ed.ac.uk/~ht/ (my homepage at the University of Edinburgh) at the beginning of 2018, whereas someone else might coin it to mean the text/html character sequence you get from the Web Archive today if you do an http GET on https://web.archive.org/web/20180101170353/http://www.ltg.ed.ac.uk/~ht/,
which are different in important ways.
--  Eld: This is only about archived materials, with the limitations that web archiving has. It is only meant to mean the latter case, which can be made more precise, of course

In general, it seems that knowing whether to use 'part' or 'page' would depend on a detailed understanding of the media type of the retrieved representation, which the average creator of a citation is unlikely to have.
-- Eld: This referencing mechanism is meant for users that verify their source before referring to it, - it is already part of standards to make clear whether it is page or site, so I would not see any problems with that. I can expand the explanation of part with several examples to make clear that that should not be an option if you mean page or site. For the rest I would assume that you would know if you were using it.
--  It is also meant as a future basis for making tools to respect and provide what the reference is covering - so you do not need to know that you have to select snapshot, - or use a get for requesting the source of a page, Also to have tools for corpora where can be made clear when a reference is to a subcorpora. The inspiration for this way of doing it came from the Internet Archives use of function e.g. the Identity Wayback function which is called by placing 'id_' after the <date> in the archive url  as for example https://web.archive.org/web/20180101170353id_/http://www.ltg.ed.ac.uk/~ht/ (different functions described on https:/en.wikipedia.org/wiki/Help:Using_the_Wayback_Machine). 

-------------
Some of this is corrigible, but only at the expense of a vast increase in detail and clarity.  Others not-so-much, insofar as so much is _necessarily_ left underspecified, to allow for _anything_ to be considered as an archive ('archive-id' value), with the consequent requirement to allow for arbitrarily idiosyncratic internal naming mechanisms ('archived-item' value).
-- Eld: It should be clarified that this is only in case an identifier can be found in the archive - actually it should probably also be mentioned under assignment that in these cases, it is identifiers assigned to objects by the archive.

I'm curious in this connection to wonder if we actually have any URN namespaces where assignment is done "independently but following an algorithm that itself guarantees uniqueness"?  Ah, there are registrations for urn:uuid:, which qualifies, and urn:oid:, which sort of does.  OK, is there such a namespace in active use?  Which provides for resolution, at least in principle?
-- Eld: we use UUIDs for identifiers for our library material, and we use WARC as packaging format, therefore it is essential for us to use the urn:uuid: since WARC only accepts URIs as identifiers

A crucial point here is that in the OID case a urn:oid:... carries a chain of responsibility with it.  If you run across one, you know how to find out who takes responsibility for every level in the tree that's involved in interpreting it.  In the UUID case, a urn:uuid: doesn't travel well/at all, so the question doesn't arise.  But for PWIDs, if I find one I have _no way_ to figure out who's responsible, or whom I can ask for clarification, or whom to blame if it doesn't appear to 'work'.
-- Eld: I am not sure I understand this - or what is causing a possible misunderstanding. As said before, anyone can make a resolver for open web archives and any archive with restricted access can make their own (only having the challenge with archive identifiers as mentioned later). You can take the PWID and "manually use it to locate you resource. If the person making the reference made mistakes or if the archive lost its resources, I don't think it is the PWID that should be blame - which I see a direct parallel to for any referencing systems - digital or analogue.
----------------
I don't see what the intended value of the discussion of the "SOLR-Wayback tool" is for the spec.
-- Eld: It is to say that you are not on a huge job if you want to collect all parts for a web page, - there actually exists tools to help you. If it is not appropriate to have it here, I can take it out.

*Interoperability*

The above comments about ambiguity/lack of identity apply here too: if different implementors interpret e.g. a coverage-spec of 'subsite'
differently, their resolvers will not interoperate.
-- Eld: I can make a more clear definition if wanted, - admittedly, I have not gone into depth with this, since (as far as I know) there are no tooling for this yet, but it is a point that it should be more precise, - also to make clear what was validated when making the reference.

*Resolution*

None of this is very much to the point.  In particular, it completely ignores the precedents set by doi.org, identifiers.org and n2t.net
(https://doi.org/10.1038/sdata.2018.95 exemplifies the first and its content discusses the second and third). 
-- Eld: I don't understand this, - as far as I know these are dealing with registered identifiers. There are no way that all web materials will have a registered identifier, - the volume is too big. That is the reason why the PWID is needed - . maybe I have misunderstood some things here.

==============

In conclusion, I think a useful way to think about this proposal is as a misguided attempt to define a urn type which _is_ a citation.  This makes sense of PWID's 3rd party nature, and clarifies that the archive-id and -time, the embedded URI and the coverage spec are just a pretty arbitrary, certainly impoverished, subset of what you would expect to find in a _real_ citation.  Trying to pack a citation into a URN just doesn't make sense to me. Neither does trying to determine exactly the _right_ subset of citations of web-hosted resources to pack up in order to be both useful and unambiguous.
-- Eld: I hope I have made it more clear in these comments

(There is work underway in various venues (see e.g. [1]) to _objectify_ citations and give PIDs to _those_, which might address at least some of the goals of this work.)

ht

[1] http://sched.co/DJ3P
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/  [mail from me _always_ has a .sig like this -- mail without it is forged spam]

-----  Comments to Juha Hakala's mail -----
3)
Hakala, Juha E [email-masked]
RE: [urn] Suggested PWID URN for Persistent Web IDentifiers - version 3
27-07-2018

Hello, 

a few comments below. 

-----Original Message-----
From: urn <[email-masked]> On Behalf Of Henry S. Thompson
Sent: perjantai 27. heinäkuuta 2018 11.58
To: Eld Zierau <[email-masked]>
Cc: [email-masked]
Subject: Re: [urn] Suggested PWID URN for Persistent Web IDentifiers - version 3

The only parts of the argument that follows that I can make sense of, as to why the proposed scheme is needed, consist of two mistaken and/or unsupported high-level claims:

 1) Citations of web-hosted resources should refer to archives, not to
    the resources themselves;

 2) Citation of archived material needs help to be sufficiently 'precise'.

The first is at best contentious as a general claim, and even in cases where an archive reference is preferred, why packaging it in a PWID urn will add value is not explained.  After all, if I know that what I want to cite is available in some archive, it's because I've found it there, in which case I already _know_ the URI I can use to retrieve it.  More on this when below when we come to the question of uniqueness.

Juha: ISO TC 46/SC 9 is currently revising ISO 690 (Guidelines for bibliographic references and citations to information resources). One of the challenges is how to deal with reference rot, which consists of link rot (404) and content drift (retrieved documents change). A large percentage of all Internet resources suffer from these problems. 

Eld has been most helpful in writing the Annex that discusses the usage of Web archives in detail, and the current version is based on what she has written, with two important exceptions: 

- I don't think that referring just to a copy of the cited resource in a Web archive is always the best solution. Archive reference only must be used when the resource is dynamic (say, https://www.ed.ac.uk/) and a specific version is cited.  But when the page is supposed to be persistent (e.g. https://www.w3.org/Provider/Style/URI) the added value provided by the archive link may be modest, because the archived copy may not always be exactly the same, and web archives themselves may not last forever. The current draft of ISO 690 (which is still only a working draft) does promote the use of web archives, but not as the only solution. And if web archives are used, utilization of Memento is something that should be considered since it provides some level of protection to disappearance of web archives.  

- the current ISO 690 draft does not mention PWID. There are three reasons for this. First, and most importantly, PWID does not (yet) have a registered URN namespace. An ISO standard cannot refer to PWIDs as long as namespace registration has not been completed. And IMO it is not obvious that it will be completed. Second, every persistent identifier that will be included in the examples or elsewhere in the standard must be actionable (expressed, for the time being, as HTTP URI). So all DOIs that were in the form doi:<DOI string> have been converted into https://doi.org/<DOI string>. This will not possible with PWIDs, since only the Danish web archive is planning to implement PWIDs, in an archive which has strict access limitations. 

When you consider what to do with PWID, keep in mind that as a national library the Royal Danish Library can always use the NBN namespace for items in the web archive. For instance, something like urn:nbn:pwid:<pwid> can be constructed, as long as the <pwid> string conforms to the syntactical requirements of the NBN namespace.    

Henry: The second surely applies to any kind of citation at all, and seems to depend on a confusion between web identity, media type and the ontology of citable things and loci associated with them -- all of FRBR lurks just around the corner here.

Just because some representation is hosted on the Web does not make it easier (or harder) than it ever was to make clear in a citation what part/aspect/property of what you _name_ in a citation is what you actual are _referring to_.  More on this below when we come to the 'coverage-spec'.

Juha:  trying to express in PWID coverage-spec which part of the of the resource has been cited may be more difficult that providing this information elsewhere in the citation. ISO 690 draft specifies such an alternative approach.  And even if PWID really has to be used for this purpose, usage of f-, r- and q-components may also be considered as an alternative. 

As Henry says below, coverage-spec would need a lot more alternatives to be truly useful. Coverage-spec "other" does not add much value, and it would be necessary to use it for every image, video, etc. And if there are several images on the page, how to indicate the right one (except with f-component)?
-- Eld: Thank you Juha - just for this last remark, I am not sure I understand it . hopefully the role of coverage-spec is more clear from my response  to Henry S. Thompson, otherwise please tell me. For the case of two images, I would say that you would have to get a reference for the exact image part of the page, otherwise you have a risk that a totally different image is displayed because of update to the archives data.

Best, 

Juha

------------
One more specific issue with this section

  "The precision regards both regards precise reference where there
   can be no doubt about that you have the correct web material as
   well as precision about what is actually referred by the reference
   (e.g. is it the page or the whole website)"

There's nothing in the proposal which follows to back up the "you have the correct web material" claim, and the subsequent inventory of 8 'type[s] of archived item[s]' is clearly not sufficient to unambiguously determine "what is actually referred [to] by the reference".

*Syntax*

I see no problems in the ABNF.
------------
The definitions of the 'coverage-spec' values are hopelessly underspecified, and heterogenous.  The 'part'/'page' distinction is particularly unclear/non-obvious/ambiguous/medai-type dependent.

*Assignment*

The initial claim here and its subsequent gloss taken together don't hold up:

 "The PWID URNs does not have to be assigned by an authority, as they
  are based on the information created at the time of archiving:"

 "In other words: the PWID URNs are created independently, but
  following an algorithm that itself guarantees uniqueness."

On a strict reading of "uniqueness" (i.e. a one-to-one relation between items and PWIDs), this amounts to a claim that _any_ 3rd party considering _any_ item in _any_ archive will always construct the _same_ PWID.  This reading is obviously false: the presence of the the two "+( unreserved )" expansions in the ABNF amount to a _guarantee_ that there will be multiple distinct PWIDs for the same item.

RFC8141 does not require one-to-one relations between URNs, even URNs within the same namespace.  But it does require, in the case of "URNs .. . . created independently" that they be created "following an algorithm that itself guarantees uniqueness".  The underspecification of the 'coverage-spec' already alluded to above makes independent _and_ inconsistently understood coinages of identical PWIDs not just possible but likely.  Consider for example

urn:pwid:web.archive.org:2018-01-01T17:03:53Z:part:http://www.ltg.ed.ac.uk/~ht/

One person might coin this to mean the text/html character sequence which was served from http://www.ltg.ed.ac.uk/~ht/ (my homepage at the University of Edinburgh) at the beginning of 2018, whereas someone else might coin it to mean the text/html character sequence you get from the Web Archive today if you do an http GET on https://web.archive.org/web/20180101170353/http://www.ltg.ed.ac.uk/~ht/,
which are different in important ways.

In general, it seems that knowing whether to use 'part' or 'page' would depend on a detailed understanding of the media type of the retrieved representation, which the average creator of a citation is unlikely to have.
-------------
Some of this is corrigible, but only at the expense of a vast increase in detail and clarity.  Others not-so-much, insofar as so much is _necessarily_ left underspecified, to allow for _anything_ to be considered as an archive ('archive-id' value), with the consequent requirement to allow for arbitrarily idiosyncratic internal naming mechanisms ('archived-item' value).

I'm curious in this connection to wonder if we actually have any URN namespaces where assignment is done "independently but following an algorithm that itself guarantees uniqueness"?  Ah, there are registrations for urn:uuid:, which qualifies, and urn:oid:, which sort of does.  OK, is there such a namespace in active use?  Which provides for resolution, at least in principle?

A crucial point here is that in the OID case a urn:oid:... carries a chain of responsibility with it.  If you run across one, you know how to find out who takes responsibility for every level in the tree that's involved in interpreting it.  In the UUID case, a urn:uuid: doesn't travel well/at all, so the question doesn't arise.  But for PWIDs, if I find one I have _no way_ to figure out who's responsible, or whom I can ask for clarification, or whom to blame if it doesn't appear to 'work'.
----------------
I don't see what the intended value of the discussion of the "SOLR-Wayback tool" is for the spec.

*Interoperability*

The above comments about ambiguity/lack of identity apply here too: if different implementors interpret e.g. a coverage-spec of 'subsite'
differently, their resolvers will not interoperate.

*Resolution*

None of this is very much to the point.  In particular, it completely ignores the precedents set by doi.org, identifiers.org and n2t.net
(https://doi.org/10.1038/sdata.2018.95 exemplifies the first and its content discusses the second and third). 

==============

In conclusion, I think a useful way to think about this proposal is as a misguided attempt to define a urn type which _is_ a citation.  This makes sense of PWID's 3rd party nature, and clarifies that the archive-id and -time, the embedded URI and the coverage spec are just a pretty arbitrary, certainly impoverished, subset of what you would expect to find in a _real_ citation.  Trying to pack a citation into a URN just doesn't make sense to me. Neither does trying to determine exactly the _right_ subset of citations of web-hosted resources to pack up in order to be both useful and unambiguous.

(There is work underway in various venues (see e.g. [1]) to _objectify_ citations and give PIDs to _those_, which might address at least some of the goals of this work.)

ht

[1] http://sched.co/DJ3P
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/  [mail from me _always_ has a .sig like this -- mail without it is forged spam]

_______________________________________________
urn mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/urn

-----Original Message-----
From: Hakala, Juha E <[email-masked]> 
Sent: Friday, July 27, 2018 1:19 PM
To: Henry S. Thompson <[email-masked]>; Eld Zierau <[email-masked]>
Cc: [email-masked]
Subject: RE: [urn] Suggested PWID URN for Persistent Web IDentifiers - version 3

Hello, 

a few comments below. 

-----Original Message-----
From: urn <[email-masked]> On Behalf Of Henry S. Thompson
Sent: perjantai 27. heinäkuuta 2018 11.58
To: Eld Zierau <[email-masked]>
Cc: [email-masked]
Subject: Re: [urn] Suggested PWID URN for Persistent Web IDentifiers - version 3

The only parts of the argument that follows that I can make sense of, as to why the proposed scheme is needed, consist of two mistaken and/or unsupported high-level claims:

 1) Citations of web-hosted resources should refer to archives, not to
    the resources themselves;

 2) Citation of archived material needs help to be sufficiently 'precise'.

The first is at best contentious as a general claim, and even in cases where an archive reference is preferred, why packaging it in a PWID urn will add value is not explained.  After all, if I know that what I want to cite is available in some archive, it's because I've found it there, in which case I already _know_ the URI I can use to retrieve it.  More on this when below when we come to the question of uniqueness.

Juha: ISO TC 46/SC 9 is currently revising ISO 690 (Guidelines for bibliographic references and citations to information resources). One of the challenges is how to deal with reference rot, which consists of link rot (404) and content drift (retrieved documents change). A large percentage of all Internet resources suffer from these problems. 

Eld has been most helpful in writing the Annex that discusses the usage of Web archives in detail, and the current version is based on what she has written, with two important exceptions: 

- I don't think that referring just to a copy of the cited resource in a Web archive is always the best solution. Archive reference only must be used when the resource is dynamic (say, https://www.ed.ac.uk/) and a specific version is cited.  But when the page is supposed to be persistent (e.g. https://www.w3.org/Provider/Style/URI) the added value provided by the archive link may be modest, because the archived copy may not always be exactly the same, and web archives themselves may not last forever. The current draft of ISO 690 (which is still only a working draft) does promote the use of web archives, but not as the only solution. And if web archives are used, utilization of Memento is something that should be considered since it provides some level of protection to disappearance of web archives.  

- the current ISO 690 draft does not mention PWID. There are three reasons for this. First, and most importantly, PWID does not (yet) have a registered URN namespace. An ISO standard cannot refer to PWIDs as long as namespace registration has not been completed. And IMO it is not obvious that it will be completed. Second, every persistent identifier that will be included in the examples or elsewhere in the standard must be actionable (expressed, for the time being, as HTTP URI). So all DOIs that were in the form doi:<DOI string> have been converted into https://doi.org/<DOI string>. This will not possible with PWIDs, since only the Danish web archive is planning to implement PWIDs, in an archive which has strict access limitations. 

When you consider what to do with PWID, keep in mind that as a national library the Royal Danish Library can always use the NBN namespace for items in the web archive. For instance, something like urn:nbn:pwid:<pwid> can be constructed, as long as the <pwid> string conforms to the syntactical requirements of the NBN namespace.    

Henry: The second surely applies to any kind of citation at all, and seems to depend on a confusion between web identity, media type and the ontology of citable things and loci associated with them -- all of FRBR lurks just around the corner here.

Just because some representation is hosted on the Web does not make it easier (or harder) than it ever was to make clear in a citation what part/aspect/property of what you _name_ in a citation is what you actual are _referring to_.  More on this below when we come to the 'coverage-spec'.

Juha:  trying to express in PWID coverage-spec which part of the of the resource has been cited may be more difficult that providing this information elsewhere in the citation. ISO 690 draft specifies such an alternative approach.  And even if PWID really has to be used for this purpose, usage of f-, r- and q-components may also be considered as an alternative. 

As Henry says below, coverage-spec would need a lot more alternatives to be truly useful. Coverage-spec "other" does not add much value, and it would be necessary to use it for every image, video, etc. And if there are several images on the page, how to indicate the right one (except with f-component)? 

Best, 

Juha

------------
One more specific issue with this section

  "The precision regards both regards precise reference where there
   can be no doubt about that you have the correct web material as
   well as precision about what is actually referred by the reference
   (e.g. is it the page or the whole website)"

There's nothing in the proposal which follows to back up the "you have the correct web material" claim, and the subsequent inventory of 8 'type[s] of archived item[s]' is clearly not sufficient to unambiguously determine "what is actually referred [to] by the reference".

*Syntax*

I see no problems in the ABNF.
------------
The definitions of the 'coverage-spec' values are hopelessly underspecified, and heterogenous.  The 'part'/'page' distinction is particularly unclear/non-obvious/ambiguous/medai-type dependent.

*Assignment*

The initial claim here and its subsequent gloss taken together don't hold up:

 "The PWID URNs does not have to be assigned by an authority, as they
  are based on the information created at the time of archiving:"

 "In other words: the PWID URNs are created independently, but
  following an algorithm that itself guarantees uniqueness."

On a strict reading of "uniqueness" (i.e. a one-to-one relation between items and PWIDs), this amounts to a claim that _any_ 3rd party considering _any_ item in _any_ archive will always construct the _same_ PWID.  This reading is obviously false: the presence of the the two "+( unreserved )" expansions in the ABNF amount to a _guarantee_ that there will be multiple distinct PWIDs for the same item.

RFC8141 does not require one-to-one relations between URNs, even URNs within the same namespace.  But it does require, in the case of "URNs .. . . created independently" that they be created "following an algorithm that itself guarantees uniqueness".  The underspecification of the 'coverage-spec' already alluded to above makes independent _and_ inconsistently understood coinages of identical PWIDs not just possible but likely.  Consider for example

urn:pwid:web.archive.org:2018-01-01T17:03:53Z:part:http://www.ltg.ed.ac.uk/~ht/

One person might coin this to mean the text/html character sequence which was served from http://www.ltg.ed.ac.uk/~ht/ (my homepage at the University of Edinburgh) at the beginning of 2018, whereas someone else might coin it to mean the text/html character sequence you get from the Web Archive today if you do an http GET on https://web.archive.org/web/20180101170353/http://www.ltg.ed.ac.uk/~ht/,
which are different in important ways.

In general, it seems that knowing whether to use 'part' or 'page' would depend on a detailed understanding of the media type of the retrieved representation, which the average creator of a citation is unlikely to have.
-------------
Some of this is corrigible, but only at the expense of a vast increase in detail and clarity.  Others not-so-much, insofar as so much is _necessarily_ left underspecified, to allow for _anything_ to be considered as an archive ('archive-id' value), with the consequent requirement to allow for arbitrarily idiosyncratic internal naming mechanisms ('archived-item' value).

I'm curious in this connection to wonder if we actually have any URN namespaces where assignment is done "independently but following an algorithm that itself guarantees uniqueness"?  Ah, there are registrations for urn:uuid:, which qualifies, and urn:oid:, which sort of does.  OK, is there such a namespace in active use?  Which provides for resolution, at least in principle?

A crucial point here is that in the OID case a urn:oid:... carries a chain of responsibility with it.  If you run across one, you know how to find out who takes responsibility for every level in the tree that's involved in interpreting it.  In the UUID case, a urn:uuid: doesn't travel well/at all, so the question doesn't arise.  But for PWIDs, if I find one I have _no way_ to figure out who's responsible, or whom I can ask for clarification, or whom to blame if it doesn't appear to 'work'.
----------------
I don't see what the intended value of the discussion of the "SOLR-Wayback tool" is for the spec.

*Interoperability*

The above comments about ambiguity/lack of identity apply here too: if different implementors interpret e.g. a coverage-spec of 'subsite'
differently, their resolvers will not interoperate.

*Resolution*

None of this is very much to the point.  In particular, it completely ignores the precedents set by doi.org, identifiers.org and n2t.net
(https://doi.org/10.1038/sdata.2018.95 exemplifies the first and its content discusses the second and third). 

==============

In conclusion, I think a useful way to think about this proposal is as a misguided attempt to define a urn type which _is_ a citation.  This makes sense of PWID's 3rd party nature, and clarifies that the archive-id and -time, the embedded URI and the coverage spec are just a pretty arbitrary, certainly impoverished, subset of what you would expect to find in a _real_ citation.  Trying to pack a citation into a URN just doesn't make sense to me. Neither does trying to determine exactly the _right_ subset of citations of web-hosted resources to pack up in order to be both useful and unambiguous.

(There is work underway in various venues (see e.g. [1]) to _objectify_ citations and give PIDs to _those_, which might address at least some of the goals of this work.)

ht

[1] http://sched.co/DJ3P
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/  [mail from me _always_ has a .sig like this -- mail without it is forged spam]

_______________________________________________
urn mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/urn

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 10 Aug 2018 21:54:43 -0400

Eld Zierau <[email-masked]> writes:
> Dale R. Worley [email-masked] writes:
> > A major component of pwid-urn is archive-id.  My assumption is that
> > the archive-id is the top-level component and defines what abstract
> > "archive" the URN is a reference into.  And that the "archive"
> > defines the exact interpretation of archival-tine, coverage-spec,
> > and archived-item.  However, in order for PWID URNs to be
> > unambiguous, it must be unambiguous what "archive" a given
> > archive-id refers to.  The draft suggests "it is recommended to use
> > the web domain as the identifier for the web archive".
> >
> > What if an institution decides to create PWID URNs that use an
> > archive-id which is a domain name that the institution does not own?
> > As written, the draft does not forbid *me* from creating PWID URNs
> > with archive-id "netarkivet.dk".
> 
> -- Eld: I think there is a misunderstanding here - it is not the
> archive that chooses the archive name, it is the creator of the
> reference, - and that is why it is recommended to use the domain
> name.

If you are forced to say "it is not the archive that chooses the archive
name", then I suggest that the "archive name" is not actually the name
*of the archive*, but an identifier of some other entity.

When you say "the domain name" here, *of what* is it the domain name?
Do you mean "the archive's domain name", or "the domain name of the
creator of the reference"?

> It would be best to have a formal registry, but I have to start
> somewhere, - and with the domain name it is possible to find a
> reference to any existing web archive (restricted access or
> not). Within the next 5-10 years, I think there will be evidence
> enough (e.g. from web archives) to reconstruct a registry in case
> there are web archives changing domains within that period.  I will
> definitely work on having such a registry formerly established.

You seem to be describing how you expect these URNs to be processed, but
it is not clear to me what the process is.  Can you describe how this
process works, starting with "I am looking at a PWID URN" and continuing
to the point where "I have discovered the particular archive in
question".  (I assume that after that point, looking up the resource in
the archive's index is straightforward, at least conceptually.)

> > archived-item can be a URI, and a URI can contain the characters '?',
> > '#', '[', and ']', among others.  However, those 4 characters may not
> > appear in the NSS part of a URN.  Note that the first 2 of these
> > characters are used in URIs only to introduce the "query" and
> > "fragment" parts, but as the draft is written, the archived-item URI
> > is not restricted to not having a query or fragment part.

> -- Eld: that is a problem if the archived URI does have fragments

Yes, it is a problem.  But your specification allows the archived URI to
have fragments, because it refers to URIs but does not forbid them to
have fragment parts.  How do you propose to resolve this contradiction?

Dale

---

From: Eld Zierau <[email-masked]>
Date: Mon, 13 Aug 2018 10:44:07 +0000

See my comments below - (marked with "--")
Best regards, Eld

-----Original Message-----
From: Dale R. Worley <[email-masked]> 
Sent: Saturday, August 11, 2018 3:55 AM
To: Eld Zierau <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [urn] Suggested PWID URN for Persistent Web IDentifiers - version 3

Eld Zierau <[email-masked]> writes:
> reference, - and that is why it is recommended to use the domain name.

If you are forced to say "it is not the archive that chooses the archive name", then I suggest that the "archive name" is not actually the name *of the archive*, but an identifier of some other entity.
-- Eld: If it says 'archive name' anywhere, it should be corrected - it is an identifier we are taking about.

When you say "the domain name" here, *of what* is it the domain name?
Do you mean "the archive's domain name", or "the domain name of the creator of the reference"?
-- Eld: the domain name of the archive - it is to identify in which archive you can find the resource in - I actually realized that it should say top level domain name of the web archive. 

> reference to any existing web archive (restricted access or not). 
> Within the next 5-10 years, I think there will be evidence enough 
> (e.g. from web archives) to reconstruct a registry in case there are 
> web archives changing domains within that period.  I will definitely 
> work on having such a registry formerly established.

You seem to be describing how you expect these URNs to be processed, but it is not clear to me what the process is.  Can you describe how this process works, starting with "I am looking at a PWID URN" and continuing to the point where "I have discovered the particular archive in question".  (I assume that after that point, looking up the resource in the archive's index is straightforward, at least conceptually.)
-- Eld: It is indirectly described under resolution, - but it is a good point that there should be a construction as well. This would be:
--  PWID URNs can be constructed in different ways. In case a single reference is wanted, it can be created by having/finding/creating the wanted archived resource and then construct the PWID URN from that. Or it can be created for a collection by using tooling producing PWIDs.
-- In the first case, there can be three situations where the creator of the reference:
--  * already has the resource from a web archive that the creator wants to reference
--  * finds resource from a web archive by using tools like Memento and choses the exact web resource to reference from a specific archive
--  * creates a resource by asking a web archive/citation service to archive a live URI, and afterwards verifies the reference in the web archive
-- In these three cases the PWID URN is constructed by filling out the PWID URN pattern 
--   urn:pwid:<archive-id>:<archival-time>:<coverage-spec>:<archived-uri>
-- By setting 
-- The <archive-id> to the top level domain of the web archive holding the reference. For open web archives like Internet Archive's publicly available web archive collection (with current wayback on https://web.archive.org/) the top level domain is archive.org. For  web archives with restricted access, choose the top level domain for where the web archive is publicly described, e.g. the Danish web archive is described on http://netarkivet.dk/in-english/  which has the top level domain netarkivet.dk.
-- The <archival-time> (on the form described in the syntax and given as UTC time) to the archival time for the selected web archive resource. In some wayback solutions, this archival time can be found in the online URI for the resource, e.g. in the online URL for an archived version of https://tools.ietf.org/html/rfc3986: https://web.archive.org/web/20180418152929/https://tools.ietf.org/html/rfc3986, the archival time is represented in the long number 20180418152929, where year is the first four digits, month the next two digits etc. In other Wayback solutions or web archive interfaces the archival date is given as part of the metadata for the web resource.
-- The <coverage-spec> to what the creator of the reference wants the reference to cover, e.g. page or subsite.
The <archived-uri> to the URI that is archived, which means what the web archive has registered as the archived URI. For URIs that still exists online, the online URI and the archive URI will usually be the same. However, there are case (e.g. for redirects) where they differ. For instance for archive.is archived in archive.org on 2018-01-06T14:08:01 UTC is archived wiith the archived URI:  http://archive.is:80.
-- An example of constructing an PWID URI for https://web.archive.org/web/20180418152929/https://tools.ietf.org/html/rfc3986 would therefore be
-- <archive-id> = archive.org (to level domain of web.archive.org)
-- <archival-time> = 2018-04-18T15:29:29Z (deduced from 20180418152929 which is always UTC time)
-- <coverage-spec> = webpage (depends on style sheets references. In case the creator of the reference wants to indicate that the contents is covered fully on this reference without dependencies to calculation of dependencies, he/she could choose to say part, - although this may result in a resolution of the html depending on the resolution tool)
-- <archived-uri> = https://tools.ietf.org/html/rfc3986
-- Which results in the PWID URN: 
--  urn:pwid:archive.org:2018-04-18T15:29:29Z:webpage:https://tools.ietf.org/html/rfc3986

> > archived-item can be a URI, and a URI can contain the characters 
> > '?', '#', '[', and ']', among others.  However, those 4 characters 
> > may not appear in the NSS part of a URN.  Note that the first 2 of 
> > these characters are used in URIs only to introduce the "query" and 

> -- Eld: that is a problem if the archived URI does have fragments

Yes, it is a problem.  But your specification allows the archived URI to have fragments, because it refers to URIs but does not forbid them to have fragment parts.  How do you propose to resolve this contradiction?
-- Eld: I will certainly make the reader aware of it, e.g. by writing:
-- "It should be noted that a valid URN is not allowed to contain the characters ?', '#', '[', and ']'. Therefore, PWID URNs for archived URIs cannot be constructed as legal URNs when following the instructions given."
-- Eld: I would like to discuss different alternative:
-- 1: we can just leave it like that - which means that that valid PWID URNs cannot cover these cases. - In all cases the PWID URN will be a huge step forward, and cover about 80-95% of all cases. In this case, it will be up to the users if they construct them anyway - just like the web archives online URIs are not valid URIs since they contain "//" as part of the path.
-- 2: we can consider encoding of these characters (which will complicate things a bit)

Dale