Subject: [art] Review draft-soilandreyes-arcp ?

From: Stian Soiland-Reyes <[email-masked]>
Date: Mon, 5 Feb 2018 14:02:17 +0000

Would you be able to comment on this Internet-Draft that is proposing
the arcp:// URI scheme?

I am intending to submit this as an Informational RFC and have
contacted arts-ads@ - however I did not receive any response, so I'm
taking it to the wider art@ list.

See also discussions on uri-review@ - note this proposal was previously called
draft-soilandreyes-app.

https://tools.ietf.org/id/draft-soilandreyes-arcp-03.html

Name:           draft-soilandreyes-arcp
Revision:       03
Title:          The Archive and Package (arcp) URI scheme
Document date:  2018-02-05
Group:          Individual Submission
Pages:          21
URL:
https://www.ietf.org/internet-drafts/draft-soilandreyes-arcp-03.txt
Status:         https://datatracker.ietf.org/doc/draft-soilandreyes-arcp/
Htmlized:       https://tools.ietf.org/html/draft-soilandreyes-arcp-03
Htmlized:       https://datatracker.ietf.org/doc/html/draft-soilandreyes-arcp-03
Diff:           https://www.ietf.org/rfcdiff?url2=draft-soilandreyes-arcp-03

Abstract:
   This specification define the Archive and Package URI scheme "arcp".

   arcp URIs can be used to consume or reference hypermedia resources
   bundled inside a file archive or an application package, as well as
   to resolve URIs for archive resources within a programmatic
   framework.

   This URI scheme provides mechanisms to generate a unique base URI to
   represent the root of the archive, so that relative URI references in
   a bundled resource can be resolved within the archive without having
   to extract the archive content on the local file system.

   An arcp URI can be used for purposes of isolation (e.g. when
   consuming multiple archives), security constraints (avoiding "climb
   out" from the archive), or for externally identiyfing sub-resources
   referenced by hypermedia formats.

-- 
Stian Soiland-Reyes
http://orcid.org/0000-0001-9842-9718

---

From: Stian Soiland-Reyes <[email-masked]>
Date: Mon, 5 Feb 2018 14:09:32 +0000

> https://tools.ietf.org/id/draft-soilandreyes-arcp-03.html

If you prefer code; here's a Python implementation for
parsing/generating arcp URIs:

https://github.com/stain/arcp-py
http://arcp.readthedocs.io/

---

From: Graham Klyne <[email-masked]>
Date: Thu, 15 Feb 2018 16:52:57 +0000

> See also discussions on uri-review@ - note this proposal was previously called

Hi Stian,

Reviewing:

Looking good!

Mostly minor editorial comments, some terminology...

## 3.1. Authority

"The authority is NOT intended to be resolvable without..."

Do you mean resolvable or dereferencable or something else?  (I'm not sure what 
it means for a URI authority to be resolvable - this term has a particular use 
at https://tools.ietf.org/html/rfc3986#section-5; see also 
https://tools.ietf.org/html/rfc3986#section-1.2.2).

I'm not sure what point item 4 is intended to convey.

## 4.2. Path semantics

"The path component of an arcp URI identify ..."

nit: "identifies" (or otherwise more consistent use of singular/plural forms)?

"undefined from this specification"  --> "undefined by this specification"

## 4.3. Resolution protocol

"This specification do not define..."  --> "... does not ..."?

"... resources according to the arcp URI scheme." --> "... resources identified 
using the arcp URI scheme"?

"Such an implementation can then resolve arcp URIs, ..."

Do you mean "dereference" rather than "resolve"? (cf. 
https://tools.ietf.org/html/rfc3986#section-1.2.2, para 2)

Similar next para: maybe worth running a global search to check tis?

"It is not undefined by this specification..." --> "... not defined ..."?

## 6. Interoperability considerations

Item 1.

on "The implementations may need to detect equality of the two UUIDs out of 
band.":  "equality" --> "equivalence"?  (I think I'd prefer something like 
"detect equivalent referents of the corresponding URIs...".)

## General

I spotted, then lost, a few more minor editorial nits - if you have a source in 
guthub, I could offer a PR for your consideration.

I wouldn't hold off requesting provisional registration until information RFC 
publication (assuming acceptance). The email address for that is different from 
the review list.

Best regards,

#g
--

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 15 Feb 2018 22:14:14 -0500

> Would you be able to comment on this Internet-Draft that is proposing

I think you could make the draft clearer -- and attract more support --
by expanding the introduction.  When I read the introduction (quickly),
it told me the general idea of what you're trying to do, but it didn't
make it clear to me how it would work in practice -- which would tell me
which of my problems it would solve.

The general idea seems to be a URI that identifies a component of an
archive.  Within an archive, components are identified by a path.  (BTW,
how the path part of a URL maps to a component name is more complicated
than you'd expect -- see the definitions of the file: and ftp: URLs for
examples that show the complexity.  If you want arcp: to be widely
adopted, you need to nail down those details.)

What isn't so clear is how the authority part is matched with the
archive itself.  If the arcp: URI is inside a component of the archive,
then it's clear how you find the archive, presumably you can verify that
the authority matches the archive that you're already processing.
(However, this is impractical if the authority part identifies the
archive via a hash of the archive's contents, as that hash will depend
on the arcp: URI contained in the component of the archive ...)

Are these URIs expected to be used in any situation where they are not
already contained within the archive that they refer to?

In any case, one or more well-chosen, fairly complete examples in the
introduction would make the concept much clearer.

Dale

---

From: Stian Soiland-Reyes <[email-masked]>
Date: Fri, 16 Feb 2018 09:23:29 +0000

Thank you for your review. I will take them into consideration and
make an updated draft.

The GitHub source is
https://github.com/stain/I-D/blob/master/arcp/arcp.md - pull requests
welcome!

On 15 February 2018 at 16:52, Graham Klyne <[email-masked]> wrote:
> Do you mean resolvable or dereferencable or something else?  (I'm not sure
> what it means for a URI authority to be resolvable - this term has a
> particular use at https://tools.ietf.org/html/rfc3986#section-5; see also
> "... resources according to the arcp URI scheme." --> "... resources
> identified using the arcp URI scheme"?
>
> I spotted, then lost, a few more minor editorial nits - if you have a source
> in guthub, I could offer a PR for your consideration.
>
> I wouldn't hold off requesting provisional registration until information
> RFC publication (assuming acceptance). The email address for that is
> different from the review list.
>
>
> Best regards,
>
> #g
> --
>
>
>
>
>
>
>>
>> Htmlized:
>> https://datatracker.ietf.org/doc/html/draft-soilandreyes-arcp-03
>> Diff:
>> https://www.ietf.org/rfcdiff?url2=draft-soilandreyes-arcp-03
>>
>> Abstract:

---

From: Stian Soiland-Reyes <[email-masked]>
Date: Fri, 16 Feb 2018 09:45:45 +0000

> which of my problems it would solve.

Thanks for your review!

Agree that the introduction needs more work, perhaps it has too much
of "Background" and too little of "Introduction".

> adopted, you need to nail down those details.)

Yes, I touched on some of those issues in earlier drafts, but removed
them after feedback that it got too complicated!

This could be quite extensive if looking across archive formats - e.g.
OS X .dmg images can have multiple roots (one per Volume), a tape
might have multiple backup sessions. The easiest approach here is
probably that if there is no "default root" then there would simply be
a new authority for each root (potentially nested from an outer arcp).
But that would mean archive-specific conventions need to be formed.

Trying to use arcp with more executable resource formats like .dll
files (like 7zip can explore) probably makes it trickier to decide on
consistent paths.

I previously had a more detailed path section:
https://tools.ietf.org/id/draft-soilandreyes-app-03.html#rfc.section.2.2
that gave the obvious mapping from URI path segments to folders in the
archive. But then many archive formats (e.g. zip) don't really have
folders, just arbitrary paths.

I feel that I probably could detail this mapping more, but as that
would bring us closer to the archive as a file container, that would
have to be at the cost of dropping the old "app" aspect with more
abstract resource paths (example 7
https://tools.ietf.org/id/draft-soilandreyes-arcp-03.html#rfc.appendix.A.7
) and would make it weirder that ?query is permitted  (I was asked to
re-permit ?query)

> archive itself.

I think my idea was that you get say a ZIP file that you then keep
"somewhere". At the same time you generate an arcp authority, e.g. a
random UUID - which you associate with the ZIP file. It is
application-specific how (and for how long) you keep this link -
perhaps you add a neighboring foo.zip.arcp file, or add it as an
extended attribute on the file system, keep it in some database, or
just always do a new UUID in memory.

> If the arcp: URI is inside a component of the archive,

Yes, I guess the possibility of self-declaring is what I hinted at in
example https://tools.ietf.org/id/draft-soilandreyes-arcp-03.html#rfc.appendix.A.4
where the arcp authority is derived from the declared
External-Identifier uuid.

There's the risk here that the archive is lying or have different
mutability expectations than the application.

> on the arcp: URI contained in the component of the archive ...)

Yes, obviously you can't declare your own hash inside - but then again
the hash can be calculated by anyone able to read the whole archive
file/stream. That is the purpose of this authority prefix, anyone
reading the same archive byte (using the same hash algo) will get the
same arcp authority, even if they got the archive through different
means.

> already contained within the archive that they refer to?

Yes, we can't expect all archives to be able to self-declare their
identities, and I think arcp should be usable for any path-like
archive found "in the wild".

> introduction would make the concept much clearer.

Thanks - I had moved all the examples to the appendix, but I can add a
quick example at the top.

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 16 Feb 2018 07:49:01 -0500

> But that would mean archive-specific conventions need to be formed.

A critical point is that arcp: can only be used with a particular
archive type if the conventions for that archive type are defined.  In
re file:, they had to figure out how to make it work with file systems
for which "/" can be part of an ordinary file name.  In regard to ftp:,
it has no "absolute directory names", the only specification of a
directory is a sequence of CWD commands, which when executed, reach the
desired directory.  Getting these details right is critical to having
the scheme work well across all archives of an archive type.

In regard to the multiple roots of .dmg, I suspect that each root/Volume
has some sort of unique identifier within the .dmg, and that can be used
as the first element of the path.

> just always do a new UUID in memory.

I think that touches on my doubt:  The only time a URI is *useful* is if
it's (1) generated by one application and (2) interpreted by another
application.  And in (2), the interpreting application must not be
treating the URI as a purely opaque string.  (In the case of URLs, this
condition is always met, because the retriever of the resource
determines the protocol to be used by parsing the URL.)

Looking at the first five examples in the appendix, the "interpreters"
seem to understand the path parts, but the authority parts seem to be
used as opaque strings; the authority parts are always resolved by
querying the application that generated them.  With that limitation on
usage, the use cases can be satisfied by just considering the URI to be
a pair:  (opaque archive identifier, path within archive).  Creating a URI
format to carry that information doesn't add any value.

Which, I think, points to an approach where there would be benefit:  The
arcp: URI *is* a pair:  (URI of the archive, path within archive).  (I'm
ignoring the syntax of how one represents this information in the URI.)
If the archive URI is a URL that fetches an archive object, then an
application can interpret and dereference the arcp: URI.

If the archive URI is an ni: URN, then the path is inside the archive
that has that URN.  That doesn't give the interpreter a way to obtain
the archive object, but if it already possesses the archive object, it
can match the ni: URI with the archive, and then locate the intended
component.

If the archive URI is a uuid: URN, then the semantics is that it
designates the archive that is named by that UUID, although the method
by which the UUID is associated with the archive is out of scope of this
document -- presumably the archive format defines how that is done and
the UUID is stored in the archive.

Etc.

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Sat, 17 Feb 2018 09:13:47 -0500

This gets more interesting the longer you look at it...

You might want to look into draft-shur-pack-uri-scheme and whatever
discussion there was around it and compare it with your work.

> application can interpret and dereference the arcp: URI.

But if course, there *is* a URI syntax for that concept:  fragment
identifiers a/k/a '#'.  Given a URI that designates a resource, the URI
to designate part of the resource is made by appending '#' and an
identifier for the part.  The syntax and semantics of the fragment
identifiers depend on the media type of the resource, not the scheme of
the URI, which makes sense, because the "parts" of a resource are
independent of how you obtained the resource.

So the natural question is, Are fragment identifiers defined for the
various archive media types?

>From the draft and this e-mail discussion, I can extract this list of
archive types:
    dmg
    ZIP
    bagit
    tar
    vnd.android.package-archive
    x-tar
    x-gtar
    x-7z-compressed

Of these, only ZIP is has a registered media type, and that gives just a
link to Paul Lindner, dated 1993.  It looks like nobody has defined
fragment identifiers for archive resource types!

(We could consider the "multipart/*" media types as archives as well.
Presumably their fragment identifiers are the msg-id values in the
Content-ID headers of the parts, or integer sequence numbers of the
parts.)

Sections 3.2 and 4.2 are the considerations for defining how fragment
identifiers for archives identify components.

If fragment identifiers were defined for archive types, and we use them,
the following changes would be seen in the examples.  Note that the
syntax changes are substantial, but the semantics don't change:

   The application generates the corresponding arcp URIs and uses those
   for URI resolutions to list resources and their hyperlinks:

   urn:uuid:32a423d6-52ab-47e3-a9cd-54f418a48571#doc.html
     -> urn:uuid:32a423d6-52ab-47e3-a9cd-54f418a48571#css/base.css
   urn:uuid:32a423d6-52ab-47e3-a9cd-54f418a48571#css/base.css
     -> urn:uuid:32a423d6-52ab-47e3-a9cd-54f418a48571#fonts/Foo.woff
   urn:uuid:32a423d6-52ab-47e3-a9cd-54f418a48571#fonts/Foo.woff

--

   From this the hash-based arcp base URL is:

   ni:///sha-256;f4OxZX_x_FO5LcGBSKHWXfwtSx-j1ncoSt3SABJtkGk

[Note that ni: URIs do not end with a slash.]

   The repository adds annotations for detected source code files within
   the archive.

   A client is browsing the annotations and discovers:

   ni:///sha-256;f4OxZX_x_FO5LcGBSKHWXfwtSx-j1ncoSt3SABJtkGk#src/luhn.c

--

   The application generates a random UUID v4 "ff2d5a82-7142-4d3f-b8cc-
   3e662d6de756" and adds the corresponding "urn:uuid" UUID to the bag
   metadata file "/gfs/bags/scan15/bag-info.txt"

   External-Identifier: urn:uuid:ff2d5a82-7142-4d3f-b8cc-3e662d6de756

[Note that uuid: URNs do not end with a slash.]

   It then generates arcp URIs for the files listed in the manifest:

   urn:uuid:ff2d5a82-7142-4d3f-b8cc-3e662d6de756#data/27613-h/q172.png
   urn:uuid:ff2d5a82-7142-4d3f-b8cc-3e662d6de756#data/27613-h/q172.txt

--

   urn:uuid:12f89f9c-e6ca-4032-ae73-46b68c2b415a

But to make this works requires registering media types for the various
archive formats, registrations which define fragment identifiers as
referencing components of the archives.  That includes dealing with any
syntactic issues regarding encoding the component names (as seen by the
archive format) into the syntax of fragment identifiers.

This allows us to consider an aspect of fragment identifiers that only
arises with "archives", resources that can contain other resources.
Suppose I have a URL for an HTML file:  http://foo.example.com/bar.html
If I want to refer to an anchor 'baz' within bar.html, I say
    http://foo.example.com/bar.html#baz
And any browser can render that.

Now suppose I have a URL for an archive:
http://foo.example.com/quux.zip.  If I want to reference bar.html within
the archive, presumably I say
    http://foo.example.com/quux.zip#bar.html

But there's no defined way to reference the anchor 'baz' inside the HTML
file bar.html inside the archive quux.zip.

What is needed are "recursive" fragment identifiers for archives, a
fragment identifier (per the URI syntax) that contains (1) the name of a
component of the archive, and (2) a general fragment identifier to be
dereferenced within that component.  The obvious syntax is

    # component # fragment-within-component

but the URI syntax doesn't allow '#' within fragment identifiers.  A
decent choice seems to be

    # component @ fragment-within-component

(Per RFC 3986, only two characters are allowed in 'fragment' that are
not 'unreserved' or 'sub-delims', ":" and "@".  They are in the class
'gen-delims', and presumably "more magic" than other characters.  I
choose "@" over ":" because my intuition is that ":" is more commonly
used in file names.  Of course, a consequence of this is that "@" within
a component name is special, and has to be %-escaped.)

With this extension, we can say

    http://foo.example.com/quux.zip#bar.htm@baz

Note that this concept has to be defined as a feature of the fragment
identifiers for each archive type; its conceptual unity comes from all
media type definitions for archive types agreeing on this syntax and
semantics.

This allows us to concisely deal with archive files that are themselves
contained within archives.  If quux.zip contains an archive corge.zip,
then corge.zip can be referenced

    http://foo.example.com/quux.zip#corge.zip

grault.html within corge.zip can be referenced

    http://foo.example.com/[email-masked]

and the HTML anchor garply in grault.html is

    http://foo.example.com/[email-masked]@garply

What this mapping of the draft's concepts does NOT cover so far is the
semantics of "name" authorities.  As I understand it, arcp URIs with
"name" authorities form an abstract namespace which is scoped to a
context, a "particular installation".  The authority part is
syntactically a domain name, and the meaning of a particular URI is
defined by the owner of that domain name.

I've tried looking through the IANA list of existing URI schemes and URN
namespaces, and I can't spot anything that looks like it has these
semantics.  So it looks like this is a good candidate for a new URI
scheme.  I notice that the scheme "name" has not been used!  However, it
might not be best to call them "names", as the usage of that word when
defining URNs requires that the resource that a "name" refers to must be
unique and never change, whereas the example in section A.7 suggests
that the mapping from URIs to resources might change with time.  The
generic "resource" is already registered.  However, "pack" (for
"package") has a provisional registration referencing an I-D that
expired in 2009, and so might be available.  Using "pack", the examples
of the draft become

   pack://app.example.com/img/logo.png

   pack://gallery.example.org/

   pack://gallery.example.org/photos/?New

   pack://gallery.example.org/photos/137

   pack://messaging.example.com/share;+{uri};{+redirect}

   pack://messaging.example.com/share
     ;uri=arcp://gallery.example.org/photos/137
     ;redirect=arcp://gallery.example.org/photos/%3fNew

Trying to distill all this into a plan, it seems to me that a way to
carry this work forward in a way that is well-aligned with existing URI
usage is:

- Register media types for the popular archive formats, defining
  fragment identifiers to identify components of the archive, aligning
  the fragment identifier definitions with each other and the
  considerations in sections 3.2 and 4.2.

- Define a method by which fragment identifiers can carry sub-fragment
  identifiers that apply to an identified component of an archive, and
  ensure that the archive media types are aligned with it.

- Possibly register (or define in an RFC?) the use of fragment
  identifiers for all multipart/* media types.

- Define a new URI type that carries the semantics that are proposed for
  "name," authorities.

The first three of these would naturally fit into one RFC, since the
definitions for the various media types need to be aligned with each
other and have common considerations.  The fourth item could be
processed as a separate RFC.

Dale