Subject: [art] Software URNs for dependency representation

From: Nico Williams <[email-masked]>
Date: Mon, 26 Jun 2017 16:29:52 -0500

The following is so obvious (I think!) that I suspect it's been done, or
at least discussed.  If so, I would appreciate pointers and I apologize
for the noise.

Suppose you have a build system that lets a recipe refer to
dependencies.  Well that's very common place, naturally.

But, how shall dependencies be identified?

In the big wide open Internet the answer is likely to be: URLs.

But URLs come with a problem: you might need to change locations (e.g.,
mirrors).  Then you have to edit all your recipes.  Worse, you have to
edit all the upstream recipes!

E.g., you depend on a library that depends on gcc, with that dependency
given as a URL to a location you cannot reach (e.g., in an intranet).

We need something better than URLs, something like... URNs!

  urn:software:foo-1.2.3rc1/git/a1b2c3d4...
  urn:software:foo-1.2.3rc1/git/tag/1_2_3rc1
  urn:software:foo-1.2.3rc1/git/tag/1_2_3rc1?default_location=<URI>

or

  urn:brew:foo-1.2.3rc1/git/tag/1_2_3rc1
  urn:debian:stretch/foo-1.2.3
  urn:SomeGitServiceProvider:foo-1.2.3rc1/tag/1_2_3rc1

What needs to be encoded?  Here's a few things:

 - source (not location)

   This would be the NID of the URN.

   In the above it's "software", and that's probably not right as you'll
   see.

   Vendors/distros/whatever could each have their own, as each may have
   different location resolution methods.  Alternatively we could have a
   sub-namespace encoded in the NSS of the URN.

   Having as few namespaces as possible would be nice, but since a name
   resolution service will be needed, it's reasonable to expect at least
   a few namespaces.

   Perhaps Debian might like a namespace per-release.  Or perhaps the
   release would be in the NSS part of the URN as in the above example.

   Everything else definitely goes in the NSS part of the URN.

 - name of software, naturally

 - software version information

   This might include things like git commit hashes, tag names, ...

 - perhaps a way to distinguish built "package" from "source" (maybe
   this should be implied by the namespace)

 - OPTIONAL: perhaps a query or fragment by which to identify one or
   more locations for the software

   This is for things that don't fit into namespaces for any packaging
   or VCS services.

 - OPTIONAL: hash of software (e.g., when other versioning information
   does not provide enough secure version identification information)

 - OPTIONAL: signature(s)

   Signatures could instead be found in the same way as the software,
   and would be bound by any hash that appears in the URN.

Finding locations for software in a relatively small number of
namespaces is easy: make it configurable in the software that needs it
and provide a default configuration with default locations.  These could
be listed in an IANA registry, naturally.

Location resolution of these sorts of URNs is easy then: first determine
if the namespace is supported by the software consuming the URN, and if
so then resolve a root location(s) (see above), lastly employ whatever
additional resolution the namespace owner specifies.

URN namespace registration information could be used to make this
location resolution automatically available in all/most cases.

Thoughts?

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Jun 2017 13:31:26 -0500

On Tue, Jun 27, 2017 at 06:38:22PM +0100, Graham Klyne wrote:
> On 26/06/2017 22:29, Nico Williams wrote:
> Strictly, I'd say *on the Web* the answer is URLs (aka URIs).  The Internet
> could and does use all sorts of identifiers.

Well, sure, if we must have URLs then a new URI scheme would be needed.

> Well, it depends on the kind of URI that you use.  There are some that are
> location-independent (e.g. uuid:, ni:).
> 
> URNs are URIs too!

Yes, quite.

> Or even with HTTP URIs, they may be primarile identifiers (e.g. DOIs via
> http://dx.doi.org, or PURLs via purl.org).  In the Web community, it's a
> well-used pattern to use purl.org (aka PURLs) for identifiers where the
> primary domain may be insufficiently stable.  (There's also another
> initiative, w3id, or something like that).  These solutions all depend on
> introducing a level of indirection (so probably sub-optimal for active
> protocol, use), and one still depends to some extent on the redirecting
> intermediary.

I think those are, indeed, all sub-optimal for this.

> URNs might be a good solution for what you propose.  But an advantage of
> using http URIs for identifiers is that it is possible (in principle) to
> dereference them to learn more about the thing identified (the "follow your
> nose" principle).

If dereferencing is required (and yeah, it probably would be) then a new
URI scheme is needed.

> Finally, Zenodo (under auspices of CERN IIRC) operate a GitHub-integrated
> redirection service for DOIs (and hence http://doi.org URIs) that can be
> registered for software.  E.g. http://doi.org/10.5281/zenodo.582881 is an
> identifier for a version of some software I'm working on.
> 
> Maybe DOIs are an answer you could use?

What I had in mind was that you use the URN's NID to lookup a base URI (URL)
and append the URN's NSS to the local part in order to obtain an actual
URI that can be dereferenced.

A new URI scheme seems better, yes.  Something like:

  sw://debian.example/stretch/foo-1.2.3

The authority would be the name of a repository as a domainname, and the
local part is... local to whatever base URIs published in DNS for the
authority.

> You provide a list of details to be "encoded" in a URI.  Identifiers don't
> really work that way: they name, not encode.  The point here is that if you
> have to know a bunch of details and an assembly rule to construct a URI,
> it's not really working as a URI, more like a URI-like syntax for encoding
> stuff.  I suspect that what you really need is to be able to generate
> different identifiers for any variation of the various variables you
> mention.

Well, semantically the details are "encoded" in the name.  I did not
mean to imply that there would be a specific way to construct a name
from such details, though I probably did accidentally imply that.

> Finally, I'll note that there was some work a while ago for URN (and
> arbitrary URI) resolution, going by the name of DDDS: see
> https://tools.ietf.org/html/rfc3401 and friends.

I'll take a look.

Thanks,

Nico
--

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 27 Jun 2017 11:41:40 -0700

> On Jun 27, 2017, at 11:31 AM, Nico Williams <[email-masked]> wrote:
> 

Hmm…

    https://dwid-org.github.io/uri/historical/draft-ietf-uri-roy-urn-urc-00.txt

....Roy

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Jun 2017 14:20:23 -0500

>     https://dwid-org.github.io/uri/historical/draft-ietf-uri-roy-urn-urc-00.txt

Yeah, something like that.

I know -I'm certain- that we've had a discussion about this many times
in the past.  This is really all about how to name things in a way that
is location-independent yet dereferenceable.  A perennial debate, no?

URIs leave it to the scheme to bake in how to dereference.

URNs don't deal with automatic deref, though where a URN is more than
just a name -- where a URN is a named object that could be retrieved --
then there's generally at least an informal way to dereference.

Maybe we should distinguish pure names (e.g., as used in protocol
elements, such as OIDs) from named resources and then require a
dereference mechanism for the latter.  Right now the one mechanism we
have for this is defining a new URI scheme.

So a new URI scheme it would have to be today, though that is somewhat
dissatisfying: there are many URNs that name resources and which could
use a deref specification.

I'd rather be able name I-Ds and RFCs with URNs and have web software
universally be able to dereference those than have to add an rfc: URI
scheme to get the same effect.  But I'd live with an rfc: scheme, and
I'd live with an sw: scheme.

DOIs for software were mentioned.  But I think with software it's nice
to be able to see names and other metadata in the URI, and it's nice to
be able to have a very light-weight location resolution algorithm.

Yes, yes, we could have every name be a hash or UUID.  But URIs have a
way of leaking into UIs, so having somewhat human-readable URIs is
somewhat handy.  (Having a hash as part of a URI is not a problem
though.)

The location resolution algorithm I have in mind is dead trivial:
resolve the location of the base respository, add local part from the
URI.  The first part can be cached for a long time -- forever even, with
cache invalidation only upon failure.  The second part is dead trivial.

Any location resolution algorithm that requires online infrastructure at
all times will probably be very dissatisfying.  Thus I oppose the use of
DOIs as a primary mechanism for identifying software.

Nico
--

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 28 Jun 2017 07:32:50 +1000

On 28 Jun. 2017 05:21, "Nico Williams" <[email-masked]> wrote:

Nico
--

I think I get what you're saying, but... aren't you literally describing
HTTP URLs with DNS names in the authority?

Cheers

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Jun 2017 17:26:10 -0500

> HTTP URLs with DNS names in the authority?

This is not at all like an https: URI.  It's a different beast, even if
there's a domainname in it.  In the case of http: and https: the
domainname *is* the location.  In this case the domainname is just an
input into a location resolution mechanism, one that allows local
overrides.

The problem with https://foo.example/somepkg-1.2.3 is that any user
agent will want to go to... foo.example to fetch this.

But the user agent may want to go to a mirror -- perhaps a private
mirror.

Now, one could edit /etc/hosts and make foo.example resolve to a local
mirror... but this... sucks, no?  That's just NOT OK.

What I want is something like sw:/foo.example/somepkg-1.2.3, and then
the UA would lookup foo.example in a local directory if there is one,
else in DNS, to find one or more base URIs, then append /somepkg-1.2.3
to those base URIs, and fetch from one of them.

And when I say DNS, I don't mean look for foo.example's IP addresses,
but rather, look for foo.example's software repository base URIs.  (What
RR type to use for this is not relevant here.  I don't want to get into
arguments about URI vs. TXT RR types!)

(Underlying all of this this is an assumption that these resources are
read-only, never-changing.  So anyone can setup a mirror anywhere at any
time for whatever reason.  Metadata might change: e.g., one might mark
something as obsolete or toxic.)

Nico
--

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 27 Jun 2017 21:54:00 -0400

> But, how shall dependencies be identified?

Of course, the answer is "they are identified by names".  But who
assigns the names, etc.?  In a sense, all naming systems are the same in
concept and only differ in details.  A system that has already been
specified is the "object identifier" system, which allows for delegation
of sub-namespaces, etc.  And object identifiers can already be turned
into URNs.

    I have a soft spot in my heart for objectids.  I once assigned names to
    RFCs:

        urn:oid:1.3.6.1.4.1.14490.5.1.xxxx is RFC xxxx

    and to releases of Sourceforge projects:

        urn:oid:1.3.6.1.4.1.14490.22.g.n.n is release n.n of Sourceforge group g

Now you seem to be proposing as a resolution system, "Contact a
well-known list of resolution servers and ask them for the information
regarding the name."

What's different and interesting about your problem is that in the real
world, a dependency actually does live in one of a small number of
servers (or their mirrors) -- at least for standard packages of common
Linux software.  Compare to web sites, which exist in millions of
servers.

But what is to be gained?  Different distributions already name
dependencies and they already store them in Internet-accessible servers.
The only thing that's missing is having a unified naming system -- right
now, a dependency is only fully identified if you know the context (the
distribution) within which that dependency name is referenced.

Dale

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 28 Jun 2017 12:00:09 +1000

On 28 Jun. 2017 08:26, "Nico Williams" <[email-masked]> wrote:

This is not at all like an https: URI.  It's a different beast, even if
there's a domainname in it.  In the case of http: and https: the
domainname *is* the location.

Not really, the IP address (and TCP port) is the location; the domain name
(along with other URL parts) is metadata sent to that location in a query.

In terms of location, the domain name is an input into a location
resolution mechanism, one that allows local overrides.

  In this case the domainname is just an
input into a location resolution mechanism, one that allows local
overrides.

Yeah, that.

I'm not sure why it's not ok. Is it because things other than just your
client use /etc/hosts for DNS? (And, presumably, you only want to redirect
your client.)

But, in general, that's what /etc/hosts is *for*.

Yeah, cool, so you want a new URI (URL) scheme; you just need to define how
to resolve the authority into a... service location? URI prefix? Something
like that.

Whether that uses DNS (A/AAAA/CNAME, SRV, TXT, whatever) or something else
(e.g. a HTTP .well-known service?) depends a bit on operational/deployment
considerations.  I don't think there's a local override mechanism for any
solution I can think of OTOH that is universal but doesn't, at some level,
need /etc/hosts to override DNS for local override.

Cheers
-- 
Matthew Kerwin

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 28 Jun 2017 12:06:34 +1000

This is not the first time this sort of thing has been suggested. I'd put forth that the pain of modifying /etc/hosts is nothing compared to the risk of designing and deploying such a system.

For the general problem (without a solution for private repositories), see:
  https://en.wikipedia.org/wiki/Metalink

> On 28 Jun 2017, at 12:00 pm, Matthew Kerwin <[email-masked]> wrote:
> 
> 
> 
> Not really, the IP address (and TCP port) is the location; the domain name (along with other URL parts) is metadata sent to that location in a query.
> 
> In terms of location, the domain name is an input into a location resolution mechanism, one that allows local overrides.
> 
> 
> I'm not sure why it's not ok. Is it because things other than just your client use /etc/hosts for DNS? (And, presumably, you only want to redirect your client.)
> 
> Yeah, cool, so you want a new URI (URL) scheme; you just need to define how to resolve the authority into a... service location? URI prefix? Something like that.
> 
> Whether that uses DNS (A/AAAA/CNAME, SRV, TXT, whatever) or something else (e.g. a HTTP .well-known service?) depends a bit on operational/deployment considerations.  I don't think there's a local override mechanism for any solution I can think of OTOH that is universal but doesn't, at some level, need /etc/hosts to override DNS for local override.
> 
> Cheers
> -- 
> Matthew Kerwin 
> _______________________________________________
> art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/art

--
Mark Nottingham   https://www.mnot.net/

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 27 Jun 2017 22:29:05 -0400

​​On Tue, Jun 27, 2017 at 10:06 PM, Mark Nottingham <[email-masked]> wrote:

> This is not the first time this sort of thing has been suggested. I'd put
> forth that the pain of modifying /etc/hosts is nothing compared to the risk
> of designing and deploying such a system.

https://tools.ietf.org/html/rfc6920

​You could also look at the .net architecture which uses signatures in a
similar fashion.

In general you have three ways to identify something, following the
trichotemy of Noth:

Firstness, the identifier is the thing itself. i.e. a data uri

Secondness, the identifier is formed from a property of the object. We do
this in two ways:

URLs are Locators, the resource is the data retrieved from
http://example.com/

Fingerprints and other hashes work in reverse, they are a pseudo-unique
identifier that is formed from either the data object or a key that is used
to sign the data object.

Thirdness, the identifier has only a purely conventional relationship to
the signified. All true names have that property, their meaning arises
purely from use, from an intersubjective agreement to use them as such.
While this is typically implemented by reference to a registry, the
registry only has authority so long as users decide to recognize it as
such. It is not possible to drop Iran from the Internet by forcing ICANN to
revoke its IP address block assignments.

The one that I think is interesting that I have not seen applied
systematically until the Mesh is the use of a fingerprint of a signature
key. Yes, there is something of the sort in OpenPGP but it is an
afterthought.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 27 Jun 2017 23:12:58 -0400

Hi,

I will note that the IAB has only recently solicited position papers
for problems shaped almost exactly like this.
https://www.iab.org/2017/06/22/call-for-participation-iab-workshop-on-explicit-internet-naming-systems/
Please note that this is not an advert: I had nothing to do with
setting up that workshop, but I think it interesting (and maybe
related to this thread).

A

-- 
Andrew Sullivan
[email-masked]

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Jun 2017 23:25:02 -0500

> (along with other URL parts) is metadata sent to that location in a query.

See below.

> input into a location resolution mechanism, one that allows local

But http: and https: don't provide for that, not really.

> I'm not sure why it's not ok. Is it because things other than just
> your client use /etc/hosts for DNS? (And, presumably, you only want to
> redirect your client.)
> 

/etc/hosts serves that purpose, this is true, but it's not manageable,
so it's not workable.

> Yeah, cool, so you want a new URI (URL) scheme; you just need to define how

Or a better URN.

> to resolve the authority into a... service location? URI prefix? Something

Yes.

> considerations.  [...]

I don't want to get into what RR types to use for this, not yet.  That's
a useful and important discussion, but it's too soon for it and we'd
rathole.

>          [...].  I don't think there's a local override mechanism for any

No, of course, but the idea is that the specification would make a point
of local overrides so that UAs that implement this get that right from
day 1.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Jun 2017 23:29:05 -0500

On Wed, Jun 28, 2017 at 12:06:34PM +1000, Mark Nottingham wrote:
> This is not the first time this sort of thing has been suggested. I'd
> put forth that the pain of modifying /etc/hosts is nothing compared to
> the risk of designing and deploying such a system.

The same could have been said of DNS.  A lot of original DNS design
decisions are suboptiomal now, and we've had a lot of pain as a result.

We should have stuck with /etc/hosts forever, huh :)

>   https://en.wikipedia.org/wiki/Metalink

Yes, we can always design a solution like that.  No support in UAs, just
build more tools that have to be invoked separately.  Screw the web?

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Jun 2017 23:37:24 -0500

On Tue, Jun 27, 2017 at 10:29:05PM -0400, Phillip Hallam-Baker wrote:
> On Tue, Jun 27, 2017 at 10:06 PM, Mark Nottingham <[email-masked]> wrote:
> You could also look at the .net architecture which uses signatures in a
> similar fashion.

I'm not at all against hashes.  But names are useful things!

Otherwise we'd never ever have had the DNS, or URNs, or anything with
names, and we'd be writing everything in assembly.

Let's set aside -for one paragraph- whether I want a URI or something
else that is name-like.  Hashes surely fit somewhere in the
architecture, but *names* are necessary for humans to deal with.  So
whatever the *name* thingy is (URI, something else), it needs
resolution.

Now come back to whether this wants to be a URI or not.  Are URIs (URNs
in particular) names that can be shown to users?  Or are they to be
hidden from users?

My answer: URIs have leaked into user interfaces.  They are very
commonplace.  While some non-name-like URI forms are certainly useful
and used and to be used, we still need some URI forms that are name-like
and that users can handle.

URIs have some very useful behaviors, not the least of which is that UAs
are an interface to many URI schemes, and usually are pluggable or
extensible to handle new URI schemes.

Nico
--

---

From: Graham Klyne <[email-masked]>
Date: Tue, 27 Jun 2017 18:38:22 +0100

URNs are URIs too!

Or even with HTTP URIs, they may be primarile identifiers (e.g. DOIs via 
http://dx.doi.org, or PURLs via purl.org).  In the Web community, it's a 
well-used pattern to use purl.org (aka PURLs) for identifiers where the primary 
domain may be insufficiently stable.  (There's also another initiative, w3id, or 
something like that).  These solutions all depend on introducing a level of 
indirection (so probably sub-optimal for active protocol, use), and one still 
depends to some extent on the redirecting intermediary.

URNs might be a good solution for what you propose.  But an advantage of using 
http URIs for identifiers is that it is possible (in principle) to dereference 
them to learn more about the thing identified (the "follow your nose" principle).

You provide a list of details to be "encoded" in a URI.  Identifiers don't 
really work that way: they name, not encode.  The point here is that if you have 
to know a bunch of details and an assembly rule to construct a URI, it's not 
really working as a URI, more like a URI-like syntax for encoding stuff.  I 
suspect that what you really need is to be able to generate different 
identifiers for any variation of the various variables you mention.

Finally, I'll note that there was some work a while ago for URN (and arbitrary 
URI) resolution, going by the name of DDDS: see 
https://tools.ietf.org/html/rfc3401 and friends.

#g
--