Subject: Re: Query on discovery algoritm for LRDD (draft-hammer-discovery-04)

From: John Panzer <[email-masked]>
Date: Mon, 5 Apr 2010 13:12:16 -0700

On Sat, Mar 27, 2010 at 2:41 PM, Eran Hammer-Lahav <[email-masked]>wrote:

>
>
> > -----Original Message-----
> > From: John Panzer [mailto:[email-masked]]
> > Sent: Friday, March 26, 2010 10:08 AM
>
> > This is my attempt to translate
> > http://tools.ietf.org/html/draft-hammer-discovery-04 into an actionable
> > algorithm, as a first step in attempting to write a truly general
> resource
> > discovery library that matches the latest spec.
> > It is written under the assumption that a client may wish to query for
> sets of
> > links with a given relvalue, and is not going to wish to retrieve and
> re-parse
> > host-meta files any more often than necessary.  Feedback welcomed!
> >
> > Definitions
> >
> > startURI: Starting point for discovery; input into the discovery process.
> > Discovery produces metadata for startURI.
> >
> > metadata: A collection of links and properties for the resource.
> >
> > link: An abstract edge between the startURI resource and another resource
> > identified by a field href.  Has a rel field indicating the relationship
> from
> > startURI to href. Has a lrdd_pri field indicating ordering within the rel
> value.
> > Has a secure boolean field indicating whether it was retrieved securely.
>  Can
> > be annotated with arbitrary properties.
> >
> > property: A piece of metadata about a resource less structured than a
> link.
> > Only present for XRD-derived data.  Has a secure boolean field indicating
> > whether it was retrieved securely.
>
> Link headers and elements can include extension parameters that are either
> rel-specific or document specific. These will be in the form of key/value
> pairs but in which the key is a short name, not a URI (like XRD <Property>
> type). In markup the can have a namespace.
>
> I generally prefer people used the XRD <Property> facility instead of
> extending the link schema (for header format).
>
> > Algorithm
> >
> > Determine priority order (Host-priority or Resource-priority):
> >
> > lookup HostMetaLRDD(startURI)
>
> What does this do in this context? Your Function below is aimed for getting
> metadata, not just raw host-meta.
>

In this context, this should be something like RawHostMeta().

>
> > if above lookup fails because Host(startURI) is not available, priority =
> > Resource Else, does result contain
> > XRD/Property@type=http://lrdd.net/priority/resource?  If yes, then
> priority
> > = Resource.  Otherwise, priority = Host
> >
> > If priority = Host:
> >   // priority order is host-meta, HTTP Link, <Link> elements
> >   metadata = lookup HostMetaLRDD(startURI, 1)
> >   augment metadata with result of lookup HeaderLinks(startURI, 2)
> >   augment metadata with lookup ResourceLinks(startURI, 3)
> >
> > If priority = Resource:
> >   // priority order is <Link> elements, HTTP Link, host-meta
> >   metadata = lookup ResourceLinks(startURI, 1)
> >   augment metadata with result of lookup HeaderLinks(startURI, 2)
> >   augment metadata with lookup HostMetaLRDD(startURI, 3)
> >
> > where
> >
> > HostMetaLRDD(uri, pri) is:
> >   Revalidate local XRD data cache for hostof(uri)/.well-known/host-meta.
>
> How do you handle http vs https?
>

What is the default ordering in which to try?  I'd much prefer to make
https:// the default scheme for the "hostof(x)" above, failing back to
http:// with a flag indicating content was not verified by a certificate.

NB:  In starting to write code for all this, I'm finding it useful to use
nonstandard ids like foo@localhost:8080 in order to preserve my sanity.
 This means "pretend you're foo@localhost, and use 8080 as the default HTTP
port for host-meta discovery, and switch off security checks too".  In
particular I end up having to swap around between foo@localhost:8080,
foo@localhost:8081, foo@localhost:8082,... dynamically quite a bit to
simulate cross-domain calls, so a static configuration would be quite
annoying/unusable.  Fortunately most of the stack I'm using doesn't care
about this, but the port needs to be passed down a few levels and get
through some regexps.  Not sure how best to do this for real without
compromising security -- perhaps allow this only for "localhost"...

> Follow redirections.

...while ensuring that one's security flag gets reset to 0 if any redirects
pass through http...

> Validate host-meta is valid for the desired host.
>

What's the procedure here?  Just match Subject and Alias against the host?

>
> >   Find the first link with rel="lrdd" in host-meta document (call
> >     the href value lrdd_url)
>
> Apply template...
>

Ah right, you don't have an href until you apply(template, uri) --> href.

>
> >   Revalidate local XRD data cache for lrdd_url,annotating each link
> >     with lrdd_pri=pri
> >   Return metadata (non-lrdd links and properties) for lrdd_uri
> >
> >
> > HeaderLinks(uri, pri) is:
> >   If uri is not dereferenceable (e.g., acct:) return empty set
>
> If you URI is not http/https, or if a GET/HEAD returns a response that does
> not contains a representation for the requested resource. See:
> http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-09#section-6.1
>
>
What if it returns a 403 Forbidden?  Or a 5xx Temporary Server error?  Or
just times out?

>  >   Revalidate local resource link cache for uri
> >   Retrieve Link: headers for resource from cache, annotating
> >     each link with lrdd_pri=pri
> >   Return links to caller
> >
> > ResourceLinks(uri, pri) is:
> >   If uri is not dereferenceable (e.g., acct:) return empty set
> >   Revalidate local resource link cache for uri
> >   Retrieve MIME-type-specific set of links known for resource, annotating
> > each
> >     link with lrdd_pri=pri
> >   Return links to caller
> >
> >
> > augment X with Y with priority pri is:
> >   For each set of links y in Y sharing link relation r:
> >     if r=="lrdd", and X does not already
> >       contain a link with rel="lrdd":
> >       augment X with results of:
> >         Revalidate local XRD data cache for first href for rel="lrdd",
> >           annotating each link with lrdd_pri=pri
> >         Augment X with resulting metadata from href
> >     else,
> >       add list of links y to X.
> >       Add all properties in Y to X.
> >
> >
> > NB: Some link types can contain space-separated rel values, which are
> > interpreted as a series of links with each of the space-separated rel
> value
> > names in the above algorithm.
> >
> > NB: Ordering of links and properties from each source with the same
> rel/type
> > values is retained throughout, with the caveat that HTTP headers do not
> > have a guaranteed stable order.
> >
> > Security is not shown here, but is based on either TLS used throughout
> the
> > retrievals or (for XRD) the XRD signature specification.  Content type
> specific
> > security may also be used to verify links within content (e.g., Magic
> > Signatures over link elements).  When set, the secure boolean indicates
> that
> > a specific link or property was retrieved securely; when not set, it
> indicates
> > that the retrieval path contained one or more insecure steps.  The
> algorithm
> > will also reject data that is red-flagged (e.g., if a CRL check shows
> that a
> > certificate has been revoked as opposed to merely expiring).  (?)
> >
> > Result:
> >
> > An ordered collection of metadata, consisting of links and properties,
> each of
> > which is annotated with its lrdd_pri in case it matters.  There is at
> most one
> > rel="lrdd" link in the collection which identifies the specific LRDD
> document
> > used, if any, and any metadata from that lrdd document is transcluded
> into
> > the metadata collection.
>
> There can be more than one LRDD document, but only one is used from each
> source. There should not be any rel='lrdd' in the collection because it is
> either expanded or dropped (not first).
>

This algorithm was just tracking the address of the one that was expanded
(not 100% necessary but useful for debuggability).

I think that "Any "lrdd" links other than the first MUST be ignored." in
section 3, step 2, is confusing:  I had interpreted "first" as the first one
encountered globally, but your note implies that you can have up to 3 LRDD
documents to peek inside (one from each source).

I'm not sure that the limitation of one LRDD link per source helps anything
in the algorithm in this case.

>
> > A client which wishes to coalesce links and properties found in all three
> > places within a given rel or type value can just query the collection for
> the
> > desired rel or type and retrieve collections of links or properties.
> >
> > Once retrieved, the set of metadata can be queried.  An important/useful
> > query is to retrieve the list of links with a given rel value:
> >
> > get_links(metadata, rel)  # Get sequence of links with given rel value,
> or
> > empty list if not found.
> >
> > Sometimes you may require that only the first set of links found
> > count:
> >
> > get_top_priority_links(metadata, rel)  # Get sequence of links with given
> rel
> > value, ignoring all but the first set found
> >
> > ...and of course you can do a similar query for properties.
>
> EHL
>
>

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Tue, 13 Apr 2010 09:24:12 -0700

On 4/5/10 1:12 PM, "John Panzer" <[email-masked]> wrote:

>>> algorithm, as a first step in attempting to write a truly general resource
>>> It is written under the assumption that a client may wish to query for sets
>>> of
>>> property: A piece of metadata about a resource less structured than a link.
>>> XRD/Property@type=http://lrdd.net/priority/resource?  If yes, then priority
> https:// the default scheme for the "hostof(x)" above, failing back to http://
> with a flag indicating content was not verified by a certificate.

Given recent discussions I'm inclined to change the draft to "SHOULD try
https:// first, then http:// second".

> nonstandard ids like foo@localhost:8080 in order to preserve my sanity.  This
> means "pretend you're foo@localhost, and use 8080 as the default HTTP port for
> host-meta discovery, and switch off security checks too".  In particular I end
> up having to swap around between foo@localhost:8080, foo@localhost:8081,
> foo@localhost:8082,... dynamically quite a bit to simulate cross-domain calls,
> so a static configuration would be quite annoying/unusable.  Fortunately most
> of the stack I'm using doesn't care about this, but the port needs to be
> passed down a few levels and get through some regexps.  Not sure how best to
> do this for real without compromising security -- perhaps allow this only for
> "localhost"...
>  
>> Follow redirections.
> 

Match the URI of the resource being discovered to the <Host> element of the
host-meta file.

> What if it returns a 403 Forbidden?  Or a 5xx Temporary Server error?  Or just
> times out?

Then you don't use Link headers for LRDD. The next draft will make this
clear. Basically, if the Link is not about what you asked for, don't use it.
A Link in a 403 response is between an anonymous resource and a target.

>>> interpreted as a series of links with each of the space-separated rel value
>>> NB: Ordering of links and properties from each source with the same rel/type
>>> Security is not shown here, but is based on either TLS used throughout the
>>> Signatures over link elements).  When set, the secure boolean indicates that
>>> that the retrieval path contained one or more insecure steps.  The algorithm
>>> will also reject data that is red-flagged (e.g., if a CRL check shows that a
>>> An ordered collection of metadata, consisting of links and properties, each
>>> of
>>> which is annotated with its lrdd_pri in case it matters.  There is at most
>>> one
>>> used, if any, and any metadata from that lrdd document is transcluded into
>>> the metadata collection.
>> 
> This algorithm was just tracking the address of the one that was expanded (not
> 100% necessary but useful for debuggability).
> 
> I'm not sure that the limitation of one LRDD link per source helps anything in
> the algorithm in this case.

I tend to agree. The extra code to iterate through multiple LRDD links in a
single source is minimal. But it does add complexity and can slow down the
client if the server goes nuts breaking its metadata over multiple LRDD
documents.

>>> get_links(metadata, rel)  # Get sequence of links with given rel value, or
>>> empty list if not found.
>>>