Subject: Re: Multi-server HTTP

From: Anthony Bryan <[email-masked]>
Date: Tue, 10 Nov 2009 18:07:31 -0500

On Sun, Oct 25, 2009 at 5:54 PM, Micah Cowan <[email-masked]> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Anthony Bryan wrote:
>> Revision 10 includes Alan Ford's merge of draft-ford-http-multi-server
>> and Henrik's additions.
>>
>> http://tools.ietf.org/html/draft-bryan-metalinkhttp
>
> I took some time to look at the current draft, and have some comments below.

Thanks Micah!

>>    o  Use of Link header to describe Mirrors.  Only send a few mirrors
>> with Link header?  Some organizations have many mirrors.
>
> My thought is that an incomplete set of mirrors should be sent, along
> with a link that is understood to refer to a more comprehensive list.

We can use a metalink XML with all mirrors. Is that what you mean?
Do you think using metalink XML for storing partial file checksums is
the best solution?

Alternatively, I believe Alan Ford's other draft had a header to show
this feature was supported & you could use that to negotiate that you
want a comprehensive mirror list.

> (other thoughts:)
>
> sec 4:
>
> sec 2, para 2:
>
>   Metalink servers are HTTP servers that use the Link header
>   [draft-nottingham-http-link-header] to present a list of mirrors of a
>   resource to a client.  They MUST provide checksums of files via
>   Instance Digests in HTTP [RFC3230], whether requested or not."
>
> To me, this implies that a server either is, or is not, a "Metalink
> server", and behaves as required for Metalink servers for any and all
> resources that it provides. I expect that in practice, a given server
> will usually be a "Metalink server" with respect to certain specific
> resources, and will fail to meet the requirements of a Metalink server
> with respect to other resources that are served.

Exactly, thanks. Is this better?

"Metalink servers are HTTP servers that use the Link header to present
a list of mirrors of certain specific resources to a client. They MUST
provide checksums of certain specific files via Instance Digests in
HTTP, whether requested or not. That is, Metalink servers are not
required to provide mirrors and checksums for all files, only certain
specific files."

> sec 3.2:
>
> I'm wondering whether a two-letter country code isn't quite granular
> enough to be completely practical; for instance, a client in the US may
> be closer to a mirror server in Canada or Mexico, than any available in
> the US; similarly, someone in France or Germany many be closer to a
> mirror in Belgium than any available in their own country.
>
> OTOH, maybe it's "good enough"; I can't speak to any system that might
> be better, apart from IP address and route analysis techniques, which
> won't be helped by anything you could put in this draft RFC.

Latitude & longitude?

MirrorBrain, the main metalink generator, does some neat stuff. See
"Sophisticated mirror selection" at
http://www.mirrorbrain.org/features/

> sec 3.4:
>
> It's not clear to me how "mirror depth" should work for URIs ending in a
> trailing slash.

I'm not sure.

What do you think is the most logical and easy way to handle that?

> sec 4:
>
> It strikes me that the Link header with a "rel" param value of
> "described-by" (and in fact the concept of a "Metainfo file" as a whole)
> covers far too many roles, without distinguishing them sufficiently.

This is what Mark Nottingham, author of the Link draft, suggested.

> - From the examples, this facility can be used to provide torrent
> locations, metalink description files, and cryptographic files, with
> probably additional possible roles to be provided as they are invented. ;)

True :)

> It would be great if a mechanism were used to explicitly describe the
> role that a "metainfo file" plays, be it mirroring, torrent tracking,
> providing cryptographic signatures for the resource identified; and not
> rely on assumptions based on the resource's content type (which should
> then be optional). Other values for the "rel" param might meet this need.

I'm sure there are cases where content type doesn't mandate how
something is used. It seems like this is flexible enough to me though,
but I'm open.

I've added this text:
Metalink clients MAY support the use of metainfo files for downloading
files, but that is not required.
Metalink clients MAY support the use of OpenPGP signatures, but that
is not required.

> various:
>
> I recommend that references to the term IRI be replaced by URI
> throughout the document, except perhaps section 10.1, "URIs and IRIs"
> itself (though I also object, if much more mildly, to the requirement
> that clients handle IRIs, as this may not make sense in all cases. A
> SHOULD use, perhaps? Really, it's a user interface issue, so I'm not
> sure it belongs in this RFC). I don't believe IRIs have a place in HTTP
> headers, where they ought to be converted to their unambiguous
> corresponding URI form first. AFAICT, IRIs are best constrained to user
> interfaces (entry and presentation), and their use to identify resources
> between non-human mechanisms should generally be avoided.

I've left IRI in section 9 and 10.1.

I'm no IRI expert. I see it used in RFC 4287 where it doesn't seem to
always deal with user interfaces.

It was nice of you to take the time for these extensive comments.
Sorry I didn't get back to you sooner, been dealing with family stuff.

On a side note, public congratulations to Henrik Nordström and Daniel
Stenberg for being nominated for the Nordic Free Software Award!
http://fscons.org/award
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: David Morris <[email-masked]>
Date: Tue, 10 Nov 2009 15:53:56 -0800 (PST)

> required to provide mirrors and checksums for all files, only certain

Uh ... no ... "certain specific ..." is not a clear phrase as it is very 
generic and and no correlation with the notion of a multi-server, etc.

I think you want to define a metalink resource ... for example:

  "Metalink resources include a Link header
   [draft-nottingham-http-link-header] to present a list of mirrors of
   in the response to a client request for the resource. A resource
   checksums via Instance Digests in HTTP [RFC3230] must be included.

   Metalink servers are HTTP servers with one or more Metalink
   resources."

---

From: Anthony Bryan <[email-masked]>
Date: Wed, 11 Nov 2009 17:48:06 -0500

>  Metalink servers are HTTP servers with one or more Metalink

Thanks, David. Here's what I've changed it to:

"Metalink resources include a Link header
[draft-nottingham-http-link-header] to present a list of mirrors in
the response to a client request for the resource. The checksum of a
resource must be included via Instance Digests in HTTP [RFC3230].

Metalink servers are HTTP servers with one or more Metalink resources."

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 23 Nov 2009 15:08:16 -0500

> Metalink servers are HTTP servers with one or more Metalink resources."

The latest revision includes this and an optional dependency on
Metalink/XML for partial file checksums, which could also be used for
the whole mirror listing if we don't want them all in HTTP headers.

I'm biased but I think that's the easiest solution for partial file checksums.

Content-MD5 isn't an option.
We could also send them in headers but that would be pretty verbose,
or define a new format.

I think repairing downloads is important, but not as many current
metalink clients go through the work of adding this feature, even
though it's available, so maybe the dependency on XML isn't too bad.

I'd love to hear everyone's thought or other solutions.

Other open issues:
Would it make more sense to use qvalue-style policies to describe
mirror priority, i.e. q=1.0 through q=0.0  instead of pri=1 ?