Subject: RE: Multi-server HTTP

From: "Ford, Alan" <[email-masked]>
Date: Fri, 2 Oct 2009 10:11:16 +0100

Hi,

> -----Original Message-----
> From: Anthony Bryan [mailto:[email-masked]]
> Sent: 25 September 2009 23:17
> To: Ford, Alan
> Cc: Henrik Nordstrom; Mark Nottingham; [email-masked]; Mark Handley
> Subject: Re: Multi-server HTTP
> 
> On Thu, Sep 24, 2009 at 8:23 AM, Ford, Alan <[email-masked]> wrote:
> > Henrik's point here goes back to the question of mirroring multiple
> > files / directory structure. This currently remains un-tackled, but I
> > think one idea that could be used is the use of a Link: header going to
> > a form of Metalink that can be used for whole mirrors. Something to come
> > back to, once the basic concept is clear.
> 
> Right, nothing so far requires XML though. Do we want to add that dependency?
> 
> I'd like tools like wget & curl to be able to support this (if they
> choose to) w/o adding extra dependencies unless required.
> 
> Thoughts?

Well it would only be an optional extension for mirroring whole directory structures, and wouldn't be a mandatory feature. So I don't see it being too much of a problem to require XML parsing for such an extension. However...

The alternative is some kind of wildcarded Link: header with a version=1.2 or similar parameter, which can then be compared against a .version file (or similar) on the mirror - putting the version controlling in the publisher's court (which I think is a reasonable assumption to make).

> >>    use the Link header [draft-nottingham-http-link-header] to indicate
> >>    them.  They also MUST provide checksums of files via Instance
> > Digests
> >>    in HTTP [RFC3230].  Mirror and checksum information provided by the
> >>    originating Metalink server MUST be considered authoritative.
> >>    Metalink servers and their associated mirror servers SHOULD all
> > share
> >>    the same ETag policy, i.e. base it on the file contents (checksum)
> >>    and not server-unique filesystem metadata.
> >
> > This must be a MUST, surely, since the whole concept breaks down if we
> > don't have verification of the same resource on all servers.
> 
> Ideally, it's a MUST. In the real world, it's hard to get mirrors to
> coordinate these extra things. I realize there are some
> inconsistencies in the draft, we want some balance though. Maybe we'll
> end up having different levels, of preferred (ETag/Instance Digest)
> mirrors vs regular mirrors.
> 
> Without them, the early mismatch detection breaks down, yes, but we
> don't have that in existing Metalink XML clients, and it's not a deal
> breaker. Of course, it's an added bonus! (Chunk checksums are used,
> but that's post download). There's always the authoritative whole file
> checksum obtained from the metalink server in the original request.

Well I guess the best case above is that the client requests an Instance-Digest from every server, and breaks the download if it finds it doesn't match. Not ideal but would work. It's still much nicer to do the test with If-Match - that's what it's there for. I do feel its use should be mandated.

> >>    Mirror servers are typically FTP or HTTP servers that "mirror"
> >>    another server.  That is, they provide identical copies of (at
> > least
> >>    some) files that are also on the mirrored server.  Mirror servers
> > MAY
> >>    be Metalink servers.  Mirror servers MUST support serving partial
> >>    content.  Mirror servers SHOULD support Instance Digests in HTTP
> >>    [RFC3230].
> >
> > I'm wondering if this should be a MUST, for the reasons above. Although
> > by ensuring ETag similarity this can probably stay as a SHOULD, since
> > that would remain at the same level of reliability as current HTTP.
> 
> As above, so below. In some cases, it's hard to get mirrors to buy into.

Maybe so, but the danger of permitting this stop-gap solution is that people will stop there! 

Anyway, that Instance-Digest is a SHOULD, but the one before is a MUST. This needs deciding one way or the other :)

> >> There is also some text about Content-MD5 for partial checksums.
> >
> > Content-MD5 seems a reasonable solution to detecting errors in chunks,
> > but I am still concerned about the overhead.
> 
> Me too. Content-MD5 was removed from the latest revision.
>
> > Fixed-size chunks may be preferable here. Indeed, a client could use a
> > Metalink XML file (which has chunks and checksums defined, and
> > potentially linked to from the original request) in the event of an
> > error to detect, to the level of a fixed-size chunk, which part of the
> > file is broken and just re-fetch that chunk.
> 
> Yep.
> 
> For large files, repairing downloads might be more important than
> early file mismatch detection.

I think so too. Pointing towards some Metalink XML may be the best way forward there - trying to cram sufficient information in HTTP headers is not ideal.

> > I don't understand what you mean here, there's almost no difference
> > between the behaviour of what we wrote in multiserver HTTP and your
> > Metalink Headers.
> >
> > Sure, we specified custom headers, but only because I was unaware that
> > Link: was sufficiently appropriate. It was always the intention (but may
> > not have been clear) that the mirror URLs could be included in a
> > metadata file on the web server edited by the user, and inserted by the
> > server into the headers. The only requirement on coordination between
> > servers is the same checksum and X-If-Checksum-Match: (as it was)
> > behaviour.
> 
> I apologize if I've mis-characterized what you wrote; it's been a
> while since I read it and I've been focused on this.
> 
> Don't all servers have to be running the same multi-server software
> etc? That isn't doable in some cases. But it is really nice and
> preferred!

We only required them to be providing the same checksums. The version header was for the version of multi-server HTTP (not the software itself), and was just for capability detection, as I described below...

> > The second issue with initial handshake is what to do regarding sending
> > data in the initial response.
> >
> > In Multi-Server HTTP, we proposed that the client should declare its
> > capability with a custom header (X-Multiserver-Version), at which point
> > the server knows to respond with the Mirrors list, and to start
> > transmitting data immediately but only of a chunk size (Content-Range)
> > that it is comfortable with. After this, the client will decided what
> > sizes of chunk it wants from each server and autonomously starts
> > fetching them.
> 
> The different sized chunks are really nice!
> 
> > It's not clear in your draft how you handle this. Reading Section 7 it
> > sounds as if no data apart from the header is sent, leaving the client
> > to start requesting Ranges, but this would break non-multilink clients.
> >
> > So a possible compromise may be for the client to request a HEAD only,
> > get all the relevant metadata and then. Not ideal since there's a RTT
> > delay before the data transfer can start, but that's not a major issue
> > for big resources, only lots of little requests.

This issue is still unanswered. What were your intentions on this question so far, and what do you think of my solution above?

> > There is also no mechanism for a server to specify priorities of mirrors
> > (yet). This could be very useful, even to the extent of being able to
> > request a client does not download anything from itself since it is just
> > a broker, or overloaded.
> 
> Right now, mirrors are listed in order of priority but more detailed
> information could be useful.

The prio parameter looks like a fine solution.

> >>     * Content-MD5 for chunk checksums could lead to many random size
> >> chunk checksum requests. Use consistent chunk sizes?
> >
> > Random sized chunks would be preferred since they allow a client to load
> > balance according to the speed of each mirror, but the overhead for the
> > servers to generate these arbitrary, non-cacheable checksums is
> > moderately high.
> 
> Exactly.
> 
> > Compromise: a client can request random sized chunks but they don't
> > necessarily get it with a Content-MD5 header.
> >
> > Question is, how does a client know what the "approved" chunk size is?
> 
> We could go by file size?

Possibly, something like chunks at 10% of file size for any files > 5MB or something. But I can see that threshold needing to change as bandwidth continues to become more plentiful.

What's more, it doesn't take into account load on servers, RTTs, etc.

Hmm. I don't immediately have a solution! Maybe this would again have to be done via Metalink XML that specifies the chunk size.

Regards,
Alan

-- 
Roke Manor Research Ltd, Romsey,
Hampshire, SO51 0ZN, United Kingdom

A Siemens company
Registered in England & Wales at:
Siemens plc, Faraday House, Sir William Siemens Square,
Frimley, Camberley, GU16 8QD. Registered No: 267550
------------------------------------------------------------------------
Visit our website at www.roke.co.uk
------------------------------------------------------------------------
The information contained in this e-mail and any attachments is
proprietary to Roke Manor Research Ltd and must not be passed to any
third party without permission. This communication is for information
only and shall not create or change any contractual relationship.
------------------------------------------------------------------------

Please consider the environment before printing this email

---

From: Anthony Bryan <[email-masked]>
Date: Fri, 2 Oct 2009 18:55:28 -0400

> The alternative is some kind of wildcarded Link: header with a version=1.2 or similar parameter, which can then be compared against a .version file (or similar) on the mirror - putting the version…

What about something simple(?) like "depth" (or something better
named) where "depth=0" is the default and means ONLY that file is
mirrored. A value of 1 means that file and everything else in the
directory are mirrored. 2 means the directory above, and everything in
it, etc

   Link: <http://www2.example.com/dir/dir2/example.ext>;
rel="duplicate"; depth=2

In the above example, /dir2/* would be mirrored...

> Well I guess the best case above is that the client requests an Instance-Digest from every server, and breaks the download if it finds it doesn't match. Not ideal but would work. It's still much ni…

Yes. I agree it's use should be mandated, if it is feasible in the real world.

If it means no one will end up using this, then I'd rather have it
used because it's better than things are now.

> Anyway, that Instance-Digest is a SHOULD, but the one before is a MUST. This needs deciding one way or the other :)

Ok...

Yep.

> This issue is still unanswered. What were your intentions on this question so far, and what do you think of my solution above?

That was an omission and fixed in a later draft. Data is sent unless
it's a HEAD request.

We can use a custom header to negotiate things, but the nice part is
that all this can be sent w/o negotiating I think, if you wanted to do
things that way.

> The prio parameter looks like a fine solution.

Indeed, cheers to Mark Nottingham. I added that to the draft.

> Hmm. I don't immediately have a solution! Maybe this would again have to be done via Metalink XML that specifies the chunk size.

Ok.

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Sander Steffann <[email-masked]>
Date: Sat, 3 Oct 2009 12:03:39 +0200

Hi,

> In the above example, /dir2/* would be mirrored...

If I understand your explanation correctly this should be /dir/*,  
right? Or do I misunderstand you?

Thanks,
Sander Steffann

---

From: Anthony Bryan <[email-masked]>
Date: Sat, 3 Oct 2009 10:44:50 -0400

On Sat, Oct 3, 2009 at 6:03 AM, Sander Steffann <[email-masked]> wrote:
> If I understand your explanation correctly this should be /dir/*, right? Or
> do I misunderstand you?

Yes... :) Thanks for noticing that, Sander.

Link: <http://www2.example.com/dir/dir2/example.ext>; rel="duplicate"; depth=2

/dir/*

depth=1

/dir2/* (which includes the file example.ext)

depth=0

only the file example.ext.

---

From: Anthony Bryan <[email-masked]>
Date: Fri, 9 Oct 2009 12:06:10 -0400

These changes are included in the latest revision.

Henrik Nordstrom and Alan Ford are authors. The next revision will
include more merges from draft-ford-http-multi-server.

http://tools.ietf.org/html/draft-bryan-metalinkhttp

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 19 Oct 2009 15:50:10 -0400

Revision 10 includes Alan Ford's merge of draft-ford-http-multi-server
and Henrik's additions.

Downloader & tools are at
http://groups.google.com/group/metalink-discussion/web/metalinkhttp

Fishing for ideas/suggestions:

   o  Should we use Metalink/XML for partial file checksums?  That
would add XML dependency to apps for an important feature. Any other
options?
   o  Use of Link header to describe Mirrors.  Only send a few mirrors
with Link header?  Some organizations have many mirrors.
   o  Would it make more sense to use qvalue-style policies to
describe mirror priority, i.e. q=1.0 through q=0.0 ?

---

From: Micah Cowan <[email-masked]>
Date: Sun, 25 Oct 2009 15:54:36 -0700

> http://tools.ietf.org/html/draft-bryan-metalinkhttp

I took some time to look at the current draft, and have some comments below.

> with Link header?  Some organizations have many mirrors.

My thought is that an incomplete set of mirrors should be sent, along
with a link that is understood to refer to a more comprehensive list.

(other thoughts:)

sec 4:

sec 2, para 2:

   Metalink servers are HTTP servers that use the Link header
   [draft-nottingham-http-link-header] to present a list of mirrors of a
   resource to a client.  They MUST provide checksums of files via
   Instance Digests in HTTP [RFC3230], whether requested or not."

To me, this implies that a server either is, or is not, a "Metalink
server", and behaves as required for Metalink servers for any and all
resources that it provides. I expect that in practice, a given server
will usually be a "Metalink server" with respect to certain specific
resources, and will fail to meet the requirements of a Metalink server
with respect to other resources that are served.

sec 3.2:

I'm wondering whether a two-letter country code isn't quite granular
enough to be completely practical; for instance, a client in the US may
be closer to a mirror server in Canada or Mexico, than any available in
the US; similarly, someone in France or Germany many be closer to a
mirror in Belgium than any available in their own country.

OTOH, maybe it's "good enough"; I can't speak to any system that might
be better, apart from IP address and route analysis techniques, which
won't be helped by anything you could put in this draft RFC.

sec 3.4:

It's not clear to me how "mirror depth" should work for URIs ending in a
trailing slash.

sec 4:

It strikes me that the Link header with a "rel" param value of
"described-by" (and in fact the concept of a "Metainfo file" as a whole)
covers far too many roles, without distinguishing them sufficiently.
- From the examples, this facility can be used to provide torrent
locations, metalink description files, and cryptographic files, with
probably additional possible roles to be provided as they are invented. ;)

It would be great if a mechanism were used to explicitly describe the
role that a "metainfo file" plays, be it mirroring, torrent tracking,
providing cryptographic signatures for the resource identified; and not
rely on assumptions based on the resource's content type (which should
then be optional). Other values for the "rel" param might meet this need.

various:

I recommend that references to the term IRI be replaced by URI
throughout the document, except perhaps section 10.1, "URIs and IRIs"
itself (though I also object, if much more mildly, to the requirement
that clients handle IRIs, as this may not make sense in all cases. A
SHOULD use, perhaps? Really, it's a user interface issue, so I'm not
sure it belongs in this RFC). I don't believe IRIs have a place in HTTP
headers, where they ought to be converted to their unambiguous
corresponding URI form first. AFAICT, IRIs are best constrained to user
interfaces (entry and presentation), and their use to identify resources
between non-human mechanisms should generally be avoided.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
Maintainer of GNU Wget and GNU Teseq
http://micah.cowan.name/