Subject: Re: Multi-server HTTP

From: Mark Nottingham <[email-masked]>
Date: Tue, 1 Sep 2009 17:08:51 +1000

Totally; we just need to be crisp about it.

My inclination would be that if we can be more inclusive without  
making it significantly more complex or risky, we should; otherwise,  
just do what's needed.

Cheers,

On 01/09/2009, at 1:49 PM, Nicolas Alvarez wrote:

> Mark Nottingham wrote:
>> That's a good start, but it deserves a bit of discussion.
>>
>> "byte-for-byte" implies that the bodes are the same, but what about
>> things like:
>>
>> * Entity headers (e.g., Content-Type)
>> * Available content-encodings
>> * Whether partial content is supported
>> * Whether the same set of methods are supported (e.g., if A is a
>> duplicate of B, will POSTing something to either have the same effect
>> as on the other?)
>>
>> I think the answer is that entity headers should generally be the
>> same, so the real question is whether we're talking about the  
>> relation
>> describing:
>>
>> a) resources with duplicate representations (i.e., a GET on any of  
>> the
>> dups will return the same reps)
>> b) duplicate resources (i.e., any method will have the same effect)
>>
>> If it's (b), we should consider whether the resources are in fact the
>> same "behind the curtains" (e.g., POSTing to A has the exact same
>> effect on the world as POSTing to B), or whether they may be in fact
>> separate systems (i.e., A and B have the same "interface", but  
>> POSTing
>> to A may affect a different part of the world to B).
>
> Well, we're talking about static GETable resources with a single
> representation. But I agree that if you make a Link relation, you'd  
> want it
> to be applicable to as many HTTP resources as possible... Or is it
> possible / reasonable to say "this relation doesn't make sense for  
> dynamic
> or POSTable resources and shouldn't be used for those"?
>
>
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 7 Sep 2009 21:19:54 -0400

Here's what I have now. More inclusive is good but I think someone
else would be better at writing it than me.

http://tools.ietf.org/html/draft-bryan-metalinkhttp

Link Relation Type Registration: "duplicate"

   o Relation Name: duplicate
   o Description: Refers to an identical resource that is a
byte-for-byte equivalence of representations.
   o Reference: This specification.
   o Notes: This relation is for static resources.  That is, an HTTP
GET request on any duplicate will return the same representation.  It
does not make sense for dynamic or POSTable resources and should not
   be used for them.

And here's the introduction (Content-MD5 is now mentioned):

   MetaLinkHeader is an alternative to Metalink, usually represented in
   an XML-based document format [draft-bryan-metalink].  MetaLinkHeader
   attempts to provide as much functionality as the Metalink XML format
   by using existing standards such as Web Linking
   [draft-nottingham-http-link-header], Instance Digests in HTTP
   [RFC3230], and Content-MD5 [RFC1864].  MetaLinkHeader is used to list
   information about a file to be downloaded.  This includes lists of
   multiple URIs (mirrors), Peer-to-Peer information, checksums, and
   digital signatures.

Here's what it looks like:

   Link: <http://www2.example.com/example.ext>; rel="duplicate";
   Link: <ftp://ftp.example.com/example.ext>; rel="duplicate";
   Link: <http://example.com/example.ext.torrent>; rel="describedby";
   type="torrent";
   Link: <http://example.com/example.ext.asc>; rel="describedby";
   type="application/pgp-signature";
   Digest: SHA=thvDyvhfIqlvFe+A9MYgxAfm1q5=

And more description:

   Metalink servers are HTTP servers that MUST have lists of mirrors and
   use the Link header [draft-nottingham-http-link-header] to indicate
   them.  They also MUST provide checksums of files via Instance Digests
   in HTTP [RFC3230].  Mirror and checksum information provided by the
   originating Metalink server is considered authoritative.

   Mirror servers are typically FTP or HTTP servers that "mirror"
   another server.  That is, they provide identical copies of (at least
   some) files that are also on the mirrored server.  Mirror servers MAY
   be Metalink servers.  Mirror servers MUST support serving partial
   content.  Mirror servers SHOULD support Instance Digests in HTTP
   [RFC3230].

   Metalink clients use the mirrors provided by a Metalink server with
   Link header [draft-nottingham-http-link-header].  Metalink clients
   MUST support HTTP and MAY support FTP, BitTorrent, or other download
   methods.  Metalink clients MUST switch downloads from one mirror to
   another if the one mirror becomes unreachable.  Metalink clients are
   RECOMMENDED to support multi-source, or parallel, downloads, where
   chunks of a file are downloaded from multiple mirrors simultaneously
   (and optionally, Peer-to-Peer).  Metalink clients MUST support
   Instance Digests in HTTP [RFC3230] by requesting and verifying
   checksums.  Metalink clients MAY make use of digital signatures if
   they are offered.

On Tue, Sep 1, 2009 at 3:08 AM, Mark Nottingham<[email-masked]> wrote:
> My inclination would be that if we can be more inclusive without making it
> significantly more complex or risky, we should; otherwise, just do what's
> needed.
>
> Cheers,
>
>
>>> same, so the real question is whether we're talking about the relation
>>> describing:
>>>
>>> a) resources with duplicate representations (i.e., a GET on any of the
>>> separate systems (i.e., A and B have the same "interface", but POSTing
>> representation. But I agree that if you make a Link relation, you'd want
>> it
>> possible / reasonable to say "this relation doesn't make sense for dynamic

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 15 Sep 2009 13:06:57 +1000

> byte-for-byte equivalence of representations.

Does this imply that each resource has exactly the same set of  
representations, or that when two resources share representations,  
those representations are duplicates?

>   Link: <http://example.com/example.ext.torrent>; rel="describedby";

Do torrents have media types yet?

>> making it

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 14 Sep 2009 23:59:19 -0400

On Mon, Sep 14, 2009 at 11:06 PM, Mark Nottingham <[email-masked]> wrote:
> representations, or that when two resources share representations, those
> representations are duplicates?

The latter.

Any suggestions for replacement text? Because what I have isn't cutting it.

> Do torrents have media types yet?

Not as far as I know.

Which is also why in draft-bryan-metalink we have this:

4.2.10.2. The "type" Attribute

   metalink:metaurl elements MUST have a "type" attribute that indicates
   the MIME type of the metadata available at the IRI.  In the case of
   BitTorrent as specified in [BITTORRENT], the value "torrent" is
   required.  Types without "/" are reserved.  Currently, "torrent" is
   the only reserved value.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Sep 2009 16:55:00 +1000

> Any suggestions for replacement text? Because what I have isn't  

Hm.

Refers to a resource whose available representations are byte-for-byte  
identical with the corresponding representations of the context IRI.

>   required.  Types without "/" are reserved.  Currently, "torrent" is

Overloading type like that is bad; register a media type (or get the  
appropriate people to do it).

Cheers,

---

From: Anthony Bryan <[email-masked]>
Date: Thu, 17 Sep 2009 18:00:47 -0400

> identical with the corresponding representations of the context IRI.

Thank you, sir. Much appreciated.

For anyone interested, you can try out Metalink in HTTP headers w/ the
software from here:

http://metalinks.svn.sourceforge.net/viewvc/metalinks/checker/
http://metalinks.svn.sourceforge.net/viewvc/metalinks/webconvert/
(Python script to convert .metalink to Apache directives)

> appropriate people to do it).

This was from Julian's comment:
http://www.ietf.org/mail-archive/web/apps-review/current/msg00155.html

BitTorrent uses "application/x-bittorrent". What would I register?
"application/vnd.bittorrent"?

Probably worse, current Metalink uses "application/metalink+xml" from
when we didn't know any better. Should I register something else?

---

From: Sam Johnston <[email-masked]>
Date: Fri, 18 Sep 2009 01:45:53 +0200

> "application/vnd.bittorrent"?

Bittorrent specifies <http://bittorrent.org/beps/bep_0003.html>
application/x-bittorrent
which is ok by RFC 2045 <http://tools.ietf.org/html/rfc2045#section-5.1>:

Private values (starting with "X-") may be defined bilaterally between two
> cooperating agents without outside registration or standardization. Such
> values cannot be registered or standardized.

You don't have the authority to register the type on their behalf anyway
(unless of course you do) and to do so would almost certainly cause
interoperability problems for implementors and users.

I think using a non-standard MIME type rather than overloading "type" is by
far the lesser of two evils...

Sam

---

From: Anthony Bryan <[email-masked]>
Date: Sat, 19 Sep 2009 13:12:40 -0400

On Fri, Aug 28, 2009 at 11:27 AM, Henrik Nordstrom
<[email-masked]> wrote:
> fre 2009-08-28 klockan 12:38 +0100 skrev Ford, Alan:
>
> So I would recommend the following slightly different approach to your
> problem.
>
> * Define a new Mirror profile object, similar to MetaLink but defining
> the mirror URL policy for groups of URLs on the server, without going
> into checksums etc (HTTP will give those).
>
> * Instance-Digest header returning the object checksum
>
> * HTTP addendum that servers participating in this mirror scheme should
> all share the same ETag policy, i.e. base it on the file contents and
> not server-unique filesystem metadata..

Henrik, I have added your suggestions about ETags to my draft (
http://tools.ietf.org/html/draft-bryan-metalinkhttp ) almost verbatim.
I didn't try to reword it, and if this is a problem, let me know.
I am looking for interested collaborators and co-authors, and you've
provided great insight. Would you like to join us?

Here is the current description:

   Metalink servers are HTTP servers that MUST have lists of mirrors and
   use the Link header [draft-nottingham-http-link-header] to indicate
   them.  They also MUST provide checksums of files via Instance Digests
   in HTTP [RFC3230].  Mirror and checksum information provided by the
   originating Metalink server MUST be considered authoritative.
   Metalink servers and their associated mirror servers SHOULD all share
   the same ETag policy, i.e. base it on the file contents (checksum)
   and not server-unique filesystem metadata.  The emitted ETag may be
   implemented the same as the Instance Digest for simplicity.

   Metalink clients use the mirrors provided by a Metalink server with
   Link header [draft-nottingham-http-link-header].  Metalink clients
   MUST support HTTP and MAY support FTP, BitTorrent, or other download
   methods.  Metalink clients MUST switch downloads from one mirror to
   another if the one mirror becomes unreachable.  Metalink clients are
   RECOMMENDED to support multi-source, or parallel, downloads, where
   chunks of a file are downloaded from multiple mirrors simultaneously
   (and optionally, from Peer-to-Peer sources).  Metalink clients MUST
   support Instance Digests in HTTP [RFC3230] by requesting and
   verifying checksums.  Metalink clients MAY make use of digital
   signatures if they are offered.

There is also some text about Content-MD5 for partial checksums.

> 4. If the object checksum does not match the instance-digest then fetch
> the recovery profile link, where partial checksums etc can be found
> allowing detection of which server returned bad information.

What do you suggest as a recovery profile link? A text file with
partial checksums? Metalink XML could also be used, but do you think
the XML dependency adds too much?

As mentioned before, you can try out Metalink in HTTP headers w/ the
software from here:

I have read draft-ford-http-multi-server and my main comment is that
the required coordination of all mirror servers may be difficult or
impossible unless you are in control of all servers on the mirror
network.
I don't see this as possible in the open source mirror networks that I
follow, but might be for commercial CDNs? In any case, this
coordination is not required in my draft.

Finally, here are some issues with my own draft:

    * Mirror negotiation. Only send a few mirrors, or only send them
if Want-Digest is used? Some organizations have many mirrors.
    * Some publishers desire stronger hashes than MD5 and SHA-1.
    * Content-MD5 for chunk checksums could lead to many random size
chunk checksum requests. Use consistent chunk sizes?
    * Do we want a way to show that whole directories are mirrored,
instead of individual files?

Comments/suggestions on these and other issues you may discover are welcome!
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: "Ford, Alan" <[email-masked]>
Date: Thu, 24 Sep 2009 13:23:40 +0100

Hi Anthony, all,

I have now had some time to read this latest draft. It seems a nice way
of linking the multi-server and metalink concepts. I am pleased that
you've considered our Multi-server HTTP ideas in this and we'd be happy
to collaborate further to create a standardised way of achieving this
efficiency across servers.

I have several comments about the operation, but I'll come to those
after the points in the below email...

> > into checksums etc (HTTP will give those).

Henrik's point here goes back to the question of mirroring multiple
files / directory structure. This currently remains un-tackled, but I
think one idea that could be used is the use of a Link: header going to
a form of Metalink that can be used for whole mirrors. Something to come
back to, once the basic concept is clear.

>    Metalink servers are HTTP servers that MUST have lists of mirrors
and

Not sure it makes sense to define everything in this document around
being a "Metalink server". After all, this is just an extension to HTTP,
and relatively separate to the existing Metalink work (AIUI).

>    and not server-unique filesystem metadata.  

This must be a MUST, surely, since the whole concept breaks down if we
don't have verification of the same resource on all servers.

>    The emitted ETag may be

OK so the purpose of this is so that we can use the If-Match: header for
matching digests. I can't immediately think there'd be any problems with
this but are there any server implementations that use ETags for any
other purpose and so this may break their assumptions? I judged from the
original responses I got that people were generally against new headers
if at all possible so this is almost certainly the best solution so long
as it doesn't hinder deployment.

>    content.  Mirror servers SHOULD support Instance Digests in HTTP

I'm wondering if this should be a MUST, for the reasons above. Although
by ensuring ETag similarity this can probably stay as a SHOULD, since
that would remain at the same level of reliability as current HTTP.

> There is also some text about Content-MD5 for partial checksums.

Content-MD5 seems a reasonable solution to detecting errors in chunks,
but I am still concerned about the overhead.

Fixed-size chunks may be preferable here. Indeed, a client could use a
Metalink XML file (which has chunks and checksums defined, and
potentially linked to from the original request) in the event of an
error to detect, to the level of a fixed-size chunk, which part of the
file is broken and just re-fetch that chunk.

> coordination is not required in my draft.

I don't understand what you mean here, there's almost no difference
between the behaviour of what we wrote in multiserver HTTP and your
Metalink Headers.

Sure, we specified custom headers, but only because I was unaware that
Link: was sufficiently appropriate. It was always the intention (but may
not have been clear) that the mirror URLs could be included in a
metadata file on the web server edited by the user, and inserted by the
server into the headers. The only requirement on coordination between
servers is the same checksum and X-If-Checksum-Match: (as it was)
behaviour.

> if Want-Digest is used? Some organizations have many mirrors.

Right, this is part of a more general issue of a server's behaviour on
initial connection. Want-Digest would be a good clue, but is not a
guarantee, about the client's intentions. However, if we are certain the
client wants the Mirror list, and the resource is large enough to
warrant their use, then I feel all should be sent, unless the server has
its own priorities regarding balancing the load across the mirrors.

The second issue with initial handshake is what to do regarding sending
data in the initial response.

In Multi-Server HTTP, we proposed that the client should declare its
capability with a custom header (X-Multiserver-Version), at which point
the server knows to respond with the Mirrors list, and to start
transmitting data immediately but only of a chunk size (Content-Range)
that it is comfortable with. After this, the client will decided what
sizes of chunk it wants from each server and autonomously starts
fetching them.

It's not clear in your draft how you handle this. Reading Section 7 it
sounds as if no data apart from the header is sent, leaving the client
to start requesting Ranges, but this would break non-multilink clients.

So a possible compromise may be for the client to request a HEAD only,
get all the relevant metadata and then. Not ideal since there's a RTT
delay before the data transfer can start, but that's not a major issue
for big resources, only lots of little requests.

There is also no mechanism for a server to specify priorities of mirrors
(yet). This could be very useful, even to the extent of being able to
request a client does not download anything from itself since it is just
a broker, or overloaded.

Is there a recognised extension to Link: that could be used for
specifying priority? (Mark?)

>     * Some publishers desire stronger hashes than MD5 and SHA-1.

RFC3230 is extensible, new digest algorithms could be added to the IANA
registry (via RFC) if required.

> chunk checksum requests. Use consistent chunk sizes?

Random sized chunks would be preferred since they allow a client to load
balance according to the speed of each mirror, but the overhead for the
servers to generate these arbitrary, non-cacheable checksums is
moderately high.

Compromise: a client can request random sized chunks but they don't
necessarily get it with a Content-MD5 header. 

Question is, how does a client know what the "approved" chunk size is?

> instead of individual files?

I would still like to see this, as I wrote earlier. But however we do
it, it will require significant metadata if we want to ensure each file
being mirrored is the same as what the original thinks it is.

Unless, we just have versioning of the directory structure? E.g. a
publisher puts a ".version" file in a directory, entirely arbitrary, and
we verify that value is the same on the mirrors, not using a checksum at
all. Not perfect, but a reasonable compromise?

Finally, do you intend that this draft should contain recommendations of
what to do at the client end - e.g. behaviour on checksum failure? While
I can see this is not normative for a document such as this, some
guidelines may be useful.

Regards,
Alan

-- 
Roke Manor Research Ltd, Romsey,
Hampshire, SO51 0ZN, United Kingdom

A Siemens company
Registered in England & Wales at:
Siemens plc, Faraday House, Sir William Siemens Square,
Frimley, Camberley, GU16 8QD. Registered No: 267550
------------------------------------------------------------------------
Visit our website at www.roke.co.uk
------------------------------------------------------------------------
The information contained in this e-mail and any attachments is
proprietary to Roke Manor Research Ltd and must not be passed to any
third party without permission. This communication is for information
only and shall not create or change any contractual relationship.
------------------------------------------------------------------------

Please consider the environment before printing this email

---

From: Nicolas Alvarez <[email-masked]>
Date: Fri, 25 Sep 2009 02:00:25 -0300

Ford, Alan wrote:
> Ant Bryan wrote:

draft-hodges-strict-transport-sec-05 (Google it) talks about "STS Servers"
all the time, even though it defines "STS Server" as just an HTTP server
that is sending a certain HTTP header.

---

From: Anthony Bryan <[email-masked]>
Date: Fri, 25 Sep 2009 18:17:04 -0400

> efficiency across servers.

Excellent!

> after the points in the below email...

Thank you for taking the time to review it & respond to both drafts.

> back to, once the basic concept is clear.

Right, nothing so far requires XML though. Do we want to add that dependency?

I'd like tools like wget & curl to be able to support this (if they
choose to) w/o adding extra dependencies unless required.

Thoughts?

> and relatively separate to the existing Metalink work (AIUI).

AIUI existing metalink work is about improving downloads. This has
mostly been done with mirrors and checksums (& more goodies when
possible). I don't see much difference.
That's why it took almost 20 lines of code to update an existing
metalink XML client to support this new draft.

A "Metalink server" by any other name would function just as sweet, as
they say. :)

As you can tell from the draft, the naming isn't totally consistent
throughout. We've used "Metalink in HTTP Headers", "MetaLinkHeader",
"Metalink/HTTP" and as we mention it is mostly a
collection/coordination of features of HTTP.

Any suggestions? Cool acronyms? :)

> don't have verification of the same resource on all servers.

Ideally, it's a MUST. In the real world, it's hard to get mirrors to
coordinate these extra things. I realize there are some
inconsistencies in the draft, we want some balance though. Maybe we'll
end up having different levels, of preferred (ETag/Instance Digest)
mirrors vs regular mirrors.

Without them, the early mismatch detection breaks down, yes, but we
don't have that in existing Metalink XML clients, and it's not a deal
breaker. Of course, it's an added bonus! (Chunk checksums are used,
but that's post download). There's always the authoritative whole file
checksum obtained from the metalink server in the original request.

> that would remain at the same level of reliability as current HTTP.

As above, so below. In some cases, it's hard to get mirrors to buy into.

> but I am still concerned about the overhead.

Me too. Content-MD5 was removed from the latest revision.

> file is broken and just re-fetch that chunk.

Yep.

For large files, repairing downloads might be more important than
early file mismatch detection.

> servers is the same checksum and X-If-Checksum-Match: (as it was)

I apologize if I've mis-characterized what you wrote; it's been a
while since I read it and I've been focused on this.

Don't all servers have to be running the same multi-server software
etc? That isn't doable in some cases. But it is really nice and
preferred!

> its own priorities regarding balancing the load across the mirrors.

Yes.

> sizes of chunk it wants from each server and autonomously starts

The different sized chunks are really nice!

> request a client does not download anything from itself since it is just

Right now, mirrors are listed in order of priority but more detailed
information could be useful.

> registry (via RFC) if required.

These were included in the latest revision.

> servers to generate these arbitrary, non-cacheable checksums is

Exactly.

> Question is, how does a client know what the "approved" chunk size is?

We could go by file size?

> all. Not perfect, but a reasonable compromise?

I really don't know what the best solution is. We don't have any
experience with this issue so far.

> guidelines may be useful.

There is a short "Error Recovery" section that will be corrected and expanded.

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 26 Sep 2009 00:20:06 +0200

> Right, nothing so far requires XML though. Do we want to add that dependency?

I don't really have an opinion on what kind of language to use. As long
as it can describe the needed data and is extensible enough to allow
future revisions without breaking existing users...

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 27 Sep 2009 08:20:56 +1000

No. I'd just do something like

Link: <http://www.example.com/foo>; rel="duplicate"; pri=5

Cheers,