Subject: Re: HTTP Extensions for Simultaneous Download from Multiple Mirrors

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 27 Jul 2009 17:03:09 +0200

This draft made a bit of surprise appearance in the transport area
meeting today:

http://tools.ietf.org/html/draft-ford-http-multi-server

My initial reaction is lots of obvious overlap with other work and
misunderstandings of basic HTTP functions like ETag.

Basic motivation behind the work may be reasonable however.

I will try to catch the author for a more in-depth discussion shortly.

Other opinions?

Regards
Henrik

---

From: Sam Johnston <[email-masked]>
Date: Mon, 27 Jul 2009 21:07:53 +0200

> I will try to catch the author for a more in-depth discussion shortly.

Sounds like a job for a Link relation (rel="replica"... I'm not convinced
rel="self" is appropriate), perhaps some additional, optional attributes
(e.g. country=us) and existing headers (e.g. Content-MD5).

There's probably a lot to be learnt from protocols like BitTorrent too.

Sam

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 27 Jul 2009 22:34:56 +0200 (CEST)

On Mon, 27 Jul 2009, Henrik Nordstrom wrote:

> This draft made a bit of surprise appearance in the transport area meeting 
> today:

Mr Ford was also mentioning the possibility of using wildcards in the 
'X-Mirrors:' header so that you could get to know other sites that have files 
in the same dir tree as the master site (for purposes like getting images from 
the fastest provider), but:

A) that isn't mentioned in this draft as far as I can see

and

B) it feels like a solution to another problem and it somehow doesn't feel as 
neat and tidy as this plain suggestion with a list of URIs for the identical 
resource. Methinks.

-- 

  / daniel.haxx.se

---

From: David Morris <[email-masked]>
Date: Mon, 27 Jul 2009 17:20:45 -0700 (PDT)

I don't discern an 'official' location for discussion of this proposal.

There is a serious termnology conflict with the term 'chunk' meaning a 
slice of the original file AND a unit of transfer encoding. Perhaps some 
underlying confusion re. transfer encoding in general.

Dave Morris

---

From: Anthony Bryan <[email-masked]>
Date: Thu, 30 Jul 2009 18:28:55 -0400

> I will try to catch the author for a more in-depth discussion shortly.

Very interesting, thanks for writing about this Henrik. I hadn't seen
or heard of it.

For those unfamiliar with Metalink, we offer solutions to the same
problems (and more) in an XML format, as opposed to HTTP extensions.

So I'm interested in what people think about it, criticism, ideas, etc
because it may allow us to improve what we are doing. We're also
seeking review for our Internet Draft at
http://tools.ietf.org/html/draft-bryan-metalink

If anyone is interested in trying Metalink out, a good amount of
software is available, in the form of download managers (most popular
ones), Firefox extensions, command line clients, and browser.
While many Metalink clients, especially download managers, download
simultaneously from multiple mirrors, it's really about giving
alternate locations for a download to complete (if a server goes down)
and also repairing downloads. Information about mirrors like location
and priority can also be included.

Projects like cURL, OpenOffice.org, and most Linux distributions use
Metalinks for downloads, especially for large files.

More info here:

http://www.metalinker.org/
http://en.wikipedia.org/wiki/Metalink

Here's the intro from draft-ford-http-multi-server-00:

"1. Introduction and Motivation

   Mirrored HTTP servers are regularly used for software downloads,
   whereby copies of data to be downloaded are duplicated on many
   servers distributed around the Internet.  Users are encouraged to
   manually choose a nearby mirror from which to download.  This is
   intended to increase both throughput and resilience, and reduce load
   on individual servers.  Manual mirror choice rarely works well; users
   do not wish to make a choice, but if they are not forced to, then the
   default server takes a disproportionate share of the load.  Even when
   they are forced to choose, they rarely have enough information to
   choose the server that will provide the best performance.

   Some popular sites automate this process using DNS load balancing,
   both to approximately balance load between servers, and to direct
   clients to nearby servers with the hope that this improves
   throughput.  Indeed, DNS load balancing can balance long-term server
   load fairly effectively, but it is less effective at delivering the
   best throughput to users when the bottleneck is not the server but
   the network.

   This document specifies an alternative mechanism by which the benefit
   of mirrors can be automatically and more efficiently realised.  These
   benefits are achieved using a number of extensions to HTTP which
   allow the discovery of mirrors, the verification of the integrity of
   files on each mirror, and the simultaneous downloading of chunks from
   multiple mirrors.  The use of this mechanism allows greater
   efficiency in resource utilisation in the Internet as a whole,
   balances server utilization, even on short timescales, and enhances
   user experience through faster downloads."

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 31 Jul 2009 11:11:43 +1200

Hi

My only comment, which springs to mind when I see wording about
downloading files in parts.

Proxies that scan for malware.

In order to allow scanning, the entire entity is required.  Therefore
range requests must be modified by the proxy.

Typically this is done by stripping the Range header in the upstream
request, and possibly stripping Accept-Ranges from server responses.

So, any system proposed that is going to use Range requests, is going
to run into problems with proxies that perform AV functions.

Even if the proxy still sends just the requested parts to the client
after downloading the whole thing and scanning it, which may satisfy
the client in terms of response, but will not satisfy it in terms of
responsiveness, retrieving parts from multiple servers will have a
hugely negative effect, since the file will cause the download the
entire file from each server that it thinks it's only getting a part of
the file from.

This is therefore only ever going to become an increasing problem,
since network administrators aren't going to be abandoning antivirus
any time soon.

I don't know what the best solution to this is.  Unless the proxy knew
that the parts were of the same file, and could piece it together
itself to scan it, it can't save that resource use.

And there's no way around getting the whole file first if you want to
use heuristic scanning for malware at the gateway.  This creates other
problems which I attempted to address in an Internet Draft a while back
(draft-decroy-http-progress).

Regards

Adrien

Anthony Bryan wrote:

Other opinions?

More info here:

--
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 31 Jul 2009 01:28:43 +0200

> to run into problems with proxies that perform AV functions.

Of course, but it's trivially detected by the first response being a 200
instead of the expected 206, before going out in parallel, so I don't
see it as a big problem for the proposed mechanism. In addition download
managers etc have to deal with this today already and it's by no means a
new problem.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 31 Jul 2009 17:34:34 +1200

Sure it's a known problem.

But building more systems on top of it just makes it a more wide-spread
known problem.

That's when knowledge of the problem enters the customer domain, which
causes support issues for proxy vendors.

Also RFC2616 mentions recommended behaviour of proxies that retrieve
the whole entity, that they should send a 206 back to the client with
just the part requested.

If they do this, then the client won't know the proxy is doing it, and
if say you were downloading a 100MB file off 20 sites, you'd end up
downloading the whole thing 20 times.  Including waiting each time for
the whole entity to get the part, since the URIs would be different,
even if the proxy cached it, it wouldn't know it was the same file
coming from different sources.

In which case... maybe an extra header to make it explicit that this is
what's going on so a proxy can act accordingly - with some sort of
universally unique key for the file (e.g. some meta URI that's not the
actual URI requested but which identifies the entity).  So that the
proxy can know it's the same file being requested from multiple
different locations, and just return the pieces from the cached version
retrieved from the first request.

Adrien

Henrik Nordstrom wrote:

Regards
Henrik

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 31 Jul 2009 09:26:44 +0200 (CEST)

On Fri, 31 Jul 2009, Adrien de Croy wrote:

> Also RFC2616 mentions recommended behaviour of proxies that retrieve the 
> whole entity, that they should send a 206 back to the client with just the 
> part requested.

While I can see the merit in doing this for AV reasons, it makes very very 
little sense for a the case when mt DVD download failed because the last 200 
bytes were ruined (I mean at storage level on the server) and all you want is 
to download those 200 bytes from a second mirror.

But as Henrik said, lots of "download accelerators" already work exactly like 
this and clearly it works fine enough for a certain amount of people.

-- 

  / daniel.haxx.se

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 31 Jul 2009 10:07:48 +0200

> the whole entity, that they should send a 206 back to the client with

Right, that one. We should open an issue on that as it's not really
practical to implement for reasons somewhat unrelated to this.

> In which case... maybe an extra header to make it explicit that this
> is what's going on so a proxy can act accordingly - with some sort of
> different locations, and just return the pieces from the cached
> version retrieved from the first request.

All of which the proposal do include..

but the scheme you mention is a bit risky from a cache pollution aspect,
without first verifying this with the second resource location.

Regards
Henrik