Subject: GOOGLE SAVES THE WORLD - Range request utility: Chrome vs FF vs IE vs Safari

From: <[email-masked]>
Date: Tue, 8 Apr 2014 15:31:33 +0000

Pardon the sensational subject line.

On Mon, 24 Mar 2014 10:26:19 -0700, Martin Thomson wrote: "Range requests serve a very narrow purpose..."
On Mon,  7 Apr 2014 14:13:54 +0800, David Krauss wrote: "Range requests don't get cached because they are uncommon..."
On Wed,  9 Apr 2014 00:12:35 +1000, Matthew Kerwin wrote: "To my mind browsers don't play any part in range discussions ... a browser wants to receive the whole object..."

We want to dispel this notion that range requests are a pathological feature of HTTP.

To quote from the introduction of HTTP-p7:
HTTP clients often encounter interrupted data transfers as a result of canceled requests or dropped connections.  When a client has stored a partial representation, it is desirable to request the remainder of that representation in a subsequent request rather than transfer the entire representation. ... [1]

For example, slow or less-reliable mobile device networks make partially downloaded content common.

To illustrate the usefulness of range requests, we investigated the behavior of the major browsers (Chrome, IE, Firefox, Safari) with respect to dropped/canceled transfers.

Specifically, we investigated the behavior when the transfer is dropped while receiving:
1) displayable content (e.g. web page), and
 2) downloadable content (e.g. installation package).

The results were mixed. The only browser which fully takes advantage of range requests is Chrome.

Why does Chrome go through all the trouble to take full advantage of range requests?  We don't believe their main motivation is to save the world. It's a competitive advantage to have a browser with a better user experience. We believe, as do the browsers (at least Chrome), that range requests should be used as much as possible, whenever it makes sense - even if its altruistic and not capitalistic :).

Chrome
Caching: Saves all content in its content-encoded format (e.g. gzip or identity)
Displayable Content: Sends range requests for partial entities stored in the cache if and only if the original response contained the Content-Length header - which somehow signifies the content isn't dynamic. It does not matter if the transfer was canceled by closing the browser or externally and it does not matter if C-E is gzip or identity. The only disappointment is that Chrome actually sends two range requests. The first one asks for a single byte after the already downloaded portion and if successful the rest of the content by specifying the exact range (we can guess why they do this, but the motivation isn't entirely clear).
Downloadable Content: In case of downloads Chrome seems to have a bug: It sends a range request, but the range starts at an offset much smaller than size of the content downloaded so far.

Below is an example of how Chrome sends two range requests for a .css file from latimes.com which is encoded with C-E gzip (superfluous headers have been omitted)...

GET /hive/stylesheets/content.css HTTP/1.1
Accept-Encoding: gzip,deflate,sdch
Range: bytes=55620-55620
If-Range: "3349f-4ef52e61f2480"

HTTP/1.1 206 Partial Content
Server: Apache
ETag: "3349f-4ef52e61f2480"
Accept-Ranges: bytes
Content-Encoding: gzip
Content-Range: bytes 55620-55620/56410
Content-Length: 1

GET /hive/stylesheets/content.css HTTP/1.1
Accept-Encoding: gzip,deflate,sdch
Range: bytes=55620-56409
If-Range: "3349f-4ef52e61f2480"

HTTP/1.1 206 Partial Content
Server: Apache
ETag: "3349f-4ef52e61f2480"
Accept-Ranges: bytes
Content-Encoding: gzip
Content-Range: bytes 55620-56409/56410
Content-Length: 790

World-wide Internet usage data are difficult to find, but since Chrome has a large market share of browsers (desktop & mobile), it is safe to assume that saving bytes with range requests for dropped/canceled transfers adds up to a total savings of a large number of bytes per day. What is clear is that Google cares deeply about saving bandwidth. They were one of the first to use C-E gzip on their servers and still heavily use it today. Assume Google has 6G search requests per day [2]. The average Google search result page has ~240 kB size -> gzipped ~60 kB -> saved 180 kB. So 6G * 180 kB > 1 PB = 1,000 TB = 1,000,000,000,000,000 B save per day.

Firefox
Caching: Saves all content in its C-E format (e.g. gzip or identity)
Displayable Content: Sends range requests for partial entities stored in the cache if and only if the original response was C-E identity.
Downloadable Content: Sends range requests, but only if the user explicitly pauses the transfer.  It will always send a range request for C-E identity. It will send range requests for C-E gzip, but if and only if the original response contained the Content-Length header.

IE
Caching: Decompresses all content (i.e. stores everything with C-E identity).
Displayable Content: Sends range requests, but only if C-E identity.
Downloadable Content: Sends range requests, but only if C-E identity.

Safari
Caching: Unknown. Apple tries to hide everything :)
Displayable Content: Never sends range requests.
Downloadable Content: Never sends range requests.

[1] http://tools.ietf.org/html/draft-ietf-httpbis-p5-range-26
[2] http://www.statisticbrain.com/google-searches/

Chris

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 8 Apr 2014 14:01:00 -0700

> We want to dispel this notion that range requests are a pathological feature

I don't think that anyone has stated that.  Range requests serve a
purpose; an important but narrow purpose.

---

From: Zhong Yu <[email-masked]>
Date: Tue, 8 Apr 2014 16:28:31 -0500

Hi Chris, how do you simulate dropped connection for chrome? First I
tried to terminate the connection from the server side (before the end
of response is reached), but Chrome then reports that the download is
success. Second I tried to "pause" download on Chrome, but apparently
Chrome does not terminate the connection; the connection stays open
until the server times out and closes the connections, and then Chrome
reports the download is success.

> Firefox
> Downloadable Content: Sends range requests, but only if the user explicitly
> pauses the transfer.  It will always send a range request for C-E identity.
> It will send range requests for C-E gzip, but if and only if the original
> response contained the Content-Length header.

This is understandable. If the server does not know the length of the
entity, it is difficult to respond to a range request with 216,
because "last-byte-pos" is unknown. Of course, it's possible that the
Content-Length is unknown on the 1st request, then becomes known
subsequently. Firefox probably thinks that is uncommon.

Here's some of my observations on Firefox and IE resumed download:

1. If the 1st response has no ETag:

    Firefox: the 2nd request is not Range request.
    IE: "pause" is disabled for the 1st response

2. If responses have changing ETags:

    Firefox/IE: send "If-Range" on resumed download, so no problem there

3. If the 1st response has no Content-Length:

    Firefox: the 2nd request is not Range request.
    IE: "pause" is disabled for the 1st response

4. If the 1st response is gzip-ed with Content-Length:

    Firefox: the 2nd request is not Range request.
    IE: "pause" is disabled for 1st response; also does not show total
length or percentage of download, as if Content-Length is ignored.

Also I remember previously on an older version of Firefox, the browser
does send Range request even if the 1st response contains no
Content-Length. That's apparently not the case any more on the current
version (28.0, win7)

Zhong Yu

---

From: Zhong Yu <[email-masked]>
Date: Tue, 8 Apr 2014 16:57:08 -0500

OK, if I "cancel" a download on Chrome, then enter the URL on address
bar, Chrome seems to be able to resume the download with a Range
request.

So my observation is such:

If the 1st response has no ETag, or has no Content-Length,
FireFox/IE/Chrome will not resume the download with a Range request.

If the 1st response is gzip-ed with Content-Length, Chrome will resume
the download with a Range request; Firefox/IE will not.

Zhong Yu

---

From: <[email-masked]>
Date: Wed, 9 Apr 2014 13:32:05 +0000

Hi Zong Yu

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
>
> Hi Chris, how do you simulate dropped connection for chrome? ...
>

My research environment is a simple program (Portmap) which can control bandwidth and latency. It is listening on 8080 and needs to be set up as the proxy for the browsers. Portmap sends everything to Fiddler2 which is setup to intercept also https traffic (acts actually as a man in the middle). Specifically for that research I added a small GUI for convenience (you can change bandwidth and close connections, individual and all). I wrote the original tool a while ago. We have to deal with: high latency networks (e.g. satellite links), non-reliable connections and both. We have frequently the issue with components which work fine for reliable low latency networks, and fail, for non-reliable and/or high latency networks (e.g. HTTP/2).
If you want I can send you the binary.

>     IE: "pause" is disabled for 1st response; also does not show total length or
> percentage of download, as if Content-Length is ignored.
>

When I said if and only, I was only talking about the mentioned parameters. There  are probably some other parameters (e.g. server type) which control the behavior. Specifically Chrome tries to use range request on content with CE != identity even without ETags.

> Also I remember previously on an older version of Firefox, the browser does
> send Range request even if the 1st response contains no Content-Length. That's
> apparently not the case any more on the current version (28.0, win7)

This is actually the point I wanted to mention. The browsers try to use all benefits of the protocol. Due to wrong implementation and misunderstandings of servers and intermediaries they seem to just throw their hands up and give up all optimizations which means they end up just starting over every time because it always works, but is slower and wasteful. I was thinking it is a good opportunity to clarify these situations (e.g. better support for range requests in future browser versions).
My Chrome is v34, my OS win7/64

Chris