Subject: clarity needed for browser behaviors when receiving 206 status codes

From: Marques Johansson <[email-masked]>
Date: Thu, 24 Jun 2010 22:20:07 -0400

The HTTP specification for ranges does not deal with a few range
related matters that I encountered while attempting to create a
streaming video script.

How can servers inform clients that all requests must be made with
ranges specified?

How should a client or server respond when the server is not willing
to fulfill the entire range requested by the client?

When a client makes a GET request without sending a Range: header, the
server can still reply with a 206 status, providing an Accept-Ranges
header, and providing some arbitrary segment of data (hopefully
starting at range byte 0).  This seems to catch many of todays
browsers off guard.  They use the data they were given, but they do
not follow up with subsequent Range requests.

According to the spec, the server can only reply with a 416 if a Range
was requested.  There should be a response status the server can use
in cases where no range was specified but a range is required.

A way to prime some web browsers to send Range: requests is to send
the Accept-Ranges: header in a 307 redirect response to the initial
un-ranged request.  The behavior of following this redirect and
sending a partial request is not taken by many text based browsers but
appears to be taken by modern desktop browsers.

Now that the browser is sending a Range with its request, what happens
when the server does not wish to return all of the data requested.
All browsers send something like: "Range: bytes=0-".  When the server
replies with an incomplete result in a 206 response browsers tend to
stop fetching (not always true - Chrome continues to fetch).  It seems
that the specification, by not bringing this point up, would have a
server respond with a 416 specifying a corrected range which the
browser would then request.  This doubles the transaction count which
is especially problematic for scenarios where the server is only
willing to return segments of content with a specific size.

If the server could respond with a "Accept-Range-Max-Length:" then
there would only be need for a single 416 response or this header
could be included with the 307 response.

In the section on content negotiation the spec seems to favor that the
client should do its best to process the response that the server
supplied.  Chrome appears to be doing this in the case of 206
responses to un-ranged requests.

The spec should permit and document a method for handling 206
responses in this continues way or provided a means for the server to
inform the client that ranges must be specified

The spec should also prescribe a behavior for handling incomplete
range results or a method for the server to state their intention to
return constrained length responses.

It should also be considered that a server could forever send a client
the wrong response data, or only tiny responses, and cause a denial of
service on the client - however, this is not too different from any
case where the server would return a simple large response.  The
client always has the option to refuse fetching the range.

See these postings for additional commentary:
 https://bugzilla.mozilla.org/show_bug.cgi?id=570755
 http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2010-June/026837.html

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 25 Jun 2010 05:47:33 +0200

>How can servers inform clients that all requests must be made with

As far as byte ranges are concerned, if the server is willing to accept
a range request for the "whole" resource, then the server should not in-
sist on the client making a range request. If it is not then the server
would need to tell the client what ranges are available, but that seems
to be more a 300 Multiple Choices kind of situation.

>to fulfill the entire range requested by the client?

That depends on why it is not willing and possibly what could sway the
server to satisfy the request. Payment Required, Service Unavailable,
Unauthorized, and so on. In doubt 403 Forbidden with an Accept-Ranges
header in the response and a suitable entity body would seem sensible;
that also goes for your first question.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Marques Johansson <[email-masked]>
Date: Fri, 25 Jun 2010 07:47:32 -0400

> to be more a 300 Multiple Choices kind of situation.

The server may be willing to accept a range for the "whole" resource
if the user purchased access to the whole video rather than paying by
the byte/second/minute.  That is really a different matter and one
that would not create any problems.

In this case, the server is not willing to accept an un-ranged request
or range request for the "whole" resource. The server wants to give
the user no more content then the user and host service have
authorized.

Example: The user purchases 10 minutes of time from a video on demand
service.  The server based on some agreement will give the browser no
more than 5 seconds of video at a time in the form of a partial
result.  This sets a streaming heartbeat that prevents the browser
from fetching the entire file and thus deducting the user for content
they did not want.  The server can also throttle the client if it
detects that the user agent is attempting to sequentially request data
faster than real-time (in terms of the video's play position).  In the
case of Chrome and the Ogg handler in Opera, the ranged requests come
from the user-agent as the content is needed - they do not attempt to
prefetch the data (the HTML video element provides attributes for
this).

There are three problems with a 300 in this case.
- there is no spec defined follow up behavior - a 307 on the other
hand informs the UA to try again
- all multiple choices have the same URL: file.mp4 (only Accept-Ranges
would differ)
- there are millions of choices available: Range:
(x1=0...size-1)-(x2=x1+1...size).

> that also goes for your first question.

The problem with a 403 is that it creates an impasse.  There is no
spec defined behavior for a browser to receive a 403 with
Accept-Ranges headers and then re-request the content with Ranges.
The 307 response gets the ball rolling which is why I suggested that
this behavior be formalized (since some of the major UAs already
support this).

This is why I suggested a new 4xx to indicate that ranges are
required.  Any other range related problems can be handled with a 416.
 If a UA doesn't understand this new error code (or how to use ranges)
then they will naturally stop processing at this point, which is the
intention of the server.

>> responses in this continues way
...
>> return constrained length responses.

If there was a "Accept-Range-Max-Length" header then the large number
of multiple choices problem would be solved without resorting to a 416
on each request that lacked an upper bound.  Or the spec could tell
UAs to follow WebKit's lead and simply expect that the server may
return less data than requested by the Range and this should prompt
the UA to follow-up with additional range requests as needed.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Jun 2010 14:37:28 +0200

> the user no more content then the user and host service have

So you have an HTTP resource that has different access rights based the 
range/length.

I don't think that HTTP range requests have been designed for this, and 
therefore you're seeing the issues you're seeing.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Jun 2010 15:00:03 +0200

> How can servers inform clients that all requests must be made with

I don't think they can.

> to fulfill the entire range requested by the client?

A 4xx status, such as 403 Forbidden, comes to mind. However, I wouldn't 
count on any generic client coming to the conclusion that other ranges 
may work on the same resource.

> was requested.  ...

Yes.

> ...  There should be a response status the server can use

A response code could be defined. Write a spec :-)

> appears to be taken by modern desktop browsers.

Me confused.

Could you show an example? If a GET on URI1 returns a 307 with 
Accept-Ranges, and a Location of URI2, why would a client assume that 
URI2 accepts range requests?

> could be included with the 307 response.

Well, this is something HTTP range requests obviously haven't been 
designed for. I'm not convinced that they should be used for this, nor 
that making these minor changes would be sufficient to deal with your 
use case.

Best regards, Julian

---

From: Marques Johansson <[email-masked]>
Date: Fri, 25 Jun 2010 11:27:11 -0400

Yes, I don't think the HTTP range spec was specifically designed for
this but it nearly fits the application of controlled content
streaming and doesn't try to block this application.  My response is
sort of long and includes various ways to address my concerns so I
added flow interrupting headings (joy).

==  Sending smaller than requested ranges in a 206 ==

If the spec spelled out how to handle smaller responses (as WebKit
handles these) we could circumvent the unnecessary error and
redirection phase:

In Webkit it works like this:
  GET /file.mp4 HTTP/1.1
  ...
  HTTP/1.1 206 partial (smaller than requested)
  Content-Range: bytes 0-99999/300000
  ...
  GET /file.mp4 HTTP/1.1
  Range: bytes=100000-
  ...
  HTTP/1.1 206 partial (smaller than requested)
  Content-Range: bytes 100000-199999/300000

Despite http://tools.ietf.org/html/draft-ietf-httpbis-p5-range-09#section-3.1
- browsers seem to be ok with a 206 response when no Range was sent -
but this should be defined somewhere (changing 'The request MUST have
included a Range header field') or by defining a 307+Accept-Ranges
behavior or by defining a new '4xx Range Required' status code.

On Fri, Jun 25, 2010 at 9:00 AM, Julian Reschke <[email-masked]> wrote:
> Accept-Ranges, and a Location of URI2, why would a client assume that URI2
> accepts range requests?

I agree that URL1 and URL2 could exist on different servers and there
is no reason why "Accept-Ranges" should be followed - but this is way
browsers seem to work.  The behavior could be restricted to relative
or same-server URIs.

Every other client does not handle shorter-than-requested range
responses the way WebKit does.  Their expectation is that the 206
response includes the full requested range and they do not follow up
with additional requests.  And when they request "Range: 0-" or send
no "Range" at all they essentially treat a 206 response as a 200
response (a whole response).

== Using 4xx to get a range the server will accept ==

> count on any generic client coming to the conclusion that other ranges may
> work on the same resource.

I mentioned that a 416 server response to requests with
larger-than-permitted ranges could solve this, but  there is no way
for the server to tell the client what a permissible upper bound for
the requested range should be:

  GET /file.mp4 HTTP/1.1
  Range: bytes=0-
  ...
  HTTP/1.1 416 range was bad
  Accept-Ranges: bytes
  Content-Range: bytes */300000

would need to be more like,

  GET /file.mp4 HTTP/1.1
  Range: bytes=0-
  ...
  HTTP/1.1 416 range was too large
  Content-Range: bytes */300000
  Accept-Ranges: bytes; max-range-length=100000;
* or to potentially prevent breaking UAs that only understand "bytes"
without a semicolon
  Accept-Range-Max-Length: bytes=100000;
  ...
  GET /file.mp4 HTTP/1.1
  Range: bytes=0-99999
  ...
  HTTP/1.1 206 partial
  Content-Range: bytes 0-99999/300000
  Accept-Ranges: bytes; max-range-length=100000;

adding a max-range-length to Accept-Ranges or a an
Accept-Range-Max-Length header.

Perhaps a 416 could send "Content-Range: bytes 0-99999/300000" (rather
than */300000) but that doesn't seem obvious.  "Content-Range: bytes
0-99999/*" or "Content-Range: bytes */99999" might also be used to
hint a client to the acceptable range but either of these are
deceptive because the server knows the true length of the file and the
client may confuse these responses with later responses for earlier
ranges that resulted in a smaller reported length.

> A response code could be defined. Write a spec :-)

Perhaps a new 4xx could be made for "Range too large" but any 4xx
response would add to the HTTP transaction count as a client would
first receive a 307+Accept-Range (rerequest with ranges please - or
4xx (range required)), then a 416 or 4xx (range too large), and then
finally a 206.

== New 2xx for shorter partial responses or defining short 206s ==

A new 2xx response could be devised for "short partial responses" but
I don't think a new 2xx response is necessary since some existing UAs
are ok with a 206 (namely WebKit).  Likewise adding a max-range-length
response would be unnecessary if 206s were specifically allowed to be
shorter than the requested range.  The spec doesn't say that servers
can't respond with less data than requested.  And the spec already
defines the necessary headers for the server to respond with in any
case (Range: bytes x-y/z).

== Cheating with a disconnect ==

An alternative to this would be for the server to take advantage of
"Client Behavior if Server Prematurely Closes Connection" and
disconnect after sending the the willing portion of the response with
an "Accept-Ranges".  The response would be less than content-length
that the server reported.  But there is no defined behavior for how a
client should respond to a successful status with an interrupted
responses so there is no guarantee that the client would follow-up
with a Ranged request continuing where the server left off.

== Winning Punch ==

> Well, this is something HTTP range requests obviously haven't been designed
> for. I'm not convinced that they should be used for this, nor that making
> these minor changes would be sufficient to deal with your use case.

Just about any one of these small changes on their own would extend
the application range of HTTP to include content-controlled seek-able
streaming video without resorting to dissecting a video file (using
additional client scripting to navigate to the appropriate video
segments) or using a proprietary streaming solution and player.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 25 Jun 2010 13:21:52 -0700

> added flow interrupting headings (joy).

This kind of application is not in the HTTP spec because it is a
poor design -- it sacrifices almost all caching and resource
identification benefits for the sake of server convenience.
If the server doesn't want to deliver huge representations
then it should be delivering a content menu containing links to
multiple subresources (e.g., Apple's use of a playlist format).

> behavior or by defining a new '4xx Range Required' status code.

"browsers" is not defined by one implementation.  If such a thing had
been tried in 1998, it would not have worked on any browsers.  In order
for us to standardize it now, it would have to be working on all browsers.
I think we would need to use specific user-agent matching to make it
work, which would also limit caching.

....Roy

---

From: Marques Johansson <[email-masked]>
Date: Fri, 25 Jun 2010 20:25:40 -0400

> multiple subresources (e.g., Apple's use of a playlist format).

In this case I am dealing with content that would insist upon not
being cached by proxies or UAs.

In the case of cacheable content, the spec already outlines how caches
can reassemble 206 responses to offer larger ranges so long as each
partial responses other headers (cache/expires) matched.

The playlist format introduces a good deal of extra work on the server
because native video files (which are to be supported by html5
browsers) will have to be disassembled into smaller files -
introducing artificial limits on the minimum amount of content a UA
can request and sacrificing cache because each piece of the file has a
unique URL.  To date, this playlist format is supported on Ipads and
Safari.

As for poor design - setting a requirement that a client specify a
range and deal with smaller available data than requested has been a
fixture in computing (filesystem and network sockets).

> work, which would also limit caching.

Accepting a 206 to an unranged request is not something I found in
Chrome alone. Chrome, Firefox, Opera, Links, wget, etc. all accept a
206 status even though they didn't send a Range in their request-
despite the spec suggesting that they should behave otherwise.  If a
server responds with a 206 and the whole content is sent todays
browsers would happily render it.  The point at which browsers deviate
is in how they react to less data being sent than what was requested.
Had these browsers sent a "Range:" in their initial request, only the
WebKit based browsers (arguably the most popular engine in modern
browsers - especially when you include mobile devices) would move to
the next block of ranged content.  (I'm not including IE when I say
"browsers" because I haven't had a chance to test it).

If a 1998 browser attempted to fetch this controlled-range content the
server would give them a '4xx Range Too Large' and that would be the
end of the matter.