Subject: Range requests and content encoding

From: Martin Thomson <[email-masked]>
Date: Tue, 22 Dec 2015 13:28:32 +1100

RFC 7233 does not mention content encoding at all.  Same for transfer
encoding.  I assume that is because this is completely unspecified and
therefore completely unreliable, however, for my sanity...

My reading is that a 206 response includes ranges of the encoded
message, and that the content-encoding applies to the complete message
body prior to being split into ranges.  Thus, if I had a "x2" content
encoding that turned "Hello World!" into "HHeelllloo  WWoorrlldd!!",
asking for bytes 3-5 would get you "eel" and not "llo".

The text in Section 4.1 suggests that you would not include a
Content-Encoding header field if the client used If-Range on the
expectation that they already know.  That seems pretty dangerous, but
it's consistent with the idea that you are repairing a larger message.

On the other hand, I have to assume that a Transfer-Encoding applies
*after* the range request.

p.s., I've opened https://github.com/httpwg/http11bis/issues/11 for this.

---

From: Mike Bishop <[email-masked]>
Date: Tue, 22 Dec 2015 04:37:10 +0000

That's murky, because C-E is often used as if it were a transfer encoding.  Do most people request a range and "Accept-Encoding: gzip" expecting to get a partial gzip that may back-reference content that isn't in the transferred portion?  If you're actually patching up a truncated download, that's fine -- if you actually only need that part of the file, though....

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Monday, December 21, 2015 6:29 PM
To: HTTP Working Group <[email-masked]>
Subject: Range requests and content encoding

RFC 7233 does not mention content encoding at all.  Same for transfer encoding.  I assume that is because this is completely unspecified and therefore completely unreliable, however, for my sanity...

My reading is that a 206 response includes ranges of the encoded message, and that the content-encoding applies to the complete message body prior to being split into ranges.  Thus, if I had a "x2" content encoding that turned "Hello World!" into "HHeelllloo  WWoorrlldd!!", asking for bytes 3-5 would get you "eel" and not "llo".

The text in Section 4.1 suggests that you would not include a Content-Encoding header field if the client used If-Range on the expectation that they already know.  That seems pretty dangerous, but it's consistent with the idea that you are repairing a larger message.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 22 Dec 2015 15:52:58 +1100

> therefore completely unreliable, however, for my sanity...

206's semantics are defined in <http://httpwg.github.io/specs/rfc7233.html#status.206>:

"""
The 206 (Partial Content) status code indicates that the server is successfully fulfilling a range request for the target resource by transferring one or more parts of the selected representation that correspond to the satisfiable ranges found in the request's Range header field (Section 3.1).
"""

Note "selected representation", which is defined in <http://httpwg.github.io/specs/rfc7231.html#representations> as:

"""
An origin server might be provided with, or be capable of generating, multiple representations that are each intended to reflect the current state of a target resource. In such cases, some algorithm is used by the origin server to select one of those representations as most applicable to a given request, usually based on content negotiation. This "selected representation" is used to provide the data and metadata for evaluating conditional requests [RFC7232] and constructing the payload for 200 (OK) and 304 (Not Modified) responses to GET (Section 4.3.1).
"""

So, Content-Range operates upon the selected representation -- which would include any Content-Encodings.

Transfer-Encoding is defined to operate on the message body in <http://httpwg.github.io/specs/rfc7230.html#message.body>, which says it is removed to get to the message payload. 

> asking for bytes 3-5 would get you "eel" and not "llo".

Correct.

> it's consistent with the idea that you are repairing a larger message.

... as per <http://httpwg.github.io/specs/rfc7233.html#combining.byte.ranges>.

--
Mark Nottingham   https://www.mnot.net/

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 22 Dec 2015 14:56:18 +1000

> p.s., I've opened https://github.com/httpwg/http11bis/issues/11 for this.
​That's been my understanding. C-E can be used to send offline-encoded
files (like .gz archives), so a range request against that representation
of that resource should target a slice of gzip-encoded data. (IOW:
bytes=0-2 of all C-E:gzip resources on the web should be identical.)

And as you say, T-E applies *after* the slicing. That fact (and the absence
of T-E in H2) was what inspired Keith Morgan to start that big discussion a
while back about gzipping sliced-up log files, and lead to
draft-kerwin-http2-encoded-data

Regarding Section 4.1 and If-Range/Range/206: as I understand it,
conditional request conditions are tested *after* the representation is
selected (e.g. RFC 7232, Section 3.1 "...conditional on the selected
representation's modification date..."), which I assume means A-E/C-E has
already been resolved before looking at the If-Range/Range request headers.
If-Range uses the strong comparison function, which should be enough to
guarantee the content encoding*. It might be nice if there was some text in
RFC 7233 that spelled that out a bit better (or even a more explicit
pointer), but I don't know how to word it.

* RFC 7232, Section 3.1: "For example, if
   the origin server sends the same validator for a representation with
   a gzip content coding applied as it does for a representation with no
   content coding, then that validator is weak."

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 22 Dec 2015 08:53:24 +0100

> So, Content-Range operates upon the selected representation -- which would include any Content-Encodings.

Right.

> ... as per <http://httpwg.github.io/specs/rfc7233.html#combining.byte.ranges>.

But it seems to me that that advice causes confusion, and including a 
correct C-E would actually make things clearer.

>> p.s., I've opened https://github.com/httpwg/http11bis/issues/11 for this.

Best regards, Julian