Subject: Range Requests vs Content Codings

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jun 2014 12:26:29 +0200

Trying to summarize the problem, and trying a solution:

Consider a text/plain resource http://example.org/test of 1000000 octets 
length (representation A), supporting content coding "gzip", yielding 
100000 octets (representation B).

Upon GET, clients can select which one to use using "Accept-Encoding". 
For instance

   GET /test HTTP/1.1
   Host: example.org
   Accept-Encoding: gzip

is likely to return the representation B, while

   GET /test HTTP/1.1
   Host: example.org
   Accept-Encoding: identity

will return representation A.

A specified range will always apply to the representation. Thus, a 
client can't easily ask for a specific range of representation A *and* 
have the server apply Content-Coding gzip.

(Compression could also be achieved by using Transfer Codings, but these 
are not implemented in practice)

One way to combine Content Codings and range requests would be to create 
a new range unit, "bbcc" (bytes-before-content-coding). In which case 
the the requested range would be applied to the non-content-coded 
representation, and the content-coding would be applied to the byte range.

Such as:

   GET /test HTTP/1.1
   Host: example.org
   Accept-Encoding: gzip
   Range: bbcc=900000-

This would retrieve the octets starting at position 900000, and apply 
content-coding gzip to the resulting octet sequence.

Note that to combine range responses using these byte range units, a 
recipient needs to understand the range unit (simple concatenation isn't 
going to work).

This also requires that both user agent and origin server understand the 
new range unit, but that appears to be easier to deploy than T-E (which 
requires all intermediaries to play along).

Thoughts?

Best regards, Julian

---

From: Roland Zink <[email-masked]>
Date: Tue, 17 Jun 2014 13:42:52 +0200

So the range request is really  for representation A and return some 
representation C, e.g. gzip content which is most likely different from 
representation B. My question is now what Etags (if there are etags) are 
send in the requests (when conditional) and responses. When the client 
sends the range request then will it send the etag for B?

A cache then can build A with the ETAG of B using several bbcc range 
requests from a client. If it gets normal byte range requests for B then 
it can't use it (cache has only A and C) and need to forward the request 
to the origin server. If the cache has B and gets a bbcc range request 
then it can respond from the cache by decompressing to A, select the 
requested range and return the compressed content. If the cache has B 
and gets a normal byte range request for A then it could do the same but 
will not match as it doesn't know about the Etag for A. Seems like more 
effort for caches, but maybe still an acceptable price.

When the client should the send the range request with the etag for A 
then how does the client get the etag for A?

Regards,
Roland

On 17.06.2014 12:26, Julian Reschke wrote:
> Consider a text/plain resource http://example.org/test of 1000000 
> octets length (representation A), supporting content coding "gzip", 
> yielding 100000 octets (representation B).
>
> (Compression could also be achieved by using Transfer Codings, but 
> these are not implemented in practice)
>
> One way to combine Content Codings and range requests would be to 
> create a new range unit, "bbcc" (bytes-before-content-coding). In 
> which case the the requested range would be applied to the 
> non-content-coded representation, and the content-coding would be 
> applied to the byte range.
>
> Such as:
>
>   GET /test HTTP/1.1
>   Host: example.org
>   Accept-Encoding: gzip
>   Range: bbcc=900000-
>
> recipient needs to understand the range unit (simple concatenation 
> isn't going to work).
>
> This also requires that both user agent and origin server understand 
> the new range unit, but that appears to be easier to deploy than T-E 
> (which requires all intermediaries to play along).
>
> Thoughts?
>
> Best regards, Julian
>
>
>
>
>

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 17 Jun 2014 23:15:03 +1000

> content-coding gzip to the resulting octet sequence.

So something like this?

| Content-Type: multipart;separator=foo
|
| --foo
| Content-Type: text/plain
| Content-Range: bbcc=900000-1000000
| Content-Encoding: gzip
| Content-Length: 1234
|
| <standalone gzip stream>
| --foo--

(Paraphrased, because I can't look up the exact terminology on my phone.)

> requires all intermediaries to play along).

I think this still requires intermediaries to play along. What does a
caching proxy do when this request-response passes through it?
Especially if it doesn't know this range unit? Does the response need
Vary:Range ? Or are they always Cache-Control:no-cache ?

To my mind, this also opens up the idea of a 'bacc' range unit (bytes
after content-coding), as an explicit signal that the client only
wants the range if it's from the content-coded representation. AFAIU
currently it's a bit ambiguous what to do when a request has both A-E
and Range headers. Of course, 'bacc' requires there to be exactly one
coding in the Accept-Encoding header, but it could be useful for
resuming a content-coded download. The same caching issues as with
'bbcc' still apply, though.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jun 2014 18:56:31 +0200

> (Paraphrased, because I can't look up the exact terminology on my phone.)

No need to for multipart unless there are multiple ranges...

> caching proxy do when this request-response passes through it?

It passes it along.

> Vary:Range ? Or are they always Cache-Control:no-cache ?

No, and no.

If the intermediary doesn't know about the range unit it'll have to 
leave it alone.

> and Range headers. Of course, 'bacc' requires there to be exactly one

The ambiguity is caused by the client sending a request that selects 
multiple representations. That's easy to avoid...

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jun 2014 19:08:38 +0200

> sends the range request then will it send the etag for B?

It's asking for a range of B, to be generated from A. So I think it 
ought to use ETags applying to B. But of course that's something to 
properly write down.

> effort for caches, but maybe still an acceptable price.

Optimal behavior of intermediaries is an interesting topic, but I don't 
see intermediaries implementing this anyway. The idea was to enable 
clients to do something they can't do now.

> ...

HEAD?

Best regards, Julian

---

From: [email-masked]
Date: Tue, 17 Jun 2014 13:19:49 -0400

I just want to weigh in that I think it's useful to be able to do range requests for bytes in both representation A and representation B. So I may want to ask for bytes X-Y of a compressed file where X and Y are offsets into the compressed representation or offsets into the original representation.

Thanks,

Peter

On Jun 17, 2014, at 1:08 PM, Julian Reschke <[email-masked]> wrote:

> It's asking for a range of B, to be generated from A. So I think it ought to use ETags applying to B. But of course that's something to properly write down.
> 
> Optimal behavior of intermediaries is an interesting topic, but I don't see intermediaries implementing this anyway. The idea was to enable clients to do something they can't do now.
>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jun 2014 19:58:14 +0200

> I just want to weigh in that I think it's useful to be able to do range requests for bytes in both representation A and representation B. So I may want to ask for bytes X-Y of a compressed file whe…

You can to that already with classical byte-range requests, no?

---

From: Peter L <[email-masked]>
Date: Tue, 17 Jun 2014 14:53:35 -0400

Yes okay thanks.

> On Jun 17, 2014, at 1:58 PM, Julian Reschke <[email-masked]> wrote:
>

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 18 Jun 2014 08:24:18 +1000

> multiple representations. That's easy to avoid...
Oh, yeah, because "identity;q=0" is allowed. Apologies for posting late at
night.

---

From: Zhong Yu <[email-masked]>
Date: Tue, 17 Jun 2014 17:33:22 -0500

This will subvert the current ordering of Range and Content-Encoding,
adding an exception for a particular range unit. I think that is a
high cost.

And the use case seems quite limited. We've heard that someone want
this feature for log tailing. But I don't understand how it could be
easier to change the HTTP spec and deployments, than to change the
server application (by adding the range to the request URI).

Zhong Yu

On Tue, Jun 17, 2014 at 5:26 AM, Julian Reschke <[email-masked]> wrote:
> length (representation A), supporting content coding "gzip", yielding 100000
> octets (representation B).
>
> Upon GET, clients can select which one to use using "Accept-Encoding". For
> instance
>
>   GET /test HTTP/1.1
>   Host: example.org
>   Accept-Encoding: gzip
>
> A specified range will always apply to the representation. Thus, a client
> can't easily ask for a specific range of representation A *and* have the
> server apply Content-Coding gzip.
>
> (Compression could also be achieved by using Transfer Codings, but these are
> not implemented in practice)
>
> One way to combine Content Codings and range requests would be to create a
> new range unit, "bbcc" (bytes-before-content-coding). In which case the the
> requested range would be applied to the non-content-coded representation,
> and the content-coding would be applied to the byte range.
>
> Such as:
>
>   GET /test HTTP/1.1
>   Host: example.org
>   Accept-Encoding: gzip
>   Range: bbcc=900000-
>
> This also requires that both user agent and origin server understand the new
> range unit, but that appears to be easier to deploy than T-E (which requires
> all intermediaries to play along).
>
> Thoughts?
>
> Best regards, Julian
>
>
>
>
>

---

From: Zhong Yu <[email-masked]>
Date: Tue, 17 Jun 2014 17:53:57 -0500

> Oh, yeah, because "identity;q=0" is allowed. Apologies for posting late at

Do servers ever honor that? For example, send this request

  GET /logos/doodles/2014/world-cup-2014-14-russia-vs-korea-5756435967246336.2-hp.gif
HTTP/1.1
  HOST: www.google.com
  Accept-Encoding: gzip, identity;q=0

and the server happily responds with 200, without Content-Encoding.

Zhong Yu

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 18 Jun 2014 10:32:58 +1000

On 17 Jun 2014, at 8:26 pm, Julian Reschke <[email-masked]> wrote:

> Consider a text/plain resource http://example.org/test of 1000000 octets length (representation A), supporting content coding "gzip", yielding 100000 octets (representation B).

Nit: I think you mean a resource with two text/plain representations; one without content-coding that's 1,000,000 octets (A), and one with 'gzip' content-coding that's 100,000 octets (B).

> 
> Upon GET, clients can select which one to use using "Accept-Encoding". For instance

Nit: the server always performs selection.

> will return representation A.

Yes.

> A specified range will always apply to the representation. Thus, a client can't easily ask for a specific range of representation A *and* have the server apply Content-Coding gzip.

Well, that's because A isn't gzipped, but OK...

> (Compression could also be achieved by using Transfer Codings, but these are not implemented in practice)
> 
> One way to combine Content Codings and range requests would be to create a new range unit, "bbcc" (bytes-before-content-coding). In which case the the requested range would be applied to the non-content-coded representation, and the content-coding would be applied to the byte range.
> 
> Such as:
> 
>  GET /test HTTP/1.1
>  Host: example.org
>  Accept-Encoding: gzip
>  Range: bbcc=900000-
> 
> This would retrieve the octets starting at position 900000, and apply content-coding gzip to the resulting octet sequence.
> 
> Note that to combine range responses using these byte range units, a recipient needs to understand the range unit (simple concatenation isn't going to work).
> 
> This also requires that both user agent and origin server understand the new range unit, but that appears to be easier to deploy than T-E (which requires all intermediaries to play along).
> 
> Thoughts?

To me, whenever we have someone propose a new range-unit, one of the deciding factors is whether it'll be useful and implemented by intermediaries -- because one of the main benefits of adding new HTTP mechanisms like this is to allow generic software to take advantage of it.

For this scheme to work in an intermediary, it looks like it'll have to cache the uncompressed response to use this range unit, and decompress responses as they come back. That doesn't seem terribly attractive, from their standpoint.

Also, this won't work with intermediaries until they actually support it.

The alternative is to do what people do now -- to use an application-specific mechanism (e.g., a URL query parameter) to fetch parts of the response.  That seems like it would satisfy the use case that was talked about earlier (from Bjoern):

> Imagine you have a remote resource that is regularily appended to, like a log file or a mailing list archive mbox file.
> You synchronise with it by making regular Range requests to it to retrieve the content that has since been appended, if any.

The natural questions seem to be:

- What benefits does standardising a range-unit bring this use case (over just using application-specific semantics)?
- Are there other use cases that are materially different?

Cheers,

P.S. Happy to give agenda time in Toronto to this if you think it's ready to discuss...

--
Mark Nottingham   https://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 18 Jun 2014 10:43:42 +1000

... also, I kind of wonder whether this could be hammered into a solution as well:
  http://tools.ietf.org/html/draft-nottingham-http-patch-status-00
(but it would need some work to get there, I think...)

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Wed, 18 Jun 2014 07:29:17 +0200

Yes.

Yes.

> Also, this won't work with intermediaries until they actually support it.

Um, no. Intermediaries can ignore it. It just makes all responses using 
the new range unit uncacheable.

Intermediaries that fail for unknown range units are broken, no?

> - What benefits does standardising a range-unit bring this use case (over just using application-specific semantics)?

That it would work with standard software such as httpd, static files, 
and a new module?

> P.S. Happy to give agenda time in Toronto to this if you think it's ready to discuss...

Maybe. I was just writing down the idea in order to find out whether 
this is actually one possible way to address this.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 18 Jun 2014 15:52:10 +1000

On 18 Jun 2014, at 3:29 pm, Julian Reschke <[email-masked]> wrote:

> Um, no. Intermediaries can ignore it. It just makes all responses using the new range unit uncacheable.
> 

Yes, of course they wouldn't break, but they also wouldn't be able to cache partial responses using the new range-unit, nor would they be able to 
service requests that use it. So there's a loss of functionality which may outweigh the usefulness of the new function.

> That it would work with standard software such as httpd, static files, and a new module?

Yes. 

I'm remembering that in the past, we've had people suggest standardising other range-units (e.g., "entries" of a blog, etc.) and there's been a fair amount of pushback, some of it for the reasons that I describe (that intermediaries won't understand / implement it).

I think what might make this different is that a) it's "closer" to the wire; it's still a byte-counting mechanism, just at a different "layer", and b) gzip is already part of HTTP.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Wed, 18 Jun 2014 08:00:29 +0200

> service requests that use it. So there's a loss of functionality which may outweigh the usefulness of the new function.

That is true. It would effectively be end-to-end with no benefits of 
intermediary caching.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 18 Jun 2014 09:11:13 +0000

>Intermediaries that fail for unknown range units are broken, no?

No, they may simply know something about the origin you do not, and
decide that delivering the full object is the way to satisfy the request.

>That it would work with standard software such as httpd, static files, 

That will only happen if people need it enough to ask for it to be
implemented.  Very few people tail gzip'ed logfiles.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 18 Jun 2014 12:10:41 +0200

> decide that delivering the full object is the way to satisfy the request.

That's not "failing".

> implemented.  Very few people tail gzip'ed logfiles.

I agree that gzipped logfiles aren't a very compelling use case.

Best regards, Julian

---

From: Roland Zink <[email-masked]>
Date: Wed, 18 Jun 2014 12:10:31 +0200

On 18.06.2014 02:32, Mark Nottingham wrote:
> To me, whenever we have someone propose a new range-unit, one of the 
> deciding factors is whether it'll be useful and implemented by 
> intermediaries -- because one of the main benefits of adding new HTTP 
> mechanisms like this is to allow generic software to take advantage of 
> it. For this scheme to work in an intermediary, it looks like it'll 
> have to cache the uncompressed response to use this range unit, and 
> decompress responses as they come back. That doesn't seem terribly 
> attractive, from their standpoint. Also, this won't work with 
> intermediaries until they actually support it.
Some intermediaries already access the uncompressed content like virus 
scanners. For such proxies it may be attractive to do this. It would 
even be more attractive when the intermediary would be enabled to do 
content-encoding gzip (or other) for non compressed content.

> The alternative is to do what people do now -- to use an 
> application-specific mechanism (e.g., a URL query parameter) to fetch 
> parts of the response. That seems like it would satisfy the use case 
> that was talked about earlier (from Bjoern):
An application specific semantic doesn't allow the caching of the 
original file in intermediaries.
It does provide a way to do a seek using range request and use 
compression at the same time.
The application can get simpler, e.g. the web server can do the work.
Applications can use this feature without the need to control the server 
side and write code for the server.
> - Are there other use cases that are materially different?
A slightly different use case is to access the end of the log file 
without reading the whole log file. When the log file is content-encoded 
gzip then a client would need to read the whole log file (possible in 
chunks) just to be able to decompress the last chunk. With the bbcc 
scheme and the addition that only the range response is content-encoded 
this is not necessary.

---

From: Zhong Yu <[email-masked]>
Date: Wed, 18 Jun 2014 11:35:14 -0500

On Wed, Jun 18, 2014 at 5:10 AM, Roland Zink <[email-masked]> wrote:
> A slightly different use case is to access the end of the log file without
> reading the whole log file. When the log file is content-encoded gzip then a
> client would need to read the whole log file (possible in chunks) just to be
> able to decompress the last chunk. With the bbcc scheme and the addition
> that only the range response is content-encoded this is not necessary.

I think this is the use case we've been talking about. Any other examples?

There is some logistic issues with this use case that makes it
unrealistic. There is just one log file (accessible from one URI) that
grows forever. Assuming the growth is very fast (otherwise we won't
need gzip+range), the file will quickly reach an impractical size.

If there is log rotation, then we are talking about different URIs at
different times, which is an app-specific protocol.

Another difficulty is at the assumption that the file is only appended
to, and previously written bytes are never altered. There is no
mechanism in HTTP to express that property. ETag obviously must evolve
when the file grows, so it cannot be relied upon. How could the client
be certain about the assumption then? There must be some size-channel
secrete shared by the server and the client.

The use case, which assumes byte-level append-only growth, is a tiny
fraction of the slightly more common case where growth is append-only,
but at various semantic units (e.g. blog article). We live by
app-specific protocols to handle these cases. What is so special about
byte unit that it must be addressed by http (in a very ugly way)?

The motivation, I think, is from subconscious association - byte range
is widely used, therefore bbcc is probably also justified.

However, byte range solves a real, important use case - that in our
current world, downloading a resource may take very long time, during
which time various events can and do interrupt the download. Resumed
download via byte range probably helps millions of users every day.

But we cannot say that about bbcc, by any measure, therefore the merit
of byte range cannot be automatically extended to bbcc. The proponents
of bbcc need to uncover a lot more benefit of the feature to justify
its cost.

Zhong Yu
http://bayou.io

---

From: <[email-masked]>
Date: Sat, 21 Jun 2014 16:34:59 +0000

On 18 June 2014 18:35, [email-masked] wrote:
> to, and previously written bytes are never altered. ...
> ...
> ... The proponents

I basically agree with everything you said, despite the fact that we actually do have this use case (but nobody cares unless you are Google or their little brother [1] :)

We have rapidly growing append-only files.  Currently we use gzip Transfer-Encoding to compress an identity range request.  But since T-E went away (again) in h2, it's no longer possible.

I realize it's a controversial issue, but if I understand correctly, h2 is supposed to replace *everything* you could do in 1.1.

As far as I can tell, there are four solutions for maintaining this capability...

1. Julian's bbcc
2. Matthew's DATA frame compression - possibly as an extension now that it was removed from the main spec.
3. My & Christoph's Message-Encoding HTTP extension (an end-to-end version of T-E that is applied after C-E, but before T-E)
4. Keep T-E in h2 (I dare say nobody wants this solution)

[1] http://www.nationmultimedia.com/breakingnews/Google-to-pay-Firefox-almost-1-billion-dollars-in--30172517.html
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: <[email-masked]>
Date: Sat, 21 Jun 2014 17:56:14 +0000

Any Chrome/Chromium devs out there???

As we pointed out in [1], Chrome goes through a bit of a bizarre "song and dance" when it resumes interrupted transfers (not just for "downloads" of large entities by the way, also for static content such as css files and scripts - which also see a performance benefit from resumed downloading after an interruption, e.g. on a mobile network).

When resuming an interrupted transfer, Chrome actually sends two range requests. The first request Chrome sends asks for a single byte directly after the already downloaded portion and, if successful, sends a second request for the rest of the content by specifying the exact range. (See [1] for an example.)

Since Google is so finicky about saving RTTs, there must be a good reason for doing the 2-range-request "song and dance". There must be some Chrome/Chromium devs out there.  I would love to hear why Chrome does that.  I can only guess why, but my theory is that it's related to dynamic C-E and range requests.

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0112.html

---

From: Zhong Yu <[email-masked]>
Date: Sat, 21 Jun 2014 13:02:43 -0500

> We have rapidly growing append-only files.  Currently we use gzip Transfer-Encoding to compress an identity range request.  But since T-E went away (again) in h2, it's no longer possible.

damn it... I just implemented the feature on my server. another fool's errand.

http://bayou.io/release/0.9/javadoc/bayou/http/HttpServerConf.html#autoGzip-boolean-

---

From: Zhong Yu <[email-masked]>
Date: Sat, 21 Jun 2014 13:45:28 -0500

> damn it... I just implemented the feature on my server. another fool's errand.

Sorry if that's misleading. I meant that I implemented it on a 1.1 server.

> I realize it's a controversial issue, but if I understand correctly, h2 is supposed to replace *everything* you could do in 1.1.

I'm not keeping up with h2, but I assume h2 will have its own
mechanism to compress bytes on transfer? And h2 intermediaries can
interact with h1 clients/servers using TE/T-E headers? Then it's not a
problem.

Zhong Yu
http://bayou.io

---

From: Roland Zink <[email-masked]>
Date: Mon, 23 Jun 2014 12:06:29 +0200

On 18.06.2014 18:35, Zhong Yu wrote:
> its cost. Zhong Yu http://bayou.io 

There is another use case where range requests are used every day by a 
lot of users and this is playing (mp4) videos. Players usually offer to 
seek forward and backward within a video. This can't be done with 
content-encoding gzip, so this is one reason why gzip is not used in 
this case. You can argue that videos are already compressed and don't 
need C-E gzip, however for something like online encoding (and 
transcoding) this is done by guessing a bitrate adding some slack to it 
and fill the unused bytes with some constant values and compress most of 
the additional bytes away. Of cause this can be done on the application 
level as well, so you get something more advanced like HLS or DASH.

Even for the download this means the resumptions can't switch from 
uncompressed to compressed and vice versa. Caches may also need to store 
both versions.

Roland