Subject: GZip Content-Encoding / Reopen #404

From: Matthew Kerwin <[email-masked]>
Date: Mon, 14 Apr 2014 12:23:30 +1000

Section 9.3 in the current draft (-11 and the editor's copy) says that
clients MUST support gzip compression for HTTP response bodies.
"Intermediaries that perform translation from HTTP/2 to HTTP/1.1 MUST
decompress payloads unless the request includes an Accept-Encoding value
that includes "gzip"."

This goes some way to resolving issue #404 -- mnot's original report: "the
implication on a HTTP/1 to HTTP/2 intermediary should be spelled out" --
but there are still a lot of implications not listed in that requirement.
According to my understanding of HTTP, the appropriate added requirements
are:

For requests in the 1.1->2 direction without Accept-Encoding:gzip :
* strip all conditional headers (If-Modified-Since, If-Unmodified-Since,
If-Match, If-None-Match, If-Range) [i]
* strip all Range headers [i]

For all responses in the 2->1.1 direction:
* ought to strip Accept-Ranges headers

For such a response with Content-Encoding:gzip that wasn't
Accept-Encoding:gzip :
* gunzip the entity, appropriately update Content-Encoding, Content-Length
[ii], and add a Warning:214
* maybe change :status: 200 to 203
* strip or update Content-MD5 [iii]
* strip Content-Location, ETag, Last-Modified headers from the response [i]
* if the request included Cache-Control:no-transform ???

There may be others. If so, my implementation (if I were writing one) would
be lacking.

There's still a big hole in "Cache-Control:no-transform". We've essentially
added a requirement that gateways break conformance, without giving any
guidance on how to do it (let alone reasonably justifying it in the first
place). We haven't even told them the appropriate way to fail.

In the absence of an opt-out mechanism for gzip C-E [iv], I think we should
remove the support requirement. In fact I think we should remove section
9.3 altogether, but I'm not opposed to a strongly worded suggestion that
everyone support gzip.

[i] or, in the case of a full-on transforming proxy, mint new Last-Modified
(possibly even ETag) header, and be responsible for mapping the
uncompressed entity on the 1.1 side to the compressed entity on the 2 side

[ii] since HTTP 2 is already fragmented by frames, a gateway is probably
going to always respond with chunked T-E, so Content-Length should probably
be stripped (if present) anyway.

[iii] thus explicitly acting as a non-transparent proxy according to RFC
2616, Section 13.5.2 (Non-modifiable Headers). I can't find the equivalent
text in the HTTPbis drafts, but I assume there's something in there.

[iv] an alternative could be to use Cache-Control:no-transform as such a
flag; i.e. a server MUST NOT apply gzip C-E if the request did not include
Accept-Encoding:gzip and did include Cache-Control:no-transform. That seems
a bit flaky to me, though, and easy to miss/get wrong.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Roland Zink <[email-masked]>
Date: Mon, 14 Apr 2014 10:49:04 +0200

+1

Even then it might be that there are some problems, e.g. my experience 
in the past with an iPhone playing a mp4 file was that it didn't 
accepted a full response on a range request and the video was not 
played. See for example 
http://stackoverflow.com/questions/10275730/php-video-play-on-ipad.

On 14.04.2014 04:23, Matthew Kerwin wrote:
> decompress payloads unless the request includes an Accept-Encoding 
> value that includes "gzip"."
>
> This goes some way to resolving issue #404 -- mnot's original report: 
> "the implication on a HTTP/1 to HTTP/2 intermediary should be spelled 
> out" -- but there are still a lot of implications not listed in that 
> requirement. According to my understanding of HTTP, the appropriate 
> added requirements are:
>
> * strip all conditional headers (If-Modified-Since, 
> If-Unmodified-Since, If-Match, If-None-Match, If-Range) [i]
> * gunzip the entity, appropriately update Content-Encoding, 
> Content-Length [ii], and add a Warning:214
> There may be others. If so, my implementation (if I were writing one) 
> would be lacking.
>
> There's still a big hole in "Cache-Control:no-transform". We've 
> essentially added a requirement that gateways break conformance, 
> without giving any guidance on how to do it (let alone reasonably 
> justifying it in the first place). We haven't even told them the 
> appropriate way to fail.
>
> In the absence of an opt-out mechanism for gzip C-E [iv], I think we 
> should remove the support requirement. In fact I think we should 
> remove section 9.3 altogether, but I'm not opposed to a strongly 
> worded suggestion that everyone support gzip.
>
>
> [i] or, in the case of a full-on transforming proxy, mint new 
> Last-Modified (possibly even ETag) header, and be responsible for 
> mapping the uncompressed entity on the 1.1 side to the compressed 
> entity on the 2 side
>
> [ii] since HTTP 2 is already fragmented by frames, a gateway is 
> probably going to always respond with chunked T-E, so Content-Length 
> should probably be stripped (if present) anyway.
>
> [iii] thus explicitly acting as a non-transparent proxy according to 
> RFC 2616, Section 13.5.2 (Non-modifiable Headers). I can't find the 
> equivalent text in the HTTPbis drafts, but I assume there's something 
> in there.
>
> [iv] an alternative could be to use Cache-Control:no-transform as such 
> a flag; i.e. a server MUST NOT apply gzip C-E if the request did not 
> include Accept-Encoding:gzip and did include 
> Cache-Control:no-transform. That seems a bit flaky to me, though, and 
> easy to miss/get wrong.
>
> -- 
>   Matthew Kerwin

---

From: <[email-masked]>
Date: Mon, 14 Apr 2014 15:55:44 +0000

+1

Two more reasons against implicit C-E gzip (in addition to Matthew's lengthy list)...

#1 In HTTP/1.1 it was possible for a client to unambiguously request a resource with a single acceptable content encoding...

e.g. "Accept-Encoding: sdch, identity;q=0"

Since opting out of gzip is not allowed in HTTP/2, this is no longer possible. The best a client can do in HTTP/2 is

e.g. "Accept-Encoding: sdch, gzip;q=0.001, identity;q=0"

- which is ugly and ambiguous.

#2 The HTTP/2 charter says "Retain the semantics of HTTP/1.1" [1]. Changing the definition of the implicit value of Accept-Encoding from "Accept-Encoding: identity" to "Accept-Encoding: gzip, identity" IMO is a change in semantics.

I agree with Patrick McManus [2] and Google researchers [3] that there is a need to require compression because of crazy intermediaries (e.g. antivirus libraries) [3], but implicit C-E gzip is the wrong way to do it.  Implicit gzip on the transfer layer would have been the right way to do it, but it seems there are too many security issues surrounding compression on the transfer layer for that to be a possibility.

[1] http://datatracker.ietf.org/wg/httpbis/charter/
[2] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1283.html
[3] https://developers.google.com/speed/articles/use-compression

On Monday,14 April 2014 10:49, Roland Zinks wrote:
>
> +1
>
> Even then it might be that there are some problems, e.g. my experience in the past with an iPhone playing a mp4 file was that it didn't accepted a full response on a range request and the video was not played. See for example http://stackoverflow.com/questions/10275730/php-video-play-on-ipad.
>
>> Section 9.3 in the current draft (-11 and the editor's copy) says that clients MUST support gzip compression for HTTP response bodies. "Intermediaries that perform translation from HTTP/2 to HTTP/1.1 MUST decompress payloads unless the request includes an Accept-Encoding value that includes "gzip"."
>>
>> This goes some way to resolving issue #404 -- mnot's original report: "the implication on a HTTP/1 to HTTP/2 intermediary should be spelled out" -- but there are still a lot of implications not listed in that requirement. According to my understanding of HTTP, the appropriate added requirements are:
>>
>> * strip all conditional headers (If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match, If-Range) [i]
>> For such a response with Content-Encoding:gzip that wasn't Accept-Encoding:gzip :
>> * gunzip the entity, appropriately update Content-Encoding, Content-Length [ii], and add a Warning:214
>> There may be others. If so, my implementation (if I were writing one) would be lacking.
>>
>> There's still a big hole in "Cache-Control:no-transform". We've essentially added a requirement that gateways break conformance, without giving any guidance on how to do it (let alone reasonably justifying it in the first place). We haven't even told them the appropriate way to fail.
>>
>> In the absence of an opt-out mechanism for gzip C-E [iv], I think we should remove the support requirement. In fact I think we should remove section 9.3 altogether, but I'm not opposed to a strongly worded suggestion that everyone support gzip.
>>
>>
>> [i] or, in the case of a full-on transforming proxy, mint new Last-Modified (possibly even ETag) header, and be responsible for mapping the uncompressed entity on the 1.1 side to the compressed entity on the 2 side
>>
>> [ii] since HTTP 2 is already fragmented by frames, a gateway is probably going to always respond with chunked T-E, so Content-Length should probably be stripped (if present) anyway.
>>
>> [iii] thus explicitly acting as a non-transparent proxy according to RFC 2616, Section 13.5.2 (Non-modifiable Headers). I can't find the equivalent text in the HTTPbis drafts, but I assume there's something in there.
>>
>> [iv] an alternative could be to use Cache-Control:no-transform as such a flag; i.e. a server MUST NOT apply gzip C-E if the request did not include Accept-Encoding:gzip and did include Cache-Control:no-transform. That seems a bit flaky to me, though, and easy to miss/get wrong.
>>

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: <[email-masked]>
Date: Thu, 17 Apr 2014 08:06:52 +0000

Another missing "implication" of forcing implicit C-E gzip...

Must a 2 -> 1.1 reverse proxy append "gzip" to the Accept-Encoding header? What if the request has no Accept-Encoding header => add "Accept-Encoding: gzip" to the headers??

If signed headers are introduced in HTTP/3 (as has been mentioned in other threads e.g. [1]), modifying/adding Accept-Encoding would break the signed headers.

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0098.html