Subject: current HTTP/2 spec prevents gzip of response to "Range" request

From: <[email-masked]>
Date: Mon, 24 Mar 2014 14:51:35 +0000

Assertion: The HTTP/2 spec, as currently written, prevents gzip compression of the response to a "Range" request, when the client wants a range of the original content (i.e. it is not possible with HTTP/2, but was possible with HTTP/1.1).

Summary: Per RFC2616 (HTTP/1.1), ranges are applied after content encoding.  Thus when a range request is combined with gzip content encoding, an HTTP/1.1 compliant server will return a range of the gzip-compressed representation of the original content, not a gzip-compressed representation of the range of the original content. On the other hand, RFC2616 specifies that transfer encodings are the final codings applied to a message body and allows gzip as a transfer coding.  Thus, the only way to transfer a requested range of bytes with gzip compression on the wire is to use "Transfer-Encoding: gzip".  (Note, the only way a client can indicate to a server that it accepts gzip transfer encoding is by sending the header "TE: gzip".)  Since the HTTP/2 spec, as currently written, disallows transfer codings and "TE" request headers that contain any value other than "trailers", it follows that gzip transfer-encoded responses to range requests are disallowed.

Proposed Actions:
1. The perfect solution, would be to change HTTP/2 Section 9.3 from implicit gzip content encoding to implicit gzip transfer encoding.  Note that a gzip content-encoded response body and a gzip transfer-encoded response body are identical if no range is requested by the client.
2. At a minimum, the value "gzip" needs to be allowed for the "TE" request header and a mechanism needs to be introduced for gzip transfer encoding. This is in the spirit of the WG mandate to maintain the core specifications for HTTP.

Soapbox Statements:
- In our opinion, the common usage of "Content-Encoding: gzip" is actually a misnomer because in most cases it is actually used as a transfer encoding i.e. compressing the data on the wire (e.g. Google seems to heavily use gzip compression on the wire).
- In a limited survey of servers, we found that some deal with the range/gzip problem by either not honouring the range request, or disabling compression for responses with range requests. Other servers strictly follow the spec and provide (unusable?) data. By switching to gzip transfer encoding, this problem would go away.
- We have scenarios where it is necessary to incrementally download growing files (strictly appended). In this case, range requests + gzip compression provide a huge performance gain, but is impossible (in a compliant manner) without using gzip transfer encoding for the reasons mentioned in the above sections.

References:
The HTTP/1.1 spec (RFC2616) requires ranges to apply to the content-encoded content:
- from Section 14.35.1: "Byte range specifications in HTTP apply to the sequence of bytes in the entity-body ..."
- from Section 7.2.1: "entity-body := Content-Encoding( Content-Type( data ) )"

The HTTP/1.1 spec (RFC2616) allows transfer encodings (e.g. gzip) as the final encoding of a response entity-body (see above for definition of entity-body):
- from Section 4.3: "The message-body (if any) of an HTTP message is used to carry the entity-body associated with the request or response. The message-body differs from the entity-body only when a transfer-coding has been applied ...
message-body = entity-body | <entity-body encoded as per Transfer-Encoding>"
- from Section 14.39: "The TE request-header field indicates what extension transfer-codings it is willing to accept in the response ... Examples of its use are: TE: deflate ..."

The HTTP/2 spec disallows transfer encodings and the associated headers:
- from Section 8.1.3: "intermediaries SHOULD also remove other connection-specific header fields, such as ... Transfer-Encoding ..."
- from Section 8.1.3: "...the TE header field, which MAY be present in an HTTP/2 request, but when it is MUST NOT contain any value other than "trailers"..."

Sincerely,
Keith Morgan & Christoph Brunhuber

--
Remote Monitoring Unit
Safeguards, International Atomic Energy Agency (IAEA)
Vienna, Austria

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Zhong Yu <[email-masked]>
Date: Mon, 24 Mar 2014 10:29:20 -0500

On Mon, Mar 24, 2014 at 9:51 AM,  <[email-masked]> wrote:
> Assertion: The HTTP/2 spec, as currently written, prevents gzip compression
> of the response to a "Range" request, when the client wants a range of the
> original content (i.e. it is not possible with HTTP/2, but was possible with
> HTTP/1.1).
>
>
>
> Summary: Per RFC2616 (HTTP/1.1), ranges are applied after content encoding.
> Thus when a range request is combined with gzip content encoding, an
> HTTP/1.1 compliant server will return a range of the gzip-compressed
> representation of the original content, not a gzip-compressed representation
> of the range of the original content. On the other hand, RFC2616 specifies
> that transfer encodings are the final codings applied to a message body and
> allows gzip as a transfer coding.  Thus, the only way to transfer a
> requested range of bytes with gzip compression on the wire is to use
> "Transfer-Encoding: gzip".  (Note, the only way a client can indicate to a
> server that it accepts gzip transfer encoding is by sending the header "TE:
> gzip".)  Since the HTTP/2 spec, as currently written, disallows transfer
> codings and "TE" request headers that contain any value other than
> "trailers", it follows that gzip transfer-encoded responses to range
> requests are disallowed.
>
>
>
> Proposed Actions:
>
> 1. The perfect solution, would be to change HTTP/2 Section 9.3 from implicit
> gzip content encoding to implicit gzip transfer encoding.  Note that a gzip
> content-encoded response body and a gzip transfer-encoded response body are
> identical if no range is requested by the client.
>
> 2. At a minimum, the value "gzip" needs to be allowed for the "TE" request
> header and a mechanism needs to be introduced for gzip transfer encoding.
> This is in the spirit of the WG mandate to maintain the core specifications
> for HTTP.
>
>
>
> Soapbox Statements:
>
> - In our opinion, the common usage of "Content-Encoding: gzip" is actually a
> misnomer because in most cases it is actually used as a transfer encoding
> i.e. compressing the data on the wire (e.g. Google seems to heavily use gzip
> compression on the wire).
>
> - In a limited survey of servers, we found that some deal with the
> range/gzip problem by either not honouring the range request, or disabling
> compression for responses with range requests. Other servers strictly follow
> the spec and provide (unusable?) data. By switching to gzip transfer
> encoding, this problem would go away.
>
> - We have scenarios where it is necessary to incrementally download growing

For the sake of offering a defense -

This use case is probably not blessed by the letter of the spec:

http://tools.ietf.org/html/draft-ietf-httpbis-p5-range-26#section-4.3

  "These ranges can only be safely combined if they all have in common the
   same strong validator"

   "A client that has received multiple partial responses to GET requests
   on a target resource MAY combine those responses into a larger
   continuous range if they share the same strong validator."

Also, multipart/byteranges may carry a gzip-ed range, though
multipart/byteranges is probably not well supported.

Zhong Yu

> files (strictly appended). In this case, range requests + gzip compression
> provide a huge performance gain, but is impossible (in a compliant manner)
> without using gzip transfer encoding for the reasons mentioned in the above
> sections.
>
>
>
> References:
>
> - from Section 14.35.1: "Byte range specifications in HTTP apply to the
> sequence of bytes in the entity-body ..."
>
> The HTTP/1.1 spec (RFC2616) allows transfer encodings (e.g. gzip) as the
> final encoding of a response entity-body (see above for definition of
> entity-body):
>
> - from Section 4.3: "The message-body (if any) of an HTTP message is used to
> carry the entity-body associated with the request or response. The
> message-body differs from the entity-body only when a transfer-coding has
> been applied ...
>
> - from Section 14.39: "The TE request-header field indicates what extension
> transfer-codings it is willing to accept in the response ... Examples of its
> use are: TE: deflate ..."
>
>
>
> - from Section 8.1.3: "intermediaries SHOULD also remove other
> connection-specific header fields, such as ... Transfer-Encoding ..."
>
> - from Section 8.1.3: "...the TE header field, which MAY be present in an
> HTTP/2 request, but when it is MUST NOT contain any value other than
> "trailers"..."
>
>
>
>
>
> Sincerely,
>
> This email message is intended only for the use of the named recipient.
> Information contained in this email message and its attachments may be
> privileged, confidential and protected from disclosure. If you are not the
> intended recipient, please do not read, copy, use or disclose this
> communication to others. Also please notify the sender by replying to this
> message and then delete it from your system.

---

From: Zhong Yu <[email-masked]>
Date: Mon, 24 Mar 2014 10:39:48 -0500

> multipart/byteranges is probably not well supported.

I take it back - not sure about the semantics if a part in a
multipart/byteranges contains header "Content-Encoding: gzip". It'll
probably be ignored by most implementation.

Zhong Yu

---

From: Martin Thomson <[email-masked]>
Date: Mon, 24 Mar 2014 09:40:16 -0700

I think perhaps that you have Transfer-Encoding confused with Content-Encoding.

HTTP/2 does not change the characteristics of the protocol with
respect to ranges.  We do change Content-Encoding to requirement that
the client support gzip, but this doesn't affect the case you
describe.

On 24 March 2014 07:51,  <[email-masked]> wrote:
> of the response to a “Range” request, when the client wants a range of the
> “Transfer-Encoding: gzip”.  (Note, the only way a client can indicate to a
> server that it accepts gzip transfer encoding is by sending the header “TE:
> gzip”.)  Since the HTTP/2 spec, as currently written, disallows transfer
> codings and “TE” request headers that contain any value other than
> “trailers”, it follows that gzip transfer-encoded responses to range
> requests are disallowed.
>
>
>
> Proposed Actions:
>
> 2. At a minimum, the value “gzip” needs to be allowed for the “TE” request
> - In our opinion, the common usage of “Content-Encoding: gzip” is actually a
> - from Section 14.35.1: “Byte range specifications in HTTP apply to the
> sequence of bytes in the entity-body …”
>
> - from Section 7.2.1: “entity-body := Content-Encoding( Content-Type( data )
> )”
>
>
>
> - from Section 4.3: “The message-body (if any) of an HTTP message is used to
> been applied …
> message-body = entity-body | <entity-body encoded as per Transfer-Encoding>”
>
> - from Section 14.39: “The TE request-header field indicates what extension
> transfer-codings it is willing to accept in the response … Examples of its
> use are: TE: deflate …”
>
>
>
> - from Section 8.1.3: “intermediaries SHOULD also remove other
> connection-specific header fields, such as … Transfer-Encoding …”
>
> - from Section 8.1.3: “…the TE header field, which MAY be present in an
> "trailers"…”

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 24 Mar 2014 17:53:26 +0100

>the client support gzip, but this doesn't affect the case you

Section 8.1.3. of the current draft is quite clear that implementations
must not indicate acceptance of or support for transfer encodings using
the `TE` header; it seems clear that `Transfer-Encoding: gzip, chunked`
is not supposed to be used in HTTP/2.0, which is exactly the issue here.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Martin Thomson <[email-masked]>
Date: Mon, 24 Mar 2014 10:04:25 -0700

> is not supposed to be used in HTTP/2.0, which is exactly the issue here.

That's because Transfer-Encoding is a hop-by-hop thing and the
non-standard "gzip" Transfer-Encoding would need to be understood by
all hops in the chain if it were to be used for ranges.  I'm not aware
of any use of Transfer-Encoding other than "chunked", because it's
virtually impossible to define a new one that interoperates.

As I said, I think that this is a confusion between two different but
oft-confused headers:

Content-Encoding: gzip
Transfer-Encoding: chunked

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 24 Mar 2014 18:14:00 +0100

>Transfer-Encoding: chunked

Imagine you have a remote resource that is regularily appended to, like
a log file or a mailing list archive mbox file. You synchronise with it
by making regular Range requests to it to retrieve the content that has
since been appended, if any. To save bandwidth, you ask the server to
compress the data on the fly. In HTTP/1.1 this can be done using Trans-
fer-Encoding: gzip. How would you ask a HTTP/2.0 sever to send you the
newly appended content in a on-the-fly compressed form?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Martin Thomson <[email-masked]>
Date: Mon, 24 Mar 2014 10:26:19 -0700

> newly appended content in a on-the-fly compressed form?

Yes, this is a feature that we have removed in HTTP/2.  We could
probably do more to make this clear.

To be fair, support for Transfer-Encoding: gzip is only hop-by-hop in
HTTP/1.1.  We're introducing a hop that *cannot* do gzip (or any
as-yet-undefined Transfer-Encoding).

For the application you describe, I would recommend a different
architecture.  Range requests serve a very narrow purpose, and this is
probably extending their usefulness a little too much to be properly
reliable in the fact of ...say changing circumstances.  Why not
consider naming the things that this client retrieves?  Then
Content-Encoding will be sufficient.

---

From: Roland Zink <[email-masked]>
Date: Mon, 24 Mar 2014 18:46:54 +0100

With content-encoding the gzip compressed content and the uncompressed 
content are two entities. By requiring the client supporting gzip 
content-encoding the server is freed from the burden to know about the 
uncompressed content. When the client works on the uncompressed content 
then this makes it difficult for the client to compute the correct 
range. For example video players doing a seek may have an issue. In 
HTTP/1.1 the client can send the range request to the uncompressed 
content if it is willing to accept uncompressed content (or it is 
already compressed like a video), but in HTTP/2 this would no longer be 
possible.

The advantage of the transfer encoding would be that the requested range 
is for the uncompressed content and the server would be able to just 
compress the requested part of the content on the fly. Also 
intermediaries can decompress the content if they wish in order to 
examine the content. Quite often content-encoding is just used to 
compress the content for the transfer without thinking about the 
potential problems like caches storing two versions.

Regards,
Roland

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 24 Mar 2014 13:19:10 -0700

> Transfer-Encoding: chunked

It seems you have confused them.  Transfer Encoding is something
that can be added or removed by the protocol.  Content Encoding is
metadata about the representation.  If the protocol modifies a value
for CE, then it breaks anything that relies on the payload being
delivered without transformation (e.g., ranges and signature metadata).

If HTTP/2 doesn't allow compression transfer encodings, then it doesn't
allow compression by intermediaries. I assumed that TE was replaced by
a framing mechanism that indicates the payload has been compressed.
If that isn't so, then HTTP/2 will be less efficient than HTTP/1 for
some use cases (like CDNs).

And, no, it isn't virtually impossible to introduce standard transfer
codings.  It just requires effort by browsers to support one.
It also isn't necessary to restrict HTTP/2 features to what a current
browser supports.

....Roy

---

From: Martin Thomson <[email-masked]>
Date: Mon, 24 Mar 2014 13:42:59 -0700

On 24 March 2014 13:19, Roy T. Fielding <[email-masked]> wrote:
> It seems you have confused them.

Let's take the "seems" out of that, I did confuse them.  And it is not
the first time that has happened in this area.

> some use cases (like CDNs).

We had a long series of discussions on this point.  It may be that we
failed to trigger the necessary reactions from folks at the time and
this needs to be reopened.  I will not attempt to defend the process
we took.

0. SPDY had a frame to indicate that frames were compressed
http://tools.ietf.org/html/draft-mbelshe-httpbis-spdy-00#section-2.2.2

1. it was removed after some discussion:
https://github.com/http2/http2-spec/issues/46
http://http2.github.io/http2-spec/#changes.since.draft-ietf-httpbis-http2-01
(point 2; -01 was 2013-01 FYI)

2. we discussed it more on-list
http://lists.w3.org/Archives/Public/ietf-http-wg/2013AprJun/0865.html

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 25 Mar 2014 08:21:04 +1000

> http://lists.w3.org/Archives/Public/ietf-http-wg/2013AprJun/0865.html
If I'm reading it right: TE:gzip was removed because servers were wasting
resources recompressing things that were already CE:gzip, and because
people wanted both a "transfer-length"-type field to do their download
progress meters and a "content-length"-type field to do allocations and
whatnot. Those both seem like they have better solutions than just "no
TE:gzip".

I'm pretty sure I read through the entire discussion in point 2, and that
whole thread seemed to be people talking at cross purposes with confusion
between TE and CE.

I'm completely fine with continuing to mandate support for TE:chunked,
however it seems like a regression to forbid other codings (such as
compression) when they may have had support in HTTP/1.1. Yes,
TE/Transfer-Encoding is a connection header, i.e. it's hop-by-hop, and if
there's a naive proxy in the way you'll lose it (and if it's a *really* naive
proxy you'll break stuff), but that's already the case, no?

On the other side, I'm not sold on a single "gzip" bit. As people have
said, gzip is widely supported but it's not very good, and I don't like
being further tied to it in the protocol. Tied to it by convention, fine,
but not in MUST (or even SHOULD) level requirements. I'm of the opinion
that an extensible header, which can support arbitrary (and experimental)
codings, is much better, despite the fact that no one seems to get the
difference between CE and TE headers. I hope I'm not picking a fight with
proxy people here; they already have to unpack all the headers to handle
cache directives and whatnot, no? So asking them to strip/modify a
TE/Transfer-Encoding header at the same time if they know that one end
doesn't play nicely doesn't seem too unreasonable.

On my soap box, I really wish we could just do away with CE. I'm happy for
foo.html.gz to be transferred as "Content-Type: application/x-gzip" instead
of "Content-Type: text/html\r\nContent-Encoding: gzip". However I know
that's never going to happen (and it's outside HTTP/2's charter anyway).

For the record: I routinely configure Apache to serve offline-compressed
versions of files [1], which I believe is the Right Way(tm) to do CE; and I've
written an API[2] that allows resources to be dynamically compressed, and
it jumps through hoops to provide content-encoded gzip (i.e. the Wrong
Way(tm)) because no one seems to do TE and I didn't want my code to go to
waste.

[1]: https://gist.github.com/phluid61/9750448
[2]: 
https://github.com/QUTlib/rest-rmr/blob/master/system/response.inc.php#L758-863

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Mar 2014 12:50:20 +1100

On 25 Mar 2014, at 7:19 am, Roy T. Fielding <[email-masked]> wrote:

> allow compression by intermediaries.

... or at least proxies. Since gateways have a relationship with the origin, they can (and often do) coordinate compression with it.

> I assumed that TE was replaced by

It'd be more accurate to say that it's less efficient *in theory*.

Let's be clear; on the browsing Web, gzip and deflate transfer-codings are basically unused. CDNs do not use them, browsers do not support them, servers do not support them. 

E.g., <https://issues.apache.org/bugzilla/show_bug.cgi?id=52860>.

> It also isn't necessary to restrict HTTP/2 features to what a current

That's true. However, transfer-codings other than chunked have resolutely failed to catch on over the past ~15 years. Content-encoding -- with warts and all -- is by far the broadest current practice.

We can certainly talk about re-introducing a flag to indicate that the payload of DATA is compressed. I don't see how we can require it to be used, however, since support for gzip transfer-codings is so poor on the existing Web.

There are also the security considerations, a la <https://github.com/http2/http2-spec/issues/423>.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Roland Zink <[email-masked]>
Date: Tue, 25 Mar 2014 10:39:52 +0100

>> allow compression by intermediaries.
Or decrompression in a HTTP/1.1 to HTTP2 gateway.
> E.g., <https://issues.apache.org/bugzilla/show_bug.cgi?id=52860>.
But you also find comments like in 
https://bugzilla.mozilla.org/show_bug.cgi?id=68517

Jon Hanna 2012-08-14 02:13:05 PDT

We started using content-encoding in cases where transfer-encoding is what we really want in the 1990s as a temporary kludge until the browsers added support, which would be any moment now...

> That's true. However, transfer-codings other than chunked have resolutely failed to catch on over the past ~15 years. Content-encoding -- with warts and all -- is by far the broadest current practi…
The question is why. For example Opera introduced support for gzip 
transfer encodings and then saw responses gziped twice (also mentioned 
in the above link).

> We can certainly talk about re-introducing a flag to indicate that the payload of DATA is compressed. I don't see how we can require it to be used, however, since support for gzip transfer-codings…
Isn't this hidden in the http2 stack? It shouldn't be visible outside 
the HTTP2 hop although I guess you will see content compressed twice in 
some cases.

Regards,

Roland Zink         http://home.zinks.de/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 25 Mar 2014 11:23:09 -0700

> ... or at least proxies. Since gateways have a relationship with the origin, they can (and often do) coordinate compression with it.

They can define another protocol, yes, but how would they
handle compression of the request content with standard semantics?
Here we are assuming that the user agent is deciding to compress and
the recipients have to be able to distinguish between content that is
just being compressed for performance of this transfer versus content
that is intended to be forever compressed.

> E.g., <https://issues.apache.org/bugzilla/show_bug.cgi?id=52860>.

That is just a chicken and egg problem.  Dynamic content encoding is
known to be broken in almost every respect, and if I ever have any time
free then that issue will be fixed.

> That's true. However, transfer-codings other than chunked have resolutely failed to catch on over the past ~15 years. Content-encoding -- with warts and all -- is by far the broadest current practi…

They aren't comparable.  Dynamic CE is deployed, but is known to break
content signatures, ranges, and caches, and is incapable of supporting
selectively compressed transfer from client to server.  Transfer
compression has limited deployment but actually does work when it is
deployed by consenting adults.  You can change that deployment in HTTP/2.
It is well within scope.

> There are also the security considerations, a la <https://github.com/http2/http2-spec/issues/423>.

I would not require it to be used -- just make it possible to be used
and required to understand (decompress by recipient).  This is what
you already do with HTTP/2 -- just stop calling it content encoding.
It is not the same thing.

FWIW, the section on Use of Compression is incorrect because it assumes
that the channel is secure (and I am not sure why it says plaintext,
because the attack is not limited to text).  What it should say:

"Implementations communicating on a secure channel MUST NOT generate
dynamically compressed content that includes both confidential data
and attacker-controlled data unless separate compression dictionaries
are used for each source of data."

....Roy

---

From: Martin Thomson <[email-masked]>
Date: Tue, 25 Mar 2014 11:45:22 -0700

I think that pinning your hopes for "Transfer-Encoding: gzip" on
HTTP/2 might be a little too optimistic, particularly for secure
channels: making decisions that depend on knowing provenance is much
harder when said provenance is several degrees removed.

But the marginal value of this new feature seems insufficient to
justify the feature, even if that new feature is just one we decided
not to port from HTTP/1.1.

On 25 March 2014 11:23, Roy T. Fielding <[email-masked]> wrote:
> Implementations communicating on a secure channel MUST NOT generate
> are used for each source of data.

Yep, that is better.

---

From: <[email-masked]>
Date: Tue, 25 Mar 2014 19:16:41 +0000

Based on the feedback from the group thus far we have written up a proposed replacement for section "9.3 GZip Content-Encoding". Our proposal is based on using gzip transfer encoding. We are proposing this route because, as Matthew pointed out, it allows more freedom than a flag which is forever locked to gzip.

**NOTE: We believe our proposal still maintains backwards compatibility with the poor misused/abused Content-Encoding prevalent in HTTP/1.1**

> On 25 Mar 2014, Roy T. Fielding <[email-masked]> wrote:

> just stop calling it content encoding. It is not the same thing.

We couldn't agree more!

Here is our proposal...

9.3 GZip Transfer-Encoding

Clients MUST support gzip compression for HTTP response bodies. Regardless of the value of the TE request header field, a server MAY send responses with gzip transfer encoding. A compressed response MUST still bear an appropriate transfer-encoding header field. This effectively changes the implicit value of the TE header field ([HTTP-p2], Section 14.39) from "chunked" to "gzip, chunked". Servers SHOULD not use gzip transfer encoding if the requested content is already compressed (e.g. images, videos, compressed files, etc.). Servers MUST not include the values "gzip" or "deflate" in a Content-Encoding header, regardless of whether the requested content is already compressed, but SHOULD include the appropriate mime type in the Content-Type header. Correspondingly, clients SHOULD not include the values "gzip" or "deflate" in a Accept-Encoding header.

The following rules apply to intermediaries that perform translation from HTTP/2 to HTTP/1.1:

1) if the request does not include an Accept-Encoding or TE header that includes the value "gzip", the intermediary MUST decompress the payload,

2) if the request includes an Accept-Encoding header that includes the value "gzip", but does not include a TE header that includes the value "gzip",

   a) the intermediary MUST decompress payloads that are gzip transfer encoded and have a :status header value of "206",

   b) the intermediary SHOULD not decompress payloads that are gzip transfer encoded and have a :status header value not "206", and if the intermediary elects to keep the payload compressed, MUST remove the value "gzip" from the Transfer-Encoding header and insert the header "Content-Encoding: gzip" in order to maintain backwards compatibility with HTTP/1.1 clients,

3) if the request includes a TE header that includes the value "gzip", the intermediary SHOULD not decompress the payload.

Clients, servers and intermediaries MAY elect to support other compression methods for transfer encoding, in which case it MUST be explicitly requested by the client in the TE header.

--

Keith S. Morgan

Remote Monitoring Unit

Vienna, Austria

Office: +43 1 2600 26672

Handy: +43 699 165 26672

---

From: Patrick McManus <[email-masked]>
Date: Tue, 25 Mar 2014 15:54:58 -0400

I'm unlikely to implement a gzip transfer encoding decoder any time soon.
The distinction between TE and CE is too confusing and I think we're better
off without the footgun and just using a single defacto CE mechanism
instead (with all its admitted warts). The mandatory implied
Accept-Encoding codifies a well worn and useful optimization..
Transfer-Encoding doesn't have the same track record to justify adding it
at the same level.

I'm neutral on adjusting the spec to allow explicit gzip in TE and
Transfer-Encoding negotiated between consenting peers.

-P

On Tue, Mar 25, 2014 at 3:16 PM, <[email-masked]> wrote:

>  Based on the feedback from the group thus far we have written up a
> proposed replacement for section "9.3 GZip Content-Encoding". Our proposal
> is based on using gzip transfer encoding. We are proposing this route
> because, as Matthew pointed out, it allows more freedom than a flag which
> is forever locked to gzip.
>
>
>
> **NOTE: We believe our proposal still maintains backwards compatibility
> with the poor misused/abused Content-Encoding prevalent in HTTP/1.1**
>
>
>
> Clients MUST support gzip compression for HTTP response bodies. Regardless
> of the value of the TE request header field, a server MAY send responses
> with gzip transfer encoding. A compressed response MUST still bear an
> appropriate transfer-encoding header field. This effectively changes the
> implicit value of the TE header field ([HTTP-p2], Section 14.39) from
> "chunked" to "gzip, chunked". Servers SHOULD not use gzip transfer encoding
> if the requested content is already compressed (e.g. images, videos,
> compressed files, etc.). Servers MUST not include the values "gzip" or
> "deflate" in a Content-Encoding header, regardless of whether the requested
> content is already compressed, but SHOULD include the appropriate mime type
> in the Content-Type header. Correspondingly, clients SHOULD not include the
> values "gzip" or "deflate" in a Accept-Encoding header.
>
>
>
> 1) if the request does not include an Accept-Encoding or TE header that
> includes the value "gzip", the intermediary MUST decompress the payload,
>
> 2) if the request includes an Accept-Encoding header that includes the
> value "gzip", but does not include a TE header that includes the value
> "gzip",
>
>    a) the intermediary MUST decompress payloads that are gzip transfer
> encoded and have a :status header value of "206",
>
>    b) the intermediary SHOULD not decompress payloads that are gzip
> transfer encoded and have a :status header value not "206", and if the
> intermediary elects to keep the payload compressed, MUST remove the value
> "gzip" from the Transfer-Encoding header and insert the header
> "Content-Encoding: gzip" in order to maintain backwards compatibility with
> HTTP/1.1 clients,
>
> 3) if the request includes a TE header that includes the value "gzip", the
> intermediary SHOULD not decompress the payload.
>
>
>
> Clients, servers and intermediaries MAY elect to support other compression
> methods for transfer encoding, in which case it MUST be explicitly
> requested by the client in the TE header.
>
>
>
>
>

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 26 Mar 2014 08:24:43 +1000

On 26 March 2014 05:16, <[email-masked]> wrote:

>  Clients MUST support gzip compression for HTTP response bodies.
> Regardless of the value of the TE request header field, a server MAY send
> responses with gzip transfer encoding. A compressed response MUST still
> bear an appropriate transfer-encoding header field. This effectively
> changes the implicit value of the TE header field ([HTTP-p2], Section
> 14.39) from "chunked" to "gzip, chunked". Servers SHOULD not use gzip
> transfer encoding if the requested content is already compressed (e.g.
> images, videos, compressed files, etc.).
>
Do you mean HTTP-p1, Section 4.3 ?

> Servers MUST not include the values "gzip" or "deflate" in a
> Content-Encoding header, regardless of whether the requested content is
> already compressed, but SHOULD include the appropriate mime type in the
> Content-Type header. Correspondingly, clients SHOULD not include the values
> "gzip" or "deflate" in a Accept-Encoding header.
>
Why? Is this an attempt to work around double-compression? It seems pretty
heavy-handed. How about this example?

* I have an Apache web server which serves two files: index.html and
index.html_gz, the latter beings a gzip-compressed version of the first.
* I have configured the Content-Type of the index.html_gz as "text/html",
and the Content-Encoding as "gzip" [a].
* I have configured my Apache server to detect Accept-Encoding:gzip on
requests for the index.html file, and internally redirect those requests to
index.html_gz
* Apache then handles setting response headers such as Last-Modified, ETag,
etc. based on the file stat, as it would any other file request.

[a]: these response headers are included if a client requests
index.html_gz even without Accept-Encoding:gzip ; my workaround here is to
not expose the _gz file, e.g. by not linking to it.

Because of the MUST NOT, I now cannot take advantage of my proactive
compression in HTTP/2, and instead either 1) require the server to compress
the response on-the-fly each time, or 2) lose the compression. Also,
because TE is hop-by-hop I risk some intermediary terminating the
compression, possibly negatively impacting my site's responsiveness in a
way that is outside my control; but CE is end-to-end, so intermediaries
shouldn't be touching the compressed data (I say "shouldn't" but I know
some still will).

Yes, I said I'd like to do away with CE, but it's widely used and very
popular, we would need to come up with a legitimate replacement. I believe
that mandating support for TE might provide a gradual path for replacement,
especially if/when HTTP/1 fades away from the web (i.e. if every hop
guarantees support for TE, I have one less need for CE).

This paragraph should be a best practice guideline, if anything, not an
interoperability requirement.

> requested by the client in the TE header.
Should probably mention the IANA HTTP Transfer Coding Registry here.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 26 Mar 2014 08:28:15 +1000

On 26 March 2014 05:54, Patrick McManus <[email-masked]> wrote:

> instead (with all its admitted warts).
>

The fact that one of these "warts" is that people think CE is a transport
feature
 is one that I strongly believe we should not perpetuate.

HTTP/2 is a brand new transport, surely it will only help to clarify the
issue if we discuss, codify, and/or revise transfer codings, and completely
doesn't mention content codings. People might see it and think "oh, TE is a
transport thing, and CE isn't". Maybe.

Incidentally, is not the current implementation of CE:gzip effectively a
"gzip transfer encoding decoder" anyway? Surely that would be *easier* to
scavenge than implementing a whole new binary, header table based, huffman
coded, bells and whistles transport mechanism. Of course, I says it as
shouldn't, since I don't know the Firefox codebase at all.

> The mandatory implied Accept-Encoding codifies a well worn and useful
> optimization.. Transfer-Encoding doesn't have the same track record to
> justify adding it at the same level.
>

It's not at the same level; CE is above TE. Making claims in HTTP/2 about
CE is too close to modifying existing protocol semantics for my liking,
whereas TE is exactly the space in which we're working. And like I said,
we're already completely rewriting the entire transport mechanism; adding
compression is hardly out of place.

Put glibly: a binary wire format for HTTP doesn't have *any* track record
at all, but we're adding that.

---

From: Patrick McManus <[email-masked]>
Date: Tue, 25 Mar 2014 22:02:57 -0400

Hi Matthew,

On Tue, Mar 25, 2014 at 6:28 PM, Matthew Kerwin <[email-masked]>wrote:

> \Surely that would be *easier* to scavenge
>

my reticence is not about implementation complexity in my client.

> It's not at the same level; CE is above TE.
>

I'm sorry to have created confusion. I mean I don't support the same level
of conformance (i.e. the suggested MUST decode) - not the same level in the
application. as I say, I'm pretty neutral on defining negotiation of it.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 26 Mar 2014 14:06:26 +1000

> my reticence is not about implementation complexity in my client.

I understand. Sorry.

> application. as I say, I'm pretty neutral on defining negotiation of it.

That makes sense. In that case I don't disagree; forcing it with a
MUST-level requirement, or even a SHOULD, is not the right approach. Maybe
a best practice guideline, if it's decided that it is, in fact, best
practice.

That said, I'm pretty strongly *for* supporting the negotiation in the
first place. Maybe it needs to be something that's clearly a part of
HTTP/2, freeing it from the stigma attached to the existing
TE/Transfer-Encoding headers.

After much deliberation I have a crazy idea: introduce a new frame type for
hop-by-hop headers.

As far as I can see, we've eliminated connection headers (and the
Connection header) from HTTP/2, with the exception of "TE: trailers", and I
don't think it's necessarily appropriate to reintroduce them, at least not
through the HEADERS frame.

I can think of a couple of benefits:
* It would be easier to clearly distinguish hop-by-hop and end-to-end
concerns (e.g. hopefully help clear up some of the CE/TE confusion).
* It would provide more flexibility than, say, a settings parameter or a
flag in a data frame, by allowing the same type of coding negotiation
currently allowed in HTTP.
* It would be extensible, allowing experimental headers, not unlike HH from
[draft-nottingham-http-browser-hints-05].
* It also makes it easier for people to ignore them ;)

Does anyone have a reason (aside from YAGNI) that we wouldn't want to
pursue this new frame type?

---

From: Martin Thomson <[email-masked]>
Date: Tue, 25 Mar 2014 21:27:13 -0700

On 25 March 2014 21:06, Matthew Kerwin <[email-masked]> wrote:
> Does anyone have a reason (aside from YAGNI) that we wouldn't want to pursue
> this new frame type?

I don't know.  YAGNI is a pretty killer argument.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 26 Mar 2014 14:45:13 +1000

> I don't know.  YAGNI is a pretty killer argument.

Especially early on in the piece, before potential uses have a chance to be
identified.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 26 Mar 2014 07:50:51 +0000

In message <CAOdDvNrOJXu8r86A1wyShaj94ccTcEZByLnEcE_=[email-masked]>

>I'm unlikely to implement a gzip transfer encoding decoder any time soon.

And I don't think Varnish will either.

Having gone through all that stuff recently, I am well aware of all the
trouble with C-E, but at least it is known trouble at this point in time.

I'd like to add one observation which I have not seen mentioned yet:

A big issue with T-E: gzip is that you need logic to recognize already
compressed content.

You can either do this by attempting compression and finding out that
it doesn't help, which requires access to about 1k+ of data before you
can trust the result, or by having a magic list of format markers
(see file(1)) to tell you, and be caught off-guard when somebody does
something new on the web.

Both is unattractive complications for very little gain.

Analytically, about the only reasons to use T-E: gzip is because the
origin server were either to stupid or too old to compress content
using C-E, and while a lot of origin servers fit that description,
I think we should just ignore them.

If we're talking "intelligent" lightbulbs, their bandwidth does not
matter and if we're talking old servers where load/bandwidth matters,
you can stick a Varnish in front of it and have that do CE: gzip
for you.

Compression is best done where the semantics and compressibility
of the content is known, attempting to fix it up later is a kludge.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Roland Zink <[email-masked]>
Date: Wed, 26 Mar 2014 10:31:01 +0100

> I think we should just ignore them.
C-E does not allow seeks, e.g. you need to read 4G of a 8G file just to 
read the 100 bytes you are interested in.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 26 Mar 2014 09:48:37 +0000

>read the 100 bytes you are interested in.

That's news to me.  Range works fine with C-E in Varnish...

---

From: Roland Zink <[email-masked]>
Date: Wed, 26 Mar 2014 10:57:26 +0100

Up to now the accept-encoding is a negotiation and the implied one is 
not well worn. Potentially a HTTP/1.1 to HTTP2 gateway may be used to 
allow HTTP/1.1 clients to talk to HTTP2 servers. Existing HTTP/1.1 
clients may not support C-E gzip and the gateway can't do the 
decompression without affecting HTTP/1.1 functionality like range 
requests. So some HTTP/1.1 clients will be incompatible to the HTTP2 
world. This is okay if there is an agreement that this is acceptable.

Roland

On 25.03.2014 20:54, Patrick McManus wrote:
> I'm unlikely to implement a gzip transfer encoding decoder any time 
> soon. The distinction between TE and CE is too confusing and I think 
> we're better off without the footgun and just using a single defacto 
> CE mechanism instead (with all its admitted warts). The mandatory 
> implied Accept-Encoding codifies a well worn and useful optimization.. 
> Transfer-Encoding doesn't have the same track record to justify adding 
> it at the same level.
>
> On Tue, Mar 25, 2014 at 3:16 PM, <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     proposed replacement for section "9.3 GZip Content-Encoding". Our
>     proposal is based on using gzip transfer encoding. We are
>     proposing this route because, as Matthew pointed out, it allows
>     more freedom than a flag which is forever locked to gzip.
>
>     **NOTE: We believe our proposal still maintains backwards
>     compatibility with the poor misused/abused Content-Encoding
>     prevalent in HTTP/1.1**
>
>     > On 25 Mar 2014, Roy T. Fielding <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>     Regardless of the value of the TE request header field, a server
>     MAY send responses with gzip transfer encoding. A compressed
>     response MUST still bear an appropriate transfer-encoding header
>     field. This effectively changes the implicit value of the TE
>     header field ([HTTP-p2], Section 14.39) from "chunked" to "gzip,
>     chunked". Servers SHOULD not use gzip transfer encoding if the
>     requested content is already compressed (e.g. images, videos,
>     compressed files, etc.). Servers MUST not include the values
>     "gzip" or "deflate" in a Content-Encoding header, regardless of
>     whether the requested content is already compressed, but SHOULD
>     include the appropriate mime type in the Content-Type header.
>     Correspondingly, clients SHOULD not include the values "gzip" or
>     "deflate" in a Accept-Encoding header.
>
>     1) if the request does not include an Accept-Encoding or TE header
>     that includes the value "gzip", the intermediary MUST decompress
>     the payload,
>
>     2) if the request includes an Accept-Encoding header that includes
>     the value "gzip", but does not include a TE header that includes
>     the value "gzip",
>
>     a) the intermediary MUST decompress payloads that are gzip
>     transfer encoded and have a :status header value of "206",
>
>     transfer encoded and have a :status header value not "206", and if
>     the intermediary elects to keep the payload compressed, MUST
>     remove the value "gzip" from the Transfer-Encoding header and
>     insert the header "Content-Encoding: gzip" in order to maintain
>     backwards compatibility with HTTP/1.1 clients,
>
>     3) if the request includes a TE header that includes the value
>     "gzip", the intermediary SHOULD not decompress the payload.
>
>     Clients, servers and intermediaries MAY elect to support other
>     compression methods for transfer encoding, in which case it MUST
>     be explicitly requested by the client in the TE header.
>
>     This email message is intended only for the use of the named
>     recipient. Information contained in this email message and its
>     attachments may be privileged, confidential and protected from
>     disclosure. If you are not the intended recipient, please do not
>     read, copy, use or disclose this communication to others. Also
>     please notify the sender by replying to this message and then
>     delete it from your system.
>
>

---

From: Roland Zink <[email-masked]>
Date: Wed, 26 Mar 2014 11:04:48 +0100

> That's news to me.  Range works fine with C-E in Varnish...
Range works fine with C-E as long as the client retrieves the whole 
beginning of the file. If the client wants to skip some parts of it then 
it is unable to decompress. Retrieving 0-999 then 1000-1999 and so on is 
fine. But retrieving 0-999 and then 2000-2999 is a problem.

Regards,
Roland

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 26 Mar 2014 10:22:37 +0000

>fine. But retrieving 0-999 and then 2000-2999 is a problem.

Can you give an example of a real-world application where this is anything
but a minor inconvenience ?

---

From: <[email-masked]>
Date: Wed, 26 Mar 2014 10:41:57 +0000

>> In message <[email-masked]<mailto:[email-masked]>>, Roland Zink writes:

>>> C-E does not allow seeks, e.g. you need to read 4G of a 8G file just

>>> to read the 100 bytes you are interested in.

This is the crux of the issue, hence the subject of this conversation!

>> That's news to me.  Range works fine with C-E in Varnish...

If Range works with C-E in Varnish, then Varnish is not HTTP/1.1 compliant.

Again, quoting the HTTP/1.1 spec, RFC2616 requires ranges to apply to the content-encoded content:
- from Section 14.35.1: "Byte range specifications in HTTP apply to the sequence of bytes in the entity-body ..."
- from Section 7.2.1: "entity-body := Content-Encoding( Content-Type( data ) )"

---

From: <[email-masked]>
Date: Wed, 26 Mar 2014 10:45:13 +0000

> - from Section 7.2.1: "entity-body := Content-Encoding( Content-Type( data ) )"

Here is an example from a HTTP/1.1 compliant server (Google's "sffe" server). Consider the following two range requests for an arbitrary url from google pointing to a static html page. (For  brevity, headers superfluous to the example have been omitted. Binary data are converted to hex. Captures were taken with Fiddler.)

1. Here is a range request without accepting gzip and the response from google...

GET https://www.google.at/insidesearch/howsearchworks/crawling-indexing.html HTTP/1.1

Host: www.google.at

User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)

Range: bytes=0-46

HTTP/1.1 206 Partial Content

Content-Range: bytes 0-46/18586

Server: sffe

Content-Length: 47

<!DOCTYPE html>

<html class="google" lang="de">

2. Now, here is the same range request, with "Accept-Encoding: gzip", and the response from google...

GET https://www.google.at/insidesearch/howsearchworks/crawling-indexing.html HTTP/1.1

Host: www.google.at

User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)

Range: bytes=0-46

Accept-Encoding: gzip

HTTP/1.1 206 Partial Content

Content-Range: bytes 0-46/4882

Content-Encoding: gzip

Server: sffe

Content-Length: 47

0000000 1f 8b 08 00 00 00 00 00 02 ff ad 5c ed 6f 1c c7

0000020 79 ff ee bf 62 72 1f 42 12 e5 dd 91 b6 e2 26 12

0000040 29 81 34 f5 56 db 32 23 d2 16 aa 24 20 e6 6e

What are these 47 bytes? And why is the Content-Range of the first "0-46/18586", but the second is "0-46/4882"?

The second request has returned the first 47 bytes of the gzip representation of the _entire_ file (not the gzip representation of just the range).

What can you do with these bytes?  Nothing.  You can't decompress a partial gzip file. The only thing you could do is request the rest of the file (i.e. range 47-4882), reassemble the gzip, decompress (yielding the entire file) and then extract the 47 bytes you are interested in.

--

Keith S. Morgan

Remote Monitoring Unit

Vienna, Austria

---

From: Roland Zink <[email-masked]>
Date: Wed, 26 Mar 2014 11:45:42 +0100

Looks like there is the log file example mentioned in the discussion. 
Where this is often used is in playing videos (although there often no 
gzip is used but I have seen this too). Many players allow the user to 
jump within the video.

Roland

---

From: <[email-masked]>
Date: Wed, 26 Mar 2014 10:53:45 +0000

On Wednesday,26 March 2014 11:23, Poul-Henning Kamp wrote:
>>beginning of the file. If the client wants to skip some parts of it
>>then it is unable to decompress. Retrieving 0-999 then 1000-1999 and so
>>on is fine. But retrieving 0-999 and then 2000-2999 is a problem.
>
>Can you give an example of a real-world application where this is anything but a minor inconvenience ?

Earlier in this conversation, Björn gave the following example...

> Imagine you have a remote resource that is regularily appended to, like a log file or a mailing list archive mbox file.
> You synchronise with it by making regular Range requests to it to retrieve the content that has since been appended, if any.
> To save bandwidth, you ask the server to compress the data on the fly. In HTTP/1.1 this can be done using
> Transfer-Encoding: gzip.
> How would you ask a HTTP/2.0 sever to send you the newly appended content in a on-the-fly compressed form?

In fact in this example, where you are requesting "Content-Encoding: gzip" and a range from the end of the file, an HTTP/1.1 compliant server would have to return "416 Requested Range Not Satisfiable" because first it would compress the whole file and then look for the range which would likely begin beyond the length of the compressed file (if it is at all compressible).

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 26 Mar 2014 11:01:09 +0000

>gzip is used but I have seen this too). Many players allow the user to 

The logfile example I'm just going to ignore because it is bogus on
so many planes that it's not even funny:  The exact same problem
exists with compressed logfiles when no HTTP is involved at alle.

Video formats defined exactly so that they are searchable, and if
anybody wastes their time gzip'ing videos, they have bigger problems
than not being able to search them.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 26 Mar 2014 11:06:25 +0000

In message <[email-masked]>, [email-masked] wr
ites:

>> Imagine you have a remote resource that is regularily appended to, like a
> log file or a mailing list archive mbox file.
>> You synchronise with it by making regular Range requests to it to retriev
>e the content that has since been appended, if any.

You must have overlooked the "real-world" I put in my question ?  :-)

If your goal is to make HTTP the protocol that can do *everything*, then you
are going to get a protocol which is good for nothing.

---

From: Roland Zink <[email-masked]>
Date: Wed, 26 Mar 2014 12:23:18 +0100

> than not being able to search them.
I know people which are doing this exactly because video formats are 
searchable. Padding is used to make search work and gzip is used to get 
rid most of the padding on the wire.

---

From: <[email-masked]>
Date: Wed, 26 Mar 2014 11:37:20 +0000

On Wednesday,26 March 2014 12:06, Poul-Henning Kamp wrote:
> The logfile example I'm just going to ignore because it is bogus on so many planes that it's not even funny:

> The exact same problem exists with compressed logfiles when no HTTP is involved at alle.

> You must have overlooked the "real-world" I put in my question ?  :-)

We do something very similar in the "real world" with data acquisition systems. But we are using a HTTP/1.1 _compliant_ server - and so the only option for compressed range requests was to do gzip transfer encoding.

> That's news to me.  Range works fine with C-E in Varnish...

You tell me "real-world" examples of Varnish using Range with C-E gzip and there you'll have all the examples you need.

> If your goal is to make HTTP the protocol that can do *everything*, then you are going to get a protocol which is good for nothing.

Not our goal at all.  _We just want to make sure HTTP/2 doesn't lock out something you can already do with HTTP/1.1._  As the HTTP/2 spec is currently written, you can't do compressed range requests because gzip transfer encoding is disallowed. But at the same time, if we can help eliminate the misuse/abuse of C-E gzip, that would be great too.

*Don't forget that HTTP isn't just for browsing the web.* Sure there are other protocols out there (e.g. FTP, SMB, CIFS, ...), but they all have their own issues (overhead, security, etc.).  But HTTP is the most widely used protocol in the world and has so many nice features that it gets used for lots of other purposes besides browsing.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 26 Mar 2014 12:11:30 +0000

In message <[email-masked]>, [email-masked] wr
ites:

>> If your goal is to make HTTP the protocol that can do *everything*, then
>>you are going to get a protocol which is good for nothing.
>
>Not our goal at all.  _We just want to make sure HTTP/2 doesn't
>lock out something you can
>already do with HTTP/1.1.

So actually yes, that is your goal...

HTTP/1 is an incredibly messy protocol, and it is being abused for all
sorts of purposes, and having a goal of carrying all abuses forward,
no matter how stupid or pointless.

And I think that is a very stupid goal.

If we are not willing to cut off some of thos abuses, in exchange
for clarity and simplicity, HTTP/2.0 ain't going to be an improvement.

---

From: Zhong Yu <[email-masked]>
Date: Wed, 26 Mar 2014 11:21:48 -0500

On Mon, Mar 24, 2014 at 10:39 AM, Zhong Yu <[email-masked]> wrote:
> I take it back

A little elaboration on the topic:

If a response looks like this

    HTTP/1.1 206 Partial Content
    Content-Type: multipart/byteranges,
    Content-Encoding: gzip

What does it mean exactly? Per "normal" interpretation, "gzip" is
applied on the multipart. This interpretation seems plausible for a
reader of RFC2616. And it would be quite understandable if an
implementation decompress the response body first, then interpret it
as a multipart.

The new bis draft makes it clearer that "gzip" is applied on the
representation and "multipart" is applied on the payload, therefore
the above interpretation is wrong. If a response is 206 and the
Content-Type is multipart/byteranges, the "normal" interpretation does
not apply. Because "Content-Type: multipart/byteranges" is abused for
this special case, the multipart/byteranges mime type must never be
used in any other situations.

Zhong Yu

---

From: <[email-masked]>
Date: Wed, 26 Mar 2014 17:11:05 +0000

On Wednesday,26 March 2014 17:22, Zhong Yu wrote:
> What does it mean exactly? Per "normal" interpretation, "gzip" is applied on the multipart.
> This interpretation seems plausible for a reader of RFC2616. And it would be quite
> understandable if an implementation decompress the response body first, then interpret
> it as a multipart.
>
> The new bis draft makes it clearer that "gzip" is applied on the representation and
> "multipart" is applied on the payload, therefore the above interpretation is wrong.
> If a response is 206 and the Content-Type is multipart/byteranges, the "normal"
> interpretation does not apply. Because "Content-Type: multipart/byteranges" is abused
> for this special case, the multipart/byteranges mime type must never be used in any
> other situations.

This is interesting.  What about for just a single range?

Can you please provide a reference to the "new bis draft"?

---

From: Roland Zink <[email-masked]>
Date: Wed, 26 Mar 2014 19:18:45 +0100

I like the idea of requiring transfer encoding instead of content 
encoding but I have concerns regarding replacing content-encoding by 
transfer encoding. When a 200 response is forwarded to a 1.1 client by 
an intermediary as content-encoding then the gzipped content is the 
same. However in HTTP/1.1 with content-encoding the gzipped content is a 
different entity than the not gzipped. If the client make further 
requests based on the content received then it will use the ETag from 
the server (for the uncompressed content) or one invented by the 
intermediary (for the compressed content). In both cases something may 
go wrong on further requests, for example a cache validation may fail. 
For a followup range request this would mean that the client does it on 
the compressed content and the server will assume the range is for the 
uncompressed content and the wrong range will be returned.

This gets especially complicated if it is assumed that the client is 
roaming between networks and cache partial responses.

Maybe something like this may work:

Clients MUST support gzip compression for HTTP response bodies. 
Regardless of the value of the TE request header field, a server MAY 
send responses with gzip transfer encoding. A compressed response MUST 
still bear an appropriate transfer-encoding header field. This 
effectively changes the implicit value of the TE header field 
([HTTP-p2], Section 14.39) from "chunked" to "gzip, chunked". Clients 
SHOULD not include the values "gzip" or "deflate" in a Accept-Encoding 
header. Intermediaries translating from HTTP/1.1 to HTTP/2 SHOULD 
forward the Accept-Encoding header. If there is an Accept-Encoding 
header containing gzip as value then servers MAY use gzip 
Content-Encoding instead of gzip Transfer-Encoding.

Mandating content-encoding gzip has the same issue as it is not mandated 
in HTTP/1.1 and a gateway can't really translate.

Regards,
Roland

On 25.03.2014 20:16, [email-masked] wrote:
> proposal is based on using gzip transfer encoding. We are proposing 
> this route because, as Matthew pointed out, it allows more freedom 
> than a flag which is forever locked to gzip.
>
> compatibility with the poor misused/abused Content-Encoding prevalent 
> in HTTP/1.1**
>
> Regardless of the value of the TE request header field, a server MAY 
> send responses with gzip transfer encoding. A compressed response MUST 
> still bear an appropriate transfer-encoding header field. This 
> effectively changes the implicit value of the TE header field 
> ([HTTP-p2], Section 14.39) from "chunked" to "gzip, chunked". Servers 
> SHOULD not use gzip transfer encoding if the requested content is 
> already compressed (e.g. images, videos, compressed files, etc.). 
> Content-Encoding header, regardless of whether the requested content 
> is already compressed, but SHOULD include the appropriate mime type in 
> the Content-Type header. Correspondingly, clients SHOULD not include 
> the values "gzip" or "deflate" in a Accept-Encoding header.
>
> that includes the value "gzip", the intermediary MUST decompress the 
> payload,
>
> intermediary elects to keep the payload compressed, MUST remove the 
> value "gzip" from the Transfer-Encoding header and insert the header 
> "Content-Encoding: gzip" in order to maintain backwards compatibility 
> with HTTP/1.1 clients,
>
The vary header should be set to include accept-encoding. What to do 
with a potential ETag?
>
> 3) if the request includes a TE header that includes the value "gzip", 
> the intermediary SHOULD not decompress the payload.
>
> compression methods for transfer encoding, in which case it MUST be 
> explicitly requested by the client in the TE header.
>
> disclosure. If you are not the intended recipient, please do not read, 
> copy, use or disclose this communication to others. Also please notify 
> the sender by replying to this message and then delete it from your 
> system.
>

---

From: Zhong Yu <[email-masked]>
Date: Wed, 26 Mar 2014 14:23:31 -0500

> Can you please provide a reference to the "new bis draft"?

http://tools.ietf.org/wg/httpbis/
http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26
http://tools.ietf.org/html/draft-ietf-httpbis-p5-range-26#section-4.1

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 27 Mar 2014 16:01:27 +1000

> of the content is known, attempting to fix it up later is a kludge.

This is true. An origin is free to choose, using whatever prior knowledge
and magic it has at its disposal, whether it's appropriate to represent a
resource with various content-encoded entities, or to serve up the same
entity using transfer encodings.

Similarly, an intermediate device can choose, maybe based on the received
Content-Type and Content-Encoding, or on the nature of the data itself, or
even on what transfer encoding the upstream peer chose, whether it's
appropriate to apply transfer encoding downstream. The knowledge about
semantics and compressibility may be less than at the origin, but that's
what you've got. You don't *have* to do anything, after all.

It might be hard for a proxy to decide whether or not to apply compression,
but that's not really relevant to the decision on whether or not to allow
transfer encoding in the protocol.

Incidentally the moment the intermediate device decides to modify the
content-encoding in any way, that device becomes the origin for the
resulting new entity, without becoming the authority for the resource. This
is a bad thing, and potentially breaks the protocol. Thus if a proxy wants
to have any say at all in the compression of responses, it has to be done
through transfer encoding.

---

From: <[email-masked]>
Date: Thu, 27 Mar 2014 21:03:58 +0000

On 26 March 2014 19:18, Roland Zinks wrote:
> I like the idea of requiring transfer encoding instead of content encoding but I have concerns regarding
> replacing content-encoding by transfer encoding...

Understood.

> Mandating content-encoding gzip has the same issue as it is not mandated in HTTP/1.1 and a gateway
> can't really translate.

Not sure what you mean exactly.  Isn't that how Section 9.3 of the current draft is written (i.e. content-encoding: gzip is explicitly supported on every request)?
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Roland Zink <[email-masked]>
Date: Fri, 28 Mar 2014 10:09:54 +0100

> Not sure what you mean exactly.  Isn't that how Section 9.3 of the current draft is written (i.e. content-encoding: gzip is explicitly supported on every request)?
Yes it is written this way and this is a problem for a gateway from my 
point of view.

---

From: <[email-masked]>
Date: Fri, 28 Mar 2014 09:30:10 +0000

On Friday,28 March 2014 10:10, Roland Zinks wrote:
>>> Mandating content-encoding gzip has the same issue as it is not
>>> mandated in HTTP/1.1 and a gateway can't really translate.
>Yes it is written this way and this is a problem for a gateway from my point of view.

It also gives the client no control.  Although unlikely, it seems a bit optimistic to assume that no security issues will be discovered down the road with compression of response bodies.  If such a security issue was discovered, clients would have to rely on servers to disable compression rather than being able to control it themselves.

What was the original motivation for mandating, implicitly, support for gzip content encoding?

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 31 Mar 2014 17:15:02 +1100

Recording this as:
  https://github.com/http2/http2-spec/issues/445

On 26 Mar 2014, at 6:54 am, Patrick McManus <[email-masked]> wrote:

> I'm unlikely to implement a gzip transfer encoding decoder any time soon. The distinction between TE and CE is too confusing and I think we're better off without the footgun and just using a single defacto CE mechanism instead (with all its admitted warts). The mandatory implied Accept-Encoding codifies a well worn and useful optimization.. Transfer-Encoding doesn't have the same track record to justify adding it at the same level.
> 
> I'm neutral on adjusting the spec to allow explicit gzip in TE and Transfer-Encoding negotiated between consenting peers.
> 
> -P
> 
> 
> 
> Based on the feedback from the group thus far we have written up a proposed replacement for section “9.3 GZip Content-Encoding”. Our proposal is based on using gzip transfer encoding. We are proposing this route because, as Matthew pointed out, it allows more freedom than a flag which is forever locked to gzip.
> 
>  
> Here is our proposal…