Subject: Re: How to handle content-encoding

From: Daurnimator <[email-masked]>
Date: Wed, 7 Dec 2016 23:34:02 +1100

On 31 May 2016 at 12:47, Daurnimator <[email-masked]> wrote:
> I'm thinking through how to add support for Content-Encoding to lua-http
> https://github.com/daurnimator/lua-http/issues/22
>
> A brief digression to lua-http structure (library terminology is borrowed
> from http2):
>   - a 'connection' encapsulates a socket, a connection has many streams
>   - a 'stream' is a request/response pair (a request can have multiple
> header blocks, and many data chunks)
>       - The same stream structure is used for both client and server
>       - You can implement a HTTP proxy by forwarding items from one stream
> to another
>   - a 'request' is a pre-prepared object consisting of a request header
> block, a function to obtain body chunks, and a destination.
>       - `request:go()` returns the 'main' response header block and a stream
> (from which you can read the body one chunk at a time)
>
> There is a desire to compress content to save bandwidth, HTTP has had two
> main ways to do this: Transfer-Encoding and Content-Encoding.
>
> To me it was simple to add support for Transfer-Encoding, without any
> ambiguities or issues. For HTTP1 in the stream logic:
>   -  (if zlib is installed) we automatically add `TE: gzip, deflate`.
>   - On reply, if Transfer-Encoding contains gzip or deflate, we decode it
> before passing it onto the caller.
> This is permitted as TE and Transfer-Encoding are hop-by-hop headers.
>
> However, HTTP2 does not support transfer-encoding.
> Furthermore, certain servers **stares at twitter.com** send
> `Content-Encoding: gzip` even if you *don't* send `Accept-Encoding: gzip`
> This seems to demand that I support Content-Encoding.
>
> As far as the specifications go, Content-Encoding is *meant* to be used to
> for end-to-end encoding that intermediate hops do not touch.
>   - Intermediaries should cache Content-Encoded bodies in their encoded form
>   - ETag is dependant on Content-Encoding
>
> This makes it hard to find a place for it in lua-http's structure.
> If I add it transparently in the stream (as done for Transfer-Encoding) then
> it will be hop-by-hop (not end-to-end)
> This seems to demand (at least for client requests) that it is switched
> on/off at the request layer.
> From there though, it seems it would need to add some sort of stream body
> filter?
>
> How should I be adding this? What have other implementations done? (and what
> do they wish they'd done differently?)
> The current state seems to be *against* the spec: should the spec be
> changed? should implementations be updated?
> HTTP2 has no transfer-encoding equivalent... why not?
>
> Regards,
> Daurn.
>
>
> Links:
>   - https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11
> Original content-encoding spec
>   - https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.5.1
> Hop-by-hop headers
>   - https://tools.ietf.org/html/rfc7231#section-3.1.2.1 Current spec
>   - https://bugzilla.mozilla.org/show_bug.cgi?id=68517 Mozilla disregards
> Content-Encoding spec
>   -
> https://stackoverflow.com/questions/11641923/transfer-encoding-gzip-vs-content-encoding-gzip
>   - https://daurnimator.github.io/lua-http/ lua-http documentation

*bump*.

Didn't get any replies :(

---

From: Cory Benfield <[email-masked]>
Date: Thu, 8 Dec 2016 09:49:00 +0000

> On 7 Dec 2016, at 12:34, Daurnimator <[email-masked]> wrote:
> 

In my experience, the gzip and deflate options for Transfer-Encoding are essentially unsupported. RFC 7230 doesn’t provide any requirement to support them, and so they are almost always not used.

>> This seems to demand that I support Content-Encoding.

Are you sending Accept-Encoding: identity? RFC 7231 Section 5.3.4 says the following things about Accept-Encoding:

1. If the request contains no Accept-Encoding header then any content-encoding is considered acceptable.
2. If the request does contain an accept-encoding header, but the server cannot satisfy it (for example, the server may not have access to an uncompressed resource), then the server may send a content-coding anyway (this is because the server SHOULD send a response without any content-coding, but is not required to)

>> HTTP2 has no transfer-encoding equivalent... why not?

Lets tackle these questions in order.

1. My implementations support only Content-Encoding. My clients support all three major compressed content-encodings. This has more or less been just fine: almost all servers also only support Content-Encoding. Generally speaking, servers that support content-encoding for static content do so not by compressing on the fly but by having compressed and uncompressed versions of each resource on disk. For dynamic responses, compression is frequently handled by middlewares that apply the content-encoding on-the-fly. This is generally not an issue: the ETag can be calculated afterwards if that’s required.

2. I don’t see that there’s any spec violations here. Can you clarify about where you believe the spec violation is occurring?

3. HTTP/2 deliberately removed all support for hop-by-hop headers of this form. For more discussion on this specific issue, see this thread from 2014: https://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1179.html <https://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1179.html>. This is a substantial thread with a number of links out of it to other discussions. The major counter-arguments were that Transfer-Encoding: gzip required logic to detect already-compressed content to avoid double-compression (that is, to avoid compressing resources that were already compressed) and that it was not widely deployed.

Some further attempts have been made to redefine Transfer-Encoding: gzip using HTTP/2 frames, but again there has been relatively lukewarm interest from implementers to support that draft, so it has stalled.

Essentially, it seems that the bulk of the HTTP/2 community isn’t interested in Transfer-Encoding: gzip: they’ve concluded, rightly or wrongly, that Content-Encoding: gzip is fine.

Cory