Subject: Support for gzip at the server #424

From: Martin Thomson <[email-masked]>
Date: Mon, 10 Mar 2014 09:59:07 +0100

Discussion offline leads me to conclude that doing this would be bad idea.

The basic problem is with 2->1.1 translation at intermediaries.  While
the HTTP/2 request might include a Content-Length, the gateway is
going to be forced to decompress and buffer the entire request body
before forwarding to a 1.1 server.

This doesn't seem like a good outcome.  Maybe this is something to
defer to the version of HTTP/x that ships when most of the world is
already using HTTP/2.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 10 Mar 2014 10:14:06 +0100

> before forwarding to a 1.1 server.

I'm not sure it is forced to. We could make it required for origin 
servers, and let intermediaries forward as-is.

In that case clients would still have to handle errors, but at least the 
chances of getting server support would be greater.

> defer to the version of HTTP/x that ships when most of the world is

In that case the same problem would come up with HTTP/3->HTTP/2 
translation...

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Mon, 10 Mar 2014 10:20:42 +0100

> In that case the same problem would come up with HTTP/3->HTTP/2

I don't think that this is quite the same.  HTTP/2 doesn't make
Content-Length mandatory, so the intermediary wouldn't be required to
know Content-Length before decompressing toward an HTTP/2 next-hop
server.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 14 Mar 2014 01:56:58 +0100

>defer to the version of HTTP/x that ships when most of the world is

I am not really following what this thread is about. HTTP/1.1 does not
(de jure) need Content-Length anywhere, requests and responses can both
be compressed and chunked. Why does that not matter here?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Roberto Peon <[email-masked]>
Date: Thu, 13 Mar 2014 18:24:22 -0700

Decompress, yes, buffer the entire body? No. Gzip can happily be
stream-decompressed.

-=R

---

From: Roberto Peon <[email-masked]>
Date: Thu, 13 Mar 2014 18:26:12 -0700

Ah I missed part of the original email-- yes, the decompressed content
length needs to be known.
That can be easily solved, though.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 14 Mar 2014 02:35:45 +0100

>That can be easily solved, though.

(One thing to note is that gzip (as opposed to raw deflate and zlib)
includes the uncompressed size in the header if it fits into 32 bit).
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 14 Mar 2014 02:38:31 +0100

>includes the uncompressed size in the header if it fits into 32 bit).

(Strike that.)

---

From: Martin Thomson <[email-masked]>
Date: Thu, 13 Mar 2014 18:43:37 -0700

> includes the uncompressed size in the header if it fits into 32 bit).

It bothers me a little that this is modulo 2^32.  Maybe we can allow
this for bodies < 2^32.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 13 Mar 2014 18:45:04 -0700

>>includes the uncompressed size in the header if it fits into 32 bit).

Oh, now you have me confused, RFC 1952 says:
         A compliant compressor must produce files with correct ID1,
         ID2, CM, CRC32, and ISIZE, [...]

Is that not correct/sufficient for files < 2^32 in length?

---

From: Roland Zink <[email-masked]>
Date: Fri, 14 Mar 2014 10:20:13 +0100

ISIZE is in the footer and not the header.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 14 Mar 2014 09:38:53 -0700

> ISIZE is in the footer and not the header.

So that leads me back to the original conclusion.  Since
intermediation from 2 to 1.1 will require the Content-Length and
extracting that from a gzip'd body would require buffering an entire
request, I'm inclined to say that this is too hard.

Does anyone disagree?

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 14 Mar 2014 17:52:57 +0100

>request, I'm inclined to say that this is too hard.

It would help if you explain why you think Content-Length is needed in
this scenario. `Transfer-Encoding: chunked` is supported by servers and
intermediaries. Likely not perfectly, but if that is the concern we
should make that very explicit.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Patrick McManus <[email-masked]>
Date: Fri, 14 Mar 2014 18:06:19 +0100

receiving chunked requests is not supported with a high enough certainty
that anyone yet wants to send them in a generic web context. (apps that
know their server's capabilities apriori are a different story). HTTP/2
negotiation at first seems to make that easier, but supporting easy
gatewaying back to http/1.0 makes it hard again :(

as much as I would really like this feature, I think its reasonable not to
include it.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 14 Mar 2014 10:42:36 -0700

Implementation experience here says that many servers barf if they don't
get content-length for an upload.

imho, the simple solution is to mandate the presence of a header that
indicates the uncompressed content-length when sending compressed data on
http/2. This is generically useful in many applications in both directions.
-=R

On Fri, Mar 14, 2014 at 10:06 AM, Patrick McManus <[email-masked]>wrote:

> On Fri, Mar 14, 2014 at 5:52 PM, Bjoern Hoehrmann <[email-masked]>wrote:
>
>> * Martin Thomson wrote:

---

From: Martin Thomson <[email-masked]>
Date: Fri, 14 Mar 2014 10:54:10 -0700

> http/2. This is generically useful in many applications in both directions.

Not that it wouldn't work, but that sounds a little too much like a
new feature to me.

---

From: Roland Zink <[email-masked]>
Date: Fri, 14 Mar 2014 18:59:35 +0100

I think this can't be mandated as sometimes the content-length may not 
be known in advance, but you can make it optional.

Roland

On 14.03.2014 18:42, Roberto Peon wrote:
> Implementation experience here says that many servers barf if they 
> don't get content-length for an upload.
>
> indicates the uncompressed content-length when sending compressed data 
> on http/2. This is generically useful in many applications in both 
> directions.
> -=R
>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 14 Mar 2014 11:07:21 -0700

On 14 March 2014 10:59, Roland Zink <[email-masked]> wrote:
> I think this can't be mandated as sometimes the content-length may not be
> known in advance, but you can make it optional.

If we were to allow this, then the sender would be required to provide
this new "uncompressed-length" header field.  As Patrick pointed out,
the alternative is breaking some servers.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 14 Mar 2014 11:21:55 -0700

Compressed uploads isn't a new feature, but it has poor interop.
This would fix the interop problem.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 14 Mar 2014 11:30:31 -0700

> This would fix the interop problem.

This is true, but I'm trying my best to ensure that we don't expand
the scope of HTTP/2.  It's one thing to fix the problem, but it's
another thing to tie the fate of HTTP/2 to the fix.

Maybe we can say MUST support gzip, in combination with a MUST NOT use
it + unless you know it's HTTP/2 to the origin server OR you can
somehow signal uncompressed content-length, using some
as-yet-unspecified method.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 14 Mar 2014 16:49:32 -0400

Roberto,

That doesn't work in a lot of situations.  Particularly with IPP, we are often dealing with gigabytes of print data and buffering that on the client prior to sending it is generally not feasible and causes a poor user experience (mainly delayed printing...)

On Mar 14, 2014, at 1:42 PM, Roberto Peon <[email-masked]> wrote:

> Implementation experience here says that many servers barf if they don't get content-length for an upload.
> 
> imho, the simple solution is to mandate the presence of a header that indicates the uncompressed content-length when sending compressed data on http/2. This is generically useful in many applications in both directions.
> -=R
> 
> 
> On Fri, Mar 14, 2014 at 10:06 AM, Patrick McManus <[email-masked]> wrote:
> receiving chunked requests is not supported with a high enough certainty that anyone yet wants to send them in a generic web context. (apps that know their server's capabilities apriori are a different story). HTTP/2 negotiation at first seems to make that easier, but supporting easy gatewaying back to http/1.0 makes it hard again :(
> 
> as much as I would really like this feature, I think its reasonable not to include it.
> 
> 

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Zhong Yu <[email-masked]>
Date: Fri, 14 Mar 2014 15:56:00 -0500

> Not that it wouldn't work, but that sounds a little too much like a

Request body compression is practically a new feature anyway - few
clients do it today. A new header is relatively a trivial addition to
the effort.

---

From: Zhong Yu <[email-masked]>
Date: Fri, 14 Mar 2014 15:58:10 -0500

> This would fix the interop problem.

RFC2616: All HTTP/1.1 applications MUST be able to receive and decode
the "chunked" transfer-coding

I was fooled by the stern language and implemented it in my server -
apparently for nothing :(

>> > on

---

From: Roberto Peon <[email-masked]>
Date: Fri, 14 Mar 2014 14:08:31 -0700

The sender doesn't know the uncompressed size before it transmits?

That seems.. odd...

Right now we have a world where nobody can *ever* use one of the features
of the protocol (compression in the upload path) because of a lack of an
interop bridge between the two worlds.
Given how cheap it is to decompress gzip, this seems strange.
-=R

On Fri, Mar 14, 2014 at 1:49 PM, Michael Sweet <[email-masked]> wrote:

> Roberto,
>
> That doesn't work in a lot of situations.  Particularly with IPP, we are
> often dealing with gigabytes of print data and buffering that on the client
> prior to sending it is generally not feasible and causes a poor user
> experience (mainly delayed printing...)
>
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 14 Mar 2014 22:09:22 +0100

> apparently for nothing :(

I've worked on clients that assume that gzip/chunked works 
client->server. It's easy enough to fallback and retry without if the 
former fails.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 14 Mar 2014 22:11:11 +0100

> gatewaying back to http/1.0 makes it hard again :(

Is gatewaying to *1.0* a goal at all?

If HTTP/2 is not the right moment to get over 1.0 limitations, when is 
it then?

> ...

Best regards, Julian

---

From: Michael Sweet <[email-masked]>
Date: Fri, 14 Mar 2014 17:15:31 -0400

Roberto,

> The sender doesn't know the uncompressed size before it transmits?

No, it doesn't.

> That seems.. odd...

The client is generally rasterizing pages of content for the printer at some agreed upon resolution, bit depth, and color space.  This raster data is typically already compressed with a simple algorithm such as PackBits (run-length encoding) and is thus already variable-length per page with no way to know ahead of time how large it will be.  Add gzip to the mix and you *really* don't know what the final length will be.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 14 Mar 2014 14:20:34 -0700

In your use-case I suppose you get away with not sending content-length at
all.
>From my perspective, that seems perfectly reasonable: If the content-length
is known a priori, then it should be signaled. If it isn't, then don't.

In an ideal world this content-length signaling wouldn't be necessary at
all, but there are a vast majority of servers that simply do not handle
*any* chunked upload without content-length since their universe has
consisted entirely of a priori known size for the uploaded entities.

-=R

On Fri, Mar 14, 2014 at 2:15 PM, Michael Sweet <[email-masked]> wrote:

> The client is generally rasterizing pages of content for the printer at
> some agreed upon resolution, bit depth, and color space.  This raster data
> is typically already compressed with a simple algorithm such as PackBits
> (run-length encoding) and is thus already variable-length per page with no
> way to know ahead of time how large it will be.  Add gzip to the mix and
> you *really* don't know what the final length will be.
>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 14 Mar 2014 14:21:16 -0700

On 14 March 2014 14:15, Michael Sweet <[email-masked]> wrote:
> The client is generally rasterizing pages of content for the printer at some
> agreed upon resolution, bit depth, and color space.  This raster data is
> typically already compressed with a simple algorithm such as PackBits
> way to know ahead of time how large it will be.  Add gzip to the mix and you
> *really* don't know what the final length will be.

This seems like a case of "I know the server capabilities well enough
to do this".  I'm not sure that we could safely do the same thing for
every HTTP client in existence.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 14 Mar 2014 21:28:40 +0000

>The sender doesn't know the uncompressed size before it transmits?

One common Varnish scenario:  ESI composition.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 14 Mar 2014 17:30:31 -0400

Martin,

I just pointed this use case out as an argument *against* requiring a new header providing the uncompressed length. HTTP/2.0 *does* effectively require chunked message body support, just like HTTP/1.1, so any interoperability issues need to be fixed by the implementors and we should not require a client or a server to provide a content length, period.  It is useful information if provided, but the onus is on the receiver to deal with content that exceeds its capabilities or configured limits, not on the protocol/transport providing it.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 14 Mar 2014 14:49:05 -0700

On 14 March 2014 14:30, Michael Sweet <[email-masked]> wrote:
> I just pointed this use case out as an argument *against* requiring a new header providing the uncompressed length.

I understand.  I don't think that we could realistically place a
blanket requirement on this.  It would be more a case of, if you don't
know who you are sending to, then it might be best to use this
mechanism: <insert new header field>.  The fact that people avoid the
issues we're talking about sometimes is great, but we still have to be
able to make recommendations about what is safe to do.

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 15 Mar 2014 10:50:19 +1300

> apparently for nothing :(

That is receive not send.
The biggest problem here is HTTP/1.0 clients not being able to receive
chunked. So it cannot be relied on by any 1.x clients on first request
(which PUT/POST usually have to be).

The only way to get reliable transfer other tan with content-length is
to send Expect:100-continue and get a working result to that.

AYJ

---

From: William Chan (陈智昌) <[email-masked]>
Date: Fri, 14 Mar 2014 15:04:21 -0700

It's unfortunate, but I agree that given the interop issues, I don't think
we can require this.

I haven't thought about it deeply, but perhaps we can opt-in using the
SETTINGS and HTTP2-Settings headers. Explicit server declaration of
support. I haven't gamed it out completely in my head yet as to whether or
not that would create other obstacles and whether or not it's better than
application specific knowledge (e.g. JS shipped from server to client).

On Fri, Mar 14, 2014 at 2:21 PM, Martin Thomson <[email-masked]>wrote:

> some
> you

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 15 Mar 2014 11:08:04 +1300

> If HTTP/2 is not the right moment to get over 1.0 limitations, when is

Good point.

Any of the client UA people here have stats on how widespread 1.0
servers and proxies still are?

Amos

---

From: Martin Thomson <[email-masked]>
Date: Fri, 14 Mar 2014 15:09:19 -0700

On 14 March 2014 15:04, William Chan (陈智昌) <[email-masked]> wrote:
> SETTINGS and HTTP2-Settings headers. Explicit server declaration of support.
> I haven't gamed it out completely in my head yet as to whether or not that
> would create other obstacles and whether or not it's better than application
> specific knowledge (e.g. JS shipped from server to client).

That would require a guarantee from the server peer that it is
handling requests itself, or can be certain that it only forwards to
servers that have made the same promise.  That's possible, I guess.

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 15 Mar 2014 11:10:36 +1300

On 15/03/2014 10:59 a.m., William Chan (陈智昌) wrote:
> On Fri, Mar 14, 2014 at 2:50 PM, Amos Jeffries wrote:
> 
>>> On Fri, Mar 14, 2014 at 1:21 PM, Roberto Peon wrote:
> Er, did you mean HTTP/1.0 server?
> 

Oops. Yes.

AYJ

---

From: William Chan (陈智昌) <[email-masked]>
Date: Fri, 14 Mar 2014 15:13:11 -0700

> servers and proxies still are?

I don't know any stats offhand, but I believe it's high enough that the
option of breaking compat with 1.0 is not even on the table from my
standpoint. I definitely see bug reports for it.

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 15 Mar 2014 11:14:41 +1300

> application specific knowledge (e.g. JS shipped from server to client).

The issue is mainly (only?) in 2.0->1.0 gateways, and only in that one
direction. Solving it in the 2.0<->2.0 or 1.x->2.0 cases does not seem
to be any benefit.

Amos

---

From: William Chan (陈智昌) <[email-masked]>
Date: Fri, 14 Mar 2014 15:22:01 -0700

Correct. But IIUC, the problem is the client doesn't know if there's a 1.0
intermediary or backend in the path, or if the 2.0 gateway is willing to
buffer (sounds crazy, but just for completeness I mentioned it). So the
client can't assume gzip support at the peer, even though we suspect most
would support it. Since we can't assume by default, I was thinking maybe we
can assume by having the server/proxy explicitly declare support.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 14 Mar 2014 23:33:54 +0100

On 2014-03-14 23:04, William Chan (陈智昌) wrote:
> It's unfortunate, but I agree that given the interop issues, I don't
> think we can require this.

HTTP/1.1 already REQUIRES the ability to process chunked encoding. What 
we're talking about is adding gzip. Streaming/buffering shouldn't be any 
more issue than it already is in 1.1.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 14 Mar 2014 23:36:05 +0100

> can assume by having the server/proxy explicitly declare support.

Another point is that I'm suspecting that gunzip on 2.0->1.x gateways will
be disabled because of seucrity issues. It's so much easy to exploit random
decompressors for size amplification that I'm having a hard time imagining
that this will remain enabled for a long time :-/

With an amplification ratio of 1000, you only need 1 Mbps upload to saturate
the Gig link of a gateway...

Willy

---

From: Roberto Peon <[email-masked]>
Date: Fri, 14 Mar 2014 15:45:26 -0700

Correct.

As far as I'm concerned, adding the additional header with the uncompressed
content-length when content-length would otherwise have been added makes
this interop just fine:
In cases where the uncompressed content-length is unknowable, nothing
changes in terms of interop.
In cases where the uncompressed content-length is knowable, nothing changes
in terms of interop.

-=R

---

From: Roland Zink <[email-masked]>
Date: Sat, 15 Mar 2014 22:25:48 +0100

In principle this is true. If compression is loosing the content-length 
then this may amplify the existing issue. Transferring entity 
information like content-length (if it is known) in addition to 
transport properties (like the compressed content-length) may help, but 
may cause issues if the information is inconsistent / wrong.

Roland

On 14.03.2014 23:33, Julian Reschke wrote:
> HTTP/1.1 already REQUIRES the ability to process chunked encoding. 
> What we're talking about is adding gzip. Streaming/buffering shouldn't 
> be any more issue than it already is in 1.1.
>
> > ...
>
> Best regards, Julian
>

---

From: James Cloos <[email-masked]>
Date: Sat, 15 Mar 2014 22:29:48 -0400

It seems from all of this that the /real/ bugs are using the gzip CE for
anything which is not already gzip(1)-compressed on disk, and http/2's
flawed idea of making any non-transparent CE implicit.

Changing how CE works clearly breaks things and lacks any real benefits.

-JimC
--
James Cloos <[email-masked]>         OpenPGP: 1024D/ED7DAEA6

---

From: William Chan (陈智昌) <[email-masked]>
Date: Sat, 15 Mar 2014 21:32:21 -0700

I must be missing something. There's no implicit content encoding, is
there? I thought it was implicit accept encoding.

---

From: Ilari Liusvaara <[email-masked]>
Date: Sun, 16 Mar 2014 08:35:27 +0200

> there? I thought it was implicit accept encoding.

Correct. The spec clearly says that if implicit A-E gzip is used, it MUST
be properly marked via C-E (and the same goes for any explicit A-E entries
server uses).

Quoting latest git master of the spec:
> "A compressed response MUST still bear an appropriate content-encoding
> header field."

-Ilari

---

From: James Cloos <[email-masked]>
Date: Sun, 16 Mar 2014 18:21:18 -0400

>>>>> "WC" == William Chan (陈智昌) <[email-masked]> writes:

WC> I must be missing something. There's no implicit content encoding, is
WC> there? I thought it was implicit accept encoding.

I was tired.  I of course meant to write implicit AE where I wrote implicit CE.

Thanks for noticing the typo/thinko.

-JimC

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 17 Mar 2014 08:59:30 +0100

> be compressed and chunked. Why does that not matter here?

Content-Length can be used by intermediaries to dispatch big downloads
(isos, etc) to specific anti-malware scanner where they won't block
processing of interactive web browsing

Regards,

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 17 Mar 2014 09:20:53 +0100

Le Ven 14 mars 2014 18:06, Patrick McManus a écrit :
> that anyone yet wants to send them in a generic web context.

Also chunked is useless in an anti malware context. You need to reassemble
the chunks at the intermediary level because typical av only works on full
files

Regards,

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 17 Mar 2014 09:23:09 +0100

> be known in advance, but you can make it optional.

Then for reliability reasons you need to be able to send a "more than foo
size" signal with foo big enough to make sense for most content-length
users (ie notzero at least)

Regards,

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 17 Mar 2014 09:28:53 +0100

> that this will remain enabled for a long time :-/

I doubt it. If http/2 is gzip all security gateways will run gunzip by
default to scan for malware. So I can't fathom why they would disable it
for the 2.0→1.x case

Regards,

-- 
Nicolas Mailhot

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 17 Mar 2014 21:45:16 +1300

> the chunks at the intermediary level because typical av only works on full

Chunked is hop-by-hop anyway so in theory it has to be reassembled and
re-chunked regardless of the scanning.

Also, some AV are already on the way towards being stream processors due
to ICAP. Not there yet to my knowledge but getting somewhere close and
this would be another incentive to correct that file fixation.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 17 Mar 2014 09:49:16 +0100

On Mon, Mar 17, 2014 at 09:28:53AM +0100, Nicolas Mailhot wrote:
> for the 2.0???1.x case

We're really not speaking about the same gateways. You're speaking about
client-side proxies, I'm speaking about server-side gateways and load
balancers. Also, the cost of gunzip vs malware analysis is minimal.
However, the cost of gunzip vs forwarding is huge, especially when
it involves using the gateway as an amplifier.

Willy

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 17 Mar 2014 14:08:49 +0100

> this would be another incentive to correct that file fixation.

I would love to hear about those unicorns if they ever materialize. I sure
could use a lot of them. AV ICAP systems seem stuck in the past, ours
can't even process a basic eclipse download from three years ago right now
(too many levels of imbricated java archives, too many small files)

Regards,

-- 
Nicolas Mailhot

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Mar 2014 09:24:55 +1100

I'm beginning to think this issue might be in the "diminishing returns" bucket. 

Does anyone think that we firmly need this, or is it just a nice-to-have?

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: William Chan (陈智昌) <[email-masked]>
Date: Mon, 17 Mar 2014 15:32:09 -0700

Just to be clear, you only mean server support for gzip, not client
support, right? The client support is very helpful for performance.

As far as server support, I think it'd be great to have too, but the
interop problems are indeed problematic.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Mar 2014 09:32:53 +1100

On 18 Mar 2014, at 9:32 am, William Chan (陈智昌) <[email-masked]> wrote:

> Just to be clear, you only mean server support for gzip, not client support, right?

Yes - just this issue, not the client side.

> The client support is very helpful for performance.
> 
> As far as server support, I think it'd be great to have too, but the interop problems are indeed problematic.
> 
>

---

From: David Morris <[email-masked]>
Date: Mon, 17 Mar 2014 18:36:58 -0700 (PDT)

On Tue, 18 Mar 2014, Mark Nottingham wrote:

>
> On 18 Mar 2014, at 9:32 am, William Chan (???) <[email-masked]> wrote:
>

I think a client and server is always involved, so I think this issue is
client compression of request content and server handling that compressed
input.

I think that server originating gzip compressed content and the client
being required to handle it is NOT the issue being discussed here?

---

From: William Chan (陈智昌) <[email-masked]>
Date: Mon, 17 Mar 2014 18:41:10 -0700

Sorry, I was too terse. Yes, I agree with your interpretation.

On Mon, Mar 17, 2014 at 6:36 PM, David Morris <[email-masked]> wrote:

> support, right?

---

From: Martin Thomson <[email-masked]>
Date: Mon, 17 Mar 2014 20:43:55 -0700

> being required to handle it is NOT the issue being discussed here?

Correct on both aspects.  This is *request* compression.

I'll note that there is nothing stopping a client from trying and
maybe failing to compress, or for specific applications to require
support of compression, or to have client send content types that use
compression in preference to uncompressed content.  But requiring
support for a specific content-encoding is, as I think we've
concluded, perhaps a little too ambitious.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Mar 2014 14:50:45 +1100

It sounds like we have consensus to close #424 with no action. Anyone have a problem with that?

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 18 Mar 2014 18:53:54 +1300

> It sounds like we have consensus to close #424 with no action. Anyone have a problem with that?

Is it worth proposing gzip as default Transfer-Encoding instead so its
only done on DATA frames between hops known to be HTTP/2 at both ends?
(ie after SETTINGS already confirms or breaks the connection).

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Mar 2014 17:21:59 +1100

We don't have transfer-codings in HTTP/2.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 08:03:16 +0100

> It sounds like we have consensus to close #424 with no action. Anyone have a problem with that?

Yes.

I believe we're giving up too early.

HTTP/1.1 already requires servers to handle chunked encoding in requests:

   "All HTTP/1.1 applications MUST be able to receive and decode the 
"chunked" transfer-coding, and MUST ignore chunk-extension extensions 
they do not understand." (RFC 2616, 3.6)

So gatewaying to 1.1 shouldn't be a problem at all, as no buffering is 
needed.

If gatewaying to 1.0 servers is the problem than we seriously should 
consider giving up on *that* goal. Speaking of which - why *exactly* is 
gatewaying to 1.0 a problem?

Best regards, Julian

---

From: Michael Sweet <[email-masked]>
Date: Tue, 18 Mar 2014 12:07:49 -0400

Nope

---

From: Daniel Sommermann <[email-masked]>
Date: Tue, 18 Mar 2014 09:14:29 -0700

I believe the issue with gatewaying to 1.0 is that the gateway would 
have to buffer the entire body and then decompress (leading to memory 
DOS). Request bodies cannot be terminated by EOF, unlike repsonse 
bodies, so a streaming approach wouldn't be possible as it is for server 
-> client 1.0 gatewaying. Even if it were, it would force the gateway to 
close the server connection, which could be considered a type of DOS as 
well.

Although this idea is nice, I think we can pass for now.

On 03/18/2014 12:03 AM, Julian Reschke wrote:
> consider giving up on *that* goal. Speaking of which - why *exactly* 
> is gatewaying to 1.0 a problem?
>
> Best regards, Julian
>
>
>
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 18 Mar 2014 09:36:17 -0700

> (ie after SETTINGS already confirms or breaks the connection).

I don't think that requiring compression is a good idea.  I can't
think of a security issue right now, but I don't want to force that on
people.

(...assuming you mean to say Content-Encoding).

---

From: Roberto Peon <[email-masked]>
Date: Tue, 18 Mar 2014 10:04:16 -0700

Note that this is true regardless of whether or not compression is in use--
anything doing chunked entity-body suffers from this when gatewaying to a
1.0 server.
-=R

On Tue, Mar 18, 2014 at 9:14 AM, Daniel Sommermann <[email-masked]> wrote:

> I believe the issue with gatewaying to 1.0 is that the gateway would have
> to buffer the entire body and then decompress (leading to memory DOS).
> Request bodies cannot be terminated by EOF, unlike repsonse bodies, so a
> streaming approach wouldn't be possible as it is for server -> client 1.0
> gatewaying. Even if it were, it would force the gateway to close the server
> connection, which could be considered a type of DOS as well.
>
>> "chunked" transfer-coding, and MUST ignore chunk-extension extensions they
>> do not understand." (RFC 2616, 3.6)
>>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 18:33:46 +0100

On 2014-03-18 18:04, Roberto Peon wrote:
> Note that this is true regardless of whether or not compression is in
> use-- anything doing chunked entity-body suffers from this when
> gatewaying to a 1.0 server.
> -=R
> ...

Ok. So support for chunked already is mandatory -- why can't we add gzip?

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 18 Mar 2014 19:13:16 +0100

>Ok. So support for chunked already is mandatory -- why can't we add gzip?

What I understand so far is that flawlessly implemented support for
chunked requests is not universally deployed, and buffering requests
in intermediaries is not a viable option in any and all cases without
exception. What is mandated is not relevant to those concerns.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 19:25:07 +0100

> exception. What is mandated is not relevant to those concerns.

I understand the concerns.

However, just because there are concerns doesn't mean we can't mandate 
it. If we all agree that client->server gzip is desirable, then the spec 
should be clear about that. Be it in non-normative prose, with a 
"SHOULD" or even a "MUST" (which would be similar to the "MUST support 
chunked encoding" in 1.1).

What I hear is that just because there are concerns we'll say nothing. 
My take is that the situation will not improve unless we say *something*.

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 18 Mar 2014 19:31:30 +0100

>My take is that the situation will not improve unless we say *something*.

That sounds pretty reasonable to me.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Martin Thomson <[email-masked]>
Date: Tue, 18 Mar 2014 11:36:50 -0700

On 18 March 2014 11:25, Julian Reschke <[email-masked]> wrote:
> However, just because there are concerns doesn't mean we can't mandate it.
> If we all agree that client->server gzip is desirable, then the spec should
> be clear about that. Be it in non-normative prose, with a "SHOULD" or even a
> "MUST" (which would be similar to the "MUST support chunked encoding" in
> 1.1).

A la, RFC 6119?

---

From: Martin Thomson <[email-masked]>
Date: Tue, 18 Mar 2014 12:03:45 -0700

>> "MUST" (which would be similar to the "MUST support chunked encoding" in

Sorry, 6919.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 20:43:56 +0100

>>> "MUST" (which would be similar to the "MUST support chunked encoding" in

No, 2119.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 18 Mar 2014 14:01:03 -0700

It might help to understand that chunked requests are universally
supported by servers but not universally supported by resources.

In particular, it is impossible to implement unlimited chunked
request bodies via a CGI script because that gateway protocol
requires the content-length be set in the environment variable
before the command is invoked. I think the same limitation is in
Servlets, because it was just a copy of CGI. In contrast, it is
trivial to implement request streaming with an Apache module, since
the server's protocol filter handles the chunks automatically
if told to do so.  It is also easy to implement limited request
buffers for a legacy back-end, configurable on a per-resource basis,
provided that sufficient protections against denial-of-sevice attacks
are in place.

It would be a terrible mistake to limit HTTP/2 to the worst
of old implementations.  That is the opposite of HTTP's design
for flexible extensibility.  There are hundreds (if not thousands) of
implementations of HTTP/1.1 that have no problem whatsoever with
compression, chunked encoding, or any of the other features of HTTP.
That is because the people installing them control the network in which
those features are enabled, and can remove any products that get them
wrong. HTTP/2 should focus on making features self-descriptive,
rather than inventing limitations on use.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 22:16:18 +0100

> Servlets, because it was just a copy of CGI. In contrast, it is

Nope. There's no problem with servlets with respect to this. You just 
obtain the ServletInputStream and start reading.

> of old implementations.  That is the opposite of HTTP's design

+1

> compression, chunked encoding, or any of the other features of HTTP.

Indeed.

Best regards. Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 19 Mar 2014 12:48:00 +1100

> rather than inventing limitations on use.

I don't think anyone is talking about *limiting* what you can do in HTTP/2 here -- what's being discussed is whether server-side support for GZIP content-coding in requests should be *required*.

Everyone should keep in mind that the only way we can make such a restriction stick in all cases is to disallow 411 (Length Required) in HTTP/2 -- which I personally think is unrealistic in the extreme, considering how common it is. 

Finally, one point of data (based upon 5 minutes of testing) - Squid 2.HEAD (in 1.1 mode) will generate 411 for POST without a Content-Length automatically. It's still one of the most widely deployed intermediaries out there, IME.

Cheers,

---

From: Roberto Peon <[email-masked]>
Date: Tue, 18 Mar 2014 18:57:22 -0700

For my part I'm aiming for having gateways interop at least as well as they
do today.
That doesn't require us to disallow anything, but it would require putting
the uncompressed body-size in there when it is known.
That isn't the same as requiring uncompressed body-size.
-=R

On Tue, Mar 18, 2014 at 6:48 PM, Mark Nottingham <[email-masked]> wrote:

> I don't think anyone is talking about *limiting* what you can do in HTTP/2
> here -- what's being discussed is whether server-side support for GZIP
> content-coding in requests should be *required*.
>
> Everyone should keep in mind that the only way we can make such a
> restriction stick in all cases is to disallow 411 (Length Required) in
> HTTP/2 -- which I personally think is unrealistic in the extreme,
> considering how common it is.
>
> Finally, one point of data (based upon 5 minutes of testing) - Squid
> 2.HEAD (in 1.1 mode) will generate 411 for POST without a Content-Length
> automatically. It's still one of the most widely deployed intermediaries
> out there, IME.
>
> Cheers,
>
> --

---

From: Julian Reschke <[email-masked]>
Date: Wed, 19 Mar 2014 12:33:12 +0100

> I don't think anyone is talking about *limiting* what you can do in HTTP/2 here -- what's being discussed is whether server-side support for GZIP content-coding in requests should be *required*.

I think it would be good if we (a) encouraged servers to do it, and (b) 
clarified error handling if you don't.

1) Define a status code for "unsupported content-encoding" (plus maybe 
discovery via a Accept-Encoding header field that can be sent with it)

2) Require servers to either support gzip content encoding or fail the 
request with the new status code.

> Everyone should keep in mind that the only way we can make such a restriction stick in all cases is to disallow 411 (Length Required) in HTTP/2 -- which I personally think is unrealistic in the ext…

So in 1.1, the description of 411 should have stated that this indicates 
a violation of a 1.1 requirement, fwiw.

> Finally, one point of data (based upon 5 minutes of testing) - Squid 2.HEAD (in 1.1 mode) will generate 411 for POST without a Content-Length automatically. It's still one of the most widely deploy…

Is there a bug report for that yet?

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 19 Mar 2014 13:25:16 +0100

> clarified error handling if you don't.

+1 for this.

Willy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Mar 2014 10:58:02 +1100

On 19 Mar 2014, at 10:33 pm, Julian Reschke <[email-masked]> wrote:

> I think it would be good if we (a) encouraged servers to do it, and (b) clarified error handling if you don't.
> 
> 1) Define a status code for "unsupported content-encoding" (plus maybe discovery via a Accept-Encoding header field that can be sent with it)

Right now this falls into 415 Unsupported Media Type:

"""
   The 415 (Unsupported Media Type) status code indicates that the
   origin server is refusing to service the request because the payload
   is in a format not supported by this method on the target resource.
   The format problem might be due to the request's indicated Content-
   Type or Content-Encoding, or as a result of inspecting the data
   directly.
"""

So, it could be a new status code (that takes "or Content-Encoding" out of the definition of 415), or it could be a header on 415 that further refines its semantics.

> 2) Require servers to either support gzip content encoding or fail the request with the new status code.

We've had repeated attempts to use the HTTP/2 effort to modify HTTP semantics, and we've repeatedly rejected them. As Chair, I get very concerned when people try to make HTTP/2 a trojan horse for "improving" a particular situation, because the experience with new features layered into HTTP/1.1 this way (e.g., expect/continue, pipelining) was poor, and also because we'll be on a slippery slope to having lots of other people throw in their hobby features too.

In truth, I'm not entirely happy about the requirement for clients to support compressed responses, but tolerate it both because we have broad support for it, and its integration is relatively simple; it might require intermediaries to selectively decompress for 1.1 clients, but that's pretty straightforward.

This proposal does not enjoy broad support, and its integration is problematic; you'd be asking intermediaries to buffer, and to translate status codes from existing servers -- a dodgy proposition at best. For these reasons, I don't see consensus to add it.

> So in 1.1, the description of 411 should have stated that this indicates a violation of a 1.1 requirement, fwiw.

Uh, no. It is perfectly legitimate and conforming to send 411, in that it's interoperable. At most we might use a SHOULD to encourage certain behaviour that's more *optimal*, but I don't think this kind of optimisation is an appropriate use of a MUST.

As Roy said, part of the confusion here is due to the difference between a server implementation and a resource implementation.

> Is there a bug report for that yet?

You mean a feature request?

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 08:01:07 +0100

> So, it could be a new status code (that takes "or Content-Encoding" out of the definition of 415), or it could be a header on 415 that further refines its semantics.

Ah. I had forgotten that we added this to the description of 415.

So if we want to go down that route, we could recommend a 415 and in 
addition elevate "Accept-Encoding" to a response header field that could 
be used with 415.

> You mean a feature request?

No, I meant bug report :-)

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 21 Mar 2014 21:06:14 +1300

On 21/03/2014 8:01 p.m., Julian Reschke wrote:
>> On 19 Mar 2014, at 10:33 pm, Julian Reschke wrote:
>>
>>>> On 19 Mar 2014, at 8:01 am, Roy T. Fielding wrote:
>>>>
 ...
>>>> 2.HEAD (in 1.1 mode) will generate 411 for POST without a
>>>> Content-Length automatically. It's still one of the most widely
>>>> deployed intermediaries out there, IME.
>>>

There may have been a while ago. Current Squid-3 versions accept chunked
request bodies AFAICS. The HTTP protocol relevance is just that Squid-2
versions which had no chunked support are still widely used. Legacy
install base etc.

Amos

---

From: Roland Zink <[email-masked]>
Date: Fri, 21 Mar 2014 09:37:51 +0100

On 21.03.2014 00:58, Mark Nottingham wrote:
> In truth, I'm not entirely happy about the requirement for clients to 
> support compressed responses, but tolerate it both because we have 
> broad support for it, and its integration is relatively simple; it 
> might require intermediaries to selectively decompress for 1.1 
> clients, but that's pretty straightforward.
I have a concern about this in the presence of range requests / 
responses. Assuming there is a HTTP/1.1 to HTTP/2 gateway and a client 
supporting range requests but not gzip. The gateway can't decompress a 
potential range response when the start isn't included. What should the 
gateway do to a ETag header when it needs to decompress?

Roland

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Fri, 21 Mar 2014 17:03:07 +0000

HTTP/1.0

955,546,143

15.2%

HTTP/1.1

5,318,719,368

84.8%

Blank

56,173

0.0%

Total

6,274,321,684

Number of requests for each, based on Wednesday’s CWS traffic.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 21 Mar 2014 12:12:51 -0700

On 21 March 2014 01:37, Roland Zink <[email-masked]> wrote:
> I have a concern about this in the presence of range requests / responses.
> Assuming there is a HTTP/1.1 to HTTP/2 gateway and a client supporting range
> requests but not gzip. The gateway can't decompress a potential range
> response when the start isn't included. What should the gateway do to a ETag
> header when it needs to decompress?

Sounds like a perfect case to apply 415 to.  ETag would (I imagine) be
the same regardless of whether gzip is applied.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Mar 2014 21:21:35 +0100

> the same regardless of whether gzip is applied.

The Etag will need to be different, as it is a different entity.

Best regards, Julian

---

From: Roland Zink <[email-masked]>
Date: Fri, 21 Mar 2014 22:12:43 +0100

> the same regardless of whether gzip is applied.

In short you mean disabling range requests in this case?

There would also the possibility for the gateway to remove the range 
request header and return the full response. There seem however some 
clients insisting on range responses. So this can be be extended to 
remove the range request header, decompress the whole content and 
extract the requested range from it.

Roland

---

From: Zhong Yu <[email-masked]>
Date: Fri, 21 Mar 2014 16:24:36 -0500

On Fri, Mar 21, 2014 at 3:37 AM, Roland Zink <[email-masked]> wrote:
>> support compressed responses, but tolerate it both because we have broad
>> support for it, and its integration is relatively simple; it might require
>> intermediaries to selectively decompress for 1.1 clients, but that's pretty
>> straightforward.
>

The gateway cannot simply relay the Range request header to the origin
server. What the client means is a range over the uncompressed data;
what the server interprets is (most likely) a range over the
compressed data.

If the gateway does not keep the decompression state, it'll have to
request the whole compressed entity from the origin server.

> header when it needs to decompress?

Maybe the gateway can do ETag transformation. If the origin response is

    Content-Encoding: gzip
    ETag: 123.gz

the gateway response can be

    (No Content-Encoding)
    ETag: 123.gz-gateway.ungzip

If the origin response is uncompressed

    (No Content-Encoding)
    ETag: 123

the gateway response should be

    (No Content-Encoding)
    ETag: 123-gateway

The gateway needs to undo the transformation on ETags in client requests.

Zhong Yu

---

From: Roland Zink <[email-masked]>
Date: Fri, 21 Mar 2014 22:26:11 +0100

So the gateway should create an ETag of its own, make it a weak etag or 
remove it? When the server doesn't know about it then the gateway 
probably need to translate the etag between the client and server legs. 
What happens if the client switch to a different gateway or talks 
directly to the server.

Regards, Roland

On 21.03.2014 21:21, Julian Reschke wrote:
>>> responses.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Mar 2014 10:36:42 +1100

On 21 Mar 2014, at 6:01 pm, Julian Reschke <[email-masked]> wrote:

> So if we want to go down that route, we could recommend a 415 and in addition elevate "Accept-Encoding" to a response header field that could be used with 415.

That seems to make sense, but it isn't something specific to HTTP/2, and it's extending the semantics of a HTTP/1 header field. That sounds like a new spec that updates p2-semantics.

Julian, do you want to sketch that out so people can have a look?

Cheers,

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 24 Mar 2014 16:49:25 -0700

> Julian, do you want to sketch that out so people can have a look?

Please, no.  Changing CE transforms the content.  That means it is
destructive unless done with full knowledge of the publication chain
(i.e., how does the origin server know whether the client wants the
representation to be compressed or just the transfer to be compressed?).

You should be talking about transfer encodings and advertised server
settings, not CE.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Mar 2014 08:53:08 +0100

>> Julian, do you want to sketch that out so people can have a look?

I can.

> representation to be compressed or just the transfer to be compressed?).

This is not about *changing* CE but simply about making it clearer how 
to signal errors (when the server doesn't support the CE used in the 
request).

> You should be talking about transfer encodings and advertised server

Yes. That too.

Best regards, Julian