Subject: Porting T-E to HTTP/2: Reasons Against

From: Martin Thomson <[email-masked]>
Date: Thu, 10 Apr 2014 16:59:55 -0700

There's been a lot of email on this topic.  I want to just summarize
the reasons for not doing this briefly and to reiterate that I don't
think that this is a good plan.  We have, on a few occasions, opted
for less than feature parity with HTTP/1.1, and I see this as a very
good example of a feature we don't need.

1. We decided once already not to do this.  SPDY had the feature, we
explicitly cut it. [1]

2. It's a very late stage to be adding features.

3. The proposal we have (Matthew's) would need significant work to get
all the details right.  In particular, the negotiation portion would
require some work, particularly as it relates to intermediaries.

4. T-E (other than chunked) is a seldom used feature in HTTP/1.1.
I've seen no evidence that a congruent feature in HTTP/2 would be any
different.

5. This feature would require serious work on the security
implications.  The main concern here is similar to the one that has
pretty much lead to the removal of generic compression from TLS 1.3
[2].  This enables compression that is removed from the contextual
information necessary to determine that compression is safe from
attacks like BEAST [3].

6. I haven't seen strong interest in implementing this from many
people on this list, other than a few.  Many of these concerns are
surmountable if a widely used implementation planned to implement the
feature (think browser, OS, server stack).

In short, while I can see some marginal benefits [4], I don't believe
that we need this feature in HTTP/2.

--Martin

[1] https://github.com/http2/http2-spec/issues/46
[2] http://www.ietf.org/mail-archive/web/tls/current/msg11619.html
[3] https://blog.torproject.org/blog/tor-and-beast-ssl-attack
[4] I think that T-E leading to worse compression than C-E supports of
my conclusion.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Apr 2014 17:07:25 -0700

Agreed. Additionally:

7. It provides an additional uncontrolled venue for requiring endpoints to
allocate memory.

-=R

---

From: Nicholas Hurley <[email-masked]>
Date: Thu, 10 Apr 2014 17:11:14 -0700

+1 to everything Roberto and Martin have said. I have no intention of
implementing this (were it to make it into the spec), for my part.

-- 
Peace,
  -Nick

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 11 Apr 2014 10:19:20 +1000

I'm happy to explore other avenues. I'm also happy to defer the discussion
until HTTP/3.

I still believe there's a significant case that HTTP/2 doesn't address,
that HTTP/1.1 (theoretically) does: allowing compression after
entity-specific operations (such as ranges).

If T-E is not the right way, I would want to discuss other mechanisms; and
since those would likely modify HTTP semantics, not to mention requiring
lengthy debate, it should wait for HTTP/3.

Others may have a stronger opinion about the importance of T-E in HTTP/2.

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: <[email-masked]>
Date: Fri, 11 Apr 2014 14:50:42 +0000

On Friday,11 April 2014 02:19, Matthew Kerwin wrote:

> I still believe there's a significant case that HTTP/2 doesn't address, that HTTP/1.1

> (theoretically) does: allowing compression after entity-specific operations (such as ranges).

We actually need that feature, but finally we accept: HTTP/2 will NOT support it. We proposed 3 options:

1)    Reintroduce "Transfert-Encoding: gzip": We agree that is not an option because it is a hop-to-hop header with all the associated consequences.

2)    Introduce a "Message-Encoding" which is in almost all cases like "Content-Encoding", specifically end-to-end. The significant difference is in the semantics of a range (range is applied after Content-Encoding; while range is applied before Message-Encoding). With Message-Encoding the server can compress a range. Due to the subtle difference we got responses like "...effectively makes it a redundant duplicate of CE" [1].

3)    Independent frame-by-frame compression with a negligible compression ratio "length(compressed data) / length(raw data)" increase. The intention was an end-to-end compression like with "Message-Encoding", but allowing the intermediary the option to directly serve range requests (most likely the first (and last frame) uncompressed) with small memory/CPU footprint.

We believe the original intent of C-E was efficient handling of pre-compressed content. Since C-E was adopted early on by almost all HTTP/1.1 implementations, C-E was also adopted for compressing dynamic content, to speed up a transfer (and despite the misuse causes no problems). For non-pre-compressed static content (likely a bad server design/usage), dynamic C-E requires, at best, useless CPU-processing when serving a range (somehow a range of something which does not exist). We think it is a DoS attack to ask a server for "Range: -1" which requires lots of CPU on the server to compress the entire content just to provide the final byte (+headers). We also figured out that ranges of Content-Encoding is somehow ambiguous (not in terms of the spec), but basically browsers don't trust the behavior of the server and vice versa (with the exception of Chrome which is the only browser that attempts heuristically to deal with it at all). We've also seen that servers and clients have changed their interpretation over time. Our motivation was to clarify that situation and allow browsers and servers to support more features, specifically in recovering from broken transfers.

Chris & Keith

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0115.html

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Roland Zink <[email-masked]>
Date: Fri, 11 Apr 2014 18:17:43 +0200

> 2. It's a very late stage to be adding features.
Mandating C-E gzip was added lately and actually this causes some 
problems which may be solved be T-E.
> I've seen no evidence that a congruent feature in HTTP/2 would be any
It is not only about C-E vs. T-E but also about seek requests. In 
HTTP/1.1 the client has the choice to prefer range requests / responses 
over compression. In HTTP2 the client will have no choice.
> information necessary to determine that compression is safe from
Actually why is C-E better here than T-E? Especially as existing proxies 
do C-E on the fly.

> [4] I think that T-E leading to worse compression than C-E supports of
Mandating C-E gzip breaks compatibility with existing HTTP/1.1 clients, 
so do we want to get rid of mandating compression at all?

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Apr 2014 09:41:06 -0700

On 11 April 2014 09:17, Roland Zink <[email-masked]> wrote:
> Mandating C-E gzip was added lately and actually this causes some problems
> which may be solved be T-E.

C-E gzip (and maybe deflate) have been in the spec since the
beginning.  The recent change was to remove the mandate for C-E
deflate (which might not have been a mandate due to confusing text).

> Actually why is C-E better here than T-E? Especially as existing proxies do
> C-E on the fly.

That's not a good idea.  For the aforementioned reason, and for other
reasons including having to re-mint ETags without coordination with
the origin server.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 11 Apr 2014 18:51:55 +0200

> reasons including having to re-mint ETags without coordination with

But don't we have that problem already (ack: [email-masked]):

"Intermediaries that perform translation from HTTP/2 to HTTP/1.1 MUST 
decompress payloads unless the request includes an Accept-Encoding value 
that includes "gzip"." -- 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-http2-11.html#rfc.section.9.3.p.2>

What are these intermediaries supposed to do with the ETag?

Best regards, Julian

---

From: Roland Zink <[email-masked]>
Date: Fri, 11 Apr 2014 19:53:47 +0200

Hi Martin,

So the following text was modified?

Roland

9.3 GZip Content-Encoding

Clients MUST support gzip compression for HTTP response bodies. Regardless of the value of the accept-encoding header field, a server MAY send responses with gzip encoding.

> Am 11.04.2014 um 18:41 schrieb Martin Thomson <[email-masked]>:
> 
I totally agree, but they do it because T-E isn't supported.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Apr 2014 11:22:32 -0700

On 11 April 2014 10:53, Roland Zink <[email-masked]> wrote:
> Clients MUST support gzip compression for HTTP response bodies. Regardless
> of the value of the accept-encoding header field, a server MAY send
> responses with gzip encoding.

draft-mbelshe-httpbis-spdy-00 included the following text:

      User-agents MUST support gzip compression.  Regardless of the
      Accept-Encoding sent by the user-agent, the server may always send
      content encoded with gzip or deflate encoding.

That has been replaced with the section you cite part of.

---

From: Roland Zink <[email-masked]>
Date: Fri, 11 Apr 2014 20:59:42 +0200

If this was always in then I problably misunderstood issue #424 
(https://github.com/http2/http2-spec/issues/424) and the consensus call 
to it http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1118.html.

The problem itself still persists and I mentioned this repeatedly, but 
see also Keith's mail 
(http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0175.html), 
Julian's message 
(http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0179.html) 
and issue 445 https://github.com/http2/http2-spec/issues/445.

The requirement to support gzip compression doesn't hold for HTTP/1.1 
clients and a HTTP/1.1 to HTTP2 gateway would either need to decompress 
violating the end to end property of content encoding or some HTTP/1.1 
clients are incompatible. Using T-E instead of C-E would be a possible 
solution to the problem.

Regards,
Roland

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 Apr 2014 12:11:35 -0700

On 11 April 2014 11:59, Roland Zink <[email-masked]> wrote:
> (https://github.com/http2/http2-spec/issues/424) and the consensus call to
> it http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1118.html.

#424 is about the client being able to compress request bodies (again
using C-E, not T-E).

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 15 Apr 2014 02:48:09 -0600

> good example of a feature we don't need.

Unless the consequence is intermediaries change C-E, thus diluting the
origin server's authority over its own resources. Content and transfer
codings are so *not* the same thing that this smacks of exceeding
HTTP2's scope by changing the semantics of the protocol, not deprecating
an "unused" feature. Matthew Kerwin also pointed this out; didn't think
I'd need to post the same point, but...

> feature (think browser, OS, server stack).

I didn't realize quantity of support trumped even one person correctly
pointing out a serious design flaw, especially when it's been well-put
by a credible outfit like IAEA, plus Roy, and others. If that's what
you want, I guess I'll be back in a few minutes (so to speak), but not
before pointing out that these concerns become insurmountable if the
major implementers bork HTTP's semantics by failing to recognize valid
use-cases which expose said design flaw, instead dismissing them as the
edge-cases they likely are -- which doesn't make the problem go away.

-Eric