Subject: Re: h2#404 requiring gzip and/or deflate

From: Zhong Yu <[email-masked]>
Date: Fri, 21 Mar 2014 16:39:35 -0500

On Fri, Feb 21, 2014 at 11:13 AM, Martin Thomson
<[email-masked]> wrote:
> On 21 February 2014 05:08, Patrick McManus <[email-masked]> wrote:
>> I think what the http/2 text means to say is that an "Accept-Encoding: gzip,
>> deflate" request header is implicit in every request and the client needs to
>> be able to process responses with those encodings.
>
>
> Yes, that is the implication.  Even if you include "Accept-Encoding:
> rot13", you effectively get *either* "Accept-Encoding: rot13,
> identity, gzip, deflate" or "Accept-Encoding: rot13, identity, gzip".
> This is a change from 1.1, which implies "Accept-Encoding: rot13,
> identity" only.

What if the request has

    Accept-Encoding: gzip;q=0

Can server ignore that directive and send gzip-ed response anyway?

>
> I had inferred the gzip+deflate, but Mark points out that this could
> also be interpreted as the latter.  Hence this thread requesting
> clarification.
>
> I hope that I'm not completely off-base on this.  No one wants to
> remove this entirely and revert to "identity" only, do they?
>

---

From: Cory Benfield <[email-masked]>
Date: Sat, 22 Mar 2014 19:35:14 +0000

According to the current draft, yes. From section 9.3:

> Regardless of the value of the accept-encoding header field, a server
> MAY send responses with gzip or deflate encoding.

The current draft leaves no way to prevent the server from sending you gzip.

Of course, servers may _choose_ to honour your request, but they aren’t
required to.

---

From: <[email-masked]>
Date: Mon, 31 Mar 2014 13:28:24 +0000

On Fri, 21 Feb 2014 11:16:21, Martin Thomson wrote:

> On 21 February 2014 11:02, Albert Lunde <[email-masked]> wrote:

>> Another question is whether compression schemes introduce side channels better to attack TLS.

> Yes, there is a concern here.  That's why we have padding. ...

> If at some point [our TLS friends] come to us and say "we have decided to forbid compression" or,

> more likely, "you are going to have to do the following things if you

> want to use compression", then we will have to consider our options.

As Albert mentioned, suppose a side-channel attack of TLS is found (particularly in light of CRIME, BREACH, etc.) and suppose (although unlikely) there are no options from "our TLS friends" other than to "forbid compression".

What are the options from the perspective of clients that may want to explicitly opt out of compression? The implicit "Accept-Encoding: gzip" leaves a client no way to tell the server to not use compression.

In other words, if an attack is found, you have to wait for all the servers around the world to be patched to manually disable compression.

On Saturday,22 March 2014 20:35, [email-masked] wrote:

>> On Friday,21 March 2014 22:40, Zhong Ju wrote:
>> What if the request has Accept-Encoding: gzip;q=0 Can server ignore that directive and send gzip-ed response anyway?

> Of course, servers may _choose_ to honour your request, but they aren’t required to.

I think an implicit opt in for accept-encoding for gzip is no problem, but at the very least there should be a way for clients to explicitly opt out of gzip.

A few thoughts on options…

1. Zhong Ju’s idea to send "Accept-Encoding: gzip;q=0" seems like the most logical option, but q=0 support is definitely mixed at best (based on a simple random check of servers in the wild). I suppose explicit support for gzip;q=0 could potentially be mandated by language in the HTTP/2 RFC?

2. Sending any “Accept-Encoding” header explicitly overrides the implicit gzip value (admittedly I’m not sure of all the implications of this option).

3. Switch back to an explicit “opt-in” for compression.  Are the really that many clients out there that don’t already send “Accept-Encoding: gzip”?? (The overhead of using this header on every request should be minimal by adding it to the HPACK static table.)

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 31 Mar 2014 09:31:16 -0700

On 31 March 2014 06:28,  <[email-masked]> wrote:
> In other words, if an attack is found, you have to wait for all the servers
> around the world to be patched to manually disable compression.

At this stage, I consider it highly unlikely that an attack is found
whereby compression has to be globally disabled.  All the attacks that
rely on compression are highly contextual.  Use of compression for
static content (images, static HTML, JS, CSS) is highly valuable from
a performance perspective, and there are no significant threats in
that area.

Note that BEAST effectively did require all servers around the world
to be patched.  It's just that only a vanishingly small proportion of
them were vulnerable to the attack.  Maybe this won't be the case next
time, but with all the recent scrutiny, I'd be willing to bet that the
next attack won't be on compression.

I'd think it unlikely that the working group would reverse this
decision and remove the implicit opt-in.

---

From: <[email-masked]>
Date: Mon, 31 Mar 2014 19:48:58 +0000

On Mar 31, 2014, at 18:31, [email-masked]<mailto:[email-masked]> wrote:
At this stage, I consider it highly unlikely that an attack is found
whereby compression has to be globally disabled.

Agreed. I said "very" unlikely - but I should have said "highly" unlikely.

I'd think it unlikely that the working group would ... remove the implicit opt-in.

I agree, but why not allow a client to explicitly opt out of gzip?

In rfc 2616, clients may opt out of the implicit "identity" content-coding by sending "identity;q=0" (see section 14.3 rule #4 at https://tools.ietf.org/html/rfc2616#section-14.3)

I suggest the following based on the exact wording from rfc 2616 14.3 rule #4 ...

The "identity" and "gzip" content-codings are always acceptable, unless specifically refused because the Accept-Encoding field includes "identity;q=0", or "gzip;q=0" respectively, or because the field includes "*;q=0" and does not explicitly include the "identity" or "gzip" content-codings respectively. If the Accept-Encoding field-value is empty, then only the "identity" or "gzip" encodings are acceptable.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 31 Mar 2014 13:13:26 -0700

> In rfc 2616, clients may opt out of the implicit "identity" content-coding by sending "identity;q=0" (see section 14.3 rule #4 at https://tools.ietf.org/html/rfc2616#section-14.3)

I have no problem with that.  Unless I hear screams, I'll take your
proposed text.

---

From: Patrick McManus <[email-masked]>
Date: Mon, 31 Mar 2014 16:33:40 -0400

Martin, I don't think you should take that as editorial as it conflicts
with some of the implicit-gzip use cases.

CE gzip was made a fixed part of the protocol, in part, because
intercepting intermediaries (or antivirus libraries as a subset of that)
were explicitly stripping negotiation for it for their own convenience.
Making it non negotiable enhances the robustness of the protocol against
that which is imo the right trade given the history here and the importance
of compression.

http://www.stevesouders.com/blog/2010/07/12/velocity-forcing-gzip-compression/
http://stackoverflow.com/questions/2645883/accept-encoding-headers-being-sent-by-browser-but-not-received-by-server

On Mon, Mar 31, 2014 at 4:13 PM, Martin Thomson <[email-masked]>wrote:

> > In rfc 2616, clients may opt out of the implicit "identity"
> content-coding by sending "identity;q=0" (see section 14.3 rule #4 at
> https://tools.ietf.org/html/rfc2616#section-14.3)
>

---

From: Roberto Peon <[email-masked]>
Date: Mon, 31 Mar 2014 13:38:18 -0700

+1

---

From: Martin Thomson <[email-masked]>
Date: Mon, 31 Mar 2014 13:44:28 -0700

On 31 March 2014 13:33, Patrick McManus <[email-masked]> wrote:
> Martin, I don't think you should take that as editorial as it conflicts with
> some of the implicit-gzip use cases.

Ack.  That's HTTP.

---

From: David Morris <[email-masked]>
Date: Mon, 31 Mar 2014 13:56:41 -0700 (PDT)

In the future, please use some form quoting ... this would have
been impossible to parse if I didn't recall the earlier message.

Reply below ...

On Mon, 31 Mar 2014, [email-masked] wrote:

> The "identity" and "gzip" content-codings are always acceptable, unless
> specifically refused because the Accept-Encoding field includes
> "identity;q=0", or "gzip;q=0" respectively, or because the field
> includes "*;q=0" and does not explicitly include the "identity" or
> "gzip" content-codings respectively. If the Accept-Encoding field-value
> is empty, then only the "identity" or "gzip" encodings are acceptable.

This wording introduces the possiblity of no valid Content-Encoding
without specification of the error handling. If we are to allow
suppression of the gzip choice, we can't end up with the null case.
Perhaps along the lines of ...
  The "identity" and "gzip" content-codings are always implied as
  acceptable when the Accept-Encoding header is omitted or empty
  valued. If the Accept-Encoding field is included it will
  specify encodings acceptable to the client.

---

From: <[email-masked]>
Date: Mon, 31 Mar 2014 21:20:11 +0000

> On Mar 31, 2014, at 22:59, "[email-masked]" <[email-masked]> wrote:
> without specification of the error handling. ...

This wording is practically identical to the wording in rfc 2616. The same thing could happen in HTTP/1.1 if you only sent "identity;q=0" or "*;q=0". The error handling for the null case is mentioned in rfc 2616 section 14.3 right after rule number four. I was under the impression that the rules from http/1.1 still applied unless explicitly stated otherwise. Otherwise you basically have to copy the whole thing.

>  specify encodings acceptable to the client.

This is essentially option #2 I proposed earlier.

---

From: <[email-masked]>
Date: Mon, 31 Mar 2014 21:45:40 +0000

On Mar 31, 2014, at 22:36, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:

Martin, I don't think you should take that as editorial as it conflicts with some of the implicit-gzip use cases.

CE gzip was made a fixed part of the protocol, in part, because intercepting intermediaries (or antivirus libraries as a subset of that) were explicitly stripping negotiation for it for their own convenience. ...

I don't doubt it. On the other hand, enterprising intermediaries are always going to do something they aren't supposed to do. (e.g. In this case they could decompress and remove the CE gzip if it's more convenient for them.)

I would prefer to see the proposed text plus a requirement that intermediaries MUST NOT change the Accept-Encoding header of a request.

Regardless, if it's decided to stick with forced acceptance of gzip no matter what, the http/2 spec section 9.3 needs to be updated to explicitly disallow "gzip;q=0".

Section 9.3 currently reads "This effectively changes the implicit value of the Accept-Encoding header field ([HTTP-p2]<http://http2.github.io/http2-spec/#HTTP-p2>, Section 5.3.4<https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/26/p2-semantics.html#header.accept-encoding>) from "identity" to "identity, gzip".

If I read this and the reference to [HTTP-p2]<http://http2.github.io/http2-spec/#HTTP-p2>, Section 5.3.4<https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/26/p2-semantics.html#header.accept-encoding> then I would assume that you can disable gzip with "gzip;q=0" because that's what you can do with the implicit identity.

BTW, the text in [HTTP-p2]<http://http2.github.io/http2-spec/#HTTP-p2>, Section 5.3.4<https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/26/p2-semantics.html#header.accept-encoding> strikes me as ambiguous. It doesn't say if you can mix entries with q values with entries without q values. It also doesn't say what the q value is for the implicit identity entry (and by extension the implicit gzip entry in h2) - this would be important if a client specifies multiple entries and wants to give preference over gzip to a new compression algorithm e.g. superawesomezip.

---

From: <[email-masked]>
Date: Mon, 31 Mar 2014 21:55:45 +0000

Sorry about the quoting issue. The iOS quoting seems to be getting striped somehow. Second try...

On Mar 31, 2014, at 22:36, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:
> Martin, I don't think you
> should take that as editorial as
> it conflicts with some of the
> implicit-gzip use cases.
>
> intercepting intermediaries (or
> antivirus libraries as a subset of
> that) were explicitly stripping
> negotiation for it for their own
> convenience. ...

---

From: Martin Thomson <[email-masked]>
Date: Mon, 31 Mar 2014 15:08:11 -0700

> If I read this and the reference to [HTTP-p2]<http://http2.github.io/http2-spec/#HTTP-p2>, Section 5.3.4<https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/26/p2-semantics.html#header.acc…

That's something I will fix with: ...```, however gzip encoding cannot
be suppressed by including ";q=0"``` based on the feedback.  The
paragraph you reference was intended to be explanatory; it's obviously
caused some confusion.