Subject: #445: Transfer-Codings

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 Apr 2014 11:44:07 +1000

<https://github.com/http2/http2-spec/issues/445>

After a fair amount of discussion on list, we have a proposal:
  <https://github.com/http2/http2-spec/pull/463>

I understand that a number of implementers are not planning to do this (just as they don't do Transfer-Encoding: gzip), but there is significant interest from non-browser folks here, and it gets us closer to feature parity with HTTP/1. 

I also don't hear any arguments that it's fundamentally harmful to include -- just that some aren't interested in implementing it for their use case. So, my inclination is to include it in the implementation draft, barring a new argument against it that is convincing.

Note that the concerns in #423 also apply to this (it could be argued even more so than with content-codings), so we'll need to discuss that and come to a resolution there. 

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Apr 2014 13:37:50 -0700

> I also don't hear any arguments that it's fundamentally harmful to include -- just that some aren't interested in implementing it for their use case. So, my inclination is to include it in the impl…

I've included the changes [1].  Again with changes:

There are security considerations now.
There is a note about Content-Length.

https://github.com/http2/http2-spec/commit/a02f5a868d99234cc19c7644ad38f91e151413fb

---

From: <[email-masked]>
Date: Wed, 23 Apr 2014 21:21:44 +0000

> https://github.com/http2/http2-spec/commit/a02f5a868d99234cc19c7644ad38f91e151413fb

Some comments regarding the security considerations...

> Intermediaries MUST NOT alter the compression of <x:ref>DATA</x:ref> frames unless additional information is available that allows the intermediary to identify the source of data.

I was under the impression that you and Matthew said this couldn't be a "MUST NOT" requirement because it's not enforceable.  Perhaps I misunderstood.

> In particular, frames that are not compressed cannot be compressed, and frames that are separately compressed cannot be merged into a single frame.

Why is merging disallowed? Two (or more) gzip files can be concatenated to form a single gzip. So unless there is a security issue I'm missing, I don't see why this is a problem.

> Compressed frames MAY be decompressed or split into multiple frames.

Do you mean split after decompressing? That's ok.  The way I read the sentence, though, is that you can split compressed frames into multiple frames.  You can't do that because each frame has its own gzip context and the second, etc. frames (and possibly even the first) would not be decompressable after a split.

Finally, what about removing and then re-applying gzip (e.g. to extract a range).  That should be ok too.

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Apr 2014 14:43:28 -0700

> I was under the impression that you and Matthew said this couldn't be a "MUST NOT" requirement because it's not enforceable.  Perhaps I misunderstood.

Perhaps you did.  Security considerations don't tend to include
requirements that are enforceable in the classic sense.  If an
intermediary compresses despite being ignorant about what it's
compressing, then the violation might be hard to detect for a client.
But it can create a security vulnerability.  That has a different sort
of enforcement associated with it.

> Why is merging disallowed? Two (or more) gzip files can be concatenated to form a single gzip. So unless there is a security issue I'm missing, I don't see why this is a problem.

As long as the compression context is shared for the merged fragments,
then it is OK I suppose.  The concern here is that an origin server
might split frames intentionally in order to make it clear to
downstream consumers that the content of those frames cannot be mixed
in the same compression context.  Having an intermediary undo that
would be bad.

> Do you mean split after decompressing? That's ok.  The way I read the sentence, though, is that you can split compressed frames into multiple frames.  You can't do that because each frame has its o…

Indeed, I thought that much was obvious, but on a re-read, I think a
few extra words might be needed here.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Apr 2014 14:46:46 -0700

Missed this one.

> Finally, what about removing and then re-applying gzip (e.g. to extract a range).  That should be ok too.

Yes.  But do you really think that more text is really necessary for that?

---

From: <[email-masked]>
Date: Thu, 24 Apr 2014 09:26:29 +0000

> As long as the compression context is shared for the merged fragments, then it is OK I suppose.  The concern here is that an origin server might split frames intentionally

> in order to make it clear to downstream consumers that the content of those frames cannot be mixed in the same compression context.  Having an intermediary undo that would be bad.

This is a really good point.  Although merging two compressed frames still keeps the compression contexts separate, merging can create a problem if later the merged frame is decompressed* and recompressed - because the data would then share a single compression context (i.e. possibly merging confidential and attacker-controlled data into a single context).  In other words, this becomes relevant if decompressing and recompressing is allowed (see below).  So I suggest keeping the requirement to not merge frames, but it would be good to add some sort of explanation as to why this is disallowed otherwise I suspect there will be others reading the spec that will have the same thought I did and disregard this as a nonsense requirement.

(*For example, as far as I know the zlib library will merrily decompress two contexts which are merged without so much as telling you that it found separate contexts because concatenated contexts is essentially a built-in feature.  From the zlib manual: "If flush is set to Z_FULL_FLUSH, all output is flushed as with Z_SYNC_FLUSH, and the compression state is reset so that decompression can restart from this point if previous compressed data has been damaged or if random access is desired. " [1]).

> Yes.  But do you really think that more text is really necessary for that?

In light of the really good point you made about the problem with merging frames, now I do think this has to be explicit.

Here is my suggested change. It’s tough to explain/justify this succinctly. I’m sure you will be able to write it more eloquently :) …

Intermediaries MUST NOT alter the compression of DATA frames unless additional information is available that allows the intermediary to identify the source of data. In particular, frames that are not compressed cannot be compressed, and frames that are separately compressed cannot be merged into a single frame. However, compressed frames MAY be decompressed (note that it may be necessary to split the uncompressed data into multiple DATA frames). Additionally, compressed frames MAY be decompressed and then recompressed (e.g. to extract a range), however the uncompressed data from separate frames MUST NOT be merged.  Merging uncompressed data from separate frames creates a shared compression context that could allow an attacker to recover secret data if merging combines confidential and attacker-controlled data.  Likewise, merging compressed frames could allow an attacker to recover secret data if a downstream intermediary elects to decompress then recompress the merged frame, potentially combining confidential and attacker-controlled data into a shared compression context.

[1] http://www.zlib.net/manual.html#Basic

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Apr 2014 07:57:01 -0700

On 24 April 2014 02:26,  <[email-masked]> wrote:
> however the uncompressed data from separate frames MUST NOT be merged.
> Merging uncompressed data from separate frames creates a shared compression
> context that could allow an attacker to recover secret data if merging
> combines confidential and attacker-controlled data.

My first impression was that you have too much text there, but this
point is valuable.  I had to ship -12, but I'll add this to the
editor's copy in preparation for -13.

Note that this is only an issue if the merged data is subsequently compressed.

I think that the text could more succinctly say:

[...] frames that are separately compressed cannot be merged into a
single compressed frame.  Either could result in the compression of
secret and attacker-controlled data within the same compression
context.  Compressed frames MAY be decompressed, in whole or part.

I'm less certain about making the downstream intermediary point so
explicit.   There's a balance to be struck.  There are plenty of other
places where our prohibitions are less-well explained than this
already is.  This is a fractal landscape, and we try to avoid
exhaustively exploring all the minute details in the interests of
readability and accessibility.

---

From: Johnny Graettinger <[email-masked]>
Date: Thu, 24 Apr 2014 14:51:35 -0400

This requirement to not merge does appear to be in tension with Martin's
point on the "Frame Length Restrictions" thread, that a padding
intermediary is able to re-frame to 16K - 9 bytes to achieve an arbitrary
padding distribution.

Conceptually, the "don't merge" and "don't compress" requirements impose
new implicit end-to-end boundaries in the data stream (for each compression
context). Perhaps it would be preferable to use the existing segment
mechanism instead? Eg, compression is per-frame, but intermediaries may
re-frame and re-compress within a segment iff the origin also compressed
that segment.

There was some concern on the the other thread about the delineation of
secret vs attacker-controlled segments being difficult for clients/servers,
but I'm unclear how coupling compression boundaries to frames helps this.
If an implementation doesn't have enough information to segment, surely it
also doesn't have enough to identify frame boundaries appropriately?

thanks,
-johnny

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Apr 2014 12:52:59 -0700

On 24 April 2014 11:51, Johnny Graettinger <[email-masked]> wrote:
> Perhaps it would be preferable to use the existing segment mechanism
> instead?

I think that this is reasonable, and on balance, probably the right solution.

I remain skeptical over claims that intermediaries are able to add
padding, partly due to lack of information, and partly due to the fact
that the intermediary is presumably getting data unpadded, which
probably needs the same sort of protection.  There's a long-standing
principle that I think is applicable in this case.  That principle
states that end-to-end is the only real security.

---

From: <[email-masked]>
Date: Thu, 24 Apr 2014 19:58:22 +0000

Sounds reasonable to me. Would that imply that all frames within a segment must be compressed or not compressed? i.e. would it be an error to have the compress flag vary from frame to frame within a segment? (That would imply a receiver has to keep state per segment to remember the flag.)

-Keith

On Apr 24, 2014, at 20:56, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:

This requirement to not merge does appear to be in tension with Martin's point on the "Frame Length Restrictions" thread, that a padding intermediary is able to re-frame to 16K - 9 bytes to achieve an arbitrary padding distribution.

Conceptually, the "don't merge" and "don't compress" requirements impose new implicit end-to-end boundaries in the data stream (for each compression context). Perhaps it would be preferable to use the existing segment mechanism instead? Eg, compression is per-frame, but intermediaries may re-frame and re-compress within a segment iff the origin also compressed that segment.

There was some concern on the the other thread about the delineation of secret vs attacker-controlled segments being difficult for clients/servers, but I'm unclear how coupling compression boundaries to frames helps this. If an implementation doesn't have enough information to segment, surely it also doesn't have enough to identify frame boundaries appropriately?

thanks,
-johnny

On Thu, Apr 24, 2014 at 10:57 AM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Apr 2014 13:06:37 -0700

> Sounds reasonable to me. Would that imply that all frames within a segment must be compressed or not compressed? i.e. would it be an error to have the compress flag vary from frame to frame within…

Yes, I think that it would have to be an error to toggle the
COMPRESSED flag for a segment.

---

From: <[email-masked]>
Date: Thu, 24 Apr 2014 21:12:37 +0000

On Apr 24, 2014, at 22:07, "[email-masked]" <[email-masked]> wrote:
>> ... i.e. would it be an error to have the compress flag vary from frame to frame within a segment? ...>

Why do you think it would have to be an error? (I'm not disagreeing, just want to understand the reasoning.) Would it introduce a security vulnerability? Increase implementation complexity? Although it might be odd for an implementation to mix compressed and uncompressed frames within a segment, I can think of examples where it might be convenient.

As I said, checking for this error condition would force the receiver to track the state of the COMPRESSED flag for the current segment of all outstanding streams - and the only reason for the state would be to detect the toggling error condition. It also essentially makes the COMPRESSED flag redundant from frame to frame because only the value of the flag in the first frame of the segment would count.

If you're going to impose a requirement on the receiver to track this state for all streams, then you might as well change from per frame compression to per segment compression. Per segment compression would improve the compression factor. It would also be more compatible with T-E from 1.1 (eg at 1.1 <-> 2 gateways).

I'm not actually advocating for per segment compression. I like the simplicity of per frame compression (particularly the statelessness).

-Keith

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Apr 2014 14:59:24 -0700

> Why do you think it would have to be an error? (I'm not disagreeing, just want to understand the reasoning.) Would it introduce a security vulnerability? Increase implementation complexity? Althoug…

I don't know what it means to get uncompressed frames interspersed
with compressed frames, if the entire segment is intended to have the
same compression context, that's all.

> As I said, checking for this error condition would force the receiver to track the state of the COMPRESSED flag for the current segment of all outstanding streams - and the only reason for the stat…

Well, this is the consequence of doing that.  You can avoid the issue
entirely by no supporting compressed frames :)

> If you're going to impose a requirement on the receiver to track this state for all streams, then you might as well change from per frame compression to per segment compression. Per segment compres…

Sorry, I should have made that clearer.  That is what I intended to
imply.  It means that you get C-E equivalent performance if you don't
mix security contexts in the same entity body (and don't segment).
However, it does increase state overheads significantly because it's
not just the single bit, it's an entire compression context that you
carry over between frames.

---

From: <[email-masked]>
Date: Fri, 25 Apr 2014 08:26:18 +0000

On Thursday,24 April 2014 23:59, Martin Thomson wrote:
> On 24 April 2014 14:12,  <[email-masked]<mailto:[email-masked]>> wrote:

>> If you're going to impose a requirement on the receiver to track [the state of the COMPRESSED flag] for all streams, then you might

>> as well change from per frame compression to per segment compression. Per segment compression would

>> improve the compression factor. It would also be more compatible with T-E from 1.1 (eg at 1.1 <-> 2 gateways).

> Sorry, I should have made that clearer.  That is what I intended to imply.  It means that you get C-E equivalent

> performance if you don't mix security contexts in the same entity body (and don't segment). However, it does

> increase state overheads significantly because it's not just the single bit, it's an entire compression context that

> you carry over between frames.

Ok.  We had a big disconnect.  I didn’t realized you were implying a single compression context per segment.

That's NOT what Johnny said when he suggested the idea of using END_SEGMENT (hence my confusion).

He said (emphasis added)...

"...compression is per-frame, but intermediaries may re-frame and re-compress within a segment iff the origin also compressed that segment."

A few thoughts follow, but most importantly, what are we supposed to implement for the -12 implementation draft?  (It quite clearly states that compression is per frame.)

Benefits of one compression context per segment:

- Per segment compression is closer in concept to Transfer-Encoding in 1.1.

- As you pointed out, it also has better compression performance (comparable to C-E gzip).

Drawbacks of one compression context per segment:

- As you pointed out, doing per segment compression “increases state overhead significantly” (and implementation complexity).

- It makes the COMPRESSED flag redundant and somewhat ambiguous.

  + It would make more sense to have a mechanism for signalling BEGIN_SEGMENT and the COMPRESSED flag is only valid when BEGIN_SEGMENT is set.

Since we already hashed all of this out once, I would prefer to stick with per frame compression, but loosen the security restrictions to include Johnny’s idea of re-frame and re-compress within a segment.  To accomplish that I would use Johnny’s statement with the following change…

"...compression is per-frame, but intermediaries may re-frame and re-compress within a segment iff the origin also compressed at least one frame within that segment."

Although it might be odd to mix uncompressed frames with compressed frames within a segment, I don’t see a security concern that makes it necessary to enforce that all frames within a segment are either compressed or not compressed.

---

From: <[email-masked]>
Date: Fri, 25 Apr 2014 08:55:00 +0000

On Thursday,24 April 2014 23:13, MORGAN, Keith Shearl wrote:

>... Although it might be odd for an implementation to mix compressed and uncompressed frames within a segment,

> I can think of examples where it might be convenient.

Sorry to reply to my own-email, but I wanted to mention a few examples where it might be convenient to allow compressed and uncompressed frames within a segment.  These examples may or may not be pathological.  Perhaps others with more experience have other examples??

1) A server under heavy load might want to dynamically switch off gzip frame compression to conserve cpu resources.  Allowing a mix of compressed and uncompressed frames within a segment would allow a server to immediately switch from compressed frames to uncompressed frames.

2) Amos mentioned here [1] that intermediaries may want to extract a range from a resource they already have cached.  I could imagine a scenario where the intermediary might want to cache the resource with the frames compressed (i.e. as received) to avoid re-compressing for clients which support gzip.  To accomplish that the intermediary could index the offsets of the underlying data contained in each compressed frame.  When later servicing a range request the intermediary would first find the frame which contains the beginning offset of the range.  That frame would likely need to be decompressed to discard the data < the beginning offset.  The data from that frame >= the beginning offset could be re-compressed or sent uncompressed.  All of the frames in-between the beginning and end offsets would be sent as-is (i.e. compressed).  The frame containing the end offset of the range would be processed in the same manner as the frame containing the beginning offset.

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0115.html

---

From: Martin Thomson <[email-masked]>
Date: Fri, 25 Apr 2014 08:56:41 -0700

On 25 April 2014 01:26,  <[email-masked]> wrote:
> A few thoughts follow, but most importantly, what are we supposed to
> implement for the -12 implementation draft?  (It quite clearly states that
> compression is per frame.)

That doesn't change.  -12 is very clear on this.  But it was fairly
clear to me that this discussion wasn't over, so here we are.

I'm perfectly happy to close the issue and call this done.  Roberto
raised some DoS concerns here as well (I'll point out that anyone
concerned about this is free to not implement the feature, which is
essentially free).