Subject: RE: #445: Transfer-codings

From: <[email-masked]>
Date: Fri, 4 Apr 2014 18:41:06 +0000

The proposals by Matthew and Mark get the job done, but they strike us as more complex than TE/Transfer-Encoding...

A (partial) list of benefits of TE/Transfer-Encoding (over framing layer transfer coding):

     1.  TE/Transfer-Encoding doesn't require any changes at the HTTP/2 framing layer!!!

     2.  Since TE and Transfer-Encoding are headers, the decision to compress can be made closer to the application layer.

          a. i.e. this makes it easier to decide when to compress.

          b. It also makes it less likely that double compression will occur.

          c. This also covers Martin’s concern that “an entity performing compression has knowledge of what is being compressed and is therefore able to more accurately determine whether compression is safe to use.”

     3. Retains the extensibility to support new compression algorithms (with all the same benefits Matthew mentioned in [1]).

     4. With TE/Transfer-Encoding, only clients have a new requirement. Servers that don’t want to support Transfer-Encoding don’t have to do anything except ignore TE headers.

     5. Since TE/Transfer-Encoding is already part of the HTTP/1.1 spec, 1.1 clients could benefit from this compression when talking to a 1.1<->2 intermediary by implementing TE/Transfer-Encoding; relieving intermediaries of the burden to decompress; and save bandwidth.

We submit the following alternative proposal for consideration:

9.4 Transfer-Encoding

HTTP/2 retains the semantics of HTTP/1.1 regarding transfer codings (see [HTTP-p1]<http://http2.github.io/http2-spec/#HTTP-p1>, Section 4<https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/26/p1-messaging.html#transfer.codings>), with the following additional requirements. Servers MUST NOT use chunked transfer coding. Clients MUST support “gzip” transfer coding. If the "TE" request header field-value is empty or if no TE field is present, the only acceptable transfer coding is "gzip".

In earlier threads, Roy Fielding gave his support for compression transfer codings, including implicitly requiring clients to understand (decompress) gzip transfer encoding (our apologies to Roy for quoting him so often):

“I would not require [gzip transfer coding] to be used -- just make it possible to be used and required to understand (decompress by recipient).  This is what you already do with HTTP/2…” [2]

“… it isn't virtually impossible to introduce standard transfer codings.  It just requires effort by browsers to support one. It also isn't necessary to restrict HTTP/2 features to what a current browser supports.” [2]

“Transfer compression has limited deployment but actually does work when it is deployed by consenting adults.  You can change that deployment in HTTP/2. It is well within scope.” [3]

(We redact our earlier proposal [4] which had several flaws.)

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0067.html

[2] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1203.html

[3] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1189.html

[4] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1205.html

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 04 Apr 2014 15:14:42 -0400

My $0.02: Since "Transfer-Encoding: gzip" would apply to both HTTP/1.1 and HTTP/2, we should define gzip in a separate document that applies to both.  That would also allow for some needed security discussions in order to make the necessary conformance statements.

The change in HTTP/2 would then just be to disallow "TE: chunked" since HTTP/2's framing provides that already and say nothing about "TR: gzip" since that has yet to be defined/standardized.

On Apr 4, 2014, at 2:41 PM, [email-masked] wrote:

> HTTP/2 retains the semantics of HTTP/1.1 regarding transfer codings (see [HTTP-p1], Section 4), with the following additional requirements. Servers MUST NOT use chunked transfer coding. Clients MUST support “gzip” transfer coding. If the "TE" request header field-value is empty or if no TE field is present, the only acceptable transfer coding is "gzip".
>  
>  

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Apr 2014 13:23:10 -0700

On 4 April 2014 11:41,  <[email-masked]> wrote:
> The proposals by Matthew and Mark get the job done, but they strike us as
> more complex than TE/Transfer-Encoding...

I don't agree.  Both are considerably simpler.

Aside from TE: Trailers (which is basically information free in my
opinion), we've removed hop-by-hop headers from HTTP/2.  I don't think
that re-introducing them would be a good idea.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Apr 2014 13:31:25 -0700

On 4 April 2014 12:14, Michael Sweet <[email-masked]> wrote:
> My $0.02: Since "Transfer-Encoding: gzip" would apply to both HTTP/1.1 and
> HTTP/2, we should define gzip in a separate document that applies to both.
> That would also allow for some needed security discussions in order to make
> the necessary conformance statements.

?

Transfer-Encoding: gzip is defined for HTTP/1.1 already.

HTTP/2 doesn't permit hop-by-hop headers, so it cannot be defined in a
way that would be applicable to both.

---

From: <[email-masked]>
Date: Fri, 4 Apr 2014 20:36:50 +0000

On Apr 4, 2014, at 21:17, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:
> My $0.02: Since "Transfer-Encoding: gzip" would apply
> to both HTTP/1.1 and HTTP/2, we should define gzip in
> a separate document that applies to both. ...

Gzip transfer coding is already explicitly defined in Section 4 of the update to RFC2616 (HTTP-p1) - which is referenced in our proposal below.

Subsection 4.2.3 defines gzip:
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26#section-4.2.3

If there are specific security issues related to gzip then they apply equally to C-E gzip (which is implicitly required for clients to support already). As we mentioned below, one of the benefits of using TE/Transfer-Encoding over a framing layer transfer coding mechanism is that the decision to TE compress can be made at the same level as deciding to CE compress.

On Apr 4, 2014, at 2:41 PM, [email-masked]<mailto:[email-masked]> wrote:

9.4 Transfer-Encoding

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Apr 2014 13:41:21 -0700

> If there are specific security issues related to gzip then they apply equally to C-E gzip (which is implicitly required for clients to support already). As we mentioned below, one of the benefits o…

This is not correct.  An origin server applies Content-Encoding.
Transfer-Encoding is hop-by-hop.  As I pointed out previously, the
entities applying T-E are likely to lack the contextual information
required to make the right decisions regarding safe use of
compression.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 04 Apr 2014 17:13:28 -0400

Martin,

> Transfer-Encoding: gzip is defined for HTTP/1.1 already.

As defined in RFC 2616, the only well-defined TE value in section 3.6 is "chunked".  The others are simply referenced back to the Content-Encoding values in section 3.5 which is confusing to say the least.  The wording in p1-messaging (section 3.3.1) *is* much clearer, but still doesn't address today's security issues surrounding compression.

> way that would be applicable to both.

FWIW, the way Transfer-Encoding is defined in p1-messaging, it makes it pretty clear that compression MAY or MAY NOT be end-to-end at the discretion of each recipient in the chain, but it is not explicitly a hop-by-hop header and it definitely is not a frame-by-frame header (which is what is being proposed.)

---

From: Michael Sweet <[email-masked]>
Date: Fri, 04 Apr 2014 17:20:50 -0400

See my response to Martin.

The origin server sets Content-Encoding and is able to make an informed decision about what content is compressed.  Intermediates are not supposed to change that.  But they *can* change Transfer-Encoding while still not knowing whether the content should be compressed.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Apr 2014 14:23:54 -0700

> FWIW, the way Transfer-Encoding is defined in p1-messaging, it makes it pretty clear that compression MAY or MAY NOT be end-to-end at the discretion of each recipient in the chain, but it is not ex…

I suspect that that is only because it allows an intermediary to
forward TE if it doesn't intend to un-Transfer-Encode the response on
the way back.

---

From: Frédéric Kayser <[email-masked]>
Date: Fri, 4 Apr 2014 23:25:20 +0200

My $0.02: why pick Gzip over Deflate?
A Deflate stream is shorter than Gzip, Gzip is a file format which eventually embeds a file name, holds a time stamp — 32 bits epoch than will not even survive past January 2038 — and defines a compression method which has always been Deflate (and will probably never change) followed by Deflate compressed data. Why bother including all this extra payload and not use a stripped down Deflate (RFC 1951) or more precisely a Zlib stream (RFC 1950)? 
I know that some server vendors had a bogus implementation of Deflate in the past and send a raw Deflate stream without the Zlib header and checksum, Billy Hoffman explained this blunder and why nowadays only Gzip is used: http://zoompf.com/blog/2012/02/lose-the-wait-http-compression .
Why not course correct now and pick the simpler Deflate over Gzip?

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Apr 2014 14:27:59 -0700

> My $0.02: why pick Gzip over Deflate?

We already had this discussion for Content-Encoding and concluded
(narrowly) that gzip was preferred.  It's probably not a good idea to
reopen that discussion, and a little consistency would be good.

---

From: <[email-masked]>
Date: Fri, 4 Apr 2014 22:18:25 +0000

See responses in-line below...

On 04 April 2014 23:20, [email-masked] wrote:
> The origin server sets Content-Encoding and is able to make an informed
> decision about what content is compressed.  Intermediates are not supposed
> to change that.  But they *can* change Transfer-Encoding while still not
> knowing whether the content should be compressed.
>
On 04 April 2014 22:41, Martin Thomson Wrote:
>> If there are specific security issues related to gzip then they apply
>> equally to C-E gzip (which is implicitly required for clients to support
>> already). As we mentioned below, one of the benefits of using
>> TE/Transfer-Encoding over a framing layer transfer coding mechanism is
>> that the decision to TE compress can be made at the same level as deciding
>> to CE compress.
>

In response to both of you, I am either totally missing something or these security concerns related to an intermediary adding compression without contextual information are dubious.

The security concerns all relate to compression within a TLS/SSL tunnel, but:
 1) In a secure tunnel, the origin server is the only one that can apply a transfer encoding.  Intermediaries only see encrypted content and *can't change the transfer coding*.
 2) In the case of plain-text communication it doesn't matter if the intermediary adds or removes a transfer encoding.

The framing layer proposal, however, suffers from this security problem, even at the origin server, because the framing layer is likely unaware of the necessary contextual information.

> I don't agree.  Both are considerably simpler.

I disagree with your disagreement :)

Implementation requirements of TE/Transfer-Coding...
Servers: None if they don't want to support TE. To support TE: logic in parallel with the existing C-E logic to decide which type to use (i.e. whether to send the C-E or TE header) - use the same compression code as for C-E
Clients: if (is_content_encoding_gzip || is_transfer_encoding_gzip) decompress();

Implementation requirements of framing layer transfer codings:
Servers: At a minimum support a new setting. To fully support transfer coding: support new setting (including settings table for acceptable encodings); add logic at the framing layer to decide if content should be compressed; add flag and encoding bytes for each data frame
Clients: support a new setting; add parsing logic for flag/encoding; add logic at the framing layer to decompress content if necessary;
BTW, as the framing layer proposal now stands, the transfer coding compression can go in either direction so both sides would require the extra logic above for servers/clients.

On 04 April 2014 22:23, Martin Thomson wrote:
> ...we've removed hop-by-hop headers from HTTP/2.  I don't think

Why not? (Please be more specific.) I get why the other Connection hop-by-hop headers don't make sense, but why are transfer-encoding headers such a bad idea? We gave a list of 5+ benefits of using TE/Transfer-Encoding.

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 05 Apr 2014 20:37:59 +1300

On 5/04/2014 11:18 a.m., [email-masked] wrote:
>> The origin server sets Content-Encoding and is able to make an
>> informed decision about what content is compressed.  Intermediates
>> are not supposed to change that.  But they *can* change
>> Transfer-Encoding while still not knowing whether the content
>> should be compressed.
>> 
>>> If there are specific security issues related to gzip then they
>>> apply equally to C-E gzip (which is implicitly required for
>>> clients to support already). As we mentioned below, one of the
>>> benefits of using TE/Transfer-Encoding over a framing layer
>>> transfer coding mechanism is that the decision to TE compress can
>>> be made at the same level as deciding to CE compress.
>> 
>> entities applying T-E are likely to lack the contextual
>> information required to make the right decisions regarding safe use
>> of compression.
> 
> In response to both of you, I am either totally missing something or
> these security concerns related to an intermediary adding compression
> without contextual information are dubious.
> 
> 1) In a secure tunnel, the origin server is the only one
> that can apply a transfer encoding.  Intermediaries only see
> encrypted content and *can't change the transfer coding*.

This is incorrect. It assumes that the secure tunnel is end-to-end
between user agent and origin. Tunnels may only exist as secured hops
which are beyond the knowledge of the origin AND user agent.
 This is a common scenario for Squid proxies tunneling between corporate
POP.

In such cases the endpoints could choose CE compression and unknowingly
cause the traffic to face all the very same vulnerabilities people are
being so worried about for TE.
With TE present the proxy endpoints for the tunnel are in a position to
remove compression (or not) as applicable to the security needs of that hop.

> 2) In the
> case of plain-text communication it doesn't matter if the
> intermediary adds or removes a transfer encoding.
> 
> The framing layer proposal, however, suffers from this security
> problem, even at the origin server, because the framing layer is
> likely unaware of the necessary contextual information.
> 

Which contextual information exactly? I've been following these threads
and only seen mention of issues compressing values from *different*
headers or requests using the same context causing actual crossover
vulnerabilities.

 There is a bunch of contextual information delivered in the HEADERS
which the intermediaries are very well able to be aware of. For example;
Content-Type/Content-Encoding to prevent double-encoding, and the URI
scheme (if not the transport layer itself) to prevent compression over TLS.

 As mentioned above there are hop-specific security contexts which the
endpoints are very much unaware of that may cause the CE vs TE security
decision to backfire.

Locking the protocol into a end-to-end decision on the assumption that
not all implementations will get security right seems to be a bad idea.
Make each *able* to secure easily with encouragement to do so properly.

>> On 4 April 2014 11:41, K.Morgan wrote:
>>> The proposals by Matthew and Mark get the job done, but they
>>> strike us as more complex than TE/Transfer-Encoding...
>> 
> Implementation requirements of TE/Transfer-Coding... Servers: None if
> they don't want to support TE. To support TE: logic in parallel with
> the existing C-E logic to decide which type to use (i.e. whether to
> send the C-E or TE header) - use the same compression code as for
> C-E Clients: if (is_content_encoding_gzip ||
> is_transfer_encoding_gzip) decompress();
> 
> Servers: At a minimum support a new setting. To fully support
> transfer coding: support new setting (including settings table for
> acceptable encodings); add logic at the framing layer to decide if
> content should be compressed; add flag and encoding bytes for each
> data frame Clients: support a new setting; add parsing logic for
> flag/encoding; add logic at the framing layer to decompress content
> if necessary; BTW, as the framing layer proposal now stands, the
> transfer coding compression can go in either direction so both sides
> would require the extra logic above for servers/clients.

In the framing proposal each intermediary endpoint can arbitrarily
decode and re-encode individual DATA frames using an encoding suitable
for the hop it is being transfered over. The only requirement is that
the encoding be accepted by the recipient.

> Why not? (Please be more specific.) I get why the other Connection
> hop-by-hop headers don't make sense, but why are transfer-encoding
> headers such a bad idea? We gave a list of 5+ benefits of using
> TE/Transfer-Encoding.

Using hop-by-hop headers would require one of two things:

 1) if present in the HEADERS/CONTINUATION frames. That would make it
mandatory to implement HEADERS decoding and re-encoding at each hop.
Including in blind relays which would otherwise just handoff the frames
untouched.

 2) if a new type of frame to be added just for these headers. That is a
blocking change on the draft while we re-hash all the decisions made
about HEADERS.

Hop-by-hop details are framing layer details in any event and mixing
them with headers again is a step backwards in efficiency.

#1 would also prevent adoption of security features such as signed
HEADERS frames which might be used on an HTTP/2 end-to-end connection to
verify no transformations are taking place. The present draft leaves
that option open for future use.

Amos

---

From: Roland Zink <[email-masked]>
Date: Sat, 05 Apr 2014 21:20:54 +0200

+1

I don't care much on how the compression is signaled.

On 04.04.2014 20:41, [email-masked] wrote:
>
> The proposals by Matthew and Mark get the job done, but they strike us 
> as more complex than TE/Transfer-Encoding...
>
>      2.  Since TE and Transfer-Encoding are headers, the decision to 
> compress can be made closer to the application layer.
>
>           c. This also covers Martin’s concern that “an entity 
> performing compression has knowledge of what is being compressed and 
> is therefore able to more accurately determine whether compression is 
> safe to use.”
>
>      3. Retains the extensibility to support new compression 
> algorithms (with all the same benefits Matthew mentioned in [1]).
>
>      4. With TE/Transfer-Encoding, only clients have a new 
> requirement. Servers that don’t want to support Transfer-Encoding 
> don’t have to do anything except ignore TE headers.
>
>      5. Since TE/Transfer-Encoding is already part of the HTTP/1.1 
> spec, 1.1 clients could benefit from this compression when talking to 
> a 1.1<->2 intermediary by implementing TE/Transfer-Encoding; relieving 
> intermediaries of the burden to decompress; and save bandwidth.
>
> *9.4 Transfer-Encoding*
> HTTP/2 retains the semantics of HTTP/1.1 regarding transfer codings 
> (see [HTTP-p1] <http://http2.github.io/http2-spec/#HTTP-p1>, Section 4 
> <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/26/p1-messaging.html#transfer.codings>), 
> with the following additional requirements. Servers MUST NOT use 
> chunked transfer coding. Clients MUST support “gzip” transfer coding. 
> If the "TE" request header field-value is empty or if no TE field is 
> present, the only acceptable transfer coding is "gzip".
>
> In earlier threads, Roy Fielding gave his support for compression 
> transfer codings, including implicitly requiring clients to understand 
> (decompress) gzip transfer encoding (our apologies to Roy for quoting 
> him so often):
>
> /“I would not require [gzip transfer coding] to be used -- just make 
> it possible to be used and required to understand (decompress by 
> recipient).  This is what you already do with HTTP/2…”/ [2]
>
> //
>
> /“… it isn't virtually impossible to introduce standard transfer 
> codings.  It just requires effort by browsers to support one. It also 
> isn't necessary to restrict HTTP/2 features to what a current browser 
> supports.”/ [2]
>
> //
>
> /“Transfer compression has limited deployment but actually does work 
> when it is deployed by consenting adults.  You can change that 
> deployment in HTTP/2. It is well within scope.”/ [3]
>
> This email message is intended only for the use of the named 
> recipient. Information contained in this email message and its 
> attachments may be privileged, confidential and protected from 
> disclosure. If you are not the intended recipient, please do not read, 
> copy, use or disclose this communication to others. Also please notify 
> the sender by replying to this message and then delete it from your 
> system.
>

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 15 Apr 2014 02:51:45 -0600

> without contextual information are dubious.

+1 because these security concerns...

>
>  2) In the case of plain-text communication it doesn't matter if the

...fail to universally apply.

> 
> I get why the other Connection hop-by-hop headers don't make sense,
> but why are transfer-encoding headers such a bad idea? We gave a list
> of 5+ benefits of using TE/Transfer-Encoding.
> 

The response to which seemed to be based on opinion rather than
computer science. Maybe because my take on this whole issue is it's
based on the same resource-representation debate I thought had been
settled. If resources have representations, then it's bad protocol
design for the C-E header to conditionally serve as transfer- or
content- encoding in a fashion that's opaque to intermediaries, thus
requiring them to be authoritative. Or as Roy said, "HTTP/2 should focus
on making features self-descriptive."

-Eric