Subject: #445: Transfer-codings

From: Matthew Kerwin <[email-masked]>
Date: Fri, 4 Apr 2014 12:39:12 +1000

Here is my proposal for reintroducing transport-level compression:

6.1 DATA

New DATA Frame Payload:

```
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | Pad High? (8) |  Pad Low? (8) |
 +---------------+---------------|
 |        Encoding? (16)         |
 +-------------------------------+-------------------------------+
 |                            Data (*)                         ...
 +---------------------------------------------------------------+
 |                           Padding (*)                       ...
 +---------------------------------------------------------------+
```

With the description:

```
Encoding: A 16 bit identifier which describes the encoding that has been
applied to the Data field. This field is optional and is only present if
the ENCODED flag is set.  A sender MUST NOT apply an encoding that has not
first been advertised by the peer in a SETTINGS_ACCEPT_DATA_ENCODING
settings frame, or was advertised with a rank of 0. Endpoints that receive
a frame with an encoding identifier they do not support MUST treat this is
a connection error of type PROTOCOL_ERROR.
```

And a new flag:

```
ENCODED (0x20): Bit 6 being set indicates that the Encoding field is
present and describes the encoding that has been applied to the Data field.
```

This is intentionally designed to make it easy for people to, for example,
detect the flag and immediately PROTOCOL_ERROR.  Hopefully it's not too
much skin off anyone's nose.

6.5.2 Defined SETTINGS Parameters

New parameter:

```
SETTINGS_ACCEPT_DATA_ENCODING (5): Indicates the sender's ability and
willingness to receive DATA frames that are encoded using the scheme
identified in the Value.

The Value field is further divided into two sub-fields, an unsigned 16 bit
encoding identifier and an unsigned 16-bit rank.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |         Encoding (16)         |           Rank (16)           |
 +-------------------------------+-------------------------------+

The following encodings are defined:

  ENCODING_COMPRESS (1):
   The "compress" coding is an adaptive Lempel-Ziv-Welch (LZW) coding that
is commonly produced by the UNIX file compression program "compress".

  ENCODING_DEFLATE (2):
   The "deflate" coding is a "zlib" data format [RFC1950] containing a
"deflate"
compressed data stream [RFC1951] that uses a combination of the Lempel-Ziv
(LZ77) compression algorithm and Huffman coding.

  ENCODING_GZIP (3):
   The "gzip" coding is an LZ77 coding with a 32 bit CRC that is commonly
produced by the gzip file compression program [RFC1952].

An endpoint MAY ignore a SETTINGS_ACCEPT_DATA_ENCODING parameter with an
encoding identifier it does not recognise or support.

The rank fulfils the same role as in the HTTP/1.1 TE header. The rank value
is an integer in the range 0 through 65,535, where 1 is the least preferred
and 65,535 is the most preferred; a value of 0 means "not acceptable".
```

The three encoding schemes are taken straight from HTTPbis-p1 section 4.2.
 Again, it's designed specifically so that people can ignore it if they
don't want to play along; but it grants us adventurous folk the chance to
play.

8.1.3.5 Malformed Messages

Clarify the content-length:

```
A request or response that includes an entity body can include a
content-length header field. A request or response is also malformed if the
value of a content-length header field does not equal the sum of the DATA
frame payload lengths, after any encoding is removed, that form the body.
```

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 04 Apr 2014 17:02:51 +1300

> Here is my proposal for reintroducing transport-level compression:

+1 liking this. I was on the way to submitting much the same proposal.

Some suggested adjustments inline...

> New DATA Frame Payload:

Causing this "ENCODING_ZLIB (2)" instead to clarify and avoid the issues
brought up earlier about two "deflate" versions causing confusion.

> and 65,535 is the most preferred; a value of 0 means "not acceptable".

rank == q-value yes? so retaining similarity with HTTP/1.1 by calling it
a q-value would be better.

> frame payload lengths, after any encoding is removed, that form the body.

Amos

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 4 Apr 2014 14:26:58 +1000

> a q-value would be better.

The HTTP/1.1 text I was ripping off says:

   When multiple transfer codings are acceptable, the client MAY rank
   the codings by preference using a case-insensitive "q" parameter
   (similar to the qvalues used in content negotiation fields, Section
<http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26#section-5.3.1>
   5.3.1 <http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26#section-5.3.1>
of [Part2 <http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26#ref-Part2>]).
 The rank value is a real number in the range 0
   through 1, where 0.001 is the least preferred and 1 is the most
   preferred; a value of 0 means "not acceptable".

and the associated BNF is:

     t-codings = "trailers" / ( transfer-coding [ t-ranking ] )
     t-ranking = OWS ";" OWS "q=" rank
     rank      = ( "0" [ "." 0*3DIGIT ] )
                / ( "1" [ "." 0*3("0") ] )

That said, I found it by searching the text for "q".

---

From: Martin Thomson <[email-masked]>
Date: Thu, 3 Apr 2014 21:34:24 -0700

> willingness to receive DATA frames that are encoded using the scheme

Settings have a single value.  Is the intent here to change this, or
are you planning to have just a single valid acceptable encoding?

I'm guessing the former, based on the existence of the rank part here.
 In that case you will need to explain how values are processed, and
how an implementation is able to limit the storage it dedicates to
storage of this new setting.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2014 16:09:08 +1100

My personal .02 - 

Given that HTTP/1 hasn't seen much use of transfer-codings, and over the years we've had them, there have only been three defined (discounting chunked), spending 16 bits on this in ever data frame doesn't seem justified.

I'd much rather just have a flag that indicates 'gzip', and a corresponding setting; that doesn't require any frame format changes at all. If another encoding becomes necessary, it can get into a subsequent version (since we've repeatedly decided to favour faster versioning over broad extensibility in the non-semantic layer).

Also, how should a recipient handle a stream that has DATA frames with different values for encoding?

Regards,

On 4 Apr 2014, at 1:39 pm, Matthew Kerwin <[email-masked]> wrote:

> Encoding: A 16 bit identifier which describes the encoding that has been applied to the Data field. This field is optional and is only present if the ENCODED flag is set.  A sender MUST NOT apply an encoding that has not first been advertised by the peer in a SETTINGS_ACCEPT_DATA_ENCODING settings frame, or was advertised with a rank of 0. Endpoints that receive a frame with an encoding identifier they do not support MUST treat this is a connection error of type PROTOCOL_ERROR.
> ```
> 
> And a new flag:
> 
> ```
> ENCODED (0x20): Bit 6 being set indicates that the Encoding field is present and describes the encoding that has been applied to the Data field.
> ```
> 
> This is intentionally designed to make it easy for people to, for example, detect the flag and immediately PROTOCOL_ERROR.  Hopefully it's not too much skin off anyone's nose.
> 
> 
> 
> SETTINGS_ACCEPT_DATA_ENCODING (5): Indicates the sender's ability and willingness to receive DATA frames that are encoded using the scheme identified in the Value.
> 
> The Value field is further divided into two sub-fields, an unsigned 16 bit encoding identifier and an unsigned 16-bit rank.
> 
>  0                   1                   2                   3
>    The "compress" coding is an adaptive Lempel-Ziv-Welch (LZW) coding that is commonly produced by the UNIX file compression program "compress".
> 
>   ENCODING_DEFLATE (2):
>    The "deflate" coding is a "zlib" data format [RFC1950] containing a "deflate" compressed data stream [RFC1951] that uses a combination of the Lempel-Ziv (LZ77) compression algorithm and Huffman coding.
> 
>   ENCODING_GZIP (3):
>    The "gzip" coding is an LZ77 coding with a 32 bit CRC that is commonly produced by the gzip file compression program [RFC1952].
> 
> An endpoint MAY ignore a SETTINGS_ACCEPT_DATA_ENCODING parameter with an encoding identifier it does not recognise or support.
> 
> The rank fulfils the same role as in the HTTP/1.1 TE header. The rank value is an integer in the range 0 through 65,535, where 1 is the least preferred and 65,535 is the most preferred; a value of 0 means "not acceptable".
> ```
> 
> The three encoding schemes are taken straight from HTTPbis-p1 section 4.2.  Again, it's designed specifically so that people can ignore it if they don't want to play along; but it grants us adventurous folk the chance to play.
> 
> 
> A request or response that includes an entity body can include a content-length header field. A request or response is also malformed if the value of a content-length header field does not equal the sum of the DATA frame payload lengths, after any encoding is removed, that form the body.
> ```
> 
> Cheers
> -- 
>   Matthew Kerwin

--
Mark Nottingham   http://www.mnot.net/

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 4 Apr 2014 15:21:19 +1000

> storage of this new setting.

Yes, multiple values. I didn't think it would be necessary to tell
implementors how to process or store the values. I guess I could add
something to the following effect:

Have a table of encodings you are able to transmit, with ranks initialised
to 0. When you receive a this setting, if it's for one of the encodings in
your table, update the rank in the table from the frame; if it's not,
ignore it. You can use your table to help inform your decision of what
encoding, if any, to apply to any DATA frames.

Does that not count as an implementation detail?

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 4 Apr 2014 15:52:24 +1000

On 4 April 2014 15:09, Mark Nottingham <[email-masked]> wrote:

> My personal .02 -
>
> Given that HTTP/1 hasn't seen much use of transfer-codings, and over the
> years we've had them, there have only been three defined (discounting
> chunked), spending 16 bits on this in ever data frame doesn't seem
> justified.
>

To be honest, the size is a completely negotiable detail. I went with 16
because the SETTINGS frame has a 32-bit Value. Old-school q-values have
1000 effective values, which needs at least 10 bits to encode; and 16 bits
is a convenient word size >=10. That leaves 16 bits in the Value, so rather
than mandate a bunch of "must be 0, fail if they're not" bits, I just used
all 16 for the identifier.

If that results in too much overhead in the DATA frames, I have no qualms
at all about changing the size, and the packing in SETTINGS. Though surely
the compression would already compensate for the extra two bytes; otherwise
why bother encoding in the first place?

I'd much rather just have a flag that indicates 'gzip', and a corresponding
> setting; that doesn't require any frame format changes at all. If another
> encoding becomes necessary, it can get into a subsequent version (since
> we've repeatedly decided to favour faster versioning over broad
> extensibility in the non-semantic layer).
>

My only issue with this is experimentation. I wouldn't want to have to
push out a whole new HTTP version just to support, say, bzip2 or LZX or xz
compression. Rather, we could allocate a range of encoding values as
"private use" or "experimental", say 32,768 to 65,535, and I could
configure my endpoints to treat 32,999 as xz, without having to change any
deeper handling of flags or frame formats or anything.

Also, how should a recipient handle a stream that has DATA frames with
> different values for encoding?

By decoding each as it arrives, the same way it would handle a stream with
the same encoding every frame. Two gzip'd frames don't add together to
make one big gzip file.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 3 Apr 2014 23:45:54 -0700

On 3 April 2014 22:21, Matthew Kerwin <[email-masked]> wrote:
> your table, update the rank in the table from the frame; if it's not, ignore
> it. You can use your table to help inform your decision of what encoding, if
> any, to apply to any DATA frames.
>

That's not the point.  The point is that an endpoint needs to be able
to limit the state commitment it makes.  Based on this, the commitment
could be as much as 256K for these settings.

---

From: Ilari Liusvaara <[email-masked]>
Date: Fri, 4 Apr 2014 14:06:38 +0300

> the same encoding every frame. Two gzip'd frames don't add together to

Are the compression contexts chained? If so, using multiple compressions
one could use some serious memory (especially as window sizes are not
limited).

bzip2 -9 neeeds ~3700kB to decode. xz -9 needs ~65MB(!).

-Ilari

---

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 04 Apr 2014 15:57:09 +0200

On Fri, 04 Apr 2014 07:09:08 +0200, Mark Nottingham <[email-masked]> wrote:

> I'd much rather just have a flag that indicates 'gzip', and a  
> corresponding setting; that doesn't require any frame format changes at  
> all. If another encoding becomes necessary, it can get into a subsequent  
> version (since we've repeatedly decided to favour faster versioning over  
> broad extensibility in the non-semantic layer).

Or you can have a per data frame flag as you propose, but define the  
actual compression algorithm applied in SETTINGS.

    SETTINGS_COMPRESSION_ALGORITHM (5):  Declares the compression
       algorithm used for all data frames with the COMPRESSION frame
       flag set. The value is a 4 octets identifer where the following
       identifiers are defined.

       "gzip"  The content is encoded according to GZIP [RFC1952], with
               one compression context per stream.

       If an unknown compression algorithm identifier is encountered
       the connection must be closed with UNKNOWN_COMPRESSION error.

I'm not sure it's needed though.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Apr 2014 09:27:53 -0700

I share Mark's concerns, but have another to add.

The security properties of compression have been highly scrutinized in
the past couple of years.  Compression oracles on the web have been
used in attacks that go beyond theoretical.

In general, the trend has been to avoid generic compression and move
to more context-aware compression.  That means that the entity
performing compression has knowledge of what is being compressed and
is therefore able to more accurately determine whether compression is
safe to use.  This can be difficult, but it is the only way to ensure
that using compression does not result in information being leaked.

This is a step away from that.  A hop-by-hop compression feature like
this will be applied without adequate knowledge of the consequences.

I don't think that we should be building features like this,
particularly when there are better alternatives available.

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 5 Apr 2014 03:50:17 +1000

> could be as much as 256K for these settings.

Or as little as zero. Or, if they do support gzip, nothing stops them
interpreting any >0 rank as 1, and remembering 1 bit per thingy. I don't
understand what you're getting at.

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 5 Apr 2014 03:52:55 +1000

> one could use some serious memory (especially as window sizes are not

No, each DATA is encoded independently. I guess that wasn't clear. Isn't
that how T-E worked in 1.1?

---

From: Ilari Liusvaara <[email-masked]>
Date: Fri, 4 Apr 2014 21:33:16 +0300

> that how T-E worked in 1.1?

No. In HTTP/1.1, gzip (just as example) T-E works in either of two ways:

- Gzip the whole response and then chunk the gzipped stream.
- Gzip the whole response, send result as-is and close the connection on
  end.

Also, HTTP/1.1 T-E supports multiple transforms (sometimes useful for e.g.
applying transforms that enhance compressiblity):

Transfer-Encoding: foo, gzip, chunked

(If foo is some filter that tries to make data more compressible by gzip).
The dataflow goes (raw data) -> foo -> gzip -> chunk -> (output)

-Ilari

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 5 Apr 2014 04:42:02 +1000

> this will be applied without adequate knowledge of the consequences.

Would more words fix that? e.g. advice for intermediaries not to compress
packets they didn't recieve compressed? (And for Apache not to blindly
compress everything it receives from PHP.)

The current semantics of HTTP force this type of compression (i.e. not at
the 'entity' level) to be at the transport level, and thus hop-by-hop. That
doesn't have to mean that the compression is applied naively hop-by-hop.

> particularly when there are better alternatives available.

This sounds like a snark, but it's not: what is a better alternative here?
C-E is an alternative, but it has deficiencies.

My goal with this proposal was partly to support compression on range
requests, and partly to de-emphasise the current practice of on-the-fly
C-E. To that end, I don't know of any alternatives at all.

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 5 Apr 2014 04:50:42 +1000

> - Gzip the whole response and then chunk the gzipped stream.

Oh, right. I guess the difference here is that, because each DATA has its
own headers (flags), I've turned this from gzip+chunked to something like
multipart+gzip.

Now I understand some of the confusion.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 04 Apr 2014 15:32:53 -0400

-1 on this.

First, why a separate flag when you can just look at a couple bytes in the DATA?

Second, what about all of the security considerations that have been brought up?

And finally, HTTP/2 is not supposed to introduce new semantics.  If you have TE: gzip/compress/zlib, then that needs to be proposed separately since it applies equally to HTTP/1.1 and HTTP/2.

> Here is my proposal for reintroducing transport-level compression:

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 5 Apr 2014 05:45:41 +1000

Apologies for replying to myself.

tl;dr: I'm happy to drop transport-level encoding for HTTP/2 as long as
HTTP/3 picks it up straight away, and as long as the transition to HTTP/2
makes the future transition to HTTP/3 much easier.

> The current semantics of HTTP force this type of compression (i.e. not at
the 'entity' level) to be at the transport level, and thus hop-by-hop.

I've had a thought (note that it's a 5am thought, after waking up at 3:30
for no good reason): introduce, outside of HTTP/2, a new semantic layer,
below the entity but above transport. Call it end-to-end transport.

It would include new headers for negotiating compression, almost exactly
like TE and Transfer-Encoding, except that the encoding can only be applied
by the origin. It would also introduce a new response header
"Transfer-Length" which holds the length of the _encoded_ payload.

Intermediaries could alter the headings the usual ways (stripping things
from TE; maybe adding things to TE as long as they decode before
retransmitting). Just not ever *encoding*.

Entity-specific metadata (e.g. ETag, Content-Length) and operations (e.g.
Range) are not affected by the encoding. I'm not sure, but I think it
doesn't affect Vary either.

For example, a client may request the range bytes 1024 to 2045 and receive
a 206 with Content-Length 1024, but a Transfer-Length of 300 and 300 bytes
of gzip'd data which, unzipped, holds the full KB range.

Since this is one of those things where participants have to explicitly opt
out (a caching proxy that ignores and forwards these headers will get
confused) I guess this is an idea to hold over for HTTP/3.

If we think that it will be relatively easy to get /3 going once /2 is out,
I'd be happy to put this entire discussion on hold until then, as long as
we're sure to address it soon.

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 5 Apr 2014 06:00:50 +1000

> First, why a separate flag when you can just look at a couple bytes in
the DATA?

Which bytes? Are you suggesting magic number heuristics? Or extra metadata
packed into DATA frames?

> Second, what about all of the security considerations that have been
brought up?

If they're the same as the ones Martin mentioned earlier, see my reply
there.

> And finally, HTTP/2 is not supposed to introduce new semantics.  If you
have TE: gzip/compress/zlib, then that needs to be proposed separately
since it applies equally to HTTP/1.1 and HTTP/2.

No, the only time TE applies equally to HTTP/1.1 and HTTP/2 is if a 1.1<->2
gateway blindly forwards transfer-encoded data; but they're not even meant
to do that in the 1.1<->1.1 case. TE is a hop-by-hop header.

HTTP/2 is explicitly supposed to alter transport, and TE is a transport
issue. That's why it's ok (but deficient) that the current draft alters
what's allowed in TE headers.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Apr 2014 13:29:14 -0700

> compress everything it receives from PHP.)

I don't think that would be a good idea.  It would end up being an RFC
6919 "MUST (BUT WE KNOW YOU WONT)".  The point being that if you build
a feature that enables intermediary compression, it will be used for
intermediary compression.

> doesn't have to mean that the compression is applied naively hop-by-hop.

It would be naive to think that it wouldn't be applied naively.

> C-E is an alternative, but it has deficiencies.

I think that - on balance - Content-Encoding is a better alternative.

> requests, and partly to de-emphasise the current practice of on-the-fly C-E.
> To that end, I don't know of any alternatives at all.

I don't want to sound too negative about range requests, but it may be
that many of the cases that depend on range requests would be better
served by giving the things that you are pushing across the network
actual names, then using content encoding.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 04 Apr 2014 16:47:29 -0400

Matthew,

On Apr 4, 2014, at 4:00 PM, Matthew Kerwin <[email-masked]> wrote:
> Which bytes? Are you suggesting magic number heuristics? Or extra metadata packed into DATA frames?
> 
If I understood your proposal correctly, your TE field is always in bytes 4-5 of the DATA frame.
> ...
> No, the only time TE applies equally to HTTP/1.1 and HTTP/2 is if a 1.1<->2 gateway blindly forwards transfer-encoded data; but they're not even meant to do that in the 1.1<->1.1 case. TE is a hop-by-hop header.
> 
Given some of the security concerns that have been voiced on this list, I think it would be reasonable for a gateway to forward gzip'd data provided that the other end supports it, but not to introduce gzip since the gateway probably lacks the context needed to know it is safe to compress.
> HTTP/2 is explicitly supposed to alter transport, and TE is a transport issue. That's why it's ok (but deficient) that the current draft alters what's allowed in TE headers.
> 
The current draft stops the use of "TE: chunked" because all HTTP/2 data is chunked.  That doesn't alter semantics or pose additional security considerations, and there is a clean mapping to/from HTTP/1.1.

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 5 Apr 2014 07:29:40 +1000

On Apr 5, 2014 6:47 AM, "Michael Sweet" <[email-masked]> wrote:
> Matthew Kerwin:
>> Michael Sweet:

"This field is optional and is only present if the ENCODED flag is set."

Otherwise everyone would pay a 2-byte tax for something only a few use.

>>
>> No, the only time TE applies equally to HTTP/1.1 and HTTP/2 is if a
1.1<->2 gateway blindly forwards transfer-encoded data; but they're not
even meant to do that in the 1.1<->1.1 case. TE is a hop-by-hop header.
>
> Given some of the security concerns that have been voiced on this list, I
think it would be reasonable for a gateway to forward gzip'd data provided
that the other end supports it, but not to introduce gzip since the gateway
probably lacks the context needed to know it is safe to compress.

Indeed. I guess that since -- as Ilari and mnot pointed out -- this differs
from TE in the compression contexts, it makes it hard for a 2->1.1 gateway,
and even harder for a 1.1->2. That's an issue with this propsal, which is
hard to address nicely.

>
> The current draft stops the use of "TE: chunked" because all HTTP/2 data
is chunked.  That doesn't alter semantics or pose additional security
considerations, and there is a clean mapping to/from HTTP/1.1.

The current draft forbids all TE but "trailers", and I think that one's
allowed because it makes gateways' lives bearable. That's one kind of clean
mapping, I suppose.

---

From: Roland Zink <[email-masked]>
Date: Sat, 05 Apr 2014 00:43:49 +0200

> It would be naive to think that it wouldn't be applied naively.
It is naive to think that Content-Encoding will not be applied 
hop-by-hop. Even though TE is hop-by-hop this doesn't mean that the 
compression and recompression must be done on each hop. If support is 
mandated then it should be sufficient to forward the pieces unmodified.
> I think that - on balance - Content-Encoding is a better alternative.
Don't think so, except it is in use.
> actual names, then using content encoding.
Actually it can be seen as a problem that by mandating content encoding 
support in the client the server decides on whether the client can use 
range requests for seeks, regardless if the client would prefer seek 
over CE. On the application layer this can be solved with some effort by 
splitting big entities on the server side and ship small pieces with 
every request, one example would be HLS compared to mp4. Such things can 
be done but come with their own complications.

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 05 Apr 2014 19:05:38 +1300

> I think that - on balance - Content-Encoding is a better alternative.

For some uses. For others TE in this proposal is better.

Content-Encoding suffers from the same weakness that allowed the Enigma
to be decrypted. The data type of each object is exposed in headers and
each data type has a known structure and commonly also a mime magic
prefix. This exposes it to decryption by comparing the known value
against encoded form. The entire object is encoded with the same context
allowing more fruit for picking.

In this proposal each DATA is able to be compress (or not) selectively,
or cycle through varous of the negotiated encodings on the DATA frames
by sending different encoding field value on each.
This encoding is also independent of the object cached by proxies. So
repeated requests for its URL do not necessarily produce the same
encoded response.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 05 Apr 2014 19:16:58 +1300

> Otherwise everyone would pay a 2-byte tax for something only a few use.

Unless that tax were perhapse the incentive for implementations to
encode data more securely than plain identity encoding? while still
allowing identity encoding (value 0x0000) to be used.

> and even harder for a 1.1->2. That's an issue with this propsal, which is

Not much more so than with the current draft. Gateways are forced to
uncompress and send as identity+chunked in HTTP/2 at present.

Also, why would a 2->1.1 gateway go to the trouble of compressing ?

The presence of a HTTP/2 hop anywhere in the path is likely to result in
*less* usage of compressed data. More bandwidth waste overall.

Amos

---

From: Albert Lunde <[email-masked]>
Date: Sat, 05 Apr 2014 04:32:20 -0500

On 4/4/2014 12:09 AM, Mark Nottingham wrote:
> Also, how should a recipient handle a stream that has DATA
> frames with different values for encoding?

In the context of that question, we must bear in mind that generic
stream-oriented compression like gzip is going to have state. I don't 
see how it could be applied to some DATA frames in a stream but
not others, without a lot of bookkeeping. I don't see how that
would serve a useful purpose.

---

From: Roland Zink <[email-masked]>
Date: Sat, 05 Apr 2014 21:18:42 +0200

Doing it on each data frame should decrease what you can save with 
compression. At the same time it doesn't require a state between frames 
and it should solve the security concerns, doesn't it?

Roland

On 04.04.2014 20:50, Matthew Kerwin wrote:
>
> On Apr 5, 2014 4:33 AM, "Ilari Liusvaara" <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >
> Oh, right. I guess the difference here is that, because each DATA has 
> its own headers (flags), I've turned this from gzip+chunked to 
> something like multipart+gzip.
>

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 7 Apr 2014 14:51:33 +1000

> allowing identity encoding (value 0x0000) to be used.

I don't know that it works that way. I think it'd be more likely to end up
a tax on our ears, from everyone complaining about the overhead.  Although
I do like the idea of having an explicit identity coding, in case someone
wants to give it a zero (or other low) rank in their settings.

> >
> > Michael Sweet <[email-masked]> wrote:
> >
 >

> Also, why would a 2->1.1 gateway go to the trouble of compressing ?

I think I was comparing the per-DATA compression context to 1.1's single
megacontext; if we had the latter, and everyone was playing along, then the
gateway's job could be a lot easier in either direction by just forwarding
blindly. However compared to the current draft (with no compression on the
2-side) then yeah, I don't think this proposal is adding too much work for
anyone.

Especially since the minimum effort for compliance is just not barfing on a
SETTINGS frame of type 5, and making sure DATA FLAG bit 6 is clear.

---

From: Roberto Peon <[email-masked]>
Date: Sun, 6 Apr 2014 22:04:29 -0700

I still don't think compression at the protocol/stream layer makes sense.
In my experience, it never worked well in our (SPDY) experiments: it added
complexity, pened the door for lots of DoS vulnerabilities, increased
memory requirements, increased CPU requirements, and rarely helped w.r.t.
bandwidth for well-constructed sites which compressed their resources.

The cost/benefit here is extremely dubious.
-=R

On Sun, Apr 6, 2014 at 9:51 PM, Matthew Kerwin <[email-masked]>wrote:

> Especially since the minimum effort for compliance is just not barfing on
> a SETTINGS frame of type 5, and making sure DATA FLAG bit 6 is clear.
>
> --
>   Matthew Kerwin

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 7 Apr 2014 15:29:13 +1000

On 7 April 2014 15:04, Roberto Peon <[email-masked]> wrote:

> bandwidth for
> well-constructed sites which compressed their resources.
> 
>
As I've said, I'm happy to not block HTTP/2 on this, if we can address it
in HTTP/3, and if H3 isn't going to be another 15 years down the track.

But I have to respond to one particular type of comment that keeps coming
up: "*well-constructed sites which compressed their resources.*"

That's a big value judgement on what constitutes good site design. Yes, in
lots of cases it makes sense to compress your resources and have multiple
representations, especially for static resources; but what about the sites
that aren't like that? Why is it bad site design to have a big resource
that can be accessed with ranges? If the answer is because doing so would
require TE in order to have compression, then it's a tautology (TE is only
needed by bad sites; those sites are bad because they need TE). If it's
because caches don't handle that properly, then it's a chicken-and-egg
problem. The only reasons I can think of for calling them bad are either
circular, or "I don't like them." Is there a real reason?

Why should I make my web API use "?start=N&end=M" when I could use "Range:
x-records=N-M" ?

---

From: Roberto Peon <[email-masked]>
Date: Sun, 6 Apr 2014 23:07:00 -0700

I wouldn't mind fixing range-requests, but I don't believe that range
requests make a whole lot of sense for dynamic objects anyway.
I'm more than happy to have my ignorance corrected :)

> in HTTP/3, and if H3 isn't going to be another 15 years down the track.

I wouldn't want it then, either, honestly. I that that having random
intermediaries attempt to apply a transformation to the content that the
origin wouldn't seems like a mistake.

> that can be accessed with ranges?
>

At best it is wasteful to have to generate a large dynamic resource, and
then serve only a portion of it.
At worst, when the dynamic resource isn't properly versioned, it is just
plain wrong.

> If the answer is because doing so would require TE in order to have
> compression, then it's a tautology (TE is only needed by bad sites; those
> sites are bad because they need TE). If it's because caches don't handle
> that properly, then it's a chicken-and-egg problem. The only reasons I can
> think of for calling them bad are either circular, or "I don't like them."
> Is there a real reason?
>

It works better for caches, which can (with little logic) cache the chunks
which are requested more often, and/or purge unneeded resources more
easily/naively, which increases cache hit rate for other resources.
In the case of video/audio, it allows for quality/encoding changes while
still allowing for caching.

-=R

---

From: David Krauss <[email-masked]>
Date: Mon, 7 Apr 2014 14:13:54 +0800

On 2014–04–07, at 1:29 PM, Matthew Kerwin <[email-masked]> wrote:

> That's a big value judgement on what constitutes good site design. Yes, in lots of cases it makes sense to compress your resources and have multiple representations, especially for static resources; but what about the sites that aren't like that? Why is it bad site design to have a big resource that can be accessed with ranges? If the answer is because doing so would require TE in order to have compression, then it's a tautology (TE is only needed by bad sites; those sites are bad because they need TE). If it's because caches don't handle that properly, then it's a chicken-and-egg problem. The only reasons I can think of for calling them bad are either circular, or "I don't like them." Is there a real reason?

Range requests don’t get cached because they are uncommon (not chicken and egg, but because there are few potential applications) and it greatly complicates cache design. The underlying resource is likely to be too large to be stored entirely by the client, so the cache would have to store ranges separately and coalesce them on demand, per query. A single cache miss potentially generates multiple sub-range requests on the network.

Custom, application-specific caching, which may know about specific usage patterns or server performance characteristics, looks better.

> Why should I make my web API use "?start=N&end=M" when I could use "Range: x-records=N-M” ?

Why not accept both? If no cache handles range requests, at least not differently from the query string, there is nothing to gain by being futuristic. You can use your extension when it’s supported, but note the high implementation cost along the entire transport chain. Is it worth it?

A simple adaptation, which surely isn’t my original invention just now, is to store large sub-ranges as virtual files. For example, go by powers of two, so you can download any two megabytes starting at any offset multiple of 2^21, any 64 kilobytes starting at a multiple of 2^16, etc. These are cache-friendly and simple to merge in client-side logic, and they work with content-encoding.

---

From: Helge Hess <[email-masked]>
Date: Mon, 7 Apr 2014 15:41:50 +0200

On Apr 7, 2014, at 8:07 AM, Roberto Peon <[email-masked]> wrote:
> But I have to respond to one particular type of comment that keeps coming up: "well-constructed sites which compressed their resources."
> 
> That's a big value judgement on what constitutes good site design. Yes, in lots of cases it makes sense to compress your resources and have multiple representations, especially for static resources; but what about the sites that aren't like that? Why is it bad site design to have a big resource that can be accessed with ranges?
> 
> At best it is wasteful to have to generate a large dynamic resource, and then serve only a portion of it. 

Maybe I'm off-track here, but aren't PDFs the common example for the usefulness of ranges plus TE?

You query the byte range required to display a specific page from the catalog, and then you would want to compress that for transfer.
I can imagine a set of resources which would work like that.

hh

P.S.: I didn't follow the whole thread, but a significant issue TE solves are ETag's in combination with content-negotiation.

---

From: Michael Sweet <[email-masked]>
Date: Mon, 07 Apr 2014 09:57:23 -0400

Helge,

> Maybe I'm off-track here, but aren't PDFs the common example for the usefulness of ranges plus TE?

PDFs are almost always compressed, typically using Flate for the page and font objects and Flate or JPEG for the image objects.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 9 Apr 2014 00:12:35 +1000

On 7 April 2014 16:13, David Krauss <[email-masked]> wrote:

>
> Range requests don't get cached because they are uncommon (not chicken and
> egg, but because there are few potential applications)
>

... they don't get cached because they're not common, and an argument
against promoting them to commonality is that they're not cached. Maybe
"Catch-22" would have been a better description than "chicken and egg".

> and it greatly complicates cache design. The underlying resource is likely
> to be too large to be stored entirely by the client, so the cache would
> have to store ranges separately and coalesce them on demand, per query. A
> single cache miss potentially generates multiple sub-range requests on the
> network.
>

Just so I can follow this paragraph, are you talking about a browser
cache? Because a client's storage capacity has no bearing on how a caching
proxy stores objects/fragments, or how that cache freshens stale data.

To my mind browsers don't play any part in range discussions, unless it's
in some new-fangled technology like video streaming. Most of the time a
browser wants to receive the whole object so it can evaluate and/or render
it. It doesn't make sense to grab half a page, or a chunk of an image. And
if it is a streaming thing, how often are those actually cached?

The model in my mind is a non-browser application that uses HTTP APIs.

Custom, application-specific caching, which may know about specific usage
> patterns or server performance characteristics, looks better.
>

Why do you assume the application doesn't have any say in the caching? If
the application layer is a javascript app, and the HTTP semantic and
transport layers are provided by a browser, then sure (if XHR requests even
pass through the browser cache). But surely a non-browser HTTP middleware
package would provide either: a super-duper opaque caching mechanism that I
know will suit my application, or an API so I can handle my own cache (like
libcurl does).

An analogy on the response-providing side is content-encoding. It's a HTTP
semantic level thing, purview of the HTTP middleware, but it's something
the application should be managing.

[snip]

> A simple adaptation, which surely isn't my original invention just now, is
> to store large sub-ranges as virtual files. For example, go by powers of
> two, so you can download any two megabytes starting at any offset multiple
> of 2^21, any 64 kilobytes starting at a multiple of 2^16, etc. These are
> cache-friendly and simple to merge in client-side logic, and they work with
> content-encoding.
>

A convenient workaround is still a workaround. Now rather than allowing
the client to request the data it wants, the server dictates what it can
have. Might as well just provide query parameters to define arbitrary
dynamic/lazy-initialised/whatever resources.

The other issue here (in either case) is that the server now has a lot more
entities to manage. And twice as many, if they're all gzip'able. That's a
lot of PUSH_PROMISEs if you update something.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 9 Apr 2014 10:47:53 +1000

On 4 Apr 2014, at 4:52 pm, Matthew Kerwin <[email-masked]> wrote:

> By decoding each as it arrives, the same way it would handle a stream with the same encoding every frame. Two gzip'd frames don't add together to make one big gzip file.

One of the useful attributes of HTTP transfer-codings is that intermediaries who want to avoid re-compressing a stream's payload can do so when both sides support compression, by simply passing it through.

I.e., if we have:

[ client ] ----( hop A: HTTP/2 )----> [ proxy ] ----( hop B: HTTP/1.1 )----> [ origin ]

... and all parties support compression, the proxy really doesn't want to have to re-compress messages because we decided to make compression frame-by-frame in HTTP/2, whereas it's message-by-message in HTTP/1.1.

Isn't it much more likely that intermediaries will actually implement this if they can do that form of pass-through?

Cheers,

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 9 Apr 2014 12:26:24 +1000

On 9 April 2014 10:47, Mark Nottingham <[email-masked]> wrote:

> different
> values for encoding?
> > By decoding each as it arrives, the same way it would handle a stream
> with the same encoding every frame. Two gzip'd frames don't add together to
> make one big gzip file.
>
> One of the useful attributes of HTTP transfer-codings is that
> intermediaries who want to avoid re-compressing a stream's payload can do
> so when both sides support compression, by simply passing it through.
>
> [snip]
>
>
> Isn't it much more likely that intermediaries will actually implement this
> if they can do that form of pass-through?
>
>
That assumes the HTTP/1.1 side is doing T-E, and I'm not sure how common
that currently is. Judging by discussions in on the list, I'd imagine "not
very." Perhaps someone with better traffic visibility could give us some
numbers.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 09 Apr 2014 13:50:34 +0200

* Matthew Kerwin wrote:
>Two gzip'd frames don't add together to make one big gzip file.

(Actually, if the frames are independently gzip'd, then concatenating
them does make one big gzip file. That's one very useful property of the
gzip format.)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 10 Apr 2014 16:16:51 +1000

I've started keeping track of my T-E proposal in github[1], and have been
attempting to come up with ways to address people's concerns.

* The easy issue was the size of identifiers, which I've reduced to 8 bits
in my current text -- i.e. 256 possible values. The worst case commitment
as a result is 256 rank values, which I've also reduced to 8 bits -- so 256
bytes. However as I and others have pointed out, the minimum commitment as
a result of this proposal is nil, and a reasonable realistic value is
probably 1-2 bytes per connection.

* I wanted to keep the rank because I'd like the ability for someone to
change their mind about an encoding. I would be happy with a single bit, if
I could come up with a way to pack it into the settings frame that doesn't
make me feel yucky. The case I'm addressing is a proxy that, having
advertised support for gzip, finds itself being zip-bombed (or possibly
just under an unexpectedly high load) and decides to stop receiving gzip
payloads from downstream.

* I've not included any suggestion of how people could or should use rank
values to determine a preferred encoding. I don't think that's a problem
though; RFC 2703 and RFC 2295/2296 exist if people really need
inspiration. It could also go away if rank is reduced to a single bit.

* An issue I'd like to address is the ability to specify multiple
encodings on a single frame. This could be done by either expanding the
proposed Encoding field in the DATA frame to some fixed multiple (e.g. 4
encodings), or with a length-prefixed or a null-terminated (er,
identity-terminated) list.

* The big issue I'm still trying to get straight in my head is the
objection that compression can reduce security when applied by part of the
transport chain that doesn't have full knowledge of the properties of the
data. The server has no way of signalling that intermediaries shouldn't
apply compression.

   My thoughts on this are that end-to-end encryption is meant to solve
that problem, by making it very hard for intermediaries to modify your
data. Also: the known compression-based vulnerabilities are based on
sharing a single compression context between secret and public data; since
this proposal uses a unique context per compressed frame, and since it's
probably not likely that intermediaries would bother coalescing data
frames, all the server has to do is send the secret and public bits in
separate frames. Not a bullet-proof solution, but not unreasonable.

Does anyone else have a concern that I've not addressed? Or possibly one
that got lost in the volume of the discussion?

[1]:
https://github.com/phluid61/http2-spec/compare/445-transfer-codings#files_bucket

---

From: <[email-masked]>
Date: Thu, 10 Apr 2014 15:58:44 +0000

On Thursday,10 April 2014 08:17, Matthew Kerwin wrote:
> I wanted to keep the rank because I'd like the ability for someone to change their mind about an encoding.
> I would be happy with a single bit, if I could come up with a way to pack it into the settings frame that doesn't make me feel yucky. ...

I'm neutral on whether you use a single value for this setting or keep a table like Amos wants. ( I always slightly lean towards KISS though.)  I just didn't realize there was a desire to dynamically switch between so many options.

Another idea for the setting...

7-bit encoding with 1-bit on/off (128 possible encodings) - you could still pack four encodings into a single setting value
```
  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |F| Encoding(7) |F| Encoding(7) |F| Encoding(7) |F| Encoding(7) |
  +-------------------------------+-------------------------------+
```

The msb in the encoding field ("T") of the DATA frame could be used to signal the presence of an optional (encoding-specific) trailer that signals the offset and length of the uncompressed data captured within the compressed frame.
This might allow intermediaries to quickly index the underlying uncompressed data.
I don't care strongly about this idea, only if the intermediaries think it would be useful. I'd rather keep it simple.

e.g.

```
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | Pad High? (8) |  Pad Low? (8) |T| Encoding?(8)|
 +-------------------------------+-------------------------------+
 |                            Data (*)                         ...
 +---------------------------------------------------------------+
 |                  Uncompressed Offset High? (32)               |
 +---------------------------------------------------------------+
 |                  Uncompressed Offset Low? (32)                |
 +---------------------------------------------------------------+
 |                    Uncompressed Length? (32)                  |
 +---------------------------------------------------------------+
 |                           Padding (*)                       ...
 +---------------------------------------------------------------+
 |                           Padding (*)                       ...
 +---------------------------------------------------------------+
```

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: <[email-masked]>
Date: Thu, 10 Apr 2014 16:53:53 +0000

> Isn't it much more likely that intermediaries will actually implement this if they can do that form of pass-through?

Actually, the intermediary can pass-through with gzip or deflate.  Concatenated gzip or deflate streams can be uncompressed as if they were one big stream.  See http://www.zlib.net/manual.html

---

From: <[email-masked]>
Date: Thu, 10 Apr 2014 18:01:54 +0000

On Saturday,05 April 2014 21:19, Roland Zinks wrote:

> Doing it on each data frame should decrease what you can save with compression.

> At the same time it doesn't require a state between frames ...

Regarding the compression factor, we did a simple investigation to get a feel for how big of a hit you take by using an independent compression context on each frame.

The results surprised us.  The increase from doing frame-by-frame compression versus single-stream compression is smaller than we expected..

For our experiments we used gzip with the following parameters: compression-level=6 (default), windows-bits=14, mem-level=7, strategy=default

Here are the results...

[units are bytes; value in () is the compression factor; % increase is frame-by-frame vs single-stream]

'Counters2.htm'

original file size:    1219829

gzip(single-stream):     17804 (0.015)

gzip(frame-by-frame):    18148 (0.015) 1.9% increase

'http2-spec.htm'

original file size:     291065

gzip(single-stream):     64165 (0.220)

gzip(frame-by-frame):    67048 (0.230) 4.5% increase

'rs.js'

original file size:     145275

gzip(single-stream):     47409 (0.326)

gzip(frame-by-frame):    48877 (0.336) 3.1% increase

'search.htm'

original file size:     236319

gzip(single-stream):     65850 (0.279)

gzip(frame-by-frame):    68353 (0.289) 3.8% increase

'search.json'

original file size:      92680

gzip(single-stream):     16110 (0.174)

gzip(frame-by-frame):    16807 (0.181) 4.3% increase

'url.htm'

original file size:       1082

gzip(single-stream):       523 (0.483)

gzip(frame-by-frame):      523 (0.483) 0.0% increase

Other points...

0. Packing frames with gzip data

An important point is trying to fill each frame with as much compressed data as possible (going over the 16383 byte limit for a frame would obviously create a corrupted stream). The implementation is simple.

1. Client decompression simplicity

We agree that doing frame-by-frame compression results in a *much simpler* logic for the client frame reader. For a single connection, the client needs to maintain only a single inflate context.

2. Deflate instead of Gzip

Using deflate instead of gzip would obviously improve the compression factor slightly. Concatenating of raw deflate streams is also well-defined.

3. Indexing

As Keith mentioned in an earlier e-mail, the compressed DATA frames could be augmented with the uncompressed offset and length for easier indexing.

4. Intermediary Efficiency

Intermediaries could store the DATA frames sequentially e.g. in a single file, and easily serve up the data (or even contiguous subsets) without recompressing.

5. 15-bit vs 14-bit frame size

The compression factor using window-bits=15 would result in a better compression factor, but would obviously require increasing the frame length to 2^15 bytes. (Probably better used than having 2 reserved bits forever.)

6. Compression for ranges

As we've mentioned many times before, using a transfer coding is the only way to compress a range response of an Identity C-E data.

Regards

Chris & Keith

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Apr 2014 16:48:38 -0700

> probably 1-2 bytes per connection.

Actually, the size committment for any receiver of something which uses T-E
would be the minimum size of the encoding context. For gzip, this is
decided by the sender and is not trivial, and thus provides for an
interesting DoS vector against the receiver.
This would completely throw out the things we've done in HTTP/2 to ensure
that the receiver controls the size of the amount of memory it needs to
use. That seems silly at best.

> though; RFC 2703 and RFC 2295/2296 exist if people really need inspiration.
> It could also go away if rank is reduced to a single bit.
>
There are a large number of places where MITM proxies are deployed,
unfortunately, and I expect this would likely increase after HTTP/2 is
widely deployed.
That means that we cannot count on end-to-end encryption for delivering
security properties.
As an example, that is why the HPACK compressor has the ability to
indicated that no intermediary should ever compress data with X.

It is simply not safe for intermediaries to do this kind of compression
unless they have some prior (and up to date) knowledge about the provenance
of the data comprising the stream.
This is unlikely at best.
-=R

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 11 Apr 2014 18:26:33 +1200

> I'm neutral on whether you use a single value for this setting or keep a table like Amos wants. ( I always slightly lean towards KISS though.)  I just didn't realize there was a desire to dynamical…

Note: KISS for a proxy is the table...

A proxy which is contacting more than one server (ie the common case).
 Each server connection may negotiate or require a separate TE
compression be used.
Translating independent server and client encodings is complex and
requires extra state. Whereas negotiating a set/table of TE with the
client allows server TE to negotiate items from that table. Thus the
proxy can then pass-through the encoded frames from multipe different
servers TE without extra compression state or logic.

> 7-bit encoding with 1-bit on/off (128 possible encodings) - you could still pack four encodings into a single setting value

Trailers are nasty. While this may sound okay to begin with, as others
have said with the LZ compression methods under discussion in most of
these threads proxies can concatenate DATA of a stream without
decompressing. It also reduces the frame payload size usable for
compressed data (which could be significant amount of decompressed content).

Amos

---

From: Roland Zink <[email-masked]>
Date: Fri, 11 Apr 2014 17:17:39 +0200

On 11.04.2014 01:48, Roberto Peon wrote:
>
>
>
> On Wed, Apr 9, 2014 at 11:16 PM, Matthew Kerwin <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     I've started keeping track of my T-E proposal in github[1], and
>     have been attempting to come up with ways to address people's
>     concerns.
>
>     * The easy issue was the size of identifiers, which I've reduced
>     to 8 bits in my current text -- i.e. 256 possible values. The
>     worst case commitment as a result is 256 rank values, which I've
>     also reduced to 8 bits -- so 256 bytes. However as I and others
>     have pointed out, the minimum commitment as a result of this
>     proposal is nil, and a reasonable realistic value is probably 1-2
>     bytes per connection.
>
>
> Actually, the size committment for any receiver of something which 
> uses T-E would be the minimum size of the encoding context. For gzip, 
> this is decided by the sender and is not trivial, and thus provides 
> for an interesting DoS vector against the receiver.
> This would completely throw out the things we've done in HTTP/2 to 
> ensure that the receiver controls the size of the amount of memory it 
> needs to use. That seems silly at best.

This seems to be a property of gzip, so isn't this the same for C-E?

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 12 Apr 2014 07:49:25 +1000

On Apr 12, 2014 1:18 AM, "Roland Zink" <[email-masked]> wrote:
>>> I've started keeping track of my T-E proposal in github[1], and have
been attempting to come up with ways to address people's concerns.
>>>
>>> * The easy issue was the size of identifiers, which I've reduced to 8
bits in my current text -- i.e. 256 possible values. The worst case
commitment as a result is 256 rank values, which I've also reduced to 8
bits -- so 256 bytes. However as I and others have pointed out, the minimum
commitment as a result of this proposal is nil, and a reasonable realistic
value is probably 1-2 bytes per connection.
>>
>>
>> Actually, the size committment for any receiver of something which uses
T-E would be the minimum size of the encoding context. For gzip, this is
decided by the sender and is not trivial, and thus provides for an
interesting DoS vector against the receiver.
>> This would completely throw out the things we've done in HTTP/2 to
ensure that the receiver controls the size of the amount of memory it needs
to use. That seems silly at best.
> This seems to be a property of gzip, so isn't this the same for C-E?

I think the argument here is that the commitment for C-E only applies go
the client*, so light-weight proxies can remain light-weight.

*apart from 1.1<->2 gateways, apparently

---

From: Roland Zink <[email-masked]>
Date: Sat, 12 Apr 2014 18:09:18 +0200

On 11.04.2014 23:49, Matthew Kerwin wrote:
>
> On Apr 12, 2014 1:18 AM, "Roland Zink" <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >
> >> On Wed, Apr 9, 2014 at 11:16 PM, Matthew Kerwin 
> <[email-masked] <mailto:[email-masked]>> wrote:
> >>>
> have been attempting to come up with ways to address people's concerns.
> >>>
> to 8 bits in my current text -- i.e. 256 possible values. The worst 
> case commitment as a result is 256 rank values, which I've also 
> reduced to 8 bits -- so 256 bytes. However as I and others have 
> pointed out, the minimum commitment as a result of this proposal is 
> nil, and a reasonable realistic value is probably 1-2 bytes per 
> connection.
> >>
> >>
> I think the argument here is that the commitment for C-E only applies 
> go the client*, so light-weight proxies can remain light-weight.
>
I think this argument not really holds. When gzip T-E is mandated like 
now C-E then light-weight proxies don't need to touch the compression 
and can forward the received bytes unmodified. The transferred data 
bytes could be the same in both cases. This would be different when 
support is optional and the next hop doesn't support it, or the 
compression is tied to a connection instead of a stream or the proxy 
want to look into the content.

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 15 Apr 2014 02:54:41 -0600

Matthew Kerwin wrote:
> 
> But I have to respond to one particular type of comment that keeps
> coming up: "*well-constructed sites which compressed their
> resources.*"
> 
> That's a big value judgement on what constitutes good site design.
> Yes, in lots of cases it makes sense to compress your resources and
> have multiple representations, especially for static resources; but
> what about the sites that aren't like that? Why is it bad site design
> to have a big resource that can be accessed with ranges? If the
> answer is because doing so would require TE in order to have
> compression, then it's a tautology (TE is only needed by bad sites;
> those sites are bad because they need TE). If it's because caches
> don't handle that properly, then it's a chicken-and-egg problem. The
> only reasons I can think of for calling them bad are either circular,
> or "I don't like them." Is there a real reason?
> 
> Why should I make my web API use "?start=N&end=M" when I could use
> "Range: x-records=N-M" ?
> 

Agreed. Mnot's "get off my lawn" RFC seems to prefer a protocol-level
solution instead of attempting to standardize range requests as part of
the http scheme. IMO, accepting that resources have representations
leads us right back to transfer codings, unless range requests to large
files are deprecated as somehow being bad design, which anyone who's
had to use them (and found it to work) knows better than. Which still
doesn't solve the problems led to by deprecating T-E.

-Eric

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 15 Apr 2014 02:59:36 -0600

> I'm more than happy to have my ignorance corrected :)

Some large dynamic objects persist long enough between changes for range
requests to be a big win, particularly for clients with intermittent
connectivity, particularly when compression is possible. I'm not at
liberty to discuss a project I was consulting architect on, which wasn't
as big as IAEA anyway so I'm sure it also pales in comparison to the
Web-browsing use case generally deferred to here.

But, the most painful bit of code I've ever written implemented T-E
compression of range requests. As Helge Hess mentioned elsewhere, I
don't get how to manage the Etag if T-E is gone and conneg is in play.
Which it is, if the large dynamic object is C-E compressed text because
the user-agent won't display *.html.gz served as application/gzip and
no support exists for displaying this served as application/html+gzip,
which would be the "right way" to do things IMO.

Resources having representations is a fact of life even if T-E is
removed and gzip mandated. Even if I didn't have personal experience
with this, I would extend the benefit of the doubt to the IAEA guys (as
I don't know the specifics of their use-case) before declaring it
nonsensical... some of us have found applications for range requests
which don't begin to resemble pre-compressed image or video content.

-Eric

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 15 Apr 2014 03:05:07 -0600

Martin Thomson wrote:
>
> >
> > This sounds like a snark, but it's not: what is a better
> > alternative here? C-E is an alternative, but it has deficiencies.
> 

Well, that's one opinion. Another would be to find an alternative that
isn't horrible protocol design exacerbating a known problem with HTTP
1.1, regardless of "how late in the process" this issue has come up.

>
> > My goal with this proposal was partly to support compression on
> > range requests, and partly to de-emphasise the current practice of
> > on-the-fly C-E. To that end, I don't know of any alternatives at
> > all.
> 

Even if it were feasible in a use-case I encountered, sharding the files
and allocating URIs to the bits and pieces solves your problem, but
leaves me writing a whole bunch of project-specific code (client and
server!) to split and re-combine what my customer (rightly) sees as one
file, when HTTP 1.1 has exactly the range-request facility I need at the
protocol layer to support remote workers with spotty connections in
Timbuktu or wherever.

Even if this were feasible, I'd rather not maintain non-portable code
for a feature HTTP2 removed before I was done using it. Client would
rather not pay me to re-code their application just so they can upgrade
to HTTP2. I'd rather not touch an application that's been working for
years on HTTP 1.1 w/ range requests and T-E.

OTOH, modifying an httpd to properly handle range-requests is something
I can file a bug against and eventually not have to worry about
maintaining. Meanwhile, I can use my fix on multiple projects because
it's part of the protocol, not the application. Once I've figured out
how a tool works, I'm likely to use it again if I find it appropriate,
so I'd prefer to keep it in the toolbox instead of missing it down the
road. YMMV.

-Eric