Subject: Making Implicit C-E work.

From: Roberto Peon <[email-masked]>
Date: Tue, 29 Apr 2014 13:29:26 -0700

Here is a proposal for keeping implicit C-E around safely.

When gzip is not in the accept-encoding, gzip is assumed to be implicitly
accepted, as opposed to explicitly accepted. This means that servers/client
which wish to indicate that it is explicitly accepted do put a-e: gzip in
the headers.

Servers seeing an implicit gzip a-e and sending a compressed entity MAY
include an additional header field, ":uncompressed-etag". An HTTP2 ->
HTTP/1 gateway which is decompressing and forwarding an entity MUST replace
ETAG with the value of this header, if it exists, or insert it if it does
not.

Additionally, to address the deployment fact that there are non-browser
usecases out there which are stuck using HTTP/1.1 servers and clients,
which only accept chunked-encoded messages with the headers include
content-length:

Servers or Clients seeing an implicit gzip a-e and sending a compressed
entity MAY include an additional header field,
":uncompressed-content-length", which contains the uncompressed content
length. An HTTP2 -> HTTP/1 gateway which is decompressing and forwarding an
entity MUST replace content-length with the value of this header, if it
exists, or insert it if it does not.

I know for a fact that compression in the client->server direction is
desirable *and* is likely to be used for non-browser usecases, and so I
think both of these end up being useful given that there is no (useful,
non-latency harming) negotiation for accepting compressed entities in the
client->server direction for HTTP/1.

-=R

---

From: Roberto Peon <[email-masked]>
Date: Tue, 29 Apr 2014 13:30:58 -0700

I'll note that the suggested text should have said that the old
content-length and old etag MUST be removed if not replaced when doing the
decompression, but hopefully we all already knew that :)
-=R

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 30 Apr 2014 07:25:54 +1000

And last-modified? And, possibly, other fields from Vary? There's more
entity-specific metadata than just ETag.

I still think you're putting too much emphasis on C-E, particularly in
cases where it doesn't (or shouldn't) apply. What you want is compressed
data (fewer bytes) on the wire. HTTP has limited options in this regard,
and the Right Way™ (T-E) is out of the picture; but using C-E is at best a
hack, but more likely a flat out misunderstanding of its purpose. And the
current text madates, in an internet standard, that everyone support the
hack; and by extension promotes the misunderstanding.

If

---

From: Roberto Peon <[email-masked]>
Date: Tue, 29 Apr 2014 14:33:45 -0700

For better or worse, C-E is what is deployed today. Many of my customers
will not be writing custom servers, and as such to be deployable, we need
solutions that will work with what is out there.
Otherwise, the feature is effectively only of theoretical use for the
majority of customers.

I don't dispute that one could use T-E over HTTP/2 for this, assuming that
it was end-to-end (which, unfortunately, it will not be for quite some
time).
-=R

On Tue, Apr 29, 2014 at 2:25 PM, Matthew Kerwin <[email-masked]>wrote:

> > When gzip is not in the accept-encoding, gzip is assumed to be
> implicitly accepted, as opposed to explicitly accepted. This means that
> servers/client which wish to indicate that it is explicitly accepted do put
> a-e: gzip in the headers.
> >

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 30 Apr 2014 07:45:00 +1000

> Otherwise, the feature is effectively only of theoretical use for the
​Sorry, my 7 year old bumped my phone before and sent my message before I
was finished writing it.

Yes, C-E is what we have, but that's no reason to promote it to a MUST
support, and even less of a reason to promote it to a MUST support and
​have no way to opt out. Some people use C-E as a hack, some other people
disable the C-E as a hack; whose hack is the worse? You've made your call
there, but we don't all have to agree.

There is no valid justification for modifying HTTP semantics, and dictating
how people use entities, in HTTP/2.

> it was end-to-end (which, unfortunately, it will not be for quite some
​​Beside the point. I'm happy for people to use (and abuse) C-E. I'm just
not happy for us to promote or mandate it in the spec.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Roberto Peon <[email-masked]>
Date: Tue, 29 Apr 2014 14:54:17 -0700

> ​Sorry, my 7 year old bumped my phone before and sent my message before I

The age of mobile has its disadvantages :)

> There is no valid justification for modifying HTTP semantics, and
> dictating how people use entities, in HTTP/2.
>

The proposal doesn't modify HTTP semantics, but preserves them while
offering a needed base capability assurance.

>
>
>
>> I don't dispute that one could use T-E over HTTP/2 for this, assuming
>> that it was end-to-end (which, unfortunately, it will not be for quite some
>> time).
>>
>>

As I described it, its use by the originator of an entity is not mandated,
instead behaviors are mandated of recipients when it IS used.
In any case, there are some observations here:
1) We know that compressing things is a huge win and should be done when
security matters don't intrude.
2) We know that software and vendors out there take shortcuts which harm
performance and have been harming interop.
3) We must continue to consider deployability and not just theoretical
usefulness-- the deployability of the protocol is the reason we've gotten
to where we are, and it doesn't make sense to stop thinking about it.

I have a real honest-to-goodness pragmatic deployment problem (myriad
pre-existing servers/clients whose deployment I do not and cannot control)
here that I cannot wish away, and this is a solution.
Can you propose another solution to dealing with this problem which solves
the problem?

An HTTP2 without C-E based compression will not achieve some of the
objectives that it would with it.

-=R

---

From: Zhong Yu <[email-masked]>
Date: Tue, 29 Apr 2014 17:45:28 -0500

On Tue, Apr 29, 2014 at 4:54 PM, Roberto Peon <[email-masked]> wrote:
>> Sorry, my 7 year old bumped my phone before and sent my message before I
>> was finished writing it.
>
>
>> support, and even less of a reason to promote it to a MUST support and have
>> no way to opt out. Some people use C-E as a hack, some other people disable
>> the C-E as a hack; whose hack is the worse? You've made your call there, but
>> we don't all have to agree.
>>
>> Beside the point. I'm happy for people to use (and abuse) C-E. I'm just

What percentage of clients do not send "Accept-Encoding: gzip"?

> usefulness-- the deployability of the protocol is the reason we've gotten to
> where we are, and it doesn't make sense to stop thinking about it.
>

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 30 Apr 2014 08:46:24 +1000

> offering a needed base capability assurance.

Well, it changes a client's ability to request an uncompressed entity.

>> ​​
> instead behaviors are mandated of recipients when it IS used.​
>
​
​​
Yeah, mandating it. Which I'm not happy about.

​

> ​​

​Indeed.

> performance and have been harming interop.

I'm just saying ​this isn't the time or the way to fix it. One battle at a
time. Use TCP better and make headers less bloaty now -- issues that affect
everyone on the web; worry about edge cases like stupid or malicious
proxies later.

> to where we are, and it doesn't make sense to stop thinking about it.

​Sure, so what does a 1.1->2 proxy do when a 1.1 client requests
Accept-Encoding:identity​ + Cache-Control:no-transform and the 2 server
responds with Content-Encoding:gzip ? That's currently possible, even if
unlikely, and we still have no suggestion at all for what the proxy is
supposed to do about it. Undocumented unlikely edge cases are the worst
kind, no? I think that makes the current draft undeployable for those
gateways, and they're the ones who are going to be doing a lot of the heavy
lifting of getting HTTP/2 out there to the masses.

> Can you propose another solution to dealing with this problem which solves
Which problem, exactly? If it's that people are doing a certain thing in
HTTP/1.1, then the simplest solution is: let them keep doing it. No need to
add any words to the spec.​ If some proxy is stripping Accept-Encoding
headers, that's a raw deal for the users behind it, but it doesn't break
the internet. Those users would still see an improvement after switching to
HTTP/2.

> objectives that it would with it.
It's not without compression, it just doesn't mandate it. What we gain in
return is: not creating paradoxes in the protocol. Arguing pragmatism is
all well and good, but you still haven't fully addressed the gateway issue.
Theoretical holes have ways of becoming real, practical holes, and often
reveal massive security issues when they do so.

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 30 Apr 2014 11:12:26 +1200

On 30/04/2014 10:46 a.m., Matthew Kerwin wrote:
> On 30 April 2014 07:54, Roberto Peon wrote:
> 
>>
>>
>> On Tue, Apr 29, 2014 at 2:45 PM, Matthew Kerwin wrote:
>>
>>> On 30 April 2014 07:33, Roberto Peon wrote:
>>>

Ahmen.

> reveal massive security issues when they do so.

Like those proxies today stripping Accept-Encoding:gzip in order to get
a fully working identity object from the server. In HTTP/2 will very
likely become proxies decompressing on the fly corrupting unknown
amounts of encoding-specific entity header values in the process. If
they dare to try and do the "right" thing by re-compressing it just
makes the new problems worse and less detectable.

So, well, uhm.

Amos

---

From: Roberto Peon <[email-masked]>
Date: Tue, 29 Apr 2014 16:47:30 -0700

It is more a question of how many intermediaries strip off a-e: gzip, and
how many servers/clients simply assume that it must be there anyway
(causing potential interop issues) because the latency problem (caused by
these intermediaries) was large enough that it mattered.

-=R

On Tue, Apr 29, 2014 at 3:45 PM, Zhong Yu <[email-masked]> wrote:

> have

---

From: Roberto Peon <[email-masked]>
Date: Tue, 29 Apr 2014 17:06:58 -0700

On Tue, Apr 29, 2014 at 3:46 PM, Matthew Kerwin <[email-masked]>wrote:

>>>> For better or worse, C-E is what is deployed today. Many of my
>>>> customers will not be writing custom servers, and as such to be deployable,
>>>> we need solutions that will work with what is out there.

An HTTP/1 client still receives an uncompressed entity if it didn't request
it compressed.

>>> ​​Beside the point. I'm happy for people to use (and abuse) C-E. I'm
>>> just not happy for us to promote or mandate it in the spec.
>>> ​​
>>>
>>
>> As I described it, its use by the originator of an entity is not
>> mandated, instead behaviors are mandated of recipients when it IS used.​
>>
> ​
>>
> ​​

Mandates support, not use.

> everyone on the web; worry about edge cases like stupid or malicious

The combination of intermediaries stripping a-e plus the competitive driver
to deliver good experience/latency is causing interop failures today where
servers will send gzip'd data whether or not the client declares support in
a-e.

I know of no case where the current (implicit gzip) behavior caused any
problems large enough to rise to the level of visibility. I *have* had to
deal with interop and performance issues because of intermediaries
stripping a-e gzip.
Thusfar, (and I guess surprisingly) the *interop* problems of not having
gzip are larger than the interop problems of implicitly assuming it between
HTTP2 actors, etag wart and all.

> lifting of getting HTTP/2 out there to the masses.

The proxy, when forwarding the server's response to the HTTP/1 client, must
ensure that the data is uncompressed when forwarding to the HTTP/1 client
since the client didn't ask for c-e gzip.
This seems easy enough to document in the HTTP/2 spec if it isn't already
clear from the 1.1 spec, and the mechanism for doing so was specified in my
first email...

>> Can you propose another solution to dealing with this problem which
>> solves the problem?
>>
>>

I'll point out that if that was applied recursively, there'd be no need to
do anything HTTP/2 because we could always let them keep doing what they've
been doing.
One of the drivers of HTTP/2 is to seek out performance. If it doesn't
deliver on that, it is not a useful effort. Ensuring that entity-bodies are
compressed when it is safe to do so is an important driver of performance.

> reveal massive security issues when they do so.
If one ensures that etag and content-length are sent as specified in my
proposal, then gateways will always produce the representation that the
HTTP/1 client requested. There is no paradox there.
-=R

---

From: "Eric J. Bowman" <[email-masked]>
Date: Tue, 29 Apr 2014 19:16:40 -0600

Matthew Kerwin wrote:
> cases where it doesn't (or shouldn't) apply. What you want is
> compressed data (fewer bytes) on the wire. HTTP has limited options
> in this regard, and the Right Way™ (T-E) is out of the picture; but
> using C-E is at best a hack, but more likely a flat out
> misunderstanding of its purpose. And the current text madates, in an
> internet standard, that everyone support the hack; and by extension
> promotes the misunderstanding.
> 

+1, the ends don't justify the means. The "Other Right Way" is still
media types, specifically appending +gzip to HTML/CSS/JS identifiers;
if gzip is explicit in Content-Type it doesn't need to be implicit in
C-E. Achieves the performance win without all the headaches, I think.

-Eric

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 30 Apr 2014 14:50:05 +1000

majority of customers.
> An HTTP/1 client still receives an uncompressed entity if it didn't
request it compressed.

It gives the gateway a bunch of extra hoops to jump through, and
obligations to take on.

> Mandates support, not use.

Kind of the same thing, from the client's POV. Server's choice.

>
>> ​
>>>
>>> 2) We know that software and vendors out there take shortcuts which
harm performance and have been harming interop.
>>
>> I'm just saying ​this isn't the time or the way to fix it. One battle at
a time. Use TCP better and make headers less bloaty now -- issues that
affect everyone on the web; worry about edge cases like stupid or malicious
proxies later.
>
> The combination of intermediaries stripping a-e plus the competitive
driver to deliver good experience/latency is causing interop failures today
where servers will send gzip'd data whether or not the client declares
support in a-e.

Wait, you're saying the whole motivator here is that servers don't comply
with the protocol? So you're changing the protocol to accommodate them?
That does not feel right to me, at all; it's not just blessing a potential
misuse of C-E, it's wallpapering over a flat out abuse.

But even so, why do you have to fix it in HTTP/2? And why does it hurt h2
to *not* fix it?

Not that surprising, really. If the current behaviour (unblessed
assumption) caused more trouble than its gains, it wouldn't have stuck
around. Blessing the assumption doesn't fix it, though, not without
extending your proposal of including before-compression metadata (you
missed last-modified, but there could be other headers too, especially
proprietary extension headers)... and if you do that you end up with my
earlier idea of a level of compression somewhere between
entity-representation and transport. But because that's just too much work
to get right, and even harder to make interoperate with older HTTP, I
deferred it until HTTP 3+.

In fact only yesterday I set up a rough and ready wiki on github to track
my thoughts on the matter.

It's at <https://github.com/phluid61/http3> IIRC.

>
> The proxy, when forwarding the server's response to the HTTP/1 client,
must ensure that the data is uncompressed when forwarding to the HTTP/1
client since the client didn't ask for c-e gzip.

Cache-Control:no-transform explicitly forbids the proxy from altering the
representation. It's not allowed to decompress it.

solves the problem?
>
> I'll point out that if that was applied recursively, there'd be no need
to do anything HTTP/2 because we could always let them keep doing what
they've been doing.

The line in the sand was drawn at transport. You're trying to optimise
entity representation. You can't fix all the web's woes with HTTP/2. Not
even all the ones you care about.

>>>
>>
>> It's not without compression, it just doesn't mandate it. What we gain
in return is: not creating paradoxes in the protocol. Arguing pragmatism is
all well and good, but you still haven't fully addressed the gateway issue.
Theoretical holes have ways of becoming real, practical holes, and often
reveal massive security issues when they do so.

You'd probably want to explicitly tell the proxies that it's ok to
partially ignore Cache-Control:no-transform, in that case. Even though that
*is* changing semantics.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 29 Apr 2014 23:44:52 -0700

On Tue, Apr 29, 2014 at 9:50 PM, Matthew Kerwin <[email-masked]>wrote:

> <snip>

True it gives some additional obligations in the HTTP2<->HTTP/1 case,
though the characterization of "a bunch" is an exaggeration.

> >>
> >>>>
> >>>>>
> >>>>> I don't dispute that one could use T-E over HTTP/2 for this,
> assuming that it was end-to-end (which, unfortunately, it will not be for
> quite some time).
> >>>>>
> >>>>

And today it is often neither the server nor the client's choice, which is
what is causing the pain. The client expresses that it wants gzip. The
intermediary doesn't do it because it makes numbers better, increases
throughput, or because they're too lazy to implement it., all at the cost
of the decreased user experience.

> <snip>
Partially.
I'm saying that intermediaries are doing things which are incenting
implementors to break compatibility with the spec, and that implementors
are doing so because it makes the users happy.
In the end, making the users happy is what matters, both commercially and
privately. The users really don't care about purity, and will migrate to
implementations that give them good/better user experience.

> to *not* fix it?

Compression is an important part of making latency decrease/performance
increase, and, frankly, there is little practical motivation to deploy
HTTP/2 if it doesn't succeed in reducing latency/increase performance.
Success isn't (or shouldn't be) defined as completing a protocol spec, but
rather, getting an interoperable protocol deployed. If it doesn't get
deployed, the effort is wasted. If it doesn't solve real problems, the
effort is wasted.

In any case, I cannot reliably deploy a T-e based compression solution.
T-e based compression costs too much CPU, especially as compared with c-e
where one simply compresses any static entity once and decompresses (which
is cheap) as necessary at the gateway.
T-e based compression isn't as performant in terms of compression/deflation
ratios.
Many deployed clients/servers wouldn't correctly support it.
T-e would require that any gateway acting as a loadbalancer/reverse proxy
would either need to know which resources it could compress,  or forces us
to not use compression. Knowing what resources to compress either requires
an oracle, or requires content authors to change how they author content
(*really* not likely to happen),

<snip>
> representation. It's not allowed to decompress it.
In fact what we're doing is offering two representations simultaneously.

> <snip>
Lol! This is one that we've already 'fixed' (and has been deployed for
years), and a bug was discovered. That is good-- we can fix the bug. You're
advocating for removing it in the pursuit of purity. I'm advocating for
patching it.

<snip>
-=R

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Wed, 30 Apr 2014 07:57:58 +0000

> It is more a question of how many intermediaries strip off a-e: gzip, 

Probably not very many. None of the Cisco ones AFAIK although we remove sdch and others in a couple of edge-cases - and those are in areas we know to be rather brittle to start with.

Richard

---

From: Roland Zink <[email-masked]>
Date: Wed, 30 Apr 2014 10:15:40 +0200

On 29.04.2014 22:29, Roberto Peon wrote:
> implicitly accepted, as opposed to explicitly accepted. This means 
> that servers/client which wish to indicate that it is explicitly 
> accepted do put a-e: gzip in the headers.
>
> Servers seeing an implicit gzip a-e and sending a compressed entity 
> MAY include an additional header field, ":uncompressed-etag". An HTTP2 
> -> HTTP/1 gateway which is decompressing and forwarding an entity MUST 
> replace ETAG with the value of this header, if it exists, or insert it 
> if it does not.
>
I think to make this work the MAY is not enough, e.g. it should be along 
the line of servers seeing an implicit gzip a-e and sending a compressed 
entity with an ETag header MUST include an additional header field, 
":uncompressed-etag". Don't think you can enforce such a thing however. 
For example PHP doesn't support you with ETag.
> Additionally, to address the deployment fact that there are 
> non-browser usecases out there which are stuck using HTTP/1.1 servers 
> and clients, which only accept chunked-encoded messages with the 
> headers include content-length:
>
> Servers or Clients seeing an implicit gzip a-e and sending a 
> compressed entity MAY include an additional header field, 
> ":uncompressed-content-length", which contains the uncompressed 
> content length. An HTTP2 -> HTTP/1 gateway which is decompressing and 
> forwarding an entity MUST replace content-length with the value of 
> this header, if it exists, or insert it if it does not.
>
>
> desirable *and* is likely to be used for non-browser usecases, and so 
> I think both of these end up being useful given that there is no 
> (useful, non-latency harming) negotiation for accepting compressed 
> entities in the client->server direction for HTTP/1.
>
> -=R
Adding the compression support on the transport level allows web servers 
to compress content which would otherwise be send uncompressed. Btw. 
making the A-E gzip implizit disables C-E gzip in current environments 
like PHP. I guess this can be repaired quite easily, but you need a 
HTTP2 aware version.

---

From: Roland Zink <[email-masked]>
Date: Wed, 30 Apr 2014 10:29:42 +0200

On 29.04.2014 23:33, Roberto Peon wrote:
> customers will not be writing custom servers, and as such to be 
> deployable, we need solutions that will work with what is out there.
You need at least new HTTP2 servers. Those servers probably could do 
gzip on the transport level.

> that it was end-to-end (which, unfortunately, it will not be for quite 
> some time).
> -=R
>
Not sure if it really needs to be end to end. If support for T-E is 
mandated in HTTP2 then proxies can just forward the compressed content 
unmodified within HTTP2 if they don't have a need to modify the content 
much the same way as they would do with C-E.

Roland

---

From: Roland Zink <[email-masked]>
Date: Wed, 30 Apr 2014 11:35:04 +0200

>     Kind of the same thing, from the client's POV. Server's choice.
For C-E this means for example the server decides if the client can do a 
seek. Some interactive clients would prefer to do seeks over getting the 
content compressed. Whereas downloaders would prefer the content to be 
compressed. T-E would allow to have both seek and compression.

> And today it is often neither the server nor the client's choice, 
> which is what is causing the pain. The client expresses that it wants 
> gzip. The intermediary doesn't do it because it makes numbers better, 
> increases throughput, or because they're too lazy to implement it., 
> all at the cost of the decreased user experience.
>
>     <snip>
>     >
>     > The combination of intermediaries stripping a-e plus the
>     competitive driver to deliver good experience/latency is causing
>     interop failures today where servers will send gzip'd data whether
>     or not the client declares support in a-e.
>     >
>
>     Wait, you're saying the whole motivator here is that servers don't
>     comply with the protocol? So you're changing the protocol to
>     accommodate them? That does not feel right to me, at all; it's not
>     just blessing a potential misuse of C-E, it's wallpapering over a
>     flat out abuse.
>
> Partially.
> implementors to break compatibility with the spec, and that 
> implementors are doing so because it makes the users happy.
> In the end, making the users happy is what matters, both commercially 
> and privately. The users really don't care about purity, and will 
> migrate to implementations that give them good/better user experience.
>
>     But even so, why do you have to fix it in HTTP/2? And why does it
>     hurt h2 to *not* fix it?
>
>
> Compression is an important part of making latency 
> decrease/performance increase, and, frankly, there is little practical 
> motivation to deploy HTTP/2 if it doesn't succeed in reducing 
> latency/increase performance.
> Success isn't (or shouldn't be) defined as completing a protocol spec, 
> but rather, getting an interoperable protocol deployed. If it doesn't 
> get deployed, the effort is wasted. If it doesn't solve real problems, 
> the effort is wasted.
>
> T-e based compression costs too much CPU, especially as compared with 
> c-e where one simply compresses any static entity once and 
> decompresses (which is cheap) as necessary at the gateway.
If it is really T-E you can do the same compression of static entities 
when the whole file is delivered, it would be different for range 
requests or the frame based approach.
> T-e based compression isn't as performant in terms of 
> compression/deflation ratios.
Don't think this is true, the same bytes can be sent as either T-E or 
C-E. For the frame based approach some numbers were given.
> Many deployed clients/servers wouldn't correctly support it.
There are no deployed HTTP2 clients or servers, or are there some?
> T-e would require that any gateway acting as a loadbalancer/reverse 
> proxy would either need to know which resources it could compress,  or 
> forces us to not use compression.
The gateway can forward the compressed content unmodified. The gateway 
is only forced to do something if either the server or the client 
doesn't support compression.
> Knowing what resources to compress either requires an oracle, or 
> requires content authors to change how they author content (*really* 
> not likely to happen),
>
Not sure that authors want to know about compression. If it is automatic 
then this would be fine. Currently there is server configuration, for 
example zlib.output_compression in php.ini, and the possibility to do 
this in the content, for example in PHP something like 
ob_start('ob_gzhandler'). I guess there is a lot more authors are not 
aware off.
>
>     <snip>
>
>     > The proxy, when forwarding the server's response to the HTTP/1
>     client, must ensure that the data is uncompressed when forwarding
>     to the HTTP/1 client since the client didn't ask for c-e gzip.
>     >
>
>     Cache-Control:no-transform explicitly forbids the proxy from
>     altering the representation. It's not allowed to decompress it.
>
Don't think this will really work. Only one can delivered and 
translating between them (for example range) seems to be difficult.

Roland

---

From: Roberto Peon <[email-masked]>
Date: Wed, 30 Apr 2014 03:26:54 -0700

On Wed, Apr 30, 2014 at 2:35 AM, Roland Zink <[email-masked]> wrote:

>    For C-E this means for example the server decides if the client can do
> a seek. Some interactive clients would prefer to do seeks over getting the

The server *always* decides what to send, whether c-e or t-e is used. The
fact that the server *may* use gzip does not *require* it to use gzip, and
with my proposal, the server knows if the client requested it explicitly or
not, and it certainly can see if there is a range request and make the
appropriate response.

T-E is theoretically wonderful if one ignores real deployments in today's
world where the majority of HTTP/1.X servers don't actually do
transfer-encoding: gzip, and thus HTTP2 gateways would have to do c-e to
t-e translation (which might be rather error prone in its own way) or have
to bear the expense of doing the compression themselves-- something which
is untenable. This ignores the the security issue of knowing when t-e is
safe, which I'll address again below.

>
>   And today it is often neither the server nor the client's choice, which
> is what is causing the pain. The client expresses that it wants gzip. The
>  But even so, why do you have to fix it in HTTP/2? And why does it hurt
>> h2 to *not* fix it?
>>
>
> when the whole file is delivered, it would be different for range requests
> or the frame based approach.
>

Now how we've thusfar spec'd it.

> Don't think this is true, the same bytes can be sent as either T-E or C-E.
> For the frame based approach some numbers were given.
>

The same bytes can't be sent in both, unless the we're willing to suffer
vastly increased DoS surface area and memory usage OR we do the frame-based
approach, which will have marginally worse compression.

> There are no deployed HTTP2 clients or servers, or are there some?

There are, but I'm not talking about those. My problem is dealing with the
rest of the world, which is mostly HTTP/1.X and is unlikely to rapidly
change.
In other words, I'm concerned mainly with HTTP/1.X clients and especially
servers.

> The gateway can forward the compressed content unmodified. The gateway is
> only forced to do something if either the server or the client doesn't
> support compression.
>

The gateway cannot know which resources it is safe to compress without
something outside the protocol. Compression via t-e without knowing whether
it is safe or not allows attackers to discern ostensibly secret
information. This is NOT acceptable.

> requires content authors to change how they author content (*really* not
> likely to happen),
>
>    Not sure that authors want to know about compression. If it is
> automatic then this would be fine. Currently there is server configuration,
> for example zlib.output_compression in php.ini, and the possibility to do
> ob_start('ob_gzhandler'). I guess there is a lot more authors are not aware
> off.
>

We definitely don't want to cause content authors to lose what little
control (and understanding) they have today, especially over matters
touching security like compression.
In general, if a resource wasn't compressed on output from an endpoint, it
shouldn't be when received by any other endpoint.
Given the necessity of interfacing with HTTP/1 servers, which rarely
support T-E: gzip, this ends up being a problem for HTTP/2 and T-e: gzip.

> Don't think this will really work. Only one can delivered and translating
> between them (for example range) seems to be difficult.
>

Why is range a problem? If a server wishes to service a range request, it
need not compress the output.
With the proposal, the server will be able to know if the client requested
gzip explicitly or not and make the correct decision w.r.t. what to output.
This is the same as it is today with HTTP/1.

-=R

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 30 Apr 2014 20:55:29 +1000

[snipping]

h2 to *not* fix it?

Do the other improvements in HTTP/2 not give those successes? Or are you
saying that they're not enough without ubiquitous payload compression?

>>
>> Cache-Control:no-transform explicitly forbids the proxy from altering
the representation. It's not allowed to decompress it.
> In fact what we're doing is offering two representations simultaneously.

It's a very messy way of doing it, though, and it makes me nervous. Too
many edge cases, too many potential holes. That's ok for a de facto
standard or something, but not for a formal IETF spec. And it strikes me as
a big disincentive to adoption.

To get it right would hold back HTTP/2 (unneccessarily, I say), so don't
rush it in there. Hold off for the next iteration. I'd be happy if HTTP/3
focused solely on fixing compression. I suspect others would be too.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 30 Apr 2014 04:07:52 -0700

> saying that they're not enough without ubiquitous payload compression?

Each helps for specific usecases. And I am saying that ubiquitous
compression of the payload is a big deal for many of the browsing cases
where the resources are larger, as is very often the case with HTML/css/js
and other resources.

> client, must ensure that the data is uncompressed when forwarding to the
> HTTP/1 client since the client didn't ask for c-e gzip.
> >>
Arguably the t-e stuff is more complicated, so I guess I disagree there.
I'm certainly up for other fixes in HTTP/3, but I don't feel that the
feature which is already implemented is something we should throw out now
when the fix is relatively easy.
-=R

---

From: Roland Zink <[email-masked]>
Date: Wed, 30 Apr 2014 13:45:42 +0200

On 30.04.2014 12:26, Roberto Peon wrote:
>
>
>
> On Wed, Apr 30, 2014 at 2:35 AM, Roland Zink <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>>     >>> As I described it, its use by the originator of an entity is
>>     not mandated, instead behaviors are mandated of recipients when
>>     it IS used.​
>>
>>         >>>​
>>         >>​
>     For C-E this means for example the server decides if the client
>     can do a seek. Some interactive clients would prefer to do seeks
>     over getting the content compressed. Whereas downloaders would
>     prefer the content to be compressed. T-E would allow to have both
>     seek and compression.
>
>
> The server *always* decides what to send, whether c-e or t-e is used. 
> The fact that the server *may* use gzip does not *require* it to use 
> gzip, and with my proposal, the server knows if the client requested 
> it explicitly or not, and it certainly can see if there is a range 
> request and make the appropriate response.
>
So when the server receives a range request then it decides based on the 
a-e if the range is for the compressed or the uncompressed content? If 
the a-e includes gzip it uses part of the compressed content and 
otherwise it returns some part of the uncompressed content and doesn't 
compress the answer?

> T-E is theoretically wonderful if one ignores real deployments in 
> today's world where the majority of HTTP/1.X servers don't actually do 
> transfer-encoding: gzip, and thus HTTP2 gateways would have to do c-e 
> to t-e translation (which might be rather error prone in its own way) 
> or have to bear the expense of doing the compression themselves-- 
> something which is untenable. This ignores the the security issue of 
> knowing when t-e is safe, which I'll address again below.
>
I agree. To avoid the c-e to t-e translation the gateway should do the 
compression. Aren't HTTP2 clients expected to detect if the server is 
HTTP/2 or HTTP/1.1 and do the appropriate thing? For clients only 
talking through a gateway it should be possible to do the compression at 
the gateway.
>>     which is what is causing the pain. The client expresses that it
>>     wants gzip. The intermediary doesn't do it because it makes
>>     numbers better, increases throughput, or because they're too lazy
>>     to implement it., all at the cost of the decreased user experience.
>>
>>         <snip>
>>         >
>>         competitive driver to deliver good experience/latency is
>>         causing interop failures today where servers will send gzip'd
>>         data whether or not the client declares support in a-e.
>>         >
>>
>>         Wait, you're saying the whole motivator here is that servers
>>         don't comply with the protocol? So you're changing the
>>         protocol to accommodate them? That does not feel right to me,
>>         at all; it's not just blessing a potential misuse of C-E,
>>         it's wallpapering over a flat out abuse.
>>
>>     Partially.
>>     I'm saying that intermediaries are doing things which are
>>     incenting implementors to break compatibility with the spec, and
>>     that implementors are doing so because it makes the users happy.
>>     In the end, making the users happy is what matters, both
>>     commercially and privately. The users really don't care about
>>     purity, and will migrate to implementations that give them
>>     good/better user experience.
>>
>>         But even so, why do you have to fix it in HTTP/2? And why
>>         does it hurt h2 to *not* fix it?
>>
>>
>>     decrease/performance increase, and, frankly, there is little
>>     practical motivation to deploy HTTP/2 if it doesn't succeed in
>>     reducing latency/increase performance.
>>     Success isn't (or shouldn't be) defined as completing a protocol
>>     spec, but rather, getting an interoperable protocol deployed. If
>>     it doesn't get deployed, the effort is wasted. If it doesn't
>>     solve real problems, the effort is wasted.
>>
>>     T-e based compression costs too much CPU, especially as compared
>>     with c-e where one simply compresses any static entity once and
>     If it is really T-E you can do the same compression of static
>     entities when the whole file is delivered, it would be different
>     for range requests or the frame based approach.
>
>
Even then the server could pre-compute the compression for the frames it 
want to send and send always the same, although you probably need a new 
tool to do so.

>     Don't think this is true, the same bytes can be sent as either T-E
>     or C-E. For the frame based approach some numbers were given.
>
>
> The same bytes can't be sent in both, unless the we're willing to 
> suffer vastly increased DoS surface area and memory usage OR we do the 
> frame-based approach, which will have marginally worse compression.
>
> There are, but I'm not talking about those. My problem is dealing with 
> the rest of the world, which is mostly HTTP/1.X and is unlikely to 
> rapidly change.
> In other words, I'm concerned mainly with HTTP/1.X clients and 
> especially servers.
>
>>     T-e would require that any gateway acting as a
>>     loadbalancer/reverse proxy would either need to know which
>>     resources it could compress,  or forces us to not use compression.
>     The gateway can forward the compressed content unmodified. The
>     gateway is only forced to do something if either the server or the
>     client doesn't support compression.
>
>
> something outside the protocol. Compression via t-e without knowing 
> whether it is safe or not allows attackers to discern ostensibly 
> secret information. This is NOT acceptable.
Is this still true when there is no compression context shared between 
frames and therefore between different sources? If a proxy is just 
forwarding what it gets I can't see a difference between C-E and T-E.

>>     requires content authors to change how they author content
>     automatic then this would be fine. Currently there is server
>     configuration, for example zlib.output_compression in php.ini, and
>     the possibility to do this in the content, for example in PHP
>     something like ob_start('ob_gzhandler'). I guess there is a lot
>     more authors are not aware off.
>
>
> In general, if a resource wasn't compressed on output from an 
> endpoint, it shouldn't be when received by any other endpoint.
>>         > The proxy, when forwarding the server's response to the
>>         HTTP/1 client, must ensure that the data is uncompressed when
>>         forwarding to the HTTP/1 client since the client didn't ask
>>         for c-e gzip.
>>         >
>>
> Why is range a problem? If a server wishes to service a range request, 
> it need not compress the output.
So turn off compression just in case the client will do a range request? 
Range requests are a problem as with C-E gzip the range is for the 
compressed content and without it is for the uncompressed content. The 
server would need to know for which representation the request is in 
order to decide what to do.

> With the proposal, the server will be able to know if the client 
> requested gzip explicitly or not and make the correct decision w.r.t. 
> what to output. This is the same as it is today with HTTP/1.
You mean when it is not explicitly requested it will not send gzip content?

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 01 May 2014 01:45:53 +1200

> shouldn't be when received by any other endpoint.

This seems wrong. The general case is a resource not compressed when
received by an endpoint it should not be compressed when leaving that
*same* endpoint.
Which I understand is what the proposals about C-E:gzip are saying
gateways should do:
  Accept implicit gzip within HTTP/2 so servers can emit it and
decompress for identity-only representations as soon as they get to any
HTTP/1 hop. The 1.1->2.0 transitions should obey the HTTP/1 senders use
of T-E:gzip (if they attempt it) or retain identity on the new HTTP/2 hop.

Essentially, resources start out compressed but anyone can decompress
and it stays uncompressed for the remainder of the journey.

I see no problem with a clause in the HTTP/2 spec regarding *T-E*
mandating that T-E:gzip can only be removed, never added.

This whole implicit C-E smells like an attempt to rename HTTP/1 T-E:gzip
as HTTP/2 C-E:gzip and lump all the resulting deployment problems on the
gateway implementers shoulders.

> support T-E: gzip, this ends up being a problem for HTTP/2 and T-e: gzip.

No problem there. The HTTP/2 gateway already has mandatory
(de)compression support in all of these proposals and the existing specs
text.

Sending traffic received with T-E:gzip into HTTP/1 is a simple
decompression.
Receiving traffic from HTTP/1 does not require any compression unless
the HTTP/1 endpoint *does* support T-E:gzip, in which case it is
optional to do anything for HTTP/2.

==> I would like to point out again that the security worries for T-E
*do not exist* unless the HTTP/2 hop is *adding* compression on its own.

Amos

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 30 Apr 2014 23:29:02 +0000

maybe it needs to be a different header.

since a C-E gzipped version of the same URI is strictly speaking a 
different entity (different E-tag etc) you can't arbitrarily swap 
between them.

however what's the goal here - simply to increase the use of compression 
correct?

Why mess with C-E or any entity headers?  What about another way of 
effectively doing T-E, but that doesn't need to be computed each time.  
Like Message-Encoding or something but which can be cached by the O-S or 
edge server or proxy or whatever, and it retains the original entity 
headers.  Maybe that is just T-E.

Adrien

------ Original Message ------
From: "Amos Jeffries" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 1/05/2014 1:45:53 a.m.
Subject: Re: Making Implicit C-E work.

>>>can do
>>The
>>today's
>>to
>>>which
>>>>>competitive
>>>>comply
>>>implementors
>>>commercially and
>>>hurt
>>>decrease/performance
>>>spec, but
>>>with c-e
>>>entities
>>>or C-E.
>>suffer
>>especially
>>>loadbalancer/reverse
>>>gateway is
>>whether
>>>not
>>>configuration,
>>endpoint, it