Subject: RE: current HTTP/2 spec prevents gzip of response to "Range" request

From: <[email-masked]>
Date: Wed, 26 Mar 2014 17:06:12 +0000

Hi Matthew-

On Wednesday,26 March 2014 05:06, Matthew Kerwin wrote:
> That said, I'm pretty strongly for supporting the negotiation in the first place.
> Maybe it needs to be something that's clearly a part of HTTP/2, freeing it from the
> stigma attached to the existing TE/Transfer-Encoding headers.

That would work too. We aren't married to the idea of using the TE/Transfer-Encoding headers. We just thought the trickle-down effect (to quote Ronald Reagan) might be beneficial.  If clients (e.g. browsers) support HTTP/2, then they would have to support TE.  Once browsers support TE then slowly HTTP/1.1 servers might start supporting Transfer-Encoding.

> After much deliberation I have a crazy idea: introduce a new frame type for hop-by-hop headers.
> I can think of a couple of benefits:
> * It would be easier to clearly distinguish hop-by-hop and end-to-end concerns (e.g. hopefully help clear up some of the CE/TE confusion).
> * It would provide more flexibility than, say, a settings parameter or a flag in a data frame, by allowing the same type of coding negotiation currently allowed in HTTP.
> * It would be extensible, allowing experimental headers, not unlike HH from [draft-nottingham-http-browser-hints-05].
> * It also makes it easier for people to ignore them ;)

I think we'd be happy with Transfer-Encoding, a flag in an existing frame type or an entirely new frame type.

One question though, this idea to use a frame type for _hop-by-hop_ headers seems to contradict what you said on Tuesday,25 March 2014 23:25: "Also, because TE is hop-by-hop I risk some intermediary terminating the compression, possibly negatively impacting my site's responsiveness ..."
Wouldn't the ideal answer be some sort of end-to-end transfer encoding?

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 27 Mar 2014 14:45:18 +1000

[email-masked] wrote:
>
>> For the record: I routinely configure Apache to serve
>> offline-compressed versions of files [1], which I believe is the
>> Right Way(tm) to do CE
>
> Are you sure this is the right way? I guess it depends on what kind
> of offline-compressed files you are talking about. For sure yes for
> static .tar.gz archives, etc. But if you are talking about your
> example of index.html that automatically redirects to
> index.html_gz, I would argue that this is actually a transport
> encoding because the client is supposed to remove the compression
> before presenting it to the user (see Roy's distinction between
> TE/CE in his e-mail dated Monday,24 March 2014 21:19) - you are
> just smartly saving server resources by not re-compressing the file
> every time a client requests it (IIS for example, also automatically
> does this for you).

In saying that you're making a judgement call about my motivation
for compressing the file, or alternatively for my decision to keep the
uncompressed one around.

It's important to remember that, in this case especially, the
user might be making a request for the resource at
http://www.example.com/foo/

It's entirely up to my server configuration to decide whether to
serve up the contents of index.html, or index.html_gz, or index.txt,
or any other random *representation* of that resource.  Importantly,
each of these representations is a distinct entity, with its own
entity-specific metadata (Content-Type, Content-Encoding,
Content-Length, Last-Modified, ETag, etc.) which Apache, through the
glory of stat() and MIME libraries and other magic, already handles
for me. It also allows entity-specific operations like range requests.

I think that puts me bang in line with what Roy said:
"Transfer Encoding is something that can be added or removed by the
protocol.  Content Encoding is metadata about the representation."

.

>
>> Because of the MUST NOT, I now cannot take advantage of my
>> proactive compression in HTTP/2, and instead either 1) require
>> the server to compress the response on-the-fly each time, or
>> 2) lose the compression.
>
> Not sure which MUST NOT you're referring to? But the idea for this
> exact scenario was to send the header "Transfer-Encoding: gzip" and
> then put the contents of index.html_gz directly in the response
> body.

This one: "Servers MUST not include the values "gzip" or "deflate"
in a Content-Encoding header ..."

If I configured Apache to serve the contents of index.html_gz
alongside a transfer-encoding header, I'd have to be very careful
to ensure that index.html and index.html_gz do contain exactly the
same contents (after inflation), whereas with CE there's no such
restriction -- it would be mean of me to send an older, or even
completely different, gzip'd version, but not illegal.

I'd also have a hell of a time convincing Apache which headers come
from which file (ETag? Content-Length?), let alone how to resolve
range requests. Actually, to align with HTTPbis I'd have to remove
the Content-Length, which brings its own unique pains.

.

>
>> Do you mean HTTP-p1, Section 4.3 ?
>
> Section 14.39 specifically talks about the TE header and specifies
> that chunked is always acceptable. We could reference both sections
> I guess.

-p2 <http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26>
doesn't even have a section 14.  You seem to be referencing RFC 2616,
which AFAIK isn't split into parts.

--

>
>> Also, because TE is hop-by-hop I risk some intermediary terminating
>> the compression, possibly negatively impacting my site's
>> responsiveness in a way that is outside my control; but CE is
>> end-to-end, so intermediaries shouldn't be touching the compressed
>> data (I say "shouldn't" but I know some still will).
>
> This is a good point. I'm not sure what the right answer is.  What
> do you propose? (The ideal solution, I think, would be an end-to-end
> content transfer encoding.)

I propose not confusing hop-by-hop transport with end-to-end content.
We already have an end-to-end compression mechanism, which works for
the most part. I just don't like it because people have gotten
confused by it. That doesn't mean that it *can't* be used correctly,
or that it doesn't have any value.

Even were gzip transport a MUST-level requirement for HTTP/2, there
would still be 2->1.1 gateways that are forced to strip the transport
compression because the machinery on the 1.1 side doesn't send TE
headers. Therefore I argue that we should definitely NOT get rid of
Content-Encoding: gzip. What I propose is that we recommend support
for TE:gzip, and hope that the 1.X parts of the web fade away enough
that nobody cares about them not having compressed data everywhere.

Hence "best practice guideline, not an interoperability requirement."

Of course there still remains the issue of double-compression.  I
agree with what Poul said up-thread: that there's no easy solution;
and that's something I want to see solved. However I don't know that
it's necessarily a reason to completely remove TE from the spec, which
is effectively where we're at today.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 27 Mar 2014 15:10:55 +1000

>
> I
> 
> agree with what Poul said up-thread
>

I meant Poul-Henning. Sorry.

---

From: <[email-masked]>
Date: Thu, 27 Mar 2014 20:55:27 +0000

Hi Matthew-

On 27 March 2014 05:45, Matthew Kerwin wrote:
> ... It's entirely up to my server configuration to decide whether to
> Content-Length, Last-Modified, ETag, etc.)...

What if a client requests directly /index.html_gz, do you still send that as "Content-Encoding: gzip\r\nContent-Type: text/html" or just "Content-Type: application/x-gzip"?

What about a resource e.g. /search.php that has dynamic results and the response body is gzip compressed to save bandwidth? Are you saying each unique result is a distinct entity and so that too should be "Content-Encoding: gzip\r\nContent-Type: text/html" or is this a case where it really should be called "Transport-Encoding: gzip\r\nContent-Type: text/html"?

> It also allows entity-specific operations like range requests.

What if user directly requests a range of /huge.resource (e.g a huge non-compressed entity), but you want to save bandwidth on the transfer with compression? Based on the requirements of RFC2616, I assert you can't compress without using "Transport-Encoding: gzip".

> which AFAIK isn't split into parts.

Got it. Thanks. I'll fix that.

> or that it doesn't have any value.

Can you give some examples of what you consider the confusion?

> Hence "best practice guideline, not an interoperability requirement."

In our original proposal we had guidelines for what intermediaries have to do for the 2->1.1 gateways if the 1.1 side doesn't send a "TE: gzip" header. I'm not sure why that wouldn't work with a MUST-level requirement for HTTP/2.

I'm also not sure exactly what part of our proposal you are referring to that should be a best practice guideline (also not exactly sure how you put a "best practice" guideline in an RFC - is there a guideline on how to write such a guideline).
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 28 Mar 2014 22:36:18 +1000

On 28 March 2014 06:55, <[email-masked]> wrote:

> What if a client requests directly /index.html_gz, do you still send that
> as "Content-Encoding: gzip\r\nContent-Type: text/html" or just
> "Content-Type: application/x-gzip"?
>

For starters we have to be clear that /foo/ and /foo/index.html_gz are
different resources, with different sets of available representations.
That said, in this case I'd respond with Content-Encoding:gzip. With a bit
more configuration I could set it up an opposite internal redirection, or
to send a 406, but I repeat that I've not ever revealed the URI for the
resource that only has a content-encoded representation. I would never link
directly to index.html_gz, nor would I include it in a Content-Location
header. If someone guesses the URL, and can't deal with the gzip encoding,
then it's kind of their own fault. Thus I violate a SHOULD-level
requirement, and don't feel bad at all.  ;)

What about a resource e.g. /search.php that has dynamic results and the
> response body is gzip compressed to save bandwidth? Are you saying each
> unique result is a distinct entity and so that too should be
> "Content-Encoding: gzip\r\nContent-Type: text/html" or is this a case where
> it really should be called "Transport-Encoding: gzip\r\nContent-Type:
> text/html"?

Well, each response *is* a distinct entity. But more significantly, each
response could be one of two different entities -- the HTML or the gzip.
Call them H1 and Z1. On a subsequent request you choose one of two
*different* entities to send in response, H2 and Z2. The fact that *1 are
different from *2 has no bearing at all on whether you choose to send H* or
Z* -- that decision is based on what you prefer, and what the client will
accept.

> What if user directly requests a range of /huge.resource (e.g a huge
> non-compressed entity), but you want to save bandwidth on the transfer with
> compression? Based on the requirements of RFC2616, I assert you can't
> compress without using "Transport-Encoding: gzip".
>

That depends on how well we're dancing. If you requested my /foo/ with
Accept-Encoding and Range, I could choose to send part of my gzipped file
in a 206 with appropriate Content-Range, Content-Encoding, ETag, and Vary
headers.

That way you can subsequently send a request with either If-Range or
If-Match, passing back my etag, and appropriate Accept-Encoding etc.; and I
can give you another chunk (or the rest, or none) of my gzipped file.

I've had a quick read through -p5, and I'm pretty sure the general gist
still holds. I'd have to add some more headers in the first response.

That's not to say that this is a *better* solution than sending it
transfer-encoded, but AFAIK it's perfectly valid and legal. It is less
likely to be useful (how much can you do with part of a gzip file?), but
has better guarantees for end-to-end compression. The only thing to
remember is that, if you don't have the pre-compressed representation
handy, you'll have to be certain that each time you generate it you end up
with the exact same bytes and the same etag, and that you do the
compression *before* doing the range slicing.

> Can you give some examples of what you consider the confusion?

Well, Martin Thomson got confused not three days ago in this thread. And
he's an editor of the HTTP spec.

> In our original proposal we had guidelines for what intermediaries have to
> do for the 2->1.1 gateways if the 1.1 side doesn't send a "TE: gzip"
> header. I'm not sure why that wouldn't work with a MUST-level requirement
> for HTTP/2.
>

The guidelines are missing a bunch of conditions, but even then, you're
suggesting people do something in direct violation of the protocol:

   b) the intermediary SHOULD not decompress payloads that are gzip
transfer encoded and have a :status header value not "206", and if the
intermediary elects to keep the payload compressed, MUST remove the value
"gzip" from the Transfer-Encoding header and insert the header
"Content-Encoding: gzip" in order to maintain backwards compatibility with
HTTP/1.1 clients,

That whole requirement is wrong. It should be: b) the intermediary MUST
decompress the payload. That's it. And that's true in a 1->2 gateway, or a
1->1 proxy, or a 2->2 proxy. That's what hop-by-hop transport is all about.
You can't just go inventing new content-encoded entities willy-nilly,
especially this way.

> I'm also not sure exactly what part of our proposal you are referring to
> that should be a best practice guideline (also not exactly sure how you put
> a "best practice" guideline in an RFC - is there a guideline on how to
> write such a guideline).

Basically, all of it. Particularly this:

Clients MUST support gzip compression for HTTP response bodies. Regardless
of the value of the TE request header field, a server MAY send responses
with gzip transfer encoding.

It's dangerously ambiguous, but I figure what you mean is that client MUST
support responses that include Content-Encoding:gzip and/or
Transfer-Encoding:gzip. I say leave the Content-Encoding part out of it,
since that's covered elsewhere, and say something along the lines of "TE
headers exist, and some of them include 'gzip'." Then somewhere else, like
a completely different document, say "I strongly recommend that everyone
support TE:gzip, because it's awesome." Here's a list of Best Current
Practice docs: http://www.apps.ietf.org/rfc/bcplist.html

---

From: <[email-masked]>
Date: Sat, 29 Mar 2014 22:42:50 +0000

On Mar 28, 2014, at 13:36, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:

... I say leave the Content-Encoding part out of it, since that's covered elsewhere, and say something along the lines of "TE headers exist, and some of them include 'gzip'."

That was exactly our original motivation to start this thread! Transfer codings and their associated headers (i.e. TE, Transfer-Encoding) are forbidden by the HTTP/2 spec. That's what we'd like to see, at a minimum, changed...

from Section 8.1.3: "intermediaries SHOULD also remove other connection-specific header fields, such as ... Transfer-Encoding ..."

from Section 8.1.3: "...the TE header field, which MAY be present in an HTTP/2 request, but when it is MUST NOT contain any value other than "trailers"..."

Then somewhere else, like a completely different document, say "I strongly recommend that everyone support TE:gzip, because it's awesome." ...

Why couldn't support for TE:gzip be at least a RECOMMENDED level requirement. Are there implications wrt getting http/2 out the door?

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 30 Mar 2014 11:40:04 +1000

On 30 March 2014 08:42, <[email-masked]> wrote:

>
> On Mar 28, 2014, at 13:36, "[email-masked]<mailto:
> [email-masked]>" <[email-masked]<mailto:
> [email-masked]>> wrote:
>
> > 
> ... I say leave the Content-Encoding part out of it, since that's covered
> elsewhere, and say something along the lines of "TE headers exist, and some
> of them include 'gzip'."
>
> That was exactly our original motivation to start this thread! Transfer
> codings and their associated headers (i.e. TE, Transfer-Encoding) are
> forbidden by the HTTP/2 spec. That's what we'd like to see, at a minimum,
> changed...
>
> from Section 8.1.3: "intermediaries SHOULD also remove other
> connection-specific header fields, such as ... Transfer-Encoding ..."
>

Yes, they should. Those headers are messages to the intermediaries, not
to the endpoints; they shouldn't be propagated.

from Section 8.1.3: "...the TE header field, which MAY be present in an
> HTTP/2 request, but when it is MUST NOT contain any value other than
> "trailers"..."
>

This is the whole crux of the issue. I'm not sure what motivated this
ruling, but I assume that it was mostly that the only usage of TE, aside
from "trailers", was for compression, and in the earlier versions of the
draft hop-by-hop compression was handled by a flag instead of negotiation
header. Then, having effectively removed transport headers from the
protocol, i.e. leaving the HEADERS frame and header tables full of
higher-level, end-to-end concerns (and I consider "trailers" an end-to-end
concern, for practicality reasons), other conveniences may emerge, like a
higher chance of retransmitting unmodified HEADERS frames, or even just
clarity of the spec.

To that end I suggested a new frame type for hop-by-hop/transport headers
(or a flag in the HEADERS frame, which amounts to essentially the same
thing).

Having a header allows us to negotiate compression (without mandating
support, unlike a flag), and allows experimentation and the introduction of
new things (TE: bzip2? Some new HH-style hints?). And having a distinct
frame type allows people to easily distinguish them from regular headers,
making them easier to ignore if that's your thing, and still allowing those
other conveniences. With a new frame type we can then reintroduce
TE/Transport-Encoding without hurting anybody.

---

From: <[email-masked]>
Date: Sun, 30 Mar 2014 05:11:39 +0000

On Mar 30, 2014, at 3:40, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:

On 30 March 2014 08:42, <[email-masked]<mailto:[email-masked]>> wrote:

On Mar 28, 2014, at 13:36, "[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>" <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>> wrote:

Yes, they should. Those headers are messages to the intermediaries, not to the endpoints; they shouldn't be propagated.

I guess it's obvious, but not very clear in the spec - an intermediary can't just remove a transfer coding and not do something about it (i.e. Remove the coding on the payload)

For example, "chunked" has to be converted to h2 framing.

Adding your proposed header type below would allow 1.1 -> 2 intermediaries to simply forward transfer codings besides chunked.

This is the whole crux of the issue. I'm not sure what motivated this ruling, but I assume that it was mostly that the only usage of TE, aside from "trailers", was for compression, and in the earlier versions of the draft hop-by-hop compression was handled by a flag instead of negotiation header. Then, having effectively removed transport headers from the protocol, i.e. leaving the HEADERS frame and header tables full of higher-level, end-to-end concerns (and I consider "trailers" an end-to-end concern, for practicality reasons), other conveniences may emerge, like a higher chance of retransmitting unmodified HEADERS frames, or even just clarity of the spec.

To that end I suggested a new frame type for hop-by-hop/transport headers (or a flag in the HEADERS frame, which amounts to essentially the same thing).

Having a header allows us to negotiate compression (without mandating support, unlike a flag), and allows experimentation and the introduction of new things (TE: bzip2? Some new HH-style hints?). And having a distinct frame type allows people to easily distinguish them from regular headers, making them easier to ignore if that's your thing, and still allowing those other conveniences. With a new frame type we can then reintroduce TE/Transport-Encoding without hurting anybody.

I understand your motivation better now and think it's a fine idea.

Are you up for writing the text and proposing it? (If you do, I think you should start a new thread as its scope is beyond just the purpose of this thread.)

BTW, by "ignore" you mean forward to the next hop right? In other words, you can't just drop a transfer coding header and not do anything about it.