Subject: Re: Support for gzip at the server #424 (Consensus Call)

From: Julian Reschke <[email-masked]>
Date: Sat, 10 May 2014 15:29:03 +0200

On 2014-03-25 00:36, Mark Nottingham wrote:
>
> On 21 Mar 2014, at 6:01 pm, Julian Reschke <[email-masked]> wrote:
>
>>>>> I don't think anyone is talking about *limiting* what you can do in HTTP/2 here -- what's being discussed is whether server-side support for GZIP content-coding in requests should be *required*.
>>>>
>>>> I think it would be good if we (a) encouraged servers to do it, and (b) clarified error handling if you don't.
>>>>
>>>> 1) Define a status code for "unsupported content-encoding" (plus maybe discovery via a Accept-Encoding header field that can be sent with it)
>>>
>>> Right now this falls into 415 Unsupported Media Type:
>>>
>>> """
>>>     The 415 (Unsupported Media Type) status code indicates that the
>>>     origin server is refusing to service the request because the payload
>>>     is in a format not supported by this method on the target resource.
>>>     The format problem might be due to the request's indicated Content-
>>>     Type or Content-Encoding, or as a result of inspecting the data
>>>     directly.
>>> """
>>>
>>> So, it could be a new status code (that takes "or Content-Encoding" out of the definition of 415), or it could be a header on 415 that further refines its semantics.
>>
>> Ah. I had forgotten that we added this to the description of 415.
>>
>> So if we want to go down that route, we could recommend a 415 and in addition elevate "Accept-Encoding" to a response header field that could be used with 415.
>
> That seems to make sense, but it isn't something specific to HTTP/2, and it's extending the semantics of a HTTP/1 header field. That sounds like a new spec that updates p2-semantics.
>
> Julian, do you want to sketch that out so people can have a look?
>
> Cheers,

Done here:

   http://tools.ietf.org/html/draft-reschke-http-cice-00

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 12 May 2014 02:12:39 +1200

>   http://tools.ietf.org/html/draft-reschke-http-cice-00

WFM.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 12 May 2014 12:03:45 +1000

LGTM. I’d tweak these to paragraphs:

>    This specification extends that definition to allow "Accept-Encoding"
>    as response header field as well.  When present, it indicates what
>    content codings a server is willing to accept in requests.  In
>    particular, a field value that contains "identity" only implies that
>    no content codings are supported at all.
> 
>    Note that this information applies to the resource to which the
>    request was addressed.  The set of supported encodings might vary for
>    different resources on the same server, and could also vary depending
>    on other aspects of the request (such as the request method).

to:

“””
This specification extends that definition to allow "Accept-Encoding"
as a response header field as well.  When present, it indicates what
content codings a resource is willing to accept in future requests.  
A field value that only contains "identity" implies that no 
content codings are supported.

Note that this information is specific to the requested resource.  The
set of supported encodings might be different for resources on the 
same server, and could also change depending on other aspects of the 
request (such as the request method).
“””

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 12 May 2014 13:05:30 +1000

Technically this looks good to me (a little better after mnot's tweaks).

I'm left wondering:

* What does it mean for a server to receive an encoded payload? E.g. if I
PUT a gzipped file, should the server store the gzip file? And/or should it
use it to generate/overwrite an unzipped copy? And if so, what metadata
does the unzipped copy get?

* How does one convey that^ decision, and other options, to the user? I'm
imagining a file upload dialog that says either "Any file you upload
(including a gzip file) will be stored as-is", or "If you upload a gzip
file it may be unzipped at the server before being processed" with a
checkbox to allow the user to instruct the UA to send it as
Content-Type:application/gzip rather than Content-Encoding:gzip. I know
this was not intended to be used in browsers, but ... <dramatic ellipsis>.

* Also: how does it play with RFC 1867/2388? At a glance I'd imagine that
any content-encoding is decoded before the multipart/form-data is parsed;
are there any emergent issues with that?

As a completionist I like that this proposal fills in a gap, and I think
there is value in having it specified, but overall it feels like we might
be packing yet more transport compression into entity encoding, rather than
addressing the existing mess.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 12 May 2014 13:14:57 +1000

On 12 May 2014, at 1:05 pm, Matthew Kerwin <[email-masked]> wrote:

> * What does it mean for a server to receive an encoded payload? E.g. if I PUT a gzipped file, should the server store the gzip file? And/or should it use it to generate/overwrite an unzipped copy? And if so, what metadata does the unzipped copy get?

You’d store the gzip’d representation, with a vanilla PUTtable resource. Remember, CE is a property of the representation, not a “transparent” encoding like TE. 

That said, the resource is free to have side effects attached to a PUT — e.g. decompressing something.

> * How does one convey that^ decision, and other options, to the user? I'm imagining a file upload dialog that says either "Any file you upload (including a gzip file) will be stored as-is", or "If you upload a gzip file it may be unzipped at the server before being processed" with a checkbox to allow the user to instruct the UA to send it as Content-Type:application/gzip rather than Content-Encoding:gzip. I know this was not intended to be used in browsers, but ... <dramatic ellipsis>.

The PUT response type and headers, and the interface that allows the PUT.

> * Also: how does it play with RFC 1867/2388? At a glance I'd imagine that any content-encoding is decoded before the multipart/form-data is parsed; are there any emergent issues with that?

Don’t think so...

> As a completionist I like that this proposal fills in a gap, and I think there is value in having it specified, but overall it feels like we might be packing yet more transport compression into entity encoding, rather than addressing the existing mess.

Well, the real question is whether anyone will implement it, surely?

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Mon, 12 May 2014 08:24:13 +0200

On 2014-05-12 05:14, Mark Nottingham wrote:
> You’d store the gzip’d representation, with a vanilla PUTtable resource. ...

Maybe. But it really doesn't matter what the server "stores", what's 
relevant is what it returns on GET.

What particular metadata are you concerned with?

> ... Remember, CE is a property of the representation, not a “transparent” encoding like TE.

But that doesn't mean the server can't do whatever it wants with it.

> That said, the resource is free to have side effects attached to a PUT — e.g. decompressing something.

Right.

>> * How does one convey that^ decision, and other options, to the user? I'm imagining a file upload dialog that says either "Any file you upload (including a gzip file) will be stored as-is", or "If…

My understanding is that you don't convey it at all. Could you explain 
what use case you're thinking of?

>> * Also: how does it play with RFC 1867/2388? At a glance I'd imagine that any content-encoding is decoded before the multipart/form-data is parsed; are there any emergent issues with that?

Correct.

> Well, the real question is whether anyone will implement it, surely?

I'm all for making T-E work in practice, but that shouldn't stop us from 
improving C-E.

This change is so tiny what it really should have been in HTTPbis. We 
just were too late thinking about it.

Best regards, Julian

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 12 May 2014 17:38:42 +1000

On 12 May 2014 16:24, Julian Reschke <[email-masked]> wrote:

>>> * What does it mean for a server to receive an encoded payload? E.g. if
>>> I PUT a gzipped file, should the server store the gzip file? And/or should
>>> it use it to generate/overwrite an unzipped copy? And if so, what metadata

I used "store" as a short hand for "provide a subsequent representation
based on what was received". I don't overly care if the file is compressed
or not in their data store.

Regarding metadata, originally I had in mind properties like Last-Modified
time. I may not be thinking clearly about it at the moment, I blame my
cold. I also think I need to look into what people currently do when a PUT
contains Last-Modified.​

> what use case you're thinking of?
As a user of HTTP I am currently able to serve a resource using three
equivalent (to a greater or lesser degree) representations:
* foo.html (c-t:text/html)
* foo.html.gz (c-t:text/html, c-e:gzip)
* foo.html.gz (c-t:application/gzip)
For my own convenience, I've statically generated two files, and use some
business logic and configuration to determine which file I use to service
each request, with what headers. But I can only do all this if I'm on the
server side.

​I'd also like to be able to upload files, and depending on what files I
have on my workstation​ PC, I'd like the option to:
* upload foo.html, updating the existing foo.html
* upload a pre-zipped foo.html.gz and have it deflated by the server,
updating foo.html
* upload foo.html.gz, updating the existing foo.html.gz

​(I imagine I'd have set up my server to accept a POST to the directory
resource, so we don't have to worry about violating the semantics of PUT.)

​For the first option, the browser can do dynamic compression. I'd argue
T-E is better here, but pragmatically if the data gets through fine, I'm
happy with whatever works.​

For the second and third options I'd choose the same file in the
file-picker, but I'd want the browser to send it with either
c-t:text/html+c-e:gzip, or c-t:application/gzip, based on ​the client-side
equivalent of my business logic and configuration. The problem is, being a
browser, it would have to use UI elements to allow me to specify which I
want.

In the current world I only have the first and third options, because they
don't use content encoding on the request; so effectively I'm after some UI
element that allows me to tell the browser that what I'm uploading uses
C-E. The same sort of static C-E I currently use today on my server.

An alternative solution is to use different PUT/POST resources (e.g. using
query parameters to tell the server whether I want to unzip or not), or
extra parameters in the POST, or something to that effect. However that's
just moving the C-E negotiation out of the protocol, into the application;
so why bother specifying it here in the first place? If it's just for the
dynamic compress case, well, HTTP/2 already has a setting/flag to allow
dynamic compression in either direction. *hint hint*

> This change is so tiny what it really should have been in HTTPbis. We just
> were too late thinking about it.
>
>
​Yeah, I'd thought in the past that it was a little hole just waiting to be
specced in​, but I never had enough of a reason to use it for me to
complain about it.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 12 May 2014 10:18:30 +0200

On 2014-05-12 09:38, Matthew Kerwin wrote:
> based on what was received". I don't overly care if the file is
> compressed or not in their data store.

I would expect the server to return the gzipped content upon GET (with 
C:E gzip) unless it determines that the client doesn't support gzip.

> Regarding metadata, originally I had in mind properties like
> Last-Modified time. I may not be thinking clearly about it at the
> moment, I blame my cold. I also think I need to look into what people
> currently do when a PUT contains Last-Modified.​

I believe most servers ignore it, but if they don't, I would expect them 
to apply it internally consistently.

Doesn't the same question apply anyway? What is L-M if the server gzips 
on the fly?

>             * How does one convey that^ decision, and other options, to
>             the user? I'm imagining a file upload dialog that says
>             either "Any file you upload (including a gzip file) will be
>             stored as-is", or "If you upload a gzip file it may be
>             unzipped at the server before being processed" with a
>             Content-Type:application/gzip rather than
>             Content-Encoding:gzip. I know this was not intended to be
>             used in browsers, but ... <dramatic ellipsis>.
>
>
>     My understanding is that you don't convey it at all. Could you
>     explain what use case you're thinking of?
>
>
> For my own convenience, I've statically generated two files, and use
> some business logic and configuration to determine which file I use to
> service each request, with what headers. But I can only do all this if
> I'm on the server side.

Ok. I can see why you'd want that when serving large files (tar.gz etc), 
but why for HTML?

> resource, so we don't have to worry about violating the semantics of PUT.)

I don't believe you'd be violating PUT semantics here.

> c-t:text/html+c-e:gzip, or c-t:application/gzip, based on ​the
> client-side equivalent of my business logic and configuration. The
> problem is, being a browser, it would have to use UI elements to allow
> me to specify which I want.
>
> In the current world I only have the first and third options, because
> they don't use content encoding on the request; so effectively I'm after
> some UI element that allows me to tell the browser that what I'm
> uploading uses C-E. The same sort of static C-E I currently use today on
> my server.

Understood.

I believe this is a browser issue, not a protocol issue.

> An alternative solution is to use different PUT/POST resources (e.g.
> using query parameters to tell the server whether I want to unzip or
> not), or extra parameters in the POST, or something to that effect.
> However that's just moving the C-E negotiation out of the protocol, into
> the application; so why bother specifying it here in the first place? If
> it's just for the dynamic compress case, well, HTTP/2 already has a
> setting/flag to allow dynamic compression in either direction. *hint hint*

That's fine, but it doesn't help with 1.1.

>             As a completionist I like that this proposal fills in a gap,
>             and I think there is value in having it specified, but
>             overall it feels like we might be packing yet more transport
>             compression into entity encoding, rather than addressing the
>             existing mess.
>
>

(to Mark: there's really little to implement here in practice (for 
gzip). Servers that already allow C-E:gzip essentially don't need a 
change at all; servers that do not would switch to from status 400 to 
415 (httpbis already says so) and add "Accept-Encoding: identity". Seems 
I should a deployment considerations chapter :-)

>     I'm all for making T-E work in practice, but that shouldn't stop us
>     from improving C-E.
>
>     This change is so tiny what it really should have been in HTTPbis.
>     We just were too late thinking about it.
>
>
> ​Yeah, I'd thought in the past that it was a little hole just waiting to
> be specced in​, but I never had enough of a reason to use it for me to
> complain about it.

Best regards, Julian

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 13 May 2014 10:21:41 +1000

On 12 May 2014 18:18, Julian Reschke <[email-masked]> wrote:

> I would expect the server to return the gzipped content upon GET (with C:E
> gzip) unless it determines that the client doesn't support gzip.
>
>
​I think that ​underpins my confusion. Thinking a bit more clearly, I see
that if I PUT a c-e:gzip resource, I can GET that c-e:gzip resource, and if
I don't accept the gzip encoding the server is free to do whatever it
likes, including 406. I had a much more convoluted and overcomplicated
state diagram in my head.

> Doesn't the same question apply anyway? What is L-M if the server gzips on
> the fly?
>
>
As long as you can guarantee that the de/compression routines produce
identical bytes on subsequent invocations, the two (or more)
representations can share last-modified (but not ETag).

Hmm.. ​That pretty concisely addresses my concern.

> Ok. I can see why you'd want that when serving large files (tar.gz etc),
​The Google told me to! <
https://developers.google.com/speed/docs/insights/EnableCompression>​

But seriously, it might be a bad example, but​ for any resource I serve I
want to choose between raw, content-encoded, or gzipped. I may make
consistent choices for a single resource, but the same decision logic can
apply to all resources.

​Your tarball example​ opened up a whole new set of questions in my head,
mostly about encoding-specific resources and representations. I'll go think
about it somewhere else, though.

>> ​[...]

> I believe this is a browser issue, not a protocol issue.
Yeah ​I think that's true. I'll stop complaining.​ :)

> That's fine, but it doesn't help with 1.1.
​This goes directly to Mark's question: if it's available to 1.1, will any
1.1 implementers actually use it? I see it as a pretty niche concern, but
then I don't have the same overview of the internet as others on the list
do.​

That said, I think my concerns were unwarranted or have been addressed.
Sorry for the noise.