Subject: Design Issue: GZIP flag on DATA Frames

From: James M Snell <[email-masked]>
Date: Tue, 21 May 2013 09:21:27 -0700

https://github.com/http2/http2-spec/issues/100

Currently the spec includes a requirement that all user-agents MUST
support gzip.. specifically:

  User-agents MUST support gzip compression.
  Regardless of the Accept-Encoding sent by
  the user-agent, the server may always send
  content encoded with gzip or deflate encoding.

If we're going to include this requirement, it makes more sense to do
this at the framing layer rather than the HTTP semantic layer. We can
do so easily by defining a GZIP flag on the DATA frame type. If set,
the payload of the DATA frame is compressed.

Doing so largely eliminates the need for the
accept-/transfer-/content-encoding mechanisms at the http semantic
layer.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 21 May 2013 16:25:37 +0000

In message <CABP7Rbfb92Vxrmxj6fKdt+jpO_Qknq8FRjsu5GZW=[email-masked]>

>If we're going to include this requirement,  ...

How about just nuking Accept-Encoding, and say that _all_ content is
gzip'ed, but that it is allowed to use the '-0' uncompressed mode ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: James M Snell <[email-masked]>
Date: Tue, 21 May 2013 09:27:45 -0700

Well... if I know that I'm sending a file that is already compressed,
why should I be required to attempt running it through the compressor
again? I get what you're saying, but requiring that content always be
compressed is not always optimal.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 21 May 2013 16:38:08 +0000

In message <CABP7RbeuieZ_tT=8ej6sQi94nRC79=[email-masked]>
, James M Snell writes:

>again?

Why would you do that if it is already compressed ?

---

From: James M Snell <[email-masked]>
Date: Tue, 21 May 2013 09:47:14 -0700

Actually, my apologies, for some reason my brain did not register the
"allowed to used the '-0' uncompressed mode" in your first note. The
approach you suggest could definitely work, I think.

---

From: Roland Zink <[email-masked]>
Date: Tue, 21 May 2013 19:13:10 +0200

This seem to make the introduction of new compression schemes more complex.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 21 May 2013 17:17:43 +0000

>This seem to make the introduction of new compression schemes more complex.

And what is the plausibility that any new compression schemes will ever
make that worth-while ?

It's not nill, but it makes a convincing impression of nill.

---

From: "Martin Nilsson" <[email-masked]>
Date: Tue, 21 May 2013 21:17:42 +0200

On Tue, 21 May 2013 18:25:37 +0200, Poul-Henning Kamp <[email-masked]>  
wrote:

> In message  
> <CABP7Rbfb92Vxrmxj6fKdt+jpO_Qknq8FRjsu5GZW=[email-masked]>
> , James M Snell writes:

The deflate format as defined in RFC 1951 requires 5 bytes added  
(injected) per 65k data. Also the two formats actually used in HTTP  
(deflate, RFC 1950, and gzip, RFC 1952) requires cheksums of the entire  
data added, so it's not like uncompressed mode is free.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 21 May 2013 22:19:51 +0200

>gzip'ed, but that it is allowed to use the '-0' uncompressed mode ?

In the Hybi Working Group many participants insisted on being able to
serve very large frames using `sendfile(...)`-style APIs. Deflate re-
quires sending additional data every couple of kilobytes, which would
interfere with that. I haven't studied the actual impact or how such
an argument applies to the most recent HTTP/2.0 proposals, and suspect
it's actually not a terribly big issue, but I would expect objections;
accordingly, such a proposal would have to come with some data that
alleviates concerns in this area.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 21 May 2013 22:45:43 +0200

>It's not nill, but it makes a convincing impression of nill.

There are many existing compression schemes that considerably outperform
Deflate in the compression ratio, some even at comparable consumption in
resources needed for decompression and, less so, compression. Especially
for something like "large JavaScript library" that you would compress
only once, something like http://en.wikipedia.org/wiki/.xz is quite near
a point where I would expect people to call for adoption of a new scheme
(with a filter optimized for such content > 10% better compression seems
very plausible to me). If the protocol cannot support adoption of a new
scheme, there would be pressure to move compression onto a lower level,
think "application/compressed-javascript", and I would rather avoid go-
ing there. My http://bjoern.hoehrmann.de/pngwolf/ was able to strip se-
veral kilobytes off the Google homepage, that already seemed quite worth
it...
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 21 May 2013 20:52:53 +0000

In message <[email-masked]>, Bjoer

>[...] something like http://en.wikipedia.org/wiki/.xz is quite near
>very plausible to me).

You forgot half the "worth-while" part:  The compatibility issues.

Even something as trivial as "deflate" vs. "gzip" was too hard for
some people to get right.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 21 May 2013 23:17:28 +0200

>some people to get right.

I think people will create an even bigger mess if they're forced to work
around lack of support for alternative compression schemes in HTTP/2.0.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: James M Snell <[email-masked]>
Date: Tue, 21 May 2013 14:24:04 -0700

This does not preclude the use of alternative compression schemes. If
someone chooses, it would be possible, for instance, to continue using
accept-/content-/transfer-encoding at the http semantic layer and
simply not set the GZIP flag on the DATA frame. Having the GZIP flag
would just provide an approach that would make that unnecessary in the
most common cases today. If, at some point in the future a new, more
efficient better compression algorithm overtakes gzip as the
predominant mechanism, the protocol can be updated to reflect that
fact (i.e. with a new protocol version string, etc).

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 21 May 2013 23:32:48 +0200

* James M Snell wrote:
>simply not set the GZIP flag on the DATA frame.

Having a "gzip flag" as, in effect, a shorthand for using those headers
would make it very clear how to deploy alternative compression schemes,
and would indeed address my concerns. I understood Poul-Henning Kamp to
be considering going one step further and offering no means to define
new compression schemes.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Roberto Peon <[email-masked]>
Date: Tue, 21 May 2013 14:36:39 -0700

The proposal does force anything other than gzip to use an alternate code
path for parsing/interpretation, which does not inspire confidence given my
experience with backup-generator problems, and it probably involves writing
*more* code today than the alternative of leaving it where it is in the
HTTP semantic layer.
-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 21 May 2013 21:53:36 +0000

In message <[email-masked]>, Bjoer
n Hoehrmann writes:

>I understood Poul-Henning Kamp to

Ok, so let me clarify:

I have always hated the confusion in HTTP/1 between content compressed
for transport purposes, and content compressed for application purposes.

In the former class you have HTML, javascript, CSS etc.

In the latter class you have things like .tgz, gzip'ed XML etc.

It's the former class I want to cut all the way into the bone
and take out of "Accept-Encoding" negotiation.

Accept-Encoding should be reserved only for the second class.

---

From: James M Snell <[email-masked]>
Date: Tue, 21 May 2013 15:07:32 -0700

If that's the case, and it's better to just handle data compression at
the higher levels, then the requirement that gzip MUST always be
supported independent of what is specified by the accept-* header
ought to be removed and implementations ought to just continue letting
the application decide which mechanism is best.

My goal here is to simplify things wherever possible. If we can
deprecate (or greatly simplify) accept-/transfer-/content-encoding at
the semantic layer and just say that the framing layer will handle
compression, then that's one less optional bit of complexity that us
application-level developers have to deal with.

On Tue, May 21, 2013 at 2:36 PM, Roberto Peon <[email-masked]> wrote:
> *more* code today than the alternative of leaving it where it is in the HTTP
> semantic layer.
> -=R
>
>

---

From: Frédéric Kayser <[email-masked]>
Date: Wed, 22 May 2013 01:15:28 +0200

Hello,
HTML5 accepts only UTF-8 encoding, there are better ways than Deflate to compress Unicode texts, bzip2 to start with. Deflate has no clue about UTF-8 since it is byte oriented, its search window is limited to 32 kilo bytes (in UTF-8 a single Devanagari character –used for Hindi and other languages in India– takes 3 bytes which seriously reduces the size of text that can actually be used as a reference for string matching, the same goes for other scripts like Cyrillic (Russian, Ukrainian, Bulgarian…), Greek,  Hebrew, Arabic… since they can no longer rely on single byte charsets and UTF-8 means 2 bytes per character for those).

For web performance having a compression scheme that could recognize and reverse/redo base64 encoding (Data URI, RFC2397) to handle "binary blobs" inside text files would be very appreciated.

Deflate misses some flexibility since it has no super fast mode à la LZ4 that would still provide decent compression but at much lower CPU cost (no entropy coding), nor something heavier on the other side (LZMA like).

Deflate was a nice compression scheme in the 90s, but the World (Wide Web) has changed since the 90s, look how archivers handle text files nowadays: they switch to PPMd, bzip2… because Deflate is outdated.

Compressing the headers is a good idea, but thinking about new compression schemes for the payload should not be overlooked.

Regards
Frédéric Kayser

---

From: Patrick McManus <[email-masked]>
Date: Tue, 21 May 2013 19:34:48 -0400

James, I think the existing spec language is fine. Its there based on
implementation experience (which has been collected in scenarios with no
implicit accept-encoding, as well as early spdy versions that applied gzip
on a spdy frame basis, along with incantation in there now which has worked
well imo) - you're sort of frenetically re-enacting the entire history of
the feature's many changes all in one day.

The way it is, the skids are greased for deflate as a LCD and the door is
open in the usual way to other approaches.

On Tue, May 21, 2013 at 6:07 PM, James M Snell <[email-masked]> wrote:

> HTTP

---

From: Roberto Peon <[email-masked]>
Date: Tue, 21 May 2013 17:27:44 -0700

I don't see why it is a one or the other choice.

>From a pragmatic viewpoint, we required gzip because it was lacking in a
lot of implementations, or went through a proxy which stripped it.

Requiring that it be supported enabled servers to send the data compressed.
This was essentially the result of years of being frustrated with not
knowing when it was safe to at least be able to send with a baseline
compression.

Unfortunately, gzip isn't the best compressor in the world, so we will
always need since mechanism for negotiating a compression scheme.
Http's scheme works, and many of its flaws are unavoidable in any system
with proxies.

Having played with compression at the protocol layer, my experience is that
it adds complexity where it is often not warranted and it results in (at
best )awkward apis.

-=R

HTTP

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 28 May 2013 16:19:53 +1000

Speaking personally - 

I'm -1 on this. The semantics you talk about *are* a property of the content, and you're embedding them in the transport.

--
Mark Nottingham   http://www.mnot.net/

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 28 May 2013 08:30:34 +0000

FWIW I agree also wrt C-E, since we have legacy issues that would 
prevent this working properly, although as a replacement for T-E gzip it 
could be an option, and could be quite useful to promote this.

Adrien

------ Original Message ------
From: "Mark Nottingham" <[email-masked]>
To: "James M Snell" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 28/05/2013 6:19:53 p.m.
Subject: Re: Design Issue: GZIP flag on DATA Frames
>Speaking personally -
>
>I'm -1 on this. The semantics you talk about *are* a property of the 
>content, and you're embedding them in the transport.
>
>
>