Subject: Compressing HTTP headers

From: Poul-Henning Kamp <[email-masked]>
Date: Mon, 07 Jul 2014 12:33:51 +0000

I have looked at HPACK and I'm pretty certain that we will want to do
better, if not now, then later[1].

As far as I can see, no space has been set aside for versioning HPACK
nor for using entirely different algorithms for compression ?

Obviously, a SETTINGS_ will be needed to tell what is supported
above or beyond HPACK, an IANA registry for same etc. etc.

But there will need to be an indication in each HEADERS frame of
what algorithm/version of compression is used.

I don't want to spend time on SETTINGS and IANA registry at this
point, that can wait till it become srelevant, for now I would be
happy if we can just reserve a field somewhere, demand its value be
zero, and note that it is for future expansion of header compression.

Can we please append a byte to the HEADERS frame for this purpose ?

Poul-Henning

[1] For background, here are some random notes I took some times back about
    domain/header specific compression:

For instance, both "User-Agent" and "Server" consists of largely well known
keywords, and using tailored dictionaries can save a lot.  There is a risk
here of tailoring *too* much, we don't want to make people think twice about
correctly declaring new version numbers for instance, but mutadis mutandis,
there is a lot to gain.

If nothing else just making 0x80 mean "compatible" would shave 9
bytes of pretty much all User-Agent headers, and with 0x81 meaning
"Mozilla", 0x82 "Win32" etc, it adds up really fast.

Such a "tokenpression" could be applied as s preprocessor before a general
purpose compressor such as HPACK or it could be done by expanding the
vocabulary of HPACKS default dictionary.  (The worries about variable
size attacks does not seem relevant in this specific case).

Likewise, the Date header can be compressed to 4 bytes in a
domainspecific (time_t) way, speeding up processing for sneaky
implemenations at the same time.

Set-Cookie, Last-Modified and various other headers also contain
dates which could be similarly compressed.

A very large percentage of Cookie/Set-Cookie headers can be compressed
by scanning the "value" for characterset and decoding/encoding
well-known-ascii-representations, so that for instance:

	Set-Cookie: foobar="0123456789abcdef"; [...]

becomes:

	foobar= <HEX> len=8 0x01 0x23 0x45 0x67 0x89 0xab 0xcd 0xef

Likewise "path=/" is so universal that we can make it tacit and only
transmit anything if it is not there or has another path.

These are the "big" ones, but there are other standard headers which show
obvious potential, Content-Type:, Vary:, Content-Length:, Age: and so on.

Some of these compressions would be CPU neutral and some, like dates
could even save CPU in many cases, all have the potential to save
memory and relevant bandwidth.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 7 Jul 2014 09:52:04 -0700

> nor for using entirely different algorithms for compression ?

There are two options:

* New setting (synchronization of use would have to rely on
acknowledgement; therefore, it would be delayed a little)

* New ALPN token

In both cases, reserving code points isn't necessary.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 07 Jul 2014 19:41:42 +0000

In message <[email-masked]>, Martin Thomson w

>acknowledgement; therefore, it would be delayed a little)

Which goes directly against any reason anybody could have to
try to improve the compression in the first place.

>* New ALPN token

Which makes it an all or nothing proposition, and precludes
using different compressions for different header-sets,
depending on circumstances etc.

No, I really think we should reserve some bits (4-8) in the
HEADERS frame for this purpose.

Considering how many bits people are willing to spend on
the priority stuff, this cannot be a big deal...

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 8 Jul 2014 17:44:05 -0700

On Mon, Jul 7, 2014 at 12:41 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <
> [email-masked]>,
> Martin Thomson w
> rites:

How do you know the peer supports the selected compression algorithm? If
there's a negotiation step, why don't we just use the ALPN token to resolve
negotiation without adding additional roundtrips?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 09 Jul 2014 06:21:32 +0000

In message <[email-masked]>, =?UTF-8?B?V2lsbG

>negotiation without adding additional roundtrips?

Primarily because I can easily see different compression algoritms or different subsets
of one algoritm being used for different headersets.

Secondarily because it would allow us to make the HPACK reference an option.

Finally, given that we are going to mandate one compression algoritm as the
default, we know there will be no interop issue, everybody will
always be able to talk to each other.

All we need is SETTINGS_I_ALSO_SQUEEZE(list) and a flag in
HEADERS/PUSH_PROMISE which says how things are squeezed.

The first HEADERS from the client may not be able to take advantage
of alternate compressions but HPACK is good enough to make that a non-issue.

---

From: <[email-masked]>
Date: Thu, 10 Jul 2014 11:15:47 +0000

On Monday,07 July 2014 14:34, [email-masked] wrote:
> As far as I can see, no space has been set aside for versioning
> HPACK nor for using entirely different algorithms for compression ?
>
> happy if we can just reserve a field somewhere, demand its value
> be zero, and note that it is for future expansion of header compression.
>

+1

I would like to see part of the field set aside for experimental use.  (Currently the only way for experimenting with new compression algorithms is via an extension.)

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.