Subject: Re: FYI: Tools to evaluate header compression algorithms

From: Mark Nottingham <[email-masked]>
Date: Sun, 6 Jan 2013 16:57:01 +1100

Quick follow-up:

I posted more about this here:
  http://www.mnot.net/blog/2013/01/04/http2_header_compression

In particular, we have graphs for all of the HAR samples I took earlier:
  http://http2.github.com/http_samples/mnot/

Cheers,

On 31/12/2012, at 1:22 PM, Mark Nottingham <[email-masked]> wrote:

> As part of his implementation, Roberto created a test harness for comparing headers. With his help, I've forked that and refactored a bit --
>  https://github.com/http2/compression-test
> Note that there's the textual output you see in the README, as well as a graph available, e.g.,:
>  http://http2.github.com/compression-test/
> 
> This makes it possible to take our collection of HAR samples:
>  https://github.com/http2/http_samples
> and get metrics on how the compression algorithms perform.
> 
> This is by no means complete. I'd like to process the HAR files so that the headers fed to the compressor more closely align with those that will be seen in a typical HTTP2 connection, and also get some more interesting metrics out.
> 
> As always, pull requests and issues welcome.
> 
> Cheers,
> 
> --
> Mark Nottingham   http://www.mnot.net/
> 
> 
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 06 Jan 2013 17:53:47 +0900

>    http://http2.github.com/http_samples/mnot/

These look very interesting. Just two points for the moment:

- Drawing connected curves seems misleading, because we are not 
mesuring/showing a continuous quantity that varies over time, but 
discrete requests and responses.

- The data sample includes big guys only. Some criticism of speedy has 
said that it is geared towards the big guys. Is there a way to get some 
more of an impression of how headers look at the long tail of websites?

Regards,   Martin.

---

From: Roberto Peon <[email-masked]>
Date: Sun, 6 Jan 2013 01:55:09 -0800

Do you have some suggestions Martin?
The obvious thing in my mind is to get submissions from site owners, but
that takes interest on their part first. :/

-=R

On Sun, Jan 6, 2013 at 12:53 AM, "Martin J. Dürst"
<[email-masked]>wrote:

>>    http://www.mnot.net/blog/2013/**01/04/http2_header_compression<http://www.mnot.net/blog/2013/01/04/http2_header_compression>
>>
>>    http://http2.github.com/http_**samples/mnot/<http://http2.github.com/http_samples/mnot/>
>>
>
> mesuring/showing a continuous quantity that varies over time, but discrete
> requests and responses.
>

---

From: Ilya Grigorik <[email-masked]>
Date: Sun, 6 Jan 2013 12:34:35 -0800

> that takes interest on their part first. :/

HTTP Archive is now scanning ~300K top domains (at least according to
Alexa). While its still "top site" biased, I think that's a pretty good
sample to work with. I believe we should be able to get the HAR files from
it.

ig

>
> On Sun, Jan 6, 2013 at 12:53 AM, "Martin J. Dürst" <[email-masked]
> > wrote:
>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 7 Jan 2013 11:35:50 +1100

On 06/01/2013, at 7:53 PM, Martin J. Dürst <[email-masked]> wrote:

> - Drawing connected curves seems misleading, because we are not mesuring/showing a continuous quantity that varies over time, but discrete requests and responses.

Yes. I tried a few other approaches, but they weren't as easy to get going (I'm pretty new to d3). If you have a better visualisation, a pull request would be very welcome (display_tsv.html in the compression-test repository is probably the best place to start).

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 7 Jan 2013 11:37:20 +1100

On 07/01/2013, at 7:34 AM, Ilya Grigorik <[email-masked]> wrote:

> The obvious thing in my mind is to get submissions from site owners, but that takes interest on their part first. :/
> 

That would be one good source, although it's just to the "top" page of each site. If someone wants to own talking to Steve and getting the HARs in suitable shape for a pull request, that'd be much appreciated.

I have a set of about 17 million links (to about 2 million distinct sites) that's more representative; it's sourced from a Wikipedia dump. Generating single-pageview HARs from them should be pretty straightforward, and it looks like getting a HAR from multiple navigations using PhantomJS is very doable: <https://github.com/ariya/phantomjs/wiki/Network-Monitoring>.

Cheers,

---

From: Roberto Peon <[email-masked]>
Date: Mon, 7 Jan 2013 11:02:36 -0800

As Mark already points out, this is pretty different as it only includes
the first page. Given how the compressors in general work, this is likely
to bias the results towards compressors that do better on only that page
load to the detriment of those which do better on further page loads.

We need traces of the "average user interaction" for each site to do a
really good job.

-=R

On Sun, Jan 6, 2013 at 12:34 PM, Ilya Grigorik <[email-masked]> wrote:

>> On Sun, Jan 6, 2013 at 12:53 AM, "Martin J. Dürst" <
>> [email-masked]> wrote:
>>

---

From: "Martin Nilsson" <[email-masked]>
Date: Thu, 10 Jan 2013 13:48:40 +0100

>   http://http2.github.com/http_samples/mnot/

One minor issue is that you are comparing both encoding and compression at  
the same time. I.e. you could have a (http, spdy) x (uncompressed, delta,  
delta-huffman, gzip) result matrix.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 14 Jan 2013 22:44:12 +1100

Perhaps, but they're not really easy to disentangle; if people are interested in the different combinations, they can easily factor the compressor code to cover the space (as we're already starting to see with James' work).

Cheers,

(of course, if you want to tackle this, a pull request would be welcome)

On 10/01/2013, at 11:48 PM, Martin Nilsson <[email-masked]> wrote:

> One minor issue is that you are comparing both encoding and compression at the same time. I.e. you could have a (http, spdy) x (uncompressed, delta, delta-huffman, gzip) result matrix.
> 
> /Martin Nilsson
> 
> --

---

From: James M Snell <[email-masked]>
Date: Mon, 14 Jan 2013 06:58:33 -0800

What would be helpful is the ability to define multiple compression runs
per compressor subdirectory... But the current design works fine too.
Pretty straightforward really.
On Jan 14, 2013 3:47 AM, "Mark Nottingham" <[email-masked]> wrote:

> Perhaps, but they're not really easy to disentangle; if people are
> interested in the different combinations, they can easily factor the
> compressor code to cover the space (as we're already starting to see with
> James' work).
>
> Cheers,
>
> > One minor issue is that you are comparing both encoding and compression
> at the same time. I.e. you could have a (http, spdy) x (uncompressed,
> delta, delta-huffman, gzip) result matrix.
> >
> > /Martin Nilsson
> >
> > --

---

From: Frédéric Kayser <[email-masked]>
Date: Mon, 14 Jan 2013 23:25:49 +0100 (CET)

Hello,
there's something I don't get, I would like to try other compression algorithms, but those are written in plain old C or assembly, sample_exec_codec.py is in Python... how I am supposed to hook something in there?

Adding New Compression Algorithms

If you wish to implement a new codec, there are two easy approaches.

1) Develop it in Python. New modules should be subdirectories of 'compressor', and should inherit from BaseProcessor there.

2) Develop it in another language, and use the 'fork' module to execute it in a separate process. See 'sample_exec_codec.py' for an example of this; it can be run like this:

./compare_compressors.py -c fork="sample_exec_codec.py" file.har

---

From: James M Snell <[email-masked]>
Date: Mon, 14 Jan 2013 14:47:06 -0800

Go here and look at the bohe directory as an example...

https://github.com/jasnell/compression-test/tree/master/compressor

Ignore the spdy3_dictionary.h file that's in there, it's not used.

- James

On Mon, Jan 14, 2013 at 2:25 PM, Frédéric Kayser <[email-masked]> wrote:

> Hello,
> there's something I don't get, I would like to try other compression
> algorithms, but those are written in plain old C or assembly,
> sample_exec_codec.py is in Python... how I am supposed to hook something in
> there?
>
>
> 1) Develop it in Python. New modules should be subdirectories of
> 'compressor', and should inherit from BaseProcessor there.
>
> 2) Develop it in another language, and use the 'fork' module to execute it
> in a separate process. See 'sample_exec_codec.py' for an example of this;
> it can be run like this:
>
> As part of his implementation, Roberto created a test harness for
> comparing headers. With his help, I've forked that and refactored a bit --

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 15 Jan 2013 10:27:14 +1100

sample_exec_codec.py is just an example; that could be any executable, as long as the format returned is similar.

Cheers,

---

From: RUELLAN Herve <[email-masked]>
Date: Wed, 16 Jan 2013 10:56:40 +0000

Hi,

I just sent a patch allowing to do that.

The following command line using twice http1 (the second time with a dummy parameter) will generate an output containing both runs.

$ python compare_compressors.py -c http1 -c http1=dummy ../http_samples/mnot/wikipedia.org.har

408 req messages processed

                   compressed | ratio min   max   std

req       http1        89,284 | 1.00  1.00  1.00  0.00

req http1=dummy        89,284 | 1.00  1.00  1.00  0.00

408 res messages processed

                   compressed | ratio min   max   std

res       http1        85,817 | 1.00  1.00  1.00  0.00

res http1=dummy        85,817 | 1.00  1.00  1.00  0.00

Hervé.

From: James M Snell [mailto:[email-masked]]
Sent: lundi 14 janvier 2013 15:59
To: Mark Nottingham
Cc: Martin Nilsson; [email-masked]
Subject: Re: FYI: Tools to evaluate header compression algorithms

What would be helpful is the ability to define multiple compression runs per compressor subdirectory... But the current design works fine too. Pretty straightforward really.
On Jan 14, 2013 3:47 AM, "Mark Nottingham" <[email-masked]<mailto:[email-masked]>> wrote:
Perhaps, but they're not really easy to disentangle; if people are interested in the different combinations, they can easily factor the compressor code to cover the space (as we're already starting to see with James' work).

Cheers,

On 10/01/2013, at 11:48 PM, Martin Nilsson <[email-masked]<mailto:[email-masked]>> wrote:

> On Sun, 06 Jan 2013 06:57:01 +0100, Mark Nottingham <[email-masked]<mailto:[email-masked]>> wrote:
>
>> Quick follow-up:
>>