Subject: HTTP Header Compaction Results

From: RUELLAN Herve <[email-masked]>
Date: Mon, 22 Oct 2012 16:26:44 +0000

Hi all,

We are currently studying the compaction results for different header encoding formats, including various SPDY versions and an internally developed format (we plan to publish its description as soon as possible).

We are wondering whether anyone is aware of a test corpus for HTTP exchanges that would be available, or could be made available. This would help us obtaining fair and realistic results.

Currently we are using web pages obtained from Alexa's top ranked sites. While computing our results, we are taking into account several parameters: separate study of requests and responses, using deflate or not (when relevant), variable number of exchanged messages (corresponding to loading from 1 to 10 web pages). Any advice on these measurements would be appreciated.

Last, we would like to share our results to help the WG discussions on this topic. As we're rather new to the IETF, we're not sure of the best way to do this. I plan to attend the IEFT 85 meeting in Atlanta, so I could use this occasion to share these results.

Regards,

Hervé

---

From: Vinayak Hegde <[email-masked]>
Date: Mon, 22 Oct 2012 22:06:40 +0530

> We are wondering whether anyone is aware of a test corpus for HTTP exchanges that would be available, or could be made available. This would help us obtaining fair and realistic results.

The internet archive stores the HTTP response as it is received and
has a huge trove of results. However I don't know if it is available
for download and replay. The arc file format is documented here -
http://crawler.archive.org/articles/developer_manual/arcs.html

Some of their code is also available on github -
https://github.com/internetarchive

Also the HTTP archive that look at some of the top alexa ranked sites
has data available - http://httparchive.org/

Regards
Vinayak

---

From: "Martin Nilsson" <[email-masked]>
Date: Tue, 23 Oct 2012 09:54:41 +0200

On Mon, 22 Oct 2012 18:36:40 +0200, Vinayak Hegde <[email-masked]>  
wrote:

>>
>> We are wondering whether anyone is aware of a test corpus for HTTP  
>> exchanges that would be available, or could be made available. This  
>> would help us obtaining fair and realistic results.
>

As discussed previously, the headers in the HTTP request may be more  
relevant to compress, since you want to fit as many requests as possible  
in the initial TCP window. In the response the headers are many times  
dwarfed by the size of the actual resource. Getting a representative  
sample for test purposes quickly gets difficult for privacy reasons, in  
particular if you capture mobile traffic, where unique identifiers like  
phone number could be added to the requests by the network operator.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Roberto Peon <[email-masked]>
Date: Tue, 23 Oct 2012 08:29:28 -0700

What Martin said... with special emphasis on the difficulty of sanitizing
the headers. :)

The other complication is that we want to see what an entire interaction
looks like (for compression), and not just the initial page load (unless
that is how the users typically interact with that site)

-=R
On Oct 23, 2012 12:57 AM, "Martin Nilsson" <[email-masked]> wrote:

>>> exchanges that would be available, or could be made available. This would
>>> help us obtaining fair and realistic results.
>>>
>>
>> http://crawler.archive.org/**articles/developer_manual/**arcs.html<http://crawler.archive.org/articles/developer_manual/arcs.html>
>>
>>
> relevant to compress, since you want to fit as many requests as possible in
> the initial TCP window. In the response the headers are many times dwarfed
> by the size of the actual resource. Getting a representative sample for
> test purposes quickly gets difficult for privacy reasons, in particular if
> you capture mobile traffic, where unique identifiers like phone number
> could be added to the requests by the network operator.
>
> /Martin Nilsson
>
> --

---

From: RUELLAN Herve <[email-masked]>
Date: Tue, 23 Oct 2012 16:00:43 +0000

Roberto,

> -----Original Message-----
> From: Roberto Peon [mailto:[email-masked]]
> Sent: mardi 23 octobre 2012 17:29
> To: Martin Nilsson
> Cc: [email-masked]
> Subject: Re: HTTP Header Compaction Results
> 
> What Martin said... with special emphasis on the difficulty of sanitizing the
> headers. :)

I've found very strange things in the response headers from our testing corpus. :-)

> looks like (for compression), and not just the initial page load (unless that is
> how the users typically interact with that site)

On this point, we are testing both the initial page load and longer sessions where the user loads several pages from the same site. The number of loaded pages clearly has a strong effect on the compaction results.

Hervé.

> 	On Mon, 22 Oct 2012 18:36:40 +0200, Vinayak Hegde
> <[email-masked]> wrote:
> 
> 
> 
> 
> 			We are wondering whether anyone is aware of a test
> corpus for HTTP exchanges that would be available, or could be made
> available. This would help us obtaining fair and realistic results.
> 
> 
> 
> <http://crawler.archive.org/articles/developer_manual/arcs.html>
> 
> 
> 
> 
> 	As discussed previously, the headers in the HTTP request may be
> more relevant to compress, since you want to fit as many requests as
> possible in the initial TCP window. In the response the headers are many
> times dwarfed by the size of the actual resource. Getting a representative
> particular if you capture mobile traffic, where unique identifiers like phone
> number could be added to the requests by the network operator.
> 
> 	/Martin Nilsson
> 
> 	--

---

From: Roberto Peon <[email-masked]>
Date: Tue, 23 Oct 2012 15:44:00 -0700

> > Subject: Re: HTTP Header Compaction Results
<snip>
> On this point, we are testing both the initial page load and longer
sessions where the user loads several pages from the same site. The number
of loaded pages clearly has a strong effect on the compaction results.

Yup, and awesome!

-=R

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 16:26:10 +1100

Hi Herve,

> Hi all,

Welcome :)

> We are currently studying the compaction results for different header encoding formats, including various SPDY versions and an internally developed format (we plan to publish its description as soo…

It's very good to hear that you're doing this. 

> We are wondering whether anyone is aware of a test corpus for HTTP exchanges that would be available, or could be made available. This would help us obtaining fair and realistic results.

I've been thinking about this too, and have started writing some software to sniff unadulterated HTTP headers (request and response) off the wire. I know that some browsers make headers available, but they also have a habit of "cleaning up" the headers before presenting them to APIs, IME.

Currently I have a pcap-based sniffer for HTTP sites <https://github.com/mnot/hdrgrab>; soon I should have a MITM proxy for HTTPS ones.

Once we start to collect a corpus, I'd suggest we put them on github: <https://github.com/http2>. That will let everyone share a common base for testing, and review it to make sure that it's appropriate.

If you'd like to put your framework for comparison up there, you'd be welcome to; please ping me. 

> Currently we are using web pages obtained from Alexa's top ranked sites. While computing our results, we are taking into account several parameters: separate study of requests and responses, using…

For the time being, mine combines all of the requests / responses on different connections to the same server into the same file, under the assumption that multiplexing will enable this. Eventually I'd like to get something more sophisticated in there.

It also removes connection headers, as they're really HTTP/1 specific.

What else should we be doing?

> Last, we would like to share our results to help the WG discussions on this topic. As we're rather new to the IETF, we're not sure of the best way to do this. I plan to attend the IEFT 85 meeting i…

If you'd like to give a short (10-20 minute) presentation in the Atlanta meeting, I'd be happy to accommodate that. Please ping me if you're interested.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 24 Oct 2012 22:25:39 +1300

> Once we start to collect a corpus, I'd suggest we put them on github: <https://github.com/http2>. That will let everyone share a common base for testing, and review it to make sure that it's approp…

FYI;  we have implemented raw HTTP header dumps on inbound/outbound 
traffic in Squid-3.2 linked with TCP socket details and timing. If 
anyone else is interested in a quick and easy source of data without 
tcpdump they are welcome to use Squid for that. "debug_options 11,2" is 
the magic config stanza to output the traffic into cache.log.

> What else should we be doing?

I think the HTTP/1 traces should leave those headers in. The data corpus 
might be used as example raw input from HTTP/1 clients or servers to 
test upgrading speeds on middleware. Which will need to also test time 
taken stripping those headers away.

The data corpus can come with an example HTTP/1->HTTP/2 convert script 
to prepare for tests of HTTP/2 native traffic speeds and/or of speed 
testing implementations gateway conversions.

> If you'd like to give a short (10-20 minute) presentation in the Atlanta meeting, I'd be happy to accommodate that. Please ping me if you're interested.

Amos

---

From: Roberto Peon <[email-masked]>
Date: Wed, 24 Oct 2012 10:05:51 -0700

On Oct 24, 2012 2:27 AM, "Amos Jeffries" <[email-masked]> wrote:
>>> We are currently studying the compaction results for different header
encoding formats, including various SPDY versions and an internally
developed format (we plan to publish its description as soon as possible).
>>
>>
>> I've been thinking about this too, and have started writing some
software to sniff unadulterated HTTP headers (request and response) off the
wire. I know that some browsers make headers available, but they also have
a habit of "cleaning up" the headers before presenting them to APIs, IME.
>>
>> Currently I have a pcap-based sniffer for HTTP sites <
https://github.com/mnot/hdrgrab>; soon I should have a MITM proxy for HTTPS
ones.

I believe that browsers using nss can be compiled to dump the TLS private
keys, and that wireshark has been patched to use them. Just another option
in case MITM proves difficult in some unknown way. It has been of great
benefit to me when debugging, regardless...

>>
>> Once we start to collect a corpus, I'd suggest we put them on github: <
https://github.com/http2>. That will let everyone share a common base for
testing, and review it to make sure that it's appropriate.
> FYI;  we have implemented raw HTTP header dumps on inbound/outbound
traffic in Squid-3.2 linked with TCP socket details and timing. If anyone
else is interested in a quick and easy source of data without tcpdump they
are welcome to use Squid for that. "debug_options 11,2" is the magic config
stanza to output the traffic into cache.log.
>>> Currently we are using web pages obtained from Alexa's top ranked
sites. While computing our results, we are taking into account several
parameters: separate study of requests and responses, using deflate or not
(when relevant), variable number of exchanged messages (corresponding to
loading from 1 to 10 web pages). Any advice on these measurements would be
appreciated.
>>
>> For the time being, mine combines all of the requests / responses on
different connections to the same server into the same file, under the
assumption that multiplexing will enable this. Eventually I'd like to get
something more sophisticated in there.
> I think the HTTP/1 traces should leave those headers in. The data corpus
might be used as example raw input from HTTP/1 clients or servers to test
upgrading speeds on middleware. Which will need to also test time taken
stripping those headers away.
>
> The data corpus can come with an example HTTP/1->HTTP/2 convert script to
prepare for tests of HTTP/2 native traffic speeds and/or of speed testing
implementations gateway conversions.

++

If we do end up with an awesome common test corpus , let's have the data
unadulterated (reformatting doesn't count as adulteration)

-=R

>
>
>
>>
>>> Last, we would like to share our results to help the WG discussions on
this topic. As we're rather new to the IETF, we're not sure of the best way
to do this. I plan to attend the IEFT 85 meeting in Atlanta, so I could use
this occasion to share these results.
>>
>> If you'd like to give a short (10-20 minute) presentation in the Atlanta
meeting, I'd be happy to accommodate that. Please ping me if you're
interested.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 24 Oct 2012 14:00:47 -0400

On Wed, Oct 24, 2012 at 1:05 PM, Roberto Peon <[email-masked]> wrote:

> keys, and that wireshark has been patched to use them.
>

for reference https://developer.mozilla.org/en-US/docs/NSS_Key_Log_Format

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 25 Oct 2012 06:39:34 +1100

> for reference https://developer.mozilla.org/en-US/docs/NSS_Key_Log_Format 

Thanks; I looked for that before, but couldn't find it. Should have asked.

I agree that the logs should be 'raw'; we can always post-process (as long as we do it in a uniform manner :)

How would people prefer to store them? I've been storing them as just text files, one per direction per stream (e.g., "response headers on this connection to 1.2.3.4"), with header blocks delimited by a blank line.  However, IIRC someone mentioned HAR as well -- any preferences?

Cheers,

---

From: James M Snell <[email-masked]>
Date: Wed, 24 Oct 2012 12:44:03 -0700

On Wed, Oct 24, 2012 at 12:39 PM, Mark Nottingham <[email-masked]> wrote:

> > for reference
> https://developer.mozilla.org/en-US/docs/NSS_Key_Log_Format
>
> How would people prefer to store them? I've been storing them as just text
> files, one per direction per stream (e.g., "response headers on this
> connection to 1.2.3.4"), with header blocks delimited by a blank line.
>  However, IIRC someone mentioned HAR as well -- any preferences?
>
>
text files would work just fine.

- James

---

From: RUELLAN Herve <[email-masked]>
Date: Thu, 25 Oct 2012 08:09:00 +0000

> From: Mark Nottingham [mailto:[email-masked]]
> Sent: mercredi 24 octobre 2012 07:26
> To: RUELLAN Herve
> Cc: [email-masked]
> > Currently we are using web pages obtained from Alexa's top ranked sites.
> While computing our results, we are taking into account several parameters:
> separate study of requests and responses, using deflate or not (when
> relevant), variable number of exchanged messages (corresponding to
> loading from 1 to 10 web pages). Any advice on these measurements would
> be appreciated.
> 

On our side, we use har files, that are generated directly by Chrome. In one har file, we have all the requests/responses to all the servers necessary to load a set of web pages. The web pages themselves come from only one server. We therefore have to do some processing to split the requests/responses according to their target server.

We've not yet implemented any specific filtering on the headers, but should probably do this to remove connection headers for example.

Regards,

Hervé.

---

From: RUELLAN Herve <[email-masked]>
Date: Thu, 25 Oct 2012 08:13:38 +0000

> From: James M Snell [mailto:[email-masked]]
> Sent: mercredi 24 octobre 2012 21:44
> To: Mark Nottingham
> Cc: Patrick McManus; Roberto Peon; Amos Jeffries; [email-masked]
> 	How would people prefer to store them? I've been storing them as
> just text files, one per direction per stream (e.g., "response headers on this

I agree that text files are OK.

HAR could also work, but are somewhat more complex to process. Moreover I think it's easier to write a HAR to text files translator than the reverse.

Hervé.

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 25 Oct 2012 22:52:12 +1300

> HAR could also work, but are somewhat more complex to process. Moreover I think it's easier to write a HAR to text files translator than the reverse.

Since the traffic is arriving in HTTP text format to produce a HAR file 
one needs to write such a translator and apply it on the data stream. 
Producing a .txt can be just a packet buffer dump.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 25 Oct 2012 21:05:55 +1100

The only thing that gives me pause about using plain text is that if we want to combine streams to the same server to simulate multiplexing, we may want to apply a time-based heuristic to do so, meaning we'll need some metadata.

Date is in responses, but not requests; I *guess* we could infer it from the paired responses, but then there's clock skew, etc.

Just a thought; I'm happy with text if that doesn't worry anyone else.

Cheers,

On 25/10/2012, at 8:52 PM, Amos Jeffries <[email-masked]> wrote:

> Since the traffic is arriving in HTTP text format to produce a HAR file one needs to write such a translator and apply it on the data stream. Producing a .txt can be just a packet buffer dump.
> 
> Amos
>

---

From: "Martin Nilsson" <[email-masked]>
Date: Thu, 25 Oct 2012 13:36:54 +0200

> What Martin said... with special emphasis on the difficulty of sanitizing

One way that perhaps is possible is to split the request headers into  
three groups, user agent-specific (accept, host, user-agent etc),  
network-specific (x-up-calling-line-id, x-msisdn etc) and site-specific  
(cookie). Processing of captured data can then generate rules for how to  
generate the headers for each set (Nokia5130c-2 device, AT&T no privilege  
network, Google search logged in repeat visit).

And immediately I realized that there are cross over issues, like the  
cookie header both being a site specific header, with user-agent specific  
formatting, as well as possibly added by the network equipment for  
specific destinations...

/Martin Nilsson

---

From: Patrick McManus <[email-masked]>
Date: Thu, 25 Oct 2012 09:04:05 -0400

+1 on the timestamp.. it also gives some idea of data rate which can be
useful for calculating cpu budgets.
And if you're going to give meta data we can include a header length
instead of the blank line. (hey, where have I heard that discussion before?
:))

-P

On Thu, Oct 25, 2012 at 6:05 AM, Mark Nottingham <[email-masked]> wrote:

> The only thing that gives me pause about using plain text is that if we
> want to combine streams to the same server to simulate multiplexing, we may
> want to apply a time-based heuristic to do so, meaning we'll need some
> metadata.
>
> Date is in responses, but not requests; I *guess* we could infer it from
> the paired responses, but then there's clock skew, etc.
>
> >>>     I agree that the logs should be 'raw'; we can always post-process
> (as
> >>> long as we do it in a uniform manner :)
> >>>
> >> HAR could also work, but are somewhat more complex to process. Moreover
> I think it's easier to write a HAR to text files translator than the
> reverse.
> >

---

From: Roberto Peon <[email-masked]>
Date: Thu, 25 Oct 2012 10:14:52 -0700

My github repo has some code for parsing the .har files. In python and js
it is trivial  (around 3 loc) to parse them. For my c++ code, I invoke
(fork)  the python to parse them and then put them in simplified http/1
header format and pipe it into the c++ code.

-=R
On Oct 25, 2012 1:13 AM, "RUELLAN Herve" <[email-masked]> wrote:

> HAR could also work, but are somewhat more complex to process. Moreover I
> think it's easier to write a HAR to text files translator than the reverse.
>
> Hervé.
>
> > - James
> >
> >
> >       Cheers,
> >
> >
> >       --

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 29 Oct 2012 11:56:32 +1100

OK, I'm going to change to produce HAR, then.

On 26/10/2012, at 4:14 AM, Roberto Peon <[email-masked]> wrote:

> My github repo has some code for parsing the .har files. In python and js it is trivial  (around 3 loc) to parse them. For my c++ code, I invoke (fork)  the python to parse them and then put them in simplified http/1 header format and pipe it into the c++ code.
> 
> -=R
>