Subject: RE: Would a header/schema compiler help?

From: Henrik Frystyk Nielsen <[email-masked]>
Date: Thu, 2 Aug 2012 20:53:40 +0000

I also have trouble with the use of compression over the headers for two reasons:

1) While we know that compression can work wonders on entity bodies, it obscures the headers which are critical for inspection and manipulation by everybody down the message path.

2) Further, it is unclear whether there is any noticeable performance gain from doing so. The only headers that today are open-ended are really User-Agent and Set-Cookie/Cookie pairs. In all our data where we don't include these headers we see no gain from using header compression whatsoever as long as you are conservative in how many headers you choose to include.

In other words, based on the data we have, focusing on optimizing UA and cookie headers may offer tangible performance gains.

Unless we can find convincing data I would argue that we should start the header discussion by focusing on the headers that we know have problems and then go from there.

Henrik

-----Original Message-----
From: Phillip Hallam-Baker [mailto:[email-masked]] 
Sent: Tuesday, July 17, 2012 07:10
To: [email-masked] Group
Subject: Would a header/schema compiler help?

One of the design choices I find a little troubling in SPDY is the use of header compression over a compact representation.

Compression algorithms worry me because the only way I can implement them is to call someone's library. I can write my own code to implement RFC822 header parsing. In fact I have done so a dozen times at least.

ASN.1 and XML schemas are undoubtedly how to do a protocol 'right' in the academic sense of get some good abstractions in there. But they both come with hideous baggage. ASN.1 started too large and has been tweaked endlessly until it became nonsense. XML encoding is bloated and XML Schema drags in the whole SGML legacy that the WG was meant to chuck in the dustbin. I dare you to use a model group in your next XML spec.

But JSON proves that it does not have to be that way. Over the past couple of months I have put together what amounts to a schema compiler for JSON. I am currently working on putting the compiler and the tools used to build it out on Github under an open source license.

Unlike your regular schema compiler, this one makes it easy to retarget the code generation at a different language. Writing a JSON back end for C# took me about a week (and I was developing the tool at the same time). I could modify that code to generate C in a day. Want python, perl, Java, no problem.

The JSON protocol synthesizer is built using the Goedel meta-synthesizer (which is of course written in itself). Comodo owns the protocol synth, I own the meta-synth. They should both be up on github under an MIT license in a few weeks.

It seems to me that it would be useful to have a compiler for RFC822 style headers. The basic strategy being similar to a 'jack up' house renovation. First we jack up the level of abstraction in the HTTP spec by defining the headers in terms of a schema that is designed for that specific purpose. Then we build in the new ground floor (aka encoding) underneath it.

--
Website: http://hallambaker.com/

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 2 Aug 2012 14:05:25 -0700

Question,

In the case that a server and client do the HTTP connection keep-alive
thing, does the client send the cookies on the first request only or
for each request?

If we have MUX, that may be where the performance gain comes.

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 2 Aug 2012 23:29:21 +0200 (CEST)

On Thu, 2 Aug 2012, Phillip Hallam-Baker wrote:

> thing, does the client send the cookies on the first request only or for 
> each request?

In each request. Stateless, remember? =)

-- 

  / daniel.haxx.se

---

From: David Morris <[email-masked]>
Date: Thu, 2 Aug 2012 14:30:47 -0700 (PDT)

> If we have MUX, that may be where the performance gain comes.

The client must send the cookies, UA, etc. on every request as their is no
state preserved between requests.

The accept headers can get quite bloated as well.

There is no reason for most headers to be repeated if the protocol
provided for reuse of previous headers. I lean toward explicit reuse,
such as a 'mark' of some kind in the re-using request or response
to indicate a reference to a previous header.

I don't advocate simple compression (LCZ et al), but I'm surprised
by the finding that only the UA and Cookie headers made a difference.
I would expect stream compression (e.g., keeping the dictionary open)
to reduce most headers to almost nothing after a very few R/R pairs.
Also, using an initial dictionary trained with header names,
common UA terms, accept tokens, etc. should make a big difference.

Dave Morris

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 2 Aug 2012 14:49:31 -0700

I remember what we wrote. What the code does is very frequently different.

A TCP connection itself represents state so I don't see any real
reason not to permit cookie caching for the duration of the session.
And I would much prefer to have explicit state than state implicit in
some compression dictionary.

---

From: Mike Belshe <[email-masked]>
Date: Thu, 2 Aug 2012 15:09:11 -0700

On Thu, Aug 2, 2012 at 1:53 PM, Henrik Frystyk Nielsen <
[email-masked]> wrote:

> 1) While we know that compression can work wonders on entity bodies, it
> obscures the headers which are critical for inspection and manipulation by
> everybody down the message path.
>

So far, this has not been a problem.

The nice thing about a general purpose compressor is that it is very
generic - it works on any body of data.  Obviously, a structured schema can
be highly tailored, but is also less flexible to changes over time.

> 2) Further, it is unclear whether there is any noticeable performance gain
> from doing so. The only headers that today are open-ended are really
> User-Agent and Set-Cookie/Cookie pairs. In all our data where we don't
> include these headers we see no gain from using header compression
> whatsoever as long as you are conservative in how many headers you choose
> to include.
>

Typical headers are ~450 bytes on the request, and ~300 on the response.
 Cookies make it vary wildly.  The compressor cuts the initial header size
in half, subsequent headers by about 90%.

I'm not sure what data you are referring to - but I was able to see
performance differences here.  When you simulate typical web pages today
(~8 domains, ~80 resources, etc) with a low-end network (385Kbps down,
128Kbps up), you can do the math yourself, but saving 200bytes per request
* 80 requests over these types of links measures in the *seconds*....

Remember that most users are using asymmetric networks, with very small
uplink capability.

>
> In other words, based on the data we have, focusing on optimizing UA and
> cookie headers may offer tangible performance gains.
>

Of course they will.

>
> Unless we can find convincing data I would argue that we should start the
> header discussion by focusing on the headers that we know have problems and
> then go from there.
>

Compression is a problem which can always be solved differently and always
be a little "better".  But at the end of the day, this is not the key part
of the protocol - users aren't going to notice as long as you get the byte
count down by ~75%.

Don't get me wrong, I really don't care what we change it to - but I hope
we talk about something else, because the compressor easily solved if we
just live with good enough.

Some recommendations I'd put out to anyone recommending new compressors for
headers:
    * Target at least 75% or more compression over today's HTTP for typical
web pages.
    * Be straightforward to implement
    * Be able to compress repeated cookies well
    * Be tolerant to long-lived sessions; if your compressor builds up
state with every new header, it must have a way to rotate out the old data
(e.g. a rolling window)
    * Try to minimize RAM and CPU
    * Do not use tricks that give advantages to incumbent browsers (e.g.
don't hard-code compression for pronouns like "WebKit", "Windows", "Chrome"
or "Firefox")
    * Don't try to "clean up" HTTP/1.1's headers; this just creates
compatibility issues, so unless done for a very specific purpose wastes time

I hope those recommendations are useful.

Mike

> One of the design choices I find a little troubling in SPDY is the use of
> header compression over a compact representation.
>
> Compression algorithms worry me because the only way I can implement them
> is to call someone's library. I can write my own code to implement RFC822
> header parsing. In fact I have done so a dozen times at least.
>
> ASN.1 and XML schemas are undoubtedly how to do a protocol 'right' in the
> academic sense of get some good abstractions in there. But they both come
> with hideous baggage. ASN.1 started too large and has been tweaked
> endlessly until it became nonsense. XML encoding is bloated and XML Schema
> drags in the whole SGML legacy that the WG was meant to chuck in the
> dustbin. I dare you to use a model group in your next XML spec.
>
>
> But JSON proves that it does not have to be that way. Over the past couple
> of months I have put together what amounts to a schema compiler for JSON. I
> am currently working on putting the compiler and the tools used to build it
> out on Github under an open source license.
>
> Unlike your regular schema compiler, this one makes it easy to retarget
> the code generation at a different language. Writing a JSON back end for C#
> took me about a week (and I was developing the tool at the same time). I
> could modify that code to generate C in a day. Want python, perl, Java, no
> problem.
>
> The JSON protocol synthesizer is built using the Goedel meta-synthesizer
> (which is of course written in itself). Comodo owns the protocol synth, I
> own the meta-synth. They should both be up on github under an MIT license
> in a few weeks.
>
>
> It seems to me that it would be useful to have a compiler for RFC822 style
> headers. The basic strategy being similar to a 'jack up' house renovation.
> First we jack up the level of abstraction in the HTTP spec by defining the
> headers in terms of a schema that is designed for that specific purpose.
> Then we build in the new ground floor (aka encoding) underneath it.
>
>
> --

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 3 Aug 2012 00:58:16 +0200

> some compression dictionary.

A TCP connection may transport requests from many different user agents
in turn. But where the size problem matters, this is rarely the case.
This is why we suggested an alternative using per-connection headers
in the network-friendly draft.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 03 Aug 2012 05:30:30 +0000

In message <[email-masked]>
, Mike Belshe writes:
>--f46d0407178542016504c64fa910

>So far, this has not been a problem.

... becuase sites which needed such inspection and manipulation deliberately
stayed away from SPDY because of the TLS+Compression requirement.

Absense of evidence is not evidence of absense.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 3 Aug 2012 10:45:04 -0700

There is a measured benefit in some situations for header compression. It
does take CPU and memory.
The question isn't if it adds CPU and memory, but rather if the tradeoff is
worthwhile?

This is really something that is best informed by measurements of
transactions in the real world.
Use cases that motivated this design, and thus are potentially interesting:
  mobile devices
  satellite links
  high-loss links (e.g. many links from Africa, where cwnd is usually
ending around 3 packets worth...)
-=R

On Thu, Aug 2, 2012 at 10:30 PM, Poul-Henning Kamp <[email-masked]>wrote:

> In message <
> [email-masked]>
> , Mike Belshe writes:

---

From: Patrick McManus <[email-masked]>
Date: Wed, 08 Aug 2012 16:24:09 -0400

On Thu, 2012-08-02 at 20:53 +0000, Henrik Frystyk Nielsen wrote:
> I also have trouble with the use of compression over the headers [..
> mcmanus deletes argument #1 just for clarity of reply..]
> 
> 2) Further, it is unclear whether there is any noticeable performance
> gain from doing so. The only headers that today are open-ended are
> really User-Agent and Set-Cookie/Cookie pairs. In all our data where
> we don't include these headers we see no gain from using header
> compression whatsoever as long as you are conservative in how many
> headers you choose to include.

Hi Henrik, Thanks for the report - but this really really doesn't mesh
with my experience. I kept it in my inbox until I had the time to do
some research to back up the value of header compression when I piped up
(I'm referring to compression generically - not a particular scheme for
doing so..).

In one sense, we must be talking about different contexts. It's obvious
that if you have very small objects that the headers are going to
dominate transfer time. They do this not so much from serialization
delay but because of interaction with CWND - you can get more headers
upstream faster in the same CWND and form a deeper pipeline (whether
that is serial http/1 style or spdy-muxxed style isn't impt). Once you
get enough requests upstream that the server can fill the BDP it doesn't
much matter, but until you reach that point it matters a lot and it
turns into significant time.

This is especially problematic when responses are smaller than requests.
The poster child for this is a 304 response to a request with a big
cookie. 

But its also a problem where clients have less aggressive TCP parameters
than the server they are talking with - which is pretty typical. A
devops environment might be tweaked up with IW=10 and set to not revert
to slowstart after small periods of idle time, but a desktop is almost
certainly going to be running vanilla IW=3 for many years. That
exacerbates the problem of the client not being able to send requests
fast enough for the server to be able to have something to respond to at
their full sending rate.

I put together some tests to illustrate it. I formed a HTTP/1 style
pipeline of 86 requests that all return 304. (click reload on one page).
I tested both without a cookie and with a modest ~1000 byte cookie. You
can look at the packet captures if you want to see the transactions, but
they are terribly vanilla - requests are 477 bytes (plus possible
cookie), and responses are 307. Both client and server are running IW=3,
and my timings exclude the handshake, and I measured over 4 latencies.
Timings are in ms.

Test    50ms    100ms    200ms    300ms
1         52    102       202      302 
2         52    102       202      302
3        358    808      1401     2108
4        256    506      1006     1542

Test 1 is with the cookie and zlib compression
Test 2 is without the cookie but with zlib
Test 3 is with the cookie and no zlib
Test 3 is without cookie and no zlib

It's pretty clear that TCP transfer (as it almost always is for http) is
dominated by round trip penalties growing CWND. No surprise there - and
no surprise that reducing the amount of data to be sent means less
penalties absorbed. The magnitude of it is a little surprising though -
if you sit behind a 300ms link this page takes 2.1 seconds to reload
without compression vs ~0.3 seconds with it.

The packet captures for these are at
https://www.ducksong.com/misc/pcaps-for-zlib/

Strategies that lossily mess with which headers to transfer are going to
create backwards compatibility problems and probably not get support.
You should see the mess we caused just trying to drop Accept-Charset
from our request headers - indeed nobody was doing charset negotiation
with it but some very big names were using it for fingerprinting and
stuff started to break :(. Something like User-Agent is a couple orders
of magnitude more intimidating.

So +1 for some form of lossless encoding of the headers.

-Patrick

---

From: Henrik Frystyk Nielsen <[email-masked]>
Date: Wed, 15 Aug 2012 11:36:50 +0000

Patrick, 

I apologize for not seeing this earlier -- this is great data!

I completely agree that if you have very small entity bodies then maturally the header size will matter getting more requests into the pipe faster. However, in the tests that we have done with actual data the size of the entity bodies was large enough that the impact was minimal. This is especially the case if you also do bundling/minification as it naturally leads you to large entities.

In other words, it is hard to beat the performance characteristic of the request you *don't* have to make :)

I completely agree that whatever transformation we do has to be lossless -- it's primarily the *mechanism* I am interested in. In particular, I am interested in its ability for easy inspection, which is why I am somewhat weary of stream-based compressing algorithms. I would much rather see some form of tokenization.

That said, we are gathering a bunch more data as we speak on the various ways of sending less headers -- I hope to be able to send out more data soon.

Thanks,

Henrik

-----Original Message-----
From: Patrick McManus [mailto:[email-masked]] 
Sent: Wednesday, August 08, 2012 13:24
To: Henrik Frystyk Nielsen
Cc: [email-masked] Group
Subject: In Defense of Header Compresson

> headers you choose to include.

Hi Henrik, Thanks for the report - but this really really doesn't mesh with my experience. I kept it in my inbox until I had the time to do some research to back up the value of header compression when I piped up (I'm referring to compression generically - not a particular scheme for doing so..).

In one sense, we must be talking about different contexts. It's obvious that if you have very small objects that the headers are going to dominate transfer time. They do this not so much from serialization delay but because of interaction with CWND - you can get more headers upstream faster in the same CWND and form a deeper pipeline (whether that is serial http/1 style or spdy-muxxed style isn't impt). Once you get enough requests upstream that the server can fill the BDP it doesn't much matter, but until you reach that point it matters a lot and it turns into significant time.

This is especially problematic when responses are smaller than requests.
The poster child for this is a 304 response to a request with a big cookie. 

But its also a problem where clients have less aggressive TCP parameters than the server they are talking with - which is pretty typical. A devops environment might be tweaked up with IW=10 and set to not revert to slowstart after small periods of idle time, but a desktop is almost certainly going to be running vanilla IW=3 for many years. That exacerbates the problem of the client not being able to send requests fast enough for the server to be able to have something to respond to at their full sending rate.

I put together some tests to illustrate it. I formed a HTTP/1 style pipeline of 86 requests that all return 304. (click reload on one page).
I tested both without a cookie and with a modest ~1000 byte cookie. You can look at the packet captures if you want to see the transactions, but they are terribly vanilla - requests are 477 bytes (plus possible cookie), and responses are 307. Both client and server are running IW=3, and my timings exclude the handshake, and I measured over 4 latencies.
Timings are in ms.

Test 1 is with the cookie and zlib compression Test 2 is without the cookie but with zlib Test 3 is with the cookie and no zlib Test 3 is without cookie and no zlib

It's pretty clear that TCP transfer (as it almost always is for http) is dominated by round trip penalties growing CWND. No surprise there - and no surprise that reducing the amount of data to be sent means less penalties absorbed. The magnitude of it is a little surprising though - if you sit behind a 300ms link this page takes 2.1 seconds to reload without compression vs ~0.3 seconds with it.

Strategies that lossily mess with which headers to transfer are going to create backwards compatibility problems and probably not get support.
You should see the mess we caused just trying to drop Accept-Charset from our request headers - indeed nobody was doing charset negotiation with it but some very big names were using it for fingerprinting and stuff started to break :(. Something like User-Agent is a couple orders of magnitude more intimidating.

-Patrick

---

From: Ilya Grigorik <[email-masked]>
Date: Wed, 15 Aug 2012 22:40:00 -0700

>
> On Wed, Aug 15, 2012 at 4:36 AM, Henrik Frystyk Nielsen <
>> I completely agree that if you have very small entity bodies then
>> maturally the header size will matter getting more requests into the pipe
>> faster. However, in the tests that we have done with actual data the size
>> of the entity bodies was large enough that the impact was minimal. This is
>> especially the case if you also do bundling/minification as it naturally
>> leads you to large entities.
>
>
>
It's worth keeping in mind that we're seeing more and more web
"applications", as opposed to pages. Frameworks like backbone, angular,
etc, all frequently make very small (usually JSON encoded) requests to
indicates record updates, or to pull specific objects on demand, from an
HTTP endpoint.. All of that to say: many of these requests are just a few
hundred bytes. Your typical CRUD operations are all great examples: a lotta
headers for a tiny payload and a 204 response. The overhead there is huge,
which is why we're seeing people starting to invent their own protocols.
e.g. SwaggerRocket:
http://blog.wordnik.com/introducing-swaggersocket-a-rest-over-websocket-protocol

They shouldn't have to do this...

Ilya

---

From: Patrick McManus <[email-masked]>
Date: Thu, 16 Aug 2012 09:45:47 -0400

This thread has primarily talked about downstream compression and the
fact that its utility is driven by the header:body ratio.

Upstream compression for http/2 is pretty interesting too.

http/2 is presumably muxxed with priority semantics. The server can only
do priority scheduling across the transactions it is aware of - so we've
moved past the simpler http/1+pipeline need of being able to send enough
requests to keep the downstream bandwidth utilized. Having extra
transactions to choose from lets the priority of a late ordered
transaction take precedence if appropriate.

upstream compression lets the client send more requests in its typically
small cwnd.

On Wed, 2012-08-15 at 22:40 -0700, Ilya Grigorik wrote:
>         On Wed, Aug 15, 2012 at 4:36 AM, Henrik Frystyk Nielsen
>                 I completely agree that if you have very small entity
>                 bodies then maturally the header size will matter
>                 getting more requests into the pipe faster. However,
>                 in the tests that we have done with actual data the
>                 size of the entity bodies was large enough that the
>                 impact was minimal. This is especially the case if you
>                 also do bundling/minification as it naturally leads
>                 you to large entities.
>         
>         
>  
> "applications", as opposed to pages. Frameworks like backbone,
> angular, etc, all frequently make very small (usually JSON encoded)
> requests to indicates record updates, or to pull specific objects on
> demand, from an HTTP endpoint.. All of that to say: many of these
> requests are just a few hundred bytes. Your typical CRUD operations
> are all great examples: a lotta headers for a tiny payload and a 204
> response. The overhead there is huge, which is why we're seeing people
> starting to invent their own protocols. e.g. SwaggerRocket:

---

From: Patrick McManus <[email-masked]>
Date: Thu, 16 Aug 2012 09:49:44 -0400

On Wed, 2012-08-15 at 11:36 +0000, Henrik Frystyk Nielsen wrote:

>  it's primarily the *mechanism* I am interested in. In particular, I am interested in its ability for easy inspection

I'm also totally open to token/table mechanisms that give good results!

---

From: "Robert Brewer" <[email-masked]>
Date: Thu, 16 Aug 2012 07:36:53 -0700

Ilya Grigorik wrote:
> > On Wed, Aug 15, 2012 at 4:36 AM, Henrik Frystyk Nielsen <[email-masked]> wrote:
> > I completely agree that if you have very small entity bodies
> > then maturally the header size will matter getting more
> > requests into the pipe faster. However, in the tests that we
> > have done with actual data the size of the entity bodies was
> > large enough that the impact was minimal. This is especially
> > the case if you also do bundling/minification as it naturally
> angular, etc, all frequently make very small (usually JSON
> encoded) requests to indicates record updates, or to pull
> specific objects on demand, from an HTTP endpoint.. All of that
> to say: many of these requests are just a few hundred bytes.
> Your typical CRUD operations are all great examples: a lotta
> headers for a tiny payload and a 204 response. The overhead
> there is huge, which is why we're seeing people starting to
> invent their own protocols. e.g. SwaggerRocket:

They shouldn't have to invent their own protocols because there are already plenty of protocols optimized for small-grained messages. That doesn't mean HTTP, which is one of the few protocols which is optimized for large-grained messages (in order to trade efficiency for scalability), has to change to become one of them.

Robert Brewer
[email-masked]

---

From: David Morris <[email-masked]>
Date: Thu, 16 Aug 2012 08:18:34 -0700 (PDT)

On Thu, 16 Aug 2012, Patrick McManus wrote:

> >  it's primarily the *mechanism* I am interested in. In particular, I
> > am interested in its ability for easy inspection
> 

I have probably spent years of my professional life analyzing data on the
wire so I sympathise with the desire to inspect data easily, but I don't
accept this as a constraint on the solution, only a tiebreaker. There
are enough ways for tools to solve the problem and infact I see a lot of
value in having motivation for the creation of more tools to support
data inspection.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 16 Aug 2012 11:24:53 -0400

What really matters for performance is the number of packets rather
than packet size. If you are worried about the packet size then you
should not expect a general purpose transport to meet your needs. UDP
is going to be better.

Similarly, if you are using JSON then the JSON bloat is going to be a
lot more overhead than HTTP bloat.

I like tokenization as a strategy as the same approach can be used for
both the JSON and the HTTP space saving. It also means that embedded
devices can even use the tokenized form as a replacement for header
and JSON syntax and only use one parser.

I don't see much advantage to compression for small message sizes
unless you can avoid transporting the compression dictionary which is
exactly what tokenization does, only with a slight reduction in
maximum efficiency and a huge reduction in complexity.

---

From: David Morris <[email-masked]>
Date: Thu, 16 Aug 2012 12:25:48 -0700 (PDT)

> and JSON syntax and only use one parser.

Tokenization, as I under stand the usage here, represents compression
using a predefined dictionary of codes and values. No different than
using a compression technique such as LZW with a predefined standard
dictionary. The key for either approach is the ability to verify
compatible dictionary versions, obtain the correct version if needed,
and for the dictionary to learn on the fly.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 16 Aug 2012 13:13:28 -0700

JSON is far more rich than we need.
name-value is really sufficient for anything the protocol should
contemplate.
I'm all for tokenization, btw, so long as the mapping for tokenization is
static for long periods of time (months/years)
-=R

On Thu, Aug 16, 2012 at 8:24 AM, Phillip Hallam-Baker <[email-masked]>wrote:

> > They shouldn't have to invent their own protocols because there are
> already plenty of protocols optimized for small-grained messages. That
> doesn't mean HTTP, which is one of the few protocols which is optimized for
> large-grained messages (in order to trade efficiency for scalability), has
> to change to become one of them.
> >
> >
> > Robert Brewer
> > [email-masked]
>
>
>
> --

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 16 Aug 2012 16:23:54 -0400

There is quite a big difference in coding.

To implement tokenization you replace the RFC822
serializer/deserializer with one that implements a different syntax.

To implement a compression technique you stack the
compressor/decompressor in front of the RFC822
serializer/deserializer.

That makes quite a difference on small devices.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 16 Aug 2012 16:26:00 -0400

Agreed, but that is not what I was proposing.

What I was saying is that if people are going to throw up JSON Web
Services as the reason for needing to compress HTTP headers they are
overlooking the main source of bloat in their code.

---

From: Henrik Frystyk Nielsen <[email-masked]>
Date: Thu, 16 Aug 2012 20:27:25 +0000

This is exactly what I am getting at :) 

I agree that upload numbers are important too.

Henrik

---

From: Henrik Frystyk Nielsen <[email-masked]>
Date: Thu, 16 Aug 2012 20:28:44 +0000

It is not just a question of tooling, it is a matter of processing on devices that either sit as part of the message path (proxies) or have limited ability either as user agent or ultimate destination.

Henrik

-----Original Message-----
From: David Morris [mailto:[email-masked]] 
Sent: Thursday, August 16, 2012 8:19 AM
To: [email-masked] Group
Subject: RE: In Defense of Header Compresson

> I'm also totally open to token/table mechanisms that give good results!

I have probably spent years of my professional life analyzing data on the wire so I sympathise with the desire to inspect data easily, but I don't accept this as a constraint on the solution, only a tiebreaker. There are enough ways for tools to solve the problem and infact I see a lot of value in having motivation for the creation of more tools to support data inspection.

---

From: Henrik Frystyk Nielsen <[email-masked]>
Date: Thu, 16 Aug 2012 20:30:11 +0000

I agree that JSON is too rich.

I completely agree that the mapping has to be static.

Henrik

From: Roberto Peon [mailto:[email-masked]]
Sent: Thursday, August 16, 2012 1:13 PM
To: Phillip Hallam-Baker
Cc: Robert Brewer; Ilya Grigorik; Henrik Frystyk Nielsen; Patrick McManus; [email-masked]
Subject: Re: In Defense of Header Compresson

JSON is far more rich than we need.
name-value is really sufficient for anything the protocol should contemplate.
I'm all for tokenization, btw, so long as the mapping for tokenization is static for long periods of time (months/years)
-=R
On Thu, Aug 16, 2012 at 8:24 AM, Phillip Hallam-Baker <[email-masked]<mailto:[email-masked]>> wrote:
What really matters for performance is the number of packets rather
than packet size. If you are worried about the packet size then you
should not expect a general purpose transport to meet your needs. UDP
is going to be better.

On Thu, Aug 16, 2012 at 10:36 AM, Robert Brewer <[email-masked]<mailto:[email-masked]>> wrote:
> Ilya Grigorik wrote:
>> > On Wed, Aug 15, 2012 at 4:36 AM, Henrik Frystyk Nielsen <[email-masked]<mailto:[email-masked]>> wrote:
> [email-masked]<mailto:[email-masked]>

---

From: Ilya Grigorik <[email-masked]>
Date: Thu, 16 Aug 2012 15:59:27 -0700

> to change to become one of them.

I believe that would be a shortsighted call to make. Besides, show me
another protocol available in the browser, which *is* the distribution
platform for all these apps? It's HTTP, plain and simple.. or build on your
own, on top of WebSockets.

Median size per GET: 2.36KB -
https://developers.google.com/speed/articles/web-metrics

Add ~800 bytes of request/response headers on top, and we're spending
25-30% on headers (that is, if you're lucky and can avoid adding cookies to
that as well). Also, there is no sign that we're going to stop putting
*more* stuff into the headers. User-agent and Cookies are the best known
offenders, but what about even things like upcoming CSP suport?

X-Content-Security-Policy: allow 'self'; img-src *; \
                           object-src media1.com media2.com *.cdn.com; \
                           script-src trustedscripts.example.com

And that's just the beginning.

In any case, my meta-point is simple: I believe Patrick's numbers may
well underestimate the prevalence of short requests/responses. We should
factor in the long-lived apps that stay open in the browser and rely on
small periodic pings (aka, any "ajax" app like gmail, docs, and so forth,
not to mention the billions of beacons for analytics and the like).

ig