Subject: Re: Header Size? Was: Our Schedule

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 01 Jun 2014 10:53:13 +0000

In message <[email-masked]>, K
[email-masked] writes:

>Could you achieve the same goal within the current spec by requiring the 
>route headers to come first and uncompressed in the first HEADERS frame of
>a stream?

Well, there's probably not much gained if they are still encrypted,
but yes, any movement towards the scheme I suggested will improve
performance.

>Are these headers you mentioned the only three proxies ever need to look at
> when forwarding requests?
>>        Host:
>>        URL sans query part  (or possibly:  URL up to first '/')
>>        (X-)Forwarded-For:

Well, proxied do the most weird things, but for the typical
load-balancer case, which is where the HTTP bandwidth is most
concentrated, those three are what's being looked at.

>What about forwarding responses back to the client? I imagine you would want
>to do a similar thing. Same headers?

In general load-balancers don't look at the response.  If they do it's
typically a very simple check of the status code, so that retries
can be done on failures (>= 400 or something)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 01 Jun 2014 10:59:05 +0000

In message <[email-masked]>
, Roberto Peon writes:

>Poul's proposal to put routing info at the front pre-supposes that all
>proxies use the same information for routing.

No it doesn't "pre-suppose".

It *optimizes* for the most common and highest-performance-pressure
use case, so that we have some chance of taking a 40gb/s or 1tb/s
firehose and split it out to a multitude of more manageable streams,
without holding state.

If a proxy wants to examine more fields than what is in the
route-header, it's free to do so, but will incur whatever complexity
and overhead any compression and/or encryption carries.

But I will grant you that putting the method (GET/PUT etc) in there
would make sense.

---

From: Jason Greene <[email-masked]>
Date: Sun, 1 Jun 2014 13:28:45 -0500

On May 31, 2014, at 2:52 PM, Poul-Henning Kamp <[email-masked]> wrote:

> In message <[email-masked]>, "Jason T. Greene"
> writes:
> 
>> IMO the only way to allow for a proxy to directly hand off requests as
>> is would be per-frame compression, where the URL is easily located and
>> matched. 
> 
> And this is why I think state-full compression of headers necessary
> for routing HTTP is a terminally bad idea for HTTP/2.0, since it
> causes state to materialize all over the place, state which increases
> cost, decrease reliability and which is vulnerable to DoS'ing at
> all the most sensitive choke-points.
> 
> My original suggestion was to have two parts to HTTP/2 transport
> messages:
> 
> 	[route headers]
> 	[payload]
> 
> Where the route headers would consist of:
> 	length of route headers
> 	length of payload
> 	Host:
> (Trivial details to cater for multiplexing and "Chunked" style
> incremental delivery left out for clarity).
> 
> Route headers are uncompressed, in order to make it possible to
> route 1Tbit/sec HTTP traffic without holding an expensive state.

Indeed. I do like the idea of separating routing from other headers, in particular because it becomes trivial for a proxy implementation to inspect and append additional routing information. 

> 
> The payload consists of metadata and objects
> 
> Metadata is the rest of the HTTP headers, which are not needed for
> transporting HTTP, but only play a role in semantic interpretation:
> Content-Type, Content-Encoding etc. etc.
> 
> Metadata and object can be compressed or encrypted how ever you like.

It still has to be limited to per-frame compression, because shared compression state means that a proxy must process and convert all metadata. 

As a server and proxy implementer I would prefer modest gains in packet size that didn’t sacrifice throughput already achievable in http 1.1, and IMO the static tables do exactly that. I can certainly appreciate that those implementing clients don’t agree with this perspective, and would like more.
 
> 
> The information leak of the Host: and first part of the URL are
> insignificant relative to privacy:  If a site wants to
> offer increased privacy, both can trivially be emptied of
> information in the site-design.
> 
> Having the route header unencrypted also means that legally mandated
> inspecting proxies can *avoid* inspecting privileged traffic for
> instance between prison inmates, their lawyers and politicians,
> by implementing a white-list.
> 
> 
> 
> -- 

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 01 Jun 2014 19:33:06 +0000

In message <[email-masked]>, Jason Greene writ
es:

>It still has to be limited to per-frame compression, because shared 
>compression state means that a proxy must process and convert all 
>metadata. 

It could be per-transaction compression, but that amounts to pretty
much the same thing if we do our job well.

The main reason to have compression is to squeeze cookies.

I still think it is a much better strategy do away with cookies and
all their problems (privacy, legal etc.) and skip compression with
all its problems (DoS, state etc.) and get smaller headers than we
would have with cookies, compressions and all their problems.

Sanitizing User-Agent: would be the next big gain (Again: privacy
problems and all that.)

Doing a sensible static enumeration of all the RFC-headers could shave
some bytes too.

>As a server and proxy implementer I would prefer modest gains in packet 
>size that didn't sacrifice throughput already achievable in http 1.1, 
>and IMO the static tables do exactly that. I can certainly appreciate 
>that those implementing clients don't agree with this perspective, and 
>would like more.

As a proxy implementer, I think a HTTP/2.0 that cannot be processed,
at the very least as a transparent proxy splitting the stream on
Host: header, at todays COTS line-rates on todays COTS servers would
be an utter embarrasment.

8-cores, 40gbit/s -> 4x10gbit/s anyone ?

What *is* the highest rate anybody has processed HTTP/2.0 according
to the current draft anyway ?

I don't recall seing anybody brag about that yet ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 01 Jun 2014 20:00:16 +0000

In the discussion about compression, and for that matter
encryption, we need to talk a bit about greenhouse gasses.

Any one microsecond, any one kilobyte of extra per-hop cost per
HTTP request, is going to scale into remarkable amounts of greenhouse
gasses by the time it is scaled up across the WWW.

Ideally, we should be able to show that processing the same
workload with HTTP/2.0 takes no more energy than with HTTP/1.1,
and preferably less.

A small power increment for significant performance gains can
probably be "sold" with some hand-waving.

Having HTTP/2.0 use significantly more energy than HTTP/1.1
would make the protocol a non-starter.  Just imagine the
gleeful CS/EE student papers and resulting headlines.

Poul-Henning

PS:  Speaking from a lot of experience, the most manageable way to
measure power consumption realtive to workload is to use really
small computers, RasberryPI, BeagleBone, Soekris etc. where the
measurements can be done on the low voltage DC supply.

If anybody wants to persue this, I'll be happy to help out.

Drop me email.

---

From: Jason Greene <[email-masked]>
Date: Sun, 1 Jun 2014 15:13:33 -0500

> Doing a sensible static enumeration of all the RFC-headers could shave

Right the static substitution elements of HPACK would likely improve over http 1.1 decoding speed.

> I don't recall seing anybody brag about that yet ?

It might be possible to keep the stateful delta operations and achieve http 1.1 throughput levels if the receiver, and not just the sender could negotiate the dynamic table size to 0.

---

From: Roberto Peon <[email-masked]>
Date: Sun, 1 Jun 2014 13:13:06 -0700

You're missing out on all of the other second order effects.

As examples, things that increase latency, increase the time
spectrum/channels are in use, requiring the radio to be on for longer,
requiring the CPU to be active for longer, requiring the display to be
active for longer, etc.
CPU is often far less costly than keeping the radio or screen on. Powering
a radio or screen is often accomplished via a battery, which has a less
than 100% energy efficiency, etc. etc.

-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 01 Jun 2014 20:27:28 +0000

In message <CAP+FsNehJkfSsnJ9=[email-masked]>
, Roberto Peon writes:

>[...]
>CPU is often far less costly than keeping the radio or screen on.

You're welcome to count any and all secondary effects of HTTP/2.0
to its advantage, provided you can show they exist, or even better
if you can actually measure them.

It would be pretty trivial to set up two Soekris boxes with a
WLAN connection and measure the total power consumption running
the same HTTP workload through HTTP/1.1 or HTTP/2.0.

Poul-Henning

---

From: Roberto Peon <[email-masked]>
Date: Sun, 1 Jun 2014 13:31:40 -0700

So, lets see.. last time we looked, we were seeing something like a 15%
reduction in latency as measured by PLT to many of the most frequented
sites on the 'net.
The time to page usefulness, which is far more difficult to measure, was
improved more, meaning that the page could be navigated to/away from faster.

The trivial tests are most often the ones that get you the most meaningless
numbers.
-=R

On Sun, Jun 1, 2014 at 1:27 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <CAP+FsNehJkfSsnJ9=
> [email-masked]>
> , Roberto Peon writes:
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 01 Jun 2014 20:42:04 +0000

In message <[email-masked]>
, Roberto Peon writes:

>So, lets see.. last time we looked, [...]

Hand-waving about potential secondary effects won't be enough.

Lets try to get some actual measurements.

---

From: Roberto Peon <[email-masked]>
Date: Sun, 1 Jun 2014 13:45:00 -0700

Sure-- I'd love to see you measure the amount of energy spent by the radio,
screen, etc. on mobile devices, and the cost in terms of wasted bandwidth
in keeping the channel open!
-=R
p.s. this line of argumentation is silly...

On Sun, Jun 1, 2014 at 1:42 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <
> [email-masked]>
> , Roberto Peon writes:
>

---

From: Jason Greene <[email-masked]>
Date: Sun, 1 Jun 2014 16:01:49 -0500

On Jun 1, 2014, at 3:56 PM, Jason T. Greene <[email-masked]> wrote:

> 
> 
> On Jun 1, 2014, at 3:48 PM, Roberto Peon <[email-masked]> wrote:
> 
>> Sure-- I'd love to see you measure the amount of energy spent by the radio, screen, etc. on mobile devices, and the cost in terms of wasted bandwidth in keeping the channel open!
>> -=R
> I think it's totally reasonable to require that http/2 be comparable to http/1.1 proxies, balancers and servers.

Sorry, premature send. What I was saying that I think its totally reasonable to expect that http/2 should offer similar performance as http/1.1 for proxies, balancers, and servers (not just clients). Otherwise, if it’s going to cost x% more for providers to switch to http/2, they just might not do it.

> 
> 
>> 
>> 
>> On Sun, Jun 1, 2014 at 1:42 PM, Poul-Henning Kamp <[email-masked]> wrote:

---

From: Roberto Peon <[email-masked]>
Date: Sun, 1 Jun 2014 14:11:38 -0700

That isn't what phk@ is arguing, however. He is arguing that HTTP/1.1 is
more 'green'.
-=R

On Sun, Jun 1, 2014 at 2:01 PM, Jason Greene <[email-masked]>
wrote:

> >> Sure-- I'd love to see you measure the amount of energy spent by the
> radio, screen, etc. on mobile devices, and the cost in terms of wasted
> bandwidth in keeping the channel open!
> >> -=R
> Sorry, premature send. What I was saying that I think its totally
> reasonable to expect that http/2 should offer similar performance as
> http/1.1 for proxies, balancers, and servers (not just clients). Otherwise,
> if it’s going to cost x% more for providers to switch to http/2, they just
> might not do it.
>
>
> >
> >
> >>
> >>

---

From: Mike Belshe <[email-masked]>
Date: Sun, 1 Jun 2014 14:32:48 -0700

I would be interested in seeing a useful study here.  However, the best
study would be one done in production environments rather than on simulated
devices with simulated models.  It would just be really hard to get a
realistic model.

Regardless, however, using less power wasn't an of objective in building
HTTP.  To revisit the requirements at this late stage is not a good idea.
 We should stick to the plan.

Mike

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 01 Jun 2014 21:42:03 +0000

In message <[email-masked]>

>That isn't what phk@ is arguing, however. He is arguing that HTTP/1.1 is

Roberto, go back and read what I wrote, will you ?

I have absolutely no idea which protocol is more "green" at this
point in time.

HTTP/1.1 is not without its touble in this respect.  For instance
the combination of Chunked and PDP-11 friendly POSIX APIs are very
expensive, so a sensible framing format in HTTP/2.0 can save a LOT
of system-calls over HTTP/1.1.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 01 Jun 2014 21:53:46 +0000

In message <[email-masked]>

> We should stick to the plan.

	"Regardless, avoiding icebergs wasn't an objective in
	building this ship.  To revisit the requirements at this
	late stage is not a good idea.  We should hold the course."

		-- Captain Smith, RMS Titanic.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 2 Jun 2014 00:12:06 +0200

> devices with simulated models.  It would just be really hard to get a

Mike,

It would be interesting to know if any large scale adopters of SPDY have
seen any significant changes in power usage in production environments - at
least on the server side.

There are significant increases in CPU demands with SPDY, but I would also
guess a lot of benefits as well.

While I tend to agree that this is not something we will be able to very
accurately measure nor to sweat any small changes, it would probably be
worthwhile for somebody with the right equipment to just verify we are not
making any huge increases in power requirements.

cheers

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: "Martin Nilsson" <[email-masked]>
Date: Mon, 02 Jun 2014 00:36:23 +0200

> probably be "sold" with some hand-waving.

With the simple compression we are looking at you are likely to see  
performance gains AND power savings. My somewhat dated data says that  
transmitting one bit on wifi takes the same amount of energy as 500 ARM  
instructions. Gzip is surprisingly well positioned in compression ration  
vs. CPU cycles. We are doing way less work here, and saving quite a lot of  
data and packages. As Roberto pointed out, getting the data to the device  
fast so the radio can move to low energy mode quickly is important, and  
much more important than compression on mobile.

We did energy benchmarking on real devices with something close to HTTP/2  
last year, and the major conclusion is that it comes down use case and  
implementation. If you correctly push dependent data you'll dramatically  
reduce radio high energy time (but as always, 50% of the energy is spent  
on getting tracking pixels from ad servers). If you on the other hand send  
ping messages to keep TCP and radio hot, you'll drain the battery in no  
time. If you benchmark load-as-you-scroll pages you'll get horrible  
results regardless of protocol, and any difference is a rounding error.

Having real numbers on a good implementation would of course be very  
interesting.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Mike Belshe <[email-masked]>
Date: Sun, 1 Jun 2014 15:40:21 -0700

> guess a lot of benefits as well.

I don't know if there are net power increases with HTTP/2 or not.

Some thoughts:

a) In the datacenter, we can generally stipulate that more CPU correlates
to more power draw.  However, in these environments, the web servers are
such a minuscule portion of the aggregate power draw that this is unlikely
to be a significant difference.

b) On handheld devices, we know that the largest draw of power is generally
the radio.  The reduced number of connections and packets in HTTP/2 will be
a significant win for the radio such that overall power consumption may be
significantly less than that with plain HTTP, in spite of using more CPU.
 Implementation optimizations (number of streams, compression parameters,
timing, etc) likely have more effect on power usage than HTTP1 vs HTTP2.
 Either way, I don't know how you'd isolate implementation-specific power
draw vs "protocol" power draw.

> making any huge increases in power requirements.

A measurement would be great.   But I don't think we'll have a good
analysis of this before HTTP2 in this form (or any other form) completes.

Mike

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 2 Jun 2014 11:19:00 +0200

On 1 June 2014 02:21, Roberto Peon <[email-masked]> wrote:
> proxies use the same information for routing.This is often not the case.
>

It is true that some proxies will want to look at arbitrary request meta
data - including cookies- to make their routing decisions.    Such proxies
are committed to full decoding.

However, there is also a large class of proxy that will want to look at
only the basic information.

So perhaps it is time that we restricted the transport meta data to only
carry transport meta data, and to move the extra meta-data to a secondary
header set?

Ie if we extended Poul's proposal so that all headers defined by the http2
specification for the use of transport, and only those headers, are in an
initial header set. A subsequent header set can then contain all of the
additional headers that an application/client/server may add.

Proxies that want to route only on the core http defined fields, can then
just decode the first header set.  Proxies that wish to route based on
additional headers (including cookies), would have to decode both sets.

I've always found it bizarre that HTTP design often results in allowing
applications to set headers like Transfer-Encoding!

This has been discussed before, I'll point out.

My limited experience of the IETF process indicates that WG such as this
often do have to re discuss design decisions made long ago.  This process
tends to iterate until either the spec and associated document because
clear enough to explain the main design decisions,  OR more likely that the
re-discussions appear to be mostly provoked from the same set of chronic
malcontents who wont accept they have not made their case.

For my own part, I have found the recent discussions very worthwhile and I
very much appreciate those who have taken the time to (re)explain why many
of the decisions have been made.   While I am critical of parts of the
spec, I am impressed at the depth of the reasoned explanations for almost
all of them.   So thanks for staying engaged and spending the time to
explain.

cheers

---

From: Roland Zink <[email-masked]>
Date: Mon, 02 Jun 2014 12:04:43 +0200

Maybe this should be the default and there should be a "energy 
considerations" section in addition to the "security considerations". 
Interesting in this case would be the comparison of HTTP2 in the clear 
compared to HTTP2 over TLS.

Regards,
Roland

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Mon, 2 Jun 2014 14:02:41 +0000

> From: Eliot Lear [mailto:[email-masked]] 
 > How is it you could have 62k worth of headers that use only 1,669 bytes?

By being a donkey and summing the request not the response. Sorry folks. Try again ...

Total Size of Response Headers (KB)	Hits	Total Bandwidth used (bytes)
0	6,150,368,648	2,089,453,788,600
1	135,595,631	180,498,938,965
2	23,400,680	57,748,788,688
3	11,201,634	39,601,339,876
4	6,087,922	27,765,408,903
5	2,813,066	15,577,485,346
6	989,603	6,515,029,811
7	518,503	3,962,960,885
8	325,843	2,821,154,088
9	207,968	2,017,299,881
10	163,296	1,754,218,517
11	136,098	1,599,819,087
12	112,042	1,429,782,500
13	86,510	1,195,309,776
14	74,170	1,099,800,313
15	61,877	980,698,602
16	53,320	900,120,875
17	46,907	839,454,243
18	39,220	742,449,495
19	32,196	642,338,712
20	29,261	613,608,158
21	21,936	482,594,306
22	19,436	447,698,254
23	15,504	372,990,015
24	13,616	341,497,628
25	14,338	374,597,107
26	13,650	369,941,596
27	11,338	319,220,899
28	9,796	285,922,219
29	10,003	301,820,285
30	5,538	172,772,085
31	4,338	139,902,554
32	2,148	71,428,243
33	1,477	50,592,932
34	1,177	41,559,715
35	928	33,674,535
36	809	30,156,933
37	373	14,319,264
38	328	12,942,841
39	222	8,958,385
40	145	6,021,609
41	100	4,252,556
42	106	4,607,330
43	59	2,619,596
44	106	4,836,936
45	56	2,603,842
46	45	2,141,898
47	42	2,042,950
48	20	989,613
49	28	1,417,397
50	25	1,291,166
51	25	1,316,187
52	23	1,237,161
53	32	1,751,656
54	35	1,950,640
55	57	3,239,257
56	88	5,096,470
57	127	7,484,887
58	419	25,168,750
59	489	29,704,989
60	1	62,316
61	3	187,461
62	1	64,093
63	2	130,387

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 2 Jun 2014 16:13:47 +0200

On 31 May 2014 10:15, Greg Wilkins <[email-masked]> wrote:

> Consider the example of merging two connection onto a connection, both
> coming from clients using large cookies.  With the current hpack, every
> time the merged connection switches from a stream from one connection to
> the other, it is going to have to removed the large cookies from the state
> table and send the other connection cookies (uncompressed because they are
> cookies).   This will be a rapid degradation in compression performance.

My apologies - I've talking a bit of nonsense on this proxy point.    As
I'm currently implementing hpack, it has dawned on me that it is possible
to remove a header from the reference set without removing it from the
header table (in fact you can't explicitly remove an entry from the header
table, you can only remove it by overflowing or resizing).

Thus my concern that a proxy merging multiple connections onto 1 connection
will end up retransmitting cookies frequently is wrong.   If the header
table is large enough, then fields from all the merged connections can
remain in the header table.  They will just be frequently added/deleted
from the reference set, using small indexed references in the header
block.  I don't see this as a significant problem.

Shows the value of working code... peddling fast as I can to catch up!

However, my concerns remain about the unconstrained size of the headers
that servers must hold; the fact that they are not flow controlled (which
will encourage the unconstrained size usage); that they cannot be
interleaved creating an easy DOS vector; that they must be processed
serially.  I also currently think it is worth evaluating separating the
transport meta-data from other meta data.

regards

---

From: Michael Sweet <[email-masked]>
Date: Mon, 02 Jun 2014 10:51:14 -0400

Greg,

On Jun 2, 2014, at 10:13 AM, Greg Wilkins <[email-masked]> wrote:
> ...
> However, my concerns remain about the unconstrained size of the headers that servers must hold; the fact that they are not flow controlled (which will encourage the unconstrained size usage); that they cannot be interleaved creating an easy DOS vector; that they must be processed serially.  I also currently think it is worth evaluating separating the transport meta-data from other meta data.

Given the default header table size of 4k, I'm guessing that those cookies will get tossed (sorry, couldn't resist :) pretty easily with a shared connection.

Another potential issue (good for header tables but not so good for compression) is the usage of cookies as security tokens ("login cookies"): I wonder whether browsers will end up sending those headers using the 'no index' encoding to avoid CRIME-like attacks? (although that would likely be a bit harder to exploit than the CRIME attacks)

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Jason Greene <[email-masked]>
Date: Mon, 2 Jun 2014 10:15:51 -0500

> It might be possible to keep the stateful delta operations and achieve http 1.1 throughput levels if the receiver, and not just the sender could negotiate the dynamic table size to 0. 

To clarify what I meant, as my wording is poor: The sender trumps the receiver until the sender processes the SETTINGS frame, allowing for a significant number of requests to be sent with a table size of 4096 (since headers have no flow control, and also the wording seems to not disallow a client from putting off processing of the inbound frame). This makes it hard to have a (mostly) stateless proxy that splits or aggregates requests, limiting the scalability and throughput of the system.