Subject: Header Size? Was: Our Schedule

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 May 2014 11:53:54 +0200

On 27 May 2014 16:49, Tatsuhiro Tsujikawa <[email-masked]> wrote:

>
>> Are there any uses **in practice on the public internet** for headers
>> longer than 64K as serialized by HTTP/1.1?
>>
>>
>  ​I don't have actual statistics of length of the header.  But server
> software have its own limit of header buffers.
> For example, HAproxy uses 8K buffer for entire request headers by default
> and its manual says it is not recommended to change the value, so limit is
> somewhere near it?
>

Jetty also imposes an 8k maximum header size on requests and responses.

I've spent the past few days trying really hard to come up with some
constructive proposals to fix some of the aspects of the draft that I'm
concerned about.  However, all the issues I'm concerned about appear to be
well justified if once accepts the need for Infinite Streaming Headers
Encoded With Highly Order Dependent Compression With Shared State
(ISHEWHODCWSS).     So the good news is that the current draft appears to
be very internally consistent and does cover a lot of corner cases. Credit
due!

But this is also a negative as the design decisions are all tightly coupled
and there exists a lot of corner cases.  Most of the coupling, if you
follow the chain, leads back to ISHEWHODCWSS.

Which is why this is so daft! We are tying ourselves in knots to implement
the complexities represented by CONTINUATION frames (and the payload they
can carry), when in practise, most clients and servers will never ever see
a CONTINUATION frame in the wild.   A 16k frame of even moderately
compressed headers far exceeds the memory commitments that most servers
will be prepared to commit to a stream.  While I accept that future
applications are likely to require more and more meta-data, there is no
reason that we should make the transfer meta data channel available for
such transfers and such applications are perfectly able to use streams,
priorities or structure within data frames to accomplish high volume meta
data.

The only time we are going to see continuation frames is in testing.
Perhaps some applications that do not have to work over the public network
will be able to stream large headers, but for the general case they wont
exist.

I really think support for ISHEWHODCWSS  does go beyond the **in practice
on the public internet** and if we just dropped CONTINUATION frames from
the draft we would be so much better off:

+ reduce the complexity of end_stream handling and the resulting state
machine.
+ No infinite data channel that allows user data to evades flow control.
+ No DOS attack possible on shared connections by sending a HEADER without
END_OF_HEADERS but no following CONTINUATION
+ Without the possibility of stretching a HPACK header set over two frames,
there are no HPACK ordering problems and thus no constraints on
interleaving.
+ No need for servers to wait for last frame before knowing common header
field (eg method).

I'm even considering making the first jetty implementation respond with
size errors if it ever sees a CONTINUATION frame.  Even if the incomplete
HEADER frame is small, it represents a reservation of server resources that
I don't know we want to commit to, as the following CONTINUATION frame may
be delayed or may never come!  It would probably fail all interoperability
tests, but work perfectly well in the wild.   That says something about
wasted effort!

The requirement for ISHEWHODCWSS is something that needs to be included in
the FAQ and well explained, as it is responsible for a lot of
dissatisfaction I see with the draft.

regards

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Cory Benfield <[email-masked]>
Date: Wed, 28 May 2014 11:15:35 +0100

On 28 May 2014 10:53, Greg Wilkins <[email-masked]> wrote:
> I'm even considering making the first jetty implementation respond with size
> errors if it ever sees a CONTINUATION frame.  Even if the incomplete HEADER
> frame is small, it represents a reservation of server resources that I don't
> know we want to commit to, as the following CONTINUATION frame may be
> delayed or may never come!  It would probably fail all interoperability

Jetty will interop fine with hyper if you take that approach. hyper
never emits a CONTINUATION frame under any circumstance, and never
will (16 kB of headers is an almost incomprehensibly large amount of
header data). I would not mourn the death of the CONTINUATION frame.

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Wed, 28 May 2014 15:51:36 +0000

> ​I don't have actual statistics of length of the header.

I do :)

The following are based off yesterday's CWS traffic. ~ 6BN requests of which only 123 fall into the > 64K category. So, yes they exist but they're a tiny edge case.
Header sizes in each case are rounded down to the nearest KB.

Richard

Total Size of Request Headers (KB)	Hits	Total Bandwidth used (bytes)
0	4,872,582,729	2,328,383,321,450
1	1,020,373,150	1,413,712,005,239
2	257,251,419	635,717,697,310
3	100,069,677	351,044,064,205
4	44,559,279	202,820,971,360
5	19,501,513	108,407,535,016
6	10,030,017	66,265,639,126
7	5,958,110	45,505,452,109
8	3,442,549	29,770,417,740
9	1,947,956	18,866,794,032
10	1,167,259	12,505,049,055
11	716,798	8,413,448,445
12	493,658	6,304,415,009
13	380,169	5,245,221,153
14	290,264	4,306,207,353
15	226,623	3,590,185,566
16	182,467	3,080,199,567
17	158,994	2,847,223,916
18	154,289	2,921,099,390
19	124,088	2,476,816,871
20	106,242	2,228,673,523
21	97,869	2,154,231,469
22	78,943	1,818,378,619
23	76,933	1,850,363,872
24	64,328	1,613,928,318
25	65,263	1,703,500,953
26	53,204	1,443,038,201
27	48,453	1,364,127,888
28	41,576	1,212,854,050
29	37,563	1,134,034,148
30	28,949	903,359,201
31	20,702	667,098,744
32	15,389	513,407,094
33	9,278	317,754,854
34	6,122	216,228,441
35	5,016	182,332,930
36	6,025	224,889,071
37	3,610	138,538,389
38	3,879	153,077,676
39	4,015	162,104,070
40	2,317	96,090,026
41	2,141	90,965,098
42	2,466	107,214,503
43	1,898	84,569,085
44	1,873	85,334,787
45	1,879	87,536,079
46	1,594	75,938,436
47	1,834	89,202,662
48	3,406	169,000,837
49	1,929	97,776,397
50	1,586	82,011,745
51	1,544	81,431,356
52	1,637	87,995,729
53	1,667	91,349,565
54	1,492	83,281,926
55	1,517	86,234,890
56	1,596	92,383,611
57	1,885	111,044,218
58	1,241	74,190,694
59	618	37,621,315
60	367	22,719,508
61	400	25,195,632
62	481	30,763,860
63	385	25,011,901
64	105	6,913,055
65	1	66,763
69	1	70,658
89	16	1,463,440

---

From: Martin Thomson <[email-masked]>
Date: Wed, 28 May 2014 10:23:27 -0700

> Header sizes in each case are rounded down to the nearest KB.

Awesome, thanks!  If we are interested in discussing who to throw off
the bus, 64K seems like good break point to discuss.  Though that
doesn't avoid the need for continuations entirely.  16K would be more
attractive, but your numbers seem to suggest that that would piss off
a considerably larger number of people.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 May 2014 22:50:53 +0200

Awesome info.
Any chance of getting that broken down by request headers verses response
headers?

cheers

On 28 May 2014 17:51, Richard Wheeldon (rwheeldo) <[email-masked]>wrote:

> The following are based off yesterday's CWS traffic. ~ 6BN requests of
> which only 123 fall into the > 64K category. So, yes they exist but they're
> a tiny edge case.

---

From: David Krauss <[email-masked]>
Date: Thu, 29 May 2014 07:54:28 +0800

On 2014–05–29, at 1:23 AM, Martin Thomson <[email-masked]> wrote:

> doesn't avoid the need for continuations entirely.  

It does if the max frame size goes back up to 64K. It was only reduced to artificially make continuations more likely, right?

As for common-case head of queue blocking, DATA frame payloads can still be limited to 16K if we like. Such a limit disparity also solves the padding granularity problem.

Again I’ll suggest that nobody gets “thrown off the bus” if the canonical translation to and from HTTP/1.1 uses an initial sequence of header blocks, with routing information going into the first block.

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 29 May 2014 10:21:03 +1000

Taking a slightly different tack here, but the issue is with a single
HEADERS+CONTINUATION+... block, right? I may need to re-read the draft, but
I don't recall anything in there that explicitly forbids anyone sending
HEADERS[EH], HEADERS[EH], DATA[ES] on a single stream.

If such a restriction exists in the transport/framing layer, it should
probably documented (more clearly, if it is already). Or, maybe better, it
should be eliminated, along with the necessity for CONTINUATION frames. As
far as I can see, that would only affect the entity who has to finally
translate from HTTP2 frames to HTTP events (e.g. the client transport
library buffering response headers so it can send a single "received
initial headers" signal to the browser, if the browser expects a single
signal).

Please correct me if I'm mistaken here, and direct me to relevant text and
discussion.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Roberto Peon <[email-masked]>
Date: Wed, 28 May 2014 17:29:28 -0700

...And we chose one size limit for everything, which included headers
frames.
On May 28, 2014 5:28 PM, [email-masked] wrote:

> No it was reduced to ensure that people mux with small enough segments
> that the priority inversion delay that occurs when higher priority data is
> blocked behind lower priority data is small enough that the protocol works
> properly.
>
> That it helps to prevent the backup generator problem is icing.
> On May 28, 2014 5:02 PM, "David Krauss" <[email-masked]> wrote:
>
>>
>> It does if the max frame size goes back up to 64K. It was only reduced to
>> artificially make continuations more likely, right?
>>
>> As for common-case head of queue blocking, DATA frame payloads can still
>> be limited to 16K if we like. Such a limit disparity also solves the
>> padding granularity problem.
>>
>> Again I’ll suggest that nobody gets “thrown off the bus” if the canonical
>> translation to and from HTTP/1.1 uses an initial sequence of header blocks,
>> with routing information going into the first block.
>>
>>
>>

---

From: Roberto Peon <[email-masked]>
Date: Wed, 28 May 2014 17:28:45 -0700

No it was reduced to ensure that people mux with small enough segments that
the priority inversion delay that occurs when higher priority data is
blocked behind lower priority data is small enough that the protocol works
properly.

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 29 May 2014 16:52:51 +1200

On 29/05/2014 11:54 a.m., David Krauss wrote:
> 
> On 2014–05–29, at 1:23 AM, Martin Thomson wrote:
> 
>> On 28 May 2014 08:51, Richard Wheeldon (rwheeldo) wrote:

Both goals can be achieved by:

* explicitly limiting CONTINUATION to 3 consecutive frames, and

* allowing other streams frames other than HEADERS, PUSH_PROMISE,
CONTINUATION to be delivered between HEADERS and CONTINUATION.

Personally I am in favour of 64K limit on headers. However, the
Cookie/Set-Cookie size problem is a hard nut to crack.

Also might I remind that Squid already has a few complaints about our
32KB default limit and people patching the code to handle >64KB
individual header length for auth tokens in NTLM/Negotiate logins when
(long) lists of groups and SID are encoded inside them.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 29 May 2014 07:25:44 +0200

> (long) lists of groups and SID are encoded inside them.

FWIW, haproxy ships with a 8kB default limit, and in our appliances
it's even 7kB. We had maybe only twice to explain to people how to
raise the limit, and each time it was because of an application bug
causing cookies to be duplicated for each request, resulting in
requests of several 10s of kB after hundreds of requests. I personally
don't expect such an application bug to drive the protocol limits :-)

Just like Greg, I think that 8kB is already a high reasonable limit
and that if we push it to 16kB we cover a most usages. It's possible
that Richard's stats include bogus applications and/or attacks BTW.

regards,
Willy

---

From: Simone Bordet <[email-masked]>
Date: Thu, 29 May 2014 09:39:55 +0200

Hi,

>From Richard's numbers, the headers greater than 16k represent the
0.026% of the hits, so 16 KiB indeed covers most usages.

-- 
Simone Bordet
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 May 2014 11:52:42 +0200

On 29 May 2014 06:52, Amos Jeffries <[email-masked]> wrote:

> Personally I am in favour of 64K limit on headers

That is an enormous increase in the resource commitment required by servers
- per stream!. It is at least 8x the defacto standard an that is not
counting compression.

Roberto says that the same 16k  size limit has been applied to everything -
which is not a bad idea.  So why exclude the poor servers from this?
Server must hold onto the all the headers to make them available throughout
the request processing, so allowing 64KB of compressed headers,  which
could easily turn into close to much more than that, is a big commitment.

The meta data channel required for transport of HTTP semantics is much
smaller than that.  8KB does almost all cases - specially on the request
side of things.

Sure future protocols are probably going to want more and more meta data -
but why do we have to make the transport meta data channel available for
such future protocols.  Let them open their own high priority stream, or
send additional header sets within the data stream.   Let's not open the
flood gates on the transport meta data channel, complete with the special
exclusions from flow control and segmentation just to make it even more
attractive to use!

regards

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Thu, 29 May 2014 13:29:43 +0000

From: Greg Wilkins [mailto:[email-masked]] 
 > Any chance of getting that broken down by request headers verses response headers?

Those were just the request headers. The following are the response header sizes from the same day for comparison. As expected, broadly similar but slightly smaller.

Total Size of Response Headers (KB)	Hits	Total Bandwidth used (bytes)
0	6,150,368,648	4,818,463,093,379
1	135,595,631	226,554,380,248
2	23,400,680	78,690,794,667
3	11,201,634	53,346,476,556
4	6,087,922	38,790,955,943
5	2,813,066	21,299,008,205
6	989,603	8,696,416,852
7	518,503	4,616,427,850
8	325,843	3,138,491,657
9	207,968	2,331,339,013
10	163,296	2,017,103,198
11	136,098	1,845,784,911
12	112,042	1,621,155,531
13	86,510	1,335,213,171
14	74,170	1,216,208,789
15	61,877	1,076,372,856
16	53,320	988,223,924
17	46,907	918,692,632
18	39,220	805,060,723
19	32,196	698,219,067
20	29,261	660,685,062
21	21,936	510,140,410
22	19,436	478,537,442
23	15,504	386,104,802
25	14,338	387,132,367
26	13,650	375,504,978
24	13,616	352,044,271
27	11,338	321,842,045
29	10,003	304,576,697
28	9,796	289,315,178
30	5,538	174,006,785
31	4,338	140,023,490
32	2,148	71,182,461
33	1,477	50,875,071
34	1,177	41,067,240
35	928	32,955,327
36	809	30,803,686
59	489	29,261,095
58	419	24,894,135
37	373	14,050,953
38	328	12,590,352
39	222	8,481,291
40	145	2,881,228
57	127	7,500,047
42	106	4,253,873
44	106	3,717,316
41	100	4,493,721
56	88	5,058,537
43	59	2,154,826
55	57	3,174,147
45	56	1,790,793
46	45	1,690,620
47	42	1,563,170
54	35	1,951,776
53	32	1,822,445
49	28	1,530,165
50	25	1,401,037
51	25	1,075,256
52	23	1,231,341
48	20	945,773
61	3	9,275
63	2	1,996
60	1	2,810
62	1	1,669

---

From: Wenbo Zhu <[email-masked]>
Date: Thu, 29 May 2014 10:41:13 -0700

On Thu, May 29, 2014 at 2:52 AM, Greg Wilkins <[email-masked]> wrote:

> That is an enormous increase in the resource commitment required by
> servers - per stream!. It is at least 8x the defacto standard an that is
> not counting compression.
>
> Roberto says that the same 16k  size limit has been applied to everything
> - which is not a bad idea.  So why exclude the poor servers from this?
URLs may actually encode data for (GET) requests.  I think 16k is a
reasonable limit, but it's also an arbitrary limit (as most limits do).

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 30 May 2014 15:51:58 +0900

This is just a thought:

Would it be possible to allow arbitrarily large amounts of header data 
(either via continuations or via multiple header frames), but to limit 
compression to a single header frame.

While in general, there is a stronger need to compress larger stuff, 
such a solution could come with various benefits:
- Simplified compression (less/no state)
- Keep the main benefit (quick start)
- Penalty against large amounts of header data
   (because that's not the way to do things anyway)

Regards,   Martin.

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 30 May 2014 17:34:18 +1000

On 30 May 2014 16:51, "Martin J. Dürst" <[email-masked]> wrote:

> While in general, there is a stronger need to compress larger stuff, such
> a solution could come with various benefits:
If you send SETTINGS_HEADER_TABLE_SIZE=0 and a HEADERS with [0x30,0x20] in
the first block fragment you effectively disable the context, and are left
with only Huffman coding (which has a per-frame context).

As Roberto reminded me yesterday, the thing about a header block is that
when it ends, you get everything else in the reference set (carried over
from the previous header block). The biggest gain in HPACK compression
comes from not actually sending identical headers again and again, which
means not only sharing context between multiple frames, but between frames
from multiple streams. I don't know if, in practice, any per-frame
compression scheme would come close to HPACK's connection-based delta
compression, and that would be a big hit to the protocol's appeal.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 30 May 2014 10:22:45 +0200

Matthew,

firstly I'm sure there are forms of header compression that can have a
shared state table that are not so highly order dependent.  The problem
with hpack is that every field in every frame of stream can mutate the
shared state table.  This gives us a really hard serialisation problem, so
that setting the table size to zero does not help, as you still have to
prevent interleaving so you can decode in order and watch for an increase
in the table size.

I think we could get a lot of benefit from a compression scheme that uses
header frames transmitted on channel 0 to set the shared state.   All the
user-agent guff could then be sent once and only once and all the stream
header decoding would then be read only (and thus could happen in any
order).    If you wanted to put cookies into the shared table, then there
are still some ordering issues, but not as hard as the current ones and
with lots of potential solutions (eg table versions or multiple table ids
etc.).

To Martins idea,  that would help in some respects if the subsequent frames
are excluded from hpack.  This would let us allow interleaving.  However,
it does not prevent the server from needing to hold onto large header
tables during request handling.  So we still should include headers in the
flow control, so the receiver can say "stop already!" when large headers
are being sent.

cheers

---

From: Michael Sweet <[email-masked]>
Date: Fri, 30 May 2014 10:14:04 -0400

Greg,

I don't see shared state like that working for proxies.

On May 30, 2014, at 4:22 AM, Greg Wilkins <[email-masked]> wrote:

> Matthew,
> 
> firstly I'm sure there are forms of header compression that can have a shared state table that are not so highly order dependent.  The problem with hpack is that every field in every frame of stream can mutate the shared state table.  This gives us a really hard serialisation problem, so that setting the table size to zero does not help, as you still have to prevent interleaving so you can decode in order and watch for an increase in the table size.
> 
> I think we could get a lot of benefit from a compression scheme that uses header frames transmitted on channel 0 to set the shared state.   All the user-agent guff could then be sent once and only once and all the stream header decoding would then be read only (and thus could happen in any order).    If you wanted to put cookies into the shared table, then there are still some ordering issues, but not as hard as the current ones and with lots of potential solutions (eg table versions or multiple table ids etc.).
> 
> To Martins idea,  that would help in some respects if the subsequent frames are excluded from hpack.  This would let us allow interleaving.  However, it does not prevent the server from needing to hold onto large header tables during request handling.  So we still should include headers in the flow control, so the receiver can say "stop already!" when large headers are being sent.
> 
> cheers
> 
> 
> 
> Would it be possible to allow arbitrarily large amounts of header data (either via continuations or via multiple header frames), but to limit compression to a single header frame.
> 
> While in general, there is a stronger need to compress larger stuff, such a solution could come with various benefits:
> If you send SETTINGS_HEADER_TABLE_SIZE=0 and a HEADERS with [0x30,0x20] in the first block fragment you effectively disable the context, and are left with only Huffman coding (which has a per-frame context).
> 
> As Roberto reminded me yesterday, the thing about a header block is that when it ends, you get everything else in the reference set (carried over from the previous header block). The biggest gain in HPACK compression comes from not actually sending identical headers again and again, which means not only sharing context between multiple frames, but between frames from multiple streams. I don't know if, in practice, any per-frame compression scheme would come close to HPACK's connection-based delta compression, and that would be a big hit to the protocol's appeal.
> 
> 
> -- 
>   Matthew Kerwin

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Fri, 30 May 2014 16:06:06 +0000

Speaking as a proxy developer, I like the idea of putting common header stuff onto frame 0. Common identity-specific stuff (user-agent) can easily be shared. We already do similar things -(re-using state from earlier requests on a keep alive connection. It's a big win over re-sending,

Richard

From: Michael Sweet [mailto:[email-masked]]
Sent: 30 May 2014 09:14
To: Greg Wilkins
Cc: Matthew Kerwin; "Martin J. Dürst"; David Krauss; Martin Thomson; Richard Wheeldon (rwheeldo); HTTP Working Group
Subject: Re: Header Size? Was: Our Schedule

Greg,

On May 30, 2014, at 4:22 AM, Greg Wilkins <[email-masked]<mailto:[email-masked]>> wrote:

Matthew,

On 30 May 2014 09:34, Matthew Kerwin <[email-masked]<mailto:[email-masked]>> wrote:
On 30 May 2014 16:51, "Martin J. Dürst" <[email-masked]<mailto:[email-masked]>> wrote:
This is just a thought:

Regards,   Martin.

--
Greg Wilkins <[email-masked]<mailto:[email-masked]>>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com<http://www.webtide.com/>  advice and support for jetty and cometd.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 30 May 2014 09:31:24 -0700

> compression to a single header frame.

That was raised, I believe.  The problem was that there were cases
where a single field was larger than the frame size limit.  That made
it more difficult.  Or at least difficult enough to want to avoid
further consideration of that option.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 30 May 2014 12:43:33 -0400

Richard,

I'm thinking from the proxy to the upstream server.  If you reuse the same upstream connection for multiple downstream clients, it is quite likely that they will not have the same User-Agent or other headers...

On May 30, 2014, at 12:06 PM, Richard Wheeldon (rwheeldo) <[email-masked]> wrote:

> Speaking as a proxy developer, I like the idea of putting common header stuff onto frame 0. Common identity-specific stuff (user-agent) can easily be shared. We already do similar things -(re-using state from earlier requests on a keep alive connection. It’s a big win over re-sending,
>  
> Richard
>

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 May 2014 11:13:28 -0700

The system properties of having order dependency (with shared-state on
stream 0) would still exist, since the receiver couldn't control when the
sender actually emitted things on stream zero.
This would also work very poorly with proxies which have multiple clients
and fewer servers-- these are muxing between clients. User-agent, etc. is
changing all the time in such circumstances.

Incidentally, HPACK allows for non-HOL-blocking compression, with the
addition of a sequence number per frame. This isn't needed right now given
that the APIs we have to TCP always deliver bytes in sequence, but if we
had APIs which delivered the data out-of-order as it was received (e.g.
TCP-minion), one could take advantage of it.

On Fri, May 30, 2014 at 1:22 AM, Greg Wilkins <[email-masked]> wrote:

> To Martins idea,  that would help in some respects if the subsequent
> frames are excluded from hpack.  This would let us allow interleaving.
> However, it does not prevent the server from needing to hold onto large
> header tables during request handling.  So we still should include headers
> in the flow control, so the receiver can say "stop already!" when large
> headers are being sent.
>
> cheers
>
>
>
>> If you send SETTINGS_HEADER_TABLE_SIZE=0 and a HEADERS with [0x30,0x20]
>> in the first block fragment you effectively disable the context, and are
>> left with only Huffman coding (which has a per-frame context).
>>

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 May 2014 11:13:40 -0700

What he said :)

On Fri, May 30, 2014 at 9:43 AM, Michael Sweet <[email-masked]> wrote:

> Richard,
>
> I'm thinking from the proxy to the upstream server.  If you reuse the same
> upstream connection for multiple downstream clients, it is quite likely
> that they will not have the same User-Agent or other headers...
>
>
>  Speaking as a proxy developer, I like the idea of putting common header
> stuff onto frame 0. Common identity-specific stuff (user-agent) can easily
> be shared. We already do similar things -(re-using state from earlier
> requests on a keep alive connection. It’s a big win over re-sending,
>
>
>
> Richard
>
>
>
> *From:* Michael Sweet [mailto:[email-masked] <[email-masked]>]
> *Sent:* 30 May 2014 09:14
> *To:* Greg Wilkins
> *Cc:* Matthew Kerwin; "Martin J. Dürst"; David Krauss; Martin Thomson;
> Richard Wheeldon (rwheeldo); HTTP Working Group
> *Subject:* Re: Header Size? Was: Our Schedule
>
>
>
> Greg,
>
>
>

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 30 May 2014 22:51:06 +0200

But Proxies will already work badly with the shared hpack table.  If the
different streams have significantly different header sets, then they will
be forever resetting the table or worse replacing each header one by one
(as I doubt many encoders will have the look ahead logic to guess if reset
or replace is best).

So a single table on channel 0 will probably not be that different.   At
least a single static table on 0 would allow streams to be processed in
parallel and/or out of order.

But I do think the proxy case is important and we should well support
aggregating streams.  hpack does not do that now, but a channel 0 mechanism
could be augmented with header set versions, so each stream would just
refer to which version it referred to.  This would well support proxies and
also allow arbitrary reordering and parallel execution of streams.

In short, I think that considering proxies is an argument against hpack
rather than for it.

cheers

On 30 May 2014 20:13, Roberto Peon <[email-masked]> wrote:

>> I'm thinking from the proxy to the upstream server.  If you reuse the
>> same upstream connection for multiple downstream clients, it is quite
>> likely that they will not have the same User-Agent or other headers...
>>
>>

---

From: "Richard Wheeldon (rwheeldo)" <[email-masked]>
Date: Fri, 30 May 2014 21:08:40 +0000

I was never thinking of a sharing an upstream connection for multiple downstream clients. In that case we’d run into even more complexity and security concerns around compression, etc. It’s an interesting use-case though. I’m curious if anyone operates multi-tenant proxies in such a mode and what their experience is of compatibility in the current Web landscape?

Richard

From: Roberto Peon [mailto:[email-masked]]
Sent: 30 May 2014 13:14
To: Michael Sweet
Cc: Richard Wheeldon (rwheeldo); Greg Wilkins; Matthew Kerwin; Martin J. Dürst; David Krauss; Martin Thomson; HTTP Working Group
Subject: Re: Header Size? Was: Our Schedule

What he said :)

On Fri, May 30, 2014 at 9:43 AM, Michael Sweet <[email-masked]<mailto:[email-masked]>> wrote:
Richard,

On May 30, 2014, at 12:06 PM, Richard Wheeldon (rwheeldo) <[email-masked]<mailto:[email-masked]>> wrote:

Richard

Greg,

Matthew,

Regards,   Martin.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 May 2014 15:06:47 -0700

Note that the shared-state problem is an issue with any compressor that
shares state (it is almost definitional! :) ). The majority (if not all)
compressors that would act on a per-frame basis offer limited benefit (e.g.
huffman, which one can do today), or have potentially poor security
properties.

We have made many design concessions on behalf of proxies. I've driven
many, if not most of them, as I've acutely felt the pain of operating
proxies at large scale for years.. This particular design point (that of
sharing one context per connection) was chosen because it cost/complexity
of having multiple contexts was not assured to be outweighed by the
benefit. The original, discarded (for the aforementioned reason), design
had stream-groups which defined which compression and flow control context
into which a stream would be binned.

Alternately said, the issue is less a matter of HPACK specifically, and
more a matter of how many compression contexts we wish to manage. The
optimal thing for proxies is generally one context per server endpoint
(e.g. per origin), or one context per client endpoint, but the complexity
cost is higher.

-=R

On Fri, May 30, 2014 at 1:51 PM, Greg Wilkins <[email-masked]> wrote:

>>>  Speaking as a proxy developer, I like the idea of putting common
>>> header stuff onto frame 0. Common identity-specific stuff (user-agent) can
>>> easily be shared. We already do similar things -(re-using state from
>>> earlier requests on a keep alive connection. It’s a big win over re-sending,
>>>
>>>
>>>
>>> Richard
>>>
>>>
>>>
>>> I think we could get a lot of benefit from a compression scheme that
>>> uses header frames transmitted on channel 0 to set the shared state.   All
>>> the user-agent guff could then be sent once and only once and all the
>>> stream header decoding would then be read only (and thus could happen in
>>> any order).    If you wanted to put cookies into the shared table, then
>>> there are still some ordering issues, but not as hard as the current ones
>>> and with lots of potential solutions (eg table versions or multiple table
>>> ids etc.).
>>>

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 31 May 2014 10:15:45 +0200

Roberto,

it is true that any shared state compression will be somewhat compromised
in the proxy situation that we are discussing.  However, hpack is
particularly compromised by its design that the emitted header set includes
precisely all the headers in the shared state that have not been
referenced.  Thus at the end of a frame, the shared state is 100% tailored
to the current stream.

In comparison, something like gzip can have a shared state that combines
compression state of multiple streams and the resulting compression will
gracefully degrade as more and more diverse streams are added.  It is
essentially blind to frame boundaries.

I can imagine variations of hpack that are not so compromised - ie by
requiring all emitted headers to be explicitly indexed.   This would allow
a shared state table to included headers from more than 1 stream.  Sure
management of that table would be a bit tricky, but then management of the
single table is already complex as the encoder has to pick a suitable
strategy and may get it wrong.

If such a shared frame boundary blind hpack state table was only mutated by
frames received on channel 0, that would free up the processing of other
streams so they could be done in parallel and out of order with respect to
each other.

If the shared table could only have headers added to in and not deleted
from it, then the processing of non-0 frame headers would be free from any
sequential access issues, but would instead have a size/growth issue
instead.   This could be addressed by simply having a fixed size, which
once hit compression was no longer dynamic, or by versioned tables that
could be deleted once all streams referencing them are complete.   None of
these things are particularly more complex than what we have today.

Consider the example of merging two connection onto a connection, both
coming from clients using large cookies.  With the current hpack, every
time the merged connection switches from a stream from one connection to
the other, it is going to have to removed the large cookies from the state
table and send the other connection cookies (uncompressed because they are
cookies).   This will be a rapid degradation in compression performance.
The natural burst nature of requests will probably help out for the case of
2 merged connections, but pretty soon with n connections the result will be
most streams having to carry it's own uncompressed cookie headers.

A shared table that is blind to frame boundaries, would instead be able to
hold the cookies from both connections.   Sure there is a table size issue
here as more and more connections are added and the table has to hold more
and more cookies.   But excluding the cookies from the share table does not
remove the requirement for the server to have to have them in memory and
make them available to the request handling.   In fact with the current
hpack, servers will end up with multiple copies of the same cookie in
memory as the merged connection adds/removes/adds/removes the same cookie
from the shared table multiple time.

Having a large shared table is going to have less memory impacts than a
smaller contested table.

I think it may well be worthwhile re-evaluating the proposals for multiple
contexts, as not only could these designs help with the proxy case, but
they could potentially free up the serialisation within receiver, giving a
more immediate benefit.

regards

On 31 May 2014 00:06, Roberto Peon <[email-masked]> wrote:

>>>> As Roberto reminded me yesterday, the thing about a header block is
>>>> that when it ends, you get everything else in the reference set (carried
>>>> over from the previous header block). The biggest gain in HPACK compression

---

From: Roberto Peon <[email-masked]>
Date: Sat, 31 May 2014 01:28:55 -0700

gzip, hpack, whatever, the issue is state management in both cases-- they
require serialization/deserialization to be done in the same order. Putting
things on stream zero doesn't change this-- it creates a sequence binding
between stream zero and the other streams, and there is no guarantee that
it wouldn't look exactly the same in terms of backreference requirements.

In the last paragraph or so, of your email you're effectively describing
the groups implementation of the original "delta" compressor-- the index
table was specific to a stream-group, whereas the state entries were shared
amongst everything in the connection. This was important to keep the code
sane when dealing with popping old state (with a single table it is pretty
obvious what to do), and seemed to also be the most effective anyway.

I agree that that design is better for proxies (which is why that is the
way I did it (and flow contol too) with stream groups, but I also decided
that it wasn't a panacea-- the encoder has to decide which context to put a
stream into (likely by guessing based on origin), and then the receiver
needs to be able to move it to the more appropriate context, assuming we
care about optimality for proxies. The requirement to use stream groups or
something similar also bloats the size of headers frames by some number of
bits, which trades-off on compression, and then maintaining separate tables
which may share entries ends up being sumoptimal in some cases.

This seemed like a fair bit of mechanism, especially when incorrect choices
result in poorer compression and a chattier protocol, and when the search
for optimal is as difficult as it is here.
-=R

On Sat, May 31, 2014 at 1:15 AM, Greg Wilkins <[email-masked]> wrote:

> If such a shared frame boundary blind hpack state table was only mutated
> by frames received on channel 0, that would free up the processing of other
>>>>> *Cc:* Matthew Kerwin; "Martin J. Dürst"; David Krauss; Martin
>>>>> Thomson; Richard Wheeldon (rwheeldo); HTTP Working Group
>>>>> If you send SETTINGS_HEADER_TABLE_SIZE=0 and a HEADERS with
>>>>> [0x30,0x20] in the first block fragment you effectively disable the
>>>>> context, and are left with only Huffman coding (which has a per-frame
>>>>> context).
>>>>>
>>>>>
>>>>>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 31 May 2014 10:28:19 +0000

In message <CAP+FsNeGtUgBNq0WN4GuYCPj0eVCm07ta6sqek=[email-masked]>

>it wouldn't look exactly the same in terms of backreference requirements.

This is why I originally suggested that we put a "envelope" (Host:
+ non-query part of URL) on the outside of all this complexity, so
that load-balancers would not have to care about it at all.

The failure to do so, is one of the main reasons why HTTP/2 will have
limited performance potential and likely become a bottleneck.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 01 Jun 2014 00:33:09 +1200

On 31/05/2014 9:08 a.m., Richard Wheeldon (rwheeldo) wrote:
> I was never thinking of a sharing an upstream connection for multiple
> downstream clients. In that case we’d run into even more complexity
> and security concerns around compression, etc. It’s an interesting
> use-case though. I’m curious if anyone operates multi-tenant proxies
> in such a mode and what their experience is of compatibility in the
> current Web landscape?

That pretty much is the current landscape regarding how ISP proxies
operating persistent connections to servers operate today in HTTP/1.

Request streams get even more complicated when there are multiple layers
of proxy involved for load balancing. As we get multiple client requests
pipelined on connections to the upstream proxy, which then may either
split again or aggregate with other downstream proxy connections (or
both) onto the backend servers.

Similar architecture is also in use many places for CDN / reverse-proxy.
Particularly in cloud access situations where multiple Enterprise
domains/services can point at a gateway proxy. Which aggregates and
relays to an upstream cloud API where the traffic may be split again (or
not) internally within the cloud service.

Amos

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 31 May 2014 19:52:48 +0000

In message <[email-masked]>, "Jason T. Greene"
 writes:

>IMO the only way to allow for a proxy to directly hand off requests as
>is would be per-frame compression, where the URL is easily located and
>matched. 

And this is why I think state-full compression of headers necessary
for routing HTTP is a terminally bad idea for HTTP/2.0, since it
causes state to materialize all over the place, state which increases
cost, decrease reliability and which is vulnerable to DoS'ing at
all the most sensitive choke-points.

My original suggestion was to have two parts to HTTP/2 transport
messages:

	[route headers]
	[payload]

Where the route headers would consist of:
	length of route headers
	length of payload
	Host:
	URL sans query part  (or possibly:  URL up to first '/')
	(X-)Forwarded-For:

(Trivial details to cater for multiplexing and "Chunked" style
incremental delivery left out for clarity).

Route headers are uncompressed, in order to make it possible to
route 1Tbit/sec HTTP traffic without holding an expensive state.

The payload consists of metadata and objects

Metadata is the rest of the HTTP headers, which are not needed for
transporting HTTP, but only play a role in semantic interpretation:
Content-Type, Content-Encoding etc. etc.

Metadata and object can be compressed or encrypted how ever you like.

The information leak of the Host: and first part of the URL are
insignificant relative to privacy:  If a site wants to
offer increased privacy, both can trivially be emptied of
information in the site-design.

Having the route header unencrypted also means that legally mandated
inspecting proxies can *avoid* inspecting privileged traffic for
instance between prison inmates, their lawyers and politicians,
by implementing a white-list.

---

From: <[email-masked]>
Date: Sun, 1 Jun 2014 00:10:42 +0000

Hi Poul-Henning-

I'm not very familiar with proxy design and constraints, but it sounds like a reasonable idea to me. From a performance point of view, it seems like a bad idea for proxies to constantly decompress (and possibly recompress) headers.

Could you achieve the same goal within the current spec by requiring the route headers to come first and uncompressed in the first HEADERS frame of a stream?

Are these headers you mentioned the only three proxies ever need to look at when forwarding requests?
>        URL sans query part  (or possibly:  URL up to first '/')

What about forwarding responses back to the client? I imagine you would want to do a similar thing. Same headers?

-keith

On 31 May 2014 21:52, [email-masked] wrote:

        [route headers]
        [payload]

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Roberto Peon <[email-masked]>
Date: Sat, 31 May 2014 17:21:57 -0700

HPACK was designed to make re-indexing when proxying possible.
It would require some complexity (one has to keep a map of
client-side-index to server-side-index when sharing state between
client-side and server-side), but such is often the tradeoff for optimality.

Poul's proposal to put routing info at the front pre-supposes that all
proxies use the same information for routing.This is often not the case.
As per my experience, his list is incomplete. Routing often happens due to
method, user-agent, cookie, accept-encoding, etc., and sometimes some
random headers that the proxy is configured to examine. Basically, I can't
predict what these may be today, and thus forcing an ordering doesn't help
my proxy in most use-cases. In many cases, I'll want to capture the entire
block of headers before forwarding to servers which my proxy may protect.
Again, having some specific headers listed first doesn't help in that use
case.

Regardless, though, if one could identify this list then with something
like HPACK one simply refers to the index of those data at the start of the
header block.
This has been discussed before, I'll point out.
-=R

On Sat, May 31, 2014 at 5:10 PM, <[email-masked]> wrote:

> Hi Poul-Henning-
>
> I'm not very familiar with proxy design and constraints, but it sounds
> like a reasonable idea to me. From a performance point of view, it seems
> like a bad idea for proxies to constantly decompress (and possibly
> recompress) headers.
>
> Could you achieve the same goal within the current spec by requiring the
> route headers to come first and uncompressed in the first HEADERS frame of
> a stream?
>
> What about forwarding responses back to the client? I imagine you would
> want to do a similar thing. Same headers?
>
> -keith
>
>
> This email message is intended only for the use of the named recipient.
> Information contained in this email message and its attachments may be
> privileged, confidential and protected from disclosure. If you are not the
> intended recipient, please do not read, copy, use or disclose this
> communication to others. Also please notify the sender by replying to this
> message and then delete it from your system.
>