Subject: Fragmentation for headers: why jumbo != continuation.

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Jul 2014 13:27:01 -0700

There are two separate reasons to fragment headers

1) Dealing with headers of size > X when the max frame-size is <= X.
2) Reducing buffer consumption and latency.

Most of the discussion thus far has focused on #1.
I'm going to ignore it, as those discussions are occurring elsewhere, and
in quite some depth :)

I wanted to be sure we were also thinking about #2.

Without the ability to fragment headers on the wire, one must know the size
of the entire set of headers before any of it may be transmitted.

This implies that one must encode the entire set of headers before sending
if one will ever do transformation of the headers. Encoding the headers in
a different HPACK context would count as a transformation, even if none of
the headers were modified.

This means that the protocol, if it did not have the ability to fragment,
would require increased buffering and increased latency for any proxy by
design.

This is not currently true for HTTP/1-- the headers can be sent/received in
a streaming fashion, and implementations may, at their option, choose to
buffer in order to simplify code.

-=R

---

From: <[email-masked]>
Date: Thu, 10 Jul 2014 20:45:28 +0000

Isn't fragmenting solved by eliminating the reference set? IIUC, removing the reference set allows the headers to be fragmented (with 1*HEADERS) and multiplexed (as long as the HEADERS frames are processed in order).

-Keith

> On Jul 10, 2014, at 22:30, "[email-masked]" <[email-masked]> wrote:
>
> I'm going to ignore it, as those discussions are occurring elsewhere, and in quite some depth :)
>
>
> Without the ability to fragment headers on the wire, one must know the size of the entire set of headers before any of it may be transmitted.
>
> This implies that one must encode the entire set of headers before sending if one will ever do transformation of the headers. Encoding the headers in a different HPACK context would count as a transformation, even if none of the headers were modified.
>
> This means that the protocol, if it did not have the ability to fragment, would require increased buffering and increased latency for any proxy by design.
>
> This is not currently true for HTTP/1-- the headers can be sent/received in a streaming fashion, and implementations may, at their option, choose to buffer in order to simplify code.
>
> -=R
This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 10 Jul 2014 23:00:12 +0200

Hi Roberto,

> buffer in order to simplify code.

Well, while implementations may do it in HTTP/1 at their option, in practice
it's only used by low-end hacks which do write(fd, string, length) and write
one header at a time. Indeed, doing so with TCP_NODELAY results in as many
packets with a PUSH flag as write() calls, which is absolutely suboptimal.

Also I tend to see it quite differently : the microsecond which is needed
to process all headers at once is very low compared to the extra work the
receiver has to do to retrieve the context of the request currently being
parsed for each chunk of header received. That's one ugliness of the
continuations frames in my opinion. I expect that a number of implementations
will very likely read each frame, convert it to HTTP/1 and apply existing
parser on the H/1 result, to discover that it's incomplete. And they'll
rebuild and re-parse the whole incomplete H/1 request after each incoming
frame. Just send one header at a time and you get an O(N^2) parsing.

Do you have numbers showing the extra latency you'd expect from passing
headers at once ? As I reported in an earlier e-mail, I'm seeing in the
order of 1 microsecond to parse a whole HTTP/1 request, and I hope that
H/2 will not increase that by orders of magnitude. Currently, operating
systems incur a much larger granularity, even a switch port gives you a
much higher granularity. I'd suspect that if you *measure* the difference
between headers-at-once and one-at-a-time, you'd get a faster sending
with all at once thanks to lower processing overhead and network overhead.

Just my two cents,
Willy

---

From: Roland Zink <[email-masked]>
Date: Thu, 10 Jul 2014 23:02:54 +0200

With HTTP/1 the URL is sent first and a proxy would know where to stream 
before the first header. With HPACK there is no guarantee that this is 
possible. Also when a proxy try to stream then it would block the whole 
connection to the server when the client doesn't send a continuation 
frame, so it need to buffer. Although this isn't a problem with 1:1 
connections in general this means the buffering is needed. Btw. this is 
true for HTTP/1 as well and in the implementations I have seen the 
headers are completely parsed before the request is forwarded.

Regards,
Roland

On 10.07.2014 22:27, Roberto Peon wrote:
> Without the ability to fragment headers on the wire, one must know the 
> size of the entire set of headers before any of it may be transmitted.
>
> This implies that one must encode the entire set of headers before 
> sending if one will ever do transformation of the headers. Encoding 
> the headers in a different HPACK context would count as a 
> transformation, even if none of the headers were modified.
>
> This means that the protocol, if it did not have the ability to 
> fragment, would require increased buffering and increased latency for 
> any proxy by design.
>
> This is not currently true for HTTP/1-- the headers can be 
> sent/received in a streaming fashion, and implementations may, at 
> their option, choose to buffer in order to simplify code.
>
> -=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 10 Jul 2014 23:20:09 +0000

In message <[email-masked]>, Roberto Peon wri
tes:

>[...]

Roberto,

I keep begging you to give me an example of what real-world
applications benefit from being able to fragment headers, and you
keep ignoring my request.

Given that the high-traffic chokepoints for HTTP traffic will benefit
a LOT from knowing the length of the compressed header-set up front,
you need to provide at least an equally compelling reason why being
able to fragment headers brings benefits elsewhere in the HTTP grid.

Where is the beef Roberto ?

Why won't you tell us ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 11 Jul 2014 09:38:21 +1000

Roberto,

I agree that this is a concern, even if large headers are only 0.01% (or
whatever it is ) of the traffic.

But I don't think fragmentation on it's own is sufficient. You need
fragmentation and flow control.       To achieve this we need to stop
treating headers as a special case and forget about any deadlines by the
end of the year. I would propose that:

   - We remove HEADERS, CONTINUATION and PUSH_PROMISE from the specification
   - We retain END_SEGMENT in DATA frames
   - Streams are created by sending a stream SETTINGs frame with a PROTOCOL
   parameter!

We now have a multiplexed framing layer that is totally devoid of any
knowledge of HTTP!   The framing supports segmented data streams that are
flow controlled and of unlimited size.    We then come up with an mapping
of HTTP semantics to this framing layer:

   - HTTP streams start with a SETTINGS frame that has PROTOCOL=h2
   - Odd data segments on the stream carry header/trailers.  So a Stream
   with  1 segment is just headers. A stream with 3 segments is headers, data,
   trailers etc.

Now we have to work out how to encode the headers into those data frames.
The stateless parts of HPACK are a pretty reasonable start, using Static-H
gives a 0.66 compression factor.  However, I think there are probably other
alternatives that are less order dependent - eg sending the header set
mutations only on stream 0 and normal decoding does not mutate the table.
If we wanted to make HTTP a bit of a special case, we could go to Linear-H,
with 0.31 compression factor, but then decoding of the headers must be done
in the order they are sent - making header compression part of the framing
layer.... but I could live with a little bit of conflation for efficiency
purposes:)

With this scheme, we could even support lazy proxies that would send
HTTP/1.1 by having PROTOCOL=h1 on a stream and just sending the http/1
bytes unaltered.   Websocket could be supported the same way or it too
could have it's own segmented data mapping.

Even if we go for a less drastic way to do fragmentation of headers, I
think the process has to be the same - start with data frame semantics and
work out how to transport compressed headers.  Don't come up with a
different fragmentation/flowcontrol regime based on the content of the
frame.

cheers

On 11 July 2014 06:27, Roberto Peon <[email-masked]> wrote:

> This is not currently true for HTTP/1-- the headers can be sent/received
> in a streaming fashion, and implementations may, at their option, choose to

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: James M Snell <[email-masked]>
Date: Thu, 10 Jul 2014 17:12:49 -0700

Roberto: This is true only because of the way HPACK's stateful
compression is currently designed. If we opted for a stateless and
streaming compression mechanism, there would not be an issue and we
would not be having to trudge through all of this yet again. You're
arguing for a solution to a problem caused by your own design.

- James

On Thu, Jul 10, 2014 at 1:27 PM, Roberto Peon <[email-masked]> wrote:
> if one will ever do transformation of the headers. Encoding the headers in a
> different HPACK context would count as a transformation, even if none of the
> headers were modified.
>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Jul 2014 17:35:08 -0700

I've mentioned a few of these in an earlier response.
-=R

On Thu, Jul 10, 2014 at 4:20 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <
> [email-masked]>,
> Roberto Peon wri
> tes:
>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Jul 2014 17:34:37 -0700

On Thu, Jul 10, 2014 at 2:00 PM, Willy Tarreau <[email-masked]> wrote:

> size
> sending
> in
Thanks Willy.

The amount of latency induced depends mainly on size and bandwidth.
Assuming that the header size is 8k, and a link is 1Mb/s (as still happens
often for mobile devices), we're talking about 62ms of buffering before it
can forward, even if it has all of the headers necessary for routing within
the first few ms.
At a 1Gb/s, it is a much more reasonable .06ms.

I'm not particularly worried about the parsing
time/copying/allocation/buffer management time either-- it will need to be
done, and it is going to be measured in us.

As a concrete usecase, think of any entity that needs to forward requests
from a slow link to a fast link.
Additionally, in cases where there are multiple proxies as part of a load
balancing infrastructure, there is no need to examine the full set of
headers on any but one of them.. and for the response headers (of one
trusts the server, which can happen if one is running both proxy and
server), there may never be a need to examine them before forwarding.
-=R

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Jul 2014 17:41:49 -0700

Flow control of headers ends up being problematic even when there is no
compressor, since, on the flip side of this coin, you can't always predict
which headers are necessary for interpretation of a request, and thus you
end up with smart attackers sending all but that last thing for every
stream and affecting a much more effective slowloris attack.

It is especially likely for proxies which are executing DoS or other
security protections to need to do this. Thankfully, in general these are
the only ones that do need to do that.

I like some of the parts of what you proposed there, and don't like other
parts. I'd prefer discussing that in a separate thread, though so we don't
dilute this conversation too much.
-=R

On Thu, Jul 10, 2014 at 4:38 PM, Greg Wilkins <[email-masked]> wrote:

>    - Streams are created by sending a stream SETTINGs frame with a
>    PROTOCOL parameter!
>
>> sending if one will ever do transformation of the headers. Encoding the
>> headers in a different HPACK context would count as a transformation, even
>> if none of the headers were modified.
>>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Jul 2014 17:44:18 -0700

I think that stateful compression only affects point #1, and we should
continue that discussion in another thread.
-=R

On Thu, Jul 10, 2014 at 5:12 PM, James M Snell <[email-masked]> wrote:

> > I'm going to ignore it, as those discussions are occurring elsewhere,
> and in
> > quite some depth :)
> >
> >

---

From: Rob Trace <[email-masked]>
Date: Fri, 11 Jul 2014 00:44:06 +0000

I am very much -1 to starting over.  I would prefer to keep the threads to the issues at hand, there is already too much traffic to read…

Thanks!!

-Rob

From: Greg Wilkins [mailto:[email-masked]]
Sent: Thursday, July 10, 2014 4:38 PM
To: Roberto Peon
Cc: HTTP Working Group
Subject: Re: Fragmentation for headers: why jumbo != continuation.

Roberto,
I agree that this is a concern, even if large headers are only 0.01% (or whatever it is ) of the traffic.
But I don't think fragmentation on it's own is sufficient. You need fragmentation and flow control.       To achieve this we need to stop treating headers as a special case and forget about any deadlines by the end of the year. I would propose that:

  *   We remove HEADERS, CONTINUATION and PUSH_PROMISE from the specification
  *   We retain END_SEGMENT in DATA frames
  *   Streams are created by sending a stream SETTINGs frame with a PROTOCOL parameter!

We now have a multiplexed framing layer that is totally devoid of any knowledge of HTTP!   The framing supports segmented data streams that are flow controlled and of unlimited size.    We then come up with an mapping of HTTP semantics to this framing layer:

  *   HTTP streams start with a SETTINGS frame that has PROTOCOL=h2
  *   Odd data segments on the stream carry header/trailers.  So a Stream with  1 segment is just headers. A stream with 3 segments is headers, data, trailers etc.

Now we have to work out how to encode the headers into those data frames.  The stateless parts of HPACK are a pretty reasonable start, using Static-H gives a 0.66 compression factor.  However, I think there are probably other alternatives that are less order dependent - eg sending the header set mutations only on stream 0 and normal decoding does not mutate the table. If we wanted to make HTTP a bit of a special case, we could go to Linear-H, with 0.31 compression factor, but then decoding of the headers must be done in the order they are sent - making header compression part of the framing layer.... but I could live with a little bit of conflation for efficiency purposes:)

With this scheme, we could even support lazy proxies that would send HTTP/1.1 by having PROTOCOL=h1 on a stream and just sending the http/1 bytes unaltered.   Websocket could be supported the same way or it too could have it's own segmented data mapping.

Even if we go for a less drastic way to do fragmentation of headers, I think the process has to be the same - start with data frame semantics and work out how to transport compressed headers.  Don't come up with a different fragmentation/flowcontrol regime based on the content of the frame.

cheers

On 11 July 2014 06:27, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
There are two separate reasons to fragment headers

-=R

--
Greg Wilkins <[email-masked]<mailto:[email-masked]>>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 11 Jul 2014 10:59:43 +1000

> stream and affecting a much more effective slowloris attack.

This is true.  But if we can break the dependencies between the order that
we decompress headers, then I think both proposals are pretty much the same.

With fragmented but not flow controlled headers, and proxy or endpoint is
still going to have to wait/buffer until the fragment holding the necessary
information arrives.

I think we should not tie ourselves too much in knots over what bad actors
can do... so long as a bad actor can only screw with their own stream and
can't take more resources from an impl that it is prepared to commit.

however, I don't think the will is there to start again from framing layer,
so happy to ignore my suggestion for the purposes of this thread.

cheers

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Jul 2014 18:08:41 -0700

> still going to have to wait/buffer until the fragment holding the necessary

Yup, but there are usecases where 'the necessary information' is the null
set (trusted senders), and others where it is found quite quickly (e.g. if
the client is smart/lucky enough to present the necessary headers).

> however, I don't think the will is there to start again from framing
> layer, so happy to ignore my suggestion for the purposes of this thread.
>

When acting as a server (or a proxy) and listening to the clients on the
untrusted internet, one will experience malicious clients.
The issue with these clients is that the memory (and cpu, etc.) they
consume could be better spent on other, hopefully non-malicious clients to
improve their experience.
A trivial example of this is the server's willingness to keep the
connection open, and thus be able to receive and react to queries without
having the client pay for the connection startup and TLS termination
latency again.
A server's willingness to do this will be related directly to the number of
clients it can handle at that moment (for a well constructed server/proxy).

In other words, if malicious clients can consume more memory (amongst other
resources, but this is the easiest attack vector), it can cause a
degradation in the quality of the service that the server can provide other
clients.
This is the reason why the design intent was to receive one set of headers
at a time, though in a streaming fashion-- it limits the attack surface
while not affecting anything in the common case where headers are small.

-=R

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 11 Jul 2014 07:01:44 +0200

Hi Roberto,

> can forward, even if it has all of the headers necessary for routing within

But here you're speaking about the serialization time on the proxy itself,
it's irrelevant to the slow bandwidth, because I'm assuming you're not
connecting your proxies using an 1 Mbps arcnet link :-)

> from a slow link to a fast link.

I'm still failing to see the use case I'm afraid, because what will be
added in this case is the time needed to serialize over the fast link.
The problematic case would be if you insert multiple proxies over a
slow link, which really does not make much sense.

> server), there may never be a need to examine them before forwarding.

It depends, because if you stack multiple proxies, each of them will have
a different usage. One may be a cache, another one may be used to perform
anti-virus processing etc... Also, we're speaking about devices where the
added time is measured in microseconds and is not noticeable by the client
(the request serialization time is lower than the TCP connection time to
the next hop).

For example, here at home a request to youtube, with all the fat cookies
tracking my activity is 669 bytes. That's 21 ms over my 256kB ADSL line.
When it reaches your gateways, it will take an extra 5 microseconds if
they are connected using GigE, or 500 nanoseconds if connected over 10GE.
The round-trip time when the link is empty is 25 milliseconds in comparison,
so that completely dominates the serialization time on all your proxies.

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Jul 2014 15:32:40 +1000

Hey Willy,

Roberto isn't making a latency argument -- it's an argument for minimal buffering in the optimistic case(s).

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 11 Jul 2014 07:37:49 +0200

> buffering in the optimistic case(s).

But that was based on the fear of adding latency on slow links if the
headers are buffered.

Cheers,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 11 Jul 2014 15:50:37 +1000

> buffering in the optimistic case(s).

Servers have to hold the headers for the entire request handling duration
anyway (unless you throw away all the general purpose servers that we
currently have), so it makes no difference to their buffering if the
headers arrive in 1 or many fragments.

Proxies cannot forward headers in fragments because:

   1. the routing information may be in the last fragment.
   2. because encoding mutates the ref set, a proxy cannot commence sending
   headers until it has all fragments, else it risks blocking the whole
   connection.

Getting rid of the ReferenceSet may fix 2 and help with 1 in special
cases.

But I maintain that if we really want to fragment headers, then put them in
a segment of DATA frames rather than invent a second mechanism.
cheers

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Jul 2014 15:57:20 +1000

On 11 Jul 2014, at 3:50 pm, Greg Wilkins <[email-masked]> wrote:

> Servers have to hold the headers for the entire request handling duration anyway (unless you throw away all the general purpose servers that we currently have), so it makes no difference to their buffering if the headers arrive in 1 or many fragments.
> 
> 	• the routing information may be in the last fragment.
> 	• because encoding mutates the ref set, a proxy cannot commence sending headers until it has all fragments, else it risks blocking the whole connection.
> Getting rid of the ReferenceSet may fix 2 and help with 1 in special cases.  

Roberto appears to be arguing that just because the routing information might be in the last fragment, it isn't always; that's pessimistic. If the necessary information is in the first, it can be acted upon and then forwarded, reducing the buffers necessary to serve this more optimistic case.

Likewise, on the response side, an intermediary can pass receive and then send fragments of headers, rather than buffering them.

I'm not taking a position here, BTW -- just trying to clarify the discussion. It also appears to be the crux of the argument regarding getting rid of or keeping CONTINUATION...

> But I maintain that if we really want to fragment headers, then put them in a segment of DATA frames rather than invent a second mechanism.

Greg, you appear to be alone here...

Cheers,

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 11 Jul 2014 22:05:56 +1200

On 11/07/2014 5:57 p.m., Mark Nottingham wrote:
>> Roberto isn't making a latency argument -- it's an argument for
>> minimal buffering in the optimistic case(s).
>> 
>> 
>> Servers have to hold the headers for the entire request handling
>> duration anyway (unless you throw away all the general purpose
>> servers that we currently have), so it makes no difference to their
>> buffering if the headers arrive in 1 or many fragments.
>> 
>> Proxies cannot forward headers in fragments because: • the routing
>> information may be in the last fragment. • because encoding mutates
>> the ref set, a proxy cannot commence sending headers until it has
>> all fragments, else it risks blocking the whole connection. Getting
>> rid of the ReferenceSet may fix 2 and help with 1 in special cases.
>> 
> 
> Roberto appears to be arguing that just because the routing
> information might be in the last fragment, it isn't always; that's
> pessimistic. If the necessary information is in the first, it can be
> acted upon and then forwarded, reducing the buffers necessary to
> serve this more optimistic case.

I believe its optimistic to expect malicious coders will write the
routing information in the last fragment if they find you have coded to
assume it is in the first one(s). Even if the spec says only to send in
the first fragment.

> 
> Likewise, on the response side, an intermediary can pass receive and
> then send fragments of headers, rather than buffering them.

*Unless* the connection is in use by any other client streams. At least
given the current draft definition of how CONTINUATION MUST NOT be
multiplexed. If it does so then it is HOL blocking the connection until
the last fragment/CONTINUATION is delivered.

> 
> I'm not taking a position here, BTW -- just trying to clarify the
> discussion. It also appears to be the crux of the argument regarding
> getting rid of or keeping CONTINUATION...

Yes. Multiplex meet CONTINUATION. Bleh.

> 
>> But I maintain that if we really want to fragment headers, then put
>> them in a segment of DATA frames rather than invent a second
>> mechanism.
> 

I agree with him on this. There is a very big "IF" at the start of the
sentence. That is the only way mentioned so far that avoids the issues
highlighted against multiplexed CONTINUATION and without using large frames.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Jul 2014 20:56:21 +1000

> assume it is in the first one(s). Even if the spec says only to send in

Not following you here...

> the last fragment/CONTINUATION is delivered.

Yes, but we’re discussing ways around that...

> Yes. Multiplex meet CONTINUATION. Bleh.

See other comments today about “bleh” arguments :)

> highlighted against multiplexed CONTINUATION and without using large frames.

Greg may not be so alone any more; see Jeff’s thread.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 11 Jul 2014 23:40:54 +1200

On 11/07/2014 10:56 p.m., Mark Nottingham wrote:
> 
> On 11 Jul 2014, at 8:05 pm, Amos Jeffries wrote:
>>>

You said it was pessimistic to expect attack. I think coding mistakes
and then attacks based on them are guaranteed to happen if the
fragmentation decision is retained.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 11 Jul 2014 23:53:22 +1200

> I wanted to be sure we were also thinking about #2.

We are. Or at least the people behind large frames proposal are.

> would require increased buffering and increased latency for any proxy by

Quite the opposite IMHO. Others have mentioned already in a roundabout
way. But here is the nutshell:

Header fragmentation (in forms similar to CONTINUATION) only reduces
buffering on the *sender*.

Proxies are a *recipient* in both traffic directions and therefore
always hit with buffering the entire sequence of fragments before they
can guarantee the set is safe to re-transmit down any shared
connections. So arguing from a proxy perspective it's 6 one way or half
a dozen the other.

The part which makes the real difference to proxies and servers, is that
receiving fragments (buffer on receiver) takes unlimited time with no
end in sight until its reached - whereas sending large frames (buffer on
sender) gives a clear size ending position up front. Which as PHK said
in earlier mails is critical data for recipients dealing with an attack
(or broken sender).

Given the size of a large frame up-front a proxy can actually stream the
start of that frame before the end has arrived if you trust the sender
will complete it cleanly.

This issue is already well-known from IPv4 protocol design and packet
routing issues that happen as a result. IIUIC the outcome of experience
with that IPv4 design was the decision to make IPv6 packets
non-fragmentable.

> buffer in order to simplify code.

Willy T already answered this one so I shall avoid repeating the same
arguments.

Amos

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 12 Jul 2014 11:43:47 +1000

Roberto,

[offlist]

I'm curious as to what parts of this thought bubble you like and what bits
you don't?

I think the current inability to reach consensus is due to some earlier
poor design decisions that forced the WG down a path that nobody is really
happy with and I fear that the current disputes are really just fighting
over where to put the deck chairs on the titanic.

I basically agree with you that headers should be fragmentable and
interleavable( I also think they should be flow controllable ), however
that requires changes to HPACK, which I did not think the WG was willing to
make, hence my proposal for a single frame header (the better of two evils
with regards to continuations).

However, if the current stalemate and additional concerns you have raised
do inspire a new interest in revisiting HPACK, then perhaps there might
come an opportunity for something a bit more radical and  I might consider
putting some effort into preparing a real proposal along the lines of this
thought bubble.

Hence I'm interested in what you did/didn't like about the idea of mapping
HTTP semantics onto data frames?

Fundamentally I would like the code that has to deal with all the framing
concerns: buffering, fragmentation, interleaving, priority, aggregation,
flow control, etc.   to be application protocol neutral.    It should not
know or care if it is transporting headers or data or websockets etc.
All it need to do is give enough indication of where the semantics are so
that actors that do need to know some more semantics (eg proxy routing
algorithm) know where in the stream to look to apply higher level
decoding.  I think stream segments are sufficient for that.

Anyway,  if you have any additional personal bandwidth,  I'd be interested
in your thoughts.

I don't think the WG is actually ready/interested in such a proposal at the
moment (hence offlist), but who knows what the future may hold.

cheers

>> Now we have to work out how to encode the headers into those data
>> frames.  The stateless parts of HPACK are a pretty reasonable start, using
>> Static-H gives a 0.66 compression factor.  However, I think there are
>> probably other alternatives that are less order dependent - eg sending the
>> header set mutations only on stream 0 and normal decoding does not mutate
>> the table. If we wanted to make HTTP a bit of a special case, we could go
>> to Linear-H, with 0.31 compression factor, but then decoding of the headers
>> must be done in the order they are sent - making header compression part of
>> the framing layer.... but I could live with a little bit of conflation for
>> efficiency purposes:)
>>
>>> and in quite some depth :)
>>>
>>>
>>> fragment, would require increased buffering and increased latency for any
>>> proxy by design.
>>>

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 12 Jul 2014 12:04:21 +1000

Doh!  Not offlist.  Apologies for predictions of sinking titanic -
hopefully we will land a compromise. But I think I've expressed pretty much
all of that publicly before anyway.

As I said, I don't think I can see any will in the WG for such a radical
change, so if there is any interest in discussing this, please respond
privately as I think there is enough bandwidth consumed here already.

regards

On 12 July 2014 11:43, Greg Wilkins <[email-masked]> wrote:

> I don't think the WG is actually ready/interested in such a proposal at
> the moment (hence offlist), but who knows what the future may hold.
>
> cheers
>
>
>
>
>
>>
>>>    - Odd data segments on the stream carry header/trailers.  So a
>>>    Stream with  1 segment is just headers. A stream with 3 segments is
>>>    headers, data, trailers etc.
>>>
>>>> sent/received in a streaming fashion, and implementations may, at their
>>>> option, choose to buffer in order to simplify code.
>>>>
>>>> -=R
>>>>
>>>
>>>
>>>
>>> --

---

From: Roberto Peon <[email-masked]>
Date: Fri, 11 Jul 2014 19:11:58 -0700

I like the larger frame max-length so long as the max-framesize setting is
there.
imho, 64k is big enough given that MTU is likely to stay below 64k for the
forseeable future (~9k is the current jumbo frame!).
When we venture into things like QUIC, smaller frame sizes will become
important, as you really want to fit frames into packets-- that way the
encryption context doesn't require HoL blocking when there is packet loss,
and things like FEC can work.

I don't like things that require buffering at the protocol level-- that
should be an implementation decision.
I think that people are underestimating the value that announcing limits
gives to attackers.
It is often better to accept a request/header that you know you won't serve
and just sit on it for a while than it is to reject it quickly.
Rejecting quickly and announcing limits allows the attacker to easily
optimize their attack. Not good.

I don't like interleaving-- it multiplicatively increases the DoS surface
(and makes it significantly worse than it was with HTTP/1)

I do wish that there was a more clear separation of session-layer and HTTP,
but I wasn't able to win that argument way back when.. :)

I think that continuations are ugly in terms of flag handling, but are
otherwise not too difficult to handle.
The state machine essentially lends itself to:
see HEADERS, stay in HEADERS state until you seen END_HEADERS on a frame.

-=R

---

From: Roberto Peon <[email-masked]>
Date: Fri, 11 Jul 2014 19:12:45 -0700

And yes I realize this is on-list, but this is how I see it, and I think it
should be clear already :)
-=R

On Fri, Jul 11, 2014 at 7:11 PM, Roberto Peon <[email-masked]> wrote:

> It is often better to accept a request/header that you know you won't
> serve and just sit on it for a while than it is to reject it quickly.
> I do wish that there was a more clear separation of session-layer and
> HTTP, but I wasn't able to win that argument way back when.. :)
>
>>> I like some of the parts of what you proposed there, and don't like
>>> other parts. I'd prefer discussing that in a separate thread, though so we
>>> don't dilute this conversation too much.
>>> -=R
>>>
>>
>> Roberto,
>>
>> [offlist]
>>
>> I'm curious as to what parts of this thought bubble you like and what
>> bits you don't?
>>
>> Hence I'm interested in what you did/didn't like about the idea of
>> mapping HTTP semantics onto data frames?
>>
>> Anyway,  if you have any additional personal bandwidth,  I'd be
>> interested in your thoughts.
>>

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 12 Jul 2014 12:43:25 +1000

> (and makes it significantly worse than it was with HTTP/1)

Ah interesting.     I don't think I had understood this objection before -
as I thought that the desire to fragment was driven by the desire to
interleave for QoS. Hence the push to drop the reference set.    But in
this case you want to fragment just to avoid buffering in the sender.

I don't  agree with the DOS part, but I can accept the buffering concern.
However I don't think that on it's own it can motivate having a dedicated
header fragmentation mechanism.   But if headers were able to be fragmented
for other reasons, then I think your concern is a good one and contrary the
proposal to allow multiple header frames only if the previous ones are full.

I also now get that you like max-framesize setting, but not a max-header
size setting.

Not sure this understanding gets us any closer to consensus, but good to
understand anyway.

cheers

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 16:55:54 +1200

On 12/07/2014 2:43 p.m., Greg Wilkins wrote:
> On 12 July 2014 12:11, Roberto Peon wrote:
> 

I think a critical detail that needs to be acknowledged by everyone
right now is that the DoS risk is from the attacker in role of sender.

Decisions which reduce the costs for senders by shifting them to
recipients actually increases the DoS vulnerability of the whole system.

Roberto, that tradeoff is a key detail within the Greg et all proposal.
Hence by requiring low sender/high recipient costs you are actually
arguing for increasing the DoS vulnerability in h2.

Amos

---

From: Roberto Peon <[email-masked]>
Date: Fri, 11 Jul 2014 22:24:23 -0700

In the case of a server, it must interpret. If it can do so in a streaming
fashion, it can do so in a streaming fashion. If not, it must buffer until
it has the entire header set. That is true of fragmented and non-fragmented
headers.
In other words, the requirement to buffer or not is unchanged and is
implementation dependent.

For a proxy, it is a sender and a receiver.
Allowing fragmentation allows the sender-half of the proxy to reduce its
memory commitment.
Again, nothing changed on the receiver side, which implies that the proxy's
memory commitment is reduced.

In other words, fragmented headers leads to equivalence at servers, and a
possible improvement at proxies.
-=R

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 12 Jul 2014 16:33:49 +1000

On 12 July 2014 15:24, Roberto Peon <[email-masked]> wrote:

> Again, nothing changed on the receiver side, which implies that the
> proxy's memory commitment is reduced.
>

I don't see this sorry.

We have already accepted that a proxy must buffer up the entire header
before starting to send it, otherwise it allows a whole connection to be
easily blocked (this is the "just don't do it" point Martin has made many
times).

So once the proxy has the entire header in it's memory then it can
definitely send it with low buffering.  It may need to do two passes over
the header, one to calculate size and another the do the actual sending,
but it is certainly doable.      So it's just an implementation detail if
the proxy wishes to trade memory for latency.

If you really want streamable headers, then the only way to achieve that is
to allow them to be fragmented just as data frames are, so a proxy can
receive/decode/update/encode/send on the fly.    HPACK would obviously need
to be updated and fragmented headers would need to be able to be
interleaved.  I'm +1000000 for going this route!

cheers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 12 Jul 2014 07:59:59 +0000

In message <[email-masked]>
, Roberto Peon writes:
>--089e0158ac8652c7fa04fdf84520
>Content-Type: text/plain; charset=UTF-8
>

I belive we have consensus to put all the :xxx headers first.

This means that the moment a receiver sees the first non-colon
header it can trust the :xxx headers.

But no other header can be trusted until the full header-set has
been received, since they can be revised (or in the case of
Content-Length: invalidate the entire request)

In other words your "streaming fashion" is hard limited to the URL.

I have a hard time seeing why anybody would ever send CONTINUATION
frames if all the server cares about is the URL ?

I have an even harder time seeing that we should bend over backwards
and inconvenience all the high-traffic implementations for their
ability to do so.

In other words:  It (still) doesn't make any sense Roberto.

>memory commitment is reduced.

Try as I might, have never found a compliant HTTP proxy which would forward
a request without receiving the full header-set first.

Do you know of any ?  If so, please share.

I know of hackish loadbalancers which operate at IP packet level, and
which expects that the information they require for load-balancing fit
inside a single ethernet frame, but given that no HTTP standard ever
talked about ethernet frames, I think we can agree that they are not
anywhere near being compliant.

So please tell us more about these mythological "header-streaming"
proxies you keep talking about, nobody else seems to have ever seen
or heard about them.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 12 Jul 2014 10:19:56 +0200

Poul-Henning,

> anywhere near being compliant.

[ One thing I like with HTTP/2 is that the crappy devices above will
  be forced to disappear and suddenly the web will start to work much
  better ]

> proxies you keep talking about, nobody else seems to have ever seen

Roberto already explained his specific use, and I understood that the
LB didn't need to inspect the response since it trusted the servers, and
that he wanted to stream it from the servers to improve latency. That was
the reason why we diverged on the subject of latency because while I full
respect this usage model (HTTP is for everyone). In this specific case, I
expected the extra latency ought to be extremely small (let even measurable)
thanks to the high speed links between the LB and the servers, and probably
not worth maintaining the extra complexity in the protocol.

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 12 Jul 2014 21:35:12 +1200

> implementation dependent.

This does not follow. The requirement to buffer is in the specification.
Fragmented HEADERS+CONTINUTATION MUST be handled as a contiguous HOL
blocking squence.
  There is no server choice in that.

Likewise connections from proxies are multiplexing many clients streams
simultaneously as per the specification.
 There is no server choice in that.

Yet the server "must" have choice whether to HOL-block the entire
connection by streaming its responses into a buffer *on the proxy* ?
that is a connection just begging to be stalled.

> Allowing fragmentation allows the sender-half of the proxy to reduce its

False. There is no sender side specific buffering state in proxies.
*Unless* fragmentation requires an area to generate inter-fragment
overheads in. Ergo, there is a net *increase* of sender requirements for
proxies with fragmentation.

> memory commitment is reduced.

False. Receiving fragmentation requires HOL-blocking of the connection
and buffering the headers multiple-fragment entirety.

Ergo, for proxies the receiver side has a net increase on both sender
and receiver sides when handling fragmentation.

> possible improvement at proxies.

Conclusion based on corrected arguments is quite the opposite:
 - Fragmentation offers server as sender (or client as sender too)
ability to freely cause HOL-block or stall/DoS the connection with
multiple end-clients suffering.
 - Fragmentation causes state commitment increases in proxies.

Amos

---

From: David Krauss <[email-masked]>
Date: Sun, 13 Jul 2014 17:56:57 +0800

> - Fragmentation causes state commitment increases in proxies.

+1. I reached the same conclusions and it was the last straw that caused me to suspend my implementation effort.

Header streaming does not practically reduce state commitment for a proxy which coalesces connections. Coalescing is one of the most promising aspects of HTTP/2 for me. It would seem to be a must for reverse proxies, but yet it also seems impossible to accomplish according to the spec.

I don’t gather that multi-session connections have been done before. So is it outside the charter or is it recommended practice? Has anyone even tried it yet?

Looking back at the archives, I alluded to the technical side of the problem in messages sent 21 Apr 2014 20:17, 21 May 2014 09:01, 26 May 2014 17:27 (PDT) but for whatever reason never started a thread to address my perception or intended application directly.

---

From: Johnny Graettinger <[email-masked]>
Date: Sun, 13 Jul 2014 13:06:46 -0400

> will complete it cleanly.

The proxy cannot know the encoded size of the block until it's run that
complete block through it's send-side HPACK context. That context is
different than the recv-side context, and it will almost certainly result
in a different block length.

Announcing the length of an encoded header-block is incompatible with
streaming.

---

From: Johnny Graettinger <[email-masked]>
Date: Sun, 13 Jul 2014 13:15:36 -0400

> Header streaming does not practically reduce state commitment for a proxy
> which coalesces connections. Coalescing is one of the most promising
> aspects of HTTP/2 for me. It would seem to be a must for reverse proxies,
> but yet it also seems impossible to accomplish according to the spec.
>

A typical reverse proxy will want to buffer full header sets on the request
path to prevent HOL blocking on coalesced connections to backends. However
it could reasonably stream fragments on the response path, from trusted
backends.

As such a proxy now doesn't need to commit to buffering full response
header sets (they're processed incrementally), it has reduced it's state
commitment.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 13 Jul 2014 18:14:50 +0000

In message <CAEn92TrULC1rniqaG-vFYw9K77d2g97=ANywW3x5UyPfPSR=[email-masked]>, Johnny Graetting
er writes:
>--20cf3079bc781d287e04fe16336c

actually, if there exists such a intimate relationship between the
backend and the proxy, the backend can compress the headers to make
that possibl.

---

From: Roberto Peon <[email-masked]>
Date: Mon, 14 Jul 2014 02:00:53 -0700

Unfortunately not true-- the backend can't have the same compressor state
as the proxy unless the proxy is doing a 1:1 connection mapping (which only
happens sometimes).
If the proxy is handling 2 FE connections and 1 BE connection, or 1 FE
connection and 2 BE connections, there can be no safe delegation of
compression to the backend, since it cannot know the state of the
compression context at the proxy.

If you need to know the size of the headers, your function will look
something like:

# Non-streaming forwarding
  header_frame = read_header_frame()
  uncompressed_headers = decompress(compressed_headers)
  outgoing_compressed_headers = compress(uncompressed_headers)
  send_header_frame(outgoing_compressed_headers)

If you don't need to know the size of the headers, you can instead do:

# Streaming forwarding. ~ 6 LoC more.
header_payload = []
while True:
  header_frame = read_header_frame()
  header_payload.append(header_frame.payload())
  partial_uncompressed_output, amount_consumed = decompress(header_payload)
  header_payload = header_payload[amount_consumed:]  # state is freed up
here while sending
  compressed_outgoing = compress(partial_uncompressed_output)
  send_header_frame(compressed_outgoing)
  if header_frame.has_end_headers():
    break

Note that in the case above, neither the full set of incoming headers, nor
the full set of outgoing headers needs to be resident in memory before
forwarding.
A decision to forward like above does involve a tradeoff between lessening
memory commitment (and latency) and the possibility of HOL blocking. This
should be a choice that an implementation makes. Some implementations will
know that the connection between the backend and a proxy is always faster
than the connection from the user to the proxy. In such cases, HOL blocking
will almost never occur, and streaming the headers to the client wins both
latency and memory reductions at almost no cost.

-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 14 Jul 2014 09:03:51 +0000

In message <CAP+FsNesMqqDa+QyV_c_sg8fud=DPi0PqxGG=[email-masked]>, Roberto Peon wri

>as the proxy unless the proxy is doing a 1:1 connection mapping (which only

If the backend compresses without relying on the state it can (ie: static
set and huffman all the way)

It obviously gives inferior compression, but it will work.

---

From: Roberto Peon <[email-masked]>
Date: Mon, 14 Jul 2014 02:05:36 -0700

True, though not likely useful.
-=R

On Mon, Jul 14, 2014 at 2:03 AM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <CAP+FsNesMqqDa+QyV_c_sg8fud=DPi0PqxGG=
> [email-masked]>, Roberto Peon wri
> tes:
>

---

From: <[email-masked]>
Date: Mon, 14 Jul 2014 09:19:24 +0000

On Monday,14 July 2014 11:06 [email-masked] wrote:
> On Mon, Jul 14, 2014 at 2:03 AM, Poul-Henning Kamp <[email-masked]> wrote:
>>

Perhaps not useful to Google.

But I think Static+Huff will be an attractive option to minimal implementations that don't want to worry about shared state.

Furthermore, my understanding has been that the primary purpose of header compression was for clients to stuff as many requests as possible into the initial TCP window.  Header compression on replies is likely to be in the noise.

-keith