Subject: HEADERS and flow control

From: Roberto Peon <[email-masked]>
Date: Fri, 9 May 2014 11:42:18 -0700

Currently HEADERS are not subject to flow control. There are a couple of
reasons for this, but the biggest one is that this helps to prevent a
deadlock which can occur when a server is waiting for a particular request
before it can make progress on other responses, but flow control is
asserted.

One of the potential users of HTTP2 whom I've spoken with wished strongly
that HEADERS after the first complete HEADERS block on a stream would be
subject to flow control. AFAICT, this would work better than what we have
today, without potentially triggering the deadlock scenario for many
non-browser usecases.

Thoughts?

-=R

---

From: Martin Thomson <[email-masked]>
Date: Fri, 9 May 2014 12:02:12 -0700

> today, without potentially triggering the deadlock scenario for many

/me needs more information.

This is unlikely to affect most use cases, but it does create another
potentially surprising corner case.  I think that I need (much)
stronger justification.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 9 May 2014 12:10:45 -0700

What creates a surprising corner case?

Lets put it another way.
HEADERS which establish streams are control frames.
HEADERS which are in the middle of a stream, however, despite being control
frames are NOT doing any controlling. They are payload.

-=R

---

From: Martin Thomson <[email-masked]>
Date: Fri, 9 May 2014 12:34:49 -0700

> What creates a surprising corner case?

What you are proposing:

function isFlowControlled(frame) {
  return frame.type === 'DATA';
}

becomes:

var firstHeadersDone = false;
function isFlowControlled(frame) {
  if (frame.type === 'DATA') {
    return true;
  }
  if (!firstHeadersDone) {
    if (frame.type === 'HEADERS' || frame.type === 'CONTINUATION') {
      firstHeadersDone = frame.flags.END_HEADERS;
    }
    return false;
  }
  return frame.type === 'HEADERS' || frame.type === 'CONTINUATION';
}

Or something like that.  That's not particularly straightforward.
Such complexity requires justification.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 9 May 2014 12:37:32 -0700

Or you can change the opcode of the other HEADERS...

What I'm proposing is effectively a rethink of the merging of HEADERs and
SYN_STREAM, because users have brought up usecases which would be
troublesome with the way it is done now.
-=R

---

From: James M Snell <[email-masked]>
Date: Fri, 9 May 2014 13:06:31 -0700

This is really the same issue that I had brought up quite a while
ago... in the context of hop-by-hop or end-to-end frame types, where
end-to-end frames were always considered to be subject to flow
control. While I contend that it still makes far more sense to define
it as a property of the opcode, having a flow-control headers frame
variant could work also.

---

From: Hasan Khalil <[email-masked]>
Date: Fri, 09 May 2014 20:21:20 +0000

+1. This sounds like yet another reason that we want a separate opcode for
HEADERS that begin a new stream and those that do not.

Nowhere in this message was SYN_STREAM mentioned. Wait...

On Fri May 09 2014 at 4:09:37 PM, James M Snell <[email-masked]> wrote:

> > On Fri, May 9, 2014 at 12:34 PM, Martin Thomson <
> [email-masked]>
> > wrote:
> >>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 9 May 2014 13:47:24 -0700

/me still needs more information.

This is a change.  The change needs greater justification than "it
might be nice".  So far, that's all I've heard.

I actually think that this is nice.  But nice doesn't cut it for me.

Given the likelihood that header blocks after the first will be used,
this is just another corner case.  If we use HTTP/1.1 as a guide, the
best analogy there is to chunk extensions and trailers, i.e.,
basically zero use.

Is the intent to flow control PUSH_PROMISE too?

(In case you haven't have noticed, I want to finish up here.)

On 9 May 2014 13:21, Hasan Khalil <[email-masked]> wrote:
>> > On Fri, May 9, 2014 at 12:34 PM, Martin Thomson
>> > <[email-masked]>
>> > wrote:
>> >>

---

From: Roberto Peon <[email-masked]>
Date: Fri, 9 May 2014 15:18:20 -0700

> might be nice".  So far, that's all I've heard.

I have non-browser customers who are looking at HTTP2 as it stands right
now and are pointing out that this will become problematic if HEADERS is
actually used as metadata (as it was intended to be) as opposed to simply
to create streams. There is no guarantee that the metadata is going to be
small. They're mostly willing to swallow the idea that a large block of
metadata would (effectively) pause multiplexing for a short time, but
worried about the fact that this large metadata actually could account for
a fair bit of buffer, and that the product that HTTP2 (etc) would replace
did have an operational problem with the lack of this in the past.

Hmm.. in scanning the document I think we don't have the requirement in
there that HEADERS and DATA have a sequence relationship that must be
maintained. We've spoken about it numberous times, but I think we
overlooked getting that in there.

> Is the intent to flow control PUSH_PROMISE too?

No-- PUSH_PROMISE is necessary for stream creation. Flow controlling it
might engender protocol-induced deadlock.

> (In case you haven't have noticed, I want to finish up here.)

I know. I'll point out I haven't been making trouble unless I have people
actually wanting to use the HTTP2 for stuff!
Better, the folks who are complaining about this to me are creating
implementations now, and as far as I can tell, no-one else out there is.
That implies that this change will not slow down interop.
-=R

---

From: Johnny Graettinger <[email-masked]>
Date: Fri, 9 May 2014 18:55:19 -0400

How would the interaction of stream flow control and continuations be
managed? A stalled stream may also stall the connection, including control
frames. Could that also lead to deadlock?

---

From: Roberto Peon <[email-masked]>
Date: Fri, 9 May 2014 17:02:12 -0700

Assuming BLOCKED is there, it works to write it when it would all fit,
though it is ugly.

Lets assume this was being used for something RPC like (e.g. websockets+).
Here is an exploration in more detail.

The alternativesare:
New opcode (e.g. METADATA), flow-controlled.
  (a) uses HPACK format, but allowed zero state
  (b) HPACK context per stream, only interpreted at the eventual endpoint--
proxies do not need to examine this state.
  (c) HPACK context per stream, SETTINGS on how much space is allowed
globally for these

Only write metadata when all of it will fit.
  (d) when blocked send BLOCKED frame
  (e) when blocked stall that stream indefinitely until enough space is
available

(f) Ignore flow control/frame doesn't get flow controlled
(g) Add a new, different setting which sets an upper bound on the total
amount of METADATA allowed to be in flight at once, and require
METADATA_ACK (or equivalent)
(h) Tell people to put each RPC onto its own stream
(i) Tell people to put each RPC onto its own stream, and express dependency
using the prioritization scheme. *Require* relative dependencies to be
expressed through proxies.
(j) Add a new flow control token (e.g. METADATA_WINDOW_UPDATE), which
allows for metadata to be sent. Define a setting for this (potentially).

As a reminder, the use-case here is something RPC and/or WebSocket like,
where we have streams of messages where order may matter.

Option (a) never deadlocks, but it allows no compression.
Option (b) never deadlocks, and allows compression, but requires an
unfortunate amount of memory of a server which wishes to interpret this
metadata.
Option (c) never deadlocks, allows compression, but requires allocations of
more memory at proxies and/or servers, and adds more complexity to
negotiations.
Option (d) never deadlocks, allows compression, but may stall a connection
for some indeterminate period of time.
Option (e) can deadlock, allows compression,
Option (f) never deadlocks but requires potentially infinite amount of
memory consumption in a proxy when a non-malicious actor wishing to make a
bunch of RPCs.
Option (g) never deadlocks, allows compression without increasing memory
footprint, but requires acknowledgement of the receipt of HEADERS/METADATA
from the remote side, which increases chattiness and complexity.
Option (h) never deadlocks, allows compression, but doesn't allow for an
expression of sequencing
Option (i) never deadlocks, allows compression, uses little memory, but
requires more state tracking and makes prioritization a requirement. Ick.
Option (j) never deadlocks, allows compression, uses little memory,
requires tracking one additional counter, slightly increases chattiness,
but arguably little since it would be tacked onto WINDOWS_UPDATE flow
control updates (even if not the same frame).

Option(f) is what would happen today. That seems problematic, and would
make the use of HTTP2 as the basis of an RPC system difficult .

Other options?
-=R

---

From: James M Snell <[email-masked]>
Date: Fri, 9 May 2014 17:10:53 -0700

Just as a reference point:
https://datatracker.ietf.org/doc/draft-snell-httpbis-ext-frames/?include_text=1

- James
On May 9, 2014 5:02 PM, "Roberto Peon" <[email-masked]> wrote:

>   (b) HPACK context per stream, only interpreted at the eventual
> endpoint-- proxies do not need to examine this state.
> (i) Tell people to put each RPC onto its own stream, and express
> dependency using the prioritization scheme. *Require* relative dependencies
> to be expressed through proxies.
> Option (c) never deadlocks, allows compression, but requires allocations
> of more memory at proxies and/or servers, and adds more complexity to
> negotiations.
>>> On Fri, May 9, 2014 at 1:47 PM, Martin Thomson <[email-masked]
>>> > wrote:
>>>
>>> I know. I'll point out I haven't been making trouble unless I have
>>> people actually wanting to use the HTTP2 for stuff!

---

From: James M Snell <[email-masked]>
Date: Fri, 9 May 2014 17:13:19 -0700

And this: http://tools.ietf.org/html/draft-snell-httpbis-ext-frames-00
On May 9, 2014 5:10 PM, "James M Snell" <[email-masked]> wrote:

>> Option (d) never deadlocks, allows compression, but may stall a
>> connection for some indeterminate period of time.
>>>> On Fri, May 9, 2014 at 1:47 PM, Martin Thomson <
>>>> [email-masked]> wrote:
>>>>
>>>> I have non-browser customers who are looking at HTTP2 as it stands
>>>> right now and are pointing out that this will become problematic if HEADERS
>>>> is actually used as metadata (as it was intended to be) as opposed to
>>>> simply to create streams. There is no guarantee that the metadata is going
>>>> to be small. They're mostly willing to swallow the idea that a large block
>>>> of metadata would (effectively) pause multiplexing for a short time, but

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Sat, 10 May 2014 10:34:17 +0900

In RPC use case, client/server can do whatever they want in DATA frames and
any metadata can be included there, which solves ordering problems as
well.  So I am under impression that they don't have to use HEADERS for
this and we don't have to introduce yet another complexity.

Best regards,
Tatsuhiro Tsujikawa

---

From: Roberto Peon <[email-masked]>
Date: Fri, 9 May 2014 18:53:47 -0700

The intent is that HTTP2 loadbalancers and other L7 things be able to deal
with things like WebSockets and RPC and HTTP without needing to know the
internals of the protocols.
-=R

On Fri, May 9, 2014 at 6:34 PM, Tatsuhiro Tsujikawa
<[email-masked]>wrote:

> In RPC use case, client/server can do whatever they want in DATA frames
> and any metadata can be included there, which solves ordering problems as

---

From: Martin Thomson <[email-masked]>
Date: Mon, 12 May 2014 09:57:01 -0700

I've been thinking about this over the weekend and I remain unmoved by
this thread.  I think that there's a kernel of something here, but I
remain unconvinced that this is something that we need to do anything
about this.

Basically, it's not HTTP.

On 9 May 2014 17:02, Roberto Peon <[email-masked]> wrote:
> an expression of sequencing

I think that this is key.  RPC protocols often depend on some sort of
ordering semantic in order to get decent throughput.  That and layer
upon layer of metadata.  The protocol Roberto looks a little like
HTTP, maybe even to the point of being a changeling [1].  I think that
we need to discuss to what extent we want to support changelings.

The alternative is that Roberto's unnamed customers need to think
about doing option (h) and put every RPC call on its own stream, using
header fields or some other mechanism to express dependencies [2].
And yes, I'm aware that this isn't the only externality in play.

--Martin

[1] https://en.wikipedia.org/wiki/Changeling
[2] Of course that this will cause some intermediaries to have
non-standard hacks in them to support backends that rely on getting
dependent streams at the same backend instance.  And that sucks, but I
believe that to be the de facto state of these sorts of intermediary
anyway.

---

From: Roberto Peon <[email-masked]>
Date: Mon, 12 May 2014 10:32:51 -0700

We are unable to express something in http2 that we previously could with
http/1 pipelining/and/or http/1.1 chunking.

That is not satisfactory.

-=R

---

From: James M Snell <[email-masked]>
Date: Mon, 12 May 2014 11:15:44 -0700

Honestly, if we had properly dealt with this a year ago like I had
suggested it would be a non-issue today. I certainly sympathize with
the use case, but given the state of things as they stand now, I'm not
convinced that it's worthwhile trying to half-heartedly jam this back
in now.

---

From: Hasan Khalil <[email-masked]>
Date: Mon, 12 May 2014 18:26:16 +0000

Care to explain which part of this would be half-hearted?

---

From: James M Snell <[email-masked]>
Date: Mon, 12 May 2014 11:32:42 -0700

half-hearted = shoehorning in one additional flow-controlled headers
frame opcode in order to meet one specific use case example as opposed
to providing a proper extensibility model.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 19 May 2014 17:41:48 +1000

Roberto,

I’ve opened <https://github.com/http2/http2-spec/issues/480> to track this.

We’ll discuss this in NYC (although please do continue discussing on-list). 

Be warned that a large amount of the discussion around this will be “Is this worth holding HTTP/2 for another implementation draft / implementation / interop cycle?”

Cheers,

On 10 May 2014, at 4:42 am, Roberto Peon <[email-masked]> wrote:

> Currently HEADERS are not subject to flow control. There are a couple of reasons for this, but the biggest one is that this helps to prevent a deadlock which can occur when a server is waiting for a particular request before it can make progress on other responses, but flow control is asserted.
> 
> One of the potential users of HTTP2 whom I've spoken with wished strongly that HEADERS after the first complete HEADERS block on a stream would be subject to flow control. AFAICT, this would work better than what we have today, without potentially triggering the deadlock scenario for many non-browser usecases.
> 
> Thoughts?
> 
> -=R

--
Mark Nottingham   http://www.mnot.net/

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 19 May 2014 16:50:05 +0200

If the protocol does not apply flow controlled to headers, then somebody
will eventually abuse this as a way to get around flow control and start
sending bulk data as headers rather than data.   Once a few start doing it,
then it will become main stream.

This is exactly what happened with browsers switching to using 6
connections instead of 2, because some started to do it to improve their
speed.  It started with individuals changing their own browsers, but then
eventually one browser switched and everybody had to follow to stay
competitive no matter how badly it abused the protocol!

Other than the meta data defined by HTTP, all other headers are essentially
application data and must be flow controlled.

The deadlock scenario described is only a problem if the initial compressed
headers are larger than the initial flow control window, which at 64kB is
highly unlikely with current usage.  However, if we don't put flow control
on headers, then I think we will very soon start seeing >64kB headers!

I definitely support putting flow control on all non control frames within
a stream.   If this means we have to do something a bit ugly to avoid the
deadlock, then I think that is far better than unconstrained meta-data.

On 19 May 2014 09:41, Mark Nottingham <[email-masked]> wrote:

> Be warned that a large amount of the discussion around this will be “Is
> this worth holding HTTP/2 for another implementation draft / implementation
> / interop cycle?”
>
> Cheers,
>
>
>
> > One of the potential users of HTTP2 whom I've spoken with wished
> strongly that HEADERS after the first complete HEADERS block on a stream
> would be subject to flow control. AFAICT, this would work better than what
> we have today, without potentially triggering the deadlock scenario for
> many non-browser usecases.
> >
> > Thoughts?
> >
> > -=R
>
> --

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Tue, 20 May 2014 00:03:51 +0900

> The deadlock scenario described is only a problem if the initial
> compressed headers are larger than the initial flow control window, which
> at 64kB is highly unlikely with current usage.  However, if we don't put
> flow control on headers, then I think we will very soon start seeing >64kB
> headers!
>
>
​We need to take into account connection-level flow control.  It is easily
exhausted and we certainly hit its barrier while sending any size of
HEADERS​ if they are flow controlled.

Best regards,
Tatsuhiro Tsujikawa

>> > Currently HEADERS are not subject to flow control. There are a couple
>> of reasons for this, but the biggest one is that this helps to prevent a

---

From: James M Snell <[email-masked]>
Date: Mon, 19 May 2014 08:10:06 -0700

It will be fairly trivial to do all kinds of things to bypass the flow
control the way things are currently defined. The right way to address
this is to have a clear delineation between frames that are always
hop-by-hop (control) vs. end-to-end (data) and to always require
end-to-end headers to be flow controlled. Making this distinction now
will make it easier later when someone decides to extend the protocol
to do more interesting things beyond just HTTP (like Roberto suggested
at the start of this thread and like I suggested last year).

On Mon, May 19, 2014 at 7:50 AM, Greg Wilkins <[email-masked]> wrote:
> This is exactly what happened with browsers switching to using 6 connections
> instead of 2, because some started to do it to improve their speed.  It
> started with individuals changing their own browsers, but then eventually
> one browser switched and everybody had to follow to stay competitive no
> matter how badly it abused the protocol!
>
> I definitely support putting flow control on all non control frames within a
> stream.   If this means we have to do something a bit ugly to avoid the
>> > we have today, without potentially triggering the deadlock scenario for many
>> > non-browser usecases.
>> >
>> > Thoughts?
>> >
>> > -=R
>>
>> --

---

From: David Krauss <[email-masked]>
Date: Wed, 21 May 2014 15:57:23 +0800

On 2014–05–19, at 10:50 PM, Greg Wilkins <[email-masked]> wrote:

> If the protocol does not apply flow controlled to headers, then somebody will eventually abuse this as a way to get around flow control and start sending bulk data as headers rather than data.   Once a few start doing it, then it will become main stream.

The user has no motivation to get around flow control.

1. The server flow control window only limits uploads. To download faster, the client only needs to set their limit high. (There’s no good reason not to do so.)
2. TCP flow control is redundant with HTTP/2 flow control at the connection level. Per-client throttling is more likely to be implemented at the TCP level.

Putting these together, bypassing flow control would at worst allow a local application to defeat its host server/user-agent’s transmission regulation. Such behavior has already been proposed as a feature (“priority level zero”). The proposal was rejected, but it’s not particularly dangerous.

> This is exactly what happened with browsers switching to using 6 connections instead of 2, because some started to do it to improve their speed.  It started with individuals changing their own browsers, but then eventually one browser switched and everybody had to follow to stay competitive no matter how badly it abused the protocol!

The user (of the user-agent) cannot decide to send bulk data as headers instead of data. That is part of the application protocol. The common methods used on the Web are mostly fixed in this respect. It takes some effort to package bulk data as cookies, and it still won’t stream unless you break it into multiple header blocks, which aren’t supported by current APIs, or multiple entity-streams.

If someone wants to bypass flow control on uploads, they will do what you already described: multi-connection. But, most folks don’t have a problem with not being able to upload fast enough, and since most uploads go to some kind of content provider who has a relationship with the user, the problem is more likely to be solved without hacking.

As for downloads, multi-connection is an obvious hack, but this is a well-known strategy and countermeasures must be applied where necessary anyway.

> Other than the meta data defined by HTTP, all other headers are essentially application data and must be flow controlled.  

The application defines what should be sent in headers. The HTTP/2 spec should include a stronger warning that header blocks should be minimized or broken-up to avoid service degradation due to flow control.

> The deadlock scenario described is only a problem if the initial compressed headers are larger than the initial flow control window, which at 64kB is highly unlikely with current usage.  However, if we don't put flow control on headers, then I think we will very soon start seeing >64kB headers!

There is already a per-implementation, arbitrary maximum header set size. I think there should also be a maximum header set size, because intermediaries are obliged to use RAM to decode headers even if they aren’t added to the reference set. I hadn’t noticed that before. In any case, letting headers grow without bound is a recipe for dropped streams and no competent app developer would do it.

> I definitely support putting flow control on all non control frames within a stream.   If this means we have to do something a bit ugly to avoid the deadlock, then I think that is far better than unconstrained meta-data.

Why not flow-control the control frames too? Then HTTP/2 would be on par with TCP.

The flow control protocol is misplaced and redundant with prioritization. I suspect that it would not exist but for legacy SPDY implementations. But, aside from adding some overhead (at least 24 bytes per TCP frame, approximately doubling the overhead of TCP) and implementation complexity, it’s harmless. Fortunately, its absence is harmless too.

---

From: David Krauss <[email-masked]>
Date: Wed, 21 May 2014 17:18:52 +0800

On 2014–05–21, at 3:57 PM, David Krauss <[email-masked]> wrote:

> I suspect that it would not exist but for legacy SPDY implementations. But, aside from adding some overhead (at least 24 bytes per TCP frame, approximately doubling the overhead of TCP) and implementation complexity, it’s harmless. Fortunately, its absence is harmless too.

Er, stream-level flow control is essential to proxying several origin servers with different TCP windows. But since header blocks suspend multiplexing, you can only proxy a single server at a time during header transmission.

I still dislike connection window updates, though.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 21 May 2014 15:50:04 +0200

> The user has no motivation to get around flow control.

Perhaps not today, but what about in 10 years time or 20 years time???

Plus, I can already consider situations where it is desirable now.  Any
time a HTTP/2.0 connection is used for shared traffic (perhaps from a
connection aggregator or even just multiple client tabs all accessing a
common host/service), then bypassing flow control can give an unfair slice
of the multiplexed channel.  Indeed not only are headers not flow
controlled, they cannot be interleave with other frames!

Finally, remember that the abuse that resulted in the break down of the two
connection limit might have started with application developers doing DNS
sharding, then got popular with users adjusting their browser
configurations, but it really only got cemented in place when the browser
vendors decided to unilaterally ignore the RFC. So any abuse of headers may
not come from user or application developers.

I remain convinced that un-flow controlled and non interleaved headers will
eventually be exploited to access unfair portions of shared resources.   I
find this an extraordinary risk to take just so we can support a specific
header compression algorithm.

regards

---

From: Michael Sweet <[email-masked]>
Date: Wed, 21 May 2014 10:32:25 -0400

+1

Even with the current header compression, there is no reason to prevent intervening DATA frames, or to omit HEADER frames from the scheduling/queuing algorithms that clients and servers must implement for HTTP/2.  The only requirement based on the header compression algorithm that has been adopted is that there can only be a single set of HEADER frames in flight in either direction, and I don't think that is a big step beyond what is already required.

On May 21, 2014, at 9:50 AM, Greg Wilkins <[email-masked]> wrote:

> Plus, I can already consider situations where it is desirable now.  Any time a HTTP/2.0 connection is used for shared traffic (perhaps from a connection aggregator or even just multiple client tabs all accessing a common host/service), then bypassing flow control can give an unfair slice of the multiplexed channel.  Indeed not only are headers not flow controlled, they cannot be interleave with other frames!
> 
> Finally, remember that the abuse that resulted in the break down of the two connection limit might have started with application developers doing DNS sharding, then got popular with users adjusting their browser configurations, but it really only got cemented in place when the browser vendors decided to unilaterally ignore the RFC. So any abuse of headers may not come from user or application developers.
> 
> I remain convinced that un-flow controlled and non interleaved headers will eventually be exploited to access unfair portions of shared resources.   I find this an extraordinary risk to take just so we can support a specific header compression algorithm.
> 
> regards
>  
> 
> 
> 
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 
> -- 

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Ilari Liusvaara <[email-masked]>
Date: Wed, 21 May 2014 18:53:54 +0300

On Wed, May 21, 2014 at 10:32:25AM -0400, Michael Sweet wrote:
> +1
> 
> Even with the current header compression, there is no reason to prevent
> intervening DATA frames, or to omit HEADER frames from the
> scheduling/queuing algorithms that clients and servers must implement
> for HTTP/2.  The only requirement based on the header compression
> algorithm that has been adopted is that there can only be a single set
> of HEADER frames in flight in either direction, and I don't think that
> is a big step beyond what is already required.

Well, this is probably fundamentially ill-defined question, but:

What are the semantics of DATA frames transmitted on stream with HEADER
block still active? If the HEADER is the first header block? If the HEADER
is subsequent HEADER block?

-Ilari

---

From: David Krauss <[email-masked]>
Date: Thu, 22 May 2014 00:01:20 +0800

> Plus, I can already consider situations where it is desirable now.  Any time a HTTP/2.0 connection is used for shared traffic (perhaps from a connection aggregator or even just multiple client tabs…

The solution (which I mentioned) is to limit the size of the decoded header set. This is already necessary to keep an implementation stable. At least, that’s sufficient if the length of the block is limited by the size of the set. If duplicates are allowed, then the encoding of a header set may be arbitrarily large. If an intermediary doesn’t catch such an exploit, then I can see commandeering the connection from a reverse proxy to a server this way.

But enabling flow control would only promote an abuse. The decoded header set should not consume unbounded resources, or more than a typical connection window, no matter what. The restriction which is needed (or really, only deserves mention in the spec), is stronger than flow control.

> Even with the current header compression, there is no reason to prevent intervening DATA frames, or to omit HEADER frames from the scheduling/queuing algorithms that clients and servers must implem…

Decoding takes resources even with a stateless compressor, because the in-flight header set needs to exist in memory. With the current algorithm, the server can at least hope that having a complete header set, and beginning to service the request, will free up resources to handle a new header set.

---

From: David Krauss <[email-masked]>
Date: Thu, 22 May 2014 00:03:56 +0800

If a header block is in progress, anything besides a matching CONTINUATION is specified as a connection error:

> A receiver MUST treat the receipt of any other type of frame or a frame on a different stream as a connection error (Section 5.4.1) of type PROTOCOL_ERROR.

---

From: Michael Sweet <[email-masked]>
Date: Wed, 21 May 2014 13:21:19 -0400

Ilari,

> is subsequent HEADER block?

On a given stream you won't see DATA frames until all of the HEADER frames have been sent (or received).

But if you are in the middle of sending HEADER frames to create a new stream (or for response headers), DATA frames for existing streams can continue to flow - semantically they are separate streams with separate state.

---

From: Michael Sweet <[email-masked]>
Date: Wed, 21 May 2014 13:24:14 -0400

David,

On May 21, 2014, at 12:01 PM, David Krauss <[email-masked]> wrote:
> ...

I'm not suggesting that implementations need to track multiple header sets, just the 2 as required today (one "up" and one "down").  All I am saying is that preventing intervening DATA frames is unnecessary for the compressor that has been chosen, and that already-established streams should not be held up for a set of HEADER frames.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 21 May 2014 10:48:47 -0700

> But if you are in the middle of sending HEADER frames to create a new stream (or for response headers), DATA frames for existing streams can continue to flow - semantically they are separate stream…

That is one possible design.  So far, each time this has been
proposed, it has not garnered any real support.  Big header blocks
will cause head of line blocking, but the answer we have thus far
chosen, is to say "don't do that then".

---

From: David Krauss <[email-masked]>
Date: Thu, 22 May 2014 11:59:56 +0800

> I'm not suggesting that implementations need to track multiple header sets, just the 2 as required today (one "up" and one "down").  All I am saying is that preventing intervening DATA frames is un…

Note that HEADER blocks are also allowed in the middle of a stream, between DATA frames. Although data could flow, the sender would still need to stall in-stream metadata. For a protocol tending toward large header blocks, this could have a snowball effect. But, I suppose the worst case is only the status quo.

We can nip this all in the bud by limiting header block size. For header-heavy protocols, dividing the metadata stream into multiple blocks should be an essential feature in any case. Should an upper limit be set on header block size? How could it be specified?

A good guideline could be that an intermediary should relay header blocks up to a megabyte (decoded, using the same accounting guidelines as for the reference set) and return RST_STREAM otherwise. The amount of metadata added by intermediaries should be capped, at maybe a half kilobyte. Categorizing implementations:

- Non-embedded clients tend to have megabytes to spare, so may be naive.
- Embedded clients are at the mercy of whatever they get. The server application is written specifically for the client, so as not to send an overload, but proxies should be generally friendly as well — they should not add many headers.
- Servers and proxies should be “serious” implementations. One megabyte per connection adds up to insane memory, but a low-memory condition may be handled by delaying processing of new header blocks.
- An origin server (sysadmin) may set the limit lower than a megabyte.
- A specific HEADER_OVERFLOW error code could be cached by a proxy and used to preemptively terminate possibly malicious subsequent attempts to the same origin.
- A proxy never needs to retain the header set, it only needs to peek at a few headers as they’re received and directly forward the encoded bytes. Counting the hypothetical decoded size isn’t much additional work, though.

Note, Huffman “compressed” binary data is 3.5x bigger on the wire, so 1 MiB decoded is up to 3.5 MiB transferred. I might be guessing high by many orders of magnitude, but I’m just being conservative on the side of the currently apparent design intent, which is to be unlimited.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 22 May 2014 03:49:01 -0700

On May 22, 2014 12:04 AM, "David Krauss" <[email-masked]> wrote:
> We can nip this all in the bud by limiting header block size. For
header-heavy protocols, dividing the metadata stream into multiple blocks
should be an essential feature in any case. Should an upper limit be set on
header block size? How could it be specified?

We have asked before about setting a hard limit. That doesn't work. There
are too many existing uses and users that wouldn't be able to use HTTP/2 as
a result.

A declared maximum (i.e., settings) might work.

You can't however reject a header block that you don't want. Not without
also dumping the connection. Common state being what it is.

The best approach is to stream headers. The HPACK design permits it, and
that allows for a bounded state commitment. The only real cost is the head
of line blocking.

> Note, Huffman “compressed” binary data is 3.5x bigger on the wire, so 1
MiB decoded is up to 3.5 MiB transferred. I might be guessing high by many
orders of magnitude, but I’m just being conservative on the side of the
currently apparent design intent, which is to be unlimited.

That is perhaps true if you do the naive thing, and only if you don't have
any syntax constraints. The real cost depends, but if you do cookie syntax,
it's closer to 1.6.

---

From: David Krauss <[email-masked]>
Date: Thu, 22 May 2014 19:03:19 +0800

On 2014–05–22, at 6:49 PM, Martin Thomson <[email-masked]> wrote:

> We have asked before about setting a hard limit. That doesn't work. There are too many existing uses and users that wouldn't be able to use HTTP/2 as a result.
> 
Even a megabyte? Who sends that many HTTP/1.1 headers? Just divide the headers into separate blocks, how hard can it be?

A 1.1<->2 proxy can add divisions arbitrarily, without adding ambiguity. END_STREAM serves as a flag not to terminate the 1.1 headers.

> A declared maximum (i.e., settings) might work.

Not proxyable.
> You can't however reject a header block that you don't want. Not without also dumping the connection. Common state being what it is.
> 
Right.
> The best approach is to stream headers. The HPACK design permits it, and that allows for a bounded state commitment. The only real cost is the head of line blocking.
> 
Unbounded head of line blocking. If a reverse proxy decides to start streaming headers to an origin, but the client stops sending them, then the server becomes a zombie. The solutions are not to use stream multiplexing upstream of a reverse proxy, or buffer headers indefinitely at the proxy. Neither is palatable, and the latter really comes down to dropping the connection after a megabyte anyway.
> That is perhaps true if you do the naive thing, and only if you don't have any syntax constraints. The real cost depends, but if you do cookie syntax, it's closer to 1.6.
I was thinking in terms of a malicious user who wants to send as many bytes as possible under the lawful constraints, for the purpose of degrading the server-side network.

---

From: Ilari Liusvaara <[email-masked]>
Date: Thu, 22 May 2014 14:47:04 +0300

> A declared maximum (i.e., settings) might work.

Unfortunately, there are rather many possible limits, e.g.:
- Maximum total uncompressed size
- Maximum compressed size of individual header backed by stream
- Maximum uncompressed size of individual header backed by stream
- Maximum size of individual header backed by header table
- <probably fair amount of limits I can't quickly come up with>

Some of those may also be effectively infinity.

> also dumping the connection. Common state being what it is.

AFAIK, It is possible to do just corrupting the stream state, not connection
state (just mark things as failed and stop really emitting headers). Then upon
end of block, if things failed, try to deal with the corrupt stream.

The amount of temporary state required for such processing is just a few
bytes (the rest of working set comes from global state, which must be committed
anyway).

> that allows for a bounded state commitment. The only real cost is the head

AFAIK, individual header sizes can still cause problems, since headers can be
very large and span multiple header frames.

Also, stuff like this can cause trouble:

foo=bar
qux=zot
... <lots of stuff> ...
:path=/something
:method=GET
:authority=example.org

These kind of cases are very nasty to handle in streaming manner, because the
webserver can't dispatch execution before almost the end of block.

-Ilari

---

From: Martin Thomson <[email-masked]>
Date: Thu, 22 May 2014 05:31:03 -0700

On 22 May 2014 04:03, David Krauss <[email-masked]> wrote:
> Even a megabyte? Who sends that many HTTP/1.1 headers? Just divide the
> headers into separate blocks, how hard can it be?

Yes, even a megabyte.  We've had reports of URLs alone that are >64K
and as far as I know, the sky's the limit for cookies.

> A 1.1<->2 proxy can add divisions arbitrarily, without adding ambiguity.
> END_STREAM serves as a flag not to terminate the 1.1 headers.
>

Right.  But that would only mean that the intermediary would be forced
to RST the stream if it couldn't forward it.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 22 May 2014 05:36:07 -0700

> Some of those may also be effectively infinity.

I imagine that we'd have to rely on the serialized size if we did
anything at all.  That way it's trivial to enforce; an important
property methinks.

> end of block, if things failed, try to deal with the corrupt stream.

Yes, you can process the updates to the header table, but dump the
output into the bitbucket and reset the stream.

> webserver can't dispatch execution before almost the end of block.

Good point.  That's why a good implementation will put all the routing
fields at the start.  But that's not possible if the routing fields
are pulled from the reference set, which I just realize is highly
likely for things like :scheme and :authority.  Bleargh.

---

From: Roland Zink <[email-masked]>
Date: Thu, 22 May 2014 15:09:26 +0200

> output into the bitbucket and reset the stream.
This probably can be done as a header block is processed as a unit, but 
currently we have:

An endpoint receiving HEADERS, PUSH_PROMISE or CONTINUATION frames MUST 
reassemble header blocks and perform decompression even if the frames 
are to be discarded. A receiver MUST terminate the connection with a 
connection error (Section 5.4.1 
<http://tools.ietf.org/html/draft-ietf-httpbis-http2-12#section-5.4.1>) 
of type COMPRESSION_ERROR if it does not decompress a header block.

> likely for things like :scheme and :authority.  Bleargh.
For intermediaries this seem to mean they need to do store and forward 
instead of streaming. If they still want to do streaming they are 
vulnerable to malicious clients not sending continuation frames.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 22 May 2014 20:08:14 +0200

Martin et al,

I really see the problems and confusion in this thread as another example
of the complexities and problems that result when you "conflate some of the
stream control functions with the application semantics functions in the
interests of efficiency."

The definition of streams should be simple and robust and entirely
independent of the semantics of the content of the streams.    A stream
should be defined to exist from the time it ID is first mentioned (either
in a frame ID or a PP) until frames have been sent in both directions with
an end frame bit set.    Between those two events, the stream should be
flow controlled regardless of content.

Once you have robust multiplexed framing, then you can layer HTTP semantics
on top of that.   If a client wants to avoid the dispatch of its requests
from being deferred, then it should just restrict it's headers to < the
stream window size.

I think the hpack tail is really wagging this protocol dog!

Currently hpack is forcing this protocol to have a user accessible non flow
controlled un-segmentable size unlimited meta data streams that has strict
ordering criteria resulting in significant parallel slow down.

Our experience with SPDY is that the vast majority of the speed up comes
from avoiding round trips rather than compression.    This means you need
good multiplexing and good push mechanisms.  Compression is worthwhile, but
I suspect that 90% of the gains can be achieved with a far simpler
algorithm that does not impose such onerous constraints on the framing.

again apologies for giving these opinions late in the process.

---

From: <[email-masked]>
Date: Thu, 22 May 2014 19:38:50 +0000

+1

On May 22, 2014, at 20:11, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:

Martin et al,

I really see the problems and confusion in this thread as another example of the complexities and problems that result when you "conflate some of the stream control functions with the application semantics functions in the interests of efficiency."

The definition of streams should be simple and robust and entirely independent of the semantics of the content of the streams.    A stream should be defined to exist from the time it ID is first mentioned (either in a frame ID or a PP) until frames have been sent in both directions with an end frame bit set.    Between those two events, the stream should be flow controlled regardless of content.

Once you have robust multiplexed framing, then you can layer HTTP semantics on top of that.   If a client wants to avoid the dispatch of its requests from being deferred, then it should just restrict it's headers to < the stream window size.

Currently hpack is forcing this protocol to have a user accessible non flow controlled un-segmentable size unlimited meta data streams that has strict ordering criteria resulting in significant parallel slow down.

Our experience with SPDY is that the vast majority of the speed up comes from avoiding round trips rather than compression.    This means you need good multiplexing and good push mechanisms.  Compression is worthwhile, but I suspect that 90% of the gains can be achieved with a far simpler algorithm that does not impose such onerous constraints on the framing.

--
Greg Wilkins <[email-masked]<mailto:[email-masked]>>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 23 May 2014 10:28:48 +1000

On 23 May 2014 04:08, Greg Wilkins <[email-masked]> wrote:

> Currently hpack is forcing this protocol to have a user accessible non
> flow controlled un-segmentable size unlimited meta data streams that has
> strict ordering criteria resulting in significant parallel slow down.
>
>
​It's also the single most complex part of the spec​ to implement. My
side-project implementation has hiccupped and stalled every time it's
bumped into continuation frames and hpack. I've currently stopped all work
on it until I have time to go through and read in much more detail the
hpack draft, and some related archived conversations on the list, and
Canonical prefix encodings and Huffman coding, etc.

If I could negotiate not using HPACK I could probably have the rest of my
implementation up to some sort of interop testing by now.

​
​

> ​
> algorithm that does not impose such onerous constraints on the framing.​
> ​​
>
>
>

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Cory Benfield <[email-masked]>
Date: Fri, 23 May 2014 19:53:26 +0100

On 23 May 2014 at 01:32:12, Matthew Kerwin ([email-masked](mailto:[email-masked])) wrote:
> ​It's also the single most complex part of the spec​ to implement. My side-project implementation has hiccupped and stalled every time it's bumped into continuation frames and hpack. I've currently stopped all work on it until I have time to go through and read in much more detail the hpack draft, and some related archived conversations on the list, and Canonical prefix encodings and Huffman coding, etc.  
>  
> If I could negotiate not using HPACK I could probably have the rest of my implementation up to some sort of interop testing by now.  

Could not agree more.

I don't want to harp on this too much because I've mentioned it before, but
HPACK is really complex and so, so easy to get wrong. The only way to find out
whether you've got it right is with really rigorous integration testing and
even that's not really sufficient to catch every edge case.

Implementing HPACK is so much harder than implementing HTTP/2. A naive HTTP/2
implementation is simply not that difficult. However, there's no such thing as
a naive HPACK implementation, only wrong implementations.

---

From: Michael Sweet <[email-masked]>
Date: Fri, 23 May 2014 15:22:59 -0400

+1

---

From: Yoav Nir <[email-masked]>
Date: Sun, 25 May 2014 17:46:49 +0300

> If I could negotiate not using HPACK I could probably have the rest of my implementation up to some sort of interop testing by now.

I might be missing something basic here. But if you set the SETTINGS_HEADER_TABLE_SIZE to zero, doesn’t this effectively eliminate compression (and state!), leaving only encoding? ISTM that it works for a minimal client.  Am I correct in assuming that compression and state are the painful parts?

A minimal server could potentially receive a bunch of requests with compressed headers before the client receives the SETTINGS frame. But I’m sure that could be worked around, for example by closing all streams that have compressed headers. Not quite efficient, but if efficiency was the primary goal for this hypothetical server, it would implement HPACK.

What am I missing?

Yoav

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 26 May 2014 13:44:36 +1000

​On 26 May 2014 00:46, Yoav Nir <[email-masked]> wrote:
>
> On 23 May 2014 03:28, Matthew Kerwin <[email-masked]> wrote:
>>
>>> Currently hpack is forcing this protocol to have a user accessible
>>> non flow controlled un-segmentable size unlimited meta data streams
>>> that has strict ordering criteria resulting in significant parallel
>>> slow down.
>>>
>>
>> bumped into continuation frames and hpack. I've currently stopped all
>> work on it until I have time to go through and read in much more
>> detail the hpack draft, and some related archived conversations on
>> the list, and Canonical prefix encodings and Huffman coding, etc.
>>
>> If I could negotiate not using HPACK I could probably have the rest
>> of my implementation up to some sort of interop testing by now.
>>
>
> I might be missing something basic here. But if you set the
> SETTINGS_HEADER_TABLE_SIZE to zero, doesn’t this effectively eliminate
> compression (and state!), leaving only encoding? ISTM that it works
> for a minimal client.

I'm not sure, that's one of those things I'd have to infer from a solid
reading of the spec, or find in the mailing list archives. I think
it's correct.

> Am I correct in assuming that compression and state are the painful
> parts?

I don't know that there are any painful parts, per se; more that there's
a whole separate 17 page spec, with 8 pages of appendices, that requires
20 pages of examples, _and_ that the streaming compression adds other
requirements and limitations to HTTP2 itself.

In terms of implementation, so far the header table is pretty easy, and
I think I've started to get a handle on the reference set; but I've
never written a Huffman coder before, nor even heard of canonical prefix
encoding. My networking experience comes from a place where RLE was
enough.

>
> A minimal server could potentially receive a bunch of requests with
> compressed headers before the client receives the SETTINGS frame. But
> I’m sure that could be worked around, for example by closing all
> streams that have compressed headers. Not quite efficient, but if
> efficiency was the primary goal for this hypothetical server, it would
> implement HPACK.
>
> What am I missing?
>

The first hurdle was when I realised that CONTINUATION frames have to be
contiguous, so my low-level architecture went from something resembling
a simple packet handler to a hybrid with the ability to effectively
block on subsequent packets.  That wasn't even an issue in the HPACK
spec, but an implication of using HPACK in HTTP/2.  Fortunately in that
case I was able to fall back on an earlier decision I'd made (out of
laziness), where I could block waiting for an ACK after sending the
initial SETTINGS.

My next issue, assuming HPACK itself is implemented correctly (or
someone gives me an awesome library, or something), is that I'll have to
redesign parts of my low-level machinery because, for the first time in
the protocol, the actual value of some received message isn't contained
inside the frames themselves. So my packet switching architecture
isn't able to discuss headers with higher-level abstraction layers.
I have an idea of how to proceed, by effectively jamming the HEADERS
(or PUSH_PROMISE) and all its associated CONTINUATIONS into one big
superframe, but it's a bit of a work around, and incidentally
complicates other earlier design decisions.

HTTP/1.1 was, generally, easy at this level, because it's a piece of
cake to read up to a ":", and then again to a "\r\n" (usual caveats for
LWS, folding, quoted strings, etc.). Then you emit the pair, and repeat.
HTTP/2 was almost easy, because I was almost able to deserialise frames
one by one and emit them as they arrived, except that I can't do that
for headers (or push_promise).

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 May 2014 12:09:15 +0200

On 9 May 2014 20:42, Roberto Peon <[email-masked]> wrote:

> deadlock which can occur when a server is waiting for a particular
> request before it can make progress on other responses, but flow control is
> asserted.
>

Circling back on this one!

Excluding HEADERS from flow control doesn't prevent deadlock.   If the
request that the server is waiting on is a post with a body and the
connection window is consumed, then we are deadlocked just the same.

Flow control is intended to limit the resources that a server needs to
commit to a stream/connection. So exempting headers just gives an incentive
to put more data in them, to thus consume more server resources and thus
increase the chances of once stream interacting with another.

Big data is not a problem for a server, as you only have to read a little
bit of it at a time and stream it to the application (which may decide to
aggregate, but that is somebody else design decision).  Big headers are a
big resource commitment for servers as we must hold them for the entire
duration of the request handling.

So if a server accepts >64KB of headers on a stream, it has to hold those
for the entire duration of the request handling as the application can ask
for a header at any time.  Even if the request handling suspends and waits
a long time for some downstream service, the headers have to be held.

To me, this is just like the browsers unilaterally deciding to break the
two connection rule, regardless of the impact on server resources, except
in this case at least servers have the ability to say no.  However, if
browser vendors decide to allow larger headers to be sent and large
corporations with big budgets are accepting of those headers, this will
have a big resource impact on the rest of use.   Sorry for my "flair for
the dramatic", but this goes somewhat towards network neutrality as we
should not be designing a protocol that will only scale for those with big
budgets!

regards

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Wed, 28 May 2014 21:31:53 +0900

> connection window is consumed, then we are deadlocked just the same.
​If we just flow control HEADERS as is in the current draft, we get
unavoidable deadlock.
This is because HEADERS cannot be arbitrary split into several frames since
the draft says that only first or last HEADERS have HTTP semantics.
We can do this in DATA frame.  For example, if there is 200 bytes available
in connection window and 250 bytes available in stream window, we send 200
bytes DATA and wait for WINDOW_UPDATE.  Meanwhile we can also send urgent
frames like PING, WINDOW_UPDATE and RST_STREAM etc..
For HEADERS we have to send entire frame and you may hit window size = 0 in
the middle of the HEADERS payload. If this happens, we just stuck until
WINDOW_UPDATE comes.
Suppose that both ends send HEADERS and hit connection window = 0 in the
middle of HEADERS.  This is nasty deadlock situation.

​Best regards,
Tatsuhiro Tsujikawa​

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 May 2014 16:55:05 +0200

Tatsuhiro,

but flow control exists to limit the resources that a server must dedicate
to a connection.  If the window is 0, then the server is saying that it
does not have any more resources available.   So the client simple should
not send a new stream as there are no server resources.

By removing the protection of excess resource consumption from the protocol
will not magically give the server any more resources.

In the case that you give, if there are 200 bytes available, but the
headers of a new stream is 250, then there can be two possibilities - the
client sends the 250 without flow control and the server resets the stream
because it has not resources  OR the client does not send the 250, but
instead uses the 200 to allow another request to progress (and maybe
complete).

In either case no progress is made on the new request, which may be holding
up some other resource.   But in the current case, bandwidth is wasted and
no other stream progresses.

Removing flow control has not fixed the problem, it has just made it
somebody elses problem.

cheers

On 28 May 2014 14:31, Tatsuhiro Tsujikawa <[email-masked]> wrote:

> This is because HEADERS cannot be arbitrary split into several frames
> since the draft says that only first or last HEADERS have HTTP semantics.
> We can do this in DATA frame.  For example, if there is 200 bytes
> available in connection window and 250 bytes available in stream window, we
> send 200 bytes DATA and wait for WINDOW_UPDATE.  Meanwhile we can also send
> urgent frames like PING, WINDOW_UPDATE and RST_STREAM etc..
> For HEADERS we have to send entire frame and you may hit window size = 0
> in the middle of the HEADERS payload. If this happens, we just stuck until
> WINDOW_UPDATE comes.

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Thu, 29 May 2014 00:16:47 +0900

> not send a new stream as there are no server resources.
Are you saying that while server retains request headers in memory, it must
not respond WINDOW_UPDATE since it is the commitment of the resource?  If
so, the connection window gets drastically small and concurrency is vastly
reduced.
 ​

> By removing the protection of excess resource consumption from the
> protocol will not magically give the server any more resources.
>
​You cannot 250 bytes HEADERS frame WITHOUT flow control if flow control is
enabled for HEADERS.​
It is mandatory and violating flow control is subject to connection error.
For the latter, I feel like a non-trivial scheduling problem, but I'm not
sure.

Best regards,
Tatsuhiro Tsujikawa

> In either case no progress is made on the new request, which may be
> holding up some other resource.   But in the current case, bandwidth is
> wasted and no other stream progresses.
>
>>> Big data is not a problem for a server, as you only have to read a
>>> little bit of it at a time and stream it to the application (which may
>>> decide to aggregate, but that is somebody else design decision).  Big
>>> headers are a big resource commitment for servers as we must hold them for
>>> the entire duration of the request handling.
>>>
>>> So if a server accepts >64KB of headers on a stream, it has to hold
>>> those for the entire duration of the request handling as the application
>>> can ask for a header at any time.  Even if the request handling suspends
>>> and waits a long time for some downstream service, the headers have to be
>>> held.
>>>

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 May 2014 18:16:15 +0200

On 28 May 2014 17:16, Tatsuhiro Tsujikawa <[email-masked]> wrote:

>> but flow control exists to limit the resources that a server must
>> dedicate to a connection.  If the window is 0, then the server is saying
>> that it does not have any more resources available.   So the client simple
>> should not send a new stream as there are no server resources.
>>
>>
> Are you saying that while server retains request headers in memory, it
> must not respond WINDOW_UPDATE since it is the commitment of the resource?
>  If so, the connection window gets drastically small and concurrency is
> vastly reduced.
>

No I'm not saying that.  I'm saying that with flow control the server makes
a fixed memory commitment when it accepts a connection.   If it is prepared
to accept more data it will send a WINDOW_UPDATE.

If the server is not prepared to accept any more data from a connection,
then it will not send a WINDOW_UPDATE.  If the client then sends enough
data to consume the window, then the window is consumed and the client
should be flow controlled.

So just saying, that the data the client wants to send is meta-data in a
header field does not some how make the server have any more resources
available in order to handle those HEADER/CONTINUATION frames.  Sending
additional headers when the flow control windows are full violates the
resource contract between client and server as the client is asking the
server to handle and store an unlimited amount of extra data.

I well written server will reset the stream, a not so well written server
may try to continue but may eventually throw an out of memory exception and
crash.   Eitherway, the non-flow controlled header data does not get
through and the dead lock is not resolved (well I guess crashing the server
will resolve the dead lock).

>> instead uses the 200 to allow another request to progress (and maybe
In your example, you said that if we flow controlled headers then a client
with a 250B header but only 200B window would not be able to send the
header.     You said this as an argument to say the client should be able
to send the 250B without flow control, otherwise an important request might
not get through and you may get deadlock.

I'm saying that a well written server that enforces resource limitations on
connections would probably  reset the stream if you did send those 250B as
it goes above the resource allocation it is prepared to make per
connection.  Perhaps an accepting server might accept some additional
requests even when they are low on resources, but servers are not infinite
and at some point they will have to reject some incremental important 250B
header set from the client (or crash) - so dead lock will result anyway.

Removing flow control from headers has not avoided deadlock, it has just
made the client unaware of the amount of data that it can send before
deadlock will occur.

regards

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 May 2014 18:35:10 +0200

Actually, it is even worse than what I just described.

If the resource constrained server does not have the resources to accept
the 250B extra header, it can RST_STREAM, but it still has to process the
headers, because of the shared state table.   So if the server really is
resource constrained, and wants to limit the resources of each connection,
then it wont just RST_STREAM, it will GO_AWAY the whole connection - and
all the work in progress on all the other streams will be lost!

Unless the server is able to exert back pressure on the client to stop it
sending more data than the server is prepared to handle, then the only
viable response the server has is to close the connection.   Saying that
there is no flow control on headers only works if you are talking to a
fantasy infinite server!

Any server that is lenient and allows clients to go significantly beyond
resource limits, just because the data is in a headers frame rather than a
data frame, is just going to be exploited!

regards

On 28 May 2014 18:16, Greg Wilkins <[email-masked]> wrote:

> No I'm not saying that.  I'm saying that with flow control the server
> makes a fixed memory commitment when it accepts a connection.   If it is
> prepared to accept more data it will send a WINDOW_UPDATE.
>
> I'm saying that a well written server that enforces resource limitations
> on connections would probably  reset the stream if you did send those 250B
> as it goes above the resource allocation it is prepared to make per

---

From: Martin Thomson <[email-masked]>
Date: Wed, 28 May 2014 10:08:39 -0700

On 28 May 2014 09:35, Greg Wilkins <[email-masked]> wrote:
> If the resource constrained server does not have the resources to accept the
> 250B extra header, it can RST_STREAM, but it still has to process the
> then it wont just RST_STREAM, it will GO_AWAY the whole connection - and all
> the work in progress on all the other streams will be lost!

Yes, if you can't tolerate the work that updating the header table
requires, then I suspect that you might find you are best dropping
connections.

I don't see any intrinsic problem with this.  We've delegated the
state commitment management to the HTTP layer: the 431 status code,
specifically.  That makes more sense to me, since header processing is
a function of that layer.  RST_STREAM remains as a secondary option.
GOAWAY as a measure of last resort.

---

From: Johnny Graettinger <[email-masked]>
Date: Wed, 28 May 2014 14:31:49 -0700

Looping back to the OP:

Under the current draft, one way in which peers could effectively negotiate
flow-control for HEADERS is first send empty DATA frame(s) padded to the
HEADERS size. This could be made efficient if DATA frames were able to
express flow-control commitment beyond the wire size of the frame. Is there
interest in this?

There are lots of ways this could be conveyed, but the least disruptive may
be as a DATA frame with padding larger than the frame size.

On Wed, May 28, 2014 at 10:08 AM, Martin Thomson
<[email-masked]>wrote:

> the
> all

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 May 2014 11:52:02 +0200

I think the simplest solution is to just include all headers in the flow
control calculations.

I believe that the argument against this (ie dead locks) is erroneous
because server may still reject important requests if they are resource
constrained, so dead lock is not avoided it is just moved to the
unknown.    With headers included in flow control, clients would have a
solid contract with the server and would know ahead of time if a request
can be sent or not.

Senders should not commence sending a headers unless there is sufficient
window available to send all the frames.     I know this effectively
prohibits streaming headers, but then streaming headers is beyond the scope
of http2 as it is not something that http/1.1 commonly supports anyway.

If we really need to support headers larger that can fit in a single frame,
then we could add a flag that set subsequent header sets sent should be
aggregated.  ie a hpack header set would have to be sent in a single frame
(making the ordering of decoding easy), but that subsequent sets could be
aggregated to make larger headers sets.  This allows other streams to
interleave frames between the aggregated header frames.

cheers

On 28 May 2014 23:31, Johnny Graettinger <[email-masked]> wrote:

> Looping back to the OP:
>
> Under the current draft, one way in which peers could effectively
> negotiate flow-control for HEADERS is first send empty DATA frame(s) padded
> to the HEADERS size. This could be made efficient if DATA frames were able
> to express flow-control commitment beyond the wire size of the frame. Is
> there interest in this?
>
> There are lots of ways this could be conveyed, but the least disruptive
> may be as a DATA frame with padding larger than the frame size.
>
>
> On Wed, May 28, 2014 at 10:08 AM, Martin Thomson <[email-masked]
> > wrote:
>

---

From: Johnny Graettinger <[email-masked]>
Date: Thu, 29 May 2014 13:27:51 -0400

> window available to send all the frames.
>

This would be my preference also, but:

It significantly complicates encoding. A header block must be encoded to
determine it's size, and the act of encoding changes encoder state. An
encoder must have a checkpoint or "undo" mechanism to throw out changes
from the overflowing representation.

An encoded header block can also be larger than the stream's send window
(particularly if a server elects to use a small initial window that it
later ramps up, ala expect 100 continue). How does a sender ask for enough
window to send the block?

A way to express flow-control commitment without sending the actual bytes
would, for example, allow an RPC protocol layered on HTTP/2 to negotiate
flow control by first committing the uncompressed header block size.

> If we really need to support headers larger that can fit in a single
> frame, then we could add a flag that set subsequent header sets sent should
> be aggregated. ie a hpack header set would have to be sent in a single
> frame (making the ordering of decoding easy), but that subsequent sets
> could be aggregated to make larger headers sets.  This allows other streams
> to interleave frames between the aggregated header frames.
>

Let me tweak this slightly; I think your suggestion is equivalent to:

* Requiring that HEADER/PUSH_PROMISE/CONTINUATION frames be broken on HPACK
opcode boundaries.
* Flow-controlling these frames under the stream & session.
* Allowing other stream's frames to interleave between continuations.

I think this has the following implications:

The expect-continuation / don't-expect-continuation state machine is moved
from the session to the stream. This doesn't seem like a big deal.

A sender can always make progress with a non-zero send window. Padding may
have to be used to completely fill the window (which allows sending
BLOCKED).

A session's decoder is always in a consistent shared state after a
HEADER/PUSH_PROMISE/CONTINUATION frame.

A sesson's encoder needs to do more work. For each emitted representation
(and importantly, *not* for the entire block), an HPACK encoder is required
to check whether the resulting opcode would overflow the window, and to not
commit it if so. There are some corner cases (eg, an encoder might prefix
an representation's opcode with other index opcodes that it would evict;
example
<https://code.google.com/p/chromium/codesearch#chromium/src/net/spdy/hpack_encoder.cc&q=hpack_encoder&sq=package:chromium&l=65>).
Still, I think this is certainly do-able, though it may be a significant
change for existing implementations.

On balance I think this is an interesting option.

cheers,
-johnny

>> On Wed, May 28, 2014 at 10:08 AM, Martin Thomson <

---

From: Johnny Graettinger <[email-masked]>
Date: Thu, 29 May 2014 13:36:34 -0400

On more implication:

You can still get deadlocked if a sender wishes to send a header
representation who's encoding is larger than the largest send window the
receiver is willing to allow.

On Thu, May 29, 2014 at 1:27 PM, Johnny Graettinger <
[email-masked]> wrote:

> * Requiring that HEADER/PUSH_PROMISE/CONTINUATION frames be broken on
> HPACK opcode boundaries.
> (and importantly, *not* for the entire block), an HPACK encoder is
> required to check whether the resulting opcode would overflow the window,
> and to not commit it if so. There are some corner cases (eg, an encoder
> might prefix an representation's opcode with other index opcodes that it
> would evict; example