Subject: H2 HEADERS and flow control

From: Greg Wilkins <[email-masked]>
Date: Thu, 3 Jul 2014 18:19:37 +1000

No I'm not intending to start the whole debate on this again..... I just
have a question on one aspect.

I totally accept that we cannot flow control headers whilst we have a
mutable shared state table, and I pretty much accept the benefits of said
shared mutable state table in this incarnation of the protocol.

But why can't we at least deduct the header frame sizes from the available
window size, so that a stream that is sending a lot of data in headers does
not necessarily get more share of the connection.

Simple example is that if a response is sent starting with a 16KB HEADERS
and a 10KB CONTINUATION, then instead of the data frames having a 64KB
window, they will be left with a 48KB window.

The caveats will of course have to be that a header/continuation frame will
not be able to reduce the window below 0, nor shall it be prevented from
being sent if the window is zero.   Thus of a 256KB header is sent, the
stream window will be zero, but the 7 continuation frames will all get
sent, but the stream will stall on the first data frame waiting for a
window update.

If we accept that most implementations will have a limit on header size
<=16KB for 99.99% of users, then including the header frame sizes in the
window accounting will at least remove the incentive for applications to
move data to the headers to avoid flow control.  Ie sending headers will
still have a cost - only that it is paid by your data frames rather than
your headers/continuations.

Has this been considered before?

regards

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 3 Jul 2014 10:33:18 -0700

> Has this been considered before?

Not to my knowledge.

Would this only affect the stream window, or the stream and connection
windows both?  I suspect that the latter would be easier to manage.

You could permit the window to become negative.  Implementations are
required to track a negative window after settings changes..

---

From: Nicholas Hurley <[email-masked]>
Date: Thu, 3 Jul 2014 10:54:15 -0700

> your headers/continuations.

I'm not convinced this is the case, though. If you have some bad (or at
least selfish) actor that is already shoving data into headers to avoid
flow control, then I would imagine this situation would make them want to
shove even MORE data into headers, since it's still not flow-controlled,
but harms their data. What's the "best" way around that for one of those
actors? Put it ALL in headers. Perhaps I'm just more pessimistic than you
are, though :)

Note, I am (like you) definitely not suggesting (or trying to re-open) the
flow controlled headers argument. I was already against flow controlled
headers. I just think that this provides the worst of both worlds.
--
Peace,
  -Nick

---

From: William Chan (陈智昌) <[email-masked]>
Date: Thu, 3 Jul 2014 10:57:12 -0700

>
> But why can't we at least deduct the header frame sizes from the
available window size, so that a stream that is sending a lot of data in
headers does not necessarily get more share of the connection.

We considered this, but if headers are being used as a control channel for
a stream, it's problematic to flow control it. Just like how we don't flew
control control frames.

>
> The caveats will of course have to be that a header/continuation frame
will not be able to reduce the window below 0, nor shall it be prevented
from being sent if the window is zero.   Thus of a 256KB header is sent,
the stream window will be zero, but the 7 continuation frames will all get
sent, but the stream will stall on the first data frame waiting for a
window update.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Thu, 3 Jul 2014 11:00:28 -0700

On Jul 3, 2014 10:57 AM, "William Chan (陈智昌)" <[email-masked]> wrote:
> > No I'm not intending to start the whole debate on this again..... I
just have a question on one aspect.
>
> We considered this, but if headers are being used as a control channel
for a stream, it's problematic to flow control it. Just like how we don't
flew control control frames.

I reread the other emails and realize now that I misread your suggestion.
Ignore my email.

>
> >
> > Simple example is that if a response is sent starting with a 16KB
HEADERS and a 10KB CONTINUATION, then instead of the data frames having a
64KB window, they will be left with a 48KB window.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 03 Jul 2014 18:17:55 +0000

>required to track a negative window after settings changes..

One nit related to windows:

Window updates are 31 bits, but we don't put a bound on the actual
window size.

At the very least we should write that there is no guarantee that
a 32bit signed integer will be enough to hold the window value.

But simpler and clearer would be to mandate that "window sizes are
64 signed values" and be done with it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Daniel Sommermann <[email-masked]>
Date: Thu, 3 Jul 2014 11:25:19 -0700

6.9.1 has it:

A sender MUST NOT allow a flow control window to exceed 2^31 - 1 bytes. 
If a sender receives a WINDOW_UPDATE that causes a flow control window 
to exceed this maximum it MUST terminate either the stream or the 
connection, as appropriate.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 03 Jul 2014 18:59:57 +0000

>connection, as appropriate.

I suggest we raise that limit to 2^63-1 bytes.

2^32 bytes only take a few seconds seconds at 100 Gbit, and files
larger than 2^32 are common as muck these days.

Raising the width of the window update itself is probably not needed,
as long as we squander the reserved bit on something else.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 3 Jul 2014 21:34:35 +0200

> larger than 2^32 are common as muck these days.

Well, 2^31 should be about 160ms at 100Gbps, but you can only achieve
such rates over TCP on very short links because TCP itself will not
allow you to grow a window that large, so in practice you'll need some
round trips at least for TCP acks. So in practice there will be more
than one TCP connection at these rates when observed out of the
datacenter, thus I don't think there's any gain in mandating a window
larger than 31 or 32 bits.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 03 Jul 2014 19:47:08 +0000

In message <[email-masked]>, Willy Tarreau writes:

>thus I don't think there's any gain in mandating a window

Right now ?  No.

But I find the margin is too small for HTTP/2, we should design for
at least 10 years lifetime before people run into trouble.

Also these four bytes don't add wire overhead so there is no
performance penalty for being on the safe side.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 3 Jul 2014 22:03:37 +0200

> at least 10 years lifetime before people run into trouble.

Given that TCP cannot do better with its 32-bit sequence numbers, and
anything but TCP/IPv4 seems unlikely to replace it anytime soon, I must
say I'm not worried even for the 10 next years (and don't get me wrong,
I too am concerned about long term).

> performance penalty for being on the safe side.

That's perfectly true, the only cost is 8 extra memory bytes per
stream (4 per direction). I'd say I don't really care about them,
it's only that I'm not convinced that there is at least a *chance*
for them to ever be used. It all depends if there are more risks
of seeing bogus implementations or not just for this.

Willy

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 4 Jul 2014 11:41:16 +1000

> actors? Put it ALL in headers. Perhaps I'm just more pessimistic than you

I've thought about these type of malevolent actors also.   I think that
there is nothing in the current draft that prevents them from acting in
this way in any case.   If they want to write custom clients to write
IPoverHTTPv2Headers, they can and probably will.

But my thought bubble is aimed more at naive actors, who are using normal
clients, but who's cookies are getting large or are including big kerberos
keys etc. etc.      Just because their headers creep up to 32KB say, should
not mean that they get 50% more fair share than a stream that has no
headers.

cheers

---

From: David Krauss <[email-masked]>
Date: Fri, 4 Jul 2014 13:28:28 +0800

On 2014–07–04, at 9:41 AM, Greg Wilkins <[email-masked]> wrote:

> But my thought bubble is aimed more at naive actors, who are using normal clients, but who's cookies are getting large or are including big kerberos keys etc. etc.      Just because their headers creep up to 32KB say, should not mean that they get 50% more fair share than a stream that has no headers.

Flow control and prioritization are different things. A server is free to penalize a stream for control overhead, it just means sending less data on that resource allocation cycle.

Correct me if I’m wrong!

To be sure, we’re discussing what a good server does for a naive client. (Supporting lots of cookies at all is a big favor.) If the server is naive, all bets are off.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 4 Jul 2014 15:54:58 +1000

> Correct me if I’m wrong!

Whilst what I'm advocating could be somewhat implemented by self
control/priorty I don't think it can entirely be done so, and certainly not
just by the server.

Currently a browser that has just sent a 64KB header is free to immediately
send another 64KB of data without any say so from the server.  This will
effectively give that client twice the share of any shared connections from
load balancer to app server vs a browser that sends small headers.

If the header size was included in the flow control calculations, then the
browser would have to wait for a WINDOW_UPDATE after sending a 64KB header
before it could send any data.   It would thus be only able to take the
same share of any amalgamated connection as a small header sender.

cheers

---

From: David Krauss <[email-masked]>
Date: Fri, 4 Jul 2014 15:18:22 +0800

On 2014–07–04, at 1:54 PM, Greg Wilkins <[email-masked]> wrote:

> Whilst what I'm advocating could be somewhat implemented by self control/priorty I don't think it can entirely be done so, and certainly not just by the server.
> 
> Currently a browser that has just sent a 64KB header is free to immediately send another 64KB of data without any say so from the server.  This will effectively give that client twice the share of any shared connections from load balancer to app server vs a browser that sends small headers.

Too many requests is exactly what ENHANCE_YOUR_CALM is for. It’s not supposed to happen under normal conditions, i.e. it’s an indication of trouble.

> If the header size was included in the flow control calculations, then the browser would have to wait for a WINDOW_UPDATE after sending a 64KB header before it could send any data.   It would thus be only able to take the same share of any amalgamated connection as a small header sender.

TCP window updates do the job you’re asking for. The load balancer should simply delay reading the delinquent socket at the time it sends or forwards ENHANCE_YOUR_CALM.

To be sure, this sucks for other users sharing the same client-side proxy (assuming it aggregates connections). But that raises doubt of whether equal sharing of server resources is a useful goal in the first place.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 4 Jul 2014 17:41:02 +1000

David,

I think we are talking at cross purposes!

I'm not talking about situations that do not happen under normal
conditions.   For better or for worse, large cookies and big kerberos
tickets are normal.

Just sending a 32KB header is not sufficient reason to be flow controlled
nor be rejected with a 420 EYC style response.   If the server supports
large headers, then this is perfectly allowable. I'm  just saying that it
should move you 32KB closer to being flow controlled (at least for frames
that can be flow controlled).

cheers

On 4 July 2014 17:18, David Krauss <[email-masked]> wrote:

> > Currently a browser that has just sent a 64KB header is free to
> immediately send another 64KB of data without any say so from the server.
>  This will effectively give that client twice the share of any shared
> connections from load balancer to app server vs a browser that sends small
> headers.
>
> Too many requests is exactly what ENHANCE_YOUR_CALM is for. It’s not
> supposed to happen under normal conditions, i.e. it’s an indication of
> trouble.
>
> > If the header size was included in the flow control calculations, then
> the browser would have to wait for a WINDOW_UPDATE after sending a 64KB
> header before it could send any data.   It would thus be only able to take
> the same share of any amalgamated connection as a small header sender.
>
> TCP window updates do the job you’re asking for. The load balancer should
> simply delay reading the delinquent socket at the time it sends or forwards
> ENHANCE_YOUR_CALM.
>
> To be sure, this sucks for other users sharing the same client-side proxy
> (assuming it aggregates connections). But that raises doubt of whether
> equal sharing of server resources is a useful goal in the first place.
>
>

---

From: David Krauss <[email-masked]>
Date: Fri, 4 Jul 2014 16:07:46 +0800

On 2014–07–04, at 3:41 PM, Greg Wilkins <[email-masked]> wrote:

> I'm not talking about situations that do not happen under normal conditions.   For better or for worse, large cookies and big kerberos tickets are normal.   
> 
> Just sending a 32KB header is not sufficient reason to be flow controlled nor be rejected with a 420 EYC style response.   If the server supports large headers, then this is perfectly allowable. I'm  just saying that it should move you 32KB closer to being flow controlled (at least for frames that can be flow controlled).

I’m not saying that should happen either. 32 KB closer to EYC is the same as 32 KB closer to a request flow control limit. However, the resource demands of requests aren’t the same as the resource demands of data streams. 32 KB containing 100,000 GETs generating 404’s is certainly EYC worthy. One 64 KB kerberos ticket is probably not.

The purpose of flow control is to allow legitimate requests to arrive faster. Telling the user that they need to wait before requesting a page, because some PUT data is still on the wire, doesn’t make a lot of sense. Measuring requests by the kilobyte is probably overlooking what the server is really doing.

In the case where legitimate headers really max out the bandwidth, that implies a lot of short-lived streams with no bodies. Raw rate limiting is already pretty much the best you can do, no?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 04 Jul 2014 08:10:51 +0000

In message <[email-masked]>
, Greg Wilkins writes:
>--089e01419ffaa7ff4b04fd543893
>

Forcing them to put it all into one HEADERS frame would be mildly
discouraging because they might bang against the default 16KB
limit.

If they feel that large HEADERS frames hurt multiplexing, that would
discourage them too.

>not mean that they get 50% more fair share than a stream that has no

But does the mitigation of this issue belong on the request side of
things ?  Isn't that something the server should rule by scheduling
the responses ?

How often do we have the congestion on the request side of things
under non-DoS conditions in the first place ?

I think flow-control should work this way:

A) Make all frames subject to the connection window so that any
   node can balance multiple competing connections against each other.

B) Make all but the first frame (ie: HEADERS or PUSH_PROMISE) on a
   stream subject to stream windows, to balance different streams
   against each other.

C) Cap the size of the first frame on streams with SETTINGS.
   All implementations must be configurable to a value of 16383
   and values less than 256 are illegal.