Subject: HTTP/2 re-sync connection level flow control?

From: "Graham King" <[email-masked]>
Date: Wed, 13 Feb 2019 23:47:40 -0500

Given a range of HTTP/2 implementations with their bugs, edge cases, varied interpretations and understandings of the spec, not to mentioned network issues, the connection-level flow control values could get out of sync between client and server. 

Once client and server get out of sync, there is no mechanism for them to re-sync. The WINDOW_UPDATE values are always relative (increment). If the client reaches a window of 0 it will block. With enough deployments, and sufficiently long running connections, it seems almost inevitable that some will block because of this.

This happened in Go 1.11. DATA frames sent after a stream close weren't being counted towards flow control on the server side, but were on the client side (It is fixed in Go 1.12): https://github.com/golang/go/issues/28204

In QUIC (at least at some point, I couldn't find this in HTTP/3 spec) the sender sends a BLOCKED frame when it's window reaches 0. QUIC also includes a flow control value in RST frames to permit resync. https://docs.google.com/document/d/1F2YfdDXKpy20WVKJueEf4abn_LVZHhMUMS5gX6Pgjl4/edit#heading=h.i0f2zh9yfysp

Should HTTP/2 include a means for client and server to resync absolute connection level flow control values?

---

From: Mike Bishop <[email-masked]>
Date: Thu, 14 Feb 2019 18:23:08 +0000

I was initially mystified by how this would happen in HTTP/2.  The frames always arrive in the same order, so if they're all processed correctly, there should never be a disagreement about what the actual allowed amount is.  Presumably the disagreement is about which sent bytes count.

TL;DR of the bug you reference: an implementation bug was sending additional DATA frames on an already-closed stream.  Per RFC7540, that's prohibited, but only fatal to the stream.  This exposed a different bug, in which connection flow control updates were not sent to replenish the flow control consumed by incoming DATA frames once a RST_STREAM had been sent.  The merged fix in Go is to always return the relevant connection flow control credit along with the RST_STREAM.

You don't see it in HTTP/3 for a couple of reasons.  First, flow control (and therefore BLOCKED) are a transport-level function.  Because there's no fixed ordering, MAX_DATA frames (the equivalent of WINDOW_UPDATE for stream 0) always carry an absolute value, so implementations can't get out of sync about what the allowed window is -- or rather, the consumer will always lag behind the issuer due to transmission delay, which is always safe.  You also don't get out of sync on the number of bytes sent -- or at least, if you do, it's immediately obvious.  A RST_STREAM tells you how many bytes were sent on the stream (the "final size"), and subsequently exceeding the final size is a connection-fatal error.

---

From: "Graham King" <[email-masked]>
Date: Fri, 15 Feb 2019 12:55:31 -0500

> I was initially mystified by how this would happen in HTTP/2.  The frames always arrive in the same order, so if they're all processed correctly, there should never be a disagreement about what the…

Yes, a disagreement on which bytes to count, possibly by accident. Imagine a client with an off-by-one error. On every frame the client thinks it sent one more byte than the server thinks. That would work correctly in a wide range of tests, and could easily go unnoticed in production for a while. When you do exhaust the flow control window, the only clue that there's a real problem is that you stay blocked forever.

Off-by-one mistakes are easy to make, but HTTP/2's relative flow control makes them hard to identify. The bug reports you'll get are "sometimes my upload gets stuck" or "I have to restart my browser every day because my email stops working"; many layers away from the problem, and easy to dismiss as networking issues.

In Go it was even harder to spot. The bug depended on how many DATA frames were in-flight when the stream was closed. The higher the latency, the faster you exhaust flow control. If a well-resourced, widely deployed project such as Go can have this bug over five minor versions (1.11.0 to 1.11.5 inclusive), it's easy to imagine other projects having similar (possibly undiagnosed) problems with flow control.

Relative flow-control is brittle. It is easy for HTTP/2 implementers to get wrong, very difficult to notice the mistake, and impossible to recover from during a connection. My goal in writing is to ask if some adjustments can be made to the protocol to eliminate this class of errors.

---

From: Mike Bishop <[email-masked]>
Date: Fri, 15 Feb 2019 19:39:53 +0000

Connection level echoing is tough, because of synchronization.  Sure, it's easy to write an extension that periodically declares how many bytes you think the peer has used -- but as of when?  The peer has more bytes in flight, so you need a sync marker to do that.  For that, you'd need a frame that says "what do you think I've used so far?" and gets an answer back for that exact point in the bytestream.  Complicated, for some processing models.  Probably better to do it stream-by-stream and just trust that both endpoints can sum all the streams correctly.

Sending on a closed stream won't work in H3, but an extension could do it in H2 by arrangement.  Imagine an extension frame emitted by receivers in response to a stream (half-)closing which declares how many flow-controlled bytes they received.  When the sender gets it back, it blows up immediately if the number doesn't match.  You'd need both peers to support it, obviously, and you couldn't send it until you know the peer supports the extension (otherwise you'd trigger STREAM_CLOSED or PROTOCOL_ERROR, per 5.1 of RFC7540).

---

From: Stefan Eissing <[email-masked]>
Date: Mon, 18 Feb 2019 10:17:25 +0100

Regarding detectoring about the state at the other end, there was a draft written
by Cory Benfield and Brad Fitzpatrick: https://tools.ietf.org/html/draft-benfield-http2-debug-state-00
based on a state handler idea that was implemented in Apache's h2.

It exposes h2 state description of the very connection it receives the request on. It does not, as Mike noted, solve the problem of synchronicity. But maybe it helps in analyzing mismatched assumptions.

As to the original question: I doubt that adding a "resync" will help interoperability. If connection state goes out of whack, it is due to bugs in the implementation of the protocol (e.g. discarding frames without counting them in flow control). Adding a 'resync' handshake will add complexity to the connection state - not simplify it, I believe.

- Stefan

> Am 15.02.2019 um 20:39 schrieb Mike Bishop <[email-masked]>:
>

---

From: Cory Benfield <[email-masked]>
Date: Mon, 18 Feb 2019 11:38:17 +0000

> As to the original question: I doubt that adding a "resync" will help interoperability. If connection state goes out of whack, it is due to bugs in the implementation of the protocol (e.g. discardi…

Agreed, I'm opposed to a resync. Fundamentally this is an
implementation bug: flow controlled frames are *always* flow
controlled, and must have their flow control state modification made
before they are checked for validity. This is the same as HEADERS
frames: you must always attempt to decode the body of the HEADERS
frame unless the frame is so ill-formed that it is not parseable and
leads to immediate connection error of type PROTOCOL_ERROR.

---

From: Mike Bishop <[email-masked]>
Date: Tue, 26 Feb 2019 00:30:22 +0000

I agree that there's no value in trying to "correct" values on the fly -- if there's a bug, it should be fixed, not patched over.  But an extension that allows immediate detection of the bug (and blows up the connection) rather than letting it bog down and eventually stall the connection?  That seems like it could have value if it were broadly implemented.

-----Original Message-----
From: Cory Benfield <[email-masked]> 
Sent: Monday, February 18, 2019 3:38 AM
To: Stefan Eissing <[email-masked]>
Cc: Mike Bishop <[email-masked]>; Graham King <[email-masked]>; [email-masked]
Subject: Re: HTTP/2 re-sync connection level flow control?

> As to the original question: I doubt that adding a "resync" will help interoperability. If connection state goes out of whack, it is due to bugs in the implementation of the protocol (e.g. discardi…

Agreed, I'm opposed to a resync. Fundamentally this is an implementation bug: flow controlled frames are *always* flow controlled, and must have their flow control state modification made before they are checked for validity. This is the same as HEADERS
frames: you must always attempt to decode the body of the HEADERS frame unless the frame is so ill-formed that it is not parseable and leads to immediate connection error of type PROTOCOL_ERROR.

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 26 Feb 2019 09:54:04 +0100

> Am 26.02.2019 um 01:30 schrieb Mike Bishop <[email-masked]>:
> 

I would implement such a thing in Apache, given that at least one client does as well.