Subject: h2 proxy and connection flow control

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Thu, 21 Aug 2014 22:39:06 +0900

I'm not sure this is already discussed, but I found a case where
stalling single stream will stall all streams in entire session
when a h2 proxy is involved while playing with h2 proxy.

Here is an example setup of the scenario:

+---------+            +----------+            +---------+
|         |+---------->| Frontend |+---------->|         |
| Browser |            | Proxy    |            | Backend |
|         |<----------+|          |<-----------|         |
+---------+            +----------+            +---------+

Browser, Frontend Proxy and Backend server all speak h2.  There
is 1 TCP connection between Browser and Frontend Proxy.  Also
there is 1 TCP connection between Frontend Proxy and Backend.
These connections are independent h2 sessions.

Browser uses 10MB receive window for stream and 10MB for
connection.  Frontend Proxy uses 64KB receive window for stream
and 64KB for connection.

Browser sends requests to Frontend Proxy, and then Frontend Proxy
issues request to Backend and sends back its response to Browser.

Now suppose that Browser issues a request to Frontend Proxy (this
is denoted by stream A).  Frontend Proxy relays its request to
Backend (this is denoted by stream B).  Suppose that response
body is 40MB.  Then Frontend Proxy sends back response for stream
A to Browser but at some point Browser stops sending stream
WIDOW_UPDATE for some reason (e.g. it is a streaming media and it
is not consumed by media player).  Eventually, Browser's receive
window for stream becomes 0.

Since Browser's receive window is 0, Frontend Proxy cannot send
more DATA for stream A to Browser.  Meanwhile, Frontend Proxy
continues to receive DATA for stream B from Backend.  Since
stream A's stream window is 0, Frontend Proxy cannot proceed and
just buffer the received data and eventually Frontend Proxy's
receive window for connection becomes 0.

After this happens, Browser can still issue new request, but
Backend cannot send response to Frontend Proxy because its recive
window is 0.  This means that if Browser stalls one stream, it
stalls all streams in entire session.

+---------+            +----------+            +---------+
|         |+---------->| Frontend |+---------->|         |
| Browser |            | Proxy    |            | Backend |
|         |<----------+|          |<-----------|         |
+---------+     ^      +----------+     ^      +---------+
                |                       |
             stream A                all streams
            was stalled              was stalled
            due to stream            due to connection
           window is 0               window is 0

If a connection between Frontend Proxy and Backend is shared by
multiple clients, then things get worse, because if one client
stalls one stream, it affects all clients sharing the connection.

I think stalling stream by Browser is reasonable behaviour and
not a bug, because it just means that data is not consumed
yet. Or else, this is just another "Don't do that" situation?

How do you deal with this issue? (or is it not an issue because
I'm completely wrong?)

Here is my observation:

o Effectively disable connection level flow control in proxy
  -- this is controversial because connection flow control is
     effective especially for proxy to limit its memory
     commitment.

o Actively timeout stalled stream by proxy
  -- this may work, but incurs delay before timeout.

o Innovate flow control algorithm not to cause this situation
  -- I'm not sure how it looks like.

Best regards,

Tatsuhiro Tsujikawa

---

From: Patrick McManus <[email-masked]>
Date: Thu, 21 Aug 2014 09:59:37 -0400

> stalls all streams in entire session.

The proxy cannot receive more data because its resources (the session
window) are exhausted.

If it did receive more data where would it put it?

If it actually has room for more data then it should just increase its
session window with a window update.

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Thu, 21 Aug 2014 23:09:57 +0900

On Thu, Aug 21, 2014 at 10:59 PM, Patrick McManus <[email-masked]>
wrote:

>
> On Thu, Aug 21, 2014 at 9:39 AM, Tatsuhiro Tsujikawa <
> [email-masked]> wrote:
>
>>
>>
​Yes
​

> session window with a window update.
​In this scenario, proxy has no room to expand. Therefore I wrote that​ all
streams on backend connections are stalled.

Best regards,
Tatsuhiro Tsujikawa

---

From: Patrick McManus <[email-masked]>
Date: Thu, 21 Aug 2014 10:12:50 -0400

On Thu, Aug 21, 2014 at 10:09 AM, Tatsuhiro Tsujikawa <[email-masked]

>
>
>
> ​In this scenario, proxy has no room to expand. Therefore I wrote that​
> all streams on backend connections are stalled.
>
>
This is my point. If the proxy really has no room to expand (its out of
ram) - then we've got a resource problem not a protocol problem. If it does
have more ram, then just use the protocol to open the window to match the
resources available.

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Thu, 21 Aug 2014 23:56:07 +0900

On Thu, Aug 21, 2014 at 11:12 PM, Patrick McManus <[email-masked]>
wrote:

>
>
>
> On Thu, Aug 21, 2014 at 10:09 AM, Tatsuhiro Tsujikawa <
​So if there is no ram to spend for that connection, is stalling entire
connection inevitable consequence (and we should not care about it)? ​
Connection flow control is limit the memory commitment for proxy then why
do we need to extend it in the response to a client behavior?

Best regards,
Tatsuhiro Tsujikawa

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 22 Aug 2014 03:48:18 +1200

On 22/08/2014 2:56 a.m., Tatsuhiro Tsujikawa wrote:
> On Thu, Aug 21, 2014 at 11:12 PM, Patrick McManus wrote:
> 
>>
>>
>>
>> On Thu, Aug 21, 2014 at 10:09 AM, Tatsuhiro Tsujikawa wrote:
>>
>>>
>>>
>>>

For the browser connection the browser has explicitly indicated with its
matching stream and connection windows that it is happy for the other
streams to stall.
The proxy should not allow itself to stall the server connection. Namely
by reducing its window on all streams from that client as soon as it
detects the clients connection window getting low.

Proxies have to operate under the assumption that a number of other
clients and/or streams will be arriving very soon and need to use
resources. So no resource can be allocated beyond what can be passed on
quickly.

Amos

---

From: Martin Thomson <[email-masked]>
Date: Thu, 21 Aug 2014 11:18:01 -0700

On 21 August 2014 07:56, Tatsuhiro Tsujikawa <[email-masked]> wrote:
> So if there is no ram to spend for that connection, is stalling entire
> connection inevitable consequence (and we should not care about it)?
> Connection flow control is limit the memory commitment for proxy then why do
> we need to extend it in the response to a client behavior?

If the proxy wants to enable a measure of concurrency, setting a
stream flow control window that is lower than the session flow control
window makes sense.  A value that is lower by some multiple will
enable concurrency of ceil(multiple).  Obviously, low values are going
to produce suboptimal performance, so there is a trade-off.

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Fri, 22 Aug 2014 13:42:44 +0900

> resources. So no resource can be allocated beyond what can be passed on

This is great hint.  The failure I made is always update stream and
connection window at the same time. It should be updated independently. So
when Frontend Proxy sends data to Browser, it only sends connection level
WINDOW_UPDATE to Backend.  Stream level WINDOW_UPDATE is deferred until
Browser sends strean level WINDOW_UPDATE for data which is not acked.
Obviously this requires that Frontend Proxy's stream window is not larger
than Browser's.

Best regards,
Tatsuhiro Tsujikawa

---

From: Ilari Liusvaara <[email-masked]>
Date: Fri, 22 Aug 2014 17:49:40 +0300

> Obviously this requires that Frontend Proxy's stream window is not larger

What if client connection gets into state where connection window is zero,
but stream windows are nonzero?

If the stream windows on proxy are nonzero and the proxy connection window
is nonzero, those streams are candidates for sending for backend.

Except that the proxy can't forward that data (due to the zero connnection
window to the client).

And if such buffered data is substracted from proxy connection window, the
window can pretty quickly shrink to zero, stalling all streams.

Now, it isn't only the framing-layer connection window. There is TCP
connection window as well. So even if framing connection window is nonzero,
it may not be possible to send data.

100% buffering for all streams would obviously get around this, but that
could use lots of memory.

-Ilari

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Sat, 23 Aug 2014 00:35:58 +0900

On Fri, Aug 22, 2014 at 1:42 PM, Tatsuhiro Tsujikawa <[email-masked]>
wrote:

> ​​
Replying to myself, this does not work, because Browser may not send
WINDOW_UPDATE if its stream window is very large (e.g. 10M) compared to
Proxies (e.g., 64K).  Most of the http2 implementation seems to send
WINDOW_UPDATE when window is reduced its half, so sending 64K data does not
make Browser send WINDOW_UPDATE...
​
​Best regards,
Tatsuhiro Tsujikawa

---

From: Martin Thomson <[email-masked]>
Date: Fri, 22 Aug 2014 10:59:16 -0700

> make Browser send WINDOW_UPDATE...

:)

You send WINDOW_UPDATE when you a) have space, and b) think that it is
necessary in order to ensure maximal data flow.