Subject: Post before acking settings?

From: Jesse Wilson <[email-masked]>
Date: Wed, 27 Apr 2016 02:22:14 +0000

In OkHttp we’re investigating a bug
<https://github.com/square/okhttp/issues/2506> where nginx v1.9.15 sends us
REFUSED_STREAM when we send our HEADERS and DATA frames before acking the
server’s SETTINGS.

Should a well behaved HTTP/2 client wait to ack the server’s SETTINGS
before sending other frames? The spec says not waiting is okay
<http://http2.github.io/http2-spec/index.html#rfc.section.3.5>:

To avoid unnecessary latency, clients are permitted to send additional
frames to the server immediately after sending the client connection
preface, without waiting to receive the server connection preface. It is
important to note, however, that the server connection preface SETTINGS
frame might include parameters that necessarily alter how a client is
expected to communicate with the server. Upon receiving the SETTINGS frame,
the client is expected to honor any parameters established.

I’d love advice on how to resolve this. My options as I see them:

   1. Change the client to ack SETTINGS before sending other frames.
   2. Petition HTTP/2 server implementors to not refuse streams that
   precede the SETTINGS ack.
   3. Recover when we get a REFUSED_STREAM.

Note that recovering from a REFUSED_STREAM isn’t an easy win for us. OkHttp
doesn’t retain the (potentially large, potentially sensitive) request body
bytes that were posted. To recover the application layer will be prompted
for new request body bytes. This is typically just serializing some JSON.
But it could be invasive, like prompting a user for their fingerprint so
that a device’s secure element can be used.

Thanks!
– Jesse
​

---

From: Martin Thomson <[email-masked]>
Date: Wed, 27 Apr 2016 13:25:07 +1000

On 27 April 2016 at 12:22, Jesse Wilson <[email-masked]> wrote:
> Should a well behaved HTTP/2 client wait to ack the server’s SETTINGS before
> sending other frames?

No.  The protocol needs to operate correctly without settings, and
blocking on the ack adds additional latency.

Though nginx is within its rights to refuse any stream; it should not
be using a lack of settings acknowledgment as the basis on which it
refuses.  Refusing in this context adds significant latency during a
time when extra latency costs a lot.

In the end, I think that nginx might be at fault here.  However, while
I get that the complexity of retry is awful, it's probably the nicest
overall outcome for you, since you are going to encounter
`REFUSED_STREAM` in future.  Consider also the other situations where
retry helps, like GOAWAY:
https://http2.github.io/http2-spec/index.html#Reliability

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 27 Apr 2016 16:28:59 +1200

On 27/04/2016 3:25 p.m., Martin Thomson wrote:
> On 27 April 2016 at 12:22, Jesse Wilson wrote:

My understanding was that endpoints should (MUST?) assume 100 streams
was permissiable up until the SETTINGS was received+ACK'd. After which
point whatever value was ACK'd is used as the limit.

If the limit was lower than the default 100 some streams the excess may
be delayed on responding or get RST_STREAM with corresponding re-try
being needed when space opened up again (possibly on other connections
for faster total latency).

> https://http2.github.io/http2-spec/index.html#Reliability

Nod.

Amos

---

From: Martin Thomson <[email-masked]>
Date: Wed, 27 Apr 2016 14:49:54 +1000

> for faster total latency).

The concurrent stream limit is actually unbounded in the first
flights.  But unless you have special magic, you won't be able to pack
that many requests into the connection before hitting the TCP
congestion window.  It might be more than 100 though.

In practice, you are right.  If you go crazy, you risk getting a RST_STREAM.

---

From: Jesse Wilson <[email-masked]>
Date: Wed, 27 Apr 2016 05:08:31 +0000

Thanks for your responses. I’ve opened an issue asking nginx to implement
the change.
https://trac.nginx.org/nginx/ticket/959

---

From: Patrick McManus <[email-masked]>
Date: Wed, 27 Apr 2016 10:04:30 -0400

On Tue, Apr 26, 2016 at 10:22 PM, Jesse Wilson <[email-masked]> wrote:

> - Change the client to ack SETTINGS before sending other frames.

arms race opportunity! Why wait for the SETTINGS to ack it? :)

---

From: Martin Thomson <[email-masked]>
Date: Thu, 28 Apr 2016 00:34:10 +1000

> arms race opportunity! Why wait for the SETTINGS to ack it? :)

Careful!  Or h3 will have a nonce that you have to echo. ;)

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 28 Apr 2016 02:57:07 +1200

On 28/04/2016 2:34 a.m., Martin Thomson wrote:
> On 28 April 2016 at 00:04, Patrick McManus wrote:

Actually on that line of thought ....

  while I was working on implementing SETTINGS in Squid I ran into a bit
of trouble. The SETTINGS can be sent multiple times with different
values. This ACK in its current form makes HTTP/2 fully reliant on the
reliability of the transport to maintain sync and a 1:1 full RTT for
each SETTINGS frame used.
 It would actually be a bit more optimal to aggregate multiple SETTINGS
and send back in the ACK a copy of the last values being ACK'd so the
sender can know what state the other end is up to receiving. (eg. TCP
seqno ACK).

I hope to experiment with that a little more for between-proxy
connections in Squid after we get h2/h2c implementation actually
finished and rolled out.

Amos

---

From: Martin Thomson <[email-masked]>
Date: Thu, 28 Apr 2016 01:23:56 +1000

> sender can know what state the other end is up to receiving. (eg. TCP

A hash might be cheaper.  A nonce is cheaper still.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 27 Apr 2016 17:41:15 +0000

However, the transport is *not* unordered or unreliable, so I don't think this is actually a weakness *of HTTP/2*.  If we were to use a different transport that was, we'd also be defining a new mapping which would need to incorporate some strategy like this.

I had envisioned a sequence number.  If we're saying that the underlying transport might be unordered/unreliable, then the server just ignores any SETTINGS that's earlier than the highest sequence number received, and ACKs any newly-adopted ones.

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Wednesday, April 27, 2016 8:24 AM
To: Amos Jeffries <[email-masked]>
Cc: HTTP Working Group <[email-masked]>
Subject: Re: Post before acking settings?

On 28 April 2016 at 00:57, Amos Jeffries <[email-masked]> wrote:
>  It would actually be a bit more optimal to aggregate multiple 
> SETTINGS and send back in the ACK a copy of the last values being 
> ACK'd so the sender can know what state the other end is up to 
> receiving. (eg. TCP seqno ACK).

---

From: Jesse Wilson <[email-masked]>
Date: Thu, 28 Apr 2016 01:46:02 +0000

One of the nginx maintainers responded
<https://trac.nginx.org/nginx/ticket/959> to my bug report and explained
how it uses flow control windows:

“The request body cannot be read and processed by nginx until some module
asks for it. To prevent the client from sending the request body that
hasn’t been asked yet, the stream’s window is kept zero.”

As I understand this, nginx sets the default window size to 0 for all
streams. The flow control window isn’t expanded from 0 until nginx finds a
module to accept the request. In terms of round trips this seems like a
regression from HTTP/1.1:

HTTP/1.1 connection
→ client sends request headers + body
← server responds with response headers + body

HTTP/2 connection with a default stream window size of 0
→ client sends request headers
← server expands flow control window
→ client sends request body
← server responds with response headers and response body

This is a surprising and disappointing regression for the common case, in
which the server accepts the request body. It’s particularly problematic
for HTTP APIs because they usually include request bodies.

I’ll continue to petition the nginx maintainers to optimistically buffer a
small incoming request body. Buffering at most 64 KiB per request would
save a network roundtrip! I think that’s an inexpensive and non-complicated
way for them to improve performance.

– Jesse
​

---

From: Martin Thomson <[email-masked]>
Date: Thu, 28 Apr 2016 18:00:03 +1000

> regression from HTTP/1.1:

Yes, think of this as equivalent to forcing 100-continue on every
request.  That has the effect of discouraging requests with bodies for
performance reasons.

This is definitely unfortunate from the perspective of performance,
but it's an understandable position.  What a server is _expected_ to
do is buffer up to the connection-level flow control window.

However, it might be that 64k per connection (plus request header
fields) is too much for some servers and RST_STREAM is the only
option.  I would hope that only constrained devices do that, but - as
Roberto is fond of pointing out - servers are effectively constrained
devices.

A better solution all around would be to set a small connection-level
flow control window and to allow that much data to be buffered.  That
would allow requests that fit into that window to avoid the round trip
tax.

(For the galleries, the SETTINGS ack thing was a red herring, nginx
rejects requests with bodies.  Which of course you won't send once the
SETTINGS are applied.)