Subject: Streams after receiving GOAWAY

From: Glen Knowles <[email-masked]>
Date: Sun, 15 Nov 2015 02:24:15 -0800

I'm about to deliberately violate the letter of a MUST NOT because I
believe it's overly strict.

"Receivers of a GOAWAY frame MUST NOT open additional streams on the
connection..."

When my server wants to close a connection it sends a GOAWAY with last
stream id set to a value somewhat higher than anything it has received. It
then keeps the connection until the reported last id is reached or enough
time goes by.

When the client gets a GOAWAY it will immediately start establishing a new
connection, continue issuing new requests up to the reported last id, and
close the old connection when it either has a new connection or has used
all the reported streams.

The goal is to avoid suspending requests in a high volume server to server
environment while waiting for new connections. I don't see how it conflicts
with any conforming implementation, am I missing something?

Thanks,
Glen

---

From: Cory Benfield <[email-masked]>
Date: Sun, 15 Nov 2015 10:52:03 +0000

> On 15 Nov 2015, at 10:24, Glen Knowles <[email-masked]> wrote:
> 
> I'm about to deliberately violate the letter of a MUST NOT because I believe it's overly strict.
> 
> "Receivers of a GOAWAY frame MUST NOT open additional streams on the connection..."
> 
> When the client gets a GOAWAY it will immediately start establishing a new connection, continue issuing new requests up to the reported last id, and close the old connection when it either has a new connection or has used all the reported streams.
> 
> The goal is to avoid suspending requests in a high volume server to server environment while waiting for new connections. I don't see how it conflicts with any conforming implementation, am I missing something?

The client conflicts with *any* conforming implementation because it ignores the section of RFC 7540 that you just quoted. You’re not just violating the letter of that MUST NOT, you’re violating the spirit of it too, which is indicated in the first paragraph of that section of the spec:

> GOAWAY allows an endpoint to gracefully stop accepting new streams while still finishing processing of previously established streams.

GOAWAY is not intended as a mechanism for saying “I will process no more than n further streams”. If it were, it would be totally acceptable to open all connections with the preamble, followed immediately by GOAWAY(last_stream_id=100). This is not the intended use of GOAWAY.

Certainly my Python implementation would reject your client’s traffic, and in this regard I believe it is conformant.

Cory

---

From: Patrick McManus <[email-masked]>
Date: Sun, 15 Nov 2015 10:47:04 -0500

> then keeps the connection until the reported last id is reached or enough

That's a bug for your server. When you send GoAway(last=N) as a server, the
client knows that streams >N can be safely replayed in another session -
they were not accepted by the server. When you send GoAway(last=N+C) (with
C as a buffer) and you continue to process some but not all of C based on
"enough time going by" the client loses this information.. So that will
lead to interop bugs and unsafe behavior.

If you're interested in breaking the connections for reasons of idle, this
shouldn't be an issue. If you're interested in migrating load gracefully
then you might take a look at alt-svc which is meant to cover that case by
setting up an alternative connection asynchronously.

hth

-P

---

From: Glen Knowles <[email-masked]>
Date: Sun, 15 Nov 2015 10:48:32 -0800

On Sun, Nov 15, 2015 at 2:52 AM, Cory Benfield <[email-masked]> wrote:

>
> GOAWAY is not intended as a mechanism for saying “I will process no more
> than n further streams”.

But that is exactly a stated purpose of GOAWAY(noerror):
"To deal with this case, the GOAWAY contains the stream identifier of the
last peer-initiated stream that was or might be processed on the sending
endpoint in this connection."

> If it were, it would be totally acceptable to open all connections with
> the preamble, followed immediately by GOAWAY(last_stream_id=100). This is
> not the intended use of GOAWAY.
>
> Reinterpreting last_stream_id to mean "last I will send" instead of "last
of your streams I may process" would be clearly wrong, and is not what I'm
doing. I'm merely giving the listener the option to widen the existing
"inherent race condition between an endpoint starting new streams and the
remote sending a GOAWAY frame."

If your server sends a GOAWAY(no error) with the last stream id set to the
last one it's processed then my client will not open any more streams on
that connection. It will only continue to open additional streams if the
last stream you report is "in the future". I see how this could be a
problem if the server doesn't set the last stream id according to the spec,
but otherwise?

---

From: Cory Benfield <[email-masked]>
Date: Sun, 15 Nov 2015 21:25:43 +0000

> On 15 Nov 2015, at 18:48, Glen Knowles <[email-masked]> wrote:
> 
> GOAWAY is not intended as a mechanism for saying “I will process no more than n further streams”.
> 
> "To deal with this case, the GOAWAY contains the stream identifier of the last peer-initiated stream that was or might be processed on the sending endpoint in this connection.”

This purpose is subordinate to the normative language in the section, which really clearly says that a peer that has received a GOAWAY frame MUST NOT open new frames. This normative language overrides the language in the section you quoted.

The way to reconcile these things is that the last_stream_id flag indicates the sending peer’s best guess about what information the application has acted on. The server promises that for any stream id > last_stream_id, no action has been taken: for any other stream, they’re potentially in a state of limbo where some action may or may not have been taken. It is not a license to continue to open new streams up to last_stream_id, and I doubt any server would accept you doing so.

> If it were, it would be totally acceptable to open all connections with the preamble, followed immediately by GOAWAY(last_stream_id=100). This is not the intended use of GOAWAY.
> 
> Reinterpreting last_stream_id to mean "last I will send" instead of "last of your streams I may process" would be clearly wrong, and is not what I'm doing. I'm merely giving the listener the option to widen the existing "inherent race condition between an endpoint starting new streams and the remote sending a GOAWAY frame.”

I apologise, I wasn’t clear enough. You could have had this flow:

Client: PREAMBLE + SETTINGS
Server: SETTINGS + SETTINGS(ACK)
Client: SETTINGS(ACK)
Server: GOAWAY(last_stream_id=101)

This is *exactly* the flow you’re proposing, taken to an extreme: here, the server is saying that the last of the client streams streams it may process is stream ID 101 (put another way, the server promises to serve 51 streams opened by the client). This is not at all the intended semantic of the GOAWAY frame, and using it like this is totally misleading. Any client that complies with RFC 7540 will see this GOAWAY and immediately refuse to open further streams.

Your client is not widening the existing race condition. That race condition applies to when the GOAWAY frame is in flight between client and server. Your client is essentially pretending that it did not receive the GOAWAY frame, deferring receipt of it until some later time.

In the above example, the server is acting incorrectly (last_stream_id does not correspond to a stream it has even seen, let alone begun to process). However, if the client follows up on this by sending a HEADERS frame, it is directly violating the normative language of RFC 7540.

> If your server sends a GOAWAY(no error) with the last stream id set to the last one it's processed then my client will not open any more streams on that connection. It will only continue to open additional streams if the last stream you report is "in the future". I see how this could be a problem if the server doesn't set the last stream id according to the spec, but otherwise?

It’s a problem because RFC 7540 quite clearly says "Receivers of a GOAWAY frame MUST NOT open additional streams on the connection”. This is not MAY, or even SHOULD NOT, it’s MUST NOT. Let me quote from RFC 2119[0], which defines the normative language of IETF drafts:

> MUST NOT   This phrase, or the phrase "SHALL NOT", mean that the definition is an absolute prohibition of the specification.

RFC 7540 says that a client MUST NOT open further streams once it has received a GOAWAY frame. This is not negotiable, and any peer that does so is not a compliant HTTP/2 peer. It is reasonably to discuss whether the semantics of GOAWAY should be extended to allow what you’re proposing here, or to propose an extension that allows this behaviour. However, if you ignore this MUST NOT in favour of non-normative language elsewhere in the specification, you are not building a HTTP/2 peer.

[0]: https://www.ietf.org/rfc/rfc2119.txt

---

From: Glen Knowles <[email-masked]>
Date: Sun, 15 Nov 2015 15:20:22 -0800

>Any client that complies with RFC 7540 will see this GOAWAY and
immediately
>refuse to open further streams.

Which is a correct outcome, since the server wants to gracefully close the
connection, otherwise it wouldn't've sent the GOAWAY(no_error).

>In the above example, the server is acting incorrectly (last_stream_id
does
>not correspond to a stream it has even seen, let alone begun to process).

I get the feeling that you're not familiar with the graceful shutdown
language, or perhaps I haven't been clear.

"A server that is attempting to gracefully shut down a
connection SHOULD send an initial GOAWAY frame with the last stream
identifier set to 2^31-1 and a NO_ERROR code.  This signals to the
client that a shutdown is imminent and that initiating further
requests is prohibited.  After allowing time for any in-flight stream
creation (at least one round-trip time), the server can send another
GOAWAY frame with an updated last stream identifier."

> MUST NOT   This phrase, or the phrase "SHALL NOT", mean that the
definition
is an absolute prohibition of the specification.

I'm well aware of what a normative MUST NOT means, I'm arguing that it's
use
in this case is to strong (or to general) - in no small part because
violating
it is undetectable by the peer.

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 16 Nov 2015 09:34:11 +1000

> GOAWAY frame with an updated last stream identifier."
​..."initiating further requests is prohibited."

This section means that 2^31-1 is a magic number, and if you (the client)
get it in your GOAWAY, you might then get another GOAWAY with a real
last-stream-id, once the server is actually done receiving streams. It
doesn't mean the client can ignore the first GOAWAY once it sees it.

> it is undetectable by the peer.
​What's to detect? It's a signal: if you get a GOAWAY with a
last-stream-id, you can safely retry any message with a greater stream id,
even if it's non-idempotent.

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Eitan Adler <[email-masked]>
Date: Sun, 15 Nov 2015 18:32:47 -0800

On 15 November 2015 at 15:20, Glen Knowles <[email-masked]> wrote:
> I'm well aware of what a normative MUST NOT means, I'm arguing that it's use

Just because a behavior is undetectable to a peer doesn't it mean it
is, or ought to be, allowed.

-- 
Eitan Adler

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 16 Nov 2015 07:33:33 +0100

> it is undetectable by the peer.

It *is* detectable through the fact that it advertises a stream ID
that has possibly not yet been sent by the client and is not the
2^31-1 magic number either.

That said, I do think that there are valid use cases for what you're
trying to do and that it could have been planned from the start, had
anyone thought about it by then.

I really think you should write a proposal for an extension. This is
a perfect use case for extensions, it just requires the two peers to
agree with it and it does not affect the framing at all. Maybe your
extension will become the default mode over time, just like everyone
supports SACK, window scaling, and timetamps on top of TCP nowadays.

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 16 Nov 2015 19:34:44 +1300

> it is undetectable by the peer.

Lets put it another way.

* GOAWAY is used to signal error conditions.

If no such stream ID was ever sent by this client, and the ID is not the
special case 2^31-1 value; then the frame interpretation state has
apparently been corrupted somehow on the server.

The client now authoritatively cannot trust _anything_ seen on that
connection. Its best course of action is termination of that corrupted
connection. Not a graceful shutdown. Immediate termination.

Amos

---

From: Stefan Eissing <[email-masked]>
Date: Mon, 16 Nov 2015 12:49:43 +0100

Maybe a better use of the protocol for the described case is to
change the SETTINGS_MAX_CONCURRENT_STREAMS to a lower and lower
value.

//Stefan

> Am 16.11.2015 um 07:34 schrieb Amos Jeffries <[email-masked]>:
>

---

From: Mike Bishop <[email-masked]>
Date: Mon, 16 Nov 2015 17:44:07 +0000

This was discussed a while ago on-list.  The server wants to stop the client from initiating new streams (the first GOAWAY) without killing requests that it may already have in flight.  The server will send a subsequent GOAWAY informing the client of the actual last stream processed.  This is behavior by a server that wants to be less strict – that doesn’t mean clients should abuse it.

There was discussion of an extension to define a DRAINING frame or setting, which is a hint to the client that a GOAWAY will be coming soon, so it may want to spin up a failover connection.  That extension was never actually defined, but if you have a use case in which it would help, write a draft….

From: Patrick McManus [mailto:[email-masked]]
Sent: Sunday, November 15, 2015 7:47 AM
To: Glen Knowles <[email-masked]>
Cc: HTTP Working Group <[email-masked]>
Subject: Re: Streams after receiving GOAWAY

On Sun, Nov 15, 2015 at 5:24 AM, Glen Knowles <[email-masked]<mailto:[email-masked]>> wrote:
When my server wants to close a connection it sends a GOAWAY with last stream id set to a value somewhat higher than anything it has received. It then keeps the connection until the reported last id is reached or enough time goes by.

That's a bug for your server. When you send GoAway(last=N) as a server, the client knows that streams >N can be safely replayed in another session - they were not accepted by the server. When you send GoAway(last=N+C) (with C as a buffer) and you continue to process some but not all of C based on "enough time going by" the client loses this information.. So that will lead to interop bugs and unsafe behavior.
If you're interested in breaking the connections for reasons of idle, this shouldn't be an issue. If you're interested in migrating load gracefully then you might take a look at alt-svc which is meant to cover that case by setting up an alternative connection asynchronously.

hth
-P