Subject: GOAWAY clarification

From: Martin Thomson <[email-masked]>
Date: Fri, 20 Mar 2015 22:53:09 -0700

This seems right to me...

@ https://github.com/http2/http2-spec/issues/731
---
We are working on a HTTP/2 library and we are a bit confused and we
would like to get it right. That's why I am kindly asking for you to
answer the following question:

Is it correct that after a sender sent a GOAWAY frame with some last
stream identifier to a receiver, the sender may still open new
streams, even with stream identifiers higher than the last stream
identifier sent in the GOAWAY frame? That is the GOAWAY frame and the
last stream identifier only limit the ability of the receiver?

If this is correct, and I believe it is, then the I find the following
sentence in the first paragraph at 6.8 GOAWAY confusing.

    Once sent, the sender will ignore frames sent on any new streams
with identifiers higher than the included last stream identifier.

In my opinion it should say something of the like

    Once sent, the sender will ignore frames sent on any new streams
created by the receiver with identifiers higher than the included last
stream identifier.

Thank you very much!
---

After all, the purpose is to create a synchronization point for
streams created by a peer, marking some set of them as safe to retry.
There is little value in delineating streams created by the sender of
GOAWAY.

That has some ramifications, that I'm sure everyone is already aware
of: a server that has decided to process a client frame might
reasonably initiate server push after sending GOAWAY.  Equally, a
client might hang around after GOAWAY and even make more requests, but
server pushes would be "ignored".

Of course, given that interpretation GOAWAY from a client could be
seen as equivalent to SETTINGS_ENABLE_PUSH=0, depending on the value
of the last stream ID provided, that is.

I'm not sure that these are new ramifications.  The question is
whether we want to somehow prevent any aspect of this interpretation.
At this extremely late stage in the process that would require
something extraordinary, but I offer the opportunity nonetheless.

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 21 Mar 2015 20:12:27 +1300

> with identifiers higher than the included last stream identifier.

Translation: "go away, Im ignoring *all* future streams after N"

> created by the receiver with identifiers higher than the included last

Translation: "I am going to ignore you now while I continue making new
streams to fill your pipe ..."

The behaviour is very different and IMHO the proposal allows senders to
violate the intent of the GOAWAY.

> There is little value in delineating streams created by the sender of

Yes, however one must take into account what type of action the
synchronisation was done for / is signalling.

Its pretty clear that the intent of the GOAWAY is to signal connection
closure gracefully. There is no sense in starting new streams after
having told the other end you are about to *terminate the connection*.

> server pushes would be "ignored".

Currently its a "might" not "must".

If the proposed alteration happens it becomes a must, because until the
sender of GOAWAY actually completes sending the last possible stream-ID
from the full set of 2^32-1, the recipient of GOAWAY cant be sure its
not going to try sending "just one more".

> something extraordinary, but I offer the opportunity nonetheless.

If a PUSH_PROMISE was sent *before* sending GOAWAY, then the sender is
free to complete it under the existing semantics.

If it was sent *after* the GOAWAY then the remote end is already in the
process of discarding state and terminating the connection. There is no
guarantee that any of it will be handled, so why would a sane sender
bother wasting bandwidth on new streams it can expect to be dropped?

Amos

---

From: Martin Thomson <[email-masked]>
Date: Sat, 21 Mar 2015 09:35:45 -0700

It would be easy to deal with your concern by having the receiver of
the GOAWAY reply with their own.  I think that avoids all of the
problems you indicate.

---

From: Martin Thomson <[email-masked]>
Date: Sat, 21 Mar 2015 19:27:23 -0700

> the GOAWAY reply with their own.  I think that avoids all of the

So @Scottmitch also notes a further bug here.  We currently prohibit
the creation of more streams after GOAWAY, which is in direct
contradiction to the graceful shutdown process.

    Receivers of a GOAWAY frame MUST NOT open additional
    streams on the connection, although a new connection can be
    established for new streams.

That contradicts the guidance we provide later in the section
regarding graceful shutdown.  It prevents a seamless transition from
one connection to another.

I've created a PR for this.  https://github.com/http2/http2-spec/pull/732

I've also taken the liberty of taking a variation on the text from @buchgr.

I think that this is erratum-worthy, so I'd like to get this in.  But
I won't do so if there are objections.  If my answer to Amos'
objection didn't satisfy you (see above; see also the PR text; Amos?)
then I can remove the second part of the change, but I tend to think
that it's more consistent with the other fix.

---

From: Martin Thomson <[email-masked]>
Date: Sun, 22 Mar 2015 08:51:03 -0700

After sleeping on it, I have taken a more thorough look at the section
and noted a few other inconsistencies.  Thus, I've made a second
proposal that is a little more thorough:

https://github.com/http2/http2-spec/pull/733

One note regarding this text, since this has already come up on github...

This shouldn't change behaviour.  If you have an implementation that
sends GOAWAY based on the stream identifiers you have seen, then you
are exposed to the "bug" in issue #458, but are otherwise unaffected.
If you implemented the graceful shutdown based on the text produced
for #458; that is, you send two GOAWAY frames, then the only
consequence is that streams might have to be retried by the client.
Ultimately, the choice of last-stream-id will determine how much
allowance is made for imminent transactions.

---

From: Martin Thomson <[email-masked]>
Date: Sun, 22 Mar 2015 09:19:07 -0700

...and I've drawn some pictures:
https://docs.google.com/presentation/d/1yGLlIUqwVy3WeVv8K9HHOSkBietWTjJPlI_pA5wqU-Q/edit?usp=sharing

On consideration, this is a technical change, albeit one that arises
out of clarifying what was previously ambiguous.  I will take the
advice of the working group on how to proceed.

---

From: Mike Bishop <[email-masked]>
Date: Mon, 23 Mar 2015 00:17:13 +0000

I'm fine with the clarification that the sender of GOAWAY may have initiated streams with a higher stream ID than the recipient of GOAWAY, and the "will ignore" only applies to the recipient-initiated streams.  I think that was always implicit, and I'm happy to make it explicit.

On the other hand, I'm still inclined to see MUST NOT create new streams as the intent of GOAWAY.  The two-phase approach was introduced as an allowance for streams the client may have already initiated, but the client still MUST stop initiating upon receipt of the GOAWAY.  Now, the reality is that the server can't actually tell for sure when the client stopped initiating new streams, so a non-compliant client could keep initiating up to the number that was actually in the GOAWAY frame and hope the server doesn't catch it.  This change just makes that particular misbehavior permitted, which doesn't seem like a net improvement to me.  If the server told the client to stop, the client should stop.

I don't think that's in contradiction to the guidance below, since the first GOAWAY tells the client to stop initiating, and the second GOAWAY confirms the last already-in-flight stream that the server chose to service.  I had not understood the two-phase approach to intend the client to keep sending while it spun up the second connection, and so didn't see the text as inconsistent in that regard.  It sounds like others interpreted the first GOAWAY as a hint that a "real" GOAWAY would follow later, but multiple are allowed.  How does the client know it's finally got the "real" GOAWAY that it needs to obey?

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Sunday, March 22, 2015 9:19 AM
To: HTTP Working Group
Subject: Re: GOAWAY clarification

On consideration, this is a technical change, albeit one that arises out of clarifying what was previously ambiguous.  I will take the advice of the working group on how to proceed.

On 22 March 2015 at 08:51, Martin Thomson <[email-masked]> wrote:
>> I've created a PR for this.  
>> https://github.com/http2/http2-spec/pull/732
>>

---

From: Martin Thomson <[email-masked]>
Date: Sun, 22 Mar 2015 20:27:13 -0700

On 22 March 2015 at 17:17, Mike Bishop <[email-masked]> wrote:
> On the other hand, I'm still inclined to see MUST NOT create new streams as the intent of GOAWAY.

This is exactly the sort of feedback that I was fishing for, and a
response I predicted :)

To yes, given that you feel this way, I think that is probably best in
the end.  Anything more than that (and what I've described certainly
goes one step further than this), is an actual change, beyond what was
implicit but poorly articulated in previous versions.  I'll open a v3
of the PR tomorrow with an attempt to fix this (it's too late for me
to formulate clear text right now).

I've been convinced that an attempt to make technical changes is folly
and any effort should merely try to clarify what currently exists.
That means that we won't get the seamless transition between
connections I described, but we will get certainty when it comes to
connection states.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 23 Mar 2015 07:36:29 -0700

>> On the other hand, I'm still inclined to see MUST NOT create new streams as the intent of GOAWAY.

I've updated the PR.  I think that this is *purely* clarification now.
Please confirm that this is the case.

https://github.com/http2/http2-spec/pull/733/files

---

From: Mike Bishop <[email-masked]>
Date: Mon, 23 Mar 2015 18:54:54 +0000

One inconsistency in the revised text:  You still say at line 2444 that initiating new requests "is inadvisable."  It's not just inadvisable, it's MUST NOT at line 2353.

I'll also note that the seamless hand-off is a perfect reason for the DRAINING frame extension that was also floated during that discussion -- DRAINING could be defined as a hint that the client should spin up a secondary connection because a GOAWAY is coming soon.  No harm if the client doesn't understand or obey it -- the loss is strictly its own.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 23 Mar 2015 12:56:14 -0700

Thanks Mike,

> One inconsistency in the revised text:  You still say at line 2444 that initiating new requests "is inadvisable."  It's not just inadvisable, it's MUST NOT at line 2353.

Fixed.

> I'll also note that the seamless hand-off is a perfect reason for the DRAINING frame extension that was also floated during that discussion -- DRAINING could be defined as a hint that the client sh…

That sounds like a great plan to me.  I look forward to someone
proposing this extension some day.

---

From: Roberto Peon <[email-masked]>
Date: Mon, 23 Mar 2015 13:03:11 -0700

Oh yea!
-=R

On Mon, Mar 23, 2015 at 12:56 PM, Martin Thomson <[email-masked]>
wrote:

> > One inconsistency in the revised text:  You still say at line 2444 that
> initiating new requests "is inadvisable."  It's not just inadvisable, it's
> MUST NOT at line 2353.
>
> Fixed.
>
> > I'll also note that the seamless hand-off is a perfect reason for the
> DRAINING frame extension that was also floated during that discussion --
> DRAINING could be defined as a hint that the client should spin up a
> secondary connection because a GOAWAY is coming soon.  No harm if the
> client doesn't understand or obey it -- the loss is strictly its own.
>

---

From: Mike Bishop <[email-masked]>
Date: Mon, 23 Mar 2015 22:03:36 +0000

(For that matter, it doesn’t necessarily need to burn a frame type – define a setting as being whether the peer knows it intends to kill the connection shortly.  It’s zero when unset, flipped to any non-zero value if the recipient needs a back-up plan.  That also retains the server’s ability to change its mind, and the ACK means the server has a good gauge of when it’s safe to start sending GOAWAYs….)

From: Roberto Peon [mailto:[email-masked]]
Sent: Monday, March 23, 2015 1:03 PM
To: Martin Thomson
Cc: Mike Bishop; HTTP Working Group
Subject: Re: GOAWAY clarification

Oh yea!
-=R

On Mon, Mar 23, 2015 at 12:56 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
Thanks Mike,

Fixed.