Subject: Re: GOAWAY and proxies (#458)

From: Daniel Sommermann <[email-masked]>
Date: Fri, 2 May 2014 10:42:36 -0700

Now that Martin has clarified the possibility of receiving multiple 
GOAWAYs in the text 
(https://github.com/http2/http2-spec/commit/d00b4f82001305a5d01e810a7b45140223cb2d08), 
I have submitted https://github.com/http2/http2-spec/pull/475/files to 
make the graceful shutdown mechanism explicit in the spec. I'd be happy 
to get hear feedback on this proposed change.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 2 May 2014 11:31:54 -0700

On 2 May 2014 10:42, Daniel Sommermann <[email-masked]> wrote:
> Now that Martin has clarified the possibility of receiving multiple GOAWAYs
> in the text
> I have submitted https://github.com/http2/http2-spec/pull/475/files to make
> the graceful shutdown mechanism explicit in the spec. I'd be happy to get
> hear feedback on this proposed change.

So that I can discuss this:

+          There is an inherent race condition when proxying from a
HTTP/1.1 client to a HTTP/2
+          server and the server goes down for maintenance. Since some
in-flight client requests may
+          be unacknowledged by the server GOAWAY and there is no way
to retry these requests,
+          servers SHOULD implement a two phase shutdown for
maintenance. In the first phase, the
+          server sends a GOAWAY with NO_ERROR and a last stream id
equal to 2^31 - 1. After at
+          least 1/2 RTT elapses, the server sends a second GOAWAY
with NO_ERROR and the last stream
+          id it actually accepted.

I think that this is a problem with intermediation in general.  And
there may even be some clients that are unable to retry for other
reasons (intermediaries aren't the only entities that attempt to
minimize their state commitment).

Half an RTT is definitely not enough.  From the server perspective it
needs to be >1RTT.

Maybe:

A client that is unable to retry requests loses all requests that are
in flight when a server closes
down a connection.  This is especially true for intermediaries that
might not be serving clients
using HTTP/2.  If the server is attempting to gracefully shut down a
connection, it can send an
initial GOAWAY frame with the last frame identifier set to
2<x:sup>31</x:sup>-1 and a
<x:ref>NO_ERROR</x:ref> code.  This signals to the client that a
shutdown is imminent and that no
further requests can be initiated.  The server can then wait at least
a round trip time to allow for
requests that the client sent prior to receiving the GOAWAY frame
before sending another GOAWAY
frame with an updated last stream identifier.  This ensures that a
connection can be cleanly shut
down without losing requests.

---

From: Daniel Sommermann <[email-masked]>
Date: Fri, 2 May 2014 11:43:28 -0700

> Half an RTT is definitely not enough.  From the server perspective it

Silly mistake. Will fix.

> minimize their state commitment).

Great point, and this was something I wasn't happy with in my first 
draft. My only concern with your text is that there is no SHOULD. That 
would be fine if this was only a problem on private networks, but this 
problem also applies to forward proxies, so HTTP/2 servers on the 
internet at large SHOULD implement this behavior.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 2 May 2014 11:45:57 -0700

On 2 May 2014 11:43, Daniel Sommermann <[email-masked]> wrote:
> Great point, and this was something I wasn't happy with in my first draft.
> My only concern with your text is that there is no SHOULD. That would be
> fine if this was only a problem on private networks, but this problem also
> applies to forward proxies, so HTTP/2 servers on the internet at large
> SHOULD implement this behavior.

I'm not that much of a fan of normative language that isn't MUST, but
I get your point.  Yes, the world would be a better place if everyone
did this.

s/
If the server is attempting to gracefully shut down a
connection, it can send
/
A server that is attempting to gracefully shut down a
connection, SHOULD send
/

---

From: Roberto Peon <[email-masked]>
Date: Fri, 2 May 2014 14:22:19 -0700

I'm not entirely happy with the GOAWAY change, honestly.

As written now, the receiver of GOAWAY has zero certainty about which
streams were successfully processed as the server gets to change its mind
and isn't limited to simply changing the number in one direction.

-=R

On Fri, May 2, 2014 at 11:31 AM, Martin Thomson <[email-masked]>wrote:

> GOAWAYs
> > (
> https://github.com/http2/http2-spec/commit/d00b4f82001305a5d01e810a7b45140223cb2d08
> ),
> make

---

From: Martin Thomson <[email-masked]>
Date: Fri, 2 May 2014 14:50:25 -0700

> and isn't limited to simply changing the number in one direction.

The purpose isn't to indicate what is successfully processed.  It's to
indicate what _might have been_ processed.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 2 May 2014 15:07:35 -0700

Agreed, but the current text doesn't allow for that to be determined with
any certainty, since there may be a subsequent GOAWAY frame which may alter
the last-potentially-processed-stream in *either* direction (as written).

The solution to that part is simple-- GOAWAY can only give a number that is
less than that which was sent before, thus indicating that a larger portion
of the stream-space was *not* processed.

Separately, I'd be shocked if many U-As actually did anything with
subsequent GOAWAYs, however-- the logic for cancelling the retries, etc.
would get very hairy very quickly if the assumption at time T was different
than T+1 and one had already acted upon the information at time T.

I still like DRAINING better for the purpose of getting clients to
reconnect, regardless of what happens with GOAWAY.
-=R

---

From: Daniel Sommermann <[email-masked]>
Date: Fri, 2 May 2014 15:24:57 -0700

This is already handled in the spec in 6.8:

"Servers SHOULD NOT increase the value they send in the last stream 
identifier, since this would cause clients to abandon automatic retry of 
requests if a GOAWAY frame is lost"

On 05/02/2014 03:07 PM, Roberto Peon wrote:
> Agreed, but the current text doesn't allow for that to be determined 
> with any certainty, since there may be a subsequent GOAWAY frame which 
> may alter the last-potentially-processed-stream in *either* direction 
> (as written).
>
> The solution to that part is simple-- GOAWAY can only give a number 
> that is less than that which was sent before, thus indicating that a 
> larger portion of the stream-space was *not* processed.
>
> subsequent GOAWAYs, however-- the logic for cancelling the retries, 
> etc. would get very hairy very quickly if the assumption at time T was 
> different than T+1 and one had already acted upon the information at 
> time T.
>
> On Fri, May 2, 2014 at 2:50 PM, Martin Thomson 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     On 2 May 2014 14:22, Roberto Peon <[email-masked]
>     <mailto:[email-masked]>> wrote:

---

From: William Chan (陈智昌) <[email-masked]>
Date: Fri, 2 May 2014 15:30:46 -0700

I skimmed this PR, but haven't read everyone's comments :) So here's my
uninformed response.

The way our code works right now is, on receipt of a GOAWAY, we put
ourselves in a lame duck mode and kill all streams above the last accepted
stream id with an error code to retry them. If we're already in the lame
duck mode, great. We don't track anything about previous accepted stream
ids. This is a key property that I'd like to see preserved. Another way of
framing that is that I'd like not to introduce new connection state
variables to handle multiple GOAWAYs. Secondarily, I kinda feel like
DRAINING is more explicit than GOAWAY, so I'd paint it that color if
anything.

I'm not completely sold on needing any solution here, but if that's the
direction the working group wants to go, I wouldn't object too strongly.

On Fri, May 2, 2014 at 3:07 PM, Roberto Peon <[email-masked]> wrote:

> The solution to that part is simple-- GOAWAY can only give a number that
> is less than that which was sent before, thus indicating that a larger
> portion of the stream-space was *not* processed.
>

---

From: Daniel Sommermann <[email-masked]>
Date: Fri, 2 May 2014 16:10:42 -0700

On 05/02/2014 03:30 PM, William Chan (陈智昌) wrote:
> ourselves in a lame duck mode and kill all streams above the last 
> accepted stream id with an error code to retry them. If we're already 
> in the lame duck mode, great. We don't track anything about previous 
> accepted stream ids.

Yeah, this is how I am hoping most clients are implemented.

> I'm not completely sold on needing any solution here

Is it because you perceive the problem to be small and not worth the 
text or that there is no problem? I'd argue strongly against both from 
experience. I am pretty sure all 1.1 to 2.0 intermediary implementations 
will run into this issue to varying degrees. It'd be better to have a 
standard way to deal with it and avoid the surprise factor and 
reinventing the wheel for new deployments.

On 05/02/2014 03:30 PM, William Chan (陈智昌) wrote:
> I skimmed this PR, but haven't read everyone's comments :) So here's 
> my uninformed response.
>
> ourselves in a lame duck mode and kill all streams above the last 
> accepted stream id with an error code to retry them. If we're already 
> in the lame duck mode, great. We don't track anything about previous 
> accepted stream ids. This is a key property that I'd like to see 
> preserved. Another way of framing that is that I'd like not to 
> introduce new connection state variables to handle multiple GOAWAYs. 
> Secondarily, I kinda feel like DRAINING is more explicit than GOAWAY, 
> so I'd paint it that color if anything.
>
> I'm not completely sold on needing any solution here, but if that's 
> the direction the working group wants to go, I wouldn't object too 
> strongly.
>
>
> On Fri, May 2, 2014 at 3:07 PM, Roberto Peon <[email-masked] 
>     Agreed, but the current text doesn't allow for that to be
>     determined with any certainty, since there may be a subsequent
>     GOAWAY frame which may alter the last-potentially-processed-stream
>     in *either* direction (as written).
>
>     The solution to that part is simple-- GOAWAY can only give a
>     number that is less than that which was sent before, thus
>     indicating that a larger portion of the stream-space was *not*
>     processed.
>
>     subsequent GOAWAYs, however-- the logic for cancelling the
>     retries, etc. would get very hairy very quickly if the assumption
>     at time T was different than T+1 and one had already acted upon
>     the information at time T.
>

---

From: Daniel Sommermann <[email-masked]>
Date: Fri, 2 May 2014 16:44:01 -0700

I updated the language on the pull request (mostly copying Martin's 
suggestions): https://github.com/http2/http2-spec/pull/475/files

It adds this text to section 6.8:

           A client that is unable to retry requests loses all requests 
that are in flight when the
           server closes the connection. This is especially true for 
intermediaries that might
           not be serving clients using HTTP/2. A server that is 
attempting to gracefully shut down
           a connection SHOULD send an initial GOAWAY frame with the 
last stream identifier set to
           2<x:sup>31</x:sup>-1 and a <x:ref>NO_ERROR</x:ref> code. This 
signals to the client that
           a shutdown is imminent and that no further requests can be 
initiated. After waiting at
           least one round trip time, the server can send another GOAWAY 
frame with an updated last
           stream identifier. This ensures that a connection can be 
cleanly shut down without losing
           requests.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Fri, 2 May 2014 16:52:30 -0700

On Fri, May 2, 2014 at 4:10 PM, Daniel Sommermann <[email-masked]> wrote:

> ids.

Yes, I wish all clients were implemented like Chromium :P

> Is it because you perceive the problem to be small and not worth the text
> or that there is no problem? I'd argue strongly against both from
> standard way to deal with it and avoid the surprise factor and reinventing
> the wheel for new deployments.
>

In spec cleanliness maybe it makes sense. Do we think we'll have many 1.1
to 2.0 intermediaries? Maybe in the non-browser case? And maybe for IE if
they support 2.0 in the clear? But Firefox and Chromium will go over TLS,
so I just kinda wonder how important this is in practice. I dunno. Just
saying I'm not completely sold, but if you and others definitely want it,
then I'm fine letting it in.

>> that is less than that which was sent before, thus indicating that a larger
>>  On Fri, May 2, 2014 at 2:50 PM, Martin Thomson <[email-masked]
>> > wrote:
>>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 2 May 2014 20:36:13 -0700

> duck mode, great. We don't track anything about previous accepted stream

Perfect.  No extra logic.  If you get an extra GOAWAY, you might kill
a few more requests.

What does the extra explicitness of DRAINING actually buy?

---

From: Roberto Peon <[email-masked]>
Date: Fri, 2 May 2014 20:44:29 -0700

DRAINING is a signal that indicates that the server will be sending a
GOAWAY in the near future, and thus that the client should create a new
connection, but, unlike GOAWAY it does not require the client to stop
sending requests.
This allows for a far more seamless switch-over without requiring the
state-tracking of ALT-SVC.

-=R

On Fri, May 2, 2014 at 8:36 PM, Martin Thomson <[email-masked]>wrote:

> accepted

---

From: Martin Thomson <[email-masked]>
Date: Fri, 2 May 2014 20:55:22 -0700

On 2 May 2014 20:44, Roberto Peon <[email-masked]> wrote:
> DRAINING is a signal that indicates that the server will be sending a GOAWAY
> in the near future, and thus that the client should create a new connection,
> but, unlike GOAWAY it does not require the client to stop sending requests.

If the client doesn't stop sending requests in a deterministic
fashion, then the server is unable to send GOAWAY without encountering
the issue that Daniel first raised.  Unless you consider this to be a
GOAWAY_REQUEST and there is some expectation that the client sends a
GOAWAY when it is ready to finish up (with the server setting some
upper bound on the time it's prepared to wait for this event, of
course).