Subject: GOAWAY and proxies

From: Daniel Sommermann <[email-masked]>
Date: Thu, 10 Apr 2014 14:56:33 -0700

In our implementation experience, GOAWAY as specified today is not an 
ideal mechanism for graceful shutdown. When there is high RTT between a 
proxy and the server, many requests may be in flight when the server 
generates a GOAWAY. For clients, it seems the solution is to reissue the 
request. The same strategy doesn't work for proxies. A proxy would have 
to buffer the headers and body of all requests until it sees the 
beginning of a response just in case a GOAWAY is received from the 
server. This is obviously not ideal as it puts high memory requirements 
on the proxy for a rare case.

The server could implement some workarounds like "pre-acking" some 
number of streams in its GOAWAY just in case there are some in flight 
requests, but this doesn't seem like a great solution.

Is there a way we can improve GOAWAY so that proxies can issue zero 
errors during a server restart and stay safe from a memory usage 
perspective? I'd be interested to hear others' implementation experience 
on this topic.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 10 Apr 2014 15:09:29 -0700

Can't the proxy just issue RST_STREAM toward its clients for all the
affected requests if it can't tolerate the memory hit?  REFUSED_STREAM
seems like the right code there.

RST_STREAM will cause an end-to-end retry, if anything, which is
slower.  But retrying requests is a feature that the proxy optionally
provides, and if it can't tolerate the costs of providing the feature,
then maybe it can do something to avoid taking on that cost.

On 10 April 2014 14:56, Daniel Sommermann <[email-masked]> wrote:
> In our implementation experience, GOAWAY as specified today is not an ideal
> mechanism for graceful shutdown. When there is high RTT between a proxy and
> the server, many requests may be in flight when the server generates a
> GOAWAY. For clients, it seems the solution is to reissue the request. The
> same strategy doesn't work for proxies. A proxy would have to buffer the
> headers and body of all requests until it sees the beginning of a response
> just in case a GOAWAY is received from the server. This is obviously not
> ideal as it puts high memory requirements on the proxy for a rare case.
>
> The server could implement some workarounds like "pre-acking" some number of
> streams in its GOAWAY just in case there are some in flight requests, but
> this doesn't seem like a great solution.
>
> Is there a way we can improve GOAWAY so that proxies can issue zero errors
> during a server restart and stay safe from a memory usage perspective? I'd
> be interested to hear others' implementation experience on this topic.
>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Apr 2014 15:12:26 -0700

Ages ago in spdy-land we discussed a LAME_DUCK frame, which would indicate
that a server would going away in a certain amount of time, but... never
seemed important enough to bother.
-=R

On Thu, Apr 10, 2014 at 2:56 PM, Daniel Sommermann <[email-masked]> wrote:

> request. The same strategy doesn't work for proxies. A proxy would have to
> buffer the headers and body of all requests until it sees the beginning of
> a response just in case a GOAWAY is received from the server. This is
> obviously not ideal as it puts high memory requirements on the proxy for a
> rare case.
>
> The server could implement some workarounds like "pre-acking" some number
> of streams in its GOAWAY just in case there are some in flight requests,
> but this doesn't seem like a great solution.
>

---

From: Daniel Sommermann <[email-masked]>
Date: Thu, 10 Apr 2014 15:17:33 -0700

> then maybe it can do something to avoid taking on that cost.
I was not aware that REFUSED_STREAM caused clients to retry. That seems 
to generally solve this issue! I am worried that some implementations 
may *not* reissue the request (the language only says the client MAY 
retry), leading to a request that could have succeeded not being serviced.

Should the language read that the client SHOULD retry, subject to some 
rate-limiting?

---

From: Martin Thomson <[email-masked]>
Date: Thu, 10 Apr 2014 15:25:57 -0700

> Should the language read that the client SHOULD retry, subject to some

A client is never obligated to do anything.  Use of "SHOULD" would
imply an obligation of the sort where there are negative consequences
for some party if they decided not to.  In this case, the only entity
affected by a client's refusal to retry is the client, so I don't see
any interoperability impact to the choice.

http://http2.github.io/http2-spec/#Reliability does make it clear that
if the client still intends to make the request, it can.

---

From: Daniel Sommermann <[email-masked]>
Date: Thu, 10 Apr 2014 16:30:24 -0700

> seems like the right code there.

It just occurred to me that this solution will only work if the client 
is speaking HTTP/2 to the proxy. This solution does not work for 
HTTP/1.1 clients connected to a proxy which speaks HTTP/2 to the server.

Seems like to fix this case you would either need something like the 
last_stream_id GOAWAY hack on the server side or buffering in the proxy.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Apr 2014 16:50:50 -0700

I'll note that REFUSED_STREAM doesn't actually solve the problem entirely,
since it does require the user (not just the proxy) to retry.

It is hoped that alt-svc (potentially to the same name) will give us the
ability to have the client (proxy or not) reconnect to a new server (and
thus there should be no need for a LAME_DUCK given that mechanism).
-=R

On Thu, Apr 10, 2014 at 4:30 PM, Daniel Sommermann <[email-masked]> wrote:

> It just occurred to me that this solution will only work if the client is
> speaking HTTP/2 to the proxy. This solution does not work for HTTP/1.1
> clients connected to a proxy which speaks HTTP/2 to the server.
>

---

From: Martin Thomson <[email-masked]>
Date: Thu, 10 Apr 2014 17:02:16 -0700

> since it does require the user (not just the proxy) to retry.

Indeed.  But each hop is given that choice (retry this request, or
fail back to my client).

> thus there should be no need for a LAME_DUCK given that mechanism).

I don't think that helps Daniel's case, just the more general one.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 10 Apr 2014 17:08:36 -0700

It helps Daniels case if the server is smart enough to begin doing alt-svc
redirecting before it goes down (i.e. it enters a period where it shifts
traffic away while processing the traffic it had been given previously).
Of course, I could be misinterpreting, but this is the retry-failure case
that I'm familiar with!
-=R

---

From: Daniel Sommermann <[email-masked]>
Date: Wed, 16 Apr 2014 07:49:49 -0700

Roberto, can you be more specific about how the proxy and server 
interact to redirect traffic using Alt-Svc? Since in this case the 
client is speaking HTTP/1.1, are you suggesting that the proxy treat the 
Alt-Svc as a hop-by-hop feature? My understanding is that this is an 
end-to-end feature.

On 04/10/2014 05:08 PM, Roberto Peon wrote:
> It helps Daniels case if the server is smart enough to begin doing 
> alt-svc redirecting before it goes down (i.e. it enters a period where 
> it shifts traffic away while processing the traffic it had been given 
> previously).
> Of course, I could be misinterpreting, but this is the retry-failure 
> case that I'm familiar with!
> -=R
>
>
> On Thu, Apr 10, 2014 at 5:02 PM, Martin Thomson 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     On 10 April 2014 16:50, Roberto Peon <[email-masked]
>     <mailto:[email-masked]>> wrote:

---

From: Daniel Sommermann <[email-masked]>
Date: Thu, 17 Apr 2014 07:41:22 -0700

I opened https://github.com/http2/http2-spec/issues/458 to track this. 
My proposal is to add a DRAINING frame rather than rely on Alt-Svc as 
Alt-Svc for shutdown would require all webservers to keep track of 
alternates. Tracking alternates falls naturally under the work of the 
proxy and not the webserver in my opinion.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 17 Apr 2014 09:44:27 -0700

On 17 April 2014 07:41, Daniel Sommermann <[email-masked]> wrote:
> I opened https://github.com/http2/http2-spec/issues/458 to track this. My
> proposal is to add a DRAINING frame rather than rely on Alt-Svc as Alt-Svc
> for shutdown would require all webservers to keep track of alternates.
> Tracking alternates falls naturally under the work of the proxy and not the
> webserver in my opinion.

For the record:

---
I first brought this up in
http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0158.html
but I wanted to open an issue to track this as I believe this is a
design flaw with HTTP/2 as it is today.

Consider an HTTP/1.1 connection to a proxy that speaks HTTP/2 to the
server. If the server needs to shut down, it will send a GOAWAY
acknowledging the last request it processed.

However, there is a race condition where a request could be in flight
or queued. When the server eventually sees the delayed requests, it
will not process it and the proxy will eventually return an error
(most likey 5xx) to the client.

Requiring proxies to buffer the request in order to replay for this
case is not feasible.

One solution is to have the server send an ALTSVC frame, but this is
not ideal as it implies each server must keep track of a list of
alternatives. It may not make sense for the origin server itself to
maintain this (potentially quickly changing) list.

Another solution is to allow acknowledging unreceived streams in the
GOAWAY. For instance, the server could acknowledge last-stream-id plus
SETTINGS_MAX_CONCURRENT_STREAMS. This might work in practice, but
seems hacky.

Another (and in my view best) solution would be to introduce a two
phase shutdown mechanism. Suggested language: "If a server needs to
shut down for maintenance, the server SHOULD immediately send a
DRAINING frame on all its client connections to indicate that it will
soon shut down. This allows the client to stop issuing new requests
and avoids unnecessary retries. After a reasonable amount of time, the
server SHOULD send a GOAWAY with status NO_ERROR on all its
connections and then close the connection. Clients MAY may open new
connections to the server after receiving a DRAINING frame."
---

HTTP/2 has a significantly higher investment per connection than
HTTP/1.1, which is why these mechanisms are suddenly hugely important.
 What would have just been executed as a connection close in HTTP/1.1
now has too much collateral loss.  I think that some text on this
issue is definitely appropriate.

As for solutions, I actually like the GOAWAY hack.  It's basically the
same as the DRAINING frame that you propose, without the additional
mechanism.  It's last stream + 2*max concurrent, BTW, and that won't
catch streams from clients that decide to skip numbers, but I don't
think that's worth worrying about.  In practice, you could set the
last stream ID to 2^31-1 and it would have the same effect.  Maybe we
could also add a new error code to accommodate the use case, though
I'll note that NO_ERROR seems like a pretty good fit already; maybe a
last stream ID of 2^31-1 is a sufficient signal.

Maybe we also want to allow for a second GOAWAY to allow for error
handling during this shutdown process too.

---

From: Daniel Sommermann <[email-masked]>
Date: Mon, 21 Apr 2014 09:45:01 -0700

I'm very interested in hearing from proxy implementors doing HTTP/1.1 
client to HTTP/2 or SPDY server conversion regarding issue #458. In our 
experience at Facebook, this becomes a significant source of 
user-visible errors during origin server restarts.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Mon, 21 Apr 2014 15:58:23 -0700

Just to be clear, this is a problem with HTTP/1.1 today, right? HTTP/2
improves on the situation with GOAWAY which allows the server, in graceful
shutdown, to explicitly indicate which streams have been "accepted", so the
client can safely retry. In your example, a proxy probably won't buffer the
request, so it can't retry itself. I'd say that RST_STREAM with
REFUSED_STREAM would solve your issue, but I see that the client<=>proxy
connection is HTTP/1.X, so that facility is not available.

When I think about what Chromium would upon receipt of a DRAINING frame, I
don't know. Unless we were to open up a new connection, there's not much we
could do, and we'd just send requests anyway. But yeah, from a load
balancing proxy talking to a set of HTTP/2 backends, I can see the utility
here. My inclination is to push back on this unless more proxy vendors
clamor for this. What do you do today in this situation? It seems like you
could build this draining semantic at the application layer on top of the
proxy<=>backend connections, by using magic path handlers
(/doyouwanttoacceptrequests), which indeed is how I would imagine doing it
with HTTP/1.X connections to the backends. This type of polling isn't
exactly elegant, but mostly works today.

So yeah, I'm open to it, but need more convincing.

On Mon, Apr 21, 2014 at 9:45 AM, Daniel Sommermann <[email-masked]> wrote:

> experience at Facebook, this becomes a significant source of user-visible
> errors during origin server restarts.
>
>

---

From: Daniel Sommermann <[email-masked]>
Date: Mon, 21 Apr 2014 16:26:22 -0700

Thanks for the response!

On 04/21/2014 03:58 PM, William Chan (陈智昌) wrote:
> Just to be clear, this is a problem with HTTP/1.1 today, right?
Correct, this is an issue for HTTP/1.1 clients only. This is a blocker 
for HTTP/1.1 <-> 2.0 proxy interop in my mind. I'm all for 
REFUSED_STREAM in general for the future.
> When I think about what Chromium would upon receipt of a DRAINING 
> frame, I don't know.
Chromium should probably issue new HTTP requests on a different 
connection and not open streams on the connection that got a DRAINING 
frame (even before getting the GOAWAY). This could avoid extra RTTs due 
to retrying requests.
> What do you do today in this situation?
We are implementing the GOAWAY hack of acknowledging the last 
transaction id we saw plus MAX_CONCURRENT_STREAMS, but we don't use this 
for publicly facing connections since it is non-standard behavior.
> It seems like you could build this draining semantic at the 
> application layer on top of the proxy<=>backend connections, by using 
> magic path handlers (/doyouwanttoacceptrequests), which indeed is how 
> I would imagine doing it with HTTP/1.X connections to the backends. 
> This type of polling isn't exactly elegant, but mostly works today.
This would work for reverse proxies, but polling as you say is not very 
elegant (wasted bytes, 1 full RTT to update state). If your polling 
interval isn't frequent enough, you could still hit the race condition. 
We have seen this kind of polling hurt CPU of origin servers in other 
contexts. Finally, this is a bunch of repeated protocol-like work that 
every proxy/server pair would have to remember to implement as a 
consequence of the broken spec, or else they would run into this issue. 
For forward proxies, polling wouldn't work at all since random HTTP/2 
servers probably don't implement the /doyouwanttoacceptrequests 
endpoint. Imo, it is better to bake a solution into the spec so every 
implementor doesn't have to reinvent the wheel.
> So yeah, I'm open to it, but need more convincing.
Hopefully this is a start :)

---

From: Martin Thomson <[email-masked]>
Date: Mon, 21 Apr 2014 16:49:39 -0700

On 21 April 2014 16:26, Daniel Sommermann <[email-masked]> wrote:
> We are implementing the GOAWAY hack of acknowledging the last transaction id
> we saw plus MAX_CONCURRENT_STREAMS, but we don't use this for publicly
> facing connections since it is non-standard behavior.

2x MAX_CONCURRENT_STREAMS, I hope.

I'm still not clear on why this is insufficient.  Can you explain
further?  (Having text in the spec would still be worthwhile, if that
was where we ended up.)

---

From: Daniel Sommermann <[email-masked]>
Date: Mon, 21 Apr 2014 17:18:40 -0700

> 2x MAX_CONCURRENT_STREAMS, I hope. 
Yeah, I didn't write out the full arithmetic.
> further? (Having text in the spec would still be worthwhile, if that 
Acking the extra streams in the GOAWAY is sufficient I think. It's just 
a bit strange, as it implies the remote is committed to servicing 
streams that may never arrive. Normally, GOAWAY is sent very shortly 
before the connection is closed. If the server sends GOAWAY and the 
client stops issuing HTTP requests, the connection could stay open for a 
while.

There are other oddities, like handling the case where a future stream 
causes a second GOAWAY with PROTOCOL_ERROR to be returned (although I 
suppose multiple GOAWAY is already possible today with NO_ERROR and then 
FLOW_CONTROL_ERROR). The second GOAWAY would probably ack a lower number 
than the first in this case.

I'm sure we could add text handling these cases, but I'm worried about 
diluting GOAWAY's current clear semantic meaning with special cases. 
What do others think?

---

From: Daniel Sommermann <[email-masked]>
Date: Mon, 21 Apr 2014 17:24:48 -0700

Actually, the extra ack GOAWAY only avoids this issue for high RPS proxy 
<-> server connections. For low RPS connections, the server's hard 
shutdown timeout would probably fire before all the extra 
MAX_CONCURRENT_STREAMS are requested, so the same old race condition 
still applies. Our proxy/server connections are extremely busy, so we 
haven't noticed this shortcoming of the workaround.

---

From: Jeff Pinner <[email-masked]>
Date: Mon, 21 Apr 2014 17:32:51 -0700

FWIW: We do out-of-band, active health-checking to signal draining.

On Mon, Apr 21, 2014 at 5:24 PM, Daniel Sommermann <[email-masked]> wrote:

> <-> server connections. For low RPS connections, the server's hard shutdown
> timeout would probably fire before all the extra MAX_CONCURRENT_STREAMS are
> requested, so the same old race condition still applies. Our proxy/server
> connections are extremely busy, so we haven't noticed this shortcoming of
> the workaround.
>
>

---

From: Daniel Sommermann <[email-masked]>
Date: Mon, 21 Apr 2014 17:45:33 -0700

> FWIW: We do out-of-band, active health-checking to signal draining.

Just to be clear: for each HTTP/2 connection from the proxy to origin, 
you periodically healthcheck the server on some endpoint on a new 
stream? As long as the healthcheck interval is shorter than the soft 
shutdown timeout, this seems like a good approach. It doesn't work for 
the forward proxy use case though.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Mon, 21 Apr 2014 17:44:13 -0700

I'd love to hear your thoughts on whether or not we should add a protocol
solution for draining, or if application level health checking is
sufficient. For reverse proxies, an application specific solution may be
sufficient, but I'd like to hear your thoughts on it. There are also the
considerations for forward proxies and other clients (browsers), and it'd
be interesting to hear your (and others) thoughts there.

On Mon, Apr 21, 2014 at 5:32 PM, Jeff Pinner <[email-masked]> wrote:

> On Mon, Apr 21, 2014 at 5:24 PM, Daniel Sommermann <[email-masked]>wrote:
>

---

From: Jeff Pinner <[email-masked]>
Date: Mon, 21 Apr 2014 17:52:34 -0700

For each server (not each connection), we periodically heathcheck the
server at some interval. The server when failing healthchecks, must
disallow new connections and allow old connections to drain gracefully. It
must only send GOAWAY frames for session level errors. The reverse proxy is
responsible for loadbalancing between new and existing connections and for
deciding when a connection becomes unusable (for example we consider it
draining when half of the available stream IDs have been used).

On Mon, Apr 21, 2014 at 5:45 PM, Daniel Sommermann <[email-masked]> wrote:

> Just to be clear: for each HTTP/2 connection from the proxy to origin, you
> periodically healthcheck the server on some endpoint on a new stream? As
> long as the healthcheck interval is shorter than the soft shutdown timeout,
> this seems like a good approach. It doesn't work for the forward proxy use
> case though.
>

---

From: Daniel Sommermann <[email-masked]>
Date: Tue, 22 Apr 2014 02:43:57 +0000

The "per server" approach only works if the server IP corresponds to a single host. If the IP actually is served by multiple hosts, then multiple healthcheck connections could be served by different instances (with different maintenance times). I think in the general case, each HTTP/2 connection needs this healthcheck.

We do healthchecking per server IP for load balancing purposes as well, but sometimes the the server IP is actually another proxy tier with many hosts (we do L4 load balancing as well).

So, I think for the reverse proxy healthchecking approach to work in general, you need to do it per connection.

At any rate, I don't think this approach can be made to work for forward proxy.

________________________________
From: Jeff Pinner [[email-masked]]
Sent: Monday, April 21, 2014 5:52 PM
To: Daniel Sommermann
Cc: Martin Thomson; William Chan (陈智昌); HTTP Working Group
Subject: Re: GOAWAY and proxies (#458)

For each server (not each connection), we periodically heathcheck the server at some interval. The server when failing healthchecks, must disallow new connections and allow old connections to drain gracefully. It must only send GOAWAY frames for session level errors. The reverse proxy is responsible for loadbalancing between new and existing connections and for deciding when a connection becomes unusable (for example we consider it draining when half of the available stream IDs have been used).

---

From: Jeff Pinner <[email-masked]>
Date: Mon, 21 Apr 2014 19:51:09 -0700

On Mon, Apr 21, 2014 at 7:43 PM, Daniel Sommermann <[email-masked]> wrote:

>  The "per server" approach only works if the server IP corresponds to a
> single host. If the IP actually is served by multiple hosts, then multiple
> healthcheck connections could be served by different instances (with
> different maintenance times). I think in the general case, each HTTP/2
> connection needs this healthcheck.
>
>
Yep, we healthcheck per server IP.

> We do healthchecking per server IP for load balancing purposes as well,
> but sometimes the the server IP is actually another proxy tier with many
> hosts (we do L4 load balancing as well).
>
>
If the server is another proxy itself (such as an L4 LB or a forward proxy,
etc.) it does its own healthchecking of the backend servers.

> So, I think for the reverse proxy healthchecking approach to work in
> general, you need to do it per connection.
>
For forward proxies, we hold onto and then retry requests :(

---

From: Martin Thomson <[email-masked]>
Date: Mon, 21 Apr 2014 21:08:12 -0700

On 21 April 2014 17:24, Daniel Sommermann <[email-masked]> wrote:
> Actually, the extra ack GOAWAY only avoids this issue for high RPS proxy <->
> server connections. For low RPS connections, the server's hard shutdown

As long the server waits >1RTT after sending the first GOAWAY before
identifying the *real* last request, isn't that enough?

Sure, if your clients exceed MAX_CONCURRENT_STREAMS before the RTT is
up, then you are pretty much guaranteed to be OK, but that's a clear
sign that you are setting MAX_CONCURRENT_STREAMS too low.

---

From: Daniel Sommermann <[email-masked]>
Date: Tue, 22 Apr 2014 09:37:19 -0700

> identifying the *real* last request, isn't that enough?
So the first GOAWAY acknowledges the last stream seen + 
2*MAX_CONCURRENT_STREAMS and then 1 RTT later the server sends a second 
GOAWAY with the actual last seen stream? This seems like it could work 
in practice, but RTT isn't always stable so it seems like this mechanism 
could be flaky.

The 2xGOAWAY approach would result in a smaller diff to the spec though. 
We would just need to add some text about expecting 2 GOAWAYs where the 
first GOAWAY may acknowledge a stream id greater than any initiated stream.

However, from an implementation point of view I think a new frame is 
more straight forward and less error prone. With the new frame, you 
don't have to measure RTT or do arithmetic. DRAINING is a completely 
empty frame. In both solutions 2 frames are sent, but with a new frame 
type, the purpose of the two frames is more clear.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 22 Apr 2014 10:01:56 -0700

On 22 April 2014 09:37, Daniel Sommermann <[email-masked]> wrote:
> GOAWAY with the actual last seen stream? This seems like it could work in
> practice, but RTT isn't always stable so it seems like this mechanism could
> be flaky.

I wasn't suggesting that it be exactly 1RTT.  Maybe worst-case RTT
plus a fair amount of slack.  That's probably still far smaller than
the timer you are running today (i.e., you could use that timer
happily.)

> The 2xGOAWAY approach would result in a smaller diff to the spec though. We
> would just need to add some text about expecting 2 GOAWAYs where the first
> GOAWAY may acknowledge a stream id greater than any initiated stream.

Yes.  I want to ensure that we explain what is going on.  That
definitely means text.  I'll put together a proposal.

> However, from an implementation point of view I think a new frame is more
> straight forward and less error prone. With the new frame, you don't have to
> measure RTT or do arithmetic. DRAINING is a completely empty frame. In both
> solutions 2 frames are sent, but with a new frame type, the purpose of the
> two frames is more clear.

I'm not sure that the problem is any different.  You still have to
ensure that you wait long enough after sending DRAINING before you
send the GOAWAY (the above math is exactly the same).  The difference
is that for a client, two GOAWAY frames are treated exactly the same,
i.e., clients don't need two code paths.

---

From: Daniel Sommermann <[email-masked]>
Date: Tue, 22 Apr 2014 10:35:32 -0700

> i.e., clients don't need two code paths. 
OK that sounds reasonable. I'm sure you were planning this already, but 
we should be explicit that the first GOAWAY with extra acknowledged 
streams does not imply the client can continue to open new streams up to 
that stream id.

I couldn't find any text in the spec for how to process multiple GOAWAY 
frames. This can actually happen today, apart from this proposal, if 
some connection level error happens on the final stream(s) after a 
GOAWAY NO_ERROR. Just to keep in mind while drafting the new text.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 22 Apr 2014 11:39:10 -0700

On 22 April 2014 10:35, Daniel Sommermann <[email-masked]> wrote:
> frames. This can actually happen today, apart from this proposal, if some
> connection level error happens on the final stream(s) after a GOAWAY
> NO_ERROR. Just to keep in mind while drafting the new text.

It's on my list of edits for today :)

It was already implicitly possible.  For instance, if a server is
shutting down and indicated acceptance of stream 19, but an error
occurred before it starts processing stream 19, it should be able to
signal the error immediately with stream 17, allowing the client to
automatically retry the request from stream 19.  That said, I doubt
that anyone would have implemented handling of the second GOAWAY
correctly.