Subject: Re: RST_STREAM(OK) after an HTTP response

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 Oct 2014 17:15:00 +1100

It seems like we've circled back to the original suggestion.

Martin, can you please prepare a pull request here?

Thanks,

On 28 Sep 2014, at 1:28 pm, Greg Wilkins <[email-masked]> wrote:

> 
> On 27 September 2014 20:24, Martin Thomson <[email-masked]> wrote:
> There is an alternative: recommend that clients reset their own
> request streams once the response is received.
> 
> This would break existing applications.   It is perfectly legal HTTP to send a response and then consume the request content.   Clients should not reset the stream after the response is read as the server may still wish to consume the body being sent.
> 
> Sure, in many/most use-cases it is entirely premature for a server to send a response before consuming the body, but there are use-cases where it does make sense... or at least where it is currently done and expected to keep working.
> 
> The only party that knows if the request body is required or not is the server.   It's perfectly able to send a reset now, and I think the only real issue is that the spec says that resets are "abnormal termination".   Renaming that to an "immediate termination" and using the existing NO_ERROR code would cover this case.
> 
> cheers
> 
> 
> -- 
> Greg Wilkins <[email-masked]> 
> http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
> http://www.webtide.com  advice and support for jetty and cometd.

--
Mark Nottingham   https://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Tue, 7 Oct 2014 15:02:10 -0700

> Martin, can you please prepare a pull request here?

https://github.com/http2/http2-spec/pull/624

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Oct 2014 09:50:44 +1100

Thanks. I think we're close to consensus on this -- anyone have further questions / pushback?

Cheers,

---

From: Matthew Cox <[email-masked]>
Date: Wed, 8 Oct 2014 16:25:15 +0000

This is still a -1 from our side.

Does RST_STREAM with NO_ERROR transition state to "closed", or is this a new state?

Obviously all data up to the other side's window must be allowed (because it may already be in flight), but will the receiver keep opening window after sending this new kind of RST_STREAM?

The way it reads right now, it almost sounds like RST_STREAM (NO_ERROR) is advisory and doesn't transition the stream to the terminal state.  

If this is the case then there is a simple workaround for us; we just ignore RST_STREAM (NO_ERROR).

Matthew

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Tuesday, October 07, 2014 3:51 PM
To: Martin Thomson
Cc: Greg Wilkins; Osama Mazahir; Jeff Pinner; Matthew Cox; Michaela LaVan; Mike Bishop; HTTP Working Group
Subject: Re: RST_STREAM(OK) after an HTTP response

Cheers,

---

From: Martin Thomson <[email-masked]>
Date: Wed, 8 Oct 2014 10:44:10 -0700

> Does RST_STREAM with NO_ERROR transition state to "closed", or is this a new state?

It's RST_STREAM, so it transitions to "closed".  It's not a special case.

The only thing special here is that once a response is complete,
RST_STREAM cannot cause the client to discard the response.  That
means that you shouldn't allow a RST_STREAM to abort an already
completed response.

That said, there are plenty of other reasons that a response might
have to be discarded, it's just that racing the RST_STREAM with the
response completion is probably a bad idea.

> Obviously all data up to the other side's window must be allowed (because it may already be in flight), but will the receiver keep opening window after sending this new kind of RST_STREAM?

Anything that the client has sent, or has commited to a send buffer,
will go out.

There's no obligation on any peer that has sent RST_STREAM to continue
to open the flow control window.  In fact, I think that it's illegal.

> If this is the case then there is a simple workaround for us; we just ignore RST_STREAM (NO_ERROR).

That would produce interesting results.  Probably nothing immediately
obvious, but you could end up exceeding MAX_CONCURRENT_STREAMS.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 8 Oct 2014 17:52:48 +0000

I wouldn't expect MAX_CONCURRENT_STREAMS to be affected -- the client would believe there was an extra active stream, and not use as many as the server allowed.  The server would start emitting RST_STREAM(STREAM_CLOSED) frames every RTT or so, though, and might eventually kick the client out for continuing to send on a closed stream.

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Wednesday, October 8, 2014 10:44 AM
To: Matthew Cox
Cc: Mark Nottingham; Greg Wilkins; Osama Mazahir; Jeff Pinner; Michaela LaVan; Mike Bishop; HTTP Working Group
Subject: Re: RST_STREAM(OK) after an HTTP response

The only thing special here is that once a response is complete, RST_STREAM cannot cause the client to discard the response.  That means that you shouldn't allow a RST_STREAM to abort an already completed response.

That said, there are plenty of other reasons that a response might have to be discarded, it's just that racing the RST_STREAM with the response completion is probably a bad idea.

> Obviously all data up to the other side's window must be allowed (because it may already be in flight), but will the receiver keep opening window after sending this new kind of RST_STREAM?

Anything that the client has sent, or has commited to a send buffer, will go out.

There's no obligation on any peer that has sent RST_STREAM to continue to open the flow control window.  In fact, I think that it's illegal.

> If this is the case then there is a simple workaround for us; we just ignore RST_STREAM (NO_ERROR).

That would produce interesting results.  Probably nothing immediately obvious, but you could end up exceeding MAX_CONCURRENT_STREAMS.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 8 Oct 2014 10:54:34 -0700

> I wouldn't expect MAX_CONCURRENT_STREAMS to be affected -- the client would believe there was an extra active stream, and not use as many as the server allowed.  The server would start emitting RST…

Right.  The last one is the potential risk, but that would only be a
problem if the server had opened the flow control window enough for
that to happen.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 12 Oct 2014 17:51:33 +1100

Matthew,

> Does RST_STREAM with NO_ERROR transition state to "closed", or is this a new state?

I really don't see how you can read it as creating a new state -- what in the proposal leads you to that impression?

> Obviously all data up to the other side's window must be allowed (because it may already be in flight), but will the receiver keep opening window after sending this new kind of RST_STREAM?

The proposal isn't for a new kind of RST_STREAM; it only illustrates how the frame interacts with HTTP semantics in a particular situation.

How do you get that? It currently reads:

> Clients MUST NOT discard responses as a result of receiving RST_STREAM in the corresponding "half-closed (remote)" state.

> If this is the case then there is a simple workaround for us; we just ignore RST_STREAM (NO_ERROR).

I think you've misunderstood the proposal. The recipient of *any* RST_STREAM is required to process it as per <http://http2.github.io/http2-spec/#RST_STREAM>, with the specified effects upon stream state. If you ignore the RST_STREAM, your implementation will not be conformant. 

This specifically includes:

> After receiving a RST_STREAM on a stream, the receiver MUST NOT send additional frames for that stream, with the exception of PRIORITY.

Cheers,

---

From: Matthew Cox <[email-masked]>
Date: Mon, 13 Oct 2014 17:10:02 +0000

> The proposal isn't for a new kind of RST_STREAM; it only illustrates how the frame interacts with HTTP semantics in a particular situation.

This is a new kind of RST_STREAM.  Previously all implementations could put their streams in an aborted state when receiving RST_STREAM and fail all I/O.  Now the reset code has to be checked and special logic must be done when getting NO_ERROR which puts the stream in a new state which I would call draining send data.  Once sends are complete (END_STREAM would have been sent if sending was allowed) then the state can be transitioned to a regular closed state (non-abortive).

Matthew

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Saturday, October 11, 2014 11:52 PM
To: Matthew Cox
Cc: Martin Thomson; Osama Mazahir; Mike Bishop; HTTP Working Group
Subject: Re: RST_STREAM(OK) after an HTTP response

Matthew,

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 14 Oct 2014 11:00:22 +1100

On 14 Oct 2014, at 4:10 am, Matthew Cox <[email-masked]> wrote:

> This is a new kind of RST_STREAM.  Previously all implementations could put their streams in an aborted state when receiving RST_STREAM and fail all I/O.

I see. This text:

"""
A server MAY request that the client abort transmission of a request without error by sending a RST_STREAM with an error code of NO_ERROR. Clients MUST NOT discard responses as a result of receiving RST_STREAM in the corresponding "half-closed (remote)" state.
"""

can be read to say that RST_STREAM causes a state transition from "open" to "half-closed (remote)", when such a state transition isn't documented; up until now, RST_STREAM has only allowed various states to transition to "closed".

I think this could be fixed by changing the text above to:

"""
When this is true, a server MAY request that the client abort transmission of a request without error by sending a RST_STREAM with an error code of NO_ERROR after sending a complete response (i.e., a frame with the END_STREAM flag). Clients MUST NOT discard responses as a result of receiving such a RST_STREAM.
"""

Does that help?

---

From: Mike Bishop <[email-masked]>
Date: Tue, 14 Oct 2014 16:59:17 +0000

On the text itself, I would make it a SHOULD NOT.  A client can already discard a response any time it wants, for any reason it wants.  Why is this one in particular deserving of an absolute prohibition?

Regardless, I don't think the issue is an incorrect implication of a state change on the wire, though if you want to make that clearer feel free.  There is an actual new state in the code that has to be introduced to support this, even if it's conceptually a "normal" RST_STREAM.

Upon receiving the RST_STREAM, we can't send any more body to the server.  But we also can't stop the layer above us from pumping body into us without aborting the request.  With any other form of RST_STREAM, we could just fail attempts to write more body, because the request was aborted by the server.  This form of RST_STREAM creates a new corner case that has to be handled in an API layer or an intermediary -- continue accepting body from the layer above, but route it to the bit bucket instead of the wire.

It's certainly possible -- it's just more code -- but our point is that permitting this does create a new state for an intermediary to handle.  (Though it's a reasonable argument that this state already existed and the text just calls attention to it in case, like us, an implementer overlooked it.)

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Monday, October 13, 2014 5:00 PM
To: Matthew Cox
Cc: Martin Thomson; Osama Mazahir; Mike Bishop; HTTP Working Group
Subject: Re: RST_STREAM(OK) after an HTTP response

I see. This text:

"""

"""

Does that help?

---

From: Martin Thomson <[email-masked]>
Date: Tue, 14 Oct 2014 10:09:22 -0700

> On the text itself, I would make it a SHOULD NOT.  A client can already discard a response any time it wants, for any reason it wants.  Why is this one in particular deserving of an absolute prohib…

It's not an absolute prohibition, it's intended to limit it to MUST
NOT discard as a direct result of this particular message (i.e., the
RST_STREAM.  I'm happy to take adjustments to that effect.

---

From: Robert Collins <[email-masked]>
Date: Wed, 15 Oct 2014 11:51:03 +1300

On 15 October 2014 05:59, Mike Bishop <[email-masked]> wrote:
> Upon receiving the RST_STREAM, we can't send any more body to the server.  But we also can't stop the layer above us from pumping body into us without aborting the request.

Isn' that dependent on the API you have? E.g. in python you could make
data being pumped at that point raise a specific exception. And then
no more data will be sent :).

Perhaps the implied but not stated thing here is that this is
different to HTTP/1.1 APIs where client code can assume that it just
needs to write all the data and then and only then it can read the
response is the problem: and this is indeed a difference to HTTP/1.1
:).

>  With any other form of RST_STREAM, we could just fail attempts to write more body, because the request was aborted by the server.  This form of RST_STREAM creates a new corner case that has to be handled in an API layer or an intermediary -- continue accepting body from the layer above, but route it to the bit bucket instead of the wire.

I don't think thats desirable, because its wasted effort - better to
signal the protocols capabilities back up the API stack more
faithfully. Its not actually a new protocol state: its a state in the
1.1->2 programming API thunk that you appear tohave :).

> It's certainly possible -- it's just more code -- but our point is that permitting this does create a new state for an intermediary to handle.  (Though it's a reasonable argument that this state al…

I'd certainly make that argument: early responses have been possible
for some time, and I presume thats not exposed in the API in question,
or your layer above would already be able to detect this in some form.

-Rob

-- 
Robert Collins <[email-masked]>
Distinguished Technologist
HP Converged Cloud

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 15 Oct 2014 18:14:52 +1300

On 15/10/2014 11:51 a.m., Robert Collins wrote:
>> On the text itself, I would make it a SHOULD NOT.  A client can
>> already discard a response any time it wants, for any reason it
>> wants.  Why is this one in particular deserving of an absolute
>> prohibition?
>> 
>> Regardless, I don't think the issue is an incorrect implication
>> of a state change on the wire, though if you want to make that
>> clearer feel free.  There is an actual new state in the code that
>> has to be introduced to support this, even if it's conceptually a
>> "normal" RST_STREAM.
>> 
>> Upon receiving the RST_STREAM, we can't send any more body to the
>> server.  But we also can't stop the layer above us from pumping
>> body into us without aborting the request.
> 
> Isn' that dependent on the API you have? E.g. in python you could
> make data being pumped at that point raise a specific exception.
> And then no more data will be sent :).
> 
> different to HTTP/1.1 APIs where client code can assume that it
> just needs to write all the data and then and only then it can read
> the response is the problem: and this is indeed a difference to
> HTTP/1.1 :).
> 
>> With any other form of RST_STREAM, we could just fail attempts to
>> write more body, because the request was aborted by the server.
>> This form of RST_STREAM creates a new corner case that has to be
>> handled in an API layer or an intermediary -- continue accepting
>> body from the layer above, but route it to the bit bucket instead
>> of the wire.
> 
> I don't think thats desirable, because its wasted effort - better
> to signal the protocols capabilities back up the API stack more 
> faithfully. Its not actually a new protocol state: its a state in
> the 1.1->2 programming API thunk that you appear tohave :).
> 
>> It's certainly possible -- it's just more code -- but our point
>> is that permitting this does create a new state for an
>> intermediary to handle.  (Though it's a reasonable argument that
>> this state already existed and the text just calls attention to
>> it in case, like us, an implementer overlooked it.)
> 
> I'd certainly make that argument: early responses have been
> possible for some time, and I presume thats not exposed in the API
> in question, or your layer above would already be able to detect
> this in some form.
> 
> -Rob
> 
> 

And I have some evidence here to back it up.
http://bugs.squid-cache.org/show_bug.cgi?id=3791
http://bugs.squid-cache.org/show_bug.cgi?id=3017

Intermediaries have been handling the early-response case for some
years in HTTP/1.x.

The referenced bug reports are for NTLM challenge with large POST/PUT
data still being sent. IIS servers are particularly well known for
doing that since years back.

It also happens frequently with CONNECT requests when server errors
are returned. SPDY and WebSocket connections are the top candidates
for causing that event.

So yes, response completion before the request has finished being sent
by the client (or intermediaries) is neither unusual nor a new event
in HTTP. Your API is just lacking the asynchronous functionality to
notice it until now. Or perhapse your API does notice internally but
is one of the many clients who abort the whole TCP connection in these
cases.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 15 Oct 2014 16:45:30 +1100

> It's certainly possible -- it's just more code -- but our point is that permitting this does create a new state for an intermediary to handle.  (Though it's a reasonable argument that this state al…

I think that's where we're at, given the data from Amos (which seems right to me based upon my experience, FWIW).

One way we could go here would be to loosen the requirement (e.g., SHOULD NOT, ought not). Arguably, this should have been clarified in HTTPbis (since the semantic is independent of version).

What do people think? 

Regards,

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 15 Oct 2014 08:22:51 +0200 (CEST)

On Wed, 15 Oct 2014, Amos Jeffries wrote:

> So yes, response completion before the request has finished being sent by 
> the client (or intermediaries) is neither unusual nor a new event in HTTP.

Completely agree. It's been going on for a very long time in HTTP 1.x land.

-- 

  / daniel.haxx.se

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 22 Oct 2014 14:22:27 +1100

Ping. 

> On 15 Oct 2014, at 4:45 pm, Mark Nottingham <[email-masked]> wrote:
> 
>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 27 Oct 2014 14:06:25 -0700

Not hearing any further feedback, I’m going to say we have consensus to adopt Martin’s pull request <https://github.com/http2/http2-spec/pull/624>, with editorial discretion (but not obligation) to adjust the last requirement’s wording. Martin, please coordinate with Mike and Matthew when you do that, and please come back to us if you guys get stuck.

Regards,

> On 21 Oct 2014, at 8:22 pm, Mark Nottingham <[email-masked]> wrote:
> 
> Ping. 
> 

--
Mark Nottingham   http://www.mnot.net/