Subject: What error code when concurrent stream limit is exceeded

From: Martin Thomson <[email-masked]>
Date: Sun, 23 Nov 2014 11:26:15 -1000

PROTOCOL_ERROR:
https://github.com/http2/http2-spec/pull/651

PROTOCOL_ERROR or REFUSED_STREAM:
https://github.com/http2/http2-spec/pull/652

There's two views on this: the zero tolerance extreme who would
probably also choose to treat the stream error as a connection error;
the permissive one who don't like to punish errors more than is
necessary.  The full range of views is accommodated by the second PR,
so I prefer that marginally.  It still lets implementations fail as
hard as possible.

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 24 Nov 2014 09:08:18 +1000

> so I prefer that marginally.  It still lets implementations fail as
I don't follow your discussion; isn't a stream error of type PROTOCOL_ERROR
still just a stream error? I don't think it's cool to treat
RST_STREAM(PROTOCOL_ERROR) the same as GOAWAY(PROTOCOL_ERROR).

That said, I prefer #652. It allows implementations to say: "things are
cool, but I didn't read those HEADERS" vs. "stop breaking the protocol you
jerk." Depending on how egregious you consider a peer ignoring/exceeding
the stream limit. (Is that what you were saying?)

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Roberto Peon <[email-masked]>
Date: Sun, 23 Nov 2014 16:15:11 -0800

IMHO, we achieve the best interop in the end/long-term by being as strict
as possible about such things, however in this case (as already noted in
the spec):

  Limits in SETTINGS parameters cannot be reduced instantaneously,
   which leaves an endpoint exposed to behavior from a peer that could
   exceed the new limits.  In particular, immediately after establishing
   a connection, limits set by a server are not known to clients and
   could be exceeded without being an obvious protocol violation.

Thus, it is impossible to be absolutely strict here without forcing
undesirable client pessimism on the first RT or so.
It would be nice to say/remind that an implementation may wish to treat
exceeding the concurrent stream limit differently when the remote endpoint
may be unaware of the settings, e.g. before they've been ack'd (especially
so just after connecting).
-=R

On Sun, Nov 23, 2014 at 3:08 PM, Matthew Kerwin <[email-masked]>
wrote:

> I don't follow your discussion; isn't a stream error of type
> PROTOCOL_ERROR still just a stream error? I don't think it's cool to treat

---

From: Brad Fitzpatrick <[email-masked]>
Date: Sun, 23 Nov 2014 19:56:41 -0800

If the peer has ACKed as many SETTINGS as you've sent, and then proceeds to
violate the latest SETTINGS, it's obvious then that said peer just can't
count (and it wasn't a network latency race). If so, what else can't they
follow the directions of? I'm at least in favor of hanging up on them in
that case.

---

From: Roberto Peon <[email-masked]>
Date: Sun, 23 Nov 2014 20:07:45 -0800

Yup, modulo the lack of knowledge on the first RT or so...
-=R

On Sun, Nov 23, 2014 at 7:56 PM, Brad Fitzpatrick <[email-masked]> wrote:

> If the peer has ACKed as many SETTINGS as you've sent, and then proceeds
> to violate the latest SETTINGS, it's obvious then that said peer just can't

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 24 Nov 2014 07:24:38 +0100

> Yup, modulo the lack of knowledge on the first RT or so...

There's another case where network latency adds some uncertainty :
when clients abort some streams from time to time and consider they
can immediately open a new one as a replacement (the stop button or
Ctrl-F5). It is possible that for internal scheduling reasons or
flow control in the client, the abort is sent after the new stream
is presented, and that from time to time a new stream is rejected.

It would cause some interop issues to break the whole connection
when this happens, and the resulting effect will certainly be that
clients will be more careful and will ensure never to reach the
limit.

In parallel I agree for being strict.

I think it can make sense to consider that whatever exceed the
configured limit ends up with RST_STREAM and that a GOAWAY happens
if "too many" consecutive RST_STREAM have to be sent, or when the
hard limit is significantly passed over (eg: 10% more or double
etc). At least clients must know that there's a risk of getting a
GOAWAY if they do not respect the rules so that they are carefully
implemented.

Willy

---

From: Brad Fitzpatrick <[email-masked]>
Date: Sun, 23 Nov 2014 22:31:51 -0800

> is presented, and that from time to time a new stream is rejected.

That's just a bad client. The frames from the client to the server are
serialized over the one connection on which the limits apply.

If the client is keeping its state under different locking than it orders
its serialized frames on the connection, it's going to have all sorts of a
bad time in any case.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 24 Nov 2014 09:26:07 +0100

> its serialized frames on the connection, it's going to have all sorts of a

It just depends on the client architecture. If the client maintains a totally
asynchronous set of streams in certain tasks (say a thread per stream) and
relies on a lower layer to handle the connection, then enforces the stream
limit in the upper layers, it might be possible that the connection layer
will not be aware of it and does not necessarily serialize optimally.

Note, I'm not saying that it's the way it should be done, I'm saying that
protocols cannot always dictate the way software is implemented. Here it
seems obvious that the limit should be enforced at the connection layer,
except that I can easily imagine that when mapping some H/1 compatible
products to H/2, things could differ quite a bit for some time.

Willy

---

From: Michael Sweet <[email-masked]>
Date: Mon, 24 Nov 2014 06:46:46 -0500

What about proxies? They can advertise a limit but the upstream server may have different limits. Short of opening extra connections to support the required amount of concurrency, seems like it should be able to tell the client it has reached a limit and to try again later...

Sent from my iPad

> On Nov 24, 2014, at 1:31 AM, Brad Fitzpatrick <[email-masked]> wrote:
> 
> 
> That's just a bad client. The frames from the client to the server are serialized over the one connection on which the limits apply.
> 
> If the client is keeping its state under different locking than it orders its serialized frames on the connection, it's going to have all sorts of a bad time in any case.
>

---

From: Brad Fitzpatrick <[email-masked]>
Date: Mon, 24 Nov 2014 08:51:52 -0800

> products to H/2, things could differ quite a bit for some time.

I don't mean to be antagonistic or beat this issue to death, but I
disagree. A protocol may not dictate how software is implemented, but it
necessarily influences it very much, or at least it's allowed to impose
some constraints within the selected design.

To imagine an absurd example, an HTTP/2 client could just write random
bytes to the socket from random threads and hope to get an HTTP/2 response.
HTTP/2 requests are small enough, this strategy will eventually work. It's
not a good strategy though.

Slightly less absurd is to have a "totally asynchronous set of streams in
certain tasks (say a thread per stream)" all speaking mostly-valid HTTP/2,
only using a lower level to coordinate access to the socket, but not
coordinating actions such as "hey, this is thread wants to open a stream...
am I allowed to?" It's still not a good strategy. It might look like HTTP/2
at the frame level, and usually get lucky, but if it can't count basic
things and track what all its threads are doing, things like violating
MAX_CONCONCURRENT_STREAMS won't be its only problems.

I'm not saying implementations can't pick their favorite architecture, but
whatever they pick, it still has to play by the rules. The protocol
shouldn't be modified to make it easier for people who choose the option of
writing random bytes.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 24 Nov 2014 18:10:32 +0100

> not a good strategy though.

What I mean is that we're not necessarily talking about designing new
products but adapting existing H/1 products to H/2. One of the goals of
H/2 was to be able to adapt existing products, and many users will simply
think that only the encoding changes. I'm already seeing users maintain
connection pools in H/1 (very common in contexts of WebServices) and I
reasonably expect that these users will try to map their architecture
onto H/2. Such users will not necessarily realize that their concurrency
limit which is currently enforced by the number of tasks/threads/connections
or whatever will not necessarily be respected when they move to a shared
encoder which picks H/1 streams one at a time and encodes them to H/2 until
the output buffer is full.

> MAX_CONCONCURRENT_STREAMS won't be its only problems.

Don't get me wrong : I'm not trying to encourage bad use, I'm just suspecting
that we'll see a number of corner cases people do not expect at all. We're
already seeing exactly the sames with TCP and local source ports because
users don't know what an orphan is, and don't understand how they can run
out of ports when they "only" have a given number of connections at the
application level. When this happens, the impact is that only their new
connection cannot be opened, the old ones do not get killed. I think it
makes sense to apply the same strategy here.

Keep in mind that the alternative for people who don't understand what is
happening will be to start again to open as many connections as they need
streams, and that's clearly not what we want to see!

> whatever they pick, it still has to play by the rules.

Absolutely, provided they understand how their architecture copes with
the rules, which is not commonly the case from what I'm seeing in field.
We're regularly seeing protocols and applications running over TCP where
the *client* is required to close first! Seeing this is a proof of a
misunderstanding of the underlying protocols, and we can't always blame
users for that, at least we should educate them.

> The protocol

That's irrelevant to what we're discussing about.

Willy

---

From: Martin Thomson <[email-masked]>
Date: Mon, 24 Nov 2014 21:09:21 +0000

That is a separate problem. We are only talking about the violation of
settings here.

You are right that this is a general problem, but not a new one. This is
partly why we encourage large concurrency limits, and only use SHOULD when
talking about the single connection.

On Mon, Nov 24, 2014, 01:46 Michael Sweet <[email-masked]> wrote:

What about proxies? They can advertise a limit but the upstream server may
have different limits. Short of opening extra connections to support the
required amount of concurrency, seems like it should be able to tell the
client it has reached a limit and to try again later...

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 25 Nov 2014 07:56:17 +1000

So, back on track...

> follow the directions of? I'm at least in favor of hanging up on them in

Are you saying you want it to be a GOAWAY if you're confident the
client should know better?

As I read the PRs, the question is about what code to put in a
RST_STREAM, and as far as I'm aware there's no reasonable way to
upgrade RST_STREAM to GOAWAY, and it's pretty rude (or pathological)
to FIN without warning or reason. Isn't it counter to interop to send
an error signal that isn't prescribed by the spec?

We *could* add a bunch more words to get to the Connection
Error/Stream Error place, but I don't know if it's worth the extra
convolution. Maybe it is, if the words are good ones.

---

From: Mike Bishop <[email-masked]>
Date: Mon, 24 Nov 2014 23:12:06 +0000

From 5.4.1:
> An endpoint can end a connection at any time. In particular, an endpoint MAY choose to treat a stream error as a connection error. Endpoints SHOULD send a GOAWAY frame when ending a connection, providing that circumstances permit it.

Saying that something is a stream error contains the implication that it could be a connection error for certain implementations.  An implementation is free to consider anything that triggers a RST_STREAM for possible upgrade to a session termination based on whatever heuristics it wants to use.  And there's not a guarantee that you'll see a GOAWAY before the connection closes in any case, particularly if the server has decided you're malicious or deranged.

For this situation, my personal opinion is that an implementation might choose to go either way.  As Roberto has pointed out, after sending SETTINGS and before receiving the ACK, the client just doesn't know better.  Kill the stream (by the time they see the RST, they'll understand why) if you want, but probably don't P-E them.  After the ACK, it's up to the implementer whether they want to be strict or just take it as input to their DoS-prevention schemes.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Matthew Kerwin
Sent: Monday, November 24, 2014 1:56 PM
To: [email-masked]
Cc: Roberto Peon; Martin Thomson; HTTP Working Group
Subject: Re: What error code when concurrent stream limit is exceeded

So, back on track...

On 24/11/2014, Brad Fitzpatrick <[email-masked]> wrote:
> If the peer has ACKed as many SETTINGS as you've sent, and then 
> proceeds to violate the latest SETTINGS, it's obvious then that said 
> peer just can't count (and it wasn't a network latency race). If so, 
> what else can't they follow the directions of? I'm at least in favor 
> of hanging up on them in that case.
>

Are you saying you want it to be a GOAWAY if you're confident the client should know better?

As I read the PRs, the question is about what code to put in a RST_STREAM, and as far as I'm aware there's no reasonable way to upgrade RST_STREAM to GOAWAY, and it's pretty rude (or pathological) to FIN without warning or reason. Isn't it counter to interop to send an error signal that isn't prescribed by the spec?

We *could* add a bunch more words to get to the Connection Error/Stream Error place, but I don't know if it's worth the extra convolution. Maybe it is, if the words are good ones.

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 25 Nov 2014 09:39:22 +1000

On 25 November 2014 at 09:12, Mike Bishop <[email-masked]>
wrote:

> From 5.4.1:
> > An endpoint can end a connection at any time. In particular, an endpoint
> MAY choose to treat a stream error as a connection error. Endpoints SHOULD
> send a GOAWAY frame when ending a connection, providing that circumstances
> permit it.
>
> Saying that something is a stream error contains the implication that it
> could be a connection error for certain implementations.  An implementation
> is free to consider anything that triggers a RST_STREAM for possible
> upgrade to a session termination based on whatever heuristics it wants to
> use.  And there's not a guarantee that you'll see a GOAWAY before the
> connection closes in any case, particularly if the server has decided
> you're malicious or deranged.
> ​
> ​
>
>
Thanks, that's the reference I was missing. It kind of changes my worldview
a bit, I'll let it soak in for a while.

> ​
> ​
> ​​
​
> For this situation, my personal opinion is that an implementation might
> choose to go either way.  As Roberto has pointed out, after sending
> SETTINGS and before receiving the ACK, the client just doesn't know
> better.  Kill the stream (by the time they see the RST, they'll understand
> why) if you want, but probably don't P-E them.  After the ACK, it's up to
> the implementer whether they want to be strict or just take it as input to
> their DoS-prevention schemes.
>
>

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 26 Nov 2014 18:47:34 +1100

This is <https://github.com/http2/http2-spec/issues/649>.

From my reading, it seems like we're leaning towards this resolution: <https://github.com/http2/http2-spec/pull/652>

Anyone have a problem with that?

Cheers,

> On 25 Nov 2014, at 10:39 am, Matthew Kerwin <[email-masked]> wrote:
> 
> On 25 November 2014 at 09:12, Mike Bishop <[email-masked]> wrote:
> From 5.4.1:
> Thanks, that's the reference I was missing. It kind of changes my worldview a bit, I'll let it soak in for a while.
>  
> 
> ​For this situation, my personal opinion is that an implementation might choose to go either way.  As Roberto has pointed out, after sending SETTINGS and before receiving the ACK, the client just doesn't know better.  Kill the stream (by the time they see the RST, they'll understand why) if you want, but probably don't P-E them.  After the ACK, it's up to the implementer whether they want to be strict or just take it as input to their DoS-prevention schemes.
> 
> 
> 
> -- 
>   Matthew Kerwin

--
Mark Nottingham   https://www.mnot.net/

---

From: Mike Bishop <[email-masked]>
Date: Wed, 26 Nov 2014 18:14:06 +0000

Works for me.

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Tuesday, November 25, 2014 11:48 PM
To: Matthew Kerwin
Cc: Mike Bishop; [email-masked]; Roberto Peon; Martin Thomson; HTTP Working Group
Subject: Re: What error code when concurrent stream limit is exceeded

Cheers,

---

From: Brad Fitzpatrick <[email-masked]>
Date: Wed, 26 Nov 2014 11:22:01 -0800

Works for me too.

On Tue, Nov 25, 2014 at 11:47 PM, Mark Nottingham <[email-masked]> wrote:

> From my reading, it seems like we're leaning towards this resolution: <
> https://github.com/http2/http2-spec/pull/652>
>
> > > An endpoint can end a connection at any time. In particular, an
> endpoint MAY choose to treat a stream error as a connection error.
> Endpoints SHOULD send a GOAWAY frame when ending a connection, providing
> that circumstances permit it.
> >
> > Thanks, that's the reference I was missing. It kind of changes my
> worldview a bit, I'll let it soak in for a while.
> >
> > 
> > ​For this situation, my personal opinion is that an implementation might

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 29 Nov 2014 19:10:48 +1100

Marking editor-ready.

> On 26 Nov 2014, at 6:47 pm, Mark Nottingham <[email-masked]> wrote:
> 

--
Mark Nottingham   http://www.mnot.net/