Subject: HTTP/2 response completed before its request

From: Jesse Wilson <[email-masked]>
Date: Tue, 1 Jul 2014 10:27:45 -0600

Here’s an HTTP/2 scenario that we ran into
<https://github.com/square/okhttp/issues/929> with OkHttp…

   1. Our client POSTs a large image. In this case, large means “larger
   than the configured initial window size”.
   2. The server reads our request headers and sends a response (headers +
   body) immediately. The response includes an END_STREAM flag, indicating
   that no further response body frames are expected.
   3. The client continues to upload the request.
   4. The client upload exhausts the window and stalls.
   5. The server never sends a RST_STREAM or WINDOW_UPDATE frame, so the
   client eventually times out.

What’s interesting…

The server sent its complete response before the request completed.
Thinking in HTTP/1.1, I hadn’t anticipated this possibility. The response
code was 500, which suggests a crash in the server somewhere. Should the
spec mention this possibility? What are the obligations on the client in
this case? What would it mean for a server to return a 200 response to a
POST-in-progress?

The error also caused the server to lose our stream. This is a partition in
the CAP sense, and timeouts are the client’s failsafe to detect this.
HTTP/2 can suffer partitions in the application layer!
​

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 1 Jul 2014 18:34:39 +0200

On Tue, Jul 01, 2014 at 10:27:45AM -0600, Jesse Wilson wrote:
> Here???s an HTTP/2 scenario that we ran into
> <https://github.com/square/okhttp/issues/929> with OkHttp???
> 
>    1. Our client POSTs a large image. In this case, large means ???larger
>    than the configured initial window size???.
> What???s interesting???
> Thinking in HTTP/1.1, I hadn???t anticipated this possibility. The response

A 200 is not that common in this scenario, however 301/302 are fairly
common there (eg: session timed out, redirect to the home page). In 1.1,
it's said that the server should drain all the client's data to avoid
the problem of sending an RST to the client, and that the client should
stop uploading if it sees the connection is closed (which generally
happens after a redirect to another domain). We need to ensure we handle
this case gracefully in HTTP/2 as well.

> the CAP sense, and timeouts are the client???s failsafe to detect this.
> ???

Regards,
Willy

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 09:45:20 -0700

Getting a response before the request has finished definitely happens
sometimes, even in HTTP/1.1
-=R

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 1 Jul 2014 10:12:16 -0700

https://code.google.com/p/chromium/issues/detail?id=330860

---

From: Jesse Wilson <[email-masked]>
Date: Tue, 1 Jul 2014 11:45:16 -0600

Cool! So this is a normal case.

One corner case of a corner case is if the server's early response has more
than a window's worth of data...

Our client wouldn't acknowledge that data because it doesn't start reading
until it's done writing.

And if the server writes its early response before discarding/resetting the
request, that write might stall waiting for a WINDOW_UPDATE that won't ever
come. Deadlock!

(This is probably just a theoretical problem!)

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 1 Jul 2014 10:55:01 -0700

This same theoretical problem happens for HTTP/1.X over TCP. If the peers
don't call read() to pull the TCP data into user space, the kernel's TCP
stack will eventually shrink the receive window to 0. Of course, the TCP
receive windows will generally be larger than HTTP/2's initial windows
(64K).

On Tue, Jul 1, 2014 at 10:45 AM, Jesse Wilson <[email-masked]> wrote:

> One corner case of a corner case is if the server's early response has
> more than a window's worth of data...
>
> And if the server writes its early response before discarding/resetting
> the request, that write might stall waiting for a WINDOW_UPDATE that won't
> ever come. Deadlock!
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 1 Jul 2014 11:41:34 -0700

> receive windows will generally be larger than HTTP/2's initial windows

The stall is half of the problem, but do you cancel the send as well?
If the server has provided a response and closed the stream, there is
no point in continuing to send them data.  Especially if they forget
to send window updates.

RST_STREAM seems appropriate, but what do you think is the right code?

---

From: Jeff Pinner <[email-masked]>
Date: Tue, 1 Jul 2014 11:47:47 -0700

RST_STREAM NO_ERROR

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 1 Jul 2014 11:48:31 -0700

> RST_STREAM seems appropriate, but what do you think is the right code?

I think Jesse didn't explicitly say that the response didn't complete. The
application code at the client that issued the request hasn't started
reading the response. It's not pumping both the write and reads, just the
writes. Therefore, its receive window can shrink to 0 and stall the
server's sending of the response. Both the request and responses are
stalled on flow control.

---

From: Mike Bishop <[email-masked]>
Date: Tue, 1 Jul 2014 18:47:48 +0000

CANCEL, because the stream is no longer needed (because you got the response).  Or NO_ERROR, since the request completed “successfully.”

Sent from Windows Mail

From: Martin Thomson<mailto:[email-masked]>
Sent: ?Tuesday?, ?July? ?1?, ?2014 ?11?:?45? ?AM
To: William Chan (陈智昌)<mailto:[email-masked]>
Cc: Jesse Wilson<mailto:[email-masked]>, HTTP Working Group<mailto:[email-masked]>

---

From: Patrick McManus <[email-masked]>
Date: Tue, 1 Jul 2014 14:54:46 -0400

are you suggesting that as required or reasonable client behavior?

It's definitely reasonable. But it seems plausible to build an application
that doesn't cancel (perhaps these data segments are the moral equivalent
of FYI trailers that don't impact the response generation)- I don't see a
reason to require the cancel.

The only firm bug there seems to be the server not sending updates.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 1 Jul 2014 11:54:08 -0700

On 1 July 2014 11:48, William Chan (陈智昌) <[email-masked]> wrote:
> writes. Therefore, its receive window can shrink to 0 and stall the server's
> sending of the response. Both the request and responses are stalled on flow
> control.

Yes, but the fix is to pump reads too.  And if the server finishes the
response (i.e., we have a transition to half closed (remote) at the
client), then the client has no reason to continue to waste uplink
bandwidth.  So RST_STREAM NO_ERROR seems about right.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 1 Jul 2014 11:56:38 -0700

> are you suggesting that as required or reasonable client behavior?

Just reasonable.  It's perfectly OK to complete the request, however
long that takes.

> The only firm bug there seems to be the server not sending updates.

Definitely.  But who gets the blame for the stall?

---

From: Jesse Wilson <[email-masked]>
Date: Tue, 1 Jul 2014 13:01:45 -0600

Because the server "acted out of turn" by responding early, I think the
server should be on the hook to either send WINDOW_UPDATEs or cancel the
stream.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 1 Jul 2014 12:03:41 -0700

> Definitely.  But who gets the blame for the stall?

The server. Even if it doesn't actually use the request data, it needs to
send to /dev/null and send WINDOW_UPDATEs, until it gets to the point that
it can send the full response and send a RST_STREAM. I assert this to be
generally true for HTTP semantics, regardless of HTTP/1.X or HTTP/2 on the
wire.

---

From: Jeff Pinner <[email-masked]>
Date: Tue, 1 Jul 2014 12:13:18 -0700

the server should read until end_stream or discard with rst_stream

---

From: Zhong Yu <[email-masked]>
Date: Tue, 1 Jul 2014 14:21:07 -0500

> sometimes, even in HTTP/1.1

A server should not do that, or it will cause deadlocks with most
major browsers.

100-continue is supposed to be helpful in this case, but it's not
really adopted in practice.

Zhong Yu
bayou.io

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 19:29:56 +0000

In message <[email-masked]>, =?UTF-8?B?V2lsbGlhbSBDaGF
uICjpmYjmmbrmmIwp?= writes:

>The server. 

I'm not so sure.

The server acts sensibly:  A client which tries to POST a 1TB file to
http://example.com should not be able to tie up bandwidth and resources
for ages before the server can tell it to forget it.

The sensible client will stop transmitting on seing END_STREAM from
the server and put an END_STREAM on the next packet going out and
abandon the stream.

At the very latest, a modestly conforming client will discover
the servers END_STREAM when the window closes to zero, since it has
to shift to receive to get WINDOW updates.

The only question is what to do about non-modestly conforming clients
and intentionally malicious clients?

I would propose the server do something like the following:

	Issue no further credits towards the window, but let it
	close to zero.

	Discard all packets received on this stream.  If any of
	them contain END_STREAM, we're done.  (sensible clients)

	Once window is at zero, wait a couple of seconds.

	Issue a single window credit, only 100 bytes or so.

	If the next frame on this stream received from the client
	does not contain END_STREAM close the connection.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Michael Sweet <[email-masked]>
Date: Tue, 01 Jul 2014 15:34:30 -0400

Maybe not for web servers, but 100-continue is used for IPP all the time...

>>>> HTTP/2 can suffer partitions in the application layer!

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 1 Jul 2014 21:42:31 +0200

> really adopted in practice.

I disagree, and there are a number of situations where it's quite desirable
to act like this. For example, imagine that I'm uploading a large image to
a site and my session has expired. I want the site to send the error as soon
as possible so that my browser stops emitting for nothing. I don't want it to 
wait minutes just to know that I need to re-login first then try again.

Browsers already handle this quite well in 1.1, and the real issue in fact
tends to be on the server side where it's not always easy to drain all the
request from the client after the response was sent, which sometimes results
in a TCP RST which risks to clear the response before the client has a chance
to see it. But correctly done, it's a very useful feature.

Willy

---

From: Johnny Graettinger <[email-masked]>
Date: Tue, 1 Jul 2014 15:43:20 -0400

Like TCP, HTTP/2 streams are full duplex. From the server's perspective,
the stream was in a half-closed-local state. As in TCP, a stream in such a
state must be consumed or reset to avoid a stall. The server had the choice
of either, and did neither.

Without knowing the application context, it's impossible to say what would
have been sensible client behavior, but it doesn't matter here.

On Tue, Jul 1, 2014 at 3:29 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <
> [email-masked]>,
> =?UTF-8?B?V2lsbGlhbSBDaGF

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 1 Jul 2014 12:46:30 -0700

> A server should not do that, or it will cause deadlocks with most

There are two other conditions that have to be met, but usually aren't, in
order for this to deadlock browsers that don't pump both writes and reads
when sending out the request:
(1) The HTTP response doesn't fit in the browser TCP receive window/buffers.
(2) The server did not pump reads, thereby causing the browser to stall on
uploading the request data.

In practice, this rarely happens.

> >> >    2. The server reads our request headers and sends a response
> (headers
> >> > +
> >> > The error also caused the server to lose our stream. This is a
> partition
> >> > in

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 19:58:30 +0000

In message <CAEn92ToDQtLd=[email-masked]>, Johnny Graettinger writes

>of either, and did neither.

Sorry, but that's a rubish argument.  When the HTTP server has sent you 
a 3xx, 4xx or 5xx, that is like getting a RST on a TCP stream:  The other
end told you "We're done, go away" and that's why your application does
not even get asked before the TCP stack replies with RST.

>have been sensible client behavior, but it doesn't matter here.

We know the application context:  The server closed the stream, and
well-behaved clients can only respect that and shut up ASAP.

The servers problem is that this is YADAV[1] in HTTP/2, so it needs a
heurstic to tell inept clients form malicious clients.

Poul-Henning

[1] Yet Another Dos Attack Vector

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 1 Jul 2014 13:05:41 -0700

On Tue, Jul 1, 2014 at 12:58 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <CAEn92ToDQtLd=
> [email-masked]>, Johnny Graettinger
> writes
> :
>

I'm with Johnny here. A HTTP/2 stream roughly mirrors a TCP connection.

I think conflating HTTP message semantics with the transport is a mistake.
If the server really wants the other end to go away, why doesn't it just
kill the connection with a RST_STREAM?

Moreover, the suggestion that the client should terminate sending the
request with an END_STREAM flag sounds wrong. Isn't that actually modifying
the HTTP request? I would think that you'd want the client to end the
request with a RST_STREAM.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 20:10:54 +0000

In message <CAA4WUYiH+oi0O0KKJA2oN8CEYJBkhw2yP=[email-masked]>, =?UTF-8?B?V2lsbGlhbSBDaGF

>kill the connection with a RST_STREAM?

Because it wants the client to accept the 3xx, 4xx or 5xx response as
valid rather than discard it ?

>request with a RST_STREAM.

Again:  The server send a reply, the client accepted it, RST'ing the
stream seems wrong to me in that case.

---

From: Johnny Graettinger <[email-masked]>
Date: Tue, 1 Jul 2014 16:16:54 -0400

On Tue, Jul 1, 2014 at 4:10 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <CAA4WUYiH+oi0O0KKJA2oN8CEYJBkhw2yP=
> [email-masked]>, =?UTF-8?B?V2lsbGlhbSBDaGF

Ah. The client received a full and complete response from the server. In
the face of a RST_STREAM / NO_ERROR, there's no reason for it to discard it.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 20:21:34 +0000

In message <[email-masked]>, Johnny Graettinger writes

>the face of a RST_STREAM / NO_ERROR, there's no reason for it to discard it.

I think that's just making semantics more complicated than they need
to be.

Let END_STREAM mark orderly mutually understood closes, and keep RST
for when things actually go wrong in some bad way.

---

From: Jeff Pinner <[email-masked]>
Date: Tue, 1 Jul 2014 13:27:41 -0700

>>the face of a RST_STREAM / NO_ERROR, there's no reason for it to discard it.

In this case the client received an END_STREAM from the server since
it received a full response.

The stream was in the half-open (local) state on the client and
half-open (remote) state on the server.

> for when things actually go wrong in some bad way.

The server can't send END_STREAM because it's already closed from it's
side, RST is the only option if it ones to tear down state before the
client finishes sending the request.

The client can't send END_STREAM to close the response early since
that would not differentiate an incomplete payload body from a
complete one. It would need to send RST to indicate early termination
of the payload.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 20:31:57 +0000

>complete one. It would need to send RST to indicate early termination

Given that the server already said "Screw the payload", I can't see
what difference that could possibly make to anybody, but if RST
makes you more happy, then send RST...

The worst that can happen is that you pollute error-log on the
server.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 13:52:30 -0700

Note that it feels like the scenario is very contrived.

If either side (including the client) isn't reading, it cannot receive a
RST_STREAM, nor WINDOW_UPDATE or anything else.
Such a client or server would be almost guaranteed to fail.

So, essentially, the bug here would be on the side that isn't reading,
regardless of the endpoint being client or server.

Any side which ceases reading/examining received bytes can cause the
session to enter into a deadlock state.

-=R

On Tue, Jul 1, 2014 at 1:31 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <
> [email-masked]>, Jeff
> Pinner writes:
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 1 Jul 2014 14:00:25 -0700

On 1 July 2014 13:52, Roberto Peon <[email-masked]> wrote:
> Any side which ceases reading/examining received bytes can cause the session
> to enter into a deadlock state.

reading, and acknowledging, yes

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 21:00:58 +0000

>session to enter into a deadlock state.

How would that ever work ?

To send any data you need to receive WINDOW updates, so there is no
way to stop paying attention to incoming frames and expect things to
work in any way.

The most likely way that could happen is mutual TCP window lockout,
but that can only happen if boths sides are either primitively
half-duplex or broken in some other way.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 14:10:32 -0700

On Tue, Jul 1, 2014 at 2:00 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <
> [email-masked]>,
> Roberto Peon writes:
>

Correct.
If an endpoint doesn't want to receive more data, it needs to set max
streams to zero, flow control to zero, etc.
Even then, it needs to read from the socket in order to respond
appropriately to PING, WINDOW_UPDATE, SETTINGS etc.

An HTTP2 endpoint must always be attempting to read from the socket.
It must always (eventually) read the amount of data it promised w.r.t. its
flow control window if it is to avoid deadlock.
It can always throw it away and RST the stream, however.

-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 21:12:15 +0000

>An HTTP2 endpoint must always be attempting to read from the socket.

... but servers would make themselves vulnerable to a number of DoS
attacks if they assume clients do so.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 14:15:06 -0700

Howso?

A server worried about the amount of memory it must use simply sets its
window size such that the client (assuming it was working properly) would
stop sending more data.
That is the reason for the flow control...

-=R

On Tue, Jul 1, 2014 at 2:12 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <
> [email-masked]>,
> Roberto Peon writes:
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 22:35:29 +0000

>Howso?

(Changed subject)

In a sense this is the evil twin of one of the design-goals:  We
want the white-hat clients to be able to send a load of requests
at the server as fast as possible, to "improve user experience".

Problem is, black-hat clients get to abuse that welcome-mat too.

The fundamental problem is that HTTP/2 fails to extract any
proof-of-good-faith from the client, it will be perfectly possible
to do things like

	nc http://www.example.com > /dev/null < huge_file

There is not a single place in the HTTP/2 protocol where the client
is actually forced to pay attention:  There are no randomly chosen
ID#'s to return to sender or unpredictable initial windows --
everything is perfectly predictable or at least replayable.

The claim that "HTTP2 streams are like TCP" is rubbish:  None of
the many expensive lessons of TCP -- or for that matter NCP -- have
been incorporated.

In fact, what HTTP/2 is proposing to do is exactly what sunk T/TCP
as a HTTP accelerator a decade ago:  Gaining perceived performance
by eliminating the "proof-of-willingness-to-work-for-it" requirement
from clients.

Admittedly it does so at a higher level than with T/TCP so at
least the attacker has to have a legitimate IP number this time,
but still...

Even if the client totally "fails" to read from the TCP socket (a
well known strategy in spamemail) the server is still tricked to
waste signficant amounts of work before it can detect that it is
under attack from black hats.  (Servers attempting to mitigate this
with non-blocking writes may only make matters worse, since they
will expend even more CPU & syscalls and still not get anyway.)

I do not want to give the impression that this is a trivial problem
to solve:  There is no material difference between a millon-machine
botnet fetching your home-page and a #neilwebfail.

But HTTP/2 doesn't do *anything* to mitigate even the kind of
"tape-recorder replay" attacks we have known about for more than
25 years, and which TCP was specifically designed to resist.

I don't know exactly how many machines it will take to DoS any given
HTTP2 implementation, but we're probably not even talking 500, maybe
as low as 100 will do.  (We should try to determine this number
before somebody gets a DefCon slot just to tell us how little it
takes.)

It's too late to address DoS attacks comprehensively in HTTP/2, no
matter what, but we can and should remove some of the most obvious
bulls-eyes. (CONTINUATION, no flow-control on HEADERS, add
max HEADER size and initial WORK-QUOTA in SETTINGS etc.)

For HTTP/3 we should look seriously at how servers can extract
situation-tuned proof-of-work "payments" from clients as condition
for expending service.

Under normal circumstances the server just hands out trivial tasks
(2234242342 + 124123123 = ?), but when a DoS is suspected, the
clients are asked to solve gradually more time consuming tasks (find
a MD5 input having hash 0x3495823543????????...), in order to limit
the number of requests per client dynamically.

It makes a really big difference if a black-hat client can spew
requests at you at wire-speed or only at a server-chosen rate
scaled to your CPU speed at the servers discretion.

The demand that a client provide proof of one second of calculations
per request will inconvenience a large class of the trivial DoS
attacks HTTP/2 is wide open for, while still being only a minor
annoyance for legitimate users, who'd rather receive slow service
than no service at all.

All this of course presumes that the WG actually cares about
DoS attacks.  I get the feeling I'm the only one who does ?

Poul-Henning

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 16:33:29 -0700

It would have been good to have had you sit in while the group involved in
creating HTTP2 discussed DoS considerations, which have been brought up
consistently over the course of the development of the protocol.

I'll attempt to rehash a small amount here.

T/TCP is not a good analog for HTTP2, and had completely different attack
surface.

To date, the browser (or "internet") usecase is likely to use TLS, since it
actually works.
If we see widespread deployment of HTTP2 in the clear, it would end up
being no worse than HTTP1 in terms of DoS potential, which is acceptable
(if not optimal) today.

In any case, your example is only useful for servers or clients which are
not deploying over TLS.
The "replay" attack vector you've chosen to bring up is not interesting in
such cases.

Lets assume, then, that we're worried about DoS attacks where the attacker
is using TLS.
In such cases, replay attacks are significantly more difficult.
That leaves non-replay attacks.

Regardless of choosing to deploy with/without TLS, when under DoS attack,
one sets the various SETTINGS to conservative values thus, as a function of
load/resource pressure, dynamically reduces the amount of memory, the
number of parallel streams one is willing to accept, perhaps the maximum
header size one is willing to accept, etc.

With the document as it is now, one might be forced to /dev/null bytes for
an RT to ensure that the more conservative values are used.
We'll see if this is good enough or not-- the fix is fairly simple: either
persist these values across connections, or have a separate profile (e.g.
h2-c) which the server offers in preference or in lieu of h2 which would
have conservative settings by default.

If you're worried about replay of cookies, etc, then yes, there is little
we can do here while continuing to use HTTP semantics as they exist today.
In such cases one should really be using TLS.

The question that should be asked is:
How is HTTP2 worse than HTTP1 in terms of DoS?
With HTTP2 servers can:
  - specify a truly tiny max-state size.
  - reduce the number of connections accepted

Given the ability in HTTP2 to set the amount of memory one is willing to
consume for any connection, and that the minimum state per connection can
be counted in a small number of ints, I think your concern doesn't have a
lot of merit.

-=R

On Tue, Jul 1, 2014 at 3:35 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <CAP+FsNcL=pJXhi2nZZgVBY2M3gPaMGiVFBqRWmAS_=
> [email-masked]>, Roberto Peon writes:
>
> >Howso?
>
> (Changed subject)
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 01 Jul 2014 23:48:09 +0000

>consistently over the course of the development of the protocol.

Yeah, well, sorry for not having a budget to spend on HTTP/2...

That said: If DoS has been brought up consistently, it seems to have
have very little to show for it.

>being no worse than HTTP1 in terms of DoS potential, which is acceptable

I disagree, defending non-TLS HTTP/2 against DoS attacks is going to be
(much) harder than defending HTTP/1

>In any case, your example is only useful for servers or clients which are

... because TLS provides the mechanism for "proof of work" for you which
plaintext HTTP/2 lack.

>one sets the various SETTINGS to conservative values thus, 

There is no setting for "I'm going to service only this many reqyests
so don't send any more until I increase it"

There is no setting for "I'm not accepting HEADERS longer than..."

There is no setting for "I'm not accepting CONTINUATION at all..."

So sorry, but no cigar...

>  - reduce the number of connections accepted

... but they can not limit the amount of work per connection.

>be counted in a small number of ints, I think your concern doesn't have a

I'd say you're not thinking creatively enough.

But here's an idea:

1.  Set up a HTTP/2 server.

2.  Announce on Defcon that it is more resistant to DoS than a HTTP/1 server.

3.  Pop-Corn!

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 17:09:46 -0700

On Tue, Jul 1, 2014 at 4:48 PM, Poul-Henning Kamp <[email-masked]>
wrote:

> In message <CAP+FsNdAmQbzSnNE9CHM8i1j6oyJ194HnVVm=
> [email-masked]>, Roberto Peon writes:
>
>

You've effectively stated that you don't believe in it and would have
nothing to do with it in the past.
I'm guessing this had little to do with budget.

In any case, you would be more productive in engaging with the group by
asking neutrally-biased questions and positing scenarios instead of making
statements-of-fact which may not be correct. This is true regardless of how
smart or wise one is, so long as it requires a group of folks to create
change.

> have very little to show for it.

You have yet to prove that, though it is another an interesting theatrical
statement.

> (much) harder than defending HTTP/1

Howso?
I've told you how to do it. Tell me how it doesn't work, or deploy and find
out for yourself, as we are doing.

> so don't send any more until I increase it"

Factually incorrect.
SETTINGS_MAX_CONCURRENT_STREAMS does exactly that.

> There is no setting for "I'm not accepting HEADERS longer than..."

No, you send the data to /dev/null after processing the HPACK stuff (which
is cheap), then return with a 413.
The problem of large headers is not unique to HTTP2, and is arguably worse
in HTTP1 with its many more connections.

> There is no setting for "I'm not accepting CONTINUATION at all..."

Yes, because that is orthogonal and has zero to do with DoS, and zero to do
with header size as written today regardless of how many times you attempt
to state that the naked emperor is wearing clothes.

> I'd say you're not thinking creatively enough.
In a debate, the onus of finding an unworkable scenario depends on the
other party (that'd be you).

> 2.  Announce on Defcon that it is more resistant to DoS than a HTTP/1

The statement can still be true, even while eating popcorn and watching the
server melt down.

'more resistant' != 'impervious'.
Nothing is completely impervious to a large enough DoS attack.

-=R

---

From: Zhong Yu <[email-masked]>
Date: Tue, 1 Jul 2014 22:02:41 -0500

> Browsers already handle this quite well in 1.1, and the real issue in fact

All the browsers I tested (firefox/chrom/safari/IE) appear to be
half-duplex - they will not read the response until the request body
is completely sent. A server can send an immediate response before
reading the request body, but the browser won't read the response
immediately.

Since sending the response before draining the request body carries
the risk of deadlock, it's probably better to drain the request body
before sending the response. That is, the server is forced to do
half-duplex, because most clients do half-duplex.

Zhong Yu
bayou.io

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 20:07:16 -0700

In HTTP2, however, it works differently since the browser must always read,
and both sides must respect flow control.

You need to try pretty hard to get it into a pathological case that
deadlocks things (e.g. overly-large/infinite/non-existent flow control
window which the application is unable/unwilling to actually adhere to
coupled with more data sent than the application is willing to read).

For my part, I would not change how the server works.
I'd have the server drop the connection to any endpoint for HTTP2 that was
not reading what the server was sending it.
Similarly, any client should drop the connection to any endpoint that was
not reading what it was sending it.

-=R

---

From: Jesse Wilson <[email-masked]>
Date: Tue, 1 Jul 2014 21:47:23 -0600

Although OkHttp's network layer is constantly reading from the socket, it
won't acknowledge response body data until it's been consumed by the
application layer. And the application layer won't consume the response
until after it's done transmitting the request.

So we're vulnerable to deadlock because our application layer is not
concurrent and our network layer refuses to buffer an unbounded amount of
data.
 On Jul 1, 2014 9:07 PM, "Roberto Peon" <[email-masked]> wrote:

> In HTTP2, however, it works differently since the browser must always
> read, and both sides must respect flow control.
>

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 20:59:44 -0700

I'd expect that OkHttp sends WINDOW_UPDATE for the connection-level flow
control, but doesn't send WINDOW_UPDATE for the receiving stream until the
application has consumed it.
If it works this way, then there is no deadlock.

If the implementation needs to be stingy with memory, then it should set a
small or zero default window size, and send a WINDOW_UPDATE at the end of
the request, allowing the server to respond then. This would force
half-duplex behavior on response entity-bodies for that stream.

-=R

On Tue, Jul 1, 2014 at 8:47 PM, Jesse Wilson <[email-masked]> wrote:

>> I'd have the server drop the connection to any endpoint for HTTP2 that
>> was not reading what the server was sending it.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 21:00:16 -0700

> half-duplex behavior on response entity-bodies for that stream.

.. and in such a case, again, there is no deadlock.
-=R

>>> Similarly, any client should drop the connection to any endpoint that
>>> was not reading what it was sending it.
>>>
>>> -=R
>>>
>>>

---

From: Jesse Wilson <[email-masked]>
Date: Tue, 1 Jul 2014 23:41:53 -0600

To be clear, only one stream is deadlocked. When the server and client both
write the same stream simultaneously, they can both become blocked waiting
for WINDOW_UPDATEs that aren't coming.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 02 Jul 2014 05:57:42 +0000

>I'm guessing this had little to do with budget.

It had a lot to do with budget:  If there were no chance of getting
a good protocol out of it, I had better things to spend my money on.

(In difference from a lot of you, I run a one-man company, so slack
or "strategic" funds are in very short supply.)

I'll leave it to the black hats to settle the debate.

Since it seems HTTP/2 is just going to be a short lived stopgap on top
of TLS only, maybe it will never become a real problem.

In HTTP/3 we'll have to be serious about it.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 23:02:43 -0700

Not even that stream one is deadlocked. It is simply paused until the
request has been fully sent.
-=R

On Tue, Jul 1, 2014 at 10:41 PM, Jesse Wilson <[email-masked]> wrote:

> To be clear, only one stream is deadlocked. When the server and client
> both write the same stream simultaneously, they can both become blocked
> waiting for WINDOW_UPDATEs that aren't coming.
>

---

From: "Eric J. Bowman" <[email-masked]>
Date: Wed, 2 Jul 2014 00:38:41 -0600

> In HTTP/3 we'll have to be serious about it.

My disillusionment with the HTTP/2 process stems from this concept that
it doesn't need to be "gotten right" because we'll address any problems
in HTTP/3. Am I the only one who thinks the horse should come before
the cart?

-Eric

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 23:41:38 -0700

phk@ has yet to demonstrate that there is an actual problem with DoS for
HTTP2 in the first place, and continues to make statements of fact about
things where he is demonstrably misinformed.

-=R

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 02 Jul 2014 06:46:34 +0000

>in HTTP/3. Am I the only one who thinks the horse should come before

What really surprises me is that we see such proposals to name&shame
proxies which do not allow random private extensions through, but
no proposals to name&shame browsers which do not want to support
HTTP/2 upgrade ?

The goals are obviously not to ensure the widest possible adoption
of HTTP/2.

I certainly looks like a number of WG participants are much more
focuses on getting HTTP/2 to work for their own private, (soon to
be walled ?), garden, than to make HTTP/2 the best possible protocol
for the web as such.

---

From: "Eric J. Bowman" <[email-masked]>
Date: Wed, 2 Jul 2014 00:51:20 -0600

> be walled ?), garden, than to make HTTP/2 the best possible protocol

Exactly. Why are so many folks talking about HTTP/3 as a solution to
the shortcomings of HTTP/2 when HTTP/2 isn't even in LC? If HTTP/2 were
"getting it right" then why all the talk of deferring proper
architecture to HTTP/3? So discouraging...

-Eric

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 2 Jul 2014 08:59:01 +0200

> generally true for HTTP semantics, regardless of HTTP/1.X or HTTP/2 on the

There's something I don't understand here. We used to say that one of the
benefits of h2 vs h1 was that it was possible to abort a stream without
breaking a connection, but here it seems to be the complete opposite.

Does the same problem happen when the client is a browser which retrieves
a very large file and the user presses "Stop" during the download ? Does
the browser continue to read all data and only stop rendering it in this
case ? That does not make much sense, because the primary purpose for
"Stop" is to release the small DSL or POTS line and be able to quickly
use it for another request. Here if every time a user presses Stop, the
client has to break the whole connection to abort, it possibly means
stopping ongoing downloads and such things, which is really not appropriate.
At least in HTTP/1 when you stop a large file transfer, you don't stop other
transfers from the same server. The typical case is on download sites, you
have a download of a tar.bz2 coming and at the same time you click on a
MANIFEST file which is huge and you're not interested in downloading.

Can someone enlighten me on this ?

Thanks,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 2 Jul 2014 09:09:06 +0200

> no point in continuing to send them data.  Especially if they forget

I don't agree.  It is certainly possible to write server application code
that commits a response and then processes the request data.     It's not a
very good way of doing it and probably invites all sorts of application
races.... but i don't think the transport layer can conclude that it is
necessarily a bad thing.

regards

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 2 Jul 2014 09:21:51 +0200

It think we need to stop conflating application state with the stream
state.   The state diagram (despite it's limitations) in the spec says it
all - a stream is close when both directions have sent/received end
stream.   There is no requirement for one direction to end_stream before
the other.

This is exactly the same in HTTP/1 with TCP/IP half closed connections.
There is no requirement at all on a TCP/IP connection for data in one
direction to shutdown before data in the other direction, so good HTTP
implementation have to just deal with that complexity.  This allows HTTP
responses to be sent even with a FIN to terminate content, before the
request has been completely sent.

HTTP/2 should do the same... in fact I could argue that our charter
requires us to do so :)

cheers

---

From: Mike Belshe <[email-masked]>
Date: Wed, 2 Jul 2014 04:30:59 -0700

> architecture to HTTP/3? So discouraging...

There is no real discussion of HTTP/3.  PHK proposed HTTP/3 discussion as a
bucket to catch the same arguments that he lost in HTTP/2 discussions. As
far as I can tell, he is the only one talking about it.  The rest of us are
still working on HTTP/2.

Mike

---

From: Julian Reschke <[email-masked]>
Date: Wed, 02 Jul 2014 14:13:58 +0200

On 2014-07-02 13:30, Mike Belshe wrote:
> ...
> There is no real discussion of HTTP/3.  PHK proposed HTTP/3 discussion
> as a bucket to catch the same arguments that he lost in HTTP/2
> discussions. As far as I can tell, he is the only one talking about it.
>   The rest of us are still working on HTTP/2.
> ...

Actually, the term "HTTP/3" as the thing we may or may not do after 
HTTP/2 has come up a lot.

In HTTP/2 we were constrained both by charter (being able to convert 
from/to 1.1) and the proposal we started with (and the audience pushing 
for it).

I hope that in future work we'll work on issues *other* than performance 
again.

Best regards, Julian

---

From: Mike Bishop <[email-masked]>
Date: Wed, 2 Jul 2014 15:43:08 +0000

Kind of.  In HTTP/1.1, if you’ve started receiving entity body and decide you don’t want it any more, you have to RST the TCP connection.  That doesn’t mean you keep reading to the end, but it does mean that until the server sees the RST, they’re going to keep sending data your way and it gets dropped either by your TCP stack or some intermediary who’s tracking TCP session state.

In HTTP/2, you can do the same thing, RST the stream.  But the scenario here is when the request is completed, not because the client doesn’t need the response any more, but because the server already sent the response and doesn’t need the request body.

I think it’s entirely appropriate for the client to RST_STREAM NO_ERROR in this situation, to alert the server that it’s abruptly stopping the (unneeded) entity body.  The concern is that complementary bugs could interact in the wild:

  *
Client doesn’t read HEADERS/DATA until it has finished sending the body
  *
Server doesn’t send WINDOW_UPDATE on a half-closed stream

Sent from Windows Mail

From: Willy Tarreau<mailto:[email-masked]>
Sent: ?Wednesday?, ?July? ?2?, ?2014 ?12?:?02? ?AM
To: William Chan (陈智昌)<mailto:[email-masked]>
Cc: Martin Thomson<mailto:[email-masked]>, Patrick McManus<mailto:[email-masked]>, Jeff Pinner<mailto:[email-masked]>, Jesse Wilson<mailto:[email-masked]>, HTTP Working Group<mailto:[email-masked]>

Thanks,
Willy

---

From: Patrick McManus <[email-masked]>
Date: Wed, 2 Jul 2014 11:51:05 -0400

On Wed, Jul 2, 2014 at 2:59 AM, Willy Tarreau <[email-masked]> wrote:There's
something I don't understand here. We used to say that one of the

> breaking a connection,

right!

> a very large file and the user presses "Stop" during the download ?

no - when the user presses stop the client sends a RST_STREAM,CANCEL and
when the server receives that it stops sending data frames. so up to 1
wasted BDP (or so - depending on over buffering) of data flows, but then
the established connection is free to do other things.

Does
> case ?

it does end up reading that BDP worth of data after the cancel and tosses
it away, sure. But it expects the CANCEL will mean that data doesn't just
flow until natural END_STREAM.

> Can someone enlighten me on this ?
I think the confusion is that the okHttp case was about an upload and a
server that responded to that upload before it was complete.. (it didn't
cancel it - the http semantics were complete).. the question is what should
happen to the rest of the upload stream.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 2 Jul 2014 18:04:23 +0200

Hi Pat,

> happen to the rest of the upload stream.

OK thanks for explaining. But I still fail to see the difference. And upload
and a download are exactly the same, in both cases we're sending a message
body and any party should be free to abort receipt saying it doesn't want it
anymore. I suspect that some intermediate states are missing. Reminds me when
I was pulling my hair on connection management in haproxy until I found I had
no other solution but adding extra closing states to handle the various end
conditions...

Willy

---

From: Martin Thomson <[email-masked]>
Date: Wed, 2 Jul 2014 09:14:35 -0700

> races.... but i don't think the transport layer can conclude that it is

If you want to provide a positive answer that is actually conditional
on something that hasn't happened yet, then you will get what you
deserve.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 2 Jul 2014 12:15:23 -0400

*All the browsers I tested (firefox/chrom/safari/IE) appear to be
half-duplex - they will not read the response until the request body is
completely sent.*
in h1 they are being half-duplex at the application stream level.. but not
at the tcp level. i.e. they are still reading the incoming TCP acks -
otherwise they wouldn't be able to continue to send the request body at
some point due to rwin exhaustion.

in h2 the equivalent of those stream tcp acks are received window updates -
so an h2 client will always be reading the tcp stream to find them even
while generating an upload stream - it simply has to. If it picks up some
data frames along the way its an implementation choice on when to pass
them to the stream sink and when to generate updates that cover them.
(presumably a related choice :))

---

From: Jason Greene <[email-masked]>
Date: Wed, 2 Jul 2014 11:26:12 -0500

On Jul 2, 2014, at 10:43 AM, Mike Bishop <[email-masked]> wrote:

> 	• Client doesn’t read HEADERS/DATA until it has finished sending the body
> 	• Server doesn’t send WINDOW_UPDATE on a half-closed stream

Is it RST_STREAM CANCEL or NO_ERROR? 

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: Patrick McManus <[email-masked]>
Date: Wed, 2 Jul 2014 12:31:24 -0400

On Wed, Jul 2, 2014 at 12:04 PM, Willy Tarreau <[email-masked]> wrote:

> anymore.

I agree - you just used the canceled download as an example - so I stayed
on track with it as it would be much more common that the scenario in this
thread.

In the okhttp case the server is the one receiving a stream it might not
want. Its state is half-closed-local (because it has already generated the
END_STREAM) and from that state it can send a RST_STREAM, NO_ERROR and the
client ought to stop uploading stuff when it receives that.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 2 Jul 2014 16:33:25 +0000

I believe that was part of the original question, however many hundred e-mails ago.  :-)

Either seems applicable (no longer needed, nothing wrong), but I'm leaning toward NO_ERROR when coming from the client, since the request did complete successfully.  CANCEL suggests to me that the response is no longer interesting.

-----Original Message-----
From: Jason Greene [mailto:[email-masked]] 
Sent: Wednesday, July 2, 2014 9:26 AM
To: Mike Bishop
Cc: Willy Tarreau; "William Chan (陈智昌)"; Martin Thomson; Patrick McManus; Jeff Pinner; Jesse Wilson; HTTP Working Group
Subject: Re: HTTP/2 response completed before its request

On Jul 2, 2014, at 10:43 AM, Mike Bishop <[email-masked]> wrote:

> 	? Client doesn’t read HEADERS/DATA until it has finished sending the body
> 	? Server doesn’t send WINDOW_UPDATE on a half-closed stream

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red Hat

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 2 Jul 2014 19:06:59 +0200

> client ought to stop uploading stuff when it receives that.

OK so it's more of an implementation issue than a protocol design issue ?
Then maybe the case should be documented while it's hot so that nobody gets
trapped again.

Thanks
Willy

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 2 Jul 2014 10:11:00 -0700

On Wed, Jul 2, 2014 at 10:06 AM, Willy Tarreau <[email-masked]> wrote:

> > > body and any party should be free to abort receipt saying it doesn't
> want
> > > it
> > > anymore.
> > >
> >

Yes, it's an implementation issue. Maybe add a new bullet to the
implementation advice wiki (https://github.com/http2/http2-spec/wiki/Advice
)?

---

From: Jason Greene <[email-masked]>
Date: Wed, 2 Jul 2014 12:14:55 -0500

Ok that makes sense. From my perspective, CANCEL felt like it only makes sense for senders, and that sounds like what you are saying. Although, I guess in practice it doesn’t really matter it only affects error logs and such.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 2 Jul 2014 17:21:36 +0000

I see use cases for CANCEL in a couple of places:
  - Sender of a request no longer wants the response.  Cancel the request.
  - Recipient of a PUSH_PROMISE discovers it already has the resource in cache and doesn't want another copy.  Cancel the push.
  - Sender of a PUSH_PROMISE discovers it no longer has the resource it intended to push.  Cancel the push.

(Or maybe they *are* all senders, and push blurs the line about who's the sender....)

-----Original Message-----
From: Jason Greene [mailto:[email-masked]] 
Sent: Wednesday, July 2, 2014 10:15 AM
To: Mike Bishop
Cc: Willy Tarreau; "William Chan (陈智昌)"; Martin Thomson; Patrick McManus; Jeff Pinner; Jesse Wilson; HTTP Working Group
Subject: Re: HTTP/2 response completed before its request

On Jul 2, 2014, at 11:33 AM, Mike Bishop <[email-masked]> wrote:

> Cc: Willy Tarreau; "William Chan (陈智昌)"; Martin Thomson; Patrick 
> McManus; Jeff Pinner; Jesse Wilson; HTTP Working Group

---

From: Zhong Yu <[email-masked]>
Date: Wed, 2 Jul 2014 14:34:28 -0500

I feel that maybe the two directions of a stream should have
independent RSTs, so that we can model more error cases.

An end point can issue an RST for its outbound only, meaning that its
outbound stream is abruptly terminated. Meanwhile its inbound is still
intact, and it is still accepting inbound data.

An end point can also issue an RST for its inbound only, meaning that
it's no longer interested in inbound data, and the peer should stop
sending more data. Meanwhile the outbound is still intact, and it *is*
still writing to the outbound.

Zhong Yu
bayou.io

---

From: Martin Thomson <[email-masked]>
Date: Wed, 2 Jul 2014 13:03:33 -0700

> independent RSTs, so that we can model more error cases.

I think that is unnecessary, and it might even be a big mistake.

An endpoint that doesn't want to continue sending can stop sending.
An endpoint that wants to stop receiving can cease sending window
updates (and buffer what it already provided credits for).  It needs
to do the latter anyway, since it can't do anything about what a peer
has already committed to send.

Would we develop rules around reciprocation for a half-RST?