Subject: Flow control for server push

From: Martin Thomson <[email-masked]>
Date: Mon, 2 May 2016 12:26:55 +1000

An issue has come up in webpush that I think bears some discussion here.

There is no flow control for server pushes that don't include data.

As long as a server push contains no data, there is no way to
practically limit to how many pushes can be generated by the server.
We have several mechanisms that might have worked, but none of these
actually engage in this case:

  - Concurrent stream limit: PUSH_PROMISE causes the promised stream
to enter a reserved state, which is not counted toward the stream
limit.  The HEADERS on that stream that completes the push will
immediately end the stream, so the concurrent streams count never
really increases.

 - Flow control: PUSH_PROMISE and HEADERS are the only messages that
are sent in this case.  Flow control for the affected stream never
enters the picture, and connection-level flow control isn't touched
either because no message payloads are sent.

 - TCP receive window: As a measure of last resort, the TCP receive
window will eventually close.  However, the best guidance we've given
implementations is to drain the TCP receive window as quickly as
possible to avoid head of line blocking and other stalling problems.
That means that a good implementation will leave this window as wide
open as possible.  In that case, the only effective limit on the rate
of inbound pushes will be the bandwidth of the connection bottleneck.

Given effective compression of the PUSH_PROMISE and HEADERS, the
number of actual pushes that could be generated is likely very high,
even if each one contains unique information.  This is not likely to
be a problem for web browsing cases, but it could be a problem for
other applications using HTTP.

The only mitigation we currently have is application-specific changes,
but I think that isn't ideal.  If things like pushing 304 are to
become more widespread, then I think that we might need to do
something about this.

Two options have been proposed:

1. A header field that limits the number of pushes in response to any
given request, maybe something that builds on Herve's push policy
work.

2. Explicit acknowledgment of each push, plus separate configuration
for the maximum number of unacknowledged pushes.  This probably needs
to be an optional, negotiated feature.

I'm interested in what people think here.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 2 May 2016 07:20:35 +0200

Hi Martin,

> There is no flow control for server pushes that don't include data.
(...)
> I'm interested in what people think here.

I think that if the purpose is to protect the client against a huge amount
of pushes, #1 would do a pretty good job and makes sense in general. It
more or less means that any agent supporting push will have to be prepared
to allocate resources to deal with these responses, and that it will sometimes
be aware of some limits. In fact SETTINGS_ENABLE_PUSH could have been a
limit instead of a bit, since not supporting push is equivalent to setting
its limit to zero.

However if the purpose is to limit network usage or protocol processing
work, then I think that we could communicate a limit on the number of
response headers. It may seem awkward at first, but over the long term
it may prove useful for a variety of purposes (eg: I'm not a browser, I
don't care about a lot of things, please be short). Then the server would
have to prevent itself from sending more pushes if the cumulated number
of push_promise+response headers reaches a limit. The limit would be used
as a budget, refilled with each request. For sure some header fields are
encoded as a single byte while others take 50 bytes, but their number does
not vary much and their average size remains more or less stable between
subsequent requests/responses over a same connection. Thus a client could
say "don't flood me with more than 200 headers fields at once" and will
know that it'll get a few push responses at best, representing at best a
few TCP segments. If I'm focusing on the number of header fields instead
of their encoded size, it's because it's significantly harder to consider
the encoding size before deciding to encode (we had an endless discussion
here regarding this in the past resulting in headers not being accounted
for the flow control).

Just my few cents,
Willy

---

From: Martin Thomson <[email-masked]>
Date: Mon, 2 May 2016 15:36:14 +1000

Hi Willy,

On 2 May 2016 at 15:20, Willy Tarreau <[email-masked]> wrote:
> In fact SETTINGS_ENABLE_PUSH could have been a

That opportunity passed already, so we'd be making a new setting.

> response headers.

The question I have is whether this is a good proxy for the thing
we're concerned about.  That's the problem I have with any option that
attempts to establish a budget.  Are we budgeting for the right thing?

I agree that we could rathole indefinitely on the question of bytes
(pre-encoding, post-encoding?) vs messages. But is the additional
granularity of header field count adding value?

(I'm purposefully ignoring secondary benefits of your proposal here: I
really wish that servers would have some incentive to stop sending me
useless headers, e.g., see slide 15 of [1].)

My experience suggests that the incremental effort of processing an
extra header field is negligible, since most header fields are
basically ignored. Whereas the push itself implies a direct and
tangible cost.

[1] https://github.com/httpwg/wg-materials/raw/gh-pages/ietf95/BC.pdf

---

From: Stuart Douglas <[email-masked]>
Date: Mon, 02 May 2016 05:44:13 +0000

> limit instead of a bit, since not supporting push is equivalent to setting

#1 would also be a solution to an issue I brought up a while ago, which is
there is no way to enable/disable push on a per request basis, with this
could just set it to zero for requests that do not support push.

> here regarding this in the past resulting in headers not being accounted

I think this would be very problematic to implement on the server side. In
general the code that generates a push has no idea how big the headers on
the pushed responses will be, and neither does the transport layer that
handles HTTP/2.

In many cases the only way to know how big a responses headers will be is
to do the work and generate it, which could result in the server having to
generate the page then discard it due to it being too big.

Stuart

---

From: Martin Thomson <[email-masked]>
Date: Mon, 2 May 2016 16:39:52 +1000

> given request, maybe something that builds on Herve's push policy

I just realized that a header field might be problematic in that
server push is a hop-by-hop mechanism and header fields would pass
intermediaries.

The problem with anything that isn't coupled to the request is that it
becomes quite imprecise.  Any request that the client makes either
counts toward a global limit, or adds an opportunity for the server to
push another tranche of messages.

---

From: Stefan Eissing <[email-masked]>
Date: Mon, 2 May 2016 10:20:33 +0200

> Am 02.05.2016 um 08:39 schrieb Martin Thomson <[email-masked]>:
> 

The undiscovered country of server pushes. It will take some time to map that out.

> push another tranche of messages.

And the order in which requests are processed is totally up to the server (guided by priority) and may encounter delays the client is unaware of.

The global limit seems to say: "I can only handle X at a time, do not send me more".
The request specific limit: "I do not want/want at most X sub-resources for this one."

I'd assume - browser devs might want to correct me on this - that for the majority of requests a client will have no real use for sub resources. While having the cache populated is nice, it is *this* request it wants to process as fast as possible.

That would be an argument for something that defaults to "no push", without intermediaries/servers having extra processing for the common case. Something like a new SETTING that says: the NOPUSH I sent you, please ignore that for streams that carry XXX. And XXX could be a header like Push-Policy.

-Stefan

---

From: Roland Zink <[email-masked]>
Date: Mon, 2 May 2016 11:54:56 +0200

I think as a measure of last resort the client can close the TCP 
connection and is then save from further pushes. Currently the number of 
subresources of a page seems to be unlimited, for example a simple 
script can generate unlimited number of transactions. This is 
independent if push or pull is used. Wouldn't it be more appropriate to 
limit the number of subresources? Even the number of connections to 
other domains is not limited. What about limiting secure request to be 
handled end to end secure 1:1, e.g. a single connection would be enough 
to retrieve the complete page.

Option 1 seems to limit the number of subresources only for push. Option 
2 limits the number of concurrent pushes like it is currently attempted. 
This allows the client to slow down the server but doesn't limit the 
total number of pushes on a page. Is this the what you want to achieve? 
Btw. the WAP WSP push has this option (confirmed push) together with a 
limit on the number of concurrent pushes.

Regards,

Roland

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 2 May 2016 23:11:21 +1200

> script can generate unlimited number of transactions. This is

Well, its not quite unlimited. There is a 2^30 limit on the stream IDs
available per endpoint per connection. But still a very big number.

The tarpit gets deeper though since CONTINUATION exists outside of flow
control. And I'll leave it at that because that chestnut has been well
and truly roasted and still survives.

> to retrieve the complete page.

That would be far too restrictive. The problem is not how many things it
takes to present a resource. Some things can be quite large and complex
just by their nature.

The problem is having all the pieces pouring back in a flood of data
regardless of need and without flow control being able to manage them.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 2 May 2016 14:22:38 +0200

Hi Martin,

> That opportunity passed already, so we'd be making a new setting.

Yes definitely.

> we're concerned about.

My gut feeling is that it could be. Pretty much inaccurate but at least
enforces *some* limits. I don't have numbers unfortunately.

> That's the problem I have with any option that

We're definitely not but I guess it could be representative enough of
a number of requests and/or on-wire volume in a given environment.

> granularity of header field count adding value?

Well, for one thing it would clearly be inexact/inaccurate so would
probably be used more as a hint than anything else. I tend to think
that there's an order of magnitude between the request count and the
header field count, and very small order of magnitude between the
header field count and the on-wire byte count. All in all we'd have
a metric which is comprised between the two we can be interested in
(requests and bytes), which is related to the two and more tied to
each of them than each of them is tied to the other one. Hence why
I suspect it could provide good enough results to address both goals.

> basically ignored. Whereas the push itself implies a direct and

I agree. My purpose is not to really limit the number of header fields,
but clearly to use them as a single proxy for both metrics that people
may be interested in. If we only address request count, some people
will complain that network traffic varies too much for a given value,
and if we fix network bytes, some people will explain that it allows
between 1 and 1000 requests and cannot serve to limit them.

Or we can decide to limit both exactly and we'd have again the issue of
the unknown output size.

At best it would be used as a courtesy setting : "please try not to send
me more than XYZ header fields, I won't care if you're slightly wrong".
A server could easily decide that a request is 10 header fields and
divide this value by 10 to limit the pushes. A client could decide that
a field is 4.37 bytes and divide a congestion window by this to limit
buffering.

That's just an idea, I know it can sound surprizing and will not be
shocked if people tell me it's bullshit, I just think it could overall
work pretty well :-)

Cheers,
Willy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 2 May 2016 12:06:50 -0700

> become more widespread, then I think that we might need to do

Are we concerned about a server accidentally sending too many pushes,
or deliberately attacking the client via too many pushes?

I fail to see the point of an attack.  Network access can be drowned
with a valid response (e.g., a 4K movie on YouTube).  Perhaps a cache
could be attacked if each push promise results in a cache eviction
before a new representation is received?  I'd blame the cache.

For accidental deluges, I think it is better to educate the implementers
than build something complicated into the protocol. Sending data that
isn't needed harms the service more than anyone else. I think it would be
better to have a way for the client to signal that it will ignore a
given push, for various reasons, as a way of providing feedback to the
service to STFU. A stream of feedback seems more useful than limits.

> I'm interested in what people think here.

Not (2) -- that's overkill.  (1) seems a shame given that it won't prevent
a server from sending pushes, and doesn't feel right given that a client
has no idea how many pushes it might need.

On the whole, I think this is addressing a theoretical problem in the absence
of a good model.  For example, what happens when h2 is being used as a channel
for notifications?  I would expect very few requests and an unlimited number
of pushes.

Unless we can think of a reason that browsers can't just discard the promises,
I don't think we need to do anything here, even though a STFU frame would be amusing.

....Roy

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 May 2016 15:35:12 +1000

> or deliberately attacking the client via too many pushes?

It's the former, accidental overloading.  And it's not really an
attack, just an infidelity.  We did a lot to provide feedback
mechanisms where there was a risk of overload, and we missed this tiny
corner case.

As with the push policy stuff, the point is to avoid having a server
send pushes when the client doesn't really want them.

> has no idea how many pushes it might need.

I agree on both counts.  I haven't been able to contrive anything that
works without ugly side-effects of one sort or other.  I'll probably
write a draft with a few ideas in it and see what people think when
confronted with specifics.

---

From: Cory Benfield <[email-masked]>
Date: Tue, 3 May 2016 10:06:09 +0100

> On 3 May 2016, at 06:35, Martin Thomson <[email-masked]> wrote:
> 

This does feel a bit like a problem we can’t do much about.

A misconfigured server can always provide more work than a client wants to deal with. There are lots of non-flow-controlled frames that a poorly-written server may emit lots of that will mandatorily consume client resources. In fact, for very nearly any of the non-flow-controlled frames that exist today it’s pretty easy to conceive of a way to accidentally mis-use them (a blizzard of WINDOWUPDATE frames that increment the flow control window by one byte each time, for example, or repeatedly emitting SETTINGS frames that contain all the settings and gradually shrink the HPACK table size by one byte each time).

I think the point that hasn’t been made clear to me yet is: why is this problem more worthy of addressing at the protocol level than any of the others? If the client is feeling overwhelmed by the pushes it is free to do minimal processing of them (pass the headers through the HPACK decoder and maintain the minimal stream state information required to ensure that nothing untoward happens, but otherwise ignore them).

And if a client is really overwhelmed, it can RST the parent stream, change the value of SETTINGS_ENABLE_PUSH to 0, and then re-request the resource. That then reduces the client to needing to maintain extremely minimal state for each pushed stream (pass through the HPACK decoder and then RST on any pushed stream).

More importantly though: if we did enshrine this at a protocol level, what would a client do in the case where a server ignores those protocol requirements? Is it going to be any different from the above?

I’m open to seeing an extension to the protocol for this, but it’s hard to see what we gain. There’s no protocol level way to avoid the client needing to process these frames, at least minimally, because it needs to keep the HPACK state in play. All we can do is have a protocol-level way to signal to a server “don’t send too many pushes at once”. I suppose that’s useful, but it doesn’t seem like we need any more than an extra SETTINGS field and a specification for what to do when a server misbehaves. We will then have to expect that servers will misbehave, and just like in HTTP/1.1, clients will need to decide what they’re doing when they do.

Cory

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 3 May 2016 11:23:35 +0200

> Am 03.05.2016 um 11:06 schrieb Cory Benfield <[email-masked]>:
> 
> 

> And if a client is really overwhelmed, it can RST the parent stream, change the value of SETTINGS_ENABLE_PUSH to 0, and then re-request the resource. That then reduces the client to needing to main…

I share your view that SETTINGS_ENABLE_PUSH is a flow control mechanism for clients/intermediaries to use in such a scenario. It's like Roy's STFU, but already there.

-Stefan

---

From: Cory Benfield <[email-masked]>
Date: Tue, 3 May 2016 10:37:57 +0100

> On 3 May 2016, at 10:23, Stefan Eissing <[email-masked]> wrote:
> 
> The difference is that this condition may arise with every implementation in the chain being well implemented. PUSHes are often trigger by an application layer that is unaware of other instances of itself and the server may not be able to decide if the PUSHes are necessary/wanted (notifications) or just cache bloating.

Yeah, that’s definitely a substantive difference, but "well implemented" in this instance is a difference of degree, not of kind. There is nothing spec-violating about incrementing flow control windows one byte at a time, it’s just obviously stupid. But that doesn’t necessarily prevent it from happening if the implementation is sufficiently incautious about what it does. Similarly to repeatedly emitting SETTINGS frames with no actual changes, or emitting over-large SETTINGS frames: the specs allow it, it’s just a bad idea. The same applies to emitting 200 pushed streams with no content: you can do it, but you should expect that the connection won’t work well for you!

But maybe the best comparison is with large header blocks. These are explicitly allowed and designed for in RFC 7540, even though they can be just as effective an accidental DOS tool as a blizzard of pushed streams. But all RFC 7540 does here is provide some way to signal to the server what a limit might be (SETTINGS_MAX_HEADER_LIST_SIZE). It doesn’t even try to claim that any size lower than this is allowed: this setting is strictly advisory, and the endpoint setting it is welcome to behave otherwise like it isn’t there. Otherwise, both servers and clients can decide that the other peer messed up (servers can 431, clients can RST_STREAM).

It would likely be enough to just add a new error code (TOO_MANY_PUSHES) and have clients emit that on a RST_STREAM when rejecting pushes that don’t want to be handled. An extra setting to signal how many pushes a client is likely to accept in the same vein as SETTINGS_MAX_HEADER_LIST_SIZE would be a nice-to-have, but isn’t really needed, especially as it’d be advisory at best.

So I agree with you and Roy: the best we really need to do is provide ways to feedback to server operators when they’re in a situation that is causing a client to discard data. Pushes are strictly optional and a client is free to ignore them without notifying the server, so the client is already equipped to handle this better than it can handle the oversized header block, so I don’t really find this problem too troubling.

Cory

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 3 May 2016 06:38:29 -0700

On Tue, May 3, 2016 at 2:06 AM, Cory Benfield <[email-masked]> wrote:

> A misconfigured server can always provide more work than a client wants to
> deal with. There are lots of non-flow-controlled frames that a
> poorly-written server may emit lots of that will mandatorily consume client
> resources. In fact, for very nearly any of the non-flow-controlled frames
> that exist today it’s pretty easy to conceive of a way to accidentally
> mis-use them (a blizzard of WINDOWUPDATE frames that increment the flow
> control window by one byte each time, for example, or repeatedly emitting
> SETTINGS frames that contain all the settings and gradually shrink the
> HPACK table size by one byte each time).
>
> I think the point that hasn’t been made clear to me yet is: why is this
> problem more worthy of addressing at the protocol level than any of the
> others? If the client is feeling overwhelmed by the pushes it is free to do
> minimal processing of them (pass the headers through the HPACK decoder and
> maintain the minimal stream state information required to ensure that
> nothing untoward happens, but otherwise ignore them).
>
> And if a client is really overwhelmed, it can RST the parent stream,
> change the value of SETTINGS_ENABLE_PUSH to 0, and then re-request the
> resource. That then reduces the client to needing to maintain extremely
> minimal state for each pushed stream (pass through the HPACK decoder and
> then RST on any pushed stream).
>
> More importantly though: if we did enshrine this at a protocol level, what
> would a client do in the case where a server ignores those protocol
> requirements? Is it going to be any different from the above?
>
> I’m open to seeing an extension to the protocol for this, but it’s hard to
> see what we gain. There’s no protocol level way to avoid the client needing
> to process these frames, at least minimally, because it needs to keep the
> HPACK state in play. All we can do is have a protocol-level way to signal
> to a server “don’t send too many pushes at once”. I suppose that’s useful,
> but it doesn’t seem like we need any more than an extra SETTINGS field and
> a specification for what to do when a server misbehaves. We will then have
> to expect that servers will misbehave, and just like in HTTP/1.1, clients
> will need to decide what they’re doing when they do.
>

​I completely agree. This doesn't feel like a terribly pressing issue, and
clients already have a mechanism (RST_STREAM) to attenuate the effects.​

---

From: RUELLAN Herve <[email-masked]>
Date: Wed, 4 May 2016 10:24:39 +0000

> -----Original Message-----
> From: Martin Thomson [mailto:[email-masked]]
> Sent: mardi 3 mai 2016 07:35
> To: Roy T. Fielding <[email-masked]>
> Cc: HTTP Working Group <[email-masked]>
> Subject: Re: Flow control for server push
> 
> It's the former, accidental overloading.  And it's not really an attack, just an
> infidelity.  We did a lot to provide feedback mechanisms where there was a
> risk of overload, and we missed this tiny corner case.
> 
> As with the push policy stuff, the point is to avoid having a server send
> pushes when the client doesn't really want them.
> 
> > Not (2) -- that's overkill.  (1) seems a shame given that it won't
> > prevent a server from sending pushes, and doesn't feel right given
> > that a client has no idea how many pushes it might need.
> 
> I agree on both counts.  I haven't been able to contrive anything that works
> without ugly side-effects of one sort or other.  I'll probably write a draft with
> a few ideas in it and see what people think when confronted with specifics.

I think that limiting the number of pushes fits well in the push-policy draft [1] as a new push-policy.
Using something like a push-policy has the advantage of covering the WebPush use case, Stuart's use case of pushing nothing for a specific request, and to be extensible to future use cases.
The main drawback is that push is hop-by-hop and that the headers may survive for more than one hop.

I can update the draft when I have some available time.

Hervé

[1] https://tools.ietf.org/html/draft-ruellan-http-accept-push-policy-01

---

From: Göran Eriksson AP <[email-masked]>
Date: Wed, 4 May 2016 19:36:57 +0000

> I'll probably

Would be useful in enabling a thorough discussion at least I think.
Will also make it easier to separate from well behaved server, crappy
server and malicious server.

Regards
Göran

---

From: Wenbo Zhu <[email-masked]>
Date: Tue, 10 May 2016 10:26:23 -0700

On Mon, May 2, 2016 at 10:35 PM, Martin Thomson <[email-masked]>
wrote:

> prevent
>  My experience with #2 and the general problem (like a push channel over
http/*):
1. "explicit" acks should be left to the app ..  to be useful since network
level delivery doesn't guarantee any commit semantics (as in the rpc case)
2. a limit on # of pending "messages" (based on network-level delivery
acks) on the sender + suspension of delivery acks on the receiver

In the absence of delivery acks (as in this web-push case), an explicit
flow-control message to the peer seems like a reasonable choice, i.e. what
the actual problem is.

---

From: Costin Manolache <[email-masked]>
Date: Wed, 11 May 2016 04:54:06 +0000

On Tue, May 10, 2016 at 10:40 AM Wenbo Zhu <[email-masked]> wrote:

> 1. "explicit" acks should be left to the app ..  to be useful since
> network level delivery doesn't guarantee any commit semantics (as in the
> rpc case)
Delivery ack would work - but they are a bit inefficient: if they're
implemented as a regular HTTP method, there is a
full extra roundtrip ( request / response frames ) for each push promise.
It would help control the rate of
send - but I think it would also limit the transmission rate.

It might help if the explicit ack could be sent at least as a one-way
message, or as an optional response to the promise - or
batched, or something like the window mechanism used for flow control.

Costin