Subject: Limiting header block size

From: Martin Thomson <[email-masked]>
Date: Thu, 22 May 2014 05:53:54 -0700

The discussion on header blocks and flow control raised the question
of whether it is appropriate to allow an endpoint to limit the size of
header block it accepts.

Due to the design of HPACK, there are many cases where a server or
intermediary is forced to buffer an entire header block.  This is
because critical routing information like :path, :authority and
:scheme can be placed anywhere in a header block.  More importantly,
some of these are highly likely to be in the reference set, causing
them to be emitted only at the end of the block.

This means that processing a header block can require an essentially
unbounded amount of state for many implementations.  Being able to
limit this commitment would be good.

However, it's not clear what an announced limit would enable at a
sender.  Conforming to a limit ultimately requires a sender to drop
header fields.  It seems unlikely that a sender would be able to
arbitrarily drop header fields in order to comply with an arbitrary
limit.  Header fields are there to express semantics, and it's
difficult to know what fields are safe to drop without knowing
application requirements in some detail.

(Someone suggested that a block could be split, but that doesn't work
due to the structure of the protocol.)

Thus, I conclude that changing the protocol is not advisable.
Implementations that receive more header information than they are
willing to tolerate can process any updates to the header table and
then reset the corresponding stream (which for a PUSH_PROMISE would be
the promised stream, of course).

I will raise an editorial issue to note the DoS implications of
excessively large header blocks and the above method for handling
them.

---

From: Michael Sweet <[email-masked]>
Date: Thu, 22 May 2014 09:08:03 -0400

Martin,

I agree that realistically we cannot limit the size of header blocks.

However, is there any reason to continue the restriction that header blocks must be contiguous? Allowing intervening DATA frames on other streams would go a long way to minimizing the impact of large header blocks on streaming content and prioritization in general.  I fear that if that issue is not addressed we'll end up with low adoption of HTTP/2 or implementations that open a separate connection for high-priority traffic rather than using the priority scheme in HTTP/2.

> excessively large header blocks and the above method for handling

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Michael Sweet <[email-masked]>
Date: Thu, 22 May 2014 09:13:15 -0400

https://github.com/http2/http2-spec/issues/481

---

From: Roland Zink <[email-masked]>
Date: Thu, 22 May 2014 15:25:13 +0200

> limit this commitment would be good.
+1
> application requirements in some detail.
The sender then shouldn't send the request and notify the user / return 
an error message. If the receiver is resetting the stream the result 
will be not much different.
> Thus, I conclude that changing the protocol is not advisable.
Another possibility would be to conclude the protocol is broken and 
should be changed :)

Roland

---

From: Martin Thomson <[email-masked]>
Date: Thu, 22 May 2014 06:51:00 -0700

On 22 May 2014 06:25, Roland Zink <[email-masked]> wrote:
> The sender then shouldn't send the request and notify the user / return an
> error message. If the receiver is resetting the stream the result will be
> not much different.

This has the advantage of bringing the error forward.

It has the disadvantage of being less flexible and introduces the need
to specify exactly how to measure the limit.

I'd say that in the common case, state commitment would have to be
based on post-decompression header fields.  But any setting we define
would need to be more deterministic and enforceable, so it would be
easiest to express a limit based solely on the size of header block
frames.

That creates a mismatch that could be exploited.  The problem then for
implementations is to choose what value to advertise.  In order to be
perfectly safe from attack, the limit would have to be so small it
would basically prevent any real messages from getting through.  Thus,
implementations are basically required to implement header block
discarding anyway.

The only advantage of a setting then is to - in some cases - cause
early detection of some errors, at the cost of more protocol
machinery.

So, I'm going to stick with my conclusion, and propose:
https://github.com/http2/http2-spec/pull/482

---

From: Roland Zink <[email-masked]>
Date: Thu, 22 May 2014 16:10:39 +0200

Hi Martin,

looks good to me, the only small comment on this is that the client not 
really knows what it did wrong. Should it retry the request at a later time?

Regards,
Roland

---

From: Martin Thomson <[email-masked]>
Date: Thu, 22 May 2014 07:14:52 -0700

> really knows what it did wrong. Should it retry the request at a later time?

This is even worse for a server that gets rejected in this fashion.
Though the error code is fairly identifiable, since that's the only
case where ENHANCE_YOUR_CALM is used in a stream error.

As for retrying, that's going to be up to the application.  In some
cases, the limit might be tripped because of transient conditions
(such as a temporary spike in load).  So retrying is possible, with
the usual caveats around non-idempotent requests and the like.  This
error code doesn't contain a promise that the stream was not
processed.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 23 May 2014 11:21:06 +1000

We already have a HTTP status code for this: <http://tools.ietf.org/html/rfc6585#section-5>

Why push it down into the framing layer?
 
There’s a LOT more chance for conveying the semantics of what went wrong in a full HTTP response; you can do it in the headers, status codes, etc. In contrast, an error code is exactly one bit, and a recipient is inevitably going to be left scratching their head and asking “HOW should I enhance my calm?”

Not to mention, doing it this way will require APIs to figure out how to surface that information, and HTTP1->2 gateways trying to figure out out to represent ENHANCE_YOUR_CALM to their clients (in this case, they’d likely use the wrong status code!).

Let’s not duplicate HTTP semantics down into the framing layer, please. I know that if we were doing this from scratch, we’d have a better separation between resource/representation semantics and protocol behaviours like this — but we’re not doing it from scratch.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Fri, 23 May 2014 06:51:13 -0700

> We already have a HTTP status code for this: <http://tools.ietf.org/html/rfc6585#section-5>

I always forget about those...

> Why push it down into the framing layer?

Only because I forgot about those...

Let me try the PR again.  That's in part why I made it a PR rather
than a direct edit :)  I think that the documentation of the problem
is still OK, just the reporting/handling part.  I'll note that 431
isn't an option for a client or intermediary that gets bombed by a
server, but I guess the implication there is that the response is
simply unusable in that case.