Subject: #551: Limiting header sizes

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Jul 2014 13:41:55 +1000

A lot of the discussion around <https://github.com/http2/http2-spec/issues/551> is around having a hard limit for header block sizes in the protocol, and the resulting ways that helps and hurts.

I wonder if we can make a small adjustment to ease some of the pain. Specifically, what if it were only advisory, and there were no default?

I.e., instead of a setting with the semantic of "You MUST NOT send header blocks larger than <x>", what if it were "If you send header blocks larger than <x>, I'll very likely discard them (responses) / respond with a 431 (requests)"?

This makes it much more flexible; if a proxy sends this setting and sees a client ignoring it, they have the choice of either soft-failing them (with a 431), accepting the larger request, or hard-failing them (ENHANCE_YOUR_CALM).

It also seems more compatible with the way that HTTP/1 works.

I think that doing this might address most of the cons listed at <https://github.com/http2/http2-spec/wiki/ContinuationProposals#limit-header-block-size-via-a-setting>.

Thoughts? 

--
Mark Nottingham   https://www.mnot.net/

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 16 Jul 2014 14:01:16 +1000

Mark,

I think that a declared header limit, regardless of exact semantics, should
be declared in terms of uncompressed header size rather than header block
size.

We should be decoupling HTTP semantics from the framing layer and with all
the other proposals being currently considered (large frames, fragmented
headers, etc.) then I think we are hopefully moving away from having any
header limits (implicit or explicit) related to the frame size.

With regards to the change in semantics that you suggest, I'm fine with
that.   It really just means that it is not a protocol error for a sender
to ignore the setting.

cheers

On 16 July 2014 13:41, Mark Nottingham <[email-masked]> wrote:

> A lot of the discussion around <
> https://github.com/http2/http2-spec/issues/551> is around having a hard
> limit for header block sizes in the protocol, and the resulting ways that
> helps and hurts.
>
> I wonder if we can make a small adjustment to ease some of the pain.
> Specifically, what if it were only advisory, and there were no default?
>
> I.e., instead of a setting with the semantic of "You MUST NOT send header
> blocks larger than <x>", what if it were "If you send header blocks larger
> than <x>, I'll very likely discard them (responses) / respond with a 431
> (requests)"?
>
> This makes it much more flexible; if a proxy sends this setting and sees a
> client ignoring it, they have the choice of either soft-failing them (with
> a 431), accepting the larger request, or hard-failing them
> (ENHANCE_YOUR_CALM).
>
> I think that doing this might address most of the cons listed at <
> https://github.com/http2/http2-spec/wiki/ContinuationProposals#limit-header-block-size-via-a-setting
> >.
>
> Thoughts?
>
>
> --

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jul 2014 07:13:30 +0200

Hi Greg,

> that.   It really just means that it is not a protocol error for a sender

I understand the point you're making and I too think that uncompressed
header limit could be useful, but I think that it will be very implementation
dependant. For instance h2->h1 gateways will have a hard limit which basically
is the h1-encoding of all the headers (including possible repetition of certain
very large multiple headers). So there's no 100%-certain uncompressed size in
this case.

And that's where I think that Mark's proposal could solve this issue
elegantly, just the in the same way we do it in h1 right now : if you
send something reasonably small, you don't need to care. If you're sending
large things, close to suggested common limits, get prepared to receive a
bunch of 431. I like it because it avoids precise accounting and allows
recipients to detect not in the framing but in the semantics and/or
processing that contents finally don't fit in the buffer anymore.

For example, right now haproxy receives h1-encoded headers and keeps a
reserve for header rewrites and additions. If you want to add far too large
headers to a large request and the result doesn't fit in the buffer anymore,
you get an error. This case very rarely happens (you have to write a special
config for this) but it's easy to handle and I don't have to advertise a
certain size that would be hard to account for on the sender's side.

Cheers,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 16 Jul 2014 06:35:03 +0000

In message <[email-masked]>, Mark Nottingham writes:
>A lot of the discussion around 
><https://github.com/http2/http2-spec/issues/551> is around having a hard 
>limit for header block sizes in the protocol, and the resulting ways 
>that helps and hurts.
>
>I.e., instead of a setting with the semantic of "You MUST NOT send 
>header blocks larger than <x>", what if it were "If you send header 
>blocks larger than <x>, I'll very likely discard them (responses) / 
>respond with a 431 (requests)"?

That works for me.

I still think the limit should apply to compressed headers, since that
is what memory allocation is required for.

Once the receiver has received the frame, it's trivial to do a
non-producing pass over it to calculate the uncompressed size, if
this is relevant to know before memory allocation happens.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 16 Jul 2014 10:44:58 -0700

> this is relevant to know before memory allocation happens.

Worst case, that requires a temporary allocation equal to the header
table size to do correctly, but I think that's perfectly feasible.
That is, if you really, really care about that.  You probably don't
need to do that in most cases; you can simply perform the accounting
as you decompress and detect when you run out of space.

---

From: Glen Knowles <[email-masked]>
Date: Wed, 16 Jul 2014 23:02:58 -0700

> is what memory allocation is required for.

As a multithreaded sender with a shared compression context it's very hard
to work with a limit of the compressed size before it's too late. The
application layer is constructing the requests/replies and submitting them
to the network layer as they are ready. Even if you communicated the header
limit to the code generating the message it can only guess at whether it's
going over because it may not be the next message to be sent.

Rejecting it at the network layer is better then sending it for the remote
system to reject, but it's much better if acceptable messages can be
deterministicly created in advance.

If we're going to have header limits (and a shared compression context) let
them be on the uncompressed size.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 17 Jul 2014 12:38:33 +0000

In message <[email-masked]>, Glen Knowles wri
tes:

>to work with a limit of the compressed size before it's too late. 

I think the proper solution would be to add an 'opcode' to HPACK that
means "reset to initial state".

One of the biggest risks of HPACK is that the two ends get out of sync
and the result is going to be really strange headers really fast which
means really strange errors comming back[1].

In that situation, the sender is currently forced to abandon the connection
and open a new to get a fresh compression state.

Having an opcode to move to start[2] that would be much cheaper.

Poul-Henning

[1] Now that I think about it, that's actually a very good argument
    for including a 100-Continue like facility.

[2] Do not collect 200 :-)

---

From: Martin Thomson <[email-masked]>
Date: Thu, 17 Jul 2014 09:33:02 -0700

> means really strange errors comming back[1].

I believe that the most likely outcome of desynchronization is that
the connection will be dumped.  Search the spec for
"ConnectionErrorHandler", grep|wc counts 45.  Good luck avoiding all
of those for long.

This is why I tend to think we should do absolutely nothing about issue #555.