Subject: H2 vs HPACK header table size... again

From: Willy Tarreau <[email-masked]>
Date: Wed, 22 Dec 2021 16:11:44 +0100

Hi all!

It was already mentioned long ago but never really settled, there is an
ambiguity in the HPACK and H2 specs regarding the header table size a
decoder assumes the encoder will use.

Today I got an haproxy user report an issue with the nghttp client when
forcing the header table to zero, because nghttp's decoder expects a table
size update from the encoder while the encoder considers it only conforms
to what was advertised and never changes it, thus never updates it.

For those who want a bit of background, the report was made here:
   https://github.com/haproxy/haproxy/issues/1498
   https://github.com/nghttp2/nghttp2/issues/1660

I discussed a bit with Tatsuhiro about this, who agreed that it was best
to bring the issue here. We're not trying to decide who's right or wrong,
and we're both fine with updating our implementations to improve
interoperability but it's annoying to bindly have to do something we
don't seem to be reading in the spec.

The H2 spec says about the SETTINGS frame:

   SETTINGS_HEADER_TABLE_SIZE (0x1):  Allows the sender to inform the
      remote endpoint of the maximum size of the header compression
      table used to decode header blocks, in octets.  The encoder can
      select any size equal to or less than this value by using
      signaling specific to the header compression format inside a
      header block (see [COMPRESSION]).  The initial value is 4,096
      octets.

Note the word "initial" here which may or may not participate to the
confusion. My reading is that until SETTINGS_HEADER_TABLE_SIZE the
value is 4096, and that SETTINGS_HEADER_TABLE_SIZE changes that value
to the newly advertised one.

The HPACK spec says in RFC7541#4.2:

   Protocols that use HPACK determine the maximum size that the encoder
   is permitted to use for the dynamic table.  In HTTP/2, this value is
   determined by the SETTINGS_HEADER_TABLE_SIZE setting (see
   Section 6.5.2 of [HTTP2]).

Thus for me it means that the value learned from the H2 layer has to
be used at the HPACK layer.

Then it continues:
   An encoder can choose to use less capacity than this maximum size
   (see Section 6.3), but the chosen size MUST stay lower than or equal
   to the maximum set by the protocol.

   A change in the maximum size of the dynamic table is signaled via a
   dynamic table size update (see Section 6.3).  This dynamic table size
   update MUST occur at the beginning of the first header block
   following the change to the dynamic table size.  In HTTP/2, this
   follows a settings acknowledgment (see Section 6.5.3 of [HTTP2]).

Thus my understanding is that an encoder that chooses to use a different
value than the one learned from the H2 layer has to notify it via an
update.

And I think this is where there are two different readings which possibly
depend on how the implementation is built:

  - if you write some client code, it seems natural that HPACK is
    instantiated before the H2 layer, and in this case I wouldn't be
    surprised that the "initial" term used above is interpreted as
    "first set your HPACK default value to 4096 and announce it if
    you learn otherwise, including from the H2 SETTINGS frame",
    which seems to be how Tatsuhiro and Pat have interpreted it in
    the thread below (do not hesitate to correct me, I'm not trying
    to put words in your mouth at all):

      https://lists.w3.org/Archives/Public/ietf-http-wg/2016JulSep/0586.html

  - if you're writing some server code, it seems natural that H2 is
    instanciated before HPACK and that HPACK comes with the negotiated
    settings. After all we could even imagine a SETTINGS parameter to
    switch to a totally different encoder. In this case HPACK starts
    with an initial value that matches the one advertised by the client,
    which seems to be what Cory, Hervé and I have interpreted in the
    thread below:

      https://lists.w3.org/Archives/Public/ietf-http-wg/2015OctDec/0107.html

I do not care about systematically emitting one extra byte to update
a table size that we already know if that improves interoperability,
but I find it pretty annoying that 6 years later we continue to face
interoperability issues because there seem to be two incompatible
readings of this corner case of the spec. And I suspect that the issue
also exists for situations where a large SETTINGS_HEADER_TABLE_SIZE
value is emitted by the decoder, the encoder is not willing to change
from 4kB and stays at 4kB, but the decoder assumes that the encoder
agreed with the initially advertised one. This can be worse because
it could work without triggering compression errors but possibly
result in damaged messages.

As such I'd like that we could collectively decide what to do for
this (even if that means relaxing both clients and servers to
accommodate for the other interpretations that are already deployed).

And maybe once we're fixed with whatever possible interpretation, we
should possibly file an errata to amend this part of the HPACK spec
in a way that doesn't leave any more room for two interpretations.

What do implementers (or others) think ?

Thanks,
Willy

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 23 Dec 2021 09:24:32 +1100

Hi Willy,

My interpretation has always been that if the value has *changed* the encoder sends a size update.  The value changes at an encoder when that endpoint sends a SETTINGS ACK for a SETTINGS that had a different value to the existing value.  The value starts at the initial 4k value.

Obviously, this is a bit silly in a few cases, but the cost is marginal.

* If the decoder opens with a >4k value in its preface and the encoder wants to keep the 4k initial value, then the encoder has to send a size change instruction.
* A server rarely has a situation where the initial value is needed as it always has the value from the client preface, but it has to send the SETTINGS ACK and size change instruction anyway, unless the client retains the 4k initial value.

Tatsuhiro's interpretation here matches my own, I think: https://lists.w3.org/Archives/Public/ietf-http-wg/2016JulSep/0499.html

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 23 Dec 2021 05:08:34 +0100

On Thu, Dec 23, 2021 at 09:24:32AM +1100, Martin Thomson wrote:
> Hi Willy,
> 
> My interpretation has always been that if the value has *changed* the encoder
> sends a size update.  The value changes at an encoder when that endpoint
> sends a SETTINGS ACK for a SETTINGS that had a different value to the
> existing value.

It can be an interpretation but at the stage you deal with settings
there's no HPACK yet.

> The value starts at the initial 4k value.

My interpretation was that HPACK's initial value starts at the SETTINGS
value.

> Obviously, this is a bit silly in a few cases, but the cost is marginal.

I totally agree and am even willing to systematically emit that byte,
as I said I just want to be certain that I'm not papering over random
stuff. And if I have to do this while I don't read this from the spec,
I think it warrants a minor text update in the spec. Alternately we
may update the new H2 spec (since it's not released yet) to mention
that the SETTINGS parameter doesn't change the HPACK's initial value
but corresponds to an update and will require a size update. It can
be confusing not to find it in the HPACK spec but could be easier to
find than by checking for errata.

> * If the decoder opens with a >4k value in its preface and the encoder wants
> to keep the 4k initial value, then the encoder has to send a size change
> instruction.

I'm fine with this one.

> * A server rarely has a situation where the initial value is needed as it
> always has the value from the client preface, but it has to send the SETTINGS
> ACK and size change instruction anyway, unless the client retains the 4k
> initial value.

That's the delicate point in my opinion, where not sending anything means
something different for the client.

> Tatsuhiro's interpretation here matches my own, I think:
> https://lists.w3.org/Archives/Public/ietf-http-wg/2016JulSep/0499.html

Thanks for sharing! And Tatsuhiro's description there confirms my
suspicion that the interpretation of the spec depends a lot on which
side you are developing, that's particularly bad for a spec.

Cheers,
Willy

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 23 Dec 2021 16:21:01 +1100

> something different for the client.

Yeah, this is the key point, but if you consider the possibility that a server might send a response before acknowledging the client's SETTINGS.  This isn't something that makes a lot of sense as the server cannot process a request before it processes the SETTINGS, so reordering the SETTINGS ACK with the response would be...bizarre, but the protocol is designed to allow for that.

Why would the protocol allow for that?  In its current form, it makes little sense because the server never sends headers without a request.  However, if you think of a server using HPACK to send non-trivial metadata to a client, something that is possible with TLS 1.3 (0.5-RTT data seems to be the term we've settled on for that data), then it makes a *tiny* bit of sense.  

Or, you could imagine using that 0.5-RTT data to prime the HPACK header table with useful values (Server, or other common values could be a good use of the bandwidth).  Priming the table is something that might also be a good use of 0-RTT, which would allow a client to use the space even when it doesn't have any requests that might be safe to send.

(Now I almost want to write a draft for that priming idea; HTTP/3 has it already, but it could still help in HTTP/2.)

> side you are developing, that's particularly bad for a spec.

I don't think so, though I think that server implementers are in a position where the normative behaviour isn't ideal, so this is a point that could be clearer.

The big question for me is whether a post-WGLC change is appropriate if we wanted to fix this in h2bis.  We're in IESG processing, so we could probably sneak something in.  How about this:

https://github.com/httpwg/http2-spec/pull/1003

As I note in the pull request, this might have been better in a revision to RFC 7541, but we don't have that document open and you have to deal with the integration of the two pieces somewhere.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 23 Dec 2021 07:33:15 +0100

On Thu, Dec 23, 2021 at 04:21:01PM +1100, Martin Thomson wrote:
> Yeah, this is the key point, but if you consider the possibility that a
> server might send a response before acknowledging the client's SETTINGS.
> This isn't something that makes a lot of sense as the server cannot process a
> request before it processes the SETTINGS, so reordering the SETTINGS ACK with
> the response would be...bizarre, but the protocol is designed to allow for
> that.

I didn't even think about that, indeed. But there's indeed little use for
this, though I can imagine that some implementers might have to face this
difficult choice.

> Why would the protocol allow for that?  In its current form, it makes little
> sense because the server never sends headers without a request.  However, if
> you think of a server using HPACK to send non-trivial metadata to a client,
> something that is possible with TLS 1.3 (0.5-RTT data seems to be the term
> we've settled on for that data), then it makes a *tiny* bit of sense.  
> 
> Or, you could imagine using that 0.5-RTT data to prime the HPACK header table
> with useful values (Server, or other common values could be a good use of the
> bandwidth).  Priming the table is something that might also be a good use of
> 0-RTT, which would allow a client to use the space even when it doesn't have
> any requests that might be safe to send.
> 
> (Now I almost want to write a draft for that priming idea; HTTP/3 has it
> already, but it could still help in HTTP/2.)

You love to write drafts ;-)

> I don't think so, though I think that server implementers are in a position
> where the normative behaviour isn't ideal, so this is a point that could be
> clearer.

No, to be clear, there's no ideal or not ideal, it's a matter of translating
what we read into code. My reading of the spec (initially as a server
implementer) makes it very clear that I do *not* have to do that. But
rereading it with my mind arranged as a client implementer makes me think
that it makes sense that it is needed. This is why I raised the issue
again. When the problem reporter told me that he worked around the
problem by sending this 0x20 byte, I was worried about the risk of
breaking other implementations that do not expect this to happen
because... it's not how I read the spec. Adapting code to match a spec
is a detail. Adapting code to match one interpretation of a spec is more
of a concern, hence my request to make sure that there cannot be two
interpretations of the spec.

> The big question for me is whether a post-WGLC change is appropriate if we
> wanted to fix this in h2bis.

Yes I know, that's delicate. But we're not changing the spec, just clarifying
something that seems to match a consensus between a majority of implementers.
In any case we're certainly not going to declare them non-compliant after
so many years, whatever side they're currently on. And those who read it
like me (apparently Cory and Hervé did) will likely not have a big
difficulty seeing the other interpretation that most other implementation
already adopted.

> We're in IESG processing, so we could probably
> sneak something in.  How about this:
> 

That looks perfectly balanced to me. It clarifies this gray area without
saying too much about how the compression works. I'm fine with adding that
initial table update with that spec.

> As I note in the pull request, this might have been better in a revision to
> RFC 7541, but we don't have that document open and you have to deal with the
> integration of the two pieces somewhere.

Yeah that was exactly why I suggested that we could possibly place a note
in h2bis instead.

Many thanks!
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 23 Dec 2021 09:02:44 +0100

> difficulty seeing the other interpretation that most other implementation

While running some tests on my side after adapting my code, I discovered
that the H2 client in the varnish-test suite doesn't seem to like very much
receiving a table size update of zero (that may be OK it's a regression
testing program). That led me to test varnish then a few other sites with
the following command:

  $ nghttp -v -H":method: HEAD" --header-table-size=0 https://$SITE

And the result is not as bright as initially thought:
  - varnish-cache.org causes a compression error
  - haproxy.org causes a compression error.
  - apache.org causes a compression error
  - h2o.examp1e.net causes a compression error
  - litespeedtech.com causes a compression error
  - lighttpd.net doesn't speak H2
  - nginx.org / .com surprisingly do not speak H2
  - akamai.com is OK
  - cloudflare.com is OK
  - fastly.com causes a compression error

It even seems to me that it's 50/50 between the two interpretations :-(

Thus it seems to me that we have to be extremely careful about what we
state and that a discussion with the various implementers is needed.
Maybe at least the proposed added note for h2bis could be turned as
a warning about known existing deployments that act differently and
that need to be taken care of.

Willy

---

From: Ilari Liusvaara <[email-masked]>
Date: Thu, 23 Dec 2021 15:23:30 +0200

>       https://lists.w3.org/Archives/Public/ietf-http-wg/2015OctDec/0107.html

My experience on this matter (having written server end) is that for
interoperability, the server has to support at least 4kB receive window.
I did not test many clients, but I saw at least Curl failing hard.

The implementation uses 0 send window, and unconditionally sends the
command to zero window in front of every HEADERS frame. Does not seem
to cause problems (and I have seen much more obscure issues).

> result in damaged messages.

IIRC, HPACK numbers headers from newest to oldest, which guarantees that
mismatched window size can never result in correctly compressed messages
to be silently misdecompressed.

-Ilari

---

From: Ilari Liusvaara <[email-masked]>
Date: Thu, 23 Dec 2021 15:46:31 +0200

On Thu, Dec 23, 2021 at 04:21:01PM +1100, Martin Thomson wrote:
> 
> Or, you could imagine using that 0.5-RTT data to prime the HPACK
> header table with useful values (Server, or other common values
> could be a good use of the bandwidth). 

Beware that some HTTP/2 clients vomit (IIRC with fatal TLS alert 10
after sending client Finished) if the server sends 0.5-RTT data (no
client certificate request). I do not know more specifics, but I
suspect that at least in some cases, there is some "anti-virus" or
"security" product involved on the client side.

-Ilari

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 23 Dec 2021 15:43:48 +0100

> "security" product involved on the client side.

Yes we've seen that more than once, they used to break WebSocket a
decade ago and more recently to break websocket-over-h2 IIRC. I don't
know what impacts it can have on such mechanisms but we must always
make sure that we can fail fast and in a detectable way.

Willy

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 24 Dec 2021 10:38:11 +1100

>   - fastly.com causes a compression error

You can add facebook.com and google.com to the "OK" list as well.

These all fail for any setting value that is less than 4096.  4095 causes the same error as zero for those servers.  I traced the HPACK instructions of a few of these (which is a bit tedious) and it's clear that the Dynamic Table Size Update instruction is not being sent by the "broken" servers here.  4097 should create a similar error, as none of the servers I checked send the instruction then either.

However, nghttp2 only checks for the instruction after a reduction but not after an increase.  Many of those servers don't send the Dynamic Table Size Update instruction if the size increases.  I don't know what they think the header table size *is* at that point, but it is being used, which is not great.

None of those servers cause a problem with Firefox with a changed setting value, so I guess that Firefox simply doesn't check for the instruction being present after the "change" from the initial values, or maybe it assumes that the value it sets in its preface is what the table is initialized to.  I don't know.

That all said, this looks like a bit of a shambles.  I've no idea what the client and server think the header table size is after this exchange.

For haproxy, what would it take to fix the problem?  More importantly perhaps, what would it take to get the fix deployed?  I should probably ask all the other folks listed here (phk, kazuho, stefan, dmitri) the same question.

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 24 Dec 2021 08:54:01 +0900

Thank you all for working on this.

> >   - fastly.com causes a compression error

In case of H2O / fastly.com, this is a FIXME that never has been fixed. H2O
ignores SETTINGS_MAX_TABLE_SIZE.

Regarding the initial size of dynamic table, I agree that it starts with
4096. When TLS 1.3 0-RTT or client authentication is used, the cilent is
first to speak. It cannot wait for the server's SETTINGS to arrive, as that
would incur a penalty of one RTT, so it was to adopt a default of 4096.

I think that part has been clarified correctly in
https://github.com/httpwg/http2-spec/issues/1003. Thank you for the work.

OTOH, I'm not sure if all the text that went in with #1003 is correct. I'm
not sure if I agree with the interpretation of RFC 7541 requires HPACK
encoders to send Dynamic Table Size Update when the table size is to remain
the same. I am also concerned that the newly added text, that suggests that
endpoints MAY error-close the connection unless peers behave as such, might
increase  interoperability failures, under the assumption that not all
servers behave like that. Please see
https://github.com/httpwg/http2-spec/issues/1004.

> These all fail for any setting value that is less than 4096.  4095 causes
> the same error as zero for those servers.  I traced the HPACK instructions
> of a few of these (which is a bit tedious) and it's clear that the Dynamic
> Table Size Update instruction is not being sent by the "broken" servers
> here.  4097 should create a similar error, as none of the servers I checked
> send the instruction then either.
>
> However, nghttp2 only checks for the instruction after a reduction but not
> after an increase.  Many of those servers don't send the Dynamic Table Size
> Update instruction if the size increases.  I don't know what they think the
> header table size *is* at that point, but it is being used, which is not
> great.
>
> None of those servers cause a problem with Firefox with a changed setting
> value, so I guess that Firefox simply doesn't check for the instruction
> being present after the "change" from the initial values, or maybe it
> assumes that the value it sets in its preface is what the table is
> initialized to.  I don't know.
>
> That all said, this looks like a bit of a shambles.  I've no idea what the
> client and server think the header table size is after this exchange.
>
> For haproxy, what would it take to fix the problem?  More importantly
> perhaps, what would it take to get the fix deployed?  I should probably ask
> all the other folks listed here (phk, kazuho, stefan, dmitri) the same
> question.
>
>

-- 
Kazuho Oku

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 24 Dec 2021 11:50:25 +1100

On Fri, Dec 24, 2021, at 10:54, Kazuho Oku wrote:
> In case of H2O / fastly.com, this is a FIXME that never has been fixed. 
> H2O ignores SETTINGS_MAX_TABLE_SIZE.

Ahh, that's not great.  It makes <4096 pretty perilous for clients.  I wonder if that is the same reason for other implementations.

I didn't realize that Cory merged #1003; I'm not that far through my email yet...  My intent was to keep this open a little.  I'll revert that and reopen it so that we can keep discussing it.

Given where things stand, the change you suggest in #1004 is probably positive from a compatibility perspective.  It allows implementations to ignore increases in the maximum.  For your implementation, all you have to do is deal with a reduction in the maximum table size more elegantly (I'd start with a connection error of type INTERNAL_ERROR).

See: https://github.com/httpwg/http2-spec/pull/1005

I thought for a while we needed to update RFC 7541 as a result, but a careful read reveals that we're OK.  The most interesting part is the caution about reducing the maximum size.  It was a little fiddly to write.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 24 Dec 2021 05:32:15 +0100

On Fri, Dec 24, 2021 at 11:50:25AM +1100, Martin Thomson wrote:
> Ahh, that's not great.  It makes <4096 pretty perilous for clients.  I wonder
> if that is the same reason for other implementations.

For haproxy we ignore it as well since we don't use it on the response
path. So this change requires us to process a setting for something we
do not use, that's a bit awkward (not *that* difficult but if forces us
to send raw HPACK bytes from the H2 code which is a direct layering
violation). And I think it's the same for other implementations as one
rarely has to implement some random tricks for things that are not used :-/

In addition I've reread RFC7541 and do not agree anymore with the change
in #1003. HPACK is clearly designed and defined as a portable compressor
and carefully avoids to mention any size, it further mentions in the text
that the limit for the table size is "determined by the protocol using
HPACK" and that "in HTTP/2 this limit is the last value of the
SETTINGS_HEADER_TABLE_SIZE parameter".

As such the magic value of 4096 has nothing to do with HPACK and is
entirely an HTTP/2 default value that is required to allow for 0.5rtt
and such to work, and that can be changed via a SETTINGS parameter.
This means that any other value advertised at the H2 layer *is* the
initial HPACK value and cannot trigger a change at the HPACK layer.

In addition, H2 was carefully designed so that HPACK could be replaced
with anything else using a new SETTINGS frame. Let's say we'd port QPACK
to H2 for example, it would seem even more gross to have to deal with
HPACK specifics in the QPACK layer for the response path.

Maybe the text could be adjusted like this:

  The dynamic table has a maximum size that is set by a decoder using the
  SETTINGS_HEADER_TABLE_SIZE setting; before receiving this setting, an
  encoder must assume an initial value of 4096 and may only use a larger
  value once it learns from this setting the size the decoder was
  initialized with.

This clearly continues to allow the client to send requests that refer
to up to 4kB of dictionary until the server announces it grows it. It
implies that a client is allowed to advertise less if desired (including
zero) since the server doesn't need to encode headers before seeing a
client's SETTINGS frame, and maintains the compatibility between existing
implementations. And more importantly it doesn't step over HPACK's feet
and makes it clear that HPACK is the only one having to speak HPACK, and
that DTSU opcodes are only related to HPACK-level changes.

Thoughts ?

Willy