Subject: HTTP/2 GREASE, Results, and Implications

From: Mike Bishop <[email-masked]>
Date: Thu, 31 Oct 2019 15:09:34 +0000

Way back when, I presented a draft (https://tools.ietf.org/html/draft-bishop-httpbis-grease-00) proposing that we adopt as an HTTP/2 extension the same behaviors that HTTP/3 is specifying, permitting the greasing of settings and frame types.  The outcome of that discussion was that, prior to considering adoption, we'd want to understand the real-world impact of deploying such a behavior.  Bence generously volunteered to add such an experiment to Chrome, which he has done.

The results are discussed at https://crbug.com/1019410.  TL;DR:  Settings are fine, but too many servers blow up on unknown frame types for this to be viable in major client deployments.  They don't even tell you what they don't like - they just PROTOCOL_ERROR on you.

Frankly, this makes me quite sad.  It means that our primary extension mechanism for HTTP/2 has already rusted shut, and it's now inadvisable to define new optional-to-understand frame types and send them without prior negotiation.

Now that we have this data, are we interested in pursuing the draft with settings only, or perhaps reserving frame types but recommending caution in their use?

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 31 Oct 2019 15:46:08 +0000

On Thu, Oct 31, 2019 at 3:14 PM Mike Bishop <[email-masked]> wrote:

> Way back when, I presented a draft (
> https://tools.ietf.org/html/draft-bishop-httpbis-grease-00) proposing
> that we adopt as an HTTP/2 extension the same behaviors that HTTP/3 is
> specifying, permitting the greasing of settings and frame types.  The
> outcome of that discussion was that, prior to considering adoption, we’d
> want to understand the real-world impact of deploying such a behavior.
> Bence generously volunteered to add such an experiment to Chrome, which he
> has done.
>
>
>
> The results are discussed at https://crbug.com/1019410.  TL;DR:  Settings
> are fine, but too many servers blow up on unknown frame types for this to
> be viable in major client deployments.  They don’t even tell you what they
> don’t like – they just PROTOCOL_ERROR on you.
>
>
>

Thanks for the experimentation and sharing the results Mike and Bence.

Is the sense that this is symmetrically broken? Do we have data about how
server-sent GREASE frames might break clients? (and if not would that move
the needle at all).

> Frankly, this makes me quite sad.  It means that our primary extension
> mechanism for HTTP/2 has already rusted shut, and it’s now inadvisable to
> define new optional-to-understand frame types and send them without prior
> negotiation.
>
>
>
> Now that we have this data, are we interested in pursuing the draft with
> settings only, or perhaps reserving frame types but recommending caution in
> their use?
>

This indeed has some practical implications to active work in the group. I
can see how there might be some merit in capturing this situation, along
with some guidance, in a draft that can be reference by people making
HTTP/2 extensions.

Based on my experience of HTTP/3 interop to date, we are doing pretty well
with GREASE perhaps it is time to capture this in the matrix. I'd also like
to highlight that today the Cloudflare edge exercises all HTTP/3 grease
mechanisms* for all connections.

* unidirectional stream type GREASE is sent when sufficient stream credit
is provided by the client e.g. more than 3

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 31 Oct 2019 16:52:59 +0100

Hi Mike!

On Thu, Oct 31, 2019 at 03:09:34PM +0000, Mike Bishop wrote:
> Way back when, I presented a draft
> (https://tools.ietf.org/html/draft-bishop-httpbis-grease-00) proposing that
> we adopt as an HTTP/2 extension the same behaviors that HTTP/3 is specifying,
> permitting the greasing of settings and frame types.  The outcome of that
> discussion was that, prior to considering adoption, we'd want to understand
> the real-world impact of deploying such a behavior.  Bence generously
> volunteered to add such an experiment to Chrome, which he has done.
> 
> The results are discussed at https://crbug.com/1019410.  TL;DR:  Settings are
> fine, but too many servers blow up on unknown frame types for this to be
> viable in major client deployments.  They don't even tell you what they don't
> like - they just PROTOCOL_ERROR on you.
> 
> mechanism for HTTP/2 has already rusted shut, and it's now inadvisable to

Ah! That's interesting, so that's the problem Yves and me have been
working on yesterday where we found frame types 0xF7 sent by a Beta
version of Chrome and rejected by haproxy! At least now we have an
explanation! And I have a bug to fix now.

But quite frankly that's exactly the expected outcome of a spec written
in directive way instead of normative. I mean, instead of indicating
state transitions it says what an agent must do. It's impossible for
such a long text to remain consistent along the whole document and you
end up with some conflicting rules. Here in the traces we've collected
yesterday, we've found a frame type 0xF7 sent in half-closed(remote)
state. And the spec says :

   half-closed (remote):
      If an endpoint receives additional frames, other than
      WINDOW_UPDATE, PRIORITY, or RST_STREAM, for a stream that is in
      this state, it MUST respond with a stream error (Section 5.4.2) of
      type STREAM_CLOSED.

So by this rule, the 0xF7 I saw was not permitted.

Further down it says:

   In the absence of more specific guidance elsewhere in this document,
   implementations SHOULD treat the receipt of a frame that is not
   expressly permitted in the description of a state as a connection
   error (Section 5.4.1) of type PROTOCOL_ERROR.  Note that PRIORITY can
   be sent and received in any stream state.  Frames of unknown types
   are ignored.

And it's just in the middle of a paragraph in 5.5 that I can see :

   Implementations MUST discard frames that have unknown or unsupported
   types.

With that said, for the cases I'm responsible of, the fixes are easy,
and I'm even fine with tagging them as major bugs so that the fixes
flow down into distros. Clearly, we try to fight protocol ossification
but by writing specs that look almost like code with very strict rules
we encourage it by accident, and we must be extremely careful about
this in the future.

I also think that for future designs, indicating prominently in a
spec that some type ranges are reserved for experimentation and
some for extensions will already ring a bell to some developers.

There's still a problem is with already deployed code, but H2 compatible
components in field are still young and experiencing frequent updates,
so users will easily apply fixes. Thus I think that it's probably not
to late to continue to enforce using unknown frame types to send such
signals from time to time is the best way to make sure they're updated.

And, just like there have been flag days for IPv6 or DNS, we could have
some H2 days once or twice a year during which browsers will send random
protocol elements. Just making this public will encourage users to care
about the correctness of their components and pressure their vendors to
provide a compliant implementation.

Just my two cents,
Willy

---

From: Mike Bishop <[email-masked]>
Date: Thu, 31 Oct 2019 15:57:08 +0000

Bence’s experiment didn’t cover anything server-sent, that I’m aware of.  Of course, if Cloudflare would like to do a corresponding experiment…?  😉

From: Lucas Pardue <[email-masked]>
Sent: Thursday, October 31, 2019 11:46 AM
To: Mike Bishop <[email-masked]>
Cc: HTTP Working Group <[email-masked]>
Subject: Re: HTTP/2 GREASE, Results, and Implications

On Thu, Oct 31, 2019 at 3:14 PM Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
Way back when, I presented a draft (https://tools.ietf.org/html/draft-bishop-httpbis-grease-00) proposing that we adopt as an HTTP/2 extension the same behaviors that HTTP/3 is specifying, permitting the greasing of settings and frame types.  The outcome of that discussion was that, prior to considering adoption, we’d want to understand the real-world impact of deploying such a behavior.  Bence generously volunteered to add such an experiment to Chrome, which he has done.

The results are discussed at https://crbug.com/1019410.  TL;DR:  Settings are fine, but too many servers blow up on unknown frame types for this to be viable in major client deployments.  They don’t even tell you what they don’t like – they just PROTOCOL_ERROR on you.

Is the sense that this is symmetrically broken? Do we have data about how server-sent GREASE frames might break clients? (and if not would that move the needle at all).

Frankly, this makes me quite sad.  It means that our primary extension mechanism for HTTP/2 has already rusted shut, and it’s now inadvisable to define new optional-to-understand frame types and send them without prior negotiation.

This indeed has some practical implications to active work in the group. I can see how there might be some merit in capturing this situation, along with some guidance, in a draft that can be reference by people making HTTP/2 extensions.

Based on my experience of HTTP/3 interop to date, we are doing pretty well with GREASE perhaps it is time to capture this in the matrix. I'd also like to highlight that today the Cloudflare edge exercises all HTTP/3 grease mechanisms* for all connections.

* unidirectional stream type GREASE is sent when sufficient stream credit is provided by the client e.g. more than 3

---

From: Ian Swett <[email-masked]>
Date: Thu, 31 Oct 2019 12:08:06 -0400

If only some servers accepted SETTINGS, that would further constrain the
negotiation process to only allowing the server to send the SETTING after
receiving the client's.  It'd be good to know if that's necessary, but
hopefully it's not.

On Thu, Oct 31, 2019 at 11:59 AM Mike Bishop <[email-masked]> wrote:

> Bence’s experiment didn’t cover anything server-sent, that I’m aware of.
> Of course, if Cloudflare would like to do a corresponding experiment…?  😉
>
>
>
> *From:* Lucas Pardue <[email-masked]>
> *Sent:* Thursday, October 31, 2019 11:46 AM
> *To:* Mike Bishop <[email-masked]>
> *Cc:* HTTP Working Group <[email-masked]>
> *Subject:* Re: HTTP/2 GREASE, Results, and Implications
>
>
>
>
>

---

From: Bence Béky <[email-masked]>
Date: Thu, 31 Oct 2019 13:08:53 -0400

Thanks, Willy, for pointing out the different sections of RFC7540
concerning unknown frame types.  It seems to me that for the past few days
Chrome (on certain channels) has been sending frames on half-closed
(remote) streams.  It might be worth fixing that and re-running the
experiment.  I'll circle back with results if we end up doing that.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 31 Oct 2019 18:15:59 +0100

> experiment.  I'll circle back with results if we end up doing that.

OK. I have a pending patch ready for testing on the haproxy front which
addresses this mismatch. The only thing is that chrome beta doesn't seem
to be available on linux so I constantly have to bother other people for
testing, which takes time :-/

Regarding the impacts on our uesrs of anoyher test, I'm not much worried
of the risk to break a few sites running unfixed versions of haproxy if
a new Chrome version enables the test again; we've faced some hard to
diagnose bugs not too long ago and users are trained to disable H2 if
something really bad happens. Of course it's not desirable but it's not
as if everything definitely broke. Also our community tends to be quite
reactive when it comes to applying important fixes. Thus I full support
restarting the test ASAP :-)

Cheers,
Willy

---

From: Mike Bishop <[email-masked]>
Date: Thu, 31 Oct 2019 17:49:47 +0000

Upon further investigation, it appears that Akamai hosts most of the sites where the connection gets closed, and Cloudflare hosts the sites which close the stream but leave the connection open (I’m guessing, from what the dev tools show).  Given that we’re both represented here, it seems likely that we should be able to drive some internal bugfixes and improve the state of the ecosystem.

From: Mike Bishop <[email-masked]>
Sent: Thursday, October 31, 2019 11:57 AM
To: Lucas Pardue <[email-masked]>
Cc: HTTP Working Group <[email-masked]>
Subject: RE: HTTP/2 GREASE, Results, and Implications

From: Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, October 31, 2019 11:46 AM
To: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Cc: HTTP Working Group <[email-masked]<mailto:[email-masked]>>
Subject: Re: HTTP/2 GREASE, Results, and Implications

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 31 Oct 2019 18:10:33 +0000

On Thu, Oct 31, 2019 at 5:49 PM Mike Bishop <[email-masked]> wrote:

> Upon further investigation, it appears that Akamai hosts most of the sites
> where the connection gets closed, and Cloudflare hosts the sites which
> close the stream but leave the connection open (I’m guessing, from what the
> dev tools show).  Given that we’re both represented here, it seems likely
> that we should be able to drive some internal bugfixes and improve the
> state of the ecosystem.
>

It's possible the Cloudflare closures are related to the client-oriented
half-closed issue that Willy and Bence have discussed. We'd be happy to
coordinate with other members of the group on a rerun of the experiment in
order to come to a more definitive answer.

Lucas

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 31 Oct 2019 19:55:43 +0100

On Thu, Oct 31, 2019 at 06:10:33PM +0000, Lucas Pardue wrote:
> > close the stream but leave the connection open (I'm guessing, from what the
> > dev tools show).  Given that we're both represented here, it seems likely
> half-closed issue that Willy and Bence have discussed.

Maybe, but regardless, if we have to accept any frame type, half-closed
makes no difference. For haproxy I still purposely decided not to accept
to create a new stream from a unknown frame type (i.e. a frame type
designating an idle stream must be either HEADERS or PRIORITY as stated
in the spec). Because if we start to let such things pass through, we
can't really count on the other end's behavior for subsequent frames :-/

> We'd be happy to

It would be nice to retest at world scale and be a bit more prepared to
assist users who report breakage.

Willy

---

From: David Benjamin <[email-masked]>
Date: Thu, 31 Oct 2019 15:05:16 -0400

> testing, which takes time :-/

Chrome beta (also dev channel) should be available on Linux. See
https://www.google.com/chrome/beta/?platform=linux. (Or if you've already
gotten the apt repository added, I believe the google-chrome-beta package
should work.) It also uses a separate profile directory, so it won't
conflict with your usual install. Hopefully that'll ease testing.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 31 Oct 2019 20:11:04 +0100

Hi David,

> conflict with your usual install. Hopefully that'll ease testing.

Sorry I used the wrong word. The beta I download there is 78.something
and does not match the latest one Yves was running (80 something). I
tried it already and it doesn't fail. I also found "chrome dev" which
is 79.something and doesn't present the problem either. When I go to
the "chrome canary" link, it clearly says in the middle of the screen
"we do not support linux". So for now I can't test, that's pretty sad :-/

Willy

---

From: David Benjamin <[email-masked]>
Date: Thu, 31 Oct 2019 15:20:15 -0400

> "we do not support linux". So for now I can't test, that's pretty sad :-/

Ah, gotcha. Yeah, we don't have a canary release for Linux sadly. Just
stable, beta, and dev. And, indeed, it looks like 80 hasn't made its way to
dev yet. :-/
https://chromiumdash.appspot.com/releases?platform=Linux

That said, it looks like we implemented GREASE for HTTP/2 quite a long time
ago. The experiment just wasn't enabled until now. You should be able to
force it on via a command-line flag on any remotely recent version. I'm not
sure off-hand what the incantation is. Bence?

David

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 31 Oct 2019 20:26:01 +0100

> sure off-hand what the incantation is. Bence?

Ah that would be great, I've been wondering the same as well but found
no info on this. And running chrome --help says "no man page" so I must
confess I've already met more verbose software :-)

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 31 Oct 2019 21:34:58 +0100

On Thu, Oct 31, 2019 at 01:08:53PM -0400, Bence Béky wrote:
> (remote) streams.

Bence, I saw that you updated the issue regarding this point. If
you're going to be stricter on the states to respect the (confusing)
wording of the spec, then you need to be very careful about other states
for completeness:

  - closed: An endpoint MUST NOT send frames other than PRIORITY on a closed
    stream.
  - reserved (remote): An endpoint MUST NOT send any type of frame other
    than RST_STREAM, WINDOW_UPDATE, or PRIORITY in this state.
  - idle: Receiving any frame other than HEADERS or PRIORITY on a stream in
    this state MUST be treated as a connection error.

I think it's reasonable to assume that only these transitions (in addition
to the already mentioned half-closed) face such restrictions. This means
that any new non-negociated frame type will still be allowed for the
connection (stream 0) and open streams. This sounds quite reasonable and
should not risk to trigger the consistency issues in the spec's wording.

And at first glance if that's the case I don't need to patch my code,
so we can expect that other implementations faced similar issues and
that the limitations above remain acceptable (but an errata will have
to be filed to keep track of this).

Hoping this helps,
Willy