Subject: Structured request headers deployment issues

From: Yoav Weiss <[email-masked]>
Date: Tue, 16 Jun 2020 00:15:06 +0200

Hey all,

Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent Client
Hints enabled by default, which is using Structured Headers.

As a result of that, we found multiple sites
<https://bugs.chromium.org/p/chromium/issues/detail?id=1091285> which seem
to have a somewhat allergic reaction to the presence of certain characters
(that are part of the SH format) in request values.
While each site in question is different (in what appears to be coming from
different stacks), we've seen sites that reject requests with quotes,
question marks or equals signs in them.
It's still early, so it's hard to know how widespread the issue is, but we
seem to be adding sites to the list at a faster pace than the pace of
removing fixed ones from it.

So, I wanted to give this group a heads-up on that front, and maybe get
folks' opinions regarding possible things we could do on that front, other
than outreach and waiting for said sites to fix themselves.

Cheers,
Yoav

---

From: Julian Reschke <[email-masked]>
Date: Thu, 18 Jun 2020 12:59:07 +0200

On 16.06.2020 00:15, Yoav Weiss wrote:
> Hey all,
>
> Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent
> Client Hints enabled by default, which is using Structured Headers.
>
> <https://bugs.chromium.org/p/chromium/issues/detail?id=1091285> which
> seem to have a somewhat allergic reaction to the presence of certain
> characters (that are part of the SH format) in request values.
> While each site in question is different (in what appears to be coming
> from different stacks), we've seen sites that reject requests with
> quotes, question marks or equals signs in them.
> It's still early, so it's hard to know how widespread the issue is, but
> we seem to be adding sites to the list at a faster pace than the pace of
> folks' opinions regarding possible things we could do on that front,
> other than outreach and waiting for said sites to fix themselves.
> ...

Thanks for the heads up.

It would be a really bad outcome if that meant that we can't define new
request header fields using certain delimiters in their values.

That said, the latest change on the ticket appears to be from Monday, so
maybe the situation is not as bad as you feared?

Best regards, Julian

---

From: Yoav Weiss <[email-masked]>
Date: Thu, 18 Jun 2020 13:35:11 +0200

> maybe the situation is not as bad as you feared?

Maybe. We'll see soon enough :)

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 19 Jun 2020 15:13:13 +1000

Hey Yoav,

> On 16 Jun 2020, at 8:15 am, Yoav Weiss <[email-masked]> wrote:
> 
> Hey all,
> 
> Chromium M84 (which Chrome equivalent is now in Beta) has User-Agent Client Hints enabled by default, which is using Structured Headers.
> 
> As a result of that, we found multiple sites which seem to have a somewhat allergic reaction to the presence of certain characters (that are part of the SH format) in request values. 
> While each site in question is different (in what appears to be coming from different stacks), we've seen sites that reject requests with quotes, question marks or equals signs in them.
> It's still early, so it's hard to know how widespread the issue is, but we seem to be adding sites to the list at a faster pace than the pace of removing fixed ones from it.
> 
> So, I wanted to give this group a heads-up on that front, and maybe get folks' opinions regarding possible things we could do on that front, other than outreach and waiting for said sites to fix themselves.

AIUI these aren't new; e.g., IIRC quite a few months ago Chrome encountered several Austrian sites that had this problem, traced back to a local(?) WAF vendor there. I believe that's been corrected since, after reaching out to them.

Personally, I think that outreach and waiting is the right approach; if browsers consistently send these headers, they'll adapt, and the numbers are still relatively small -- or at least small enough that it's not likely the numbers will be reduced if the syntax is changed (due to _other_ WAFs' opinions about what a "good" request is).

Also, if we get these headers through, it seems like it would give us good protection (of a sort) for future Structured request headers.

Related, we're also seeing more examples WAFs limiting how we can evolve the protocol (e.g., <https://github.com/coreruleset/coreruleset/pull/1777>). There's been a bit of background chatter about writing something about this and creating better communication with that community; I'm not sure what that will look like yet, but if anyone has ideas or is interested, please say so.

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Mike West <[email-masked]>
Date: Fri, 19 Jun 2020 09:03:18 +0200

On Fri, Jun 19, 2020 at 7:13 AM Mark Nottingham <[email-masked]> wrote:

> > As a result of that, we found multiple sites which seem to have a
> somewhat allergic reaction to the presence of certain characters (that are
> part of the SH format) in request values.
> from different stacks), we've seen sites that reject requests with quotes,

Yoav, have we experimented at all with alternate spellings? Do
single-quotes have the same impact as double-quotes, for example? Are other
(uglier) non-alphanumeric delimiters less likely to cause trouble? There's
a limited appetite for breakage, and it might be worth exploring alternate
aesthetics rather than forcing through the particular spelling we like the
most.

> AIUI these aren't new; e.g., IIRC quite a few months ago Chrome
> encountered several Austrian sites that had this problem, traced back to a
> local(?) WAF vendor there. I believe that's been corrected since, after
> reaching out to them.
>

I think you might be referring to some conversations around the
(since-renamed) `Sec-Metadata` header in 2018, like those captured in
https://bugs.chromium.org/p/chromium/issues/detail?id=861678#c11.

Personally, I think that outreach and waiting is the right approach; if
> browsers consistently send these headers, they'll adapt, and the numbers
> are still relatively small -- or at least small enough that it's not likely
> the numbers will be reduced if the syntax is changed (due to _other_ WAFs'
> opinions about what a "good" request is).
>

The flip-side of this is that we break sites users rely upon. That's tough
to do at scale. In this case in particular, I think I agree with the
underlying assertion that we'd like this syntax to work, and it seems
reasonable to me to roll it out to a small percentage of stable users. That
said, if we end up breaking the internet for even a small percentage of
users, it doesn't seem like a good idea to hold them hostage in the hopes
of increasing pressure on WAF vendors.

My expectation is that we'd roll this out to a small percentage of stable,
see a spike in bug reports (especially from enterprise folks who are likely
to have WAF deployments, unlikely to broadly deploy beta channels, and
unlikely to choose to enable usage statistics or histograms), see a
corresponding spike in error codes for top-level navigations, and roll it
back.

I think it might be reasonable to explore alternate spellings at the same
time, perhaps with some A/B testing to evaluate how well each weaves its
way through middleware.

Related, we're also seeing more examples WAFs limiting how we can evolve
> the protocol (e.g., <https://github.com/coreruleset/coreruleset/pull/1777>).
> There's been a bit of background chatter about writing something about this
> and creating better communication with that community; I'm not sure what
> that will look like yet, but if anyone has ideas or is interested, please
> say so.
>

I am interested!

-mike

---

From: Yoav Weiss <[email-masked]>
Date: Fri, 19 Jun 2020 10:38:09 +0200

> aesthetics rather than forcing through the particular spelling we like the

We have not. But given the fact that we've seen issues with the use of `"`,
`?` and `=` separately, I'm somewhat assuming that those separate
implementations are using an allow-list approach.
With that said, I can try to throw various values at those known-to-break
servers and see if I can come up with an alternative spelling that would
work with them. It won't give us any guarantees that these spellings would
work at scale, but it might be a good starting point...

---

From: Yoav Weiss <[email-masked]>
Date: Fri, 19 Jun 2020 12:15:00 +0200

On Fri, Jun 19, 2020 at 10:38 AM Yoav Weiss <[email-masked]> wrote:

>>> > It's still early, so it's hard to know how widespread the issue is,
>>> but we seem to be adding sites to the list at a faster pace than the pace
>>> of removing fixed ones from it.
>>>
>>
> We have not. But given the fact that we've seen issues with the use of
> `"`, `?` and `=` separately, I'm somewhat assuming that those separate

I played around with trying to find a safe sete of chars that would work as
delimiters on the set of sites we know are broken. I have failed
<https://bugs.chromium.org/p/chromium/issues/detail?id=1091285#c10> to
find such a subset :/

>> The flip-side of this is that we break sites users rely upon. That's
>> tough to do at scale. In this case in particular, I think I agree with the
>> My expectation is that we'd roll this out to a small percentage of
>> stable, see a spike in bug reports (especially from enterprise folks who
>> are likely to have WAF deployments, unlikely to broadly deploy beta
>> channels, and unlikely to choose to enable usage statistics or histograms),
>> see a corresponding spike in error codes for top-level navigations, and
>> roll it back.
>>
>>> the protocol (e.g., <
>>> https://github.com/coreruleset/coreruleset/pull/1777>). There's been a
>>> bit of background chatter about writing something about this and creating
>>> better communication with that community; I'm not sure what that will look
>>> like yet, but if anyone has ideas or is interested, please say so.
>>>
>>
>> I am interested!
>>
>> -mike
>>
>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 22 Jun 2020 13:19:37 +1000

Hey Mike,

> On 19 Jun 2020, at 5:03 pm, Mike West <[email-masked]> wrote:
> 
> The flip-side of this is that we break sites users rely upon. That's tough to do at scale. In this case in particular, I think I agree with the underlying assertion that we'd like this syntax to work, and it seems reasonable to me to roll it out to a small percentage of stable users. That said, if we end up breaking the internet for even a small percentage of users, it doesn't seem like a good idea to hold them hostage in the hopes of increasing pressure on WAF vendors.

I don't think we disagree. The problem is that there is always going to be a small percentage that break, unless we severely limit the evolution of the Web (i.e., don't use new headers at all, especially given the data Yoav is bringing!), so the relevant question is *how much*.

In other words, there's a theshold below which you're comfortable breaking a *few* sites, and those few sites presumably feel the pressure to change. Above that threshold, you're unwilling to cause too much damage, and the sites presumably feel comfortable in persisting their behaviour (acknowledging that there's some fuzziness on both sides).

> My expectation is that we'd roll this out to a small percentage of stable, see a spike in bug reports (especially from enterprise folks who are likely to have WAF deployments, unlikely to broadly deploy beta channels, and unlikely to choose to enable usage statistics or histograms), see a corresponding spike in error codes for top-level navigations, and roll it back.
> 
> I think it might be reasonable to explore alternate spellings at the same time, perhaps with some A/B testing to evaluate how well each weaves its way through middleware.

What's missing here are the ability to engage the WAF community, and time. Unlike Web sites, there's a relatively smaller number of WAF vendors out there, so it's not unreasonable to engage with them to bring the numbers down below the threshold. That will take time, especially since WAFs don't yet have auto-update capabilities on par with browsers, but I think it's worth doing.

What I don't want to see is the Web become ossified on what a few WAFs happened to do in June 2020.

>> Related, we're also seeing more examples WAFs limiting how we can evolve the protocol (e.g., <https://github.com/coreruleset/coreruleset/pull/1777>). There's been a bit of background chatter about…

Great. I've heard from a few other folks too, so we'll try to figure out the appropriate venue for further discussion.

Cheers,

---

From: Mike West <[email-masked]>
Date: Mon, 22 Jun 2020 07:32:49 +0200

On Mon, Jun 22, 2020 at 5:19 AM Mark Nottingham <[email-masked]> wrote:

> I don't think we disagree. The problem is that there is always going to be
> a small percentage that break, unless we severely limit the evolution of
> the Web (i.e., don't use new headers at all, especially given the data Yoav
> is bringing!), so the relevant question is *how much*.
>
> In other words, there's a theshold below which you're comfortable breaking
> a *few* sites, and those few sites presumably feel the pressure to change.
> Above that threshold, you're unwilling to cause too much damage, and the
> sites presumably feel comfortable in persisting their behaviour
> (acknowledging that there's some fuzziness on both sides).
>

I agree that we're not disagreeing. :)

> > I think it might be reasonable to explore alternate spellings at the
> same time, perhaps with some A/B testing to evaluate how well each weaves
> its way through middleware.
>
> What's missing here are the ability to engage the WAF community, and time.
> Unlike Web sites, there's a relatively smaller number of WAF vendors out
> there, so it's not unreasonable to engage with them to bring the numbers
> down below the threshold. That will take time, especially since WAFs don't
> yet have auto-update capabilities on par with browsers, but I think it's
> worth doing.
>

For better or worse, it seems to me that this asymmetry in update
capability puts more onus on user agents to fit themselves to the world as
it exists, as they're likely to be able to move more quickly (or at all!).

> happened to do in June 2020.

To this end, I wonder if some sort of GREASEd header would be reasonable
for user agents to start sending on some subset of requests. The user agent
client hints might end up accidentally serving as that header, but I don't
think Chromium's implementation even attempts to exercise all the bits and
pieces of https://tools.ietf.org/html/draft-ietf-httpbis-header-structure.

> Great. I've heard from a few other folks too, so we'll try to figure out
> the appropriate venue for further discussion.
>
> Cheers,
>
>
> --