Subject: Attack research on HTTP/2 implementations

From: "Martin Thomson" <[email-masked]>
Date: Fri, 06 Aug 2021 10:43:00 +1000

https://portswigger.net/research/http2

The introduction claims to have found imperfections in the RFC, so I read this fairly carefully.  There's solid work here in terms of attacking implementations, but no concrete specification problems.

In terms of actual changes to specifications, the work we did in the HTTP/2 revision on field validation should already cover all of these attacks.  Not that RFC 7540 didn't, but we're a lot, lot clearer about it now.

There's a lesson in here for our industry regarding how implementations deal with untrustworthy inputs.  The thing we might each reflect on is why we haven't already internalized that lesson.  It's not like this is a new class of attack or anything.

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 6 Aug 2021 04:31:01 +0200

IMHO, there is a limit in how far even the best pure specification
of DO/DONT's can go in creating perfect code. When developers
would be able to more easily read "how to build exploits against bad implementations"
before starting their implementation, i am sure that would go further
in creating better implementations than the best spec alone.

Maybe i am overlooking it, but i can not find a description of e.g.:
the H2.CL exploit in the HTTP/2 revision security section.

writing out exploits in security sections or add-on documents
(http2-bis and http2-break) will give coders also more ammunition
why good implementations cost more money (but also avoid more loss),
require more negative/exploit testing and ideally help coders to proactively
think about similar exploits and how to avoid them (extrapolation).

Once we're beyond this point and bad code is still written it
gets a lot more hairy and becomes an even bigger societal
problem. Instead of getting money from companies like Netflix
for reporting exploits as explained in your linked article, 
you might instead get sued as a security researcher even when
you comply with all well defined reporting procedures for security
weaknesses.  Just yesterday in Germany:

https://www.tellerreport.com/news/2021-08-05-criminal-charges-for-app-notice--cdu-germany-apologizes-to-security-researcher.B1GlL7EFJF.html

Her description of the exploit:

https://lilithwittmann.medium.com/wenn-die-cdu-ihren-wahlkampf-digitalisiert-a3e9a0398b4d

(explanatory screenshots are broken after you use google chrome translate).

Of course, this was the typical HTTP application exploit, which
leaves me with the fatalistic conclusion: Why even make HTTP secure
when HTTP application are going to mess up security anyhow *sigh*

Cheers
    Toerless

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 6 Aug 2021 05:43:05 +0200

Hi Martin,

> https://portswigger.net/research/http2

Thanks for the link, pretty interesting stuff there!

> The introduction claims to have found imperfections in the RFC, so I read
> this fairly carefully.  There's solid work here in terms of attacking
> implementations, but no concrete specification problems.

I agree, unless I'm mistaken, everything that was attacked there is
already dealt with in the spec (allowed characters in values & names
etc).

> In terms of actual changes to specifications, the work we did in the HTTP/2
> revision on field validation should already cover all of these attacks.  Not
> that RFC 7540 didn't, but we're a lot, lot clearer about it now.

Yes the new one is way better and more readable. In 7540 you often have
to compare a series of "must" with a series of "must not" from another
section.

> There's a lesson in here for our industry regarding how implementations deal
> with untrustworthy inputs.  The thing we might each reflect on is why we
> haven't already internalized that lesson.  It's not like this is a new class
> of attack or anything.

I suspect that some of the attacked sites might be using outdated
implementations of some of the usual suspects. We've all had such
weaknesses in our early implementations precisely because they were
not easy to spot in the spec or because some of them were hard to
implement and there was no justification in the spec. For example I
remember that the very first H2 implementation in haproxy didn't
explicitly compare the content-length with the amount of transferred
bytes in the H2 layer since that was already done in the inner HTTP
layers. I don't *think* it could have exposed it to one of these
vulnerabilities, but it's certain that by then I could easily have
overlooked some of them!

In that sense, the new trend of wording around "don't do that because
it exposes to this risk" that we're seeing in the core spec is way
more powerful to encourage to carefully follow all important rules.

Cheers,
Willy

---

From: Nick Harper <[email-masked]>
Date: Thu, 5 Aug 2021 21:01:33 -0700

> already dealt with in the spec (allowed characters in values & names

I saw one thing in the paper that I don't think is addressed by RFC 7540:
the handling of a request that contains both an :authority pseudo-header
and a Host header. I see that draft-ietf-httpbis-http2bis-03 has new
language to mostly cover that issue. I say "mostly" because I don't see any
specification of what should happen if multiple :authority pseudo-headers
are present. (I would argue that that is a malformed request.)

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 6 Aug 2021 06:57:27 +0200

On Thu, Aug 05, 2021 at 09:01:33PM -0700, Nick Harper wrote:
> and a Host header.

This is in fact addressed by HTTP: when you have :authority, it becomes
the authority part of the URI and must match the Host header field. But
I agree that it's not always trivial to address.

> I see that draft-ietf-httpbis-http2bis-03 has new

Yep it's malformed. In 7540#8.1.2.3, it was already said:

   All HTTP/2 requests MUST include exactly one valid value for the
   ":method", ":scheme", and ":path" pseudo-header fields, ...

Willy

---

From: Nick Harper <[email-masked]>
Date: Thu, 5 Aug 2021 22:10:21 -0700

>    ":method", ":scheme", and ":path" pseudo-header fields, ...

Unless I’m misreading something, that only covers some pseudo-headers, but
it doesn’t include :authority. (The same language missing :authority is in
http2bis section 8.3.1.)

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 6 Aug 2021 08:56:48 +0200

On Thu, Aug 05, 2021 at 10:10:21PM -0700, Nick Harper wrote:
> Unless I'm misreading something, that only covers some pseudo-headers, but
> it doesn't include :authority. (The same language missing :authority is in
> http2bis section 8.3.1.)

Indeed, you're absolutely right, I never noticed that. This perfectly
illustrates what I mentioned regarding detailed enumeration versus
intent. I summarized that by making sure no pseudo-header could appear
more than once (and of course making sure all those above are present)
but I made my implementation resistant against this by pure accident :-/

Willy

---

From: Eric J Bowman <[email-masked]>
Date: Fri, 06 Aug 2021 00:04:50 -0700

I think you could just as easily title that document "Request-Response Protocols Considered Harmful."

" 

Web developers are advised to shed assumptions inherited from HTTP/1. It's historically been possible to get away without performing extensive validation on certain user inputs like the request method, but HTTP/2 changes this.

"

To be expected (IMO) when major version numbers are tightly bound to a lower layer of the stack (/1 = TCP, /2 = SPDY, /3 = QUIC) when what I'd like to see is an HTTP spec (RFC) with generic language applicable to any underlying stack, otherwise what, /4 = SCTP anyone? Every time this happens, new assumptions are bound to be inherited, so to speak.

"There's too much confusion, I can't get no relief!"

An accompanying BCP covering input validation *and* sanitization would go a long way to avoiding common attack scenarios regardless of stack choice.

-Eric

---

From: Eric J Bowman <[email-masked]>
Date: Fri, 06 Aug 2021 00:09:27 -0700

---- On Thu, 05 Aug 2021 19:31:01 -0700 Toerless Eckert <[email-masked]> wrote ----

 ------------------

After a decade-long gap in webserver coding, I was almost overwhelmed catching up with all the new exploits I'd neither considered nor heard of before. But that's exactly what's delayed my hobby webserver project, and why I'd like to see an HTTP spec that isn't bound to any underlying stack, accompanied by a BCP for input validation/sanitation.

My solution is a very HTTP-like protocol bound to SCTP, in a server that switches requests based on file permissions. I call it Montage (MONT) protocol, it's back-end -- CDN nodes (if I build my own lol) take any HTTP request and convert to MONT -- a MONT browser would need another layer on the stack (UDP-SCTP) to get around the port 80/443 problem but I don't care. HTTP "DELETE" method is "DE" in MONT, chmod'ing an existing resource to 0000 -- responds 202 followed by 410. Files set to 0000 only accept (if permitted and authenticated) LN (link or maybe REstore) with 205, or RP (replace aka PUT) with 201, followed by 200 OK. 0444 for robots.txt, 0555 for dynamic content, 0666 for .js / .css / .jpg and whatnot.

So the file permissions assigned to any URI (or URI pattern) dictate available methods, in conjunction with authentication. To guide me, I've taken a cleaver to obsolete aspects of HTTP and a scalpel to its bindings and stitched up something useful, at least to myself. Kind of a mishmash of HTTP, what little I can discern about Waka, a touch of FTP, WebDAV, Gopher and Veronica, and extensive perusal of open-source httpd communication methods to create a monster that's impervious to downgrade, injection, and smuggling attacks *by design*.

Montage is no snowflake. It's 90% cribbed from HTTP 1.1. I think the problems have more to do with implementations than protocol specs. I'm just changing the bindings from TCP/SPDY/QUIC to SCTP for back-end communication, it would be nice to have a "generic" HTTP that gives said approach its blessing. Versatile nonetheless, given the inherent nature of the Web (as defined by REST).

-Eric

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 6 Aug 2021 10:22:01 +0200

On Fri, Aug 06, 2021 at 12:04:50AM -0700, Eric J Bowman wrote:
> To be expected (IMO) when major version numbers are tightly bound to a lower
> layer of the stack (/1 = TCP, /2 = SPDY, /3 = QUIC) when what I'd like to see
> is an HTTP spec (RFC) with generic language applicable to any underlying
> stack, otherwise what, /4 = SCTP anyone? Every time this happens, new
> assumptions are bound to be inherited, so to speak.

You mean like this maybe ?

   https://github.com/httpwg/http-core
   https://github.com/httpwg/http2-spec

Willy

---

From: Mike Bishop <[email-masked]>
Date: Fri, 6 Aug 2021 15:04:33 +0000

:authority isn’t included in the list because it’s slightly different – it’s not mandatory.  So the others must occur exactly once (which is stated), while :authority must occur no more than once (which isn’t).

But yes, that should be called out explicitly.

From: Nick Harper <[email-masked]>
Sent: Friday, August 6, 2021 1:10 AM
To: Willy Tarreau <[email-masked]>
Cc: HTTP Working Group <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Attack research on HTTP/2 implementations

On Thu, Aug 5, 2021 at 9:57 PM Willy Tarreau <[email-masked]<mailto:[email-masked]>> wrote:

Unless I’m misreading something, that only covers some pseudo-headers, but it doesn’t include :authority. (The same language missing :authority is in http2bis section 8.3.1.)

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 6 Aug 2021 18:36:12 +0200

On Fri, Aug 06, 2021 at 03:04:33PM +0000, Mike Bishop wrote:
> :authority isn't included in the list because it's slightly different - it's
> not mandatory.

That's what I understood as well.

> So the others must occur exactly once (which is stated),
> while :authority must occur no more than once (which isn't).
> 

Definitely :-)

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 7 Aug 2021 21:30:34 +1200

On 7/08/21 3:04 am, Mike Bishop wrote:
> :authority isn’t included in the list because it’s slightly different – 
> it’s not mandatory.  So the others must occur exactly once (which is 
> stated), while :authority must occur no more than once (which isn’t).
> 

The lack of "exactly one" for :authority has always given me an odd 
feeling. This thread has just brought to my attention the detail that 
may be behind that.

The requirement to provide :scheme without a requirement to provide 
:authority conflicts with the RFC3986 requirement that URLs containing 
scheme MUST also contain an authority section.

Are we at the stage of HTTP compliance yet where :authority can be 
promoted to SHOULD send (or if we want to go all the way MUST), with 
Host downgraded to SHOULD NOT when :authority is present?

Amos

---

From: Erik Aronesty <[email-masked]>
Date: Fri, 6 Aug 2021 00:46:50 -0400

is anyone interested in adding an adaptive ddos-mitigation defense
into the TLS layer so that attackers cannot cause servers to
re-compute public keys in a tight loop?

(server provides nonce + bits + hash algo, client provides a lightweight pow)

i have no idea how to propose this properly in the http2 protocol, but
i do think it would be useful

i dont see it in there

On Fri, Aug 6, 2021 at 12:04 AM Nick Harper <[email-masked]> wrote:
> I saw one thing in the paper that I don't think is addressed by RFC 7540: the handling of a request that contains both an :authority pseudo-header and a Host header. I see that draft-ietf-httpbis-http2bis-03 has new language to mostly cover that issue. I say "mostly" because I don't see any specification of what should happen if multiple :authority pseudo-headers are present. (I would argue that that is a malformed request.)
>>
>>
>> > revision on field validation should already cover all of these attacks..  Not

---

From: Eric J Bowman <[email-masked]>
Date: Mon, 09 Aug 2021 02:51:45 -0700

Needs more meta. As XML is to XHTML. I think it's time for a generic language for HTTP-ish protocols. 

I'm using XProc 3.0 to describe and configure Montage servers (my protocol i.e. MONT, with Diameter support), and define functionality of methods in terms of file permissions and response status codes. What I need is a vocabulary in there to define my method semantics, and their bindings to various transport protocols, specifically SCTP, without the result having to be HTTP/4 i.e. new transport layer for every major revision of an application protocol? Sorry, makes no sense to me.

This vocabulary's spec would strip down section 8.6 to just an explanation of why it's good for clients and servers to know message lengths explicitly, explain the security risks of making CL and TE "just work" over TCP/UDP, then suggest that other transport-layer protocols might be better suited to HTTP (and isn't that just what /2 and /3 attempt). Would define bindings to TCP, UDP, DCCP, SCTP or whatnot. Depends on whether you want to muddle layer boundaries to "solve" the HoL problem, or choose a transport protocol that doesn't have that problem to start with.

Instead of "All general-purpose servers MUST support the methods GET and HEAD" I'd say "MUST support methods for dereferencing URIs with or without content" or somesuch. MONT has no method matching the semantics of HTTP DELETE, ultimate removal of a resource is the job of a housekeeping bot or sysadmin, to reset 404 status. DElete, ReMove, INhibit, or UNlink could result in 410, 451, 401-3. So instead of defining DELETE, I'd maybe define generic removal semantics as changing resource status from 200 to 4xx when subsequently dereferenced. Why should my origin server obliterate all representations of a resource on DELETE? Plenty of use cases where I may want to REstore, RePlace, LiNk, or ENable the resource *without* making it a sysadmin restore-from-backup task.

Compressed HTTPS connections being a security risk is one of those "wow, why didn't I see that coming 20 years ago" things. Probably because I didn't buy into the 7-layer OSI network model, then. But now, I see how the layering violations in HTTP inevitably turn into exploits. Implementing various features of HTTP 1-3 at the transport layer, like by using SCTP, should dramatically reduce the "attack surface area" of the origin server, while eliminating many well-known attacks outright.

So, I think there's a way to more generically define HTTP (as opposed to reconciling nuances between 1-3), while blessing the use of SCTP and custom methods. In a way, I'm advocating security-through-obscurity over having to use HTTP/2 or /3 every step of the way from user-agent to origin server to avoid desync/downgrade/smuggling exploits. Don't get me wrong, I'm also highly praising the state of HTTP these days, in that I think any competing protocol *anyone* can design, will be very HTTP-ish. Just maybe without the layering violations. ;)

-Eric

---- On Fri, 06 Aug 2021 01:22:01 -0700 Willy Tarreau <[email-masked]> wrote ----

 
 
Willy

---

From: Eric J Bowman <[email-masked]>
Date: Mon, 09 Aug 2021 03:05:04 -0700

Ungarbled, that first line should read as follows:

"Needs more meta. As XML is to XHTML. I think it's time for a generic language for HTTP-ish protocols."

-Eric

---

From: "Martin Thomson" <[email-masked]>
Date: Tue, 10 Aug 2021 10:00:51 +1000

Hi Erik,

This is not a new idea.  We've batted the idea around a bit in the TLS working group ([email-masked]).  There are drafts.

Most of these never went anywhere for a range of reasons.  One of those being that proof of work is an abomination (and I don't say that lightly).

Cheers,
Martin

---

From: Paul Vixie <[email-masked]>
Date: Tue, 10 Aug 2021 00:11:03 +0000

On Tue, Aug 10, 2021 at 10:00:51AM +1000, Martin Thomson wrote:
> ...
> 
> Most of these never went anywhere for a range of reasons.  One of those
> being that proof of work is an abomination (and I don't say that lightly).

in cryptocurrency circles, PoW is Proof of Waste. and yes, it's an
abomination. we ran into something similar in NSEC3 for DNSSEC, which
was not only hard to implement interoperably, but also hard to operate
correctly, and ultimately irrelevant even in these pre-quantum times.

(just sayin'.)

-- 
Paul Vixie

---

From: Stephen Farrell <[email-masked]>
Date: Tue, 10 Aug 2021 01:19:48 +0100

It probably isn't, but just in case this is a bit of less
well known history...

On 10/08/2021 01:00, Martin Thomson wrote:
> Most of these never went anywhere for a range of reasons.  One of
> those being that proof of work is an abomination (and I don't say
> that lightly).
I think there were also ideas of less onerous proofs that
went by the name "client puzzles" but that also fell by the
wayside, IIRC due to what's by now likely expired IPR.

Cheers,
S.

---

From: Erik Nygren <[email-masked]>
Date: Mon, 9 Aug 2021 22:25:18 -0400

> wayside, IIRC due to what's by now likely expired IPR.

As an example see:

    https://datatracker.ietf.org/doc/html/draft-nygren-tls-client-puzzles-02

A real challenge is finding a balance that can let a range of legitimate
clients
through while blocking a range of attackers.  (For example, that draft
has both CPU-hard and memory-hard puzzles.)

To avoid battery-life and environmental / energy impact, these would
presumably only get employed during attack scenarios, but figuring
out when that is is a challenge.

The protocol side of this isn't hard, but it adds some real complexity
and understanding how to use it effectively isn't straight-forward.

In one of the TLS WG interims, a request was for a deeper
analysis of the scenarios where client puzzles would and wouldn't help,
which seems like a great academic modeling project for someone
so-inclined.

    Erik

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 10 Aug 2021 13:29:19 +0900

Hello Amos, others,

> scheme MUST also contain an authority section.

Sorry, that's not the case. It's easy to understand when looking at the 
"hier-part" production at
https://datatracker.ietf.org/doc/html/rfc3986/#section-3. An authority 
component is only needed if the scheme is followed by two slashes. There 
are even schemes that allow an empty path
(https://datatracker.ietf.org/doc/html/rfc3986/#section-3.3). That was 
"pioneered" by "dav:", which very reluctantly lead to allowing empty 
paths is RFC 3986.

Regards,   Martin.