Subject: Dealing with header injection through reverse proxies

From: Eric Rescorla <[email-masked]>
Date: Mon, 17 Jul 2017 08:18:10 -0700

i folks,

We had a discussion today in TOKBIND about handling security-sensitive
indications in HTTP headers (this came up in the context of
https://tools.ietf.org/html/draft-campbell-tokbind-ttrp-01).  The
setting here is that you have a network with a TLS reverse proxy
serving the origin server, and the TLS proxy is responsible for doing
some security check and telling the server about it. E.g.,

    Client                    Proxy                     Server

    <--- TLS w/ client auth ---> <----- HTTP with cert --->

The client does TLS client authentication with the proxy and then
passes the certificate to the back-end server in an injected HTTP
header (e.g., X-Client-Certificate). In order for this to be secure,
the proxy has to *strip* any security sensitive headers sent by the
client. Otherwise, the client could inject their own headers that
would appear to come from the proxy.

Obviously, this design doesn't fail safe if the proxy fails to
strip the headers. One way in which this can happen is if you
have a large network of load balancers fronting a server network
and you somehow incompletely misconfigure either the servers
or the proxies, so that a server which supports this mechanism
ends up behind a proxy which does not (and hence does not strip
the headers). So, it would be nice to do something better that
wasn't too heavyweight, especailly as a general solution.

One natural design is simply to have a shared key between the
proxy and the server. In that case, it's easy to demonstrate
that the header is injected, as in [0][1]

   X-Client-Certificate: <key>, <certificate>

The obvious objection to this design is that it requires you to
establish the shared key, and people were concerned about having to
configure it into the proxy. I'm aware of a number of designs here:

- Configure it via some sort of remote configuration mechanism.
- The proxy could send it to the server on its first request
  (might require some trickery on the server).
- If you have a TLS channel between Proxy and Server you can
  use a TLS exporter.
- If you have a H2 connection between the two, you can have the
  server provide it in a settings frame.
- Use a signature by the proxy using its private key over something
  e.g., S(K, "Token binding"). If you use a mostly fixed string,
  then the server just needs to verify it once.

Do people have other thoughts on this problem? Other good ways to
establish the key? Other ideas for how to address it?

-Ekr

[0] I'd originally suggested HMACing, but that's actually not
necessary, though obviously stronger.

[1] Note that you could have a generic solution, like a header which
listed all the injected headers, but with the same security mechanism.

---

From: Patrick McManus <[email-masked]>
Date: Mon, 17 Jul 2017 16:25:40 +0000 (UTC)

> so that a server which supports this mechanism
> the headers).

If the problem can be reduced to the above then I think there are some http
features that are worth considering if you're willing to look at it as a
single hop requirement:

a] if the proxy->server link is h1 then the x-client-certificate name can
be included in the Connection header sent to the server (and the server can
enforce this property). Any naive proxy receiving that same combination
from a malicious client would remove the request-header of the same name
before forwarding it on to the server and then generate its own Connection
header.

b] if the proxy->server link is h2 then you can inject connection-specific
information into the stream with an extension frame type (and the server
can enforce this property). You don't need to negotiate it with SETTINGS
(which is nice, because that's a round trip.) and these frames are hop to
hop (proxies that don't understand a frame type MUST drop them).

Its not too hard to imagine a generalization of (b).. a
CONNECTION-SPECIFIC-HEADERS or somesuch that has strict hop-to-hop
semantics.

-P

---

From: John Bradley <[email-masked]>
Date: Mon, 17 Jul 2017 18:33:56 +0200

It would be good if the new mechanism could also be applied to existing reverse proxy headders like SSL_CLIENT_CERT and SSL_CLIENT_CERT_CHAINn without breaking the existing parsing.

That might lead to a preference for a header that lists the injected headers as a possibility.

John B.

> On Jul 17, 2017, at 5:18 PM, Eric Rescorla <[email-masked]> wrote:
> 
> i folks,
> 
> https://tools.ietf.org/html/draft-campbell-tokbind-ttrp-01 <https://tools.ietf.org/html/draft-campbell-tokbind-ttrp-01>).  The
> _______________________________________________
> Unbearable mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/unbearable

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 17 Jul 2017 20:45:07 +0200

Hi Eric,

> configure it into the proxy. I'm aware of a number of designs here:
(...)

What I've seen and used was slightly different :
  1) proxies unconditionally remove the header field
  2) proxies unconditionally add the new header field even with no
     certificate
  3) servers verify that there is exactly one header field

This way even if step 1 above fails (eg: usual typo in the rule needed
to strip the header field which nobody notices since nobody injects
such a field name), step 2 ensures that any injection will be detected
in step 3.

In my opinion, it's important to stick to simple principles that can
be troubleshooted and fixed in field. For example I've had to deal
with signed header fields in the past and it used to be a great pain
to debug for everyone involved, to the point that people undervalue
the extra security. Simply prepending a clearly visible static value
into a field could help without adding much hassle but that's more or
less where the realistic limit lies in my opinion.

Cheers,
Willy

---

From: Piotr Sikora <[email-masked]>
Date: Tue, 18 Jul 2017 11:16:16 +0200

Hey Patrick,

> a] if the proxy->server link is h1 then the x-client-certificate name can be
> included in the Connection header sent to the server (and the server can
> enforce this property). Any naive proxy receiving that same combination from
> a malicious client would remove the request-header of the same name before
> forwarding it on to the server and then generate its own Connection header.

This would work... but only in a single-hop deployments.

> information into the stream with an extension frame type (and the server can
> enforce this property). You don't need to negotiate it with SETTINGS (which
> is nice, because that's a round trip.) and these frames are hop to hop
> (proxies that don't understand a frame type MUST drop them).

Putting "internal" headers into a new HTTP/2 frame instead of mixing
them with client headers in the HEADERS frame is the idea that I've
been toying for a while now, mostly for multi-hop proxy deployments.

However, it's unclear whether the backend server should receive only
client headers (which wouldn't solve this particular issue) or both,
and how should this work with "internal" headers crossing
organizational boundaries (i.e. CDN terminating TLS and forwarding
"X-Client-Certificate" header to the origin server).

Also, this requires end-to-end HTTP/2, which is pretty much
non-existent nowadays, since most proxies don't support HTTP/2 to
backends (yet).

Having said that, this is something that I definitely want to see
done, regardless of whether it's the right solution for this
particular issue or not.

Best regards,
Piotr Sikora

---

From: Piotr Sikora <[email-masked]>
Date: Tue, 18 Jul 2017 11:20:16 +0200

Hey Willy,

> such a field name), step 2 ensures that any injection will be detected

This is exactly what the current draft suggests and what EKR objects,
because misconfigured proxy that doesn't know about
"X-Client-Certificate" won't execute steps 1-3 for the
"X-Client-Certificate" header.

Best regards,
Piotr Sikora

---

From: Piotr Sikora <[email-masked]>
Date: Tue, 18 Jul 2017 11:23:39 +0200

Hey Brian,

> I perhaps didn't articulate it well in the meeting yesterday but I'd like to
> reiterate that, in my opinion and I don't think I'm alone, it would be very
> inappropriate for -tokbind-ttrp to define a one-off mechanism for the origin
> server to detect client injection of the headers. The potential problem of
> client header injection is not at all unique to the functionality of that
> draft so the draft shouldn't define a unique solution. It would be useful if
> there were a common standardized mechanism for doing detecting/preventing
> client header injection that the -tokbind-ttrp draft could refer to (the
> generic solution that Ekr mentions in his [1] seems preferable precisely
> because it is generally applicable). In the absence of a generic solution
> existing currently, stripping/sanitizing the headers is the de facto means
> of dealing with the situation in practice today, is sufficient when properly
> implemented, and is normatively required by the text in -tokbind-ttrp. It's
> true that, if the reverse proxy is defective/misconfigured, it doesn't fail
> safe but in the context of -tokbind-ttrp that failure mode is far from
> catastrophic. Such a failure loses the protections afforded by token
> binding, which is not ideal, but it is the current state of just about
> everything on the web today so it's not *that* bad.

I agree. I don't believe that TOKBIND-TTRP draft should be blocked on this.

Best regards,
Piotr Sikora

---

From: Graham Leggett <[email-masked]>
Date: Tue, 18 Jul 2017 12:53:59 +0200

On 17 Jul 2017, at 5:18 PM, Eric Rescorla <[email-masked]> wrote:

>     <--- TLS w/ client auth ---> <----- HTTP with cert —>

A few years ago I needed to solve this problem.

I needed a secure, typo-proof way of passing the original IP address from the frontend endpoints, through various reverse proxies, through various service layers which in turn were fronted with various reverse proxies, to an eventual service that cared what the IP address was.

In this scenario, it is trivial to misconfigure things and have the wrong IP address passed on, which was a problem for us.

What we had was a signed header:

X-Foo: SIGNATUREORERROR;value

The frontend endpoint would sign this header, turning X-Foo: value into the above.

Each layer that the header passed through that cared would then verify the header, and if the header verified successfully it would be passed through unchanged. If the header did not verify, the contents of the header was replaced with an error message like so and the value discarded:

X-Foo: error:signature-expired-or-whatever;

What this did was two things:

- Any value that couldn’t be verified was discarded, ie the part after the semicolon was made blank. This stopped developers ignoring the failed signatures.

- The error message was put there so that a bone could be thrown to whoever had to troubleshoot the header. The error would include the IP address of the node that discarded the value, and what the error was.

I can dig out the pattern we used, it solved a lot of practical problems with signing headers.

Regards,
Graham
—

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 18 Jul 2017 13:49:19 +0200

Hi Piotr,

> "X-Client-Certificate" header.

In fact, all depends on the amount of misconfiguration expected. If we
have to consider that a proxy suddenly becomes totally transparent, then
prepending a secret token before the actual value detects it.

Willy

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Jul 2017 14:24:35 +0200

[ taking tokbind off as it's not really for them ]

> On 17 Jul 2017, at 6:25 pm, Patrick McManus <[email-masked]> wrote:
> 
> b] if the proxy->server link is h2 then you can inject connection-specific information into the stream with an extension frame type (and the server can enforce this property). You don't need to negotiate it with SETTINGS (which is nice, because that's a round trip.) and these frames are hop to hop (proxies that don't understand a frame type MUST drop them).
> 
> Its not too hard to imagine a generalization of (b).. a CONNECTION-SPECIFIC-HEADERS or somesuch that has strict hop-to-hop semantics.

If I had a do-over, I'd still allow hop-by-hop headers in h2. HTTP/2 was designed to be a "drop-in" replacement for H1, and therefore there aren't any consistent (or often, existent) APIs for developers to use to affect things like extension frames or SETTINGS, and navigating the transition between H2 and H1 that's so common still in intermediaries.

Yes, it means touching the headers a bit more. *shrug*

Maybe we should have a SETTING_I_PROCESS_CONNECTION_HEADERS.

--
Mark Nottingham   https://www.mnot.net/