Subject: X-Forwarded-For and HTTP2

From: Fedor Indutny <[email-masked]>
Date: Wed, 08 Jul 2015 08:28:09 +0000

Hello again!

I have a question/suggestion with regards to HTTP2 spec, and TLS
terminator implementations.

When HTTP2 (or previously SPDY) is used under the TLS terminator,
there is a big question about how the address of the original
client should be sent to the back end.

One of the ways is to add `x-forwarded-for` header for every
request, but it requires parsing the underlying protocol (be it
HTTP2, SPDY, or HTTP/1.1), which is rather costly.

Another approach is to send (so called) proxyline. This is what
[HAProxy][0], and stud ([hitch][1]) are doing at the moment. In
a few words this means prefixing all decrypted data with a single
line specifying the client's IP address and port (and sometimes
more metadata, like TLS servername and other protocol-specific
things).

And the third approach that is to the best of my knowledge is
implemented only in [bud][2] TLS terminator - is to add special
SPDY frame, asking the SPDY server to automatically add
`x-forwarded-for` for every incoming connection.

What I was wondering if this frame could be standardized, or at
least if it might be possible to allocate the frame type for it.
So that people may use it for their own purposes, without being
afraid of the possible frame type conflicts.

What are your thoughts on this?

Thank you,
Fedor.

[0]: http://www.haproxy.org/ <http://www.haproxy.org/>
[1]: https://github.com/varnish/hitch <https://github.com/varnish/hitch>
[2]: https://github.com/indutny/bud <https://github.com/indutny/bud>

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 08 Jul 2015 22:51:28 +1200

On 8/07/2015 8:28 p.m., Fedor Indutny wrote:
> [HAProxy][0], and stud ([hitch][1]) are doing at the moment.

FWIW, Squid now also support this.

> In

XFF header is now deprecated. Forwarded (RFC ) was standardized in its
place.

> What are your thoughts on this?

Software properly supporting the X-Forwarder-For, Forwarded and/or Via
headers will send them appropriately without being told to. Any
difference from that is an explicit sysadmin policy on the senders side.

The security model behind XFF and Forwarded is that the message *sender*
uses it to indicate the level of responsibility it is willing to take on
for the message. Naming itself as the responsible party, or pointing at
some other source which may or may not be anonymized in turn.

The recipient of messages has no place imposing requirements on the
senders internal security policy. It can only determine its own level of
trust for the information provided, if and when its provided.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Jul 2015 19:51:13 +0900

Hi Fedor,

> On 8 Jul 2015, at 5:28 pm, Fedor Indutny <[email-masked]> wrote:
> 
> Hello again!
> 

Yup. BTW, there's also Forwarded <http://tools.ietf.org/html/rfc7239>, which is a bit more of a flexible format.

> more metadata, like TLS servername and other protocol-specific

*nod* I've been thinking we should have a discussion of that spec at some point.

> afraid of the possible frame type conflicts.

Hmm. In the past there's been some reluctance to duplicate or supplement what might be consider application-layer information in the framing layer; doing so means there will need to be new APIs to expose it, potential confusion about precedence, etc. etc. 

Having the semantic of the frame mean "this header is present in all requests on this connection" might be an interesting workaround for that (provided that both sides support it, of course), and might be useful for other things too.

Cheers,

> [0]: http://www.haproxy.org/
> [1]: https://github.com/varnish/hitch
> [2]: https://github.com/indutny/bud
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Mike Bishop <[email-masked]>
Date: Wed, 8 Jul 2015 17:59:23 +0000

But if the terminator is speaking HTTP/2 on both sides, is it muxing requests into fewer connections?  A extension that adds "present on all requests" semantics would work if every client connection has a corresponding server connection, but that's not necessarily the case.  Once the terminator uses fewer connections on the back-end, each request needs to be separately marked again.  On the other hand, I could see browsers using that for the things they always send and never vary over the lifetime of a connection....

The fact that HPACK was changed to be less stateful means that inserting a header can be a simple append to the HPACK frame -- you don't have to parse everything else in the frame, just tack your extra headers onto the end as non-indexed literals and adjust the frame length accordingly.  (You would have to handle the case where the incoming frame is a few bytes shy of the maximum; alternately, you could also do this by always dropping the END_HEADERS flag on the HEADERS frame and sticking the appended content in a CONTINUATION.)

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Wednesday, July 8, 2015 3:51 AM
To: Fedor Indutny
Cc: [email-masked]
Subject: Re: X-Forwarded-For and HTTP2

Hi Fedor,

> there is a big question about how the address of the original client 
> should be sent to the back end.
> 
> One of the ways is to add `x-forwarded-for` header for every request, 
> but it requires parsing the underlying protocol (be it HTTP2, SPDY, or 
> HTTP/1.1), which is rather costly.

> [HAProxy][0], and stud ([hitch][1]) are doing at the moment. In a few 
> words this means prefixing all decrypted data with a single line 
> specifying the client's IP address and port (and sometimes more 
> metadata, like TLS servername and other protocol-specific things).

> implemented only in [bud][2] TLS terminator - is to add special SPDY 
> frame, asking the SPDY server to automatically add `x-forwarded-for` 
> for every incoming connection.
> 
> What I was wondering if this frame could be standardized, or at least 
> if it might be possible to allocate the frame type for it.
> So that people may use it for their own purposes, without being afraid 
> of the possible frame type conflicts.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 8 Jul 2015 20:11:13 +0200

On Wed, Jul 08, 2015 at 05:59:23PM +0000, Mike Bishop wrote:
> But if the terminator is speaking HTTP/2 on both sides, is it muxing requests
> into fewer connections?  A extension that adds "present on all requests"
> semantics would work if every client connection has a corresponding server
> connection, but that's not necessarily the case.

I think there's no need for such a thing in the protocol, it can easily
be implementation-specific as we all do with various header additions on
the gateway such as x-ssl or whatever.

The simple idea of making people *think* there's a correlation between
both sides worries me about possible outcomes such as broken products
which consider that their mode is the correct one and who will expect
all agents around them to maintain the 1:1 relation between both sides
connections. That sounds a bit scary to me to be honnest.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 8 Jul 2015 20:17:32 +0200

> HTTP2, SPDY, or HTTP/1.1), which is rather costly.

Once you start working with a multiplexed protocol, processing
the headers is only a fraction of the total cost anyway. Dealing
with multiplexed frames still requires some advanced magics so I
wouldn't consider the addition of xff costly here.

> more metadata, like TLS servername and other protocol-specific

Yes but it's only for protocol gateways where there's a 1:1 relation
between front and back connections. It was designed mainly for SSL
offloading gateways. It doesn't make sense to use this mechanism on
multiplexed protocols.

> What are your thoughts on this?

In fact what could be done would be to send a frame as a preambule
for each stream indicating such information when relevant. That could
make sense if/when we start to extend the protocol to transport
something non-http with one connection per stream. But at this stage
we're only talking HTTP in the protocol frames so I don't see any
benefit in not using XFF (or the Forwarded header since it was
specified for this purpose).

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 8 Jul 2015 20:26:00 +0200

On Wed, Jul 08, 2015 at 11:20:33AM -0700, Fedor Indutny wrote:
> It is not adding the XFF to every HEADERS frame that is costly,
> it is parsing the protocol itself. Most of the TLS terminators at the
> moment skip this stage and just do one client connection = one
> backend connection.

Sure but they don't process the multiplexed frames either. That's
what I'm saying is expensive and complex. If you can process the
frames and multiplex multiple connections from one side to one or
multiple ones on the other side, adding HTTP processing on top of
them is relatively cheap.

Willy

---

From: Fedor Indutny <[email-masked]>
Date: Wed, 08 Jul 2015 18:21:24 +0000

With this in mind - adding parsing layer, and changing the frames
on the fly is very costly.

On Wed, Jul 8, 2015 at 11:17 AM, Willy Tarreau <[email-masked] <mailto:[email-masked]>> wrote:

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 08 Jul 2015 20:53:17 +0000

--------

>client should be sent to the back end.

I think we should actively discourage this practice going forward.

Lets turn the question around:

Why is it people think they need the network address in the first place ?

About the only thing X-F-F style data can tell you is *roughly*
where in the world the HTTP Proxy, NAT gateway or TOR exit-node
filtered the HTTP client onto the "open" network.

Very often it is badly misleading.

This can be for benign reasons, see for instance Opera story about
how "virtual norvegians" were the most common visitors to a US (fire
brigade ?) web-page.  (Didn't save the link, sorry.)

Or it can be for hostile reasons, such as people trying to bypass
restriction mechanisms on A/V content or sneak through ill-advised
ACLs.

(Strictly speaking people have only themselves to blame if they trust
X-F-F headers from untrustworthy proxies or clients, but that fine
print seems to be universally not be in HOWTOs and I fear the
practice is much more widespread than is good for anybody.)

On the plus side: The network address has *some* value as debugging
aid, but the frequency of that happening, makes it a very weak
argument for the overhead of managing/communicating the NA's.

If we decide that NA's are useful enough that they should be provided
down-stream, they need to be per-request, that follows directly
from the HTTP specs: proxies are allowed to mux.

If we provide NA's they almost certainly would have to be in
text-string format, because as far as I know, there is no universally
agreed binary representation format for IPv4/TCP and IPv6/TCP
addresses (whereas we have *far* too may textual represenations
to handle).

There *is* the on-wire format in the actual TCP/IP packet address
fields, but that is not supported by the sockaddr based APIs, so
that would be a pretty bad choice.

In other words: it's X-F-F or nothing, and nothing would be the
wiser choice for the long term.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 09 Jul 2015 10:35:37 +1200

> on the fly is very costly.

In order to generate or receive frames (even the extension one you
mention) one must have a *frame* parser. That is different from and far
simpler than a frame payload parser.

As Mike pointed out appending non-indexed headers to the HEADERS frame
payload is almost cost-free. All that is needed is the basic frame
parser/receiver and ability to correctly update the frame payload-size
on altered frames. This is no more difficult than high speed TCP or IP
layer processing by any router today. The appended header bit can even
be a pre-generated binary blob which is fixed data per inbound connection.

Actually consuming the results or destructivly modifying existing header
contents is an HTTP level operation and there is no avoiding the full
HTTP processing stack for those. Thats business for a full HTTP agent
anyhow so no additional costs their either.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 9 Jul 2015 14:43:17 +0800

> On 9 Jul 2015, at 1:59 am, Mike Bishop <[email-masked]> wrote:
> 
> But if the terminator is speaking HTTP/2 on both sides, is it muxing requests into fewer connections?

My assumption was that this wasn't the use case; Fedor, is this correct? 

It's pretty common for load balancers, etc. to spread connections between a number of back-end servers, but not do any protocol processing (or only minimal processing). This is so that the LB doesn't become a SPOF.

> A extension that adds "present on all requests" semantics would work if every client connection has a corresponding server connection, but that's not necessarily the case.

Certainly, it's not *always* the case. I think the question is whether these devices are still useful with HTTP/2…

>  Once the terminator uses fewer connections on the back-end, each request needs to be separately marked again.  On the other hand, I could see browsers using that for the things they always send and never vary over the lifetime of a connection....
>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 9 Jul 2015 10:27:11 +0200

On Thu, Jul 09, 2015 at 02:43:17PM +0800, Mark Nottingham wrote:
> It's pretty common for load balancers, etc. to spread connections between a
> number of back-end servers, but not do any protocol processing (or only
> minimal processing). This is so that the LB doesn't become a SPOF.

Yes but then if we have 1 connection forwarded from end to end, the proxy
protocol header at the beginning of the TCP payload which is now supported
by haproxy, squid, varnish, nginx, stunnel, stud and others is enough and
low overhead.

Here we're talking about devices which would perform all the hard work of
dealing with TLS deciphering/ciphering, parsing the protocol frames but not
have any knowledge of the HTTP headers inside. That doesn't make much sense
to me given that the most complicated part is dealing with the framing (and
that was the reason many people objected against the binary protocol that
could not be produced anymore from shell scripts).

Thus I tend to think that such devices doing only 90% of the job have very
little reasons of ever being created. And as Amos said, adding this header
on the fly even without parsing the HTTP frames is not complicated at all.

> Certainly, it's not *always* the case. I think the question is whether these
> devices are still useful with HTTP/2???

I think not. One of the goals seeked by HTTP/2 (and formerly by SPDY) was to
make it harder to implement half-assed devices which enforce their bogus
behaviour everywhere around them. I think at least this goal was reached and
we'll see correct protocol stacks because there's no benefit in doing half of
the job.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 9 Jul 2015 10:47:20 +0200

On Thu, Jul 09, 2015 at 01:35:49AM -0700, Fedor Indutny wrote:
> Willy,
> 
> I don't think that those TLS terminators actually do the half of the job.
> There is no way on earth they could support every kind of underlying
> protocol at once. They are just following unix philosophy and doing one
> thing right at the time, offloading the rest of the work to the other
> application.

I think you misunderstand me. I'm not speaking about TLS terminators,
they're doing the job right and have no problem with HTTP/2 at all.
Some people already do use haproxy as a TLS termination which injects
the proxy header at the beginning of the connection to serve HTTP/2
and/or SPDY.

What I'm saying is that HTTP/2 is a multiplexed protocol and that the
dumb offloader which has no knowledge about the protocol won't process
frames inside the protocol thus it will not multiplex connections and
instead will maintain the one-to-one relation between front and back
connections. In this case, a proxy protocol header at the beginning of
each backend connection does the job perfectly. And I'm all in favor
of TLS offloading since I tend to consider this method to scale better
than the other ones by moving the heavy processing out of the smart
components, thus reducing the number of smart ones and keeping the
ability to take more centralized decisions.

For a TLS offloader to break the one-to-one relation (and not being
compatible with a simple protocol header), it would have to be able
to multiplex connections and for this to parse all HTTP/2 frames in
the protocol. *That* is the difficult part which doesn't make sense
*alone* without the ability to inject an HTTP/2 header.

So in short, for an offloading gateway :

    TLS ----> plain TCP  : 1:1 connections, protocol header is enough
    TLS ----> HTTP       : XFF works fine
    TLS ----> multiplexed HTTP/2 frames over TCP : doesn't make sense

Hoping this helps,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 9 Jul 2015 11:02:45 +0200

On Thu, Jul 09, 2015 at 01:54:59AM -0700, Fedor Indutny wrote:
> Aaah, sorry for this then. I got you wrong.

No problem.

> I see your logic about the protocol header, but it does add
> unnecessary complexity to the backend side. From my
> experience, this usually means stripping down default HTTP(2)
> parser and working with the TCP socket at first, and only after
> the parsing the protocol header execute the default HTTP code.

Normally it's done just after the accept() so that it doesn't
interfere with the parsing. For example I have no idea how to
parse SSL, yet I can parse the protocol before SSL does its job.

> This statement implies that backends usually run on some kind
> of existing platform like node.js, or anything else. And those
> platforms have protocol implementations, so why not use just it
> instead of adding PROXYLINE parsing code to them?

I see your point but then I'd propose different semantics : instead
of fiddling with HTTP/2 frames, maybe you'd want to propose something
similar to what the proxy protocol header does, but as a TLS extension
so that it continues to repeat the information it got on the other side
of the connection. Note I'm really meaning a connection here. I really
find it wrong to say something like "all these frames belong to the
same connection", I'd rather say "the connection I'm sending to you
has such properties", because I have no idea what the frames transported
inside look like nor if someone upstream has put another semantics on
their relation.

Willy

---

From: Fedor Indutny <[email-masked]>
Date: Thu, 09 Jul 2015 08:55:49 +0000

On Thu, Jul 9, 2015 at 1:47 AM, Willy Tarreau <[email-masked] <mailto:[email-masked]>> wrote:

Hoping this helps,
Willy

---

From: Fedor Indutny <[email-masked]>
Date: Thu, 09 Jul 2015 09:00:09 +0000

Continuing my thoughts:

Adding such logic to backend also makes it impossible to
run and test the backend without the TLS terminator, because
it will always expect the PROXYLINE from it.

This adds complexity to the staging infrastructure as well.

On Thu, Jul 9, 2015 at 1:54 AM, Fedor Indutny <[email-masked] <mailto:[email-masked]>> wrote:
Aaah, sorry for this then. I got you wrong.

Hoping this helps,
Willy

---

From: Fedor Indutny <[email-masked]>
Date: Thu, 09 Jul 2015 08:36:39 +0000

Willy,

I don't think that those TLS terminators actually do the half of the job. There is no way on earth they could support every kind of underlying protocol at once. They are just following unix philosophy and doing one thing right at the time, offloading the rest of the work to the other application.

Strictly speaking, this is a sole purpose of having TLS terminator as a separate application. If one can afford to have it embedded - one does not need it at all, and may use `nginx` or something else that does HTTP(2) as well.

Prepending one frame using ALPN/NPN is very easy, parsing every possible protocol is something that many people do not want to have in a single-purpose tool. Especially, considering that there is no pros for them in re-framing incoming things.

Fedor.

On Thu, Jul 9, 2015 at 1:27 AM, Willy Tarreau <[email-masked] <mailto:[email-masked]>> wrote:
On Thu, Jul 09, 2015 at 02:43:17PM +0800, Mark Nottingham wrote:
>
> > On 9 Jul 2015, at 1:59 am, Mike Bishop <[email-masked] <mailto:[email-masked]>> wrote:
> >

Willy

---

From: Fedor Indutny <[email-masked]>
Date: Thu, 09 Jul 2015 07:08:16 +0000

Mark,

This is exactly the case I'm talking about. One client connection to one backend connection, no multiplexing in TLS terminator.

> From: Mark Nottingham [mailto:[email-masked] <mailto:[email-masked]>]
> Cc: [email-masked] <mailto:[email-masked]>
>> On 8 Jul 2015, at 5:28 pm, Fedor Indutny <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> Hello again!
>>
> Yup. BTW, there's also Forwarded <http://tools.ietf.org/html/rfc7239 <http://tools.ietf.org/html/rfc7239>>, which is a bit more of a flexible format.
>
>
> Mark Nottingham   https://www.mnot.net/ <https://www.mnot.net/>
>
>
>
>
>
>

--
Mark Nottingham   https://www.mnot.net/ <https://www.mnot.net/>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 9 Jul 2015 11:32:27 +0200

> it will always expect the PROXYLINE from it.

In fact not because usually implementations expect the proxy line
from well-known, trusted sources only (the offloading gateway). That
would be a security issue to require it from any source.

Willy