Subject: New Draft: draft-ohanlon-transport-info-header

From: "Piers O'Hanlon" <[email-masked]>
Date: Thu, 21 Nov 2019 08:32:38 +0000

Hi all,

We like would to present a new draft - Abstract:

  The Transport-Info header provides a mechanism to inform a client of the
last-mile server's view of the network transport related information such
as current delivery rate and round-trip time.  This information has a wide
range of uses such as client monitoring and diagnostics, or allowing a
client to adapt to current network conditions.

The draft is available in a few places - on Github:
https://bbc.github.io/draft-ohanlon-transport-info-header/draft-ohanlon-transport-info-header.html

Github Issues/code:
https://github.com/bbc/draft-ohanlon-transport-info-header

Datatracker:
https://datatracker.ietf.org/doc/draft-ohanlon-transport-info-header/
Tools: https://tools.ietf.org/html/draft-ohanlon-transport-info-header

Thanks for fitting in James’ presentation of our draft at today’s meeting.
I wanted follow up on some of the questions:

- “How does it compare to ServerTiming Header”
It’s pretty different as the ServerTiming header provides info server
processing times whereas Transport-Info provides info on transport
connection metrics.

- “Privacy concerns”
We have tried to address some of these in the security section of the draft
- let us know if there’s specific concerns or things that aren’t tackled.

- “It only provides information per response which isn’t very often”
We mention in the draft that with H2+ one can send and arbitrary number of
requests (using OPTIONS/HEAD) to obtain more measurements responses per
unit time.

We would welcome any more feedback by email and/or Github issues

Thanks,

Piers O'Hanlon

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 22 Nov 2019 18:50:53 +0800

Hi Piers,

> requests (using OPTIONS/HEAD) to obtain more measurements responses per

I have an observation but not sure it belongs in a document.
Implementations such as nginx have a soft max number of HTTP/2 requests
before closing the connection (default 1000 last I checked). If you're
trying to sample the transport info to frequently you may end ip blowing it
up. This seems unfortunate because the client use case is designed around
making smarter transport related decisions.

Cheers
Lucas

---

From: "Piers O'Hanlon" <[email-masked]>
Date: Fri, 22 Nov 2019 12:56:13 +0000

Hi Lucas,

Thanks for bring that up - as you say Nginx's default for
http2_max_requests is 1000, although it can be changed (and appears to
be done so for RPC applications). I'm not sure how many other server
implementations do this?

Firstly, we can detect this through of the use of the dstport
parameter - as a new TCP connection would use a different port.
Although it could potentially lead to temporary loss of information
for a time as discussed below.

Secondly, the affect of this would be only apparent each time the
request count is exceeded - so say every 1000 requests - when a switch
over would occur. When a switch over does occur then it depends on the
comparative duration of the data responses of interest, versus how
often one wants to perform parallel HEAD/OPTION requests for
Transport-Info. So for the case where where the frequency of parallel
requests is about the same then I think it shouldn't matter much as
with most server systems these days the congestion control parameters
are cached in the kernel so a subsequent connection to the same
destination would be preloaded with the cached metrics so the
Transport-Info header would contain these. In the case where there's a
series of long running responses then it might be an issue as after
switch over point there would also be two parallel TCP connections to
the same point but they would exist separately for a longer period so
potentially the metrics obtained via subsequent HEAD/OPTIONS could be
different as the cwnd can be reduced for low volume flows, though
these would generally not be used since the dstport would not match so
there could be a loss of information in this case.

Cheers

Piers

On Fri, 22 Nov 2019 at 10:51, Lucas Pardue <[email-masked]> wrote:
>> We mention in the draft that with H2+ one can send and arbitrary number of requests (using OPTIONS/HEAD) to obtain more measurements responses per unit time.
>
>
> I have an observation but not sure it belongs in a document. Implementations such as nginx have a soft max number of HTTP/2 requests before closing the connection (default 1000 last I checked). If you're trying to sample the transport info to frequently you may end ip blowing it up. This seems unfortunate because the client use case is designed around making smarter transport related decisions.
>
> Cheers
> Lucas
>
>>

---

From: Patrick McManus <[email-masked]>
Date: Sat, 23 Nov 2019 05:41:13 +0800

To the extent that this leaks information across origins to js its probably
a problem too.

On Fri, Nov 22, 2019 at 8:58 PM Piers O'Hanlon <[email-masked]> wrote:

> >> We mention in the draft that with H2+ one can send and arbitrary number
> of requests (using OPTIONS/HEAD) to obtain more measurements responses per
> unit time.
> >
> >

---

From: "Piers O'Hanlon" <[email-masked]>
Date: Mon, 25 Nov 2019 12:18:48 +0000

Hi Patrick,

Thanks for your feedback.

I'm assuming most cross-origin issues should be dealt with by CORS but
were you concerned with that suggestion that the header could be
contained in an OPTIONS response? I can see that since such a response
is not subject to CORS - such as in a CORS pre-flight request - then
we could drop that as an approach and just keep to using HEAD requests
as a mechanism to obtain the Transport-Info.

Let me know what if that doesn't address your issue or if you have any
other concerns.

Piers

On Fri, 22 Nov 2019 at 21:41, Patrick McManus <[email-masked]> wrote:
>
> To the extent that this leaks information across origins to js its probably a problem too.
>

---

From: Patrick McManus <[email-masked]>
Date: Mon, 25 Nov 2019 10:22:43 -0500

A value like CWND is connection based - it is a side channel for all of the
transactions on the connection that is fed back through an http response
header on one particular transaction. These transactions don't necessarily
share the same origin (or the same auth properties).

On Mon, Nov 25, 2019 at 7:19 AM Piers O'Hanlon <[email-masked]> wrote:

> >> >> We mention in the draft that with H2+ one can send and arbitrary
> number of requests (using OPTIONS/HEAD) to obtain more measurements
> responses per unit time.
> >> >
> >> >

---

From: "Piers O'Hanlon" <[email-masked]>
Date: Mon, 25 Nov 2019 16:50:14 +0000

> On 25 Nov 2019, at 15:22, Patrick McManus <[email-masked]> wrote:
> 
> A value like CWND is connection based
Yes the server sender cwnd for that particular connection between the “origin”* and the client.

> - it is a side channel for all of the transactions on the connection that is fed back through an http response header on one particular transaction.
The only transactions that traverse that connection are between the client and the “origin”.

> These transactions don't necessarily share the same origin (or the same auth properties).
> 
If they don’t share the same origin they’re transported on a different connection so would have a different CWND(s).

The CWND sent in a Transport-Info header would only be related to a single transport connection between one “origin" and one client over which flows transactions related to those entities. The transactions from the same origin to other clients would be over a different transport connection so the cwnd would be different.  Transactions that don’t share the same origin would be a different connection and contain yet another cwnd. 

*when I say “origin” I mean the last hop edge node of whatever CDN serves the origin.

Apologies if I’m missing your point but from what have you have described the side channel isn’t clear to me?

Piers

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 25 Nov 2019 17:02:50 +0000

IIUC Patrick correctly, I think one threat model relates to connection
coalescing. Imagine two resources: a.example.com/index.html includes
b.example.com/foo.js and the properties of these resources (i.e. authority
and certs) satisfy the requirements for HTTP/2 connection reuse as
described in https://tools.ietf.org/html/rfc7540#section-9.1.1.

a.example.com and b.example.com are in different administrative domains but
requests for b.example.com/foo.js are able to obtain information about the
state of the connection including the effect of requests to a.example.com.
This could be used for fingerprinting or some other form of attack from one
domain to the other.

Lucas

---

From: Patrick McManus <[email-masked]>
Date: Mon, 25 Nov 2019 12:51:11 -0500

> The only transactions that traverse that connection are between the client
no - connections are not limited to a single origin as far back as the h1
proxy use case and its definitely not true in h2/h3 at all. And even if it
were, that's not enough protection. HTTP exchanges are explicitly stateless
- mechanisms that ignore that to create some kind of ambient state always
cause trouble (e.g. ntlm).

> If they don’t share the same origin they’re transported on a different
> connection so would have a different CWND(s).
>

no.

>
> The CWND sent in a Transport-Info header would only be related to a single
> transport connection between one “origin" and one client over which flows
> transactions related to those entities.

I want to put up a flag here and say that even same-origin exchanges need
to be isolated from each other. Two different sets of login cookies on the
same origin don't share the same security context.

> The transactions from the same origin to other clients would be over a
> different transport connection so the cwnd would be different.
> Transactions that don’t share the same origin would be a different
> connection and contain yet another cwnd.
>
>
no.

> *when I say “origin” I mean the last hop edge node of whatever CDN serves
origin is the combination of scheme host and port. A CDN edge serves many
origins.

> the side channel isn’t clear to me?
obviously the HTTP protocol is aware of this sharing - indeed it initiates
it! but the ramifications of it are generally opaque to the semantics of
HTTP exchanges (roughly expressed by headers and messages). So perhaps what
you want is better carried in the protocol (e.g. as a frame).. if you are
proposing exposing it to content it needs to be scrutinized for the same
reason that it is useful. This can be subtle - see CRIME and BREACH for
example.

hth

---

From: "Piers O'Hanlon" <[email-masked]>
Date: Tue, 26 Nov 2019 17:10:37 +0000

Hi Lucas,

Thanks for the explanation - that helped. I think to keep the discussion simple I’ll reply with my points to Patrick.

Piers

> On 25 Nov 2019, at 17:02, Lucas Pardue <[email-masked]> wrote:
> 
> IIUC Patrick correctly, I think one threat model relates to connection coalescing. Imagine two resources: a.example.com/index.html includes b.example.com/foo.js and the properties of these resources (i.e. authority and certs) satisfy the requirements for HTTP/2 connection reuse as described in https://tools.ietf.org/html/rfc7540#section-9.1.1.
> 
> a.example.com and b.example.com are in different administrative domains but requests for b.example.com/foo.js are able to obtain information about the state of the connection including the effect of requests to a.example.com. This could be used for fingerprinting or some other form of attack from one domain to the other.
>

---

From: "Piers O'Hanlon" <[email-masked]>
Date: Tue, 26 Nov 2019 17:12:20 +0000

> On 25 Nov 2019, at 17:51, Patrick McManus <[email-masked]> wrote:
> 
> On Mon, Nov 25, 2019 at 11:50 AM Piers O'Hanlon <[email-masked]> wrote:
> 
> no - connections are not limited to a single origin as far back as the h1 proxy use case and its definitely not true in h2/h3 at all. And even if it were, that's not enough protection. HTTP exchanges are explicitly stateless - mechanisms that ignore that to create some kind of ambient state always cause trouble (e.g. ntlm).

Thanks for the clarifications. I did mention in the draft that there may be issues with the use of client proxies, but as Lucas and you have pointed out the aspect of connection reuse by H2/3 means that this is more of a general issue.

Although this kind of attack would be limited to domains that share the same origin where such coalescing could occur - but I agree that this may not be insignificant. However, the H2 RFC [rfc7540#section-9.1.1], does also warn that with connection reuse "it is possible for clients to send confidential information to servers that might not be the intended target for the request, even though the server is otherwise authoritative” so there are already issues with cross-origin protection in this scenario. So one could say that there may be less confidential transactions sharing such a link so the threat might be considered as lower.

Another approach could be for an origin utilising the header could be set up to disallow cross-origin connection reuse by returning a 421 response to thwart abuse.

> 
> I want to put up a flag here and say that even same-origin exchanges need to be isolated from each other. Two different sets of login cookies on the same origin don't share the same security context.
>  
This a good point - I guess the browser needs to handle this correctly when sharing a connection between, for example, two tabs. 

> The transactions from the same origin to other clients would be over a different transport connection so the cwnd would be different.  Transactions that don’t share the same origin would be a different connection and contain yet another cwnd. 
> 
> 
> no.
>  
> origin is the combination of scheme host and port. A CDN edge serves many origins.
> 
Sure - I guess I didn’t put it very well - I was trying to say that the Transport-Info header would be only be transferred between that CDN edge and the client - it’s a hop-by-hop header - for the last last hop. I guess the CDN edge could employ connection reuse where appropriate.

> obviously the HTTP protocol is aware of this sharing - indeed it initiates it! but the ramifications of it are generally opaque to the semantics of HTTP exchanges (roughly expressed by headers and messages). So perhaps what you want is better carried in the protocol (e.g. as a frame).. if you are proposing exposing it to content it needs to be scrutinized for the same reason that it is useful. This can be subtle - see CRIME and BREACH for example.
> 
Yes I had considered that frame based transport might be better but what is also required is to extract the transport/flow information for that stream - but if one could do that then it would potentially be ok to transport it in a header as the issue is that there is access to shared state.

It could be argued that one could achieve similar kind of fingerprinting by the client measuring the response times of requests. This would provide similar information as the cwnd isn’t static so it’s not going to be exactly the same between responses. Although it is a little different as with the transport-info one has the throughput for the whole connection.

Also for H2/3 since this is an issue about transport rate the fact padding is used on each connection means that there is always going to be some uncertainty introduced by padding in such attack. Such an attack where one flow to try to learn about another flow it would need to calculate it’s own portion of the flow so it could subtract it from the total flow rate provided by the header but’s going to be limited by not knowing about the padding.

Another potential mitigation technique would be to add some noise to the measurements which would make it harder for collusion to occur with coordinated attacks. The choice of the level of noise could be a function of the number of flows the server knows are sharing the connection, or maybe just some randomness so that two simultaneous requests don’t obtain the same result - although in practice they’re not going to obtain the same result as the connection parameters are constantly changing, although such random noise could mitigate attempts to compare trends.

Piers