Subject: Re: H2 Implementation Debug State URI

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Aug 2016 08:01:52 +0000

--------
In message <[email-masked]>, Cory Benfield writes:

>Brad Fitzpatrick and I ended up prototyping a sample approach for 
>getting this information out of server implementations, based on issuing 
>a GET request to a specific well-known URI. This GET would return a JSON 
>document [...]

>https://http2.golang.org/.well-known/h2interop/state and 
>https://shootout.lukasa.co.uk/.well-known/h2interop/state.

>draft here: 
>https://tools.ietf.org/html/draft-benfield-http2-debug-state-00

As I said in the workshop, this has security considerations.

A lot of people tend to forget that client side proxies exist in H2.

They may not be popular, they may not be COTS (yet), but nanny-proxies
are not going to go away when they are mandated by law in an
increasing number of contexts.

For security reasons we cannot allow this well-known-URL to be
fetched through a shared client-side proxy.

Exporting the HPACK header table gives you the other clients
cookies etc, and even exporting the streams table would
probably leak more info than acceptable.

Short of a dedicated H2 frame to get this, I don't know of any
method which can enforce this being used only hop-to-hop, which
is the only way it can be used safely.

So at the very least this needs to be firmly turned off by default
and have a major warning posted on the enabling button.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Cory Benfield <[email-masked]>
Date: Fri, 5 Aug 2016 10:07:45 +0100

> On 5 Aug 2016, at 09:01, Poul-Henning Kamp <[email-masked]> wrote:
> 

You’re going to love my time machine: https://github.com/python-hyper/draft-http2-debug-state/commit/532c5315c8b8ab28993eff69e232dd261096359d <https://github.com/python-hyper/draft-http2-debug-state/commit/532c5315c8b8ab28993eff69e232dd261096359d>

I agree with you, and there’s some extensive conversation going on on the GitHub page about the way this interacts with proxies. In the short term, draft-01 will contain a strong admonition not to use the “hpack" key on uncontrolled networks.

The longer term question is about how to deal with the fact that a misbehaving H2 connection could be the result of the H2 state in any of the hops on the connection. For example, if you have a connection that goes: client -> forward proxy A -> reverse proxy B -> origin server, and all those hops are H2, you really need to see H2 state in *all* of them to understand what the problem might be. However, the well-known URI doesn’t allow for that kind of debugging without requiring the awkward step of needing proxies to rewrite the response body to include their own data.

A frame might be better, but then we lose some of the introspective power, as well as somewhat increasing the overhead of an implementation deploying this kind of support. Certainly it’s much harder to see the value of it when you can’t easily demo in a browser.

Cory

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Aug 2016 09:12:12 +0000

--------
In message <[email-masked]>, Cory Benfield writes:

>You're going to love my time machine: 

Cool.

Maybe it is one of those places where crypto could be useful:

Make the request contain the ID of a pre-shared-key.  If its on
the list you get the full monty, if not, you only get a safe summary.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 Aug 2016 22:30:04 +1000

> the list you get the full monty, if not, you only get a safe summary.

Or encrypt the response using TLS exporters.  If you are on the same
connection, then you will be able to extract the keys and decrypt.

---

From: Cory Benfield <[email-masked]>
Date: Fri, 5 Aug 2016 13:17:01 +0100

> On 5 Aug 2016, at 10:12, Poul-Henning Kamp <[email-masked]> wrote:
> 

So that kind of model certainly could work. At a certain point, though, one wonders if we’re severely over-engineering in order to solve a problem that’s not important enough to justify the effort.

Cory

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 5 Aug 2016 14:39:17 +0200

> Am 05.08.2016 um 14:17 schrieb Cory Benfield <[email-masked]>:
> 
> 

The intend here is to help debugging when client X does not work with Server Y. I fail to see how pre-shared keys could facilitate this, given that the client could be a browser and the bug reporter not a developer.

I'd rather limit the amount of information it exposes than its usefulness.

-Stefan

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 5 Aug 2016 10:00:32 -0600

On 08/05/2016 03:07 AM, Cory Benfield wrote:

> The longer term question is about how to deal with the fact that a
> misbehaving H2 connection could be the result of the H2 state in any of
> the hops on the connection. For example, if you have a connection that
> goes: client -> forward proxy A -> reverse proxy B -> origin server, and
> all those hops are H2, you really need to see H2 state in *all* of them
> to understand what the problem might be. However, the well-known URI
> doesn’t allow for that kind of debugging without requiring the awkward
> step of needing proxies to rewrite the response body to include their
> own data.

If you combine the well-known debugging URI with a Max-Forwards header,
then you may get pretty close to the desired "and here is my state too"
effect. It is not going to be "the state of each agent at the same
moment of time", but nothing is going to give you that precision anyway.

Alex.

---

From: Cory Benfield <[email-masked]>
Date: Sat, 6 Aug 2016 17:33:19 +0100

> On 5 Aug 2016, at 17:00, Alex Rousskov <[email-masked]> wrote:
> 

This was discussed, but was discarded in part because many proxies do not respect that header. Additionally, it’s only defined on OPTIONS requests.

Cory

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 8 Aug 2016 13:45:09 -0600

> This was discussed, but was discarded in part because many proxies do not respect that header. 

I do not understand that reasoning: Surely those proxies that add
support for the new feature (debugging URI) can be required to support
Max-Forwards, at least in the context of that feature. Is it really
better to add Max-Forwards-2 or require response body adaptations?

> Additionally, it’s only defined on OPTIONS requests.

It is not prohibited for any other method. TRACE and OPTIONS are the two
methods where the protocol requires certain actions, but that does not
mean new features (like a debugging URI) cannot use it.

Alex.

---

From: Cory Benfield <[email-masked]>
Date: Tue, 9 Aug 2016 10:50:30 +0100

> On 8 Aug 2016, at 20:45, Alex Rousskov <[email-masked]> wrote:
> 

The problem is not those proxies that add support for the feature: the problem is those that *don’t*. For those that don’t add support, they blindly forward the request on, running the risk of information leakage and invalid/incorrect responses.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 9 Aug 2016 09:59:07 -0600

On 08/09/2016 03:50 AM, Cory Benfield wrote:

> For those that don’t add support, they blindly forward the request
> on, running the risk of information leakage and invalid/incorrect
> responses.

I do not see how Max-Forwards unsupporting proxies can have a
significant negative effect on the problem you are trying to solve.
Obviously, you will not get their debugging state, but that is going to
happen no matter what -- there will always be proxies that do not
support your new feature. Why would Max-Forwards unsupporting proxies
produce invalid responses? They will just forward the response the next
hop gives them, and that response would be correct or incorrect
regardless of what they do.

AFAICT, Max-Forwards essentially lets you interrogate supporting hops.
Unless you change the protocol to require debugging support, nothing you
can do will let you interrogate unsupporting hops.

This overall problem feels very similar to traceroute -- it does not
always work and not all hops support ICMP TTLs, but it works well enough
in many cases to remain useful.

Alex.

---

From: Cory Benfield <[email-masked]>
Date: Wed, 10 Aug 2016 10:50:46 +0100

> On 9 Aug 2016, at 16:59, Alex Rousskov <[email-masked]> wrote:
> 

I think this summary is true if HPACK information is removed from the debugging output. So using Max-Forwards is conditional on not dumping HPACK. That may be an acceptable compromise.

Cory

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 10 Aug 2016 10:16:36 -0600

On 08/10/2016 03:50 AM, Cory Benfield wrote:

> using Max-Forwards is conditional on not dumping
> HPACK. That may be an acceptable compromise.

What is wrong with receiving HPACK state from hop #(N+1) that has
allowed its HPACK state sharing, even if you actually asked to see HPACK
state of the unsupporting hop #N? I understand that #(N+1) may not be
very useful in some triage cases, but I am sure it will be useful in others.

The Via headers and/or equivalent will tell the client which hop
responded to the Max-Forwards:N debugging request (and, in most cases,
whether any unsupporting hops were skipped).

Alex.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 10 Aug 2016 17:39:57 +0000

The problem is that upstream HPACK context might well contain headers from requests that were made by clients other than you.  Leakage between connections is a bad thing.  You might be able to turn this on for in-depth debugging, but you'd *never* want that bit enabled on a live server.

-----Original Message-----
From: Alex Rousskov [mailto:[email-masked]] 
Sent: Wednesday, August 10, 2016 9:17 AM
To: Cory Benfield <[email-masked]>
Cc: HTTP Working Group <[email-masked]>
Subject: Re: H2 Implementation Debug State URI

On 08/10/2016 03:50 AM, Cory Benfield wrote:

> using Max-Forwards is conditional on not dumping HPACK. That may be an 
> acceptable compromise.

What is wrong with receiving HPACK state from hop #(N+1) that has allowed its HPACK state sharing, even if you actually asked to see HPACK state of the unsupporting hop #N? I understand that #(N+1) may not be very useful in some triage cases, but I am sure it will be useful in others.

The Via headers and/or equivalent will tell the client which hop responded to the Max-Forwards:N debugging request (and, in most cases, whether any unsupporting hops were skipped).

Alex.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 10 Aug 2016 17:48:08 +0000

--------
In message <[email-masked]>, Mike Bishop writes:

If hop #N is a shared proxy, the HPACK state at hop #N+1 will disclose 
info about other clients traffic.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 10 Aug 2016 14:11:55 -0600

On 08/10/2016 11:39 AM, Mike Bishop wrote:

> The problem is that upstream HPACK context might well contain headers
> from requests that were made by clients other than you. 

It is not a problem under the stated precondition -- "hop that has
allowed its HPACK state sharing". Somebody has made a decision that such
sharing is OK in their specific environment, for that specific request, etc.

And it is easy to make HPACK sharing conditional so that the interface
can remain more open to less dangerous queries:

  # I know you are going to deny my request if I [implicitly] ask for
  # HPACK, so please note that I actually do not want to see it:
  uri = .well-known/h2/state?with-hpack=0

> Leakage
> between connections is a bad thing.  You might be able to turn this
> on for in-depth debugging, but you'd *never* want that bit enabled on
> a live server.

Yes, of course. I doubt any sane admin would enable this debugging by
default in a unprotected environment anyway, with or without HPACK. This
is a triage aid. Code like that is rarely of high quality and must be
protected.

These valid security concerns are not tied to the Max-Forwards mechanism
IMHO. This is like telling a carpenter to avoid the really heavy hammer
because somebody will get hurt if that hammer is misused.

Alex.

---

From: Cory Benfield <[email-masked]>
Date: Thu, 11 Aug 2016 09:48:43 +0100

> On 10 Aug 2016, at 21:11, Alex Rousskov <[email-masked]> wrote:
> 
> 

So draft-01 now explicitly says that the HPACK state dictionary should almost always be omitted. This may allow us to reconsider the Max-Forwards notion.

Cory