Subject: Server Push and Content Negotiation

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Aug 2016 14:26:31 +1000

The interaction of Content Negotiation and Server Push isn't really specified. Depending on how it's implemented, it could be quite tricky, because it seemingly requires the server to guess what the client would have sent, in order to negotiate upon it.

However, it becomes much simpler if we assume that the client SHOULD NOT check a `PUSH_PROMISE` request's headers to see whether or not it would have sent that request.

This means, for example, that if you `PUSH_PROMISE` the "wrong" `User-Agent`, `Accept-Encoding`, `User-Agent` or even `Cookie` header field, the client SHOULD still use the pushed response; all they're looking for is a matching request method and URL.

However, this does imply a few things:

* The pushed request and response MUST still "agree"; i.e., if you're using gzip encoding, `Accept-Encoding` and `Content-Encoding` should be pushed with appropriate values.
* The pushed response MUST have an appropriate `Vary` header field, if it is cacheable. This is so that the cache operates properly.

Additionally, the server needs to know what the base capabilities and preferences of the client are, to allow it to select the appropriate responses to push. To aid this, servers SHOULD create a PUSH_PROMISE's request by copying the values of the request header fields mentioned in the `Vary` response header field from the request that is identified by the `PUSH_PROMISE` frame's Stream ID.

So, for example, if the first request for a page had the following headers:

~~~
:method: GET
:scheme: https
:authority: www.example.com
:path: /
User-Agent: FooAgent/1.0
Accept-Encoding: gzip, br
Accept-Language: en, fr
Accept: text/html,s application/example, image/*
Cookie: abc=123
~~~

and the server wishes to push these response headers for `/style.css`:

~~~
:status: 200
Content-Type: text/css
Cache-Control: max-age=3600
Content-Encoding: gzip
Vary: Accept-Encoding
~~~

then it should use these headers for the `PUSH_PROMISE`:

~~~
:method: GET
:scheme: https
:authority: www.example.com
:path: /style.css
Accept-Encoding: gzip, br
~~~

This approach has its limits. For example, use of Client Hints might not be practical with server push (since in some circumstances, hints might change between the base page request and the request for what's been pushed).

I'd be tempted to go even further and say that PUSH_PROMISE headers SHOULD NOT contain `DNT`, `User-Agent`, `Cookie` or similar headers UNLESS they were specified in Vary.

What do people think -- is this worth specifying? How are implementations currently doing this? 

--
Mark Nottingham   https://www.mnot.net/

---

From: Patrick McManus <[email-masked]>
Date: Wed, 24 Aug 2016 10:36:35 -0400

[implementer hat]

> What do people think -- is this worth specifying? How are implementations

push is one way - negotiation doesn't make any sense and yet the
alternative to negotiation is some kind of sniffing. sniffing is awful and
is even harder here where you don't have a request to sniff - you'd have to
do some whole connection guess.

how would you push some feature that relies on conneg today - like webp or
brotli?

I think as a client realistically you have to wait for the response headers
and check them out to see if its an acceptable push. Whatever is in the
request headers isn't really enough for that - which is a bit of a change
from what I currently do.

fwiw, this was one of the reasons gzip was a mandatory-to-implement CE in
spdy and probably should have stayed. It meant you could reliably push it.
Doing so now is really just an assumption that gzip is a defacto standard.

Or this all might be another argument about the excessive complexity of
push - maybe that should be a document :)

---

From: Tom Bergan <[email-masked]>
Date: Wed, 24 Aug 2016 10:12:14 -0700

[sorry for the dup post, used the wrong email]

> push - maybe that should be a document :)

:-)

On Tue, Aug 23, 2016 at 9:26 PM, Mark Nottingham <[email-masked]> wrote:

> The interaction of Content Negotiation and Server Push isn't really
> specified. Depending on how it's implemented, it could be quite tricky,
> because it seemingly requires the server to guess what the client would
> have sent, in order to negotiate upon it.

This sounds right to me. Though, if the server and client are well-behaved,
the worst-case is wasted bandwidth, not an incorrect response:
- If the server pushes response with a non-empty Vary (or Key) field, it
should make a guess for all headers mentioned in Vary (or Key) and include
those guesses in the PUSH_PROMISE
- The client should is match pushed (request, response) pairs to client
requests using the Vary (or Key) field in the usual way

Additionally, the server needs to know what the base capabilities and
> preferences of the client are, to allow it to select the appropriate
> responses to push. To aid this, servers SHOULD create a PUSH_PROMISE's
> request by copying the values of the request header fields mentioned in the
> `Vary` response header field from the request that is identified by the
> `PUSH_PROMISE` frame's Stream ID.
>

For fields like Accept-Encoding and User-Agent, that's probably sufficient.
But in general, it's not quite that simple -- e.g., what if the response
has "Vary: Cookie", which can in theory change between requests? You make a
similar observation when talking about how Client Hints might change
between requests. Also see the discussion for "Rule #4" in this doc:
https://docs.google.com/document/d/1K0NykTXBbbbTlv60t5MyJvXjqKGsC
VNYHyLEXIxYMv0

I'd be tempted to go even further and say that PUSH_PROMISE headers SHOULD
> NOT contain `DNT`, `User-Agent`, `Cookie` or similar headers UNLESS they
> were specified in Vary.
>

I'd agree with that. I would say that PUSH_PROMISE only needs to contain
header fields that describe the cache key of the response (which includes
:method, :scheme, :path, :authority, and any header field referenced by
Vary or Key).

> What do people think -- is this worth specifying? How are implementations

Chrome is not doing anything (yet):
https://bugs.chromium.org/p/chromium/issues/detail?id=554220

I think most readers come to roughly the same conclusion when reading the
RFC, but there is enough uncertainty that I think this is worth specifying
in more detail.

---

From: Mike Bishop <[email-masked]>
Date: Wed, 24 Aug 2016 18:40:52 +0000

(From memory, without consulting devs)

On the server-side, there's an additional complication you're eliding over here -- the push may be cross-app, even on the same origin.  Say you have an app that dynamically renders things, but pushes other resources that are handled by a static file handler elsewhere in the system's request routing structure.  We essentially punt this choice to the app -- the app sends us a set of request headers.  We then simultaneously dispatch the PUSH_PROMISE on the wire and begin routing that request as if it had just been received from the client.  So whatever negotiation headers are specified by the app are those which will be seen by the recipient of the pushed request.  Requiring that we inspect the response to the push *before constructing* the request would require a total rearchitecting -- essentially requiring that the app hand us a fully-formed response and only then being able to declare the push.  That's going to delay processing of the initial request, which we really don't want blocked.

On the client side, there are two stages -- first, we compare the pushed request with the cache to see if we would have been able to serve that request out of cache.  If so, we reset the stream.  Otherwise, we receive and hold the content.  When the browser makes a request, we perform the same checks of whether the browser's request could be served by the held response.  So yes -- Vary is definitely necessary.

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Tuesday, August 23, 2016 9:27 PM
To: HTTP Working Group <[email-masked]>
Subject: Server Push and Content Negotiation

~~~
:method: GET
:scheme: https

~~~
:status: 200
Content-Type: text/css

~~~
:method: GET
:scheme: https