Subject: [#153] PUSH_PROMISE headers

From: Martin Thomson <[email-masked]>
Date: Fri, 28 Jun 2013 17:21:04 -0700

https://github.com/http2/http2-spec/issues/153

The current text describes PUSH_PROMISE as having a few request
headers, plus some response headers, but it's quite vague.

I think that if this is going to be properly workable across a wide
range of uses with lots of different headers, PUSH_PROMISE needs to
include two sets of headers: the ones that it overrides from the
associated request (:path being foremost of those) and the ones that
it provides as a "preview" of the response (e.g., ETag might allow
caches to determine if they were interested in the rest of the
response).

---

From: James M Snell <[email-masked]>
Date: Fri, 28 Jun 2013 21:44:17 -0700

Let's take a step back and consider what a pushed stream is...

A pushed stream is essentially an "Implied GET". This means that a
server is going to assume that the client was going to send a GET for
the pushed resource. This also means that the server has to make some
assumptions about the make up of that implied GET.

Now, consider how HTTP caching works. When a cache receives a request
for a resource, how does it determine whether or not it has a
representation of the resource already available? Does it look at the
request headers or the response headers? Obviously, it looks at the
request headers. It uses the response headers when populating the
cache.

So, if we look at the pushed resource sent by the server, what we need
is for A) the server to first let us know about the implied GET
request.. which means pushing down a set of request headers then B)
the server to send the actual resource, which means pushing down the
response headers.

Already in our design for pushed resources, we have the server sending
a PUSH_PROMISE frame that contains a header block, followed by a
HEADERS frame that also contains a headers block. It stands to reason
that the PUSH_PROMISE frame would contain the set of request headers
that the server is assuming for the implied GET. These are delivered
to the client, which uses those to determine whether or not a cached
representation of the resource is already available (just as any cache
would do using the request headers). The server would then send it's
response headers in a HEADERS frame, just as it would any response to
any other kind of GET.

Two examples to show how this naturally fits... First, let's look at a
normal GET request sent by the client to the server...

Client                 Server
------                 ------
  |                        |
  | ---------------------> |
  |   HEADERS              |
  |     GET                |
  |     /images/f.jpg      |
  |     If-Match: etag1    |
  |     Accept: image/jpeg |
  |                        |
  | <--------------------- |
  |   HEADERS              |
  |     200                |
  |     Content-Type:      |
  |       image/jpeg       |
  |     Content-Length:    |
  |       123              |
  |                        |
  | <--------------------- |
  |   DATA....DATA....     |
  |                        |

Now consider the same resource being pushed by the server using PUSH_PROMISE...

Client                 Server
------                 ------
  |                        |
  | <--------------------- |
  |   PUSH_PROMISE         |
  |     GET                |
  |     /images/f.jpg      |
  |     If-Match: etag1    |
  |     Accept: image/jpeg |
  |                        |
  | <--------------------- |
  |   HEADERS              |
  |     200                |
  |     Content-Type:      |
  |       image/jpeg       |
  |     Content-Length:    |
  |       123              |
  |                        |
  | <--------------------- |
  |   DATA....DATA....     |
  |                        |

Note that the only difference here is the direction and type of the
first frame. Everything else is identical. The PUSH_PROMISE contains
everything the client needs to determine whether or not it already has
the resource in it's local cache (request URI, etag, content-type...).

There's no need to get any more complicated than this. We already
require two distinct header blocks for every request. We already send
two distinct header blocks for each pushed stream. We already indicate
that a pushed stream is an implied GET. To make it work, we simply
state that the PUSH_PROMISE contains the Request headers that the
server has assumed for the implied GET request, while the HEADERS
frame sent later contains the Response headers. If the request headers
in the PUSH_PROMISE end up not being adequate enough to properly
determine if the resource is already cached, then we treat it as just
another cache miss.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 28 Jun 2013 21:55:27 -0700

Depending on how the request might have been been constructed, response
headers may be necessary to identify the resource in the cache, as compared
to the resource specified in the HTML (I'm thinking about vary: stuff).

-=R

---

From: James M Snell <[email-masked]>
Date: Fri, 28 Jun 2013 22:25:44 -0700

Have an example handy?

Here's an example that shows that response headers in the PUSH_PROMISE
would not be necessary... Let's say I send a PUSH_PROMISE with the
following bits of info...

PUSH_PROMISE
  :path = /images/f.jpg
  :method = GET
  :host = example.org
  :scheme = http
  accept = image/jpeg
  if-match: "my-etag1"
  cache-control: max-age=1000

These headers are giving me everything I would need to determine if
there is a matching resource in my local cache. I have the method, I
have the etag, I have the cache-control parameters, accept... There's
no need for response headers at this point.

Later, once I start accepting the frames for the pushed content, I
would get something like...

HEADERS
  :status = 200
  content-type: image/jpeg
  content-length: 123
  etag: "my-etag1"
  vary: accept
  cache-control: public

On the off chance that the PUSH_PROMISE doesn't give me what I need,
the follow on HEADERS frame will give me the rest.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 28 Jun 2013 22:34:39 -0700

Any content negotiation would be an appropriate example. :)

You don't want to have to wait for the HEADERS frame to indicate to the
client which resource it might already have (it should have the opportunity
to RST_STREAM if it has it in cache, for instance).
-=R

---

From: Roberto Peon <[email-masked]>
Date: Fri, 28 Jun 2013 22:36:10 -0700

... because by then you've opened up a stream., and you're back into
problematic territorry.
PUSH_PROMISE exists because we need to indicate to the browser all of the
information it needs to make a determination about whether or not it wants
the stream (and to short circuit the inlining/push mechanism when it
already has what it needs!)
-=R

---

From: James M Snell <[email-masked]>
Date: Fri, 28 Jun 2013 22:44:51 -0700

I just gave an example that uses content-negotiation so that argument
doesn't really work. The server is crafting the request headers for an
implied GET, so it gets to construct those request headers to be as
specific as it wants them to be, without much need for actual
content-negotiation.

For example, suppose the server receives a request for an HTML page like this...

GET /index.html HTTP/1.1
Accept: text/html, image/jpeg, image/gif

The server decides that it wants to push jpeg files to the client... it sends

PUSH_PROMISE
  :path = /images/f.jpg
  :method = GET
  :host: example.org
  :if-match: "my-etag1"
  accept: image/jpeg

That's pretty darn unambiguous. I can easily check to see if I have a
representation of "/images/f.jpg" with etag "my-etag1" and
content-type "image/jpeg" in my local cache, without the need to have
*any* response headers in the PUSH_PROMISE.

Perhaps you have another, more specific example in mind?

On Fri, Jun 28, 2013 at 10:36 PM, Roberto Peon <[email-masked]> wrote:
> the stream (and to short circuit the inlining/push mechanism when it already
> has what it needs!)
> -=R
>
>

---

From: Roberto Peon <[email-masked]>
Date: Fri, 28 Jun 2013 22:53:57 -0700

Actually, I think you're right, since the resource is identified by the URL
name (unfortunate in some ways). I was thinking that you'd want to indicate
both the canonical name for the served resource and the name used to
identify the request, but that'd be a bigger change.
-=R

---

From: David Morris <[email-masked]>
Date: Fri, 28 Jun 2013 23:04:39 -0700 (PDT)

Without PUSH_PROMISE in the picture, the client would only have the
request headers to decide if something in the cache applies to
a request.

If we expect the PP response header set to be meaningful, then the server
needs to do much of the processing before generating the PP headers. With
the ordering requirement, this gets really difficult and possibly
introduces a new form of head of line blocking.

The fly in the ointment that I see is that the client may not actually
create the request header set the way the server guessed it would. For
example, the client would likely include an eTag if available and the
server has no knowledge of what eTag should be in the PP headers.

I continue to suspect that PUSH_PROMISE is either not going to be
heavily used on the general web or it is going to waste significant
bandwidth and server capacity. A PUSH_OFFER approach might be slightly
slower (1/2 RTT for the whole set?) but the resource consumption could
be similar to the case without either form of pre-fetch. But I'm not
advocating a change at this time. Once there are implementations we
need to evaluate alternatives carefully.

Dave Morris

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 29 Jun 2013 20:00:21 +1200

> Perhaps you have another, more specific example in mind?

The above example is a good one.

The cache will store the pushed response under:
   hash(URL)+hash(Vary:"image/jpeg")

When looking up future HIT the cache for this client on explicit-GET it 
will look up
   hash(URL)+hash(Vary:"text/html, image/jpeg, image/gif")

I assume you can spot the difference.

Middleware which looks up its cache on future PUSH_PROMISE will also 
find the does the hash(URL)+hash(Vary:"image/jpeg") version and probably 
RST_STREAM for teh redundant data. Other clients will however still MISS 
on that object, and may even MISS on the second version above if they 
for example drop text/html on their explicit-GET for images.

... PUSH_PROMISE will only work on middleware which supports the 
proposed Key header. It will not work easily with Vary as you seem to think.

Amos

---

From: Yoav Nir <[email-masked]>
Date: Sat, 29 Jun 2013 09:14:39 +0000

On Jun 29, 2013, at 9:04 AM, David Morris <[email-masked]> wrote:

> bandwidth and server capacity. 

There's been very little talk (at list on-list) about how a server decides what resources to push. One way is to push down all the resources referenced from an HTML document. That would interact badly with caching, both in clients and in proxies.

Another possible approach is statistical. Begin with the server not pushing any resources, and then collect statistics. What percentage of requests for index.html were followed by requests for logo.gif? If that percentage is high enough, you add logo.gif to the resources pushed following index.html.  Once a resource is pushed, you keep another statistic of what percentages of pushes were rejected (with RST_STREAM). If that's above a certain (low) threshold, you stop pushing it. 

Done this way, pushes will be mostly successful, and I don't see why they would waste significant bandwidth for the server.

Yoav

---

From: James M Snell <[email-masked]>
Date: Sat, 29 Jun 2013 07:27:04 -0700

On Jun 29, 2013 1:03 AM, "Amos Jeffries" <[email-masked]> wrote:
[ snip ]
will look up
> I assume you can spot the difference.

I didn't miss that. Pushed resources and content negotiation used together
have rather, um, "interesting" complications. However, those don't change
the request headers in the push promise and response headers in the HEADERS
question which is what I was addressing.

- James

---

From: Jeff Pinner <[email-masked]>
Date: Sat, 29 Jun 2013 11:54:04 -0700

+1

At least for the first implementation draft I would suggest we heed James'
suggestion and restrict the headers inside the pushed header block to those
that would have been sent in the request.

If we find later that we *really really* need to add response headers to
the pushed header block then we can reevaluate. Or perhaps after some
implementation experience we figure out that with some thought into
selection of the pushed headers there is not a compelling enough reason to
add the extra complexity.

---

From: Michael Sweet <[email-masked]>
Date: Sun, 30 Jun 2013 21:52:05 -0400

I haven't had a proper amount of time to review the current draft, but I just wanted to throw something out...

> There's been very little talk (at list on-list) about how a server decides what resources to push. One way is to push down all the resources referenced from an HTML document. That would interact ba…

Currently a client will send GET requests with If-Modified-Since for every resource it needs; perhaps if there was a way for the client to pass along a list of already-fetched resources with the fetch date, the server could push any new resources back to the client?

(that might defeat any performance advantage, or maybe not - something to try?)

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: David Morris <[email-masked]>
Date: Sun, 30 Jun 2013 20:28:32 -0700 (PDT)

> (that might defeat any performance advantage, or maybe not - something to try?)

I think the basic premis is that PUSH_PROMISE is generated by the server
before the client has had a chance to receive and parse the original
request and identify what it has and might need.

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 01 Jul 2013 15:51:36 +1200

> request and identify what it has and might need.

There does seem to be an interesting case for RST_STREAM+If-Not-Modified 
here to optimize bandwidth usage.

Amos