Subject: Re: i37: Vary and non-existant headers

From: Mark Nottingham <[email-masked]>
Date: Wed, 6 May 2009 15:59:19 +1000

Looking back at this discussion, I think we have agreement to drop  
'present' (as per Roy), and add the "When a resource's  
representations..." sentence.

I don't see consensus to add language about canonicalising request  
headers (beyond the process described in p6 2.6).

On 29/07/2008, at 1:18 AM, Mark Nottingham wrote:

>
> http://tools.ietf.org/wg/httpbis/trac/ticket/37
>
> Roy sort-of proposed deleting "present" in <http://lists.w3.org/Archives/Public/ietf-http-wg/2006OctDec/0090.html 
> >
>
> In addition, I'd propose:
>
> * adding: "When a resource's representations vary, an origin server  
> SHOULD include the Vary header even when the selecting header(s) are  
> not present; otherwise, caches will use that representation  
> inappropriately."
>
> * adding: "Caches MAY canonicalise request headers before comparing  
> them for purposes of determining whether they match during variant  
> selection."
>
>
> --
> Mark Nottingham     http://www.mnot.net/
>
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 06 May 2009 08:43:22 +0200

> representations..." sentence.

The current text in draft 06 is different from the one that was 
discussed in earlier emails; so I'll have to re-read the thread and 
compare what's in now with what's in RFC 2616...

> headers (beyond the process described in p6 2.6).

That's issue <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/147>, 
right? Do we expect consensus to emerge in the future (that is, do we 
leave the issue open)?

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 6 May 2009 16:55:44 +1000

Good point. Current text:
> When a cache receives a request that can be satisfied by a stored  
> response that includes a Vary header field (Section 3.5), it MUST  
> NOT use that response unless all of the selecting request-headers in  
> the presented request match the corresponding stored request-headers  
> from the original request.

> That's issue <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/ 
> 147>, right? Do we expect consensus to emerge in the future (that  
> is, do we leave the issue open)?

My reading of *this* thread was that doing so would violate the  
prohibition against changing headers by intermediaries, but now that  
you remind me, IIRC there was also a subsequent view expressed that as  
long as the component the cache resided in didn't change them on the  
wire (i.e., it was only for purposes of caching), this was OK.

However, the wording would need to be very careful, and we'd risk  
having to define canonicalisation for all possible headers, something  
that I'm loathe to do... at some point, it's better to just design a  
new, better function for doing this (as has been discussed before).

Do you have proposed text?

---

From: Yves Lafon <[email-masked]>
Date: Wed, 6 May 2009 03:33:01 -0400 (EDT)

On Wed, 6 May 2009, Mark Nottingham wrote:

>>> 'present' (as per Roy), and add the "When a resource's representations..." 
>>> sentence.
>> 
>> The current text in draft 06 is different from the one that was discussed 
>> in earlier emails; so I'll have to re-read the thread and compare what's in 
>> now with what's in RFC 2616...
>
>> When a cache receives a request that can be satisfied by a stored response 
>> that includes a Vary header field (Section 3.5), it MUST NOT use that 
>> response unless all of the selecting request-headers in the presented 
>> request match the corresponding stored request-headers from the original 
>> request.
>
>
>
>>> I don't see consensus to add language about canonicalising request headers 
>>> (beyond the process described in p6 2.6).
>> 
>> right? Do we expect consensus to emerge in the future (that is, do we leave 
>> the issue open)?
>
> My reading of *this* thread was that doing so would violate the prohibition 
> against changing headers by intermediaries, but now that you remind me, IIRC 
> there was also a subsequent view expressed that as long as the component the 
> cache resided in didn't change them on the wire (i.e., it was only for 
> purposes of caching), this was OK.

Yes, when used internally by a cache, it is valid to canonicalize, 
however, ticket 147 raises one question,
is Accept-Encoding: gzip equivalent to Accept-Encoding: gzip, identity and 
Accept-Encoding: identity, gzip

So while not talking about canonicalization of all headers, we may need to 
have a story for such defaulted values, at least an advice for 
implementors.

> However, the wording would need to be very careful, and we'd risk having to 
> define canonicalisation for all possible headers, something that I'm loathe 
> to do... at some point, it's better to just design a new, better function for 
> doing this (as has been discussed before).
>

How about:
"Implementations might define header matching as the equivalence of their 
internal representations".

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Julian Reschke <[email-masked]>
Date: Wed, 06 May 2009 10:08:01 +0200

Mark Nottingham wrote:
>> response that includes a Vary header field (Section 3.5), it MUST NOT 
>> use that response unless all of the selecting request-headers in the 
>> presented request match the corresponding stored request-headers from 
>> the original request.

"...selecting request-headers in the presented request..."

What's essential is that this includes also headers *not present* in the 
request, if the cached response varied on them. For instance, if the 
cache as a response for a previous request with "Accept-Encoding: gzip", 
it MUST NOT use that for a subsequent request that lacks an 
"Accept-Encoding" header.

How about:

"When a cache receives a request that can be satisfied by a stored 
response that includes a Vary header field (Section 3.5), it MUST NOT 
use that response unless all of the selecting request-headers stored 
from the original request match those in the presented request."

?

That at least makes it a bit clearer that what's relevant is the set in 
the stored request. It also fixes the problem that the set of "selecting 
request header" is only defined once you have the origin server's 
response for that request.

Also, I just noticed that the definition of "selecting headers" now is 
down in the definition of "Vary". Maybe we should undo this, add a 
forward reference, or move it into the Terminology section...

> ...

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 6 May 2009 10:10:09 -0500

Yves Lafon wrote:
> How about:
> "Implementations might define header matching as the equivalence of
> their internal representations".

That looks like "Header matching is implementation-defined."

It makes sense that transparent proxies must use character-by-character
comparison (no canonicalization). Non-transparent proxies are already free
to rewrite the "Accept-Encoding" header of requests, so they are free to do
whatever canonicalization they want (RFC 2616 section 13.5.2), AFAICT. That
is, a proxy cache that treats "Accept-Encoding: inflate, identity" the same
as "Accept-Encoding: inflate" is the same as a non-transparent proxy cache
that rewrites "Accept-Encoding: inflate, identity" to "Accept-Encoding:
inflate". Such a cache can't claim to be transparent (because it isn't).

- Brian

---

From: Yves Lafon <[email-masked]>
Date: Wed, 6 May 2009 15:48:24 -0400 (EDT)

> That looks like "Header matching is implementation-defined."

Exactly.

> inflate". Such a cache can't claim to be transparent (because it isn't).

Well, most transparent proxies I ran into did not support "TRACE *", so it 
is hard to find a truly transparent proxy anyway.

         ~~Yves

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 7 May 2009 09:47:33 +1000

On 06/05/2009, at 6:08 PM, Julian Reschke wrote:

>>>> Looking back at this discussion, I think we have agreement to  
>>>> drop 'present' (as per Roy), and add the "When a resource's  
>>> discussed in earlier emails; so I'll have to re-read the thread  
>>> and compare what's in now with what's in RFC 2616...
>>> NOT use that response unless all of the selecting request-headers  
>>> in the presented request match the corresponding stored request- 
>>> headers from the original request.
>
> What's essential is that this includes also headers *not present* in  
> the request, if the cached response varied on them. For instance, if  
> the cache as a response for a previous request with "Accept- 
> Encoding: gzip", it MUST NOT use that for a subsequent request that  
> lacks an "Accept-Encoding" header.
>
> How about:
>
> NOT use that response unless all of the selecting request-headers  
> stored from the original request match those in the presented  
> request."

Possibly.

How about:

When a cache receives a request that can be satisfied by a stored  
response that includes a Vary header field (Section 3.5), it MUST NOT  
use that response unless all of the selecting request-headers  
nominated by the Vary header match in both the original request  
associated with the stored response, and the presented request.

and adding this to the subsequent paragraph (defining what it means to  
match):

If a header field is absent from a a request, it can only match  
another request if it is also absent there.

> Also, I just noticed that the definition of "selecting headers" now  
> is down in the definition of "Vary". Maybe we should undo this, add  
> a forward reference, or move it into the Terminology section...

Yes, that needs some massaging.

---

From: David Morris <[email-masked]>
Date: Wed, 6 May 2009 17:38:11 -0700 (PDT)

On Thu, 7 May 2009, Mark Nottingham wrote:

> If a header field is absent from a a request, it can only match another 
> request if it is also absent there.
>

Any reason we can't make an empty value == to absent for this purpose?

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 6 May 2009 18:32:22 -0700

On May 6, 2009, at 5:38 PM, David Morris wrote:
>> and adding this to the subsequent paragraph (defining what it  
>> means to match):
>>

Because they don't mean the same thing in HTTP.

....Roy

---

From: David Morris <[email-masked]>
Date: Wed, 6 May 2009 20:35:10 -0700 (PDT)

For my dense brain, could you provide an explicit example of a header 
which would have a different meaning?

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 6 May 2009 20:41:09 -0700

On May 6, 2009, at 8:35 PM, David Morris wrote:
> For my dense brain, could you provide an explicit example of a  
> header which would have a different meaning?

Accept and User-Agent, off the top of my head.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 07 May 2009 11:00:50 +0200

Mark Nottingham wrote:
> use that response unless all of the selecting request-headers nominated 
> by the Vary header match in both the original request associated with 
> the stored response, and the presented request.

That still makes it sound as if "selecting request headers" is 
well-defined in absence of a response from origin server. So, how about:

"When a cache receives a request that can be satisfied by a stored
response that includes a Vary header field (Section 3.5), it MUST NOT
use that response unless all of the selecting request-headers nominated
by the *stored* Vary header match in both the original request 
associated with the stored response, and the presented request."

> request if it is also absent there.

Yes (if we can agree on allowing proxies to canonicalize, this will of 
course need tuning).

BR, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 7 May 2009 16:23:18 +0100

> is hard to find a truly transparent proxy anyway.

It's not hard to detect them though, unless they are very good.  Just
send something which isn't HTTP syntax.  Most transparent proxies will
mangle it.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 7 May 2009 16:51:17 +0100

> from the original request match those in the presented request."

No, because it is wrong.  Example:

   Stored original request does not have Accept-Encoding.
   Stored response has "Vary: Accept-Encoding".

   New request has "Accept-Encoding: gzip"
        => Cache MUST NOT use the stored response.

   New request has "Accept-Encoding: deflate"
        => Cache MUST NOT use the stored response.

   New request has "Accept-Encoding:"
        => Cache MUST NOT use the stored response.
 
   New request does not have Accept-Encoding.
        => Cache MAY use the stored response.

The original request's _lack_ of Accept-Encoding header is being
varied on.

The proposed text doesn't make this any clearer than the existing
text, it just moves the ambiguity from missing headers in the new
request to missing headers in the original request.

For each header mentioned in Vary, the value or absence of that header
must be the same for the original and new requests.

Here is proposed replacement:

   "When a cache receives a request that can be satisfied by a stored
   response that includes a Vary header field (Section 3.5), it MUST
   NOT use that response unless, for every header identified by the
   stored Vary, the header's value or absence in the original request
   matches its value or absence in the presented request."

(When the stored Vary is "*", _any_ more or fewer headers (except
transport headers?) in the presented request must prevent a match).

-- Jamie

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 7 May 2009 12:31:17 -0500

> associated with the stored response, and the presented request."

This is wrong, but not for the reason Jamie stated.

Consider this:

1. Client requests "GET /foo"
2. Cache forwards the request to the server "GET /foo" with
   an If-None-Match: "A", "B", "C"
3. Server returns "304 Not Modified" with ETag "A".

According to the statement above, the cache could only return the cached
response with ETag "A" if the Vary'd headers match; otherwise, it couldn't
return any response at all. However, the cache must be able to return the
response with ETag "A" regardless of whether the Vary'd headers match.

If all of the Vary'd headers match, then the cache can satisfy the request
without forwarding it on (if it is fresh enough), but otherwise the cache
must forward the request on; which cached response (if any) the cache can
return will be decided by the upstream servers.

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 07 May 2009 21:43:15 +0200

> response with ETag "A" regardless of whether the Vary'd headers match.

Why?

What you describe sounds like a bug. If the selecting headers that 
resulted in the response with entity tag "A" had a "Accept-Encoding: 
gzip", then the response can't be used for a request without 
"Accept-Encoding", even if it may be fresh.

> return will be decided by the upstream servers.

I do not disagree with this part, but it doesn't seem to be related to 
the stuff above...

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 7 May 2009 14:08:23 -0700

On May 7, 2009, at 12:43 PM, Julian Reschke wrote:

>>> NOT
>>> nominated
> resulted in the response with entity tag "A" had a "Accept- 
> Encoding: gzip", then the response can't be used for a request  
> without "Accept-Encoding", even if it may be fresh.

Because the origin server just told the cache to deliver "A" in (3).
The origin server is authoritative.

We've had this discussion before.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 07 May 2009 23:29:25 +0200

> We've had this discussion before.

Right, sorry.

So the text is only needs to consider the case where the origin server 
gives its ok.

""When a cache receives a request that can be satisfied by a stored
response that includes a Vary header field (Section 3.5), it MUST NOT
use that response unless all of the selecting request-headers nominated
by the *stored* Vary header match in both the original request 
associated with the stored response, and the presented request, *or 
after validating with the origin server*".

?

BR, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 08 May 2009 10:57:21 +1200

In the case of say Accept-Encoding, what if a server sends a vary tag 
with Accept-Encoding in it, yet there is no Content-Encoding field (e.g. 
no encoding, but the server selected on Accept-Encoding)?

It would then seem pointless for a cache to refuse to send an unencoded 
response (that would otherwise be valid) simply because a subsequent 
request didn't state it would be willing to accept an encoding.

Or is that second-guessing the server and prohibited?

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 7 May 2009 19:34:43 -0500

> after validating with the origin server*".

Section 2.2 already says this (almost), much more clearly. How about this:

  - Use of server-driven content negotiation (Section 4.1 of [Part3])
  - alters the conditions under which a cache can use the response for
  - subsequent requests. [ed: completely untrue]
  - 
  - When a cache receives a request that can be satisfied by a stored
  - response that includes a Vary header field (Section 3.5), it MUST NOT
  - use that response unless all of the selecting request-headers in the
  - presented request match the corresponding stored request-headers from
  - the original request. [ed: redundant with 2.2]

    The selecting request-headers from two requests are defined to match
    if and only if the selecting request-headers in the first request can
    be transformed to the selecting request-headers in the second request
    by adding or removing linear white space [[anchor11&anchor12]].
  + Any selecting request-headers absent from the first request must 
  + be absent from the second request. [ed: Mark's suggestion,
  + reworded to match the terminology from the first sentence.]

The first sentence in Section 2.2 needs to be changed:

  - For a presented request, a cache MUST NOT return a stored
  - response, unless:
  + A cache MUST NOT return a stored response without
  + validating it, unless:

The last bullet point in Section 2.2 needs to be changed:

  -  o the stored response is either:
  -
  -     *  fresh (see Section 2.3), or
  -
  -     *  allowed to be served stale (see Section 2.3.3), or
  -
  -     *  successfully validated (see Section 2.4).
  +  o the stored response is fresh (see Section 2.3) or allowed
  +    to be served stale (see Section 2.3.3).

Regards,
Brian

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 7 May 2009 19:48:21 -0500

> no encoding, but the server selected on Accept-Encoding)?

That is the normal case. If the server decided to send a non-encoded version
due to the absence of an Accept-Encoding, then it has to include
Accept-Encoding in the Vary. 

> Or is that second-guessing the server and prohibited?

Yes, the cache must return what the server told it to return:

   If instead the cache receives a full response (i.e., one with a
   response body), it is used to satisfy the request and replace the
   stored response. [[anchor8: Should there be a requirement here?]]

and:

   If the server responds with 304 (Not
   Modified) and includes an entity tag or Content-Location that
   indicates the entity to be used, that cached response MUST be used
   to satisfy the presented request [...]

- Brian

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 08 May 2009 13:06:44 +1200

> Accept-Encoding in the Vary. 
that's not what I meant.  I meant maybe the original request had an 
Accept-Encoding header, the server sent back a response with Vary 
Accept-Encoding, but no Content-Encoding header, since it wasn't 
encoded.  this was then cached.

Then another request came in without an Accept-Encoding header.

What I'm trying to get at is: is there any case to be made for 
validation between the Accept-* headers and the matching Content-* 
headers of the original request and response, and whether this should 
have any bearing on matching for subsequent requests.

Regards

Adrien

---

From: Julian Reschke <[email-masked]>
Date: Fri, 08 May 2009 10:38:29 +0200

> Accept-Encoding in the Vary. 

Yes. As a matter of fact, that's the reason why we have this issue open 
-- many people do not get that from the current spec text.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 8 May 2009 06:40:58 -0500

> have any bearing on matching for subsequent requests.

A transparent cache cannot do that, but a non-transparent proxy can, unless
the no-transform attribute is used. My understanding is that whenever the
specification refers to a cache, it is referring to a transparent one. Maybe
that is something that needs to be more explicit.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 8 May 2009 09:31:18 -0500

Julian Reschke wrote:
> What you describe sounds like a bug.

I was thinking about what you said and I noticed this sentence in 2.6:

   If no stored response matches, the cache MAY forward the presented
   request to the origin server in a conditional request, and SHOULD
   include all ETags stored with potentially suitable responses in an
   If-None-Match request header.  

What is a "potentially suitable response"? Doesn't that just mean any or all
stored responses for that request-URI? If no stored response matches, the
cache MUST forward the presented request to the origin server (what else
could it do?), and that request SHOULD be conditional (see 2.4), right?. If
"potentially suitable responses" doesn't have any real meaning then what is
the point of this statement?

- Brian

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 8 May 2009 16:07:02 +0100

> return will be decided by the upstream servers.

That's a very interesting point.

Here's another:

  4. Another client sends a request whose headers match the stored request
     from step 1 (_not_ the originally stored request), over the same Vary.

Can the cache return the response with Etag "A" without consulting the
server?

I'm thinking the only useful and sensible answer is yes, it fits the
obvious intentions of the caching model, and that is probably what
good cache implementations do.

But that means the cache is allowed to store the request from step 1
associated with the response that the cache sent, with Etag "A".  (Or
equivalently, as an internal implementation detail, it's allowed to
create an additional "cache key" for the same response with Etag "A").

And I'm not sure that's permitted by the RFC text.

Assuming that is permitted, should the newly stored response with Etag
"A", associated with the response's Vary and the Vary-selected
requesting headers for the second request (step 1) - should it use the
original response's Vary, or the Vary that comes with the "304 Not
Modified" response?

I'm thinking the Vary that comes with the "304 Not Modified" if there
is one, otherwise the Vary that came with the originally stored
response, is the safest conservative combination.

Always using the Vary that came with the originally stored response is
wrong, because it is not always sufficient to encode the dependencies
that were used to make the decision that Etag "A" is suitable for the
second request at step 1.

-- Jamie