Subject: Re: ETags vs Variants, was: Revising RFC2616 - what's happening

From: Lisa Dusseault <[email-masked]>
Date: Sat, 4 Nov 2006 11:14:16 -0800

Sounds like a trigger to at least consider obsoleting, dropping or  
deprecating the feature.

Lisa

On Oct 22, 2006, at 5:22 AM, Henrik Nordstrom wrote:

>
> What we see today is that neither component really cares. Most servers
> forgets to send Vary headers when they should, instead using no- 
> cache to
> solve the problem. And most caches sees Vary too complex and reads it
> the same as no-store, or in some user-agent cases reads Vary  
> wrongly as
> "no-cache" (need to revalidate on every request) and additionally
> getting the validation completely wrong.

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 04 Nov 2006 21:04:38 +0100

> Sounds like a trigger to at least consider obsoleting, dropping or

It's not much needed to make Vary work, and it's becoming more and more
important if caching of HTTP should at all be considered.

The main problem is that the current specifications allows for multiple
interpretations so nobody trusts it to anything meaningful.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Sun, 05 Nov 2006 20:04:41 +0100

> Sounds like a trigger to at least consider obsoleting, dropping or 

Dropping it would be an incompatible change, breaking lots of existing 
servers (which do send it, and if recipients ignored them caches would 
break).

I don't think there's an alternative to clarifying it.

Best regards, Julian

---

From: Lisa Dusseault <[email-masked]>
Date: Sun, 5 Nov 2006 14:00:32 -0800

On Nov 5, 2006, at 11:04 AM, Julian Reschke wrote:

> Dropping it would be an incompatible change, breaking lots of  
> existing servers (which do send it, and if recipients ignored them  
> caches would break).
>

There are certainly more creative alternatives.  For example, if the  
spec said roughly that
	- Vary headers are deprecated in the sense that servers implementing  
this RFC SHOULD discontinue use of the header, and instead simply use  
"no-cache"
	- Clients MUST continue to handle the Vary header.  A client MAY  
treat the Vary header as defined in RFC2616, or it MAY simply treat  
the header as an indication not to cache the response.
	(plus some more text and ABNF for completeness)

While it can be a little painful to live with a deprecation period,  
it ends at some point, and I can't see how the deprecation period  
would be worse than the situation Henrik described as the current  
reality.

Lisa

---

From: "Robert Sayre" <[email-masked]>
Date: Sun, 5 Nov 2006 17:13:54 -0500

> would be worse than the situation Henrik described as the current

You're deprecating a feature people actually use, so the result is
much worse than the current reality. I understand why the header is
difficult for Henrik, because Squid is designed to be placed between
parties it knows nothing about. Caches that have more out-of-band
information, such as AOL's or Django's, have an easier time with it.

-- 

Robert Sayre

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 05 Nov 2006 23:47:17 +0100

> this RFC SHOULD discontinue use of the header, and instead simply use  

Which isn't far from deprecating caching of HTTP in general, not only
the Vary header. The amount of Vary:ing content served today is quite
large and rapidly growing thanks to the deployment of content-encoding. 

So I would not see this as an option.

Vary is kind of usable today, problem with the specs is mainly
interoperability issues resulting in sub-optimal performance when both
sides doesn't agree entirely on how to read the specs. The result is
always correct as long as they conform to their own interpretation of
the specs..

The other problem is that many forget that they SHOULD send Vary at all,
but that's a different story.. changing it into another SHOULD isn't
likely to help that situation..

The biggest mess wrt Vary is the widespread confusion about the meaning
of ETag and entities / entity variants which is tightly coupled with
Vary.

Regards
Henrik

---

From: Lisa Dusseault <[email-masked]>
Date: Sun, 5 Nov 2006 14:54:37 -0800

I wouldn't want to deprecate a feature that was useful (although the  
fact that it is used isn't necessarily proof it's useful if it's only  
servers that use it).  I appreciate discussion on whether it's too  
useful to deprecate.

Does anybody have counter-examples of clients handling Vary properly  
and sites benefiting from that?

thanks,
Lisa

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 06 Nov 2006 00:00:27 +0100

> and sites benefiting from that?

Squid has cached Vary responses for quite some time (4 years now) with
good results, both to clients (when used as internet proxy) and servers
(when used as surrogate / reverse proxy).

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 06 Nov 2006 00:03:33 +0100

> information, such as AOL's or Django's, have an easier time with it.

I don't have a problem with understanding Vary. Apart from the small
detail what change in the Vary seen for an URI is actually supposed to
mean, which I choose to read as if the rules for that URI has changed.
If server authors reads it differently there will be a performance
penalty at worst due to stuff which in theory could have been served
from our cache no longer is and the request gets forwarded to the origin
server, and a bit of trashing in our store.

I'd like to see the specs clarified on what a new Vary value for an URI
means, but I can live with the current specifications just as server
authors have to live with what we implement if they want to benefit from
it.

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sun, 5 Nov 2006 17:34:14 -0800

It would make me feel better if proposals to change HTTP/1.1 were based
on hard facts and not random conjecture.

On Oct 22, 2006, at 5:22 AM, Henrik Nordstrom wrote:
> sön 2006-10-22 klockan 04:35 +0100 skrev Jamie Lokier:
>
>>> I would say that if the value for "Vary" changes between to HTTP
>>> requests, the sever implementation/configuration has somehow  
>>> changed,
>>> and a proxy should invalidate all cached entries for that URI.
>>
>> No, no.
>
> Yes yes yes ;-)

No, and you can read the specification to learn why.  Vary is a  
statement
by the origin server about how intermediate caches should behave in  
regards
to *this* response.  It is impossible for the cache to know anything  
about
how the resource works or how responses to similar requests in the  
future
might actually vary -- it is only responsible for obeying the origin  
server's
wishes for *this* response, and *this* response remains valid for as  
long
as it remains fresh.

It is not the cache's responsibility to ensure that the server correctly
implements Vary.  It is not the cache's responsibility to exhaustively
check every possible combination of client request headers.  The Vary
field states what the origin server cares about for *this* response.
What it cares about for this response may be entirely different from
what it cares about for the next response -- some responses are more
specific than others, and some responses are more generic than others.

>> The natural implementation is for the server to note each time a
>> request header is examined to compute the response, and to emit Vary
>> with those headers.
>
> True, but this creates quite a bit of a nightmare at the cache  
> level. So
> with this requirement Vary: will still become equal to "no-store" in
> most implementations, perhaps with hardcoded special cases for the  
> most
> common uses or more likely caches trying to outguess the servers and
> implementing their own content negotiation schemes. This simply  
> because
> general caching of Vary entities then becomes too complicated to even
> care trying to index the variants in the cache.

Nonsense.  It is a trivial linear algorithm of store and compare that
has been implemented correctly in every implementation of HTTP/1.1 that
has actually attempted to implement it.  Even the lousy Microsoft DLL
that turns off caching when Vary is present is a "correct  
implementation",
even though it is absurdly inefficient.  Others have done better.

>> If you specify that a cache must purge all variants when receiving a
>> Vary header which is different from previously received Vary, then
>> servers will realistically have to send "Vary: Accept-Encoding,
>> User-Agent" even in the case that the response _doesn't_ depend on
>> User-Agent.
>
> Which is fine to me. Especially if the server supports ETag and
> If-None-Match on larger responses.

Flushing the cache is a correctness-preserving action for an HTTP
intermediary, regardless of the contents of Vary.  In other words,
you will be compliant with HTTP even if your caching sucks.  If you
implement Vary as it is specified in RFC 2616, your implementation will
be both correct and cache when appropriate.  I don't see a problem here.

>> However, when the server's use of request headers is less tightly
>> coupled, it's _much_ harder to do that.
>
> True.
>
> So question then becomes multifold:
>
> 1) Is caching of Vary responses worth the effort to get it working
> proper?

Yes.

> 2) If caching of Vary is desireable, what component of the network
> should have to deal with the complexity involved?

There is no complexity involved.  An origin server makes its own choices
about what is important to Vary upon, and can set the header field
accordingly using any number of simple configuration mechanisms.
A cache simply follows those instructions.

> getting the validation completely wrong.

That is conjecture.  Most caches implement Vary correctly or haven't
been updated to HTTP/1.1 yet.  The Microsoft client DLL implements
Vary in the least efficient way, but no sane protocol designer will
base an RFC on one of Microsoft's implementations when the rest of
the world has no problem dealing with that feature.  HTTP/1.1 offers
"no-cache" as an option that may be used for any number of reasons,
so its presence or absence has nothing whatsoever to do with Vary.

There is no issue here.  Vary works in many implementations and there
has never been a single report of interoperability problems between
clients and servers that have implemented Vary as specified.  It is an
integral part of HTTP/1.x caching that cannot be deprecated.

Cheers,

Roy T. Fielding                            <http://roy.gbiv.com/>
Chief Scientist, Day Software              <http://www.day.com/>

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 06 Nov 2006 10:43:57 +0100

> integral part of HTTP/1.x caching that cannot be deprecated.

While I agree with you in principle I am still a bit curious. Can you
name an implementation where Vary really works?

MSIE -> no-store

Firefox -> Stores at most one variant per URL. Broken if no ETag in the
response. In such case it's processed as kind of a no-cache and Sends
If-Modified-Since based on the old response, and the real error being
taking a 304 response to mean that the old response is valid for the new
request headers as well. ETag + If-None-Match prevents the bug from
surfacing if supported by the server.

Squid -> kind of attempts to obey Vary, but with limitations

Delegate -> no-store

Regards
Henrik

---

From: Stefan Eissing <[email-masked]>
Date: Mon, 6 Nov 2006 09:49:12 +0100

Am 05.11.2006 um 23:54 schrieb Lisa Dusseault:

>
> I wouldn't want to deprecate a feature that was useful (although  
> the fact that it is used isn't necessarily proof it's useful if  
> it's only servers that use it).  I appreciate discussion on whether  
> it's too useful to deprecate.

I would appreciate if the one willing to change the spec argues for  
the changes and makes a compelling case. Having to argue to keep  
features "in" does not compute.

In case of doubt, deployed protocols are innocent.

//Stefan

---

From: Justin Chapweske <[email-masked]>
Date: Mon, 6 Nov 2006 16:03:36 -0600

I think you guys are too quick to state that Squid and Apache define  
the entire HTTP world.  There are a lot of load balancers, switches,  
content delivery networks, etc. out there that make use of these  
features and continue to improve their support over time.