Subject: Re: draft-ietf-httpbis-p6-cache-06

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Jun 2009 14:00:23 +1000

On 28/05/2009, at 1:31 AM, Brian Smith wrote:

> Thomas Broyer wrote:
>> Brian Smith <[email-masked]> wrote:
>>> It's not invalid, but it doesn't mean anything. POST responses are
>>> never cacheable; caches must write POST requests through to the
>>> server, and a cache can't use a POST response to respond to a
>>> GET/HEAD request.

2616 says:
> Responses to this method are not cacheable, unless the response  
> includes appropriate Cache-Control or Expires header fields.

>> Even when the POST response has a Content-Location?
>
> There's nothing in RFC2616 or the latest HTTPbis drafts that says a  
> cached POST response can be returned in response to a GET/HEAD  
> request.

No, but there is accommodation for it; see p6 2.1 and 2.2, in  
particular the TODO note in the latter.

> Applying Postel's rule, a cache shouldn't return a cached POST  
> response to a GET/HEAD request, and servers shouldn't include Cache- 
> Control/Expires headers in POST responses. That should be explicit  
> in the specification.

There has been considerable discussion on this, and your conclusion  
wasn't suggested AFAIK, nor was it the direction we've chosen to move  
in. See:
    http://trac.tools.ietf.org/wg/httpbis/trac/ticket/139
    http://www.w3.org/mid/08345F97-7D4D-40AD-98E2- 
[email-masked] (entire thread)

--
Mark Nottingham     http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Jun 2009 15:46:33 +1000

Hi Adrien,

Thanks for the feedback. I think others have covered most of this, but  
see below for a few responses...

On 27/05/2009, at 3:17 AM, Adrien de Croy wrote:
> Wrt POST (or any method).  If the response to a POST is marked  
> explicitly by the origin server as cachable, why should a subsequent  
> POST invalidate that contrary to other Cache-control directives?   
> Surely this should only apply if the original method was not POST?

Because POST changes state on the server; it's a useful pattern to  
have POST (or other responses) cached, but invalidated upon a visible  
update.

> S 2.6 Caching Negotiated responses
> ----------------------------------
> Should I then be referring to Section 4.1 of [part3] to resolve the  
> issues around content negotiation?  If so, maybe a mention in S 2.2  
> would be useful.

Well, 2.2 refers to 2.6 for a full treatment, which then refers to 4.1  
of p3.

> I also don't understand in para 5 the sentence "If the server  
> responds with 304 (Not Modified) and includes an entity tag or  
> Content-Location that indicates the entity to be used"  How can  
> Content-Location be used to select an entity?  Do you match on  
> previously returned Content-Location headers for requests for the  
> same URI?  Is that what the final para is getting at?  Maybe the  
> wording could be a bit clearer.

Interesting question; not sure if this is implemented anywhere.  
Recording this as issue #167 <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/167 
 >.

> S 3.2 Cache-Control
> -------------------
> first sentence states that directives MUST be obeyed.   This doesn't  
> fit with a strategy of ignoring unhandled directives if you get a  
> mixture of request and response directives in a message (which is  
> still allowed in the ABNF).

As discussed, this could use some tweaking. #168 (editorial); <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/168 

> Some cache control directives are confusingly similar, especially  
> for response directives.

Yes. There's not a lot we can do about this except improve the doc  
(which we've tried to do in this revision; compare to 2616, which IMO  
was even more confusing). #168 should incrementally improve this (in a  
small way).

> I can't see anywhere easily where it says what to do in some cases  
> either.
> 1. private directive with headers.
>
> It states that these headers then are all that is private, and these  
> must not be stored. However it does not state what to do when you  
> get a subsequent request for that URI that would match.  Does this  
> mean that the stored response must be revalidated with the server,  
> or is it appropriate to send the stored response (if still fresh) to  
> the client without those headers?
>
> 2. no-cache.
>
> It's not clear to me what the point of revalidating an entire entry  
> vs just revalidating header fields.  Especially when you consider  
> that a conditional request will revalidate both or either.  So I  
> can't see any point in having headers specified in a no-cache  
> response directive.  Unless it's intended that failure to revalidate  
> just headers could still result in the entry being served, but that  
> in that case those headers would need to be omitted?

Good question. My experience is that this isn't implemented, and that  
caches just ignore the extensions and treat them as bare private and  
no-cache, respectively. Anybody else?

Even if they aren't widely implemented, it seems like they're not  
harmful as defined (since caches take a more drastic but still  
conformant step). However, we can consider deprecating this behaviour.

Now issue #169 <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/169>.

> S 3.4 Pragma. -------------
> The BNF for this mentions extension-pragma.  Were there ever any of  
> these?  Does it make sense to continue to support an extension  
> mechanism on a deprecated header that no-one extended?

IIRC there are a few floating out there, so we need to allow them  
syntactically, at least.

> I've also seen some responses lately that have multiple Cache- 
> Control headers - is this valid?

You mean something like

Cache-Control: max-age=60
Cache-Control: public

?

If so, yes, that's completely allowed; p1 talks about headers being  
allowed to be serialised in different forms.

Cheers,

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 2 Jun 2009 02:04:13 -0500

> [email-masked] (entire thread)

My concern is that there are many origin servers that assume that a POST
response will not be returned in response to a subsequent GET/HEAD
request--in other words, they assume the method is part of the cache key.
But, I can see that my concern is unjustified, since even if a POST response
is cached, in cannot be returned in a subsequent request until it has been
revalidated (as required by RFC 2616 section 13.10). Since the cache has to
do one (possibly conditional) GET after every POST anyway, there is very
little chance of any problematic behavior.

Still, I expect that almost all caches will continue to avoid caching any
POST responses, just to avoid the issue altogether--especially since caching
POST responses doesn't reduce the number of requests that need to be
forwarded to the origin server at all (because of mandatory revalidation).

- Brian

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 03 Jun 2009 12:51:10 +1200

Mark - thanks for your comments, I've a few more below

> have POST (or other responses) cached, but invalidated upon a visible 
so when the language mentions invalidation of a cache entry, it doesn't 
necessarily mean deletion from the cache, purely flagging it as 
requiring revalidation?

> Recording this as issue #167 
> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/167>.
>
Further reading on the Content-Location header helped quite a lot in the 
end.

> As discussed, this could use some tweaking. #168 (editorial); 
> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/168>
>
>
>> Some cache control directives are confusingly similar, especially for 
>> response directives.
>
I definitely think the HTTPbis docs are a lot easier to read.

>> must not be stored. However it does not state what to do when you get 
>> a subsequent request for that URI that would match.  Does this mean 
>> that the stored response must be revalidated with the server, or is 
>> it appropriate to send the stored response (if still fresh) to the 
>> client without those headers?
>>
>> 2. no-cache.
>>
my reading is that a private with headers is more like a public, since 
it means only those headers are private.  So to treat it as private 
whilst being safe (in that it won't cache any part of the resource), 
won't also be optimal in terms of cache efficiency.

>> I've also seen some responses lately that have multiple Cache-Control 
>> headers - is this valid?
>
> You mean something like
>

thanks.

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 3 Jun 2009 17:47:46 +1000

> request--in other words, they assume the method is part of the cache  

2616 already states that a POST response can only be cached if there's  
explicit freshness information.

> revalidated (as required by RFC 2616 section 13.10).

That requirement is placed upon the POST, not a subsequent GET.

> POST responses, just to avoid the issue altogether

Very possibly; for the foreseeable future, I think using POST in this  
manner is a fairly specialised thing. However, it's also useful in a  
number of cases (esp. as "RESTful" Web Services become more prevalent).

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 8 Jun 2009 08:57:19 +1000

On 03/06/2009, at 10:51 AM, Adrien de Croy wrote:

> so when the language mentions invalidation of a cache entry, it  
> doesn't necessarily mean deletion from the cache, purely flagging it  
> as requiring revalidation?

 From 2616, section 13.10:
> In this section, the phrase "invalidate an entity" means that the  
> cache will either remove all instances of that entity from its  
> storage, or will mark these as "invalid" and in need of a mandatory  
> revalidation before they can be returned in response to a subsequent  
> request

>> Good question. My experience is that this isn't implemented, and  
>> that caches just ignore the extensions and treat them as bare  
>> private and no-cache, respectively. Anybody else?
>>
> my reading is that a private with headers is more like a public,  
> since it means only those headers are private.  So to treat it as  
> private whilst being safe (in that it won't cache any part of the  
> resource), won't also be optimal in terms of cache efficiency.

Of course, but that's an implementation decision. A cache isn't  
required to be maximally efficient :)

Cheers,

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 8 Jun 2009 00:38:32 +0100

Mark Nottingham wrote:
> It definitely isn't perfectly reliable, but it is very often good  
> enough.

I suppose it is.

I try to make my HTTP applications have results which are predictable
and the same for everyone, so you can build reliable things on top of
them.  Mostly that involves Etags, and hash-encoded URL components
where round trips are better avoided.

I don't like "program/person A sees a different result from
program/person B because of luck of the network", even though I agree
that a lot of applications are good enough that way and it undoubtedly
reduces loads.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 8 Jun 2009 00:47:01 +0100

Adrien de Croy wrote:
> However order therefore is still important, so matching between
> 
> Content-Language: en-nz, fr
> and
> Content-Language: fr, en-nz
> 
> would fail, even though logically these are identical since order is not 
> significant in the spec.  I can understand this makes matching simpler 
> (normalise and string compare), and probably shields against arbitrary 
> selection algorithms (e.g. order-based) being used by the origin server, 
> but it's not clear from the spec as to why, and anyone wanting to do 
> things a bit "smarter" may consider doing a different sort of match with 
> potentially questionable results.

I believe the idea is that it isn't for caches to override or police
the origin server's policy, but to provide a generic caching mechanism
that supports negotiation, and let the origin server implement the
negotiation policy however it likes.

That is much more extensible to the future, and works with all request
headers, including types of negotation which aren't mentioned in any
spec.

I don't see any advantage in making the cache "smarter" than this, and
several disadvantages.

For example, imaging a server which responds with this text
as a diagnostic service:

    HTTP/1.1 200 UK
    Vary: Content-Language
    Content-Type: text/html
    Etag: 5346593659365

    <html><head></head><body>

    Thank you for your request.<br>
    Your request provided the following language list: "en-nz, fr".

    </body></html>

I think that's a legitimate cacheable response, and one where you
would want the origin server to be consulted when a request comes
along with Content-Language in a different order.

-- Jamie

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 8 Jun 2009 09:28:18 +1000

It's not dependable in the sense that authors can rely upon *all*  
cached copies of a response being invalidated all of the time (and a  
note to this effect would probably be worthwhile).

However, there is still very much a point; for many kinds of data, the  
most relevant cached copies are those in between the user who made a  
change (with a POST, PUT or whatever) and the origin server; e.g., if  
you submit a blog comment, it's important that you see the change  
quickly, less so that someone halfway around the world see the change  
as quickly.

As far as going through different proxies -- that's largely an  
implementation and deployment issue. With HTCP <http://www.rfc-editor.org/rfc/rfc2756.txt 
 >, you can in fact coordinate such invalidations between peered  
proxies, and in fact my employer has just finished funding changes to  
Squid to ensure that this happens correctly, because we use have uses  
for this type of deployment.

Cheers,

On 08/06/2009, at 9:21 AM, Jamie Lokier wrote:

> But it's unreliable, because requests can go via different proxies.
>
> What is the point in mandating that proxies support an unreliable
> mechanism, which suggests (wrongly) to server authors that they can
> depend on it, when there are good reliable caching mechanisms in the
> spec already?
>
> -- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 8 Jun 2009 00:28:37 +0100

Brian Smith wrote:
> Adrien de Croy wrote:
> > This leaves problems if there's no Vary header but content negotiation
> > was used.  It's not possible to reliably heuristically determine if
> > content-negotiation was used without the Vary header.  Vary is only a
> > SHOULD level.  Maybe it should be a MUST level?
> 
> As you noted below, this rule applies:
> 
>    Caches MUST use the most recent response (as determined by the Date
>    header) when more than one suitable response is stored.  They can
>    also forward a request with "Cache-Control: max-age=0" or "Cache-
>    Control: no-cache" to disambiguate which response to use.
> 
> Using this rule, along with other rules for caches, results in predictable
> behavior, right?

Requests with "Cache-Control: no-cache" bypass the cache, and do _not_
always result in the most recent response (as determined by the Date
header) being forwarded.  They require the exact corresponding
response to be forwarded.  Think about overlapping concurrent requests
and responses in various orders, various times to forward parts of
each message and generate the response, and with various Date values.

> > There needs to be another set of checks, at the minimum that
> > Accept-Encoding matches the stored Content-Encoding.  E.g. you
> > can't serve gzip content if there's no Accept-Encoding: gzip.
> > Arguably others as well (like Accept-Language
> > matching on Content-Language , with q values etc).
> 
> Actually, a server *is* allowed to do that. And, caches don't have to worry
> about that because they don't have to interpret Accept-* and Content-*
> headers. (See my previous message in the other thread.)

I agree.  A server can serve whatever it likes, and cache is not
supposed to interpret those headers itself.

> > It says if there are several stored representations serve the one with
> > the most recent Date header (MUST level), but this may not be the
> > appropriate one if Vary headers aren't available, and you are say
> > selecting based on language.
> 
> The server needs to provide a Vary header in that case.

I agree.  Selecting based on language is a server decision - the cache
does not know what policy the server uses to decide the language of
responses.

> See the discussion about whether the method is part of the cache key. Caches
> really need to be very conservative here (that is, MUST invalid) as there
> seems to be a lot of disagreement amongst implementers and standardistas
> regarding this issue.

Conservative is good, but not too conservative that anything with
Vary, varying Vary, or '?' in the URL is uncacheable, as some caches
implement :-)

-- Jamie

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 8 Jun 2009 10:33:26 +1000

Just FYI, we do have an open issue on these lines:
   http://trac.tools.ietf.org/wg/httpbis/trac/ticket/147

Cheers,

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 8 Jun 2009 00:21:06 +0100

-- Jamie