Subject: Meaning of Vary header applied to non-existent headers

From: Jamie Lokier <[email-masked]>
Date: Mon, 23 Feb 2004 20:40:41 +0000

This is a request for clarification in the HTTP/1.1 standard.  Perhaps
something for the accumulating errata.

1. When an origin server selects the response by looking at a request
   header, and the header does not exist, is it correct for the server to
   add the name of that header to Vary?

   For example, suppose a server's logic is like this pseudo-code:

       reponse_headers (Vary) = "Accept-Language"

       IF request_headers (Accept-Language) contains "ja" THEN
	   serve the japanese page
       ELSE 
           serve the english page
       ENDIF

   What is the correct HTTP response when the request doesn't have an
   Accept-Language header?

   I would guess that the correct response is to "include Vary:
   Accept-Language", but that begs the next question:

2. How should a cache treat missing requests headers which are
   mentioned in Vary, when deciding if a stored entity matches a
   request?

   a. If the stored entity was requested _without_ the header mentioned
      in Vary, and the new request is also _without_ the header, does
      the stored entity match?  (This would come from scenario 1 above).

   b. If the stored entity was requested _without_ the header mentioned
      in Vary, and the new request is _with_ the header, does the
      stored entity match?
     
   c. If the stored entity was requested _with_ the header mentioned
      in Vary, and the new request is _without_ the header, does the
      stored entity match?

I would imagine that in case a, the entity matches, and in cases b and
c, the entity doesn't match and must be revalidated.

But the language of RFC 2616 is a unclear:

   When the cache receives a subsequent request whose Request-URI
   specifies one or more cache entries including a Vary header field,
   the cache MUST NOT use such a cache entry to construct a response to
   the new request unless all of the selecting request-headers present
   in the new request match the corresponding stored request-headers in
   the original request.

Can the selecting request-headers "match" if they are absent?  What
about case a?

   The selecting request-headers from two requests are defined to match
   if and only if the selecting request-headers in the first request can
   be transformed to the selecting request-headers in the second request
   by adding or removing linear white space (LWS) at places where this
   is allowed by the corresponding BNF, and/or combining multiple
   message-header fields with the same field name following the rules
   about message headers in section 4.2.

Is this supposed to mean that an absent header is equivalent to an
empty header, or that an absent header is semantically a distinct
value from an empty header, or that an absent header is not considered
at all in the decision, or that an absent header cannot be matched
even with an absent header in the original request for the cached
entity?

3. Does this mean that server code which inspects a header value, and
   finds the header missing, should look like this instead?

       IF exists request_headers (Accept-Language) THEN
           reponse_headers (Vary) = "Accept-Language"
       ELSE
           response_headers (Vary) = "*"
       ENDIF

       IF request_headers (Accept-Language) contains "ja" THEN
	   serve the japanese page
       ELSE 
           serve the english page
       ENDIF

This question has come up because I am writing some simple server code
which creates Vary headers when the application examines the value of
a request-header.  Yet, reading RFC 2616 doesn't seem to clarify what
the server code should do in these cases; it sheds no light on it at all.

The conservative thing to do is to disable cacheing when an examined
request-header is absent, but that seems excessively pessimistic.

I was prompted to ask here, after reading the HTTP/1.1 errata document
and seeing nothing that clarifies this question.

Can anyone shed any light on this?

Thanks,
-- Jamie

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 23 Feb 2004 14:34:02 -0700 (MST)

>    add the name of that header to Vary?

If cachable response depends on header X, the server must add a
	Vary: X
or
	Vary: *
header. This behavior should be independent from actual headers
present in the request (IMO).

>    mentioned in Vary, when deciding if a stored entity matches a

As you know, RFC 2616 defines Vary matching based on the existence of
a LWS/merge transformation. Such transformation cannot add new or
remove existing headers. Thus, IMO, a present header can never match
an absent header. An absent header matches an absent header.

> ...
> I would imagine that in case a, the entity matches, and in cases b
> and c, the entity doesn't match and must be revalidated.

I agree.

> even with an absent header in the original request for the cached

If both requests lack a varied header X, then a null transformation
(i.e., transformation that does nothing) matches them.

To match a header without a field-value with an absent header one
would need to add a header name to one of the requests, which is
probably not a part of the allowed transformation. However, a header
without field-value and/or a header without field-content (RFC 2616
BNF is ambiguous in this context) are strange animals, and I would not
recommend counting on any specific implementation of related
requirements.

>        IF request_headers (Accept-Language) contains "ja" THEN

I would add Vary header unconditionally. Whether you want to disable
caching depending on request properties is a question that goes beyond
protocol semantics. The right thing to do would depend on the
trade-off between more hits and fewer incorrect objects delivered.

Note that a valid Accept-Language cannot be empty. It is not clear,
however, how an invalid header is supposed to be treated by the
matching algorithm in the RFC.

Alex.

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 23 Feb 2004 23:55:27 +0000

> an absent header. An absent header matches an absent header.

I agree totally with you, Alex.

> trade-off between more hits and fewer incorrect objects delivered.

Sure, forget about my caching preferences.  I wanted to know if it was
technically ok to add the Vary header unconditionally - and more
importantly, if that would work according to everyone else's reading
of the standard.

> matching algorithm in the RFC.

Ok, but that's a different question from the one I'm interested in.

I would like to see the language of the standard clarified to cover
the case of absent headers mentioned in Vary, so that there is no
confusion.  The two paragraphs I cited seem contradictory, given a
strict reading of the language.

To cut to the precise language which is ambiguous:

   *all of the selecting request-headers present in the new request
   match the corresponding stored request-headers in the original
   request.*

In my understanding of English, if the marked text begin with "each"
instead of "all", it's unambiguous but means the wrong thing.  (It
means to try each selecting request-header in the new request, and one
at a time match it against the corresponding stored request-header).

"All" in that position can mean _either_ the same as "each", or that
all the new request-headers are taken as a set and matched against all
the stored request-headers as a set, in one single match operation.  It
depends how you parse the quantifier.

Matching as a set is wanted, and the next paragraph confirms that
interpretation, although it wasn't until you drew my attention to it
that it made sense to me that way.

Better language may be:

   When the cache receives a subsequent request whose Request-URI
   specifies one or more cache entries including a Vary header field,
   the cache MUST NOT use such a cache entry to construct a response to
-  the new request unless all of the selecting request-headers present
-  in the new request match the corresponding stored request-headers in
+  the new request unless the set of selecting request-headers present
+  in the new request matches the set of selecting request-headers stored in
   the original request.

+  Headers may be named in the Vary field value which do not exist in
+  one or both requests.  Headers which are named in the Vary field
+  value and do not exist in one request are matched if and only if
+  the corresponding headers do not exist in the other request.

That last paragraph is a logical consequence of the preceding two
paragraphs, if you read them the right way.  However, I have read the
standard many times and implemented a HTTP/1.1 server, and not
understood that logical consequence until now.  So the extra
clarifying paragraph is suggested.

Thanks,
-- Jamie