Subject: If-Range with other conditionals

From: Alex Rousskov <[email-masked]>
Date: Mon, 23 Sep 2002 18:13:04 -0600 (MDT)

Hi there,

RFC 2616 says:

   An HTTP/1.1 origin server, upon receiving a conditional request that
   includes both a Last-Modified date (e.g., in an If-Modified-Since or
   If-Unmodified-Since header field) and one or more entity tags (e.g.,
   in an If-Match, If-None-Match, or If-Range header field) as cache
   validators, MUST NOT return a response status of 304 (Not Modified)
   unless doing so is consistent with all of the conditional header
   fields in the request.

As far as I can see, a 304 (Not Modified) response code SHOULD NOT be
generated for an If-Range request. If-Range definition implies that
only 200 (OK) and 206 (Partial Content) responses are valid. Can a 304
response be ever "consistent with" an If-Range field?

Can a 412 response be ever "consistent with" an If-Range field?

And a related question regarding proxy behavior; RFC 2616 says:

   An HTTP/1.1 caching proxy, upon receiving a conditional request that
   includes both a Last-Modified date and one or more entity tags as
   cache validators, MUST NOT return a locally cached response to the
   client unless that cached response is consistent with all of the
   conditional header fields in the request.

What does "return a locally cached response" mean exactly? Does it
include returning a 304 response (cached headers, no content)? Or just
responses with locally cached _content_ (such as 200 and 206
responses, but not 412).

I find it curious that all "If-*" sections in the RFC have "the
behavior when both If-Foo and If-Bar are present is undefined"
clauses, _except_ for the If-Range section. It sort of implies that
If-Range can be used in combination with all others, but valid server
behavior upon receiving certain combinations is not clear to me. Thus
the questions above.

Can the author(s) or anybody clarify the intent?

Thank you,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 24 Sep 2002 18:24:52 -0600 (MDT)

On Tue, 24 Sep 2002, Jeffrey Mogul wrote:

> Anyway, I can't think of any reasonable interpretation for this
> example:
>
> 	GET /foo.html HTTP/1.1
> 	Host: whatever.com
> 	If-Range: "a"
> 	Range: bytes=100-200
> 	If-Match: "b"

What, in your opinion, should an origin server return when receiving
the above request (we can assume that entity has an ETag of "c")? It
looks like returning 200/206 would violate If-Match while returning
anything else would violate If-Range SHOULD requirements. Should the
server return "Bad Request"? Or is this simply an "undefined behavior"
case?

> It *might* make sense to do:
>
> 	GET /foo.html HTTP/1.1
> 	Host: whatever.com
> 	If-Range: "a"
> 	Range: bytes=100-200
> 	If-None-Match: "b"
>
> which means
> 	if the current entity tag is "b" then send 304

OK. It looks to me that you are saying that the presence of If-Range
headers does not limit response codes that can be returned. In other
words, the existing SHOULD "definition" of If-Range is valid only when
no other If-* headers are present:

   If the entity tag given in the If-Range header matches the current
   entity tag for the entity, then the server SHOULD provide the
   specified sub-range of the entity using a 206 (Partial content)
   response. If the entity tag does not match, then the server SHOULD
   return the entire entity using a 200 (OK) response.

Here is what we currently do to test the proxy-related MUST in
question: generate all possible combinations of If-* headers that do
not violate other MUSTs, make on of the If- header mismatch LMT/tag of
the cached entity, and then make sure that the proxy under test does
not return any cached information (including headers). The proxy can
return any status codes. Do you think that's the best strategy of
verifying the MUST in question?

> P.S.: Alex, I owe you answers for several previous questions you've
> posted.  I'm trying to catch up on my email.

Please feel free to ignore any that make little sense or are not
important. You certainly do not owe me any explanations!

Thanks a lot for a detailed explanation,

Alex.

---

From: Jeffrey Mogul <[email-masked]>
Date: Tue, 24 Sep 2002 19:47:14 -0400 (EDT)

Alex Rousskov asks:

I think that's probably "me", for these excerpts, but it's
possible other people had some say in the wording here.

(1) Alex writes:

    RFC 2616 says:

    
It would have been helpful to note the specific section number from
RFC2616.  This particular text is in 13.3.4 (Rules for When to Use
Entity Tags and Last-Modified Dates) and isn't meant to be normative
with respect to what status codes are valid under other circumstances.
That is, this is an extra restriction (imposed when a request has
multiple conditions), and can't be taken as implying a lack of
restrictions in other cases.

My guess is that we (I?) wrote this without fully thinking through
whether If-Range could ever return ("be consistent with") 304.
But on the "better safe than sorry" principle, I see nothing
really wrong with this paragraph.

Consider at least this case:

	GET /foo.html HTTP/1.1
	Host: whatever.com
	If-Range: *
	Range: bytes=100-200
    
Remember that section 14.27 (If-Range) informally defines If-Range
as a sort of combination of GET+If-Match+Range, followed by a plain
GET if the If-Match fails.

Section 14.24 (If-Match) says:

   If none of the entity tags match, or if "*" is given and no current
   entity exists, the server MUST NOT perform the requested method, and
   MUST return a 412 (Precondition Failed) response. This behavior is
   most useful when the client wants to prevent an updating method, such
   as PUT, from modifying a resource that has changed since the client
   last retrieved it.

All of the other language that defines what 412 really means
looks a lot like this.

Therefore, I suppose it is plausible that if "/foo.html" simply
doesn't exist, then 412 is ""consistent with" If-Range.  However,
this seems to be a nonsensical special case (why not simply return
404 here, and in any case what does "If-Range: *" mean?) so I think
we can rule it out.  The other reason for returning 412 with a
plain If-Match (none of the entity tags match) is precisely the
case where If-Range wants the server to return 200.

Thus, I don't think If-Range should ever give rise to a 412 response,
except in this one probably nonsensical example for which a 404
response would be more appropropriate.

But again, this is not really the point being addressed in section
13.3.4!

    

(Again, we're in 13.3.4.)  This is one of those places where the
terminology ("response") used in RFC2616 fails us.  (I won't rehash
that debate here.)  I think the most reasonable interpretation of this
paragraph is that a cache entry of any sort (i.e., with any status
code) is useless if it's not consistent with all of the conditional
header fields in the request.

This follows my basic philsophy about caching: if there's any
doubt, don't use a cache entry; contacting the origin server
might be slow but it won't give the wrong content or headers.

Perhaps an oversight on my part.  On the other hand, we made an
explicit decision (discussed on the mailing list, if my memory
is correct) not to list in the spec all possible combinations of
headers that were not meaningful.  The fact that we did list a
few specific "undefined" combinations does not mean that all
other combinations are reasonable!

	GET /foo.html HTTP/1.1
	Host: whatever.com
	If-Range: "a"
	Range: bytes=100-200
	If-Match: "b"

Note that unlike "If-Match", which can carry many entity tags,
If-Range is explictly defined to carry only one validator.  The
assumption is that If-Range is defined to fill in holes for
a specific cached instance, not to grab a chunk of some
finite subset of the instances of a resource.

	GET /foo.html HTTP/1.1
	Host: whatever.com
	If-Range: "a"
	Range: bytes=100-200
	If-None-Match: "b"

which means
	if the current entity tag is "b" then send 304
	else if the current entity tag is "a" then send 206 (w/bytes=100-200)
	else send 200

but believe me, I haven't thought this through all the way.

-Jeff

P.S.: Alex, I owe you answers for several previous questions
you've posted.  I'm trying to catch up on my email.

---

From: Jeffrey Mogul <[email-masked]>
Date: Tue, 24 Sep 2002 21:04:03 -0400 (EDT)

> Anyway, I can't think of any reasonable interpretation for this
    
    What, in your opinion, should an origin server return when
    receiving the above request (we can assume that entity has an ETag
    of "c")? It looks like returning 200/206 would violate If-Match
    while returning anything else would violate If-Range SHOULD
    requirements. Should the server return "Bad Request"? Or is this
    simply an "undefined behavior" case?
    
According to the spec, it's "undefined behavior" (because we
don't define it) but I think you are correct that neither 200
nor 206 is allowed in this case, and 304 seems forbidden, too.
So as a server implementor, I would probably go for 400 (Bad
Request).  I'm generally a "be liberal in what you accept"
kind of guy, but returning a 200 response in this case does
violate the intent of If-Match.

    

Like I wrote, "believe me, I haven't thought this through all the way."
    
Anyway, "SHOULD" means (per RFC2119):

   that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.

so that we could argue that RFC2616 isn't truly self-contradictory :-)

This seems plausible, but I'm not an expert on formal verification,
and such experts have found bugs in spec language that I helped
write (see the unfortunately-titled "Safe Composition of Web
Communication Protocols for Extensible Edge Services", by
Adam D. Bradley, Azer Bestavros, Assaf J. Kfoury in the 7th
WCW, which is really about the bugs that arise w.r.t. the
Expect/100 Continue specification).

In general, a proxy "cache" should be able to meet all of the
caching-related parts of RFC2616 by shrinking its cache size to
zero, so a cache that passes your test could be compliant, but
I'm not sure that it's *necessarily* compliant.  That is, I'm
not sure that your approach actually tests all of the MUSTs
in the spec.  There might be some interactions with Cache-Control
headers that you need to test, too.

I'm also assuming that you limit your testing to GET/HEAD
methods, since nobody has really worked out how HTTP caches
deal with PUTs or POSTs in all possible cases (except by
simply forwarding these).

-Jeff

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 25 Sep 2002 09:08:27 -0600 (MDT)

> headers that you need to test, too.

Yes, of course. The above approach is used for the "combination of
If-* headers" test cases only. There are more than 500 test cases
covering everything else, including Cache-Control handling. For the
majority of [caching] MUSTs it is possible to test in a more
specific/direct way when the cache has to act on stored content.  The
above is simply a workaround because we seem to be hitting an
"undefined behavior" ground when many If-* headers are combined in a
request.

> I'm also assuming that you limit your testing to GET/HEAD methods,
> since nobody has really worked out how HTTP caches deal with PUTs or
> POSTs in all possible cases (except by simply forwarding these).

Hmm... RFC 2616 has a few MUSTs related to non-GETs and caches/proxies
that we do test. For example, some proxies retry POST requests, many
proxies cannot handle chunked POSTs, and most caches do not invalidate
cached entities mentioned in *Location request headers.  Co-Advisor
can detect those MUST violations quite well.

We do not do formal protocol verification though (as opposed to Adam
Bradley's research). Our test subject is the implementation, not the
spec.

Thank you,

Alex.