Subject: Re: i37 - Vary and non-existant headers

From: Adrien de Croy <[email-masked]>
Date: Wed, 14 Nov 2007 17:11:08 +1300

Just been reading the old discussion on issue i37 and thought I'd throw 
in my 2c on this, since I'm currently re-writing our caching module.

IMO the condition for a match depends on the meaning of the header.  In 
general if a header in a Vary tag advertises a capability, and has any 
possible values which are optional under the spec, then lack of the 
header would need to be deemed a mis-match, otherwise the cache would be 
assuming that an optional capability was available at the UA where it 
was not advertised.  Where the header advertises a restriction, then 
lack of an advertised restriction should be deemed lack of the 
restriction.  Where the definition of the header explicitly states how 
to handle lack of the header, then this overrides.

For instance, an Accept-Language header is a way for a UA to indicate 
what languages it is prepared to accept.  Absence of this header IMO 
indicates that the UA is prepared to accept any language, and that 
therefore if a cached entity had "Accept-Language" in its Vary header, 
then this would match.  An alternative view could be that absence of a 
header implies a default value, e.g. English, and so if the stored 
entity was not English, this would not match.  Personally I believe in 
this case that lack of advertisement of a restriction implies lack of 
the restriction.

However for Accept-Encoding, I believe this is an advertisement of 
capability by the client, and absence of this advertisement should be 
taken to mean the capability isn't there, and therefore this is a 
non-match. (PS, this is another instance of where a reference to the 
"identity" coding should be removed).  There is also explicit wording 
for how to deal with a missing header in this case also which disagrees 
with me - states that "the server MAY assume that the client will accept 
any content coding".  This however IMO is dangerous, and the SHOULD 
requirement to send "identity" should be replaced by MUST send 
unencoded.  Otherwise we are making gzip, deflate etc all mandatory.

Also, for Accept-Charset there is explicit mention of how to determine 
default values.  It explicitly states that the lack of an advertised 
restriction is deemed to be no restriction.

PS, a gripe about "q" values.  Surely it would have been a lot simpler 
to just get the UA to specify the preference by the order, e.g.

Accept-Language: en-nz, en, fr, *

This clearly indicates a preference for NZ English, followed by any 
other English, followed by any French, followed by anything else.  
Absolute quality shouldn't factor in the server's decision (it 
contravenes client's advertised willingness to accept something), only 
relative quality, and that is ranking.  I guess this wouldn't cover the 
q=0 case, but absence of advertised acceptance should be taken as lack 
of willingness?  And the use case for accepting a range of things, and 
denying part of the range, whilst plausible, makes me wonder if anyone 
uses it.

In terms of how this affects the spec, what about in these Accept* 
headers, allowing a list of accepted values to be interpreted as a 
priority list where no q values are present.

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 14 Nov 2007 05:33:47 +0100

>unencoded.  Otherwise we are making gzip, deflate etc all mandatory.

I am not sure what you are suggesting here. The requirement is a poorly
phrased "SHOULD unless" one, do you want to change this into "MUST un-
less" or into simply "MUST"? I don't think the latter is appropriate
(if I know for some reason the client can handle the response, there is
nothing wrong with sending an encoded entity body), and turning it into
a "MUST unless" is not meaningful, since "SHOULD" is already "MUST un-
less".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 14 Nov 2007 18:24:15 +1300

> a "MUST unless" is not meaningful, since "SHOULD" is already "MUST un-

How can you know for sure that a client has a capability when the client 
doesn't advertise it in that particular request?  How can the 
server/proxy/cache even know it's the same client, or going through the 
same request chain?  (e.g. load-balancing through a heterogenous proxy 
farm) If it's from a previous request advertising some capability, why 
would a sane client arbitrarily (?) decide to cease advertising that 
capability if in fact it's still available?

Some capabilities may have been inserted in previous requests by a proxy 
in the chain, which may not be in the chain for this request.  e.g. 
Accept-Encoding: gzip inserted by a proxy trying to save upstream 
bandwidth intending to expand it before sending it to the client which 
doesn't support gzip itself.

IMO if the client doesn't advertise a capability for a request, it 
should not be assumed to be available for that request, regardless of 
any previous requests that may have advertised a capability. Therefore 
to act as if an optional capability is available when not expliocitly 
advertised is dangerous.

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 14 Nov 2007 19:10:47 +1300

Adrien de Croy wrote:
>
> Bjoern Hoehrmann wrote:
>> * Adrien de Croy wrote:
>>   
> Some capabilities may have been inserted in previous requests by a 
> proxy in the chain, which may not be in the chain for this request.  
> e.g. Accept-Encoding: gzip inserted by a proxy trying to save upstream 
This lead me to an interesting discussion on ETags and Variants.

Until about 25 minutes ago, I was a proponent of different encodings of 
the same resource being treated as the same resource, due to the fact 
that the encoding should be lossless.  However, this I now believe is 
impossible.  The reason I haven't seen discussed, so thought I'd toss it 
out there.

Servers don't want the overhead of compressing a file every time someone 
requests it compressed, vs serving it straight when not.  So, the server 
maintains a cache of pre-compressed versions of the file. Or a server 
may use a downstream accelerator to do this.

It's impossible to guarantee that changes in the original document can 
be made and propagated in step with the (possibly cached) compressed 
version.  Therefore there will be windows in time where the compressed 
version does not reflect the current version of the file.  It's possible 
even the resource could change whilst the server is updating the 
compressed version.  Therefore all encodings must be treated as a 
separate entities with their own ETag (unless the server takes the hit 
of on-the-fly compression).  Whilst a server could feasibly take 
(brutal) steps to guarantee synchronisation and use the same ETags, it 
can't guarantee that a downstream cache would.

This makes it (very unfortunately) a bad idea for a proxy to decide to 
try and save bandwith by inserting an Accept-Encoding: gzip header as 
mentioned above, since that invalidates the ETag that the client may 
have provided.  I guess that's why the spec says proxies must not touch 
end-to-end headers.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 14 Nov 2007 07:13:08 +0100

>capability if in fact it's still available?

Well I was really asking what change you would like to see. There are
three paragraphs dealing with the lack of Accept-Encoding, are you pro-
posing to replace all of them by saying, if you can send it without any
encoding, then you must send it without any encoding if A-E is missing?
Or would you go even further and prohibit sending an encoded body even
if the server cannot send an unencoded body (you'd send 406 instead)?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 14 Nov 2007 19:29:18 +1300

> if the server cannot send an unencoded body (you'd send 406 instead)?
OK.

Just basically that since the encodings are optional, and you can't
guarantee a client can use any content other than unencoded, then the
only safe option where there is no C-E header is to send unencoded or
406 if you don't have an unencoded one (when will that ever happen on an
origin server?).

I guess it's all very well in custom applications for a server to know a
particular client say has a particular capability without that client
explicitly advertising it (e.g. some media apps), but this isn't
something we could expect an intermediary caching proxy to be aware of.
Those clients should still advertise the capability.  A 406 certainly
would prompt UA developers to start advertising capabilities.

Actually if there is no unencoded content, then you should be able to
deem the content to be unencoded.  E.g if a certain resource is only
ever available as gzipped, and clients only ever get it as gzipped, then
Content-Encoding: gzip shouldn't be used (the gzipped version is the
"identity"), since that entices intermediaries to try and unzip it and
send unzipped versions.

Or do you know of other use cases I'm missing?

Regards

Adrien

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 14 Nov 2007 19:24:56 +0000

> have provided.  I guess that's why the spec says proxies must not touch 

A proxy shouldn't do that for the reason you give.

But a proxy _is_ allowed to use the TE header instead for the same
effect.  In that case, the same Etag _should_ be used for compressed
(with Transfer-Encoding) and uncompressed representations of the same
entity.

This is allowed because TE and Transfer-Encoding are hop-by-hop
headers, while Accept-Encoding and Content-Encoding are end-to-end
headers.

A server, or even another proxy in the chain, can still use stored (or
cached on the fly) compressed/uncompressed versions, for efficiency as
you describe.

The details of how a server (or proxy) keeps both versions in sync if
it stores both are an implementation detail, outside the scope of the
protocol, but it's not complicated or difficult.

So you can get the performance you want of not compressing on the fly,
at the same time as implementing proxies which
auto-compress/auto-decompress by inserting a hop-by-hop request
header.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 14 Nov 2007 19:10:51 +0000

> to act as if an optional capability is available when not expliocitly 

Adding to this: whatever assumptions a server _might_ make (e.g. for
policy reasons, or knowing something about the clients another way),
in general caches should be conservative, and err on the side of
unnecessary cache misses, _never_ cache hits based on assumptions that
aren't the same as assumptions made by all compliant servers present,
past or future.  Whenever there's uncertainty, it's important to ask
the server if the request hits, and leave it to server policy to decide.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 14 Nov 2007 19:06:40 +0000

> this case that lack of advertisement of a restriction implies lack of 

What about a server generating something dynamic like this pseudo-code?

    <html><body>
    <? if (exists headers["Accept-Language"]) {
           print "Your request asked for these: ", headers["Accept-Language"];
       } else {
           print "Your request didn't specify any particular languages"
       }
    ?>
    </body></html>

That doesn't conform to the semantic equivalences you describe, but
it's valid and sometimes useful.

A result like that can be cached by putting Accept-Language in the
Vary header, and some systems would do that automatically, generating
the Etag and Vary header automatically from code like that.  (The Vary
header is generated due to seeing that header referenced in the page
logic.)

But if you say that a client or proxy can match a request with no
Accept-Language with a stored earlier request that has any
Accept-Language, then those dynamic response generation systems will
behave not as the programmer expects.  Some requests with no
Accept-Language will receive responses like "Your request asked for
these: <language>", which is clearly misleading.

That would require all systems which automatically make pages
cacheable by generating Vary headers to have knowledge of all header
semantic-equivalence rules, or declare that code in those systems
which doesn't fit those rules (like the example) is invalid and may
give misleading responses from caches, or always use Vary * which is
quite some overhead.

Note that there is no harm in a mis-match, and a mis-match is only
likely when requests are from differently configured clients.

As there's no harm in occasional mis-matches, and I think there is
some harm in preventing code like the example from working in a simple
automatic dynamic caching system, I propose that empty headers be
treated as distinct from present headers with any value, and that the
rules are the same for all headers, unless particularly good reasons
why that's a problem for a particular header can be shown.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Nov 2007 02:39:59 +0100

> to handle lack of the header, then this overrides.

No. Vary indicates the headers the server used / looked for when
determinig which variant to satisfy the request with, telling caches
which headers it must account for when comparing requests.

A request not having a certain header is a valid data point, and only
matches other requests not having that header.

> this case that lack of advertisement of a restriction implies lack of 

Vary and the cache rules around Vary do not care about semantics of
headers. Just their existence and value.  Vary is a selector telling
caches which headers of the request need to be included in the
comparison when comparing two requests. 

The logic actually selecting the correct variant depending on the
request content is all deferred to the web server, except when there is
exact matches filtered by Vary.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Nov 2007 02:47:46 +0100

> impossible.  The reason I haven't seen discussed, so thought I'd toss it 

It's been discussed several times already, and is where at least one of
the major web servers goof up..

It's very clearly spelled in the RFC. ETag is unique for the variant of
the resource, not the resource. Responses with different
content-encoding is different variants.

> version.

Further, the two is certainly not range compatible, allowing merging of
sub-ranges.

> have provided.

No it doesn't. But it then becomes the proxys responsibility to manage
ETag mappings properly or remove ETag entirely, and yes it's generally a
bad idea as you break the object identity chain.

> I guess that's why the spec says proxies must not touch 
> end-to-end headers. 

semantically transparent proxies MUST NOT ...

semantically non-transparent proxyes MAY, but then the specs do not
really care much about those. It's the implementers obligation to verify
that any semantic non-transparency done by the proxy do not violate
specifications.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Nov 2007 02:55:26 +0100

> 406 if you don't have an unencoded one (when will that ever happen on an

When the server only has an encoded variant. Consider a request for
http://www.example.com/some.file and the server only has some.file.gz
and the request has "Accept-Encoding: [blank]".

Clients not sending Accept-Encoding is assumed to accept any encoding
which may exists now or in future. Not sending Accept-Encoding is
basically saying "I don't care how it's encoded, just send me whatever
you have, but preferably unencoded if possible".

> A 406 certainly

So would getting incompatible media.. but in real life it's not such big
deal. All the server need to do is to include User-Agent (or whatever
header identifies the client in question) in Vary.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Nov 2007 03:04:50 +0100

> why that's a problem for a particular header can be shown.

It's in fact what the specs says. The problem is the language which
doesn't fire the right associations in many peoples mind. It's meant to
fire thoughts about headers as data sets, but for many people just
leaves a questionmark.

Vary is a selector on what to compare between two request. Think of it
as a grep selecting the interesting data of the requests before compare.

If you read "13.6 Caching Negotiated Responses" it's spelled out quite
in detail. Reading this section is mandatory before discussing ETag or
Vary. If you know this section then most (all?) questionmarks regarding
ETag or Vary in other sections kind of disappears.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 15 Nov 2007 08:51:53 +0000

> exact matches filtered by Vary.
  ^^^^^

What about whitespace differences (e.g. leading/trailing whitespace,
whitespace following the header name colon), multi-line breaks, and
multiple occurrences of the same header?

Does exact means _exactly_ in all those respects, or is it acceptable
for a cache to canonicalise certain things, like converting multiple
occurences of a header to a single header by joining with ", " (or
","), converting multi-line headers to single-line, and stripping
leading/trailing whitespace from the header value?

Thanks,
-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 15 Nov 2007 14:21:36 +0100

> leading/trailing whitespace from the header value?

Ofcourse there is some canonisation taking place, you can't compare two
sets of headers otherwise as the LWS and list rules allows the exact
same thing to be transmitted in many ways.. But there is no knowledge of
the meaning of any of the headers involved in the process.

The process is spelled out in quite detail in 13.2 Caching Negotiated
Documents, with very little room for error other than implementation
bugs. That's provided one cares to read 13.2 before jumping to
conclusions about how this should work.

Please note that in case of Vary request headers compare it's safe to
err on the cautious side and get false "Vary request" misses. So if you
optimize by doing a string literal compare instead of exact value then
things will still work, just not optimally so..

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 16 Nov 2007 09:46:52 +1300

> the meaning of any of the headers involved in the process.

If we don't take into account any meaning of headers, we lose many 
opportunities to cache.

E.g. if an original request for something contained an Accept-Language 
header specifying many acceptable languages, the response comes back 
with one of those languages.  If a subsequent request comes in which 
will accept the language that is cached, should that be served back or not?

I guess there are difficulties there where for one request it was the 
preferred language and for another it was a less-preferred language.

In which case how would you compare Accept-Language tags?  You'd need to 
compare them as a list with q values.  Or would you do it so that if the 
q value was 1 in the original request and subsequent request then its ok

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 15 Nov 2007 21:25:52 +0000

> If we don't take into account any meaning of headers, we lose many 

This is true, but that meaning is taken into account at the server.

It can still return a 304 Not Modified response, and the bulk of data
transfer is cached even though there's a round trip time.

If caches interpret headers in order to find more cache equivalents,
then it's _essential_ that this agrees with _all_ conformant servers,
including those already deployed, otherwise caching is not safe.

> will accept the language that is cached, should that be served back or not?

In that case, the request is sent to the server, which might decide to
return 304 Not Modified with the same Etag, and after that the cache
can server the entity without contacting the server for both forms of
the request.

> q value was 1 in the original request and subsequent request then its ok 

No, because it's permitted to have a server whose policy differs from
what you're assuming, and in any way you haven't thought of.

For example, a server might decide that Engish with q=1 and Polish
with q=0.75 should be served the Polish version because the English
version is a poor translation of the Polish original.  You can't know
the server's policy.

If there was a real value in removing a round trip in these cases, a
cache extension could be defined, where the server returns something
like "Cache-control: accept-language-standard-policy", and there was a
standard policy.  That would permit the cache to apply the policy.

(I doubt if there would be much gained by that feature, because it
only helps the _first_ request with a different header than previous
matching requests).

> >conclusions about how this should work.

And, indeed, a large number of server depend on caches behaving this
way.  It can only be changed by definining something in the response
which flags that additional rules may optionally be applied.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 16 Nov 2007 00:10:00 +0100

> If we don't take into account any meaning of headers, we lose many 

Maybe, unless the server supports ETag.

> will accept the language that is cached, should that be served back or not?

See 13.2 Caching Negotiated Documents for a detailed description of what
should happen in the point of the RFC.

> preferred language and for another it was a less-preferred language.

Yes, and is why the procedure in 13.2 exists, whereby the cache uses
If-None-Match to ask the origin which variant should be used for a new
kind of request.

> q value was 1 in the original request and subsequent request then its ok 

As a list-header, which is the compare function used for all headers in
this context (vary). Nothing special just because it's the
Accept-Language header. Same compare function used for Accept-Content or
X-FooBar.

The only one actually parsing Accept-Language (or the other Accept
headers) is the origin server when performing it's server driven content
negotiation dance selecting which variant to return for this request.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 17 Nov 2007 21:58:14 +0100

> will accept the language that is cached, should that be served back or not?

Depends on if the cache have seen that Accep-language header previosly,
and if not if the server supports If-None-Match.

Please read RFC2616 13.6  Caching Negotiated Responses one more time.

Regards
Henrik