Subject: IE vs vary header

From: "Julian Reschke" <[email-masked]>
Date: Wed, 19 Jun 2002 13:34:05 +0200

Hi,

seems that it's an old and well-known issue ([1]) that Internet Explorer
can't handle Vary headers correctly when passing content to external
applications. The Apache bug report is from 1999, yet unfortunately, even IE
6.0 still has this bug, and according to what I hear from Microsoft, *they*
don't consider it a bug at all.

The Apache-recommended workaround is to disable sending the Vary header
based on the User-Agent header. However, I have my doubts that this is a
workable solution.

Consider:

1a) Resource X is requested by a non-IE user agent. Server sends entity with
vary header. Entity and headers are cached by a proxy.
1b) Same resource is requested by IE. Entity and headers (including vary
header) are returned by caching proxy.

In this case, IE will get a result including the vary header it's not
supposed to see.

On the other hand:

2a) Resource X is requested by IE. Server sends entity without vary header.
Response is cached by a proxy.
2b) Same resource is requested ny non-IE user agent. Entity and headers
(vary header missing) are returned by caching proxy.

Here, the user agent (or other intermediates) will not see the vary header,
breaking their cache logic.

So otherwise I'm misunderstanding something about the mechanics of the Vary
header and caches, this workaround will (1) not always work with IE and
(2) - worse - break other user agents.

Feedback appreciated,

Julian

[1] <http://bugs.apache.org/index.cgi/full/4118>

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 Jun 2002 09:11:42 -0600 (MDT)

On Wed, 19 Jun 2002, Julian Reschke wrote:

> seems that it's an old and well-known issue ([1]) that Internet
> Explorer can't handle Vary headers correctly when passing content
> to external applications. The Apache bug report is from 1999, yet
> unfortunately, even IE 6.0 still has this bug, and according to
> what I hear from Microsoft, *they* don't consider it a bug at all.
> 
> The Apache-recommended workaround is to disable sending the Vary
> header based on the User-Agent header. However, I have my doubts
> that this is a workable solution.
> 
> Consider:
> 
> 1a) Resource X is requested by a non-IE user agent. Server sends
> entity with vary header. Entity and headers are cached by a proxy.
>
> 1b) Same resource is requested by IE. Entity and headers
> (including vary header) are returned by caching proxy.

I am afraid (1b) is incorrect for caches that know about Vary. A
compliant HTTP cache will at least revalidate the cached entity with
the origin server because IE User-Agent header will not match the
cached User-Agent header. A correctly behaving server would then
return a new entity, without the Vary header, and that entity will be
forwarded to IE. To quote RFC 2616:

   When the cache receives a subsequent request whose Request-URI
   specifies one or more cache entries including a Vary header field,
   the cache MUST NOT use such a cache entry to construct a response to
   the new request unless all of the selecting request-headers present
   in the new request match the corresponding stored request-headers in
   the original request.

If a cache does not handle Vary correctly, then (1b) is correct.

In general, I would simply avoid using Vary headers for server-side
negotiation. If you control the server, use other, more robust,
mechanisms to achieve the same goal. I cannot confirm or deny that IE
is broken in this way, but our HTTP compliance testing shows that at
least some popular HTTP/1.1 proxies do not handle Vary correctly.

Alex.

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 19 Jun 2002 17:23:14 +0200

Alex,

thanks for the feedback.

> ...

Of course you are right -- if and only if "user-agent" is added to the Vary
header. I'd like to avoid that, because that adds yet another dimension. And
even if I do that, the second scenario still can happen, right?

> least some popular HTTP/1.1 proxies do not handle Vary correctly.

What are these other, more robust mechanisms?

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 Jun 2002 10:03:20 -0600 (MDT)

On Wed, 19 Jun 2002, Julian Reschke wrote:

> Of course you are right -- if and only if "user-agent" is added to
> the Vary header. I'd like to avoid that, because that adds yet
> another dimension. 

Oh, I see. In this context, you cannot negotiate depending on user
agent without adding User-Agent to the Vary header and you have to
negotiate based on user agent if you want to work around user agent
bugs.

> And even if I do that, the second scenario still can happen,
> right?

The second scenario should not happen if you mark Vary-less responses
as uncachable. You have to mark them as uncachable for the
Apache-suggested work-around to make sense in your case (IMO).

> What are these other, more robust mechanisms?

There are probably several such mechanisms. The simplest is to mark
all negotiated content as uncachable and return no Vary headers, but
there may be more elegant solutions. For example, would it be possible
to change the URL (using an HTTP redirect response) based on the
request headers or based on whatever you are using to negotiate? You
could redirect all IE clients to their own URL "space" that does not
use Vary while using default URLs for other clients.  Changing URLs
solves all solvable problems caused by broken caches. The redirection
and its side-effects must be handled transparently to the user and
content author, of course.

Alex.

---

From: "Julian Reschke" <[email-masked]>
Date: Wed, 19 Jun 2002 18:13:05 +0200

Again,

thanks for the feedback...

> negotiate based on user agent if you want to work around user agent

Great :-)

> Apache-suggested work-around to make sense in your case (IMO).

I think if I mark them as un-cacheable, the *same* problem in IE will
occur -- it fails to pass content to external applications if it decides
that the content is not cacheable (their logic is: we pass data to external
programs by caching it, but the response headers indicate that the result
isn't cacheable, thus we can't pass it on -- funny, isn't it?).

> content author, of course.

That's very good input -- I'll try that.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 Jun 2002 10:20:14 -0600 (MDT)

On Wed, 19 Jun 2002, Julian Reschke wrote:

> I think if I mark them as un-cacheable, the *same* problem in IE
> will occur -- it fails to pass content to external applications if
> it decides that the content is not cacheable (their logic is: we
> pass data to external programs by caching it, but the response
> headers indicate that the result isn't cacheable, thus we can't
> pass it on -- funny, isn't it?).

You may be able to mark them with "Cache-Control: private" to avoid
the above problem. Also, as somebody suggested to me off-list, you can
try a must-revalidate cache-control directive so that the content is
cachable by shared caches but the server has some control based on
client request headers. The latter option requires must-revalidate
support in caches and may cause the same IE problems you mention
above; simple but not a very robust solution.

Good luck,

Alex.

---

From: "Julian Reschke" <[email-masked]>
Date: Fri, 28 Jun 2002 09:05:52 +0200

> From: [email-masked] [mailto:[email-masked]]On
> Behalf Of Alex Rousskov
> Sent: Wednesday, June 19, 2002 6:20 PM
> To: Julian Reschke
> Cc: [email-masked]
> Subject: RE: IE vs vary header
>
>
>

Alex,

this was actually a very good suggestion -- for some reason *this* is
accepted by IE.

(It's really time that MS starts listening to reports about serious bugs).

Julian