Subject: Require UAs and intermediary caches to assume Vary: User-Agent

From: "Simon Pieters" <[email-masked]>
Date: Wed, 17 Oct 2012 09:19:46 +0200

See http://www.w3.org/mid/[email-masked]

It would be useful if the spec required that UAs and intermediary caches  
act as if all responses had Vary: User-Agent specified, since many pages  
on the Web serve different content depending on the User-Agent header but  
do not bother specifying Vary: User-Agent.

I would like this to be filed as a new issue in the issue tracker.

cheers,
-- 
Simon Pieters
Opera Software

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 17 Oct 2012 18:32:14 +1100

Um, no.

Not only will this retroactively make all intermediary caches non-conformant, it'll also make them completely useless, because of the large (and unnecessary) amount of variance in User-Agent headers.

I understand there are security issues here caused by CORS, but that doesn't give you the ability to force changes on the rest of the Web. 

Regards,

On 17/10/2012, at 6:19 PM, "Simon Pieters" <[email-masked]> wrote:

> It would be useful if the spec required that UAs and intermediary caches act as if all responses had Vary: User-Agent specified, since many pages on the Web serve different content depending on the User-Agent header but do not bother specifying Vary: User-Agent.
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 17 Oct 2012 08:15:51 +0000

--------
In message <[email-masked]>, "Simon Pieters" writes

>I would like this to be filed as a new issue in the issue tracker.

Fine with me, provided you _first_ reduce the number of distinct
U-A strings on the web to something reasonable, lets say no more than 25 ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Simon Pieters" <[email-masked]>
Date: Wed, 17 Oct 2012 11:00:42 +0200

On Wed, 17 Oct 2012 09:32:14 +0200, Mark Nottingham <[email-masked]> wrote:

> Um, no.
>
> Not only will this retroactively make all intermediary caches  
> non-conformant, it'll also make them completely useless, because of the  
> large (and unnecessary) amount of variance in User-Agent headers.

OK, I can see now that it would make them useless.

> I understand there are security issues here caused by CORS,

The security issue under discussion in the referenced thread would  
materialize if browsers start allowing changing the User-Agent header in  
XHR without sanitizing it. However, that's not the reason I sent the  
email. The reason is that bz argued that intermediary caches are broken,  
which they are for pages on the Web that vary but don't say they vary,  
however that's not actually limited to the User-Agent header and is not a  
valid reason to require intermediary caches be useless instead of broken.

Also see  
http://lists.w3.org/Archives/Public/public-webapps/2012OctDec/0216.html
-- 
Simon Pieters
Opera Software

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 17 Oct 2012 22:28:55 +1300

On 17/10/2012 10:00 p.m., Simon Pieters wrote:
>> non-conformant, it'll also make them completely useless, because of 
>> the large (and unnecessary) amount of variance in User-Agent headers.
>
> materialize if browsers start allowing changing the User-Agent header 
> in XHR without sanitizing it. However, that's not the reason I sent 
> the email. The reason is that bz argued that intermediary caches are 
> broken, which they are for pages on the Web that vary but don't say 
> they vary, however that's not actually limited to the User-Agent 
> header and is not a valid reason to require intermediary caches be 
> useless instead of broken.
>
> Also see 

Exactly. Vary is mandatory for resources with negitiated variations - 
even if it is the nasty "Vary: *". Servers which omit it while varying 
the representation are non-compliant with HTTP already - thus the broken 
complainants have no ground to stand on.

However, if you were to propose an implicit Vary:ETag that would be 
another matter entirely and something I wholeheartedly support. Although 
the smart servers already do that anyway.

AYJ

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 17 Oct 2012 07:30:48 -0400

How much of the Web actually passes through client side proxies these days?
Of those, how many have  caches?

There was a good reason for caching proxies in 1993. I don't see much
justification or utility in fiddling with that part of the spec now. The
content that is relevant for caching these days is huge chunks of video,
audio and images, the part that is generated is text.

But in any case, I can't see why a server that cares can't generate the
vary header for the user agent header...

Bigger problem these days is that there are many sites that just break with
the 'wrong' user agent. All those clueless 'mobile' versions of sites that
don't actually link to any content. Just a page congratulating the
programmer on producing a mobile edition of the site instead of the linked
content.

Net is that even if a site specified the vary header on user agent, it is
probably best ignored just as I switched from safari to chrome on my iPhone
because it gives fewer stupid problems like that.

On Wed, Oct 17, 2012 at 3:32 AM, Mark Nottingham <[email-masked]> wrote:

> I understand there are security issues here caused by CORS, but that
> doesn't give you the ability to force changes on the rest of the Web.
>
> Regards,
>
>
> act as if all responses had Vary: User-Agent specified, since many pages on
> the Web serve different content depending on the User-Agent header but do
> not bother specifying Vary: User-Agent.
> >

-- 
Website: http://hallambaker.com/

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 18 Oct 2012 10:03:05 +1300

> Of those, how many have  caches?

Unknown, but still a large enough portion of the population to matter.

There are still countries like .au and .nz which have very high 
international transit costs or low bandwidth using large proxy farms in 
the major telecos. The historical reasons for caching have not 
disappeared, the bandwidth increases these late few decades have just 
spoilt the user base and added to the need if anything. Educational 
institutions and corporate sites are also still using proxies a lot as 
gateway control servers.

Ubuntu and Debian provide software install counters and inform me there 
are over half a million Squid, Varnish, HAproxy, polipo, oos still 
installed and running around the world - thats anything up to 500 M 
users right there. Other more popular distros, other proxies, unreported 
installs - who knows how big the total actually is.

> audio and images, the part that is generated is text.

With most of user generated www traffic being videos and media the case 
for caching those still stands and is gaining popularity amongst 
sysadmin.

AYJ

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 17 Oct 2012 17:48:17 -0400

My point was that in most cases it is only the text that varies.

While there is prior art for varied images back to 1993, this is still very
rare. A site may send links to different static content depending on who is
accessing the site but I suspect that the case where the image returned
from a URL changes is negligible.

On Wed, Oct 17, 2012 at 5:03 PM, Amos Jeffries <[email-masked]> wrote:

> international transit costs or low bandwidth using large proxy farms in the
> major telecos. The historical reasons for caching have not disappeared, the
> bandwidth increases these late few decades have just spoilt the user base
> and added to the need if anything. Educational institutions and corporate
> sites are also still using proxies a lot as gateway control servers.
>
> installed and running around the world - thats anything up to 500 M users
> right there. Other more popular distros, other proxies, unreported installs
> - who knows how big the total actually is.
>
>
>
> for caching those still stands and is gaining popularity amongst sysadmin.
>
> AYJ
>
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 18 Oct 2012 08:07:28 +0200

>From what I have seen two years ago at some mobile phone operators, I'd
say close to 100% of smartphones. And since smartphones become predominant
internet users, this is already a very high part. Not to mention enterprises,
schools and hotels.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 18 Oct 2012 08:11:29 +0200

> U-A strings on the web to something reasonable, lets say no more than 25 ?

One of my customers encodes them in logs to reduce log size, so basically
each new UA string gets assigned a 16-bit ID which is reused everytime the
same string is encountered. Last time we checked, we saw something like
30k different strings in one day worth of logs ! Some browsers emit all
the combinations of their plugins and versions, making the UA currently
useless.

Another benefit of getting much less different UAs is that their
identification string could significantly shrink then.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 18 Oct 2012 07:27:12 +0000

--------

>the combinations of their plugins and versions, making the UA currently

Many of the, now illegal in EU, tracking schemes use U-A as a major
component in disambiguation factor, for exactly this reason.

>identification string could significantly shrink then.

Personally I think the U-A is just plain wrong and should be dropped.