Subject: Re: [perpass] HTTP user-agent fingerprinting

From: Patrick Pelletier <[email-masked]>
Date: Fri, 13 Sep 2013 12:18:32 -0700

Forwarding this idea along to httpbis as you (Stephen) suggested.   
Although this could be retrofitted onto existing HTTP, not just  
httpbis, since it's merely recommending practices which are already  
legal in HTTP.

On Sep 13, 2013, at 5:17 AM, Stephen Farrell wrote:

> On 09/13/2013 04:12 AM, Patrick Pelletier wrote:
>> On 9/12/13 1:18 PM, Dave Crocker wrote:
>>
>>>    "privacy properties of IETF protocols and concrete ways in which
>>>     those could be improved."
>>
>> One obvious thing is the amount of (usually unnecessary) information
>> leaked by the User-Agent field in HTTP.
>>
>> Should we downgrade the User-Agent field (section 14.43 of RFC 2616)
>> from a SHOULD to a MAY?
>
> I think everyone finds those values problematic, and not only for
> privacy reasons. But yes, if you believe [1] then its probably the
> biggest contributor to browser fingerprinting that's in an IETF
> spec. (No idea if that site's evaluation is sound myself though.)
>
>   [1] https://panopticlick.eff.org/
>
>> Or, if that's too radical, should we standardize a small number of  
>> fixed
>> strings to use in the User-Agent field?  (For example, "Desktop/ 
>> 1.0" for
>> desktop browsers, "Mobile/1.0" for mobile browsers, "Text/1.0" for  
>> text
>> browsers like Lynx, "Batch/1.0" for non-interactive clients like curl
>> which are performing a task more specific than crawling the web, and
>> "Robot/1.0" for clients which are crawling the web?)
>
> Interesting. An IANA registry of those kinds of value might just end
> up like the UA string though, which also started out nice and simple.

I agree that things always start out simple and get messy.  However, I  
think there are some differences:

* The original User-Agent field was not designed with privacy in  
mind.  In fact, it was designed specifically to identify the product  
and version the user is using.  So, with a different goal (privacy  
first), we will hopefully get different results.

* By specifying only a single product token, omitting comments, and  
fixing the version number at 1.0, we've already eliminated a fair  
amount of information.  And then we further limit the information by  
making the product name not the actual name of the software, but  
merely a generic indication of the type of User-Agent; whatever is the  
minimal amount of information necessary for any legitimate browser  
sniffing that needs to occur.  (Such as differentiating desktop and  
mobile clients.)

And, of course, using the simplified User-Agent strings was just one  
of my two proposals.  My other proposal, which was even simpler,  
though perhaps more radical, was to downgrade the requirement on User- 
Agent from SHOULD to MAY, and encourage browsers not to send User- 
Agent at all.  (We could even change it to a SHOULD NOT if we feel  
really heavy-handed.)  One could argue that by using other techniques  
such as responsive layout, no browser sniffing should be necessary at  
all.

> Maybe ask this on httpbis if you don't get more feedback here? That's
> where you'd find folks who know if it could be done and who could do
> it.

--Patrick

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 13 Sep 2013 13:26:33 -0700

One could argue a lot of things, but disabling the often-used
and very useful User-Agent string (used by almost every site
to avoid known bugs and incompatible implementation by both
past and current browsers) is not going to happen.  Among other
things, it plays a role in CSRF security filters.

Hiding the User-Agent field-value would not have any impact
on browser fingerprinting.  Common browsers do not send unique
User-Agent strings (the TCP packeting is far more unique per
user than the UA string); non-browser tools do, but they aren't
a concern.  Using an extremely uncommon browser will, of course,
lead to more identifiable traces, but the same is true because of
how it uniquely composes requests, how it buffers sends, and
how it reacts to embedded subrequests.

In any case, the primary source for fingerprinting information
in browsers is the DOM interfaces, and I've seen very little to
suggest that browser developers are willing to remove them.

....Roy

On Sep 13, 2013, at 12:18 PM, Patrick Pelletier wrote:

> Forwarding this idea along to httpbis as you (Stephen) suggested.  Although this could be retrofitted onto existing HTTP, not just httpbis, since it's merely recommending practices which are already legal in HTTP.
> 
>>> Or, if that's too radical, should we standardize a small number of fixed
>>> strings to use in the User-Agent field?  (For example, "Desktop/1.0" for
>>> desktop browsers, "Mobile/1.0" for mobile browsers, "Text/1.0" for text
> I agree that things always start out simple and get messy.  However, I think there are some differences:
> 
> * The original User-Agent field was not designed with privacy in mind.  In fact, it was designed specifically to identify the product and version the user is using.  So, with a different goal (privacy first), we will hopefully get different results.
> 
> * By specifying only a single product token, omitting comments, and fixing the version number at 1.0, we've already eliminated a fair amount of information.  And then we further limit the information by making the product name not the actual name of the software, but merely a generic indication of the type of User-Agent; whatever is the minimal amount of information necessary for any legitimate browser sniffing that needs to occur.  (Such as differentiating desktop and mobile clients.)
> 
> And, of course, using the simplified User-Agent strings was just one of my two proposals.  My other proposal, which was even simpler, though perhaps more radical, was to downgrade the requirement on User-Agent from SHOULD to MAY, and encourage browsers not to send User-Agent at all.  (We could even change it to a SHOULD NOT if we feel really heavy-handed.)  One could argue that by using other techniques such as responsive layout, no browser sniffing should be necessary at all.
>

---

From: David Morris <[email-masked]>
Date: Fri, 13 Sep 2013 13:28:57 -0700 (PDT)

I think you'll have a very hard time convincing web developers that they
can solve the browser bug issues with this approach. The User-Agent string
is an awful hack, but it has clasically allowed creativity on web sites
which wouldn't be possible if the web sites were forced to conform to
the LCD. In addition, I've noticed that some plug-ins add their version
information to the UA String so that down stream processes will know
about the specific features.

Web developers need pretty specific knowledge about browsers and their
enviornment. I would be hard to convince that any alternative mechanism
which provided the platform target profile that web apps need wouldn't 
expose the user to the same or worse issues re. fingerprinting.

On Fri, 13 Sep 2013, Patrick Pelletier wrote:

> Forwarding this idea along to httpbis as you (Stephen) suggested.  Although
> this could be retrofitted onto existing HTTP, not just httpbis, since it's
> merely recommending practices which are already legal in HTTP.
> 
> * The original User-Agent field was not designed with privacy in mind.  In
> fact, it was designed specifically to identify the product and version the
> user is using.  So, with a different goal (privacy first), we will hopefully
> get different results.
> 
> * By specifying only a single product token, omitting comments, and fixing the
> version number at 1.0, we've already eliminated a fair amount of information.
> And then we further limit the information by making the product name not the
> actual name of the software, but merely a generic indication of the type of
> User-Agent; whatever is the minimal amount of information necessary for any
> legitimate browser sniffing that needs to occur.  (Such as differentiating
> desktop and mobile clients.)
> 
> And, of course, using the simplified User-Agent strings was just one of my two
> proposals.  My other proposal, which was even simpler, though perhaps more
> radical, was to downgrade the requirement on User-Agent from SHOULD to MAY,
> and encourage browsers not to send User-Agent at all.  (We could even change
> it to a SHOULD NOT if we feel really heavy-handed.)  One could argue that by
> using other techniques such as responsive layout, no browser sniffing should
> be necessary at all.
>

---

From: Karl Dubost <[email-masked]>
Date: Fri, 13 Sep 2013 16:49:49 -0400

Patrick Pelletier [2013-09-13T15:18]:
> encourage browsers not to send User-Agent at all.

It will not happen. This simple decision would break most of the Web. (Unfortunately). My daily work is to fight against bad user agent detection. It's so deep rooted in the Web infrastructure that even simplifying it takes a lot of energy.

Today I was going through the stats of a very high traffic web sites that shared with me the list of all unique UA strings they collected on 1 week. 

cat access-ua-log-7days.txt | wc -l
386 844

Yes… each of these are different. Some of them are just the usual one, such as 
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0"
with million of occurences

From looking at the file I have, there are things which can be improved though, some nasty things are done with some of the UA strings. With two areas worse than others:

* IE products such as 3rd party toolbars modifying the UA with a unique ID per user.
* some Mobile products with unique ID (most of the time put by Operators)

The spec could forbid it, but I guess it would be more a question of legal matter. Operators are sometimes using these unique ID for services.

btw it's not only User-Agent, there are secondary things such as 

X-Original-User-Agent
X-Device-User-Agent
Device-Stock-UA
X-OperaMini-Phone-UA

-- 
Karl Dubost
http://www.la-grange.net/karl/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 13 Sep 2013 20:51:30 +0000

In message <[email-masked]>, "Roy T. Fielding" w
rites:

>and very useful User-Agent string [...]

How about making it intelligently usable instead ?

Right now everybody wastes bandwidth claiming to be "Mozilla/5.0"
with "Mozilla/4.0" being a distant second:

	root@phk:/usr/local/www/logs # grep -c Mozilla/4.0 thttpd.log
	44445
	root@phk:/usr/local/www/logs # grep -c Mozilla/5.0 thttpd.log
	369977
	root@phk:/usr/local/www/logs # wc -l thttpd.log
	  520850 thttpd.log

with the result that those 12 bytes (incl the next SP) is just
a total waste of bandwidth.

HTTP/2.0 would be a great chance to stop this race to the bottom
where everybody sticks everything they can think of into User-Agent
in the hope that the dudes in the other end are incompetent enough
to actually cater for broken browsers.

If nothing else, putting a hard 32 byte limit on the string would
be a BIG improvement, since that would force people to transmit
only the necessary and useful information.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 13 Sep 2013 13:51:44 -0700

> suggest that browser developers are willing to remove them.

Speaking as someone actively involved in expanding the browser
fingerprinting surface, the DOM is definitely where the worst
fingerprinting problems exist.

That's not to say that the right response is to throw our hands up and
say "the Chinese are doing nothing about climate change, why should we
suffer hardship?"  Putting the fact that the premise is completely
wrong aside for a moment, there are things that can be done.  On the
down side, we're not really the right group to do it.  On the up side,
the browsers - who are the right people - already are doing something.

There aren't that many browsers in wide use.  Most of those
automatically update.  The number of fingerprinting bits available
from User-Agent if you use one of these browsers is actually very low.
 The value derived from those bits is simultaneously diminishing as
more capability detection moves to the DOM.

It may be that at some future point, the value of User-Agent
diminishes to the point that browsers will cease sending it (or they
will all send the same thing).  At which point it contains zero bits
of fingerprint entropy.  For the moment, it's still very useful in
some contexts, particularly mobile, so I suspect that it would very
hard to go cold turkey.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 13 Sep 2013 14:07:22 -0700

On Sep 13, 2013, at 1:51 PM, Poul-Henning Kamp wrote:

> In message <[email-masked]>, "Roy T. Fielding" writes:
> 

These concerns are already addressed (to the extent we can) in p2.

Nobody is being forced to send long UA header fields.
I also find it silly to send TMI, particularly when browser
folks complain about request latency (sending and parsing
the UA string is always in the critical path).  However,
there is a huge difference between TMI and sending enough
information for a server to identify and work around
known protocol issues.

....Roy

---

From: Karl Dubost <[email-masked]>
Date: Fri, 13 Sep 2013 17:13:35 -0400

trimming -perpass

> 	root@phk:/usr/local/www/logs # wc -l thttpd.log

You can add

Dalvik/***
Opera/9.80 (kept for Web compatibility reasons)

Note that anything which is removed will break something.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Fri, 13 Sep 2013 14:22:22 -0700

Speaking as a browser vendor, we do try to limit the info we send in the UA
string. It's sadly difficult. Please refer to
http://webaim.org/blog/user-agent-string-history/. Changing the UA string
breaks web compat, and when the end user's favorite site foo.com fails to
render in browser X, the end user switches to browser Y.

For a fairly recent discussion of this on the Chromium developer's mailing
list, feel free to refer to
https://groups.google.com/a/chromium.org/forum/#!msg/chromium-dev/N78vBMOZlF8/tQ-xnJV1lVsJ
.

On Fri, Sep 13, 2013 at 12:18 PM, Patrick Pelletier <
[email-masked]> wrote:

>  Although this could be retrofitted onto existing HTTP, not just httpbis,
> since it's merely recommending practices which are already legal in HTTP.
>
> user is using.  So, with a different goal (privacy first), we will
> hopefully get different results.
>
> * By specifying only a single product token, omitting comments, and fixing
> the version number at 1.0, we've already eliminated a fair amount of
> information.  And then we further limit the information by making the
> product name not the actual name of the software, but merely a generic
> indication of the type of User-Agent; whatever is the minimal amount of
> information necessary for any legitimate browser sniffing that needs to
> occur.  (Such as differentiating desktop and mobile clients.)
>
> And, of course, using the simplified User-Agent strings was just one of my
> two proposals.  My other proposal, which was even simpler, though perhaps
> more radical, was to downgrade the requirement on User-Agent from SHOULD to
> MAY, and encourage browsers not to send User-Agent at all.  (We could even
> change it to a SHOULD NOT if we feel really heavy-handed.)  One could argue
> that by using other techniques such as responsive layout, no browser
> sniffing should be necessary at all.
>

---

From: Karl Dubost <[email-masked]>
Date: Fri, 13 Sep 2013 17:43:00 -0400

Patrick,

As William Chan (陈智昌) [2013-09-13T17:22] said:
> Changing the UA string breaks web compat, and when the end user's favorite site foo.com fails to render in browser X, the end user switches to browser Y.

I guess I could write yet another article to explain the 1001 ways a Web site breaks because of UA strings. :) but in the mean time background reading

https://hacks.mozilla.org/2013/09/user-agent-detection-history-and-checklist/
https://wiki.mozilla.org/Compatibility/Mobile
https://wiki.mozilla.org/Compatibility/Mobile/WipeOutUAOverides
http://my.opera.com/community/openweb/
http://www.opera.com/docs/browserjs/

You can help fix some of these, it's a required experience for understanding the insanity going on around ;)
https://bugzilla.mozilla.org/describecomponents.cgi?product=Tech%20Evangelism

That's my job for the last 3 years. First at Opera, then now at Mozilla. :)

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 13 Sep 2013 21:55:08 +0000

In message <[email-masked]>, Karl Dubost wr

>Note that anything which is removed will break something. 

Yes, undoubtedly.

But if HTTP/2.0 limits User-Agent to 32 bytes, that means that
no HTTP/2.0 browser will send more than 32 bytes, and no website
will support HTTP/2.0 until they can do their job with just 32 bytes
of User-Agent.

HTTP/2.0 transition is a unique chance for getting this monster
under control, we should not waste it.

---

From: Fred Akalin <[email-masked]>
Date: Fri, 13 Sep 2013 15:18:41 -0700

> under control, we should not waste it.

It seems more likely that the end result of this would just be that
websites won't support HTTP/2.0.

---

From: James M Snell <[email-masked]>
Date: Fri, 13 Sep 2013 15:41:46 -0700

An alternative possible approach could build on the typed codec work..
that is, use compact opaque binary values for user agent as an
alternative to the current string-based value. For those who need the
backwards compatibility, the legacy string values would still be
available and usable.. but we'd have a simple transition path for
something better.

On Fri, Sep 13, 2013 at 3:18 PM, Fred Akalin <[email-masked]> wrote:
> It seems more likely that the end result of this would just be that websites
> won't support HTTP/2.0.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 13 Sep 2013 15:48:18 -0700

> available and usable.. but we'd have a simple transition path for

As it pertains to tracking, I don't see an opaque binary blob to have
characteristics different from a string.  It does make the identifier
less useful for the aforementioned purposes, which leads me to
conclude that something like that would be lose-lose.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 13 Sep 2013 15:55:17 -0700

There are conflicting goals here:
1) Modify the presentation of a site based on the user-agent.
2) Ensure that pages are using feature-probing instead of assumptions based
on the user-agent.

There are valid reasons for both.

I'm ambivalent because I can't think of a good solution that resolves the
conflict in a manner that is useful.
Feature probing is great, but can cause harm (i.e. some features crash
browsers).

-=R

---

From: Leif Hedstrom <[email-masked]>
Date: Mon, 16 Sep 2013 09:00:24 -0600

> will support HTTP/2.0 until they can do their job with just 32 bytes

I imagine that for some time during HTTP/2 early deployments, the primary server technology would include HTTP/2.0 -> HTTP/1.1 proxies. As such, those same HTTP/1.1 servers would presumably still require the same UA fingerprinting as they do today.

From prior experiences, most IPv6 deployments I've done used v6tov4 proxies. We would roll v6 out on a small number of servers, yet still provide the feature for all existing IPv4 services. It seems reasonable to think that HTTP/2 would be deployed (initially) in a similar fashion?

Yes, I know, the HTTP/2->HTTP/1.1 proxies could rewrite the UA. :) I probably would if we changed the UA requirements (e.g. 32-byte size limit).

Cheers,

-- Leif

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 16 Sep 2013 18:34:57 +0200

Le Ven 13 septembre 2013 21:18, Patrick Pelletier a écrit :

> really heavy-handed.)

Please don't. If any change it from SHOULD to MUST

User-Agent is invaluable for filtering out pathologic web clients in a
network without bothering legitimate users. And in fact 9 web clients out
of ten that do not declare user-agent are broken one way or another
protocol-wise (and they're a PITA because you can't filter them out
without breaking the few correctly implemented clients that take advantage
of the SHOULD)

In fact no-user-agent should be "I swear on the penalty of [insert cruel
and deserved punishment] to never take liberties with the HTTP protocol
and never add a bug to my software".

Right now no-user-agent-is "I don't want to understand HTTP but it gets me
through the firewall, simulate a real web client and only implement the
parts I need most of the time with no error handling except for retrying
in a loop with no sleeps"

-- 
Nicolas Mailhot

---

From: Karl Dubost <[email-masked]>
Date: Mon, 16 Sep 2013 14:42:07 -0400

Nicolas Mailhot [2013-09-16T12:34]:
> User-Agent is invaluable for filtering out pathologic web clients in a network without bothering legitimate users.

* At which level do you need to filter out? (proxy, server, …)
* Which type of clients? (bot, browser, etc?)
* Why do you need to filter out?
* What are you looking for in the UA string?

/me is interested to know. I'm kind of collecting how the UA string is used out in the wild in good and bad ways.

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 16 Sep 2013 21:59:59 +0200

Le Lun 16 septembre 2013 20:42, Karl Dubost a écrit :
>> network without bothering legitimate users.
>

proxy

> * Which type of clients? (bot, browser, etc?)

pretty much everything that tries to access the web in http(s)

> * Why do you need to filter out?

because the internet got so fast web client authors do pretty idiotic
things at the http level and it's lost in the youtube noise. OTOH those
idiotic things tend to clutter my logs with errors that make investigating
actual problems difficult, so it's easier to just blacklist miscreants

Major browsers almost never cause problems, but pretty much anything else
that does http is likely to make shortcuts

> * What are you looking for in the UA string?

Anything likely to identify the problem client and nothing else. For example:
1. corp policy is that windows stations must go through corp wsus →
blacklist windows update UA ;
2. something is hammering the proxy with corrupted connects (control chars
before CONNECT) → it says it is "Google Mail" → go away Google Mail,
3. someone deployed a widget that wants to go to the internet every 5
minutes to display a clock no one will look at → kill clock (not ntp, we
have perfectly fine ntp servers)

Actually, I don't care about the volume as long as it's not pathological
real-time updates, what I do care is the flood of errors that makes our
job more difficult than it should be.

>
> /me is interested to know. I'm kind of collecting how the UA string is
> used out in the wild in good and bad ways.
>

-- 
Nicolas Mailhot

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 17 Sep 2013 10:56:09 +1000

This has been discussed ad nauseum in the communities it affects; folks really need to read the links below as well as the related browser bugs before discussing here.

See also:
  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/232

Cheers,

--
Mark Nottingham   http://www.mnot.net/