Subject: [#232] User-Agent Guidelines (proposal)

From: Mark Nottingham <[email-masked]>
Date: Mon, 20 Sep 2010 19:14:48 +1000

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/232>

UA strings have become quite long, which makes requests bigger than necessary, can reveal too much information about the user (including allowing them to be "fingerprinted"), and potentially can encourage inappropriate sniffing.

Guidance should be provided regarding what's appropriate / inappropriate in a UA string, and the privacy aspect should be added to Security Considerations.

See also:

 https://bugzilla.mozilla.org/show_bug.cgi?id=572650 
 http://blogs.msdn.com/b/ie/archive/2010/03/23/introducing-ie9-s-user-agent-string.aspx 
 https://panopticlick.eff.org/

Current text <http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-11#section-9.9>:

---8<---

The "User-Agent" request-header field contains information about the user agent originating the request.  This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations.

User agents SHOULD include this field with requests.  The field can contain multiple product tokens (Section 6.3 of [Part1]) and comments (Section 3.2 of [Part1]) identifying the agent and any subproducts which form a significant part of the user agent.  By convention, the product tokens are listed in order of their significance for identifying the application.

--->8---

Proposal:

---8<---

The "User-Agent" request-header field contains information about the user agent originating the request.  User agents SHOULD include this field with requests.  

Typically, it is used for statistical purposes, the tracing of protocol violations, and tailoring responses to avoid particular user agent limitations.

The field can contain multiple product tokens (Section 6.3 of [Part1]) and comments (Section 3.2 of [Part1]) identifying the agent and its significant subproducts.  By convention, the product tokens are listed in order of their significance for identifying the application.

Because this field is usually sent on every request a user-agent makes, implementations are encouraged not to include needlessly fine-grained detail, and to limit (or even prohibit) the addition of subproducts by third parties. Overly long and detailed User-Agent field values make requests larger and can also be used to identify ("fingerprint") the user against their wishes.

Likewise, implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them, as this circumvents the purpose of the field. Finally, they are encouraged not to use comments to identify products; doing so makes the field value more difficult to parse.

--->8---

Current text <http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-11#section-11.1>:
  
---8<---

The User-Agent (Section 9.9) or Server (Section 9.8) header fields can sometimes be used to determine that a specific client or server have a particular security hole which might be exploited. Unfortunately, this same information is often used for other valuable purposes for which HTTP currently has no better mechanism.

--->8---

Proposal: 

---8<---

The User-Agent (Section 9.9) or Server (Section 9.8) header fields can sometimes be used to determine that a specific client or server have a particular security hole which might be exploited. Unfortunately, this same information is often used for other valuable purposes for which HTTP currently has no better mechanism.

Furthermore, the User-Agent (Section 9.9) header field may contain enough entropy to be used, possibly in conjunction with other material, to uniquely identify the user.

--->8---

--
Mark Nottingham     http://www.mnot.net/

---

From: Adam Barth <[email-masked]>
Date: Mon, 20 Sep 2010 02:24:53 -0700

On Mon, Sep 20, 2010 at 2:14 AM, Mark Nottingham <[email-masked]> wrote:
> Likewise, implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them, as this circumvents the purpose of the field.

Although noble, many implementations will ignore this advice.  I can
tell you that there are several folks in the Chromium project that are
adamant about fixing the User-Agent mess, but they're crushed by the
compatibility requirements.  In particular, including the token
"Safari" in Chrome's User-Agent string is massively important to
making the browser actually work in the real world.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 20 Sep 2010 19:27:09 +1000

> making the browser actually work in the real world.

Yes, I think that's probably the most tenuous part of the proposal -- and one reason why it's just "encouraged." 

OTOH, if things are really that dire, maybe we should standardise the value to start with "Mozilla/4.0" and just move on ;)

Cheers,

---

From: Adam Barth <[email-masked]>
Date: Mon, 20 Sep 2010 02:35:40 -0700

> OTOH, if things are really that dire, maybe we should standardise the value to start with "Mozilla/4.0" and just move on ;)

Some of us still hold onto the dream that this can be fixed at some point.  :)

Adam

---

From: Julian Reschke <[email-masked]>
Date: Mon, 20 Sep 2010 11:59:07 +0200

On 20.09.2010 11:27, Mark Nottingham wrote:
>> On Mon, Sep 20, 2010 at 2:14 AM, Mark Nottingham<[email-masked]>  wrote:
> ...

I think it's good advice, so I'd be in favor of leaving it in.

Best regards, julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Mon, 20 Sep 2010 13:58:19 +0200

On Mon, 20 Sep 2010 11:14:48 +0200, Mark Nottingham <[email-masked]> wrote:
> [...]
>
> Because this field is usually sent on every request a user-agent makes,  
> implementations are encouraged not to include needlessly fine-grained  
> detail, and to limit (or even prohibit) the addition of subproducts by  
> third parties. Overly long and detailed User-Agent field values make  
> requests larger and can also be used to identify ("fingerprint") the  
> user against their wishes.

Having this is great I think. I think we should also give advice to people  
using the header (i.e. against user-agent sniffing), similar to what HTML5  
does for navigator.userAgent:

http://whatwg.org/C#client-identification

(HTML5 should probably also mention the fingerprinting issue.)

-- 
Anne van Kesteren
http://annevankesteren.nl/