Subject: Re: SPDY Header Frames

From: "Nicolas Mailhot" <[email-masked]>
Date: Tue, 17 Jul 2012 20:50:41 +0200

Julian Reschke <julian.reschke@...> writes:

>
> On 2012-07-17 15:38, Poul-Henning Kamp wrote:

> > There must be a smarter way than "User-Agent:"...
>
> Actually one nice potential optimization is if the server can declare
> that it's not interested in the User-Agent at all; see
> <http://tools.ietf.org/html/draft-nottingham-http-browser-hints-03#section-5.7>

The server may not be interested by intermediaries may still be

(while ugly user-agent special-casing is quite useful for proxy operators
that have to contend with web clients that were never really tested with
proxies and misbehave big way)

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jul 2012 21:03:25 +0200

> proxies and misbehave big way)

Could you elaborate? What kind of misbehavior are you referring to?

Best regards, Julian

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Tue, 17 Jul 2012 21:23:36 +0200

> Could you elaborate? What kind of misbehavior are you referring to?

Typical misbehaviour is inability to cope with intermediary-inserted
redirect and error codes, and retrying in a loop (because the developer
could not bother to write error handling code, considers that all errors
on the Internet are transient, so with enough retrying he will get
whatever he expected to get at first)

Then you get the web clients that try to interpret error pages as if they
were whatever they expected to get at first, with sometimes weird results.

Then you get the 'pump as much as you can' web client that will starve
everything else.

It's usually less invasive to blacklist or limit just the specific
troublesome client instead of targeting the URLs it tries to access or the
system it runs on (because the buggy part is the web client,the user who
installed it may do legitimate work with other web clients at the same
time, the broken web client will misbehave the same way tomorrow with
other web sites, and this way the protection is in place before other
systems are infected with brokenware)

I really hope HTTP/2 makes intermediaries first-call citizens and
clarifies intermediary/web client signalling so such things happen less
often in the http/2 future (though bugs will always exist, so a client
signature to home on is nice)

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Tue, 17 Jul 2012 21:30:40 +0200

> Then you get the 'pump as much as you can' web client that will starve

(lastly in a corporate environment systems are locked down, apps are
supposed to only come in versions the heldesk know to deal with, which has
not stopped ISVs from multiplying the web updaters that run sneakily in
the background and try to pump new buggy updates no one requested. When we
find the user agent of such a beast it ends in the blacklist directly –
much less trouble than trying to follow which update web site they use
this week)

-- 
Nicolas Mailhot

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 19:49:59 +0000

In message <[email-masked]>, "Nicol
as Mailhot" writes:

>so a client signature to home on is nice.

Agreed, we can't do without some form of User-Agent, if nothing else
then for debug purposes, but if we could make it less of landfill
and more meaningfull that would be A Good Thing.

Anybody know what WAP did for this ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Karl Dubost <[email-masked]>
Date: Tue, 17 Jul 2012 16:06:19 -0400

Nicolas, 

interesting but I would like to understand the issues you are having.

Le 17 juil. 2012 à 15:23, Nicolas Mailhot a écrit :
> Typical misbehaviour is inability to cope with intermediary-inserted redirect and error codes, and retrying in a loop 

Do you have a precise example of such an issue?

> Then you get the web clients that try to interpret error pages

which clients?

> Then you get the 'pump as much as you can' web client that will starve everything else.

which clients, which circumstances?

> troublesome client

seen from the client side. Note that it happens quite often that people use a "regular user agent" to run a spambot. So the user agent string is not an absolute signature of the user agent behind.

-- 
Karl Dubost - http://dev.opera.com/
Developer Relations, Opera Software

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 13:05:27 -0700

Not sure what WAP's strategy for this is but, off the top of my head, given
how User-Agent's are currently used (and abuse), one approach may be to
redefine User-Agent to limit it to a single product token and version, and
define a new Capabilities header whose value is a list of meaningful tokens
that identify stuff the user-agent is able to support.

  User-Agent: Foo/1.0
  Capabilities: u, mobile, something-else

Using the revised header structure I've suggested, it would be something
like...

  [0, 9, 1, 0, 7, 'F', 'o', 'o', '/', '1', '.', '0']

  [16, 2, 1, 23, 'u', 0, 'm', 'o', 'b', 'i', 'l', 'e',
    0, 's', 'o', 'm', 'e', 't', 'h', 'i', 'n', 'g', '-',
   'e', 'l', 's', 'e']

This definitely won't stop folks from lying about their user agents, but by
providing a clear location for "capabilities", perhaps we can stem many of
the ugly hacks that user-agent strings are currently used for.

- James

---

From: Roland Zink <[email-masked]>
Date: Tue, 17 Jul 2012 22:15:56 +0200

WAP did stick with the user-agent header. WAP has an extension named 
user agent profile sending a device profile URL in a header. This allows 
vendors to place an XML description of the clients capabilities on the 
Internet, see 
http://www.openmobilealliance.org/Technical/release_program/uap_v2_0.aspx. 
The client is able to add/overwrite properties using the 
x-wap-profile-diff header.

Regards,
Roland

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 20:18:14 +0000

In message <CABP7RbftZjB3dKTroubA1xGtrGX5dh9+=Y1T-s=[email-masked]>

>  Capabilities: u, mobile, something-else

Isn't that just moving the problem to a different header ?

I guess making the Capabilities a Registry could stem the tide
a bit, but do we (in this WG) even have a clue about what
capabilities make sense for the initial set ?

I guess the WURFL database can be used for a clue about what
capabilities would be interesting, but without any slight or malice
intended, it seems slightly overengineered, in that good old german
fashion we all love in cars and cameras.

Is that a task we could fork out to another more web-app centric
WG or maybe W3c ?

---

From: Karl Dubost <[email-masked]>
Date: Tue, 17 Jul 2012 16:24:24 -0400

Le 17 juil. 2012 à 16:05, James M Snell a écrit :
> given how User-Agent's are currently used (and abuse),

I have plenty of horror stories, if you want me to share and help understand the problem space. (It's partly my daily job to deal with those.)

> define a new Capabilities header whose value is a list of meaningful tokens that identify stuff the user-agent is able to support.
> 
>   User-Agent: Foo/1.0

I have the feeling that will not really work. 
In the browser world, the way we recommend to do things lately is feature detection [1]. APIs are complex and not always implemented fully at the same time. So Javascript developers can programmatically determine if the device has this or that capability, down sometimes to one specific property of an API.

The "Capabilities" would become huge if we wanted to it an exact picture of the user agent capabilities. So that would not really scale.

On the other side, I see another reason why "user agent maker" will want to keep their identity visible: market shares statistics (Faust). Don't want to be put apart/Want to have an existence.

---

From: Karl Dubost <[email-masked]>
Date: Tue, 17 Jul 2012 16:26:37 -0400

Le 17 juil. 2012 à 16:18, Poul-Henning Kamp a écrit :
> capabilities 

Any user agent capabilities Databases are future fail.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 20:26:33 +0000

In message <[email-masked]>, Roland Zink writes:

>WAP has an extension named 

I like it.

Lets replace
	 "User-Agent: <sedimentary garbage>"
with
	 "Capabilities: <url>"

And fork of the job of defining what's at the end of the URL
to somebody else (WG/W3C) already now.

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Tue, 17 Jul 2012 22:34:07 +0200

Le Mar 17 juillet 2012 22:06, Karl Dubost a écrit :
>> redirect and error codes, and retrying in a loop
>

we have seen broken web clients trying to get the proxy to proxify its own
auth portal (messy loop especially when the load balancer moved the client
to another proxy, so the proxies started looping over one another, fat web
client written in delphi or another obsolete windows tech, we protect
against this pattern now), clients that retried the same access several
thousand times a second all day round (stuffed our logs and made resolving
an actual incident that happened concurrently hard) infra load-testing
when akamai launched its hd service and users pushed us to he limit (still
investigating how to best block the akamai P2P video downloader), some
google chrome service that really could not accept the proxy refusal (not
chrome itself), windows widgets that thought wall-time was better queried
over the internet (it's nice to learn a computer can't emulate a basic
clock without the web) etc

Though most offenders do not bother to declare a user-agent at all which
is why I'm currently getting trying to get approval to block any access
attempt without user agent header

> seen from the client side. Note that it happens quite often that people
> use a "regular user agent" to run a spambot. So the user agent string is
> not an absolute signature of the user agent behind.

spambots either do not know how to authentify over the proxy (not because
of the evil way we do proxy auth, for lack of a good standard, but because
they don't know the user password), are caught by its anti-malware
scanning (for http), the IPS probes, or the evil-sites list provided by
the manufacturer (that's why full-url-in-clear is useful at the
intermediary level, with only the ip or hostname security equipments would
not catch bot access patterns)

-- 
Nicolas Mailhot

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Jul 2012 22:40:47 +0200

On 2012-07-17 22:15, Roland Zink wrote:
> ...

OMG; that sounds so sane that the only thing I can complain about is the 
"x-" prefixed header field :-)

---

From: Karl Dubost <[email-masked]>
Date: Tue, 17 Jul 2012 17:01:23 -0400

Le 17 juil. 2012 à 16:40, Julian Reschke a écrit :
> OMG; that sounds so sane that the only thing I can complain about is the "x-" prefixed header field :-)

:) wait a minute

GET / HTTP/1.1
Host: www.example.org
Capabilities: <http://example.com/acmeCom/new/shiny/browser/23>

What is happening when example.org meets the client

1.  for the first time, example.org might download the URI which might be huge. It means big latency for replying to the client with the right content.

2. to avoid that example.org will be clever, they will cache the capabilities file and do negotiation on the URI becoming an ID.

3. Some business will start by selling list of URIs and capabilities files (the thing which is happening), create Databases.

* Some business will be abandoned. Databases will not be updated. Future fail
* The cache system will not be maintained. future fail.
* Some scripts will use the URL-ID (new useragent string) to block, filter, redirect.

Basically exactly back to the same situation we have today.

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 14:08:17 -0700

+1... I was getting ready to respond to Julian's post with a "+1 YEAH!" but
then stopped and thought about it... using a URI, while potentially sound
in theory, would likely just end up with a different form of the same mess
we're in now. Reducing the User-Agent field to nothing more than a single
token with a version identifier seems to me to be the Least Bad Option.

- James

On Tue, Jul 17, 2012 at 2:01 PM, Karl Dubost <[email-masked]> wrote:

> 1.  for the first time, example.org might download the URI which might be
> huge. It means big latency for replying to the client with the right
> content.
>
> 2. to avoid that example.org will be clever, they will cache the
> capabilities file and do negotiation on the URI becoming an ID.
>
> 3. Some business will start by selling list of URIs and capabilities files
> (the thing which is happening), create Databases.
>
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 21:10:22 +0000

In message <[email-masked]>, Karl Dubost writes

>* Some business will be abandoned. Databases will not be updated.

Too bad ?  It's not like User-Agent strings are always correct
either...

>* The cache system will not be maintained. future fail.

We cannot out-standardize incompetence.

>* Some scripts will use the URL-ID (new useragent string) to block,

And what is the harm in this ?  If it works, it works, if it doesn't
work, it only hurts the website which took the shortcut...

>Basically exactly back to the same situation we have today.

With one very big difference:  We stop the User-Agent string
from growing another 10 characters every year...

I still think it is a very superior solution to User-Agent,
but it can obviously be modified in various ways to address
some of the concerns you raise (make it IANA registry, 
make it an ISO OID, make a repository for the files etc.)

But given that there is stuff and standards to levarage from
WAP I think we should just make it an URL and leave the
problems to other people to solve.

---

From: Karl Dubost <[email-masked]>
Date: Tue, 17 Jul 2012 17:15:32 -0400

Le 17 juil. 2012 à 17:10, Poul-Henning Kamp a écrit :
> if it doesn't work, it only hurts the website which took the shortcut...

nope. It hurts

1. the users 
2. then the blocked user agent market share.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 17 Jul 2012 21:26:39 +0000

In message <[email-masked]>, Karl Dubost writes

>2. then the blocked user agent market share.

And this is different from today in what way ?

The problem we are trying to solve is transmission efficiency
not that people are morons od make mistakes...

---

From: James M Snell <[email-masked]>
Date: Tue, 17 Jul 2012 14:35:10 -0700

> not that people are morons od make mistakes...
If it's only about transmission efficiency, then using a URI doesn't really
help matters much, to be honest. If we specified the use of a URI, more
often than not, the URI would be used as nothing more than an opaque
identifier at best. Dereferencing the URI would quickly become viewed as
Something Not To Do. If an application needs anything more than a simple
identifier, they always have the option of using an extension header.

- James

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 18 Jul 2012 10:24:22 +1000

Just a clarification (without commenting on what roles intermediaries should play) --

The intent of the small-hdrs hint is NOT to omit the UA header;

>    o  Description: When true, this hint indicates that clients can omit
>       the Accept and Accept-Charset request headers when communicating
>       with the resource, and that they can use a shortened version of
>       the User-Agent header.

This is because browser vendors tend to emit a long (and somewhat unnatural) UA header because any significant changes will hurt interop. small-hdrs says "don't worry, I'm not doing weird browser sniffing on historic UA strings, so send me a compact, simple UA" -- not "don't send me a UA".

This may be an opportunity to introduce a profile of the UA header, but I didn't attempt that in the current draft (hoping that browser implementers can figure out a shorter UA header on their own).

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 18 Jul 2012 10:41:27 +0900

> token with a version identifier seems to me to be the Least Bad Option.

So why did that not happen up to now, and why is the User-Agent string 
growing as it does?

Because servers/proxies check on substrings in that string, so UAs are 
using the strategy to copy the whole header of some other browser 
(usually the most popular one around at that time) and then tack some 
additional stuff on at the back to make sure they can be uniquely 
identified.

This is a cat-and-mouse game where the servers are as much (or as 
little) to blame as the browsers. As long as each side does what's most 
convenient for them, changing the name of the header, or offloading the 
data to a separate URI, or any such thing, is only a cosmetic change.

Regards,   Martin.

---

From: Anil Sharma <[email-masked]>
Date: Wed, 18 Jul 2012 11:33:32 +0000

I was trying to understand impact of HTTP2.0 or SPDY on intermediaries.

I think Metadata will always be in cleartext so redirect or proxies will still be low cost operation..

Even if major traffic sources such as Google, Facebook and Twitter go the TLS route, I think valid intermediaries (SPs doing traffic classification, traffic policy control, CDNs doing embedded video optimization) can always use TLS proxy functionality to achieve their goal.

Not sure if I understood the impact perfectly yet but I think intermediaries won't be severely impact even if most of the major players chose to use TLS....

Anil 

-----Original Message-----
From: Nicolas Mailhot [mailto:[email-masked]] 
Sent: Wednesday, July 18, 2012 12:54 AM
To: Julian Reschke
Cc: Nicolas Mailhot; HTTP Working Group
Subject: Re: Minimizing/avoiding User-Agent, was: SPDY Header Frames

-- 
Nicolas Mailhot

---

From: James M Snell <[email-masked]>
Date: Wed, 18 Jul 2012 08:11:11 -0700

On Tue, Jul 17, 2012 at 6:41 PM, "Martin J. Dürst"
<[email-masked]>wrote:

> using the strategy to copy the whole header of some other browser (usually
> the most popular one around at that time) and then tack some additional
> stuff on at the back to make sure they can be uniquely identified.
>
> This is a cat-and-mouse game where the servers are as much (or as little)
> to blame as the browsers. As long as each side does what's most convenient
> for them, changing the name of the header, or offloading the data to a
> separate URI, or any such thing, is only a cosmetic change.
>
>
As has been mentioned several times, we cannot stop people from doing
stupid things. We can, however, make it easier to do the right thing and
work on setting a right precedent. Placing additional constraints on the
User-Agent is the right thing to do. If implementors go off and create a
new header to do the same stupid kinds of things they've been doing, then
that's on them.

- James