Subject: Proposal: Cookie Priorities

From: Mike West <[email-masked]>
Date: Thu, 3 Mar 2016 16:58:06 +0100

Way back in 2013, folks at Google put together a proposal for a `priority`
attribute for cookies with the intent of allowing servers to influence a
user agent's retention policy[1]. Chrome has been shipping this feature
since ~November 2013[2], and Google servers have been using it since then.
It would be lovely to get more feedback on the concept from other folks
outside the company, so I've just submitted a copy/pasted version of the
original proposal[3] as
https://tools.ietf.org/html/draft-west-cookie-priority-00. Apologies for
the years of delay. :/

Like many other excitingly huge companies, Google has both internal and
external servers hosted on subdomains of `google.com`, and employees hit
the user agent's cookie retention limit on a regular basis. In order to
insure that this doesn't result in lost sessions, Google marks certain
cookies as `Priority=High`, and others as `Priority=Low`. As you might
imagine, the latter are evicted more frequently than regular cookies, the
former less frequently. The document describes how Chrome takes these
priorities into account when evicting cookies from the cookie store.
Anecdotally, folks internally have found it quite helpful in terms of
retaining session state.

There's still some work to do to bring the document up to date with
proposals like
https://tools.ietf.org/html/draft-ietf-httpbis-cookie-alone-00 which also
aim to alter the browser's eviction policy. I'm working through the
implications of that document on this proposal in Chrome right now, and
will document whatever merger we end up considering sane once we figure out
what it it might be. :)

One of the original authors (Erik) has left Google, and I haven't been
successful at getting in contact with him: I'm hopeful that we can get him
involved again. Regardless, Samuel and I would be thrilled to hear what
this group thinks of the proposal.

Thanks!

[1]:
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/xK4IJ1-5oJE
[2]: https://codereview.chromium.org/54303010
[3]:
https://docs.google.com/a/google.com/file/d/0B3o1IlTKoADVRllKWGlyWGxIVTg/edit

-mike

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Mar 2016 11:02:55 +1100

Thanks, Mike.

As I understand it, this is already implemented in one browser, which is good in that we're looking for implementation.

What do folks -- both other browser implementers and site folks -- think about this?

Regards,

> On 4 Mar 2016, at 2:58 AM, Mike West <[email-masked]> wrote:
> 
> Way back in 2013, folks at Google put together a proposal for a `priority` attribute for cookies with the intent of allowing servers to influence a user agent's retention policy[1]. Chrome has been shipping this feature since ~November 2013[2], and Google servers have been using it since then. It would be lovely to get more feedback on the concept from other folks outside the company, so I've just submitted a copy/pasted version of the original proposal[3] as https://tools.ietf.org/html/draft-west-cookie-priority-00. Apologies for the years of delay. :/
> 
> Like many other excitingly huge companies, Google has both internal and external servers hosted on subdomains of `google.com`, and employees hit the user agent's cookie retention limit on a regular basis. In order to insure that this doesn't result in lost sessions, Google marks certain cookies as `Priority=High`, and others as `Priority=Low`. As you might imagine, the latter are evicted more frequently than regular cookies, the former less frequently. The document describes how Chrome takes these priorities into account when evicting cookies from the cookie store. Anecdotally, folks internally have found it quite helpful in terms of retaining session state.
> 
> There's still some work to do to bring the document up to date with proposals like https://tools.ietf.org/html/draft-ietf-httpbis-cookie-alone-00 which also aim to alter the browser's eviction policy. I'm working through the implications of that document on this proposal in Chrome right now, and will document whatever merger we end up considering sane once we figure out what it it might be. :)
> 
> One of the original authors (Erik) has left Google, and I haven't been successful at getting in contact with him: I'm hopeful that we can get him involved again. Regardless, Samuel and I would be thrilled to hear what this group thinks of the proposal.
> 
> Thanks!
> 
> [1]: https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/xK4IJ1-5oJE
> [3]: https://docs.google.com/a/google.com/file/d/0B3o1IlTKoADVRllKWGlyWGxIVTg/edit
> 
> -mike

--
Mark Nottingham   https://www.mnot.net/

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 4 Mar 2016 10:52:56 +1000

On 04/03/2016 10:09 AM, "Mark Nottingham" <[email-masked]> wrote:
>
> Thanks, Mike.
>
> As I understand it, this is already implemented in one browser, which is
good in that we're looking for implementation.
> What do folks -- both other browser implementers and site folks -- think
about this?

It's a practical issue with a simple (demonstrated) improvement. And hints
like this usually fall under "handy at best, harmless at worst" so I see no
harm in and of itself.

That said, maybe I've been listening to PHK for too long, but I wonder if
all these cookie patches aren't just putting band-aids over a fundamentally
flawed system. How many (and how big) cookies do you need to receive before
this priority comes to the fore?

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Mar 2016 11:56:35 +1100

> On 4 Mar 2016, at 11:53 AM, Martin Thomson <[email-masked]> wrote:
> 
> On 4 March 2016 at 11:02, Mark Nottingham <[email-masked]> wrote:
> This is a pretty nice hole Google dug for themselves.  Though I have
> heard the same from folks at other similarly large and crufty
> organizations; it's a real problem.

FWIW - this has been my observation as well (explicitly not pointing fingers :)

> I have a small suggestion:
> 
> if (request.url.scheme == 'http') {
>  cookie.priority = 'floor';
> }
> 
> Related story, I believe that some of those people run servers that
> forcibly evict all cookies other than those on a small whitelist to
> prevent this sort of craziness.  That turns out to have beneficial
> properties.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 4 Mar 2016 11:53:23 +1100

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 7 Mar 2016 15:28:35 +1300

On 4/03/2016 1:52 p.m., Matthew Kerwin wrote:
> On 04/03/2016 10:09 AM, "Mark Nottingham" wrote:
>>
>> Thanks, Mike.
>>

Oh, its absolutely a battered, abused, and broken mechanism. I am
doubtful that many would even attempt to argue the contrary.

My take has been from the beginning that we should start pruning away
pieces of Cookie to prevent bad usages a much as possible.

This particular baindaid proposal looks like it might double as a nice
way to allow any recipient or relay to proactively prune away the
low-priority Cookies in traffic when bandwidth gets overloaded. Can it
be the beginning of an efficient Cookie deletion mechanism?

Amos

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 7 Mar 2016 13:29:18 +1000

> be the beginning of an efficient Cookie deletion mechanism?
​Maybe, but I have reservations...

Regarding "Priority=High": Currently cookies are often used on the critical
path, and these existing cookies don't have a "Priority" attribute. We
can't define a new spec that retroactively mandates that implementers have
to add the attribute, so I imagine we would probably want to define it so
that "no Priority" is as high as "High Priority." (Unless we're proposing
to define a new supe..uh..'more important' cookie.) So all the existing
dead wood remains, and the path of least resistance for implementers who
want to comply with the spec is: don't do anything. Thus it's not a
valuable addition.

Regarding "Priority=Low": this allows/encourages people to add even more
cookies, because "they're low priority, so they're less harmful." Telling
people to add a bunch of fluffy cookies because 'they can be pruned if
there are too many' doesn't seem like an improvement to me. Better advice
would be: don't send so much cruft in cookies.

(At the risk of sounding idealistic:) Do we want to make it easier for
middleboxen to cover up the abuses of pathological web applications?

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 7 Mar 2016 09:12:39 +0100 (CET)

> https://tools.ietf.org/html/draft-west-cookie-priority-00. Apologies for

Count me as another skeptic.

Since implementing support different cookie priority levels requires changing 
the server ends, wouldn't it be better to ask server admins to instead stop 
polluting the same domain with excessive amounts of cookies? I presume the 
primary cookie limit you're talking about number of cookies per domain. 
Alternatively, advocate for a higher limit (even though RFC6265 only suggests 
certain limits, browsers are free to user higher)?

What happens to cookies that are actually called 'Priority' ? It seems like a 
very standard name for a cookie and to me it seems like there's room for 
confusion for services that already use Priority set to Low/High etc. I don't 
see any mentioning of how to handle this.

This mechanism adds more complexity to an already complicated and messed up 
concept.

I'm also a bit sad to hear that Chrome+Google already implement this, as it 
feels like a certain degree of the old web war tactics all over again. It 
won't really matter what we say in this work group, as Google services will 
work less good without this feature and Chrome already works like this, so in 
order to keep users happy, user agents are strong-armed into following...

-- 

  / daniel.haxx.se

---

From: Mike West <[email-masked]>
Date: Mon, 7 Mar 2016 10:25:46 +0100

Thanks for the conversation, folks. A few replies inline:

On Mon, Mar 7, 2016 at 9:12 AM, Daniel Stenberg <[email-masked]> wrote:

> Since implementing support different cookie priority levels requires
> changing the server ends, wouldn't it be better to ask server admins to
> instead stop polluting the same domain with excessive amounts of cookies? I
> presume the primary cookie limit you're talking about number of cookies per
> domain. Alternatively, advocate for a higher limit (even though RFC6265
> only suggests certain limits, browsers are free to user higher)?
>

Chrome and Firefox both set limit at ~150 cookies per domain. Chrome has a
different eviction mechanism than Firefox, allowing a domain to flex up to
180 cookies before purging it back down to 150. My (vague) understanding is
that we arrived at that mechanism for performance reasons: purging just one
cookie lead to some level of churn for users who hovered around the limit.

Raising the limit is certainly possible, but also has performance impacts
and storage limitations. We'd need to measure those carefully before
bumping the limit, as determining which cookies to send ends up being on
the critical path for every request.

> What happens to cookies that are actually called 'Priority' ? It seems
> like a very standard name for a cookie and to me it seems like there's room
> for confusion for services that already use Priority set to Low/High etc. I
> don't see any mentioning of how to handle this.
>

Like cookies named "HttpOnly" or "MaxAge", this is handled by step 1 of
https://tools.ietf.org/html/rfc6265#section-5.2, which splits the cookie
string on the first ';' into the name/value pair, and the set of attributes.

> This mechanism adds more complexity to an already complicated and messed
> up concept.
>

It also solves a real problem that really exists, at least in the complex
ecosystems of corporate intranets. Both Mark and Martin seem to have
confirmed that the anecdata I provide isn't limited to Google.

> work less good without this feature and Chrome already works like this, so
> in order to keep users happy, user agents are strong-armed into following...

On this point, I can only apologize. We didn't follow through on our
commitment to discuss shipping this feature openly, we didn't report our
experimental results, and we didn't get buy-in from other browser vendors.
You're correct that this is bad behavior. If it turns out that Chrome is
the only vendor that's going to implement this, then I'll do my best to get
it removed.

All that said, I think the Chrome team has gotten significantly better at
this since 2013. We have a clear launch process
<http://www.chromium.org/blink#launch-process> for web-facing features, and
if we were launching cookie priorities _today_, the conversation would have
gone differently.

On Mon, Mar 7, 2016 at 4:29 AM, Matthew Kerwin <[email-masked]>
 wrote:

> Regarding "Priority=High": Currently cookies are often used on the
> critical path, and these existing cookies don't have a "Priority"
> attribute. We can't define a new spec that retroactively mandates that
> implementers have to add the attribute, so I imagine we would probably want
> to define it so that "no Priority" is as high as "High Priority." (Unless
> we're proposing to define a new supe..uh..'more important' cookie.) So all
> the existing dead wood remains, and the path of least resistance for
> implementers who want to comply with the spec is: don't do anything. Thus
> it's not a valuable addition.
>

The spec defines a missing `priority` attribute as setting "medium"
priority, which allows an origin to just set priority on those cookies
which are particularly important, or particularly discardable. For
instance. Google sets `priority=high` on authentication cookies, and
`priority=low` on cookies which only contain information that could be
derived (at expense) from the user session data.

Also, just so it's clear: the `priority` attribute is only considered in
the context of a single domain. We don't discard `example.com`'s "low"
priority cookies in order to keep `google.com`'s "high" priority cookies.
We only consider priority when determining which of a particular domain's
to evict, once we know that we need to evict a few. It is quite limited in
scope, and does not override any of the other mechanisms which might cause
a cookie to be removed. In particular, `priority=high` does not change
cookie expiration. I don't think it's fair at all to allude to it as a
supercookie.

> would be: don't send so much cruft in cookies.

Given that `priority` only comes into play when cookies are evicted for
exceeding a domain's limit, it doesn't appear that developers have needed
much encouragement. :)

In the particular set of cases I'm concerned with, the problem isn't a
single developer or even a single application stuffinh a user's cookie jar
with 150+ cookies, but a collusion of multiple applications on a single
registrable domain. For each individual application, cookies might be
totally legitimate and not at all crufty; that doesn't change the overall
impact on the domain.

>   cookie.priority = 'floor';

Note that the eviction changes in
https://tools.ietf.org/html/draft-ietf-httpbis-cookie-alone-00 would still
be effective; I'm working through how those changes impact Google at the
moment, and hope to have a more concrete proposal of what an eviction
algorithm that takes care of both of these proposals might look like. The
initial pass that we planned to start shipping in Chrome 50 caused some
disruption that I'm still working through.

> prevent this sort of craziness.  That turns out to have beneficial

Indeed, Google has put a ton of effort into understanding all the cookies
that are set by Google services, and my understanding is that we do
actively cull cookies coming into/set by our frontend servers. The intranet
(`*.corp.google.com`) is less centralized, and less controllable; I'm told
that the sign-out rates we see internally are orders of magnitude higher
than we see externally, for this reason.

-mike

---

From: Martin Thomson <[email-masked]>
Date: Mon, 7 Mar 2016 20:42:29 +1100

On 4 March 2016 at 02:58, Mike West <[email-masked]> wrote:
>  https://tools.ietf.org/html/draft-west-cookie-priority-00

Another bit of feedback, mostly in jest.  Mostly.

http://memedad.com/memes/829452.jpg

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 7 Mar 2016 10:53:04 +0100 (CET)

On Mon, 7 Mar 2016, Mike West wrote:

>> What happens to cookies that are actually called 'Priority' ?
>

IOW: they'll stop working or get mixed up since we can't tell them apart?

HttpOnly isn't done using a key/value so it actually easy is to separate from 
a cookie named HttpOnly. Max-Age for cookies was first suggested in RFC 2109 
(from what I can tell) from 1997, which in theory would make it less likely to 
ever have worked very good as a cookie name.

Priority, however, has been a perfectly fine cookie name since the dawn of 
cookies. I of course have no idea how common it might be though.

Or am I missing some subtlety?

-- 

  / daniel.haxx.se

---

From: Mike West <[email-masked]>
Date: Mon, 7 Mar 2016 11:09:50 +0100

> IOW: they'll stop working or get mixed up since we can't tell them apart?

We can tell them apart when we're parsing the cookie string. I'll explain
in more detail below, but I think you can convince yourself by sending a
`Set-Cookie: Priority=HUGE; Priority=High` header to Chrome and examining
the cookie jar.

> Or am I missing some subtlety?

Consider a `Set-Cookie` header whose value is `Priority=HUGE;
Priority=High`, which I think is what you're worried about.

Step 1 of the algorithm in https://tools.ietf.org/html/rfc6265#section-5.2
splits that string into a `name-value-pair` consisting of "Priority=HUGE",
and an `unparsed-attributes` of `; Priority=High`.

Steps 3-6 set the cookie's `cookie-name` to "Priority" and `cookie-value`
to "HUGE".

Then the second part of that algorithm parses `unparsed-attributes` to
extract an attribute named "Priority" with a value of "High".

-mike

---

From: Francesco Chemolli <[email-masked]>
Date: Mon, 7 Mar 2016 11:09:30 +0100

> path, and these existing cookies don't have a "Priority" attribute. We can't
> define a new spec that retroactively mandates that implementers have to add
> the attribute, so I imagine we would probably want to define it so that "no
> Priority" is as high as "High Priority." (Unless we're proposing to define a
> new supe..uh..'more important' cookie.) So all the existing dead wood
> remains, and the path of least resistance for implementers who want to
> comply with the spec is: don't do anything. Thus it's not a valuable
> addition.

..why?
Cookie priorities are relative and anchored to an authority. It's up
to the UA to define pruning strategies but I doubt the implementors
would consider priority as an absolute value.
The proposal mentions "Medium" as being the default. I see no harm in it.
This said, I'm adding my voice to the chorus hoping for a new
mechanism replacing cookies to be defined.

Francesco

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 7 Mar 2016 11:16:53 +0100 (CET)

> Priority=High`, which I think is what you're worried about.

I was actually thinking of the case when 'Priority=High' or 'Priority=Low' is 
used for an existing cookie, but I think I spoke up a little too early about 
that since in the case of only one 'Priority' and no (other) cookie name, it 
should indeed be distinguishable.

Sorry for the noise.

-- 

  / daniel.haxx.se

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 7 Mar 2016 13:09:20 +0100 (CET)

On Mon, 7 Mar 2016, Daniel Stenberg wrote:

> I was actually thinking of the case when 'Priority=High' or 'Priority=Low' 
> is used for an existing cookie, but I think I spoke up a little too early 
> about that since in the case of only one 'Priority' and no (other) cookie 
> name, it should indeed be distinguishable.

(sorry for replying to myself)

...execept for older clients that don't know about these cookie priorities of 
course. For those, they will appear as duplicate cookie names in the headers 
and will most likely cause problems to legacy client-side implementations.

libcurl will treat "Set-Cookie: Priority=Low; favcolor=blue" as a cookie named 
'Priority' and discard the favcolor part. Reversing the string order will make 
it store 'favcolor' instead. Most surely other implementations will act 
differently.

Thus, a server needs to know if the client supports Priority cookies before it 
can reliably send them.

-- 

  / daniel.haxx.se

---

From: Mike West <[email-masked]>
Date: Mon, 7 Mar 2016 13:24:32 +0100

On Mon, Mar 7, 2016 at 1:09 PM, Daniel Stenberg <[email-masked]> wrote:

> ...execept for older clients that don't know about these cookie priorities
> of course. For those, they will appear as duplicate cookie names in the
> headers and will most likely cause problems to legacy client-side
> implementations.
>
> libcurl will treat "Set-Cookie: Priority=Low; favcolor=blue" as a cookie
> named 'Priority' and discard the favcolor part. Reversing the string order
> will make it store 'favcolor' instead. Most surely other implementations
> will act differently.
>
> Thus, a server needs to know if the client supports Priority cookies
> before it can reliably send them.

I'm confused. Are there clients that process things in the reverse order
from what RFC6265 lays out?

I mean, according to the algorithm I quoted in the previous response,
`Priority=Low; favcolor=blue` _is_ a cookie named `Priority`. Just like
`Max-Age=1; favcolor=blue` is a cookie named `Max-Age` today. I think
that's the way browsers process cookies today. Does `curl` do things
differently?

-mike

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 7 Mar 2016 13:37:41 +0100 (CET)

> from what RFC6265 lays out?

I'm sure there are several. The curl one just happens to be the one I know the 
best.

> `Max-Age=1; favcolor=blue` is a cookie named `Max-Age` today. I think that's 
> the way browsers process cookies today. Does `curl` do things differently?

It does! It basically detects a set of names used for properties and treats 
the first unknown name value pair on the header as the cookie name, in a 
left-to-right order on the header.

(I'm not suggesting it is a "proper" or "good" implementation, just that it 
works with the vast majority of sites using cookies and it was written long 
before we created RFC 6265 and I guess nobody felt the need to update it since 
to that aspect.)

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Mon, 7 Mar 2016 09:13:32 -0800

On Mon, Mar 7, 2016 at 4:37 AM, Daniel Stenberg <[email-masked]> wrote:

> I'm sure there are several. The curl one just happens to be the one I know
> the best.
>
> It does! It basically detects a set of names used for properties and
> treats the first unknown name value pair on the header as the cookie name,
> in a left-to-right order on the header.
>
> (I'm not suggesting it is a "proper" or "good" implementation, just that
> it works with the vast majority of sites using cookies and it was written
> long before we created RFC 6265 and I guess nobody felt the need to update
> it since to that aspect.)

It might be worth improving the libcurl implementation to match RFC6265.
The vast majority of user agents using the parsing approach described in
RFC6265.  Having a hard-coded list of attribute names makes it (even more)
difficult to extend and improve cookies over time.  Regardless of what you
think of this particular extension, I think most people would agree that
extensibility in protocols is valuable.

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 7 Mar 2016 20:19:30 +0100 (CET)

On Mon, 7 Mar 2016, Adam Barth wrote:

>> before we created RFC 6265 and I guess nobody felt the need to update it 
>> since to that aspect.)
>

No objections there, I think it would! But you know, whatever works in 
practise rarely gets attention and fixes just because it would make it follow 
an RFC closer.

Besides, I think the point is that 1. curl is probably not the only naive 
implementation in the world and 2. even if we'd fix this in curl today, we'd 
see applications in the wild for many years to come still using the previous 
implementation... (we have some users that are notoriously slow on upgrading 
and regularly get bug reports and questions on a decade old code!)

-- 

  / daniel.haxx.se

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 8 Mar 2016 07:50:13 +1000

Doesn't that last paragraph counter the previous a bit? You don't discard
example.com's low cookies to keep google.com's high ones, but you evict
google.com/foo's low ones to keep google.com/bar's high ones. Even though
the foo and bar teams are clearly independent of each other (else surely
they could synergise their cookies a bit better in the first place.)

How many domains host 150 completely independent apps that the user is
actively logged into simultaneously? Even 75? Hell, even 35? And four-five
cookies per app is pushing what I'd normally consider reasonable, we're
definitely pushing into cruft territory here.

Maybe I'm too conservative.

If the wg/community decides that fixing the problem is intractible, then
sure, patch the symptoms -- but please take care to do it in a way that
doesn't make things worse.

> [...] it doesn't appear that developers have needed much encouragement.

What if the current state of things is just not making it worse?

Cheers