Subject: [http-state] Ticket 5: Cookie ordering

From: Adam Barth <[email-masked]>
Date: Tue, 19 Jan 2010 14:10:48 -0800

Ticket 3 is still open for discussion, but I'd like to get started
talking the next ticket:
http://trac.tools.ietf.org/wg/httpstate/trac/ticket/5

== Overview ==

Currently the draft defines the order in which cookies should appear
in the Cookie header.  In particular, cookies are ordered first by the
length of the Path attribute (longest first) and then by creation date
(earliest first).  The majority of the most widely used user agents
use this ordering. (I can look up exactly which browsers follow the
ordering if that's important.)

Sending cookies with longer (i.e., more specific) paths first is
important for compatibly because some servers host multiple (mutually
trusting) web applications at different (possibly overlapping) paths.
For example, mail.google.com hosts a public instance of Gmail at
https://mail.google.com/ and corporate instances of Gmail at
https://mail.google.com/a/example.com/.  These web applications expect
their own cookies (with the more specific path) to be presented before
cookies for other web applications (even if the cookies have the same
names).  In fact, this behavior is required by the original Netscape
cookie spec.

== Proposal ==

(1) Continue to specify the ordering.

== Alternatives ==

(2) Do not specify an ordering for cookies in the Cookie header.  This
alternative reduces interoperability between user agents because user
agents that serialize cookies in different orders will operate
different than each other.  As the most popular user agents already
order cookies, user agents with less market share can gain
interoperability by matching the consensus ordering.

(3) Require that cookies with longer paths appear before cookies with
shorter paths, but do not require an ordering among cookies with equal
path lengths.  This approach maximizes user agent flexibility while
meeting the most important compatibility needs, but it's unclear to me
that there is value in maximizing user agent flexibility in this
regard.  It seems like if we're going to specify an ordering for
cookies, we might as well specify the total ordering used by the most
popular user agents.

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 19 Jan 2010 23:50:16 +0100 (CET)

On Tue, 19 Jan 2010, Adam Barth wrote:

> Currently the draft defines the order in which cookies should appear in the 
> Cookie header.  In particular, cookies are ordered first by the length of 
> the Path attribute (longest first) and then by creation date (earliest 
> first).  The majority of the most widely used user agents use this ordering. 
> (I can look up exactly which browsers follow the ordering if that's 
> important.)

My take on cookies is from the other direction. I think we should specify and 
document what needs to be done in order to work with the vast majority of 
sites. I don't think mimicing browsers is a good idea if there's no demand for 
that from the other side.

I know that just about every behavior in browsers can be tracked back to a bug 
in at least one of them and then the others followed or had their own bug 
report and acted independently. Still, many of those acts were more or less 
server-side bugs that SHOULD (imho of course but I suspect I'm not alone in 
this belief) have been fixed in the servers but browser vendors have been too 
scared to loose users so they've instead adapted to stupidities.

But this said, I'm willing to say that a specific "quirk" is no longer a 
server side bug if a large enough amount of servers are already using it. Say 
if three or more of the top-500 sites do it (I'm just throwing out a way to 
count, I didn't really think the numbers through).

> Sending cookies with longer (i.e., more specific) paths first is important 
> for compatibly because some servers host multiple (mutually trusting) web 
> applications at different (possibly overlapping) paths. For example, 
> mail.google.com hosts a public instance of Gmail at https://mail.google.com/ 
> and corporate instances of Gmail at https://mail.google.com/a/example.com/. 
> These web applications expect their own cookies (with the more specific 
> path) to be presented before cookies for other web applications (even if the 
> cookies have the same names).  In fact, this behavior is required by the 
> original Netscape cookie spec.

(This is a change compared to my previous statements in this matter.) I now 
agree that there's in fact a need to be certain that the "most specific" 
version of two names is sent first, and I quite like the phrasing the netscape 
doc:

   cookies with a more specific path mapping should be sent before cookies with
   less specific path mappings

I don't think it needs to say that _all_ cookies need to be sorted on path 
length, as that's not what servers need.

The purpose of the suggested secondary sort key on creation date however is 
less clear to me. That would be for sites using overlapping paths (with the 
same lengths), identical cookie names and then it would depend on which order 
the cookies were created. Is there really a presently existing site that 
requires this functionality in browsers?

> shorter paths

If they have the same name, yes.

> but do not require an ordering among cookies with equal path lengths.  This 
> approach maximizes user agent flexibility while meeting the most important 
> compatibility needs, but it's unclear to me that there is value in 
> maximizing user agent flexibility in this regard.  It seems like if we're 
> going to specify an ordering for cookies, we might as well specify the total 
> ordering used by the most popular user agents.

Since user agents can also at whim decide to not accept cookies (and perfectly 
allowed to do so according to previous and this spec), a server cannot 
reliably know in advance which cookie that was created first so it couldn't 
reliably be written to depend on that order. Could it?

Again: alternative implementations do not necessarily seek to act like other 
browsers, they may just try to work against all servers.

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Tue, 19 Jan 2010 17:29:16 -0800

Let me try rephrasing your proposal to make sure I understand it properly:

(4) Require that among cookies with the same name, user agents ought
to present cookies with longer path attributes before cookies with
shorter path attributes.  The spec should not require or recommend
other ordering constraints.

Can you explain why (4) provides a better cost / benefit trade-off
than (1) or (3)?  In particular, what is the advantage of us adopting
(4) over (1) or (3)?

On Tue, Jan 19, 2010 at 2:50 PM, Daniel Stenberg <[email-masked]> wrote:
> I think we should specify
> and document what needs to be done in order to work with the vast majority
> of sites. I don't think mimicing browsers is a good idea if there's no
> demand for that from the other side.

Do you have data about the relative compatibility of (1), (3), and
(4)?  If not, are you willing to generate this data?  It seems like
there is some interoperability / compatibility risk to (4) that would
be mitigated in (3) and (1).  Without a way to quantify this risk, we
are flying blind.

> I know that just about every behavior in browsers can be tracked back to a
> bug in at least one of them and then the others followed or had their own
> bug report and acted independently. Still, many of those acts were more or
> less server-side bugs that SHOULD (imho of course but I suspect I'm not
> alone in this belief) have been fixed in the servers but browser vendors
> have been too scared to loose users so they've instead adapted to
> stupidities.

That's right.  Browser vendors are reticent in introduce compatibility
problems with web sites.  That's why we need to be careful about these
issues.  Whether we think that's a good idea on there part, if
shouldn't ignore them if we want them to care about our spec.

> server side bug if a large enough amount of servers are already using it.
> Say if three or more of the top-500 sites do it (I'm just throwing out a way
> to count, I didn't really think the numbers through).

Quirk is a very judgmental term.  I'd rather talk about behaviors.  In
particular, this sorting behavior seems harmless to me, whereas not
sorting appears risky in the absence of data.

> version of two names is sent first, and I quite like the phrasing the
> netscape doc:
>

Why introduce interoperability problems where none exist?  If the most
widely used user agents are already doing something sensible, it seems
like the path of least resistance is to adopt that behavior.  I'd
rather save my effort for trying to change things that matter.

> same lengths), identical cookie names and then it would depend on which
> order the cookies were created. Is there really a presently existing site
> that requires this functionality in browsers?

I'm sure there's a site somewhere that depends on the consensus ordering.

>> but do not require an ordering among cookies with equal path lengths.
>>  This approach maximizes user agent flexibility while meeting the most
>> important compatibility needs, but it's unclear to me that there is value in
> Since user agents can also at whim decide to not accept cookies (and
> perfectly allowed to do so according to previous and this spec), a server
> cannot reliably know in advance which cookie that was created first so it
> couldn't reliably be written to depend on that order. Could it?

A common case is that a server will send two Set-Cookie headers in the
same HTTP response:

Set-Cookie: foo=bar
Set-Cookie: baz=qux

With sort order (1), the user agent will respond

Cookie: foo=bar; baz=qux

or omit the cookie header entirely.  I can easily imagine server-side
code that would break if presented with "Cookie: baz=qux; foo=bar".

> browsers, they may just try to work against all servers.

The easiest way to work with all servers is to sort the cookies as
proposed in (1).

Adam

---

From: David Morris <[email-masked]>
Date: Tue, 19 Jan 2010 20:51:30 -0800 (PST)

> trusting) web applications at different (possibly overlapping) paths.

I think 'longer' is not a precise term ... likewise, 'more specific' isn't
precise enough to avoid confusion. I think what would be expected is that
the path sith the most 'levels' (as noted by '/' characters) is longer in 
a deeper into the hierarachy sense. Same depth, the age rule could apply.

If 'more specific' is already defined similar to what I've outlined,
you can igore this comment.

Dave Morris

---

From: Adam Barth <[email-masked]>
Date: Tue, 19 Jan 2010 21:13:47 -0800

On Tue, Jan 19, 2010 at 8:51 PM, David Morris <[email-masked]> wrote:
> the path sith the most 'levels' (as noted by '/' characters) is longer in a
> deeper into the hierarachy sense. Same depth, the age rule could apply.
>

In this case, all the cookie paths are prefixes of the Request-URI
path, so these all amount to the same thing.  FWIW, longer is a
precise term: literally the one whose path attribute contains more
characters.

Adam

---

From: David Morris <[email-masked]>
Date: Tue, 19 Jan 2010 21:21:07 -0800 (PST)

> precise term: literally the one whose path attribute contains more

More characters doesn't seem like the right definition, even if it works 
most of the time. If the argument is, as you've made, that the requirement
is for a server to handle cookies in the most precise order ... it will
be related to logical depth of the hierarchy and not character length.

---

From: Adam Barth <[email-masked]>
Date: Tue, 19 Jan 2010 21:37:36 -0800

On Tue, Jan 19, 2010 at 9:21 PM, David Morris <[email-masked]> wrote:
>>> a

I'm not sure what you mean by the "right" definition.  Can you provide
a test case where the two definitions differ?

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 20 Jan 2010 09:13:56 +0100 (CET)

> than (1) or (3)?  In particular, what is the advantage of us adopting

I like specifying only what needs to be specified and leave the rest to the 
implementor. In every spec I read or work with.

The (3) "sort-them-all" approach is indeed the easier way to implement this, 
as figuring out exactly which cookies that have the same name and only 
special-case the order of those will probably be a bigger effort than to 
simply unconditionally sort all cookies based on path length.

Since the only cookies that NEED an order are those that have the same name, I 
think we should only specify that.

> Do you have data about the relative compatibility of (1), (3), and (4)?  If 
> not, are you willing to generate this data?  It seems like there is some 
> interoperability / compatibility risk to (4) that would be mitigated in (3) 
> and (1).  Without a way to quantify this risk, we are flying blind.

I've not seen anyone present any numbers regarding cookie sort order so I 
don't quite see why I would be different here. No, I don't have any numbers 
and I don't think I'm able to come up with any numbers either.

My assumptions and views of reality here comes from the fact that I've written 
a cookie implementation that has been used rather widely by applications, 
frameworks and browsers for quite a number of years by now. Like most other 
people here base their views on work with other implementations and/or users. 
I assume we're all similar in that aspect.

> Why introduce interoperability problems where none exist?

The interoperability here is against servers. Can you show me two servers in 
active use today that a client doesn't work with if the cookie order is as 
"free" as I suggest?

> If the most widely used user agents are already doing something sensible

Web user agents, browsers, sure. There is also a wide range of non-gui clients 
that use HTTP and cookies that we don't know how much they're used and as I 
showed before, lots of them have no sorting whatsoever and lots of them work 
fine against the vast majority of the cookie-using sites out there.

I don't think browser usage share is the only and final measurement here.

(I should add that curl will sort cookies based on path length starting soon, 
as I've been convinced that there will be a few rare sites that cannot work 
properly without it.)

Some of those non-web tools that deal with cookies know how to import cookies 
from the traditional cookies file in the netscape file format, and that format 
has no "creation time" field and thus importing cookies from such a file makes 
it impossible to send the cookies back sorted on the original "creation time" 
but instead it could only do it based on the time of the read from a file.

> I'm sure there's a site somewhere that depends on the consensus ordering.

In my view there is no consensus ordering.

> or omit the cookie header entirely.  I can easily imagine server-side code 
> that would break if presented with "Cookie: baz=qux; foo=bar".

Really? I mean, sure I can imagine that there is also sites that require that 
the Cookie: header is the 5th header in the request but that doesn't make it a 
correct assumption even though it might work with a browser or two.

Is it really an assumption that is widely enough used so that we need to 
document it as something clients should do?

I'm only against specifying it in the spec if it's a very rare assumption that 
we can declare is a server bug. If the assumption is spread and used more than 
rarely then I am wrong and it should be specified.

> The easiest way to work with all servers is to sort the cookies as proposed 
> in (1).

The easiest way is to just document the exact behavior of one browser and one 
server implementation and we're done. That's however not our job here, I 
think.

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Wed, 20 Jan 2010 01:48:23 -0800

It sounds like you're now advocating:

path lengths.

In particular, because you wrote:

> simply unconditionally sort all cookies based on path length.

and

> (I should add that curl will sort cookies based on path length starting
> soon, as I've been convinced that there will be a few rare sites that cannot
> work properly without it.)

You didn't answer my question, which I'll repeat here re-phrased for
your current position:

Can you explain why (3) provides a better cost / benefit trade-off
than (1)?  In particular, what is the advantage of us adopting (3)
over (1)?

I claim that (1) has a better cost / benefit trade-off than (3) for
the following two reasons:

A) There is a risk that some sites will fail to function properly if
the cookies are presented in another order.

B) It is essentially cost-free for user agents to sort cookies in a
particular order, especially in light of the fact that we agree that
user agents must at least sort cookies using a partial order based on
path length.

Detailed comments below.

> implementor. In every spec I read or work with.

I'm not sure what your personal preference has to do with anything.
I'm sure some folks would prefer that we specify things in more detail
and others would prefer that we have a looser specification.  In the
end, technical arguments and data carry more weight.

> Since the only cookies that NEED an order are those that have the same name,
> I think we should only specify that.

You haven't presented any evidence for this claim.

>> Do you have data about the relative compatibility of (1), (3), and (4)?
>>  If not, are you willing to generate this data?  It seems like there is some

If there were some cost associated with sorting the cookies, then we
would need to balance that cost against the compatibility cost of not
sorting the cookies.  As it stands, there is no cost to sorting the
cookies.  Only risk.  You claim that the risk is small, but you
haven't presented any evidence as to its magnitude.  My argument
doesn't depend on the magnitude of the risk.  That's why your argument
requires data and mine does not.

> My assumptions and views of reality here comes from the fact that I've
> written a cookie implementation that has been used rather widely by
> applications, frameworks and browsers for quite a number of years by now.
> Like most other people here base their views on work with other
> implementations and/or users. I assume we're all similar in that aspect.

I'd rather made decisions based on data and technical arguments
instead of assumptions.

> active use today that a client doesn't work with if the cookie order is as

That's not a good criterion to use when deciding to relax a
requirement.  For example, if you'd asked me to show you a server that
would fail to operate if we changed the number of cookies allowed per
domain from 20 to 50, I would not have been able to produce one.  Yet,
when IE changes to allow 50 cookies per domain, they rendered a major
financial institution's web site completely inoperable:

http://blogs.msdn.com/ieinternals/archive/2009/08/20/WinINET-IE-Cookie-Internals-FAQ.aspx

In that case, however, there was a good reason for allowing 50 cookies
per domain (which I can elaborate).  In the case of cookie sorting, I
haven't heard a good reason not to sort the cookies.

Here's what Eric Lawrence learned from this episode:

"That, in turn, is one reason why the IE team must exercise great care
when making any change to IE’s cookie implementation."

We would be well served to heed this advice.  If we ignore it, user
agent implementors will ignore us and we'll have another dead-letter
spec like 2109.

> Some of those non-web tools that deal with cookies know how to import
> cookies from the traditional cookies file in the netscape file format, and
> that format has no "creation time" field and thus importing cookies from
> such a file makes it impossible to send the cookies back sorted on the
> original "creation time" but instead it could only do it based on the time
> of the read from a file.

This is a good technical argument.  Maybe we should state that the
creation time ordering is required only if the creation time
information is available to the user agent.

> Really? I mean, sure I can imagine that there is also sites that require
> that the Cookie: header is the 5th header in the request but that doesn't
> make it a correct assumption even though it might work with a browser or
> two.

The HTTP spec would be quite different if I were writing it.  For
example, this poor proxy vendor wouldn't have had this problem:

http://lists.w3.org/Archives/Public/ietf-http-wg/2009JulSep/0264.html

However, my job is not to write the HTTP spec.  My job is to write the
highest quality cookie spec I can.

> I'm only against specifying it in the spec if it's a very rare assumption
> that we can declare is a server bug. If the assumption is spread and used
> more than rarely then I am wrong and it should be specified.

Suppose we declare it a server bug.  What then?  Are user agent
vendors going to spend their evangelism budget to convince site
operators to unbreak their servers?  In reality, the user agents with
high market share will continue to sort cookies.  User agents with low
market share will be blissfully ignorant until they come across a
server that breaks.  If they're lucky enough to get a bug report and
reduce the issue, they can either spend evangelism dollars to get the
site to change or they can sort the cookies in the same way as the
user agents with high market share.

As someone who's been around these decisions, I can tell you that a
lot of evangelism dollars get taken up on things like fixing the way
servers parse the user agent string.  In these cases, the user agent
has no recourse except to copy the user agent string of a high market
share user agent.

> The easiest way is to just document the exact behavior of one browser and
> one server implementation and we're done. That's however not our job here, I
> think.

Our job is the following:

"Where commonalities exist in the most widely used implementations,
the working group will specify the common behavior."

In my view, option (1) best accomplishes this goal.

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 20 Jan 2010 13:02:28 +0100 (CET)

On Wed, 20 Jan 2010, Adam Barth wrote:

> shorter paths, but do not require an ordering among cookies with equal path 
> lengths.

I think that cookies using the same name should be presented in an order so 
that the version with the longest specified path gets presented first.

>> simply unconditionally sort all cookies based on path length.

I mentioned that in an attempt to provide context: I can see why implementors 
sort all cookies based on the path lengths, and there's nothing wrong in doing 
that, but I don't think our spec needs to specify that as the sort order is 
only necessary between cookies using the exact same name (unless a server 
implementation relies on that order somehow).

> You didn't answer my question, which I'll repeat here re-phrased for your 
> current position:
>
> Can you explain why (3) provides a better cost / benefit trade-off than (1)? 
> In particular, what is the advantage of us adopting (3) over (1)?

I did answer it but I'll happily do it again slightly clarified:

   I want us to specify only what needs to be specified and leave the rest to
   the implementors.

I don't think the creation-time of a cookie is relevant so I don't think a 
client needs to consider it when sending cookies. The advantage for us is that 
we get one less thing to include in the spec and we get a larger amount of 
current implementations to qualify as already compliant with the spec.

> A) There is a risk that some sites will fail to function properly if the 
> cookies are presented in another order.

Yes, but this risk already exist for all the implementations of today that 
don't sort cookies (like that) and given the widespread use of non-sorting 
implementations I guess that server-side reliance on such strict sorting is 
fairly rare.

> particular order, especially in light of the fact that we agree that user 
> agents must at least sort cookies using a partial order based on path 
> length.

No, it's not cost-free - nothing really is. I already described one scenario 
in which the cookie's creation time gets discarded (the netscape cookie file 
format), and I can also easily find several cookie implementations that 
currently completely ignore the creation-time. They would be deemed 
incompatible and they would need to be fixed. The current implementations that 
already sort wouldn't break the spec so thus we would cover more existing 
implementions by not specifying the sort order like that.

Not to mention how every added complexity in the spec makes it harder for all 
future implementations.

It would deem servers that rely on the date-sorting non-compliant.

> I'd rather made decisions based on data and technical arguments instead of 
> assumptions.

Me too, but we don't have such data in this discussion - even though I think 
we all would appreciate and enjoy it. And even if we had data and numbers 
(that we all thought was the objective truth), there would still be a 
discussion around what percentage of servers or clients that would matter and 
count as "existing practise". Is 10% enough? 1% 0.1%? 0.01%? etc.

> This is a good technical argument.  Maybe we should state that the creation 
> time ordering is required only if the creation time information is available 
> to the user agent.

Isn't that rather half-baked and pointless?

It is required, but if you don't have the info it isn't required anymore. How 
could a server side implementation then assume anything about the sort order 
since they cannot possibly know when the client had the required info or not? 
In reality that takes away the requirement. And without the requirement, why 
specify it at all?

I wish we knew if there are any actual servers that rely on the time sort. It 
would help this discussion a lot I think.

-- 

  / daniel.haxx.se

---

From: Dan Winship <[email-masked]>
Date: Wed, 20 Jan 2010 09:19:53 -0500

On 01/20/2010 03:13 AM, Daniel Stenberg wrote:
> cookies from the traditional cookies file in the netscape file format,
> and that format has no "creation time" field and thus importing cookies
> from such a file makes it impossible to send the cookies back sorted on
> the original "creation time" but instead it could only do it based on
> the time of the read from a file.

FTR, Netscape/Mozilla/Firefox always write the cookie file out in
creation-time-order, so while you can't recover the *exact* creation
time of each cookie, you can still sort by creation time. (Rather than
talking about "creation time", it probably makes more sense in this case
to think about "serial number", assigning the cookies in the cookies.txt
file numbers from 1 to N, and then assigning each cookie received after
that point the next number in sequence.)

(Yes, I'm sure there are non-mozilla-based tools that write out
cookies.txt files that *don't* do it in creation order.)

-- Dan

---

From: Dan Winship <[email-masked]>
Date: Wed, 20 Jan 2010 09:43:22 -0500

> (1) Continue to specify the ordering.

+1

But really this discussion is just a proxy for the still-unresolved
"conformance classes" / "what exactly is it that we are specifying"
discussion...

-- Dan

---

From: Maciej Stachowiak <[email-masked]>
Date: Sat, 23 Jan 2010 20:48:33 -0800

> ordering if that's important.)

That would be useful information. For the browsers that don't follow this behavior, I'd also like to know how they differ. It sounds like not specifying the order at all isn't viable, so I'd like to understand the tradeoffs between fully specifying it and partially specifying it in various ways.

Regards,
Maciej

---

From: "Anne van Kesteren" <[email-masked]>
Date: Sat, 30 Jan 2010 14:51:33 +0100

+1

-- 
Anne van Kesteren
http://annevankesteren.nl/