Subject: [http-state] Comments on draft-ietf-httpstate-cookie-08

From: "Yngve Nysaeter Pettersen" <[email-masked]>
Date: Sat, 29 May 2010 03:05:58 +0200

Hello all,

Here are some more comments about the httpstate draft.

* Sec 3 (overview)

  - "In subsequent requests, the user agent returns a Cookie request header  
to the origin server"

Should this say that a client may send a Cookie header (if it decides to  
accept the Cookie)? (I also prefer "send" instead of "returns",  
alternatively "return the stored cookie in a Cookie header")

  - The section also mentions gateways,concerning changes to the Set-Cookie  
headers. Might it be useful to also specify that gateways must not insert  
Set-Cookie headers?

* Sec 3.1

- Should the expiration example mention max-age?

- Should "a expiration date" be changed to "a validity period" or words to  
that effect?

* Sec 4.1.1

- "Servers SHOULD NOT include two Set-Cookie header fields in the same  
response with the same cookie-name."

Should a caution about "unpredictable results" be added to this?

* Sec 4.1.2.3

-   For example, some user agents will reject Domain attributes of "com"  
or "co.uk".

This seems to imply that there are agents that will accept a TLD as a  
valid domain (except ".local" from 2965, which is a special case). I'd  
rather have the section say that a domain attribute need to specify at  
least a second level domain.

* sec 4.1.2.4

- %XX escaping and path compares. Maybe add a reference to the URI compare  
definition in 2616?

* Sec 4.1.2.6

- Maybe an idea to mention that HttpOnly is a relatively new attribute,  
which may not be implemented by legacy clients?

* sec 7.1

  - Should this section have a recommendation to servers about graceful  
handling if the client does not send cookies? (Yes, it is mentioned  
generally further up)

* Double Qoutes in the cookie-value field

While not permitted by the Server side part of the spec, the draft seem to  
allow the double qoute character <"> (DQUOTE) anywhere in a cookie value,  
whether they are balanced, or not.

This is IMO of significant concern since it breaks with how parameter  
values are defined and used in both MIME, HTTP, and RFC 2965, as (token |  
quoted-string), and may cause interoperability problems with clients and  
servers that support RFC 2965 (Opera supports 2965, and AFAIK the Apache  
Tomcat server also supports it).

The handling of cookie values that are not tokens, and contain double  
quotes inside the value in a form that does not match the 2616 definition  
of quoted-string is according to my quick testing inconsistent between a  
few browsers (IE8, FF 3.6 and Opera 10.53), and AFAICT there is likely to  
be problem with Python based server-environments, like Django 1.1, which  
is the only one I have tested.

Examples

   foo=""";

      IE: foo=""";
      FF: foo="";
      Opera: discarded

   foo="xx"xx";
      IE: foo="xx"xx";
      FF: foo="xx";
      Opera: discarded

   foo="xx"xx;
      IE: foo="xx"xx;
      FF: foo="xx";
      Opera: foo="xx";

   foo=",;";
      IE:  <foo=",>; (Cookie terminated at the first semicolon, not the  
second, <> added for emphasis)
      FF:  foo=",;";
      Opera:  foo=",;";

As for the server-side, when sending this header

    Cookie:  name1="foo1; name2=bar"; name3=foo2"2; name4=bar"4; name5=foo

to Django 1.1 or the Python 2.6 Cookie module, the result is the following  
set of cookies:

    name1="foo1; name2=bar"
    name3=foo2
    name4=bar
    name5=foo

In other words, when using quotes inside a value with any syntax other  
than quoted-string (which permit internal quoting if backslash escaping is  
used), the result does not just depend on the client the server is sending  
it to, it also depends on the script engine and its cookie parser.

IMO the specification should specifically comment on these issues, and  
preferably allow clients to discard cookies with quotes that does not  
match the quoted-string syntax, as well as specifically tell servers not  
to use double quotes, except as quoted-string (if they absolutely want to  
break the spec's requirement of only using tokens). Perhaps one way to do  
that is to specifically say that the result of using quotes in this  
fashion is undefined?

-- 
Sincerely,
Yngve N. Pettersen
********************************************************************
Senior Developer		     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
********************************************************************

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 29 May 2010 14:50:47 +0200

>least a second level domain.

As I understand it, there are a number of frameworks that will auto-
matically use the server name, or failing that its address, as default
value for the domain attribute. It does not strike me as a good idea to
prohibit `Domain=localhost` for instance.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Date: Sat, 29 May 2010 16:48:02 +0200

> prohibit `Domain=localhost` for instance.

The formal hostname for localhost would be "localhost.local", but in any  
case, in such a situation the hostname would also be non-dotted  
("localhost") and it must exactly match the domain attribute (localhost).  
This kind of situation is handled by Opera.

What I am (implicitly) referring to above is a situation where the domain  
attribute is not equal to the hostname, but domain-matches the hostname,  
*and* where the domain attribute is a *single* level (IOW a TLD). Those  
cases should not be allowed, even by a public suffix list.

If somebody creates a server "com" (for example, setting a cookie for it  
will not (in Opera, and probably other browsers) cause the cookie to be  
valid for all the dot-com servers, since the cookie will be set for the  
implied FQDN "com.local".

Come to think of it, is the single label (local intranet server) situation  
covered in the domain sematics?

-- 
Sincerely,
Yngve N. Pettersen

---

From: Adam Barth <[email-masked]>
Date: Sat, 29 May 2010 09:43:16 -0700

> * Double Qoutes in the cookie-value field

I think you're testing with an old version of Firefox.  Can you try a
nightly build?  I believe the handling of double quotes in cookies
values is consistent across the spec, IE, Firefox (nightly), and
Chrome.

Adam

---

From: Adam Barth <[email-masked]>
Date: Sat, 29 May 2010 09:51:41 -0700

I'll reply to your full email in detail later, but another note:

On Fri, May 28, 2010 at 6:05 PM, Yngve Nysaeter Pettersen
<[email-masked]> wrote:
> preferably allow clients to discard cookies with quotes that does not match
> the quoted-string syntax,

User agents already have the option to discard any cookie for any
reason.  Adding this text to the spec would be redundant.

> as well as specifically tell servers not to use
> double quotes, except as quoted-string (if they absolutely want to break the
> spec's requirement of only using tokens).

Add this text to the spec would be redundant because the spec already
requires servers (at the SHOULD level) not to use quotes (either
double or single) at all.  More precisely, adding this text to the
spec would be a contrary-to-duty imperative, which, generally
speaking, are quite dubious logically.

> Perhaps one way to do that is to
> specifically say that the result of using quotes in this fashion is
> undefined?

Leaving things like this undefined hurts interoperability.

Adam

---

From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Date: Sat, 29 May 2010 19:54:00 +0200

On Sat, 29 May 2010 18:51:41 +0200, Adam Barth <[email-masked]> wrote:

>> match

I think it may still be useful to point out specific issues where such  
discarding policies are used automatically. My impression of that text is  
that it focus more on cleanup and user controlled filtering, rather than  
what policies the client may have implemented. I suspect that most  
administrators will read it that way too. For example, the draft  
specifically mentions public suffix as a reason why a client can discard a  
cookie.

> Leaving things like this undefined hurts interoperability.

The situation for this particular value syntax is already undefined, and  
while it is a corner case, it can have unanticipated effects, particularly  
in combination with different cookie ordering algorithms. Putting the  
server administrators on notice about the issue should hopefully increase  
interoperability because they can then avoid the problematic cases.

If we are going to document cookies as used on the Internet, then I think  
that includes pointing out the areas that should be avoided, particularly  
the ones where the results will be undefined. That should not just be done  
by how the syntax is defined, but also directly mentioning them.

This information can be mentioned either as notes (or Notes) in the  
definitions themselves, or collected in a separate "Implementation  
pitfalls"-section.

-- 
Sincerely,
Yngve N. Pettersen

---

From: Adam Barth <[email-masked]>
Date: Sun, 30 May 2010 13:48:16 -0700

> alternatively "return the stored cookie in a Cookie header")

I've changed this to "send."  As this is just the overview, I've left
it focusing on the common case, which is that the user agent does
actually end up sending the cookie.

> headers. Might it be useful to also specify that gateways must not insert

Why must gateways not insert Set-Cookie headers?  Granted, it might or
might not be friendly behavior, but it seems ok from a protocol
standpoint.

> - Should the expiration example mention max-age?

These are just examples.  I think its fine to focus on the common case.

> - Should "a expiration date" be changed to "a validity period" or words to

Hum...  The Expires attribute doesn't contain a period.  It contains a date...

> Should a caution about "unpredictable results" be added to this?

The results are predictable, just nutty.

> * Sec 4.1.2.3
>
> -   For example, some user agents will reject Domain attributes of "com" or
> "co.uk".
>
> This seems to imply that there are agents that will accept a TLD as a valid
> domain (except ".local" from 2965, which is a special case). I'd rather have
> the section say that a domain attribute need to specify at least a second
> level domain.

There are some user agents that will accept a TLD as a valid domain.
I'd advise against building such a user agent, but not everyone will
take my advice.

> - %XX escaping and path compares. Maybe add a reference to the URI compare

Can you provide a test case that demonstrates that %-escaping is
relevant in this context?  My testing shows that user agents ignore
%-escaping in the Path attribute.  Now, some user agents canonicalize
%-escaping in URLs, but that's a matter best left to another working
group, IMHO.

> which may not be implemented by legacy clients?

I believe HTTPOnly is broadly implemented among user agents for which
you can detect whether it's implement.  Note that for the vast
majority of user agents, you can't tell whether they implement
HTTPOnly, so we might as well act as if they did implement it.

> handling if the client does not send cookies? (Yes, it is mentioned

We could, but I don't think it would make much of a difference.

> * Double Qoutes in the cookie-value field

I've replied to this section earlier.

Generally, this whole business of double-quote being a meaningful
character in cookies values was a fantasy invented by RFC 2109.  As
far as I can tell, IE has never treated double-quote different from
any other character in cookie values, which means that the vast
majority of web sites will work fine if we don't either.

The cargo cult of double-quotes being meaningful in cookie attributes
has caused nothing but pain and misery in the world and should be
shot.

(Details below.)

> whether they are balanced, or not.

Indeed.

> This is IMO of significant concern since it breaks with how parameter values
> are defined and used in both MIME, HTTP, and RFC 2965, as (token |

The syntax of the Set-Cookie header differs from the syntax of other
things related to HTTP.  That's just an unfortunate fact of life.  If
we were designing things to be beautiful from the start, we would
probably do something different, but that's not how things work today.

As for RFC 2965, we're not creating any problems that don't exist
today.  We're just writing down how the cookie protocol works.  If
Cookie2 had deployment problems, that's a topic for another time.

> The handling of cookie values that are not tokens, and contain double quotes
> inside the value in a form that does not match the 2616 definition of
> quoted-string is according to my quick testing inconsistent between a few
> browsers (IE8, FF 3.6 and Opera 10.53),

Indeed.  Cookie values are not quoted strings.

> and AFAICT there is likely to be
> problem with Python based server-environments, like Django 1.1, which is the
> only one I have tested.

I suspect that Django 1.1 works fine in IE5, IE6, IE7, and IE8, which
means it will work fine w.r.t. the spec's treatment of quotes in
cookie values as (in this respect) the spec's behavior matches every
version of IE I could dig up for testing.

> to Django 1.1 or the Python 2.6 Cookie module, the result is the following

That's only an issue if Django sends a Set-Cookie header that
generates that Cookie header.  If somewhere there's a Django server
that does send such a Set-Cookie header, the author of that server is
likely expecting the Cookie header to parse that way because that's
how it works for the majority of user agents.  If they aren't
expecting that parsing, they'll discover it as soon as they test their
site with any version of IE.

By contrast, if we adopt Opera's parsing of the Set-Cookie header,
then we're much more likely to cause interoperability problems for
sites that rely on the more common parsing.

> In other words, when using quotes inside a value with any syntax other than
> quoted-string (which permit internal quoting if backslash escaping is used),
> the result does not just depend on the client the server is sending it to,
> it also depends on the script engine and its cookie parser.

Indeed.  Such is life.

> the quoted-string syntax, as well as specifically tell servers not to use
> spec's requirement of only using tokens). Perhaps one way to do that is to

I replied to this paragraph earlier.  Essentially, everything you're
asking for is already provided by the current draft.

> prohibit `Domain=localhost` for instance.

The spec just works the same way user agents work today.  If folks
with such servers are living happy lives today, they'll continue to
live happy lives.  If they're living unhappy lives, then we're not
doing them any additional harm.

On Sat, May 29, 2010 at 7:48 AM, Yngve N. Pettersen (Developer Opera
> covered in the domain sematics?

Yes.  Feel free to read the spec for details, but what happens
(essentially) is that you can set a host-only cookie for intranet
hosts but you cannot set a cookie without the host-only flag.

On Sat, May 29, 2010 at 10:54 AM, Yngve N. Pettersen (Developer Opera
> specifically mentions public suffix as a reason why a client can discard a

Yes.  The draft suggests that people discard cookies in ways that are
productive.  I'd rather not encourage folks to discard cookies in ways
that don't interoperate, but, from a protocol point of view, we need
to allow them to do that.  Hence the current text in the spec.

You're welcome to build a user agent that discards cookies with
unbalanced double-quotes.  However, you'll find your user agent in the
minority.  Instead, I'd recommend you build a user agent that doesn't
treat double-quote as a special character and we'll all live happier
lives.

> in combination with different cookie ordering algorithms. Putting the server
> administrators on notice about the issue should hopefully increase

The behavior for this particular value syntax is indeed defined.  You
can find the definition in Section 5.  We're already advising server
administrators not to use double-quotes.  That's the most helpful
piece of advise I know as it avoids the complexities here.  For
servers, there's no real reason to use double-quotes anyway.  It's
just a trail of tears.

> definitions themselves, or collected in a separate "Implementation

I agree that we should discuss this issue in more depth in the
deviation description document.  However, I think we should fill up
this document with a bunch of informative text about legacy user
agents except in cases where understanding that information is of
critical importance (e.g., for security).

Adam

---

From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Date: Sun, 30 May 2010 23:47:57 +0200

> %-escaping in URLs, but that's a matter best left to another working

target URI is /foo/%41/bar, path is /foo/a/

> site with any version of IE.

You are assuming that Django is the only framework involved in the system.  
And AFAICT this behavior comes from *Python's* module for cookie handling,  
and is *not* Django-only, meaning that potentially all Python based  
systems could encounter this problem. And keep in mind that this was just  
the frameworks I had immediately at hand, it may well be (probably very  
likely) that other frameworks are behaving in the same fashion.

I think it is quite possible that a parser such as Django/Python's can be  
integrated into an existing system that under some circumstances use  
unbalance quotes by another system. If so, testing might not reveal the  
issue until much later. One possibility is that the cookie's value is  
directly or indirectly derived from user input (which could be very  
dangerous in any case).

> sites that rely on the more common parsing.

Not too frequently I suspect; I can recall only one, maybe two, cases (and  
the one I remember showed up in my mailbox last week)

>> server

By "undefined" I mean that there are already deployed systems that behave  
in different fashions from what might be expected if unbalanced quotes are  
used.  Perhaps "the result is unpredicatable" would be a better  
description.

> administrators not to use double-quotes.  That's the most helpful

Not in that many words, but it is admittedly implied by the token  
specification of cookie-value. I think that in this case it is better to  
have a little redundancy by pointing out that this is one character where  
the "opaque" part is ignored by a number of clients (and may be ignored by  
serverside frameworks), and we are not just talking about Opera, we are  
also talking about a major client like FireFox 3.6 and earlier versions  
(whether or not nightlies have changed behavior in this area)

-- 
Sincerely,
Yngve N. Pettersen

---

From: Adam Barth <[email-masked]>
Date: Sun, 30 May 2010 16:52:36 -0700

On Sun, May 30, 2010 at 2:47 PM, Yngve N. Pettersen (Developer Opera
> target URI is /foo/%41/bar, path is /foo/a/

Right, so that depends on whether the user agent canonicalizes %41 to
"a" in its URL processing code.  I don't think we should be concerned
with that in this working group.  I'm also working on URL parsing and
canonicalization, so hopefully we'll get that stuff ironed out as
well.

> and is *not* Django-only, meaning that potentially all Python based systems
> could encounter this problem. And keep in mind that this was just the
> frameworks I had immediately at hand, it may well be (probably very likely)
> that other frameworks are behaving in the same fashion.
>

Regardless, either those systems work today or they don't.  If they
work today, then they'll work with the cookie protocol as specified.
If they don't work today, well, then they should be fixed.  We're not
introducing any new behavior that hasn't been deployed for many, many
years.

> the one I remember showed up in my mailbox last week)

By contrast, I don't think IE has gotten such an email in the past
decade despite shipping this behavior for as long as I can tell.

> By "undefined" I mean that there are already deployed systems that behave in
> different fashions from what might be expected if unbalanced quotes are
> used.  Perhaps "the result is unpredicatable" would be a better description.

Depends what sets peoples expectations.  In my experience, the first
thing to set people's expectations is the behavior of Internet
Explorer (especially in a standards vacuum, like we've had
historically with cookies).  That might not be a pleasant fact, and we
might wish to pretend it's not true, but that is the world in which we
find ourselves.

> serverside frameworks), and we are not just talking about Opera, we are also
> talking about a major client like FireFox 3.6 and earlier versions (whether
> or not nightlies have changed behavior in this area)

There are lots of exciting characters that cause wacky behavior.  Just
because your user agent happens to get tripped up on this one doesn't
mean it's worth calling out in the spec.  We could add reams of
rationale for everything in the document.  In this case, I think we'd
be better of explaining the life and times of the double-quote
character in the deviation description document.

Actually, the story is pretty engaging when told from the beginning as
a narrative (if you're into this sort of thing).  I'd rather set out
the full context and explanation in the other document.

Adam