Subject: [http-state] Ticket 11: Character encoding for non-ASCII cookies values

From: Adam Barth <[email-masked]>
Date: Tue, 2 Mar 2010 16:24:44 -0800

We had some earlier discussion about what to do with non-ASCII
characters in cookie values.

== On the wire ==

* IE, Firefox, Chrome, and Opera seem to treat non-ASCII characters as
opaque octets on the wire
* Safari seems to drop cookies with non-ASCII characters (although
Maciej said code inspection leads him to believe Safari's behavior is
a bit more complicated).

== In document.cookie ==

In <http://github.com/abarth/http-state/blob/master/notes/2010-02-03-Julian-Reschke.txt>,
Julian Reschke wrote:
[[
I just did a quick test with an ISO-8859-1 encoded cookie value,
client-side javascript and "alert(document.cookie)":
- IE and Firefox appear to treat the cookie as ISO-8859-1
- Safari appears to ignore the cookie
- Chrome and Opera appear to try to decode as UTF-8 (and returns a
REPLACEMENT CHAR in place of the umlaut I tried)
]]

== Proposal ==

The draft treats the cookie values as opaque octets throughout for use
on the wire.  I've added a SHOULD-level requirement to use a UTF8 when
converting the octets to characters (e.g., for use in the user agent's
user interface).

Given that the encoding issue doesn't appear to affect
interoperability on the wire, I think a SHOULD-level recommendation is
appropriate here.  If specific APIs (e.g., document.cookie) have more
specific needs, they can add additional requirements.

Thoughts?

Adam

---

From: Dan Witte <[email-masked]>
Date: Tue, 2 Mar 2010 16:43:03 -0800 (PST)

----- "Adam Barth" <[email-masked]> wrote:

> == In document.cookie ==
> 
> In
> <http://github.com/abarth/http-state/blob/master/notes/2010-02-03-Julian-Reschke.txt>,
> Julian Reschke wrote:
> [[

Firefox uses UTF-8, I believe.

A SHOULD level requirement for UTF-8 sounds good to me.

Regards,
Dan

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 2 Mar 2010 17:08:24 -0800

> specific needs, they can add additional requirements.

I think that is fine if it is made clear that UTF-8 is only applicable
after the field value is extracted from the rest of the message.  I.e.,
the HTTP parser must be ASCII-based and thus not vulnerable to
invalid Unicode byte sequences.

....Roy

---

From: Adam Barth <[email-masked]>
Date: Tue, 2 Mar 2010 21:46:32 -0800

> invalid Unicode byte sequences.

Hopefully that should be clear in the draft.  The encoding is mention
at the end of the serialization section (which is two sections after
the parsing section).

Adam

---

From: Achim Hoffmann <[email-masked]>
Date: Wed, 03 Mar 2010 15:32:53 +0100

> at the end of the serialization section (which is two sections after

IIRC previous discussions revealed that some browsers allow arbitrary data
for the cookie.
If the draft now (phase 1) recommends a special encoding, i.e. UTF-8, then
it violates the status quo. Does it?

If a coding like UTF-8 is recommended in phase 2, then this may result in
a transforming/canonicalisation/best-fit-mapping nightmare again.
Think of browser APIs (like JavaScript) which might use a different
encoding (i.e. UCS-2 as in ECMA-262).

Do I miss something here? If not all cookie data (key=value) SHOULD (phase 1)
or MUST (phase 2) be URL encoded.
This leaves the final data format open to whatever the application and/or
the browser wants but is transparent and secure on protocol level.

Achim

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Mar 2010 06:44:05 -0800

> IIRC previous discussions revealed that some browsers allow arbitrary data

That's correct.

> it violates the status quo. Does it?

Not quite.  The draft still allows arbitrary data for the on-the-wire
protocol, which is what I meant when I wrote: "The draft treats the
cookie values as opaque octets throughout for use on the wire."  The
UTF-8 recommendation is for other uses of cookie data, such as display
in the user agent's user interface.

> encoding (i.e. UCS-2 as in ECMA-262).

We can worry about phase 2 issues in phase 2.

> or MUST (phase 2) be URL encoded.

No one is saying anything about URL encoding.

> the browser wants but is transparent and secure on protocol level.

The server is free to use URL encoding.  In fact, the draft recommends
that the server encodes whatever data it wants to store into a
server-selected printable ASCII encoding (URL encoding, base64, etc).

Adam

---

From: Mark Pauley <[email-masked]>
Date: Wed, 3 Mar 2010 09:27:49 -0800

Yes, CFNetwork (the component responsible for cookie handling in Safari) currently drops any cookies created with non-ascii values.

In the future, we ought to treat these as opaque octets.  However, the current cookie spec would lead me to believe that we should reject any cookies that contain control characters, which would be most non-ascii UTF-8 sequences, right?

On Mar 2, 2010, at 4:24 PM, Adam Barth wrote:

> _______________________________________________
> http-state mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/http-state

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 3 Mar 2010 21:53:12 +0100 (CET)

On Wed, 3 Mar 2010, Mark Pauley wrote:

> In the future, we ought to treat these as opaque octets.  However, the 
> current cookie spec would lead me to believe that we should reject any 
> cookies that contain control characters, which would be most non-ascii UTF-8 
> sequences, right?

Isn't the RFC2616 'token' a bit too strict for cookie-value ? The netscape 
spec is _very_ liberal ("a sequence of characters excluding semi-colon, comma 
and white space") so the current wording is a great deal more restrictive.

Don't cookie implementations already allow and use for example ()<>:@? etc?

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Mar 2010 15:14:20 -0800

> In the future, we ought to treat these as opaque octets.  However, the current cookie spec would lead me to believe that we should reject any cookies that contain control characters, which would be…

That's not the intent of the current draft.  May I ask what leads you
to believe that?

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Mar 2010 15:16:48 -0800

On Wed, Mar 3, 2010 at 12:53 PM, Daniel Stenberg <[email-masked]> wrote:
> spec is _very_ liberal ("a sequence of characters excluding semi-colon,
> comma and white space") so the current wording is a great deal more
> restrictive.

To which wording are you referring?  We recommend that servers send
only tokens but require that user agent process a wide range of exotic
characters.

> Don't cookie implementations already allow and use for example ()<>:@? etc?

Yes they do.  They also allow $, which is problematic for Yngve's
(Phase 2) $Origin proposal.

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 4 Mar 2010 00:27:58 +0100 (CET)

On Wed, 3 Mar 2010, Adam Barth wrote:

> To which wording are you referring?  We recommend that servers send only 
> tokens but require that user agent process a wide range of exotic 
> characters.

Oh. That's not very clear with the current wording methinks.

I read the section 4.1.1 and it says that servers "SHOULD NOT send Set-Cookie 
headers that fail to conform to the following". Where is the text saying that 
clients MUST accept all those other characters?

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Mar 2010 15:32:01 -0800

On Wed, Mar 3, 2010 at 3:27 PM, Daniel Stenberg <[email-masked]> wrote:
>>> Isn't the RFC2616 'token' a bit too strict for cookie-value ? The
>>> netscape spec is _very_ liberal ("a sequence of characters excluding
>>> semi-colon, comma and white space") so the current wording is a great deal
>>> more restrictive.
>>
> I read the section 4.1.1 and it says that servers "SHOULD NOT send
> Set-Cookie headers that fail to conform to the following". Where is the text
> saying that clients MUST accept all those other characters?

It's a consequence of the parsing algorithm in
<http://tools.ietf.org/html/draft-ietf-httpstate-cookie-04#section-5.2>.
 In particular, how the name-value-pair and the cookie-value are
extracted from the set-cookie-string.

Adam

---

From: Mark Pauley <[email-masked]>
Date: Wed, 3 Mar 2010 16:51:52 -0800

You're correct, we do allow those.  What I'm worried about though is that valid UTF-8 sequences will contain \r\n .   In any case, writing good parsers for a variable-length is really tough.

To give you a sense of what Safari will accept, we'll accept the following for names or values in cookies:

CTL = 		[\x00-\x1F\x7F];
NETSCAPE_SPECIAL = ("," | ";" | "=" | SPACE | HTAB);
NONETSCAPE_VALCHAR = (CTL | NETSCAPE_SPECIAL);
NETSCAPE_CHAR = ((CHAR \ NONETSCAPE_VALCHAR) | COMMA_NOSPACE);
COMMA_NOSPACE = ((",")(CHAR \ (SPACE | HTAB | ";")));
NETSCAPE_VAL = (CHAR \ NONETSCAPE_VALCHAR)((NETSCAPE_CHAR | SPACE | HTAB)* NETSCAPE_CHAR)?;

And NETSCAPE_VAL is effectively our TOKEN.
We allow for quoted values, but those still reject the CTL class of character.

So, we certainly will choke on many UTF-8 character sequences.  In that case, we'll just ditch the cookie header as a whole.

One of the main problems so far with cookies has been that the Date spec used in the Netscape implementation clashes with the standard of HTTP to separate values in headers by a comma token.  That is I believe, some sites wish to set multiple cookies by using a comma separator whereas the comma is also used as part of an unquoted field.  If we can verify that nobody is actually trying to set multiple cookies per Set-Cookies header, then we can easily pull this functionality and be a bit more blind to the character set of the Set-Cookie header value.

On Mar 3, 2010, at 12:53 PM, Daniel Stenberg wrote:

> Isn't the RFC2616 'token' a bit too strict for cookie-value ? The netscape spec is _very_ liberal ("a sequence of characters excluding semi-colon, comma and white space") so the current wording is a great deal more restrictive.
>

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Mar 2010 17:17:28 -0800

See inline.

On Wed, Mar 3, 2010 at 4:51 PM, Mark Pauley <[email-masked]> wrote:
> You're correct, we do allow those.  What I'm worried about though is that
> valid UTF-8 sequences will contain \r\n .   In any case, writing good
> parsers for a variable-length is really tough.

This grammar is significantly more complicated than what appears to be
necessary.  Is there some reason the paring algorithm in
<http://tools.ietf.org/html/draft-ietf-httpstate-cookie-04#section-5.2>
is insufficient?

> We allow for quoted values, but those still reject the CTL class of

Safari is currently the only browser to treat quote characters in
cookie values differently from other characters.  (Note that Firefox
used to have this behavior but changed recently to match IE and
Chrome.)  Safari seems to have adopted this behavior in the past two
years.  Source inside Apple have indicated that this change was made
to improve compliance with RFC 2109 and not because of specific web
compatibility concerns.

> So, we certainly will choke on many UTF-8 character sequences.  In that
> case, we'll just ditch the cookie header as a whole.

Indeed.  Safari is currently the only browser that has this behavior.

> One of the main problems so far with cookies has been that the Date spec
> used in the Netscape implementation clashes with the standard of HTTP to
> separate values in headers by a comma token.  That is I believe, some sites
> wish to set multiple cookies by using a comma separator whereas the comma is
> also used as part of an unquoted field.  If we can verify that nobody is
> actually trying to set multiple cookies per Set-Cookies header, then we can
> easily pull this functionality and be a bit more blind to the character set
> of the Set-Cookie header value.

The date parsing algorithm is specified in detail in
<http://tools.ietf.org/html/draft-ietf-httpstate-cookie-04#section-5.1.1>.

Unlike most HTTP headers, a comma cannot be used to combine multiple
Set-Cookie headers.  Safari appears to be the only browser that
attempts to support this behavior.

If you haven't already, I'd encourage you to read the latest draft
available at <http://tools.ietf.org/html/draft-ietf-httpstate-cookie>.
 Hopefully that will answer some number of your questions.

Adam

---

From: Mark Pauley <[email-masked]>
Date: Wed, 3 Mar 2010 17:24:42 -0800

> to improve compliance with RFC 2109 and not because of specific web

That's partially correct.  We received a bug from an external source that drove this, though we didn't immediately change because of the lack of compatibility issues as you say.

> Hopefully that will answer some number of your questions.

Interesting.  Well, this could certainly simplify much of our internal cookie parsing mechanism.  Is there going to be a push to deprecate RFC2109?  Could you point me at a page with results on how the browsers handle different Set-Cookie forms in practice?  Changing our behavior is nearly always dictated by broken compatibility with other sites.

---

From: Adam Barth <[email-masked]>
Date: Wed, 3 Mar 2010 17:54:10 -0800

On Wed, Mar 3, 2010 at 5:24 PM, Mark Pauley <[email-masked]> wrote:
> Interesting.  Well, this could certainly simplify much of our internal cookie parsing mechanism.  Is there going to be a push to deprecate RFC2109?

Our charter is to depreciate RFC2109 in favor of the spec we're writing.

> Could you point me at a page with results on how the browsers handle different Set-Cookie forms in practice?  Changing our behavior is nearly always dictated by broken compatibility with other sites.

We're going to produce such a document once the spec settles down a
bit more.  Currently, the best documentation we have is the spec
itself and the test suite I referenced before.  The spec is intended
to be very close to how browsers actually behave.  In particular, IE,
Firefox, and Chrome pass the vast majority of the tests in the test
suite.

If you have a question about a specific form, I can usually tell you
the compatibility matrix.  If I can't, that means we need to write
more tests.  :)

Adam

---

From: Mark Pauley <[email-masked]>
Date: Wed, 3 Mar 2010 18:08:17 -0800

Perfect.  I'll stay tuned.

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 4 Mar 2010 09:23:43 +0100 (CET)

On Wed, 3 Mar 2010, Adam Barth wrote:

>> Set-Cookie headers that fail to conform to the following". Where is the 
>> text saying that clients MUST accept all those other characters?
>
> <http://tools.ietf.org/html/draft-ietf-httpstate-cookie-04#section-5.2>. In 
> particular, how the name-value-pair and the cookie-value are extracted from 
> the set-cookie-string.

Yeah, that is in my eyes strikingly hard to comprehend and find info in when 
used as a reference. The step-by-step parsing algorithm is to me a very 
strange way to document a protocol syntax. Not the least because there's 
already ABNF syntax for 4.1.1 and even in 5.1.1.

I'm not sure how I'd like to see this fixed but I would prefer an ABNF 
approach with added appropriate prose rather than the long free text algorithm 
in 5.2. We're documenting a protocol, and if that protocol can be implemented 
using other code flows then I don't think it is our spec's job to limit that 
as long as the protocol is followed.

I'll try to write up a suggestion for this soonish.

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Thu, 4 Mar 2010 05:32:46 -0800

On Thu, Mar 4, 2010 at 12:23 AM, Daniel Stenberg <[email-masked]> wrote:
> approach with added appropriate prose rather than the long free text
> algorithm in 5.2. We're documenting a protocol, and if that protocol can be
> implemented using other code flows then I don't think it is our spec's job
> to limit that as long as the protocol is followed.
>

Thanks.  I too would prefer a declarative presentation, but we need to
be careful not to sacrifice accuracy.

Adam

---

From: Adam Barth <[email-masked]>
Date: Wed, 10 Mar 2010 15:18:11 -0800

I'm going to close this ticket because no one objected to this resolution.

Adam