Subject: Set-Cookie vs list header parsing (i129), was: NEW ISSUE: repeating non-list-type-headers

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Aug 2008 22:07:53 +0200

Julian Reschke wrote:
> 
> Hi.
> 
> It seems to me that we really should open a separate issue (*) for this 
> one, so that it doesn't get lost.
> ...

Opened as <http://tools.ietf.org/wg/httpbis/trac/ticket/129>.

BR, Julian

---

From: Dave Kristol <[email-masked]>
Date: Thu, 14 Aug 2008 11:47:54 -0400

On 08/13/2008 04:07 PM, Julian Reschke wrote:
> 
> Julian Reschke wrote:
>>
>> Hi.
>>
>> It seems to me that we really should open a separate issue (*) for 
>> this one, so that it doesn't get lost.
>> ...
> 

Regarding this comment:

(That nobody implements RFC2109 is implied in RFC2965, which obsoletes
RFC2109 and in section 9 talks about using Set-Cookie2 alongside
Netscape style Set-Cookies, not mentioning RFC2109 style Set-Cookiess. I
think this reflects the observation at the time that the change of
Set-Cookie syntax promoted in RFC2109 wasn't taken up, probably because
it's not backward compatible.)

I wrote a paper that describes the standardization process for cookies
in excruciating detail.  You can get it at
<http://arxiv.org/abs/cs.SE/0105018>.  I'll refer to some of its
sections below

Appendix section A.2, particularly A.2.3, discusses the problem of
"folding" multiple Cookie headers, that is the problem of "," and ";"
separators.  I suspect (but have no proof) that, in self-defense,
current clients and servers treat Cookie as a special case and are
careful to send each cookie in its own header, rather than merge them.

Appendix B describes where Set-Cookie2 came from.  It had nothing to do
with "," vs. ";", at least originally.  Work on what became RFC 2965
began shortly after RFC 2109 came out, to fix an incompatibility we
found.  That work began well before RFC 2109 would have had any time to
be adopted.  The long time gap between RFC 2109 and RFC 2965 arose from
other factors.  See section 4.3.3.

It was certainly our goal (see section 4.3) to introduce upward- (or is
it downward- ?) compatible changes, though we had to deal with the hand
that Netscape's specification dealt us.  We obviously didn't succeed.

Dave Kristol

---

From: Julian Reschke <[email-masked]>
Date: Fri, 15 Aug 2008 20:21:53 +0200

> that Netscape's specification dealt us.  We obviously didn't succeed.

Dave,

thanks for the pointer. I wish we had more documentation like this to 
help those who joined the WG much later.

With respect to the original issue -- I think it would be a service to 
readers to point out that this:

    Multiple message-header fields with the same field-name MAY be
    present in a message if and only if the entire field-value for that
    header field is defined as a comma-separated list [i.e., #(values)].
    It MUST be possible to combine the multiple header fields into one
    "field-name: field-value" pair, without changing the semantics of the
    message, by appending each subsequent field-value to the first, each
    separated by a comma. The order in which header fields with the same
    field-name are received is therefore significant to the
    interpretation of the combined field value, and thus a proxy MUST NOT
    change the order of these field values when a message is forwarded. 
-- <http://tools.ietf.org/html/rfc2616#section-4.2>

...is wishful thinking in practice, because of Cookie and Set-Cookie (as 
used in practice, not defined in RFC2109).

How about:

--- snip ---

Note: the "Cookie" and "Set-Cookie" headers as implemented in practice 
(as opposed to how they are specified in [RFC2109]) can occur multiple 
lines, but do not use the list syntax, and thus can not be combined into 
a single line (see [Kri2001] for details). Also note that the 
Cookie2/Set-Cookie2 headers specified in [RFC2965] do not share this 
problem).

--- snip ---

BR, Julian

---

From: Dave Kristol <[email-masked]>
Date: Fri, 15 Aug 2008 14:44:46 -0400

Julian Reschke wrote:
> [...]

Sounds good.

Corrections:
can occur multiple lines -> can occur multiple times
line (see [Kri2001] for details). -> line.  (See [Kri2001] for details.)
do not share this problem). -> do not share this problem.

Dave

---

From: Julian Reschke <[email-masked]>
Date: Fri, 15 Aug 2008 21:46:23 +0200

> do not share this problem). -> do not share this problem.

OK. Proposed patch: 
(<http://www3.tools.ietf.org/wg/httpbis/trac/attachment/ticket/129/i129.diff>):

at the end of 4.2:

       Note: the "Cookie" and "Set-Cookie" headers as implemented in
       practice (as opposed to how they are specified in [RFC2109]) can
       occur multiple times, but do not use the list syntax, and thus can
       not be combined into a single line.  (See [Kri2001] for details.)
       Also note that the Cookie2/Set-Cookie2 headers specified in
       [RFC2965] do not share this problem.

Informative References:

    [Kri2001]  Kristol, D., "HTTP Cookies: Standards, Privacy, and
               Politics", ACM Transactions on Internet Technology Vol. 1,
               #2, November 2001, <http://arxiv.org/abs/cs.SE/0105018>.

    [RFC2109]  Kristol, D. and L. Montulli, "HTTP State Management
               Mechanism", RFC 2109, February 1997.

    [RFC2965]  Kristol, D. and L. Montulli, "HTTP State Management
               Mechanism", RFC 2965, October 2000.

...more feedback appreciated.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Fri, 15 Aug 2008 15:31:29 -0500

Julian Reschke wrote:
>   practice (as opposed to how they are specified in [RFC2109])
>   can occur multiple times, but do not use the list syntax, 
>   and thus can not be combined into a single line.
>   (See [Kri2001] for details.) Also note that the
>   Cookie2/Set-Cookie2 headers specified in [RFC2965] do not
>   share this problem.

"Cannot" is better than "can not" here. But, what exactly does "cannot" mean
in a specification? Instead of making this a note, it is better to make it a
normative part of the specification using RFC 2119 language:

    The original definition of the "Cookie" and "Set-Cookie"
    header fields used ";" instead of "," as the list separator,
    and this legacy syntax is still in common use. To maximize
    compatibility, clients MUST send at most one "Cookie"
    header field, origin servers MUST send at most one "Set-Cookie"
    header field, and proxies MUST NOT combine multiple "Cookie"
    or "Set-Cookie" header field values. See section A.2.3 of
    [Kri2001] for details.

As Dave said, this seems to be what the majority of implementations actually
do.

Dave Kristol's paper is excellent but it is large and it talks about
political things as much as it talks about technical concerns. That is why I
changed the citation to mention section A.2.3 specifically.

Regards,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 15 Aug 2008 23:07:27 +0200

> "Cannot" is better than "can not" here. But, what exactly does "cannot" mean

Ack.

> normative part of the specification using RFC 2119 language:

I doubt that we can get consensus to add more than a note here. But 
let's hear what the WG thinks.

> changed the citation to mention section A.2.3 specifically.

Ack.

-> 
<http://www.tools.ietf.org/wg/httpbis/trac/attachment/ticket/129/i129.2.diff>

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Sat, 16 Aug 2008 07:23:20 +0200

>            #2, November 2001, <http://arxiv.org/abs/cs.SE/0105018>.

Fascinating.  This sounds even worse than the MARID bloodshed.

[censoring my rant about ping attributes, 3rd party cookies,
 HTML5, doubleclick, hostile to privacy, and more off topics]

> ...more feedback appreciated.

Maybe add a reference to RFC 2964 for the complete cookie zoo.

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Wed, 20 Aug 2008 09:32:25 +0200

> Maybe add a reference to RFC 2964 for the complete cookie zoo.

To be complete we would also need to cite the original spec 
(<http://www.netscape.com/newsref/std/cookie_spec.html>, 404s...). We 
already have three cookie-related references; enough is enough, isn't it?

The currently proposed text is at: 
<http://www3.tools.ietf.org/wg/httpbis/trac/attachment/ticket/129/i129.diff>

Brian also proposed to make this REQUIRED behavior. I don't think we 
have agreement on this -- for once, handling Set-Cookie and/or 
Set-Cookie2 is optional anyway.

So unless something new comes up soon, I'll apply the patch in the next 
few days. If we later agree on making this normative, we still can do that.

BR, Julian

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 20 Aug 2008 09:38:44 +0200 (CEST)

> already have three cookie-related references; enough is enough, isn't it?

BTW, I noticed this 404 a while ago and we're now offering that same document 
here:

 	http://curl.haxx.se/rfc/cookie_spec.html

Just in case someone missed it before netscape took it away.

Oh, and you can also see the last version archive.org knows of it at the 
previous URL:

http://web.archive.org/web/20060615215758/wp.netscape.com/newsref/std/cookie_spec.html

---

From: Dan Winship <[email-masked]>
Date: Wed, 20 Aug 2008 08:58:03 -0400

> already have three cookie-related references; enough is enough, isn't it?

Well, but that one is more worth citing than some of the others, since
it's pretty much what people actually implement in practice.

> <http://www3.tools.ietf.org/wg/httpbis/trac/attachment/ticket/129/i129.diff>

AFAIK, the problem is only with "Set-Cookie", not "Cookie". (There's no
need to send multiple Cookie headers; the spec says you're supposed to
include all of the cookies, semicolon-delimited, in a single Cookie header.)

> Brian also proposed to make this REQUIRED behavior.

FWIW, 3 out of the big 4 browsers also don't correctly parse multiple
WWW-Authenticate headers that have been merged into one (even though
2617 explicitly points out this possibility). So it might be best to
just say that intermediaries SHOULD NOT merge headers, except in cases
where they know it's safe.

-- Dan

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 20 Aug 2008 14:04:38 +0100

> where they know it's safe.

Do they handle unmerged, multiple WWW-Authenticate headers correctly?

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Wed, 20 Aug 2008 15:34:16 +0200

> it's pretty much what people actually implement in practice.

It's indirectly referenced through RFC2965, which now has an erratum 
pointing out the backup URL (thanks, Daniel) -- see 
<http://www.rfc-editor.org/errata_search.php?rfc=2965>.

> include all of the cookies, semicolon-delimited, in a single Cookie header.)

OK, see 
<http://www3.tools.ietf.org/wg/httpbis/trac/attachment/ticket/129/i129.3.diff>.

> where they know it's safe.

Time for test cases and bug reports, I think.

BR, Julian

---

From: Dan Winship <[email-masked]>
Date: Wed, 20 Aug 2008 09:54:36 -0400

> Do they handle unmerged, multiple WWW-Authenticate headers correctly?

Yes. This results in Basic auth being performed:

    WWW-Authenticate: UnknownAuthType realm=Realm1, prop1=value1,
            prop2=value2
    WWW-Authenticate: Basic realm=Realm2

But this does not:

    WWW-Authenticate: UnknownAuthType realm=Realm1, prop1=value1,
            prop2=value2, Basic realm=Realm2

-- Dan

---

From: Julian Reschke <[email-masked]>
Date: Thu, 21 Aug 2008 10:17:58 +0200

> <http://www3.tools.ietf.org/wg/httpbis/trac/attachment/ticket/129/i129.3.diff>. 

For now, I have submitted the change, and closed the issue (see 
<http://tools.ietf.org/wg/httpbis/trac/changeset/310>).

The new text now states:

       Note: the "Set-Cookie" header as implemented in practice (as
       opposed to how it is specified in [RFC2109]) can occur multiple
       times, but does not use the list syntax, and thus cannot be
       combined into a single line.  (See Appendix A.2.3 of [Kri2001] for
       details.)  Also note that the Set-Cookie2 header specified in
       [RFC2965] does not share this problem.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 27 Aug 2008 15:15:03 -0500

Julian Reschke wrote:
> For now, I have submitted the change, and closed the issue 
> (see <http://tools.ietf.org/wg/httpbis/trac/changeset/310>).

The ticket is still open in the tracker.

>        combined into a single line.  (See Appendix A.2.3 of 
>        [Kri2001] for details.)  Also note that the Set-Cookie2 header
>        specified in [RFC2965] does not share this problem.

I am not against having this note but it doesn't address the real problem.
RFC 2616 required everybody to define header fields with syntax that is
compatible with the list syntax, but that requirement hasn't been followed.
In general, an intermediary cannot tell the difference between a header
field that is multi-valued using the list syntax, a header field that is
multi-valued but (illegally) using some other list syntax, and a header
field that is single-valued but erroneously repeated. If an intermediary
combines two single-valued header fields together then it could change the
meaning of the request/response if the combined value is also a legal value
for that field (see http://tools.ietf.org/wg/httpbis/trac/ticket/93).

Like Dan said, it is best to say an intermediary SHOULD NOT combine headers
fields in requests. Also, a client SHOULD NOT send requests with multiple
header fields with the same name, because many commonly used server gateways
(e.g. CGI and derivatives like Python's WSGI and Ruby's Rack) only allow a
single instance of each request header field. If the client provides more
than one header field with the same name, then the application server (for
these frameworks) will have to combine them. Some implementations don't do
this correctly (especially WSGI implementations).

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Aug 2008 22:35:06 +0200

> The ticket is still open in the tracker.

Indeed. The updates of Trac and SVN seem to have affected the commit 
scripts (which, in this case, should have closed the ticket).

> compatible with the list syntax, but that requirement hasn't been followed.

Yes (mod. ordering of events in time).

> for that field (see http://tools.ietf.org/wg/httpbis/trac/ticket/93).

Does this affect more headers than Set-Cookie?

> this correctly (especially WSGI implementations).

I'm not totally sure what your suggestion is.

I don't think that changing things just because some implementations get 
them wrong is on our agenda.

The case for Set-Cookie is different, because any implementation that 
does support (old-style) cookies must be aware of the difference wrt 
RFC2616's rules.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 27 Aug 2008 19:17:56 -0500

> Does this affect more headers than Set-Cookie?

Dan pointed out that it also affects WWW-Authenticate. Dan's point (which I
agree with) is that since we've already found two specific header fields
where combining is problematic, it is safer to just recommend that
implementors avoid the problem generally. Attempting to solve the problem by
enumerating the header fields that are known to be problematic is too
brittle.

> > Like Dan said, it is best to say an intermediary SHOULD NOT combine 
> > headers fields in requests. Also, a client SHOULD NOT send requests 
> > with multiple header fields with the same name, because 
> > many commonly used server gateways (e.g. CGI and derivatives like 
> > Python's WSGI and Ruby's Rack) only allow a single instance of each
> > request header field. If the client provides more than one header
> > field with the same name, then the application server (for these
> > frameworks) will have to combine them. Some implementations don't
> > do this correctly (especially WSGI implementations).
> 

Proxies SHOULD NOT combine header fields. Clients SHOULD combine header
fields in requests so that  there is at most one instance of each header
field in the request. 

> I don't think that changing things just because some 
> implementations get them wrong is on our agenda.

AFAICT, this is the primary motivation for HTTPbis--to clarify parts of the
specification where there are known interoperability problems. A
proxy/client that follows the two recommendations above will interoperate
correctly with more implementations than one that doesn't follow those
recommendations.

> The case for Set-Cookie is different, because any 
> implementation that does support (old-style) cookies must be 
> aware of the difference wrt RFC2616's rules.

Again, WWW-Authenticate has the same problem, and other header fields
probably do to.

Regards,
Brian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 27 Aug 2008 18:27:51 -0700

> enumerating the header fields that are known to be problematic is too

That is irrelevant.  We are specifying a deployed protocol, not
something we make up as we go along.  HTTP as deployed says that
all repeated header fields can be folded and that is exactly what
implementations do, with a specific exception for Set-Cookie
(because it was defined outside the IETF process).  I do not know
of any such exception for WWW-Authenticate.

If an implementation can't handle folding, then fix it.

....Roy

---

From: Dan Winship <[email-masked]>
Date: Thu, 28 Aug 2008 09:54:45 -0400

> them wrong is on our agenda.

I didn't mean to suggest actually changing the header merging rules.
Maybe I should have said "proxies should not merge" rather than "proxies
SHOULD NOT merge". Advice, not requirements.

Basically, we know that multiple implementations get this section wrong
in different ways (the cookie spec, the WWW-Authenticate bugs, the
ignoring-multiple-header bugs Brian mentioned), so this is a really good
place to "be conservative in what you send" (meaning multiples of
Set-Cookie, WWW-Authenticate, and Proxy-Authenticate, and no multiples
of anything else).

-- Dan

---

From: Yves Lafon <[email-masked]>
Date: Thu, 28 Aug 2008 10:32:48 -0400 (EDT)

> for that field (see http://tools.ietf.org/wg/httpbis/trac/ticket/93).

If there an example of a repeated single-valued header that, once folded 
in one list-valued header becomes legal?
  Content-Length: 12
  Content-Length: 42
is as illegal as
  Content-Length: 12,42

(and more than likely to trigger a 400 reply)

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 28 Aug 2008 10:04:51 -0500

Yves Lafon wrote:
> > field that is single-valued but erroneously repeated. If an 
> > intermediary combines two single-valued header fields 
> together then it 
> > could change the meaning of the request/response if the 
> combined value 
> > is also a legal value for that field (see 
> http://tools.ietf.org/wg/httpbis/trac/ticket/93).
> 
> If there an example of a repeated single-valued header that, 
> once folded in one list-valued header becomes legal?
>   Content-Length: 12
>   Content-Length: 42
> is as illegal as
>   Content-Length: 12,42

Off the top of my head: Location, Content-Location, and Referer. 

Location: http://foo.com/a
Location: http://foo.com/b

Location: http://foo.com/a,http://foo.com/b

Regards,
Brian

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 28 Aug 2008 10:08:09 -0500

Dan Winship wrote:
> Maybe I should have said "proxies should not merge" rather 
> than "proxies SHOULD NOT merge". Advice, not requirements.

IMO, that is not much different. "SHOULD" is only used for advise; by
definition it means the same thing as "RECOMMENDED."

> Basically, we know that multiple implementations get this 
> section wrong in different ways (the cookie spec, the 
> WWW-Authenticate bugs, the ignoring-multiple-header bugs 
> Brian mentioned), so this is a really good place to "be 
> conservative in what you send" (meaning multiples of 
> Set-Cookie, WWW-Authenticate, and Proxy-Authenticate, and no 
> multiples of anything else).

I agree 100%.

Dan, you mentioned 3 of the top 4 browsers cannot handle a merged
WWW-Authenticate. Which one got it right?

Thanks,
Brian

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Thu, 28 Aug 2008 13:14:58 -0500

> (and more than likely to trigger a 400 reply)

You understand that both representations MUST cause a 400 reply, of course?
The folding is irrelevant to the underlying flaw/contradiction.

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 28 Aug 2008 14:04:50 -0500

> The folding is irrelevant to the underlying flaw/contradiction.

That isn't what the spec says, AFAICT. Surely the proxy or the server MAY
report an error but they are not required to. More generally, there is
currently no requirement for the server to report syntax errors that do not
affect its processing of the message. For example, if the request included a
malformed Content-Location header, then the server wouldn't have to report
an error if it ignores Content-Location. Content-Length is a special case
because of its wide-ranging effects on parsing, but even for it there isn't
any such requirement. See issue 95 [1] and issue 93 [2].

[1] http://tools.ietf.org/wg/httpbis/trac/ticket/95
[2] http://tools.ietf.org/wg/httpbis/trac/ticket/93

- Brian

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Thu, 28 Aug 2008 14:15:39 -0500

> [2] http://tools.ietf.org/wg/httpbis/trac/ticket/93

Good points.  A most lenient server/client could choose the smaller of the
numbers, and assume connection: close as it doesn't know where to begin
(the contradiction inhibits the ability to determine valid keep alive/next
request point).  This is similar to the presence of both content length and
content encoding of chunked.

But error handling is left to the client.  The condition is an error,
so you are correct when you say the spec has nothing to say about the 
contradiction.  I wonder if your notes on 95 are better left to commentary
under Security Considerations?

---

From: Yves Lafon <[email-masked]>
Date: Thu, 28 Aug 2008 15:19:07 -0400 (EDT)

On Thu, 28 Aug 2008, William A. Rowe, Jr. wrote:

>> If there an example of a repeated single-valued header that, once folded in 
>> one list-valued header becomes legal?
>>  Content-Length: 12
>>  Content-Length: 42
>> is as illegal as
>>  Content-Length: 12,42
>> 

Yes, that was the point of my email, an error in two headers is still an 
error once folded :)
The Location use case in interesting, as it depends if in the 
implementation the folding is done before verification of the HTTP headers 
structure, or at the same time, triggering different level of errors in 
the application.

         ~~Yves

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 28 Aug 2008 23:18:13 +0200

On ons, 2008-08-27 at 15:15 -0500, Brian Smith wrote:

> fields in requests.

No. It's fine to do so on known list headers, such as Cache-Control or
If-None-Match, and quite many do.

"SHOULD NOT combine unknown headers" is perhaps a better wording.

Regards
Henrik