Subject: Multiple header fields with the same field name - unwritten assumption about quoted commas in values?

From: Piotr Dobrogost <[email-masked]>
Date: Wed, 9 Jan 2013 01:28:53 +0100

Hi!

In http://trac.tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-21#section-3.2
one reads

"Multiple header fields with the same field name MUST NOT be sent in a
 message unless the entire field value for that header field is
 defined as a comma-separated list [i.e., #(values)]."

Is the unwritten assumption that each value out of "values" has no
unquoted comma?

Regards,
Piotr Dobrogost

---

From: Zhong Yu <[email-masked]>
Date: Tue, 8 Jan 2013 20:22:12 -0600

> Is the unwritten assumption that each value out of "values" has no

Good question. I think it is true for all headers defined in the spec
that use the #rule; I guess it is intended to be true in general.

Zhong Yu

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jan 2013 12:00:04 +0100

> that use the #rule; I guess it is intended to be true in general.

It's a good idea to do, but not always the case. Example: "If-Match" and 
"If-None-Match". So to separate values in a list production, code needs 
to understand the syntax of the individual values.

Best regards, Julian

---

From: Piotr Dobrogost <[email-masked]>
Date: Wed, 9 Jan 2013 16:32:06 +0100

On Wed, Jan 9, 2013 at 12:00 PM, Julian Reschke <[email-masked]> wrote:
> "If-None-Match". So to separate values in a list production, code needs to
> understand the syntax of the individual values.

I don't quite understand what you meant to say above.
What is a good idea?
What's special about "If-Match" and "If-None-Match" headers?

> So to separate values in a list production, code needs to understand the syntax of the individual values.

True but is this mentioned somewhere?

Regards,
Piotr Dobrogost

---

From: Karl Dubost <[email-masked]>
Date: Wed, 9 Jan 2013 17:10:24 +0900

Le 9 janv. 2013 à 09:28, Piotr Dobrogost a écrit :
> "Multiple header fields with the same field name MUST NOT be sent in a message unless the entire field value for that header field is defined as a comma-separated list [i.e., #(values)]."

To add to Piotr comment and give a bit of context: 

Let's imagine a situation where I want to send multiple qualified Link: such as:

   Link: </2012/07/26/csstests/csshttplink.css>;rel=stylesheet
   Link: </>;rel=next

According to the quoted paragraph, The sender (server in this case) MUST not send that but 

   Link: </2012/07/26/csstests/csshttplink.css>;rel=stylesheet, </>;rel=next

Is it a correct understanding?

Then there are cases where some Headers have values with commas. Am I correct?

http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-21#section-9.3.1

> Because commas (",") are used as a generic delimiter between field-
> values, they need to be treated with care if they are allowed in the
> field-value's payload.  Typically, components that might contain a
> comma are protected with double-quotes using the quoted-string ABNF
> production (Section 3.2.4 of [Part1]).
> 
> For example, a textual date and a URI (either of which might contain
> a comma) could be safely carried in field-values like these:
> 
>   Example-URI-Field: "http://example.com/a.html,foo",
>                         "http://without-a-comma.example.com/"
>   Example-Date-Field: "Sat, 04 May 1996", "Wed, 14 Sep 2005"
> 
> Note that double-quote delimiters almost always are used with the
> quoted-string production; using a different syntax inside double-
> quotes will likely cause unnecessary confusion.

-- 
Karl Dubost - http://dev.opera.com/
Developer Relations, Opera Software

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 09 Jan 2013 16:59:41 +0100

>Is it a correct understanding?

The Link header appears to be defined as taking a comma-separated list,
<http://tools.ietf.org/html/rfc5988#section-5>, so the "MUST NOT" above
does not apply to the Link header.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Zhong Yu <[email-masked]>
Date: Wed, 9 Jan 2013 10:29:51 -0600

On Wed, Jan 9, 2013 at 5:00 AM, Julian Reschke <[email-masked]> wrote:
>>> In
>>> http://trac.tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-21#section-3.2
>>> one reads
>>>

The commas in entity tags are "quoted", so it's not a problem for a
generic parser which pairs quotes.

    If-Match: "12,34", W/"56,78"

The Link header can contain unquoted commas, since comma is legal in URIs

    Link: </page,6>; rel="Previous",  </page,8>; rel="Next"

fortunately these commas will be surround by <>, no ambiguity there.

The same bis section also specifically mentions the exception to
`Set-Cookie` - each cookie must be declared in its own line, because
unquoted comma can appear in the line

    Set-Cookie: name=value; path=/ab,1; expires=Wed, 09 Jun 2021 10:18:14 GMT

If we are to put multiple cookies on one line, the comma in the expiry
date does not cause ambiguity, but the comma in the path value could

    Set-Cookie: n1=v1; path=/ab,1 , n2=v2

Zhong Yu

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jan 2013 17:30:54 +0100

> I don't quite understand what you meant to say above.

It is a good idea to define header fields in a way so that they can be 
parsed unambiguously.

> What's special about "If-Match" and "If-None-Match" headers?

The allow ",", but do not use quoted-string syntax.

> True but is this mentioned somewhere?

Not at the moment, but maybe we should mention it.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jan 2013 17:38:07 +0100

>      If-Match: "12,34", W/"56,78"

But they do not use the quoted-string ABNF production (anymore). Note 
that that would imply handling of "\" which we found almost nobody does. 
This also means that just "pairing" double quotes is not sufficient.

> fortunately these commas will be surround by <>, no ambiguity there.

Yes, but the parser needs to understand the ABNF of the field value in 
order to decide which is which.

>      Set-Cookie: n1=v1; path=/ab,1 , n2=v2

Yes, Set-Cookie is a mess. That's why we needed that to call out 
specifically.

Best regards, Julian

---

From: Piotr Dobrogost <[email-masked]>
Date: Wed, 9 Jan 2013 18:05:40 +0100

On Wed, Jan 9, 2013 at 5:38 PM, Julian Reschke <[email-masked]> wrote:
> But they do not use the quoted-string ABNF production (anymore). Note that
> that would imply handling of "\" which we found almost nobody does. This
> also means that just "pairing" double quotes is not sufficient.

I'm not sure I follow - I'll try to rephrase it. In the past entity
tags were specified as using quoted-string ABNF production but because
existing implementations did not implemented this grammar production
fully (they did not handle backslash as they should) you removed this
production from specs. The outcome is that "standard" parser which
parses quoted-string ABNF production can't parse entity tags anymore.
Does it mean that If-Match header field's value does no longer is in
the form of comma-separated list [i.e., #(values)] thus there can't be
multiple If-Match header fields?

> Yes, but the parser needs to understand the ABNF of the field value in order
> to decide which is which.

This leads to some questions:
What are the requirements on parser of conforming user agent in regard
to parsing header fields' values? Is it required to parse header
fields' values of all registered header fields? Is it required to
parse header fields' values for header fields which it handles? Is it
required to parse quoted-string values?

Regards,
Piotr Dobrogost

---

From: Zhong Yu <[email-masked]>
Date: Wed, 9 Jan 2013 11:08:36 -0600

On Wed, Jan 9, 2013 at 10:38 AM, Julian Reschke <[email-masked]> wrote:
>>> to

Do you mean if a server responds

    Entity-Tag: "abc\""

most clients will fail to parse it? But if the server changes it to

    Entity-Tag: "abc\"

it'll fail on good clients which are conformant to RFC2616.

Zhong Yu

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jan 2013 18:22:59 +0100

> parses quoted-string ABNF production can't parse entity tags anymore.

It won't parse entity tags that contain a backslash. We made that change 
because we believe that it reflects would current software actually 
does. And as entity tags are always minted by the server, there really 
shouldn't be a case where the client reformats the entity tag at all.

> multiple If-Match header fields?

No. Why don't you just look it up? -> 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p4-conditional-latest.html#header.if-match>

> required to parse quoted-string values?

I don't understand. Why (and how) would you "parse" field values that 
you don't know? To "parse", you need an ABNF, no?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jan 2013 18:25:45 +0100

> most clients will fail to parse it? But if the server changes it to

What we tested is whether servers that send an ETag of "X" will also 
process "\X" and treat it as equivalent. I believe the outcome was that 
they don't.

> it'll fail on good clients which are conformant to RFC2616.

Yes. Do you have information that this happens in practice? ((a) servers 
using backslash in entity tags, and/or (b) clients parsing them as 
quoted-string?)

Best regards, Julian

---

From: Alexander Dutton <[email-masked]>
Date: Wed, 09 Jan 2013 17:32:10 +0000

Also worth noting the WWW-Authenticate header, which takes
comma-separated values, themselves containing unquoted commas (and so
breaks the alleged unwritten rule), e.g.:

WWW-Authenticate: Bearer realm="example.org", error="invalid_token",
Basic realm="example.org"

The division between items is the comma-space before a token not
followed by a comma-space.

(Yes, I've had fun because of this)

Yours,

Alex

-- 
Alexander Dutton
Linked Open Data Architect, Office of the CIO; data.ox.ac.uk, OxPoints
IT Services, University of Oxford, ℡ 01865 (6)13483

---

From: Piotr Dobrogost <[email-masked]>
Date: Wed, 9 Jan 2013 18:36:28 +0100

On Wed, Jan 9, 2013 at 6:22 PM, Julian Reschke <[email-masked]> wrote:
> I don't understand. Why (and how) would you "parse" field values that you
> don't know? To "parse", you need an ABNF, no?

Right, you need ABNF to parse; I did not say otherwise. However I
don't see how this answers any of the above questions.

Regards,
Piotr

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jan 2013 18:40:23 +0100

> (Yes, I've had fun because of this)

Indeed. WWW-Authenticate is as bad as Set-Cookie, just in a different 
way. At least it's *possible* to process properly.

See also <http://greenbytes.de/tech/tc/httpauth/#multidisgscheme>.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jan 2013 18:47:32 +0100

> don't see how this answers any of the above questions.

"Is it required to parse header fields' values of all registered header 
fields?" -> No, why would it?

"Is it required to parse header fields' values for header fields which 
it handles?" -> If it "handles" them it needs to parse them, no?

"Is it required to parse quoted-string values?" -> Yes, to the extent 
it's needed to extract the information it needs (charset param on 
Content-Type comes to mind).

Best regards, Julian

---

From: Zhong Yu <[email-masked]>
Date: Wed, 9 Jan 2013 13:37:22 -0600

On Wed, Jan 9, 2013 at 11:25 AM, Julian Reschke <[email-masked]> wrote:
>>> that
> What we tested is whether servers that send an ETag of "X" will also process
> "\X" and treat it as equivalent. I believe the outcome was that they don't.
>
>
>>      Entity-Tag: "abc\"
>>

Well this is very funny, apparently, it *is* a valid quoted string per RFC2616

       quoted-string  = ( <"> *(qdtext | quoted-pair ) <"> )
       qdtext         = <any TEXT except <">>
       quoted-pair    = "\" CHAR

so bare backslashes are allowed, e.g. "\" is a valid quoted string.

That got to be a bug in the RFC, isn't it?

The new syntax for entity-tag also allows bare backslashes. If it was
a bug in the old syntax, the new syntax should probably forbid bare
backslashes.

Zhong Yu

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Jan 2013 21:55:56 +0100

On 2013-01-09 20:37, Zhong Yu wrote:
> ...

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/31>

> a bug in the old syntax, the new syntax should probably forbid bare

I don't recall why we didn't exclude them; probably in order to keep the 
change as minimal as possible.

Best regards, Julian

---

From: Piotr Dobrogost <[email-masked]>
Date: Tue, 15 Jan 2013 12:47:29 +0100

To summarize, from the point of view of http client library (see
https://github.com/kennethreitz/requests/issues/741):

- The safe approach is to not merge any header fields with the same field name.
- If merging, merge only those fields which are known to be safe to
merge ie. those, which can be parsed after merging. Also, if the top
most production in BNF specyfing field's value is #(values) it does
NOT mean the field is safe for merging although this seems to be
implied by the statement in the spec starting with "Multiple header
fields with the same field name MUST NOT be sent (...)"

Regards,
Piotr

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Jan 2013 13:21:44 +0100

> - The safe approach is to not merge any header fields with the same field name.

Yes.

> fields with the same field name MUST NOT be sent (...)"

If a spec uses the list production but then doesn't allow proper parsing 
then that spec is buggy (such as Set-Cookie).

Best regards, Julian

---

From: Zhong Yu <[email-masked]>
Date: Tue, 15 Jan 2013 08:28:49 -0600

> then that spec is buggy (such as Set-Cookie).

It's good to know whether Set-Cookie is the only exception among
well-known headers existed before rfc2616?

(Any headers introduced after rfc2616 should follow the rule; no slack for them)

Zhong Yu

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Jan 2013 15:31:22 +0100

> (Any headers introduced after rfc2616 should follow the rule; no slack for them)

It's the only exception I'm aware of (and, btw, this hasn't changed 
since rfc 2068 or even earlier).

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 15 Jan 2013 14:45:26 +0000

Content-Type: text/plain; charset=ISO-8859-1
--------
>> - The safe approach is to not merge any header fields with the same field name.

I reached the same conclusion in Varnish, if you want headers
merged, you must do so explicity in VCL.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Karl Dubost <[email-masked]>
Date: Wed, 16 Jan 2013 00:23:57 +0900

Le 15 janv. 2013 à 21:21, Julian Reschke a écrit :
>> - If merging, merge only those fields which are known to be safe to merge ie. those, which can be parsed after merging. Also, if the top most production in BNF specyfing field's value is #(values) it does NOT mean the field is safe for merging although this seems to be implied by the statement in the spec starting with "Multiple header fields with the same field name MUST NOT be sent (...)"
> 
> If a spec uses the list production but then doesn't allow proper parsing then that spec is buggy (such as Set-Cookie).

I still have very hard time to understand how the statement "MUST NOT be sent " is working, if it seems good to in fact not enforce it. Should it be dropped?

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Jan 2013 16:30:48 +0100

> I still have very hard time to understand how the statement "MUST NOT be sent " is working, if it seems good to in fact not enforce it. Should it be dropped?

What exactly do you think should be dropped?

Note: current text in editor's copy is:

"A sender MUST NOT generate multiple header fields with the same field 
name in a message unless either the entire field value for that header 
field is defined as a comma-separated list [i.e., #(values)] or the 
header field is a well-known exception (as noted below)." -- 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#rfc.section.3.2.2>

Best regards, Julian

---

From: Piotr Dobrogost <[email-masked]>
Date: Tue, 15 Jan 2013 16:38:29 +0100

On Tue, Jan 15, 2013 at 4:30 PM, Julian Reschke <[email-masked]> wrote:
> "A sender MUST NOT generate multiple header fields with the same field name
> in a message unless either the entire field value for that header field is
> defined as a comma-separated list [i.e., #(values)] or the header field is a
> well-known exception (as noted below)." --

Is the change compared to
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-21#section-3.2
was made due to discussion in this thread?

Piotr

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Jan 2013 16:56:16 +0100

> was made due to discussion in this thread?

AS far as I can tell, it was made due to earlier feedback (not that it 
matters, though).

The important point being that *senders* MUST NOT generate multiple 
header fields with the same name unless the syntax of the header field 
allows it.

Best regards, Julian

---

From: Karl Dubost <[email-masked]>
Date: Wed, 16 Jan 2013 01:23:14 +0900

Le 16 janv. 2013 à 00:30, Julian Reschke a écrit :
> "A sender MUST NOT generate multiple header fields with the same field name in a message unless either the entire field value for that header field is defined as a comma-separated list [i.e., #(values)] or the header field is a well-known exception (as noted below)." -- <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#rfc.section.3.2.2>

but then to my earlier message, Bjoern replied:

Le 10 janv. 2013 à 00:59, Bjoern Hoehrmann a écrit :
> <http://tools.ietf.org/html/rfc5988#section-5>, so the "MUST NOT" above does not apply to the Link header.

and it seems to contradict my understanding.
My initial point was:

  GET /foo/bar HTTP/1.1
  Host: blah.example.org
  […]

If I understand the spec, this above is forbidden. It MUST be

  GET /foo/bar HTTP/1.1
  Host: blah.example.org

The special case I know is "Set-Cookie" which can't be merged. others?

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Jan 2013 17:39:19 +0100

> If I understand the spec, this above is forbidden. It MUST be

Why would that be forbidden?

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 15 Jan 2013 17:41:05 +0100

> merged, you must do so explicity in VCL.

FWIW, haproxy doesn't merge them either. However it considers commas as
delimiters for any headers.

Willy

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 15 Jan 2013 19:33:09 +0100

>and it seems to contradict my understanding.

The structure of the sentence above is "MUST NOT $z UNLESS $a OR $b".
Take it as "MUST NOT <play video games> UNLESS <completed chores> OR
<no school tomorrow>". When <no school tomorrow> then this does not
say anything about <play video games>. Same when <completed chores>.
When you are caught at <play video games> while neither <no school
tomorrow> nor <completed chores> however, then that's non-compliant.
The Link header is a case of <completed chores>.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Jan 2013 09:09:32 +1100

Saying that headers can only be combined under certain circumstances doesn't mean that they're required to be combined.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Nico Williams <[email-masked]>
Date: Tue, 15 Jan 2013 17:22:40 -0600

> Saying that headers can only be combined under certain circumstances doesn't mean that they're required to be combined.

It might help to be able to say that all new headers must be
mergeable.  That is: how can a proxy or what have you, know whether
it's OK to merge a given header's multiple instances?  And I think the
answer is as Poul said: you should never do it.  But then shouldn't we
say so?

Whatever was the point of this feature in the first place?  Was it a
form of header compression?  If so, isn't it best to stop merging
multiple instances of headers and just go with whatever header
compression scheme we settle on?

Nico
--

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Jan 2013 10:28:37 +1100

We're talking about HTTP/1.x here, not 2.0. We can't retroactively make implementations non-conformant. 

Cheers,

---

From: Nico Williams <[email-masked]>
Date: Tue, 15 Jan 2013 17:37:08 -0600

> We're talking about HTTP/1.x here, not 2.0. We can't retroactively make implementations non-conformant.

Ah, yes.  But we could stop encouraging implementors to merge multiple
header instances.  Then we only have to say that it happens and
explain the pitfalls.

---

From: Piotr Dobrogost <[email-masked]>
Date: Wed, 16 Jan 2013 00:38:49 +0100

> We're talking about HTTP/1.x here, not 2.0. We can't retroactively make implementations non-conformant.

I started this thread citing
http://trac.tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-21#section-3.2
so I guess we are talking about 2.0. Also, we talk about existing
behavior so we talk about 1.x as well.

I guess what Nico had in mind is if this subject could be clarified in
2.0 so that we won't have problems we have now in the future.

Piotr

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Jan 2013 10:47:35 +1100

> header instances.  Then we only have to say that it happens and

It's extremely common to do something like:

Cache-Control: max-age=60, must-revalidate

Are you really saying that this should be discouraged?

---

From: Nico Williams <[email-masked]>
Date: Tue, 15 Jan 2013 17:57:55 -0600

> Are you really saying that this should be discouraged?

No.  I'm saying that it's OK for apps to do that but not any other
entities (middleboxes), mostly because middleboxes can't possibly know
about headers that hadn't been registered when they were implemented.

Nico
--

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Jan 2013 11:01:22 +1100

> about headers that hadn't been registered when they were implemented.

OK. Is this an actual problem you've encountered? 

I'm fine with adding some clarifying text if it helps implementers, but I haven't seen this confusing any middlebox vendors; they tend to leave the bits alone...

---

From: Nico Williams <[email-masked]>
Date: Tue, 15 Jan 2013 18:07:48 -0600

> I'm fine with adding some clarifying text if it helps implementers, but I haven't seen this confusing any middlebox vendors; they tend to leave the bits alone...

I noticed Poul's and someone else's replies that in their middlebox
implementations they concluded that it's never safe to merge headers.
If that's the case (and I do think it follows from the facts that it
is the case) then we should say so rather than leave each implementor
to figure this out on their own.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Jan 2013 01:53:47 +0100

> so I guess we are talking about 2.0. Also, we talk about existing

That is 1.1.

> 2.0 so that we won't have problems we have now in the future.

What problem, exactly?

Given a valid message, you can combine header fields as specified in the 
spec. But you don't have to.

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Tue, 15 Jan 2013 19:02:55 -0600

> spec. But you don't have to.

You shouldn't if you're a middlebox.

---

From: Zhong Yu <[email-masked]>
Date: Tue, 15 Jan 2013 19:05:36 -0600

> Whatever was the point of this feature in the first place?  Was it a

Ideally no header should be repeated - it adds another layer of
structure that needs to be explained.

RFC 822 seems to allow, but discourage multiple headers of the same name

4.1.
            This specification permits multiple  occurrences  of  most
            fields.   Except  as  noted,  their  interpretation is not
            specified here, and their use is discouraged.

RFC 2616 probably wanted to clarify the meaning of multiple
occurrences, instead of leaving them undefined.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 16 Jan 2013 02:10:14 +0100

> You shouldn't if you're a middlebox.

Please explain why.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 15 Jan 2013 17:12:51 -0800

> to figure this out on their own.

The requirement is on generating headers, so it does not concern
middlebox vendors other than for the fields that they add to a message.

Apache httpd merges header fields as they are read from the network.
So does any correctly written client that reads internet message format
(like all of the common MIME and libwww libraries).  If you want to
interoperate on the Web, field generators must obey that requirement.

....Roy

---

From: Zhong Yu <[email-masked]>
Date: Tue, 15 Jan 2013 19:29:25 -0600

> You shouldn't if you're a middlebox.

On the server side, it is a good idea to merge headers. There's no
point for a server framework to return multiple values for one header
name - application will have to merge them anyway before parsing.
(Java Servlet for example does not merge headers; it's a source of
bug; fortunately clients usually don't split headers)

On the client side, it's probably the same story (except Set-Cookie)

Zhong Yu

---

From: Karl Dubost <[email-masked]>
Date: Wed, 16 Jan 2013 15:30:02 +0900

> The structure of the sentence above is "MUST NOT $z UNLESS $a OR $b". 

Ah gotcha! :) The double negation got me. 
Thanks a lot Bjoern. Really.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jan 2013 07:52:54 +0100

> interoperate on the Web, field generators must obey that requirement.

Roy, you're totally right here. If Apache merges multiple instances of
a header, it is because the client did so, and as per the spec, it was
allowed to do so because the headers were mergeable.

However I've seen some corner cases caused by applications (not middleboxes)
sometimes adding a header that was already present and not mergeable (eg:
the Date or Location header). I know that Apache has a special handling of
the Set-Cookie header, I don't know if it does so for other well-known
non-mergeable headers which are supposed to contain commas.  

Due to this mess caused by bogus applications, I finally decided that I
won't merge headers in haproxy, I didn't want to start dealing with a list
of exceptions. And I think it's the same reasoning for Poul Henning.

That said, I disagree with having the spec declare Apache non-compliant,
as it is doing the correct job in my opinion, as long as clients are
compliant.

But probably we could discourage future middlebox implementors from merging,
by making them aware of the difficulty of doing it right. This would just be
a non-normative warning then.

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 16 Jan 2013 22:27:37 +1300

> On the client side, it's probably the same story (except Set-Cookie)

In the middle it is simply difficult to merge correctly without adding 
further mistakes. However for those willing to go to the effort to 
optimize the headers merging can be very useful tool for reducing bytes. 
A warning that many senders still generate corrupted headers may be 
useful but I see no reason to make it a "should not" as these problems 
will face a compliant client as much as any middle box.

Amos

---

From: Zhong Yu <[email-masked]>
Date: Wed, 16 Jan 2013 12:32:22 -0600

> the Date or Location header). I know that Apache has a special handling of

If different applications interpret duplicate headers differently, we
have a serious problem

    Location: http://abc.com
    Location: http://xyz.com

Some chooses the 1st one, some chooses the 2nd one, and some may see
the merged `http://abc.com,http://xyz.com` (a valid URI)

We probably should require that an application either reject the
message, or merge the header first if it wants to parse the header.

Zhong Yu

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Jan 2013 20:41:51 +0100

Hi,

> the merged `http://abc.com,http://xyz.com` (a valid URI)

Obviously and the issue is the same with many headers. I was just explaining
that this is an example of a *non-compliant* input which is then turned into
a different non-compliant output by a middlebox. The problem clearly is with
the input and not with the operation performed by the middlebox. So we could
recommend middlebox authors against merging the headers because it's hard to
do it right, but we have no reason to add a SHOULD NOT which would turn some
existing implementations to non-compliant for no reason.

Regards,
Willy

---

From: Nico Williams <[email-masked]>
Date: Wed, 16 Jan 2013 13:54:37 -0600

> existing implementations to non-compliant for no reason.

SHOULD NOT does not do that.  MUST NOT would.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jan 2013 13:51:20 +1100

> existing implementations to non-compliant for no reason.

I think the middlebox aspect of this is a red herring. What's happening above is that a non-conformant message is being turned into, potentially, conformant input (if there's no space around the comma) -- regardless of where it takes place. I.e., that combination into a single value could just as easily be done in the UA, before the Location is followed.

It seems to me that that's worth at least a note in Security Considerations. I do agree that a requirement -- especially targeted at a particular kind of recipient, rather than *all* recipients -- is probably too much. This is more on the scale of implementation advice. YMMV.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Jan 2013 13:53:39 +1100

Now <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/429>.

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 20 Jan 2013 08:15:17 +0100

Hi Mark,

> Now <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/429>.

Quite frankly, I'd prefer to stay on Roy's side which consists in saying
that when a compliant message is passed to an intermediary, the output is
a compliant message, and when a non-compliant message is passed, the
output is indetermined.

Otherwise we'll have to document all possible corner cases, which will
result in even worse implementations givent that we won't be exhaustive.

Probably that all the trouble comes from the obligations made to senders,
with senders sometimes being intermediaries. I've been bothered by this
in the past. So the point above at least would solve the issue for them :
they have to emit clean things but if they forward stupid things, well,
it's the other side's fault.

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Jan 2013 11:17:14 +1100

> it's the other side's fault.

I know, and agree with the spirit of what you're saying. 

Digging around, it's gratifying to see that we already cover this somewhat in [1]:

> Whether the field is a single value, or whether it can be a list (delimited by commas; see Section 3.2 of [Part1]).
> 
> If it does not use the list syntax, document how to treat messages where the field occurs multiple times (a sensible default would be to ignore the field, but this might not always be the right choice).
> 
> Note that intermediaries and software libraries might combine multiple header field instances into a single one, despite the field's definition not allowing the list syntax. A robust format enables recipients to discover these situations (good example: "Content-Type", as the comma can only appear inside quoted strings; bad example: "Location", as a comma can occur inside a URI).

I think the remaining questions are:

 - Is it necessary to say anything regarding Location? The security aspect here is pretty limited, AFAICT; while it can certainly cause interop problems (the easy answer to which is "don't do that"), I don't see how it's useful to say anything about the security risks, because if an attacker can insert a new Location header in your response, they can do pretty much anything else they want too... And, unlike Content-Length (where we *have* said something), it doesn't affect framing.

 - Are there any other headers that we define where something should be said? I think we've already reviewed and said no. Anyone?

So, absent any further discussion, I'll close the issue.

Cheers,

1. https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#considerations.for.new.header.fields

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 21 Jan 2013 01:42:53 +0100

Hi Mark,

On Mon, Jan 21, 2013 at 11:17:14AM +1100, Mark Nottingham wrote:
> > If it does not use the list syntax, document how to treat messages where
> > the field occurs multiple times (a sensible default would be to ignore the
> > field, but this might not always be the right choice).
> > 
> > Note that intermediaries and software libraries might combine multiple
> > header field instances into a single one, despite the field's definition
> > not allowing the list syntax. A robust format enables recipients to
> > discover these situations (good example: "Content-Type", as the comma can
> > only appear inside quoted strings; bad example: "Location", as a comma can
> > occur inside a URI).

Oh Great.

>  - Is it necessary to say anything regarding Location? The security aspect
>  here is pretty limited, AFAICT; while it can certainly cause interop
>  problems (the easy answer to which is "don't do that"), I don't see how it's
>  useful to say anything about the security risks, because if an attacker can
>  insert a new Location header in your response, they can do pretty much
>  anything else they want too... And, unlike Content-Length (where we *have*
>  said something), it doesn't affect framing.

Location has no special status here in my opinion. I provided it as an
example but it's harmless (at least as harmless as many other fields).

>  - Are there any other headers that we define where something should be said?
>  I think we've already reviewed and said no. Anyone?

In my opinion, as long as framing is not affected, I don't think so.
And the other fields which affect framing already support commas (eg:
Connection, Transfer-Encoding). I suspect that Content-Length was the
only one to have such a nasty effect.

One might consider that Max-Forwards could affect how far the message
goes, but this is seldom used and more for tracing than anything else.

> So, absent any further discussion, I'll close the issue.

I think that's OK.

Cheers,
Willy