Subject: draft-ietf-httpbis-header-structure: handling multiple field values

From: Julian Reschke <[email-masked]>
Date: Tue, 12 May 2020 18:53:40 +0200

Hi there,

while working on an implementation I encountered the following question.

Consider something defined as "sh-list". If a value is received spread
over multiple field instances, recipients *can* recombine the value
before processing. So for

   Foo: "1
   Foo: 2"

...the parser would see the string "1,2" (or maybe "1, 2").

What's not totally clear to me is whether recipients are *allowed* to
process the field values separately, in which case parsing would fail.

If this is allowed, I would expect tests in
<https://github.com/httpwg/structured-header-tests> where "can_fail" is
set to true, but I couldn't find any.

FWIW; one reason for processing them separately would be to catch syntax
errors; another one performance, as it would avoid concatenating things
that can be parsed separately.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 12 May 2020 19:23:47 +0200

Hi Julian,

> process the field values separately, in which case parsing would fail.

Note that they might not be aware that these were two values
because the folding might have been performed by an upstream
gateway.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 May 2020 19:38:13 +0200

On 12.05.2020 19:23, Willy Tarreau wrote:
> ...

Yes, understood. (and that's actually mentioned in the spec)

Best regards, Julian

---

From: Ian Clelland <[email-masked]>
Date: Tue, 12 May 2020 13:39:18 -0400

This is mentioned in
https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#rfc.section.4.2
--
"parsers MUST combine all lines in the same section (header or trailer)
that case-insensitively match the field name into one comma-separated
field-value", (with the warning given that strings split across multiple
field values will have "unpredictable results") -- So I don't think you're
allowed to parse them separately. If both exist in the same message, they
must be combined before parsing.

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 12 May 2020 19:46:29 +0200

> https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#rfc.section.4.2

Thanks, so the behavior is indeed well defined and easy to enforce.

Willy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 May 2020 19:47:45 +0200

On 12.05.2020 19:39, Ian Clelland wrote:
> This is mentioned in
> https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#rfc.section.4.2 --
> field values will have "unpredictable results") -- So I don't think
> you're allowed to parse them separately. If both exist in the same
> message, they must be combined before parsing.
> ...

Indeed. Looking at this again, I realize that a paragraph below then
confused me:

"Strings split across multiple field lines will have unpredictable
results, because comma(s) and whitespace inserted upon combination will
become part of the string output by the parser. Since concatenation
might be done by an upstream intermediary, the results are not under the
control of the serializer or the parser."

I read this to mean that errors might be detected early or not, but
maybe this is just a warning that the actual string used for
concatenation can vary?

If that's the intent, I'd call that a spec bug. A string value split
across multiple field instances is very clearly a violation of what HTTP
says about list-shaped header fields, and not allowing a recipient to
detect that seems incorrect to me.

Best regards, Julian

---

From: Ian Clelland <[email-masked]>
Date: Tue, 12 May 2020 14:45:02 -0400

> https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#rfc.section.4.2

Definitely a spec bug -- not sure which spec though.
7230 reads:

> A sender MUST NOT generate multiple header fields with the same field name
> in a message unless either the entire field value for that header field is
> defined as a comma-separated list [i.e., #(values)] or the header field is
> a well-known exception (as noted below).

Perhaps what it should also mention is that the header must be defined as a
comma-separated list, *and* the split must be between list elements, in
cases where the field value can contain commas with other semantic meanings.

It goes on to say:

> A recipient MAY combine multiple header fields with the same field name
> into one "field-name: field-value" pair, without changing the semantics of
> the message, by appending each subsequent field value to the combined field
> value in order, separated by a comma.

and maybe the phrase "without changing the semantics of the message" means
that the server is only free to join the fields if it doesn't change the
semantics (implying indirectly that the field shouldn't have been split up
within a quoted string in the first place), but it doesn't really read that
way.

Ian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 May 2020 21:25:38 +0200

On 12.05.2020 20:45, Ian Clelland wrote:
>
>
> On Tue, May 12, 2020 at 1:47 PM Julian Reschke <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     A sender MUST NOT generate multiple header fields with the same
>     field name in a message unless either the entire field value for
>     that header field is defined as a comma-separated list [i.e.,
>     #(values)] or the header field is a well-known exception (as noted
>     below).
>
>
> Perhaps what it should also mention is that the header must be defined
> as a comma-separated list, *and* the split must be between list
> elements, in cases where the field value can contain commas with other
> semantic meanings.

AFAIU, that was the intent in RFC 2616 and 7230: every single field
value must conform to the header field's grammar.

> It goes on to say:
>
>     A recipient MAY combine multiple header fields with the same
>     field name into one "field-name: field-value" pair, without changing
>     the semantics of the message, by appending each subsequent field
>     value to the combined field value in order, separated by a comma.
>
>
> and maybe the phrase "without changing the semantics of the message"
> means that the server is only free to join the fields if it doesn't
> change the semantics (implying indirectly that the field shouldn't have
> been split up within a quoted string in the first place), but it doesn't
> really read that way.

No, whoever joins the header fields does not need to know the syntax of
the field (because that would defeat extensibility). IOW, if the input
is garbage, so will be the output.

Going back to the SH spec: I'm afraid that the spec *disallows* to fail
early on garbage - is this *really* the intent?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 12 May 2020 19:34:56 +0000

--------

>early on garbage - is this *really* the intent?

Ehhh... what ?

	1.1.  Intentionally Strict Processing

	   This specification intentionally defines strict parsing and
	   serialisation behaviours using step-by-step algorithms; the only
	   error handling defined is to fail the operation altogether.

Since the only failure is total failure, why or even how would you go
about postponing it ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 May 2020 22:17:39 +0200

> Since the only failure is total failure, why or even how would you go

The issue here is that combining

   Foo: "a

and

   Foo: b"

into

   Foo: "a, b"

*hides* an issue, and no failure will occur. Furthermore, we not only
get no failure, but the resulting string may vary depending on how the
fields were combined.

I get that it's unavoidable that this *can* happen, but does this mean
we should *disallow* failing for that input?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 12 May 2020 21:22:31 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>we should *disallow* failing for that input?

Yes, we should disallow it, because that would cause a valid request
to fail unless a proxy implemented a particular way is in the path.

I think we say this very clearly on page 23:

   When generating input_bytes, parsers MUST combine all lines in the
   same section (header or trailer) that case-insensitively match the
   field name into one comma-separated field-value, as per [RFC7230],
   Section 3.2.2; this assures that the entire field value is processed
   correctly.

Poul-Henning

PS: I think your example is wrong, the combined header would be:

    Foo: "a,b"

Because there is an important asymmetry in RFC7230/3.2.2.

A sender is required to know the syntax of the header is a comma
separated list, before can split the header.

A receiver on the other hand is under no such obligation, and therefore

   "…by appending each subsequent field value to
   the combined field value in order, separated by a comma."

should be interpreted narrowly:  Only a comma, not whitespace.

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 12 May 2020 23:35:19 +0200

On Tue, May 12, 2020 at 09:22:31PM +0000, Poul-Henning Kamp wrote:
>    "...by appending each subsequent field value to

Yes it is valid because whitespaces before and after the value are not
part of the value itself, so:

   Foo: b
   Foo:    b

are equivalent, as such you can simply fold using a comma and whatever
number of spaces. I remember a very old discussion where someone suggested
that some limited equipements that must not change the size could replace
the CR with the comma and the LF plus all subsequent chars of the header
field name and the colon with spaces and have it done.

In haproxy we used to replace the CR by comma, the LF with a space,
and memmove the value. Now we're simply folding using <comma> <space>.

Cheers,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 May 2020 05:18:19 +0200

> to fail unless a proxy implemented a particular way is in the path.

How can the request be valid, if the resulting string is not fully
predictable?
(https://greenbytes.de/tech/webdav/draft-ietf-httpbis-header-structure-18.html#rfc.section.4.2.p.6)

> should be interpreted narrowly:  Only a comma, not whitespace.

(see Willy's reply) FWIW, the latest spec says:

"A recipient MAY combine multiple field lines with the same field name
into one field line, without changing the semantics of the message, by
appending each subsequent field line value to the initial field line
value in order, separated by a comma and optional whitespace. For
consistency, use comma SP."
<https://greenbytes.de/tech/webdav/draft-ietf-httpbis-semantics-latest.html#rfc.section.4.1.p.2>

We go on with

"This means that, aside from the well-known exception noted below, a
sender MUST NOT generate multiple field lines with the same name in a
message (whether in the headers or trailers), or append a field line
when a field line of the same name already exists in the message, unless
that field's definition allows multiple field line values to be
recombined as a comma-separated list [i.e., at least one alternative of
the field's definition allows a comma-separated list, such as an ABNF
rule of #(values) defined in Section 4.5]."

I wonder whether we should point out that each *individual* field line
needs to conform to the syntax defined for the field value, so that
splitting strings over lines would be disallowed.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 13 May 2020 05:33:45 +0200

> splitting strings over lines would be disallowed.

I think that would be a nice addition. It would implicitly allow a
recipient to reject a message when it knows some fields are not
properly formatted, before recombining them. Your example with a
single double-quote could serve as an encouragement for some
intermediaries which want to be strict to systematically reject
such individual values early.

Willy