Subject: Re: [#95] Multiple Content-Lengths

From: Mark Nottingham <[email-masked]>
Date: Wed, 9 Mar 2011 10:04:20 -0800

I've scheduled this for -13.

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/95#comment:20>

On 20/02/2011, at 11:12 PM, Mark Nottingham wrote:

> So, I propose:
> 
> * adding text that allows duplicates explicitly, and
> 
> * upgrading the SHOULD to a MUST in this requirement:
> 
>   If this is a response message received by a user-agent, it SHOULD be treated
>   as an error by discarding the message and closing the connection.

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Mar 2011 23:21:46 +0100

>>    as an error by discarding the message and closing the connection.

...clarifying: you say "adding text that allows duplicates 
explicitly"... that could be read to REQUIRE recipients to accept those 
duplicates -- are we really going to declare recipients that do not do 
that to be non-compliant?

If we do, we *probably* need to adjust the header field ABNF (because 
"x, x" doesn't parse), which I'd rather do not...

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 9 Mar 2011 14:32:35 -0800

On Mar 9, 2011, at 2:21 PM, Julian Reschke wrote:

> ...clarifying: you say "adding text that allows duplicates explicitly"... that could be read to REQUIRE recipients to accept those duplicates -- are we really going to declare recipients that do not do that to be non-compliant?

We need to require that they process received duplicates in the
same way as all other recipients in order to avoid response
smuggling.

> If we do, we *probably* need to adjust the header field ABNF (because "x, x" doesn't parse), which I'd rather do not...

No, we still require that duplicates not be sent.  The ABNF
only defines valid messages.  This new requirement is for
exception handling in the case of an invalid received message.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Mar 2011 23:44:20 +0100

> same way as all other recipients in order to avoid response

I can think of three ways for recipients to handle these:

a) fail to parse C-L, and treat the message as invalid (closing the 
connection because of broken framing)

b) accept the duplicate value, and use the C-L as if it wasn't repeated

c) fail to parse C-L, and just treat the C-L header field as invalid, 
but continue processing by reading until the end of connection

Smuggling could only happen if some recipients did c), right? Those that 
do this IMHO are already non-compliant, so I'm not sure how mandating b) 
helps...

> exception handling in the case of an invalid received message.

Ack.

Best regards, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 10 Mar 2011 11:55:31 +1300

On 10/03/2011 11:44 a.m., Julian Reschke wrote:
> Smuggling could only happen if some recipients did c), right? Those 
> that do this IMHO are already non-compliant, so I'm not sure how 
> mandating b) helps...
>

What should a proxy do?  It has the task of putting something together 
to send a client.

it seems to me the only safe option is a.  It's also the only option 
that provides any incentive for people to fix their sites.

Regards

Adrien

>>> If we do, we *probably* need to adjust the header field ABNF 
>>> (because "x, x" doesn't parse), which I'd rather do not...
>>

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 9 Mar 2011 23:58:29 +0100

> that to be non-compliant?

That would be a big mess, because the real issue comes from the
non-compliant servers which emit them twice. I'd rather use a MAY
here.

> "x, x" doesn't parse), which I'd rather do not...

That makes me realize that content-length is not the only header in this
case. It's probably one of the most dangerous ones when the differing
content-lengths are observed though. But I've already seen buggy apps or
app servers returning a bunch of duplicate headers. So maybe instead of
writing something explicit for content-length, we could suggest something
along those lines (I don't like that much but I'm trying to help find a
pragmatic solution) :

  Headers which are not defined as coma-separated values and which appear
  multiple times MAY be folded into a single one by the recipient only if
  they all have the exact same value. Similarly, a header which does not
  support comas and which is received with multiple exactly identical
  values delimited by comas MAY be folded into one such single value.
  All non-matching situations MUST be rejected.

  Examples :

      Content-length: 100
      Content-length: 100, 100

    1) reduce individual header lines to single values :

      Content-length: 100
      Content-length: 100

    2) reduce repetitive occurrences to a single line :

      Content-length: 100

  Non-working example :

      Content-length: 100
      Content-length: 100, 102

    per-line reduction not possible on the second one, and both header lines
    don't match.

I've seen those duplicates often with Content-type for instance.

Note that every time I observed a duplicate, there were really two header
lines, so probably that most components capable of folding them sometimes
also take care of cleaning the mess.

Thus maybe we could just amend the folding paragraph with a duplicate
removal suggestion and still not allow comas in Content-Length. That
should cover most situations I have observed without opening a dangerous
can of worms.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 10 Mar 2011 00:03:40 +0100

> do this IMHO are already non-compliant, so I'm not sure how mandating b) 

I think that doing only c) could have impacts on pipelined requests,
because if a client sends 3 requests, and receives a duplicate C-L in
the first response, then it will read all three responses as the first
one because it ignores C-L and waits for the close. Also we don't know
how other intermediaries in the chain will proceed. So basically treating
the C-L as invalid on *some* components seems more dangerous to me than
trying to keep in sync with the stream. I'd say you've listed the solutions
in the order of the least risky to the most risky ones, with the first one
causing issues in some environments.

Regards,
Willy

---

From: "William A. Rowe Jr." <[email-masked]>
Date: Wed, 09 Mar 2011 17:06:08 -0600

On 3/9/2011 4:55 PM, Adrien de Croy wrote:
>> a) fail to parse C-L, and treat the message as invalid (closing the connection because
>> of broken framing)
>>
>> c) fail to parse C-L, and just treat the C-L header field as invalid, but continue
>> processing by reading until the end of connection
>>
>> Smuggling could only happen if some recipients did c), right? Those that do this IMHO
>> are already non-compliant, so I'm not sure how mandating b) helps...
>>
> 
> What should a proxy do?  It has the task of putting something together to send a client.
> 
> it seems to me the only safe option is a.  It's also the only option that provides any
> incentive for people to fix their sites.

Well, d) was omitted, fail with a 400, having read the rest of the headers off the
wire, but without draining the [misrepresented] body.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 10 Mar 2011 00:10:03 +0100

> that provides any incentive for people to fix their sites.

The problem is that it does not work. When you go deploying your
appliance at a large customer and some applications don't work
behind it, whatever the bugs on the applications, the customer
only says your appliance does not work while many other products
do work there. That's why we got so many diverging workarounds
for many issues. Everything that is declared illegal will still
be done in a random way :-/

I even have one example. I had haproxy deployed at a bank and which
was blocking a few invalid responses which contained a header named
"Content/type". Guess what ? They disabled HTTP processing and only
used TCP until I was able to provide an option with a scary name to
relax the header parser, because fixing the app was expected to take
6 months (and it did take more).

I'd really like that standards propose hints for how to handle
exceptions when there's no alternative, without creating vulnerabilities
and without having all products behave differently.

Content smuggling exists precisely because everyone had to find a
different solution to an issue that was not considered something
that had to be handled at one point.

Regards,
Willy

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 10 Mar 2011 13:40:28 +1300

> be done in a random way :-/

There's another way to look at it.  If all software bounced the invalid 
response, the problems would soon be fixed.

So, there are some vendors that would choose to bounce, and some would not.

Those that do not may be opening themselves up to security 
vulnerabilities or other problems.

As a vendor of a product that bounces it, you could then argue that your 
software was more secure.  In my experience, bank IT managers are among 
the more security-conscious of customers.

If all the C-L values are the same, it's perhaps not so dangerous to 
strip duplicates, but otherwise choosing the smallest value may result 
in truncation of the entity, and choosing the biggest may result in 
response smuggling.  There's no way of knowing at the time you need to 
formulate a response for a waiting client.  If it's an attempted 
smuggle, it could possibly be detected by looking for something like a 
response at the offset specified by the smallest C-L value.  But if a 
proxy has already sent the larger C-L value on to the client it's 
hosed.  Likewise if the proxy sends the smaller C-L value on to the 
client and the resource keeps coming.

So maybe we need to take a tougher stance.

In the end, we're talking about the spec here.  People sending multiple 
C-Ls are already ignoring the spec.  People who want to not bounce 
multiple C-Ls could likewise ignore the spec.  The spec however should 
be clear on what should happen - what we want as a community the 
direction to be. Otherwise it seems like a slippery slope.

Regards

Adrien

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 10 Mar 2011 02:17:50 +0100

> response, the problems would soon be fixed.

In theory yet, but in practice it's not what we observe. Incompetent
developers write whatever code that enters production and nowadays with
N-tier applications, they write poor code on both sides, and call their
implementation "HTTP" because one of the function they use at one point
surely has in its name.

When any compliant device enters the arena, things break. More or less
seriously, and more or less quickly. I spend my time arguing against
those people that respecting standards is extremely important because
the cost of staying away from them increases as time passes. Still,
it's not easy because they just read what they want to understand. So
in the end they believe strongly they're right and the rest of the
world is wrong. Since they take a lot of time to discover the rest of
the world, once in a while they fix a bug or two in their apps until
they start to work even with similar products. It's just a matter of
randomness in those environments, and that's why flexibility is very
important there.

> vulnerabilities or other problems.

Yes, indeed. Reason why I'm saying that when fallbacks are known to be
possible without taking risks, we should indicate them, it avoids the
worst things to be done.

> software was more secure.

Oh they know that, but when their apps are broken, being more secure
is the least of their concerns.

> In my experience, bank IT managers are among the more security-conscious
> of customers.

That's true on the design side, not on the operations side. The cost of
breakage is so high that any solution will fit. The first example is the
common "I want to be able to replace this firewall with a cable in no
delay in case of trouble with it".

> response smuggling.

Exactly ! That's why I was advocating for accepting them only when they're
exact duplicates. All other cases are very dangerous, and only the duplicate
can happen by accident and/or poor implementations somewhere.

> There's no way of knowing at the time you need to 

I agree we should not try to guess anything. When I was confronted to
this question for haproxy, it was a tough one. I first chose not to
accept duplicates, but I already had encountered some of them a few
times. Finally, I realized that duplicates were acceptable due to
their nature, and I reject anything else. I've never ever had any
issue on production with that solution.

> direction to be. Otherwise it seems like a slippery slope.

As I said in another mail, it's not only C-L. There are some components
or frameworks where developers have little control over headers received
and/or sent. Sometimes they blindly "set" a header, hoping it was not
there, assuming the risk of duplicate, and you get two of them on the
wire once in a while. That's pretty disgusting but that's what many
applications we're using on a daily basis are doing, unfortunately. If
there weren't that amount of protocol normalization performed by reverse
proxies at boundaries, we would see even much more ugly things.

In my experience, duplicates do happen. People emitting them have no real
excuse for that, they should read specs and manuals before coding and file
bug reports if needed, but those things do exist. However I've not seen
conflicting values of a same header (whether it was content-length or
content-type).

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 27 Mar 2011 16:13:57 +0200

Roy, Julian and I discussed this, and agreed that (a) and (b) are both valid. So, we propose this patch which will show up in -14:

  http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1214

... which should let us close #95. Any thoughts / objections?

Cheers,

On 09/03/2011, at 11:44 PM, Julian Reschke wrote:

> a) fail to parse C-L, and treat the message as invalid (closing the connection because of broken framing)
> 
> c) fail to parse C-L, and just treat the C-L header field as invalid, but continue processing by reading until the end of connection
> 
> Smuggling could only happen if some recipients did c), right? Those that do this IMHO are already non-compliant, so I'm not sure how mandating b) helps...
>

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 27 Mar 2011 16:35:22 +0200

> ... which should let us close #95. Any thoughts / objections?

Seems fine to me. It indicates that this annoying case exists, suggests
how to openly deal with it without forcing anyone to support it.

Willy