Subject: what constitutes an "invalid" content-length

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 12 Jul 2016 13:31:32 +0000

Hi all

just dealing with a site that sends more payload data than is indicated 
in the Content-Length header.

If the browser connects directly, the page loads fine, if via the proxy, 
the proxy is truncating the length to that advertised and the client 
isn't displaying a page (of course this is the .css file).

RFC7230 sections 3.3.2 (Content-Length), 3.3.3 (Message body length), 
and 3.3.4 (Handling incomplete messages) only contemplate issues around 
Content-Length specifying more bytes than are received, not fewer.

I guess one could argue that a wrong C-L value is "invalid", but it's 
not clear that invalid in this context simply means it doesn't parse, or 
is otherwise non-compliant with the ABNF.

So, it's not clear what the browser and/or proxy response should be.  If 
we deem a wrong value to be "invalid" (s3.3.3 para 4), a client is 
supposed to discard the response.  This isn't happening.

For the proxy, it only sees that the content length is wrong once it 
receives too many bytes.  By this stage, the horse has bolted so it 
cannot really comply either.

I would expect it's in everyone's best interest if sites that have 
broken framing are forced to be fixed.  This won't happen if browsers 
"just work" for the site.

Is there a special behaviour we should agree on for such cases?

Regards

Adrien de Croy

---

From: Patrick McManus <[email-masked]>
Date: Tue, 12 Jul 2016 10:19:32 -0400

H1 Framing is badly broken in practice.. we tried to tighten up what
Firefox would accept about a year ago and it was basically a disaster that
was reverted. Some of the story is in the 300+ comments in
https://bugzilla.mozilla.org/show_bug.cgi?id=237623

You can set network.http.enforce-framing.http1 to true if you'd like a more
error filled web and lean on content-providers. That didn't get us very far
because we're talking about the tail of the Internet - basically one off
gateways to old legacy machinery like pdf generators, and broken dynamic
gzip modules and stuff that insists 304 should have a body. While that
doesn't dominate the Internet it does cause persistent breakage that a high
number of people will see (and be frustrated by) over a fairly short period
of time. Its extremely well baked in at this point and its very hard to fix
- we're much better off enforcing strict standards around h2 and
encouraging a migration than beating our heads against the h1 legacy imo.

I know there is some talk of trying to remove support for 0.9 in another
browser. I sincerely wish that team nothing but happiness, good karma, and
good luck in their trail blazing - but I wonder if its going to go a
similar route.

-Patrick

On Tue, Jul 12, 2016 at 9:31 AM, Adrien de Croy <[email-masked]> wrote:

> Hi all
>
> just dealing with a site that sends more payload data than is indicated in
> the Content-Length header.
>
> the proxy is truncating the length to that advertised and the client isn't
> displaying a page (of course this is the .css file).
>
> I guess one could argue that a wrong C-L value is "invalid", but it's not
> clear that invalid in this context simply means it doesn't parse, or is
> otherwise non-compliant with the ABNF.
>
> we deem a wrong value to be "invalid" (s3.3.3 para 4), a client is supposed
> to discard the response.  This isn't happening.
>
> receives too many bytes.  By this stage, the horse has bolted so it cannot
> really comply either.
>
> I would expect it's in everyone's best interest if sites that have broken
> framing are forced to be fixed.  This won't happen if browsers "just work"
> for the site.
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 12 Jul 2016 15:24:05 +0000

In this case, it's a fairly popular site in Norway (mytribe.no) running 
through a relatively recent nginx (1.8.1 Jan 2016).

I would have expected that the static css file would come from the 
reverse proxy cache and so should have had C-L validated/fixed, maybe it 
didn't and the bad C-L is coming from the back end.

But in any case, this isn't really an example of an old or legacy site.

I guess it's always a problem when any single browser makes a stand.  
What we need is for all the browsers to do the same thing.  Then the 
users can't say "but it works in xxx browser", and site operators have 
no choice but to fix it.

Maybe there's some security angle that would prompt a harder look at 
this.

Regards

Adrien

------ Original Message ------
From: "Patrick McManus" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 13/07/2016 2:19:32 a.m.
Subject: Re: what constitutes an "invalid" content-length

>Firefox would accept about a year ago and it was basically a disaster 
>that was reverted. Some of the story is in the 300+ comments in 
>You can set network.http.enforce-framing.http1 to true if you'd like a 
>more error filled web and lean on content-providers. That didn't get us 
>very far because we're talking about the tail of the Internet - 
>basically one off gateways to old legacy machinery like pdf generators, 
>and broken dynamic gzip modules and stuff that insists 304 should have 
>a body. While that doesn't dominate the Internet it does cause 
>persistent breakage that a high number of people will see (and be 
>frustrated by) over a fairly short period of time. Its extremely well 
>baked in at this point and its very hard to fix - we're much better off 
>enforcing strict standards around h2 and encouraging a migration than 
>beating our heads against the h1 legacy imo.
>
>I know there is some talk of trying to remove support for 0.9 in 
>another browser. I sincerely wish that team nothing but happiness, good 
>karma, and good luck in their trail blazing - but I wonder if its going 
>to go a similar route.
>
>-Patrick
>
>>just dealing with a site that sends more payload data than is 
>>indicated in the Content-Length header.
>>
>>If the browser connects directly, the page loads fine, if via the 
>>proxy, the proxy is truncating the length to that advertised and the 
>>client isn't displaying a page (of course this is the .css file).
>>
>>and 3.3.4 (Handling incomplete messages) only contemplate issues 
>>around Content-Length specifying more bytes than are received, not 
>>fewer.
>>
>>not clear that invalid in this context simply means it doesn't parse, 
>>or is otherwise non-compliant with the ABNF.
>>
>>So, it's not clear what the browser and/or proxy response should be.  
>>If we deem a wrong value to be "invalid" (s3.3.3 para 4), a client is

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 12 Jul 2016 10:20:29 -0600

>From the standards point of view, that is _not_ what you are dealing
with. You are dealing with a site that sends two responses, the first
response is proper HTTP. The second response is garbage.

>From the standards point of view, it is impossible for the
Content-Length to specify fewer bytes than the message has. Irrelevant
for this discussion cases aside, the message end is defined by the
Content-Length header value. One cannot have more than what was promised
because one stops assembling the message [body] after the promised
number of bytes were added. Any "leftovers" are another message or
garbage, depending on Connection:close, pipelining, and similar factors.

> is otherwise non-compliant with the ABNF.

It is valid from protocol point of view. You know it is "wrong" only
because you can (or you think you can) distinguish garbage from the end
of the content.

> So, it's not clear what the browser and/or proxy response should be.

There is no single right answer to that. A compliant client (including
proxies) ought to treat leftovers as post-message gardbage or another
message. A real-world client may identify specific cases where leftovers
are likely to be the end of the message content and ignore
Content-Length in those cases. The cases where such behavior would be a
good idea would vary from agent to agent, from one deployment to another.

> "just work" for the site.

The ever-popular "force sites to be fixed" approach rarely fixes enough
real-word sites to remove special treatment code. See Patrick's response
for a good illustration.

> Is there a special behaviour we should agree on for such cases?

We could agree to violate the standard in one or two special cases, but
any formal agreement would probably result in a few more broken sites
because more folks will tolerate them, decreasing the probability that
they will be fixed.

I can think of one special case where it is more-or-less safe to ignore
response Content-Length:

* the HTTP/1 connection is not persistent,
* no additional outstanding pipelined requests on that connection,
* the unique Content-Length header field is syntactically valid, and
* more bytes were read during the last network read than C-L promises.

The combination of these conditions can trigger [optional] "robustness"
code that reads until connection closure and re-sends leftovers/garbage
to the next hop (or displays it to the user), opening a message
smuggling attack vector.

Needless to say, there are benign leftover cases that the above
conditions do not cover.

Cheers,

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 12 Jul 2016 23:40:03 +0200

Hi Adrien,

On Tue, Jul 12, 2016 at 03:24:05PM +0000, Adrien de Croy wrote:
> I would have expected that the static css file would come from the reverse
> proxy cache and so should have had C-L validated/fixed, maybe it didn't and
> the bad C-L is coming from the back end.

If it was bad in the backend, normally the front reverse proxies would not
forward the extra data. So i guess instead that the frontend is broken. Also
you're probably seeing nginx only because it's the one advertising its name
but you don't necessarily see what blackbox runs in front of it. If it's an
ad injector or whatever similar device, you can reasonably expect it to be
broken by default :-/

> I guess it's always a problem when any single browser makes a stand.  What
> we need is for all the browsers to do the same thing.  Then the users can't
> say "but it works in xxx browser", and site operators have no choice but to
> fix it.

You forget that people first consider that their browser got broken by an
upgrade and revert to the old version. And I'm the first one to do this,
I have a terrible experience of constantly changing interfaces forcing me
to stay as long as possible on a given version. Similarly if users see that
their browser reports broken objects on their favorite web site after an
upgrade, they'll simply revert to the previously working version. And that
can be much worse because they suddenly stop to apply any security fixes
for as long as they can...

Willy

---

From: Tim Bray <[email-masked]>
Date: Tue, 12 Jul 2016 14:53:48 -0700

I've written two large-scale web crawlers, processing billions of links,
and since my principle was to err on the side of inclusiveness, I totally
ignored Content-length and wrote the necessary code to deal with whatever I
got, including the occasional GET returning an infinite stream of
smiley-emoji or null bytes or whatever.

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 12 Jul 2016 22:38:47 +0000

Looks like this turns out to be a red herring sorry all.

improper header wrapping / bare linefeed in a response header value was 
pushing our header byte count out by 2 bytes.

------ Original Message ------
From: "Alex Rousskov" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "Adrien de Croy" <[email-masked]>
Sent: 13/07/2016 4:20:29 a.m.
Subject: Re: what constitutes an "invalid" content-length

>>around
>>or

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Jul 2016 10:21:03 +1000

Hey Patrick,

Just curious -- is the heuristic you use more than just "if we only have one outstanding request, keep reading bytes and consider them part of the response?"

Cheers,

> On 13 Jul 2016, at 12:19 AM, Patrick McManus <[email-masked]> wrote:
> 
> H1 Framing is badly broken in practice.. we tried to tighten up what Firefox would accept about a year ago and it was basically a disaster that was reverted. Some of the story is in the 300+ comments in https://bugzilla.mozilla.org/show_bug.cgi?id=237623
> 
> You can set network.http.enforce-framing.http1 to true if you'd like a more error filled web and lean on content-providers. That didn't get us very far because we're talking about the tail of the Internet - basically one off gateways to old legacy machinery like pdf generators, and broken dynamic gzip modules and stuff that insists 304 should have a body. While that doesn't dominate the Internet it does cause persistent breakage that a high number of people will see (and be frustrated by) over a fairly short period of time. Its extremely well baked in at this point and its very hard to fix - we're much better off enforcing strict standards around h2 and encouraging a migration than beating our heads against the h1 legacy imo.
> 
> I know there is some talk of trying to remove support for 0.9 in another browser. I sincerely wish that team nothing but happiness, good karma, and good luck in their trail blazing - but I wonder if its going to go a similar route.
> 
> -Patrick
> 
> just dealing with a site that sends more payload data than is indicated in the Content-Length header.
>  
> If the browser connects directly, the page loads fine, if via the proxy, the proxy is truncating the length to that advertised and the client isn't displaying a page (of course this is the .css file).
>  
> RFC7230 sections 3.3.2 (Content-Length), 3.3.3 (Message body length), and 3.3.4 (Handling incomplete messages) only contemplate issues around Content-Length specifying more bytes than are received, not fewer.
>  
> I guess one could argue that a wrong C-L value is "invalid", but it's not clear that invalid in this context simply means it doesn't parse, or is otherwise non-compliant with the ABNF.
>  
> So, it's not clear what the browser and/or proxy response should be.  If we deem a wrong value to be "invalid" (s3.3.3 para 4), a client is supposed to discard the response.  This isn't happening.
>  
> For the proxy, it only sees that the content length is wrong once it receives too many bytes.  By this stage, the horse has bolted so it cannot really comply either.
>  
> I would expect it's in everyone's best interest if sites that have broken framing are forced to be fixed.  This won't happen if browsers "just work" for the site.
>  

--
Mark Nottingham   https://www.mnot.net/

---

From: Patrick McManus <[email-masked]>
Date: Wed, 13 Jul 2016 08:40:42 -0400

On Tue, Jul 12, 2016 at 8:21 PM, Mark Nottingham <[email-masked]> wrote:

> Just curious -- is the heuristic you use more than just "if we only have
> one outstanding request, keep reading bytes and consider them part of the
> response?"

we don't actually do that - it doesn't mesh well with how fast we want to
either recycle connections or notify the in-browser consumer (i.e. the
parser, the decoder, etc..) about EOM.

If doing persistent connections - 'extra' bytes get silently thrown away as
leading garbage on the subsequent response... otherwise we just read to EOF
and ignore C-L/Chunked. (A surprising portion of the servers that get C-L
wrong are closing the connection anyhow) as a message delimiter.

My biggest frustration in this space is actually around the unreliability
of truncation detection. *lots* of non persistent h1 transactions that seem
to be strongly framed come up short by some criterion - c-l, missing a
zero-chunk terminator, or unclean close termiations like RST or TLS Alerts.
For all practical purposes you need to silently accept what you have
received. Of course some actual network driven truncations look exactly the
same. That's created a problem in the file downloader (do I show a retry or
not?) and there is a security implication as well (do you really need all
that javascript?) when carried over a non-authenticated transport (or when
we are forced to ignore some aspects of it.)

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 14 Jul 2016 07:26:04 +0200

> zero-chunk terminator, or unclean close termiations like RST or TLS Alerts.

I actually second this observation. In haproxy we have to process socket
errors *after* processing the message so that we get a chance to pass a
complete message in case it ends with a reset. And since haproxy is most
often installed very close to the server, we cannot even blame some
random blackboxes, it's the server which does nasty stuff. I found that
one cause of such issues is when the max_orphans setting on the system
is low and applications immediately close() after the last send(), and
haproxy itself used to be caught doing this from time to time as well.
As a workaround for this specific case in haproxy, we do a shutdown(SHUT_WR)
before the close(), which manages to send the last FIN before trying to
turn the socket into an orphan, so that if there's no more orphans and
an RST has to be sent, at least a FIN was sent before.

But what it means is that quite a number of resets are actually caused
by this and also indicate an intentional close() by the application,
meaning that you likely got the last chunk... unless it was killed in
the TCP stack upon receipt of the reset of course.

Willy