Subject: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 20:49:37 +0000

The language in RFC 7230 section 3.3.2 is extremely non-commital about 
whether Content-Length needs to be correct or not.

I'm currently having a dispute about this with someone who quoted these 
sections at me as being proof that you can use any value for C-L 
regardless of the body length.

I think it could be a lot more forcefully written

Or is the person correct and we don't need to have C-L match the body 
length?

In this particular case the discussion is around request messages.

Adrien

---

From: Loïc Hoguin <[email-masked]>
Date: Tue, 14 Feb 2017 22:05:46 +0100

> Or is the person correct and we don't need to have C-L match the body

It sounds pretty explicit to me:

    4.  If a message is received without Transfer-Encoding and with
        either multiple Content-Length header fields having differing
        field-values or a single Content-Length header field having an
        invalid value, then the message framing is invalid and the
        recipient MUST treat it as an unrecoverable error.  If this is a
        request message, the server MUST respond with a 400 (Bad Request)
        status code and then close the connection.

If it's both invalid and required for handling the request, send a 400 
and close the connection.

I suppose the spec allows you to have an invalid Content-Length if and 
only if the request also has a Transfer-Encoding header, however:

        If a message is received with both a Transfer-Encoding and a
        Content-Length header field, the Transfer-Encoding overrides the
        Content-Length.  Such a message might indicate an attempt to
        perform request smuggling (Section 9.5) or response splitting
        (Section 9.4) and ought to be handled as an error.

So sending a 400 and closing does not sound crazy even in that case, 
despite the spec not requiring it.

-- 
Loïc Hoguin
https://ninenines.eu

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 14 Feb 2017 21:11:55 +0000

--------
In message <[email-masked]>, =?UTF-8?Q?Lo=c3=afc_Hoguin?= writes:

>        invalid value,

Last year, I think, I have had some dude claim that a "valid value"
was a sequence of digits, but that there is no explicit requirement
that it match the length of the body of that request.

I told him to get stuffed.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 21:12:38 +0000

I did quote that section, but it doesn't define what an invalid C-L is.

Nowhere does it explicitly state that C-L value must equal the body size 
in order to be valid.  Sure it should be obvious, but the language at 
the start of 3.3.2 doesn't help there.

It says

"Any Content-Length field value greater than or equal to zero is
valid."

well no, to be valid it must match the body size.

The first sentence in 3.3.2

"When a message does not have a Transfer-Encoding header field, a
Content-Length header field can provide the anticipated size, as a
decimal number of octets, for a potential payload body."

"can provide"???
"anticipated size"???
"potential payload body"???

There's either a payload or there isn't.
C-L is either required or it isn't.
If size isn't known you can't send C-L, If it's a request you must use 
T-E chunked, since you can't close after sending and still receive the 
reply.

Adrien

------ Original Message ------
From: "Loïc Hoguin" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>; "[email-masked]" 
<[email-masked]>
Sent: 15/02/2017 10:05:46 AM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

>-- Loïc Hoguin

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 21:13:27 +0000

I wonder if it's the same guy.

------ Original Message ------
From: "Poul-Henning Kamp" <[email-masked]>
To: "Loïc Hoguin" <[email-masked]>
Cc: "Adrien de Croy" <[email-masked]>; "[email-masked]" 
<[email-masked]>
Sent: 15/02/2017 10:11:55 AM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

---

From: Mike Bishop <[email-masked]>
Date: Tue, 14 Feb 2017 21:15:02 +0000

Valid and matching the body length aren't quite the same, though.  I agree, it's a bit wishy-washy -- which sounds like one or more major implementations do something odd here that we didn't quite agree on invalidating.

However, in message framing, it does say that the length of the expected body is taken from the Content-Length header, and if the body doesn't match that you'll fail in odd ways.

-----Original Message-----
From: Loïc Hoguin [mailto:[email-masked]] 
Sent: Tuesday, February 14, 2017 1:06 PM
To: Adrien de Croy <[email-masked]>; [email-masked]
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

On 02/14/2017 09:49 PM, Adrien de Croy wrote:
> I'm currently having a dispute about this with someone who quoted 
> these sections at me as being proof that you can use any value for C-L 

If it's both invalid and required for handling the request, send a 400 and close the connection.

I suppose the spec allows you to have an invalid Content-Length if and only if the request also has a Transfer-Encoding header, however:

So sending a 400 and closing does not sound crazy even in that case, despite the spec not requiring it.

--
Loïc Hoguin
https://ninenines.eu

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 15 Feb 2017 07:15:52 +1000

On 15 Feb 2017 06:54, "Adrien de Croy" <[email-masked]> wrote:

I'm currently having a dispute about this with someone who quoted these
sections at me as being proof that you can use any value for C-L regardless
of the body length.

Adrien

"For messages that do include a payload body, the Content-Length
field-value provides the framing information necessary for determining
where the body (and message) ends."

That seems pretty significant to me. Especially when you consider how
pipelining works. Then in the next section:

"If a valid Content-Length header field is present without
Transfer-Encoding, its decimal value defines the expected message
body length in octets.  If the sender closes the connection or
the recipient times out before the indicated number of octets are
received, the recipient MUST consider the message to be
incomplete and close the connection."

That, along with everything else in 3.3.2 & 3.3.3, and 3.4 makes me
wonder how anyone could find ambiguity there. If C-L is present, it
has to be correct.

Cheers (and sorry if gmail messes up the formatting)

---

From: "Jason T. Greene" <[email-masked]>
Date: Tue, 14 Feb 2017 16:47:52 -0500 (EST)

> I told him to get stuffed.

Out of morbid curiosity, what semantics did he expect?

---

From: Jacob Champion <[email-masked]>
Date: Tue, 14 Feb 2017 13:53:29 -0800

> "potential payload body"???

Seems to me that those weird qualifications are there mostly for the 
case where Content-Length is sent without a payload body. From the same 
paragraph:

    [...] For messages
    that do include a payload body, the Content-Length field-value
    provides the framing information necessary for determining where the
    body (and message) ends.  For messages that do not include a payload
    body, the Content-Length indicates the size of the selected
    representation (Section 3 of [RFC7231]).

Without the (admittedly vague) language, you'll have people trying to 
parse bodies from responses to HEAD requests if they have a 
Content-Length, or people complaining because the server changed the 
size of a representation between the HEAD and the GET, or whatever. 
Thus, "anticipated size" and "potential payload body".

I'm curious to know what use case someone could possibly create for a 
client to send an incorrect Content-Length.

--Jacob

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 14 Feb 2017 21:57:13 +0000

--------
>Out of morbid curiosity, what semantics did he expect?

I think it was "It should just work" ?

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 14 Feb 2017 23:05:12 +0100 (CET)

On Wed, 15 Feb 2017, Matthew Kerwin wrote:

> That, along with everything else in 3.3.2 & 3.3.3, and 3.4 makes me wonder 
> how anyone could find ambiguity there. If C-L is present, it has to be 
> correct.

... but we still see a significant amount of content on the web today that 
serve bodies with sizes that aren't matching the C-L value.

I went several rounds against that, trying to make Firefox's C-L checks 
stricter, but finally lost in the end and had to resort to keeping funky 
heuristics and guessworks.

The web is a harsh place! =)

-- 

  / daniel.haxx.se

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 14 Feb 2017 23:05:04 +0100

On Tue, Feb 14, 2017 at 09:12:38PM +0000, Adrien de Croy wrote:
> Nowhere does it explicitly state that C-L value must equal the body size in
> order to be valid.  Sure it should be obvious, but the language at the start
> of 3.3.2 doesn't help there.
> 
> It says
> 

In fact that's not true. It's not the content-length that matches the
body size, it's the body which ends after the content-length. So whatever
numeric value >= 0 is indeed valid *and* defines the body size.

> "potential payload body"???

I think that's for 304 or responses to HEAD. Though out of the context,
the wording looks scary.

Regardless the text explaining how to determinate the body size (hence
where it stops once you've found where it starts) is pretty clear and
I hardly see how to interpret it differently.

You can make a parallel with a piece of string. Tell him "I'm giving you
2 meters of string, just pull it and cut at 2 meters, this is your body,
and what remains in my hand is mine and is not your body ; if I don't
tell you where to cut before pulling, you pull everything until I run
out of it and only then you can determine the length".

Anyway it's sad to see that there are people having so many difficulties
understanding so obvious concepts and that such people are allowed to
demand that a technical component works one way or another...

Willy

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 22:13:56 +0000

------ Original Message ------
From: "Willy Tarreau" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Loïc Hoguin" <[email-masked]>; "[email-masked]" 
<[email-masked]>
Sent: 15/02/2017 11:05:04 AM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

>numeric value >= 0 is indeed valid *and* defines the body size.
I think there's an argument you could make against this.

If the C-L is smaller than the transmitted body, then the remainder 
after C-L bytes is processed as an invalid new request, or maybe 
smuggling request.

If the C-L is larger than the transmitted body then the message is 
incomplete.  In a response message the connection would need to be 
closed, and in a request message there's no recovery from this.

An implementation may choose to just truncate the message at C-L bytes 
of body, but this is an implementation decision, not necessarily a 
specification?

I think it should be made clear that for the body and C-L to be both 
correct in terms of framing, the C-L value must match the size of the 
body in bytes, if there's a mis-match you can't claim C-L is correct.

>I think that's for 304 or responses to HEAD. Though out of the context,

Where did we end up with C-L on 304 responses?  I didn't think it was 
another case like HEAD where the 304 is the size of what would have been 
sent?

>demand that a technical component works one way or another...

Or telling others this is how it works in a public Q&A forum.  It 
doesn't help anyone for this interpretation to propagate.

Adrien

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 22:15:14 +0000

Yes, you can do this in a U-A, but in a proxy we have to be strict :)

I guess those sites don't work behind proxies.

Adrien

------ Original Message ------
From: "Daniel Stenberg" <[email-masked]>
To: "Matthew Kerwin" <[email-masked]>
Cc: "Adrien de Croy" <[email-masked]>; "HTTP Working Group" 
<[email-masked]>
Sent: 15/02/2017 11:05:12 AM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

>... but we still see a significant amount of content on the web today 
>that serve bodies with sizes that aren't matching the C-L value.
>
>stricter, but finally lost in the end and had to resort to keeping 
>funky heuristics and guessworks.
>

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 14 Feb 2017 15:38:17 -0700

> I did quote that section, but it doesn't define what an invalid C-L is.

The term "valid" in that section means "syntactically correct". 123 is
valid. 0x123 is not. 0123 is valid unless the recipient is paranoid.

> in order to be valid.

You are correct. The message framing rules (3.3.3.1-5) establish that
C-L value and body length are the same concept (for the applicable cases
where C-L value is used for framing and only for those cases).

In other words, one should not add a "C-L value MUST match the body
length" or "the body length MUST match the C-L value" rule because the
body length _is_ the C-L value (for the applicable cases). Adding such a
rule would be like saying "an object with a weight of 5 tons MUST weigh
5 tons".

HTH,

Alex.

---

From: Jacob Champion <[email-masked]>
Date: Tue, 14 Feb 2017 14:43:43 -0800

> closed, and in a request message there's no recovery from this.

I think Willy's point is that, from a protocol perspective, C-L 
*defines* the body (for the types of messages we're discussing here). 
It's not possible to transmit a body -- for HTTP's definition of "body" 
-- that is "smaller" or "larger" than the C-L. In one case, the endpoint 
runs out of bytes to read and eventually times out. In the other, it 
reaches the end of the body, and now there happens to be junk data at 
the start of the next received message.

> of body, but this is an implementation decision, not necessarily a

Well, there's this part of 3.3.3:

    If the final response to the last request on a connection has been
    completely received and there remains additional data to read, a user
    agent MAY discard the remaining data or attempt to determine if that
    data belongs as part of the prior response body, which might be the
    case if the prior message's Content-Length value is incorrect.  A
    client MUST NOT process, cache, or forward such extra data as a
    separate response, since such behavior would be vulnerable to cache
    poisoning.

which to me seems like a MAY that is doomed to fail in the general case. 
Broken pipelining, race conditions, cache poisoning...

--Jacob

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 22:54:37 +0000

I have no problem with the concept of adding a rule that states that

objects labelled as weighing 5T MUST weigh 5T or the label is 
incorrect/invalid.

The Content-Length field value is an assertion of an attribute.

------ Original Message ------
From: "Alex Rousskov" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "Adrien de Croy" <[email-masked]>
Sent: 15/02/2017 11:38:17 AM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

>>>>these

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 14 Feb 2017 16:13:56 -0700

> objects labelled as weighing 5T MUST weigh 5T or the label is

Thinking of Content-Length as a packaging label gets you into the very
trap you want to escape: Yes, rules for labeling accuracy would be fine,
but Content-Length (in relevant contexts) is _not_ a label!
Content-Length does not merely document weight that you can
independently measure and validate. Content-Length _is_ weight.

Alex.

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 23:18:24 +0000

I disagree

Content-Length the concept is indeed the content length.

Content-Length the header is a _header_ which _represents_ 
Content-Length the concept.

It's a promise made by the sender that the sender will send that many 
bytes of body.

The only true size of a body is what you obtain by counting its bytes.  
The Content-Length header is not part of that body, it's sent in the 
headers, we parse it, convert the string to a number.

Yet we have no rule stating it must be consistent with what is sent.

I think we could be a little less timid about this and call it like it 
is.

Adrien

------ Original Message ------
From: "Alex Rousskov" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>; "[email-masked]" 
<[email-masked]>
Sent: 15/02/2017 12:13:56 PM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

>>>>   Nowhere does it explicitly state that C-L value must equal the 
>>>>body
>>>>  size
>>>>   in order to be valid.
>>>
>>>>>          recipient MUST treat it as an unrecoverable error.  If 
>>>>>this
>>>>>  is a
>>>>>          Content-Length header field, the Transfer-Encoding 
>>>>>overrides
>>>>>  the

---

From: Jacob Champion <[email-masked]>
Date: Tue, 14 Feb 2017 15:24:48 -0800

> The only true size of a body is what you obtain by counting its bytes.

Can you explain how you would count the bytes in a message body if you 
don't know where that body ends?

--Jacob

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 23:25:07 +0000

and I understand the argument that the header value defines the length 
of the body.

But that doesn't help with what happens if the actual body sent is 
different to what is advertised.

You can say that by definition if moe are sent, then they are not body 
by definition, but what if fewer are sent?

It's an incomplete message.

Maybe this should be turned around, so that byte sent MUST equal C-L 
advertised.  I'm just saying they should match and we shouldn't have a 
problem stating that, or stating that a mis-match is a fatal error.

Adrien

------ Original Message ------
From: "Adrien de Croy" <[email-masked]>
To: "Alex Rousskov" <[email-masked]>; 
"[email-masked]" <[email-masked]>
Sent: 15/02/2017 12:18:24 PM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

>>>>>>>   The language in RFC 7230 section 3.3.2 is extremely 
>>>>>>>non-commital
>>>>>>>  about

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 14 Feb 2017 23:34:30 +0000

A receiver cannot.

A sender can however (e.g. get file size, string length whatever).

A receiver therefore depends on the sender sending the correct value.

This is presuming the sender knows the length.  If the sender doesn't 
know the length it should not (MUST) set Content-Length header, but 
instead use chunking.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 14 Feb 2017 17:32:04 -0700

> The only true size of a body is what you obtain by counting its bytes. 

I disagree. The only true size of a body is the Content-Length value (in
relevant contexts). The number of bytes counted by the sender or
recipient do not change the message body length. Yes, there could be
garbage after the body and, yes, something may prevent sending or
receiving of the whole body, but none of that changes what the body size is.

If you define the message body size as the number of bytes sent or
received (in relevant contexts), then you may add rules about those
numbers, but HTTP specs do not define the message body size that way.

Alex.

> I think we could be a little less timid about this and call it like it is.
> 
> Adrien
> 
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 15 Feb 2017 00:42:50 +0000

------ Original Message ------
From: "Alex Rousskov" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>; "[email-masked]" 
<[email-masked]>
Sent: 15/02/2017 1:32:04 PM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

>relevant contexts).

What about for a sender piecing a message together.  Where does 
Content-Length come from?

The content existed before you derived or obtained its length.

>The number of bytes counted by the sender or
>>>>>   C-L value and body length are the same concept (for the 
>>>>>applicable
>>>>>  cases
>>>>>>>           request message, the server MUST respond with a 400 
>>>>>>>(Bad
>>>>>>>   Request)
>>>>>>>    If it's both invalid and required for handling the request, 
>>>>>>>send
>>>>>>>  a 400
>>>>>>>    I suppose the spec allows you to have an invalid 
>>>>>>>Content-Length
>>>>>>>  if and

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 14 Feb 2017 18:15:49 -0700

> The content existed before you derived or obtained its length.

The content may have existed, of course, but the HTTP message (and,
hence, HTTP message body) did not.

When the agent creates an HTTP message with a Content-Length header (no
T-E, etc.), that agent determines the HTTP message body size. Whether
the agent first computes the C-L header value or imports the body is an
irrelevant implementation detail from the _protocol_ point of view. On
the protocol/conceptual level, the header value and the body size are
the same thing. We could add qualifiers like expected or anticipated
message body size but they actually make the text less accurate IMHO.

Consider an agent that reads body content from a 100-byte file, counts
each byte read, but forgets to read the last byte (bug!) and then sets
the Content-Length header value to 42 (another bug!). From the HTTP
point of view, the message body is going to be 42 bytes long even though
that body does not accurately represent the content the agent read from
the disk or the content it ought to be actually sending.

And the above example will still stand if I replace 100 with 10!

Alex.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 15 Feb 2017 11:18:43 +1000

On 15 February 2017 at 10:42, Adrien de Croy <[email-masked]> wrote:

> To: "Adrien de Croy" <[email-masked]>; "[email-masked]" <
> [email-masked]>
When we say "content" do we mean content of the message, content of the
representation, or content of the resource? I've always taken
content-length to mean the content of the representation[*] -- which, since
the demise of T-E, basically means message payload.

When we say "body", I hope we're all always talking about message payload.
(i.e.: representation · Range · T-E)

Reasoning about the _*resource*_ (including file size) is the purview of
the application, outside of HTTP transport or semantics. At least, that's
how it looks from my ivory tower.

[*] something something Range something

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 15 Feb 2017 01:45:27 +0000

Maybe this discussion is indicative of the underlying problems in the 
(lack of?) definition, which results in things like browsers accepting 
smaller payloads than advertised without complaining.

If we turn it around, under what circumstances could we consider that

a user-agent emitting a request message which includes a Content-Length 
header whose value is greater than the number of bytes it then transmits 
after the headers

Is not an error condition, or that the message should be processed as if 
it were complete?

I also wonder whether we would be having a similar discussion on the TLS 
WG about record lengths.

When security is at stake I thought we tended to take a firmer stand.

It seems from the language in 3.3.3 par 4 that the intention is that the 
C-L MUST match the body payload size.

The language in 3.3.2 however does not reinforce this.

Adrien

------ Original Message ------
From: "Matthew Kerwin" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Alex Rousskov" <[email-masked]>; 
"[email-masked]" <[email-masked]>
Sent: 15/02/2017 2:18:43 PM
Subject: Re: Sections 3.3.2 and 3.3.3 allow bogus Content-Length?

>>From: "Alex Rousskov" <[email-masked] 
>><mailto:[email-masked]>>
>content-length to mean the content of the representation[*] -- which, 
>since the demise of T-E, basically means message payload.
>
>When we say "body", I hope we're all always talking about message 
>payload. (i.e.: representation · Range · T-E)
>
>Reasoning about the _resource_ (including file size) is the purview of 
>the application, outside of HTTP transport or semantics. At least, 
>that's how it looks from my ivory tower.
>

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 14 Feb 2017 22:38:02 -0700

> Is not an error condition,

Never. That is, it is always an error condition.

> or that the message should be processed as if it were complete?

When the receiving agent is configured to work around certain well-known
error conditions (because the decision makers have decided that
rejecting the request under those conditions will make things worse for
them).

FWIW, I do not recall any such conditions on the request receiving side
in Squid (other than those already documented as tolerable in the HTTP
specs), but there are some framing-related hacks on the response
receiving side IIRC.

> I also wonder whether we would be having a similar discussion on the TLS

There is essentially no difference at the protocol level AFAICT. Are
there any hacks in some SSL implementations to accommodate known error
conditions? I do not know. Compared with plain text HTTP, the difficulty
in writing an SSL agent from scratch combined with outlawing older
protocol versions probably helps minimize the long-term need for such
hacks. Said that, Squid still has to deal with ancient SSLv2 records
(containing TLS v1 messages)...

> When security is at stake I thought we tended to take a firmer stand.

I do not think this thread contained any recommendations not to take a
firm stand. The only disagreement was whether the protocol is missing
some MUSTs that would help making that stand easier.

> C-L MUST match the body payload size.

Since C-L defines the body/payload size, the two always match. This
equivalence allows the recipient to determine when it got a truncated
message or post-message garbage.

Alex.

> From: "Matthew Kerwin" <[email-masked]
> <mailto:[email-masked]>>
> To: "Adrien de Croy" <[email-masked] <mailto:[email-masked]>>
> Cc: "Alex Rousskov" <[email-masked]
> <mailto:[email-masked]>>; "[email-masked]"
> <[email-masked] <mailto:[email-masked]>>
>> On 15 February 2017 at 10:42, Adrien de Croy <[email-masked]
>> <mailto:[email-masked]>> wrote:
>>
>>
>>
>>     To: "Adrien de Croy" <[email-masked] <mailto:[email-masked]>>;
>>     "[email-masked] <mailto:[email-masked]>"
>>     <[email-masked] <mailto:[email-masked]>>
>> When we say "content" do we mean content of the message, content of
>> the representation, or content of the resource? I've always taken
>> Reasoning about the _/resource/_ (including file size) is the purview
>> of the application, outside of HTTP transport or semantics. At least,

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 15 Feb 2017 16:56:46 +1100

Hi Adrien,

Interesting conversation ;) My .02 -

The number of bytes that the sender sends and the recipient receives might be different (e.g., network segments, prematurely closed connections, machine crashes). The fact that the recipient receives less bytes than Content-Length indicated doesn't make the sender any less conformant.

A pathological sender might send *more* bytes than Content-Length indicates. In the strictest sense, this behaviour is conformant, because it doesn't violate a MUST, but it clearly violates the intent of the field, and harms interoperability. Unfortunately, this isn't theoretical.

As Alex and others have said, Content-Length defines the message body, in the contexts where it's used. A recipient can try to handle errors gracefully (e.g., doing something useful with an incomplete representation, or treating bytes following it on the connection as part of the representation when there isn't another outstanding message). As per 2.5:

> Unless noted otherwise, a recipient MAY attempt to recover a usable protocol element from an invalid construct. HTTP does not define specific error handling mechanisms except when they have a direct impact on security, since different applications of the protocol require different error handling strategies.

Cheers,

> On 15 Feb 2017, at 7:49 am, Adrien de Croy <[email-masked]> wrote:
> 
> 
> The language in RFC 7230 section 3.3.2 is extremely non-commital about whether Content-Length needs to be correct or not.
> 
> I'm currently having a dispute about this with someone who quoted these sections at me as being proof that you can use any value for C-L regardless of the body length.
> 
> Or is the person correct and we don't need to have C-L match the body length?
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 15 Feb 2017 07:53:18 +0100

On Tue, Feb 14, 2017 at 10:13:56PM +0000, Adrien de Croy wrote:
> If the C-L is smaller than the transmitted body, then the remainder after
> C-L bytes is processed as an invalid new request, or maybe smuggling
> request.
> 
> incomplete.  In a response message the connection would need to be closed,
> and in a request message there's no recovery from this.
> 
> An implementation may choose to just truncate the message at C-L bytes of
> body, but this is an implementation decision, not necessarily a
> specification?

But that's exactly what happens in practice. In a request you don't want
to forward more than C-L because whatever follows is a new request (eg:
pipelining). In a response, if some bytes follow the C-L, it can be an
error message when the server closes on time-out. If the C-L announces
too large a message, the server may wait forever to consume the request
message, while too long a C-L in a response, followed by a close simply
indicates to the client a bad network condition with a connection cut
in the middle of the transfer (ie it may retry with a byte-range to get
the missing part).

> I think it should be made clear that for the body and C-L to be both correct
> in terms of framing, the C-L value must match the size of the body in bytes,
> if there's a mis-match you can't claim C-L is correct.

But that's really the point I do not understand well given that it defines
the body size. Imagine you transmit a body made of many requests, followed
by a series of similar requests. You can easily imagine that only the C-L
can tell where the end is :

POST / HTTP/1.1
Host: foo
Content-length: 100

POST / HTTP/1.1
Host: foo
Content-length: 100

POST / HTTP/1.1
Host: foo
Content-length: 100

POST / HTTP/1.1
Host: foo
Content-length: 100

POST / HTTP/1.1
Host: foo
Content-length: 100

POST / HTTP/1.1
Host: foo
Content-length: 100

...
etc.

Only the C-L here can let you now how many real requests you have because
even if visually some look like requests, they are not, they are just an
opaque body.

> another case like HEAD where the 304 is the size of what would have been

It's specified that you have to ignore it for the framing (as well as on
HEAD and 204). I've seen it wrongly set in the past on some servers (a
long time ago), causing interoperability breakage (it was set to 0,
causing intermediary caches to truncate their bodies). By definition,
HEAD, 204 and 304 don't carry a body.

> Or telling others this is how it works in a public Q&A forum.  It doesn't
> help anyone for this interpretation to propagate.

Stupid people claiming to be experts on public forums have always existed,
and it's only their attendees' fault if they believe them instead of reading
the official specs. We are responsible (and possibly faulty) for any unclear
info in the spec but at least it's written with the expectation that any
reader will have a measurable IQ and we've done a lot to even address some
lack of common sense... It could also be used as a test to say "if you can't
get it you're not allowed to implement anything for me"...

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 15 Feb 2017 07:23:32 +0000

--------
In message <[email-masked]>, Mark Nottingham writes:

My 0.02 DKK:

In H1 Content-Length is part of the "framing layer", and therefore
we should not be tolerant of senders transmitting bogus values,
just like we don't allow H2 senders the option of bogus frame length.

>A pathological sender might send *more* bytes than Content-Length 
>indicates. In the strictest sense, this behaviour is conformant, because 
>it doesn't violate a MUST, but it clearly violates the intent of the 
>field, and harms interoperability. Unfortunately, this isn't 
>theoretical.

This is where skipping this discussion during the H2 phase have
created a new class of problems of us:  In H2 the body has a true
length, as delinieated by the framing, in H1 that is only the
case with chunked.

In H2 C-L can be given any role/significance we prefer, but I
personally think we should treat H1 and H2 senders the same:  C-L
can only be sent if it is precise[1] otherwise leave it out, (and
use chunked in H1).

How the receiver reacts to a mismatch is a policy decision, which
should probably lean tolerant for missing bytes at human-driven
clients, in particular for huge objects, and be zero tolerance
for mistakes in all other cases[2].

Poul-Henning

[1] I do recoginize that significant performance improvements could
be had if we had C-L-hints in H2 and in H1 with chunked, but
overloading Content-Length would be too dangerous, maybe:

	 "Body-Length: 1024?"  -> This is only an estimate

	 "Body-Length: >1024"  -> At least this many bytes

	 "Body-Length: <1024"  -> No longer than this

[2] Postel's rule was for a network shared with all your
friends, we live in a network with all our enemies.

---

From: Cory Benfield <[email-masked]>
Date: Wed, 15 Feb 2017 10:37:27 +0000

> On 15 Feb 2017, at 07:23, Poul-Henning Kamp <[email-masked]> wrote:
> 
> My 0.02 DKK:
> 

My 0.02 GBP is in agreement with PHK here. Content-Length is not only metadata, it is also data about the framing. A Content-Length header defines where the request body ends (except in the specific few exceptions as defined in the RFCs, which are well-known and understood). Any appeal to the platonic ideal of what a resource should look like is nonsensical.

> length, as delinieated by the framing, in H1 that is only the

So, as a data point, the Python HTTP/2 implementations all strictly enforce content-length. This is an unappreciated benefit of H2. In HTTP/1.1 a message that does not match the content length is moderately tricky to detect: messages that are too short are indistinguishable from timeouts or other network weather, and messages that are too long require that the recipient deploy heuristics to check whether the next bytes are actually a new response or appear to be part of the previous representation. Most Python implementations don’t bother trying to detect that because it’s not worth spending the CPU time, so in practice messages that are too long are just truncated and a parsing error occurs if the connection is re-used.

In H2, however, there is a very clear way to police content-length: count the data bytes as they come in until END_STREAM. If they don’t match the content-length header that was provided, that’s an error: GOAWAY(PROTOCOL_ERROR). There is no excuse for H2 implementations to get request/response sizes wrong, especially as PHK points out that implementations that are not certain of body size can simply strip the C-L header.

So I don’t really consider this a new class of problems, so much as H2 giving us the solution to this problem. In my view, for H2 implementations should either treat the content-length is an absolute right-or-wrong, or just ignore it completely. Either works, and implementations may decide which they prefer. I suspect most will go for the former (except browser UAs, who are generally speaking inclined to tolerate most misbehaviour if possible).

---

From: Patrick McManus <[email-masked]>
Date: Wed, 15 Feb 2017 08:33:57 -0500

On Wed, Feb 15, 2017 at 5:37 AM, Cory Benfield <[email-masked]> wrote:

> So I don’t really consider this a new class of problems, so much as H2
> giving us the solution to this problem. In my view, for H2 implementations
> should either treat the content-length is an absolute right-or-wrong, or
> just ignore it completely. Either works, and implementations may decide
> which they prefer. I suspect most will go for the former (except browser
> UAs, who are generally speaking inclined to tolerate most misbehaviour if
> possible).

content-length in h2 doesn't define the framing - as Cory points out we are
saved from the sad h1 reality by consistent and simple framing rules in h2.
It should just be used informationally - e.g. for drawing download
thermometers. or making sensible pre-allocations/quota checks (assuming you
don't also make a TOCTOU bug out of it). Firefox treats all h2 transactions
as if they were h1 chunked.

with the exception of the anecdote that started this thread (funny!), I'm
not aware of anyone actually trying to publish incorrect CL and call it
compliant. The exceptions in implementations that allow consuming such
behavior are there as explicit choices about bugwards-interop, changing
spec language wrt h1 isn't going to change those decisions. The
implementors already either understand the tradeoffs or made an
unintentional mistake in generating the crud in the first place. This
however is a reason why the language in h2 is much stronger - so that early
interop would force implementations to clean up their act during the launch
phase. I'm sure that entropy will inevitably win, but the road is
considerably cleaner for a while.

I will say that during h2 dev chrome firefox and msft were all vigilant in
situations where there was a bug report from some new content source of "it
works in A, but not in B" (which happened for all variations of A and B)
and the root cause turned out to be a sender violation with a more lenient
variant allowing it.. the resolution was always changes that created a
stricter enforcement across the board. (i.e. now it doesn't work in A or
B!)  this applies to more than browsers of course, but reporters love to
play us off against each other.

That's a lesson to be learned for quic - and its a lesson that we should
stay vigilant in h2 even though it is now deployed. Otherwise entropy wins
and you end up with defacto specs. This is a good forum to raise those
issues as they come up.

-Patrick