Subject: P1: Content-Length SHOULD be sent

From: Mark Nottingham <[email-masked]>
Date: Tue, 27 Nov 2012 16:21:25 +1100

Currently, p1 says:

> When a message is allowed to contain a message body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been sent, the sender should send a Content-Length header field to indicate the length of the payload body as a decimal number of octets.

This unqualified SHOULD leads people to convoluted readings of the spec where Content-Length is required to be sent on a GET request:
  https://github.com/kennethreitz/requests/issues/223#issuecomment-10745532

Proposal:

> When a message is allowed to contain a body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been sent, the sender should send a Content-Length header field to indicate the length of the payload body as a decimal number of octets, unless the message is a request and the payload length is zero (in which case the Content-Length header MAY be sent). 

--
Mark Nottingham   http://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 27 Nov 2012 07:05:16 +0100

> > When a message is allowed to contain a body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been s…

Seems fine to me too.

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 27 Nov 2012 22:32:52 +1300

>    https://github.com/kennethreitz/requests/issues/223#issuecomment-10745532

Reading that thread I don't see anyone requiring CL on a GET. Just CL:0 
on a POST with no body - which seems correct.
>> When a message is allowed to contain a body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been se…

If I'm reading that right any recipient MUST consider a request with no 
Content-Length or Transfer-Encoding header as being 0-length.
   That opens a request smuggling loophole when overly zealous 
privacy/anonymizer config has been implemented. When proxy-A is known to 
erase CL headers (but obeys them) it can be sent a POST with smuggled 
request and victim request in pipeline. Proxy-A duly erases the CL and 
passes what server X is now required to interpret as three requests, 
resulting in proxy-A getting the smuggled requests response stored as 
the victums reply - and some garbage at the end of the pipeline.
  Bit rare, but I have seen people erasing every header they thought was 
optional because "some requests dont have it".

That paragraph also looks like a confusingly long set of conditionals to me.

What about something along the lines of:
   When a message is allowed to contain a body Content-Length SHOULD be 
sent, if the length of body is not known or is not easily retrieved in 
adavnce Transfer-Encoding:chunked SHOULD be used instead.

I am also inclined to suggest a MUST on there being some indication of 
length other than close. If that does not cause too many existing 
implementation to become invalid we have a chance to kill off that pesky 
close signal earlier rather than later.

Amos

---

From: Zhong Yu <[email-masked]>
Date: Tue, 27 Nov 2012 09:03:22 -0600

>> When a message is allowed to contain a body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been se…

Isn't there a semantic difference between a request without a payload
and a request with an empty payload?

If a POST request payload is empty (which is perfectly valid, e.g. an
HTML form post without parameters), and it doesn't include
"Content-Length:0", it may break server implementations, because the
server checks whether a request has an "entity" based on
Content-Length/Transfer-Encoding, regardless of request method. If
Content-Length:0 is missing, the server will think that this is a POST
request without "entity", which it is probably unprepared for.

Zhong Yu

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 27 Nov 2012 10:34:19 -0500

On Tue, Nov 27, 2012 at 4:32 AM, Amos Jeffries <[email-masked]> wrote:
> resulting in proxy-A getting the smuggled requests response stored as the
> victums reply - and some garbage at the end of the pipeline.

Why isn't the answer to the above corner case simply 'you lose' ?

Seems to me that a lot of potential for forward progress in the HTTP world
is being blocked by people dredging up the most bizarre corner cases
imaginable. Often times corner cases that probably should not be fixed.

What is a privacy proxy anyway? And why would a person be using it? And why
would it be a good idea for the HTTP protocol to provide a way to
circumvent the control?

If people write proxies that break on very frequent cases such as the POST
request then they are going to be broken no matter what we write in the
spec.

I don't think it is worth any working group spending time on a
non-conforming implementation that has less than a 5% deployed base. For
HTTP that is a LOT of deployed base.

-- 
Website: http://hallambaker.com/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 27 Nov 2012 11:28:19 -0800

>> When a message is allowed to contain a body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been se…

That would be incorrect, so I don't see why it is being suggested.
Try it with POST on a valid CGI script and it will result in a
parser failure (if not a segfault).

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 28 Nov 2012 09:58:45 +1100

> parser failure (if not a segfault).

Because the current text reads that you SHOULD send C-L: 0 on a GET.

How about:

> When a message is allowed to contain a body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been sent, the sender should send a Content-Length header field to indicate the length of the payload body as a decimal number of octets, unless the message is a request and the method does not define any meaning for the body (in which case the Content-Length header MAY be sent).

---

From: Zhong Yu <[email-masked]>
Date: Tue, 27 Nov 2012 17:34:12 -0600

> Because the current text reads that you SHOULD send C-L: 0 on a GET.

In the old term of "entity" (since I'm not too sure about the precise
meaning of "payload")

If a GET request doesn't carry an entity, which is usually the case,
RFC2616 doesn't require C-L:0.

Actually C-L:0 must not be sent in this case. If the receiver sees
C-L:0, it means the request carries an empty entity, which means
something different than if the request carries no entity.

>> When a message is allowed to contain a body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been se…

You probably meant

unless the message is a request **and the payload length is zero** and
the method does not define any meaning for the body (in which case the
Content-Length header MAY be sent).

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 28 Nov 2012 12:59:57 +1300

On 28.11.2012 04:34, Phillip Hallam-Baker wrote:
> On Tue, Nov 27, 2012 at 4:32 AM, Amos Jeffries wrote:
>>

Because the behaviour enabled is a cache poisoning vulnerability not 
just a screwed up user experience.
It does not matter how unlikely the edge case, if its creating a 
vulnerability somebody will take advantage eventually.

> What is a privacy proxy anyway? And why would a person be using it?

I guess you have not spotted Privoxy in action yet (~2% of the 
Debian/Ubuntu installs run it - only $diety$ knows how many Edubuntu 
end-users that covers). With some big players on the server end pushing 
the user tracking game to full speed there is a growing force of 
over-reaction from those end-users. They seems to finally be getting fed 
up with having to enumerate the long and growing list of badness. This 
year the demand shifted from 'how do we disable feature X' as feature-X 
was announced as a tracker to 'how do we disable *everything* and only 
enable header X and Y' (not even whole features, *individual* headers). 
If they could revert HTTP back to HTTP/0.9 with only a request-line sent 
they would. Demand for this seems to be on par with demand for 
high-performance optimizations FWIW.

> And why

> HTTP that is a LOT of deployed base.

You seem to think privacy/anonymity is the concern of a small 
deployment base. For starters, schools in most western countries have 
mandatory tracking prevention by law on all students connectivity; the 
restrictions are paranoid and consistently similar. That covers a lot of 
%-points, well into your 5% before we even get near the paranoia in 
military and corporate sectors, or the edge-case crazy folks.

Amos

---

From: Martin Thomson <[email-masked]>
Date: Tue, 27 Nov 2012 16:10:07 -0800

> When a message is allowed to contain a body, does not have a Transfer-Encoding header field, and has a payload body length that is known to the sender before the message header section has been sen…

IANAL, so this was hard to parse for me.  How about:

--8<--
The Content-Length header indicates the size of a message body as a
decimal number of octets.  The sender SHOULD set Content-Length if:
   o  the message is allowed to contain a body,
   o  the Transfer-Encoding header field is not included, and
   o  the size of the message body is known at the time that headers are sent.
Messages that do not define a meaning for the body MAY either omit the
Content-Length header field or include a value of 0.
-->8--

I'm leery of the "SHOULD" still.  It seems weak.  I'd almost go with
MUST include and a fourth condition: the message size is unknown and
the end of the body is signaled by closing the connection (in which
case the Connection header MUST include the 'close' token).

---

From: Zhong Yu <[email-masked]>
Date: Tue, 27 Nov 2012 19:17:00 -0600

In RFC2616, there's a clear distinction between
  1. a message without an entity body
  2. a message with an entity body of 0 bytes.
We should not remove the distinction

Quotes from RFC2616
=====

(HTTP messages) consist of ... possibly a message-body.

The message-body (if any) of an HTTP message is used to carry the entity-body

The presence of a message-body in a request is signaled by the
inclusion of a Content-Length or Transfer-Encoding header field in the
request's message-headers.

messages MAY transfer an entity

An entity consists of entity-header fields and an entity-body,
although some *responses* will only include the entity-headers.

An entity-body is only present in a message when a message-body is present

====

We can conclude that

If a request does not have "Content-Length" or "Transfer-Encoding", it
does not contain a message body, it does not contain an entity body,
it does not contain an entity.

If a request has "Content-Length: 0", it contains a message body of 0
bytes, it contain an entity body of 0 bytes, it contain an entity.

This is true regardless of the request method.

Zhong Yu