Subject: Distinguishing 0-byte request body in HTTP/2

From: Kazuho Oku <[email-masked]>
Date: Thu, 15 Sep 2016 06:17:16 +0900

Hi,

Is there any way for a H2 server to distinguish between a request
without a body and a request with 0-byte body?

In HTTP/1, the distinction has been possible by looking for a
content-length or a transfer-encoding header. And H1 applications have
been actually looking for the headers to see if a request is
accompanied by a body by checking the existence of these headers.

OTOH, HTTP/2 does not seem to provide a method to distinguish between the two.

A HTTP/2 client is allowed to send a request accompanied by a body
without using the content-length header. It is also allowed to send a
HEADERS frame with END_STREAM flag set in case the size of the body is
zero-byte, omitting the DATA frame.

In such case, a request with zero-byte body becomes indistinguishable
from a request without a body.

The fact becomes an issue when we need to transcode a HTTP/2 request
to a HTTP/1 request (e.g. when a H2 proxy transmits a request to an H1
server running upstream), because, some applications try to see if a
POST request is accompanied by a body by checking the existence of
content-length or transfer-encoding header, or to assert that a GET
request is _not_ accompanied by a body by checking the non-existence
of the headers.

As a mitigation, it is certainly possible for a H2 proxy transcoding
to H1 to use the method of the request to see if content-length or
transfer-encoding header should be set in such case. But my
understanding is that generally speaking in HTTP whether if a request
is accompanied by a body is orthogonal to which method is being used.

Could somebody clarify what I am missing, or provide a method I should
use to accommodate the issue?

-- 
Kazuho Oku

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 15 Sep 2016 09:21:19 +1000

The rules in <http://httpwg.org/specs/rfc7230.html#message.body.length> still apply:

 - Any response to a HEAD has no body
 - Any 1xx, 204 and 304 response has no body
 - A 2xx response to a CONNECT has no body (because it's no longer HTTP after the header fields)
 - Otherwise, the message has a body which might be 0-length.

We intentionally made the set of messages without a body unable to be extended; see <http://httpwg.org/specs/rfc7231.html#considerations.for.new.methods>:

"""
Since message parsing (Section 3.3 of [RFC7230]) needs to be independent of method semantics (aside from responses to HEAD), definitions of new methods cannot change the parsing algorithm or prohibit the presence of a message body on either the request or the response message. Definitions of new methods can specify that only a zero-length message body is allowed by requiring a Content-Length header field with a value of "0".
"""

and <http://httpwg.org/specs/rfc7231.html#considerations.for.new.status.codes>:

"""
To allow existing parsers to process the response message, new status codes cannot disallow a payload, although they can mandate a zero-length payload body.
"""

Cheers,

> On 15 Sep 2016, at 7:17 AM, Kazuho Oku <[email-masked]> wrote:
> 
> Hi,
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 15 Sep 2016 09:09:42 +0900

Hi,

Thank you very much for the clarification.

So to paraphrase, the general rule for handling request body is
defined in section 3.3.3 of RFC 7230 as:

   6.  If this is a request message and none of the above are true, then
       the message body length is zero (no message body is present).

which means that in HTTP, there is no distinction between a request
with zero-length body and a request _without_ a body.

That means it is completely up to the HTTP client to whether or not to
send `content-length: 0` for such requests, though each implementation
may decide to send or not, depending on interoperability issues that
might exist.

-- 
Kazuho Oku

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Sep 2016 09:00:00 +0200

Hi Kazuho,

> may decide to send or not, depending on interoperability issues that

In fact there exists a corner case, some servers return 411 in response
to a POST without a content-length. Thus I'd suggest that the client
always emits the content-length when it considers that semantically it
emits a body even if it's empty. Let's imagine a disk backup tool uploading
files over HTTP, using a few header fields to pass file name, permissions
and various meta-data. Some files may be empty, and regardless of this they
are sent. In this case it totally makes sense to emit "content-length: 0".

I'd be tempted to simplify this as "if you're sending a body even an empty
one, announce its size in content-length". Methods like POST and PUT expect
a message body so that should always be done.

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 15 Sep 2016 18:47:32 +1000

> On 15 Sep 2016, at 5:00 PM, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Kazuho,
> 
> to a POST without a content-length.

That's a bit different; the message is still delimited in the same way, but the server is saying that it wants to know the length up-front (for purposes other than figuring out where it ends; usually, to figure out if it's willing to accept it).

> Thus I'd suggest that the client

That's a bit too simple. Keep in mind that for HTTP/1.1,

"A sender MUST NOT send a Content-Length header field in any message that contains a Transfer-Encoding header field." <http://httpwg.org/specs/rfc7230.html#header.content-length>

Also, don't put C-L: 0 on 304 et al! :)

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Sep 2016 11:24:03 +0200

Hi Mark,

On Thu, Sep 15, 2016 at 06:47:32PM +1000, Mark Nottingham wrote:
> "A sender MUST NOT send a Content-Length header field in any message that
> contains a Transfer-Encoding header field."
> <http://httpwg.org/specs/rfc7230.html#header.content-length>

When I'm saying "c-l", I mean "or equivalent" or more precisely "announce the
message length one way or another". I totally agree with you on all this, but
I know that Kazuho is fine as well with language shortcuts like this :-)

Cheers,
Willy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 15 Sep 2016 13:07:29 -0700

> On Sep 14, 2016, at 5:09 PM, Kazuho Oku <[email-masked]> wrote:
> 
> Hi,
> 

Yes, there is no semantic distinction.

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 15 Sep 2016 13:13:01 -0700

> On Sep 15, 2016, at 12:00 AM, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Kazuho,
> 

No.  It is never a good idea to send extra information just in case you
might encounter a broken server.  It is better to send less information and
let people fix their own broken code.  Otherwise, the Internet becomes a
cesspool of poorly imagined cases that are far less likely to exist than
the keel-over-waiting-for-the-extra-TCP-packets cases that always exist.

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Sep 2016 00:06:37 +0200

Hi Roy,

> the keel-over-waiting-for-the-extra-TCP-packets cases that always exist.

But if c-l:0 is supposed to be exactly equivalent to no c-l, then what's
the purpose of status code 411 ? My understanding no c-l means there is
no body while c-l: 0 means the body is empty, both of which are totally
equivalent from a framing perspective, but not necessarily from a
semantics perspective.

Regards,
Willy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 15 Sep 2016 15:15:08 -0700

> On Sep 15, 2016, at 3:06 PM, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Roy,
> 

CL is present only for message framing and can be removed at any hop.
CL:0 versus no CL has no semantic distinction whatsoever, so any recipient
that chooses to interpret it as a distinction is inherently broken.

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 15 Sep 2016 15:21:57 -0700

> that chooses to interpret it as a distinction is inherently broken.

And the purpose of 411 is to say that Transfer-Encoding chunked isn't useful
because the resource requires a fixed length before it can be invoked (CGI).
[Actually, I think most servers attempt to read at least one buffer full
of body as chunked, to see if it happens to be a short body, and will internally
translate that to a fixed CL before continuing; otherwise, sending 411.]

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Sep 2016 08:00:03 +0200

> because the resource requires a fixed length before it can be invoked (CGI).

OK, thanks for this clarification Roy!

Willy

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 20 Sep 2016 10:07:16 +0900

Hi, Willy, Mark, Roy,

Thank you for your insights.

> a message body so that should always be done.

Although there is no distinction between a request with a zero-byte
body and a request without a body, I agree that it might be a good
idea to send `content-length: 0` for some methods (e.g. POST), whereas
for others it would be a good idea to _not_ send `content-length: 0`
(e.g. GET).

I now understand that suggestions for existing methods are given in
RFC 7231. For example, section 4.3.1 states:

   A payload within a GET request message has no defined semantics;
   sending a payload body on a GET request might cause some existing
   implementations to reject the request.

Similar statements exist for other methods.

And the recommendation made for methods that are defined outside of
HTTP/1.1 is defined in section 8.1.2 (thanks to Mark for pointing it
out), quote:

   Since message parsing (Section 3.3 of [RFC7230]) needs to be
   independent of method semantics (aside from responses to HEAD),
   definitions of new methods cannot change the parsing algorithm or
   prohibit the presence of a message body on either the request or the
   response message.  Definitions of new methods can specify that only a
   zero-length message body is allowed by requiring a Content-Length
   header field with a value of "0".

-- 
Kazuho Oku