Subject: H2 vs responses which should not carry any payload

From: Willy Tarreau <[email-masked]>
Date: Fri, 23 Oct 2020 06:54:26 +0200

Hi all,

we've recently faced a stupid case in haproxy with H2 and I realized that
I didn't find the good response in the spec.

What we've seen is that a client sends a HEAD request, which we forward
to the server. In response the server returns an error with some payload
(possibly a typical pre-made error page that doesn't care about the method),
and haproxy forwards both the HEADERS and DATA frames to the client, then
the client complains about protocol violations (I don't know yet what the
client is for now but I don't think it's important).

We were wondering where we ought to trim the payload in this case (and
for 204/304 as well), whether we ought to do this while reading the
response from the server or when sending the response do the client, and I
figured that nowhere at all in 7540 is mentioned anything about 204/304/HEAD
and that made me start to wonder if adjusting this at the H2 level is the
right solution, and if we ought to do anything about it or not (since
after all maybe everyone is right in this whole chain).

We all know that 204/304/HEAD are between transport and semantics because
for H1 these directly affect the parsing. From this perspective it would make
sense to consider that H2 should drop these. But if we consider semantics
only, it also makes sense to consider that H2 should let everything pass
through.

And even then, do all implementations accept, say, a HEADERS frame with
no ES flag in response to a HEAD request, followed by an empty DATA frame
carrying the ES flag ? At the semantic level it's OK since there's no
payload, but I can understand how some could find it annoying to wait
for DATA frames when no payload is expected (it's our case as well as
part of the possible fixes for this).

For those who want a bit more details, internally we're not directly
forwarding frames but transcoding these into a version-agnostic HTTP
representation that allows us to have either H1 or H2 on any side. This
internal version carries the semantics. If we decide that H2 has nothing
to do with this, we can decide to perform the filtering at the semantics
layer, while knowing that when it comes to H1 it still has to take these
special cases for the messaging anyway. It even makes me suspect that
the contraints are double, in that HEAD/204/304 ought to see no response
payload at the semantic layer, and that H1 is a special case in that it
cannot accept that either at the transport layer to respect the messaging
and that it's not a problem if it duplicates that check.

I'm interested in any opinion on this subject (or any pointers to anything
I could have missed).

Thanks!
Willy

---

From: Bence Béky <[email-masked]>
Date: Fri, 23 Oct 2020 07:45:03 -0400

Hi Willy,

> part of the possible fixes for this).

At some point during the GREASE experiments Chrome removed the
END_STREAM flag from every HEADERS frame, then sent a frame of
reserved type, then an empty DATA frame with END_STREAM, and I found
that not every server accepts this.  To my best knowledge WinHTTP
still fails to process such a request (ever without the reserved
frame) if the request method is GET.  My interpretation of RFC7540 is
that such a request is spec compliant, but in practice Chrome cannot
send them at this point.  (The GREASE experiment continues only on
requests with a request body.)

Bence

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 23 Oct 2020 13:55:39 +0200

Hi Bence,

> requests with a request body.)

Pretty useful indeed, thank you. I'm more concerned about the response
message however, since it's where we may have to perform some
adjustments to trim excess data. So this tends to confirm that we'll
have to deal with this at the H2 layer on the output path, to detect
that we're processing a HEADERS frame for a payload-less message, that
we need to set ES and that we ought to drop whatever else follows.

Cheers,
Willy

---

From: Bence Béky <[email-masked]>
Date: Fri, 23 Oct 2020 10:42:05 -0400

Hi Willy,

Sorry, my previous e-mail was incorrect.  Chrome did run into some
issues with buggy servers, and a fix to those servers has been
deployed since.  Separately, the same bug is thought to exist in
WinHTTP.  However, WinHTTP is a client-only stack, so this should only
affect responses, not requests.  Which is exactly what you are worried
about.

Thanks to Mike Bishop for correcting me in private.

Also cc'ing Eric Lawrence who might have more details to share.

Bence

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 23 Oct 2020 16:46:25 +0200

> affect responses, not requests.  Which is exactly what you are worried

OK, thanks to you and Mike for the details :-)

This comforts me in the idea that we really need to do our best to
consider a bit of semantics when serializing the response and emit
ES immediately.

Cheers,
Willy

---

From: Mike Bishop <[email-masked]>
Date: Fri, 23 Oct 2020 14:59:44 +0000

Yes, client-role, not client machines.  Thanks for the details, Eric!

-----Original Message-----
From: Eric Lawrence <[email-masked]> 
Sent: Friday, October 23, 2020 10:57 AM
To: Bence Béky <[email-masked]>; Willy Tarreau <[email-masked]>
Cc: HTTP Working Group <[email-masked]>; Mike Bishop <[email-masked]>
Subject: RE: H2 vs responses which should not carry any payload

The term "client-only stack" is potentially ambiguous. WinHTTP can and is sometimes used as a HTTP client running on servers.

In the case in question (broken Microsoft's web properties in spring), the problem was is that the http.sys HTTP Server code on our "Azure front door" servers would accept the client's H2 request, see the header block did not set the END_STREAM flag indicating that there was no body, and convert that request to a GET with a Transfer-Encoding: chunked. That request was then handed off to WinHTTP for forwarding to the backend servers, but WinHTTP would reject the request because it does not permit Transfer-Encoding to appear on GET or HEAD requests. 

The June "fix" was to have our "front door" servers simply skip adding the TE:Chunked to inbound GETs even if the H2 headers did not indicate there would be no body, allowing WinHTTP to forward the request without breaking.

WinHTTP continues to disallow TE:Chunked on GET/HEAD, and we believe that an issue very similar to the one described previously will also occur for servers running the IIS ARR load balancer software.

-Eric

-----Original Message-----
From: Bence Béky <[email-masked]>
Sent: Friday, October 23, 2020 9:42 AM
To: Willy Tarreau <[email-masked]>
Cc: HTTP Working Group <[email-masked]>; Mike Bishop <[email-masked]>; Eric Lawrence <[email-masked]>
Subject: Re: H2 vs responses which should not carry any payload

Hi Willy,

Sorry, my previous e-mail was incorrect.  Chrome did run into some issues with buggy servers, and a fix to those servers has been deployed since.  Separately, the same bug is thought to exist in WinHTTP.  However, WinHTTP is a client-only stack, so this should only affect responses, not requests.  Which is exactly what you are worried about.

Bence

On Fri, Oct 23, 2020 at 7:45 AM Bence Béky <[email-masked]> wrote:
> > And even then, do all implementations accept, say, a HEADERS frame 
> > with no ES flag in response to a HEAD request, followed by an empty 
> > DATA frame carrying the ES flag ? At the semantic level it's OK 
> > since there's no payload, but I can understand how some could find 
> > it annoying to wait for DATA frames when no payload is expected 
> > (it's our case as well as part of the possible fixes for this).
> >
> >
>

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 24 Oct 2020 13:07:19 +1100

My .02 -

204/304/1xx and head responses are defined by HTTP's core semantics to not have a body. 

HTTP/1 enforced those in the framing mechanism (if you didn't, you lost framing reliability and Bad Things happened).

HTTP/2 doesn't explicitly enforce them, which means that we're now in similar territory to "Can GET have a request body?" ... and we've all seen how much angst that causes.

Whether or not you should strip them, generate an error, or just pass them through depends on what you're doing. In general, HTTP wants implementations that don't have to "know" the whole protocol, to allow extensions to be deployed successfully, and to allow simpler implementations to succeed. So while HTTP/1 requires that implementations understand these situations to maintain framing, I think it's reasonable to not have that requirement (and we don't) in HTTP/2, because framing doesn't depend upon it.

That doesn't mean that putting a response body on a 304 or HEAD response is interoperable or a good thing to do; it just means that you're not obligated to refuse that message -- and we should be super crisp in http-core about what this means.

Cheers,

> On 23 Oct 2020, at 3:54 pm, Willy Tarreau <[email-masked]> wrote:
> 
> Hi all,
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 24 Oct 2020 08:16:22 +0200

Hi Mark,

On Sat, Oct 24, 2020 at 01:07:19PM +1100, Mark Nottingham wrote:
> Whether or not you should strip them, generate an error, or just pass them
> through depends on what you're doing. In general, HTTP wants implementations
> that don't have to "know" the whole protocol, to allow extensions to be
> deployed successfully, and to allow simpler implementations to succeed. So
> while HTTP/1 requires that implementations understand these situations to
> maintain framing, I think it's reasonable to not have that requirement (and
> we don't) in HTTP/2, because framing doesn't depend upon it.
> 
> That doesn't mean that putting a response body on a 304 or HEAD response is
> interoperable or a good thing to do; it just means that you're not obligated
> to refuse that message -- and we should be super crisp in http-core about
> what this means.

That's exactly the difficulty. We know that responses to HEAD containing
body were quite frequent due to error pages. Typically a hard-coded 403
or 500 returned directly based on the URI or due to a server malfunction
would not even check the method. But I think that it's getting tricky for
intermediaries, as they want to have the smallest possible impact on the
existing infrastructure, thus if possible not add errors on top of existing
errors. Also they have a greater responsibility here, for example when
translating H1 requests to H2: if an H2 response contains a body where
it's not supposed to, it must in no way be serialized to H1. So this is
what makes me think that the "cleanup" if any should be enforced at the
semantic layer, and that the messaging layers should only produce errors
if the unexpected situation would break the protocol.

In haproxy's case, this would mean always passing headers of 204/304/HEAD
responses to a client, and silently dropping any body for H2 or breaking
the server-side connection for H1 if a body appears there.

For H2 remains the case of trailers. My view on this (possibly skewed by
H1) is that trailers extend the body and must not be delivered when no
body is expected. In H1 there's no doubt about it since it was explicitly
stated that bodyless responses end after the "first empty line" and
"regardless of headers" which means that T-E is ignored and that there's
no way to append trailers. But in H2 they can be used to trivially deliver
an operational status and I can understand how some implementations would
prefer to always send their status in a trailer rather than in a header.
This could result in HEAD/204/304 to contain trailers. And I'm even
wondering if gRPC doesn't do that with their "grpc-status" trailer.
Thus I suspect that for now we'll stick to strict semantics by trimming
any received body but possibly appending trailers for H2. We may revisit
this later if that causes trouble :-/

Thanks!
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 25 Oct 2020 16:09:29 +0100

Willy,

You describe having a semantic layer that can have H1 or H2 either side.
So let's consider the case of H1->Poxy->H2.  In this case if the server
gives you a body to a 204/304 response or a HEAD request, then you have no
way of passing them back to the client  - as it breaks H1.  You must filter
them in that case in your semantic layer, thus it would make sense to me to
also filter them in the H2->Proxy>H2 case.

cheers

-- 
Greg Wilkins <[email-masked]> CTO http://webtide.com

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 25 Oct 2020 19:47:21 +0100

Hi Greg,

> also filter them in the H2->Proxy>H2 case.

Yes, I totally agree on this point and that's what we want to do. However
thinking about deleting the trailers in H2->H2 worries me because I suspect
some might already use them.

In any case, it's clear that having more technical capabilities in H2 has
opened a new can of worms regarding HTTP semantics that once used to be
limited by the transport layer!

Cheers,
Willy

---

From: Eric Lawrence <[email-masked]>
Date: Fri, 23 Oct 2020 14:58:32 +0000

-Eric

-----Original Message-----
From: Bence Béky <[email-masked]> 
Sent: Friday, October 23, 2020 9:42 AM
To: Willy Tarreau <[email-masked]>
Cc: HTTP Working Group <[email-masked]>; Mike Bishop <[email-masked]>; Eric Lawrence <[email-masked]>
Subject: Re: H2 vs responses which should not carry any payload

Hi Willy,

Bence