Subject: Line-folding in HTTP/2

From: Mike Bishop <[email-masked]>
Date: Sat, 22 Aug 2015 19:44:21 +0000

We've encountered a site (whom we shall leave nameless, unless they choose otherwise) that included a 0x0A20202020202020 sequence inside an HPACK value -carriage return followed by several spaces, which is to say line-folding.  Edge rejects that (as will IIS if clients send it), because that's an HTTP/1.1 legacy artifact.  RFC 7540 just references RFC 7230 section 3.2 for header values, which in turn says in section 3.2.4:

   Historically, HTTP header field values could be extended over

   multiple lines by preceding each extra line with at least one space

   or horizontal tab (obs-fold).  This specification deprecates such

   line folding except within the message/http media type

   (Section 8.3.1).  A sender MUST NOT generate a message that includes

   line folding (i.e., that has any field-value that contains a match to

   the obs-fold rule) unless the message is intended for packaging

   within the message/http media type.

While there's no requirement for HTTP/1.1 clients to reject this pattern (for back-compat), it seems like an HTTP/2 implementation might want to hold its peers to that MUST NOT.  Osama has additionally pointed out that being tolerant of this could make HTTP/1.1 to HTTP/2 conversion "wreak havoc and bugs."  In an offline thread with a few others, the general feeling seems to be that we should all change to reject this as broken header framing.

Does anyone have objections and think this should be tolerated by an h2 parser?  If not, I don't know that this qualifies as an erratum, but it would be nice to officially preserve this somewhere.

---

From: Robert Collins <[email-masked]>
Date: Sun, 23 Aug 2015 08:11:54 +1200

On 23 Aug 2015 7:49 am, "Mike Bishop" <[email-masked]> wrote:
... with a few others, the general feeling seems to be that we should all
change to reject this as broken header framing.
>
>
>
> Does anyone have objections and think this should be tolerated by an h2
parser?  If not, I don’t know that this qualifies as an erratum, but it
would be nice to officially preserve this somewhere.

IMO reject is entirely appropriate.

-rob

---

From: Cory Benfield <[email-masked]>
Date: Mon, 24 Aug 2015 09:24:59 +0100

On 22 August 2015 at 20:44, Mike Bishop <[email-masked]> wrote:
> While there’s no requirement for HTTP/1.1 clients to reject this pattern
> (for back-compat), it seems like an HTTP/2 implementation might want to hold
> its peers to that MUST NOT.  Osama has additionally pointed out that being
> tolerant of this could make HTTP/1.1 to HTTP/2 conversion “wreak havoc and
> bugs.”  In an offline thread with a few others, the general feeling seems to
> be that we should all change to reject this as broken header framing.

Presumably the rule would be that any header containing an octet
string that matches the regex '\r?\n[ \t]' is treated as invalid
(PROTOCOL_ERROR). That's reasonable enough.

What is the proposed rejection? RST_STREAM for the given stream?
GOAWAY? Should we consider adding a new error code to handle this case
for ease of debugging? Are intermediaries expected to also reject the
stream, or should they simply strip the whitespace from the folded
line?

I'm not opposed to this in principle, but I'd like to see it fleshed out a bit.

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 25 Aug 2015 03:56:04 +1200

On 24/08/2015 8:24 p.m., Cory Benfield wrote:
> On 22 August 2015 at 20:44, Mike Bishop wrote:

If we are just mandating the reject option from the list mentioned in
RFC 7230 then the requirement is status code 400. Since its likely
coming from a 1.1 client via a broken gateway that is still reasonable.

If an explanation is desired the only choice is a canned 400-status
response with payload explaining. Simply closing the stream with that as
if it were a normal h2 completion. The error and problems are all in the
message layers outside the h2 itself anyway.

RST_STREAM vs GOAWAY is purely an HTTP/2 optimization for this error in
teh case where no explanation is desired. Selection of which is based on
criteria unrelated to the error itself. Which one gets sent (if either)
should follow the same choice made on any other h2 error.

If a 1.1 gateway is actually involved it should be mapping
PROTOCOL_ERROR on either of RST_STREAM or GOAWAY to a 400 status anyway.
So it does not matter really - all three options result in roughly the
same thing. Senders choice as to what side effects they want to see
happen, which comes back to that RST/GOAWAY selection algorithm being
generic.

>From a proxy/gateway perspective it would be more beneficial if servers
used RST_STREAM or the canned response. So only the one stream was
affected. But GOAWAY is also good - after all it is the h2 proxies or
gateways job to filter this out first.

Amos

---

From: "Kulkarni, Saurabh" <[email-masked]>
Date: Mon, 24 Aug 2015 21:47:36 +0000

We (i.e. Akamai) are leaning towards coalescing the obs-fold into one more spaces for http/1.1 to http/2 translation as recommended by the old http/1.1 spec (I was told that this was a recommendation for http/1.0 spec too). This would avoid having folded headers as part of http/2 headers and still be backward compatible with old http/1.1. I do agree with Mike Bishop that h2 parsers need to basically be consistent in rejecting the obs-fold headers.

- Saurabh Kulkarni

From: Amos Jeffries <[email-masked]<mailto:[email-masked]>>
Date: Monday, August 24, 2015 at 8:56 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: Line-folding in HTTP/2
Resent-From: <[email-masked]<mailto:[email-masked]>>
Resent-Date: Monday, August 24, 2015 at 8:57 AM

While there's no requirement for HTTP/1.1 clients to reject this pattern
tolerant of this could make HTTP/1.1 to HTTP/2 conversion "wreak havoc and
bugs."  In an offline thread with a few others, the general feeling seems to

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Aug 2015 10:06:54 +1000

> On 23 Aug 2015, at 5:44 am, Mike Bishop <[email-masked]> wrote:
> 
> We’ve encountered a site (whom we shall leave nameless, unless they choose otherwise) that included a 0x0A20202020202020 sequence inside an HPACK value –carriage return followed by several spaces, which is to say line-folding.  Edge rejects that (as will IIS if clients send it), because that’s an HTTP/1.1 legacy artifact.  RFC 7540 just references RFC 7230 section 3.2 for header values, which in turn says in section 3.2.4:
> While there’s no requirement for HTTP/1.1 clients to reject this pattern (for back-compat), it seems like an HTTP/2 implementation might want to hold its peers to that MUST NOT.  Osama has additionally pointed out that being tolerant of this could make HTTP/1.1 to HTTP/2 conversion “wreak havoc and bugs.”  In an offline thread with a few others, the general feeling seems to be that we should all change to reject this as broken header framing.
>  
> Does anyone have objections and think this should be tolerated by an h2 parser?  If not, I don’t know that this qualifies as an erratum, but it would be nice to officially preserve this somewhere.

Reading a bit further down:

"""
A server that receives an obs-fold in a request message that is not within a message/http container MUST either reject the message by sending a 400 (Bad Request), preferably with a representation explaining that obsolete line folding is unacceptable, or replace each received obs-fold with one or more SP octets prior to interpreting the field value or forwarding the message downstream.

A proxy or gateway that receives an obs-fold in a response message that is not within a message/http container MUST either discard the message and replace it with a 502 (Bad Gateway) response, preferably with a representation explaining that unacceptable line folding was received, or replace each received obs-fold with one or more SP octets prior to interpreting the field value or forwarding the message downstream.

A user agent that receives an obs-fold in a response message that is not within a message/http container MUST replace each received obs-fold with one or more SP octets prior to interpreting the field value.
"""

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Mike Bishop <[email-masked]>
Date: Tue, 25 Aug 2015 01:04:10 +0000

So the argument would be that clients should be more permissive and un-fold based on that guidance?

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Monday, August 24, 2015 5:07 PM
To: Mike Bishop <[email-masked]>
Cc: [email-masked]
Subject: Re: Line-folding in HTTP/2

"""

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 25 Aug 2015 11:09:46 +1000

I think so. We deprecated folding, but couldn't require implementations to reject it, because (as you've seen) it still happens in the wild.

A later version of HTTP might be able to introduce a stronger requirement about rejecting it. H2 isn't that version, though.

Cheers,

> On 25 Aug 2015, at 11:04 am, Mike Bishop <[email-masked]> wrote:
>

---

From: Mike Bishop <[email-masked]>
Date: Tue, 25 Aug 2015 20:50:40 +0000

Implementations might not be required to reject it, but I'd argue that implementations should be permitted to reject it in HTTP/2.  No one is hand-typing a request with HPACK.  Stripping it out should be a normal part of preparing to HPACK-compress headers before you send them -- if not required by the standard, at least de facto in all the major implementations.  That goes back to the discussion of seeing the multiplexing layer as a 1.1/2 intermediary.

Servers and intermediaries are already allowed to be intolerant in the HTTP/1.1 spec -- I think HTTP/2 is the right time for user agents to also choose to reject this behavior.  Maybe it's not time to mandate it in a spec, or rather maybe it's too late to get it into the HTTP/2 spec, but it's a good time to further extinguish this through market action so we can fully remove it from the spec next time around.

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Monday, August 24, 2015 6:10 PM
To: Mike Bishop <[email-masked]>
Cc: [email-masked]
Subject: Re: Line-folding in HTTP/2

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 26 Aug 2015 09:41:26 +1000

Indeed, I forgot that we already covered this:

<http://httpwg.github.io/specs/rfc7540.html#rfc.section.10.3>

"""
Similarly, HTTP/2 allows header field values that are not valid. While most of the values that can be encoded will not alter header field parsing, carriage return (CR, ASCII 0xd), line feed (LF, ASCII 0xa), and the zero character (NUL, ASCII 0x0) might be exploited by an attacker if they are translated verbatim. Any request or response that contains a character not permitted in a header field value MUST be treated as malformed (Section 8.1.2.6). Valid characters are defined by the field-content ABNF rule in Section 3.2 of [RFC7230].
"""

Note that it references field-content, whose ABNF doesn't allow CR or LF.

> On 26 Aug 2015, at 6:50 am, Mike Bishop <[email-masked]> wrote:
>

---

From: Mike Bishop <[email-masked]>
Date: Tue, 25 Aug 2015 23:43:51 +0000

There we go -- thanks for finding that!

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Tuesday, August 25, 2015 4:41 PM
To: Mike Bishop <[email-masked]>
Cc: [email-masked]
Subject: Re: Line-folding in HTTP/2

"""

---

From: "Jain, Shakesh" <[email-masked]>
Date: Wed, 26 Aug 2015 00:27:35 +0000

> Note that it references field-content, whose ABNF doesn't allow CR or LF.

I agree with the intent of this part of the HTTP/2 spec but the problem we
are facing is that the spec does not clearly say how to *safely* translate
requests coming from a sender which was potentially written before RFC
7230 came into being. And for us (Akamai) that is really important because
any error response on HTTP/2 to a request that works fine on HTTP/1 posses
risks for customer traffic.

I think if we were to add that obs-fold MUST be replaced by a space when
translating from HTTP/1 to HTTP/2 would make it much more safer for us.
For obvious reasons we will anyways do it but just thought of asking is
such a correction is possible and if other people might be interested in
this too.

Shakesh

On 8/25/15, 4:43 PM, "Mike Bishop" <[email-masked]> wrote:

>Similarly, HTTP/2 allows header field values that are not valid. While
>most of the values that can be encoded will not alter header field
>parsing, carriage return (CR, ASCII 0xd), line feed (LF, ASCII 0xa), and
>the zero character (NUL, ASCII 0x0) might be exploited by an attacker if
>they are translated verbatim. Any request or response that contains a
>character not permitted in a header field value MUST be treated as
>malformed (Section 8.1.2.6). Valid characters are defined by the
>field-content ABNF rule in Section 3.2 of [RFC7230].
>"""
>
>> Implementations might not be required to reject it, but I'd argue that
>>implementations should be permitted to reject it in HTTP/2.  No one is
>>hand-typing a request with HPACK.  Stripping it out should be a normal
>>part of preparing to HPACK-compress headers before you send them -- if
>>not required by the standard, at least de facto in all the major
>>implementations.  That goes back to the discussion of seeing the
>>multiplexing layer as a 1.1/2 intermediary.
>> 
>> Servers and intermediaries are already allowed to be intolerant in the
>>HTTP/1.1 spec -- I think HTTP/2 is the right time for user agents to
>>also choose to reject this behavior.  Maybe it's not time to mandate it
>>in a spec, or rather maybe it's too late to get it into the HTTP/2 spec,
>>but it's a good time to further extinguish this through market action so
>>we can fully remove it from the spec next time around.
>> 
>> I think so. We deprecated folding, but couldn't require implementations
>>to reject it, because (as you've seen) it still happens in the wild.
>> 
>> A later version of HTTP might be able to introduce a stronger
>>requirement about rejecting it. H2 isn't that version, though.
>> 
>> Cheers,
>> 
>> 
>> 
>>>> We¹ve encountered a site (whom we shall leave nameless, unless they
>>>>choose otherwise) that included a 0x0A20202020202020 sequence inside
>>>>an HPACK value carriage return followed by several spaces, which is
>>>>to say line-folding.  Edge rejects that (as will IIS if clients send
>>>>it), because that¹s an HTTP/1.1 legacy artifact.  RFC 7540 just
>>>>references RFC 7230 section 3.2 for header values, which in turn says
>>>>in section 3.2.4:
>>>> While there¹s no requirement for HTTP/1.1 clients to reject this
>>>>pattern (for back-compat), it seems like an HTTP/2 implementation
>>>>might want to hold its peers to that MUST NOT.  Osama has additionally
>>>>pointed out that being tolerant of this could make HTTP/1.1 to HTTP/2
>>>>conversion ³wreak havoc and bugs.²  In an offline thread with a few
>>>>others, the general feeling seems to be that we should all change to
>>>>reject this as broken header framing.
>>>> 
>>>> Does anyone have objections and think this should be tolerated by an
>>>>h2 parser?  If not, I don¹t know that this qualifies as an erratum,
>>>>but it would be nice to officially preserve this somewhere.
>>> 
>>> A server that receives an obs-fold in a request message that is not
>>>within a message/http container MUST either reject the message by
>>>sending a 400 (Bad Request), preferably with a representation
>>>explaining that obsolete line folding is unacceptable, or replace each
>>>received obs-fold with one or more SP octets prior to interpreting the
>>>field value or forwarding the message downstream.
>>> 
>>> A proxy or gateway that receives an obs-fold in a response message
>>>that is not within a message/http container MUST either discard the
>>>message and replace it with a 502 (Bad Gateway) response, preferably
>>>with a representation explaining that unacceptable line folding was
>>>received, or replace each received obs-fold with one or more SP octets
>>>prior to interpreting the field value or forwarding the message
>>>downstream.
>>> 
>>> A user agent that receives an obs-fold in a response message that is
>>>not within a message/http container MUST replace each received obs-fold
>>>with one or more SP octets prior to interpreting the field value.
>>> """
>>> 
>>> Cheers,
>>> 
>>> 
>>> --

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 26 Aug 2015 10:32:50 +1000

> On 26 Aug 2015, at 10:27 am, Jain, Shakesh <[email-masked]> wrote:
> 

That's already specified in 7230.

> such a correction is possible and if other people might be interested in

Hopefully, all of this will be a little less convoluted once we update the 723x series to Full Standard.

Cheers,