Subject: p1: whitespace in request-target

From: Mark Nottingham <[email-masked]>
Date: Thu, 18 Apr 2013 10:49:10 +1000

p1 3.1.1 says:

> Unfortunately, some user agents fail to properly encode hypertext references that have embedded whitespace, sending the characters directly instead of properly encoding or excluding the disallowed characters. Recipients of an invalid request-line SHOULD respond with either a 400 (Bad Request) error or a 301 (Moved Permanently) redirect with the request-target properly encoded. Recipients SHOULD NOT attempt to autocorrect and then process the request without a redirect, since the invalid request-line might be deliberately crafted to bypass security filters along the request chain.

  http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-22#section-3.1.1

I note that the practice of correcting this is fairly widespread; e.g., in Squid, the default is to strip the whitespace, and IIRC has been for some time:

  http://www.squid-cache.org/Doc/config/uri_whitespace/

I think that the Squid documentation needs to be corrected, because the text in RFC2396 (and later in 3986) is about URIs in contexts like books, e-mail and so forth, not protocol elements:

  http://tools.ietf.org/html/rfc3986#appendix-C

My question is why this is a SHOULD / SHOULD NOT. We say that SHOULD-level requirements affect conformance unless there's a documented exception here:

  http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-22#section-2.5

... but these requirements don't mention any exceptions. Is the security risk here high enough to justify a MUST / MUST NOT? If not, they probably need to be downgraded to ought (or an exception needs to be highlighted).

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 18 Apr 2013 15:48:09 +1200

>    http://tools.ietf.org/html/rfc3986#appendix-C

The relevant portion there being:
  "

    For robustness, software that accepts user-typed URI should attempt
    to recognize and strip both delimiters and embedded whitespace.
"
Note that Squid *does* accept user-typed HTTP messages and the software traditionally causing whitespace issues
are usually the specialized clients sending URI through HTTP-compatible messages in the URL field (Outlook and Exchange with RCP, some shockwave ICY clients).

> ... but these requirements don't mention any exceptions. Is the security risk here high enough to justify a MUST / MUST NOT? If not, they probably need to be downgraded to ought (or an exception ne…

The biggest risk is software truncating portions of the URL, or (like 
Squid when there is a trailing SP on the line) determining that the 
HTTP/1 version label is part of the URL on a HTTP/0.9 syntax GET request 
- which results in any HTTP/1.x header features being ignored. The 
actual security worst-case risk of this undeterminable, but its not 
going to be good for the transaction at the best of times.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 18 Apr 2013 07:55:53 +0200

Hi,

On Thu, Apr 18, 2013 at 10:49:10AM +1000, Mark Nottingham wrote:
> p1 3.1.1 says:
> 
> > Unfortunately, some user agents fail to properly encode hypertext
> > references that have embedded whitespace, sending the characters directly
> > instead of properly encoding or excluding the disallowed characters.
> > Recipients of an invalid request-line SHOULD respond with either a 400 (Bad
> > Request) error or a 301 (Moved Permanently) redirect with the
> > request-target properly encoded. Recipients SHOULD NOT attempt to
> > autocorrect and then process the request without a redirect, since the
> > invalid request-line might be deliberately crafted to bypass security
> > filters along the request chain.
> 
> I note that the practice of correcting this is fairly widespread; e.g., in
> Squid, the default is to strip the whitespace, and IIRC has been for some
> time:
> 

Does Squid log something which helps seeing when it fixes this ?

> I think that the Squid documentation needs to be corrected, because the text
> in RFC2396 (and later in 3986) is about URIs in contexts like books, e-mail
> and so forth, not protocol elements:
> 
> My question is why this is a SHOULD / SHOULD NOT. We say that SHOULD-level
> requirements affect conformance unless there's a documented exception here:
> 
> ... but these requirements don't mention any exceptions. Is the security risk
> here high enough to justify a MUST / MUST NOT? If not, they probably need to
> be downgraded to ought (or an exception needs to be highlighted).

Well, FWIW, haproxy is strict on this and rejects requests which don't
exactly match the expected format, which means that embedded spaces are
rejected with a 400 badreq. At the same time when such a bad request
happens, the whole request is captured. I must say that all the ones
I have seen to date (which are extremely rare) were made by poorly
written attack scripts, or by stupid web scraping tools that resolve
URL-encoding in links found on web pages before doing the request.
These are the same which append '">' at the end of a URL because they
failed to properly parse the "<a href=..." tag.

So in my opinion, we could reasonably use a MUST here. Suggesting recipients
to fix this significantly increases the risk that they do it wrong and become
vulnerable to certain classes of attacks. And we're clearly not contributing
to cleaning the web by accepting such erroneous behaviours, especially if
browsers managed to get it right.

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 19 Apr 2013 21:48:38 +1000

Tracking this as:
  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/443

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 30 Apr 2013 13:18:43 +1000

So, I'm not hearing you say "don't make this a MUST" -- just noting that some broken software out there; correct?

On 18/04/2013, at 1:48 PM, Amos Jeffries <[email-masked]> wrote:

> The biggest risk is software truncating portions of the URL, or (like Squid when there is a trailing SP on the line) determining that the HTTP/1 version label is part of the URL on a HTTP/0.9 syntax GET request - which results in any HTTP/1.x header features being ignored. The actual security worst-case risk of this undeterminable, but its not going to be good for the transaction at the best of times.
> 
> Amos
>

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 30 Apr 2013 07:52:51 +0200

Amos' last sentence makes me understand "please don't make this a MUST" :

> The actual security worst-case risk of this undeterminable, but its
> not going to be good for the transaction at the best of times

And I also agree with him that the wording currently is ambiguous because
it can either be understood as "servers and intermediaries should do this
since they're accepting user-typed URIs" or as "clients should fix user-
typed requests before sending".

Thus in order to avoid any ambiguity, I would propose two sentences instead

>   to recognize and strip both delimiters and embedded whitespace.

Would become :

    Clients MUST NOT send user-typed delimiters and embedded whitespaces
    as-is in URIs, and SHOULD either encode them, strip them. Alternatively
    they MAY simply refuse to perform the request.

    Servers and intermediaries MUST NOT try to fix embedded spaces and
    delimiters in URIs, as doing so could lead to interoperability issues
    and make several components in the chain understand different things.
    When a request does not parse exactly as defined in the ABNF, an error
    400 (Bad Request) MUST be returned to the client.

Comments ?

Willy