Subject: Restricting the HTTP method definition

From: James M Snell <[email-masked]>
Date: Tue, 20 Aug 2013 16:22:15 -0700

HTTPbis currently defines the request method as a "token" of unbounded-length.

Specifically:

   tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
    "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA
   token = 1*tchar
   method = token

This definition is overly broad and does not reflect real world use
[http://tools.ietf.org/html/draft-ietf-httpbis-method-registrations-12].

I propose that in HTTP/2 we tighten this definition up significantly
and place an upper bound on the length a request method ought to be:

  UPPER = %x41-5A
  method = UPPER *20( UPPER / "_" / "-" )

This is obviously a strictly limited subset of what's allowed by the
current definition. It limits the length of method names to no more
than 20 characters, requires that methods be all uppercase, requires
that methods always start with a letter and limits non-letter
characters to the dash and underscore. The rule would be that all
*newly registered* HTTP methods MUST conform to the new rule but
implementations MAY choose to support the old definition if necessary
for backwards compatibility.

It's a fairly minor issue, yes, but tightening this up ought to make
it easier for developers to create parsers that are both efficient
*and* compliant [http://www.chmod777self.com/2013/08/sigh.html]

- James

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 21 Aug 2013 10:14:19 +1000

James, 

The place to do that is HTTP/1, not HTTP/2.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: James M Snell <[email-masked]>
Date: Tue, 20 Aug 2013 17:18:51 -0700

Ok. Same note. Same mailing list. Same proposal. s/HTTP\/2/HTTP\/1/g.

Regardless of exactly which doc to do it in, it ought to be done.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 20 Aug 2013 17:37:49 -0700

I don't really care about this-- it solves none of my problems.

I typically scan for the EOL, checking length while doing this 8 or more
characters at a time, then, once the EOL is discovered, attempt to parse
the tokens within the first line.
I care about the total length of the first line, but even then only because
I care about the total size of the headers since I don't want to exceed
available memory and die.
I don't care if it is spent in the method, uri, headers, p3p, cookie, or
whatever. It is all the same crud to me-- it just has to fit in the
available space.
If the particular server cares about accepting a limited set of methods
(which is always true-- if you don't know what a verb is, you don't know
what to do with it...), then the request is rejected.

What problem does limiting the verb sizes solve? Are people doing weird
things with them? Are people abusing them in ways that would cause problems
that they cannot otherwise do (e.g. with a very long path)?
-=R

---

From: Roberto Peon <[email-masked]>
Date: Tue, 20 Aug 2013 17:41:30 -0700

I admit that I haven't thought hard about how this might potentially be
abused with others' frameworks/servers, but I imagine the pattern is nearly
always the same:

Servers/Clients which care about security/memory utilization will ensure
that this won't be abused.
Servers/Clients that trade reliability for simplicity will occasionally
suffer decreased reliability when they receive strange input.

In the end, I don't think there is anything we can do for the
servers/clients in the latter case. If we constrain the input so much that
it is always safe, it is likely we have created something which is not
generally useful.

-=R

On Tue, Aug 20, 2013 at 5:37 PM, Roberto Peon <[email-masked]> wrote:

> I care about the total length of the first line, but even then only
> because I care about the total size of the headers since I don't want to
> exceed available memory and die.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 20 Aug 2013 18:06:26 -0700

>  method = UPPER *20( UPPER / "_" / "-" )

No.  The specification already explains why that isn't desirable
for gateways, and it actually makes the security properties
of applications worse if they think they can rely on the ABNF
to limit received syntax lengths.

....Roy

---

From: James M Snell <[email-masked]>
Date: Tue, 20 Aug 2013 19:50:58 -0700

On Tue, Aug 20, 2013 at 6:06 PM, Roy T. Fielding <[email-masked]> wrote:
>>[snip]

Well, I'm familiar with the passage in the current spec that deals
with request line length, but not any discussion of request line
lengths and what's desirable for gateways. I may have missed it, have
a pointer?

Regardless, I was talking primarily about requirements for new methods
and defining a subset that is a closer reflection of what
implementations do. If it makes it better, however, I'm fine with
dropping the *20 part... just make it:

  UPPER = %x41-5A
  method = UPPER *( UPPER / "_" / "-" )

how's that?

- James

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 21 Aug 2013 22:19:45 +1200

On 21/08/2013 12:41 p.m., Roberto Peon wrote:
> I admit that I haven't thought hard about how this might potentially 
> be abused with others' frameworks/servers, but I imagine the pattern 
> is nearly always the same:
>
> Servers/Clients which care about security/memory utilization will 
> ensure that this won't be abused.
> Servers/Clients that trade reliability for simplicity will 
> occasionally suffer decreased reliability when they receive strange input.
>
> servers/clients in the latter case. If we constrain the input so much 
> that it is always safe, it is likely we have created something which 
> is not generally useful.

For middleware proxies which are validating the request-line it is one 
of the outstanding problems that we cannot determine explicitly that a 
extremely long string of ASCII alphanumerics is an invalid request. 
Often times we run out of available buffer space and have to reject the 
request even if it was valid for the application layer doing funky 
REST-ful "extensions", meanwhile the client delivered it in small chunks 
and held up a lot of network resources (slow-loris etc).

I've not seen this in practice in legitimate use outside of software 
fingerprinting tests.

At the very least the proposal allows a much faster rejection of the 
nastiness.
+1 for limiting the method length.

Amos

>
> -=R
>
>
> On Tue, Aug 20, 2013 at 5:37 PM, Roberto Peon <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     I typically scan for the EOL, checking length while doing this 8
>     or more characters at a time, then, once the EOL is discovered,
>     attempt to parse the tokens within the first line.
>     I care about the total length of the first line, but even then
>     only because I care about the total size of the headers since I
>     don't want to exceed available memory and die.
>     I don't care if it is spent in the method, uri, headers, p3p,
>     cookie, or whatever. It is all the same crud to me-- it just has
>     to fit in the available space.
>     If the particular server cares about accepting a limited set of
>     methods (which is always true-- if you don't know what a verb is,
>     you don't know what to do with it...), then the request is rejected.
>
>     What problem does limiting the verb sizes solve? Are people doing
>     weird things with them? Are people abusing them in ways that would
>     cause problems that they cannot otherwise do (e.g. with a very
>     long path)?
>     -=R
>
>
>
>     On Tue, Aug 20, 2013 at 5:18 PM, James M Snell <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>         On Tue, Aug 20, 2013 at 5:14 PM, Mark Nottingham
>         <[email-masked] <mailto:[email-masked]>> wrote:
>         > James,
>         >
>         > On 21/08/2013, at 9:22 AM, James M Snell <[email-masked]
>         <mailto:[email-masked]>> wrote:
>         >

---

From: Zhong Yu <[email-masked]>
Date: Wed, 21 Aug 2013 10:07:04 -0500

> *and* compliant [http://www.chmod777self.com/2013/08/sigh.html]

I don't see how the bug mentioned in the blog has anything to do with
what you are proposing. It looks like node.js is accepting any "GE<*>"
as "GET" where <*> can be any octet. Maybe node.js was assuming that
the request has been validated by an upstream parser?

Zhong Yu

---

From: James M Snell <[email-masked]>
Date: Wed, 21 Aug 2013 08:18:37 -0700

The connection with what I described in that blog post is that many
implementers have, apparently, take a number of liberal shortcuts in
terms of how they parse and accept http methods. Node's parser, for
instance, will immediately error out if it receives a method that
doesn't start with an alpha character (which is reasonable). It also
does not support the use of any non-alpha character other than "-"
(dash). Their implementation also chooses not to support extension
methods because the implementers feel there is no way of supporting an
arbitrary unknown set of tokens in a performant way... and truthfully,
because there is no upper bound on the length of the methods and
because there is a significantly large value space, they do have a
point.

Now, I did say right up front that this is a fairly minor issue, and
if it doesn't happen, so be it. But the restrictions I suggest ought
to make it at least some degree easier for implementations to
generically handle extension methods in a performant manner.. while at
the same time more accurately reflecting the reality of what many
implementers are already doing.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 21 Aug 2013 12:14:02 -0700

On Aug 21, 2013, at 8:18 AM, James M Snell wrote:

> terms of how they parse and accept http methods.

As is allowed by HTTP, though some of those "shortcuts" are
actually security holes in that implementation.

> Node's parser, for

No origin server is required to implement all methods.  That choice
has nothing to do with limitations in the protocol itself, which
would prevent ANY implementation from supporting longer methods
or gatewaying to distributed object systems.

> implementers are already doing.

Rubbish.  There are plenty of implementations to choose from that
do not have protocol limitations baked into stupid parsing code.
Any implementation can choose a maximum method length that it is
willing to support -- systems that depend on longer methods will
use different software.  This has zero effect on performance
because the request line is going to be read as one buffer that
has to be very long (to support the request-target) regardless.

The rationale for method is given in

http://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/23/p2-semantics.html#method.overview

It would be really helpful if folks here would understand that
typical browser traffic represents less than 1% of implementations.
Yes, it is the bulk of traffic, but that doesn't mean the rest
of HTTP is defined by it.  We optimize for the common case and
let the uncommon cases fend for themselves; we do not kill the
uncommon cases because we cannot know what good things would
be broken by doing so.

....Roy

---

From: Larry Masinter <[email-masked]>
Date: Sat, 24 Aug 2013 13:10:07 -0700

> use different software.

How can a proxy choose, if it wants to handle all expected (legal) traffic? Origin server, sure. Proxy?

Larry
--
http://larry.masinter.net

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 24 Aug 2013 21:47:42 -0700

> How can a proxy choose, if it wants to handle all expected (legal) traffic? Origin server, sure. Proxy?

The same way it does now ... it reads the request-line until
it finds the end of the method or it decides to send an error
instead.  This is not a problem in practice because the method
occurs before the request target (URI), which means the proxy
is reading the line into an 8kB buffer (typically). The method
is just another string, and is subject to the same length and
overflow protections as any other string parsed by a server.

....Roy