Subject: HTTP request validation guidelines for implementers

From: João Penteado <[email-masked]>
Date: Thu, 08 Jul 2021 18:18:23 +0000

Greetings everyone,

This is my first time posting on a IETF mailing list, so I apologize in advance
if this isn't the proper channel for this discussion.

Input validation is a critical security component of most information systems,
and this is especially true for public services available over a untrusted
network such as the Internet.

In the context of HTTP requests, there're several opportunities for early
request validation and many systems employ an dedicated initial validation layer
either internally, as "request middleware", or externally in the form of layer 7
firewalls, AKA Web Application Firewalls (WAFs), or reverse proxies.

It is usually in both the server's and the client's best interest that an
invalid request be rejected as soon as possible and that meaningful error
information be sent back to the client. For this very purpose, RFC7231 defines a
whole class of HTTP error status code (4XX).

However, when implementing an HTTP server, there is no standardization or
consolidated guideline that I could find that address these two points:

1. Which components of the request MUST and SHOULD be validated.
2. In which order SHOULD these components be validated.

There're various flowcharts[1][2] available online that propose different
approaches of which components of the request should be validated first and
which HTTP status should have a "higher priority" when a request has multiple
errors. For instance, if the both Body and the URI exceed the maximum size
allowed by the server and client also exceeds its agreeded rate limit, should
the response code be 413 (Payload Too Large), 414 (Request-URI Too Long) or 429
(Too Many Requests).

As I mentioned earlier, currently, as it is in the server's best interest to
reject an invalid request as soon as possible, the "higher priority" status code
usually is one whose error is found earlier in the validation chain
implementation.

Should a standard or best practice recommendation for the points above be
adopted, clients that receive a client error response code will be able to tell
which checks were successful and servers will have a clear recommendation to
validate certain aspects of the requests they receive, which is still
unfortunately overlooked by many in the wild. It would also reduce inconsistent
behavior among different implementations of a same service that may return
different error codes for the same given request.

As someone who is currently implementing such a validator, I would like to start
this discussion proposing that the order of such an evaluation chain takes into
consideration the following factors, in priority order:

1. The computational complexity of the validation check.
2. The specificity of the returned status code (i.e. checks that when failed
return a generic "400 Bad Request" should be latter in the validation chain).
4. The order in which the (sub)component appears in the request (start-line,
headers and body). Null, size, length and boundary range checks should probably
come first.

The same factors should also apply when evaluating subcomponents of the request,
such as individual headers or body parts.

Finally, additional security concerns, like the classic 404 vs 403 unintentional
information disclosure dilemma, should also probably be addressed in such a
document.

I'd love to get the input of this working group on the relevance and topics I
raised above.

Best regards,

João Penteado

External links

[1] https://www.loggly.com/blog/http-status-code-diagram/
[2] https://www.codetinkerer.com/2015/12/04/choosing-an-http-status-code.html

---

From: Rafal Pietrak <[email-masked]>
Date: Fri, 9 Jul 2021 09:18:23 +0200

Hi,

I'm also very-very new to the list, and on one occasion I was kindly
reminded, that my "wisdom" is pointless to share.

Despite that, IMHO:
1. form validation belongs to an application level, not HTTP level. A
"form" is a payload of HTTP.
2. HTTP error codes 4xx are exactly that - HTTP level error codes.
Mixing design levels is not a good thing.
3. within the form, the decision of what is valid and what is not is
very application dependent. In particular, fields that go into the SQL
RDBMS will be validated against cooked SQL syntax. But should an
application use another database with different query syntax,
form-fields may require quite different validation.
4. then, an application level programmer may choose to gather all
possible inputs (for attack vector detection) or he/she may choose to
file just those "meaningful". So recommending what SHOULD or what MUST
be validated can be hard to determine for general purpose.

Then again. Should you have a clear view of that should be defined as
standard SHOULD/MUST regarding field validation, you may put that in
writing as "draft-proposal", for everybody to see how such
recommendations could fit their work.

With best regards,

-R

-- 
Rafał Pietrak

---

From: Julian Reschke <[email-masked]>
Date: Fri, 9 Jul 2021 12:42:55 +0200

Am 08.07.2021 um 20:18 schrieb João Penteado:
> ...

I agree that (1) is an interesting question, and in a perfect world the
spec will state that. If it does not, we may want to improve it.

With respect to (2) I'm very sceptical: ordering of validation steps
often depends on implementation details. Why does it matter in practice?
(I understand that it would help in comparing validators, but besides
that???)

Best regards, Julian

---

From: João Penteado <[email-masked]>
Date: Fri, 09 Jul 2021 17:51:45 +0000

>     recommendations could fit their work.

I mostly agree with you in regards to form validation, but I am not reffering
to form key and value validation for the most part here. I am talking about
validating HTTP request (sub)components like:

0. Connection parameters
- Remote address
- TLS version, cipher suite, etc.

1. Start-line
- Method
- Path
- Protocol major and minor version

2. Headers
- Header key
- Header value

3. Body

4. Trailers
- Trailer key
- Trailer value

For each request (sub)component you can perform _at least_ the following
validation checks:
- Nullity checks: (in)existence of certain request (sub)components
- Numeric comparison checks: equality and range checks of request
(sub)components count and length/size.
- String comparison checks: equality, prefix, suffix, substring, character
boundary, regex, etc.
- Rate-limiting checks (429)

For instance, today a well configured web server may check:
- TLS settings (i.e. ensure you're using TLS 1.2)
- Request rate of your IP and/or the current method+path
- Request URI length
- Payload size
- If the current method is supported at all in any path of the service
(before request routing).
- If the path exists
- If the Host header is unique and matches the host itself
- If you're using trailers which the sever does not support
- If you're authenticated and authorized for this action
- ...

Today, the problem is that there's no clear recommendation of which validation
checks MUST/SHOULD be made on which request (sub)components and in what order
those checks SHOULD be made (i.e. if both your body size and request URI length
are larger than what the server is willing to process, which one of these should
the server check for first and return the appropriate HTTP error status code).

> (I understand that it would help in comparing validators, but besides

There're two main benefits of recommending a validation step order in the spec:

1. With the appropriate research/tests made with each type of check, we could
determine the optimal check order performance and security-wise, making the
implementer's job much easier.

2. If the most servers out there adopt the same validation order, clients will
gain additional information unavailable before. If, for instance, every server
checks URI length before checking payload size, and I get a "413 Request Entity
Too Large" error, I would know for sure that my URI length is fine and all the
previous checks passed successfully.

Best regards,

João Penteado

---

From: Julian Reschke <[email-masked]>
Date: Fri, 9 Jul 2021 20:45:29 +0200

> previous checks passed successfully.

You lost me here.

If a client sends both a too large URI *and* a too large request body,
why does it matter in practice which one is reported first? At the end
of the day, to fix the issue, both problems need to be resolved, no?

Best regards, Julian

---

From: João Penteado <[email-masked]>
Date: Fri, 09 Jul 2021 20:51:07 +0000

On Friday, July 9th, 2021 at 15:45, Julian Reschke <[email-masked]> wrote:

> If a client sends both a too large URI and a too large request body,
>

You're correct, in pratice, the validation order doesn't matter as much as
what is being validated, as the client would still have to address all issues
in order to have its request accepted. Which is why I believe that reason no. 1
is a more relevant argument for having a well definied validation order.

Despite this, with a well defined validation order additional information would
be conveyed to the client, given HTTP's limitation of only returning one error
code at a time, which might be helpful in debugging.

If there's no consensus on the need of establishing such a validation check
order, what we could do instead is focus first on establishing what SHOULD and
what MUST be validated and then suggest on the spec some considerations
implementers may want observe regarding validation order.

Best regards,

João Penteado

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 11 Jul 2021 00:35:35 +0200

> is a more relevant argument for having a well definied validation order.

It's difficult to impose an order, as it will really depend on the
implementation. Parsing errors will almost always trigger as soon as
the rest of the message is not parsable anymore. But even then, you
could find implementations that stop on the first bad character in a
header field name while others will first tokenize in lines and only
then split each line into name+value.

Then you can have significant differences in ordering between the layer
of evaluation. Message delimitation is critical, but assuming that
components are properly formed, some implementations will have
opportunities to validate some fields on the fly (e.g. content-length,
transfer-encoding while others will rather postpone this to after the
whole header block is parsed.

The URI length or header length limitation issues will be even less
enforceable because they do not indicate something invalid in the message
but that one element in the chain couldn't cope with them. So this can be
triggered by the 3rd gateway in the chain after all other elements were
properly validated, just like they can be triggered by the very first
gateway before any other element is validated.

In any case you should consider that an error report doesn't imply that
once fixed everything else will work, but that *at least* this one needs
to be addressed. Thus the order doesn't really matter in the end.

Hoping this helps,
Willy