Subject: Denial of Service using invalid Content-Length header

From: Jan Starke <[email-masked]>
Date: Tue, 14 Jun 2011 06:47:16 +0000

Hello,

during a pentest we oberved an interesting behaviour of some web
servers: When we send a POST-request to the web server using some
value in the Content-Length header >0 (say, n), but the length of the
content of the request is less than this value (e.g. n-k), then the
server waits for the remaining (k) bytes.

Because TCP is a very simple protocol, it is possible to send a
POST-request without maintaining a client state. Therefore, a client
can send a large number of requests in a very short time; the server
creates a connection for every request until there are no more
resources available. The number of simultanious "connections" depends
on the implementation, but it is limited, which is the point. An
attacker may continously try to send POST-requests to the server,
which result in a denial of service of the web server, as long as the
attacker process runs. This is some kind of HTTP-specific SYN-flood.

This kind of vulnerability exists in nearly every network protocol,
that dooes not stream; it works as follows:
attacker: "I have <n> byte for you"
server: "Okay, I'm listening..."
attacker: <sending <n-k> bytes>
server: <waiting for <k> bytes> until the connection times out

As attachment you receive a proof of concept exploit.

A possible mitigation would be to require SSL or TLS, as this forces
the attacker to maintain a state, so that the attacker needs as much
resources as the attacked server.

Another (not very mighty) mitigation could be to provide an
intermediate layer between TCP and HTTP to handle meta information.
Only after receiving the full content, the immediate layer would
forward the content to the HTTP layer. The immediate layer could use
the round trip time to calculate a timeout that is as short as
possible...

One could also restrict the number of TCP-Segments (and IP-packets)
per POST-requests.

Restricting the number of parallel connections per IP address does not
work, as many clients use proxy servers.

I have no idea how to really prevent this kind of attack, maybe
someone in this mailing-list knows how...

Regards, Jan

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 20 Jun 2011 16:21:55 +0000

In message <[email-masked]>, Jan Starke wri
tes:

>A possible mitigation would be to require SSL or TLS, [...]

Economically impossible at the bandwidths many sites run.

>intermediate layer between TCP and HTTP to handle meta information.

This is what FreeBSD's "accept-filters" do.  None of the presently
implemented filters catch this particular case though.  Not really
sure it is a feasible way to deal with POST bodies though.

>someone in this mailing-list knows how...

There is no way to prevent it, it is a direct consequence of the
protocols, at best you can mitigate it.

The best mitigation is to have high-level detection software that
says "Funny, we've seen a lot of those, lets just summarily
close all unauthenticated POST attempts until they get bored" and
similar.

The second best mitigation is to write your code to spend as few
resources as possible, until you can commit to the request.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 20 Jun 2011 18:38:13 +0200

> resources as possible, until you can commit to the request.

I would add that the *first* protection obviously is to have the
server correctly implement timeouts, because if it is sensible to
this attack, it's also sensible to simple client failure.

Quite honnestly, I see nothing new here, it been abused for at least a
decade by DDoSers and I'm not sure there are not that many servers which
are *that* sensible. In fact, if the slowloris attack was playing on
headers, it precisely is because playing on data us much less effective
with servers, but the principle is the same (ie: maintain a server state
for a long time).

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 20 Jun 2011 17:03:32 +0000

>this attack, it's also sensible to simple client failure.

There is no possible timeout value which will both serve slow clients
in bad connectivity (iPhone4 ?) and prevent DoS attacks.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 20 Jun 2011 23:19:11 +0200

> in bad connectivity (iPhone4 ?) and prevent DoS attacks.

Yes in practice you can because even with bad connectivity you're generally
interested by covering holes as large as 30-60 seconds, not much more. After
that, many other stateful components between the client and the server will
have cut the connection anyway. For instance, most haproxy setups use between
20 and 30 seconds and don't report any particular issue. Those using larger
values generally try to cover long application response times and make use
of the same timeout for all values to keep their configs simple.

And with a 30 seconds timeout, even if you're able to send 1000 connections
per second, which is quite not possible with a simple xDSL line, that's still
only 30k concurrent connections, which is quite small to medium for todays
gateways.

Clearly I'm not speaking about DDoSes here, botnets have virtually unlimited
bandwidth, and they don't care about doing anything smart such as trying to
fool your content-length. I'm just talking about the "easy DoS" that any joe
user might ignite from his own access. Slowloris typically was such one. A
variant can be made to work with POST requests by sending one byte at a time
before the idle timeout strikes, unless the server supports a total transfer
timeout too.

Cheers,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 20 Jun 2011 21:41:10 +0000

>interested by covering holes as large as 30-60 seconds, 

Well your sever may not crash, but it does not serve legitimate
traffic either.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 20 Jun 2011 23:56:03 +0200

> Well your sever may not crash, but it does not serve legitimate

I'm sorry, I don't see your point. Why are you saying that the server does
not serve legitimate traffic ? It will only break the dead connection but
still serve all other ones well, that's the point of timeouts.

Also that's why some protocols with very long sessions implement an
application-level keep-alive (eg: SSH). That way it's possible to have
reasonable timeouts (eg. twice the keep-alive interval) without keeping
dead connections forever.

Willy

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Jun 2011 10:05:49 +1200

I think the inference was that some servers will struggle to provide 
service to others when they maintain 30.000 idle connections.

But in the end, I don't see any real difference between sending the POST 
or even just sending nothing.

The server will have some sort of timeout.  It might be a different 
timeout used for waiting for anything vs waiting for a completion of a 
POST message body.

but I think the timeout in general will be long enough for many 
connections to have been initiated.

Things that mitigate that are features like connection rate limiting 
(per source IP), per-IP overall concurrent connection limits etc.

Adrien

On 21/06/2011 9:56 a.m., Willy Tarreau wrote:
>> In message<[email-masked]>, Willy Tarreau writes:

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 beta out now - http://www.wingate.com/getlatest/

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 21 Jun 2011 00:18:54 +0200

> service to others when they maintain 30.000 idle connections.

This is less and less common with todays hardware. Having been involved
10 years ago in designing an infrastructure that was able to support 10k
concurrent users with Apache 1.3 as the first gateway, I remember how hard
it was to support even a few thousands of concurrent connections on a
server with 1-2 gigs of RAM at this time. Now common sizing for medium-
sized servers are 32, 48 or even 64 gigs, and I know quite a bunch of
people working in DCs who are used to buy that as the default hardware.
Even with Apache 1.3, this allows you to reach such high numbers with
some careful tuning.

> or even just sending nothing.

Agreed. It's similar to when the server is waiting for the end of
headers for starting processing.

> timeout used for waiting for anything vs waiting for a completion of a 

Yes that clearly is common and mandatory anyway. You can't rely on the
OS only to kill the connections for you, as most default timeouts are
much too large.

> connections to have been initiated.

Indeed, which is why I was explaining the change that was made possible
with RAM sizing nowadays.

> (per source IP), per-IP overall concurrent connection limits etc.

I 100% agree. That's the most efficient solution but not all servers are
able to do that. Still that's what the antiloris module was doing :-)

Cheers,
Willy