Subject: [tcpm] PoC for draft-moncaster-tcpm-rcv-cheat-02

From: Stefanos Harhalakis <[email-masked]>
Date: Wed, 26 Mar 2008 00:29:33 +0200

Hello there,

While experimenting I've written a (small) python program that utilizes the 
TCP weakness that is addressed in draft-moncaster-tcpm-rcv-cheat-02 (which I 
discovered latter). Using it, I was able to make an (under my control) HTTP 
server generate a constant 5-minute (until I interrupted it) 900Mbps traffic 
over the Internet using just a DSL line. Theoretically it could also be done 
using a 64Kbps ISDN line, it is 100% reproduceable and it can be used on many 
web servers with fast internet connections.

I'm considering wether I should make this (small) tool public or not.

On the plus side: It can be used as a proof of concept and for easily testing 
future implementations of this draft (if it becomes an RFC)

On the minus side: Since it is very easy to use (no kernel patches etc), it 
can be easily abused by people that are not currently able to re-implement 
it.

So, I kindly ask for your advice:
Based on your experience, would it be of any use if I made this public?

Thank you in advance,
Harhalakis Stefanos
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Rob Sherwood <[email-masked]>
Date: Wed, 26 Mar 2008 00:25:15 -0400

> I'm considering wether I should make this (small) tool public or not.

As a data point, I have choosen not to publish the PoC code I created
for this attack ("TCP Receivers Can Cause Internet-Wide Congestion
Collapse" CCS 2005).  While I am a strong proponent of full disclosure,
I don't think releasing the code will speed the acceptance of the
proposed solution. 

- Rob
.

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 26 Mar 2008 14:38:53 +0000

Stefanos,

I'd tend to agree with ROb - publication is unlikely to get the 
solution out there any quicker.

If you're willing, you can always make the code available but require 
the downloader to contact you personally for a copy. Not ideal, but a 
decent compromise.

BOb

At 04:25 26/03/2008, Rob Sherwood wrote:
> > TCP weakness that is addressed in 
> draft-moncaster-tcpm-rcv-cheat-02 (which I

____________________________________________________________________________
Bob Briscoe, <[email-masked]>      Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Pekka Savola <[email-masked]>
Date: Wed, 26 Mar 2008 20:21:49 +0200 (EET)

> solution out there any quicker.

First, I suggest you or someone contact the most interesting vendors 
(some BSD and Linux communities) about this and potentially obtain a 
CVE number.  Fernando Gont might be willing to help with this process 
as he has been through it before.

However, on your main point I disagree; publication of PoC code is 
likely to trigger getting a CVE number, and that is likely to trigger 
some response especially in most security-aware (Linux, BSD) operating 
system vendors.

There is ample evidence that vendors are willing to react even if the 
IETF progress is slow or nonexistent (e.g. ICMP errors draft); there 
is also ample evidence that vendors are not interested in putting (and 
should not be expected to put) the IETF on the critical path of 
shipping a product (or fixing a flaw).

If the issue is not progressing in the IETF, I think it's exactly the 
right thing to do to release the PoC code; it would be nicer to 
contact vendors' security teams first and give them a heads-up e.g. 3 
or 6 months in advance though.

But if you believe this issue is progressing in the IETF, maybe 
publishing the PoC is premature.  If the progress is slow I'd suggest 
contacting vendors' security teams now.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Rob Sherwood <[email-masked]>
Date: Wed, 26 Mar 2008 15:33:38 -0400

> as he has been through it before.

When I first published my paper (2005), I brought the issue up with
CERT and they issued the problem [VU#102014]
(http://www.kb.cert.org/vuls/id/102014).  While not a CVE, I believe it
has the same effect (maybe others may disagree).  

> contacting vendors' security teams now.

I contact vendors in 2005, and the response was luke warm (as was the
response on this list).  My take on this issue is that people are
divided into three[1] camps:

a) people who do not believe this attack practical
b) people who believe the attack is practical, but not dangerous enough
	to motivate patching TCP implementations
c) people who believe this attack is practical and dangerous, but felt
	my randomly-skipped segments solution was not "good" for
	reasons of efficiency, correctness, etc.

In my mind, publishing the PoC could convince people in group (a) to
move to group (b).  If the attack were to become active in the wild,
then maybe people might move from (b) to (c).  But that still leaves
the bottleneck on people in group (c) to come up with a "good"
solution.  Bob Briscoe et al. have come up with a variant on my
solution, but I am not certain how well its been scrutinised or if it
has even been implemented.

So, in my mind, the best case is that publishing the PoC could get more
people looking at potential solutions to the attack, at the cost of
potentially causing widespread DoS.  Given that analysis, I chose not
to publish my code, but I will leave it to Stefanos to decide what to
do with his.

- Rob
.

[1] I guess there is a forth group (namely, me) that believed the attack
was practical and dangerous, and that the skipped segment solution was
good, but I guess we can discount that :-)
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: "Caitlin Bestler" <[email-masked]>
Date: Wed, 26 Mar 2008 15:41:12 -0400

> Rob Sherwood wrote

I'd rephrase that as:

b) people who believe the attack is practical, but that the problem is
an
   application layer problem that should be solved at the application
layer
 -- and especially not by transport layer solutions that would have a
    negative impact on honest connections.

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Wed, 26 Mar 2008 13:00:05 -0700

Ok the following a general comment/observation and just my opinion. 

There are a class of problems that CAN be solved in application layers,
but those solution doesn't scale ( all applications need to be fixed),
in such cases if we make minor tweaks in the transport layer with larger
benefits ( meaning a multitude of transport apps), and if the benefits
of doing so outweigh the drawbacks, then we should hoenstly pursue or
make an attempt to pursue it. In other words a rejection of a transport
proposal cannot be simply based on the fact that "you can possibly solve
this in application layer", wait a minute, I am also not advocating that
all applications problems need to be solved in transport layers.

Well, David Clark's quote on layering comes to mind as well.

S0.02,
-Anantha

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Caitlin Bestler
> Sent: Wednesday, March 26, 2008 12:41 PM
> To: Rob Sherwood; Pekka Savola
> Cc: [email-masked]
> Subject: Re: [tcpm] PoC for draft-moncaster-tcpm-rcv-cheat-02
> 
> > Rob Sherwood wrote
> > 
> b) people who believe the attack is practical, but that the 
> problem is an
>    application layer problem that should be solved at the 
> application layer

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 26 Mar 2008 13:05:27 -0700

Intentionally dropping segments at the sender
has an unacceptable negative impact on honest
connections. No wonder this never flew.

You can detect a possible attacker of this nature
just by looking for acks of segments you have
not yet sent. After all, the attacker is sending
acks for what he didn't receive, so he has no idea
how far into the future to ack.
Even if you look for future acks, they may be legit
ones from a previous connection or old ones from
the same connection if the sequence space has wrapped.

So I suggest as a defense, if you receive an ack of
the future, to halve the congestion window.

-- 
Jakob Heitz. x5475. 510-566-2901
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Rob Sherwood <[email-masked]>
Date: Wed, 26 Mar 2008 16:18:26 -0400

>     negative impact on honest connections.

I would call this a different camp all together, mostly because it's
one I least understand the rationale for.  Maybe you could reiterate
your points for me (off list, if you think this issue is dead)?

I feel like this view comes from a misunderstanding of the attack.  In
my view, the goal of the attack is not to saturate the first hop of a
single victim.  Instead, the attacker's goal is to fool many victims in
parallel into saturating their shared access and backbone links by the
*aggregate traffic*.  Thus, no one victim is sending abnormally fast,
but the aggregate level of traffic ( summed across victims) is
sufficient to cause DoS.  To paraphrase, this is an attack against the
network not an individual.  

So, when one views the attack from this standpoint, I don't see how
application level rate limits (which, IIRC, is what you were proposing)
would stop this attack: no one victim is sending at an excessive rate.

- Rob
.

---

From: David Malone <[email-masked]>
Date: Wed, 26 Mar 2008 20:20:45 +0000

> the future, to halve the congestion window.

In the demo at:

	http://www.hamilton.ie/gavinmc/drop_dupack_attack/

Gavin only acked packets that were sent - there is no need to ack
into the future.

	David.

---

From: Rob Sherwood <[email-masked]>
Date: Wed, 26 Mar 2008 16:25:01 -0400

> connections. No wonder this never flew.

My experiments have show this to be a <0.01% impact on perfomance and
no impact on correctness.  You might argue that my experiments do not
have a wide deployment and might depend on network conditions, but I
don't know why you say the solution is "unacceptable".

> the future, to halve the congestion window.

As I say in my paper, this is not a reasonable defense.

1) The attack does not necessarily send Acks ahead of what is sent and
	I outline a number of ways to avoid doing so.
2) If this is the detection condition, it opens a new DoS where a different
	type of attacker can inject out-of-window Acks into valid connections
	to create false possitives in this test.  Thus, if what you propose 
	were implemented, a third party could force the congestion window
	to be arbitarily small for a connection between two honest nodes,
	without guessing the sequence number.

- Rob
.

---

From: Stefanos Harhalakis <[email-masked]>
Date: Thu, 27 Mar 2008 02:39:50 +0200

Hello Caitlin, 

>     negative impact on honest connections.

I believe that people that find the above logical do not actually understand 
the issue. Let me explain:

First, it is impossible for the application layer to distinguish legitimate 
clients from this kind of misbehaving receivers.

Second, this has to do with exploiting TCP and not a group of applications.

Third, this affects the network and the end-points and not the applications. 
This means that it has to be solved either in the intermediate network 
(routers) or at the end-point operating systems. 

Forth, it is a security issue that (at least) should be mentioned in 
congestion control related RFCs in the security considerations section.

Since this only affects TCP and the network layer has (almost) nothing to do 
with congestion control, this becomes 100% a transport layer TCP issue. 

Perhaps we should just document this in an informational RFC and cooperate 
with or wait for vendors to solve it.
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Fernando Gont <[email-masked]>
Date: Wed, 27 Feb 2008 13:58:23 -0300

>Based on your experience, would it be of any use if I made this public?

This issue was disclosed by US-CERT. That is, vendors were contacted, 
and had the chance to do something about it.

It has a CVE name: 
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2005-3675 , and 
there's a US-CERT report about it: http://www.kb.cert.org/vuls/id/102014

According to US-CERT's report, there has not been much of a response 
from vendors.

I suggest you contact CERT and let them know that your planning to 
disclose your PoC. Set your own deadline as to until when you'd be 
willing to get a response on a way forward (i.e., are vendors going 
to be as unresponsive as they have been so far?)

Additionally, contact developers at FreeBSD, NetBSD, OpenBSD and 
Linux. I can provide a few names at each, if you want. But they 
usually have some form of security@ contact. I know quite a few 
people at these projects that do care about security issues, and 
would be willing to implement counter-measures if they make sense.

If the process with CERT doesn't go as expected, (and provided you 
have already talked with the open source projects and given them some 
time to do something about it), post your code to bugtraq and CC 
full-disclosure. Also include a pointer to Savage's paper and to 
Sherwood's paper, too. And also include a pointer to Moncaster's draft.

I could also put you in contact with the right people at UK CPNI 
(http://www.cpni.gov.uk). In my own experience, they have proved to 
be more responsive, and much nicer to work with.

Kind regards,

--
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Stefanos Harhalakis <[email-masked]>
Date: Thu, 27 Mar 2008 23:28:54 +0200

> and had the chance to do something about it.

Thank you for the information. It was very helpfull. After thinking it a lot, 
I decided to first disclose it in the form of a paper which should further 
alert vendors and security organizations (I'll contact them when it is ready) 
but it won't be used by script-kiddies reading the bugtraq. I believe that 
this is the best approach (tm) right now and that it won't do any harm.
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: John Kristoff <[email-masked]>
Date: Thu, 27 Mar 2008 17:11:47 -0500

On Thu, 27 Mar 2008 23:28:54 +0200

> this is the best approach (tm) right now and that it won't do any harm.

From an operational perspective my personal opinion is that this is
probably the best approach at this time.  Releasing code is likely
going to result in two things, pain for some and public notoriety
for you.  Sounds like your ego is already well intact.  Thats good. :-)

Perhaps you or a designated colleague would be willing to give a
lightning talk and expose the attack(s) to a forum such as USENIX,
RIPE or NANOG to help raise awareness?  A few well formed graphs
showing real-world results might get some influential operators and
implementors to put mitigation higher on the priority list?

John

---

From: "Caitlin Bestler" <[email-masked]>
Date: Thu, 27 Mar 2008 19:14:44 -0400

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Rob Sherwood
> Sent: Wednesday, March 26, 2008 1:25 PM
> To: Jakob Heitz
> Cc: [email-masked]

As a *test* described in an informational RFC there is no real 
problem with it. As a routine counter-measure it would have
severe problems.

On very high speed links the NIC and/or driver has very little
choice but to implement one or more strategies to reduce the number
of interactions with the TCP consumer. Delivering each TCP segment
as soon as it is deliverable is not a viable option.

These techniques could include interrupt coalescing, Large Receive
Offload, TCP offload and others. Large Receive Offload in particular
will be negatively impacted whenever the TCP sender does not faithfully
send entire  messages in order and as promptly as the TCP congestion
control algorithms expect. LRO seeks to reduce the number of distinct
notifications required to deliver a sequence of TCP Segments for the
same connection, but in a manner that minimizes latency delays.

When a gap indicates a lost packet the answer is simple, aggregate
what you have and deliver it. When a gap really means that the sender
is testing you then you still aggregate what you have and deliver it.
And then deliver the deliberately mis-ordered segment separately.

So the issue is not just the memory resources, but the number of
distinct inter-layer notifications required, the number of times
that a context switch between TCBs is required and the number of
times a user context is switched. These all have a negative impact
on not only memory usage, but memory bandwidth and cache efficiency.

Indeed, if a sender engaged in deliberate TCP segment misordering
on a regular basis it could be considered a Denial-of-service attack.

This is something that can very easily be solved at the application
layer, by simply requiring the application layer to periodically
echo an application layer cookie back to the sender. A trivial AJAX
script could handle this without a single change to the TCP layer.
Using timestamps would also probably work. And if the client was
unwilling to provide these echoes, just limit the bandwidth that
you will provide to them.

As for the "network" implications, this attack is a very poor 
candidate for a successful DoS attack because it requires the
attacker to saturate a link that actually leads to them. The
pattern of the attack would be easily detected, and once detected
the destination IP address is easily traceable.

Work in the IEEE's Data Center Bridging task group should lead to 
reduction or elimination of congestion drops within IP subnets.
That means that TCP handling optimizations in NICs, drivers or
TCP stacks that expect in-order delivery will be even more effective.

The NICs and Bridges supporting these techniques will be going through
a lot of extra work to provide largely lossless and in-order delivery
of TCP segments. Deliberately sending them in the wrong order borders
on being ungrateful.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 28 Mar 2008 01:25:10 +0000

Rob,

Just to be clear, our original aim was merely to put what you had 
done into IETF standards-speak because we believed you (and previous 
researchers on the same tack) had identified a serious vulnerability 
and a reasonable solution. However, on writing up the I-D, we quickly 
realised the TCPM w-g would likely have issues with some of the 
precise details of your approach, so we ended up inventing a slightly 
different but broadly similar approach.

It's been fairly closely scrutinised technically on the list. I think 
we have satisfied most people that a 'good' solution is possible that 
has no harmful side-effects - at least in theory.

You're right that we haven't simulated or implemented the deltas from 
your approach. This was originally a sideline activity - just a write 
up that turned into more than we had planned. Nonetheless, Toby 
Moncaster (the primary author) now has plans to simulate it and write 
up the outcome. I can't be more precise 'cos I'm responding on his 
behalf, as he's on his sick-bed at the mo. I guess if it became a WG 
item, he'd get simulations done sharpish.

Bob

At 19:33 26/03/2008, Rob Sherwood wrote:
>My take on this issue is that people are
>---snip---
>---snip---
>But that still leaves

____________________________________________________________________________
Notice: This contribution is the personal view of the author and does 
not necessarily reflect the technical nor commercial direction of BT plc.
____________________________________________________________________________
Bob Briscoe,                           Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Rob Sherwood <[email-masked]>
Date: Fri, 28 Mar 2008 10:59:26 -0400

> And then deliver the deliberately mis-ordered segment separately.

Again, I think we have missed each other's point here.  I think you are
over estimating the _frequency_ that the test need be applied.  How
often the test is applied is a local decision made by the sender, in a
trade off between efficiency and how pro-actively attackers are
detected.  My intuition as to why the test has such low over head is
that it is applied less frequently then the regular packet loss rate
(caused by congestion and the TCP saw tooth) so it does not increase
the apparent packet loss rate significantly.  Applied to your
high-speed connections, this means that the rate of notifications
between the layers would not increase significantly either.

Further, the parameters for how often the test is applied are tunable:
my experiments recommend skipping a segment once every 1-200 segments,
which, for for the machines I had available to me, seemed to be a
reasonable compromise between overhead and detection.  The higher rate
connections that you speak of may be better suited with different
parameters: skipping a segment every 1-2000 or something similar.

In short, I don't see how you can say that this solution is too
inefficient; one can reduce the overhead of the test arbitrarily ( at
the cost of reduced detection),

> on a regular basis it could be considered a Denial-of-service attack.

I believe that the security implications of this design are independent
of the test I've proposed here.

> Using timestamps would also probably work. 

I guess this is another argument that I don't understand: you're saying
it's better to changes all applications individually than solve this in
one single place?  I agree that adding a challenge-response to all
applications would solve the problem, but it assumes modifying the
entire world several times over: all applications * all clients * all
servers.  I just don't see how that is feasible - to say nothing of
having to do a per-application analysis of what challenge response is
best for the least amount of overhead.

> And if the client was

As I mention in my previous mail, using "each host limits the bandwidth
each connection gets", even as a fall back for compatibility, does not
stop the attack.  The attacker can simply target more victims to
generate the same aggregated amount of traffic.

> the destination IP address is easily traceable.

Assuming the source ISP was responsive, this might work.  But, I
contend that most operators looking at the traffic patterns would
assume that the attack was backwards, i.e., that it was the victims
targeting the attacker.  I don't believe that the actions that an
operator would be likely to take to stop the mistaken backwards attack
(blocking traffic from victim to attacker) would stop the actual
attack.  This attack works (with reduced efficiency) when the attacker
is totally blind with respect to the victims traffic.  But yes, IF the
source ISP cared, IF they detect the attack, and IF they do the right
thing by blocking the senders Acks, then the attack would stop for that
one attacker.  It just seems like a lot of "if"'s and a lot of
dependence on other people doing the right thing when it is your
network that is congested.

> of TCP segments. Deliberately sending them in the wrong order borders

At least from your description here, it would seem that they have DoS
implications independent of this test.  I will have to read up on these
issues, thank you for pointing them out.

- Rob
.