Subject: Re: [tcpm] Is this a problem?

From: MURALI BASHYAM <[email-masked]>
Date: Tue, 6 Nov 2007 10:37:24 -0800 (PST)

We want to limit the timer to the sender's persist state only, any other timeout overloads the semantics of the application,
and may also cause unnecessary timeouts for the application. In fact there are instances where the application may not
even have the connection state around to implement the timout correctly, like the tail of a long HTTP response. After sending the tail of the 
response, the application can close the connection and release its connection memory, during this window only TCP has the 
connection state. We've done a lot of testing and design  for handling all these cases, and to implement this correctly, 
we need TCP's involvement.

Murali

----- Original Message ----
From: Joe Touch <[email-masked]>
To: MURALI BASHYAM <[email-masked]>
Cc: Lloyd Wood <[email-masked]>; [email-masked]
Sent: Tuesday, November 6, 2007 10:24:49 AM
Subject: Re: [tcpm] Is this a problem?

MURALI BASHYAM wrote:
> In the spirit of keeping changes to TCP protocol minimal, how about
> having the application specify a time limit via the socket API when
 TCP gets into this
>  state and TCP cuts its indefinite wait down to this timeout? 

Why not set the timer in the application and ABORT or CLOSE the
connection when the timer goes off?

Joe

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Tue, 06 Nov 2007 10:40:10 -0800

> we need TCP's involvement.

It still sounds like you have a poorly written application to me; anyone
else?

Joe

---

From: Ethan Blanton <[email-masked]>
Date: Tue, 6 Nov 2007 14:08:45 -0500

Joe Touch spake unto us the following wisdom:
> MURALI BASHYAM wrote:
> > We want to limit the timer to the sender's persist state only, any
> > other timeout overloads the semantics of the application, and may also
> > cause unnecessary timeouts for the application. In fact there are
> > instances where the application may not even have the connection state
> > around to implement the timout correctly, like the tail of a long HTTP
> > response. After sending the tail of the response, the application can
> > close the connection and release its connection memory, during this
> > window only TCP has the connection state. We've done a lot of testing
> > and design for handling all these cases, and to implement this
> > correctly, we need TCP's involvement.
> 

This comment of Murali's, specifically, is a case that the application
*cannot* (portably or cleanly) handle, at least under the Berkeley
sockets API.  He is correct that once an application writes the end of
its data and calls close() or shutdown(), the TCP socket may persist
indefinitely in the kernel, and the application would never know (if
there are no application-level acknowledgments, as there are not in
simple HTTP responses).

It does not seem unreasonable to add a zero-window timeout tunable to
any given TCP implementation; I don't necessarily think it is a TCP
standardization issue, however, as there is no wire impact.

Ethan

-- 
The laws that forbid the carrying of arms are laws [that have no remedy
for evils].  They disarm only those who are neither inclined nor
determined to commit crimes.
		-- Cesare Beccaria, "On Crimes and Punishments", 1764
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Tue, 06 Nov 2007 11:12:09 -0800

> there are no application-level acknowledgments, as there are not in

Why would a blocking CLOSE not solve that?

> standardization issue, however, as there is no wire impact.

Protocols are more than just what is on the wire. They include state at
the endpoints, and APIs.

Joe

---

From: Ethan Blanton <[email-masked]>
Date: Tue, 6 Nov 2007 14:27:46 -0500

(Apologies for the duplicate, Joe, I didn't reply-to-list.)

> Why would a blocking CLOSE not solve that?

It is my understanding (correct me if I am wrong) that close() returns
immediately, regardless of queued data in the send buffer.  In
addition, I am not aware of any way to force a reset on a TCP socket
without pending data in the read buffer.

> Protocols are more than just what is on the wire. They include state at

Agreed, to some extent.

Ethan

---

From: Ethan Blanton <[email-masked]>
Date: Tue, 6 Nov 2007 14:39:12 -0500

> without pending data in the read buffer.

SO_LINGER is indeed more flexible than I have used it for -- it can be
used to solve both of these problems.  For the benefit of those
listening:

1) Set SO_LINGER with l_onoff != 0 and l_linger = timeout in seconds.
2) If a nonblocking close returns EWOULDBLOCK, the linger timeout
   expired, so:
   2a) set SO_LINGER with l_onoff != 0 and l_linger = 0
   2b) call close, which will emit a RST

I recant my previous claims -- this can be handled cleanly with
standard sockets.

> > > standardization issue, however, as there is no wire impact.

I do stand by this, however.

Ethan

---

From: Mahesh Jethanandani <[email-masked]>
Date: Tue, 06 Nov 2007 12:08:54 -0800

Ethan,

Would we not receive a EWOULDBLOCK if the connection was throttled 
because of congestion in the network? Under the scenario you describe we 
would have timed out a connection that was either slow or congested. Is 
that true?

Ethan Blanton wrote:
> ------------------------------------------------------------------------
>

-- 

*Mahesh Jethanandani*
*Technical Leader*
**Corporate Development*
*
[email-masked] <mailto:[email-masked]>
Phone :*+1 408 527-8230*
Fax :*+1 408 527-6351*

**
170 West Tasman Drive
San Jose, CA 95070
United States
www.cisco.com <http://www.cisco.com>

This e-mail may contain confidential and privileged material for the 
sole use of the intended recipient. Any review, use, distribution or 
disclosure by others is strictly prohibited. If you are not the intended 
recipient (or authorized to receive for the recipient), please contact 
the sender by reply e-mail and delete all copies of this message.

---

From: Ethan Blanton <[email-masked]>
Date: Tue, 6 Nov 2007 15:32:12 -0500

> would have timed out a connection that was either slow or congested. Is 

Yes; however, I agree with Joe (and others) that in the situation you
are *actually* trying to solve, this is not particularly relevant.  If
you are short on resources, the distinction between "not making
progress due to zero window" and "not making progress for some other
reason" does not seem important.  As previously mentioned, zero window
does _not_ indicate a malicious host in any way.

[Snip *24 lines* of signature]

Ethan

---

From: Joe Touch <[email-masked]>
Date: Tue, 06 Nov 2007 13:35:55 -0800

Ethan Blanton wrote:
> immediately, regardless of queued data in the send buffer.  ...

See, e.g.:
http://www.developerweb.net/forum/archive/index.php/t-2982.html

> In

ABORT is defined to do this, and does not require pending data in the
buffer AFAICT.

Joe

---

From: Mahesh Jethanandani <[email-masked]>
Date: Thu, 08 Nov 2007 08:56:57 -0800

> does _not_ indicate a malicious host in any way.
Where is the fairness argument in knocking connections off without 
making distinctions? A connection that is not making progress because of 
congestion in the network is a transient behavior and is very different 
from a connection that continues to advertise a zero window. The former 
is much harder to do and requires TCP stack changes. Zero window 
connection can be made to happen on a whim (as we successfully did) on a 
large number of connections from a user level program with little or no 
privileges.

/mahesh

---

From: Ted Faber <[email-masked]>
Date: Thu, 8 Nov 2007 09:49:30 -0800

> I do stand by this, however.

Let's not lose sight of this point.

Do we need a new standard interface to the TCP stack, or is this an
implementation issue?

-- 
Ted Faber
http://www.isi.edu/~faber           PGP: http://www.isi.edu/~faber/pubkeys.asc
Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: "Caitlin Bestler" <[email-masked]>
Date: Thu, 8 Nov 2007 12:55:25 -0500

*If* you were going to solve this problem as a *transport* problem
then you would have to define it in transport terms. I'll attempt
to offer such a description, and show that it is still in conflict
with the application layer problem.

The apparent transport layer issue is that the application does not
know whether the lack of credits are due to lack of receive buffers
or network congestion. The presumption here is that the server is
willing to suffer a bit if the problem is in the network rather
than the client.

The problem is that as long as the client is using TCP over a
conventional stack the receive buffers actually belong to the
OS and not to the client. So the absence or presence of receive
buffers does not really tell us anything about the client's state.
For example the reason the available receive buffering for the
connection is so small might be that the OS or Hypervisor has
swapped the client out.

So the lack of adequate receive windows can have *three* causes:
network congestion, lack of OS buffering resources or failure
of the client to drain the connection. And the latter case can
be a failure of the client to drain the connection that is based
on malice and/or sufficiently sloppy coding that terminating the
connection is justified, or it could just be that the client
is not getting scheduled for OS reasons.

There is no way for the transport layer to distinguish between
any of the latter three cases. There are many reasons for a
lack of receive buffering, and they all look the same on
the wire.

Now you could use a transport that guaranteed availability of
specific buffers that are fully guaranteed. You might even call
it something like Remote Direct Memory Access. But before you
could deploy this solution you would have to convince the clients
to use it. If you are trying to do this as a DoS protection then
you must convince *All* clients to make the shift so that you
can turn off support for legacy modes. If you were to actually
attempt such a project you should not do it with TCP. HTTP would
be far better implemented over SCTP and/or RDMA than over TCP
(if you are willing to ignore the billion or so installed clients).

More generally, while there is some merit in separating feedback
about end-to-end buffer credits from congestion tracking, such
a change would not be a "minor" tweak to TCP. If the market was
willing to make major changes to TCP it would already be using
SCTP instead.

---

From: John Heffner <[email-masked]>
Date: Wed, 14 Nov 2007 10:46:25 -0500

> large number of connections from a user level program with little or no 

It's actually an easy and generally more effective attack to just drop 
all the packets on the floor rather than continue ACKing with zero 
window.  The connection will eventually time out, but it takes so long, 
it's easy to initiate another one in this time to take its place.  Once 
again, please look carefully at Stas's netkill: 
<http://shlang.com/netkill/>.  It's a simple short perl script.  The 
defense you are proposing does nothing for this attack.

I don't think it's critical to determine whether a connection is stalled 
while limited by cwnd or rwin.  This *may* be useful information, but 
it's not the most important.  In my view, the more important thing to 
know is (1) whether the connection is making progress, and (2) how much 
memory (the contended resource) it is using.   Using this information, 
you can implement a policy that resets connections that are consuming 
resources with no benefit (progress).  This solves the more general 
problem -- both the netkill attack and a persist attack.

Also, I have a quibble with some of the language in the draft that I 
think reflects a view of where the solution must go:

    In most implementations, TCP runs in kernel mode as part of the
    operating system.  In this mode the operating system may share the
    same address space as TCP.  For the purposes of discussion, this
    draft considers TCP protocol implementation to be a separate module
    responsible for all resources such as buffers and connection control
    blocks that it borrows from the operating system.  The operating
    system can enforce the maximum number of buffers it is willing to
    give to TCP but beyond that it lets TCP decide how to manage them.

Even if you view TCP as running with an inaccessible address space, it 
can still export all sorts of information, in both standard (see RFC 
4022 and RFC 4898) and non-standard ways.  From a standards point of 
view, I think RFC 4898 exports all the necessary information.

And even if the solution must be implemented in an independent TCP 
module, this does not mean the solution should be defined and 
standardized as a transport layer mechanism.  A TCP module could 
implement a memory-based fairness policy if it's controlling its own 
memory pool as you describe.  This does not mean you need to change the 
definition of how the TCP protocol behaves in the persist state.

   -John

---

From: Mahesh Jethanandani <[email-masked]>
Date: Wed, 28 Nov 2007 11:06:41 -0800

John Heffner wrote:
> window.  The connection will eventually time out, but it takes so 
> long, it's easy to initiate another one in this time to take its 
> place.  Once again, please look carefully at Stas's netkill: 
That is true. The attack is very similar to the one we describe. It 
applies to a different state in TCP - FIN_WAIT_1. I would suggest that 
your attack is not just a "mbuf exhaustion" but also a TCP connection 
resource attack. It is possible to run either or both of them out. The 
defense we propose is specifically for connections in persist condition. 
We found it more tricky to simulate the situation you describe because 
it required the amount of data requested to fit in the receiver 
advertised window. In addition, it does have a timeout (albeit a long 
one) and requires more active clients to attack the server. However ...
>
> I don't think it's critical to determine whether a connection is 
> stalled while limited by cwnd or rwin.  This *may* be useful 
> information, but it's not the most important.  In my view, the more 
> important thing to know is (1) whether the connection is making 
> progress, and (2) how much memory (the contended resource) it is 
> using.   Using this information, you can implement a policy that 
> resets connections that are consuming resources with no benefit 
> (progress).  This solves the more general problem -- both the netkill 
> attack and a persist attack.
... our solution keeps track of connections that are in persist 
condition and keeps track of connections that are not making progress 
and consuming memory. The framework of the solution is amenable to 
include connections in FIN_WAIT_1 state.