Subject: Re: [tcpm] draft-ananth-tcpm-persist-00.txt as a WG document

From: Murali Bashyam <[email-masked]>
Date: Thu, 2 Oct 2008 17:13:50 -0700

> -----Original Message-----
> From: John Heffner [mailto:[email-masked]]
> Sent: Monday, September 29, 2008 11:10 PM
> To: David Borman
> Cc: [email-masked]; Anantha Ramaiah (ananth); Murali Bashyam
> Subject: Re: [tcpm] draft-ananth-tcpm-persist-00.txt as a WG document
>
> On Fri, Sep 26, 2008 at 12:37 PM, David Borman
> <[email-masked]> wrote:
> > At Dublin a presentation was made on draft-ananth-tcpm-persist-
> 00.txt.  Not
> > a lot of people had read the document at the time of the
> presentation, but
> > there didn't seem to be any objections to adopting it as a TCPM WG
> document.
> >
> > The authors posted a summary to the mailing list, have responded to
> all
> > issues that were raised, and have a new version of the document ready
> for
> > publication.  We (Wes and I) feel that there is support for adopting
> this as
> > a WG document, but as always, we need to verify this on the mailing
> list.
> >  So, if you have any objections to adopting this as a WG document,
> please
> > speak up now.  Also speak up if you support adopting it as a WG
> document.
> >
> > I'll be the first to say I support adopting this as a WG document.
> If there
> > are no serious objections, the authors can submit their updated
> version next
> > week and Wes and I will add it to our list of WG documents.
>
>
> I have reservations about moving forward with this draft as a wg
> document.  While the information is technically correct, it is overly
> specific to the point of possibly being misleading rather than
> clarifying.  An operating system may terminate a tcp connection at any
> time, not just when the connection is in the persist state.  Further,
> I would actually argue that terminating a connection *because* it's in
> the persist state is a bad idea and should be discouraged.  (The
> reason for termination should be for its use of resources, not TCP's
> state.)

No implementation today (the well-known ones BSD, Windows and Linux) terminates the TCP connection in that persist state as long as ACKs are being reliably received from the peer. Those three implementations are not doing what you are saying they should be doing, and if it's crystal clear from the standard that they should be doing so, why aren't they? They are all following the letter of the standard (1122) which dictates that they MUST do so. Linux even has a comment in the code that says RFC 1122 prevents it from doing so.

The *informational* document's goals are the following:
 - Clarify 1122's position on this sender behaviour.
 - Bring to light a consequence of persisting indefinitely, leading to a DoS vulnerability triggered easily by a user-space receiving application. The mitigation techniques for this are not going to be described in this doc, as discussed in the mailing list before. It's possible that some of the OS's existing techniques for mitigating resource exhaustion can already address this scenario, but that's out of scope of this document.
 - Expose the indefinite persisting behavior to allow the application to control that behavior and terminate the connection (possibly even when resources are available).

Murali

>   -John
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Oct 2008 17:54:49 -0700

Murali Bashyam wrote:
...
> No implementation today (the well-known ones BSD, Windows and Linux)
> terminates the TCP connection in that persist state as long as ACKs are
> being reliably received from the peer.

No OS I'm aware of cleans much of anything up when it runs out of space.
They tend to just run out of space and silently hang or thrash incessently.

The point is that an OS that wants to do this can. That's an OS
implementation recommendation that I support - the question is whether
such OS implementation issues are useful as RFCs.

> Those three implementations are
> not doing what you are saying they should be doing, and if it's crystal
> clear from the standard that they should be doing so, why aren't they?

The standard already says, IMO, that they MAY do this. Not that they
MUST. The standard says nothing about what to do when you run out of
resources to run the connections you're currently supporting - it says
nothing about what to do when new connections arrive in that state
either. Memory management is the OS's job, not the protocol's.

> They are all following the letter of the standard (1122) which dictates
> that they MUST do so. Linux even has a comment in the code that says RFC
> 1122 prevents it from doing so.

That's a great reason to:
	a) write code to enable this behavior in the OS's you
	want to fix
	b) correct the Linux comment

This is all an OS implementation issue, not a protocol standards issue.

Joe

---

From: Murali Bashyam <[email-masked]>
Date: Thu, 2 Oct 2008 23:51:36 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Joe Touch
> Sent: Thursday, October 02, 2008 5:55 PM
> To: Murali Bashyam
> Cc: Anantha Ramaiah (ananth); [email-masked]; David Borman
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Murali Bashyam wrote:
> ...

Which standard are u referring to? I'm quoting from RFC 1122, page 91
the section on probing zero windows :

"As long as the receiving TCP continues to send acknowledgments in response to the
Probe segments, the sending TCP MUST allow the connection to stay open."

Can you point me to the word "MAY" in that sentence that you are referring to?
The important thing here is to clarify that sentence and the sender's behaviour and prevent what appears to be a conflict between memory management and protocol violation.

>
> Joe
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkjlbVkACgkQE5f5cImnZrs/fQCfYqFOdyjmg9aVMxyuMjY2eDZd
> svkAnjvVH/Rq5E93BOsp5A/ZzKv4LBjL
> =p3nQ
> -----END PGP SIGNATURE-----

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 Oct 2008 00:19:36 -0700

Murali Bashyam wrote:
...
>> Can you point me to the word "MAY" in that sentence that you are referring to?

It's implied by the conditional. I.e., if the receiving TCP stops
sending ACKs, the sending TCP can drop the connection.

As to the sending side, there's the ABORT call in 793 - which we
discussed on this list before. The point is that _TCP_ can't decide to
stop the connection because of the TCP state of the other end, the local
TCP state, or behavior on the wire. Nothing in 793 says that the OS
can't kill the connection - any connection - at any time.

>> The important thing here is to clarify that sentence and the sender's behaviour
>> and prevent what appears to be a conflict between memory management
and protocol violation.

Please describe anywhere in 1122 or 793 where memory management is
described.

As I've noted before, this isn't a protocol violation - it's a failure
of an OS to support arbitrary numbers of instances of a protocol.

Joe

---

From: Ted Faber <[email-masked]>
Date: Fri, 3 Oct 2008 11:37:26 -0700

I don't mean to be answering from John, but your comments interested me.

On Thu, Oct 02, 2008 at 05:13:50PM -0700, Murali Bashyam wrote:
> terminates the TCP connection in that persist state as long as ACKs
> are being reliably received from the peer. Those three implementations
> are not doing what you are saying they should be doing, and if it's
> crystal clear from the standard that they should be doing so, why
> aren't they?

Your comment is a little ambiguous.  John says that he believes that
persist state is not necessarily a good marker for aborting connections
when resources are scarce.  To take him to task because implementations
are not using it as a marker - because the implementers agreed with
his position - is a little unfair, IMHO.

On the WG item:

As a bigger picture issue, I find it helpful to remember that
protocol standards, like 1122, are primarily interoperation documents.
As such, they tend to focus on what protocol implementers must do to
talk with one another successfully, not to nail down all possible
choices.

Once we start offering advice (or stronger) to developers, we're leaving
the interoperability domain.  We, as a standards body, have to assume
that implementers know their application and their environment better
than we do, and that they can and will make appropriate decisions if we
give them the room to do so.

Where a standard has unnecessarily tied the hands of implementers, that
standard should be changed, IMHO.  I am also sympathetic to clarifying
the standards body's intent where poor language has obscured it.  I
don't see that kind of confusion in the text the authors are addressing
(that is RFC 1122 Section 4.2.2.17), but reasonable people can disagree
on such things.

As such, taking on this work item - clarifying 1122 4.2.2.17 - doesn't
excite me, but I don't oppose it.

The draft should, IMHO, point out that 1122 has nothing to say about
resource management.  Designers and implementers are free to be as
clever or foolish as they'd like.

End of thoughts on the WG item.

As for the draft itself, I'm concerned that it seems to be advocating a
particular resource management position more strongly than simply
pointing out that RFC1122 should not affect resource allocation
decisions.  I also see some misplaced standards language that is
somewhat confusing.  I'm looking at this paragraph in Section 2:

	An extensive discussion took place recently about this issue on
	the TCPM WG mailing list [TCPM].  The general opinion seemed to
	be that terminating a TCP connection in persist condition does
	not violate RFC 1122.  In particular the operating system, a
	resource manager, or an application can instruct TCP to abort a
	connection in the persist condition.  TCP itself SHOULD not take
	any action and continue to keep the connection open as mandated
	by RFC 1122 unless otherwise instructed to do so.  The exact
	mechanism by which the instruction to abort the connection is
	conveyed to TCP is an implementation decision and falls beyond
	the scope of the current memo. 

There's no instruction going on, and one doesn't request an abort.
Abort is a command in the TCP interface (defined in RFC 793, p. 50) that
destroys a TCP connection.  Abort works in any state.  Anything with
access to the TCP interface, including the OS, a resource manager, or an
application, can call it.  A TCP abort is "the exact mechanism by which
the instruction to abort the connection is conveyed to TCP", and the
only reason that it's outside the scope of this document is that it's
defined in 793.

In light of that I think that the sentence about "TCP itself SHOULD
not[sic] take any action..." is more confusing than the 1122 section.
It seems to imply that TCP might react differently to an abort than
the single paragraph in 793, and I don't think that's the case.

Now, a strictly defined abort command seems to somewhat contradict my
earlier comments about allowing implementers some freedom.  This is a
spot where the standards body has required a specific tool to be
available to applications (and OSes and resource managers): there must
be a way to annihilate a TCP connection from outside without regard to
its state or anything else.  The standard is silent about how the
command is used; that's the flexibility here.

-- 
Ted Faber
http://www.isi.edu/~faber           PGP: http://www.isi.edu/~faber/pubkeys.asc
Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: "Murali Bashyam" <[email-masked]>
Date: Fri, 3 Oct 2008 12:37:10 -0700

> his position - is a little unfair, IMHO.

My intent was that even where TCP implementations have a scheme for
dropping connections in the event
of resource exhaustion (such as the Linux one does), they have excluded
connections in this state.
The point i am trying to convey is that the 1122 portion has (mis)guided
them in that direction.

> clever or foolish as they'd like.

I agree, the primary intention of this draft is not to restrict or present
any resource management schemes, it's to clarify 1122.
In addition,  we are trying to highlight the need for resource management in
this scenario because it leads upto a DoS potential.

> the single paragraph in 793, and I don't think that's the case.

What's meant here is that it's okay for the connection to get aborted in
this state, whatever the mechanism for doing
so maybe: TCP can abort the connection and inform the application, similar
to the retransmit/persist timeout mechanisms
already in place, or the application can choose to abort the connection
based policy, or the OS can do so based
on resource management criteria. We can re-word that sentence and paragraph
to reflect this.

> command is used; that's the flexibility here.

The abort semantics as specified by 793 are clear, they are not specific to
a particular state, it's the opinion
of the authors that 1122 excludes it from being applied in this state.

Murali

>
>
> --
> Ted Faber
> http://www.isi.edu/~faber <http://www.isi.edu/%7Efaber>           PGP:
> http://www.isi.edu/~faber/pubkeys.asc<http://www.isi.edu/%7Efaber/pubkeys.asc>
> Unexpected attachment on this mail? See
> http://www.isi.edu/~faber/FAQ.html#SIG<http://www.isi.edu/%7Efaber/FAQ.html#SIG>
>

-- 
Rgds,
Murali Bashyam
(c) (510)6736915

----------------------------- CONFIDENTIAL  --------------------
*****************************************************
This telecommunication and any data attached to, or included in it
is considered confidential, and is intended only for use by the named
recipient. The contents may be legally protected as any one or more of:
copyrighted material, trade-secret protected material, attorney-client
privileged material, attorney workproduct, or as material covered by
any other legally available means. If you received this material in
error, please notify the sender and destroy the original and all copies,
whether electronic or otherwise. Thank you.
*****************************************************

---

From: David Borman <[email-masked]>
Date: Fri, 3 Oct 2008 15:03:45 -0500

(WG co-chair hat off)

First, I agree with everything that Ted said.

It might be helpful in this discussion to know why the wording on  
connections in persist state was put into RFC 1122 in the first  
place.  Everything that is in 1122 and 1123 is in there for a reason,  
whether or not that reason is included in the document.

In the case of connections in persist state, my fuzzy recollection was  
that there were TCP implementations that were timing out and  
terminating TCP connections stuck in persist state, the same way that  
they timed out and terminated non-responsive connections.  An example  
scenario where this is bad is if you have TCP connection open to a  
printer, and the printer has stopped accepting data because it is  
waiting for someone to load more paper; you don't want TCP arbitrarily  
timing out and closing that connection, just because the printer can't  
accept any more data.  The printer might sit without paper for a  
couple of hours over a long lunch break, or overnight, and once  
someone comes back and reloads paper, the printer should be able to  
continue receiving data on TCP connection, and continue printing where  
it left off.

That's an example of why RFC 1122 explicitly calls out keeping  
connections in persist state open, and is also an example of why *TCP*  
does not have enough information by itself to decide to shut down a  
connection in persist state.

I do understand that in many TCP stacks, you can create a TCP socket,  
write a bunch of data and close the socket, and TCP will keep the  
state and data, and continue to send the queued up data.  If the  
connection goes into persist state, there is no longer an application  
associated with the socket to say this is taking to long and abort the  
connection, hence as long as the other side responds to the probes,  
the state & data stay around.

But terminating that connection is an OS implementation issue of how  
to reclaim data when the system is out of resources.  If it decides to  
abort existing TCP connections, it could choose to abort first those  
connections in persist state where the application has closed the  
socket.  Or it might have some other criteria.  And in terms of the  
protocol definition, as Ted points out, the way you do that is with an  
ABORT.  Perhaps the OS should provide an interface to abort arbitrary  
TCP connections, so that the sys admin can kill off any TCP  
connection.  But that is all implementation details, it has nothing to  
do with how the TCP protocol operates.

If the other side is being malicious, it can just as easily  
acknowledge one byte of data every so often to keep the connection  
from staying perpetually in persist state, effectively defeating any  
remediation based on looking at how long a connection has been in  
persist state.  So, the connection is making progress now, but  
slowly.  So do we now need to call out how to deal with connections  
that are stuck with very slow progress, in addition to those that are  
making no progress?  It's a slippery slope.

It might be useful in the draft to note that this is just one specific  
example of how a malicious client can tie up resources on the server,  
and that the server application or OS is perfectly free to terminate  
any TCP connection in any state for any reason, and that a connection  
stuck in in persist state is just one example.

My personal feeling is that there isn't a problem here that is crying  
out to be documented, it is an implementation issue of how to deal  
with a lack of resources.  On the other hand, if this item in 1122 has  
caused implementors of resource recovery to treat connections in  
persist state different than other TCP connections, then I have no  
objection to an informational document that can give some  
clarification.  And it should be vetted through the TCPM WG.

			-David Borman

---

From: Ted Faber <[email-masked]>
Date: Fri, 3 Oct 2008 13:21:39 -0700

Somewhat snipped up.

On Fri, Oct 03, 2008 at 12:37:10PM -0700, Murali Bashyam wrote:
> What's meant here is that it's okay for the connection to get aborted
> in this state, whatever the mechanism for doing so maybe: TCP can
> abort the connection and inform the application, similar to the
> retransmit/persist timeout mechanisms already in place, or the
> application can choose to abort the connection based policy, or the OS
> can do so based on resource management criteria. We can re-word that
> sentence and paragraph to reflect this.

I disagree with "TCP can abort the connection and inform the
application."  RFC 1122 says that TCP must allow the connection to stay
open.  I believe that TCP closing or aborting such a connection (without
an abort command being issued) violates 1122.  External entities (OS,
resource allocator, applications) can abort the connection, as they can
abort any connection.

The point is that the OS and applications (and other entities in a
system) can both annhilate TCP connections regardless of the state of
those connection to meet their needs.  I believe that's a perfectly
adequate tool for dealing with resource starvation.  Delivering that
message seems a valid WG item.

> The abort semantics as specified by 793 are clear, they are not
> specific to a particular state, it's the opinion of the authors that
> 1122 excludes it from being applied in this state.

Can you say that again?  What you typed says both that the abort command
is independent of connection state and that it doesn't work on
connections in a certain state.  That sentence contradicts itself, so
I must be misunderstanding something.

For the record, I believe that when a TCP stack receives an abort
command for a connection in persist state the stack must abort the
connection as per RFC 793, p. 50.

---

From: David Borman <[email-masked]>
Date: Fri, 3 Oct 2008 16:04:48 -0500

On Oct 3, 2008, at 2:37 PM, Murali Bashyam wrote:
> of resource exhaustion (such as the Linux one does), they have  
> excluded connections in this state.
> The point i am trying to convey is that the 1122 portion has  
> (mis)guided them in that direction.

If this is true, it should be clarified.  1122 does not in any way say  
that ABORT does not apply to connections in persist state (probing a  
zero window).

...
> What's meant here is that it's okay for the connection to get  
> aborted in this state, whatever the mechanism for doing
> so maybe: TCP can abort the connection and inform the application,  
> similar to the retransmit/persist timeout mechanisms
> already in place, or the application can choose to abort the  
> connection based policy, or the OS can do so based
> on resource management criteria. We can re-word that sentence and  
> paragraph to reflect this.

Just to be clear: TCP DOES NOT ABORT THE CONNECTION!  The OS or the  
application can choose to ABORT the connection, but TCP does not make  
this decision.  TCP does have retransmit timers, and it is allowed to  
terminate unresponsive connections, but it does not abort connections  
just because they are in persist state for long periods of time.   
Until you are running out of resources there is no good reason to  
terminate connections in persist state, and when you are running out  
of resources, it is the OS that is deciding whether or not to ABORT  
any TCP connections.

> specific to a particular state, it's the opinion

Yes, that is an opinion, but it is wrong.  In no way does 1122 say  
that connections in persist state cannot be ABORTed.  What 1122 does  
say is that a connection being in persist state for a long time is not  
sufficient reason *by itself* to terminate a TCP connection, i.e.,  
don't time it out like you would a connection to a non-responsive host.

			-David Borman

---

From: Murali Bashyam <[email-masked]>
Date: Sat, 4 Oct 2008 21:48:37 -0700

> From: David Borman [mailto:[email-masked]]
> Sent: Friday, October 03, 2008 2:05 PM
> To: Murali Bashyam
> Cc: Ted Faber; [email-masked]; Anantha Ramaiah (ananth); Murali Bashyam

I agree that the retransmit/persist timeout analogy doesn't apply to this situation.
It is ONLY the OS or resource manager or the application that can terminate the connection
In this state.

> don't time it out like you would a connection to a non-responsive host.

Just so that we are all clear, i want to summarize the goals of the draft here once again:

1. Clarify the 1122 ambiguity/stance on the sender behavior in persist state.
2. Document (for informational purpose) the consequence of this behavior leading to
the DoS possibility.
3. Specify that the OS/application can terminate the connection in this state using the 793 ABORT interface.

We are NOT documenting the various memory management considerations/issues surrounding the DoS scenario, that's left to the implementers discretion.

We will change the section that's ambiguous regarding the sentence that goes "TCP itself should not take any action...".