Subject: RE: [Ietf-behave] TCP equivalent of UDP Req-5 (Was: RE: Req-5)

From: "Dave Hudson" <[email-masked]>
Date: Wed, 3 May 2006 02:03:08 -0700 (PDT)

Hi Saikat,

* -----Original Message-----
* From: ietf-behave-bounces@xxxxxxxxxxxxxxxxxxx 
* [mailto:ietf-behave-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of 
* Saikat Guha
* Sent: 03 May 2006 09:15
* To: Philip Matthews
* Cc: Behave
* Subject: Re: [Ietf-behave] TCP equivalent of UDP Req-5 (Was: 
* RE: Req-5)
* 
* Keeping it open means that for each an every properly-closed 
* TCP connection, the NAT can't evict the connection-table 
* entry for 4 minutes. The demerits are obvious: this cuts the 
* connection-per-second throughput of the NAT in 
* non-exceptional settings. The merit of this is that packets 
* for the old connection can still be routed to the endpoint 
* (only to be dropped by the kernel). Either way, it doesn't 
* affect the application at all since the app has already 
* close()'ed the socket.
* 
* Allowing the NAT to (possibly) delete state increases the 
* connections-per-second throughput of the NAT -- the point of 
* compromise recently discussed for UDP. It doesn't affect the 
* application still.
* 
* Consequently, the proposal is to leave NAT behavior in this 
* regard (TIME_WAIT state) undefined. Vendors may keep state 
* for 4 minutes, or 0 minutes as they choose. Applications are 
* not affected.

One real problem I've seen with NATs that discard state after 0 minutes
is when the NATs themselves don't then send well-formed RST:ACK packets
in response to TCP traffic that doesn't match a known flow in the
connection table.

With P2P file sharing apps it's very easy to see congestion problems
causing packets to get dropped after one NAT's sent them but before a
remote NAT receives them (especially true with many residential
broadband connections since they're relatively low bandwidth - 40 or 50
peers sending traffic at once can and will cause congestion problems).
Increasingly though packets are also getting lost on the local side too
- it's very common to see wirelessly-connected systems lose packets
between a NAT and the local system (this happens all the time - switch
on your microwave oven when you're using 802.11b/g/n and watch the TCP
retransmission rates :-)).

The problem is that just because a NAT has seen all of the FINs (and
most seem to completely ignore ACKs) or even the ACKs of FINs in a
congested or lossy network, it doesn't follow that the endpoints
themselves saw those things.  If a NAT evactuates a connection from the
connection table without allowing any time for retransmissions and
doesn't then send RSTs in response to traffic for unknown connections
(ignoring initial SYNs of course) then instead it ends up with one
endpoint or the other sending FINs at it until the endpoint eventually
gets bored and times-out.

Having the TIME-WAIT timer isn't guaranteed to stop problems either
(since retransmissions can also get lost) but the incidence of failure
is significantly lower.

Incidently one interesting failure mode I see in a lot of traffic I've
seen from systems behind NATs is where the NAT will create a new
connection in the connection table without ensuring that it was in
response to a SYN, thus you can see retransmitted FINs creating new
connection entries with different port bindings to the original traffic
(and thus you see response packets coming back from the wrong port
number).

Regards,
Dave
*****************************************************************************
We've Moved!

Please note our new address and phone numbers:

Ubicom, Inc.
510 N. Pastoria Ave
Sunnyvale, CA 94085
(408) 789-2200 - Main
(408) 739-2427 - Fax
www.ubicom.com
 
******************************************************************************

---

From: Saikat Guha <[email-masked]>
Date: Wed, 3 May 2006 02:55:41 -0700 (PDT)

On Wed, 2006-05-03 at 02:02 -0700, Dave Hudson wrote:
> connection table[...]
> (ignoring initial SYNs of course)

Just to clarify, the relaxation for TIME_WAIT does _not_ affect the
cases where the FIN is lost after it crosses the NAT but before it
reaches the destination. Those states are LAST_ACK, FIN_WAIT_1,
FIN_WAIT_2, and CLOSING. As per the requirement, NATs must wait 4
minutes for retransmissions etc. As you noted, the timer itself doesn't
solve this problem, but decreases the chances of failure by allowing the
endpoints to make a few retransmissions. 

The relaxation for TIME_WAIT affects connections where both the FINs
have been received but the very last ACK may have been lost. In these
situations, the OS has already notified the application that the
connection has terminated. The OS internally lets the TCP state linger
for some time. As a result, allowing vendors to pick a timeout for
TIME_WAIT (and only TIME_WAIT) doesn't affect the application per se.

Now considering TIME_WAIT, if the last ACK is lost and endpoint OS
re-sends the FIN (in the background while the application has moved on).
If the NAT has dropped the associated state, then I agree that it should
respond with an ACK to the sender. The question then becomes -- does the
draft need to state this? i.e. something along the lines of if the timer
for TIME_WAIT is short, then the NAT should ACK unexpected FIN packets.
Or, should the draft stay silent about it since it doesn't affect the
application.

My vote would be to not leave this unspecified. What do others think?

> response to a SYN

Aye. There are some really broken ones out there.

cheers,
-- 
Saikat
Attachment:
signature.asc

---

From: Philip Matthews <[email-masked]>
Date: Wed, 3 May 2006 06:04:25 -0700 (PDT)

Saikat:

Thanks for your reply. I think that Dave has more direct experience  
in this area,
so I will simply give some general comments.

Speaking personally, I am concerned with both the application and the  
TCP state machine in the
underlying OS. I would like to ensure that both of them have a "good"  
experience
when they are behind a NAT.

So when things are left unspecified (like how long the NAT's state  
machine
should remain in the TIME_WAIT state), then I personally would like  
to see the document
give some suggestions or guidelines to NAT implementors on what are
acceptable implementation choices, and what to watch out for.
Specifically, the document might say something like
  "The time to remain in the TIME_WAIT state is left unspecified to  
allow for
   some variations in implementation. A NAT should remain in this  
state for at
   most 4 minutes. Some NAT implementations will choose stay in this  
state for exactly
   4 minutes. Other NAT implementations may choose
   to reduce this time. However, NATs that reduce this time must  
still behave
   as follows: ....  "
How this should be split between requirement text and informative  
text is something
I am not yet sure of.

To me, the thing about NATs is that, unlike many other products, a  
bad choice by a NAT
implementor affects not only the NAT itself, but all products behind  
the NAT,
so reasonable behavior is very important.

Of course, all this is just my own opinion, and others may disagree.
Anyway, hope this helps. Thanks very much for working on this document.
- Philip

On 3-May-06, at 05:55 , Saikat Guha wrote:

On Wed, 2006-05-03 at 02:02 -0700, Dave Hudson wrote:
One real problem I've seen with NATs that discard state after 0  
minutes
is when the NATs themselves don't then send well-formed RST:ACK  
packets
in response to TCP traffic that doesn't match a known flow in the
connection table[...]
(ignoring initial SYNs of course)
Just to clarify, the relaxation for TIME_WAIT does _not_ affect the
cases where the FIN is lost after it crosses the NAT but before it
reaches the destination. Those states are LAST_ACK, FIN_WAIT_1,
FIN_WAIT_2, and CLOSING. As per the requirement, NATs must wait 4
minutes for retransmissions etc. As you noted, the timer itself  
doesn't
solve this problem, but decreases the chances of failure by  
allowing the
endpoints to make a few retransmissions.

The relaxation for TIME_WAIT affects connections where both the FINs
have been received but the very last ACK may have been lost. In these
situations, the OS has already notified the application that the
connection has terminated. The OS internally lets the TCP state linger
for some time. As a result, allowing vendors to pick a timeout for
TIME_WAIT (and only TIME_WAIT) doesn't affect the application per se.
Now considering TIME_WAIT, if the last ACK is lost and endpoint OS
re-sends the FIN (in the background while the application has moved  
on).
If the NAT has dropped the associated state, then I agree that it  
should
respond with an ACK to the sender. The question then becomes --  
does the
draft need to state this? i.e. something along the lines of if the  
timer
for TIME_WAIT is short, then the NAT should ACK unexpected FIN  
packets.
Or, should the draft stay silent about it since it doesn't affect the
application.

Incidently one interesting failure mode I see in a lot of traffic  
I've
seen from systems behind NATs is where the NAT will create a new
connection in the connection table without ensuring that it was in
response to a SYN

cheers,
--
Saikat

---

From: Saikat Guha <[email-masked]>
Date: Sun, 7 May 2006 18:21:45 -0700 (PDT)

On Wed, 2006-05-03 at 09:05 -0400, Philip Matthews wrote:
> underlying OS.

Fair enough.

> acceptable implementation choices, and what to watch out for.

Ok. I'll incorporate the text you suggested into the draft. Essentially
to the effect that it is up to the vendor and it would be nice if they
were to stick to 4 minutes, but there may be reasons to reduce it, which
may in turn cause certain side-effects and one way to handle such
effects.

cheers,
-- 
Saikat
Attachment:
signature.asc

---

From: Philip Matthews <[email-masked]>
Date: Fri, 12 May 2006 07:31:16 -0700 (PDT)

Hi Saikat:

Sorry to be slow in responding, but I was away.

What you said below sounds fine to me.

- Philip

On 7-May-06, at 21:21 , Saikat Guha wrote:

So when things are left unspecified (like how long the NAT's state
machine
should remain in the TIME_WAIT state), then I personally would like
to see the document
give some suggestions or guidelines to NAT implementors on what are
acceptable implementation choices, and what to watch out for.
Ok. I'll incorporate the text you suggested into the draft.  
Essentially
to the effect that it is up to the vendor and it would be nice if they
were to stick to 4 minutes, but there may be reasons to reduce it,  
which
may in turn cause certain side-effects and one way to handle such
effects.

cheers,
--
Saikat