Subject: RE: [Ietf-behave] Catalogue of problems with deployed NATs

From: "Francois Audet" <[email-masked]>
Date: Mon, 19 Dec 2005 07:39:56 -0800 (PST)

This is quite interesting, and hardly surprising.

We had find bizarre behaviors like this on UDP where it seemed to defy
all logic.

If you are seeing this in more than one NAT, it might be a good idea
to document it: it might help people figuring out the root causes.

> -----Original Message-----
> From: ietf-behave-bounces@xxxxxxxxxxxxxxxxxxx 
> [mailto:ietf-behave-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of 
> Dave Hudson
> Sent: Monday, December 19, 2005 05:59
> To: Pyda Srisuresh; ietf-behave@xxxxxxxxxxxxxxxxxxx
> Subject: RE: [Ietf-behave] Catalogue of problems with deployed NATs
> 
> 
> Hi Suresh,
> 
> Thanks for the pointers.  I was familiar with most of the 
> documents you mentioned but I'd not seen anything that 
> specifically discussed many of the things I've been seeing.  
> While I've seen a lot of discussion about general problems 
> with NATs I concluded I was seeing other 
> implementation-related problems but haven't been able to find 
> so much discussion of this.
> 
> Essentially much of what I've been seeing appear to be a 
> consequence of the high volumes of NAT sessions that can be 
> established by some P2P apps.  I'm also seeing the effect of 
> the high levels of flux seen in many of the P2P file-sharing 
> networks where nodes in the network disappear (although in 
> some cases I suspect this may be the result of symmetric NATs 
> or NATs that exhibit symmetric behaviour under some circumstances).
> 
> I normally find that after running for an hour or so (on a slow DSL
> link) I get more than 5 egressing and 5 ingressing new 
> sessions per second (often this number is more than 20 in 
> each direction for periods of a few seconds) so when coupled 
> with a recommendation that TCP sessions in the Syn-Sent state 
> hold state for 4 minutes and UDP sessions hold state for 5 
> minutes this can generate around 2500 sessions. Coupled with 
> this I may have 700 to 800 live sessions and another 1000 or 
> so half-open (or half-closed :-)) TCP sessions.
> 
> At one point I was seeing far more half-open TCP sessions but 
> I then adjusted the timings in the NAT I'm working with to 4 
> minutes for Syn-Sent, Syn-Received, Time-Wait, Last-Ack, 
> while making them 130 minutes for Established, Fin-Wait-[1/2] 
> and Close-Wait.  I also modified the code to allow sessions 
> that had seen a RST to stay in a "closed" state that would 
> allow packets to pass for 4 minutes because it was amazing 
> how many RST packets were apparently not being handled (I 
> suspect many were being dropped in transit as a result of too 
> much traffic being sent to remote peers in bursts), leading 
> to situations where one peer would send a RST and then 2 or 3 
> minutes later the other peer would send (and then continue to 
> resend FIN/ACK);  The extra time in this closed state allows 
> for RST packets to be resent in response to packets from the 
> peer that didn't see the original RST.
> 
> Anyway here are some of the things I've observed so far (I've 
> seen all of these with a surprisingly high frequency - 
> typically I see one of these every 4 to 5 seconds during 
> tests where I send about 70 packets per second and typically 
> receive 60 per second):
> 
> 1) Bad ICMP destination (port) unreachable or redirect 
> messages.  These are messages where the remote peer is 
> apparantly no longer running where it was but the remote NAT 
> fails to translate the payload of the message. As a result 
> the packet is sent from the remote NAT to my NAT but the ICMP 
> message refers to a remote host that is clearly a private IP 
> of some host behind that NAT (typically 192.168.x.y) or 
> occasionally 0.0.0.0.  These messages are usually for UDP 
> sessions but I do see some for TCP too.
> 
> 2) FIN/RST/ACK packets in response to SYN requests (this may 
> just be odd behaviour from a remote TCP stack though).
> 
> 3) RST packets being sent to the wrong local port.  In these 
> cases a remote NAT sends a FIN/ACK to close an existing 
> session and follows it immediately with a RST but the RST 
> ends up sent to a random port at my end despite the FIN/ACK 
> having been sent the correct port (this double approach to 
> closing sessions is surprisingly common but normally the port 
> numbers are the same).  All subsequent FIN/ACKs from my end 
> get the same mismatched RST in response.
> 
> 4) Remote NAT bindings change mid-session after some period 
> of inactivity.  My guess is that the remote NAT times-out its 
> binding part way through a TCP session and suddenly traffic 
> that was coming from one port suddenly starts to arrive from 
> a different port (this is similar to 3 but in 3 the port 
> binding for my host gets changed).  I'd guess that these 
> remote NATs aren't really tracking the TCP state transitions 
> as anything stateful probably wouldn't allow a new session 
> for anything other than SYN-related transactions.
> 
> 5) Inconsistent port binding when remote systems use port 
> forwarding. In these cases the remote NAT has a port 
> forwarding rule to ensure that ingressing traffic will be 
> sent to some specific service, however any egressing traffic 
> from the remote NAT appears from a different (but
> consistent) port number (this effectively renders it 
> symmetric even though it may nominally be a NAT that would 
> normally employ a persistent port binding).  This is 
> particularly noticeable as a variant of 4.
> 
> One problem here is that by not sending RST packets in 
> response to some of the resulting bad TCP packets we avoid 
> the possibility of being a reflector for a DDoS attack but do 
> end up with far more bad traffic. I've experimented with a 
> compromise that sends RSTs in response to bad packets from 
> hosts with which we already have an active session and saw 
> about a 25% drop in packets rejected at my end.
> 
> 
> Regards,
> Dave
> 
> * -----Original Message-----
> * From: ietf-behave-bounces@xxxxxxxxxxxxxxxxxxx 
> * [mailto:ietf-behave-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of 
> * Pyda Srisuresh
> * Sent: 19 December 2005 05:06
> * To: Dave Hudson; ietf-behave@xxxxxxxxxxxxxxxxxxx
> * Subject: Re: [Ietf-behave] Catalogue of problems with deployed NATs
> * 
> * Dave,
> * 
> * RFC 3027 was done a while back documenting compliations with NATs.
> * draft-ford-behave-top-00.txt documents complications with the 
> * use of NATs in certain topologies. The ID has expired. We 
> * will get a new rev out shortly.
> * Outside of these, I dont know of any other. 
> * 
> * Additional comments inline below.
> * 
> * cheers,
> * suresh
> * 
> * --- Dave Hudson <dhudson@xxxxxxxxxx> wrote:
> * 
> * > Does anyone have any pointers to documents/websites that describe 
> * > problems seen by widely-deployed NATs?  I've been looking 
> into some 
> * > very odd behaviour I've seen from systems using P2P file-sharing 
> * > applications and it's not at all unusual for me to see 50 or 60 
> * > packets per minute arriving at my test system (around 0.5% of all 
> * > packets) where there's very strong evidence of something 
> * wrong in the 
> * > translations being applied by remote NATs.
> * > 
> * > Many of these problems are leading to me seeing excessively high 
> * > numbers of TCP connections left in effectively a half-open 
> * state (e.g. 
> * > with a 2 hour timeout applied to traffic in Establish, 
> Fin-Wait-1, 
> * > Fin-Wait-2 and Close-Wait I can see 2k or 3k such 
> * connections after an 
> * > hour or two even when only using at a slow DSL connection) with a 
> * > consequent impact on overall performance.
> * > 
> * 
> * [suresh] Sounds like, you are saying you noticed a lot of 
> * half-open or half-close conenctions on the NATs. This may be 
> * specific to the file sharing Peer-to-peer (P2P) application 
> * you are using and the algorithm the application is using to 
> * traverse NATs. I dont know what the problem is. But, I 
> * suspect, the problem is to do with symmteric NATs being enroute.
> * 
> * > I'm investigating strategies that will help me reduce both 
> * the amount 
> * > of bad traffic that I see arriving and the number of half-open 
> * > connections but a more comprehensive list of possible 
> failure modes 
> * > would help (if there is one).
> * > 
> * [suresh] You might want to take a look at 
> * draft-srisuresh-behave-p2p-state-01.txt for a description of 
> * the various algorithms in use by P2P apps for NAT traversal.
> * 
> * > Failing that I'll be happy to summarise problems I have 
> observed if 
> * > anyone's interested.
> * > 
> * [suresh] That sounds good in any case. Thanks.
> * 
> * cheers,
> * suresh
> * 
> * > 
> * > Thanks,
> * > Dave
> * > _______________________________________________
> * > Ietf-behave mailing list
> * > Ietf-behave@xxxxxxxxxxxxxxxxxxx
> * > https://list.sipfoundry.org/mailman/listinfo/ietf-behave
> * > 
> * 
> * 
> * 
> * _______________________________________________
> * Ietf-behave mailing list
> * Ietf-behave@xxxxxxxxxxxxxxxxxxx
> * https://list.sipfoundry.org/mailman/listinfo/ietf-behave
> * 
> _______________________________________________
> Ietf-behave mailing list
> Ietf-behave@xxxxxxxxxxxxxxxxxxx 
> https://list.sipfoundry.org/mailman/listinfo/ietf-behave
> 
>

---

From: "Dave Hudson" <[email-masked]>
Date: Wed, 21 Dec 2005 03:59:26 -0800 (PST)

Hi Suresh, 

Comments inline...

Regards,
Dave

* -----Original Message-----
* From: Pyda Srisuresh [mailto:srisuresh@xxxxxxxxx] 
* Sent: 20 December 2005 16:45
* To: Dave Hudson; ietf-behave@xxxxxxxxxxxxxxxxxxx
* Subject: RE: [Ietf-behave] Catalogue of problems with deployed NATs
* 
* Hi Dave, 
* 
* It seems you can control some of the NAT devices enroute, but 
* not all in this setup. Is that right? Results are not 
* predictable when you change settings on just one of the NAT devices.

I have no control over anything but my own local NAT.  I have, at times,
removed that and watched the same behaviour with a PC directly connected
in the P2P networks though (it's just easier to log things with my NAT
and I was looking at changes I could make to it that would help improve
its behaviour or to have it generate fewer "noise" log messages about
packets received that it couldn't make sense of).

Much as I'd like to have made this a more controlled test I don't have a
network of a thousand systems available for me to test with (at any one
time it's quite normal for some of these apps to be talking to over a
thousand remote systems).  The thing I was surprised about were the
numbers of different remote systems (scattered all over the world when I
checked the IP addresses I'd collected) that all exhibited the same set
of quirks.  This leads me to believe that the behaviour I've observed
must be found in quite widely deployed systems.

* Also, do you have know the NAT traversal algorithm being 
* employed by the P2P application? If so, what is it? 

The applications I've been looking at don't do hole punching for the TCP
streams - they rely on using UPnP or having the user configure specific
forwarding rules within the NAT to forward all traffic on a particular
port to the P2P apps.  This is one of the reasons I find them an
interesting (and relatively simple) test case because I could discount
hole-punching attempts as a cause of the behaviour I was seeing.

* > Essentially much of what I've been seeing appear to be a 
* consequence 
* > of the high volumes of NAT sessions that can be established by some 
* > P2P apps.  I'm also seeing the effect of the high levels of 
* flux seen 
* > in many of the P2P file-sharing networks where nodes in the network 
* > disappear (although in some cases I suspect this may be the 
* result of 
* > symmetric NATs or NATs that exhibit symmetric behaviour under some 
* > circumstances).
* > 
* 
* [suresh] Right. It is possible that some of the NAT devices 
* are hitting their resource limits (say, the number of NAT 
* Sessions) with P2P applications, especially when the P2P apps 
* are aggressive with multiple session initiations. 

I'd be quite surprised if that wasn't the case.  I know that many of the
NATs deployed in SOHO routers have very limited session tables (say 1024
sessions).  I tend to be very interested in how things behave under
these sort of conditions though as that's often the cause of bad
behaviour that systems I develop have to then handle :-)

* [suresh] Once a TCP session is established, RST handling for 
* P2P sessions is same as for regular client-server sessions. 
* The dropping of RST packets may be due to unusual traffic volume.

I'm pretty certain that's what's happening.  The reason I note the RST
issue is that I've seen suggestions that a NAT can delete a session
mapping once it's seen a RST - just because it saw it doesn't mean the
remote host did though.  Given that we now don't want to mandate that a
NAT send a RST in response to a packet that has no session mapping (and
it will just silently discard the packet) then we end up with the NAT
being the cause of half-open connections in hosts and half-open sessions
in other NATs.  When I made the session persist for 2 MSL after a RST  I
saw a significant reduction in the rate of half-open connections that
I'd previously seen since this gave an opportunity for lost RSTs to be
resent in response to traffic from the remote host that never saw the
original.

* Also, when you tune just one of the NAT devices enroute, the 
* results are not necesaarily predictable. there may be 
* different cascading effects.

Agreed - I'd be happier if I could do my tests under more controlled
conditions but the size of the network required precludes this.

* > 1) Bad ICMP destination (port) unreachable or redirect messages.  
* > These are messages where the remote peer is apparantly no longer 
* > running where it was but the remote NAT fails to translate 
* the payload of the message.
* > As a result the packet is sent from the remote NAT to my 
* NAT but the 
* > ICMP message refers to a remote host that is clearly a 
* private IP of 
* > some host behind that NAT (typically 192.168.x.y) or occasionally 
* > 0.0.0.0.  These messages are usually for UDP sessions but I do see 
* > some for TCP too.
* >
* 
* [suresh] Couple of possibilities here. The NAT device may 
* have bad code for translation of ICMP error packets (or) the 
* NAT device retired the NAT Session too early and couldnt 
* locate the NAT session to match when the ICMP error message 
* came. The ICMP message may have come from an intermediate 
* device, not necessarily the end node.

Hmm - I guess there could be a race here between my UDP packet and the
error reply with the session timing out in between but then I'd expect
the NAT in question to discard a packet for which there's no mapping.
Forwarding it to me isn't helpful as I can't do anything with it.

* > 2) FIN/RST/ACK packets in response to SYN requests (this 
* may just be 
* > odd behaviour from a remote TCP stack though).
* > 
* 
* [suresh] Is it possible these are coming from remote NAT 
* instead of remote host?

My original guess was that this was the remote NAT but it could also be
odd behaviour from a remote host.  I'm still trying to determine this.

* > 3) RST packets being sent to the wrong local port.  In 
* these cases a 
* > remote NAT sends a FIN/ACK to close an existing session and 
* follows it 
* > immediately with a RST but the RST ends up sent to a random 
* port at my 
* > end despite the FIN/ACK having been sent the correct port 
* (this double 
* > approach to closing sessions is surprisingly common but 
* normally the 
* > port numbers are the same).  All subsequent FIN/ACKs from 
* my end get 
* > the same mismatched RST in response.
* > 
* [suresh] It is not normal for the NAT devices to directly 
* engage in FIN/ACK 3-way handshake, unless the NAT device is 
* doing transparant proxy (where the NAT device actually 
* terminates the TCP session). Some vendors do implement 
* transparant proxy on their NAT devices. But, transparant 
* proxy is different and not an integral part of NAT, as IETF 
* defines it. 

In this particular case though I would have expected the NAT to retain
the session until it saw a responding FIN at least.  The quirk here
though is that the RST is being sent to a different port on my side - I
can understand how a deleted session could cause it to be sent from a
different port but not to a different one.