Subject: RE: [Ietf-behave] Catalogue of problems with deployed NATs

From: "Dave Hudson" <[email-masked]>
Date: Mon, 19 Dec 2005 05:59:41 -0800 (PST)

Hi Suresh,

Thanks for the pointers.  I was familiar with most of the documents you
mentioned but I'd not seen anything that specifically discussed many of
the things I've been seeing.  While I've seen a lot of discussion about
general problems with NATs I concluded I was seeing other
implementation-related problems but haven't been able to find so much
discussion of this.

Essentially much of what I've been seeing appear to be a consequence of
the high volumes of NAT sessions that can be established by some P2P
apps.  I'm also seeing the effect of the high levels of flux seen in
many of the P2P file-sharing networks where nodes in the network
disappear (although in some cases I suspect this may be the result of
symmetric NATs or NATs that exhibit symmetric behaviour under some
circumstances).

I normally find that after running for an hour or so (on a slow DSL
link) I get more than 5 egressing and 5 ingressing new sessions per
second (often this number is more than 20 in each direction for periods
of a few seconds) so when coupled with a recommendation that TCP
sessions in the Syn-Sent state hold state for 4 minutes and UDP sessions
hold state for 5 minutes this can generate around 2500 sessions.
Coupled with this I may have 700 to 800 live sessions and another 1000
or so half-open (or half-closed :-)) TCP sessions.

At one point I was seeing far more half-open TCP sessions but I then
adjusted the timings in the NAT I'm working with to 4 minutes for
Syn-Sent, Syn-Received, Time-Wait, Last-Ack, while making them 130
minutes for Established, Fin-Wait-[1/2] and Close-Wait.  I also modified
the code to allow sessions that had seen a RST to stay in a "closed"
state that would allow packets to pass for 4 minutes because it was
amazing how many RST packets were apparently not being handled (I
suspect many were being dropped in transit as a result of too much
traffic being sent to remote peers in bursts), leading to situations
where one peer would send a RST and then 2 or 3 minutes later the other
peer would send (and then continue to resend FIN/ACK);  The extra time
in this closed state allows for RST packets to be resent in response to
packets from the peer that didn't see the original RST.

Anyway here are some of the things I've observed so far (I've seen all
of these with a surprisingly high frequency - typically I see one of
these every 4 to 5 seconds during tests where I send about 70 packets
per second and typically receive 60 per second):

1) Bad ICMP destination (port) unreachable or redirect messages.  These
are messages where the remote peer is apparantly no longer running where
it was but the remote NAT fails to translate the payload of the message.
As a result the packet is sent from the remote NAT to my NAT but the
ICMP message refers to a remote host that is clearly a private IP of
some host behind that NAT (typically 192.168.x.y) or occasionally
0.0.0.0.  These messages are usually for UDP sessions but I do see some
for TCP too.

2) FIN/RST/ACK packets in response to SYN requests (this may just be odd
behaviour from a remote TCP stack though).

3) RST packets being sent to the wrong local port.  In these cases a
remote NAT sends a FIN/ACK to close an existing session and follows it
immediately with a RST but the RST ends up sent to a random port at my
end despite the FIN/ACK having been sent the correct port (this double
approach to closing sessions is surprisingly common but normally the
port numbers are the same).  All subsequent FIN/ACKs from my end get the
same mismatched RST in response.

4) Remote NAT bindings change mid-session after some period of
inactivity.  My guess is that the remote NAT times-out its binding part
way through a TCP session and suddenly traffic that was coming from one
port suddenly starts to arrive from a different port (this is similar to
3 but in 3 the port binding for my host gets changed).  I'd guess that
these remote NATs aren't really tracking the TCP state transitions as
anything stateful probably wouldn't allow a new session for anything
other than SYN-related transactions.

5) Inconsistent port binding when remote systems use port forwarding.
In these cases the remote NAT has a port forwarding rule to ensure that
ingressing traffic will be sent to some specific service, however any
egressing traffic from the remote NAT appears from a different (but
consistent) port number (this effectively renders it symmetric even
though it may nominally be a NAT that would normally employ a persistent
port binding).  This is particularly noticeable as a variant of 4.

One problem here is that by not sending RST packets in response to some
of the resulting bad TCP packets we avoid the possibility of being a
reflector for a DDoS attack but do end up with far more bad traffic.
I've experimented with a compromise that sends RSTs in response to bad
packets from hosts with which we already have an active session and saw
about a 25% drop in packets rejected at my end.

Regards,
Dave

* -----Original Message-----
* From: ietf-behave-bounces@xxxxxxxxxxxxxxxxxxx 
* [mailto:ietf-behave-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of 
* Pyda Srisuresh
* Sent: 19 December 2005 05:06
* To: Dave Hudson; ietf-behave@xxxxxxxxxxxxxxxxxxx
* Subject: Re: [Ietf-behave] Catalogue of problems with deployed NATs
* 
* Dave,
* 
* RFC 3027 was done a while back documenting compliations with NATs.
* draft-ford-behave-top-00.txt documents complications with the 
* use of NATs in certain topologies. The ID has expired. We 
* will get a new rev out shortly.
* Outside of these, I dont know of any other. 
* 
* Additional comments inline below.
* 
* cheers,
* suresh
* 
* --- Dave Hudson <dhudson@xxxxxxxxxx> wrote:
* 
* > Does anyone have any pointers to documents/websites that describe 
* > problems seen by widely-deployed NATs?  I've been looking into some 
* > very odd behaviour I've seen from systems using P2P file-sharing 
* > applications and it's not at all unusual for me to see 50 or 60 
* > packets per minute arriving at my test system (around 0.5% of all 
* > packets) where there's very strong evidence of something 
* wrong in the 
* > translations being applied by remote NATs.
* > 
* > Many of these problems are leading to me seeing excessively high 
* > numbers of TCP connections left in effectively a half-open 
* state (e.g. 
* > with a 2 hour timeout applied to traffic in Establish, Fin-Wait-1, 
* > Fin-Wait-2 and Close-Wait I can see 2k or 3k such 
* connections after an 
* > hour or two even when only using at a slow DSL connection) with a 
* > consequent impact on overall performance.
* > 
* 
* [suresh] Sounds like, you are saying you noticed a lot of 
* half-open or half-close conenctions on the NATs. This may be 
* specific to the file sharing Peer-to-peer (P2P) application 
* you are using and the algorithm the application is using to 
* traverse NATs. I dont know what the problem is. But, I 
* suspect, the problem is to do with symmteric NATs being enroute.
* 
* > I'm investigating strategies that will help me reduce both 
* the amount 
* > of bad traffic that I see arriving and the number of half-open 
* > connections but a more comprehensive list of possible failure modes 
* > would help (if there is one).
* > 
* [suresh] You might want to take a look at 
* draft-srisuresh-behave-p2p-state-01.txt for a description of 
* the various algorithms in use by P2P apps for NAT traversal.
* 
* > Failing that I'll be happy to summarise problems I have observed if 
* > anyone's interested.
* > 
* [suresh] That sounds good in any case. Thanks.
* 
* cheers,
* suresh
* 
* > 
* > Thanks,
* > Dave
* > _______________________________________________
* > Ietf-behave mailing list
* > Ietf-behave@xxxxxxxxxxxxxxxxxxx
* > https://list.sipfoundry.org/mailman/listinfo/ietf-behave
* > 
* 
* 
* 
* _______________________________________________
* Ietf-behave mailing list
* Ietf-behave@xxxxxxxxxxxxxxxxxxx
* https://list.sipfoundry.org/mailman/listinfo/ietf-behave
*

---

From: Saikat Guha <[email-masked]>
Date: Mon, 19 Dec 2005 10:09:29 -0800 (PST)

On Mon, 2005-12-19 at 05:59 -0800, Dave Hudson wrote:
> 1) Bad ICMP destination (port) unreachable or redirect messages.  

I found a lot of NATs filter any ICMP message that is not a echo
request/response, or frag-required. I've heard of NATs that
mis/don't-translate ICMP payloads, but I never picked them up in my
tests -- perhaps a bug in my test.

> 3) RST packets being sent to the wrong local port.  

Could possibly be the bug WindowsXP SP1 and earlier. Once (If) you
trigger simultaneous open, the TCP state-machine jumps out of the window
and the stack runs amok returning any packet it pleases. This could all
to already broken NAT behavior.

> inactivity.

Some NATs have very short timers (<15min) for TCP connections in the
established state. See http://nutss.net/stunt-results.php?sort=-33

> 5) Inconsistent port binding when remote systems use port forwarding.

Static port mappings behavior is implemented differently by different
vendors. Some use it only for inbound initiations, while others for
both. David Barrett raised this issue earlier I think; the final
resolution was to stay silent about it in the UDP NAT RFC. Consequently,
this behavior will remain vendor specific.

cheers,
-- 
Saikat
Attachment:
signature.asc

---

From: Pyda Srisuresh <[email-masked]>
Date: Tue, 20 Dec 2005 08:45:11 -0800 (PST)

Hi Dave, 

It seems you can control some of the NAT devices enroute, but not all in this
setup. Is that right? Results are not predictable when you change settings on
just one of the NAT devices.

Also, do you have know the NAT traversal algorithm being employed by the P2P
application? If so, what is it? 

My comments below inline.

cheers,
suresh

> implementation-related problems but haven't been able to find so much
[suresh] OK.

> symmetric NATs or NATs that exhibit symmetric behaviour under some

[suresh] Right. It is possible that some of the NAT devices are hitting their
resource limits (say, the number of NAT Sessions) with P2P applications,
especially when the P2P apps are aggressive with multiple session initiations. 

> or so half-open (or half-closed :-)) TCP sessions.
[suresh] Right. The NAT devices could indeed be hitting resource limits, if the
session initiation rate is high (say, 40 per second) and TCP session are being
left in half-open state for long periods of time, without being established.

What is the algorithm being used to setup ingress sessions? Is this
simultaneous hole punch (or) a port prediction based algorithm? Saikat and Paul
Francis had run several experiments under the NUTSS project and published their
results in "Characterization & measurement of TCP traversal through NATs and
Firewalls". You might want to take a look at this. As I recall, the best case
for P2P session setup time was in the order of 200 msec (i.e, 5 sessions/sec).
But, the median and average were much less - no more than 3 sessions per sec.

> packets from the peer that didn't see the original RST.
[suresh] Once a TCP session is established, RST handling for P2P sessions is
same as for regular client-server sessions. The dropping of RST packets may be
due to unusual traffic volume.

Also, when you tune just one of the NAT devices enroute, the results are not
necesaarily predictable. there may be different cascading effects.

> 0.0.0.0.  These messages are usually for UDP sessions but I do see some

[suresh] Couple of possibilities here. The NAT device may have bad code for
translation of ICMP error packets (or) the NAT device retired the NAT Session
too early and couldnt locate the NAT session to match when the ICMP error
message came. The ICMP message may have come from an intermediate device, not
necessarily the end node.

> behaviour from a remote TCP stack though).

[suresh] Is it possible these are coming from remote NAT instead of remote
host?

> same mismatched RST in response.
[suresh] It is not normal for the NAT devices to directly engage in FIN/ACK
3-way handshake, unless the NAT device is doing transparant proxy (where the
NAT device actually terminates the TCP session). Some vendors do implement
transparant proxy on their NAT devices. But, transparant proxy is different and
not an integral part of NAT, as IETF defines it. 

This characteristic may be specific to the particular NAT vendor. 

> other than SYN-related transactions.

[suresh] Right, it is a problem if NAT drops the state (i.e., NAT Session)
midway through the session. Usually this results in either silent dropping of
the expired session packets or a RST being sent. As you said, most NAT device
wait for a SYN to establish the next session state.

> port binding).  This is particularly noticeable as a variant of 4.
[suresh] By port forwarding for an ingress session, let us say  you are
refering to the tuple of (Ext-node-ip, Ext-Node-port, NAT-IP, NAT-ingress-port)
in the Internet realm. Now, when you say egress traffic, are you referign to
the traffic emanating for this session (or) any session, but from the same
endpoint that represents (NAT-IP, NAT-Ingress-Port)?

Given that you are talking about P2P applications (with multiple simlutaneous
sessions from the same endpoint), I suspect, you are refering to the latter. In
that case, it is normal behavior for symmetric NATs. It sounds like, the NAT
traversal algorithm being adapted by the application doesnt work for multiple
simultaneous sessions.

cheers,
suresh