Subject: [Ietf-behave] TCP without relays

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 7 Oct 2004 10:05:19 -0700 (PDT)

Mostly we've been discussing UDP issues, and I wanted to raise a TCP 
issue that I think is relatively important.

One of the objectives in the UDP work has been to allow it to be 
possible for protocols like STUN and TEREDO to work through NAT without 
requiring an external relay to act as a rendezvous point. Today, that's 
not really possible for TCP. There are hacks that people have tried, 
relying on simultaneous opens, to get this behavior to work for TCP, but 
they are extremely unreliable afaik.

However, I think it would be a tremendous accomplishment if we could 
allow for clients to establish direct TCP connections to each other in a 
reliable way. One of my real concerns are mumblings that I hear about 
p2p file sharing folks using UDP for file transfer, because they can use 
STUN to get UDP traffic to work directly, but not TCP. Suffice it to 
say, widespread usage of UDP protocols for large file transfer on the 
Internet would be nothing short of catastrophic, and must be avoided.

If there were a way to offer up a TCP-based solution, it would hopefully 
provide a way out.

Frankly, I'm not much of an expert of existing NAT treatment of TCP, and 
in particular, the state machines that they maintain and the specific 
behaviors of those machines which are problematic. Perhaps someone could 
contribute some experience here to help work on this problem?

Thanks,
Jonathan R.
--
Jonathan D. Rosenberg, Ph.D.                600 Lanidex Plaza
Chief Technology Officer                    Parsippany, NJ 07054-2711
dynamicsoft
jdrosen@xxxxxxxxxxxxxxx                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Bryan Ford <[email-masked]>
Date: Thu, 7 Oct 2004 11:54:07 -0700 (PDT)

> they are extremely unreliable afaik.

I've actually done some experiments in this direction, and it looks like the 
situation may not be as grim as I and others previously thought it was.  
There appear to be two main challenges to implementing NAT hole punching for 
TCP:

1. To make TCP NAT hole punching work reliably, the NAT must NOT respond to 
unsolicited incoming TCP connection attempts (i.e., incoming SYN packets) by 
sending back an RST - instead, the NAT must just silently drop the incoming 
SYN.  If the NAT silently drops incoming SYNs _and_ preserves port number 
identities (i.e., has "cone NAT" rather than "symmetric NAT" port allocation 
behavior), then TCP NAT hole punching can be done reliably and is no more 
timing-sensitive than UDP hole punching is.  My impression some time ago was 
that sending back an RST was common behavior for NATs, but on the basis of 
more recent experiments I'm suspecting that the more hole-punching-friendly 
behavior of just dropping unsolicited incoming SYNs on the floor might 
actually be much more common, making TCP-based NAT hole punching actually 
more-or-less practical.

2. Implementing TCP-based NAT hole punching in applications involves the use 
of obscure sockets API features that aren't exercised often and whose details 
therefore are apparently not very well standardized or portable.  In 
particular, each P2P application endpoint has to allocate a TCP listen socket 
_and_ potentially several outgoing sockets all bound to the same IP 
address/port on the local host.  In addition, the app has to manage all of 
these sockets in a nontrivial asynchronous fashion.  All of the important 
OS's I know of - e.g., Win32, linux, *bsd, OS X - support this functionality, 
but they all do it in subtly and annoyingly different ways.  This is 
obviously not an issue of NAT behavior, and is therefore not directly in the 
scope of this group - it's just an annoying issue for P2P apps that want to 
be portable, and perhaps calls for better POSIX network API standardization.

So in short, I now think that NAT hole punching for TCP is feasible and 
certainly should be addressed in this group, though my experimentation so far 
is limited and certainly not conclusive, especially as far as evaluating the 
behavior of existing NATs.  You can find my current NAT hole punching test 
code for both TCP and UDP on http://midcom-p2p.sourceforge.net/, although be 
warned that the NAT Check servers are currently broken (and it looks like the 
results database might now be broken too).  I just haven't had time lately to 
look into it in more detail.

Cheers,
Bryan

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 11 Oct 2004 03:02:41 -0700 (PDT)

I'm still not up to speed on this so let me ask things that are probably
obvious....

A is trying to set up a TCP connection to B and B is behind a NAT. We need B
to be able to "punch a hole" or more specifically be able to get the NAT to
create a mapping such that a SYN packet received by the NAT will get
forwarded to B. B also needs to be able to find out the public address/port
on the NAT that A needs to sent to. Is that a fair way of describing the
problem?

In the TCP whole punching approach, how does B find out the port that the
NAT allocated? Does it just assume that port preservation worked?

I'm going to propose something here which I suspect is a really bad hack and
not a good idea but, well perhaps it will lead to a better idea. Build NAT
such that when a SYN arrived an the public interface and there was no TCP
binding for it, the NAT looks at the UDP bindings and forwards the SYN to a
device that the UDP binding for that port. So in the case above, B could
open UDP port 5060 and be listening on TCP port 5060 then do something like
STUN to punch a UDP hole in the NAT and find out that the NAT mapped this
to, say address 1.2.3.4:6000, Now A sends a TCP SYN to 1.2.3.4:6000. The NAT
looks at it's UDP binding table and sends this as a TCP SYN to port 5060 on
B. Everything else happens as normal from this point on. Lots of details
hand waived here but you get the idea.

I have been thinking of other things that allow a STUN like operation using
TCP packets but I have a hard time imagining how to get these to interact
with existing client stacks in a clean way.

Cullen

On 10/7/04 11:54 AM, "Bryan Ford" <baford@xxxxxxx> wrote:

> _______________________________________________
> Ietf-behave mailing list
> Ietf-behave@xxxxxxxxxxxxxxxxxxx
> https://list.sipfoundry.org/mailman/listinfo/ietf-behave

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 11 Oct 2004 05:32:02 -0700 (PDT)

Hi, Cullen,

I haven't seen this mentioned previously, so I'm wondering - when NATs 
allocate a new port number, do they take the transport protocol into 
account?

I knew that IANA allocates both UDP and TCP for registered ports, even 
though most application protocols only use one or the other (DNS and 
SIP being the most obvious exceptions I can think of off the top of my 
head) and I'd always semi-wondered why - could we take advantage of 
this obscure fact to justify asking NATs to ignore the transport 
protocol in port mapping, so STUN would work, and then your host with 
modified applications but no modified transports uses TCP as normal 
(based on your question below, is this the goal?)?

... and one wonders what NATs do with SCTP, and what they will do with 
DCCP, but I'll follow up with Aaron and Lyndon and report back ...

Spencer

I'm still not up to speed on this so let me ask things that are 
probably
obvious....

A is trying to set up a TCP connection to B and B is behind a NAT. 
We need B
to be able to "punch a hole" or more specifically be able to get the 
NAT to
create a mapping such that a SYN packet received by the NAT will get
forwarded to B. B also needs to be able to find out the public 
address/port
on the NAT that A needs to sent to. Is that a fair way of describing 
the
problem?

In the TCP whole punching approach, how does B find out the port 
that the
NAT allocated? Does it just assume that port preservation worked?

I'm going to propose something here which I suspect is a really bad 
hack and
not a good idea but, well perhaps it will lead to a better idea. 
Build NAT
such that when a SYN arrived an the public interface and there was 
no TCP
binding for it, the NAT looks at the UDP bindings and forwards the 
SYN to a
device that the UDP binding for that port. So in the case above, B 
could
open UDP port 5060 and be listening on TCP port 5060 then do 
something like
STUN to punch a UDP hole in the NAT and find out that the NAT mapped 
this
to, say address 1.2.3.4:6000, Now A sends a TCP SYN to 1.2.3.4:6000. 
The NAT
looks at it's UDP binding table and sends this as a TCP SYN to port 
5060 on
B. Everything else happens as normal from this point on. Lots of 
details
hand waived here but you get the idea.

I have been thinking of other things that allow a STUN like 
operation using
TCP packets but I have a hard time imagining how to get these to 
interact
with existing client stacks in a clean way.

Cullen

---

From: Bryan Ford <[email-masked]>
Date: Mon, 11 Oct 2004 06:49:32 -0700 (PDT)

On Sunday 10 October 2004 13:25, Cullen Jennings wrote:
> A is trying to set up a TCP connection to B and B is behind a NAT. We need
> B to be able to "punch a hole" or more specifically be able to get the NAT
> to create a mapping such that a SYN packet received by the NAT will get

Suppose both A and B are behind NATs, and C is a rendezvous server.

1. Both A and B have active TCP connections to C.  C knows both A's (public 
IP,port#) for the A-C connection and B's (public IP,port#) for the B-C 
connection.

2. In addition to these connections to C, both A and B have active _listen_ 
sockets on _exactly the same private local TCP port numbers_ as the local 
port numbers they are using for the A-C and B-C connections, respectively.  
(Setting this up requires the use of SO_REUSEADDR/SO_REUSEPORT.)

3. A asks C for help setting up a connection to B.  C returns to A the (public 
IP,port#) for B, and also forwards a request to B with the (public IP,port#) 
of A.  We're still talking about the same port numbers as the ones being used 
for the existing A-C and B-C connections.

4. A initiates a new outgoing TCP connection to B's (public IP,port#) pair 
_from the same private TCP port number that its existing A-C connection and 
its existing listen socket use_, and B likewise initiates a new outgoing TCP 
connection _from the same local TCP port number as its other sockets_ to A's 
(public IP,port#) pair.  Again, SO_REUSEADDR/SO_REUSEPORT required.

5. The SYN packets for each outgoing TCP connection will punch a hole in the 
local firewall to the other client's (public IP,port#) pair.  Assuming the 
NAT preserves port identities (i.e., is a cone NAT and not a symmetric NAT), 
the NAT will assign the same (public IP,port#) pairs for the new A-B and B-A 
connections as it did for the original A-C and B-C connections.  One or more 
SYNs might get dropped at the opposite firewall depending on the timing, but 
before long (assuming neither NAT sends back an RST), both holes will have 
been punched and the SYNs will get through all the way.

6. Now one of two things might happen, depending on the timing and the 
operating systems involved.

        Case (a): One client's (say A's) SYN reaches the other client (B), and 
since 
B has an active listen socket at the targetted port, B accept()s this 
connection, and we get a usable TCP connection established in more or less 
the usual symmetric fashion.  But now since B's not-yet-established outgoing 
connection attempt to A was trying to use exactly the same (src IP, src port, 
dest IP, dest port) tuple as this new TCP connection that was just 
established, B's outgoing TCP connection attempt fails with a "port in use" 
or similar error.  No matter; we have the TCP connection we want.

        Case (b): One client's (say A's) SYN reaches the other client (B), and 
B's OS 
notices that it also has an outstanding, outgoing connection attempt to A 
with exactly matching IP addresses and port numbers.  B therefore recognizes 
that a "simultaneous TCP open" is being attempted (as per the TCP spec), and 
establishes an active TCP connection in that fashion.  From the viewpoint of 
A's and B's application code, _both_ of their outgoing TCP connection 
attempts succeeded, forming a single connection, and nothing at all happened 
with their listen sockets.  Again, we have the TCP connection we want.

In my testing, case (a) is the usual behavior under Linux, while case (b) is 
usual for BSD.  Either way it works though.  I can't remember what the usual 
case was for Windows, but I think it worked too in one way or another.  I 
tried to get it working on Mac OS X, but ran into some stupid API 
incompatibilities with my code that I just didn't have time to fix - I think 
it should work fine too once those API portability issues are resolved.

The main point to notice here is that what is happening is really _exactly_ 
the same as for UDP hole punching, just using TCP instead, with a lot of help 
from the SO_REUSEADDR/SO_REUSEPORT options.  Which is really what we want.  
And the success of this trick for TCP depends on _exactly_ the same 
conditions as the success of the trick for UDP (namely cone NAT behavior), 
with the minor additional proviso that firewalls must simply drop 
unrecognized incoming SYNs on the floor and not send back RSTs.  Which 
apparently a lot of firewalls if not most already do.

> I'm going to propose something here which I suspect is a really bad hack
> and not a good idea but, well perhaps it will lead to a better idea. Build
> NAT such that when a SYN arrived an the public interface and there was no
> TCP binding for it, the NAT looks at the UDP bindings and forwards the SYN
> to a device that the UDP binding for that port. So in the case above, B
> could open UDP port 5060 and be listening on TCP port 5060 then do
> something like STUN to punch a UDP hole in the NAT and find out that the
> NAT mapped this to, say address 1.2.3.4:6000, Now A sends a TCP SYN to
> 1.2.3.4:6000. The NAT looks at it's UDP binding table and sends this as a
> TCP SYN to port 5060 on B. Everything else happens as normal from this
> point on. Lots of details hand waived here but you get the idea.

I thought of hacks like that too at some point (I think some existing NATs 
even couple the TCP and UDP port spaces, so this hack might even work 
already), but given that a cleaner alternative is available I really don't 
think this route should be taken.

Bryan