Subject: [Ietf-behave] TCP Simultaneous Open, NATs and Bryan's paper

From: Saikat Guha <[email-masked]>
Date: Fri, 10 Dec 2004 02:55:50 -0800 (PST)

I am trying to interpret my results from testing, among others, the TCP
hole-punching technique proposed in Bryan's paper
(http://list.sipfoundry.org/archive/ietf-behave/msg00243.html). 
I came up with something odd. Maybe someone familiar with nat-check can
point out my mistake. Bryan?

First, regarding TCP simultaneous-open, it is known that Windows XP (pre
SP2) has a buggy implementation. Windows XP SP2 fixes this problem.
Other than XP does anyone know of other stacks that don't support it?
Regardless, my observation pertains to Windows XP pre SP2.

I am wondering if nat-check checks if two windows boxes each behind a
NAT can establish a TCP or if one side has to be linux. In my tests, if
both ends are windows then it doesn't seem to work as I'll explain
later. I went through the nat-check client code and I think the
discrepancy can be explained if one of the ends is linux. Has anyone
successfully used the technique in the paper above to establish TCP
between NAT'ed windows boxes?

In the following diagrams, A and B are windows XP pre-SP2 and N and M
are NATs. S is a linux box. Here is what I observed.

     A        N              M       B
      --------SYN----------->x
      <-----------------SYN----------
      --------SYNACK---------------->
      <-----------------ACK----------
      <----------------DATA----------
      --------SYNACK-----------------> (retransmission)
      <-----------------ACK----------
      <----------------DATA----------
      --------SYNACK-----------------> (retransmission)
        ...
      <-----------------ACK----------
      <----------------DATA----------
      --------RST-------------------->

This is with a server-socket and client-socket both open and bound to
the same port on A and similarly for B (as proposed by the paper). One
side (in this case B) gets a half-open pipe while the other end appears
to be horribly confused. This always happens if you can somehow arrange
for an incoming SYN while in SYN-Sent state.

Now here is what I am speculating could be happening with NAT-check.
Correct me if I am wrong.

Case 1: 
NAT is Cone ... so the S's SYN packet makes its way through to A. S =
server #3.

     A        N                      S
      <-----------------SYN----------
      --------SYNACK---------------->
      <-----------------ACK----------

the SYN packet is picked up by A's server socket, TCP is established
accept() on A succeeds and the subsequent connect() fails (since there
is already an established TCP between the respective addresses). Life is
good.

Case 2:
NAT is not cone. Server's SYN is dropped.

     A        N                      S
              x<------------SYN------
      -----------SYN---------------->
      <--------------SYNACK----------
      --------------ACK------------->

Since the server is linux, it correctly responds with SYNACK to A's SYN
and A sees a 3-way handshake. On A, connect() succeeds, accept() doesn't
(no incoming SYN). Again it works.

Essentially, the Windows XP bug is not triggered in either of the cases
since no incoming SYN is seen while in SYN-Sent state. There is a remote
case 3 where the bug could be triggered with nat-check.

Case 3:

     A        N                      S
                  +---------SYN------
      --------SYN-|----------------->
      <-----------+

This is a race condition that is not triggered, I believe, because the
client initiates a connect() long after the server has sent the SYN. In
the rare occasion that the race were to play out as above then I believe
nat-check too would trigger the bug ... though for that the server's
timing needs to be adjusted.

That said, I do agree that the technique mitigates the problem. This is
explained as follows:

Case 1:
Pure simultaneous open. No NATs. Both A and S have client sockets. No
server sockets anywhere.

     A                               S
      <------------------SYN---------
      -----------RST---------------->
      <------------------SYN---------
      -----------RST---------------->
                  +---------SYN------
      --------SYN-|----------------->
      <-----------+

bug triggered. That S keeps retrying increases the probability that a
SYN in encountered in the SYN-Sent state.

Case 2:
Bryan's technique. 

     A                               S
      <-----------------SYN----------
      --------SYNACK---------------->
      <-----------------ACK----------

The server socket on A soaks up the SYN and serves two purposes. 1)
Stops S from sending any more SYNs and establishes the connection. 2)
Stops A from sending out SYNs by failing the connect() thereby avoiding
SYN-Sent altogether.

Now all this breaks down in we throw in NATs and make both A and B
windows boxes, which is what I observed.
     
     A        N                      B
              x<------------SYN------
      -----------SYN---------------->

the bug is triggered (this time at B). 

Thoughts?

-- 
Saikat
Attachment:
signature.asc

---

From: Pyda Srisuresh <[email-masked]>
Date: Sun, 12 Dec 2004 22:37:19 -0800 (PST)

Saikat,

I believe, the traces you outline are taken between M & N, right? It woudl be
easy to debug if you have the actual traces (with full frames, along with
seqno/ackno) at A, and at B, in addition to between M & N. 

For example, is it possible that the Ackno. on the ACK frame coming from B does
not acknowledge the seqno. on the SYN (or) SYNACK from node A, for whatever
reason - introduced by the NAT devices enroute or the buggy end nodes. 

BTW, what is the bug you are referign to w.r.t Windows XP/SP1?

cheers,
suresh

> ATTACHMENT part 1.2 application/pgp-signature name=signature.asc
> _______________________________________________
> Ietf-behave mailing list
> Ietf-behave@xxxxxxxxxxxxxxxxxxx
> https://list.sipfoundry.org/mailman/listinfo/ietf-behave
> 

=====

---

From: Saikat Guha <[email-masked]>
Date: Sun, 12 Dec 2004 23:45:23 -0800 (PST)

> BTW, what is the bug you are referign to w.r.t Windows XP/SP1?

If TCP simultaneous open was attempted on a Windows 2K or XP/SP1 box
then the stack would get quite confused. Instead of taking the 
      SYN_SENT -> SYN_RCVD -> ESTABLISHED
transition, it would essentially take 
      SYN_SENT -> SYN_RCVD -> SYN_RCVD -> ...
where it would keep on sending out SYNACK packets before timing out even
after seeing SYNACK and ACK packets. This has been fixed in XP2 and 2k3
we have been told.

On a side note, in the draft I cited earlier, Section 4.3 bullet 1
describes the usual TCP simultaneous open as it works under linux, XP
SP2 etc. Which implementations correspond to bullet 2? 

>Subject: RE: Simultaneous 2-way opening
>Date: Tue, 16 Nov 2004 14:36:37 -0800
>X-MS-Has-Attach:
>X-MS-TNEF-Correlator:
>Thread-Topic: Simultaneous 2-way opening
>Thread-Index: AcTMI22Fq3Cx25pBSjeR9CgPiv3K8wABkGpw
>From: "Richard Draves" <richdr@xxxxxxxxxxxxx>
>To: "Paul Francis" <francis@xxxxxxxxxxxxxx>
>Cc: <minshall@xxxxxxx>
>X-OriginalArrivalTime: 16 Nov 2004 22:36:35.0921 (UTC) 
>FILETIME=[BAE1E410:01C4CC2C]
>
>As I noted below, they were already aware of the bug so we can't take
>credit for bringing it to their attention. I would be surprised to see
a
>regression on this in the future, since it's in their test suite now.
>
> > -----Original Message-----
> > From: Paul Francis [mailto:francis@xxxxxxxxxxxxxx]
> > Sent: Tuesday, November 16, 2004 1:30 PM
> > To: Richard Draves
> > Cc: minshall@xxxxxxx
> > Subject: RE: Simultaneous 2-way opening
> >
> >
> > My student tells me that service pack 2 of XP now has
> > symmetric open for TCP!
> >
> > Is this thanks to you????  If not, do you know the story?
> > I'm especially interested in whether this is a long-term
> > change (i.e. future versions of windows OS will allow
> > symmetric establishment).
> >
> > PF
> >
> >
> > At 07:07 PM 9/8/2004, Richard Draves wrote:
> > >I checked with the Windows TCP folks.
> > >
> > >First, they do now test the state transitions
> > >SYN_SENT->SYN_RCVD->ESTD->FIN_WAIT1->TIME_WAIT.
> > >
> > >Second, they had a regression in Win2k/XP. It should be
> > fixed in Win2k3
> > >and XPSP2.
> > >
> > >So I suggest that you test this again with XPSP2 and see if it
works.
> > >
> > >Rich
> > >
> > > > -----Original Message-----
> > > > From: minshall@xxxxxxx [mailto:minshall@xxxxxxx]
> > > > Sent: Wednesday, September 08, 2004 4:27 PM
> > > > To: Paul Francis
> > > > Cc: Richard Draves
> > > > Subject: Re: Simultaneous 2-way opening
> > > >
> > > > i didn't know p2p cared, but if it can be fixed, so much
> > the better!

-- 
Saikat
Attachment:
signature.asc

---

From: Bryan Ford <[email-masked]>
Date: Mon, 13 Dec 2004 16:32:20 -0800 (PST)

> Regardless, my observation pertains to Windows XP pre SP2.

I've heard of this long-standing Windows bug, but never knew the details of 
it, and I'm glad to hear it's finally been fixed.

> between NAT'ed windows boxes?

NAT Check as written normally performs its tests between the client machine 
you run it on and a fixed machine here in my lab that happens to run FreeBSD.  
So you're right - NAT Check doesn't provide a direct test for hole punching 
between two Windows machines.  Since my purpose in writing NAT Check was to 
test the properties of the NAT(s) on the path rather than to test the 
properties of the end hosts, this seemed reasonable, and it eliminated the 
need for the user to have TWO different clients between different firewalls 
to run NAT Check on simultaneously.  But obviously there are some hole 
punching issues that are caused by certain combinations of broken end host 
OSes, in addition to those issues caused by the NATs themselves. :)

> for an incoming SYN while in SYN-Sent state.

Building on what Suresh already suggested, here is what I suspect might be 
happening in the above figure:

1. A connects to B while listening on the same source port, sending a SYN to 
B, which B's NAT drops.  A's (buggy) TCP stack creates a TCP state data 
structure for this outgoing connection attempt and stores (among other 
things) the initial sequence number it chose for this first outgoing SYN in 
that data structure.

2. B connects to A while listening on the same source port, sending a SYN to 
A, which gets through A's now-open hole.  Since it's just a raw SYN rather 
than a SYNACK, A does not associate B's SYN with A's outgoing connection 
attempt, but instead accepts it via A's outstanding listen socket.  Now,
rather than re-using the TCP state structure A already created for its 
outgoing connect() attempt to B, A just creates a new TCP state structure 
representing an accept()ed server-side socket, and A's TCP stack probably 
chooses a new and potentially different initial sequence number for its 
SYNACK response.  The fact that A's first SYN might have a different initial 
sequence number from its subsequent SYNACK might theoretically cause 
problems, but it presumably isn't the direct cause of THIS specific problem 
since A's first SYN was never actually seen by B.

3. Now, A has TWO TCP state structures for the same source and destination 
endpoints - how does it deal with them?  Linux immediately cancels the first 
one (the one created for the outgoing connect()), leaving only the one for 
the accept() socket, which makes things work right.  BSD apparently never 
creates the two TCP state structures in the first place, which is even better 
IMHO.  But I could very easily envision the buggy Windows TCP stack doing 
something stupid like keeping the original connect()-initiated state 
structure around in some fashion, so that when B starts sending real data to 
A, A tries to associate this data with A's original connect() socket rather 
than with the accept() socket from which A's SYNACK was produced.  Since the 
TCP stack may well have chosen different initial sequence numbers for the two 
TCP state structures, A's TCP stack will in this case think that B is sending 
ACKs and data with totally inappropriate sequence numbers, and just keep 
retrying its SYNACK as shown above.

This is only a vague guess, of course - I've never seen the Windows TCP stack, 
so I really have no clue exactly how it's managing to screw this up.  I'm 
just happy to hear that they finally fixed it.

> Correct me if I am wrong.

I think your analysis of the reason the bug does not manifest itself when the 
client is a Windows machine but the server is FreeBSD or Linux is most likely 
exactly correct.

Cheers,
Bryan

---

From: Saikat Guha <[email-masked]>
Date: Mon, 13 Dec 2004 20:37:35 -0800 (PST)

Here is an annotated version of the trace. (Pcap files attached). 

A                N                  M                B
(192.168.1.137)  (128.84.154.97)    (128.84.154.99)  (192.168.1.1)

----SYN    SEQ:425311996----------->x 
<---SYN    SEQ:2632191422----------------------------
----SYNACK SEQ:425311996,ACK:2632191423------------->
<---ACK    SEQ:2632191423,ACK:425311997--------------
<---ACK    SEQ:2632191423,ACK:425311997, DATA--------
----SYNACK SEQ:425311996,ACK:2632191423-------------> (retransmit)
<---ACK    SEQ:2632191423,ACK:425311997-------------- (retransmit)
<---ACK    SEQ:2632191423,ACK:425311997, DATA-------- (retransmit)
----SYNACK SEQ:425311996,ACK:2632191423-------------> (retransmit)
<---ACK    SEQ:2632191423,ACK:425311997-------------- (retransmit)
<---ACK    SEQ:2632191423,ACK:425311997, DATA-------- (retransmit)
<---ACK    SEQ:2632191423,ACK:425311997, DATA-------- (retransmit)
----RST    SEQ:425311997---------------------------->

(Bryan's original mail didn't show up on the list, at-least for me, so I
am including relevant portions of his email)

> just happy to hear that they finally fixed it.

I haven't seen the Windows stack either, but I observed the same
behavior with just two facing client sockets (without the server
sockets). So I am inclined to write it off as a bug as opposed to a
premeditated implementation trait. The reason I bring this up is
because, to the best of my knowledge, this is the reason why
simultaneous-open has not succeeded under windows so far. Windows (SP2)
and Linux (and likely the BSDs too) can all now establish a TCP
connection with just two facing client sockets. Buggy implementations
and NATs blocking incoming SYNs aside, I want to know if there are other
stacks that don't  support simultaneous-open. 

The answers to the questions above help me in evaluating the merits of
the various TCP traversal techniques that have been proposed -- 1) pure
simultaneous open with two client sockets, 2) Bryan's technique with two
client and two server sockets, and 3,4) the two mentioned in the NUTSS
paper (http://www.cs.cornell.edu/People/francis/nutss-fdna.pdf). 

-- 
Saikat
Attachment:
A-N.pcap
Description: Binary data
Attachment:
B-M.pcap
Description: Binary data
Attachment:
N-M.pcap

---

From: Bryan Ford <[email-masked]>
Date: Tue, 14 Dec 2004 07:50:44 -0800 (PST)

> Here is an annotated version of the trace. (Pcap files attached).

Very interesting - the sequence numbers all look fine, so it presumably is 
indeed just utter internal confusion on the part of one of the Windows end 
hosts.  Thanks for taking the time to do this analysis!

> stacks that don't  support simultaneous-open.

A good question - I haven't heard specific reports of any, but that of course 
doesn't really mean much. :)

Cheers,
Bryan