Subject: [MMUSIC] ICE-tcp implementation concerns

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 05 Jul 2007 18:21:01 -0400

Most folks probably missed this, but deep within one of the threads on 
core ICE, was an important issue raised by Guylain Lavoie.

With ICE-tcp, if you have passive or SO candidates, the basic sequence 
of events is as follows:

1. open a tcp connection to your stun server
2. send Binding Request, learn a mapped address A:B
3. send an INVITE with A:B as a passive or SO candidate
4. be prepared to receive TCP incoming connection requests on the IP and 
port from which you connected to the STUN server

Guylain had indicated that, with BSD sockets at least, this could not be 
implemented. He said:

> It seems that you cannot do a listen and then do a
> connect on the same socket. It does not work. At least on the platforms
> I tested (Linux and Windows). The only way to achieve this is by binding
> two sockets to the exact same address/port and this required special
> support from the OS (SO_REUSEPORT and/or SO_REUSEADDR combinason). From
> what I read, the needed fonctionality is not available on Linux. You are
> right that at the TCP level, nothing should prevent us from doing this.
> However, it's the API level that does not follow.

If this is true, it would mean that ICE-tcp is basically useless. That 
would be bad. However, this same problem would exist in any solution for 
direct tcp through NAT (STUNT for example), so it didn't seem right. I 
sent a note to Saikat Guha, and got the following reply:

> The following (preudo) code works, and has been tested to do TCP
> traversal on Windows, Linux and OSX:
> 
> for i in 1 to 100
>    sock_i = socket()
>    set(sock_i, SO_REUSEPORT)
>    bind(sock_i, local)
> 
> listen(sock_0)
> connect(sock_1, stun)
> connect(sock_2, remote_a)
> connect(sock_3, remote_b)
> ...
> 
> The key thing is that all the sockets must be bound to the local port
> before listen is called. So the pool of potential sockets to use for
> ICE-TCP would need to be pre-allocated. Once you exhaust the pool, a
> whole new pool needs to be created (with its own listen socket, and N
> connect sockets).

So, it seems this can be made to work, though its a bit awkward. But, 
such is the penalty. I don't think there is really an alternative. 
However I wanted to invite comments on this.

I think it might be useful having an informational appendix in ice-tcp 
which mentions this issue. Otherwise this is going to be a huge problem 
for implementors.

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                   600 Lanidex Plaza
Cisco Fellow                                   Parsippany, NJ 07054-2711
Cisco Systems
[email-masked]                              FAX:   (973) 952-5050
http://www.jdrosen.net                         PHONE: (973) 952-5000
http://www.cisco.com

_______________________________________________
mmusic mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/mmusic

---

From: "Guylain Lavoie" <[email-masked]>
Date: Thu, 5 Jul 2007 23:05:09 -0400

Hi Jonathan,

Frankly, I am very happy to learn that there is finally one way to make
this work, albeit not very obvious. The effort deployed for this
document will not have been done in vain.

Having to implement our ICE stack for many platforms, I was beginning to
be a bit worried about how we could manage to make this work. Thank you
for this information.

You are right, adding an appendix to explain this is a very good idea.

Best Regards,
Guylain

-----Original Message-----
From: Jonathan Rosenberg [mailto:[email-masked]] 
Sent: July 5, 2007 6:21 PM
To: IETF MMUSIC WG
Cc: Saikat Guha
Subject: [MMUSIC] ICE-tcp implementation concerns

1. open a tcp connection to your stun server 2. send Binding Request,
learn a mapped address A:B 3. send an INVITE with A:B as a passive or SO
candidate 4. be prepared to receive TCP incoming connection requests on
the IP and port from which you connected to the STUN server

implemented. He said:

> It seems that you cannot do a listen and then do a connect on the same

> socket. It does not work. At least on the platforms I tested (Linux 
> and Windows). The only way to achieve this is by binding two sockets 
> to the exact same address/port and this required special support from 
> the OS (SO_REUSEPORT and/or SO_REUSEADDR combinason). From what I 
> read, the needed fonctionality is not available on Linux. You are 

-Jonathan R.

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Fri, 6 Jul 2007 09:45:24 +0300

> > connect(sock_3, remote_b)

Unlikely to work as is. Linux nor Windows even define SO_REUSEPORT at all. 
Supposedly, that would rather be SO_REUSEADDR on every platform, and 
SO_REUSEPORT in addition for BSDs...

I'll have a look.

-- 
Rémi Denis-Courmont

---

From: "Guylain Lavoie" <[email-masked]>
Date: Fri, 6 Jul 2007 09:16:37 -0400

Hi Jonathan,

While thinking about this, I found one additional thing that you should
add to your appendix.

As I said, the technique is not obvious, but if it works, the
implementation should be doable without to much sacrifice on the
architecture.

However, while thinking more about binding all those sockets at once, I
stumble upon one less trivial issue. I am thinking about SIP forking
where additional sockets will need to be bound much later in the
process. We will need to create a poll of sockets already bound to the
same port and use them later if there is forking. For example, we could
bind a certain amount of additional sockets, that may not be sufficient
or may be way too much, wasting valuable resources. This problem does
not affect Active candidates because the port is not important. Neither
does it affect Passive candidates because we are listening of the socket
and one server socket is enough for this purpose. The issue show up with
the SO candidates. This issue will have a much bigger impact on the
implementation architecture and than I originally thought, at least in
our implementation. 

It would be good to discuss this also in the appendix.

Best Regards,
Guylain

Guylain Lavoie
Technical Product Manager
M5T, a Division of Media5 Corporation
Web: http://www.m5t.com
Email: [email-masked]
Tel: +1-819-829-1008, ext: 662
Fax: +1-819-829-1342

-----Original Message-----
From: Guylain Lavoie [mailto:[email-masked]] 
Sent: July 5, 2007 11:05 PM
To: Jonathan Rosenberg; IETF MMUSIC WG
Cc: Saikat Guha
Subject: RE: [MMUSIC] ICE-tcp implementation concerns

Hi Jonathan,

Best Regards,
Guylain

-----Original Message-----
From: Jonathan Rosenberg [mailto:[email-masked]]
Sent: July 5, 2007 6:21 PM
To: IETF MMUSIC WG
Cc: Saikat Guha
Subject: [MMUSIC] ICE-tcp implementation concerns

-Jonathan R.

---

From: Saikat Guha <[email-masked]>
Date: Thu, 5 Jul 2007 16:09:00 -0700

On Thu, 2007-07-05 at 18:21 -0400, Jonathan Rosenberg wrote:
> > connect on the same socket.

Just to clarify, pure TCP S-O can be triggered using the BSD api simply
using: bind() then connect(). For pure S-O listen() doesn't need to be
called. The connect will make the OS send the SYN, and when it receives
a SYN in response, the OS stack with do S-O and the userspace will be
none the wiser.

The problem with this is getting the SYNs to cross in the network. And
if either SYN reaches the remote endpoint before connect is called on
that endpoint, RSTs are generated by the OS stack and the whole thing
breaks down.

> >    set(sock_i, SO_REUSEPORT)
                   ^^^^^^^^^^^^
My mistake. This should be REUSEADDR. REUSEPORT isn't necessary.
> > whole new pool needs to be created (with its own listen socket, and N

Just to clarify, the whole business of calling listen on sock_0 above is
to catch a SYN that didn't cross in the network. 

If the SYN's cross:
- TCP S-O happens
- one of the connect()s on the local host succeeds, 
  and one of the connect()s on the remote hosts succeeds.

If SYNs don't cross:
- Regular TCP happens
- On one host, one of the connect()s succeeds
- On the other host, a new socket can be accept()ed from sock_0

So more implementation advise:

After calling listen() to catch an un-crossed SYNs, and connect()s to
attempt either S-O or 3WHS ... an implementation needs to
*asynchronously* wait for either a successful connect() or a successful
accept(). 

-- 
Saikat

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sat, 07 Jul 2007 16:27:13 -0400

Yes, I added text on this in the appendix. The recommendation is that 
the number of sockets to allocate should be equal to 25, which 
represents a worst case of five forked receipients, each with five SO 
candidates. As you point out, this problem is only for SO.

-Jonathan R.

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Mon, 9 Jul 2007 09:59:23 +0300

> candidates. As you point out, this problem is only for SO.

That's quite big though.

select() is limited to 64 sockets (by default) on a bunch of platforms (at 
least Winsock, Cygwin, Symbian). You might as well add an implementation hint 
to increasing FD_SETSIZE if more than one session may be pending at a time.

-- 
Rémi Denis-Courmont