Subject: [P2PSIP] Peer protocol, DHT and running code

From: Enrico Marocco <[email-masked]>
Date: Sun, 17 Jun 2007 10:50:53 +0200

Folks,

Emil and I have just submitted two drafts specifying the peer protocol
(XPP) and the overlay algorithm (CAN based) we have implemented in the
last version of sipdht (http://sipdht.sourceforge.net/sipdht2/).  Till
they appear in the repository, you can find them at the following links
(just s/txt/html/ or s/txt/xml/ if you prefer html or xml):
http://sipdht.sourceforge.net/drafts/draft-marocco-p2psip-xpp-00.txt
http://sipdht.sourceforge.net/drafts/draft-marocco-p2psip-xpp-pcan-00.txt

We hope this contribution will stimulate discussion on some key issues
we've tried to address, among which:

+ transport and encoding for the peer protocol;
+ DHT algorithm;
+ peer joins.

On the latter, we propose a mechanism to let nodes join the overlay only
when they are explicitly invited by existing peers.  Such a "passive"
approach (yeah, Dean anticipated that the term was going to get
overloaded) has some benefits which deserve to be investigated.

Needless to say, any feedback is very very appreciated.

-- 
Ciao,
Enrico
_______________________________________________
P2PSIP mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/p2psip

---

From: Philip Matthews <[email-masked]>
Date: Tue, 19 Jun 2007 15:30:07 -0400

Here are some questions and comments on draft-marocco-p2psip-xpp-00.txt
I haven't read the other document yet, so my apologies if these  
questions
are answered in the other document.

1) It seems that XPP is a transport protocol.
That is, it provides reliable, in-order delivery of messages, along  
with segmentation
and re-assembly so messages will fit within the Path MTU.
However, unlike TCP which is stream-oriented, XPP seems to be message- 
oriented.
In that regards, XPP seems more like SCTP.
Though XPP runs over UDP, UDP seems to be just used for encapsulation,
and XPP could run directly over IP if there are no NATs in the path.
Is this correct?

2) It seems that an XPP session is intended to be set up using SIP  
signaling,
in a way similar to how SIP can set up RTP sessions.
That is, the SDP body of the SIP message, rather than specifying RTP,  
would specify XPP.
Furthermore, it seems that you expect ICE to used with SIP to handle  
the NAT
traversal aspects, including the selection of a transmission path.
Is this correct?      (I will assume "yes" in the rest of my  
questions below).

3) Why do you think we need XPP as a new transport protocol, rather  
than using TCP or SCTP?
You argue that UDP is easier than TCP to set up through NATs, but  
Saikat Guha has shown
that TCP connections can be reliably set up through NATs in about 90%  
of the time, and 100%
of the time through common pairs of NATs, so I don't really buy this  
argument. Especially when
the BEHAVE recommendations on TCP will be published as an RFC soon.
Moreover, one can easily work around this problem if desired by  
running TCP over UDP,
in the same way that you run XPP over UDP.

4) If XPP is set up using ICE, then why is a simultaneous session  
establishment important?
If ICE has already handled the UDP hole-punching, then the  
simultaneous nature
is unimportant.

5) If XPP is set up using ICE, then presumably the NAT bindings are  
kept alive using
STUN indications as specified in ICE. Thus the keepalives in XPP are  
used for something
other than keeping NAT bindings open. My guess is they are just there  
to make sure the far-end
software is still alive. Am I correct?

6) XPP uses 16-bit endpoint identifiers. What are these? I am  
guessing that these are not
peer IDs  as described in the Concepts draft, since they are so  
short. Is this correct?
How are these allocated?

7) Is an ACK a response to a request? Or must a response be something  
else?
Say I wanted to transport a file from one endpoint to another
(most IM clients today support the ability to transfer a file to the  
client that one is talking or IMing with)
-- how would I do this?

8) Are you assuming that the security is handled by the signaling  
protocol (e.g., SIP),
so there is no need to handle this at the XPP layer?

9) How and when do you see XPP sessions being set up? That is, say  
peer X and peer Y are
behind (different) NATs and there is no XPP session between them? How  
does this session get
set up? How do the signaling messages get transported from X to Y?

10) It seems that there is no provision in XPP for XPP messages to be  
transported hop-by-hop.
That is, if peer X and peer Y want to transport a message using XPP,  
they have to first set up an
XPP session.
How do you see this working if X and Y just want a quick message  
exchange? Say X wants to send
a short 40-byte request and get back an equally short answer. Does X  
have to set up an XPP session
with Y first?  It seems to me that session setup is a pretty heavy- 
weight operation (e.g., send one or more
STUN requests and receive one or more STUN responses to gather ICE  
candidates, then send a  SIP INVITE,
get back a 100 TRYING, get back a 200 OK, send an ACK, send ICE  
connectivity checks to various addresses,
receive ICE connectivity checks on various addresses, send and  
receive a XPP handshake,
then finally send the 40-byte query and get back the 40-byte response).

- Philip

---

From: "Adam Fisk" <[email-masked]>
Date: Tue, 19 Jun 2007 18:17:07 -0400

I'm wondering if you considered the "Reliable Data Protocol (RDP)" for the
peer protocol.  It's at http://www.faqs.org/rfcs/rfc1151.html.

Seems like an oldie but a goodie with many of the same characteristics,
although I don't believe there's the option to turn off reliability.  Are
you thinking something like XPP would be easier to implement?  RDP at least
has addressed all the thorny reliability details.

It seems to me the session negotiation issues could be worked out for any of
these since ICE is designed to be relatively transport agnostic with a
little SDP fiddling.

I agree the NAT hold punching is a little more straightforward with UDP
since you just have more control over the timing of packets, making
something along these lines worthwhile.

-Adam

---

From: Philip Matthews <[email-masked]>
Date: Tue, 19 Jun 2007 20:06:20 -0400

Adam:

I was not familiar with RDP, but I took a quick look.
 From what I can see, RDP is an old protocol that who state is  
roughly comparable to the original specification of TCP.
As such, it does not have all the enhancements that TCP has undergone  
over the years (congestion control, etc.).
My guess is that the IESG would strongly urge us to look at TCP,  
SCTP, and DCCP (the latter two having been
invented with all the existing TCP work in mind) before trying to  
resurrect RDP.

Furthermore, using a transport protocol other that TCP or UDP means  
that we have to do all the work to define
how SIP and other protocols get transported over this transport  
protocol. I don't think we should underestimate
how much time that would take.

- Philip

On 19-Jun-07, at 18:17 , Adam Fisk wrote:

> I'm wondering if you considered the "Reliable Data Protocol (RDP)"  
> for the peer protocol.  It's at http://www.faqs.org/rfcs/rfc1151.html.
>
> Seems like an oldie but a goodie with many of the same  
> characteristics, although I don't believe there's the option to  
> turn off reliability.  Are you thinking something like XPP would be  
> easier to implement?  RDP at least has addressed all the thorny  
> reliability details.
>
> It seems to me the session negotiation issues could be worked out  
> for any of these since ICE is designed to be relatively transport  
> agnostic with a little SDP fiddling.
>
> I agree the NAT hold punching is a little more straightforward with  
> UDP since you just have more control over the timing of packets,  
> making something along these lines worthwhile.
>
> -Adam
>
>
> On 6/17/07, Enrico Marocco < [email-masked]> wrote:
> Folks,
>

---

From: "Krishna Sankar \(ksankar\)" <[email-masked]>
Date: Tue, 19 Jun 2007 20:18:02 -0700

Yep, agreed. I would suggest SCTP which is relatively better in may
ways. 
 
I am going to ask couple of stupid questions (as I didn't get enough
time to go over all the nuances, summer time you know ;o))
 
a)    Enrico, are you using XPP to achieve reliable UDP ?
b)    Did you consider SCTP ? Any glaring problems ?
c)    Knowing that TCP is very well optimized in all the devices incl
the NICs, why should we explore XPP/RDP et al ?
d)    What is our goal ? The perfect protocol or a "good-enough" one
that has maximum co-existence ?
 
I am just asking, no judgment calls. 
 
Cheers
<k/>

________________________________

	From: Philip Matthews [mailto:[email-masked]] 
	Sent: Tuesday, June 19, 2007 5:06 PM
	To: Adam Fisk
	Cc: Emil Ivov; P2PSIP Mailing List
	Subject: Re: [P2PSIP] Peer protocol, DHT and running code

	Adam:

	I was not familiar with RDP, but I took a quick look.
	From what I can see, RDP is an old protocol that who state is
roughly comparable to the original specification of TCP.
	As such, it does not have all the enhancements that TCP has
undergone over the years (congestion control, etc.).
	My guess is that the IESG would strongly urge us to look at TCP,
SCTP, and DCCP (the latter two having been
	invented with all the existing TCP work in mind) before trying
to resurrect RDP.

	Furthermore, using a transport protocol other that TCP or UDP
means that we have to do all the work to define
	how SIP and other protocols get transported over this transport
protocol. I don't think we should underestimate
	how much time that would take.

	- Philip

		I'm wondering if you considered the "Reliable Data
Protocol (RDP)" for the peer protocol.  It's at
http://www.faqs.org/rfcs/rfc1151.html.  
		
		Seems like an oldie but a goodie with many of the same
characteristics, although I don't believe there's the option to turn off
reliability.  Are you thinking something like XPP would be easier to
implement?  RDP at least has addressed all the thorny reliability
details.  
		
		It seems to me the session negotiation issues could be
worked out for any of these since ICE is designed to be relatively
transport agnostic with a little SDP fiddling.
		
		I agree the NAT hold punching is a little more
straightforward with UDP since you just have more control over the
timing of packets, making something along these lines worthwhile. 
		
		-Adam

		On 6/17/07, Enrico Marocco <
[email-masked] <mailto:[email-masked]>
> wrote: 

			Folks,
			
			Emil and I have just submitted two drafts
specifying the peer protocol
			(XPP) and the overlay algorithm (CAN based) we
have implemented in the 
			last version of sipdht
(http://sipdht.sourceforge.net/sipdht2/).  Till
			they appear in the repository, you can find them
at the following links 
			(just s/txt/html/ or s/txt/xml/ if you prefer
html or xml): 

http://sipdht.sourceforge.net/drafts/draft-marocco-p2psip-xpp-pcan-00.tx
t
			
			We hope this contribution will stimulate
discussion on some key issues
			we've tried to address, among which:

			On the latter, we propose a mechanism to let
nodes join the overlay only
			when they are explicitly invited by existing
peers.  Such a "passive"
			approach (yeah, Dean anticipated that the term
was going to get 
			overloaded) has some benefits which deserve to
be investigated.
			
			Needless to say, any feedback is very very
appreciated.
			
			--
			Ciao,
			Enrico

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 20 Jun 2007 10:35:00 +0200

Philip, let me firstly answer the question which hits the main point
separately. Inline.

Philip Matthews wrote:
> 3) Why do you think we need XPP as a new transport protocol, rather than
> using TCP or SCTP?
> You argue that UDP is easier than TCP to set up through NATs, but Saikat
> Guha has shown
> that TCP connections can be reliably set up through NATs in about 90% of
> the time, and 100%

The problem with TCP is that the mechanism you need to establish a
connection heavily depends on the environment.  The simultaneous
handshake, for example, is generally required to establish the
connection if the two peers are behind NATs, but fails 50% of the times
if one isn't (because the non-natted peer would send back a RST if it
hasn't started sending SYNs yet) and ~100% of the times if none of them
is (because most likely one of the peers would receive a SYN on an
uninitialized socket).  So, basically, it is true that ~88% of the times
you can successfully establish a TCP connection between natted peers,
but to do that you really need to implement something like ICE-TCP to
find every time the effective way to do that.

We decided to go for a compromise to keep ICE a "SHOULD", and defined
the simultaneous establishment mechanism over UDP, which is supposed to
work both if the peers are behind 4787-compliant NATs and if any of them
has a public address.  Note that the full XPP implementation (including
session establishment, fragmentation, reliability and request-response
matching) is just about 2500 lines of code.

Then, using UDP it is possible to turn reliability off; although it
sounds interesting especially in distributed overlays, I think it should
be discussed separately.

> Moreover, one can easily work around this problem if desired by running
> TCP over UDP,

I don't see XPP as a general purpose transport protocol; it is really
conceived for executing DHT operations, and in this is similar to P2PP
(we even came out with the same encoding).

Finally, TCP over UDP would bring too much complexity, and too much
unnecessary and redundant functionality, which contradicts our design goals.

---

From: "Beck01, Wolfgang" <[email-masked]>
Date: Wed, 20 Jun 2007 11:00:23 +0200

Philip Matthews wrote:
> My guess is that the IESG would strongly urge us to look at TCP, SCTP,
and DCCP (the latter two having been
resurrect RDP.

I share Philip's reservations about inventing yet another UDP-based
transport protocol.
However, I'd guess that many firewalls out there don't know about SCTP
and DCCP and
will simply discard such packets. 

What are our requirements?
- transport of larger pieces of data;
    TCP shines here, but how much continuous traffic do P2P protocols
create?
- reliable session setup through all NAT (advantage UDP?)
- measurement of the network's performance for peer selection;
    Linux apps can access the TCP parameters of a running session, do
other OSs have a similar functionality?
    UDP-based transport could do this in a more portable way
- ease of implementation
    getting segmentation and reliable transport right requires some work

As much as I'd like to see a TCP-based solution, a UDP-based transport
seems to have
some important advantages. But please, either merge xpp with p2pp or
drop one of the protocols.

Wolfgang

---

From: Emil Ivov <[email-masked]>
Date: Wed, 20 Jun 2007 11:20:48 +0200

Hello Krishna,

> a)    Enrico, are you using XPP to achieve reliable UDP ?

Reliability is only one of the transport features that we'd need in an 
overlay maintenance protocol. We'd also like to be able to turn it off, 
make it handle fragmentation and remain as light as possible in the same 
time.

> b)    Did you consider SCTP ? Any glaring problems ?

SCTP has a lot of functionality that we don't need, such as: recording 
paths, transporting multiple flows simultaneously, etc. Then, I am also 
wondering how many of the existing NATs and firewalls would let it pass.

> the NICs, why should we explore XPP/RDP et al ?

You may want to have a look at Enrico's reply to Philip, but here's a 
summary: We need it to behave as well as possible behind NATs. We'd like 
to be able to turn reliability off. We'd like it to remain a light and 
easy to implement solution.

> d)    What is our goal ? 

First, make sure that it would work in as many cases as possible. In 
other words we'd like to handle any NAT or firewall that would let UDP 
pass through.

Second, keep it as lite and easy to implement as possible.

Cheers
Emil

> The perfect protocol or a "good-enough" one 

>     ------------------------------------------------------------------------
>     *From:* Philip Matthews [mailto:[email-masked]]
>     *Sent:* Tuesday, June 19, 2007 5:06 PM
>     *To:* Adam Fisk
>     *Cc:* Emil Ivov; P2PSIP Mailing List
>     *Subject:* Re: [P2PSIP] Peer protocol, DHT and running code
> 
>     Adam:
> 
>>     turn off reliability.  Are you thinking something like XPP would
>>     be easier to implement?  RDP at least has addressed all the thorny
>>     reliability details. 
>>
>>     I agree the NAT hold punching is a little more straightforward
>>     with UDP since you just have more control over the timing of
>>     packets, making something along these lines worthwhile.
>>
>>     -Adam
>>
>>
>>     On 6/17/07, *Enrico Marocco* < [email-masked]
>>     <mailto:[email-masked]>> wrote:
>>
>>         Folks,
>>
>>         <http://sipdht.sourceforge.net/drafts/draft-marocco-p2psip-xpp-00.txt>
>>         [email-masked] <mailto:[email-masked]>
>>     [email-masked] <mailto:[email-masked]>
> ------------------------------------------------------------------------
>

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 20 Jun 2007 12:17:59 +0200

Philip Matthews wrote:
> Furthermore, using a transport protocol other that TCP or UDP means that
> we have to do all the work to define

Actually, I don't think we really need to tunnel SIP over the peer
protocol.  In the PCAN draft
(http://tools.ietf.org/id/draft-marocco-p2psip-xpp-pcan-00.txt) we
address the issue of NAT traversal the following way:

1) clients that need to do so would keep sip-outbound connections with
   peers.
2) peers that handle sip-outbound flows for some clients would store
   routing records containing the full path to the client.

Here's an example: imagine client C registers with peer B and
establishes an outbound flow with it. C's routing record (inserted in
the overlay by B) will contain two hops: B's contact and C's
contact/gruu.  In case B is also suffering restricted connectivity, this
would mean that it too would have an outbound flow with another peer,
say A. In this case (which is pretty unlikely, because if B is
restricted it probably won't be able to receive REGISTER requests form
C), C's routing record would contain three hops.

Having said that, it's probably worth noting that we have not yet
decided whether this approach is better compared to routing SIP messages
inside the overlay. Both are easily implementable with XPP and XPP-PCAN
so I guess we'd need to discuss this further before we really made up
our minds.

---

From: Emil Ivov <[email-masked]>
Date: Wed, 20 Jun 2007 13:41:48 +0200

Hello Wolfgang,

> drop one of the protocols.

XPP, P2PP, and dSIP are all proposals made by contributors to this 
working group. The drafts that we submitted represent our take on the 
problem, based on our work on the SIPDHT2 project. We are, by no means, 
expecting that they would be adopted the way they are. We do hope, 
however, that the working group would find parts of them useful and we 
are, most certainly, willing to work together with other wg 
members/authors on a version combining the strengths of all proposed 
solutions.

Cheers
Emil

---

From: "Brian Rosen" <[email-masked]>
Date: Wed, 20 Jun 2007 08:51:05 -0400

<as individual>
I am bothered by this discussion:

> connection heavily depends on the environment.  ....
> So, basically, it is true that ~88% of the times
> has a public address.  

88% of the time is going to be a WHOLE lot more that "any of them has a
public address".  I doubt 2% of a normal overlay will have peers with public
addresses behind 4787 compliant NATs.  To me, this is the biggest problem
with p2psip; peers have non-public addresses behind very restrictive NATs.
That is the way the Internet looks like these days.

I am also bothered by ICE as a should.  There is a whole lot of effort that
was put into making ICE work in as many circumstances as possible, and I
think the code size is not going to be much of an issue.  It's going to be
common to have ICE implementations with any SIP implementation.  I don't
think investing much effort in ICE-light is worthwhile.

Brian

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 20 Jun 2007 15:42:13 +0200

> That is the way the Internet looks like these days.

I think I've been unclear.  Obviously XPP is supposed to work when none,
one or both peers are behind NATs compliant with RFC 4787 (clearly if
they are behind a NAT, they don't have public addresses).

> think investing much effort in ICE-light is worthwhile.

Yes, this is indeed a big question.  I'm curious to see what people
think about ICE;  on one hand, it is pretty difficult to implement and
to integrate with existing libraries, on the other hand, it is the most
complete mechanism for NAT traversal.  For the first version of our
solution we tried to keep it optional and accepted a cheap compromise
which should /often/ work (and which does not preclude ICE).  It's a
trade off... I would really like to get feedback on this.

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 20 Jun 2007 17:13:23 +0200

Philip Matthews wrote:
> That is, it provides reliable, in-order delivery of messages, along with
> segmentation
> However, unlike TCP which is stream-oriented, XPP seems to be
> message-oriented.

In a way yes, probably. Right now  we are only using UDP for
encapsulation (and checksum), pretty much the same way RTP is using it.

> Furthermore, it seems that you expect ICE to used with SIP to handle the
> NAT
> Is this correct?      (I will assume "yes" in the rest of my questions
> below).

Yes, XPP is negotiated just like RTP.  Anyway, we don't want to mandate
ICE; we recommend it, because it may help in many ways (especially to
find best paths), but we want to specify something as simple as possible
which could work in as many cases as possible.  XPP without ICE is
supposed to work when peers are behind firewalls and NATs compliant with
RFC 4787 performing endpoint-dependent filtering.

> in the same way that you run XPP over UDP.

Answered separately.

> If ICE has already handled the UDP hole-punching, then the simultaneous
> nature
> is unimportant.

We wouldn't want to mandate ICE.  Would you? (I'm quite curious to know
what others think about this)

> 5) If XPP is set up using ICE, then presumably the NAT bindings are kept
> alive using
> other than keeping NAT bindings open. My guess is they are just there to
> make sure the far-end

That, and keep NAT bindings alive if ICE is not used.

> 6) XPP uses 16-bit endpoint identifiers. What are these? I am guessing
> that these are not
> peer IDs  as described in the Concepts draft, since they are so short.
> Is this correct?

Yes, local and remote IDs have noting to do with peer IDs.  They are
used to identify XPP sessions.  Think of them as the dialog tags in SIP.

> How are these allocated?

Locally, one per endpoint, the same way SIP dialog tags get generated.

> 7) Is an ACK a response to a request? Or must a response be something else?

An ACK is not a response.  Requests and responses are messages which can
be fragmented; each fragment is acked.

> client that one is talking or IMing with)

Well, in such a case I'd establish an MSRP session with the other guy.
There is no reason why this should be handled by the peer protocol. We
could of course craft some way of doing it with XPP, but that would be
unnecessary and redundant. Instead, we should simply be seeing file
transfer as call sessions.

> so there is no need to handle this at the XPP layer?

No, not at all.  Current proposal acknowledges the need to add support
for DTLS, but security issues definitely require more investigation.

> 9) How and when do you see XPP sessions being set up? That is, say peer
> X and peer Y are

Sessions are initiated and negotiated with SIP, which, in the case of
P2PSIP, is routed by the overlay. This is explained in detail in our
other draft.

> with Y first?  It seems to me that session setup is a pretty
> heavy-weight operation (e.g., send one or more
> receive ICE connectivity checks on various addresses, send and receive a
> XPP handshake,

Hop-by-hop routing of requests and responses is supported, but is
specified in XPP extensions (see e.g. PCAN) as it is generally specific
to the implemented algorithm.

---

From: Philip Matthews <[email-masked]>
Date: Wed, 20 Jun 2007 14:02:03 -0400

Enrico:

If we decide that TCP is the right way to go, then I would strongly  
suggest
that we ALWAYS use ICE-TCP to set up the TCP connection,
even if one or both peers have public addresses.  If we did this,
then the connection would be set up correctly even in these cases.

I make this suggestion for the following reasons:
a) In many many cases, it is difficult or impossible to know whether  
a node
is "behind" a NAT or not.
b) ICE-TCP has advantages other than NAT traversal. For example,
it also allows you to select between an IPv4 and an IPv6 address.

I agree that ICE is not a trivial protocol. But I think it is the  
right thing to do,
because it solves an important problem, and attempts to  make it a  
SHOULD
will just lead to difficulties.

One concern I have with XPP is similar to my concern with the   
suggestion to use RDP.
There has be A LOT of work put into TCP, and it is a very well  
understood protocol.
A lot of this work has since gone into SCTP and DCCP. Inventing a new  
transport
protocol is not  a trivial matter, and I think we will get strong  
push-back from the IESG if
we try to propose a new one without looking hard at the existing  
choices.

Though you think that XPP is simple right now, how much attention  
have you paid
to all the corner cases? And what about things like congestion  
control, path MTU discovery,
and the like? If our peer protocol becomes successful, then these are  
all things that it
will need to have. After all, peer-to-peer traffic is a very large  
and growing part
of Internet traffic today, and if we are successful, then we will  
likely capture a large
part of this traffic.

- Philip

On 20-Jun-07, at 04:35 , Enrico Marocco wrote:

>> Saikat
>> running

---

From: "David Barrett" <[email-masked]>
Date: Wed, 20 Jun 2007 11:12:11 -0700

> -----Original Message-----
> From: Enrico Marocco [mailto:[email-masked]]

I certainly agree with the first point.  But has "most complete mechanism
for NAT traversal" been really demonstrated yet?

Are there any empirical studies (testing how good it "actually" is in the
real world, rather than inferring how good it "should" be) that have studied
ICE outside a limited lab environment?

Furthermore, until it's been shown that multiple, separately-developed
implementations actually interoperate, I'd be hesitant to even call ICE
well-specified.

So far as I can tell, there's a lot of optimism, hope, and hard work, but
I've yet to see the real world confirmation.  Can someone point me to this?

-david

---

From: Philip Matthews <[email-masked]>
Date: Wed, 20 Jun 2007 14:16:07 -0400

On 20-Jun-07, at 05:20 , Emil Ivov wrote:

> SCTP has a lot of functionality that we don't need, such as:  
> recording paths, transporting multiple flows simultaneously, etc.  
> Then, I am also wondering how many of the existing NATs and  
> firewalls would let it pass.

I suspect that you are right in saying that many NATs and firewalls  
do not currently support SCTP traffic (this may change if P2PSIP  
selects SCTP as its transport protocol?).
However, it is also true that many (all?) NATs and firewalls do not  
currently support XPP traffic.

The solution that you adopted for XPP (= encapsulate it inside UDP)  
also works for SCTP.
Furthermore, it would be possible to define an ICE-for-SCTP that  
would detect if the intervening NATs support SCTP, and eliminate the  
UDP encapsulation wherever that is possible. A similar idea is being  
investigated as part of the NAT Traversal for HIP work, which has a  
similar problem (= most NATs today don't support HIP).

- Philip

---

From: "Brian Rosen" <[email-masked]>
Date: Wed, 20 Jun 2007 14:28:56 -0400

There is a fair amount of work on investigating the characteristics of
deployed NATs, which then guided the ICE effort.  It's a moving target.
There are a number of commercial STUN servers out there, which is a big
piece of ICE.  I think the direct answer to your question is that there is
not a lot of hard data on the effectiveness of ICE, but I would bet on that
a whole lot more than any new, supposedly simpler set of NAT traversal
techniques, even in more restricted use cases.

Brian

> From: David Barrett [mailto:[email-masked]]
> Sent: Wednesday, June 20, 2007 2:12 PM
> To: 'Enrico Marocco'; 'Brian Rosen'
> Cc: 'Emil Ivov'; 'Philip Matthews'; 'P2PSIP Mailing List'
> Subject: RE: [P2PSIP] Peer protocol, DHT and running code
>

---

From: Philip Matthews <[email-masked]>
Date: Wed, 20 Jun 2007 14:38:11 -0400

>> how much time that would take.

First, a small correction.
SCTP transport of SIP has already been defined:  RFC 4168.

Now on to the main discussion...

> so I guess we'd need to discuss this further before we really made up

I assume that your peers have public IP addresses.
If so,  this is "superpeer" approach that is discussed in the "NATs  
and Overlays"  draft
and in my NAT traversal presentation in Prague. Dean has also suggested
a similar approach.

We have had some discussions of the pros and cons of this approach vs  
the approach
that Eric and I have been advocating of routing messages through a  
partial mesh of connections.

The whole goal of draft-matthews-p2psip-nats-and-overlays-01 is to  
discuss these two approaches
and their pros and cons.  Feedback on that draft would be much  
appreciated, and would help
the  WG decide on a direction.

- Philip

---

From: Philip Matthews <[email-masked]>
Date: Wed, 20 Jun 2007 14:45:04 -0400

> trade off... I would really like to get feedback on this.

Personally, I think we should just use ICE.
Yes, it is complex, but it is solving a complex problem, and the  
solution
works well. The problem with other "cheaper" solutions is that they  
often
break, and break unexpectedly.

It might be useful to cite a recent post by Dan Wing to the BEHAVE list:
http://www1.ietf.org/mail-archive/web/behave/current/msg02396.html
Read the rest of the messages in the thread for more details.

>>>  1. ICE works.  This has been well demonstrated by its successful
>>>     deployment on the Internet by millions of endpoints.  It works
>>>     without explicit support by subscriber's NATs or ISP's NATs.
>>>
>>
>> Just curious, what statistics are available on ICE's
>> real-world usage and NAT-penetration success rates?  Specifically:
>>
>> - How did you come up with the "millions of endpoints" number?
>>
>
> GoogleTalk's API, Google Talk itself, Microsoft's Windows Live
> Messenger (formerly called MSN), and Yahoo Messenger all use
> ICE.
>
>
>> - What does "works" mean?  (I'd suggest this be measured in
>> "direct peer connectivity ratio" and "connection setup time",
>> but any measure will do.)
>>
>
> I found this, "Improving ICE Service Selection in a P2P System
> using the Gradient Topology",
> http://www.jimdowling.info/pubs/dowling-j_icep2p_saso07.pdf
> which includes the following quote:
>
>   [...] Google's estimate that 8% of the GoogleTalk traffic
>   is routed using relay nodes.
>
> which cites
>   Google. Google talk, Accessed, Feb 22, 2007.
>   http://code.google.com/apis/talk/libjingle
>
> which leads one to:
>   http://code.google.com/apis/talk/libjingle/important_concepts.html

- Philip

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 20 Jun 2007 21:14:18 +0200

Philip Matthews wrote:
> If we decide that TCP is the right way to go, then I would strongly suggest

I totally agree.  If we chose TCP, then we would definitely need to
mandate ICE-TCP.

> a) In many many cases, it is difficult or impossible to know whether a node

Yes, that's and issue especially for TCP.

> it also allows you to select between an IPv4 and an IPv6 address.

Yep.

> I agree that ICE is not a trivial protocol. But I think it is the right
> thing to do,
> because it solves an important problem, and attempts to  make it a SHOULD

Unfortunately I don't have such clear ideas.  Maybe listing some of
those difficulties may help me and others to better understand.

> we try to propose a new one without looking hard at the existing choices.
> 
> Though you think that XPP is simple right now, how much attention have
> you paid
> to all the corner cases? And what about things like congestion control,
> path MTU discovery,
> will need to have.

Well, path MTU discovery may be done using ICMP or the default 1500
value may be adopted, as for SIP.  Congestion control is not an issue,
because we send fragments one by one, only after the previous fragment
has been acked.  Yes, that's not much efficient, but if fits well in our
restricted scope (i.e. for implementing DHTs).

>                    After all, peer-to-peer traffic is a very large and
> growing part
> of Internet traffic today, and if we are successful, then we will likely
> capture a large
> part of this traffic.

I agree here too.  By the way, not that it really matter, but, IIRC some
of the most popular p2p applications (emule and azureus) use UDP to
maintain the internal DHT, don't they?

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 20 Jun 2007 21:20:16 +0200

> I assume that your peers have public IP addresses.

No, not all peers need to have public addresses.  Restricted peers
receive SIP requests on sip-outbound connections.

> If so,  this is "superpeer" approach that is discussed in the "NATs and
> Overlays"  draft

Actually our proposal uses a partial mesh too.  The difference is that
SIP messages do not need to follow peer protocol paths.

---

From: "David Barrett" <[email-masked]>
Date: Wed, 20 Jun 2007 12:57:27 -0700

> > Messenger (formerly called MSN), and Yahoo Messenger all use

Don't forget later in the thread:

[Dan] "ICE isn't finished, and early ICE versions are not compatible
with each other.  Thus, there isn't much hope those different
networks would be compatible until until ICE has gone to RFC
or if those networks had worked out a version of ICE they 
mutually agreed they would interoperate with (thus creating a
defacto standard)."

In other words, there are zero endpoints using the ICE protocol (because it
doesn't even exist yet), and millions of endpoints that have protocols
somewhat like some version of ICE.

> >   http://code.google.com/apis/talk/libjingle/important_concepts.html

Likewise, later in that same thread:

[Dan] "Google's analysis is based on their own (GoogleTalk) network, I 
would expect -- not GoogleTalk-to-something-else.  I expect their
92% number is based on how often two users, each behind 
NATs with the 'Address and Port-Dependent Mapping' property
(RFC4787) attempt to talk with each other (this property was
formerly called 'symmetric NAT').  With ICE, since approximately
ice-10, that is the only case that causes a media relay to be
utilized."

In other words, those numbers aren't measured, they're guesses.

Which brings me back to my original point that ICE hasn't been standardized,
and hasn't been measured in the real world.  Thus it's reasonable to be
skeptical of its claims until it's been demonstrated that the standard is
sufficiently-well defined that a "clean" implementation will actually
interoperate, and actually solve its intended job.  So far as I can tell,
neither of these have been proven, and until they have, it's reasonable to
consider alternatives.  

-david

---

From: "Brian Rosen" <[email-masked]>
Date: Wed, 20 Jun 2007 16:35:07 -0400

<as individual>
David

It used to be that the IETF prided itself on "rough consensus and running
code".  We've lost the "running code" part of our process, but in its place,
we have a lot of really good people who really look hard at the specs.  Our
experience over the last decade or so is that we are fair-to-middling good
at writing specs that, with a reasonable amount of interoperability testing,
work.

ICE has had A LOT of very good eyeballs looking at it.  Yep, it's
complicated, and it's complicated because the world is complicated, and
there have been enough good eyeballs on the spec to deal with the
complications.

It's ALWAYS possible that those eyeballs missed some not so obvious
simplification.  If you spot one, go tell them (in MMUSIC).

But, I, for one, don't believe that we here in p2psip can make something
that works as well as ICE for our problem.  YMMV, but I don't want to waste
a lot of my time looking at ideas unless there really is a big breakthrough.
If there is, I'm interested.  If you are in the "there must be a simpler
way" state, by all means, keep trying, but until you get to a really big
"aha" moment, I'd rather work on other parts of the problem and assume ICE.

Brian

> Sent: Wednesday, June 20, 2007 3:57 PM
> To: 'Philip Matthews'; 'Enrico Marocco'
> Cc: 'Emil Ivov'; 'P2PSIP Mailing List'

---

From: "Bruce Lowekamp" <[email-masked]>
Date: Wed, 20 Jun 2007 17:12:03 -0400

> ICE outside a limited lab environment?

yes, although I'm not aware of any that are actually published/publishable.

> I've yet to see the real world confirmation.  Can someone point me to this?

What has been shown is that previous attempts to solve almost the
whole problem are frequently tricky and don't work as well as they
should.  Moreover, it doesn't really matter whether a solution
supports 90% or even 99% of the cases.  If it doesn't work with a
potential user/customer's hardware (or that of the people they want to
call) then it doesn't work.

You're absolutely right in that ICE, and particular the
interoperability, isn't as thoroughly tested as would be ideal.  But I
don't think that's an entirely reasonable standard to hold a brand-new
protocol to, because it adds a chicken and egg problem to developing a
new complicated protocol.  Right now what we have is that it's the
best that some of the best people in the world at working with NATs
can develop to address the problem.  It's finally in WGLC, so
hopefully we will start to see interoperability tests soon.  If it
turns out that there are problems with it, I'm sure they will be
addressed.

If we want to support peers behind NATs (and as far as I'm concerned,
that's most of the Internet right now, so I don't think we have a
choice), ICE is our best hope at getting that to work.  Anything else
risks trying to reinvent ICE, anyway.

Bruce

---

From: "David Barrett" <[email-masked]>
Date: Wed, 20 Jun 2007 15:46:34 -0700

Well, my primary goal is to just push back on suggestions that ICE is a
proven solution, when in fact it's largely untested (except for perhaps some
private study hinted at by Bruce). 

I'll admit, I'm uncomfortable narrowing the field of choices to a single,
unstandardized, largely-untested protocol, but I can understand the overall
process.  It's reasonable to say "NAT penetration is not part of our
mandate, we're delegating this to ICE, and we'll accept whatever it produces
-- it's not our job to investigate alternatives".  I just wish there was
some force pushing back on ICE to show measurable results.

Again, I'm totally open to the possibility ICE is great.  But I haven't seen
the data yet.  Indeed, the only data I do have is personal experience, which
suggests ICE is an impossibly complex, ever-changing morass that cannot be
implemented except in proprietary, non-standard subsets of obsolete drafts,
cemented together with undocumented tricks and voodoo.  I'd like to see data
that shows otherwise, and I'm hoping I can rally the troops to raise the
priority of obtaining such data.

-david

> From: Brian Rosen [mailto:[email-masked]]
> Sent: Wednesday, June 20, 2007 1:35 PM
> To: 'David Barrett'; 'Philip Matthews'; 'Enrico Marocco'

---

From: Enrico Marocco <[email-masked]>
Date: Thu, 21 Jun 2007 08:35:20 +0200

> techniques, even in more restricted use cases.

Just to be clear, XPP does not introduce any _new_ NAT traversal
technique.  As it is UDP based (and it is UDP based not only for NAT
traversal!), it has a NAT-friendly handshake mechanism.  Which isn't new
at all, as it is almost the same as symmetric RTP.

And it doesn't preclude ICE either.

---

From: Emil Ivov <[email-masked]>
Date: Thu, 21 Jun 2007 10:18:46 +0200

Hello all,

Some more on the subject.

As Enrico mentioned, we are very ICE friendly and we are not trying to 
suggest a replacement technique.

However, as some of you have already mentioned, ICE is big, complex and 
time consuming. Are we certain that we want it to be a MUST and call 
incompatible all implementations that are bypassing it?

How about seamless mobility? Imagine a peer that moves from one subnet 
to another and needs to reconnect to the overlay as quickly as possible. 
  Is it reasonable to impose the extra lag that comes with ICE?

Then there are sensor (possibly IPv6 only )networks and other similar 
environments where the obligation to add an extra stack would simply 
preclude against using a specific solution.

To sum it up, I am thinking that we might have a lot of cases where ICE 
would be undesirable so it might be a bit harsh to shun as 
"incompatible" implementations that won't be using it.

Cheers
Emil

---

From: Philip Matthews <[email-masked]>
Date: Thu, 21 Jun 2007 08:14:23 -0400

On 21-Jun-07, at 04:18 , Emil Ivov wrote:

> As Enrico mentioned, we are very ICE friendly and we are not trying  
> to suggest a replacement technique.
>
> However, as some of you have already mentioned, ICE is big, complex  
> and time consuming. Are we certain that we want it to be a MUST and  
> call incompatible all implementations that are bypassing it?

If the system you produce will only run in a single address space,  
then ICE is not needed.
However, if either a client or a peer can be in a different address  
space (= behind a NAT), then ICE is needed, unless you always use a  
media relay. Otherwise, how do you set up a direct connection for the  
media when the destination is behind a NAT?

> How about seamless mobility? Imagine a peer that moves from one  
> subnet to another and needs to reconnect to the overlay as quickly  
> as possible.  Is it reasonable to impose the extra lag that comes  
> with ICE?
>
> Then there are sensor (possibly IPv6 only )networks and other  
> similar environments where the obligation to add an extra stack  
> would simply preclude against using a specific solution.

Is this a code space problem? Or something else?
I know how memory can be tight on embedded devices. However, I can  
say that by the time the P2PSIP specs are published as RFCs, the  
amount of memory on typical new embedded device will probably have  
doubled :-). In the meantime, stop programming in C++, or at least  
give up templates. :-).

- Philip

---

From: Emil Ivov <[email-masked]>
Date: Thu, 21 Jun 2007 14:55:28 +0200

> If the system you produce will only run in a single address space,  

This means you agree that "MUST use ICE" is too strong and SHOULD fits 
better. Right?

> space (= behind a NAT), then ICE is needed 

ICE is helpful but not absolutely needed.

>, unless you always use a  

Using only STUN (not TURN), you could still be successful in many
many cases.  Then, if you cannot establish some connections, hell, the
overlay is designed to be robust:  There will be more featured (and
expensive) ICE-aware peers doing some work for you...

> doubled :-). 

And there will be smaller and cheaper devices with same amount resources
(and is memory the only bottleneck on such devices?).

> In the meantime, stop programming in C++, or at least  
> give up templates. :-).

Now, isn't this a wise argument! :-)

Cheers
Emil

---

From: Enrico Marocco <[email-masked]>
Date: Thu, 21 Jun 2007 15:09:54 +0200

Brian Rosen wrote:
> we have a lot of really good people who really look hard at the specs.

Some real life examples -- to name but a few: Jabber, Bonjour and
arguably HTTP1.0 -- would probably make some people argue that the
"running code" part is essential for protocols to be successful,
probably more important than specs.  Obviously, I don't completely agree
with such an argument, for if I did, I wouldn't be discussing any more,
but I still think that, especially for highly unpredictable
decentralized systems as P2PSIP, running code is an extremely important
part.

I certainly hope I am not the only one left to believe this.

---

From: "David Barrett" <[email-masked]>
Date: Thu, 21 Jun 2007 12:47:40 -0700

Totally agree with you.

> Sent: Thursday, June 21, 2007 6:10 AM
> To: Brian Rosen
> Cc: 'David Barrett'; 'Philip Matthews'; 'Emil Ivov'; 'P2PSIP Mailing List'

---

From: "Adam Fisk" <[email-masked]>
Date: Thu, 21 Jun 2007 17:55:14 -0400

> I totally agree.  If we chose TCP, then we would definitely need to

I think it would be mistake to choose only TCP.  I have yet to see
robustly-running code for NAT traversal in TCP.  I have a lot of faith in
Saikat and co., but I sure haven't been able to get it to work.  I actually
think TCP should be one option while also supporting a UDP-based solution.

> restricted scope (i.e. for implementing DHTs).

Hmnn...I didn't notice that part of XPP (the lock-step approach), but it
raises a red flag for me.  If you're doing lock-step, why not just use TFTP
(Trivial File Transfer Protocol).  If you're looking for simplicity, that's
the way to go!

> maintain the internal DHT, don't they?

eMule and Asureus, as well as I believe LimeWire's new DHT, all are
UDP-based.  That's primarily because they're all based on Kademlia, though.
None of them to anything with NAT traversal.

-Adam

---

From: "Adam Fisk" <[email-masked]>
Date: Thu, 21 Jun 2007 18:07:03 -0400

> The solution that you adopted for XPP (= encapsulate it inside UDP)

That's an interesting idea to me.  SCTP doesn't make sense to me alone
(since NAT traversal is undefined), but encapsulation in UDP for NAT
traversal could make a lot of sense.  That was the heart of my RDP
suggestion -- use an existing transport for reliable UDP transmission.

Like you, I don't like the idea of inventing another transport that combines
much of the functionality of existing transports but that also crams in
specialized DHT handling.  I think we should choose existing protocols that
solve the transport problem better than we should be thinking about here,
and then focus on the peer messages themselves.

I'm not too familiar with SCTP, though, so encapsulating it in UDP could be
a huge task.

-Adam

---

From: "Brian Rosen" <[email-masked]>
Date: Thu, 21 Jun 2007 18:12:28 -0400

<as individual>
We have a WHOLE LOT of experience with UDP and sip.

It was a mistake.  A big one.  We should NEVER have defined sip on UDP.
Henning told me he regrets not insisting on that way back when.

Let's not make that mistake again.

Fragmentation always gets you when you least expect it
Reliable message passing always is needed, even when you think you don't
Message order always is a pain to keep straight
Connections really are helpful, and simulating your own is a pain

And the big one

Congestion control is required, don't ever believe that it isn't for your
protocol

There is so much good engineering in TCP that, unlike ICE, is very well
proven, and we should not re-invent it.
We may not need some of TCP.  It may get in the way a bit.  It's MUCH better
than UDP for our purposes.

Brian

________________________________________
From: Adam Fisk [mailto:[email-masked]] 
Sent: Thursday, June 21, 2007 5:55 PM
To: Enrico Marocco
Cc: Emil Ivov; Philip Matthews; P2PSIP Mailing List
Subject: Re: [P2PSIP] Peer protocol, DHT and running code

-Adam

---

From: "Adam Fisk" <[email-masked]>
Date: Thu, 21 Jun 2007 18:25:22 -0400

I agree 100% UDP should never have been used for SIP, and I've had the same
conversation with Henning.  The key difference, though, is that SIP doesn't
need to worry about NAT traversal in the same way we're talking about here
because SIP proxy and registrars are on the public Internet.

I agree reliable message passing makes sense for this purpose, and that's
one of the reasons I was pointing to RDP.

I think we're missing each other a little bit.  The first question is:

1) Do we need NAT traversal for the Peer protocol?

If the answer to that question is "yes", I think the peer protocol should be
*a separate protocol* implemented over 2 transports:

1) TCP
2) some sort of reliable UDP.

If the answer to that question is no, then yes, of course, just use TCP like
should have always been done with SIP.

-Adam

On 6/21/07, Brian Rosen <[email-masked]> wrote:
> I totally agree.If we chose TCP, then we would definitely need to
> mandate ICE-TCP.
>
> value may be adopted, as for SIP.Congestion control is not an issue,
> has been acked.Yes, that's not much efficient, but if fits well in our
> I agree here too.By the way, not that it really matter, but, IIRC some

---

From: "Brian Rosen" <[email-masked]>
Date: Thu, 21 Jun 2007 20:37:54 -0400

Huh.

I must not understand firewalls and UDP.

I was told most firewalls, and especially enterprise firewalls did not allow
arbitrary incoming UDP.  Some allow symmetric UDP with the outbound traffic
first.  I know I can't run a SIP proxy behind my firewall at home without
configuring the firewall to have a wide open port, and I know what I'm
doing.  I know I can't get incoming or outgoin UDP in my office unless I
have a really good reason to ask the IT guys to open a port.  YMMV I guess.

I think reliable+congestion control+fragmentation control+nat traversal UDP
= TCP+ICE.  

Brian

________________________________________
From: Adam Fisk [mailto:[email-masked]] 
Sent: Thursday, June 21, 2007 6:25 PM
To: Brian Rosen
Cc: Enrico Marocco; Emil Ivov; Philip Matthews; P2PSIP Mailing List
Subject: Re: [P2PSIP] Peer protocol, DHT and running code

1) TCP

-Adam

And the big one

Brian

-Adam

---

From: Philip Matthews <[email-masked]>
Date: Thu, 21 Jun 2007 20:38:06 -0400

I agree, which another reason I like our proposal
(draft-matthews-p2psip-hip-hop-00).
It allows you to run SIP over TCP or run SIP over TLS .
Or SIP over SCTP if you prefer that.

It gives you all the existing transport protocols and application  
layer APIs that
are used today, because the changes to work in an overlay happen  
(mostly)
below this layer.

Of the proposals published this week, it is the only one that  
leverages all the existing
Internet protocols.

- Philip

---

From: Philip Matthews <[email-masked]>
Date: Thu, 21 Jun 2007 20:46:22 -0400

On 21-Jun-07, at 18:07 , Adam Fisk wrote:

> That's an interesting idea to me.  SCTP doesn't make sense to me  
> alone (since NAT traversal is undefined), but encapsulation in UDP  
> for NAT traversal could make a lot of sense.  That was the heart of  
> my RDP suggestion -- use an existing transport for reliable UDP  
> transmission.
>
> Like you, I don't like the idea of inventing another transport that  
> combines much of the functionality of existing transports but that  
> also crams in specialized DHT handling.  I think we should choose  
> existing protocols that solve the transport problem better than we  
> should be thinking about here, and then focus on the peer messages  
> themselves.
>
> I'm not too familiar with SCTP, though, so encapsulating it in UDP  
> could be a huge task.

I couldn't find a reference quickly when I went looking yesterday,  
but I am under the impression that a UDP encapsulation for SCTP has  
already been defined.
I have discovered recently that encapsulating a protocol in UDP for  
NAT traversal reasons is a pretty common trick. Here I am talking  
about a protocol that normally can run directly over IP.

- Philip

---

From: "David Barrett" <[email-masked]>
Date: Thu, 21 Jun 2007 17:56:42 -0700

> arbitrary incoming UDP.  

The demographic for my clients is mostly home users, but I see less than 1%
total UDP blockage overall.  So it happens, but so far as I can tell, not
often.  Seems like a good candidate for a relay service, rather than
justification for adoption of TCP.

> Some allow symmetric UDP with the outbound
> traffic

To clarify, your enterprise network even blocks UDP outbound?  Because
that's the only thing that will stop a port-hopping UDP-based
NAT-penetration protocol (other than deep packet inspection).

> I think reliable+congestion control+fragmentation control+nat traversal

It seems to me that congestion control only matters for bulk data delivery;
not media streaming.  For anything where throughput-maximization isn't an
issue, you can safely skip congestion control and do something like TFTP (as
Adam suggested).

Anyway, I'm with Adam: UDP NAT traversal is hard enough.  Even the
proponents of TCP NAT traversal admit it's harder than with UDP, so it seems
crazy to standardize on one of the highest risk, least-proven networking
techniques to accommodate a small subset of the network.

-david

---

From: "Adam Fisk" <[email-masked]>
Date: Thu, 21 Jun 2007 22:32:05 -0400

Hi Brian-

> I was told most firewalls, and especially enterprise firewalls did not

That's true in my experience as well.

Some allow symmetric UDP with the outbound traffic
> doing.

That makes perfect sense as well. To run a SIP proxy, you need to have a
public address in normal SIP land, as I'm sure you know better than me.
With p2p SIP, we'd ideally like to be able to do what you're talking about
without manually opening the port, although it's still not clear to me
that's a requirement.

I know I can't get incoming or outgoin UDP in my office unless I
> have a really good reason to ask the IT guys to open a port.  YMMV I

Any firewall not allowing outgoing UDP is pretty darn strict.  You're
completely sure they disallow outgoing?

> = TCP+ICE.

I agree with that 100% except for the NAT traversal.  The simplicity win of
straight TCP+ICE = HUGE (win that is).  I've had trouble getting the TCP
traversal stuff to work reliably, though, and I know there are cases where
UDP traversal works where TCP doesn't.  If NAT traversal is a requirement, I
think you'd want to be super robust there and allow both TCP traversal and
UDP traversal using a reliable UDP layer.

Maybe starting with only TCP traversal and just supporting ICE to allow for
future extensions would make sense?  I'd like to see more people getting
really solid results with the TCP traversal stuff before betting that big on
it though.  I could never get Saikat's stuff to work, and I tried it with
about 6 different NAT configurations.  I fully admit the problem was
probably on my end, and I could have stuck with it longer, but that was my
experience.  I also haven't tried their latest release, although it sounds
awesome.

-Adam

---

From: "Greger V. Teigre" <[email-masked]>
Date: Fri, 22 Jun 2007 09:25:11 +0200

From my experience with enterprises (i.e. not SMBs with LinkSys and 
dlink etc) deploying ipsec and sip-based apps: They come mostly in two 
flavors, those who have an open attitude and those with a close-it-all 
attitude. In the first category, you will mostly expect outgoing UDP and 
TCP to work, but normally no incoming at all. In the second category, 
you should expect no outgoing UDP and TCP allowed expect those 
explicitly allowed (and no, p2p is normally not allowed...) You may also 
see web proxies and a full block, i.e. only outgoing and incoming goes 
through the web proxy and no connections go through the firewall 
directly from the personal computer. Finally, you may also see some deep 
packet inspection or even intrusion detection prevention (i.e. active 
dropping based on application-level logic). Financial companies tend to 
be in this last category, but also others ex. IPR-based companies, like 
engineering, oil&gas etc can be in this category.

The close-it-all group is probably 5-10% of companies, but they tend to 
be big, so in terms of employees the percentage is larger. I have no 
numbers though.  The tendency is towards closing down...
So, one cannot solve the NAT/firewall-traversal without having IT 
department support. Thus, the solution should be something the IT guys 
find non-obtrusive and easy to understand.  They understand UDP, they 
understand TCP, they know how to create rules to open ports based on 
inside-triggers. The rest of the acronyms in this thread is grouped 
under one word: scary, let's stop it.
g-)

Adam Fisk wrote:
> That makes perfect sense as well. To run a SIP proxy, you need to have 
> a public address in normal SIP land, as I'm sure you know better than 
> me.  With p2p SIP, we'd ideally like to be able to do what you're 
> talking about without manually opening the port, although it's still 
> not clear to me that's a requirement. 
>
> I agree with that 100% except for the NAT traversal.  The simplicity 
> win of straight TCP+ICE = HUGE (win that is).  I've had trouble 
> getting the TCP traversal stuff to work reliably, though, and I know 
> there are cases where UDP traversal works where TCP doesn't.  If NAT 
> traversal is a requirement, I think you'd want to be super robust 
> there and allow both TCP traversal and UDP traversal using a reliable 
> UDP layer. 
>
> Maybe starting with only TCP traversal and just supporting ICE to 
> allow for future extensions would make sense?  I'd like to see more 
> people getting really solid results with the TCP traversal stuff 
> before betting that big on it though.  I could never get Saikat's 
> stuff to work, and I tried it with about 6 different NAT 
> configurations.  I fully admit the problem was probably on my end, and 
> I could have stuck with it longer, but that was my experience.  I also 
> haven't tried their latest release, although it sounds awesome.
>
> -Adam
>
>  
>
>

---

From: Enrico Marocco <[email-masked]>
Date: Fri, 22 Jun 2007 10:20:24 +0200

Adam Fisk wrote:
> raises a red flag for me.  If you're doing lock-step, why not just use
> TFTP (Trivial File Transfer Protocol).  If you're looking for
> simplicity, that's the way to go!

Actually it's not much stressed in the draft, but it's how we
implemented it; you could get a clue in section 5.2 "Retransmissions".
Thank you for pointing it out, we'll make it clearer.

Anyway, XPP collects features which are present in several other
protocols (TFTP, RDP, RTP, SCTP, DCCP, TCP...).  Unfortunately, none of
them has all those we think we need together: reliability,
fragmentation, NAT-friendly session establishment, and, yes,
unreliability too (i.e. a switch to turn reliability off).

---

From: Philip Matthews <[email-masked]>
Date: Fri, 22 Jun 2007 08:27:28 -0400

Regarding the choice of protocols (UDP vs TCP vs ...), here is my take:

There are three aspects that we are discussing:
1) Which transport protocol provides the right transport layer  
features needed for our protocol?
2) Which protocol provides the best NAT traversal properties?
3) Which protocol provides the best Firewall traversal properties?

Since I think our protocols have the possibility of being used very  
widely, I think it is important for us to make the right choice or  
choices for (1). If UDP offers the right semantics for (1), then we  
should use it, but if TCP or SCTP or DCCP or some other transport  
protocol offers the right semantics, then we should choose that  
instead. As Brian points out, SIP made the wrong choice for (1), and  
has been paying for it ever since.

If the right choice for (1) is a poor choice for (2), then what do we  
do? Well, we first note that not every host in the world is behind a  
NAT (only about 80%), and we also note that NATs get replaced every  
few years (I have heard some figure for this, but sorry, no solid  
reference). So we do the right thing for (1), and if needed between  
two specific peers, we encapsulate our chosen protocol for (1) over  
UDP which most people think provides the best NAT traversal  
properties. To be more precise, we arrange our connectivity checks to  
first try sending sending protocol (1) un-encapsulated, and then try  
sending it encapsulated over UDP and see which makes it through. This  
is not something that ICE supports today, but it is not difficult to  
add. In this way, we use un-encapsulated traffic where that is  
possible, and we use encapsulated traffic where that is needed. If we  
are successful, and if we make sure that BEHAVE has published specs  
to say how NATs should handle protocol (1), then we can hope that  
NATs will be upgraded so that the un-encapsulated case works more and  
more often as times goes on.

And what do we do about (3)?  What if our choice for (1) is a poor  
choice for (3)? Well, one option is to say that we encapsulate (1)  
with TCP rather than UDP in these cases. That is, in our connectivity  
checks, we try protocol (1) encapsulated over TCP as well as the  
other two choices and see what gets through. However, we need to be a  
careful here, because (3) is about policy and doing this is trying to  
circumvent that policy. So I think I agree with Greger when he says  
that you need IT department support. If the applications we run over  
our overlays are compelling, then the IT departments will open up  
their firewalls to allow it to come through. And if they don't find  
it compelling, then that is their decision and we shouldn't be trying  
to circumvent that  decision.

- Philip

On 22-Jun-07, at 03:25 , Greger V. Teigre wrote:

> >From my experience with enterprises (i.e. not SMBs with LinkSys  
> and dlink etc) deploying ipsec and sip-based apps: They come mostly  
> in two flavors, those who have an open attitude and those with a  
> close-it-all attitude. In the first category, you will mostly  
> expect outgoing UDP and TCP to work, but normally no incoming at  
> all. In the second category, you should expect no outgoing UDP and  
> TCP allowed expect those explicitly allowed (and no, p2p is  
> normally not allowed...) You may also see web proxies and a full  
> block, i.e. only outgoing and incoming goes through the web proxy  
> and no connections go through the firewall directly from the  
> personal computer. Finally, you may also see some deep packet  
> inspection or even intrusion detection prevention (i.e. active  
> dropping based on application-level logic). Financial companies  
> tend to be in this last category, but also others ex. IPR-based  
> companies, like engineering, oil&gas etc can be in this category.
>
> The close-it-all group is probably 5-10% of companies, but they  
> tend to be big, so in terms of employees the percentage is larger.  
> I have no numbers though.  The tendency is towards closing down...
> department support. Thus, the solution should be something the IT  
> guys find non-obtrusive and easy to understand.  They understand  
> UDP, they understand TCP, they know how to create rules to open  
> ports based on inside-triggers. The rest of the acronyms in this  
> thread is grouped under one word: scary, let's stop it.
> g-)
>
> Adam Fisk wrote:
>> Hi Brian-
>>
>>
>> That makes perfect sense as well. To run a SIP proxy, you need to  
>> have a public address in normal SIP land, as I'm sure you know  
>> better than me.  With p2p SIP, we'd ideally like to be able to do  
>> what you're talking about without manually opening the port,  
>> although it's still not clear to me that's a requirement.
>>
>> Any firewall not allowing outgoing UDP is pretty darn strict.   
>> You're completely sure they disallow outgoing?
>>
>> I agree with that 100% except for the NAT traversal.  The  
>> simplicity win of straight TCP+ICE = HUGE (win that is).  I've had  
>> trouble getting the TCP traversal stuff to work reliably, though,  
>> and I know there are cases where UDP traversal works where TCP  
>> doesn't.  If NAT traversal is a requirement, I think you'd want to  
>> be super robust there and allow both TCP traversal and UDP  
>> traversal using a reliable UDP layer.
>>
>> allow for future extensions would make sense?  I'd like to see  
>> more people getting really solid results with the TCP traversal  
>> stuff before betting that big on it though.  I could never get  
>> Saikat's stuff to work, and I tried it with about 6 different NAT  
>> configurations.  I fully admit the problem was probably on my end,  
>> and I could have stuck with it longer, but that was my  
>> experience.  I also haven't tried their latest release, although  
>> it sounds awesome.
>>
>> -Adam
>>
>>
>>
>>

---

From: "Brian Rosen" <[email-masked]>
Date: Fri, 22 Jun 2007 08:49:20 -0400

Right, no incoming UDP (or TCP) in something like 80% of enterprise.  That's
what I thought.  They usually have a DMZ and will allow incoming for several
ports into specific servers in the DMZ.  That's where the proxy goes.

I have lived in the 20% side for a while (one company ago).  They were
really stingy about opening a port.  They were more willing to open TCP
ports than UDP ports.  It's because the firewall has a better shot at
spotting bad behavior with TCP than with UDP.  Took me six months to get
them to open 5060 to  a specific server.  Of course, as Greger said, if the
firewall can be statefull for the protocol you want, then it's easier to get
them to approve.

Brian

  _____  

From: Greger V. Teigre [mailto:[email-masked]] 
Sent: Friday, June 22, 2007 3:25 AM
To: Adam Fisk
Cc: Brian Rosen; Emil Ivov; Philip Matthews; P2PSIP Mailing List
Subject: Re: [P2PSIP] Peer protocol, DHT and running code

>From my experience with enterprises (i.e. not SMBs with LinkSys and dlink
etc) deploying ipsec and sip-based apps: They come mostly in two flavors,
those who have an open attitude and those with a close-it-all attitude. In
the first category, you will mostly expect outgoing UDP and TCP to work, but
normally no incoming at all. In the second category, you should expect no
outgoing UDP and TCP allowed expect those explicitly allowed (and no, p2p is
normally not allowed...) You may also see web proxies and a full block, i.e.
only outgoing and incoming goes through the web proxy and no connections go
through the firewall directly from the personal computer. Finally, you may
also see some deep packet inspection or even intrusion detection prevention
(i.e. active dropping based on application-level logic). Financial companies
tend to be in this last category, but also others ex. IPR-based companies,
like engineering, oil&gas etc can be in this category.

The close-it-all group is probably 5-10% of companies, but they tend to be
big, so in terms of employees the percentage is larger. I have no numbers
though.  The tendency is towards closing down...
So, one cannot solve the NAT/firewall-traversal without having IT department
support. Thus, the solution should be something the IT guys find
non-obtrusive and easy to understand.  They understand UDP, they understand
TCP, they know how to create rules to open ports based on inside-triggers.
The rest of the acronyms in this thread is grouped under one word: scary,
let's stop it.
g-)

Adam Fisk wrote: 

Hi Brian-

arbitrary incoming UDP.  

 

-Adam

 

  _____

---

From: "Bruce Lowekamp" <[email-masked]>
Date: Fri, 22 Jun 2007 09:38:39 -0400

We need to look carefully at the requirements for P2PSIP before we
make a decision on transport protocol.

TCP may have (I would vote would have) been a better choice for SIP
than UDP.  But that's not necessarily relevant.  They're different
protocols.

SIP requires a large number of clients to connect with small numbers
of well-provisioned (including well-networked) servers.

P2PSIP requires large numbers of peers to connect with each other.
There is no requirement (in many use cases) about the quality of
provisioning for these peers.  Typically a peer will establish log(N)
connections in a routing table and may be in regular correspondence
with some other number of peers based on the activity of the user.

Given enough resources (I think someone in this or another thread
recently mentioned using P2PSIP on motes, so it might be an issue) to
establish and maintain that many TCP connections, and given that they
are able to punch TCP through any intervening NATs and firewalls
without resorting to a relay, I think TCP (or similar stream-based) is
probably the right transport protocol for those connections.  But
there will be many, many cases where a UDP connection can be punched
through and TCP can't.

I don't think TCP is the right transport for iterative routing.  It
makes no more sense there than it does for DNS queries.

Supporting multiple transport choices shouldn't be a problem for a
peer protocol.  Speaking for RELOAD, peers are able to advertise
multiple PEER-IP-PORT attributes in their SOURCE-INFO, which means
they can specify both a TCP and UDP port (or either if the peer
doesn't want to or can't support both).   When NAT traversal is
required, the OPEN method includes an SDP attribute for ICE
negotiation that can, of course, include both udp and tcp candidates.

I think we need to look carefully at the requirements we have, but I
personally think there are enough arguments for leaving UDP in that I
don't think we should say anything stronger than SHOULD prefer TCP.

Bruce

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 22 Jun 2007 10:22:25 -0400

Since this canard seems to pop up again and again, without any basis  
in measurement: We've been running actual experiments, as opposed to  
hear-say, on large-volume TCP servers. We have tested up to 400,000  
simultaneous connections on a Linux box. I don't think a P2P system  
is going to maintain that many - the NAT is going to run out of  
mappings looong before that... Motes aren't going to be limited by  
TCP connections; they are going to be energy-limited, which makes  
participation in large-scale P2P a rather dubious idea to begin with.

---

From: "Bruce Lowekamp" <[email-masked]>
Date: Fri, 22 Jun 2007 11:15:01 -0400

Speaking of canards, are you suggesting that all P2PSIP peers are
going to be large-volume linux servers?

Motes, cell phones, hardphones, etc are slightly different devices.
Many, of course, can maintain more than enough TCP connections, but I
fail to see how the large-volume TCP server capacity illustrates that.
 If there are published numbers on the cpu power, memory footprint,
and power consumption requirements of udp vs tcp for periodic
messaging on embedded devices, that would be very interesting.  A
comparison that included TLS and DTLS (or other secure protocols)
would be even more interesting, as I suspect the cpu & power tradeoffs
are different.

None of this addresses the issues of iterative routing or NAT/firewall
traversal that are also strong arguments that TCP cannot be the only
transport protocol.

Bruce

---

From: "David Barrett" <[email-masked]>
Date: Fri, 22 Jun 2007 11:05:03 -0700

That matches what I've seen in terms of UDP nat and firewalls:

-          20% not firewalled, not natted

-          13% natted with "full cone" behavior

-          6% inbound-firewalled but not natted

-          45% natted with "restricted cone" behavior

-          16% natted with "symmetric" behavior (of some flavor)

-          1% outbound-firewalled (unknown of they're also natted)

The first two (comprising 33% of clients) can accept arbitrary incoming
connections once the port is open, so all both are more or less globally
available.  The next 51% you can establish connections with using
simultaneous connection.  And the remaining 16% get trickier, but still
work.

I've only seen ~1% of the network truly blocked from using UDP, but there
could be some "self-selection" going on that is skewing those results.

-david

  _____  

From: Brian Rosen [mailto:[email-masked]] 
Sent: Friday, June 22, 2007 5:49 AM
To: 'Greger V. Teigre'; 'Adam Fisk'
Cc: 'Emil Ivov'; 'Philip Matthews'; 'P2PSIP Mailing List'
Subject: RE: [P2PSIP] Peer protocol, DHT and running code

Brian

  _____  

Adam Fisk wrote: 

Hi Brian-

arbitrary incoming UDP.  

 

-Adam

 

  _____

---

From: Dean Willis <[email-masked]>
Date: Sat, 23 Jun 2007 17:33:06 -0500

> have a really good reason to ask the IT guys to open a port.  YMMV I guess.

Right. UDP hole-punching techniques do not, in general, work on
enterprise firewalls, such as the one operated by the IT department of
my most recent former employer. Said firewalls block all incoming UDP,
except for certain allowed ports to certain allowed hosts (needed for DNS).

On the other hand, Skype worked just fine from behind that same firewall.

--
Dean

---

From: Salman Abdul Baset <[email-masked]>
Date: Sat, 23 Jun 2007 19:04:05 -0400 (EDT)

> On the other hand, Skype worked just fine from behind that same firewall.

Most likely it used TCP.

---

From: Emil Ivov <[email-masked]>
Date: Sun, 24 Jun 2007 02:04:24 +0200

> except for certain allowed ports to certain allowed hosts (needed for DNS).

This probably means that the people administrating the firewall most 
likely did not want any media/voip traffic coming in and were explicitly 
blocking it. If this is really the case then I wonder whether we should 
be worried about it since the same people could very easily find ways to 
block media and voip even if they were transported by tcp.

> On the other hand, Skype worked just fine from behind that same firewall.

Are you certain that the reason for this was protocol related. Isn't it 
likely that this was either intentional or the firewall was not advanced 
enough and was not analyzing traffic any further beyond transport and 
port. In either case I doubt the phenomenon is something that TCP would 
always be a solution for.

Cheers
Emil

---

From: "Wei Gengyu" <[email-masked]>
Date: Sun, 24 Jun 2007 14:50:15 +0800

Hi, 

If you use MS windows xp, you can see your wirewall configurations. 
The administrator of your PC can make whether to let any traffic flow in. 
The IT staffs can have the enterprise wirewalls work in the same way.

However, most end users do not know or bother to change the firewall configurations.
But most enterprise IT staffs often check and change that to meet the requirements of their companys.

In a word, people can control flows at firewall actively, no matter which protocols and ports to adopt.
If people let the firewall configurations as default values, UDP hole-punching techniques do work often.   

Regards, 

Gengyu

---

From: "Henry Sinnreich" <[email-masked]>
Date: Sun, 24 Jun 2007 15:46:14 -0700

> I still think that, especially for highly unpredictable

> I certainly hope I am not the only one left to believe this.

Count me in as well and also add _measured performance_ on the network.

Thanks, Henry

-----Original Message-----
From: Enrico Marocco [mailto:[email-masked]] 
Sent: Thursday, June 21, 2007 3:10 PM
To: Brian Rosen
Cc: 'Emil Ivov'; 'Philip Matthews'; 'P2PSIP Mailing List'
Subject: Re: [P2PSIP] Peer protocol, DHT and running code

-- 
Ciao,
Enrico

---

From: "David Barrett" <[email-masked]>
Date: Sun, 24 Jun 2007 17:06:49 -0700

> From: Henry Sinnreich [mailto:[email-masked]]

So I think everyone would agree that running code is important.  But where
the rubber meets the road is when deciding what to do in the *absence* of
running code.

Personally, I believe the IETF should follow, not lead, the standardization
process.  So if there is no running code to measure and refine, then it's a
sign that the IETF has gotten out ahead of the process.  If this happens,
rather than charging boldly ahead and designing in a vacuum, I think the
priority should be on:

1) Document and instrument widely-deployed running code to be measurable
2) Help nascent running code grow to a point where it's widely-deployed.

Right now it seems that ICE is in the (1) camp: code out there is widely
deployed, but it's unclear exactly what it is, and it's not instrumented to
generate the data we need (peer connectivity and connection setup latency).

P2PSIP appears to be in the (2) camp: code is out there but it isn't used on
a widespread basis and thus might not be really representative of the
problem set it's designed to solve.

In the latter case, we should ask ourselves "why aren't the current P2PSIP
implementations being widely used, and what can we do to help?"  In the
former case, we should focus on "Who is behind these widespread
implementations ICE (MSN, Google, etc) and can we have them share exactly
what their doing and provide data on how well it works?"

Regarding P2PSIP, at the very least, we could start "eating our own
dogfood": we should get the code together in a central location, write all
our drafts to be no more or less than what that code does, and then start
using it.  As we encounter real-world problems that prevent us from using it
(the hardest one being "nobody I want to call is using it"), then we focus
on solving *those* problems before any other.

Regarding ICE, it seems that we could put together some sort of formal
ambassador to approach these teams with a clear request for a specification
of what they're doing and data on how well it works.  We should understand
what their concerns are about getting us that data (probably some
engineering, intellectual property, and possibly privacy concerns), work to
resolve those, and then base our standardization efforts on iterations of
that.

But if neither of these can be accomplished now, then I think a totally
reasonable course of action is to simply wait until they can be.  At least,
I feel that's more reasonable than simply assuming measurable running code
and real-world data aren't necessary to standardize these protocols.

-david

---

From: "Henry Sinnreich" <[email-masked]>
Date: Sun, 24 Jun 2007 17:50:34 -0700

David,

> But where the rubber meets the road is when deciding what to do in the

> *absence* of running code.

There is an abundance of running code and performance measurements for
most DHTs, see for example openDHT.org, Chord, etc.

What we don't have now is open source code for recent ICE specs (as far
as I know) and most of all, measurements with ICE, even for CS SIP
systems. 
This has been mentioned by others on this list and makes me uneasy as
well.

IMHO standards that are not backed up by OS code may fail in spectacular
ways. We don't need to mention here some examples :-)

Thanks, Henry

-----Original Message-----
From: David Barrett [mailto:[email-masked]] 
Sent: Monday, June 25, 2007 2:07 AM
To: Henry Sinnreich; 'Enrico Marocco'; 'Brian Rosen'
Cc: 'Emil Ivov'; 'Philip Matthews'; 'P2PSIP Mailing List'
Subject: RE: [P2PSIP] Peer protocol, DHT and running code

> Count me in as well and also add _measured performance_ on the
network.

So I think everyone would agree that running code is important.  But
where
the rubber meets the road is when deciding what to do in the *absence*
of
running code.

Personally, I believe the IETF should follow, not lead, the
standardization
process.  So if there is no running code to measure and refine, then
it's a
sign that the IETF has gotten out ahead of the process.  If this
happens,
rather than charging boldly ahead and designing in a vacuum, I think the
priority should be on:

Right now it seems that ICE is in the (1) camp: code out there is widely
deployed, but it's unclear exactly what it is, and it's not instrumented
to
generate the data we need (peer connectivity and connection setup
latency).

P2PSIP appears to be in the (2) camp: code is out there but it isn't
used on
a widespread basis and thus might not be really representative of the
problem set it's designed to solve.

In the latter case, we should ask ourselves "why aren't the current
P2PSIP
implementations being widely used, and what can we do to help?"  In the
former case, we should focus on "Who is behind these widespread
implementations ICE (MSN, Google, etc) and can we have them share
exactly
what their doing and provide data on how well it works?"

Regarding P2PSIP, at the very least, we could start "eating our own
dogfood": we should get the code together in a central location, write
all
our drafts to be no more or less than what that code does, and then
start
using it.  As we encounter real-world problems that prevent us from
using it
(the hardest one being "nobody I want to call is using it"), then we
focus
on solving *those* problems before any other.

Regarding ICE, it seems that we could put together some sort of formal
ambassador to approach these teams with a clear request for a
specification
of what they're doing and data on how well it works.  We should
understand
what their concerns are about getting us that data (probably some
engineering, intellectual property, and possibly privacy concerns), work
to
resolve those, and then base our standardization efforts on iterations
of
that.

But if neither of these can be accomplished now, then I think a totally
reasonable course of action is to simply wait until they can be.  At
least,
I feel that's more reasonable than simply assuming measurable running
code
and real-world data aren't necessary to standardize these protocols.

-david

---

From: "David Barrett" <[email-masked]>
Date: Sun, 24 Jun 2007 22:33:13 -0700

> ways. We don't need to mention here some examples :-)

Heh, totally agree.

I guess what I'd like us to do is choose our top use case and build/adopt an
application for real-world use and testing.  For example, I'd like to see
"OpenSkype".  Some complete, end-to-end implementation of P2PSIP, ICE, and
STUN that demonstrates the combination does in fact produce effective
results.

Then, we can measure the hell of that thing and figure out where it's good
and bad.  And once it's been proven to be "good enough" in one area (ie,
meeting the base requirements for a given use case), then we can discuss how
to extend that to cover the next-highest-priority use case, and so on down
the list.

In this way we focus our energy on quantifying and solving real-world
problems that affect actual users, reducing distractions and maximizing the
use of our limited resources.

Does anyone agree with this plan?  Anybody know of a good OpenSkype to
adopt?

Or, if we can't do this plan, what an alternative plan?

-david

---

From: "Greger V. Teigre" <[email-masked]>
Date: Mon, 25 Jun 2007 11:15:18 +0200

I share the concerns on the absence of running open-source code. In my 
understanding, there is (mostly) code for various pieces, but we are 
struggling in finding example systems (preferably open-source) that 
cover everything we want to cover.  The risk is thus to over-engineer 
(design by committee) instead of standardizing based on experience about 
what works.

IMHO, this has a lot to do with design-choices: If we strive to 
logically separate the various components from each other, we are in a 
better position to choose what is known to work, as well as more 
directly rely on other working groups' work (I think this point has been 
made earlier, but it is worth bringing up again). However, if we try to 
mix all functions together in a new P2PSIP protocol (stack?), we will 
struggle to adapt and integrate, and we have no running code we really 
can rely on, because measurements are no longer valid.

Unless this working group points in a direction where the existing 
implementors of SIP-based p2p quickly can amend/change their 
implementation, I think we will go astray.
g-)

---

From: Philip Matthews <[email-masked]>
Date: Mon, 25 Jun 2007 07:47:17 -0400

I agree with Emil.
- Philip

On 23-Jun-07, at 20:04 , Emil Ivov wrote:

> This probably means that the people administrating the firewall  
> most likely did not want any media/voip traffic coming in and were  
> explicitly blocking it. If this is really the case then I wonder  
> whether we should be worried about it since the same people could  
> very easily find ways to block media and voip even if they were  
> transported by tcp.
>
> Are you certain that the reason for this was protocol related.  
> Isn't it likely that this was either intentional or the firewall  
> was not advanced enough and was not analyzing traffic any further  
> beyond transport and port. In either case I doubt the phenomenon is  
> something that TCP would always be a solution for.
>
> Cheers
> Emil
>> --
>> Dean

---

From: "Henry Sinnreich" <[email-masked]>
Date: Mon, 25 Jun 2007 11:14:04 -0700

David Barrett wrote:

>Some complete, end-to-end implementation of P2PSIP, ICE, and STUN that
>demonstrates the combination does in fact produce effective results.

Completely agree! 

Maybe the WG chairs could suggest a process to keep running code and P2P
SIP Internet Drafts in sync; that is put a brake on drafts until someone
catches up with code and measurements.

As mentioned, there is ample code measurements and research reports for
NAT-less P2P, such as Bamboo, etc., but the ICE part for P2P though
conceptually OK as reported here may have some surprises. 

How much time would a call setup take when routing over multiple hops
with NATs in the way? This is a critical measurement objective IMHO.

Question: Can the P2PSIP WG avoid the avalanche of code-less I-Ds seen
in other SIP WGs? Or is this idea too outlandish :-)

Henry

-----Original Message-----
From: David Barrett [mailto:[email-masked]] 
Sent: Sunday, June 24, 2007 10:33 PM
To: Henry Sinnreich; 'Enrico Marocco'; 'Brian Rosen'
Cc: 'Emil Ivov'; 'Philip Matthews'; 'P2PSIP Mailing List'
Subject: RE: [P2PSIP] Peer protocol, DHT and running code

Heh, totally agree.

I guess what I'd like us to do is choose our top use case and
build/adopt an
application for real-world use and testing.  For example, I'd like to
see
"OpenSkype".  Some complete, end-to-end implementation of P2PSIP, ICE,
and
STUN that demonstrates the combination does in fact produce effective
results.

Then, we can measure the hell of that thing and figure out where it's
good
and bad.  And once it's been proven to be "good enough" in one area (ie,
meeting the base requirements for a given use case), then we can discuss
how
to extend that to cover the next-highest-priority use case, and so on
down
the list.

In this way we focus our energy on quantifying and solving real-world
problems that affect actual users, reducing distractions and maximizing
the
use of our limited resources.

-david

---

From: Dean Willis <[email-masked]>
Date: Fri, 29 Jun 2007 00:32:06 -0500

> very easily find ways to block media and voip even if they were  

No, the inbound rule looks like "DISCARD UDP ANY ANY" unless a prior  
rule allowed DNS through to the DNS server.

They weren't singling out "media" traffic.  They were just using a  
large blunt object to club all UDP traffic into submission, because  
there's no good way to tell whether a node inside the firewall wants/ 
needs to receive arbitrary UDP traffic right now.

This is a VERY common firewall policy for enterprises basing the  
firewalls on conventional router ACLs, Linux iptables, Cisco firewall  
feature set, and so on.

And yes, I installed a large number of enterprise firewalls that did  
exactly this back when I was doing that for a living.

> something that TCP would always be a solution for.

Huh?

Yeah, it was protocol related. The firewall effectively blocked all  
UDP, and allowed TCP only when the connection was established from  
within the firewall. Again, this is pretty much standard for  
companies that don't restrict things further by only allowing HTTP  
traffic using an HTTP proxy in their firewall, and otherwise also  
blocking TCP. That's why applications that REALLY work through  
firewalls can do things like tunnel media over HTTP requests.

As to whether this means the "firewall was not advanced enough", I  
suppose that's open to interpretation. But this IS what most  
enterprise firewalls do, and it was at a company that makes a large  
share of the enterprise firewalls in use today (so they ought to  
know) . . . and they're still using it now as far as I know.

--
Dean

---

From: Emil Ivov <[email-masked]>
Date: Fri, 29 Jun 2007 12:16:17 +0200

Dean Willis wrote:
> large blunt object to club all UDP traffic into submission

And according to you what were they trying to achieve?

I guess I did not express myself clearly. My point is that if someone 
does not want or does not care about UDP traffic, it's most probably 
because they don't want or do not care about real-time traffic. We 
should not try to find ways of circumventing firewall policies and cheat 
firewall administrators (... we couldn't do that reliably even if we 
wanted to).

Almost all NATs/firewalls previewed for residential use allow responses 
to UDP traffic started from inside their network and therefore allow for 
hole punching.

Enterprise firewalls on the other hand are strictly configured to comply 
with a predefined policy and it is not our business to care what 
protocols these policies include.

The reasoning that we had while working on XPP was that it should work 
as easily and in as many network configurations as possible. That's why 
we chose UDP and we still believe it is the right way to go as it would 
greatly facilitate implementation and deployment.

We are currently building a public overlay using SIPDHT2 (our current 
draft implementation available on 
http://sipdht.sourceforge.net/sipdht2/index.html). We should have it 
ready in a month and a couple of months later we should also have 
support inside SIP Communicator and thus a fully functional p2p 
real-time communications environment. We'll publish results as soon as 
we have them, but so far the situation is quite promising just as it is 
(using UDP).

Still, it is most certainly up to the working group to make the decision 
and if there is consensus on using TCP (rather than UDP) and a MUST for 
ICE usage is a reasonable requirement to impose on implementors (rather 
than a SHOULD), then we would try to come up with a revised version that 
uses TCP as a transport.

Cheers
Emil

>, because

---

From: Dean Willis <[email-masked]>
Date: Fri, 29 Jun 2007 10:27:27 -0500

On Jun 29, 2007, at 5:16 AM, Emil Ivov wrote:

>
>
> Dean Willis wrote:
>> No, the inbound rule looks like "DISCARD UDP ANY ANY" unless a  
>> prior  rule allowed DNS through to the DNS server.
>> They weren't singling out "media" traffic.  They were just using  
>> a  large blunt object to club all UDP traffic into submission
>

A reasonable level of security.

> I guess I did not express myself clearly. My point is that if  
> someone does not want or does not care about UDP traffic, it's most  
> probably because they don't want or do not care about real-time  
> traffic. We should not try to find ways of circumventing firewall  
> policies and cheat firewall administrators (... we couldn't do that  
> reliably even if we wanted to).

It wasn't that they didn't want real-time traffic. It is a matter of  
simple expedience -- since UDP traffic is hard to manage, it gets  
blocked. If we could have come up with a reasonable way to allow RTP  
only when a node inside the network was expecting it, we'd have made  
that work. And no, coupling a SIP proxy to the firewall via a control  
protocol is NOT a reasonable solution in a large enterprise.

> Almost all NATs/firewalls previewed for residential use allow  
> responses to UDP traffic started from inside their network and  
> therefore allow for hole punching.
>

True.

It's a question of reach. Are you happy having only residential  
users? If you're happy, I'm happy for you.

If that doesn't make you happy, then you might want to think about  
how to get to enterprise users, keeping in mind that their security  
people aren't evil -- just conservative, lazy, and using technology  
that might not be what you want them to use.

--
dean

---

From: Emil Ivov <[email-masked]>
Date: Sat, 30 Jun 2007 01:31:45 +0200

Dean Willis wrote:
>>> prior  rule allowed DNS through to the DNS server. They weren't
>>> singling out "media" traffic.  They were just using a  large
>>> blunt object to club all UDP traffic into submission

I am not so convinced since I don't see how allowing tcp connections is
more secure than having UDP with address and port dependent filtering
(which is what conntrack in iptables is doing for example). Can you
please give us some references or describe the risks that this may be
causing?

I'd be more inclined to think that most of the enterprises
unconditionally blocking UDP, do so in order to prevent their employees
from using Internet access for purposes, not directly related to their
work. Most such enterprises also tend to block all traffic other than
http/https.

>>  policies and cheat firewall administrators (... we couldn't do
>> that reliably even if we wanted to).
> 
> blocked.

Implementing address and port specific filtering for UDP in a firewall
is only slightly different from keeping track of TCP connections. As
far as I know most firewall management interfaces allow users to do it.

> If we could have come up with a reasonable way to allow RTP only when
> a node inside the network was expecting it, we'd have made that work.
> And no, coupling a SIP proxy to the firewall via a control protocol
> is NOT a reasonable solution in a large enterprise.

You really lost me here. Why is this a problem? And why do you need to
have a SIP proxy on the firewall to do it? You can do this for UDP just
as easily as you could for TCP.

If this would help convince you, you might want to have a look at the
following iptables example:

iptables -A INPUT  -p udp -m state --state ESTABLISHED     -j ACCEPT
iptables -A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT

> True. It's a question of reach. Are you happy having only residential

Well even though I'd really like all enterprises in the world to
contribute to our overlay, I know that I can spend the whole weekend
thinking about it and I still won't be able to figure how to standardize
a protocol in a way that would allow me to get enterprise users if they
don't want me to get them. (Unless of course standardization includes
physical intervention and violation of human rights :) ).

Emil

> -- dean

---

From: Dean Willis <[email-masked]>
Date: Sat, 30 Jun 2007 11:14:03 -0500

> please give us some references or describe the risks that this may be

It probably isn't any more secure, but historically (and currently)  
large-scale enterprise firewalls often don't do connection tracking  
for UDP. This probably has a lot to do with the problem of high-speed  
connection state replication across a large pool of firewalls.

> work. Most such enterprises also tend to block all traffic other than

My experience differs, but certainly some enterprises do have such  
rules and that accounts for Skype and other successful applications  
use of tunneling over HTTP. Of course we're not going to standardize  
that in the IETF, but protocols need to be amenable to this sort of  
tunneling even if we don't standardize it.

> far as I know most firewall management interfaces allow users to do  

But the FIREWALL doesn't keep track of TCP connections. It just looks  
at syn/ack to determine whether an incoming packet appears to be part  
of an established TCP flow. If it is an initial connection "opening"  
from outside, it gets discarded. This is completely stateless  
inspection on the firewall. It does leave one a bit open to SYN-flood  
DOS attacks, but there are other workarounds for that.

> iptables -A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT

ok, now make this work on an enterprise that's multihomed 10 times,  
with 4 firewall engines in parallel on each access link to provide  
the needed capacity and reliability, and where any two-packet  
sequence of incoming UDP might hit either two different firewall  
engines or even appear on two entirely different access links. To  
make this work, you've got to have the entire state pool of the UDP  
connection tracking mechanism mirrored in real-time across a whole  
bunch of boxes, and the mirroring has to be fast enough to work  
between the first packet of a flow and the second packet of that  
flow. But given physics (186,000 miles an hour isn't just a good  
idea, it's the law, except maybe for quantum twinning), it's going to  
be very, vary hard to replicate that fast.

That's why little residential/SOHO boxes can do things like uPNP,  
connection tracking for symmetric UDP, and so on, and big guys either  
can't or just can't afford too.

Also, make it work for SIP early media, where a flow from the  
"outside" starts before the "inside" node sends a single UDP media  
packet and before the "outside" node has even returned a 200 OK  
indicating from where it will be listening (and note that it NEVER  
says from where it will be sending -- if it's polite enough to use  
symmetric RTP that will help, but we won't KNOW it's going to).

Now if you can figure all this out, let me know and I'm sure I can  
find some startup money to make us both rich!

--
Dean

---

From: Emil Ivov <[email-masked]>
Date: Sun, 01 Jul 2007 02:25:23 +0200

Dean Willis wrote:
>> I am not so convinced since I don't see how allowing tcp 
>> connections is more secure than having UDP with address and port 
>> dependent filtering (which is what conntrack in iptables is doing 
>> for example). Can you please give us some references or describe 
>> the risks that this may be causing?
> 

Large scale enterprises are quite often behind NATs. Few companies
today could get their hands on enough IPv4 addresses and this is even
more so for the big ones (especially when outside the US). Therefore
most of these are already capable of handling connection tracking.

> that in the IETF,

Yes, and this is exactly the point I am trying to make here.

> but protocols need to be amenable to this sort of tunneling even if 
> we don't standardize it.
> 
>  at syn/ack to determine whether an incoming packet appears to be 
> part of an established TCP flow. If it is an initial connection 
> "opening" from outside, it gets discarded. This is completely 
> stateless inspection on the firewall. It does leave one a bit open to
>  SYN-flood DOS attacks, but there are other workarounds for that.

Well, isn't it quite obvious that this kind of firewalls won't work for
P2P even when TCP based because it won't allow for end-to-end
simultaneous connection establishment?

> find some startup money to make us both rich!

Heey, no problem. I am sure that if we start entangling photons right
now in a month or two we'll be able to build a quantum firewall and
it'll be doing all kinds of magic, so you better get that money ready! :-))

Emil

> -- Dean