Subject: [Sip] The cost of UDP

From: "Avshalom Houri" <[email-masked]>
Date: Mon, 15 Jul 2002 12:08:54 +0300

Was caught in a middle of a row during the meeting and could not get to 
the microphone.

Wanted to remark on the remark that UDP is less resource consuming then 
TCP and you can
get more users on a machine with UDP.

Well, it is very much dependent on your implementation. I have seen an 
implementation that can
hold 20-30K users on a simple NT machine quite easily. Saying that UDP 
enable you to support
more users is a bit misleading what happens is that you consume less 
system resources but you
end up in consuming much more application resources since now you need to 
do what TCP does
for you in the application level. Using TCP and TLS also goes with the 
lines of using existing
protocols to solve problems and not having to solve these problems again 
and again in the
applications.

As can be understood, I am all for deprecating UDP while SIP is still 
young and it can be done.
Regarding backward compatibility issues, I am not so worried about it. I 
am not aware of
a very large SIP deployments that are used by millions of users currently. 
It will be relatively
easy to modify proxies and servers to accept and initiate only TCP. 
Regarding clients, in the time frame that we
are speaking about, 12-18 months, the clients can also be modified.

Avshalom

---

From: "Dean Willis" <[email-masked]>
Date: Tue, 16 Jul 2002 02:00:36 -0500

Avshalom said:
--------------

end up in consuming much more application resources since now you need
to do what TCP does 
---------------

The "TCP limit" is based on the fact that most sockets-based
implementations of TCP use up one filesystem file handle per connected
socket. Since most kernel limitations use an unsigned 16-bit index for
the filehandle table, this puts a fairly hard upper bound of 64k
connected sockets. Given that one is likely to need a few to support the
OS and other applications, this makes a firm upper bound of around
64,000. Of course, many operating systems don't actually support this
many, with many TCP implementations actually breakikng at a much smaller
number, perhaps 16 or 256, but we're not likely to see commercial-grade
proxy systems built on these systems.

Of course, with connection-reuse in a proxy-proxy scenario, one socket
might actually be supporting thousands of users. The real issue is on
"boundary" proxies, where there is fan-in (or fan-out) to many UAs or
small (residential?) proxies.

What's the impact?

Let's say we average one 7-message dialog sequence per hour per device.
That's 7/messages per hour, or about 2 messages per second per 1,000 UA.
With an average proxy capable of handling 1,000 messages per second, we
should be able to handle 500,000 UA per proxy. That's clearly a larger
number than 64k, so there IS a potential for fan in/out limitations
being induced by TCP on larger servers.  Of course, if one factors in
presence and a little more traffic, it's not at all unreasoablle to
expect more than 70 messages per hour per UA, which with this size proxy
gets us CPU bound at around 50,000 clients, so the TCP constraint isn't
binding.

In other words, my analysis seems to indicate that for any "reasonable"
model of proxy fanout that the TCP connected socket limitation is only
to likely to impact really BIG proxy servers using a monolithic
"mainframe" style architecture not common in today's distributed
environments. In other words, it's just not much of a problem.

Another aspect to consider is the "half open connection limit". There
exists a constraint in many TCP implementations as to how many sockets
may be in a connecting (not connected) state at any given time. Just a
few years back, this number was something like 16 in a certain "robust
commercial UNIX". I think it's commonly much larger now. This was highly
problematic with HTTP 1.0 and it's very short-lived TCP connections, but
our model of persistent TCP/TLS connections and connection reuse greatly
reduce this liability.

Consider that 50,000 client server. If every single TCP connection is
broken and restored over the course of an hour, that gives us something
like 5 half/open connections at any given time. If however every
connection is restarted at exactly the same time, we DO have a bit more
of a problem, so there may well be an issue in this regard.

--
Dean

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: "Shane McCormack" <[email-masked]>
Date: Tue, 16 Jul 2002 10:24:19 +0100

Hi,
Would SCTP solve any resource issues here?

Shane

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of Dean
Willis
Sent: 16 July 2002 08:01
To: 'Avshalom Houri'; [email-masked]
Subject: RE: [Sip] The cost of UDP

Avshalom said:
--------------

What's the impact?

--
Dean

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 16 Jul 2002 08:24:58 -0400

The "reboot building" restart issue is something we probably need to do 
something about in any event. Using TCP to the outbound proxy compounds 
the problem that already exists due to the REGISTER flood. But maybe all 
the devices will be fighting for DHCP IP addresses and thus spread out 
the onrush.

---

From: "Dean Willis" <[email-masked]>
Date: Tue, 16 Jul 2002 07:42:39 -0500

> Would SCTP solve any resource issues here?

I don't know that we have enough deployment experience to really say.
Many people think so. A lot depends on the way that the SCTP library is
actually implemented, and I personally haven't implemented one yet . . .
Maybe somebody with implementation experience can say. Approaches that
follow the TCP Berkeley-sockets model of file-handle allocation would
presumably have the same problem. Other "raw socket" approaches might
not. Of course, one COULD presumably do a TCP implementation that isn't
file-handle bound, but I wouldn't recommend the one I did as a class
project n-many years ago as a starting point.

One place that SCTP should REALLY help is on optimizing high-volume
server-server links. TCP in this role has this annoying "head of line
blocking" quirk around retransmission of lost packets that SCTP fixes
nicely.

--
Dean

---

From: "Cullen Jennings" <[email-masked]>
Date: Tue, 16 Jul 2002 17:55:47 -0700

Few thoughts on this ....

64,000 subscriber per host is not very many - people that are going to
deploy millions of users don't want a boat load of servers to manage. 64,000
could correspond to around 50 cps. I can imagine proxies that easily do 500
cps on a cheap machine today and I can imagine proxies that do 30,000 cps on
a $1000 machine 10 years from now. Perhaps no one will care about that level
of performance but the point I am making is simple - 64,000 users per host
is very low.

If a host supported several IP address on a single interface, it might be
able to do more than 64k TCP connections. Should work in theory but probably
does not work in practice.

The following scenario is quite realistic - We end up with a city with
several million IP phones. This city regularly looses power resulting in all
UA simultaneously rebooting and trying to register.

Cullen

> Sent: Tuesday, July 16, 2002 12:01 AM

---

From: "Shane McCormack" <[email-masked]>
Date: Tue, 16 Jul 2002 13:59:53 +0100

I agree, I know Coritel university in Italy has done some SIP SCTP work
using the lksctp open source linux kernel implementation of SCTP
http://sourceforge.net/projects/lksctp
and kphone/dissipate sip stack.
http://www.wirlab.net/kphone/

Maybe some more research should be done? or is being done?

Regards
Shane

-----Original Message-----
From: Dean Willis [mailto:[email-masked]]
Sent: 16 July 2002 13:43
To: 'Shane McCormack'; 'Avshalom Houri'; [email-masked]
Subject: RE: [Sip] The cost of UDP

*This message was transferred with a trial version of CommuniGate(tm) Pro*

--
Dean

---

From: "Dean Willis" <[email-masked]>
Date: Wed, 17 Jul 2002 04:15:00 -0500

> If a host supported several IP address on a single interface, 
> it might be able to do more than 64k TCP connections. Should 
> work in theory but probably does not work in practice.

Nope. It is a kernel limitation on the number of file handles available,
as I understand it.

--
Dean

---

From: Michael Thomas <[email-masked]>
Date: Wed, 17 Jul 2002 08:54:53 -0700 (PDT)

> Nope. It is a kernel limitation on the number of file handles available,

This doesn't really strike me as a reason to avoid
something. It may be crufty to change, but there
is nothing sacrosanct about 16 bit file
handles. There are *lots* of services where you'd
like to pile as many occassional users as possible
onto the server. IM comes to mind here too.

That said, I think that this entire discussion is
somewhat missing the point: how does post dial
delay fare with all things considered (e2e, qos,
security) with TCP and UDP? A self-contained
signed UDP exchange (kerberos, SMIME...) would
take 2 messages (invite/response). The equivalent
with TCP and TLS would take 10 messages (3, 5, 2).
This is in the realm of IKE la-la land. It's
probably even gnarlier with QoS preconditions.

	      Mike

---

From: "Medhavi Bhatia" <[email-masked]>
Date: Wed, 17 Jul 2002 17:59:21 -0400

I dont think it comes from file descriptors. There are
several packages out there, which keep on stretching this limit.
Moreover I assume one file descriptors will be 64 bits.

I think its the TCP port size in the protocol, 16 bits.

----- Original Message -----
From: "Michael Thomas" <[email-masked]>
To: "Dean Willis" <[email-masked]>
Cc: "'Cullen Jennings'" <[email-masked]>; <[email-masked]>; "'Avshalom
Houri'" <[email-masked]>
Sent: Wednesday, July 17, 2002 11:54 AM
Subject: RE: [Sip] The cost of UDP

---

From: "Medhavi Bhatia" <[email-masked]>
Date: Wed, 17 Jul 2002 20:56:28 -0400

Several systems store the 32 bit descriptor in
a 16 bit quantity, like the sfio package from AT&T.

My thinking is that all these limits still come from the idea
of ports used in udp/tcp.

----- Original Message -----
From: "Jiri Kuthan" <[email-masked]>
To: "Medhavi Bhatia" <[email-masked]>; "Michael Thomas"
<[email-masked]>; "Dean Willis" <[email-masked]>
Cc: "'Cullen Jennings'" <[email-masked]>; <[email-masked]>; "'Avshalom
Houri'" <[email-masked]>
Sent: Wednesday, July 17, 2002 6:49 PM
Subject: Re: [Sip] The cost of UDP

> TCP connections in operating systems I know are identified by
> quad-tupel {src_ip, dst_ip, src_port, dst_port}. Many more bits
> than just 16. If an operating system has 64k limitation on
> number of open TCP, than it is an implementation choice.
>
> -Jiri
>
> At 11:59 PM 7/17/2002, Medhavi Bhatia wrote:

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 17 Jul 2002 20:49:52 -0400

Just in case there is confusion: The port limitation does not limit the 
number of concurrent connections. A TCP connection is identified by four 
quantities: source/destination port, source/destination address. Thus, 
there's no limit imposed by 16-bit ports unless you count the 
uninteresting case where all such connections go to the same destination 
address.

---

From: Jiri Kuthan <[email-masked]>
Date: Thu, 18 Jul 2002 00:49:56 +0200

-Jiri

---

From: "Dean Willis" <[email-masked]>
Date: Wed, 17 Jul 2002 20:38:31 -0500

> probably even gnarlier with QoS preconditions.

Well, there is a little issue with fragmentation, so while only two
MESSAGES might be required, it could well be many PACKETS. And given any
probability of loss, a retry of an entire message occurs, many many more
packets. And of course there is always the unstated issue of key
management -- somewhere in there you also have to take into account the
talking to the ticket server and so on.

But Mike has a real point here -- the reality is much messier than the
simplfied models that often get discussed, and we probably do need to
make a habit of looking at a lot of these issues in more detail.

In my experience, proper security of any kind is non-trivial and often
considered prohibitively expensive. I believe this accounts for the
generally poor level of security we find in the real world. In other
words, experience dictates that some things may be worth doing, but NOT
doing right.

Thanks,

--
Dean