Subject: [Tsvwg] SCTP latency

From: humaira <[email-masked]>
Date: Fri, 26 Nov 2004 13:52:46 -0800 (PST)

Hi,

We have been looking at SCTP (one-to-one style) latency performance on a basic socket application (described below), and compared the results obtained with an equivalent TCP application. We are using lksctp-2.6.6 on Linux kernel 2.6.7. According to our results SCTP is an order of magnitude slower than TCP. Other people have mentioned that they have done similar latency tests and have obtained similar outputs. (Reference:
http://sourceforge.net/mailarchive/forum.php?thread_id=5987343&forum_id=1856 )
We are very interested in finding out where, deep down, SCTP is spending so much time since this is just a vanilla sctp port. We would have expected to see SCTP performance comparable to TCP on a clean network.

We have read other postings on the subject such as
http://www1.ietf.org/mail-archive/web/tsvwg/current/msg04879.html
which mention CRC32c as a possible performance overhead in sctp. However, looking at the order of difference in our results it seems unlikely that CRC32C is the only factor here.

It would be very helpful if somebody could provide insight into other factors for this behavior.

Thank you for your help.
Humaira

Our application does the following.
There are two nodes (one Pentium II, 400.9MHz and the other Pentium III,
999.981MHz both with Linux 2.6.7) Node1 sends a message of size X to Node2 and then receives message of size X from Node2. Node2 does the opposite,i.e.receives message size X from Node1 and then sends message size X to Node1.

We measure the time taken to carry out this operation using rdtsc which
returns the number of Pentium clock cycles. The above operation was
repeated 1000 times and the time obtained is the average of these
1000 iterations. We warm-up the cache before doing any time measurement.

Our SCTP program uses the one-to-one style and we send and receive using
sctp_sendmsg and sctp_recvmsg. The TCP counterpart simply uses send and recv.
The NODELAY option is set to 1 in both SCTP and TCP (i.e.Nagles's algorithm is disabled). The default values for socket send and receive buffers was used in both cases. In our case the default tcp socket send and receive buffer sizes were 16384 and 87380 respectively and the default SCTP socket send and receive buffer sizes were both 110592.

The results for different message sizes are copied below.

Latency results for TCP
Msg_size=1 tsecs=0.000176
Msg_size=2 tsecs=0.000176
Msg_size=4 tsecs=0.000168
Msg_size=8 tsecs=0.000169
Msg_size=16 tsecs=0.000175
Msg_size=32 tsecs=0.000175
Msg_size=64 tsecs=0.000186
Msg_size=128 tsecs=0.000219
Msg_size=256 tsecs=0.000261
Msg_size=512 tsecs=0.000371
Msg_size=1024 tsecs=0.000571
Msg_size=2048 tsecs=0.000856
Msg_size=4096 tsecs=0.001182
Msg_size=8192 tsecs=0.001905
Msg_size=16384 tsecs=0.003300
Msg_size=32768 tsecs=0.006069
Msg_size=65536 tsecs=0.011681
Msg_size=131072 tsecs=0.022805
Msg_size=262144 tsecs=0.045151
Msg_size=524288 tsecs=0.089808
Msg_size=1048576 tsecs=0.178923
Msg_size=2097152 tsecs=0.357190

Latency results for SCTP
Msg_size=1 tsecs=0.005685
Msg_size=2 tsecs=0.005686
Msg_size=4 tsecs=0.005697
Msg_size=8 tsecs=0.005693
Msg_size=16 tsecs=0.005700
Msg_size=32 tsecs=0.005706
Msg_size=64 tsecs=0.005704
Msg_size=128 tsecs=0.005726
Msg_size=256 tsecs=0.005738
Msg_size=512 tsecs=0.005733
Msg_size=1024 tsecs=0.005772
Msg_size=2048 tsecs=0.007902
Msg_size=4096 tsecs=0.002706
Msg_size=8192 tsecs=0.004739
Msg_size=16384 tsecs=0.023130
Msg_size=32768 tsecs=0.048449
Msg_size=65536 tsecs=0.050129
Msg_size=131072 tsecs=0.078571
Msg_size=262144 tsecs=0.258564
Msg_size=524288 tsecs=0.716646
Msg_size=1048576 tsecs=0.981830
Msg_size=2097152 tsecs=2.056492

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Mon, 29 Nov 2004 09:15:03 -0500

Humaria:

Comments below...

humaira wrote:
> Hi,
> 
> We have been looking at SCTP (one-to-one style) latency performance
> on a basic socket application (described below), and compared the
> results obtained with an equivalent TCP application. We are using
> lksctp-2.6.6 on Linux kernel 2.6.7. According to our results SCTP is
> an order of magnitude slower than TCP. Other people have mentioned
> that they have done similar latency tests and have obtained similar
> outputs. (Reference: 
> http://sourceforge.net/mailarchive/forum.php?thread_id=5987343&forum_id=1856
> ) We are very interested in finding out where, deep down, SCTP is
> spending so much time since this is just a vanilla sctp port. We
> would have expected to see SCTP performance comparable to TCP on a
> clean network.
> 
> which mention CRC32c as a possible performance overhead in sctp.
> However, looking at the order of difference in our results it seems
> unlikely that CRC32C is the only factor here.

CRC32c is a large part of the performance difference... however I think
in the case of lk-sctp it is more an issue with a new stack. I have just
completed most of another performance pass on the KAME-Bsd stack and
I get much better numbers then you did... I hope to re-run my tests
and get both a gprof output and a transfer throughput to post on
my web site....

As to what I was seeing roughly..

My network was:

   P-IV-2.8Gig dell server  <------------------>  IBM Laptop T-40 (1.5Gig
                                                  M Pentium)

                                  1000Mbit copper via
                                  2950 cisco switches

Sending from the Dell box to the IBM I saw:
0.0 53.925576 # rb:0 size:2000000000:sctp: (296.705222 Megabits per sec)
0.0 54.160244 # rb:0 size:2000000000:tcp: (295.419644 Megabits per sec)
0.0 53.914660 # rb:0 size:2000000000:sctp: (296.765295 Megabits per sec)
0.0 54.082463 # rb:0 size:2000000000:tcp: (295.844514 Megabits per sec)

The above is what comes out of my test utility.. it runs the same code
only with TCP or SCTP using the SOCK_STREAM (or 1 to 1 model).

Now watching the CPU idle'ness the Dell (sending side) seemed to run
about 55% idle while sending TCP and 50% idle while sending SCTP.

The IBM laptop (receiving side) tendend to run about 58% idle on the
TCP pass but only 30% idle on the SCTP passes..

Another important note here is that both machines have a TCP offload
on the NIC I am using.. this means that the TCP connections do not
suffer any extra time for the checksum..

Testing on my laptop locally... KAME-BSD by default will disable the
CRC32c (controlable by a sysctl), I get over the loopback with the
CRC disabled about 3Gigbit's per second .. however if I change the
sysctl and run it with the CRC enabled I drop to 600Megabit...

Now when I reverse things and have my P-IV  be the receiver things
get a bit more interesting.. I zip up to about 400Megabit with
SCTP slightly faster... however in looking at TCP and SCTP I
am seeing fast retransmissions.. this tells me I am loosing
packets.. and since I use cat-5 wire I think this is my
problem (I even took the switch out and did a back-2-back
wire and had the same problem)...

Note that I am quite a bit faster on the transfer the opposite
way.. this tells me that I still need to work on the
receive path of KAME-Bsd.. I really need to write my
own steam-lined sctp_soreceive instead of using the
stock one.. this, I think, could get the BSD stack
close to the same idle time on receive.. I am hoping...

I have noted too that message sizes play a big role
in the transfer... TCP  can always pack down a
full MTU.. SCTP is limited in this, somewhat, in that
it must fragment to its max size and can do bundling
but not always will the pieces fit together.

To pull this out of the factor (since I don't have an
app that does message framing for a fair comparison)
I use 1452 has the message write size... This is the
largest chunk of data that SCTP can place in one
packet... for FreeBSD (which is what I was using 4.10) the
max TCP stuffs is about 1448 (I think timestamps is the
reason)...

> Thank you for your help. Humaira
> 
> Our application does the following. There are two nodes (one Pentium
> II, 400.9MHz and the other Pentium III, 999.981MHz both with Linux
> 2.6.7) Node1 sends a message of size X to Node2 and then receives
> message of size X from Node2. Node2 does the opposite,i.e.receives
> message size X from Node1 and then sends message size X to Node1.
> 
> We measure the time taken to carry out this operation using rdtsc
> which returns the number of Pentium clock cycles. The above operation
> was repeated 1000 times and the time obtained is the average of these
> Our SCTP program uses the one-to-one style and we send and receive
> using sctp_sendmsg and sctp_recvmsg. The TCP counterpart simply uses
> send and recv. The NODELAY option is set to 1 in both SCTP and TCP
> (i.e.Nagles's algorithm is disabled). The default values for socket
> send and receive buffers was used in both cases. In our case the
> default tcp socket send and receive buffer sizes were 16384 and 87380
> respectively and the default SCTP socket send and receive buffer
> sizes were both 110592.
> 
> Latency results for TCP Msg_size=1 tsecs=0.000176 Msg_size=2
> tsecs=0.000176 Msg_size=4 tsecs=0.000168 Msg_size=8 tsecs=0.000169 
> Msg_size=16 tsecs=0.000175 Msg_size=32 tsecs=0.000175 Msg_size=64
> tsecs=0.000186 Msg_size=128 tsecs=0.000219 Msg_size=256
> tsecs=0.000261 Msg_size=512 tsecs=0.000371 Msg_size=1024
> tsecs=0.000571 Msg_size=2048 tsecs=0.000856 Msg_size=4096
> tsecs=0.001182 Msg_size=8192 tsecs=0.001905 Msg_size=16384
> tsecs=0.003300 Msg_size=32768 tsecs=0.006069 Msg_size=65536
> tsecs=0.011681 Msg_size=131072 tsecs=0.022805 Msg_size=262144
> tsecs=0.045151 Msg_size=524288 tsecs=0.089808 Msg_size=1048576
> tsecs=0.178923 Msg_size=2097152 tsecs=0.357190
> 

Does your TCP app do actuall message framing? I.e. are you doing
a read and then processing the buffer to frame it correctly?

Also is this just one message transfered? I.e. are you incurring
the a connection setup per message?

Maybe we can share the test utilties we are using.. I can gladly
post mine (or email it to you) and I could run yours
on my setup as well..

R

> Latency results for SCTP Msg_size=1 tsecs=0.005685 Msg_size=2
> tsecs=0.005686 Msg_size=4 tsecs=0.005697 Msg_size=8 tsecs=0.005693 
> Msg_size=16 tsecs=0.005700 Msg_size=32 tsecs=0.005706 Msg_size=64
> tsecs=0.005704 Msg_size=128 tsecs=0.005726 Msg_size=256
> tsecs=0.005738 Msg_size=512 tsecs=0.005733 Msg_size=1024
> tsecs=0.005772 Msg_size=2048 tsecs=0.007902 Msg_size=4096
> tsecs=0.002706 Msg_size=8192 tsecs=0.004739 Msg_size=16384
> tsecs=0.023130 Msg_size=32768 tsecs=0.048449 Msg_size=65536
> tsecs=0.050129 Msg_size=131072 tsecs=0.078571 Msg_size=262144
> tsecs=0.258564 Msg_size=524288 tsecs=0.716646 Msg_size=1048576
> tsecs=0.981830 Msg_size=2097152 tsecs=2.056492
> 
> 
> _______________________________________________ tsvwg mailing list 
> [email-masked] https://www1.ietf.org/mailman/listinfo/tsvwg
> 
> 

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)