Subject: Re: [Tsvwg] SCTP latency

From: humaira <[email-masked]>
Date: Mon, 29 Nov 2004 17:51:40 -0800 (PST)

Hi,

Thank you for your reply, it helps a lot. Can you point us to where we
can find design documentation about KAME SCTP implementation?

We had a few questions about your tests on KAME-Bsd stack.
The first one is about the 2Gbyte message size in the output of your test utility. We have been experimenting with sending large messages in lksctp, however, it is limited by the default send buffer size and sctp_sendmsg fails for message sizes above SO_SNDBUF size. (Reference:
http://sourceforge.net/mailarchive/message.php?msg_id=9664532). Therefore, we break-up messages that are larger than the send buffer size into multiple smaller messages that can be handled by lksctp and then reassemble them into one large message at the receiving side.

Is KAME capable of sending and receiving messages to the order of 2Gbytes on its own without requiring the user to break it up? If so, does KAME use some sort of kernel buffering?

Your results for CPU idleness on IBM laptop are very interesting. Is the 30% idle figure for SCTP receiver because of the CRC32c overhead on the slower machine or something else?

We would be very glad to make use of your test utility. We are emailing our basic socket application to you (For others: if you want a copy, please let me know). Regarding your question about our application,
we are not incurring connection setup per message. The connection is
established once and then we send a few messages to warm-up the cache before doing any time measurements. Different sized messages are sent in a for-loop on the same socket. Each message X of some particular size is sent and received a 1000 times and the time taken is the average of these 1000 iterations. Our TCP application is not currently doing message framing, which is of course something that should be added to it in order to make the comparison fair, but we were a bit surprised to see the order of difference in our TCP and SCTP latencies in our initial results to attribute it only to the lack of message framing.

Thank you for your help.

Humaira

-----Original Message-----

> Date: Mon Nov 29 06:15:03 PST 2004
> From: "Randall Stewart" <[email-masked]>
> Subject: Re: [Tsvwg] SCTP latency
> To: 
>
> Humaria:
> 
> Comments below...
> 
> humaira wrote:
> > Hi,
> > 
> > We have been looking at SCTP (one-to-one style) latency performance
> > on a basic socket application (described below), and compared the
> > results obtained with an equivalent TCP application. We are using
> > lksctp-2.6.6 on Linux kernel 2.6.7. According to our results SCTP is
> > an order of magnitude slower than TCP. Other people have mentioned
> > that they have done similar latency tests and have obtained similar
> > outputs. (Reference: 
> > http://sourceforge.net/mailarchive/forum.php?thread_id=5987343&forum_id=1856
> > ) We are very interested in finding out where, deep down, SCTP is
> > spending so much time since this is just a vanilla sctp port. We
> > would have expected to see SCTP performance comparable to TCP on a
> > clean network.
> > 
> > We have read other postings on the subject such as 
> > http://www1.ietf.org/mail-archive/web/tsvwg/current/msg04879.html 
> > which mention CRC32c as a possible performance overhead in sctp.
> > However, looking at the order of difference in our results it seems
> > unlikely that CRC32C is the only factor here.
> 
> CRC32c is a large part of the performance difference... however I think
> in the case of lk-sctp it is more an issue with a new stack. I have just
> completed most of another performance pass on the KAME-Bsd stack and
> I get much better numbers then you did... I hope to re-run my tests
> and get both a gprof output and a transfer throughput to post on
> my web site....
> 
> As to what I was seeing roughly..
> 
> My network was:
> 
>    P-IV-2.8Gig dell server  <------------------>  IBM Laptop T-40 (1.5Gig
>                                                   M Pentium)
> 
>                                   1000Mbit copper via
>                                   2950 cisco switches
> 
> 
> Sending from the Dell box to the IBM I saw:
> 0.0 53.925576 # rb:0 size:2000000000:sctp: (296.705222 Megabits per sec)
> 0.0 54.160244 # rb:0 size:2000000000:tcp: (295.419644 Megabits per sec)
> 0.0 53.914660 # rb:0 size:2000000000:sctp: (296.765295 Megabits per sec)
> 0.0 54.082463 # rb:0 size:2000000000:tcp: (295.844514 Megabits per sec)
> 
> The above is what comes out of my test utility.. it runs the same code
> only with TCP or SCTP using the SOCK_STREAM (or 1 to 1 model).
> 
> Now watching the CPU idle'ness the Dell (sending side) seemed to run
> about 55% idle while sending TCP and 50% idle while sending SCTP.
> 
> The IBM laptop (receiving side) tendend to run about 58% idle on the
> TCP pass but only 30% idle on the SCTP passes..
> 
> Another important note here is that both machines have a TCP offload
> on the NIC I am using.. this means that the TCP connections do not
> suffer any extra time for the checksum..
> 
> Testing on my laptop locally... KAME-BSD by default will disable the
> CRC32c (controlable by a sysctl), I get over the loopback with the
> CRC disabled about 3Gigbit's per second .. however if I change the
> sysctl and run it with the CRC enabled I drop to 600Megabit...
> 
> Now when I reverse things and have my P-IV  be the receiver things
> get a bit more interesting.. I zip up to about 400Megabit with
> SCTP slightly faster... however in looking at TCP and SCTP I
> am seeing fast retransmissions.. this tells me I am loosing
> packets.. and since I use cat-5 wire I think this is my
> problem (I even took the switch out and did a back-2-back
> wire and had the same problem)...
> 
> Note that I am quite a bit faster on the transfer the opposite
> way.. this tells me that I still need to work on the
> receive path of KAME-Bsd.. I really need to write my
> own steam-lined sctp_soreceive instead of using the
> stock one.. this, I think, could get the BSD stack
> close to the same idle time on receive.. I am hoping...
> 
> I have noted too that message sizes play a big role
> in the transfer... TCP  can always pack down a
> full MTU.. SCTP is limited in this, somewhat, in that
> it must fragment to its max size and can do bundling
> but not always will the pieces fit together.
> 
> To pull this out of the factor (since I don't have an
> app that does message framing for a fair comparison)
> I use 1452 has the message write size... This is the
> largest chunk of data that SCTP can place in one
> packet... for FreeBSD (which is what I was using 4.10) the
> max TCP stuffs is about 1448 (I think timestamps is the
> reason)...
> 
> 
> > 
> > It would be very helpful if somebody could provide insight into other
> > factors for this behavior.
> > 
> > Thank you for your help. Humaira
> > 
> > Our application does the following. There are two nodes (one Pentium
> > II, 400.9MHz and the other Pentium III, 999.981MHz both with Linux
> > 2.6.7) Node1 sends a message of size X to Node2 and then receives
> > message of size X from Node2. Node2 does the opposite,i.e.receives
> > message size X from Node1 and then sends message size X to Node1.
> > 
> > We measure the time taken to carry out this operation using rdtsc
> > which returns the number of Pentium clock cycles. The above operation
> > was repeated 1000 times and the time obtained is the average of these
> >  1000 iterations. We warm-up the cache before doing any time
> > measurement.
> > 
> > Our SCTP program uses the one-to-one style and we send and receive
> > using sctp_sendmsg and sctp_recvmsg. The TCP counterpart simply uses
> > send and recv. The NODELAY option is set to 1 in both SCTP and TCP
> > (i.e.Nagles's algorithm is disabled). The default values for socket
> > send and receive buffers was used in both cases. In our case the
> > default tcp socket send and receive buffer sizes were 16384 and 87380
> > respectively and the default SCTP socket send and receive buffer
> > sizes were both 110592.
> > 
> > The results for different message sizes are copied below.
> > 
> > Latency results for TCP Msg_size=1 tsecs=0.000176 Msg_size=2
> > tsecs=0.000176 Msg_size=4 tsecs=0.000168 Msg_size=8 tsecs=0.000169 
> > Msg_size=16 tsecs=0.000175 Msg_size=32 tsecs=0.000175 Msg_size=64
> > tsecs=0.000186 Msg_size=128 tsecs=0.000219 Msg_size=256
> > tsecs=0.000261 Msg_size=512 tsecs=0.000371 Msg_size=1024
> > tsecs=0.000571 Msg_size=2048 tsecs=0.000856 Msg_size=4096
> > tsecs=0.001182 Msg_size=8192 tsecs=0.001905 Msg_size=16384
> > tsecs=0.003300 Msg_size=32768 tsecs=0.006069 Msg_size=65536
> > tsecs=0.011681 Msg_size=131072 tsecs=0.022805 Msg_size=262144
> > tsecs=0.045151 Msg_size=524288 tsecs=0.089808 Msg_size=1048576
> > tsecs=0.178923 Msg_size=2097152 tsecs=0.357190
> > 
> 
> Does your TCP app do actuall message framing? I.e. are you doing
> a read and then processing the buffer to frame it correctly?
> 
> Also is this just one message transfered? I.e. are you incurring
> the a connection setup per message?
> 
> Maybe we can share the test utilties we are using.. I can gladly
> post mine (or email it to you) and I could run yours
> on my setup as well..
> 
> R
> 
> > Latency results for SCTP Msg_size=1 tsecs=0.005685 Msg_size=2
> > tsecs=0.005686 Msg_size=4 tsecs=0.005697 Msg_size=8 tsecs=0.005693 
> > Msg_size=16 tsecs=0.005700 Msg_size=32 tsecs=0.005706 Msg_size=64
> > tsecs=0.005704 Msg_size=128 tsecs=0.005726 Msg_size=256
> > tsecs=0.005738 Msg_size=512 tsecs=0.005733 Msg_size=1024
> > tsecs=0.005772 Msg_size=2048 tsecs=0.007902 Msg_size=4096
> > tsecs=0.002706 Msg_size=8192 tsecs=0.004739 Msg_size=16384
> > tsecs=0.023130 Msg_size=32768 tsecs=0.048449 Msg_size=65536
> > tsecs=0.050129 Msg_size=131072 tsecs=0.078571 Msg_size=262144
> > tsecs=0.258564 Msg_size=524288 tsecs=0.716646 Msg_size=1048576
> > tsecs=0.981830 Msg_size=2097152 tsecs=2.056492
> > 
> > 
> > _______________________________________________ tsvwg mailing list 
> > [email-masked] https://www1.ietf.org/mailman/listinfo/tsvwg
> > 
> > 
> 
> 
> -- 
> Randall Stewart
> 803-345-0369 <or> 815-342-5222(cell)

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Tue, 30 Nov 2004 07:12:46 -0500

humaira wrote:
> Hi,
> 
> Thank you for your reply, it helps a lot. Can you point us to where
> we can find design documentation about KAME SCTP implementation?'

I have never really documented it.. the closests would be
in the SCTP book since its a rewrite of the original app
layer library into the kernel... of course there were
a lot of changes ... but it does follow the general
overview layed out in the book...

But I really should circle back and update it to
reflect the code.... sigh.. never enough time ...

> 
> We had a few questions about your tests on KAME-Bsd stack. The first
> one is about the 2Gbyte message size in the output of your test
> utility. We have been experimenting with sending large messages in
> lksctp, however, it is limited by the default send buffer size and
> sctp_sendmsg fails for message sizes above SO_SNDBUF size.

The test is setup that you give it a size of transfer
and a record size. It does NOT send a 2G message.. KAME
would also fail in this since the socket send buffer
limits the largest single size message one can send.

How my test works is the client does the active
open, sets any receive buffer on the socket, and
then sends a request to the peer to send
N bytes in X byte sizes. Also included is the
send buffer size. If the send buffer size is
specified then the server sets it on the
new "accepted" fd and then it starts to
write data in X size pieces until N bytes
are transfered.

The control file to the client looks like:

N:X:snd_buf:rcv_buf

For snd_buf and rcv_buf a 0 indicates use
the default for the transport.

The run you looked at has a control file that looks like:

# Control file needs
# sizetos:blksize:snd_buf:recv_buf\n
2000000000:1452:0:0

So each write was 1452 bytes until 2 Gig was transfered

The program first connects SCTP and does the transfer then
it connects TCP and does the transfer ... it repeats these
until it hits the number of passes requested or forever :-D

Of course a ctl-c will get the program to stop at the end
of the next pass (sometimes forever is to long :-D)

> (Reference: 
> http://sourceforge.net/mailarchive/message.php?msg_id=9664532).
> Therefore, we break-up messages that are larger than the send buffer
> size into multiple smaller messages that can be handled by lksctp and
> then reassemble them into one large message at the receiving side.
> 
I have not looked at message sized type tranfer... its one of
the things I have on my to-do list... i.e. have a
app that sends varied size messages between a range of
values... that way the TCP side must parse the messages
into complete pieces... vs SCTP that does not...

Most apps I have ran across in my various roles have all
had very few messages over a few hundred bytes... I was
thinking of doing things between 1 - 1400 bytes.. but throwing
in a 5k or 8k message might also be good.... :-D

> Is KAME capable of sending and receiving messages to the order of
> 2Gbytes on its own without requiring the user to break it up? If so,
> does KAME use some sort of kernel buffering?
Well its possible but it would mean the socket send buffer would
need to be 2Gig... and in reality I would think not likely.. It does
not have a "partial deliver send"... Now I have sent a 2 Meg message
(by raising the socket buffer defaults up large enough that I
  could raise up SCTP's socket send buffer)... but never 2G
> 
> Your results for CPU idleness on IBM laptop are very interesting. Is
> the 30% idle figure for SCTP receiver because of the CRC32c overhead
> on the slower machine or something else?

I think its a combination of things...

First of all the 30% number is un-scientific.. its me
running top while the TCP and SCTP transfers happen and
observing... no fine grained stuff.

I do, of course, have gprof info on what the kernel is
doing.. and I have done most of the tweaks that were
easy... A lot of the remaining time is in the standard
so_receive() call. This is really optimized for (surprise)
TCP and is not very efficent for SCTP.. but Free and Net
BSD allow a transport to change these out.. so I really
need to re-write an optimized version.. this, I think, will
get the "idle-ness" up quite a bit.. as well as speed the
transfer up more towards where I start hitting FR's

> 
> We would be very glad to make use of your test utility. We are
> emailing our basic socket application to you (For others: if you want
> a copy, please let me know). Regarding your question about our
> application, we are not incurring connection setup per message. The
> connection is established once and then we send a few messages to
> warm-up the cache before doing any time measurements. Different sized
> messages are sent in a for-loop on the same socket. Each message X of
> some particular size is sent and received a 1000 times and the time
> taken is the average of these 1000 iterations. Our TCP application is
> not currently doing message framing, which is of course something
> that should be added to it in order to make the comparison fair, but
> we were a bit surprised to see the order of difference in our TCP and
> SCTP latencies in our initial results to attribute it only to the
> lack of message framing.

I think a lot of your problem may be that lk-sctp has not
finished any performance work as well... just a guess...
I know KAME was not steller when I started..

I will send you my apps seperate.

R
>> Date: Mon Nov 29 06:15:03 PST 2004 From: "Randall Stewart"
>> <[email-masked]> Subject: Re: [Tsvwg] SCTP latency 
>> To:
>> 
>> Humaria:
>> 
>> Comments below...
>> 
>> humaira wrote:
>> 
>>> Hi,
>>> 
>>> We have been looking at SCTP (one-to-one style) latency
>>> performance on a basic socket application (described below), and
>>> compared the results obtained with an equivalent TCP application.
>>> We are using lksctp-2.6.6 on Linux kernel 2.6.7. According to our
>>> results SCTP is an order of magnitude slower than TCP. Other
>>> people have mentioned that they have done similar latency tests
>>> and have obtained similar outputs. (Reference: 
>>>  ) We are very interested in finding out where, deep down, SCTP
>>> is spending so much time since this is just a vanilla sctp port.
>>> We would have expected to see SCTP performance comparable to TCP
>>> on a clean network.
>>> 
>>>  However, looking at the order of difference in our results it
>>> seems unlikely that CRC32C is the only factor here.
>> 
>> CRC32c is a large part of the performance difference... however I
>> think in the case of lk-sctp it is more an issue with a new stack.
>> I have just completed most of another performance pass on the
>> KAME-Bsd stack and I get much better numbers then you did... I hope
>> to re-run my tests and get both a gprof output and a transfer
>> throughput to post on my web site....
>> 
>> P-IV-2.8Gig dell server  <------------------>  IBM Laptop T-40
>> (1.5Gig M Pentium)
>> 
>> 1000Mbit copper via 2950 cisco switches
>> 
>> 
>> Sending from the Dell box to the IBM I saw: 0.0 53.925576 # rb:0
>> size:2000000000:sctp: (296.705222 Megabits per sec) 0.0 54.160244 #
>> rb:0 size:2000000000:tcp: (295.419644 Megabits per sec) 0.0
>> 53.914660 # rb:0 size:2000000000:sctp: (296.765295 Megabits per
>> sec) 0.0 54.082463 # rb:0 size:2000000000:tcp: (295.844514 Megabits
>> per sec)
>> 
>> The above is what comes out of my test utility.. it runs the same
>> code only with TCP or SCTP using the SOCK_STREAM (or 1 to 1 model).
>> 
>> 
>> Now watching the CPU idle'ness the Dell (sending side) seemed to
>> run about 55% idle while sending TCP and 50% idle while sending
>> SCTP.
>> 
>> The IBM laptop (receiving side) tendend to run about 58% idle on
>> the TCP pass but only 30% idle on the SCTP passes..
>> 
>> Another important note here is that both machines have a TCP
>> offload on the NIC I am using.. this means that the TCP connections
>> do not suffer any extra time for the checksum..
>> 
>> Testing on my laptop locally... KAME-BSD by default will disable
>> the CRC32c (controlable by a sysctl), I get over the loopback with
>> the CRC disabled about 3Gigbit's per second .. however if I change
>> the sysctl and run it with the CRC enabled I drop to 600Megabit...
>> 
>> get a bit more interesting.. I zip up to about 400Megabit with SCTP
>> slightly faster... however in looking at TCP and SCTP I am seeing
>> fast retransmissions.. this tells me I am loosing packets.. and
>> since I use cat-5 wire I think this is my problem (I even took the
>> switch out and did a back-2-back wire and had the same problem)...
>> 
>> way.. this tells me that I still need to work on the receive path
>> of KAME-Bsd.. I really need to write my own steam-lined
>> sctp_soreceive instead of using the stock one.. this, I think,
>> could get the BSD stack close to the same idle time on receive.. I
>> am hoping...
>> 
>> I have noted too that message sizes play a big role in the
>> transfer... TCP  can always pack down a full MTU.. SCTP is limited
>> in this, somewhat, in that it must fragment to its max size and can
>> do bundling but not always will the pieces fit together.
>> 
>> To pull this out of the factor (since I don't have an app that does
>> message framing for a fair comparison) I use 1452 has the message
>> write size... This is the largest chunk of data that SCTP can place
>> in one packet... for FreeBSD (which is what I was using 4.10) the 
>> max TCP stuffs is about 1448 (I think timestamps is the reason)...
>> 
>> 
>> 
>>> It would be very helpful if somebody could provide insight into
>>> other factors for this behavior.
>>> 
>>> Our application does the following. There are two nodes (one
>>> Pentium II, 400.9MHz and the other Pentium III, 999.981MHz both
>>> with Linux 2.6.7) Node1 sends a message of size X to Node2 and
>>> then receives message of size X from Node2. Node2 does the
>>> opposite,i.e.receives message size X from Node1 and then sends
>>> message size X to Node1.
>>> 
>>>  which returns the number of Pentium clock cycles. The above
>>> operation was repeated 1000 times and the time obtained is the
>>> average of these 1000 iterations. We warm-up the cache before
>>> doing any time measurement.
>>> 
>>> Our SCTP program uses the one-to-one style and we send and
>>> receive using sctp_sendmsg and sctp_recvmsg. The TCP counterpart
>>> simply uses send and recv. The NODELAY option is set to 1 in both
>>> SCTP and TCP (i.e.Nagles's algorithm is disabled). The default
>>> values for socket send and receive buffers was used in both
>>> cases. In our case the default tcp socket send and receive buffer
>>> sizes were 16384 and 87380 respectively and the default SCTP
>>> socket send and receive buffer sizes were both 110592.
>>> 
>>> tsecs=0.000176 Msg_size=4 tsecs=0.000168 Msg_size=8
>>> tsecs=0.000169 Msg_size=16 tsecs=0.000175 Msg_size=32
>>> tsecs=0.000175 Msg_size=64 tsecs=0.000186 Msg_size=128
>>> tsecs=0.000219 Msg_size=256 tsecs=0.000261 Msg_size=512
>>> tsecs=0.000371 Msg_size=1024 tsecs=0.000571 Msg_size=2048
>>> tsecs=0.000856 Msg_size=4096 tsecs=0.001182 Msg_size=8192
>>> tsecs=0.001905 Msg_size=16384 tsecs=0.003300 Msg_size=32768
>>> tsecs=0.006069 Msg_size=65536 tsecs=0.011681 Msg_size=131072
>>> tsecs=0.022805 Msg_size=262144 tsecs=0.045151 Msg_size=524288
>>> tsecs=0.089808 Msg_size=1048576 tsecs=0.178923 Msg_size=2097152
>>> tsecs=0.357190
>>> 
>> 
>> Does your TCP app do actuall message framing? I.e. are you doing a
>> read and then processing the buffer to frame it correctly?
>> 
>> post mine (or email it to you) and I could run yours on my setup as
>> well..
>> 
>> R
>> 
>> 
>>> tsecs=0.005686 Msg_size=4 tsecs=0.005697 Msg_size=8
>>> tsecs=0.005693 Msg_size=16 tsecs=0.005700 Msg_size=32
>>> tsecs=0.005706 Msg_size=64 tsecs=0.005704 Msg_size=128
>>> tsecs=0.005726 Msg_size=256 tsecs=0.005738 Msg_size=512
>>> tsecs=0.005733 Msg_size=1024 tsecs=0.005772 Msg_size=2048
>>> tsecs=0.007902 Msg_size=4096 tsecs=0.002706 Msg_size=8192
>>> tsecs=0.004739 Msg_size=16384 tsecs=0.023130 Msg_size=32768
>>> tsecs=0.048449 Msg_size=65536 tsecs=0.050129 Msg_size=131072
>>> tsecs=0.078571 Msg_size=262144 tsecs=0.258564 Msg_size=524288
>>> tsecs=0.716646 Msg_size=1048576 tsecs=0.981830 Msg_size=2097152
>>> tsecs=2.056492
>>> 
>>> 
>>> _______________________________________________ tsvwg mailing
>>> list [email-masked] https://www1.ietf.org/mailman/listinfo/tsvwg
>>> 
>>> 
>> 
>> 
>> -- Randall Stewart 803-345-0369 <or> 815-342-5222(cell)
> 
> 
>