Subject: [tsvwg] A query on SCTP client’s (running under RHEL4) performance in terms of aggregate throughput rate

From: Chinmaya Dwibedy <[email-masked]>
Date: Thu, 31 May 2012 05:29:45 -0700 (PDT)

Hi.
 
We have SCTP client program (C++) running under RHEL4 (using kernel version: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using one-to-one socket (also called TCP-style sockets) using connect () system call to setup an association with a server. Then it calls sctp_sendmsg () library function to send a message from a socket while using the following advanced features of SCTP i.e., SCTP_UNORDERED for un-ordered delivery of the message. The sctp_sendmsg () API takes a pointer to "sockaddr" as a parameter indicating the address of the destination. Since our client application knows well that, it will be sending the multiple messages to the same configured destination address, it would become more efficient if we will connect the socket explicitly.  In summary, we can say that a SCTP client can call connect only if that process uses the SCTP socket to communicate with exactly one peer. I mean,calling connect () on the SCTP socket once simply records the
 specified IP address and port number as being the desired communications partner and send/write call.Anything written to a connected SCTP socket is automatically sent to the protocol address (e.g., IP address and port) specified by connect. 
After connect() system call,  there will be absolutely no need of again passing socket address structure containing the destination IP address and port for each message in sctp_sendmsg(). It would be unnecessary memory copy between user space and kernel space, which is a performance hit. Rather we should pass the destination address of the message and length of the destination address to zero in sctp_sendmsg () API. I believe, with this change, it will improve performance (i.e., aggregate throughput rate) when using a lot (32K) associations on a single system with 8 logical CPUs (speed of the processor is 2.13GHz) and 12GB RAM. Can someone please validate my understanding and correct me if I am wrong? Thanks in advance for your response.
Regards,
Chinmaya

---

From: Chinmaya Dwibedy <[email-masked]>
Date: Thu, 31 May 2012 11:35:17 -0700 (PDT)

Hi, 
We have SCTP client program (C++) running under RHEL4 (using kernel version: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using one-to-one style socket using connect () system call to setup an association with a SCTP server (i.e., SUT of vendor). Then it calls sctp_sendmsg () library function to send a message from a socket while using the following advanced features of SCTP (i.e., SCTP_UNORDERED for un-ordered delivery of the message) and timetolive (TTL) is set to 1000 milliseconds for a given message. We have typical defaults values for these RTO.Initial = 3000, RTO.Min = 1000, RTO.Max = 60000 milliseconds.The PR-SCTP extension is enabled in the kernel (by default).
We are using lot (4000) SCTP associations on a single system, all are in ESTABLISHED state and data communication goes on fine without any drop/loss of message. But when the peer (i.e., SCTP server) gets restarted, 
a)    All the SCTP clients receive INIT messages (with same address to the association) from SCTP server. The INIT ACK is being sent with the new InitTag and Verification Tag set to the InitTag received in INIT for each association. 
b)    The four-way handshake gets completed. I believe the existing association, including its current state, and the corresponding TCB does not get changed. 
c)    Afterward, all SCTP clients use DATA chunks to exchange information with SCTP server.
The trouble is that, after sometime (30-45 minutes), DATA chunks start to get dropped or lost and that goes on increasing as the time advances.The data was not transmitted to the peer at least once. Note that, the time after which this problem is being encountered is also not consistent. I mean to say, sometime we noticed after 30 minutes, 45 minutes and an hour. But when we lessen the number of SCTP associations to 2000, such problem does not appear.    

a)    Is there any chance that, that data messages time-out within the RTO although both the values are same? As a result, the data will be skipped and no longer transmitted.
b)    Does it indicate that the message could never be transmitted to the peer (e.g., flow control prevented the message from being sent before its lifetime expired), so the peer never received it? Please note that, we are using 1 Gig Intel NIC. 
The preliminary  analysis says that, timetolive (in sctp_sendmsg() API) can be set on a message independent of Partial Reliability.The timer (i.e., sinfo_timetolive) only runs while the message is in the kernel send buffer and has not yet been put on the wire (for the first time). As soon as the message is put on the wire the first time, the timer is dropped. It is used to indicate that a time based lifetime is being applied to the data.  It is then a number of milliseconds for which the data is attempted to be transmitted. If that many milliseconds elapsed and the data has not been transmitted, As a result, they move the abandoned list and are never retransmitted. Please feel free to correct me if I am wrong.
Should we try with 0 ms timetolive which indicates no timeout should occur on this message and is treated as reliable? Please suggest and also let me know if you need any additional information.
Thanking you in advance for your response. 
Regards,
Chinmaya

---

From: Chris Benson <[email-masked]>
Date: Thu, 31 May 2012 11:43:13 -0700 (PDT)

Chinmaya,

Please forgive me if this is too basic and obvious a question.

Are you sure that your "application" (SCTP-user) at the client 
side is consuming the incoming data AT LEAST AS FAST as it arrives?

If not, then you will see something like the behaviour you 
describe. With thanks, from Chris

On Thu, 31 May 2012, Chinmaya Dwibedy wrote:

>>  Date: Thu, 31 May 2012 11:35:17 -0700 (PDT)
>>  From: Chinmaya Dwibedy <[email-masked]>
>>  To: "[email-masked]" <[email-masked]>
>>  Subject: [tsvwg] When SCTP server is restarted,
>>      DATA chunks start to get dropped or lost after sometime at SCTP client
>>       end.
>>  
>>  Hi,

---

From: Chinmaya Dwibedy <[email-masked]>
Date: Thu, 31 May 2012 12:20:08 -0700 (PDT)

Hi Chris,
Thanks for your response.
Our application consumes the message fast enough. Note that, we are using single Intel Xeon Quad-Core L5518 processor, 8 logical CPUs (speed of the processor is 2.13GHz) and 12GB RAM. Moreover there are no other apps running on besides our application. This problem appears after SCTP server gets restarted. There is absolutely no packets drop if we won’t restart the server.
 
Do you recommend us to try with 0 ms timetolive which indicates no timeout should occur on this message and is treated as reliable?
 
Regards,
Chinmaya

________________________________
From: Chris Benson <[email-masked]>
To: Chinmaya Dwibedy <[email-masked]> 
Cc: "[email-masked]" <[email-masked]> 
Sent: Friday, June 1, 2012 12:13 AM
Subject: Re: [tsvwg] When SCTP server is restarted, DATA chunks start to get dropped or lost after sometime at SCTP client’s end.

Chinmaya,

---

From: Randy Stewart <[email-masked]>
Date: Thu, 31 May 2012 19:34:40 -0400

Chinmaya:

If you are using a connected socket aka one-2-one model, then you can also use
send() and that should work.

The sctp_sendmsg() is also used by the one-2-many model socket, and there you need a
socket address. Some implementations allow you to place a NULL in the socket address
of a connected socket for sctp_sendmsg(), not sure if linux does. But this is all
implementation specific stuff.

R
On May 31, 2012, at 8:29 AM, Chinmaya Dwibedy wrote:

> 
> 
> Hi.
>  
> We have SCTP client program (C++) running under RHEL4 (using kernel version: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using one-to-one socket (also called TCP-style sockets) using connect () system call to setup an association with a server. Then it calls sctp_sendmsg () library function to send a message from a socket while using the following advanced features of SCTP i.e., SCTP_UNORDERED for un-ordered delivery of the message. The sctp_sendmsg () API takes a pointer to "sockaddr" as a parameter indicating the address of the destination. Since our client application knows well that, it will be sending the multiple messages to the same configured destination address, it would become more efficient if we will connect the socket explicitly.  In summary, we can say that a SCTP client can call connect only if that process uses the SCTP socket to communicate with exactly one peer. I mean, calling connect () on the SCTP socket once simply records the specified IP address and port number as being the desired communications partner and send/write call. Anything written to a connected SCTP socket is automatically sent to the protocol address (e.g., IP address and port) specified by connect.

-----
Randall Stewart
[email-masked]

---

From: Randy Stewart <[email-masked]>
Date: Thu, 31 May 2012 19:35:41 -0400

Chinmaya:

I would use that has a starting point. Don't put a time-to-expire on it and see if
you no longer see this behavior. If you do, then it sounds like a bug in the linux kernel
implementation

R

-----
Randall Stewart
[email-masked]