Subject: [tsvwg] Source IP address of the HEARTBEAT_ACK in multi-homing setup

From: Chinmaya Dwibedy <[email-masked]>
Date: Fri, 1 Jun 2012 07:26:27 -0700 (PDT)

Hi,
We have a SCTP multi-homing setup with host and peer having two network interfaces with IP addresses belong to two different subnets.
 Host       IP eth5 (192.168.16.125 – primary IP),
  IP eth6 (192.168.17.125 – secondary IP): Multi-Homed endpoint
 Peer-end IP eth5 (192.168.16.129 – primary IP),
 The network interfaces of host and peer are connected with crossover cable.
We have received one issue saying that, Host (SCTP endpoint) should use the source IP (i.e., 192.168.17.125 – secondary IP) on which it received the Heart_beat_req on while sending back the response (Heart_beat_ack)”. 
Here goes my analysis. Can anyone please validate and feel free to correct me if I am wrong? Thanks in advance for your response.
As per section 3.3.6 of rfc-2960/rfc-4960, a HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ACK is responding. It says nothing about whether the source of the response has to be the same as the destination of HEART_BEAT_REQ. The RFC/standard does not mandate to keep the source IP address of the HEARTBEAT_ACK has to be the same as the destination IP address of HEART_BEAT_REQ. I think in this case, it is quite expected behavior in compliance with rfc-2960/rfc-4960. It would be wrong to say that, it is an issue in RHEL4 (kernel version: 2.6.9-55.ELsmp). 
Even if there are two paths, SCTP does not have any way to determine whether two paths share links and routers when traversing the network. The paths used by the IP   packets across the network might be different depending on the destination IP address.  If a message fails to reach its destination, SCTP may retransmit the message using a different destination IP address. If somewhere along the path a link or/and router fails then SCTP can detect the failure of the path via a heartbeat message. These actions done by SCTP are independent of the actions performed by the lower layers for failure detection and restoration. Under the assumption that every IP address will have a different, separate path towards the remote endpoint, (this is the responsibility of the routing protocols or of manual static configuration), if the transport to one of the IP addresses (= 1 particular path) fails then the traffic can migrate to the other remaining IP address (=
 other paths) within the SCTP association.
 
Regards,
ChinmayaD

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 1 Jun 2012 16:42:54 +0200

On Jun 1, 2012, at 4:26 PM, Chinmaya Dwibedy wrote:

> As per section 3.3.6 of rfc-2960/rfc-4960, a HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ACK is responding. It says nothing about whether the source of the response has to be the same as the destination of HEART_BEAT_REQ. The RFC/standard does not mandate to keep the source IP address of the HEARTBEAT_ACK has to be the same as the destination IP address of HEART_BEAT_REQ. I think in this case, it is quite expected 
> behavior in compliance with rfc-2960/rfc-4960. It would be wrong to say that, it is an issue in RHEL4 (kernel version: 2.6.9-55.ELsmp).
RFC 4960 does require you to use any specific source address. The only constraint is,
that you use one of the addresses known by the peer (which means listed in the INIT or
INIT-ACK or the source address of the INIT or INIT-ACK).

Best regards
Michael
> Even if there are two paths, SCTP does not have any way to determine whether two paths share links and routers when traversing the network. The paths used by the IP   packets across the network might be different depending on the destination IP address.  If a message fails to reach its destination, SCTP may retransmit the message using a different destination IP address. If somewhere along the path a link or/and router fails then SCTP can detect the failure of the path via a heartbeat message. These actions done by SCTP are independent of the actions performed by the lower layers for failure detection and restoration. Under the assumption that every IP address will have a different, separate path towards the remote endpoint, (this is the responsibility of the routing protocols or of manual static configuration), if the transport to one of the IP addresses (= 1 particular path) fails then the traffic can migrate to the other remaining IP address (= other paths) within the SCTP association.
>  
> Regards,
> ChinmayaD

---

From: Randy Stewart <[email-masked]>
Date: Tue, 5 Jun 2012 10:10:14 -0400

> INIT-ACK or the source address of the INIT or INIT-ACK).
The HB section also says that you *SHOULD* use the source address the packet was received on.

There is a known bug in the linux implementation that has been since fixed where it would
incorrectly reply from a different address…

SHOULD in general means you really really really should do this unless you have some other
purposeful reason not to. A bug is not a purposeful reason IMO. So yes, you are allowed to
respond with any valid source. But when responding with a HB you SHOULD use the src address. Which
basically means you *should* unless you have a specific reason (for example you are a user
land stack and you cannot control the src address). Not responding due to a bug, is not being
conformant to the RFC IMO…  ;-)

In fact in general, if I remember correctly you SHOULD respond whenever possible with
the source address that sent you the packet. There are times when thats not possible
(A sack for example acking a packet that came in from two different source addresses, or
 a situation where you suspect the reverse path is broke), but in general that SHOULD
also applies here as well. 

R

>> Even if there are two paths, SCTP does not have any way to determine whether two paths share links and routers when traversing the network. The paths used by the IP   packets across the network mi…

-----
Randall Stewart
[email-masked]

---

From: Chinmaya Dwibedy <[email-masked]>
Date: Thu, 7 Jun 2012 11:24:49 -0700 (PDT)

Hi Mr. Randy Stewart,
Thanks a lot for your valuable suggestion. We turned off Partial Reliability and set the 0 ms timetolive in sctp_sendmsg () API. Thereafter there was no loss in DATA chunks, upon restarting the peer (i.e., SCTP server). 
But when the peer (i.e., SCTP server) gets restarted,
1)     All the SCTP clients receive INIT messages (with same address to the association) from SCTP server. The INIT ACK is being sent with the new InitTag and Verification Tag set to the InitTag received in INIT for each association.  The four-way handshake gets completed.
2)    Though there is no loss but we encountered another issue i.e., for one or two associations, it performs chunk bundling by coalescing of multiple DATA chunks into a single SCTP packet and unnecessary delays are introduced. Unexpectedly it takes more than 20 seconds to put the data on the wire, which was never the case before restart.
Notes: 
a)    Our SCTP client application has requested kernel to not to bundle by making SCTP_NODELAY on. Of course the kernel does   not in itself stop all bundling from occurring (i.e., in case of congestion or retransmission.
b)    Peer has sent a_rwnd of 131072 bytes sent in the INIT when it restarts. It indicates that the peer has sufficient buffer space.
c)    Our client program (C++) running under RHEL4 (using kernel version: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1).
What I understand, the excess delay most likely due to a stall caused by congestion or peer flow control. Does it reset ssthresh to the correct value (peer's a_rwnd) if the association has been restarted or it consists of SCTP_DEFAULT_MAXWINDOW (i.e., 65535)?
When restarting association,
a)    Does the Linux kernel reset the transport congestion variables?
b)    Does it reset the cwnd (Congestion window values for destination address)? 
What I think, if it will be consisting the old values (i.e.,  cwnd, ssthresh) after occurrence of restart then, it must not be transmitting new data to a peer since it will have wrong impression of old cwnd bytes of data outstanding to peer.Can anyone please validate my understanding and correct me if I am wrong.  Please point me to right direction to fix this issue. Thanking you in advance for your response.
Regards,
Chinmaya

________________________________
From: Randy Stewart <[email-masked]>
To: Chinmaya Dwibedy <[email-masked]> 
Cc: Chris Benson <[email-masked]>; "[email-masked]" <[email-masked]> 
Sent: Friday, June 1, 2012 5:05 AM
Subject: Re: [tsvwg] When SCTP server is restarted, DATA chunks start to get dropped or lost after sometime at SCTP client’s end.

Chinmaya:

I would use that has a starting point. Don't put a time-to-expire on it and see if
you no longer see this behavior. If you do, then it sounds like a bug in the linux kernel
implementation

R
On May 31, 2012, at 3:20 PM, Chinmaya Dwibedy wrote:

> Hi Chris,
> Thanks for your response.
> Our application consumes the message fast enough. Note that, we are using single Intel Xeon Quad-Core L5518 processor, 8 logical CPUs (speed of the processor is 2.13GHz) and 12GB RAM. Moreover there are no other apps running on besides our application. This problem appears after SCTP server gets restarted. There is absolutely no packets drop if we won’t restart the server.
>  
> Do you recommend us to try with 0 ms timetolive which indicates no timeout should occur on this message and is treated as reliable?
>  
> Regards,
> Chinmaya
> 
> From: Chris Benson <[email-masked]>
> To: Chinmaya Dwibedy <[email-masked]> 
> Cc: "[email-masked]" <[email-masked]> 
> Sent: Friday, June 1, 2012 12:13 AM
> Subject: Re: [tsvwg] When SCTP server is restarted, DATA chunks start to get dropped or lost after sometime at SCTP client’s end.
> 
> Chinmaya,
> 
> Please forgive me if this is too basic and obvious a question.
> 
> Are you sure that your "application" (SCTP-user) at the client 
> side is consuming the incoming data AT LEAST AS FAST as it arrives?
> 
> If not, then you will see something like the behaviour you 
> describe. With thanks, from Chris
> 
> On Thu, 31 May 2012, Chinmaya Dwibedy wrote:
> 
> >>  Date: Thu, 31 May 2012 11:35:17 -0700 (PDT)
> >>  From: Chinmaya Dwibedy <[email-masked]>
> >>  To: "[email-masked]" <[email-masked]>
> >>  Subject: [tsvwg] When SCTP server is restarted,
> >>      DATA chunks start to get dropped or lost after sometime at SCTP client
> >>      end.
> >>  
> >>  Hi, 
> We have SCTP client program (C++) running under RHEL4 (using kernel version: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using one-to-one style socket using connect () system call to setup an association with a SCTP server (i.e., SUT of vendor). Then it calls sctp_sendmsg () library function to send a message from a socket while using the following advanced features of SCTP (i.e., SCTP_UNORDERED for un-ordered delivery of the message) and timetolive (TTL) is set to 1000 milliseconds for a given message. We have typical defaults values for these RTO.Initial = 3000, RTO.Min = 1000, RTO.Max = 60000 milliseconds.The PR-SCTP extension is enabled in the kernel (by default).
> We are using lot (4000) SCTP associations on a single system, all are in ESTABLISHED state and data communication goes on fine without any drop/loss of message. But when the peer (i.e., SCTP server) gets restarted, 
> a)    All the SCTP clients receive INIT messages (with same address to the association) from SCTP server. The INIT ACK is being sent with the new InitTag and Verification Tag set to the InitTag received in INIT for each association. 
> b)    The four-way handshake gets completed. I believe the existing association, including its current state, and the corresponding TCB does not get changed. 
> c)    Afterward, all SCTP clients use DATA chunks to exchange information with SCTP server.
> The trouble is that, after sometime (30-45 minutes), DATA chunks start to get dropped or lost and that goes on increasing as the time advances.The data was not transmitted to the peer at least once. Note that, the time after which this problem is being encountered is also not consistent. I mean to say, sometime we noticed after 30 minutes, 45 minutes and an hour. But when we lessen the number of SCTP associations to 2000, such problem does not appear.    
> 
> 
> a)    Is there any chance that, that data messages time-out within the RTO although both the values are same? As a result, the data will be skipped and no longer transmitted.
> b)    Does it indicate that the message could never be transmitted to the peer (e.g., flow control prevented the message from being sent before its lifetime expired), so the peer never received it? Please note that, we are using 1 Gig Intel NIC. 
> The preliminary  analysis says that, timetolive (in sctp_sendmsg() API) can be set on a message independent of Partial Reliability.The timer (i.e., sinfo_timetolive) only runs while the message is in the kernel send buffer and has not yet been put on the wire (for the first time). As soon as the message is put on the wire the first time, the timer is dropped. It is used to indicate that a time based lifetime is being applied to the data.  It is then a number of milliseconds for which the data is attempted to be transmitted. If that many milliseconds elapsed and the data has not been transmitted, As a result, they move the abandoned list and are never retransmitted. Please feel free to correct me if I am wrong.
> Should we try with 0 ms timetolive which indicates no timeout should occur on this message and is treated as reliable? Please suggest and also let me know if you need any additional information.
> Thanking you in advance for your response. 
> Regards,
> Chinmaya
> 

-----
Randall Stewart
[email-masked]

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 7 Jun 2012 22:12:59 +0200

On Jun 7, 2012, at 8:24 PM, Chinmaya Dwibedy wrote:

> What I think, if it will be consisting the old values (i.e.,  cwnd, ssthresh) after occurrence of restart then, it must not be transmitting new data to a peer since it will have wrong impression of old cwnd bytes of data outstanding to peer. Can anyone please validate my understanding and correct me if I am wrong.  Please point me to right direction to fix this issue. Thanking you in advance for your response.
> Regards,
> Chinmaya
Chinmaya: This question is related to the Linux implementation, not the protocol. So please
please use the mailing list for the Linux Kernel implementation. (You even got an answer
there from the Linux SCTP maintainer).

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 19 Jun 2012 20:29:44 +0200

On Jun 19, 2012, at 4:18 PM, Chinmaya Dwibedy wrote:

> Hi,
> We are testing SCTP behavior in the following multihoming configuration: 2 RHEL4 PCs, each pc has got 2 NICs, and these interfaces are connected through 2 crossed cables. We established an SCTP association between two signaling endpoints: A and B. The association comprises of two routing paths: path#1 (primary path) and path#2(secondary path). The signaling traffic (data chunks, mostly 250 bytes) is sent on primary path.  SCTP continuously monitors the reachability on primary and secondary paths-on an active primary path SCTP probes for reachability using transferred data packets themselves, and on idle secondary path heartbeat packets are used. Some seconds after transmission's beginning, I physically disconnected the primary path (disconnecting one crossed cable). In this situation, as I had expected, I did not experience the retransmission on the secondary path and then notification of inactive primary path, and consequent path's switching. Rather I found kernel notifies with SCTP_COMM_LOST as the association is closed. Please have a look into the below stated and feel free to correct me if you find me wrong at any point.
> Thanks in advance for your clarifications and comments.
> My general understanding of rfc-2960 says that,
> 1)     For each path (i.e., network destination), SCTP keeps an error counter that counts the number of consecutively missed acknowledgments to data or heartbeat packets. A path is considered unreachable when the error counter of the path exceeds the value of the SCTP parameter Path.Max.Retrans. I mean to say that, SCTP sends the data to the primary address. If timeout occurs, it resends the data to a secondary address. All new data will still be sent to the primary address. If a timeout occurs again, it resends the data to the secondary address. It continues this process until path-retransmit is reached on the primary address.
>  
> 2)    A separate RTO is kept for each path. SCTP computes RTO values based on RTT measurements. When packet retransmission occurs, the timeout value is doubled for each retransmission, with an upper limit of max RTO. The SCTP keeps the track of the number of RTOs and missing heartbeat acknowledgements. If the sum of these two counts exceeds a so-called Path.Max.Retrans threshold, the primary path will be declared to be inactive, and the association enters the "Failover," which continues to send all data to the secondary address. I mean to state that, the number of path-retransmits multiplied by the retransmission timer ultimately controls how fast a secondary address becomes the primary path for multihomed nodes.
>  
> 3)    For example, assuming the default (i.e., recommended values of section 14 of RFC- 2960) Path.Max.Retrans = 5, RTO = RTO.Min = 1, and RTO.Max = 60, Association.Max.Retrans=10 and HB.interval= 30 seconds, an SCTP failover from minimal RTO will take 1 + 2 + 4 + 8 + 16 + 32 = 63 seconds. Shortening this time can be achieved by setting the RTO.Min, RTO.Max, and Path.Max.Retrans, to smaller values. In this multi-homing setup, we have aggressively set parameters as follows i.e., RTO.Init= 250ms, RTO.Min= 250ms, RTO.Max=400ms, Path.Max.Retrans=5) in order to allow an IP network to meet the time-sensitive requirements of LTE network. I think, the SCTP will take changeover time to 250ms + 400ms + 400ms + 400ms + 400ms+ 400ms=2.250 seconds
You could also reduce Path.Max.Retrans...
Or you make use of
http://tools.ietf.org/html/draft-nishida-tsvwg-sctp-failover-05
which is implemented in FreeBSD... But it could be implemented in Linux. too.

Best regards
Michael
>  
> The association gets closed due to either an unreachability threshold being triggered (i.e., the SCTP endpoint timed out multiple times and hit its threshold, which indicates the peer is no longer reachable), or the peer performed an abortive close. In this case, peer has not performed abortive close. Thus is there any chance of getting SCTP_COMM_LOST event with aforesaid configured parameters? What might be the possible causes of this association lost when network cable of primary path gets unplugged?
> If we need faster failover, do you recommend us to change the Path.Max.Retrans setting (without altering RTO.Init, RTO.Min and RTO.Max), to limit the number of consecutive timeouts that will trigger the failover?
> Regards,
> Chinmaya
>