Subject: [tsvwg] Q about path failure detection in SCTP

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 10 Aug 2009 17:14:20 +0900 (JST)

Hello people,

I sometimes write sctp applications for multihome environments and notice that
there is always significant long delay (over 30 secs) to switch over to secondary 
path when the primary path becomes unavailable. 

This is because sctp sender tries to retransmit data packets (usualy) 5 times to the
primary path before SCTP considers it is unavailable. In my understanding, this is 
a normal behavior of SCTP since RFC4960 specifies that an endpoint should always 
transmit to the primary path.

However, it seems to me that this behavior impairs the usability of SCTP. 
For example, if we can utilize the retransmission scheme proposed in the 
CMT paper [1] which use "potentially-failed" state, we might be able to 
switch over to the other path more smoothly.

I just would like to know if there is any reason that we don't adopt this kind of 
approach in the standard. It seems to me there is no major drawback in this approach.
Or am I misunderstanding something?

[1]
P. Natarajan, N, Ekiz, P. Amer, R. Stewart, 
"Concurrent Multipath Transfer using SCTP Multihoming: Transmission Policies using
Potentially-failed Destination State", LNCS 4982

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Randy Stewart <[email-masked]>
Date: Mon, 10 Aug 2009 13:23:08 -0700

Nishida-san:

I think you are probably correct. You can also find
some similar ideas put out by Armando Caro in some
of his research... don't have the citations at my finger
tips..

His idea on multiple triggers for failure were I think what
Jana pulled in to the CMT work...

I am not sure about putting it in the standard I would have
to think about if its allowed (it might well be in the current
standard)...

R

> Potentially-failed Destination State", LNCS 4982

-----
Randall Stewart
[email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 11 Aug 2009 08:59:36 +0900 (JST)

Dear Randy,

Thank you so much for your feedback. 
In my understanding, there is no good way to solve this in the current standard
without some cooperations from applications.
The section 6.4 in RFC4960 says:

  By default, an endpoint SHOULD always transmit to the primary path,
  unless the SCTP user explicitly specifies the destination transport
  address (and possibly source transport address) to use.

If we follow this, we have to transmit data to the primary path until it has been changed.
However, the threshold of 'Path.Max.Retrans' prevents to change primary path quickly.

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: Randy Stewart <[email-masked]>
Subject: Re: [tsvwg] Q about path failure detection in SCTP
Date: Mon, 10 Aug 2009 13:23:08 -0700
Message-ID: <[email-masked]>

---

From: Randy Stewart <[email-masked]>
Date: Mon, 10 Aug 2009 17:17:30 -0700

> However, the threshold of 'Path.Max.Retrans' prevents to change  

True, but I was more thinking of the mechanisms that have to
do with primary path... i.e. when does one failover..... this is
what some of Armando's work talked about. I need to go reread
that sections... and only if its a MUST do you violate the
spec by the way... even though SHOULD means you really really really
should do it unless you have a good reason not to. And if it causes
no harm to switch path's early (as some of Armando's work indicated) I
think we might be able to do it..

But let me look at it tonight when I get back to my hotel room and
have some time to think on it.

R

>>> This is because sctp sender tries to retransmit data packets  
>>> (usualy)
>>> 5 times to the
>>> approach in the standard. It seems to me there is no major  
>>> drawback in
>>> this approach.

-----
Randall Stewart
[email-masked]

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 11 Aug 2009 09:43:22 +0200

On Aug 10, 2009, at 10:14 AM, Yoshifumi Nishida wrote:

> (usualy) 5 times to the
Have you you tried to reduce the number of retransmissions before  
considering
a path as failed? This setting does have an impact on the failover time.
> Or am I misunderstanding something?
Getting this adopted needs an analysis if it does not hurt the network,
as the CMT stuff is not standardized due due to congestion control  
issues.
However, this might change in the near future, since there seems to be
a way to resolve this issue.

However, if these technical point is resolved it needs people willing
to write the ID and the TSVWG to adopt it as a WG item.

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 11 Aug 2009 17:29:43 +0900 (JST)

From: Michael Tüxen <[email-masked]>
Subject: Re: [tsvwg] Q about path failure detection in SCTP
Date: Tue, 11 Aug 2009 09:43:22 +0200
Message-ID: <[email-masked]>

 > a path as failed? This setting does have an impact on the failover

Do you mean reducing Path.Max.Retrans? If so, yes. that's an option.
However, it might cause path flapping. also the recommended value in RFC4960 is 5.
I think it would be better to have more sophisticated way for this.

 > to write the ID and the TSVWG to adopt it as a WG item.

My rough idea is something like this.

1) If timeout happens on the primary path, set the status of this path to "suspicious"
   
2) If the status of the primary path is "suspicious", send data to the secondary,
   while send keep-alive to the primary.

3) If keep-alive ack is received from the primary, set the status of the primay to
   "normal" and back to normal data transmission.
   If keep-alive cannot be received, increase error counter on each timeout.
   If the counter exceeds Path.Max.Retrans, change primary

As you suggest, we need to analyze the impact of 2), but I tend to think 
it's mostly OK in many cases..

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 11 Aug 2009 11:15:55 +0200

> 1) If timeout happens on the primary path, set the status of this  
So a path having experienced a congestion indication (timeout) is  
suspicious?

---

From: Randy Stewart <[email-masked]>
Date: Tue, 11 Aug 2009 06:28:14 -0700

On Aug 11, 2009, at 2:15 AM, Michael Tüxen wrote:

>>>> I just would like to know if there is any reason that we don't  
>>>> adopt
>>>> this kind of

And one could end up with bouncing paths...

I believe Armando's work actually had this "suspicious" state but what
he did is switch the primary to the new one... and did not switch back.
That kept the bouncing effect.... which by the way might limit  
throughput...

R

-----
Randall Stewart
[email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 12 Aug 2009 03:57:40 +0900 (JST)

From: Randy Stewart <[email-masked]>
Subject: Re: [tsvwg] Q about path failure detection in SCTP
Date: Tue, 11 Aug 2009 06:28:14 -0700
Message-ID: <[email-masked]>

 > And one could end up with bouncing paths...

Yes, and my point is I would like to discuss bouncing more.

Let's say, the primary path is a bit lossy and causes timeout (congestions or whatever).
In case of normal SCTP, retransmit packets will be transmited to secondary.

   "Furthermore, when its peer is multi-homed, an endpoint SHOULD try to
   retransmit a chunk that timed out to an active destination transport
   address that is different from the last destination address to which
   the DATA chunk was sent."

Do we call this bouncing?  Probably not.
What I'm going to propose is, when packet is retransmitted to the secondary, heart-beat is 
transmitted to the primary. If we can receive heart-beat ACK, then use primary for the next 
transmission. Otherwise, use secondary for the next transmission. 
I tend to believe this approach does not cause much bouncing effects. It behaves mostly 
like the current SCTP in case of congestion. In addition, by sending small heart-beat 
instead of data chunk to potentially congested path, it can reduce the risk of congestion 
collapse.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "David Lehmann" <[email-masked]>
Date: Tue, 11 Aug 2009 15:27:39 -0400

Yoshifumi raised a good point.

Waiting for the primary destination to time out doesn't fly with telecom
apps.  Our implementation has being doing something similar for years to
what Yoshifumi has suggested.  Our algorithm checks the primary path.
If the error count is zero, use it.  Otherwise, look for a path with the
least amount of errors.  As Randall mentioned, the spec says "SHOULD";
so it is RFC compliant.

As soon as the error is cleared, the primary will automatically be used,
given the above algorithm. (When an alternate path is used, that doesn't
change the primary.  The primary can only be changed by the user.)

Yes, a heartbeat will speed recovery of the primary path. (I think it
was Armando who suggested this to me at the UofD bake off.) Fast
recovery of the primary path seems to help throughput possibly because
data is being sent back on the primary path which will have a larger
cwnd.

We have seen no down side for this approach.  In tests, when we pulled
the Ethernet cable used for the primary path, we hardly notice a blip in
the throughput.

--
David Lehmann
Ulticom, Inc.
856-787-2952

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Yoshifumi Nishida
> Sent: Tuesday, August 11, 2009 2:58 PM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]
>  > >> 1) If timeout happens on the primary path, set the status of
this
> path
>  > >> to "suspicious"
> I tend to believe this approach does not cause much bouncing effects.
It
> behaves mostly

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 11 Aug 2009 21:50:38 +0200

> least amount of errors.  As Randall mentioned, the spec says "SHOULD";
This might be a good idea in a telco environment where the different
paths share no common link. But if for example all paths share a single
link and this one is responsable for the loss observed, switching
to a different path is counter productive.

The best way to handle it is to use all paths simultaneously, but
that is not yet standardized.
>> transmitted to the primary. If we can receive heart-beat ACK, then  
>> use
>> primary for the next
>> instead of data chunk to potentially congested path, it can reduce  
>> the
>> risk of congestion
>> collapse.
>>
>> Thanks,
>> --
>> Yoshifumi Nishida
>> [email-masked]
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 12 Aug 2009 14:01:34 +0900 (JST)

From: Michael Tüxen <[email-masked]>
Subject: Re: [tsvwg] Q about path failure detection in SCTP
Date: Tue, 11 Aug 2009 21:50:38 +0200
Message-ID: <[email-masked]>

 > to a different path is counter productive.

I think it's not very harmful even if all paths share a single link.

If all paths share a single link and the shared link is congested, 
even with normal SCTP, it tries to retransmit data to the non-primary paths.
Hence, the data go toward the shared congested link anyhow.

The slight difference between the current SCTP and my proposal is heart-beat
packets transmitted to the primary path. Yes, this is a negative factor
in this approach, but I tend to believe it's mostly acceptable.

CMT might be the ideal goal, but I would like to think what we can do on 
the current SCTP with minimal change.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "David Lehmann" <[email-masked]>
Date: Wed, 12 Aug 2009 08:56:54 -0400

I would think that most apps (not just telco apps) would benefit from and _expect_ a quick (i.e. hardly noticeable) failover when using multihomed endpoints.

As far as being "counter productive", waiting 30 seconds to failover to an alternate path is counter productive.  I would venture to guess that this characteristic is unacceptable to most apps.

--
David Lehmann
Ulticom, Inc.
856-787-2952

> From: Michael Tüxen [mailto:[email-masked]]
> Sent: Tuesday, August 11, 2009 3:51 PM
> To: David Lehmann
> Cc: Yoshifumi Nishida; [email-masked]; [email-masked]

---

From: Randy Stewart <[email-masked]>
Date: Wed, 12 Aug 2009 06:07:29 -0700

On Aug 11, 2009, at 10:01 PM, Yoshifumi Nishida wrote:

>>> apps.  Our implementation has being doing something similar for  
>>> years
>>> to

Actually, I believe we do something along these lines
when a congestion event occurs anyway. This came out
of Armando's work and was not in violation of the RFC... since
you can send a HB at any time ;-)

What I think you really want is what Armando studied in
his PHd Thesis. And showed to have no real harm.. i.e
a two level trigger... can't remember his terms for it..

Lets call it Trigger 1 and Trigger 2..

When you hit T1 you switch from your active to your
secondary until a HB is ack'd on the old primary.
He might have even stayed on the secondary.. I know
he looked at this. Since when you switch you pretty
much enter slow start and start ramping up the secondary
path... and switching back slows you again...

If you hit T2 with the old primary it is like hitting
the normal threshold.. i.e. you tell the user its unreachable.

He played with different values for the values of T1 and T2 and I  
believe
setting T1 to 1 (which is what you are proposing) was not shown to
be harmful.

You really should google his work.. in fact maybe he will chime in
here since I know he still watches this list...

Armando? You out there (probably just to busy at BBN ;-D)?

Anyway, I think doing what you propose is within the bounds of the spec
since what it goes against is in the SHOULD department.

It might be worth writing up a separate I-D documenting how it
works and why you can try it. I would think one would want to
pitch it has experimental.. I would also think one should offer
to have Armando on board as a co-author if he is interested.

Of course once the M-TCP folks figure out the right congestion control,
I will simply turn-on by default SCTP-CMT in FreeBSD after we put in
whatever the CC is they come up with...  that is, in my opinion, the
only good thing to come out of the M-TCP work ;-)

R

-----
Randall Stewart
[email-masked]

---

From: Michael Tüxen <[email-masked]>
Date: Wed, 12 Aug 2009 17:04:28 +0200

On Aug 12, 2009, at 2:56 PM, David Lehmann wrote:

> I would think that most apps (not just telco apps) would benefit  
> from and _expect_ a quick (i.e. hardly noticeable) failover when  
> using multihomed endpoints.
I agree. But this needs diverse paths...
>
> As far as being "counter productive", waiting 30 seconds to failover  
> to an alternate path is counter productive.  I would venture to  
> guess that this characteristic is unacceptable to most apps.
What I mean by counter productive is the following:
Assume multiple paths which share a link which is responsable for the  
message loss.
So switching paths doesn't help. But if you switch you always have a  
new (small)
congestion window...

If you know that you have diverse paths (like in telco setups) you can  
reduce
the threshold for switching the primary path and it will help. But if  
you
do not know what the setup is, switching due to a single timeout might  
be
suboptimal, since you do not know if the alternate path provides better
service.

Best regards
Michael
>>> years to
>>>> like the current SCTP in case of congestion. In addition, by  
>>>> sending
>>>> small heart-beat

---

From: Alberto Cortés <[email-masked]>
Date: Wed, 12 Aug 2009 18:01:36 +0200

On Wed, Aug 12, 2009 at 3:07 PM, Randy Stewart<[email-masked]> wrote:
> He played with different values for the values of T1 and T2 and I believe

I have explore the use of HBs to reduce failure detection delay,
please take a look at:

"Decoupling path failure detection from congestion control to improve
SCTP failovers"
Cortes, A.   Garcia-Rubio, C.   Campo, C.   Marin, A.   Almenarez, F.
 Diaz, D.
Dept. of Telematic Eng.,, Univ. Carlos III of Madrid, Leganes;

This paper appears in: Communications Letters, IEEE
Publication Date: November 2008
Volume: 12,  Issue: 11
On page(s): 858-860
ISSN: 1089-7798
INSPEC Accession Number: 10346278
Digital Object Identifier: 10.1109/LCOMM.2008.080884
Current Version Published: 2008-11-25
Abstract
SCTP handover is too slow to be useful as a mobility solution for most
applications. With quicker handovers, SCTP will be an interesting
solution to mobility. SCTP uses retransmissions as probes for path
failure detection. Congestion control forces slow paced
retransmissions but quick failovers need fast paced probes. Our
solution is to stop using retransmissions as probes. Instead, we
propose active path monitoring using unreliable heartbeats.
Applications can tune this algorithm to meet their own requirements.
We show data from a simple experiment in a real environment.

-- 
Alberto Cortés
Telematic Engineering Dept. at UC3M

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 13 Aug 2009 17:25:39 +0900 (JST)

From: Randy Stewart <[email-masked]>
Subject: Re: [tsvwg] Q about path failure detection in SCTP
Date: Wed, 12 Aug 2009 06:07:29 -0700
Message-ID: <[email-masked]>

 > only good thing to come out of the M-TCP work ;-)

OK. I'll check Armando's work and other related stuff again and try to prepare I-D 
and see how many people have an interest.
Thanks for your advice. 
--
Yoshifumi Nishida
[email-masked]

---

From: Scott Brim <[email-masked]>
Date: Fri, 14 Aug 2009 15:22:44 -0400

Excerpts from Randy Stewart on Wed, Aug 12, 2009 06:07:29AM -0700:
> whatever the CC is they come up with...  

and I assume you'll be there helping to test the algorithms.

> that is, in my opinion, the

I see the wink in that smiley.

Scott

---

From: Randy Stewart <[email-masked]>
Date: Sat, 15 Aug 2009 04:42:43 -0700

> and I assume you'll be there helping to test the algorithms.

Of course :-)

We will be glad to implement and test things in real O/S's

R

-----
Randall Stewart
[email-masked]