Subject: Re: [Tsvwg] sctp retrans count

From: devayya <[email-masked]>
Date: Thu, 01 Jun 2006 16:49:25 +0530

Hi,

Please see my inline comments.

regards,
devayya

Brian F. G. Bidulock wrote:

>devayya,
>
>Please see comments below...
>
>devayya wrote:                          (Fri, 19 May 2006 15:18:18)
>  
>
>>Hi Michael,
>>
>>In our present test set up, we have two paths from originating side and 
>>two from the destination, for single  association to support SCTP 
>>multi-homing. When there is no traffic flowing between these paths, and 
>>when the primary path is disabled, the secondary  path becomes primary 
>>path.
>>However when there is traffic flowing(5000 msus/sec), when the primary 
>>path is disabled it takes some time for the secondary path to become 
>>primary and carry the traffic. By this time upper layer timer, in our 
>>case m2pa delay ack timer expires(approx 2 sec), and there is traffic loss.
>>
>>the default values of sctp protocol parameters as follows,
>>
>>    3,      /* ASSOCIATION MAX. RETRANSMISSIONS */
>>    3,      /* PATH MAX. RETRANSMISSIONS */
>>    
>>
>
>If you have two paths, the association will be brought down when the primary
>fails due to retransmissions with the above settings.  In general, assoc_max_
>retrans would be better set to N * path_max_retrans, where 'N' is the number
>of paths (in your case 6 instead of 3).  In that case the switchover from the
>primary to the secondary will occur before the association peg takes the
>entire association down.
>
>The recommended values are 10 and 5 respectively, that works well for 2 paths
>because 10 == 2 * 5.
>  
>
If we set the path MAX retrans count to 3 and assoc max retans count to 
6 for two paths(3 *2), then when both the paths are down the sctp 
association remains in established state, while both paths are inactive. 
This is not correct.

The reason being,
if primary path goes down, then the path retrans error count and assoc 
error count is incremented, but the assoc error count is reset once the 
heartbeat or SACK is received for that association, probably through 
secondary path. So the path retrans error count for primary goes on 
incrementing and reaches maximum of 3 but the association retrans error 
count would get reset and would never reach more than 1 unless the 
secondary path goes down. and when secondary path goes down the path 
retrans error coutn for second path incremented and assoc error count 
incremented. Once the path retrans count reaches 3 the path is 
considered not reachable and moved inactive state. Now with no paths 
available there is no point in trying or incrementing assoc retrans 
error count beyond 3.
SO in NO condition the assoc error count  goes beyond 3 for this 
perticular association. So even though both path becomes unavailable, 
SCTP still thinks that the end point is reachable because assoc error 
count has not reached 6 and the state remains in established state.
What do you think?

Thanks and regards,
devayya

---

From: Randall Stewart <[email-masked]>
Date: Thu, 01 Jun 2006 11:46:54 -0400

Devayya:

No matter how you tweak and set the thresholds you
can always have this problem..

Consider I set them as you prescribe..

3/6

And we have two different events Failure on A
Failure on B at least 20 minutes apart.

1) We quickly move the primary to B when we
    hit the threshold and report that A is down.

2) We are using are dest B .. everything is fine..

3) B fails.. and we hit that threshold.. but no
    matter what our overall error threshold is 3 when
    that happens.. because we reset it to 0, once we
    got something through on B (when A first failed)..

At that point.. we hit the dormant state..

If you set the threshold to 3/3... you defeat
multi-homing.. since the minute one destination fails you
destroy the association..

Pick 3/4.. same thing above occurs..

The dormant state is NOT a problem IMO.. its just a fact
of how things work.. In fact the BSD-Kame stack goes ahead
and still trys to send data to one of the destinations (as
long as its not un-confirmed :-D)...

And maybe one of them will come back up...

There are other implementations that on failure of
the last destination, tear down the association..

There are others that just HB and send no data..

All of these are valid implementation options for the dormant
state...

R

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 1 Jun 2006 11:41:56 -0600

Randall,

Gosh, take a look at RFC 2960 Section 8.1 first paragraph again.

One does not stop pegging association error counts just because
a path error count reaches its maximum.

I would say that both other implementation you mention are incorrect:
the implementation that tears down the association on the failure
of the last destination is incorrect because it does not wait for
the association consecutive retransmissions to reach Association.
Max.Retrans; the other is incorrect because it does not stroke the
association consecutive retransmissions count while it is timing
out on heartbeats.

When all paths have reached their Path.Max.Retrans count, successive
retransmissions no longer increase the path error count; however,
they continue to increase the association error count, which will
eventually exceed Association.Max.Retrans as described in the first
paragraph of RFC 2960 Section 8.1 and "the endpoint shall consider
the peer endpoint unreachable and shall stop transmitting any more
data to it (and thus the association enters the CLOSED state)."

The mistake, Devayya, is that the implementation is not pegging
association error counts on each successive retransmission under the
circumstances you  mention.

--brian

Randall Stewart wrote:                       (Thu, 01 Jun 2006 11:46:54)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: devayya <[email-masked]>
Date: Fri, 02 Jun 2006 09:14:00 +0530

Brian,

Does this mean that , eventhough all the paths are down, the association 
still keeps trying until it reaches the maximum threshold and only then 
the association is taken to closed state?
Because at the time of my second path goes down, the association retrans 
count is at 3 and it has to try 3 more times before it takes association 
to closed state!!! Why is it necessary to try 3 more times, eventhough 
all paths are down?
I appreciate your clarifications.

Thanks and regards,
devayya

---

From: Randall Stewart <[email-masked]>
Date: Fri, 02 Jun 2006 07:19:43 -0400

Brian:

I did not say in my dialog below anywhere that you
stopped pegging error counts on HB or Retransmissions
in the dormant state... of course I did not
mention to do it either.. since it seems obvious and
is explicit in the spec... as you point out...

> association consecutive retransmissions count while it is timing

I will agree with you that the first "other implementation" is
spec-wise incorrect.. since they do not await the Max.retran of
the asoc to be hit before tearing down the assoc...  I never
liked this type of behavior.

The second type... as long as they continue to peg the assoc
error counter with each HB and don't send data.. finally
tearing down the association when you hit the max.retrans is
perfectly legal... this is what I was trying to describe...

Some will send data (BSD does this) and HB's

others will only send HB and hold data sends until
they get an active address...

Both of the above will peg assoc.error.count... at retran
of HB or data... failing at max....

R

---

From: Randall Stewart <[email-masked]>
Date: Fri, 02 Jun 2006 07:31:34 -0400

> the association is taken to closed state?

yep.

> to closed state!!! Why is it necessary to try 3 more times, eventhough 

Well.. what if its a temporary network outage?

You also have a choice.. send data and HB's to the paths
OR
just send HB's to the path not sending data...

An implementation that continues to try to send data and HB's will
remove the TCB quicker since HB's are at a slower rate...  but
on the other hand just sending HB's will give the peer
longer to either get the network back together ... maybe a
routing reconvergance is going on... so you might end up
recovering.. its a trade off..

I like the idea of continuing to try to send data as well
as HB's (yes and we always peg the assoc error count)...

Hmm.. and digging a little deep on Brian's comments.. 8.1 has
no MUST in it .. so in theory if your implementation does
not wish to stay in the dormant state and when all paths
fail remove the TCB.. this is not forbidden by the spec.

I would not recommend it.. since as I said, the path may
have a temporary event happening on it and by going dormant
for a while you MAY recover...  some folks even would like
this as a feature I bet (disconnected networking :-D)

Your choices in the dormant state are:

1) Continue to send Data and HB's
<or>
2) Continue to just send HB's
<or>
3) Destroy the association right away..

I know of different implementations that have all
made different choices in this as well BSD does <1>..
and yes.. if you do 1 <or> 2 you must continue to peg the
association error counter and when you reach the max
destroy the assocaition..

But since there are no MUST's nor SHOULD's for that
matter this is really an implementation choice... as
it probably should be :-D

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 2 Jun 2006 09:01:40 -0600

devayya,

devayya wrote:                                                     (Fri, 02 Jun 2006 09:14:00)
> 
>    Brian,
>    Does  this  mean  that  ,  eventhough  all  the  paths  are  down, the
>    association  still keeps trying until it reaches the maximum threshold
>    and only then the association is taken to closed state?
>    Because  at  the  time  of  my  second path goes down, the association
>    retrans  count  is at 3 and it has to try 3 more times before it takes
>    association  to  closed  state!!!  Why  is  it necessary to try 3 more
>    times, eventhough all paths are down?

Because of the way the parameters were set.  The association max retrans count
is the number of consecutive retransmissions for the association.  If you set
this equal to or below the path max retrans count, you defeat multi-homing
(the association will fail before a destination is marked inactive).  If you
want to be really brutal, set it to 4.  But then if a spurious retransmission
occurs on the good interface just after the bad one fails, the association will
fail.

Also, understand that one interface failing and then the other failing some
seconds later is rare:  what is more common is network anomalies causing
retransmission timeouts to occur for brief periods.  In the later case, it is
usually better to wait rather than fail and incur the cost of re-establishing
the association (which is what the application will invariably try to do).

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/