Subject: Re: [tsvwg] draft-ietf-tsvwg-sctp-failover: comments and suggestions

From: <[email-masked]>
Date: Thu, 4 Jul 2013 18:38:49 +0300

Hi All,

Please find some additional comments inline below.

>
>- Section 5.1: CWND Handling for Potentially Failed Destinations
>
<snip>
>
>I agree with you to stick to the behavior of RFC4960 as default behavior. We've
>been exploring several possibilities for performance improvement and discussing
>using 2 MTU might be an idea, although we think we'll need further discussions.
>Preethi might have some more comments on this point.
>

Please accept the following addition comments:

The motivation for the 2MTU, I now understand, is to aim to mitigate unwanted effects of 
path bouncing resulting from a spurious T3-timeout followed by failover 
and subsequent "immediate" switchback operation of RFC4960 once HB ACK
is acknowledged within the back-off'ed RTO. 

While I think that one can very well say in this document that the CWND could be set to 2MTU 
in case the first HB sent on a PF path is successfully ack'ed, then I would like to note that 
this then is motivated by a desire, first and foremost, to be backward compatible with RC4960, more than it
really necessarily addresses how to best mitigate throughput degradation effects of spurious T3-timeouts.
Indeed for implementations that operate with different switchback modes than the immediate switchback operation of RFC4960,
 (e.g. operate with certain permanent failover modes), then the set of the CWND to 2 MTU on the previous PF path 
is not of a big significance as it can be most optimal, from a performance perspective,
to stay on the new data carrying path, which at this point in time may have a CWND of IW, IW+1MTU or even higher.
I think that the motivation for setting the CWND to 2MTU possibly should be made conditionally on whether the SCTP indeed follows
the immediate switchback operation of RFC4960.

One further observation is that in a situation where it is only the 2dn or the 3rd (or 4th..) of the HBs sent on the PF path, that is successful, then 
setting the CWND to 2 MTU may actually constitute a decrease of the CWND. This is the case if the implementation 
follows RC4960 and raise the CWND to IW (4MTU) after an idle period of one RTO following the T3 timeout.
When in the text it is said that the CWND should be set to 1MTU at the state transition from PF to ACTIVE does this then very explicitly
addresses the fact that unsuccessful HBs, sent on a PF path, shall keep the CWND down at 1MTU as otherwise data failures would ?
If so then I think that it could be good to add this clarification.

Final thoughts:

We have in real deployment experienced fault situations where HBs consistently were able to get through on the primary path, 
but where the path broke due to congestion once it got loaded with data. There are different ways in which one can improve the
protocol operation to best tackle such a situation. Our present solution, which is to deploy a permanent failover mode together with PF
and looking for a way to add reliability to the T3 timeouts, may not be the ultimate way to solve this, but I would like to note that 
in such a situation then consistently returning to the primary path and setting CWND = 2MTU is not the right thing to do.

>- Section 5.3: Permanent Failover
>
>We have good experiences from live networks with running SCTP in a "no
>switchback" mode of operation with no automatic migration back to a preferred
>primary path. Once the primary path is left due to it being potentially failed, then a
>selected alternative path automatically becomes the new primary path.
>
>This mode of operation corresponds to a simplified variant of the permanent
>failover operation proposed in [CAR002] with alpha = beta =
>SCTP_PEER_ADDR:THLDS. The mode of operation is run with notifications of
>change of primary path being provided to upper layers via RFC6458
>SCTP_PEER_ADDR_CHANGE semantics (SCTP_ADDR_MADE_PRIM).
>
>The mode of operation does not suit all purposes, but in networks environments
>where a number of equally preferred paths are available, then by this mode of
>operation one minimizes the effects of unnecessary path bouncing and the thereby
>associated effects of throughput degradation due to path changes and slow start
>operation.
>
>Based on our experiences we would recommend and support that the Permanent
>Failover mode of operation, at least in the simple variant described above, is
>promoted as a valid alternative approach. For the purpose of activation of this
>mode of operation a switchback (or failover) mode operation parameter could be
>introduced as configurable by the introduction of an appropriate new read/write
>socket option in the Socket API.
>
>
>Yes. I agree that there will be a situation where Permanent Failover work effectively.
>OTOH, supporting this prevents us from keeping RFC4960 behavior as RFC4960
>doesn't have this mode.
>But, I believe this is a very interesting point and we would like to get more feedback
>in order to think about this more.
>

I understand to continue to support RFC4960 behaviour, but I suppose that one can add to RFC4960
behaviour :-) We have the simple "no switchback mode" described above running in deployment.

>- Section 5.4: The impact of HB timeouts in association error counter
>
>The draft recommends for excluding HB timeouts from contributing to the
>(association) error counter. We fully support this approach for all timeouts of HBs
>that are send to probe the state of destinations addresses that are in PF state and
>we would even propose for the draft to demand that generally timeouts stemming
>from HBs, which are not sent on the Primary Path, will be excluded from the
>association error counter.
>
>Thereby the association error counter and thus the association fate reflects the
>forward progress of the data transmission to peer endpoint while at the same time it
>will be appropriately isolated from timeouts of HBs sent on non-data carrying paths.
>
>Allowing timeout of HBs sent on the Primary Path to impact the association error
>counter will serve to validate the aliveness of the peer endpoint when there is no
>data in transit.
>
>This suggestion follows the thought expressed in http://www.ietf.org/mail-
>archive/web/tsvwg/current/msg11294.html and it is thus believed to fulfil the original
>intend of RFC4960.
>
>This is a bit difficult point for me. I don't know much about the original intention of
>RFC4960, but if we follow what is written there, i think we need to increment
>association error count. (Please correct me if I misunderstand this point) So, my
>personal preference is to set bigger AMR to avoid early termination so that we can
>keep the RFC4960 behavior.
>However, we don't mind discussing to update the logic described in 4960 for more
>sophisticated solution.
>

Thanks !

Yes, I concur that the text of section 8.1 of RFC4960 stipulates that all HB Acks and HB timeouts 
influence the association error counter. But from the referenced discussion thread,
 http://www.ietf.org/mail-archive/web/tsvwg/current/msg11294.html  (simply copying it in here to avoid repeating the arguments) 
 it also seems that by purpose no keywords were used when writing these paragraphs. 
Further as the Potentially Feature was not defined at the time, I am not sure that
it would be valid to say that the text of RFC4960 apply (at least) to the HBs sent as part of this function.

>From our perspective what we would like to support and see happening is 
that all HBs, whether sent as part of the PF function or not, influence the association error counter 
if and only if they are transmitted on the Primary Path (and then with standard logic running making sure that all data transmissions 
prevent (or cancel in race conditions) any influence from HBs).

Thereby the association error counter will be influenced by (and will thus reflect) the data transmission only.
Except in idle situation where HBs on Primary Path will serve to update the association error counter in a similar manner as data would.
Thereby one get a very clean and reliable implementation of the association error counter. By reliable, I mean that one can use
the same AMR value regardless of the level of HB probing that one have activated and regardless of whether the Potentially Failed feature is activated or not.
Such an implementation also makes the AMR value be independent from the number of paths supported in the association context. 
One can choose to use a value of AMR proportional with the number of paths in order to have all paths be tried a number of times before association closure, but
with the choice suggested, one is not forced to "artificially" raise the AMR value to make room for the HBs that will fire as part of the potentially failed feature as well
as for some fraction of the Path Monitoring HBs that can happen to fire during the same timeframe.

An additional thought is that in the odd cases where HBs get through whereas data do not, then I think that it is contra-productive to have the association error counter be reset
as long as data consistently fail. 

I, personally, does not so much like the suggestion to use a "maximal" or "bigger" AMR. I think that it will be complex to deduce exactly
what failure time (aka number of data retransmission)  that a certain AMR value will guarantee you, 
whereas when only data transmission influences the AMR, things will be clean, simple and predictable.

I would be very happy if we by this communication could solicit information from others.

>
>- Section 5.1 Thoughts on the destination address state machine
>
>In RFC4960 address state machine, notification of cease of use of the current
>primary path as transmission path for new data is provided to upper layer via
>(coupled with) the announcement of the destination as being UNREACHABLE. With
>the introduction of the potentially failed state as a semi-hidden state (not resulting in
>notified state transition to UL) the information on the fail-over to use a new data
>carrying path is lost.
>While there may be merits in suppressing notifications on potentially failed
>addresses, then we believe that it is unfortunate that the notification of failover from
>usage of the primary path for new data is lost.
>This is especially problematic in path bouncing scenarios where the primary path is
>unstable but still works some. E.g., HBs may go through, but the path breaks (data
>transmission fails) as soon as it is more loaded. In such a situation no notifications
>may be generated to UL, even if new data transmission consistently cycles to and
>from the primary path. The solution to such path bouncing scenarios will have to
>come from more advanced path failover mechanism, like [CAR002] (and would not
>be solved just by notification), but it would be a great help if appropriate
>notifications were provided.
>
>I think it will be OK to send notification as long as it's a new notification type. But
>we should not send notifications defined in RFC6458.

ok - I assume this is for backward compatible reasons. 

>I guess having a new notification type won't cause problems, but I would like to get
>more feedback on this.

Ok - we can come back with a suggestion.

Thanks a lot !

BR, Karen

>Thanks!
>--
>Yoshifumi

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 15 Jul 2013 09:49:02 -0700

Hi Karen,

Thanks for your comments. I also added my comments in line.

On Thu, Jul 4, 2013 at 8:38 AM, <[email-masked]> wrote:

> Indeed for implementations that operate with different switchback modes
> than the immediate switchback operation of RFC4960,
> to stay on the new data carrying path, which at this point in time may
> have a CWND of IW, IW+1MTU or even higher.
> I think that the motivation for setting the CWND to 2MTU possibly should
> be made conditionally on whether the SCTP indeed follows
> One further observation is that in a situation where it is only the 2dn or
> the 3rd (or 4th..) of the HBs sent on the PF path, that is successful, then
> When in the text it is said that the CWND should be set to 1MTU at the
> state transition from PF to ACTIVE does this then very explicitly
> addresses the fact that unsuccessful HBs, sent on a PF path, shall keep
> the CWND down at 1MTU as otherwise data failures would ?

Ok. I see your point.
I think we should explicitly say that we should set CWND to 1MTU after
unsuccessful HBs.
I don't think using 4MTU after seeing HB timeout is good approach. Thanks
for pointing out.

> Final thoughts:
>
> We have in real deployment experienced fault situations where HBs
> consistently were able to get through on the primary path,
> but where the path broke due to congestion once it got loaded with data.
> There are different ways in which one can improve the
> protocol operation to best tackle such a situation. Our present solution,
> which is to deploy a permanent failover mode together with PF
> and looking for a way to add reliability to the T3 timeouts, may not be
> the ultimate way to solve this, but I would like to note that
> in such a situation then consistently returning to the primary path and
> setting CWND = 2MTU is not the right thing to do.
>
>

Yes, I think that's doable. I think the question here is the degree of
recommendation for this.
The easiest one is just saying "An implementation MAY support no switchback
mode", but I'm gueesing some people might prefer more explicit expressions.

> if and only if they are transmitted on the Primary Path (and then with
> standard logic running making sure that all data transmissions
> Except in idle situation where HBs on Primary Path will serve to update
> the association error counter in a similar manner as data would.
> Thereby one get a very clean and reliable implementation of the
> association error counter. By reliable, I mean that one can use
> the same AMR value regardless of the level of HB probing that one have
> activated and regardless of whether the Potentially Failed feature is
> activated or not.
> Such an implementation also makes the AMR value be independent from the
> number of paths supported in the association context.
> One can choose to use a value of AMR proportional with the number of paths
> in order to have all paths be tried a number of times before association
> closure, but
> with the choice suggested, one is not forced to "artificially" raise the
> AMR value to make room for the HBs that will fire as part of the
> potentially failed feature as well
> An additional thought is that in the odd cases where HBs get through
> whereas data do not, then I think that it is contra-productive to have the
> association error counter be reset
> I, personally, does not so much like the suggestion to use a "maximal" or
> "bigger" AMR. I think that it will be complex to deduce exactly

I agree that the approach described in the draft is not the best approach.
But, since it doesn't violate RFC4960, we recommend it in the draft.
I think what you suggest sounds reasonable and prefer to update 4960 based
on this, although I guess we'll need more discussions.

> ok - I assume this is for backward compatible reasons.

Yes. As far as I can think that's the reason.

> Ok - we can come back with a suggestion.

Thank you so much for many valuable feedback. We really appreciate!

Regards,
--
Yoshifumi

---

From: <[email-masked]>
Date: Mon, 22 Jul 2013 13:00:02 +0300

Hi Yoshifumi,

Please find some follow up below.

BR, Karen

Ok. I see your point.
I think we should explicitly say that we should set CWND to 1MTU after unsuccessful HBs.
I don't think using 4MTU after seeing HB timeout is good approach. Thanks for pointing out.

[Karen] I think that there could an issue here. On a data carrying path then the CWND is set to 1 MTU after a data T3 timeout.
But on other paths, then I think that it could be unfortunate if a random loss of one HB (a periodic path monitoring HB)
made the CWND be reduced to 1 MTU.

I would opt for the set CWND to 1MTU after unsuccessful HBs to be performed ONLY on a path in PF state, i.e., not for the HB that made the path enter PF state nor for HBs done of Paths that are INACTIVE state, thereby one would make the CWND reductions be a little more robust towards random packet drops (of HBs).

Alternatively one should do nothing special at all and keep the situation of RFC4960 where the fate of the HBs do not impact the CWND at all.

Side note: A single drop of a data packet on a sparsely loaded path (or in data tail - not enough data for Fast Recovery to kick in) today also results in CWND reduction and this of course compares with the loss of HB situation. But I think that this is a problem, rather than a desired effect, in the present algorithms that one continuously tries to find solutions for. We should not make this problem worse by also introducing if for single drop of HBs. I think.

Final thoughts:

Yes, I think that's doable. I think the question here is the degree of recommendation for this.
The easiest one is just saying "An implementation MAY support no switchback mode", but I'm gueesing some people might prefer more explicit expressions.

[Karen] Yes, I agree that better than MAY is desirable.
But as a starting point then I think that a MAY support other switchback modes is much better than discouraging such,
which is like I read the draft at present.

Thanks !

I agree that the approach described in the draft is not the best approach. But, since it doesn't violate RFC4960, we recommend it in the draft.
I think what you suggest sounds reasonable and prefer to update 4960 based on this, although I guess we'll need more discussions.
[Karen] I agree, I will try to initiate such more general discussions.

[Karen] Ok.

Regards,
--
Yoshifumi

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 26 Jul 2013 00:11:41 -0700

Hi Karen,

On Mon, Jul 22, 2013 at 3:00 AM,  <[email-masked]> wrote:

> [Karen] I think that there could an issue here. On a data carrying path then
> the CWND is set to 1 MTU after a data T3 timeout.
>
> But on other paths, then I think that it could be unfortunate if a random
> loss of one HB (a periodic path monitoring HB)
>
> I would opt for the set CWND to 1MTU after unsuccessful HBs to be performed
> ONLY on a path in PF state, i.e., not for the HB that made the path enter PF
> state nor for HBs done of Paths that are INACTIVE state, thereby one would
> make the CWND reductions be a little more robust towards random packet drops
> (of HBs).
> Alternatively one should do nothing special at all and keep the situation of
> RFC4960 where the fate of the HBs do not impact the CWND at all.

Agreed. I meant for only a path in PF status since PF only sends HBs a
path in PF status.
I think the behavior of HB timeout on a path in other states should follow 4960.

> Alternatively one should do nothing special at all and keep the situation of
> RFC4960 where the fate of the HBs do not impact the CWND at all.
>
>
>
> Side note: A single drop of a data packet on a sparsely loaded path (or in
> data tail - not enough data for Fast Recovery to kick in) today also results
> in CWND reduction and this of course compares with the loss of HB situation.
> But I think that this is a problem, rather than a desired effect, in the
> present algorithms that one continuously tries to find solutions for. We
> should not make this problem worse by also introducing if for single drop of
> HBs. I think.

I also think we need something here as single drop of HB might not be
an indication of severe congestion.
I'm thinking that something like draft-ietf-tcpm-newcwv might need to
be considered.

Thanks,
--
Yoshifumi