Subject: [L2tpext] Re: Questions about draft-ietf-l2tpext-failover-02.txt

From: Vipin Jain <[email-masked]>
Date: Wed, 1 Oct 2003 14:49:47 -0700 (PDT)

Paul,

You have some good points and feedback.
I am responding to your original email while
keeping in mind the discussion you and Keyur have
had so far and bringing them to the list.
my comments inline..

> I've got some questions about the failover draft.
> 
> sec 2.3 - "Endpoints MUST avoid sending any control packets over the old 
> tunnel that is being recovered".   What is the behavior for in-transit 
> control packets?   I presume the receiver should silently discard any 
> control packets on the old tunnel (no acknowledgement, no action based 
> on received control packet).   I presume such behavior should start once 
> recovery is detected (either by failed endpoint sending new SCCRQ or by 
> failed endpoint's peer receiving new SCCRQ)?
True. Probably we we should clarify that the packets received on receiving end
MUST be sliently discarded.

> 
> sec 2.3.1, last bullet - "...it SHOULD clear the tunnel and sessions 
> within that tunnel on its end".   I presume this is a silent clearing 
> (no transmit of StopCCN or CDN)?
True. Probably we should clarify that too explicitly.

> sec 2.3.3, last bullet - Would it not be potentially premature to start 
> establishing new sessions prior to re-synchornization of the session 
> state?  
As Keyur pointed out, the session re-synchronization is not mandatory
aspect and is provided to bring predictability at an end to know session's
state. Therefore we have to allow the tunnel to accept new sesisons upon it
getting established.
Later in your response, you pointed out that it could result in a problem where
session ids are reused while the session states are being synchronized. Well,
that is a possibility and was discussed earlier. It was decided to leave it to
the implementation. From you comments it does look it should be at least
clearly identified and following suggestions could be made for an
implementation to handle it:
- upon detecting the reuse, both the old and new session MUST be discarded (the
new one discarded with CDN other one silently dicarded). This would lead to
some sessions not being able to establish temporarily. But then, considering
the number of sessions that would be in incosistent states on the two
endpoints, in average situations, it may be an acceptable solution.
- The session-id allocation is LRU basis would ensure the sessions do not get
recycled quick enough. This is true even for tunnels, though it doesn't go
along with section 

The reason we did not go for mandating synchronizing session ahead of calling
news sessions being allowed was following:
- If session state is synchronized, then all sessions would have to be
synchronized to keep both ends on the same page. This process for a scalable
setup (about 32k sessions on 8k tunnels or so) might take long enough (in order
of tens of seconds) by which it may be too late to respond to new sessions
anyways, considering default PPP timeout values.
- The idea was to identify the sessions that have idled-out by some means. One
ways is to observe the traffic. Besides, the FSQ/FSR message construct allows
an implementation to identify if the session is considered alive or is is idle
just due to 'traffic not present on that session during a given time'. Assuming
PPP Echo request would trigger some packets in a targeted time period.

> sec 2.3.4 - "The Session State Inconsistency mechanism SHOULD be carried 
> out only for sessions that are not getting any traffic".   It would seem 
> that this would be based on observation of data flow after determining 
> that a failover has started.   How would the failed endpoint do this 
> reliably?   Consider that data is in-transit from the failed endpoint's 
> peer when the failed endpoint starts tunnel recovery.   If this 
> in-transit data is the last data to be sent and the failed endpoint's 
> peer has initiated a disconnect then it would seem possible to have the 
> failed endpoint consider the session established with data flow (and 
> therefore not query it's peer) and for the peer to consider it not 
> established (and thus not query the peer).   The result would seem to be 
> a mis-match in the session state even after exchange of the FSQ and FSR.
To answer 'how would an endpoint do this reliably?' I'd say it need not be
done that accuracy, because FSQ and FSR would ultimately resolve that and do
achieve the same reliably. Say an endpoint suspects a session that might have
been removed on the peer because there is no traffic. Then it could ask the
peer the same in an FSQ, thereby reliably determining if it needs to consider
that session 'hung' or not.
FSQ and FSR are facilitated to exchange the state to bring them in sync, not to
conclude that they are inconsistent and should be kept that way.
I think in the draft we could emphasize that:
Upon detecting a session gone away on the peer, an endpoint MUST silently clean
up the session on its end.

Let me comment here on discussion you and Keyur had, to quickly bring all of us
on the same page:
>KRP> again, my interpretation here would be that, if not in this period, in 
> the next period - due to inactivity the failed node would sync the session
> and  should clean up.  The idea here was (correct if I am wrong Vipin)
> to avoid permanent "hang" of the session.
>
> PWH> But, the draft also mentions the importance of time limiting the 
> acceptability of these messages.  It doesn't really seem possible that 
> an endpoint can really wait to determine which sessions are potentially 
> out of sync.   If we have one implementation waiting for a period of 
> inactivity on the new TID and the other implementation locking out FSQ 
> after a period, we are going to end up with the FSQ lockout period < 
> inactivity period and thus end up with "hung" sessions.   It would seem 
> that a positive handshake of the established sessions would eliminate 
> any issues here with timing.   Using dedicated handshake messages (which 
> include the session resync data) instead of SCCRQ, SCCRP, SCCCN would 
> allow accomplishment of this task with a potential reduction in the 
> number of control messages required. 

The importance of timing of FSQ and FSR comes from the fact that this mechanism
should not be used in any post failover operation. And it should be considered
concluded once a peer think its sessions are in sync. However if it is running
out of time to conclude such a thing, it SHOULD use FSQ/FSR messages to resolve
state of such sessions.

> sec 2.3.4 - "Data Plane Behavior" - When should the data flow be 
> transitioned from the old TID to the new TID?   Should data flow be 
> interrupted upon start of tunnel recovery until completion of tunnel 
> recovery?   What about in-transit data that arrives after the switch to 
> the new TID?   Should data on the old TID be accepted until data is 
> received on the new TID?
The trigger to new tunnel is the establishment of new tunnel.

Based on the discussions Keyur, myself and other folks had so far, the idea was
not to disturb the data path as much as possible. Keeping that in mind, the
data on the old tid would be allowed to go through till the new tunnel is
established. Upon having established (not just received SCCRQ, for example) the
new tunnel the switch over should trigger and the old tunnel would be
considered stale from there on.

> sec 2.3.4 - "Data Plane Behavior" - "...the non-failed endpoint MUST set 
> the next expected Ns based on the incoming Ns value..." - I presume the 
> next expected Ns is set based on the incoming Ns value of the first data 
> received on the new TID?   Why is this only required at the non-failed 
> endpoint?

The data plane getting affected on the failed node is likely. I don't know why
you say on non failed node data plane getting affected is a possibility at all?

Considering the possibility of data plane not being affected by failure on the
endpoint - then it need not reset the Ns, it could simply use whatever it was
using (thereby allowing peer to choose that number).
Where as if data plane was affected then the failed node may choose to reset
the Ns value to zero.

> sec 3.1 - I presume multiple FSQs are allowed to handle situations with 
> large numbers of sessions on a single tunnel?   Similarly with multiple 
> FSRs?
True. 

> sec a.2 - "target time of 3-5 seconds for 30K tunnels" - Does this 
> really set a realistic expection?   It would seem that we're looking at 
> on the order of 210K control packets to recovery 30K tunnels (SCCRQ, 
> SCCRP, SCCCN, FSQ(2), FSR(2)).   This works out to 40K to 70K pps of 
> control traffic at a time when one peer is doing failover recovery and 
> is likely to be rather busy with activity over and above L2TP 
> requirements.  
sure, that was the design consideration. Not to set the realistic expectations.
But it was the trigger point for us to say that we do not want to exchange the
state of each and every session right away before we can conclude the recovery
process to be complete.

>General - What was the reasoning behind creating a new tunnel instead of 
>just doing a resync exchange on the extant tunnel?   Depending upon the 
> >structure of the implementation this could potentially be a rather 
>expensive operation to "move" sessions to the new tunnel resulting in 
>increased forwarding interruption, increased cpu intensiveness, and 
>slower recovery.  I guess I don't understand the value in changing the 
>TIDs.  
>
>KRP> Interesting you bring this up - Vipin and I had a long discussion
> on this way back.  The conclusion was that we need a 3 way handshake
> to resync after failover - to authenticate each other.  Since we
> already have a tunnel setup handshake, why not augment that? and
> hence the procedure in this draft.  The other question is why
> assigned a different TID after a failover? and honestly I can't 
> remember why it has to be a different TID - it can very well be the
> old TID and the implementation supports that - does not recommend
> that a implementation SHOULD do that when possible - may be
> Vipin can shed more light on this.

>
> PWH> I think unless there is specific value in using a new TID, then it 
> shouldn't be part of the procedure.   I think the expectation of a 
> reader of the draft will be that a new TID gets assigned and thus 
> implementations won't be necessarily expecting re-use of the existing 
> TID.   Changing the TID has a potentially signficant peformance hit 
> depending upon how the implementation has structured their forwarding 
> database and how expensive it is to update said forwarding database.   
> It would seem that what we are really trying to do here is reset the 
> control plane (and if necessary the data plane) sequence numbers and 
> detect any sessions that are in an inconsistent state.   Changing the 
> TID seems to be an unnecessary complication.   If there's real value in 
> changing the TID, then we should do so, but I'd like to understand what 
> this value is.
The new tunnel id was choosen to avoid interference with old control messages.
Even though it has side effects of having a hit on data plane, it was
considered to give control plane negotiations a mechanism for recovery.
However if we change the tid, after having established the new tunnel we would
minimize the data plane. Would it be truly hitless, probably not.

thanks,
-- vipin

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

_______________________________________________
L2tpext mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/l2tpext

---

From: Vipin Jain <[email-masked]>
Date: Wed, 1 Oct 2003 14:49:47 -0700 (PDT)

Paul,

True. 

thanks,
-- vipin

---

From: Paul Howard <[email-masked]>
Date: Tue, 14 Oct 2003 14:56:18 -0400

Vipin,

Thanx for your comments.   See [pwh2] inline.

Thanx,

Paul

Vipin Jain wrote:

Paul,

[pwh2] In the case of a data plane failover, one side (the failed side) has
lost it's next Ns and it's next expected Ns.   Thus the failed endpoint must
reset it's next expected Ns based on the Ns from the next incoming data frame
from the non-failed endpoint.   On the non-failed endpoint, neither the next
Ns or the next expected Ns have been lost.   However, the next incoming data
frame from the failed endpoint will not have an Ns matching the next expected
Ns.   Thus the non-failed endpoint must also reset it's next expected Ns
based on the incoming Ns.   Thus the sentence should really read "....each
endpoint MUST set the next expected Ns based on the incoming Ns value..."

True.

thanks,
-- vipin

---

From: Paul Howard <[email-masked]>
Date: Tue, 14 Oct 2003 14:56:18 -0400

Vipin,

Thanx,

Paul

Vipin Jain wrote:

Paul,

True.

thanks,
-- vipin

---

From: Vipin Jain <[email-masked]>
Date: Wed, 15 Oct 2003 11:40:21 -0700 (PDT)

Paul,

I am snipping out plenty of text that you didn't comment
on in this email..

> <pwh> sec 2.3.4 - "Data Plane Behavior" - "...the non-failed endpoint
> MUST set the next expected Ns based on the incoming Ns value..." - I
> presume the next expected Ns is set based on the incoming Ns value of
> the first data received on the new TID?   Why is this only required at
> the non-failed endpoint?    
>   
> <vipin> The data plane getting affected on the failed node is likely.
> I don't know why you say on non failed node data plane getting affected
> is a possibility at all?  
>
> [pwh2] In the case of a data plane failover, one side (the failed side)
> haslost it's next Ns and it's next expected Ns.   Thus the failed endpoint 
> mustreset it's next expected Ns based on the Ns from the next incoming data
> framefrom the non-failed endpoint.   On the non-failed endpoint, neither the 
> nextNs or the next expected Ns have been lost.   However, the next incoming 
> dataframe from the failed endpoint will not have an Ns matching the next 
> expectedNs.   Thus the non-failed endpoint must also reset it's next expected

> Nsbased on the incoming Ns.   Thus the sentence should really 
> read "....eachendpoint MUST set the next expected Ns based on the incoming Ns

> value..."
I see your point; The clause to omit 'failed node' resulted from the assumption
we made that upon detecting failure the non-failed endpoint would also reset
its Ns to 'zero' as if the session was started afresh and so the expected Ns on
the failed node would be 'zero'.
However, to keep smoother transition, it may not be desired particularly when
data plane is not failing over. And as you point out, mandating both ends reset
their 'expected Ns' is safer approach in all situations.

thanks,
-- vipin

---

From: Vipin Jain <[email-masked]>
Date: Wed, 15 Oct 2003 11:40:21 -0700 (PDT)

Paul,

thanks,
-- vipin