Subject: Re: updated text on HA section

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Thu, 13 Oct 2005 15:50:16 -0700

Good point Furquan.
 
I am not sure we can solve this in an elegant way or any way without
some sort of CE-to-CE communication mechanism.
I'll send out the text with the addition of the CE Multicast ID case as
Jamal suggested, but that might not be directly applicable for this
case.
 
We can definitely mention this as an issue in the draft and propose some
solutions based on CE-to-CE mechanisms, which are not yet defined.
 
Thanks for reviewing this!
Hormuzd

________________________________

From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Furquan A. Ansari
Sent: Thursday, October 13, 2005 9:03 AM
To: [email-masked]
Subject: Re: [FORCES] updated text on HA section

Hello Hormuzd/all,
    Haven't been closely following the discussion recently  on the
mailing list since I was quite 
busy with a few things. However, after quickly going through the HA
support section, I would be
hesitant to put this in the current draft. I feel that the HA situation
is more complicated than
spelled out here and in fact could/should come as a separate draft in
its own right.

Here are a few things that complicate the situation:
1) In an NE with multiple CEs (Active and backups) and FEs, it is
mentioned that a communication
failure between a CE and an FE - due to either link reasons or CE
failure - it would trigger the FE to
switch over to the backup (using PL/TML) CE. This can lead to a "very
bad situation" in case of loss
of communication due to link failure. We do not want to be in a
situation where only "one" of the FEs is
now being managed/controlled by the backup CE while all other FEs are
being controlled by the active
CE (since the active CE is still functional and the loss of association
between the active CE and one of 
the FEs in question occured either due to link failure or TML channel
failure etc.). This not only
destroys the notion of the having a single active CE within an NE (since
we now have the original active
CE still controlling the other FEs along with the backup CE now acting
as an active CE for "one" FE), but
making the whole collection of CE and FEs function together as a single
NE would be close to impossible,
since higher layer applications such as routing protocols etc. will now
have to be run from both the CEs.

So, essentially, I would say that there are a few choices we have:
1) If one of the FEs is switching over to the backup CE, then all other
FEs will have to switch over the
backup CE as well
    - Don't like this choice very much, because this would cause great
disruption at the network level since applications will have to be
restarted at the backup (e.g. routing protocols)
2) When a majority/all of the FEs are reporting loss of the connectivity
to the active CE (reported to the backup CE), only then should the
backup CE take over the functionality as the active CE.
   - Better choice than before, but still not the best one - since what
happens to the FEs that are not
reachable by the active CE? How do we establish control over them since
the communication is lost?
Should the backup CE somehow help this FE in trying to establish the
connection/association back to the active CE?

I can go on with many scenarios, but will stop here for now.

Regards,

Khosravi, Hormuzd M wrote:

	The text is attached as per the protocol team discussions.
	 
	regards
	Hormuzd

________________________________

	8. High Availability Support
	
	The ForCES protocol provides mechanisms for CE redundancy and
failover, 
	in order to support High Availability as defined in [RFC3654].
FE redundancy 
	and FE to FE interaction is currently out of scope of this
draft.  There 
	can be multiple redundant CEs and FEs in a ForCES NE.  However,
at any 
	time there can only be one Primary CE controlling the FEs and
there can 
	be multiple secondary CEs.  The FE and the CE PL are aware of
the primary 
	and secondary CEs.  This information (primary, secondary CEs) is
configured 
	in the FE, CE PLs during pre-association by FEM, CEM
respectively.  Only 
	the primary CE sends Control messages to the FEs. 
	
	There are 2 modes of HA defined in the ForCES protocol, Report
Primary Mode 
	and Report All Mode. The Report Primary Mode is the default mode
of the protocol, 
	in which the FEs only associate with one CE (primary) at a time.
The Report All 
	mode is for future study and not part of the current protocol
version. In this mode, 
	the FE would establish association with multiple CEs (primary &
secondary) and 
	report events, packets, Heart Beats to all the CEs. However,
only the primary CE 
	would configure/control the FE in this mode as well. This helps
with keeping state 
	between CEs synchronized, although it does not guarantee
synchronization. 
	
	The HA Modes are configured during Association setup phase,
currently only Report 
	Primary Mode is configured.  A CE-to-CE synchronization protocol
will be needed 
	to support fast failover, however this will not be defined by
the ForCES protocol.
	
	During a communication failure between the FE and CE (which is
caused due to CE 
	or link reasons, i.e. not FE related), the TML on the FE will
trigger the FE PL 
	regarding this failure.  This can also be detected using the HB
messages between 
	FEs and CEs.  The communication failure (detected via either of
the two means 
	described before) is considered as a loss of association between
the CE and 
	corresponding FE and the FE PL must establish association with
the secondary CE at 
	this point.  During this phase, if the original primary CE comes
alive and starts 
	sending any commands to the FE, the FE should ignore those
messages and send an 
	Event to all CEs indicating its change in Primary CE.  Thus the
FE only has one 
	primary CE at a time.
	
	An explicit message (Config message setting Primary CE attribute
in ForCES Protocol 
	object) from the primary CE, can also be used to change the
Primary CE for an FE 
	during normal protocol operation.

-- 
Furquan Ansari
Bell Labs Research			Ph : 732-949-5249
Holmdel, NJ				Fax: 732-949-4556

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 13 Oct 2005 22:55:03 -0400

I think the main point we are making here is that while we are putting in 
hooks to support full redundancy, we are not claiming to have built all the 
pieces that are needed.  We have not even built all the hooks probably.
We need to be clear about that.

Yours,
Joel

At 06:50 PM 10/13/2005, Khosravi, Hormuzd M wrote:
>Good point Furquan.
>
>I am not sure we can solve this in an elegant way or any way without some 
>sort of CE-to-CE communication mechanism.
>Jamal suggested, but that might not be directly applicable for this case.
>
>----------
>hesitant to put this in the current draft. I feel that the HA situation is 
>more complicated than
>spelled out here and in fact could/should come as a separate draft in its 
>own right.
>
>failure between a CE and an FE - due to either link reasons or CE failure 
>- it would trigger the FE to
>switch over to the backup (using PL/TML) CE. This can lead to a "very bad 
>situation" in case of loss
>of communication due to link failure. We do not want to be in a situation 
>where only "one" of the FEs is
>CE still controlling the other FEs along with the backup CE now acting as 
>an active CE for "one" FE), but
>making the whole collection of CE and FEs function together as a single NE 
>would be close to impossible,
>to the active CE (reported to the backup CE), only then should the backup 
>CE take over the functionality as the active CE.
>>The ForCES protocol provides mechanisms for CE redundancy and failover,
>>in order to support High Availability as defined in [RFC3654].  FE 
>>redundancy
>>and FE to FE interaction is currently out of scope of this draft.  There
>>can be multiple redundant CEs and FEs in a ForCES NE.  However, at any
>>time there can only be one Primary CE controlling the FEs and there can
>>be multiple secondary CEs.  The FE and the CE PL are aware of the primary
>>and secondary CEs.  This information (primary, secondary CEs) is configured
>>in the FE, CE PLs during pre-association by FEM, CEM respectively.  Only
>>There are 2 modes of HA defined in the ForCES protocol, Report Primary Mode
>>and Report All Mode. The Report Primary Mode is the default mode of the 
>>protocol,
>>in which the FEs only associate with one CE (primary) at a time. The 
>>Report All
>>mode is for future study and not part of the current protocol version. In 
>>this mode,
>>report events, packets, Heart Beats to all the CEs. However, only the 
>>primary CE
>>would configure/control the FE in this mode as well. This helps with 
>>keeping state
>>between CEs synchronized, although it does not guarantee synchronization.
>>
>>The HA Modes are configured during Association setup phase, currently 
>>only Report
>>Primary Mode is configured.  A CE-to-CE synchronization protocol will be 
>>needed
>>to support fast failover, however this will not be defined by the ForCES 
>>protocol.
>>
>>During a communication failure between the FE and CE (which is caused due 
>>to CE
>>or link reasons, i.e. not FE related), the TML on the FE will trigger the 
>>FE PL
>>regarding this failure.  This can also be detected using the HB messages 
>>between
>>FEs and CEs.  The communication failure (detected via either of the two 
>>means
>>described before) is considered as a loss of association between the CE and
>>corresponding FE and the FE PL must establish association with the 
>>secondary CE at
>>this point.  During this phase, if the original primary CE comes alive 
>>and starts
>>sending any commands to the FE, the FE should ignore those messages and 
>>send an
>>Event to all CEs indicating its change in Primary CE.  Thus the FE only 
>>has one
>>primary CE at a time.
>>
>>An explicit message (Config message setting Primary CE attribute in 
>>ForCES Protocol
>>object) from the primary CE, can also be used to change the Primary CE 
>>for an FE