Subject: updated text on HA section

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Wed, 12 Oct 2005 14:38:33 -0700

The text is attached as per the protocol team discussions.
 
regards
Hormuzd

---

From: "Furquan A. Ansari" <[email-masked]>
Date: Thu, 13 Oct 2005 12:02:48 -0400

Hello Hormuzd/all,
    Haven't been closely following the discussion recently  on the 
mailing list since I was quite
busy with a few things. However, after quickly going through the HA 
support section, I would be
hesitant to put this in the current draft. I feel that the HA situation 
is more complicated than
spelled out here and in fact could/should come as a separate draft in 
its own right.

Here are a few things that complicate the situation:
1) In an NE with multiple CEs (Active and backups) and FEs, it is 
mentioned that a communication
failure between a CE and an FE - due to either link reasons or CE 
failure - it would trigger the FE to
switch over to the backup (using PL/TML) CE. This can lead to a "very 
bad situation" in case of loss
of communication due to link failure. We do not want to be in a 
situation where only "one" of the FEs is
now being managed/controlled by the backup CE while all other FEs are 
being controlled by the active
CE (since the active CE is still functional and the loss of association 
between the active CE and one of
the FEs in question occured either due to link failure or TML channel 
failure etc.). This not only
destroys the notion of the having a single active CE within an NE (since 
we now have the original active
CE still controlling the other FEs along with the backup CE now acting 
as an active CE for "one" FE), but
making the whole collection of CE and FEs function together as a single 
NE would be close to impossible,
since higher layer applications such as routing protocols etc. will now 
have to be run from both the CEs.

So, essentially, I would say that there are a few choices we have:
1) If one of the FEs is switching over to the backup CE, then all other 
FEs will have to switch over the
backup CE as well
    - Don't like this choice very much, because this would cause great 
disruption at the network level since applications will have to be 
restarted at the backup (e.g. routing protocols)
2) When a majority/all of the FEs are reporting loss of the connectivity 
to the active CE (reported to the backup CE), only then should the 
backup CE take over the functionality as the active CE.
   - Better choice than before, but still not the best one - since what 
happens to the FEs that are not
reachable by the active CE? How do we establish control over them since 
the communication is lost?
Should the backup CE somehow help this FE in trying to establish the 
connection/association back to the active CE?

I can go on with many scenarios, but will stop here for now.

Regards,

Khosravi, Hormuzd M wrote:

>------------------------------------------------------------------------
>
>8. High Availability Support
>
>The ForCES protocol provides mechanisms for CE redundancy and failover, 
>in order to support High Availability as defined in [RFC3654].  FE redundancy 
>and FE to FE interaction is currently out of scope of this draft.  There 
>can be multiple redundant CEs and FEs in a ForCES NE.  However, at any 
>time there can only be one Primary CE controlling the FEs and there can 
>be multiple secondary CEs.  The FE and the CE PL are aware of the primary 
>and secondary CEs.  This information (primary, secondary CEs) is configured 
>in the FE, CE PLs during pre-association by FEM, CEM respectively.  Only 
>the primary CE sends Control messages to the FEs. 
>
>There are 2 modes of HA defined in the ForCES protocol, Report Primary Mode 
>and Report All Mode. The Report Primary Mode is the default mode of the protocol, 
>in which the FEs only associate with one CE (primary) at a time. The Report All 
>mode is for future study and not part of the current protocol version. In this mode, 
>the FE would establish association with multiple CEs (primary & secondary) and 
>report events, packets, Heart Beats to all the CEs. However, only the primary CE 
>would configure/control the FE in this mode as well. This helps with keeping state 
>between CEs synchronized, although it does not guarantee synchronization. 
>
>The HA Modes are configured during Association setup phase, currently only Report 
>Primary Mode is configured.  A CE-to-CE synchronization protocol will be needed 
>to support fast failover, however this will not be defined by the ForCES protocol.
>
>During a communication failure between the FE and CE (which is caused due to CE 
>or link reasons, i.e. not FE related), the TML on the FE will trigger the FE PL 
>regarding this failure.  This can also be detected using the HB messages between 
>FEs and CEs.  The communication failure (detected via either of the two means 
>described before) is considered as a loss of association between the CE and 
>corresponding FE and the FE PL must establish association with the secondary CE at 
>this point.  During this phase, if the original primary CE comes alive and starts 
>sending any commands to the FE, the FE should ignore those messages and send an 
>Event to all CEs indicating its change in Primary CE.  Thus the FE only has one 
>primary CE at a time.
>
>An explicit message (Config message setting Primary CE attribute in ForCES Protocol 
>object) from the primary CE, can also be used to change the Primary CE for an FE 
>during normal protocol operation.
>  
>

-- 
Furquan Ansari
Bell Labs Research			Ph : 732-949-5249
Holmdel, NJ				Fax: 732-949-4556

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 13 Oct 2005 14:07:23 -0400

On Thu, 2005-13-10 at 12:02 -0400, Furquan A. Ansari wrote:
> hesitant to put this in the current draft. I feel that the HA
> situation is more complicated than
> CE (since the active CE is still functional and the loss of
> association between the active CE and one of 
> destroys the notion of the having a single active CE within an NE
> (since we now have the original active
> making the whole collection of CE and FEs function together as a
> single NE would be close to impossible,
> since higher layer applications such as routing protocols etc. will
> now have to be run from both the CEs.
> 
Hrm.
Reading this and factoring in that text on multicast IDs is missing at
the moment makes it imperative that such text gets added. Hormuzd please
update the text.

Lets take the case of the CE first (the same concept would apply to the
FE, so ignore it for now):

If we have a CEID which is from a group that is multicast, then a link
loss does not necessarily translate to a TML loss. In the case of a TCP
TML for example, a socket getting an eof is when you decided your
association is lost and therefore require a failover. A physical
interface loss would be sufficiently dealt with by other mechanisms
(VRRP if you are using IP without affecting the socket/TML level.
In other words it is fair to say that link level loss does not equate to
a failover, no?

Now if you do use a multicast CEID:

- the FE PL does not distinguish which CE failed; that is a TML
responsibility. Only when all CEs that map to a multicast CEID are lost
does the TML declare that the association is lost.

- in an active/backup setup, you MAY need to have things like routing
protocols running on both and synchronized or booted on
failover(depending on your scheme). But this is outside our scope.

- in an active/active setup(which is really outside our scope again),
the FE PL should be able to receive messages from N CEs as long as they
are mapped to the multicast CEID, no?
 
> 1) If one of the FEs is switching over to the backup CE, then all
> other FEs will have to switch over the
> backup CE as well

Refer to my comment above.

> 2) When a majority/all of the FEs are reporting loss of the
> connectivity to the active CE (reported to the backup CE), only then
> should the backup CE take over the functionality as the active CE.
>    - Better choice than before, but still not the best one - since
> what happens to the FEs that are not
> reachable by the active CE? How do we establish control over them
> since the communication is lost?

Please review the case of multicast CEID and see if it resolves your
issues.

cheers,
jamal