Subject: Re: An approach to modeling ForCES events

From: "Chawla, Shuchi" <[email-masked]>
Date: Thu, 21 Apr 2005 08:33:35 -0700

Jamal,

Comments inline...

Regards,
Shuchi

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Jamal Hadi Salim
Sent: Wednesday, April 20, 2005 5:37 AM
To: [email-masked]
Subject: Re: An approach to modeling ForCES events

Shuchi,

On Tue, 2005-19-04 at 13:35 -0700, Chawla, Shuchi wrote:

[..]

> -       In certain cases, it is possible that a large number of
> events/alarms get generated since a condition on object X could lead
> to a different condition on objects associated with X, and so on.  If
> the FE is not doing event correlation and root cause analysis, it
> might land up sending all of these events to the CE, and it can result
> in event flooding.  As an example, if an FE is associated with a large
> number of channelized interfaces, each configured to provide a
> service, if the physical interface experiences a loss of signal, all
> channelized interfaces will also have an alarm, as will the services
> associated with the channelized interfaces.  Depending on the alarm
> detection mechanisms and alarm integration times, it is possible that
> the root cause alarm is detected after the alarm conditions that
> result from it.
> 

The alarm management is an NE level/layer detail (derived from the
happenings in the FE(s) and maybe in the CE no doubt).
You seem to agree on this after looking at your last response; 
[As an example - I would expect no changes whatsoever in the way system
root cause analysis or fault management schemes would work just because
of how we do events between CE-FE. To the Forces level CE, the link
going down is just another event and to the alarm management relating
one or more NE that may end up being a high priority fault that need to
be repaired; forces will send any "repair" messages without needing to
understand the high level details etc etc]. 

[Shuchi] I believe both cases are possible -- alarms directly generated
(and cleared) by the FE with the CE acting more as a pass-through or the
FE just generating events, and some entity on the CE doing the needful
to convert these to alarms to be sent to some NMS.  In a system I had
worked on, the former model was followed.  However, my assumption is
that there are some events/alarms from the FE that are also going to be
used by the CE on the basis of which it will take some specific action.
I think there are going to be events/alarms that are of critical nature
and those that are more informative.  Minimally, it seems there should
probably be some mechanism to prioritize the critical ones from the FE
to the CE so that they don't get held up behind a burst of informative
events.  Maybe this is where message priorities get used, I suppose.
This would be especially true of events related to faults or loss of
signal type events etc. where there is a need to failover to a standby
entity.

You do bring an interesting example though with the channelized
intefaces in relation to the hierachies of possible events.
If the CE is told that the physical interface went down; isnt that
sufficient as a detail to mean all the logical interfaces also went
down? i.e would this still mean that the FE sends events for all
channelized interfaces as well? Refer to my views on earlier email on
desire to have hierachical semantics to avoid ambiguities (but now based
on your example also to avoid unnecessary events).
Likewise if a table is flushed it should be sufficient to indicate that
all entries have been deleted without sending events for every row.

[Shuchi] If the FE did detect that the physical interface went down
prior to the channelized interfaces going down, then it is possible for
the FE to correlate the problem on the channelized interfaces to that of
the physical interface and hence only send the event for the physical
interface.  However, it is possible based on alarm integration times and
how the detection is implemented, that one or more logical interfaces is
detected to be down prior to the physical interface being detected.
It's up to the implementation on the FE to then maybe check the physical
interface status prior to sending up these logical interface events.
There can be several levels in this hierarchy of interfaces, so there's
no guarantee how this might be implemented within an FE. 

> -       In my opinion, if we can avoid support for deferred events, we
> should.  

You may be refering to what i said: I used the word "defered" but didnt
intend for it to mean defering events rather it is to defer processing
of events.
What i said is: If you register to receive events you typically do so 
in order to process them - otherwise you dont register for them; 
i.e, when you receive notification of an event at the CE, you create
work to process it. In other words, such a notification is a
work-scheduling signal (this would not be necessarily true if you just
received an event you had no interest in). Typically, the work is
scheduled to run immediately. If you allow edge vs level triggering of
events then you could use that scheduling signal to _defer_ the work
(and not the event).

[Shuchi] My mistake -- I had misunderstood the reference to "deferred".

So the desired support is for the two kinds of triggers.
In one scheme the event trigger stays disabled (after sending a event to
the CE) until re-enabled explicitly by the CE and in the second case
it is always enabled.
This is very easy to add to the definition of the event; brute force: it
becomes an XML-level-attribute at definition time although my preference
would be it gets set at runtime and therefore is a boolean within the
definition of the event. A registration to the event could be used to
reenable it every time.

[Shuchi] Isn't registration for an event a "one-time" thing?  I'm
unclear about a registration being used to reenable it every time...

Again like i said earlier - I know we will do this in our implementation
(based on some of experiences in implementation thus far); it would be
nice if Forces supported it, however if consensus is not to then we will
keep it proprietary. I do hope consensus is derived from people
understanding what i am proposing though.
  
> If there is an issue with the CE not being able to keep up with the
> events from the FE, maybe the severity of the condition (or some other
> threshold) can be used as a filter to determine what to send to the
> CE.

All these are attempts to help offload the CE and reduce abuse of
resources (bandwidth and CPU in this case). Same with event-trigger type
being edge vs level.

>   I agree that packets that need processing by the CE do need to get
> sent by the FE to the CE.  I wasn't viewing these as "events" however.
> If we have the concept of a "data" versus a "control" channel, these
> packets would be considered for transmission over the data channel,
> whereas events/alarms would be reported over the control channel (I
> may have misunderstood what was being stated).
> 

But these are not intentionaly redirected packets (such as OSPF for
example).
They are "units of work" for some event that needs to be processed at
the CE. Example are ones to be exception handled in particular; i am not
so sure these belong to the data channel. 
The other " units of work" are packets built as a result of other
packets arriving at the FE(sic);->. I gave the example of an IPSEC
ACQuire being sent to the CE as a result of a missing security
association in the FE.

[Shuchi] Would you consider these to be "events" or just other "control
messages" (such as those from the CE to the FE)?

> -       Thresholds being used as triggers for events/alarms are
> definitely a good thing to support in my opinion.  In large systems,
> where the number of objects being managed and monitored is
> significant, thresholds provide a mechanism to support manageability
> of such system without overwhelming the CE and running out of space on
> the event/alarm logs that are maintained in persistent storage.
> 

I agree - an optimization to offload the CE. Nota Bene the same logic
reasoning is used for the event-trigger-type ;->

> -       I think event subscription/registration should only be
> possible on events explicitly defined by an LFB.

This is agreed on.

>   It seems it would be rather complex to express the "situations"
> under which to use different verbosity levels for the different event
> types.  Are there specific use cases that require this flexibility? 

Optimization is a good enough reason in my opinion.

[Shuchi] Do you have any specific examples of this?  I'm still no too
sure how exactly this would be specified and used.

>  Else, it would seem reasonable to limit this to registration time.

Verbosity - I dont agree that should be at registration time; it is
trivial to add to both the model and protocol that it is done at 
run time. 

>  I assume multiple entities on the CE can register for the same event
> from an FE.  What if one registers with a high verbosity level, and
> one with a low verbosity level?  Would the FE now have to send 2
> separate events with the different levels of verbosity, or is it the
> intent that the FE always send all the info., and an "event subsystem"
> on the CE will do the filtering for the different registrations?  

Good point to bring up: Upto this point ive thought of the CE
as only registering once per event. 
On the systems i have used this scheme on, the highest verbosity level
is sent in such a case and it would be up to the CE to discriminate.
i.e any intelligence will have to be embedded at the CE.

> If so, have we gained that much with this support for different
> verbosity levels?  Is this case valid?

I think the case is valid - whether we have gained much will depend on
the implementation. Clearly it is a win over the case where you send two
events and they are all verbose for two different registrations
(I assume this to be the case if you didnt have a level differentiator).

Ok, that was long ;-> Blame it on coffee.
I didnt proof read it so i may be repeating myself in a few places.
Dont disappear now.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 28 Apr 2005 09:25:37 -0400

Hello Shuchi,
I asked you not to disappear, so to set an example i am back ;->

> used by the CE on the basis of which it will take some specific action.

Agreed, although the scope of their interpretations is different.

> signal type events etc. where there is a need to failover to a standby

nod on prioritization.

BTW, this is a good example of scope. The Forces scope for recovery and
failover to a standby is restricted within an NE. As an example, at the
forces NE level, a link failure or FE failure would mean a failover
within the NE (to another link/FE depending on policy). An alarm system
looking at the same issue would have a broader scope of failing over to
a different NE altogether.

> no guarantee how this might be implemented within an FE. 

Agreed - maybe that was a bad example.
I wanted to demonstrate that if you could have hierachical events, that
it would make sense to send less event notifications by only sending the
"top of hierachy event".
As an example, assume the physical went down - then you only want to
mention that to the CE and not the other 100 channels. The CE should be
able to deduce.

> unclear about a registration being used to reenable it every time...

Theres nothing that says event registration/deregistration cant happen
multiple times at runtime. Infact not allowing that maybe
counterproductive.

> messages" (such as those from the CE to the FE)?

Depends on the implementation i think - and I dont wanna be biased
(and believe we should cover both to be usable).
Lets take the example of the Acquire event in the case of IPSEC. Its
essentially a table lookup failure of an arriving packet. Typically,
such a lookup generates an Acquire event to the key manager. What gets
sent to the key manager is not a packet but rather a description of some
important things in the packet as well as some metadata. The result of
the acquire is a table update on the FE.
A different scheme maybe to send the packet in its entirety to the CE.
The former would fit as an "event" and the later is closer to 
"control".

> sure how exactly this would be specified and used.

Refer to my email to Joel. I really dont need to have to list a specific
LFB that would benefit from this - its a classical scheme when you talk
events.  I am not inventing anything new here.

cheers,
jamal