Subject: Re: An approach to modeling ForCES events

From: "Chawla, Shuchi" <[email-masked]>
Date: Tue, 19 Apr 2005 13:35:15 -0700

I haven't been following the entire thread on event modeling in detail,
but here are some of my thoughts - some of them are on the general
aspects of event and alarm reporting.  So, it is a slight digression
from the conversation, but hopefully, still relevant.

There are some ITU-T specs for data networks/communication systems that
specify the alarm and event reporting functions/requirements.  It's been
a long time since I've looked at these specs. and I don't currently have
access to them, but I think it would be worth it to scan through them to
see the type of functionality that such systems should support, and what
might be considered overkill.  The two that I saw on the ITU site are:
ITU-T X.733 and ITU-T X.734 (I assume these were the ones I had looked
at in the past, but I don't remember for sure!).

-	There should be support for both events and alarms, since they
have differences.  If events are thresholded, and they exceed their
threshold, it may lead to an alarm condition.  Alarms may be generated
and cleared.  Events do not have a concept of a "clear".  An attribute
change on some object may be classified as an event.  A Loss of Signal
on a physical interface would classify as an alarm.  An alarm should be
generated when the condition is detected (after the appropriate
integration time), and then a "clear" generated when the condition
clears. 
-	From what I remember, these specs. classify alarms into
different classes (equipment, QoS etc.).  Each alarm has a default
severity (which is not modifiable) and perceived severity (which is
modifiable).  The support for perceived severity arises from the fact
that what may be considered a "major" condition on some network element
in a certain network, may be considered "minor" in some other network
element based on its function and topology in the network etc. Alarms
define probable cause and proposed repair information (among other
information, which I don't remember).
-	In certain cases, it is possible that a large number of
events/alarms get generated since a condition on object X could lead to
a different condition on objects associated with X, and so on.  If the
FE is not doing event correlation and root cause analysis, it might land
up sending all of these events to the CE, and it can result in event
flooding.  As an example, if an FE is associated with a large number of
channelized interfaces, each configured to provide a service, if the
physical interface experiences a loss of signal, all channelized
interfaces will also have an alarm, as will the services associated with
the channelized interfaces.  Depending on the alarm detection mechanisms
and alarm integration times, it is possible that the root cause alarm is
detected after the alarm conditions that result from it.
-	In my opinion, if we can avoid support for deferred events, we
should.  If there is an issue with the CE not being able to keep up with
the events from the FE, maybe the severity of the condition (or some
other threshold) can be used as a filter to determine what to send to
the CE.  I agree that packets that need processing by the CE do need to
get sent by the FE to the CE.  I wasn't viewing these as "events"
however.  If we have the concept of a "data" versus a "control" channel,
these packets would be considered for transmission over the data
channel, whereas events/alarms would be reported over the control
channel (I may have misunderstood what was being stated).
-	Thresholds being used as triggers for events/alarms are
definitely a good thing to support in my opinion.  In large systems,
where the number of objects being managed and monitored is significant,
thresholds provide a mechanism to support manageability of such system
without overwhelming the CE and running out of space on the event/alarm
logs that are maintained in persistent storage.
-	I think event subscription/registration should only be possible
on events explicitly defined by an LFB.  It seems it would be rather
complex to express the "situations" under which to use different
verbosity levels for the different event types.  Are there specific use
cases that require this flexibility?  Else, it would seem reasonable to
limit this to registration time.  I assume multiple entities on the CE
can register for the same event from an FE.  What if one registers with
a high verbosity level, and one with a low verbosity level?  Would the
FE now have to send 2 separate events with the different levels of
verbosity, or is it the intent that the FE always send all the info.,
and an "event subsystem" on the CE will do the filtering for the
different registrations?  If so, have we gained that much with this
support for different verbosity levels?  Is this case valid?

Regards,
Shuchi

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Jamal Hadi Salim
Sent: Tuesday, April 19, 2005 3:53 AM
To: [email-masked]
Subject: Re: An approach to modeling ForCES events

On Mon, 2005-18-04 at 21:42 -0700, Deleganes, Ellen M wrote:
> Deferring events seems more difficult for the FE than just turning
them
> on and off because in order to defer sending events to the CE, the FE
> would have to maintain state and possibly buffer the events. (I'm
> assuming that if the CE doesn't want notifications that the FE doesn't
> have to keep them around.)

There are two scenarios:
 
1) nothing needs to be buffered because the state is part of the
attribute you are already maintaining at the FE. This applies to
<change> as the trigger.
Events just dont get generated at the FE anymore in the period those
events are turned off for the target. 
As an example for a simple implementation approach: if the event of
interest was attribute X of LFB Y changing, then every time attribute X
is changed localy within an FE, the FE will send an event to the CE if
the send-event-flag-for-attribute-X is true. If this flag is turned off
no such event is generated (but the local state of attribute X is still
updated). So in this case, the first event generated would alert the CE
and turn itself off until the CE sets back the boolean to process again.
Note turning off events for attribute X does not equate to turning off
all events at the FE.

2) Buffering needed.
The "units of work" need to be transported to the CE - without the "
unit of work" the CE cant process the event. As an example of such
things would be packets that need to be transported to the CE for
further resolution.  Another example would be queries built as a result
of packets arriving at the FE (eg a IPSEC security association acquire
message whose resolution happens at the CE). Overall such units of work
need to be buffered.
 
>  What happens if the FE runs out of space to buffer events? 

So you are talking about case #2. I would say simple FIFO schemes would
do. Overrun of the FIFO should be flagged.

> Seems complex to deal with in the first place and the
> error cases are even worse. I would hope events don't come in "huge
> bursts" but are relatively rare and we don't need to build into the
> model or the protocol the extra complexity to defer events.
> 

What do you mean by error cases? Lost messages? There are several ways
to deal with that issue.

To digress a little:
I think the threshold triggers we are trying to build in are equally as
complex as this issue and serve the same purpose. 
The only real value the thresholds provide is offloading the CE by
providing ability to deffer the CE work scheduling point (which is
exactly what the edge/level triggering does). 
In other words, the thresholds dont reduce the work that the FE has to
do. It still has to constantly poll to check the stats in place of the
CE doing that work. Note: i am not arguing against the threshold
triggering - just pointing out the similarities because you brought up
the point of complexity (I still think its a good necessary evil).

> If there is a case where one cause generates a burst of events, it may
> be reasonable to provide a mechanism to allow the CE to request only
the
> first event in the burst. Do you happen to have a scenario where a
large
> burst of events would occur? The only realistic scenario I can think
of
> is when the CE is controlling a large number of FEs and they all want
to
> send events at once. But this isn't the scenario outlined.

The general class of problems is: more events coming into the CE than
the CE can handle within reasonable time. Essentially, like the
threshold triggering, it is an optimization. More importantly, its a
classical event-processing scaling mechanism. Assuming in the worst case
your CE cant keep up with the rate at which events are generated from
the FEs, then the CE will end up thrashing. The ability to selectively
defer events processing helps the CE not to thrash.

I hadnt thought of the example you provided above - but its a reasonable
one.
The examples i can think of require processing of something in the
CE to update state at the FE[1]. These could be "stateful" classifiers
at the FE which require that the CE make the state update decision or
IDSes etc. The specific problem i am faced with at the moment is IPSEC
ACQuires being issued at the FE to the CE (assuming IKE runs at the
CE). 

cheers,
jamal

[1] We have spent many electrons in the past arguing about this very
issue. i.e whether the full processing of such things should happen at
the FE instead of being sent to the CE. I dont wanna go there with this
thread - Assume this is a legit way as well and let the benchmarks
decide which of the two approaches is better.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 19 Apr 2005 16:56:51 -0400

You have some good points here.
Two things to keep in mind:
1) We are focused more on integrating into the IETF/ SNMP management world 
view, while paying attention to the useful input from ITU.
2) This particular problem is not about the reporting of events and alarms 
to external entities.  it is not even about reporting SNMP traps or 
notifications.  Rather, it is about how the FE will give the CE enough 
information that it can generate appropriate indications (variables, traps, 
notifications, alarms, etc) to the external OA&M environment.

Yours,
Joel M. Halpern

At 04:35 PM 4/19/2005, Chawla, Shuchi wrote:

>I haven’t been following the entire thread on event modeling in detail, 
>but here are some of my thoughts – some of them are on the general aspects 
>of event and alarm reporting.  So, it is a slight digression from the 
>conversation, but hopefully, still relevant.
>
>specify the alarm and event reporting functions/requirements.  It’s been a 
>long time since I’ve looked at these specs. and I don’t currently have 
>ITU-T X.733 and ITU-T X.734 (I assume these were the ones I had looked at 
>in the past, but I don’t remember for sure!).
>
>threshold, it may lead to an alarm condition.  Alarms may be generated and 
>cleared.  Events do not have a concept of a “clear”.  An attribute change 
>on some object may be classified as an event.  A Loss of Signal on a 
>physical interface would classify as an alarm.  An alarm should be 
>integration time), and then a “clear” generated when the condition clears.
>
>-       From what I remember, these specs. classify alarms into different 
>classes (equipment, QoS etc.).  Each alarm has a default severity (which 
>is not modifiable) and perceived severity (which is modifiable).  The 
>support for perceived severity arises from the fact that what may be 
>considered a “major” condition on some network element in a certain 
>network, may be considered “minor” in some other network element based on 
>its function and topology in the network etc. Alarms define probable cause 
>and proposed repair information (among other information, which I don’t 
>remember).
>
>events/alarms get generated since a condition on object X could lead to a 
>different condition on objects associated with X, and so on.  If the FE is 
>not doing event correlation and root cause analysis, it might land up 
>sending all of these events to the CE, and it can result in event 
>the events from the FE, maybe the severity of the condition (or some other 
>threshold) can be used as a filter to determine what to send to the CE.  I 
>agree that packets that need processing by the CE do need to get sent by 
>the FE to the CE.  I wasn’t viewing these as “events” however.  If we have 
>the concept of a “data” versus a “control” channel, these packets would be 
>considered for transmission over the data channel, whereas events/alarms 
>would be reported over the control channel (I may have misunderstood what 
>was being stated).
>
>-       Thresholds being used as triggers for events/alarms are definitely 
>a good thing to support in my opinion.  In large systems, where the number 
>of objects being managed and monitored is significant, thresholds provide 
>a mechanism to support manageability of such system without overwhelming 
>the CE and running out of space on the event/alarm logs that are 
>maintained in persistent storage.
>
>-       I think event subscription/registration should only be possible on 
>events explicitly defined by an LFB.  It seems it would be rather complex 
>to express the “situations” under which to use different verbosity levels 
>for the different event types.  Are there specific use cases that require 
>this flexibility?  Else, it would seem reasonable to limit this to 
>registration time.  I assume multiple entities on the CE can register for 
>the same event from an FE.  What if one registers with a high verbosity 
>level, and one with a low verbosity level?  Would the FE now have to send 
>2 separate events with the different levels of verbosity, or is it the 
>intent that the FE always send all the info., and an “event subsystem” on 
>the CE will do the filtering for the different registrations?  If so, have 
>we gained that much with this support for different verbosity levels?  Is 
>this case valid?
>
>Regards,
>
>Shuchi
>
>[<mailto:[email-masked]>mailto:[email-masked]] 
>On Behalf Of Jamal Hadi Salim
> > Deferring events seems more difficult for the FE than just turning them
>
> > be reasonable to provide a mechanism to allow the CE to request only the
>
> > first event in the burst. Do you happen to have a scenario where a large
>
> > burst of events would occur? The only realistic scenario I can think of
>
> > is when the CE is controlling a large number of FEs and they all want to
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 20 Apr 2005 08:36:52 -0400

Shuchi,

On Tue, 2005-19-04 at 13:35 -0700, Chawla, Shuchi wrote:

[..]

> events/alarms get generated since a condition on object X could lead
> to a different condition on objects associated with X, and so on.  If
> the FE is not doing event correlation and root cause analysis, it
> might land up sending all of these events to the CE, and it can result
> in event flooding.  As an example, if an FE is associated with a large
> number of channelized interfaces, each configured to provide a
> service, if the physical interface experiences a loss of signal, all
> channelized interfaces will also have an alarm, as will the services
> associated with the channelized interfaces.  Depending on the alarm
> detection mechanisms and alarm integration times, it is possible that
> the root cause alarm is detected after the alarm conditions that
> result from it.
> 

The alarm management is an NE level/layer detail (derived from the
happenings in the FE(s) and maybe in the CE no doubt).
You seem to agree on this after looking at your last response; 
[As an example - I would expect no changes whatsoever in the way system
root cause analysis or fault management schemes would work just because
of how we do events between CE-FE. To the Forces level CE, the link
going down is just another event and to the alarm management relating
one or more NE that may end up being a high priority fault that need to
be repaired; forces will send any "repair" messages without needing to
understand the high level details etc etc]. 

You do bring an interesting example though with the channelized
intefaces in relation to the hierachies of possible events.
If the CE is told that the physical interface went down; isnt that
sufficient as a detail to mean all the logical interfaces also went
down? i.e would this still mean that the FE sends events for all
channelized interfaces as well? Refer to my views on earlier email on
desire to have hierachical semantics to avoid ambiguities (but now based
on your example also to avoid unnecessary events).
Likewise if a table is flushed it should be sufficient to indicate that
all entries have been deleted without sending events for every row.

> should.  

You may be refering to what i said: I used the word "defered" but didnt
intend for it to mean defering events rather it is to defer processing
of events.
What i said is: If you register to receive events you typically do so 
in order to process them - otherwise you dont register for them; 
i.e, when you receive notification of an event at the CE, you create
work to process it. In other words, such a notification is a
work-scheduling signal (this would not be necessarily true if you just
received an event you had no interest in). Typically, the work is
scheduled to run immediately. If you allow edge vs level triggering of
events then you could use that scheduling signal to _defer_ the work
(and not the event).
So the desired support is for the two kinds of triggers.
In one scheme the event trigger stays disabled (after sending a event to
the CE) until re-enabled explicitly by the CE and in the second case
it is always enabled.
This is very easy to add to the definition of the event; brute force: it
becomes an XML-level-attribute at definition time although my preference
would be it gets set at runtime and therefore is a boolean within the
definition of the event. A registration to the event could be used to
reenable it every time.
Again like i said earlier - I know we will do this in our implementation
(based on some of experiences in implementation thus far); it would be
nice if Forces supported it, however if consensus is not to then we will
keep it proprietary. I do hope consensus is derived from people
understanding what i am proposing though.
  
> If there is an issue with the CE not being able to keep up with the
> events from the FE, maybe the severity of the condition (or some other
> threshold) can be used as a filter to determine what to send to the
> CE.

All these are attempts to help offload the CE and reduce abuse of
resources (bandwidth and CPU in this case). Same with event-trigger type
being edge vs level.

>   I agree that packets that need processing by the CE do need to get
> sent by the FE to the CE.  I wasn’t viewing these as “events” however.
> If we have the concept of a “data” versus a “control” channel, these
> packets would be considered for transmission over the data channel,
> whereas events/alarms would be reported over the control channel (I
> may have misunderstood what was being stated).
> 

But these are not intentionaly redirected packets (such as OSPF for
example).
They are "units of work" for some event that needs to be processed at
the CE. Example are ones to be exception handled in particular; i am not
so sure these belong to the data channel. 
The other " units of work" are packets built as a result of other
packets arriving at the FE(sic);->. I gave the example of an IPSEC
ACQuire being sent to the CE as a result of a missing security
association in the FE.

> where the number of objects being managed and monitored is
> significant, thresholds provide a mechanism to support manageability
> of such system without overwhelming the CE and running out of space on
> the event/alarm logs that are maintained in persistent storage.
> 

I agree - an optimization to offload the CE. Nota Bene the same logic
reasoning is used for the event-trigger-type ;->

> -       I think event subscription/registration should only be
> possible on events explicitly defined by an LFB.

This is agreed on.

>   It seems it would be rather complex to express the “situations”
> under which to use different verbosity levels for the different event
> types.  Are there specific use cases that require this flexibility? 

Optimization is a good enough reason in my opinion.

>  Else, it would seem reasonable to limit this to registration time.

Verbosity - I dont agree that should be at registration time; it is
trivial to add to both the model and protocol that it is done at 
run time. 

>  I assume multiple entities on the CE can register for the same event
> from an FE.  What if one registers with a high verbosity level, and
> one with a low verbosity level?  Would the FE now have to send 2
> separate events with the different levels of verbosity, or is it the
> intent that the FE always send all the info., and an “event subsystem”
> on the CE will do the filtering for the different registrations?  

Good point to bring up: Upto this point ive thought of the CE
as only registering once per event. 
On the systems i have used this scheme on, the highest verbosity level
is sent in such a case and it would be up to the CE to discriminate.
i.e any intelligence will have to be embedded at the CE.

> If so, have we gained that much with this support for different
> verbosity levels?  Is this case valid?

I think the case is valid - whether we have gained much will depend on
the implementation. Clearly it is a win over the case where you send two
events and they are all verbose for two different registrations
(I assume this to be the case if you didnt have a level differentiator).

Ok, that was long ;-> Blame it on coffee.
I didnt proof read it so i may be repeating myself in a few places.
Dont disappear now.

cheers,
jamal

---

From: Tom Petch <[email-masked]>
Date: Wed, 20 Apr 2005 14:31:50 +0200

As an aside, X.733, X.736 and M.3100 were used by the Disman WG in the creation
of the Alarm MIB, RFC 3877, and the ITU events and alarms are listed there.  As
Joel points out, the internal alarms between FE and CE are likely to be
different in kind but the Alarm MIB provides a useful checklist of what the
ITU-T were thinking in 1992.  If and when there is a list of alarms for this
work, then I see a case for reusing the same technical terms for the same
alarms.  The name/number mappings are registered with IANA (as an SNMP Textual
Convention).

Tom Petch

----- Original Message -----
From: "Chawla, Shuchi" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, April 19, 2005 10:35 PM
Subject: Re: An approach to modeling ForCES events

Regards,
Shuchi

<snip>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 21 Apr 2005 08:29:48 -0400

> definition of the event. A registration to the event could be used to

After chewing on this some more:
I think perhaps using terms like edge vs level triggering may mud what i
am trying to say. So perhaps a more descriptive term is event-mode
(because event-type is taken).
This is analogous to what is used in timers. When you register for
a timer you typically could select a one-shot or continuous mode timer.
In one-shot mode, the timer runs once and you have to reenable it if you
need it again.
In continous mode, it runs periodically, forever unless you explicitly
stop it.

To apply this to event registration - when you register for events, you
set a flag to select one-shot mode. Otherwise continous mode is assumed.
Does this make more sense?

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 21 Apr 2005 09:42:53 -0400

That (one shot vs continuous) makes somewhat more sense.

I am having trouble building a mental model of when a CE would need 
one-shot notifications.  Essentially, this would reflect some situation 
where the CE needed to know the first time something happened, but not 
later times?

I can see how we would support this.  I am just trying to figure out if it 
is a complication worth having.  Can you give some examples of one-shot 
event registrations (for events that themselves happen more than once.  The 
registration mode doesn't matter if the event only occurs once.)

Yours,
Joel

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 28 Apr 2005 08:46:08 -0400

Apologies for taking this long to respond. Too tied up.

> where the CE needed to know the first time something happened, but not 

Well, two things:
- A one shot timer (but thats not why i proposed this). I think timers
do fit in the events semantics. 
- The real reason I was pushing for this is the classical case where it
is used - optimization. In cases where you/CE are overloaded you may
wanna switch to a one-shot event mode for certain events in order to
reduce the load on the CE. The one-shot event tells the CE of work that
needs to be done, but lets the CE decide when to do work.
Some of these schemes go to extremes from say migrating from
asynchronous event notification to pure polling and back
when things get better. The application of these sorts of event-modes
ranges from use in servers and associated events to device drivers where
interupts are the events. In other words there is " no invention" here.

> registration mode doesn't matter if the event only occurs once.)

Refer to above.
I dont think it imposes any complications or introduces breakages. When
you register for an event, you specify mode=continous/oneshot.

cheers,
jamal