Subject: [CCAMP] Opsdir last call review of draft-ietf-ccamp-alarm-module-07

From: Joe Clarke via Datatracker <[email-masked]>
Date: Mon, 18 Mar 2019 10:00:40 -0700

Reviewer: Joe Clarke
Review result: Has Nits

I have been assigned to review this draft on behalf of the OPS DIR.  Overall, I
found the draft ready with some nits.  I found relatively easy to read and
well-thought-through.  Broadly speaking, I found the term "Alarm Shelving" to
be confusing.  You do define this, but it is not something that many operators
are familiar with.  I had originally thought alarm suppression was better, but
after I read more, I see what you're trying to do with this.

Additionally, some of the features like alarm-history and alarm-shelving have a
potential operational impact with respect to local server resources, and
perhaps that's worth calling out.

The leaf "has-clear" seems like it would read better as "has-cleared" from an
English standpoint.  Thoughts?

Finally, I can see where it might be desirable to suppress alarm notifications
without shelving them per se.  That is, the alarm remains in the list, but
notifications are not sent.  That doesn't seem possible, and I'm wondering if
it's worth considering.

My per-section nits (typos) are found below.

Section 3.2

s/an hierarchy/a hierarchy/

===

Section 3.5.1

In the example, ((GigabitEthernet0/25, link-
   alarm,""), false, T, major, "Interface GigabitEthernet0/25 down"), maybe
   replace 'T' with a sample timestamp.  It took me a few seconds to grok that
   as I was associating it with a boolean.

===

Section 3.5.2

s/criteria/criterion/

===

Section 3.6

s/the the disk full/the disk full/

===

Section 3.7

s/this criteria/these criteria/

===

Section 4.7

s/alarn/alarm/

===

YANG Module section

s/identifiying/identifying/

s/exampla/example/

s/alarm is not longer active using other/alarm is no longer active using other/

---

From: "Adrian Farrel" <[email-masked]>
Date: Mon, 18 Mar 2019 17:12:53 -0000

Hi,

I see where Joe is coming from with the term 'Alarm Shelving', but I think it is a term of art in the alarm management world (entirely random example provided by duckduckgo... https://www.exida.com/Blog/Alarm-Shelving-Relieve-the-Symptoms-of-Nuisance-Alarms).

Actually, I found the term to be quite evocative of the process.

His point about the impact on server resources is worth exploring.

And finally, yes, it seems to me that shelving and suppression of notifications seem to be approaching orthogonal. That is, suppression of notifications is not the same as suppression of alarms.

> Broadly speaking, I found the term "Alarm Shelving" to

Cheers,
Adrian
--
Want to buy a signed copy of a book of fairy stories for adults of all ages?
Send me an email and I'll bring one to Prague for you.
"Tales from the Wood"
"More Tales from the Wood"
"Tales from Beyond the Wood"
https://www.feedaread.com/profiles/8604/

---

From: Joe Clarke <[email-masked]>
Date: Tue, 19 Mar 2019 06:55:02 -0400

> Actually, I found the term to be quite evocative of the process.

That's why I backed out my original comment.  As I read, it made more
and more sense.  I was appreciative of the glossary to set the context,
though.

> And finally, yes, it seems to me that shelving and suppression of notifications seem to be approaching orthogonal. That is, suppression of notifications is not the same as suppression of alarms.

I had originally thought to propose this as a name for shelving, but
they are different as you say.  I see value in the suppression-only case
for an operational standpoint.

Joe

---

From: stefan vallin <[email-masked]>
Date: Tue, 19 Mar 2019 15:07:14 +0100

Hi Joe!
Thanks for your review and comments!
See inline

br Stefan

> after I read more, I see what you're trying to do with this.
Good :)
> perhaps that's worth calling out.
Good comment: suggested edits 
(could of course go somewhere else than in the feature desc)

     feature alarm-history {
       description
         "This feature indicates that server maintains a history of
          state changes for each alarm.  For example, if an alarm
          toggles between cleared and active 10 times, these state
          changes are present in a separate list in the alarm.
ADDED:
          The alarm-history feature has an impact on server memory 
          resources.         
          ";
     }

     feature alarm-shelving {
       description
         "This feature indicates that the system supports shelving
          (blocking) alarms.
ADDED:
          Alarm shelving has an impact on server processing resources
          in order to match alarms against shelf criterion";
     }

> English standpoint.  Thoughts?
Note well that this leaf is for the alarm *inventory*, not the alarm itself.
For the alarm itself the leaf is called "is-cleared".
The alarm inventory lists all candidate alarms and it is important for
operations to know if the server/instrumentation is capable of detecting
and generating alarm clear. So "has-clear" states that a specific alarm-type
will be cleared by the instrumentation when the alarm condition disappears.
I guess "has-clear" works in this context?
> notifications are not sent.  That doesn't seem possible, and I'm wondering if
It is possible using the /alarms/control/notify-status-changes choice:
 leaf notify-all-state-changes {
             type empty;
             description
               "Send notifications for all status changes.";
           }
           leaf notify-raise-and-clear {
             type empty;
             description
               "Send notifications only for raise, clear, and re-raise.
                Notifications for severity level changes or alarm text
                changes are not sent.";
           }
           leaf notify-severity-level {
             type severity;
             description
               "Only send notifications for alarm state changes crossing
                the specified level.  Always send clear notifications.";
           }

You could consider turning all of by adding another leaf:
leaf notify-none {
   type empty;
   description
     "Don't send any notifications.";
 }
Not sure if that would make sense, recommend to leave that out for further study.
Note well that you can always stop to subscribe to the notification stream.

> My per-section nits (typos) are found below.
Thanks!
Will correct

---

From: stefan vallin <[email-masked]>
Date: Tue, 19 Mar 2019 15:10:05 +0100

Adrian and Joe
Thanks for your discussion :)
I think we have come to agreement on the “shelving” term?
See my previous response on the notification suppression topic
br Stefan

> On 19 Mar 2019, at 11:55, Joe Clarke <[email-masked]> wrote:
>

---

From: Joe Clarke <[email-masked]>
Date: Tue, 19 Mar 2019 10:16:17 -0400

>           in order to match alarms against shelf criterion";

This is a good start, but people will miss this buried in the YANG
module.  I think it deserves some text in the document itself.  With
respect to resources, I recommend memory get called out especially
since, as I recall, you want all changes to be recorded in the history.

> I guess "has-clear" works in this context?

Got it.  I do see the is-cleared, and I got a bit confused here.  After
re-reading the has-clear leaf definition, I think will-clear makes more
sense for this node.

> Note well that you can always stop to subscribe to the notification stream.

What if I want to control notifications on a per-alarm basis?  Like
shelving filtering, but keeping the alarms on the list instead of a shelf.

Joe

---

From: stefan vallin <[email-masked]>
Date: Tue, 19 Mar 2019 16:16:50 +0100

Hi!

> On 19 Mar 2019, at 15:16, Joe Clarke <[email-masked]> wrote:
> 
Ok, I called out memory for alarm-history in my suggested edit above.
For alarm shelving it will mostly be processing resources and not memory 
since the shelved  alarms would anyhow stay in the larm list if not shelved.
I will suggest a text snippet in the document as well. 
> re-reading the has-clear leaf definition, I think will-clear makes more
Ok, we will consider change to "will-clear"
> shelving filtering, but keeping the alarms on the list instead of a shelf.
We have excluded that by design. If we introduce too flexible notification filtering and at the same
time support alarm shelving it will be hard to understand what is going on at the end, with these
two mechanisms interfering. We had a fairly long thread in the CCAMP mailing group where we ended
up where we are right now.

/stefan

---

From: Joe Clarke <[email-masked]>
Date: Tue, 19 Mar 2019 11:23:00 -0400

> I will suggest a text snippet in the document as well. 

Sorry.  I thought I had seen memory, but I got a bit distracted.  Thanks
for drafting some supporting text.

> up where we are right now.

Okay.  The fact that shelving without history wouldn't be too much
different than per-alarm suppression us likely sufficient.  I was trying
to think through some of the operational use cases I have.  Do you
envision a non-shelved alarm to periodically generate notifications
while active, or is it one-and-done?

Joe

---

From: stefan vallin <[email-masked]>
Date: Tue, 19 Mar 2019 16:30:21 +0100

Hi!

> On 19 Mar 2019, at 16:23, Joe Clarke <[email-masked]> wrote:
> 
> Do you
Oh yes, see
3.5.1.  Resource Alarm Life-Cycle

   From a resource perspective, an alarm can for example have the
   following life-cycle: raise, change severity, change severity, clear,
   being raised again etc.  All of these status changes can have 
   different alarm texts generated by the instrumentation

That is the purpose of having the resource and alarm-type as key
As long as you do not purge (delete) the alarm it can change:
* severity
* alarm-text
* is-cleared

Each of the generating a notification.

If that was your question….
/s

---

From: Joe Clarke <[email-masked]>
Date: Tue, 19 Mar 2019 11:35:06 -0400

On 3/19/19 11:30, stefan vallin wrote:
> Hi!
> 
>> On 19 Mar 2019, at 16:23, Joe Clarke <[email-masked]
>> <mailto:[email-masked]>> wrote:
>>
>> Do you

Not exactly.  I saw the life-cycle stuff -- and this edge trigger makes
sense -- but I was asking that if the state is raise, will multiple
notifications be sent for raise (or could multiple, periodic
notifications be sent)?  For example, while status is "raise" and
without any other change, send a notification every five minutes.

And now that you call out this text, "change severity" is listed twice :-).

Joe

---

From: stefan vallin <[email-masked]>
Date: Tue, 19 Mar 2019 18:00:25 +0100

Hi!
> without any other change, send a notification every five minutes.
Ah, ok. 
Notifications are only sent for changes not repeated for same state
> And now that you call out this text, "change severity" is listed twice :-).
Yes, intended :)
Could be minor, major, minor, major for example. 
Imagine a set of thresholds

---

From: Joe Clarke <[email-masked]>
Date: Tue, 19 Mar 2019 13:08:15 -0400

> Notifications are only sent for changes not repeated for same state

Okay.  Takes care of that line of thinking.

In general, what about shelving for a time window?  Is it envisioned the
Management System would enable and disable shelving during maintenance
windows?

> Imagine a set of thresholds

Ah, it's a sample life-cycle, not a list.

Joe

---

From: stefan vallin <[email-masked]>
Date: Wed, 20 Mar 2019 08:50:31 +0100

Hi!

> On 19 Mar 2019, at 18:08, Joe Clarke <[email-masked]> wrote:
> 

From a user perspective this make sense but it should be controlled by the manager/client.

Shelving is configuration, and configuration that "changes itself” 
kind of breaks the NETCONF configuration pattern.  No one reconfiguring
the system and still the config has changed.

> Ah, it's a sample life-cycle, not a list.
Depends on what you mean :)

      +--ro alarm* [resource alarm-type-id alarm-type-qualifier]
      |  +--ro is-cleared               boolean
      |  +--ro perceived-severity       severity
      |  +--ro alarm-text               alarm-text
      |  +--ro status-change* [time] {alarm-history}?
      |  |  +--ro time                  yang:date-and-time
      |  |  +--ro perceived-severity    severity-with-clear
      |  |  +--ro alarm-text            alarm-text

Each alarm has a leaf indicating the last severity level, alarm text and clearance status
And each alarm has a list of status changes.

So an alarm composite state could be:

+ alarm [disc-partition-foo, disc-utilization, “”]
   + is-cleared false
   + perceived-severity major
   + alarm-text “disc utilization high, major threshold”
   + status-change
      + time T3
      + perceived-severity major
      + alarm-text “disc utilization high, major threshold”
      + time T2
      + perceived-severity minor
      + alarm-text “disc utilization high, minor threshold”
      + time T4
      + perceived-severity major
      + alarm-text “disc utilization high, major threshold”

Note well, if the alarm gets cleared it will appear in the status list as well,
and it might be raised again

+ alarm [disc-partition-foo, disc-utilization, “”]
   + is-cleared false <<< (was True at T4)
   + perceived-severity major
   + alarm-text “disc utilization high, major threshold”
   + status-change
      + time T5
      + perceived-severity major
      + alarm-text “disc utilization high, major threshold”
      + time T4
      + perceived-severity cleared
      + alarm-text “disc utilization ok”
      + time T3
      + perceived-severity major
      + alarm-text “disc utilization high, major threshold”
      + time T2
      + perceived-severity minor
      + alarm-text “disc utilization high, minor threshold”
      + time T4
      + perceived-severity major
      + alarm-text “disc utilization high, major threshold”

/s

---

From: Joe Clarke <[email-masked]>
Date: Wed, 20 Mar 2019 04:26:38 -0400

> the system and still the config has changed.

Fair enough.  I just wanted to talk through the use cases.

> Depends on what you mean :)

I meant this is an example of a possible life-cycle versus a list of
possible life-cycle states.  I do certainly understand the escalation
nature within a given life-cycle.

Thanks for your discussion.

Joe