Subject: Re: [OPSAWG] [OPS-AREA] suggested updates to draft-asai-vmm-mib-03.txt

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 21 May 2013 10:57:36 +0200

On Tue, May 07, 2013 at 11:27:05AM -0700, Michael MacFaden wrote:
> 
> Juergen wrote:
> >can you please send this message to the OPSAWG mailing list? We need
> >to move discussions over to the list.
> 
> Attached please find my suggested patch to the draft-asai-vmm-mib-03.txt
> 
> In short:
> 
> 1. Introduce a new TC called VirtualMachineList. It
>    is a clone of Q-BRIDGE-MIB PortList
> 
> 2. Introduce a new managed object, vmBulkNotificationsEnabled, to report which type 
>    of events to receive, either bulk or existing per vm notifications.
> 
> 3. Adds a parallel set of notifications to match: vmRuning, vmShutdown, vmPaused,
> vmSuspended, vmCrashed, vmDeleted that only send a new accessible-for-notify 
> variable vmAffectedVMs using SYNTAX VirtualMachineList.

Thanks Michael. I am wondering whether we should really have two sets
of notifications.

I know that we discussed this before but lets me ask again: How large
is the number of VMs per hypervisor going to be we have to plan for? I
am asking so that I can get an idea what the length of vmAffectedVMs
might be - I guess we are fine but I just thought I ask. And of
course, with your notification design, it makes a lot of sense to
assign small VirtualMachineIndex values (right now, I can choose
2147483647 as vmIndex but that would be a really bad choice for your
new notifications).

The old notifications carry more information. Hence, if we would move
to the new notifications, a management application may send subsequent
read requests to obtain that information. But perhaps this is OK since
this is under the control of the management application (while a
notification blast is not).

> Lastly,  I've added some comments (grep mrm) about what clients
> should expect with vmName, vmUUID, vmIndex based on my prior
> deployment experience with the similarly structured VMWARE-VMINFO-MIB.

These make sense to me.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Michael MacFaden <[email-masked]>
Date: Tue, 21 May 2013 15:24:00 -0700 (PDT)

> Thanks Michael. I am wondering whether we should really have two sets

Indeed. Let's discuss how the application would be expected to use
either notification.
 
> is the number of VMs per hypervisor going to be we have to plan for? 

1000 virtual machines per hypervisor, 100 hypervisors 
per mgmt controller system is the present design points is where we are 
today.

> might be - I guess we are fine but I just thought I ask.

Sure. 

> And of course, with your notification design, it makes a lot of sense to

Yes, and starting a simple index from 1 remains a strong SNMP
convention since RFC 1213 through RFC 2863 for ifIndex.

> read requests to obtain that information.

The value of the additional information for a mgmt system that uses
notifications for trap-directed polling is redundant. 

The procedure I had in mind for the mgmt app would be as follows:
1. poll the entire vm table say every X minutes.
2. any traps received would trigger an ad-hoc poll to update
   only those indicies using get or get bulk.
3. lather, rinse and repeat

Now if a hypervisor boots with 1000 vms, then you've got 1000
notifications instead of one and the initial state isn't going
to be very useful, mgmt system knows the hypervisor just booted
if it was either polling it sysUptime and/or got a coldStart notification

> But perhaps this is OK since

Right, if the design for the mgmt app is something like:
1. do an initial poll the agent vm table and collect all state 
2. from then on mgmt uses the notifications to maintain state
3. reload agent table based on a consistency check or agent cold|warmStart/sysUptime rollover
   A consistency check is typically done by: 
     + comparing a sequence number varbind in the event to the last one received
     + comparing a scalar lastChange varbind to a regular get of lastChange.0 on the vm table

As you can see from my patch, by including the managed object vmBulkNotificationsEnabled, I
believe both models merit inclusion in the standard.

Cheers,
Mike MacFaden

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 22 May 2013 09:20:22 +0200

> convention since RFC 1213 through RFC 2863 for ifIndex.

We better be explicit if we adopt your notifications. I have seen
boxes with rather large ifIndex values that seemed to resemble memory
locations of device drivers.

> As you can see from my patch, by including the managed object
> vmBulkNotificationsEnabled, I believe both models merit inclusion in
> the standard.

Yes, but whether an application is fine with bulk notifications or
prefers separate notifications is actually a function of the
application behind the notification receiver. In other words, I can
see situations where some notification receivers prefer bulk
notifications while others prefer the other type of notifications.  In
principle, SNMPv3 has a mechanism to deal with this, namely the
notification filtering mechanism described in section 6 of RFC 3413.

So perhaps the way to go is to have two 'enable' objects that enable
the generation of the two classes of notifications (so you can enable
both, only one of them, or none) and to add explicit text how RFC 3413
notification filtering can be used to control which notification
receiver receives certain notifications.

Makes sense?

/js

---

From: Michael MacFaden <[email-masked]>
Date: Wed, 22 May 2013 14:46:52 -0700 (PDT)

Juergen writes:
> notifications while others prefer the other type of notifications. 

At scale per-vm notifications require more resources to avoid
classic producer-consumer issues. Having both will be unlikely to exist 
for long in a given operator's IT system as it grows however 
I do concede that it will happen.

An CSP (cloud service provider) will not use both because:
a) at scale if they did enable both they'd quickly discover that notifications (from this mib module and
others) were being dropped as multiple hypervisors would pummel their notification receivers
with per-vm notifications and so would quickly shut off/filter out per-vm notifications. 
b) CSP would not deploy two applications that do the same thing

The case that where both might exist will happen in the enterprise account or
maybe a small CSP where the shop scale ranges from say tens of hypervisors to hundreds of VMs
(eg not Verizon).  Again CSP don't deploy multiple monitoring solutions for the same thing but
I have seen multiple solutions deployed in the enterprise IT systems, typically
where monitoring is less, shall we say, a core competency.

> In principle, SNMPv3 has a mechanism to deal with this, namely the

This is true in practice as well. Most agents have a way to allow operators
to configure their agents to filter out particular notifications they don't want.

In VMware ESXi 5.1 its (esxcli system snmp set -n <oid-of-trap>,...)
 
> receiver receives certain notifications.

Yes it does. 

And I'd leave the defaults as to which is enabled to the vendors particular 
implementations as anyone who remembers the pain of some product thinking
it knew best about what to enable by default, cisco ios enabling proxy-arp by default
comes to mind.

Cheers,
Mike MacFaden