Subject: [OPSAWG] Yet another hypervisor MIB

From: Hirochika Asai <[email-masked]>
Date: Wed, 25 Apr 2012 03:44:02 +0900

Hi all,

As I previously posted, we have implemented and been testing a prototype code
of yet another hypervisor MIB.  Now you can browse the current version of code
in python and MIB here: <http://hg.scyphus.co.jp/virtsnmp/>.

Since we implemented this before <draft-schoenw-opsawg-vm-mib-00> was
submitted, our MIB and implementation don't follow it.
At the moment, we don't care on write operations and trap notifications.
Moreover, we haven't implemented the storage part although we define
some objects for virtual disks in our MIB file.  This is because storage
is more complex than other resources, i.e., CPU, memory, and network
interfaces.  For example, current resource allocation (e.g., actual size
of sparse virtual disks) and I/O statistics rather than the static definition of
the storage size are more important from the viewpoint of operations and
management of hypervisor storage.  Thus, deep discussions are required
for storage MIB, so we haven't implemented the storage part yet.

After the implementation of the prototype code, I'd like to discuss
the "direction" of interface statistics, which should be defined, here.
This is a common issue to <draft-schoenw-opsawg-vm-mib-00>.

Looking from hypervisors, TX and RX mean packets to a VM and those from a VM,
respectively.  On the other hand, looking from VMs, TX and RX mean packets
from a VM (to HV) and those to a VM (from HV).  Libvirt API follows the
former manner, but our implementation follows the later.
The former is the same way of switches, but the vmIfTable/vmIfEntry is
defined under vmEntry (perspective from VMs), not directly under the
hypervisor MIB root.  Consequently, we have chosen the later.
I agree that some of you think the former is more natural.  So let me bring
it up for discussion in this community.

And, one minor question to WG chairs: Is it better to submit a document
summarizing MIB with our target environment and requirements in detail as an I-D?
Or better to avoid the conflict with draft-schoenw-opsawg-vm-mib (and keep discussion in ML)?

Thanks,
Hirochika

-- 
Hirochika Asai <[email-masked]>, The University of Tokyo

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 24 Apr 2012 20:53:54 +0200

On Wed, Apr 25, 2012 at 03:44:02AM +0900, Hirochika Asai wrote:
 
> As I previously posted, we have implemented and been testing a
> prototype code of yet another hypervisor MIB.  Now you can browse
> the current version of code in python and MIB here:
> <http://hg.scyphus.co.jp/virtsnmp/>.

I will look at this in more detail tomorrow but it is clear already
that this is not a valid MIB module. SMIv2 does not allow SEQUENCES
inside of SEQUENCES. Anyway, I will try to figure out what the
modeling differences are.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Michael MacFaden <[email-masked]>
Date: Tue, 24 Apr 2012 14:18:35 -0700 (PDT)

> modeling differences are.

Re: http://hg.scyphus.co.jp/virtsnmp

The vm nic state is represented by copying selected objects
from ifTable/ifXTable. 

Designing how to model the plumbing of a vm's nic
needs to support quite a few configurations.

a) hypervisors that plumb a virtual machines set of nics directly to one or
more virtual layer 2 switches : (IEEE8021-BRIDGE-MIB / index
ieee8021BridgeBasePortComponenId and IEEE8021BridgePortNumber)
for the up/down state, source vm's nic mac address, and per vm/per port/per q-vlan 
in/out/error byte/packet stats.

b) virtualization software that plumbs to pseudo interfaces in host OS
visible in ifTable/ifStackTable. (seen with ifconfig -a)

c) virtualization software that shunts nics directly to the host os nic 
(no pseudo device).

Others?

Mike MacFaden

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 25 Apr 2012 09:10:57 +0200

> The vm nic state is represented by copying selected objects

Copying MIB objects is generally not the way we like to do things.
Interface statistics are in the ifTable - so all that is needed is to
identify the interfaces associated with virtual machine.
 
> c) virtualization software that shunts nics directly to the host os nic 

I think c) is pretty much out of scope for a standards solution. The
architectural model used in the IETF is that IP packets flow through
interfaces. For bridges, we have ports but there is a common mapping
of bridge ports to interfaces. Implementations that do pass packets
magically to a virtual machine might exist but I doubt it is something
worth spending extra efforts on. (I think such implementations should
better be encouraged to expose their mechanism as an interface.)

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 25 Apr 2012 11:32:47 +0200

> modeling differences are.

Ignoring a number of technical issues, I believe the major differences
are that

a) your MIB duplicate objects already found in other MIBs (SNMPv2-MIB,
   IF-MIB) which I tried to avoid

b) your MIB exports information about each virtual CPU assigned to a
   VM - I have no clear opinion whether that is valuable or not, in
   particular since CPU states may change pretty quickly (it would
   probably then make more sense how much type a CPU has spent in the
   various states)

c) your MIB explicitely models a notion of disks as storage units
   which is probably a good thing

Do you agree? Is this is fair summary?

/js

---

From: Hirochika Asai <[email-masked]>
Date: Wed, 25 Apr 2012 20:04:02 +0900

> identify the interfaces associated with virtual machine.

Thanks, I understood.  Just copying isn't a good nor smart idea.  As you
suggested, vmEntry can have pointers to ifTable or ifEntry objects
instead of holding vmIfTable in vmEntry.

> better be encouraged to expose their mechanism as an interface.)

Our current implementation assumes b).  Both Xen and KVM create pseudo
NICs (e.g., vif/tap/vnet) on a hypervisor and each VM attaches one or
more of them when using bridge mode.

I agree to js's opinion.
Moreover, as far as I know, a) is the case of VMware; which plumbs
vSwitches and each VM's NIC attaches a vSwitch, although I don't know
the internal implementation of VMware's virtual NICs.  c) is cases such
as using MacVTap and directly attaching a physical NIC.  However, in
both cases, hypervisors must internally have virtual (pseudo) NICs or
equivalent mechanisms to forward packets from/to VMs.  Thus, I also
think we don't have to care the internal mechanisms, and SNMP agents can
generalize and provide virtual NICs' information in the same scheme.

Only one exception is the case that a hypervisor bypasses "PCI" (i.e.,
physical NIC) to a VM using IOMMU.  In this case, hypervisors cannot
recognize any packets passing through the NIC.

regards,
Hirochika

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 25 Apr 2012 15:02:46 +0200

> recognize any packets passing through the NIC.

Yes, but in this case the hypervisor can't be expected to report about
traffic it never sees. Hence in this case, a management application
has to read the statistics directly from the virtual machine that is
directly controlling the physical NIC.

/js

---

From: Hirochika Asai <[email-masked]>
Date: Thu, 26 Apr 2012 11:56:54 +0900

> Do you agree? Is this is fair summary?
Yes, and thank you for your review.

I have comments about b).
As for virtual CPUs, I agree that the state of each vCPUs is not much
valuable for SNMP because CPU state frequently changes.  I defined this
because it may be valuable for local operation tools like the "top"
command in case that this MIB object is locally used through a lighter
interface, not SNMP, but I also think it's not much valuable.
I think CPU time of each virtual CPU is important to monitor its CPU
utilization.  In our cloud frontend system, we monitor the virtual CPU
time to provide CPU utilization graph like <http://jar.jp/tmp/cc-snapshot.png>.

---

From: Hirochika Asai <[email-masked]>
Date: Thu, 26 Apr 2012 12:00:39 +0900

> directly controlling the physical NIC.

I want to say the same.  Sorry that my expression wasn't good.
The management domain of NICs in this case is completely separated from hypervisors,
and consequently, we don't have to care about it.  It's VM's operators' job.