Subject: [OPSAWG] Hypervisor MIB document

From: Melinda Shore <[email-masked]>
Date: Mon, 26 Mar 2012 05:45:41 -0800

A few weeks ago Juergen posted the -00 draft of a hypervisor MIB.
One concern about doing this work in the IETF is making sure we've
got input from people who actually build hypervisors, and that they're
interested in implementing a standard MIB.  So, at this point we're
interested in feedback from the mailing list - who's interested,
is there interest in the vendor community, is this something that
operators would see as a requirement, etc.?

Thanks,

Melinda

---

From: "So, Ning" <[email-masked]>
Date: Mon, 26 Mar 2012 09:51:21 -0400

Yes, I do see value in the area of work as provider/operator.

Best regards,
 
Ning So
Verizon Corporate Technology
(office) 972-729-7905
(Cell) 972-955-0914

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Melinda Shore
Sent: Monday, March 26, 2012 8:46 AM
To: [email-masked]
Subject: [OPSAWG] Hypervisor MIB document

A few weeks ago Juergen posted the -00 draft of a hypervisor MIB.
One concern about doing this work in the IETF is making sure we've got input from people who actually build hypervisors, and that they're interested in implementing a standard MIB.  So, at this point we're interested in feedback from the mailing list - who's interested, is there interest in the vendor community, is this something that operators would see as a requirement, etc.?

Thanks,

Melinda
_______________________________________________
OPSAWG mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/opsawg

---

From: Melinda Shore <[email-masked]>
Date: Mon, 26 Mar 2012 05:52:40 -0800

> Yes, I do see value in the area of work as provider/operator.

Is this something that you'd include as a requirement for a purchase?

Melinda

---

From: "So, Ning" <[email-masked]>
Date: Mon, 26 Mar 2012 10:01:27 -0400

The work in the area has just begun, so it is premature to list them as requirements of purchase immediately.  As you correctly pointed out, hypervisor and server vendor's interest of development is an important component in this work as well.  I would really like to see some active discussion and consensus on this topic.    

Best regards,
 
Ning So

-----Original Message-----
From: Melinda Shore [mailto:[email-masked]] 
Sent: Monday, March 26, 2012 8:53 AM
To: So, Ning
Cc: [email-masked]
Subject: Re: [OPSAWG] Hypervisor MIB document

Melinda

---

From: Benson Schliesser <[email-masked]>
Date: Mon, 26 Mar 2012 16:14:06 +0200

Hi, Ning.

As a point of clarification: Do you find value in a MIB specifically for SNMP, or rather in a common management model? E.g. would YANG / NetConf or some other management model be sufficient (or preferred)?

Cheers,
-Benson

---

From: Hirochika Asai <[email-masked]>
Date: Mon, 26 Mar 2012 23:20:26 +0900

Hi,

I'm interested in the area of hypervisor MIB.  Actually, we have also
worked on a hypervisor MIB document and implemented an SNMP agent
running with libvirt.  Unfortunately the documentation process is
delaying though a prototype implementation is running on our operating
hypervisors.  We will summarize a specification of our implementation
in a couple of weeks, and post it to this mailing list.

I'm sorry that I cannot go to this IETF, but if the WG members are also
interested in this area, we will summarize specifications of our
implementation in a couple of weeks, and post it to this mailing list.

Hirochika

-- 
Hirochika Asai <[email-masked]>, The University of Tokyo

---

From: Melinda Shore <[email-masked]>
Date: Mon, 26 Mar 2012 06:26:09 -0800

> in a couple of weeks, and post it to this mailing list.

This is excellent - exactly the sort of feedback we're looking for.
Have you had an opportunity to review 
http://tools.ietf.org/html/draft-schoenw-opsawg-vm-mib-00?

Thanks,

Melinda

---

From: "So, Ning" <[email-masked]>
Date: Mon, 26 Mar 2012 10:27:10 -0400

Benson,

Vast majority of our existing OSS systems and tools are SNMP based.  However, YANG/NetConf model is certainly very attractive and useful.  Unfortunately provider OSS systems are very difficult to change, so I see SNMP and YANG/NetConf will co-exist for quite a while.  There are practical value for both.  

Best regards,
 
Ning So

-----Original Message-----
From: Benson Schliesser [mailto:[email-masked]] 
Sent: Monday, March 26, 2012 9:14 AM
To: So, Ning
Cc: Melinda Shore; [email-masked]
Subject: Re: [OPSAWG] Hypervisor MIB document

Hi, Ning.

Cheers,
-Benson

---

From: Hirochika Asai <[email-masked]>
Date: Tue, 27 Mar 2012 00:04:30 +0900

> Have you had an opportunity to review http://tools.ietf.org/html/draft-schoenw-opsawg-vm-mib-00?
I had no opportunity before your post, but now I'm reading this document, and have found that some of the defined objects are same as ours.  IMPO, this draft seems very good to begin the discussion about the hypervisor MIB.

Thanks,
Hirochika

On Mar 26, 2012, at 11:26 PM, Melinda Shore wrote:

> Have you had an opportunity to review http://tools.ietf.org/html/draft-schoenw-opsawg-vm-mib-00?
> 
> Thanks,
> 
> Melinda
>

---

From: Robert Story <[email-masked]>
Date: Mon, 26 Mar 2012 12:51:13 -0400

On Mon, 26 Mar 2012 06:26:09 -0800 Melinda wrote:
MS> This is excellent - exactly the sort of feedback we're looking for.
MS> Have you had an opportunity to review 
MS> http://tools.ietf.org/html/draft-schoenw-opsawg-vm-mib-00?

I just took a quick look at it, and had a few comments..

>  GuestState ::= TEXTUAL-CONVENTION
>  [...]
>           other(2)       The state has been obtained but it does
>                          not a known state.

s/does/is/

>     -- The number of CPUs and the amount of memory can be found
>     -- in the objects of the HOST-RESOURCES-MIB

This should be clarified to indicate that it means the CPU/memory for the
hypervisor host..

> vmGuestTable OBJECT-TYPE

This table has information on the number of CPUs, but not information on
the type of CPUs. Hypervisors can sometimes provide a different CPU to the
guest than the host architecture (i.e. 32bit guest on 64bit host, or even
emulation of another processor entirely).

> vmStorageEntry OBJECT-TYPE
> [...]
>     INDEX       { vmGuestIndex, vmStorageIndex }
>     ::= { vmStorageTable 1 }
>
> VmStorageEntry ::= SEQUENCE {
>     vmStorageGuest      GuestIndexOrZero,
>     vmStorageIndex      StorageIndex,
>     vmStorageName       SnmpAdminString
> }

Shouldn't vmStorageGuest and vmStorageIndex be swapped in position, since
vmStorageIndex is part of the table index and vmStorageGuest is not?

> vmStorageGuest OBJECT-TYPE
> [...]
>         "Identifies the guest (virtual machine) this storage has
>          been allocated to. The value 0 indicates that the storage
>          is currently not allocated to a guest (virtual machine)."

What about storage that is shared with multiple guests?

> vmStorageIndex OBJECT-TYPE
> [...]
>         "A unique value identifying a logical storage area. On
>          systems implementing the HOST-RESOURCES-MIB, the value
>          must be the same value that is used as the index into
>          the hrStorageTable (hrStorageIndex)."

What about storage that is not in hrStorageTable? A good example would be
file based storage devices.

Most hypervisors have multiple network configurations available. An
interface may be bridged to a hosy interface, use NAT on a host
interface, or be a private network (which may or may not be shared with
other VMs). I think there should be a vmNetTable. The vmIfTable should have
a column indication which network, if any, it is attached to.

Robert

--
Senior Software Engineer
SPARTA, Inc., a Parsons Company

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 27 Mar 2012 01:10:40 +0200

> implementation in a couple of weeks, and post it to this mailing list.

I am surely interested in your data model. The data model in the
hypervisor MIB document posted two weeks ago was extracted out of
exising proprietary MIB modules and the goal was (and still is) to
identify what a common core might be and at the same time to avoid
duplication of stuff already covered by other MIB modules.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 28 Mar 2012 10:07:24 +0200

> >           other(2)       The state has been obtained but it does

fixed
 
> This should be clarified to indicate that it means the CPU/memory for the

This comment has already been moved into a separate section in the
document discussing relationships to other MIB modules.  The current
text is this:

        <t>
          The HOST-RESOURCES-MIB implemented on the physical machine
          provides information about the number of CPUs and the amount
          of memory available. Furthermore, the HOST-RESOURCES-MIB
          provides information about logical storage devices.
        </t>
        <t>
          The MIB module defined in this memo provides a mapping of
          logical storage devices to virtual machines. Further details
          about the storage devices (such as the size and the amount
          of allocated storage) is provided by the HOST-RESOURCES-MIB.
          Note that the number of storage types can be extended
          through the IANA maintained HOST-RESOURCES-TYPES MIB module.
        </t>

> emulation of another processor entirely).

Yes, this is a good point. I added action item to add this.

> vmStorageIndex is part of the table index and vmStorageGuest is not?

Oops, this really should have been:

      INDEX { vmStorageGuest, vmStorageIndex }

> What about storage that is shared with multiple guests?

Is this being done in reality? If it is, we surely have to revisit the
indexing. My idea was to make it easy to obtain all storage areas
attached to a specific guest quickly (without having to read the whole
table) - hence the double indexing. But if storage areas can be
shared, then this does obviously not work.

> file based storage devices.

But could storage areas that are backed up by files not show up in the
hrStorageTable? Or are you saying that tight integration of the
HOST-RESOURCES-MIB implementation with this MIB can't be assumed?
(Which boils down to replicating stuff defined in the
HOST-RESOURCES-MIB.)

> a column indication which network, if any, it is attached to.

Can you elaborate? What would be in the vmNetTable?

/js

---

From: <[email-masked]>
Date: Wed, 28 Mar 2012 06:22:23 -0400

Juergen,

Regarding storage sharing:

> Is this being done in reality?

Yes - it's rather common.  Here are two important examples:

1) When certain HA cluster software is deployed within multiple VMs, the quorum device/volume (used to maintain cluster membership, including revising cluster membership after a failure) is shared among the VMs that run the HA cluster software.

For at least VMware vSphere (ESX & ESXi hypervisors), it's very common to put multiple virtual machines (including virtual disk images) on a single storage resource seen by the hypervisor (NFS filesystem, SCSI-based SAN storage device) - for SCSI-based SAN storage, this is done via VMFS.  In addition, hypervisor clusters typically share the storage resources across the cluster in order to avoid copying storage when a VM is live-migrated across servers.

Thanks,
--David (storm [STORage Maintenance] WG co-chair)

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Juergen Schoenwaelder
> Sent: Wednesday, March 28, 2012 4:07 AM
> To: Robert Story
> Cc: [email-masked]

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 28 Mar 2012 13:13:42 +0200

> For at least VMware vSphere (ESX & ESXi hypervisors), it's very common to put multiple virtual machines (including virtual disk images) on a single storage resource seen by the hypervisor (NFS file…

David,

this is useful information. While some of the sharing seems to be
happening below the 'image' given to virtual machine as a file system
(and thus might not be relevant from the hypervisor perspective), it
seems assuming there is no sharing in the indexing structure is likely
a mistake. So it seems reasonable to change the current indexing.

/js

---

From: Robert Story <[email-masked]>
Date: Wed, 28 Mar 2012 08:01:51 -0400

On Wed, 28 Mar 2012 06:22:23 -0400 [email-masked] wrote:
DBC> SCSI-based SAN storage device) - for SCSI-based SAN storage, this is
DBC> done via VMFS.  In addition, hypervisor clusters typically share the
DBC> storage resources across the cluster in order to avoid copying storage
DBC> when a VM is live-migrated across servers.

Which reminds me of something else I meant to mention.. We probably need to
add 'migrating' to the list of machine states..

Robert

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 28 Mar 2012 14:46:50 +0200

> add 'migrating' to the list of machine states..

Yes, I agree.

/js

---

From: Robert Story <[email-masked]>
Date: Wed, 28 Mar 2012 08:50:40 -0400

On Wed, 28 Mar 2012 10:07:24 +0200 Juergen wrote:
JS> > > vmStorageIndex OBJECT-TYPE
JS> > > [...]
JS> > >         "A unique value identifying a logical storage area. On
JS> > >          systems implementing the HOST-RESOURCES-MIB, the value
JS> > >          must be the same value that is used as the index into
JS> > >          the hrStorageTable (hrStorageIndex)."
JS> > 
JS> > What about storage that is not in hrStorageTable? A good example
JS> > would be file based storage devices.
JS> 
JS> But could storage areas that are backed up by files not show up in the
JS> hrStorageTable? Or are you saying that tight integration of the
JS> HOST-RESOURCES-MIB implementation with this MIB can't be assumed?
JS> (Which boils down to replicating stuff defined in the
JS> HOST-RESOURCES-MIB.)

I'm not familiar with high-end/bare metal hypervisors, but Parallels,
VMWare player/workstation, KVM and Xen all support various file-based
storage devices (e.g. qcow, qcow2, vmdk, cow). These do not appear as
filesystems on the host os, so they most likely would not appear in
hrStorage. (It's possible, but unlikely, that an HR implementation could be
hypervisor aware and parse Vm configurations to extract paths for these
files.)

JS> > Most hypervisors have multiple network configurations available. An
JS> > interface may be bridged to a hosy interface, use NAT on a host
JS> > interface, or be a private network (which may or may not be shared
JS> > with other VMs). I think there should be a vmNetTable. The vmIfTable
JS> > should have a column indication which network, if any, it is attached
JS> > to.
JS> 
JS> Can you elaborate? What would be in the vmNetTable?

 vmNetIndex - index of network
 vmNetName - name
 vmNetDesc (optional)
 vmNetPhysIfIndex - ifIndex of physical interface (or 0 if none)
 vmNetifType - vmNetNAT(0), vmNetPublicBridge(1), vmNetPrivateBridge(2)

Usually these virtual network offer DHCP addresses to guests, so it might
be useful to have that configuration information available too, either in
additional columns or in another table.

The vmIfTable would then look something like

  vmIfIndex - index of virtual interface; no relation to ifIndex
  vmGuestIndex - index of guest to which interface is assigned
  vmNetIndex - index of network to which interface is assigned
  vmIfState - vmIfConnected(0), vmIfDisconnected(0)
  vmIfPhysAddr

I don't think that guest or network index should be part of the index for
this table, as either one can be changed at any time.

It may be useful to have a few convenience tables:

* vmGuestIfTable { vmGuestIndex, vmGuestIfIndex } vmNetIndex
* vmNetIfTable { vmNetIndex, vmIfIndex} vmGuestIndex

Robert