Subject: [OPSAWG] Hypervisor MIB document

From: Michael MacFaden <[email-masked]>
Date: Tue, 10 Apr 2012 22:40:33 -0700 (PDT)

Hey Juergen, 

Am copying the opsawg list per your suggestion as we continue our
conversation on the draft hypervisor mib module. 

>[email-masked] wrote: 
>>[email-masked] wrote: >> 1) Which operators are we trying to service with this mib module? 
>> - enterprise 
>> - service providers (old school telcos or new school 'cloud') 
>> - all of the above 
>> - other 

>I guess it is all of the above. Do you think there is a major 
>difference from the MIB module perspective between enterprise clouds 
>and clouds provided as a service? 

Yes. I'll explain below. 

>And if there are differences, would 
>there not likely also be differences be clouds run by native cloud 
>providers and clouds operated by telcos as part of their portfolio? 

Don't personally know if ops models being used differ for native cloud 
providers vs telcos. Maybe someone on the list will chime in here. 
Today I work mostly with large corporate IT accounts and integrators that 
embed VMware ESX/vCenter directly into their products 
(ex: Cisco Integrated Service Router, Motorola Astro 25) and (in the past) 
with telcos and the nanog community. 

>I assume the difference (if any) is the integration into existing 
>management and monitoring infrastructures where SNMP likely plays a 
>more important role for telcos and enterprises than for specialized 
>cloud operators (but I am guessing here). What is your experience with 
>VMware MIBs? 

The VMWARE-VMNINFO-MIB[1] has been in deployed in product since 2001. 
In that time it has not changed all that much. Other than converting
to SMIv2, here are three changes I can recall off hand: 

1) A common problem among mgmt apps was unique vm identification. 

Apps would use the VMWARE-VMINFO-MIB::vmwVmTable table index vmwVmIdx . 
At point I added vmwVmUUID to allow tracking of virtual machine movement cross 
ESX systems as this could confuse mgmt apps like CA SPECTRUM. 
Notifications being sent still generate power off and power on 
and its up to mgmt apps to sync on UUID to know a VM moved. 

2) Then when virtual SMP was added new managed objects were needed to 
track that, vmwVmCpus. A comment on the opsawg list pointed out that 
virtualized CPUs (VCPU) can be 32 or 64 but its even more than that. 
In order to move a live virtual machine to move from one host to another there 
is a whole raft of VCPU properties that must also be identical. 

3) At some point counters for network i/o were made obsolete & not 
provided in ESXi (VMWARE-OBSOLETE-MIB:: vmwNetPktsTx, etc), yet 
these were highly popular. Modeling these as part of the VM were a mistake 
to begin with imo as there are standard objects that should 
have been populated in the virtual switch/bridge mib port 
that a given VM is attached to. 

So the right balance of what to show as "config" and 
what to show for "performance" will take some time to get right 
but then again that's what SNMP community has shown time again 
it is good at coming up with -- a data model done right 
(regardless of SPPI/YANG/SMIv2 syntax). Also would point out
DMTF has existing standards for virtual machines described here: 
http://www.dmtf.org/sites/default/files/standards/documents/DSP2013_1.0.0.pdf 

Anyway I believe a main difference between an Enterprise and 
a Service provider is how they do tenancy and scale. 
Some background on tenancy: 
http://www.netapp.com/us/technology/secure-multi-tenancy.html 
http://www.cisco.com/en/US/solutions/ns340/ns414/ns742/ns743/ns1050/landing_dcVDDC.html 

Also for scale, here is what Igor Gashinsky/YAHOO posted on ARMD list Feb/2012: 
>To me, (and perhaps i'm in a very small minority here?), massive implies cloud scale datacenters, so, 10-20k physical hosts *per >cluster*, and somewhere around 400-500k VM's is the absolute *minimum* for what I would concider to be massive (and, really, I >think 100k physical, 2M+ logical is what i believe a realistic aiming point these days should be). 

Have never seen such large numbers in enterprise accounts. 
So might consider using timeFilter indexed tables and other techniques for scale.

Mike MacFaden 
Staff Engineer, R&D Apps, VMware Palo Alto 

[1] http://communities.vmware.com/community/developer/forums/managementapi

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 23 Apr 2012 13:41:16 +0200

> and its up to mgmt apps to sync on UUID to know a VM moved. 

Good. The VM-MIB has a UUID for each guest so we are save.
 
> is a whole raft of VCPU properties that must also be identical. 

There was already a comment that we need to identify the CPU type
especially since some platforms allow to emulate CPUs. So the question
is how we model this properly. I should probably start a separate
thread on this. Can you provide more information what the other "whole
raft of VCPU properties" entails?

> Also for scale, here is what Igor Gashinsky/YAHOO posted on ARMD
> list Feb/2012: >To me, (and perhaps i'm in a very small minority
> here?), massive implies cloud scale datacenters, so, 10-20k physical
> hosts *per >cluster*, and somewhere around 400-500k VM's is the
> absolute *minimum* for what I would concider to be massive (and,
> really, I >think 100k physical, 2M+ logical is what i believe a
> realistic aiming point these days should be).
> 
> Have never seen such large numbers in enterprise accounts.  So might
> consider using timeFilter indexed tables and other techniques for
> scale.

Since the VM-MIB is implemented on physical hosts, what matters is
really the ration of guests (VMs) per physical host. With the numbers
above, 2M+ logical and 100k physical, we are still at an average of 20
guests per physical host. Assuming a long tailed distribution, we
might consider 100 guests per physical machine, if its a cluster
perhaps even more but nothing really unmanageable. The big number
mentioned above is the huge number of physical machines.

Whether timeFilter indexed tables are needed I do not know. Perhaps
just having a few lastChanged objects is sufficient to allow for
simple polls to check whether new guests popped up or guests changed
their status?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Robert Story <[email-masked]>
Date: Mon, 23 Apr 2012 09:02:19 -0400

On Mon, 23 Apr 2012 13:41:16 +0200 Juergen wrote:
JS> There was already a comment that we need to identify the CPU type
JS> especially since some platforms allow to emulate CPUs. So the question
JS> is how we model this properly. I should probably start a separate
JS> thread on this. Can you provide more information what the other "whole
JS> raft of VCPU properties" entails?

KVM virtual machines keep track of features (e.g. sse2, apic) supported by
the virtual machine's CPU.  When you migrate a VM to another  host, that
host must support the same feature set. Here's are two pages about this:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Getting_Started_Guide/para-CPU_Models.html

http://berrange.com/posts/2010/02/15/guest-cpu-model-configuration-in-libvirt-with-qemukvm/

Robert

--
Senior Software Engineer
SPARTA, Inc., a Parsons Company

---

From: Michael MacFaden <[email-masked]>
Date: Tue, 24 Apr 2012 09:58:13 -0700 (PDT)

> thread on this. Can you provide more information what the other
> "whole raft of VCPU properties" entails?

CPU Features which make the VMware products capable of migrating
VMs (what VMware calls vMotion) between physical processors
are described here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1993

Hence the following must be exposed for a given hardware system
Cpu mfg, process family, feature sets.

IMO this data would not be added to this draft mib module, 
we just augment RFC 2790 HOST-RESOURCES-MIB or if need be 
have a separate mib module for CPU related reporting in the draft.

Notice that if you know your workload you can mask out features
to increase compatibility which would be something for this draft.

> above, 2M+ logical and 100k physical, we are still at an average of
> 20 guests per physical host. Assuming a long tailed distribution, we

Indeed. There are two general types of servers, those that run 
backend vms (databases, etc) where the ratio of vms to host system is low and
desktop vms, where the ratio is hundreds of vms per server.

> simple polls to check whether new guests popped up or guests changed

I'll see if I can collect some evidence of what  frequently 
vs what does not. I prefer to not have to sync full tables to find
one out difference when a last changed goes off would be my preference
when the tables are large.

Thanks,
Mike MacFaden

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 25 Apr 2012 15:48:34 +0200

> to increase compatibility which would be something for this draft.

Just to be clear: My understanding is that you are talking about the
physical CPUs of a system, not the CPUs emulated and given to virtual
machines (as qemu for instance does). Am I correct?

/js

---

From: Michael MacFaden <[email-masked]>
Date: Wed, 25 Apr 2012 10:16:21 -0700 (PDT)

> machines (as qemu for instance does). Am I correct?

In short, HOST-RESOURCES-MIB should report what it is capable of.
The draft should report what it expects.

One fielded implementation allows for masking out what features will
be compared in order to determine if a virtual machine can migrate.

Mike

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 25 Apr 2012 20:11:03 +0200

> be compared in order to determine if a virtual machine can migrate.

Well, the HOST-RESOURCES-MIB is likely not revised anytime soon; all
we can reasonably do is to extend it. That said, there is also the
ENTITY-MIB that is reporting about the physical components and it has
a CPU class. It is not clear to me whether the HOST-RESOURCES-MIB or
the ENTITY-MIB should be able to report these hardware characteristics
(as long as we talk about physical CPUs).

For the matching of physical CPUs and their features, I am not even
sure a MIB is the right tool. There usually is a software tool that
controls migration of VMs and I am not sure such a tool will rely on
information provided by MIBs. In fact, I see the MIBs more as a way to
do scalable monitoring and hence I am not sure we really have to solve
the CPU matching problem.

/js

---

From: Robert Story <[email-masked]>
Date: Wed, 25 Apr 2012 15:45:32 -0400

On Wed, 25 Apr 2012 20:11:03 +0200 Juergen wrote:
JS> For the matching of physical CPUs and their features, I am not even
JS> sure a MIB is the right tool. There usually is a software tool that
JS> controls migration of VMs and I am not sure such a tool will rely on
JS> information provided by MIBs. In fact, I see the MIBs more as a way to
JS> do scalable monitoring and hence I am not sure we really have to solve
JS> the CPU matching problem.

I don't care about solving the matching problem either.. Just reporting the
CPU type for each VM is fine with me..

Robert

---

From: Michael MacFaden <[email-masked]>
Date: Wed, 25 Apr 2012 17:45:48 -0700 (PDT)

js writes:
> There usually is a software tool that
> information provided by MIBs. 

Sure. Then strike a use case where an operator would use this mib module
as a means 'after the fact' to understand why vm migration between two hosts failed
or for collecting an inventory report the physical host systems to obtain this data.

FWIW, here is the DMTF standard for reporting on CPUS:
http://software.intel.com/sites/manageability/AMT_Implementation_and_Reference_Guide/DOCS/Implementation%20and%20Reference%20Guide/default.htm?turl=HTMLDocuments%2FWS-Management_Class_Reference%2FCIM_Processor.htm

>In fact, I see the MIBs more as a way to do scalable monitoring and 
>hence I am not sure we really have to solve the CPU matching problem.

Please define what scaleable monitoring trying to accomplish?
who is using this mib module and what are they using it for 
needs to be expressly stated at some point.

Mike

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 26 Apr 2012 09:44:15 +0200

On Wed, Apr 25, 2012 at 05:45:48PM -0700, Michael MacFaden wrote:
> Sure. Then strike a use case where an operator would use this mib
> module as a means 'after the fact' to understand why vm migration
> between two hosts failed or for collecting an inventory report the
> physical host systems to obtain this data.

If I attempt to migrate a VM, would the software tool not report a
suitable error explaining why the migration can't be carried out? Are
tools really written in such a way that operators need to investigate
SNMP MIBs what potential reasons for not migrating a VM were?
 
> http://software.intel.com/sites/manageability/AMT_Implementation_and_Reference_Guide/DOCS/Implementation%20and%20Reference%20Guide/default.htm?turl=HTMLDocuments%2FWS-Management_Class_Reference%2FC…

Thanks. There is lots of stuff - so let me try to see what
specifically is relevant in this context. I see:

Family            - An enueration of processor families.
Stepping          - A free-form string that indicates the revision 
		    level of the Processor within the Processor

And then there is a collection of clock speeds and states. I do not
see something that addresses the CPU matching problem.

Anyway, my current thinking is that what we are discussing here really
boils down to an extension of the ENTITY-MIB, i.e. an ENTITY-CPU-MIB
providing an entPhyCPUTable, sparsely augmenting the entPhysicalTable
for entities with entPhysicalClass = cpu.

> needs to be expressly stated at some point.

Such text is difficult to write and usually use cases evolve during
the work on the module. For me, the prime interest is to enable
monitoring and thresholding of virtual machines using existing SNMP
tools out there including tracking of resource usage to support longer
term load balancing and planning etc. But I am sure there are other
usages that people find important and that's fine as well. As we move
forward, we will start to understand what is possible and feasible.

/js

---

From: Michael MacFaden <[email-masked]>
Date: Thu, 26 Apr 2012 11:41:53 -0700 (PDT)

js writes:
> suitable error explaining why the migration can't be carried out?

Yes it would. 

> Are tools really written in such a way that operators need to investigate

Depends on the problem that causes the failure. While operator configuration
errors tend to account for the bulk of IT systems failures, networking issues, 
hardware health issues, software issues (OS/cpu, memory etc
contribute on average something like 10-15% of the failure cases. 

SNMP MIB modules excel at providing near real-time diagnostics, 
generalized configuration/inventory and performance state.

So if one can't query via SNMP mib modules the physical inventory pertinent
to running virtual machines then I think that needs to be addressed
at some point in a separate mib 4133 or 2970 and discussed in
in your front matter of the draft.
 
> Anyway, my current thinking is that what we are discussing here
> really boils down to an extension of the ENTITY-MIB, i.e. an ENTITY-CPU-MIB

Ok, sounds good.

> For me, the prime interest is to enable monitoring and thresholding of 
> virtual machines using existing SNMP tools out there including tracking 
> of resource usage to support longer term load balancing and planning etc. 

I get monitoring of VMs on a hypervisor and reporting basic resource usage.
So I think your needs align with operators deploying hypervisors that do 
not use clustering protocols though. 

Today's commercial hypervisors are running with clustering protocols
that use a wooly set of internal/possibly proprietary metrics 
that provide the input to the load balancing algorithms running distributed across a
set(cluster) of hypervisors. In addition to load balancing vms they deal with hypervisors 
that die suddenly or get disconnected(split-brain). Hypervisors under control
of clustering software then automatically will shut down vms a or power up 
a copy of a given vm lost from another system automatically as configured.

Further like BGP meds are to network admins, host admins teak lots of knobs on 
resource pools at a cluster level not hypervisor level. There's another
bit of software system (call it XVCX) controlling the various hypervisors and this 
XVCX may also have its own SNMP agent. 

Here's how large datacenters are setup today --

Operators now build datacenters by ordering racks of servers at a time.
A truck delivers a full 46RU high rack pre-loaded and wired up with servers.
Admins simply connect the built-in (top-of-rack or TOR) ethernet switch 
to their network and supply power.

The x86 systems PXE boot, get their hypervisor, configuration and then connect to 
XVCX and join their respective clusters, optionally connect
to their san storage and vms being powering up or w/out power up are migrated to the
storage of the new hardware, and all controlled by the clustering protocols which
are controlled from the XVCX. 

> But I am sure there are other

Ok, let's just be clear to summarize the discussion and
document why something is there or not either in the draft or otherwise.

Mike