Subject: [OPSAWG] Comments on draft-asai-vmm-mib-04

From: Joe Marcus Clarke <[email-masked]>
Date: Sun, 18 Aug 2013 19:28:39 -0400

Sorry for the big delay between IETF-87 and now.  I wanted to officially
record my comments about this proposed MIB module on the mailing list to
backup what I said on the mic in the working group session.  Please see
JMC below.

Section 1:

The design of this MIB module has been derived from enterprise
   specific MIB modules, namely a MIB module for managing guests of the
   Xen hypervisor, a MIB module for managing virtual machines controlled
   by the VMware hypervisor, and a MIB module using the libvirt
   programming interface to access different hypervisors.

JMC: While I think the intention is to say that this MIB pulls from
other, existing MIBs for common virtualization platforms, it may come
across as being limited to only those platforms.  Perhaps it is good to
say that this MIB then attempts to generalize those modules to support
other hypervisors as well?

Section 3.3:

Concerning vmTable

JMC: I think there should be an object in this table such as vmMgmtAddr,
which lists the management IP address of the VM.  This would be akin to
what one would see in vCenter for a VMware-controlled VM.  As a manager,
this would be very useful in being able to discover other potential SNMP
agents in the network.

JMC: The purpose of this MIB is not to provide configuration of the
hypervisor.  Quoting section 3.2:

"The MIB module provides a few writable objects that can be used to
   make non-persistent changes, e.g., changing the memory allocation or
   the CPU allocation.  It is not the goal of this MIB module to provide
   a configuration interface for virtual machines since other protocols
   and data modeling languages are more suitable for this task."

To that end, I find objects like vmAutoStart, vmMinCpuNumber,
vmMaxCpuNumber, vmMinMem, and vmMaxMem strange.  For starters,
vmAutoStart mentions the word "configuration" in its description.  And
the other objects, at least from a VMware standpoint are actually
specified in the VM's configuration.  One would have to, for example,
shutdown the VM to increase the number of vCPUs allocated to the VM.
Therefore, I feel all of these objects represent persistent changes to
the hypervisor in support of the respective VM.

I think these objects should go away and only the read-only objects
showing current vCPU allocation, affinity, and memory stats should
remain.  As for auto-start, that can be read-only.

The alternative is to say that some persistent changes will be allowed
(such as auto-start), but general VM resource configuration will be out
of scope.

Section 3.3:

vmNetworkEntry OBJECT-TYPE
       SYNTAX       VmNetworkEntry
       MAX-ACCESS   not-accessible
       STATUS       current
       DESCRIPTION
               "An entry for one virtual storage device attached to the
               virtual machine."
       INDEX { vmIndex, vmNetworkIndex }
       ::= { vmNetworkTable 1 }

JMC: Note how the description describes this as "storage" device...

Section 3.3:

Concerning vmNetworkTable.

JMC: I think there should be a vmNetworkVlan object in this table that
describes the VLAN ID associated to the underlying interface.  This need
not be used for every hypervisor, but I think it is very important in
some (like VMware).

JMC: What would one expect if I have a VM connected to multiple
vSwitches in a VMware environment?  What object will tell me to what
vSwitch I am connected?  Should there also be a name here for the
virtual interface?  I'm thinking of the case where something like a
Cisco Nexus 1000v switch is used.  In that case, knowing that I am
connected to Ethernet1/1 might be useful.

JMC: Like in the VM table, I wonder if it makes sense to have a network
management address for the case where the associated virtual network
device can be independently managed.

Section 3.3:

Concerning vmNotifications.

JMC: I think there should be a general notification for VM changing
state.  The reason for this is that some of the state transitions
described in section 3.2 Figure 2 can take a long time (e.g., running ->
shuttingdown -> shutdown).  In that case, an administrator may have to
wait minutes before being alerted to the fact that her VM is offline.
If there was a notification for vmStateChange, then the admin would be
notified immediately if a VM entered a degraded state.

I think that's all I noticed.  Thanks.

Joe

-- 
Joe Marcus Clarke, CCIE #5384,         |          |
SCJP, SCSA, SCNA, SCSECA, VCP        |||||      |||||
Distinguished Services Engineer ..:|||||||||::|||||||||:..
Phone: +1 (919) 392-2867         c i s c o  S y s t e m s
Email: [email-masked]

----------------------------------------------------------------------------

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 10:55:37 -0700 (PDT)

Hi Joe,

Thanks for sharing this, my comments inline.

Marking this item as Proposal: Joe-1

> other hypervisors as well?

So is this just the wording that concerns you?
Juergen and others have raised generalization issue/asked others to contribute. 

Regards,
Mike MacFaden

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 11:02:15 -0700 (PDT)

Marking this item as Proposal: Joe-2

> this would be very useful in being able to discover other potential SNMP

Do you want this managed object to report only if there is a configured
mgmt agent running in the guest OS(GOS)? There can be more than one mgmt 
address per address family.

So I did something similar in VMware ESXi 5.1 for the hypervisor
and its relation to the hardware it is installed on. With blade servers, 
Cisco ISR, or Cisco UCS or the new HP Gen8 hardware for example it
was clear to me there were going to be multiple SNMP agents and so to
ease mgmt apps discovery I made it simple for hardware vendors
to insert conceptual rows into entLogicalTable/RFC 4133 so
that ESXi SNMP can point back to the supervisor card or other embedded
hardware mgmt agent.

An example from the wild...(not a marketing pitch)

To discover the mgmt address hypervisor's snmp agent if a mgmt
app has first discovered the hardare agent, poll RFC 4133

SNMPv2-MIB::sysDescr.0 = STRING: Integrated Lights-Out 4 1.10 Jul 17 2012
SNMPv2-MIB::sysObjectID.0 = OID: CPQHOST-MIB::compaq.9.4.10
....
ENTITY-MIB::entPhysicalIndex.1 = INTEGER: 1
ENTITY-MIB::entPhysicalDescr.1 = STRING: HP ProLiant BL460c Gen8
ENTITY-MIB::entPhysicalVendorType.1 = OID: SNMPv2-SMI::zeroDotZero
ENTITY-MIB::entPhysicalContainedIn.1 = INTEGER: 0
ENTITY-MIB::entPhysicalClass.1 = INTEGER: backplane(4)
ENTITY-MIB::entPhysicalParentRelPos.1 = INTEGER: -1
ENTITY-MIB::entPhysicalName.1 = STRING: smbios
ENTITY-MIB::entPhysicalHardwareRev.1 = STRING:
ENTITY-MIB::entPhysicalFirmwareRev.1 = STRING: I31
ENTITY-MIB::entPhysicalSoftwareRev.1 = STRING:
ENTITY-MIB::entPhysicalSerialNum.1 = STRING: USE246M0JN
ENTITY-MIB::entPhysicalMfgName.1 = STRING: HP
ENTITY-MIB::entPhysicalModelName.1 = STRING: ProLiant BL460c Gen8
ENTITY-MIB::entPhysicalAlias.1 = STRING:
ENTITY-MIB::entPhysicalAssetID.1 = STRING:
ENTITY-MIB::entPhysicalIsFRU.1 = INTEGER: true(1)
ENTITY-MIB::entPhysicalMfgDate.1 = STRING: 2012-8-20
ENTITY-MIB::entPhysicalUris.1 = STRING: "urn:uuid:30313436-3631-5355-4532-34364d304a4e"
ENTITY-MIB::entLogicalIndex.1 = INTEGER: 1
ENTITY-MIB::entLogicalDescr.1 = STRING: ESXi 5.1.0 Releasebuild-1021289 Update 0 SNMP Agent over IPv4
ENTITY-MIB::entLogicalType.1 = OID: SNMPv2-SMI::mib-2
ENTITY-MIB::entLogicalCommunity.1 = STRING: "public"
ENTITY-MIB::entLogicalTAddress.1 = STRING: "10.131.0.171"  <<<- VMware ESXi instance
ENTITY-MIB::entLogicalTDomain.1 = OID: TRANSPORT-ADDRESS-MIB::transportDomainUdpIpv4
ENTITY-MIB::entLogicalContextEngineID.1 = STRING: "00000063000000a10a8300ab"

To discover the hardare agent given discovery of a hypervisor agent,
SNMPv2-MIB::sysDescr.0 = STRING: VMware ESXi 5.1.0 build-799733 VMware, Inc. x86_64
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.6876.4.1
...
ENTITY-MIB::entPhysicalDescr.1 = STRING: HP ProLiant BL460c Gen8
ENTITY-MIB::entPhysicalVendorType.1 = OID: SNMPv2-SMI::zeroDotZero
ENTITY-MIB::entPhysicalContainedIn.1 = INTEGER: 0
ENTITY-MIB::entPhysicalClass.1 = INTEGER: module(9)
ENTITY-MIB::entPhysicalParentRelPos.1 = INTEGER: -1
ENTITY-MIB::entPhysicalName.1 = STRING: smbios
ENTITY-MIB::entPhysicalHardwareRev.1 = STRING:
ENTITY-MIB::entPhysicalFirmwareRev.1 = STRING: 2.7
ENTITY-MIB::entPhysicalSoftwareRev.1 = STRING:
ENTITY-MIB::entPhysicalSerialNum.1 = STRING: USE246M0JM
ENTITY-MIB::entPhysicalMfgName.1 = STRING: HP
ENTITY-MIB::entPhysicalModelName.1 = STRING: ProLiant BL460c Gen8
ENTITY-MIB::entPhysicalAlias.1 = STRING:
ENTITY-MIB::entPhysicalAssetID.1 = STRING:
ENTITY-MIB::entPhysicalIsFRU.1 = INTEGER: true(1)
ENTITY-MIB::entPhysicalMfgDate.1 = STRING: 2012-8-20,0:0:0.0,
ENTITY-MIB::entPhysicalUris.1 = STRING: "urn:uuid:50e8484b-6e6d-734c-539c-b4b52f6fa410"
ENTITY-MIB::entLogicalDescr.1 = STRING: VMware vCenter Server
ENTITY-MIB::entLogicalDescr.2 = STRING: HP Integrated Lights Out SNMP Management Agent
ENTITY-MIB::entLogicalType.1 = OID: VMWARE-PRODUCTS-MIB::vmwVC
ENTITY-MIB::entLogicalType.2 = OID: SNMPv2-SMI::enterprises.11.1
ENTITY-MIB::entLogicalTAddress.1 = STRING: "192.168.2.104"
ENTITY-MIB::entLogicalTAddress.2 = STRING: "10.131.16.172"
ENTITY-MIB::entLogicalTDomain.1 = OID: TRANSPORT-ADDRESS-MIB::transportDomainUdpIpv4
ENTITY-MIB::entLogicalTDomain.2 = OID: TRANSPORT-ADDRESS-MIB::transportDomainUdpIpv4
ENTITY-MIB::entLastChangeTime.0 = Timeticks: (42096100) 4 days, 20:56:01.00

So my question here is even if we want to report the mgmt IP address of the GOS,
is this the right MIB module to do that in?

Mike MacFaden
Staff Engineer R&D Apps VMware, Inc Palo Alto CA

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 14:05:17 -0400

> Juergen and others have raised generalization issue/asked others to contribute. 

Yup.  This was just a nit that I thought the wording as is mught give
the impression that this MIB is limited in scope to hypervisors of the
specified flavors.

Joe

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 14:13:53 -0400

> address per address family.

That's fair, and I understand that.  My point was that when one is
looking at a VM, at least in vCenter, one can see an IP of the GOS.  I
thought it would be useful to pass that information down the MIB as
well.  And, yes, I get that this may not be known the HV, and thus may be 0.

> that ESXi SNMP can point back to the supervisor card or other embedded

But this is driven from the GOS, right?

[SNMP Walk elided]

> is this the right MIB module to do that in?

I think that showing that information, where available is helpful,
especially since this is designed to be cross-HV, where the ENTITY
support may be more limited, and vendor-specific.

Joe

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 11:26:06 -0700 (PDT)

Marking this proposal as Joe-3

>    and data modeling languages are more suitable for this task."

Right.

> shutdown the VM to increase the number of vCPUs allocated to the VM.

Such behavior really depends on the hypervisor.

Adding a CPU while VM is running has been a feature of VMware for some time
now: http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.vmadmin.doc_41/vsp_vm_guide/configuring_virtual_machines/t_change_cpu_hotplug_settings.html

> remain.  As for auto-start, that can be read-only.

My position as well. Once you go read-create/write the complexity
grows and description fall short of any real standard behavior in
response to a mutational set is almost always lost.

> (such as auto-start), but general VM resource configuration will be out

Mark this as Joe-3A.

As I said above, I'm not interested in standardizing configuration
of Virtual Machines via SNMP. Neither are the operators I know.

Mike MacFaden

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 19 Aug 2013 20:30:14 +0200

> address per address family.

In our systems, the hypervisor has no clue which IP addresses are used
by the guest OSes (and surely the hypervisor does not know which IP
addresses would be "management" addresses).

We use a virtual bridge to connect the VMs. Can we not use the same
discovery techniques that can be used today to discover devices
(e.g. connected to a physical bridge) to discover virtual machines?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 19 Aug 2013 20:34:43 +0200

> of Virtual Machines via SNMP. Neither are the operators I know.

We seem to agree on the general scope. Now we have to determine which
objects reasonably have a MAX-ACCESS or read-write. For me, it seems
that vmAutoStart likely should indeed be read-only. However, as far as
I know, Xen allows to change vmMinCpuNumber, vmMaxCpuNumber, vmMinMem,
and vmMaxMem at runtime without touching persistent config state.

/js

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 14:36:14 -0400

> now: http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.vmadmin.doc_41/vsp_vm_guide/configuring_virtual_machines/t_change_cpu_hotplug_settings.html

Ah, I haven't played with hotplug.  Nonetheless, I would expect
something like this to be persistent vs. available for the current like
of the VM, correct?  Something more transient would be the option to
boot into the BIOS on next boot...

> of Virtual Machines via SNMP. Neither are the operators I know.

Yeah, I gather, and I agree.  That's why options like CPU and memory
make me a bit uneasy as I see them as more configuration.

Joe

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 11:38:02 -0700 (PDT)

> well.  And, yes, I get that this may not be known the HV, and thus may be 0.

What I'm trying to get from you is why we need this?
So far you've stated the use is to discover other (SNMP) agents.
And I buy that and demonstrated how its done using ENTITY-MIB
implemented in two separate vendors agents given bi directional
discovery capability.

Are there any other use cases here? 
Say the address is IPv6 and the stat is not preferred what would we show?

Trying to get more details why mgmt agents (as opposed to 
operators using VC) want this info.

Such data can also be obtained by reading various network address tables
if the VM is at all active. 

In the case of the product you mention, VC is reporting IP address
ranges it see only if the GOS has the "Tools" agent installed.
It also just reports on the network range seen on the vswitch to VM
connection.

> But this is driven from the GOS, right?

No GOS involved. If one turns on the ESXi snmp agent (no vms need be powered up)
and polls RFC 4133, they can find other snmp mgmt agent. Hardware
vendors install vibs (like .deb/.rpm) to populate the data.

> support may be more limited, and vendor-specific.

One of the nice features of MIB modules is that they are modular. 
That means you can write code once and it works in most deployments.
After two decades my admiration for the design that went into these
modules (2790 on up) has only grown.

Time and time again, operators tell me they wan to learn something once
and leverage over and over again. So I would make acounter-proposal,
show you how the feature that relates a GOS agent to a hypervisor
is no different that relating a hypervisor to the supervisor/hardware card.

Regards,
Mike MacFaden Staff Engineer R&D Apps VMware, Inc Palo Alto CA

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 14:38:40 -0400

> (e.g. connected to a physical bridge) to discover virtual machines?

Sure.  This just presents another, maybe simpler way to find VMs in
particular in the network.  I fully understand that this address may be
0, but I know at least for the VMs I maintain, it would be a useful value.

Joe

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 11:45:19 -0700 (PDT)

Marking this proposal as Joe-4

> JMC: Note how the description describes this as "storage" device...

Agreed.

Mike MacFaden

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 19 Aug 2013 20:50:54 +0200

> 0, but I know at least for the VMs I maintain, it would be a useful value.

I do not question the value (although an IP address still requires
guess-work to talk to a management interface successfully).

I am saying that it is not at all easy to implement such an object
since the hypervisor may simply have no clue which IP addresses are
used by the guest OSes (and which subset would be 'management'
addresses). It might work in your case - but we need to maintain a
somewhat broader view I think.

/js

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 11:55:21 -0700 (PDT)

Marking this proposal as Joe-5

> not be used for every hypervisor, but I think it is very important in

Are we talking about the configuration in the GOS or the virtual switch?

> connected to Ethernet1/1 might be useful.

If the virtual switches in the hypervisor implement either the IEEE
or the IETF Q bridge mib then you can easily make this determination.
Why? the VM-MIB reports MAC address. From that we find the mac in the
FDB table and locate the switch port and vlan.
 
> device can be independently managed.

It does.

My counter-proposal is that the ENTITY-MIB would be populated
contain the mgmt address(s) for any virtual switch. My prior
mib walk actually had entries for how one product does it already :-)

Regards,
mike

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 12:00:07 -0700 (PDT)

Marking this proposal as Joe-7

> notified immediately if a VM entered a degraded state.

Can you help me understand the relationship between agent and operator?
Is the app polling and getting events (aka trap-directed polling) or is
it just getting events or just polling?

Regards,
Mike MacFaden

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 16:35:03 -0400

> What I'm trying to get from you is why we need this?

Sorry for the delay.  I was on a plane previously, and now I'm back on
solid ground.

My use case is around VM discovery.  If I have this table, and I can
enumerate VMs, it would be useful, if available, to see what address I
might use to connect to the VM.

> Say the address is IPv6 and the stat is not preferred what would we show?

That would be up to the HV and the underlying interface with the VM.
The HV is perfectly capable of using a 0 address to indicate that no VM
address is available.  But if populated, a manager can then, without
necessarily doing other host discovery operations, attempt to further
manage the VM.

> if the VM is at all active. 

Yes, this is a typical approach, but in seeing this MIB, I was proposing
a shortcut.

> It also just reports on the network range seen on the vswitch to VM

Yes, I understand that.

> vendors install vibs (like .deb/.rpm) to populate the data.

Right, I misunderstood.  I had thought the VM was inserting conceptual
rows.  That's what altitude will do to you.  While this is very useful
for the host, it isn't the same thing as showing the VM IP address.

> is no different that relating a hypervisor to the supervisor/hardware card.

That is a fair point.  If I understand you right this time, and the VM
inserts entLogicalTable entries in the HV ENTITY-MIB, would it not make
sense to add a pointer in the VMM-MIB to indicate their index in the
entLogicalTable?

Joe

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 16:38:28 -0400

> and vmMaxMem at runtime without touching persistent config state.

If the WG's intent is to leave them as RW (and I can see that certain
HV's allow this sans persistence), then there should be stronger
guidance (I think) that indicates that these are operational-only
objects and the new values should not be persisted.  But that may be too
messy.

Is there a strong push from operators to toggle these values via SNMP?

Joe

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 16:40:18 -0400

> Are we talking about the configuration in the GOS or the virtual switch?

I was referring to the virtual switch.

> FDB table and locate the switch port and vlan.

You're right.  Again, I was aiming for more of a shortcut with data that
would generally be useful.

> mib walk actually had entries for how one product does it already :-)

Again, that's fair.  My corollary question would then be should there be
a entLogicalTable pointer in this table?

Joe

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 16:46:22 -0400

> it just getting events or just polling?

Potentially both (I know Cisco at least uses both types of managers).
My use case, based on reading the MIB, is that I'd like an object that
tells me when I make a negative state transition so I can be notified
immediately when a VM moves to, for example, shuttingdown from running
(i.e., similar to bgpBackwardTransition from the BGP4-MIB).

Joe

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 13:52:15 -0700 (PDT)

> If I understand you right this time, and the VM

Yes, exactly. Create a row in entLogicalTable if there is data to provide
regarding pointers to agents in GOS and have a way to get at it to/from 
the VMM-MIB.

Regards,
Mike MacFaden

---

From: Joe Marcus Clarke <[email-masked]>
Date: Mon, 19 Aug 2013 16:54:22 -0400

> regarding pointers to agents in GOS and have a way to get at it to/from 

Okay, so in that case I would change Joe-2 to ask for an object in this
table to point to such a conceptual row (or 0).

Joe

---

From: Michael MacFaden <[email-masked]>
Date: Mon, 19 Aug 2013 14:09:49 -0700 (PDT)

----- Original Message -----
> I was referring to the virtual switch.

Ah, I was thinking it was the vid was installed into the GOS ethernet driver.
LoL.

>Again, I was aiming for more of a shortcut with data that

Right, definitely a shortcut for mib-browser users for sure.
Yet its more work for me to build the test to validate it when it changes!

> My corollary question would then be should there be a entLogicalTable 
> pointer in this table?

Not the entity mib in this case but a RowPointer to the vswitch's port could work.[1] 
The complexity here is that to get a VID, not of the VM, but of the 
uplink it connects to, that really depends on how the VM's virtual port
is wired. Some hypervisors might drop down to a real hardware switch, som
are virtual/software switch, some are just host Ethernet adapters.

RFC 2790/HOST-RESOURCES-MIB also comes to mind but its networking 
is limited to pointing to an ifIndex. In any case you raise a good
item for the list to push all sorts of possibilities.

Regards,
Mike MacFaden

[1] btw, the model I've been working with, which I think goes against current
standards thinking has vswitches that span hypervisors tied together with 
entries in entLogicalTable. These entries provide the uuid of the vswitch 
and the local ieee component id so mgmt apps can collect the entire state 
of a given vswitch across hypervisors by looking up the uuid then getting
the component id from which to poll ieee IEEE8021-BRIDGE-MIB, 
IEEE8012-Q-BRIDGE-MIB, IEEE8023-LAG-MIB

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 20 Aug 2013 08:37:54 +0200

> objects and the new values should not be persisted.  But that may be too

There is already text like this:

      Changes to this object may not persist across restarts of the
      hypervisor.

What is your proposal to make this clearer / stronger?
 
> Is there a strong push from operators to toggle these values via SNMP?

Remeber that this is a MAX-ACCESS. RFC 2578 section 7.3 says that it
'defines whether it makes "protocol sense" to read, write and/or
create an instance of the object, or to include its value in a
notification'. The compliance statement vmReadOnlyCompliances says
that write access is not required to be implemented. I believe this is
how we commonly do things in a MIB module - we allow read-write
implementations but we do not require read-write implementations.
Hence, I prefer to make no changes (except perhaps vmAutoStart but I
need to check whether there are hypervisors that actually allow to
change autostart behaviour of the running instance without touching
persistent config - this might actually be possible).

/js

---

From: Joe Marcus Clarke <[email-masked]>
Date: Wed, 21 Aug 2013 14:27:08 -0400

>> objects and the new values should not be persisted.  But that may be too

Sorry for the delay.  I've been in meetings this week.

> What is your proposal to make this clearer / stronger?

This leaves it open to one persisting it.  You could use normative
language and say, MUST NOT persist...

> persistent config - this might actually be possible).

That's fair.  I was not aware of the ability to adjust VM CPU and memory
on the fly without touching the config.  That said, I struggle to
understand the use case of doing this via SNMP versus a more reliable API.

I would also ask that some language be added to the compliance section
like you have in Section 3.2 about only implementing read-write if the
changes can be made dynamically and independent of the config.

Joe

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 22 Aug 2013 17:14:22 +0200

> understand the use case of doing this via SNMP versus a more reliable API.

In the MIB world, we have ways to express things in such a way that
people who want to implement a writable object can do so without
causing costs to those who prefer to not do so. This is why we have
MAX-ACCESS (what makes 'protocol sense' - means in principle this is a
writable object) and MIN-ACCESS (the minimal level of access - means
the minimum access one has to implement in order to comply to a
conformance definition).

/js

---

From: Michael MacFaden <[email-masked]>
Date: Thu, 22 Aug 2013 09:56:13 -0700 (PDT)

Juergen, co-authors and spectators,

Given our initial draft did not have writeable objects the
original name of this mib module name was appropriate:

                  'Virtual Machine Monitoring MIB'

Since the addition of configuration support, which I personally
do not think is necessary to standardize, it has gone from monitor 
to 'manage':

http://datatracker.ietf.org/doc/draft-asai-vmm-mib/

   1.  Introduction

   This document defines a portion of the Management Information Base
   (MIB) for use with network management protocols in the Internet
   community.  In particular, this specifies objects for managing  <===
   virtual machines controlled by a hypervisor (a.k.a. virtual machine
   managers).

I have the following proposal in order of my preference for your consideration:

1) Drop config objects from VMM-MIB
  My argument is that we are not standardizing industry practice.
  A counter argument would be to point to one fielded MIB module implementation 
  presently used in production a environment that can configure VMs today or have
  datacenter/network operators speak up about wanting this standardized.

2) Move any/all config objects to a second MIB module either in this draft or make another

In any case if read-write variables will be standarized, then let's update the
   title and textual matter where appropriate to make clear the increased scope.

Regards,
Mike MacFaden

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 22 Aug 2013 19:21:55 +0200

>    virtual machines controlled by a hypervisor (a.k.a. virtual machine

This is standard MIB boilerplate language. People probably read too
much into all this.
 
>   datacenter/network operators speak up about wanting this standardized.

There are no config objects.

>    title and textual matter where appropriate to make clear the increased scope.

Since there are no config objects, 2) is a mood point. ;-)

/js

---

From: Michael MacFaden <[email-masked]>
Date: Thu, 22 Aug 2013 10:34:37 -0700 (PDT)

> There are no config objects.

Right, there are no VM config objects, there are two config objects:

 -- Notification definitions:

   vmPerVMNotificationsEnabled OBJECT-TYPE
       SYNTAX       TruthValue
       MAX-ACCESS   read-write
       STATUS       current
       DESCRIPTION
               "Indicates if notification generator will send
               notifications per VM."
       ::= { vmObjects 9 }

   vmBulkNotificationsEnabled OBJECT-TYPE
       SYNTAX       TruthValue
       MAX-ACCESS   read-write
       STATUS       current
       DESCRIPTION
               "Indicates if notification generator will send
               notifications per set of VMs."
       ::= { vmObjects 10 }

Need more coffee this morning :-), I meant future edits to the mib module
in the resolution of Proposal: Joe-3,

I'm not interested in expanding the scope of the mib module to make MAX-ACCESS
anything more than read-write so again my proposals would be on direction
we'd take.

Mike

---

From: Tina TSOU <[email-masked]>
Date: Thu, 22 Aug 2013 22:16:39 +0000

Dear all,
I agree with Juergen's points below.

Thank you,
Tina

On Aug 22, 2013, at 10:22 AM, "Juergen Schoenwaelder" <[email-masked]> wrote:

> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

---

From: Joe Marcus Clarke <[email-masked]>
Date: Fri, 23 Aug 2013 14:56:49 -0400

> the minimum access one has to implement in order to comply to a

I understand MAX-ACCESS and MIN-ACCESS.  At this point I'm not arguing
the access type as you've pointed out this can be non-persistently
writable, and thus not a configuration object.

In this comment I was merely inquiring, mainly out of curiosity, about
the use case of an operator changing these values via SNMP vs. some
other API mechanism.  Meaning, today I see most MIB objects being
read-only and used for monitoring or data collection.  Writable objects
are typically being pushed to APIs such as NETCONF.  In _this_ case, I
just assumed operators would be more comfortable changing aspects of a
VM using a connection-oriented API vs. an SNMP SET.

My other comment (not quoted here) is that I was asking if some more
explicit text could be added to the compliance section of the MIB that
says these objects should be implemented read-write if they can be done
in a non-persistent way (essentially incorporating text from Section 3.2
into the MIB module itself).

Joe

---

From: Michael MacFaden <[email-masked]>
Date: Fri, 23 Aug 2013 14:01:24 -0700 (PDT)

Hi Joe,

Think there is agreement on the list that changes to the objects reported in VMM-MIB are not
coming in through SNMP.

If VMM-MIB has managed objects whose value are configured
via other means *and* the hypervisor in question separates 
out (in cisco parlance) running-config (in memory)
vs config (persisted) then what value will the agent report?

Thanks,
Mike

---

From: Joe Marcus Clarke <[email-masked]>
Date: Fri, 23 Aug 2013 17:31:47 -0400

> vs config (persisted) then what value will the agent report?

Operational status would be the way I would read it given the current
MIB definition.  Now, if the VM is shutdown, that operational state
would match the config state (for something like memory/CPU).

Joe

---

From: Hirochika Asai <[email-masked]>
Date: Sat, 24 Aug 2013 18:27:03 +0900

Thank you for your comments and I'm sorry for responding late.

> persistent config - this might actually be possible).

Now I've confirmed that vmAutoStart touches persistent config in libvirt.
I agree that vmAutoStart probably touches persistent config in the most
of implementations because it takes the effect at the next boot sequence
of the hypervisor.  If an implementation has one-shot config only for the
next boot, it should be another object.
Thus, I'll change the MAX-ACCESS of vmAutoStart to read-only.

Thank you.
Hirochika

-- 
Hirochika Asai <[email-masked]>, The University of Tokyo

---

From: Joe Marcus Clarke <[email-masked]>
Date: Sat, 24 Aug 2013 15:03:10 -0400

> Thus, I'll change the MAX-ACCESS of vmAutoStart to read-only.

Thanks.

Joe

---

From: Hirochika Asai <[email-masked]>
Date: Fri, 30 Aug 2013 10:17:28 +0900

Hi Joe,

Thank you for your comments.

I have a question on the management address.  Do you mean a
management interface that is attached to virtual machine as a
network interface and that cannot be distinguished as a
management interface from the hypervisor w/o VMware tools etc?
Or, do you mean a management interface provided by the hypervisor
for a virtual machine (let's call it a virtual IPMI here)?

If you mean the former, I think it is reasonable to include it in
entLogicalTable of ENTITY-MIB but not in the VMM-MIB .  However,
if you mean the latter, I think it should be included in the VMM-MIB.

There is a patch of the virtual IPMI for qemu though it haven't been
merged the trunk as follows:
https://lists.gnu.org/archive/html/qemu-devel/2012-07/msg02516.html
https://lists.gnu.org/archive/html/qemu-devel/2012-07/msg02525.html

Does anybody know such other implementations of the virtual IPMI?
If there are, we may need to take into account them.

Thank you,
Hirochika

---

From: Hirochika Asai <[email-masked]>
Date: Fri, 30 Aug 2013 10:35:00 +0900

Oops, I'm sorry that there was the answer earier in this thread.  Joe
mentioned the former, i.e., management IP addr of a VM.  So please
ignore my question to Joe.

But let me ask to the list about the virtual IPMI implementation.
> If there are, we may need to take into account them.
I couldn't find any other implementations of the virtual IPMI, so my
opinion is that we do not need an object related to the virtual IPMI
or something like it.  If you find any, I'd like you to let me know.

Thank you,
Hirochika

---

From: Michael MacFaden <[email-masked]>
Date: Thu, 29 Aug 2013 18:44:07 -0700 (PDT)

> or something like it.  If you find any, I'd like you to let me know.

I'm not aware of either a virtualized or just pass-through access to IPMI
hardware in VMware products or I'd have been happily usig it many times 
over to allow me to test ESXi SNMP agent RFC 2790 implementation in a VM[1]

If there are none, lets just make sure the MIB module doesn't preclude
adding that...

Regards,
Mike

[1] 
$ esxcfg-info | grep Product Name
      |----Product Name.............................................IBM System x3200 M2 -[4368AC1]-
# diagnostics report
$/bin/snmpd -S
INFO: fetching sensors...
record id: 8, length: 42, type: 2 entity: 7 instance: 1
record id: 9, length: 35, type: 2 entity: 7 instance: 1
record id: 14, length: 37, type: 2 entity: 3 instance: 1
record id: 15, length: 37, type: 2 entity: 20 instance: 1
record id: 17, length: 36, type: 2 entity: 23 instance: 1
record id: 18, length: 37, type: 2 entity: 7 instance: 1
record id: 20, length: 42, type: 2 entity: 7 instance: 1
record id: 21, length: 40, type: 2 entity: 7 instance: 1
record id: 22, length: 40, type: 2 entity: 7 instance: 1
record id: 29, length: 40, type: 2 entity: 23 instance: 1
record id: 34, length: 39, type: 2 entity: 3 instance: 1
record id: 36, length: 34, type: 2 entity: 7 instance: 1
record id: 60, length: 41, type: 2 entity: 4 instance: 2
record id: 61, length: 41, type: 2 entity: 4 instance: 1
record id: 62, length: 41, type: 2 entity: 4 instance: 4
record id: 63, length: 41, type: 2 entity: 4 instance: 5
record id: 64, length: 41, type: 2 entity: 4 instance: 6
record id: 65, length: 41, type: 2 entity: 4 instance: 7
record id: 66, length: 41, type: 2 entity: 4 instance: 8
record id: 74, length: 41, type: 2 entity: 4 instance: 3
record id: 75, length: 38, type: 2 entity: 21 instance: 1
record id: 76, length: 38, type: 2 entity: 21 instance: 2
record id: 77, length: 41, type: 2 entity: 21 instance: 3
record id: 78, length: 41, type: 2 entity: 21 instance: 4
record id: 79, length: 38, type: 2 entity: 21 instance: 5
record id: 80, length: 37, type: 2 entity: 0 instance: 1
INFO: sensor sweep completed, using total of 11 sensor entries in sdr
[0] name: VRD Status, inst: 1, type: POWER, status: D_Normal, errors: 0
[1] name: PS 1 Status, inst: 1, type: POWER, status: D_Normal, errors: 0
[2] name: PS 2 Status, inst: 2, type: POWER, status: D_Normal, errors: 0
[3] name: PS Overspec, inst: 5, type: POWER, status: D_Normal, errors: 0
[4] name: Fan 1 Tach, inst: 1, type: FAN, status: D_Normal, errors: 0
[5] name: Fan 2 Tach, inst: 2, type: FAN, status: D_Normal, errors: 0
[6] name: Fan 3 Tach, inst: 3, type: FAN, status: D_Normal, errors: 0
[7] name: PS 1 Fan Fault, inst: 3, type: FAN, status: D_Normal, errors: 0
[8] name: PS 2 Fan Fault, inst: 4, type: FAN, status: D_Normal, errors: 0
[9] name: CPU Status, inst: 1, type: CPU, status: D_Normal, errors: 0
[10] name: CPU OverTemp, inst: 1, type: CPU, status: D_Normal, errors: 0
INFO: report complete

---

From: Hirochika Asai <[email-masked]>
Date: Fri, 30 Aug 2013 14:34:47 +0900

Yes, I agree that the MIB module shouldn't preclude adding any functionalities.
So we'd better not to have the virtual IPMI object in the MIB in order not to
prevent adding them from developers.

Best regards,
Hirochika

---

From: Michael MacFaden <[email-masked]>
Date: Thu, 29 Aug 2013 22:40:22 -0700 (PDT)

> prevent adding them from developers.

I agree.

Mike