Subject: [OPSAWG] OPS-DIR review of draft-ietf-opsawg-vmm-mib-02

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 14 May 2015 16:45:19 +0000

I have reviewed this document as part of the Operational directorate's
ongoing effort to review all IETF documents being processed by the IESG.  These
comments were written with the intent of improving the operational aspects of the
IETF drafts. Comments that are not addressed in last call may be included in AD reviews
during the IESG review.  Document editors and WG chairs should treat these comments
just like any other last call comments.

Document Status: Ready with Issues

This is a MIB document - however my review focuses on the perspective of an operational and manageability review. I assume that a separate RFC 4181 review is or will be performed.

5706 check list:

A.1.  Operational Considerations

   1.  Has deployment been discussed?  See Section 2.1.

       *  Does the document include a description of how this protocol

          or technology is going to be deployed and managed?

       *  Is the proposed specification deployable?  If not, how could

          it be improved?

       *  Does the solution scale well from the operational and

          management perspective?  Does the proposed approach have any

          scaling issues that could affect usability for large-scale

          operation?

       *  Are there any coexistence issues?

DR: there is some discussion about deployment, especially in relation to the read-only vs. write capabilities.

   2.  Has installation and initial setup been discussed?  See

       Section 2.2.

       *  Is the solution sufficiently configurable?

       *  Are configuration parameters clearly identified?

       *  Are configuration parameters normalized?

       *  Does each configuration parameter have a reasonable default

          value?

       *  Will configuration be pushed to a device by a configuration

          manager, or pulled by a device from a configuration server?

       *  How will the devices and managers find and authenticate each

          other?

DR: no, but I do not believe that there are any special or different problems than the ones encountered when installing any new SNMP agent.

   3.  Has the migration path been discussed?  See Section 2.3.

       *  Are there any backward compatibility issues?

DR: N/A - this is the first version of this MIB module. Relationship with other MIB modules is discussed.

   4.  Have the Requirements on other protocols and functional

       components been discussed?  See Section 2.4.

     *  What protocol operations are expected to be performed relative

          to the new protocol or technology, and what protocols and data

          models are expected to be in place or recommended to ensure

          for interoperable management?

DR: N/A

   5.  Has the impact on network operation been discussed?  See

       Section 2.5.

       *  Will the new protocol significantly increase traffic load on

          existing networks?

       *  Will the proposed management for the new protocol

          significantly increase traffic load on existing networks?

       *  How will the new protocol impact the behavior of other

          protocols in the network?  Will it impact performance (e.g.,

          jitter) of certain types of applications running in the same

          network?

       *  Does the new protocol need supporting services (e.g., DNS or

          Authentication, Authorization, and Accounting - AAA) added to

          an existing network?

DR: the number of notifications issued by the agent may be a problem and the tuning is quite course. See the more detailed comments.

   6.  Have suggestions for verifying correct operation been discussed?

       See Section 2.6.

       *  How can one test end-to-end connectivity and throughput?

       *  Which metrics are of interest?

       *  Will testing have an impact on the protocol or the network?

DR: N/A

   7.  Has management interoperability been discussed?  See Section 3.1.

       *  Is a standard protocol needed for interoperable management?

       *  Is a standard information or data model needed to make

          properties comparable across devices from different vendors?

DR: this is a standard data model, relationship with other MIB modules is discussed

   8.  Are there fault or threshold conditions that should be reported?

       See Section 3.3.

       *  Does specific management information have time utility?

       *  Should the information be reported by notifications?  Polling?

          Event-driven polling?

       *  Is notification throttling discussed?

*  Is there support for saving state that could be used for root

          cause analysis?

DR: standard MIB implementation conditions apply (like system time for time object). As in any SNMP agent implementations the operators strategy may be notifications-triggered polling, but this is not explicitly mentioned. Notification throttling is not applied, and it may be useful - see comment #18

   9.  Is configuration discussed?  See Section 3.4.

       *  Are configuration defaults and default modes of operation

          considered?

       *  Is there discussion of what information should be preserved

          across reboots of the device or the management system?  Can

          devices realistically preserve this information through hard

          reboots where physical configuration might change (e.g., cards

          might be swapped while a chassis is powered down)?

DR: This is a mostly read-only MIB module. Two objects are defined to enable/disable notifications, the compliance section does not make them mandatory

A.2.  Management Considerations

   Do you anticipate any manageability issues with the specification?

   1.  Is management interoperability discussed?  See Section 3.1.

       *  Will it use centralized or distributed management?

       *  Will it require remote and/or local management applications?

       *  Are textual or graphical user interfaces required?

       *  Is textual or binary format for management information

          preferred?

DR: remote management is possible, this MIB module makes possible interoperable management

   2.  Is management information discussed?  See Section 3.2.

       *  What is the minimal set of management (configuration, faults,

          performance monitoring) objects that need to be instrumented

          in order to manage the new protocol?

DR: yes, this is a MIB specification

   3.  Is fault management discussed?  See Section 3.3.

       *  Is Liveness Detection and Monitoring discussed?

       *  Does the solution have failure modes that are difficult to

          diagnose or correct?  Are faults and alarms reported and

          logged?

DR: Yes

   4.  Is configuration management discussed?  See Section 3.4.

       *  Is protocol state information exposed to the user?  How?  Are

          significant state transitions logged?

DR: Yes, configuration management is discussed, but only read operations are possible with the exception of enabling and disabling notifications.

5.  Is accounting management discussed?  See Section 3.5.

DR: N/A

   6.  Is performance management discussed?  See Section 3.6.

       *  Does the protocol have an impact on network traffic and

          network devices?  Can performance be measured?

       *  Is protocol performance information exposed to the user?

DR: Yes

   7.  Is security management discussed?  See Section 3.7.

       *  Does the specification discuss how to manage aspects of

          security, such as access controls, managing key distribution,

          etc.
DR: Yes, see detailed comments

A few more detailed comments:

1.       Page 5: s/the MIB objects are managed at a hypervisor/ the MIB objects are managed at the hypervisor/

2.       Page 6: s/the relationship between other MIB modules/ s/the relationship with other MIB modules/

3.       Page 8: figure 2 - at the top did you mean 'Notation' rather than 'Notification'?

4.       Same - in general this figure does not really help. It is not clear how are the Blocked and Crashed states linked to the rest. I would at least add the numbers of the states in the VirtualMachineOperState TC

5.       Page 9: The statement about the MIB module providing writeable objects that can be used for non-persistent changes like changing memory allocation or CPU allocation seems to be a leftover from previous versions - in any case it is not true any longer for this version of the MIB module.

6.       I frankly do not know if the OID tree structure that spreads in pages 9 to 11 is useful at all. Who needs it? And if anybody needs this, it can be obtained from any MIB tool. I believe that a short list of the MIB groups and tables and a few words about what each of these does is sufficient.

7.       Page 12: I find the second paragraph in section 5 confusing - maybe it's a language issue. The last sentence for example does not makes sense (what does 'the objects of this information' mean?)

8.       Same about the 4th paragraph in the same section and page. What does the first sentence mean? ('the objects ... are not also included')

9.       Page 15:  the explanation of the destroy(5) value in the VirtualMachineAdminState TC may confuse. We do not allow changing VM admin states by write operations. Of course, an agent implementing this MIB module may read this state and reflect it in an object that uses this SYNTAX. I believe that some text about this usage of the TC needs to be added.

10.   Page 20: is there any reason that the TC VirtualMachineStorageAccess has no 'unknown' enumeration defined?

11.   Page 20-21: Why is the VirtualMachineStorageType limited to two media types only? Would not an IANA-administered TC be more appropriate here?

12.   Page 24: It is not clear to me what is an 'administered region' mentioned in the DESCRIPTION clause of the vmUUID object

13.   Page 25: why 'power' in the DESCRIPTION of the vmAdminState?

14.   Page 26: it looks like the intent of the vmMemUnit is to use bytes as unit of expressing memory size - this should be explicitly mentioned

15.   Page 33: same comment relative to vmStorageSizeUnit

16.   Page 34-35: I have a big question mark about how the vmStorageReadLatency and vmStorageWriteLatency objects need to be used. First the usage of the Counter64 syntax seems odd as these objects do not count anything and their increase is not unitary. Second I do not know how to interpret them. What does say a reading of 'a million' or of 'a billion' mean? It really makes no sense if not interpreted in conjunction with the number of respective I/O operations, media type, etc. I guess that the implementation of these objects is not trivial and probably requires HW support taking into consideration the microseconds resolution. At a minimum I guess that some text recommending to the operators how they are supposed to be used is needed.

17.   I am concerned that the current way of defining the notifications switches is too course. Hypervisors may have many VMs in charge, and if each generates one notification per each state changes the numbers can become big even in normal operation. Maybe some throttling mechanism would be useful. Or maybe a couple of more switches that allow to enable only 'critical' notifications (e.g. vmCrashed).

18.   The Security Considerations section does not include a description of the security hazards of mis-configuration of the writeable objects (danger of flooding the network with unwanted notifications)

Thanks and Regards,

Dan

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 23 May 2015 08:37:16 +0200

On Thu, May 14, 2015 at 04:45:19PM +0000, Romascanu, Dan (Dan) wrote:

[...]

Dan,

many thanks for the review. I am skipping over the smaller nits.

> 16.  Page 34-35: I have a big question mark about how the
> vmStorageReadLatency and vmStorageWriteLatency objects need to be
> used. First the usage of the Counter64 syntax seems odd as these
> objects do not count anything and their increase is not
> unitary. Second I do not know how to interpret them. What does say a
> reading of 'a million' or of 'a billion' mean? It really makes no
> sense if not interpreted in conjunction with the number of
> respective I/O operations, media type, etc. I guess that the
> implementation of these objects is not trivial and probably requires
> HW support taking into consideration the microseconds resolution. At
> a minimum I guess that some text recommending to the operators how
> they are supposed to be used is needed.

The only SMIv2 requirement for a Counter64 is that it increases
monotonically which is the case here. There is no requirement that a
counter increases unitary. By making this a counter, it is clear that
one has to to look at the first derivative, that is the change of the
counter over time. Obviously, on a lightly loaded hypervisor, the
latency should be small but in situations where it increases
significantly, you are likely hitting a slowdown of your VMs due to
I/O contention instead of CPU limits. These objects are there to
identify such situations (and the virtual storage devices involved).
 
> 17.  I am concerned that the current way of defining the
> notifications switches is too course. Hypervisors may have many VMs
> in charge, and if each generates one notification per each state
> changes the numbers can become big even in normal operation. Maybe
> some throttling mechanism would be useful. Or maybe a couple of more
> switches that allow to enable only 'critical' notifications
> (e.g. vmCrashed).

The only situation I can think of where the number of notifications
will be significant is during restart of a whole rack with many
hypervisors inside. But even then, things usually take time. (I think
our Xen hypervisors actually create virtual machines sequentially and
hence the notifications get actually spaced over time.) It is not
unlikely that the operating systems inside the hypervisors during
startup generate significantly more network traffic compared to a few
hypervisor notifications. That said, during the development of the MIB
module we moved from generic state change notifications to a set of
specific notifications and this allows to use SNMPv3 notification
filtering to filter out notifications people find not useful.

> 18.   The Security Considerations section does not include a description of the security hazards of mis-configuration of the writeable objects (danger of flooding the network with unwanted notifica…

Perhaps text can be added but I am not sure I see that a data center
network will experience any significant load due to these notifications.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Keiichi SHIMA <[email-masked]>
Date: Tue, 26 May 2015 10:45:48 +0900

Hi all,

> filtering to filter out notifications people find not useful.

I and Asai talked locally and we agree Juergen that the number of notification events is manageable.

However we noticed that the 'vmBlocked' notification may be a problem.  The 'blocked' state is defined as 'The operational state of the virtual machine indicating the execution of the virtual machine is currently blocked, e.g., waiting for some action of the hypervisor to finish.  This is a transient state from/to other states.'  This state transition event may appear more frequently than other events, since the state is caused by the I/O scheduler of the hyper visor implementation.

We think we can simply remove the 'vmBlocked' notification.  The 'blocked' state is a transient state and typically return to the previous state immediately (once the pending I/O requests completed).

---
Keiichi SHIMA (島 慶一)
WIDE project <[email-masked]>
Research Laboratory, IIJ Innovation Institute, Inc <[email-masked]>

> On 2015/05/23, at 15:37, Juergen Schoenwaelder <[email-masked]> wrote:
> 
> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 26 May 2015 08:47:13 +0200

> We think we can simply remove the 'vmBlocked' notification.  The 'blocked' state is a transient state and typically return to the previous state immediately (once the pending I/O requests completed…

Yes, this makes sense to me.

/js

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 26 May 2015 07:23:17 +0000

Hi,

I would still recommend that you add text in the Security Considerations section in which you describe the risk of intentional or unintentional misconfiguration of the two writeable objects in the MIB module. It would also be useful to mention explicitly that the authors considered the risks of multiplication of notification and evaluated it as manageable for all reasonably designed and sized networks. 

Thanks and Regards,

Dan

> -----Original Message-----
> From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-
> university.de]
> Sent: Tuesday, May 26, 2015 9:47 AM
> To: Keiichi SHIMA
> Cc: Romascanu, Dan (Dan); [email-masked];
> [email-masked]; [email-masked]
> Subject: Re: [OPSAWG] OPS-DIR review of draft-ietf-opsawg-vmm-mib-02
> 
> > >> some throttling mechanism would be useful. Or maybe a couple of
> > >> more switches that allow to enable only 'critical' notifications
> > >> (e.g. vmCrashed).
> > >
> > > hypervisors inside. But even then, things usually take time. (I
> > > think our Xen hypervisors actually create virtual machines
> > > sequentially and hence the notifications get actually spaced over
> > > time.) It is not unlikely that the operating systems inside the
> > > hypervisors during startup generate significantly more network
> > > traffic compared to a few hypervisor notifications. That said,
> > > during the development of the MIB module we moved from generic
> state
> > > change notifications to a set of specific notifications and this
> > > allows to use SNMPv3 notification filtering to filter out notifications
> people find not useful.
> >
> >
> > However we noticed that the 'vmBlocked' notification may be a problem.
> The 'blocked' state is defined as 'The operational state of the virtual machine
> indicating the execution of the virtual machine is currently blocked, e.g.,
> waiting for some action of the hypervisor to finish.  This is a transient state
> from/to other states.'  This state transition event may appear more
> frequently than other events, since the state is caused by the I/O scheduler
> of the hyper visor implementation.
> >
> > We think we can simply remove the 'vmBlocked' notification.  The 'blocked'
> state is a transient state and typically return to the previous state
> immediately (once the pending I/O requests completed).
> >
> 
> Fax:   +49 421 200 3103
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.jacobs-
> 2Duniversity.de_&d=AwIBAg&c=BFpWQw8bsuKpl1SgiZH64Q&r=I4dzGxR31O
> cNXCJfQzvlsiLQfucBXRucPvdrphpBsFA&m=MgxJDgRk1TpLrZ2hiPqtqSLH_gUN
> Xo4hIzz7Czjm6c0&s=0FOhbeoR7FEAGZddNHtPDimJFq1MNfZlAEOjldkr1tA&e
> = >

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 26 May 2015 10:52:36 +0200

Keiichi,

note that there is outdated text on page 9 that should be updated:

   The MIB module provides a few writable objects that can be used to
   make non-persistent changes, e.g., changing the memory allocation or
   the CPU allocation.  It is not the goal of this MIB module to provide
   a configuration interface for virtual machines since other protocols
   and data modeling languages are more suitable for this task.

In the security considerations, I assume Dan wants:

OLD

   There are two objects defined in this MIB,
   vmPerVMNotificationsEnabled and vmBulkNotificationsEnabled, that have
   a MAX-ACCESS clause of read-write.  Such objects may be considered
   sensitive or vulnerable in some network environments.

NEW

   There are two objects defined in this MIB,
   vmPerVMNotificationsEnabled and vmBulkNotificationsEnabled, that
   have a MAX-ACCESS clause of read-write. Enabling notifications can
   lead to a noticeable number of notifications if many virtual
   machines change their state concurrently. Hence, such objects may
   be considered sensitive or vulnerable in some network environments.

/js

---

From: Keiichi SHIMA <[email-masked]>
Date: Tue, 26 May 2015 18:00:36 +0900

Juergen, thanks,

I'm now working on updating the text with Hirochika.

>   and data modeling languages are more suitable for this task.

OK, we've removed the above paragraph.  The latest text in the repository does't have it.

>   be considered sensitive or vulnerable in some network environments.

OK, the above text has been merged.

---
Keiichi SHIMA (島 慶一)

> On 2015/05/26, at 17:52, Juergen Schoenwaelder <[email-masked]> wrote:
> 
> Keiichi,
>

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 26 May 2015 09:04:18 +0000

Yes - this addresses my concern. 

Thanks and Regards,

Dan

> Sent: Tuesday, May 26, 2015 11:53 AM
> To: Romascanu, Dan (Dan)
> Cc: Keiichi SHIMA; [email-masked]; ops-
> [email-masked]; [email-masked]
> > I would still recommend that you add text in the Security Considerations
> section in which you describe the risk of intentional or unintentional
> misconfiguration of the two writeable objects in the MIB module. It would
> also be useful to mention explicitly that the authors considered the risks of
> multiplication of notification and evaluated it as manageable for all
> reasonably designed and sized networks.
> >
> > Thanks and Regards,
> >
> > Dan
> >
> >
> > > Cc: Romascanu, Dan (Dan);
> > > [email-masked];
> > > > >> notifications switches is too course. Hypervisors may have many
> > > > >> VMs in charge, and if each generates one notification per each
> > > > >> state changes the numbers can become big even in normal
> > > > >> operation. Maybe some throttling mechanism would be useful. Or
> > > > >> maybe a couple of more switches that allow to enable only
> > > > >> 'critical' notifications (e.g. vmCrashed).
> > > > >
> > > > > The only situation I can think of where the number of
> > > > > notifications will be significant is during restart of a whole
> > > > > rack with many hypervisors inside. But even then, things usually
> > > > > take time. (I think our Xen hypervisors actually create virtual
> > > > > machines sequentially and hence the notifications get actually
> > > > > spaced over
> > > The 'blocked' state is defined as 'The operational state of the
> > > virtual machine indicating the execution of the virtual machine is
> > > currently blocked, e.g., waiting for some action of the hypervisor
> > > to finish.  This is a transient state from/to other states.'  This
> > > state transition event may appear more frequently than other events,
> > > since the state is caused by the I/O scheduler of the hyper visor
> implementation.
> > > >
> > > state is a transient state and typically return to the previous
> > > state immediately (once the pending I/O requests completed).
> > > >
> > >
> cNXCJfQzvlsiLQfucBXRucPvdrphpBsFA&m=qQg5JHHnGYDVuGEwHHPY2bhg
> mu5R0g-
> xo8jGJBsC_nI&s=A7HwS6XPjZEsREWKTr7TANnfo2PZlxIg2bhFUlPeZUw&e= >

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 26 May 2015 10:17:11 +0000

Dear Hirochita Asai,

Thank you for your response and for addressing the comments in my review. 

Most of the proposals you made are fully addressing my comments. 

A couple of open issues are left. See below.

Regards,

Dan

> From: Hirochika Asai [mailto:[email-masked]]
> Sent: Tuesday, May 26, 2015 1:00 PM
> To: Romascanu, Dan (Dan)
> Cc: [email-masked]; [email-masked];
> [email-masked]
> Dear Dan,
> > 11.   Page 20-21: Why is the VirtualMachineStorageType limited to two
> media types only? Would not an IANA-administered TC be more appropriate
> here?
> Although few people are using, the type of floppy disk was missing.  Then we
> added it.
> As for the IANA-administrated TC, if I understand correctly, StorageType in
> SNMPv2-TC defines the storage types. However, they are not what we
> expect as the media types of virtual storages attached to a virtual machine.
> 

Two issues here: 
- maybe VirtualMachineStorageMediaType is a better name for the TC? 
- my suggestion to make the TC IANA-administered would allow for new media types to be added in the future without the need to modify the core MIB module. Hopefully, this MIB module will be used many years ahead and you cannot predict in advance what new media types will be invented or become popular in the future. 

> > 12.   Page 24: It is not clear to me what is an ‘administered region’
> mentioned in the DESCRIPTION clause of the vmUUID object
> We would like to mean an “administrative domain”.  Does this make sense?
> 

Yes.

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 26 May 2015 11:07:00 +0000

Dear Hirochita Asai,

Exactly, with two observations.

1. you need to prefix the new MIB module with IANA
2. You also need to add text about expanding the TC in the  IANA considerations section. 

(see RFC 6933 as a working example)

Thanks and Regards,

Dan

> Sent: Tuesday, May 26, 2015 2:00 PM
> To: Romascanu, Dan (Dan)
> Thank you.
> >> media types only? Would not an IANA-administered TC be more
> >> appropriate here?
> >> Although few people are using, the type of floppy disk was missing.
> >> Then we added it.
> >> As for the IANA-administrated TC, if I understand correctly,
> >> StorageType in SNMPv2-TC defines the storage types. However, they are
> >> not what we expect as the media types of virtual storages attached to a
> virtual machine.
> > Two issues here:
> It has already been the name VirtualMachineStorageMediaType.
> 
> > - my suggestion to make the TC IANA-administered would allow for new
> media types to be added in the future without the need to modify the core
> MIB module. Hopefully, this MIB module will be used many years ahead and
> you cannot predict in advance what new media types will be invented or
> become popular in the future.
> 
> Do you recommend to separate VirtualMachineStorageMediaType from VM-
> MIB like the following?
> (Please find the attached for the revised draft.)
> 
> NEW:
> 
> VM-STORAGE-TC-MIB DEFINITIONS ::= BEGIN
> 
>    IMPORTS
>        MODULE-IDENTITY, mib-2
>            FROM SNMPv2-SMI
>        TEXTUAL-CONVENTION
>            FROM SNMPv2-TC;
> (snip)
> VirtualMachineStorageMediaType ::= TEXTUAL-CONVENTION
> (snip)
> END
> 
> VM-MIB DEFINITIONS ::= BEGIN
> (snip)
> 
> 
> Best,
> Hirochika Asai
> 
> --
> Hirochika Asai <[email-masked]>, The University of Tokyo