Subject: [OPSAWG] VMM-MIB: question about vmAdminState values

From: Keiichi SHIMA <[email-masked]>
Date: Sat, 24 Aug 2013 20:52:47 +0900

Hi,

Let me ask one question about vmAdminState values.

In the latest VMM-MIB draft (http://www.ietf.org/id/draft-asai-vmm-mib-04.txt), the values for vmAdminState are defined as below.

INTEGER {
                       unknown(1),
                       other(2),
                       preparing(3),
                       running(4),
                       blocked(5),
                       suspending(6),
                       suspended(7),
                       resuming(8),
                       paused(9),
                       migrating(10),
                       shuttingdown(11),
                       shutdown(12),
                       crashed(13)
                    }

Also, how the state transits is shown in Figure 2.

However, in the above description, the 'blocked' state is quite unclear to me.

What precisely does the 'blocked' state mean?  When does a virtual machine transit to the 'blocked' state?  I think we need more text about the state, if we want to keep the state.

Regards,
---
Keiichi SHIMA (島 慶一)
WIDE project <[email-masked]>
Research Laboratory, IIJ Innovation Institute, Inc <[email-masked]>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 26 Aug 2013 12:59:55 +0200

> In the latest VMM-MIB draft (http://www.ietf.org/id/draft-asai-vmm-mib-04.txt), the values for vmAdminState are defined as below.

You are actually asking a question about VirtualMachineOperState...

> However, in the above description, the 'blocked' state is quite unclear to me.

The current description is this:

               blocked(5)     The virtual machine is currently blocked.    
 
> What precisely does the 'blocked' state mean?  When does a virtual
> machine transit to the 'blocked' state?  I think we need more text
> about the state, if we want to keep the state.

The blocked state covers situations where a virtual machine is blocked
because it waits for certain service requests to complete. For
example, if you use Xen, then a DomU delegates most of the low-level
input/output to Dom0 and while waiting for Dom0 to finish processing a
request, the DomU issuing the request is 'blocked'.

Would this clarification help:

             blocked(5)     The virtual machine is currently blocked
	     		    waiting for example for some action of 
                            the hypervisor to finish.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: SHIMA Keiichi <[email-masked]>
Date: Thu, 29 Aug 2013 19:27:40 +0900

Hello,

> You are actually asking a question about VirtualMachineOperState…

Ah, yes, you are right.  My fault.

> request, the DomU issuing the request is 'blocked'.

OK, thank you, now the meaning of the blocked state is much clearer to me than before.

Then, my following question is that, do we need the state?  I wonder how the state is used in a real operation.

It seems that the blocked state is a kind of temporal/short-lived state caused by the scheduling process of virtual machine entities running in a hypervisor.  It is sure that we will have a blocked state, but it won't last long.  Are we trying to catch that short-lived state using this VMM-MIB?

Do we have any usage scenario of the blocked state?

Regards,
---
Keiichi SHIMA (島 慶一)

On 2013/08/26, at 19:59, Juergen Schoenwaelder <[email-masked]> wrote:

> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 29 Aug 2013 12:59:50 +0200

On Thu, Aug 29, 2013 at 07:27:40PM +0900, SHIMA Keiichi wrote:

[...]
 
> Do we have any usage scenario of the blocked state?

I thing this is backwards. I like my SNMP agent to access the current
state and report it. It does not matter whether I believe the state is
short lived or not. I like to see the actual state - not a translation
of it. The worst experience when I debug problems is when software
tries to hide reality. And, btw, being blocked is really not uncommon
in a Xen setup.

$ sudo xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  4160     8     r----- 267537.1
foo                                          3   512     1     -b----  39148.0
bar                                         17  2048     1     -b----  71467.1
baz                                         20  2048     1     -b----  80750.0
bing                                         6  1024     1     -b----  66408.8
bong                                         8   256     1     -b----  57365.8
bang                                        21  1024     1     -b----  36534.1
yum                                         10   256     1     -b----  41611.5
yam                                         15  1024     1     -b----  49363.2
yim                                         16  2048     1     -b----  44081.6

(Taken from a live machine with names obfuscated.)

/js

---

From: Joe Marcus Clarke <[email-masked]>
Date: Thu, 29 Aug 2013 11:06:04 -0400

>                             the hypervisor to finish.

This reminds me of another problem I had initially noticed, but forgot
to raise.  That is, the use of the state name to define the state.  I
think what you have written here is good, but is there another way to
say "blocked" that would define it more clearly?

I have the same comment for just about all states.  Their definition
contains the state name.  If those descriptions were more descriptive,
it might short-circuit emails like this in the future.

Joe

-- 
Joe Marcus Clarke, CCIE #5384,         |          |
SCJP, SCSA, SCNA, SCSECA, VCP        |||||      |||||
Distinguished Services Engineer ..:|||||||||::|||||||||:..
Phone: +1 (919) 392-2867         c i s c o  S y s t e m s
Email: [email-masked]

----------------------------------------------------------------------------

---

From: SHIMA Keiichi <[email-masked]>
Date: Fri, 30 Aug 2013 13:00:28 +0900

Hello,

On 2013/08/29, at 19:59, Juergen Schoenwaelder <[email-masked]> wrote:

> of it. 

I understand that you want to see the 'raw' state of virtual machines.  I agree with you as an engineer :)
But as an operator, I tend to think we need some kind of 'translation', or in other words, higher level state information here.

My previous wording of 'short-lived' might not be a proper word.  Let me try to make myself clearer.

Maybe my question is that whether the 'blocked' state is a primary state or not, in the sense of virtual machine operation.  I think we need to have some abstraction here.  How it is implemented is a different discussion.

Well, we may provide the implementation specific state to MIB users, but it is different level of information.  I'm not saying that we should not provide any detailed status information through VMM-MIB.  For example, in the KVM environment, we may want to know the process state information of a running 'kvm' process as a detailed state information.

What I want say here is that we probably need 'different' level of information, or a model of virtual machine operation.  It seems to me that the 'blocked' state is not a primary state.  It is rather a kind of a sub-state of primary states, or an attribute of primary states (e.g. a VM is running (and blocked)).

I might think that the name of the 'running' state is misleading.  If I rephrase the 'running' state, for example, to 'in-operation' state, does my opinion become clearer?

I may be wrong.  Any comments are welcome :)

---
Keiichi SHIMA (島 慶一)

---

From: Michael MacFaden <[email-masked]>
Date: Thu, 29 Aug 2013 22:02:50 -0700 (PDT)

> Maybe my question is that whether the 'blocked' state is a primary state or
> not, in the sense of virtual machine operation.  I think we need to have
> some abstraction here.  How it is implemented is a different discussion.

Yes.

> What I want say here is that we probably need 'different' level of
> information, or a model of virtual machine operation.  It seems to me that
> the 'blocked' state is not a primary state.  It is rather a kind of a
> sub-state of primary states, or an attribute of primary states (e.g. a VM is
> running (and blocked)).

The key thing in any standard is that different folks read the standard
and implement it. We then have a bakeoff/implementation report and see if
we got it right nor not.

I haven't put my two cents into the state machine yet either. And to me
blocked given what is written so far is quite generic.

Question is: what would an operator do if they saw a vm in blocked state?

Here's some blocked states in VMware's implemtation.

1) Operator moved a vm fileset (vmx config file, vmdk virtual disks from 
a usb to a new machine and powered on. vmx process will block waiting for
a connection to the vmx to answer the question (did you move or copy the vm?)
It then updates the vmx config file appropriately. There are quite a number
of "questions" that can block a VM.

2) VM can block operation during hot add/delete of devices, when doing
live migration (vmotion (tm)) and so forth.

Again is blocked something the operator has to deal with or not?

Mike MacFaden Staff Engineer R&D Apps VMware, Inc Palo Alto CA

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 30 Aug 2013 07:16:59 +0200

> I might think that the name of the 'running' state is misleading.  If I rephrase the 'running' state, for example, to 'in-operation' state, does my opinion become clearer?

I do not get your point, probably because of a lack of definition of
'primary state'. ;-)

Perhaps you want to write management software that provides other
abstractions on top of what the various MIB modules provide - thats
fine. I doubt the SNMP agent is the right place to provide
abstractions though. And for the KVM case, please note that the
process state is already available from other existing MIB modules.

I repeat myself: For troubleshooting purposes, you want the real state
and not a classification of states - which may be fuzzy. Or more
precisely, whether a blocked VM is good or bad depends on the context.
Requiring the SNMP agent to figure out the context correctly and to
classify the state accordingly is adding significant complexity and
agent code is much harder to change and adapt than management software.

/js

---

From: SHIMA Keiichi <[email-masked]>
Date: Fri, 30 Aug 2013 18:07:54 +0900

Hello,

> of "questions" that can block a VM.

Thank you for showing some examples in VMware.  Since I'm not very familiar with it, the above information is quite useful to me.

And, the above examples are cases what I was not thinking about.  They are events caused by procedures of virtual machine 'operation'.  I tend to agree that the VirtualMachineOperState should be able to distinguish the above situations…

I still think the above situations and the blocked state of Xen (caused by I/O wait between DomU and Dom0, which is not cased by operator's actions) are a bit different context.  In the current VMM-MIB draft though, both situations are marked as blocked.  But I'm not sure it is a correct design or not.

I noticed that I have one assumption that we haven't shared among us.  In my understanding, the state transition is initiated by operation actions (except the crash case).  So, the transition to/from the blocked state looks a bit strange to me.  But Juergen doesn't have such an assumption.  That is the reason of the gap between I and others, maybe.

> live migration (vmotion (tm)) and so forth.

This case is also a confusing case to me.  In the current draft, when a VM is migrated, its state is changed to the 'migrating' state.  But in the VMware case, the state is blocked (or migrating & blocked, maybe).

It may be necessary to review and refine the meanings of VirtualMachineOperState and the model of the state machine more carefully.  Though I'm not sure if it must be done in the current version.  Or maybe in the next major revision based on inputs from implementors and operators.

Regards,
---
Keiichi SHIMA (島 慶一)

---

From: SHIMA Keiichi <[email-masked]>
Date: Fri, 30 Aug 2013 18:08:10 +0900

Hello,

Thank you for your comments.

On 2013/08/30, at 14:16, Juergen Schoenwaelder <[email-masked]> wrote:

> abstractions though.

So, you are saying that the MIB should provide a kind of raw data, and how it is translated (or understood) should be done in upper layer applications.  Is my understanding correct?

Yes, I must agree with you.

As I mentioned in my previous mail, my understanding of VirtualMachineOperState and yours is a bit different, I guess.  I was thinking the state transition is initiated by an operator, and the next state is determined by the result of the operator's action.   But you are thinking that VirtualMachineOperState is also changed by other factors (like I/O blocking in a hypervisor level).

Once we have a concrete definition of VirtualMachineOperState, maybe my question will be solved.

> And for the KVM case, please note that the

Ok, thank you for this information!

> agent code is much harder to change and adapt than management software.

I understand your concern.

I am not saying we should not provide real (or detailed) state information.  It is certainly useful to get state information as is.

As I mentioned above, I believe that the definition of the meaning of VirtualMachineOperState is the key to conclude this discussion.  In section 3.2, there is a text about how the vmOperState changes.

"The transition
   of `vmOperState' by the write access to `vmAdminState' and the
   notifications generated by the operational state changes are
   summarized in Figure 2."

If I'm not missing, there is no other explanation about how the vmOperState should behave.  Maybe we need more concrete description in the vmOperState section.  Currently, it just says as below.

"The current operational state of the virtual machine."

Also, if we keep the blocked state and the state machine figure, then it would be better to provide a complete state transition table.  The current figure-style state machine lacks transition flows between the blocked state and other states.  I don't want to draw that part in the figure, since it will mess up the figure :)  How about adding a separate table in a appendix section?

Regards,
---
Keiichi SHIMA (島 慶一)

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 30 Aug 2013 11:29:58 +0200

> "The current operational state of the virtual machine."

I think this is pretty clear - it is the _current_ _operational_ state
and indeed a virtual machine may cycle between states or it may go
through a sequence of state changes while carrying out a complex
administrative action like migrate from hypervisor A to B. The MIB
module is not modeling such administrative actions - its aim is only
to report what is actually happening for monitoring or troubleshooting
purposes. In other words, if you instruct your hypervisor controller
to migrate a virtual machine from hypervisor A to B, then the MIB
module allows you to monitor how this migration process is carried
out.

I see this very similar to process states and process state changes at
the operating system level. If you, for example, tell an application
to terminate, then the underlying processes still go through a
sequence of state changes until they finally (hopefully) disappear.

> Also, if we keep the blocked state and the state machine figure,
> then it would be better to provide a complete state transition
> table.  The current figure-style state machine lacks transition
> flows between the blocked state and other states.  I don't want to
> draw that part in the figure, since it will mess up the figure :)
> How about adding a separate table in a appendix section?

Perhaps we can add an arrow from running to blocked and another from
blocked to running. But I would not try to cover all possible
transitions since you can transition into the crashed state from
many states.

/js