Subject: [OPSAWG] discussion on network automatic adaption to VM migration?

From: "Bert (IETF) Wijnen" <[email-masked]>
Date: Tue, 22 Mar 2011 14:18:29 +0100

>From some discussion I had with Yingjie Gu on document

draft-gu-dc-management-problem-statement

> The following is brief introduction of the problem and you can find the draft at:
> http://datatracker.ietf.org/doc/draft-gu-dc-management-problem-statement/
> I fail to post it to OPSAWG, so you cannot find it at OPSAWG page.
>

In order to post, you have to be subscribed. Are you.
The other thing is that it will probably not accept any (large) or weird (.doc) attachements.
If that all fails, pls send an email to the OPSA WG chairs ([email-masked])
because they can make sure your postings are accepted.

> The background is that VM are allowed to relocate at any place in DataCenter without interruption to running service on the VM.
> I partition the problem into two parts:
> 1) Policy migration, which similar to the slides Linda sent to you. Our further consideration on this part is that, NETCONF/NETMOD is a possible way to reconfigure the policy on new location, but it needs automatic mechanism. We also think that policy-base management protocol COPS-PR could also resolve the problem. COPS-PR may be better, because we can easily complete automatic configuration based on the PEP-PDP architecture. Assuming there is a migration notification from VM to network, the devices involved in can connect PDP to get configuration automatically.  What's your opinion on this point?
>
Well, COPS-PR has not really been picked up by any products (as far as we know).
There is no-one anymore interested in it in the IETF (at least not as far as I know).
In fact a few IETFs ago, we were even discussing to reclassify the do COPS-PR documents
to HISTORIC. But since that requires work, nobody seemed interested to do that either.
So they are just lingering on.
You could bring this up in the OPSAWG session/presentation and solicit for feedback.
Don't be surprised or offended if you get severe push back on it.
> 2) Dynamic information also needs to migrate with VM, if we hope the running service on VM is not 
> interrupted. By dynamic information, I mean such items like TCP Connection tables and dynamic
> ACLs, not configured by NM, that is generated on devices according to real traffic. My draft has
> given some reason why dynamic information needs to be migrated.  For this part of problem,
> we think a new system and protocol should be defined to transfer the dynamic information.
> The reason that existing NM is not applicable  is
>   a) dynamic information migration from one device to another differs from configuration
>        sending from NM to elements,
>   b)dynamic information migration is time-sensitive, meaning it should not be earlier than a
>       specific time, refer to my draft for detail please.
>
> Sincerely hope to hear your opinion on the problems.
>
I need to read your latest document.
Not sure I have time for that before the upcoming IETF (too many documents to read as always).
But I will give it a try.

It would be good to do this sort of discussion on the OPSA WG mailing list.
That way other people get to see it and... others will probably have other
things to say as well.

Bert

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Wed, 23 Mar 2011 13:38:44 +0800

Hi Bert,
Thank you very much for your comments.  
Some of my considerations to your comments.

> Well, COPS-PR has not really been picked up by any products (as far as we  
  > know). There is no-one anymore interested in it in the IETF (at least not as far as I 
  > know). In fact a few IETFs ago, we were even discussing to reclassify the do  
  > COPS-PR documents to HISTORIC. But since that requires work, nobody seemed 
  > interested to do that either. So they are just lingering on. You could bring this up in 
  > the OPSAWG session/presentation and solicit for feedback. Don't be surprised or 
  > offended if you get severe push back on it.

Thanks for the information. You are right, COPS-PR is not wildly used. Maybe we don't use COPS-PR, it has some cons and it's binary encoding (RFC3535), we could only refer to the architecture of COPS. 3GPP uses COPS architecture, where each QoS request for bearer traffic is queried and authorized by PDP. 

I also considered NETCONF for Policy migration. NETCONF is competent for policy reconfiguration on destination devices where VM migrates to. However some parts lack if we want a complete mechanism. 1) The difference between VM migration and 'VM delete and add' is the former is much more sensitive to delay, because there are some running services on it. So we need a 'Trigger' to let NM know 'this is right the time to reconfigure policies'; 2) NM need to know where to reconfigure the policies, i.e. where the VM migrates to. 

Two ways to achieve these. 

1) 'Sideway'    Let IT Manager/VM Manager notifies NM every time IT Manager wants to move a VM. The notification could be minutes even hours before real migration. This is non-standard way, or say there is no standard for this intention yet. And this 'Sideway' cannot accurately migrate Dynamic information, because Dynamic Information is very sensitive to migration time, it's best to migrate when VM stop running. 

2) 'in-band'    Let network feels the migration and triggers policy and Dynamic Information migration. VDP (VSI Discovery and Configuration Protocol), defined in IEEE 802.1Qbg, associates/de-associate a VM with switch, that is the switches can feel the 'coming' and 'leaving' of VM. If switch feel the coming a VM, it can easily notify the NM to reconfigure policies and Dynamic information on it. Doesn't it seem very like the COPS architecture? PEP (switch) query PDP(NM) for action indication (reconfigure).  That is why I mentioned COPS-PR. 

The purpose of my document includes two parts: 1) I believe there is some standard we need to do to migrate Dynamic Information.  2) We may also need to analyze whether static policy migration can be achieved by existing NM with some extended mechanism, and whether we need to standardize the extended mechanism. 

Hope to hear more opinions from the WG.

Thanks again.

Yingjie

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Bert (IETF) Wijnen
> Sent: Tuesday, March 22, 2011 9:18 PM
> To: [email-masked]
> Subject: [OPSAWG] discussion on network automatic adaption to VM migration?
> 
> > The background is that VM are allowed to relocate at any place in DataCenter
> without interruption to running service on the VM.
> > 1) Policy migration, which similar to the slides Linda sent to you. Our further
> consideration on this part is that, NETCONF/NETMOD is a possible way to
> reconfigure the policy on new location, but it needs automatic mechanism. We also
> think that policy-base management protocol COPS-PR could also resolve the
> problem. COPS-PR may be better, because we can easily complete automatic
> configuration based on the PEP-PDP architecture. Assuming there is a migration
> notification from VM to network, the devices involved in can connect PDP to get
> configuration automatically.  What's your opinion on this point?
> >
> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 23 Mar 2011 08:51:28 +0100

> Hope to hear more opinions from the WG.

Can you say something about the VM installations you have operational
experience with (size, software platforms used, ...) and that makes
you believe there is a lack of a standard?

As far as I understand, you are talking about an interface between VM
live migration systems and network management systems and maybe
middleboxes and I like to understand the ecosystem since such a
standard will only have a chance to be deployed if all organizations
involved have a keen interest in solving this problem by implementing
and deploying a standard.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Thu, 24 Mar 2011 10:55:04 +0800

We have a small experimental platform to verify some of the problem I described in the draft.
In our platform, we used the following components:
1) Virtualization platform: VMware ESX 4.0 and vCenter Trial version
2) Network side manager (not traditional NM for the network), we call it nCenter: OpenNMS
3) Servers: Huawei Product
4) Switches: Huawei Product.
5) NM protocol: SNMP
6) SAN
During the experiment, we find no standard to follow to deal with the problem I described in the previous email. 
I mean we can use SNMP to SET devices, but we have to use VMware API (non-standard) to learn the migration of VM. The API doesn't provide the source and destination network devices, so we have to design yet another proprietary way to learn the source and destination devices.  

And we have to define several proprietary MIBs for some features we need, e.g. DHCP Snooping table, topology discovery between physical and virtual switch, and dynamic ACL. Later on, maybe we can use NETCONF, but still need to define data module. 

> From: Juergen Schoenwaelder [mailto:[email-masked]]
> Sent: Wednesday, March 23, 2011 3:51 PM
> To: Yingjie Gu(yingjie)
> Cc: 'Bert (IETF) Wijnen'; [email-masked]
> Subject: Re: [OPSAWG] discussion on network automatic adaption to VM
> migration?
> 
> > The purpose of my document includes two parts: 1) I believe there is some
> standard we need to do to migrate Dynamic Information.  2) We may also need to
> analyze whether static policy migration can be achieved by existing NM with some
> extended mechanism, and whether we need to standardize the extended
> mechanism.
> >

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 24 Mar 2011 09:27:36 +0100

> And we have to define several proprietary MIBs for some features we need, e.g. DHCP Snooping table, topology discovery between physical and virtual switch, and dynamic ACL. Later on, maybe we can u…

There seems to be ongoing IEEE work to help among other things with VM
migrations. What exactly do you want to do in the IETF?

I guess I am still not sure what exactly problem is to be solved. Here
is what I understood (correct me if I am wrong):

* If a switch discovers the movement of a VM (e.g. via VDP), then you
  want to have a network management system notified. Correct? 

  It seem there is work ongoing in 802.1Qbg to define a MIB module.
  Perhaps you should ask them to add appropriate notifications?

* Once a network manager has learned about a migration, you seem to
  want to do state transfers. It is not particularly clear to me which
  state you want to transfer - it seems you are thinking of state held
  in middleboxes, but I am not sure.

Am I missing something?

/js

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Thu, 24 Mar 2011 17:39:41 +0800

> Sent: Thursday, March 24, 2011 4:28 PM

> > We have a small experimental platform to verify some of the problem I described

> in the draft.

> > 2) Network side manager (not traditional NM for the network), we call it nCenter:

> OpenNMS

> > During the experiment, we find no standard to follow to deal with the problem I

> described in the previous email.

> > I mean we can use SNMP to SET devices, but we have to use VMware API

> (non-standard) to learn the migration of VM. The API doesn't provide the source and

> destination network devices, so we have to design yet another proprietary way to

> learn the source and destination devices.

> > And we have to define several proprietary MIBs for some features we need, e.g.

> DHCP Snooping table, topology discovery between physical and virtual switch, and

> dynamic ACL. Later on, maybe we can use NETCONF, but still need to define data

> module.

>   Perhaps you should ask them to add appropriate notifications?

I have been following and participating IEEE 802.1 Qbg. Qbg doesn't define MIB, it defines protocol that works only between Hypervisor (general name for Virtual Switch) and adjacent physical switch to associate a VM to an adjacent switch. The following is VDP TLV, you can find detail information at IEEE.

Let’s skip the VDP part for now. Assuming the adjacent switch already learned VM. Here are the parts that we look for to make a complete mechanism for policy and dynamic information migration:

1) When? For static polices, NM can start polices reconfiguration any time (operated by a human) or start at the same time as dynamic information migration (automatically, without human intervention), but should be finished not too later than VM restart at destination devices.  For dynamic information, we need to know the exact time when VM-NOT-RUNNING.  

2) Where? Network manager needs to know the source and destination devices. API from VM Manager can not provide such information. Both adjacent switch and upper layer devices, e.g aggregation switch, router and middleboxes, need to know the migration state if we want to migrate dynamic information. VDP can not achieve this.

3) How? NETCONF and other existing NMP can work for policy reconfiguration, as long as the above two parts can be resolved pretty well. 

For dynamic information, we need a new mechanism (including architecture and protocols), I don’t think NETCONF covers this. 

>   in middleboxes, but I am not sure.

If you can review my draft, I list some examples in my drafts. 

Examples are TCP connection table, Multicast membership on ports, DHCP Snooping, cumulative data, etc. 

> Am I missing something?

Thanks for your attention and comments.

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 24 Mar 2011 10:41:44 +0100

Hi Yingjie,

IEEE 802.1Qbg will define the MIB extensions for protocols defined in
the document. This will be part of Section 17 of the IEEE 802.1Qbg
draft, by now a place holder, which will be filled in with content in
the coming versions.

Thanks and Regards,

Dan

________________________________

	From: [email-masked] [mailto:[email-masked]]
On Behalf Of Yingjie Gu(yingjie)
	Sent: Thursday, March 24, 2011 11:40 AM
	To: 'Juergen Schoenwaelder'
	Cc: 'yingjie gu'; [email-masked]
	Subject: Re: [OPSAWG] discussion on network automatic adaption
to VMmigration?

	> From: Juergen Schoenwaelder
[mailto:[email-masked]]

	>   Perhaps you should ask them to add appropriate
notifications?

	I have been following and participating IEEE 802.1 Qbg. Qbg
doesn't define MIB, it defines protocol that works only between
Hypervisor (general name for Virtual Switch) and adjacent physical
switch to associate a VM to an adjacent switch. The following is VDP
TLV, you can find detail information at IEEE.

	Let's skip the VDP part for now. Assuming the adjacent switch
already learned VM. Here are the parts that we look for to make a
complete mechanism for policy and dynamic information migration:

	1) When? For static polices, NM can start polices
reconfiguration any time (operated by a human) or start at the same time
as dynamic information migration (automatically, without human
intervention), but should be finished not too later than VM restart at
destination devices.  For dynamic information, we need to know the exact
time when VM-NOT-RUNNING.  

	2) Where? Network manager needs to know the source and
destination devices. API from VM Manager can not provide such
information. Both adjacent switch and upper layer devices, e.g
aggregation switch, router and middleboxes, need to know the migration
state if we want to migrate dynamic information. VDP can not achieve
this.

	3) How? NETCONF and other existing NMP can work for policy
reconfiguration, as long as the above two parts can be resolved pretty
well. 

	For dynamic information, we need a new mechanism (including
architecture and protocols), I don't think NETCONF covers this. 

	Examples are TCP connection table, Multicast membership on
ports, DHCP Snooping, cumulative data, etc.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 24 Mar 2011 10:57:08 +0100

On Thu, Mar 24, 2011 at 05:39:41PM +0800, Yingjie Gu(yingjie) wrote:
 
> I have been following and participating IEEE 802.1 Qbg. Qbg doesn't
> define MIB, it defines protocol that works only between Hypervisor
> (general name for Virtual Switch) and adjacent physical switch to
> associate a VM to an adjacent switch. The following is VDP TLV, you
> can find detail information at IEEE.

So what about the MIB listed here?

http://www.ieee802.org/1/pages/802.1bg.html

And terminology wise, I disagree with the notion that a Hypervisor is
a general name for a Virtual Switch.

> For dynamic information, we need a new mechanism (including architecture and protocols), I don’t think NETCONF covers this. 

Sounds too abstract too me. You are already sure you need new
architecture and protocols while I am still not sure what the problem
to be solved is. It sounds like you want communication between the VM
management system and the network management system. If so, to have a
chance of success in terms of implementation of deployment, you would
have to have organizations building VM management system and network
management systems excited about the idea.

> Examples are TCP connection table, Multicast membership on ports, DHCP Snooping, cumulative data, etc. 

I already choked up on the TCP connection table. What I call the TCP
connection table sits in the kernel of the OS being migrated, so I
fail to see what needs to be done there. I had the impression you were
thinking more about connection state maintained by middleboxes, but
your document just did not make this clear enough to me.

/js

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Fri, 25 Mar 2011 20:41:16 +0000

Hi Juergen, 
See inline please. 

____________
____________________________
From:Juergen Schoenwaelder [[email-masked]]
Time : 2011年3月24日 17:57
To: Yingjie Gu(yingjie)
Cc: 'Bert (IETF) Wijnen'; [email-masked]; 'yingjie gu'
Subject: Re: [OPSAWG] discussion on network automatic adaption to VM migration?

http://www.ieee802.org/1/pages/802.1bg.html
[Yingjie] Sorry for the mistake. 
And terminology wise, I disagree with the notion that a Hypervisor is
a general name for a Virtual Switch.
[Yingjie] Well, not formally, but Hypervisor refers to VEPA/VEB/other kind of virtual switch type.

[Yingjie] Not communication between VM manager and network manager. This is only a way of migration notification, and I said that is not a standadized way, and even there is a need for standard, it's not IETF SCOPE I guess.

I already choked up on the TCP connection table. What I call the TCP
connection table sits in the kernel of the OS being migrated, so I
fail to see what needs to be done there. I had the impression you were
thinking more about connection state maintained by middleboxes, but
your document just did not make this clear enough to me.
[Yingjie] TCP connection table on OS needn't to be done, you are right.
I am talking the TCP connection table on Middlebox, and it's only an example. Maybe there is another way to name it. 
I will try to make it clearer. 
Thank you for your comments.
[end]

/js