Subject: [nvo3] server2nve signaling: VM migration

From: Kireeti Kompella <[email-masked]>
Date: Mon, 16 Jul 2012 12:07:54 -0700

Hi,

This discussion is to understand the process of VM migration (to the extent
needed), and the consequences on address mappings.  This is for all
(mostly, me) to understand the issues; not all of what's discussed here
will be put in the draft.  I've tried to incorporate what David has said so
far.

Situation: VM X is running on server S connected to lNVE L.  X will move to
server S' connected to lNVE L'.

(For now, we'll assume a push model where addressing information is sent
out "eagerly" rather than "lazily".)

A high level picture of what the VM orchestration system does: (please let
me know if I have some significant details wrong)

M1a) tell S' that VM X will move from S to S'.  S' prepares to receive a
copy of the VM image, memory, etc.
M1b) tell S to begin copying X from S to S'.

M2a) tell S to pause X
M2b) tell S' to start X

M3a) if X runs fine on S', tell S to destroy X (migration successful)
M3b) if not, tell S' to destroy X, and tell S to continue running X
(migration aborted)

M1x happen before M2x which happen before M3x.  The order of orchestration
cannot be assumed to reflect in the order of signaling messages.

----

M1a results in N1:
N1) S' sends a "pre-associate" message to L', which in turn sends out an
update that "X will move to L' ".

(This can be done by an address update with a lower preference, such that
other VMs in X's VM still use the old location L, but those who want to
know where X is moving (like L) can find out.)

M2a results in N2a:
N2a) S sends a dissociate message to L with a timeout.  When L receives
this, it points its route for X to L' (if it already learned this from L')
or to discard.

M2b results in N2b:
N2b) S' sends an associate message to L', whereupon L' makes itself the
"official" nexthop for X (normal preference; VM moved).

M3b (error condition) will be left for future discussion.

----

Race conditions:
R1) if N1 BEFORE N2a => traffic to X still arriving at L gets redirected to
L'
      If not, traffic to X arriving at L gets dropped

R2) if N2a BEFORE N2b => traffic redirected from L to L' gets dropped until
N2b occurs (and is processed).

R3) N2a times out => redirection at L stops.  Traffic sent to L from NVEs
that still haven't received the control plane update from N2b (i.e., VM
moved) will be dropped.

Please let me know of other interesting race conditions!

-- 
Kireeti

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 18 Jul 2012 20:27:27 +0000

Kireeti,

Lots of description in your draft on VM addition/deletion/moving are not how it is done today in the deployed environment. There are many reasons for the deployed approaches stay the way they are today.
Study has shown that network is only a very small portion (15%) of the overall DC infrastructure.

So DC is heavily server focused. Most of available VM/server management systems don't really consider too much of network setup, let along VNID or VLANs.

So the Step #4/#8 in your draft's Introduction session are usually performed by separate crew. Yes, it would be nice to have #5 & #6. But they are not available  yet. Maybe there should be a standard group to define those interfaces.

Comments to your questions are inserted below:

From: [email-masked] [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Monday, July 16, 2012 2:08 PM
To: [email-masked]
Subject: [nvo3] server2nve signaling: VM migration

Hi,

This discussion is to understand the process of VM migration (to the extent needed), and the consequences on address mappings.  This is for all (mostly, me) to understand the issues; not all of what's discussed here will be put in the draft.  I've tried to incorporate what David has said so far.

Situation: VM X is running on server S connected to lNVE L.  X will move to server S' connected to lNVE L'.

(For now, we'll assume a push model where addressing information is sent out "eagerly" rather than "lazily".)

A high level picture of what the VM orchestration system does: (please let me know if I have some significant details wrong)

M1a) tell S' that VM X will move from S to S'.  S' prepares to receive a copy of the VM image, memory, etc.

[Linda] The orchestration system will start with downloading VM image to S' via a dedicated port.

[Linda] many VM orchestration system will copy VM X image from S to its own storage, prepare to download to S'.

M2a) tell S to pause X

[Linda] normally tell S' to start X before pausing X on S.

M2b) tell S' to start X

M3a) if X runs fine on S', tell S to destroy X (migration successful)
M3b) if not, tell S' to destroy X, and tell S to continue running X (migration aborted)

M1x happen before M2x which happen before M3x.  The order of orchestration cannot be assumed to reflect in the order of signaling messages.

----

M1a results in N1:
N1) S' sends a "pre-associate" message to L', which in turn sends out an update that "X will move to L' ".

[Linda] L' can't trust the message sent from S' unless S' carries some valid KEY for L'. If S' can get the valid KEY from trusted authorities, why not let the L' manager (or management systems) inform L' of the new VM X is to be attached?

(This can be done by an address update with a lower preference, such that other VMs in X's VM still use the old location L, but those who want to know where X is moving (like L) can find out.)

M2a results in N2a:
N2a) S sends a dissociate message to L with a timeout.  When L receives this, it points its route for X to L' (if it already learned this from L') or to discard.

[Linda] Today's VM hypervisor don't send any "Dissociate" message when a VM is removed. We may not have enough power to change the behavior, especially there are widely deployed base. Maybe it is easier to use the "timeout" approach. It is not that important to timeout a VM, but more important to timeout a VN. When a VN is timed out from a NVE, all the broadcast messages within the VN can be filtered out.

M2b results in N2b:
N2b) S' sends an associate message to L', whereupon L' makes itself the "official" nexthop for X (normal preference; VM moved).

[Linda] Should consider using an external entity to inform L' of the new association.

----

Race conditions:
R1) if N1 BEFORE N2a => traffic to X still arriving at L gets redirected to L'
      If not, traffic to X arriving at L gets dropped

R2) if N2a BEFORE N2b => traffic redirected from L to L' gets dropped until N2b occurs (and is processed).

R3) N2a times out => redirection at L stops.  Traffic sent to L from NVEs that still haven't received the control plane update from N2b (i.e., VM moved) will be dropped.

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Sat, 21 Jul 2012 08:48:36 +0200

A few comments (primarily based on my experience with VMware, Xen might be a bit different). I’m also assuming Kireeti had live VM migration in mind (since he’s mentioning pausing the VM), not cold one (where you shut down the VM, move its disk image to wherever, and start the VM from scratch on another hypervisor)

[M1a] the orchestration system is not in the data path of VM move. The memory image is transferred directly between the hypervisors. A dedicated interface (or vNIC) or subnet may be used but that’s not mandatory. All you need is IP connectivity between source and target hypervisor.

[M1b] The orchestration system is not involved in VM memory move. That would place unnecessary burden on the orchestration system and increase the latency/time-to-move.

[M2a] Linda’s comments are totally wrong. X has to be paused on S before it’s started on S’, or you’ll get nice split brain problems. Yet again, the orchestration system is not involved. The source and target hypervisor figure it out themselves once they are ordered to move the VM.

N1) Linda’s comment that L’ cannot trust the message from S’ unless it’s signed is correct ... yet in practice we rely on gratuitous ARP generated by S’ (more precisely, vSwitch in S’) as being good enough. We might have to settle on some middle ground.

N2a) Linda’s comment that “today’s hypervisors don’t send disassociate message” is (almost) correct. They don’t send a message to the network, but they do report the change to the orchestration system, and they definitely indicate the change to the embedded vSwitch, which can act on it (be it Open vSwitch in Xen/KVM or Nexus 1000V/DVS 5000V in VMware, with at least Nexus 1000V and potentially Open vSwitch being available in Windows 8). The “we may not have the power to change it” argument is totally bogus and results in crazy “solutions” where the network elements desperately try to track VM movements by either snooping the network traffic or tracking the orchestration system state. Furthermore, the behavior has already been changed – a fine example is Nicira’s NVP, which uses OpenFlow, or IBM’s DVS 5000V which supports VDP.

As for Kireeti’s race conditions R1 and R2 – current solutions accept dropping the traffic sent to the VM that’s being moved between the moment it’s paused on S and the moment the new path to S’ is fully established (through the fine mechanism of gratuitous APR). If we’re not OK with that, we can either use a mechanism that enforces transactional consistency (example: OpenFlow with barriers) or rely on S or L to perform temporary forwarding toward S’ or L’ (something like MPLS/VPN BGP PIC on PE-routers).

Race condition R3 can only be solved with a two-phase commit, where all the elements in the path (S, L, S’, L’) report their readiness for VM move to the orchestration system, which then orders all involved elements to complete the move. Not sure whether we really need this level of consistency ... it might be simpler (and good enough) to settle for the temporary packet drops during the VM move event.

Best,

Ivan

From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Wednesday, July 18, 2012 10:27 PM
To: Kireeti Kompella; [email-masked]
Cc: Yakov Rekhter; [email-masked]
Subject: Re: [nvo3] server2nve signaling: VM migration

Kireeti, 

 

So DC is heavily server focused. Most of available VM/server management systems don’t really consider too much of network setup, let along VNID or VLANs.  

So the Step #4/#8 in your draft’s Introduction session are usually performed by separate crew. Yes, it would be nice to have #5 & #6. But they are not available  yet. Maybe there should be a standard group to define those interfaces. 

 

Hi,

[Linda] The orchestration system will start with downloading VM image to S’ via a dedicated port. 

[Linda] many VM orchestration system will copy VM X image from S to its own storage, prepare to download to S’. 

M2a) tell S to pause X

[Linda] normally tell S’ to start X before pausing X on S. 

M2b) tell S' to start X

 

----

M1a results in N1:

[Linda] L’ can’t trust the message sent from S’ unless S’ carries some valid KEY for L’. If S’ can get the valid KEY from trusted authorities, why not let the L’ manager (or management systems) inform L’ of the new VM X is to be attached? 

M2a results in N2a:

[Linda] Today’s VM hypervisor don’t send any “Dissociate” message when a VM is removed. We may not have enough power to change the behavior, especially there are widely deployed base. Maybe it is easier to use the “timeout” approach. It is not that important to timeout a VM, but more important to timeout a VN. When a VN is timed out from a NVE, all the broadcast messages within the VN can be filtered out. 

M2b results in N2b:

[Linda] Should consider using an external entity to inform L’ of the new association. 

----

Race conditions:

 

-- 
Kireeti

---

From: Melinda Shore <[email-masked]>
Date: Sat, 21 Jul 2012 13:58:24 -0800

On 7/20/12 10:48 PM, Ivan Pepelnjak wrote:
> [M1b] The orchestration system *is not *involved in VM memory move. That
> would place unnecessary burden on the orchestration system and increase
> the latency/time-to-move.

It is not clear to me that the amount of latency introduced would
actually be a burden given the existing time-to-move, but I think
it's worth considering how any associated network state (middleboxes,
primarily - i.e. firewalls) would be migrated in parallel with the
VM, and which network elements would need to be involved.

Melinda

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 23 Jul 2012 15:52:23 +0000

Ivan,

See comments inserted below:

From: Ivan Pepelnjak [mailto:[email-masked]]
Sent: Saturday, July 21, 2012 1:49 AM
To: Linda Dunbar; 'Kireeti Kompella'; [email-masked]
Cc: 'Yakov Rekhter'; [email-masked]
Subject: RE: [nvo3] server2nve signaling: VM migration

[Linda] Some orchestration CAN trigger VM memory move, even though vMotion requires VM image to be accessible by both Source and Destination host.

[Linda] it depends on the applications running on the VMs.

Best,
Ivan

Kireeti,

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Monday, July 16, 2012 2:08 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] server2nve signaling: VM migration

Hi,

M2a) tell S to pause X

M2b) tell S' to start X

----

M1a results in N1:

M2a results in N2a:

M2b results in N2b:

----

Race conditions:

--
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Mon, 23 Jul 2012 18:26:19 +0200

@Comment#1 – there might be a difference between orchestration system triggering the move (which it definitely does) and being involved in the move (which, to me, means being involved in the memory and/or disk image copy and associated handover activities) ... or maybe it’s just my lack of English skills.

@Comment#2 – I would love to see an example of an application that can be started on the second hypervisor first and only after that stopped on the first hypervisor ... more so if the said application performs an I/O operation in between.

From: Linda Dunbar [mailto:[email-masked]] 
Sent: Monday, July 23, 2012 5:52 PM
To: Ivan Pepelnjak; 'Kireeti Kompella'; [email-masked]
Cc: 'Yakov Rekhter'; [email-masked]
Subject: RE: [nvo3] server2nve signaling: VM migration

Ivan, 

 

Best,

Ivan

Kireeti, 

 

 

Hi,

M2a) tell S to pause X

M2b) tell S' to start X

 

----

M1a results in N1:

 

M2a results in N2a:

 

M2b results in N2b:

 

----

Race conditions:

 

-- 
Kireeti

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Mon, 23 Jul 2012 18:33:44 +0200

Oh, did you have to open that can of worms ;)

Network state needs to be migrated only if the hypervisor (or a kernel plug-in) provides the service (example: VMware vShield App/Zones, Juniper VGW, Xen OpenFlow-based filters). If you use physical or virtual appliances connected to a virtual L2 or L3 segment, then there's no need to move the state, as the appliance (where the state is) hasn't moved.

Interestingly, it's quite easy to move the session state(s) between hypervisors if they use OpenFlow like Open vSwitch (read the flows from the first hypervisor, copy them to the second hypervisor).

Ivan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Melinda Shore
> Sent: Saturday, July 21, 2012 11:58 PM
> To: [email-masked]
> > [M1b] The orchestration system *is not *involved in VM memory move.
> > That would place unnecessary burden on the orchestration system and
> > increase the latency/time-to-move.
> 
> It is not clear to me that the amount of latency introduced would actually
> be a burden given the existing time-to-move, but I think it's worth
> considering how any associated network state (middleboxes, primarily -
> i.e. firewalls) would be migrated in parallel with the VM, and which
> network elements would need to be involved.
> 
> Melinda
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Melinda Shore <[email-masked]>
Date: Mon, 23 Jul 2012 08:36:51 -0800

On 7/23/12 8:33 AM, Ivan Pepelnjak wrote:
> Network state needs to be migrated only if the hypervisor (or a
> kernel plug-in) provides the service (example: VMware vShield
> App/Zones, Juniper VGW, Xen OpenFlow-based filters). If you use
> physical or virtual appliances connected to a virtual L2 or L3
> segment, then there's no need to move the state, as the appliance
> (where the state is) hasn't moved.

Only in a tromboning topology, which we're trying to avoid.

Melinda

---

From: Truman Boyes <[email-masked]>
Date: Mon, 23 Jul 2012 12:40:25 -0400

On Mon, Jul 23, 2012 at 12:33 PM, Ivan Pepelnjak <[email-masked]>wrote:

> Network state needs to be migrated only if the hypervisor (or a kernel
> plug-in) provides the service (example: VMware vShield App/Zones, Juniper
> VGW, Xen OpenFlow-based filters). If you use physical or virtual appliances
> connected to a virtual L2 or L3 segment, then there's no need to move the
> state, as the appliance (where the state is) hasn't moved.
>
> Interestingly, it's quite easy to move the session state(s) between
> hypervisors if they use OpenFlow like Open vSwitch (read the flows from the
> first hypervisor, copy them to the second hypervisor).
>
> Ivan

It's easy to move session state on the nodes that participate in the
overlay network  (ie. NVEs, kernel module plugins, etc); however
communicating with external elements that are part of the service would
need to occur. An example would be firewalls or NAT instances that reside
outside the NVE space. Orchestration would help to localize traffic or
provide the necessary network state changes.

Truman

> actually

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Mon, 23 Jul 2012 18:44:46 +0200

Much as I'd love to get rid of traffic trombones, we might have to live with dedicated L4-7 appliances (physical or virtual) and associated trombones to satisfy the whims of security/compliance auditors (PCI comes to mind).

On a more technical note, firewalls are be easy (more so if they don't do deep packet inspection), but architecting a scalable distributed load balancer w/o traffic trombones might be an interesting task.

Ivan

> From: Melinda Shore [mailto:[email-masked]]
> Sent: Monday, July 23, 2012 6:37 PM
> To: Ivan Pepelnjak
> Cc: [email-masked]
> > plug-in) provides the service (example: VMware vShield App/Zones,
> > Juniper VGW, Xen OpenFlow-based filters). If you use physical or
> > virtual appliances connected to a virtual L2 or L3 segment, then
> > there's no need to move the state, as the appliance (where the state
> > is) hasn't moved.
>

---

From: Melinda Shore <[email-masked]>
Date: Mon, 23 Jul 2012 08:47:39 -0800

On 7/23/12 8:44 AM, Ivan Pepelnjak wrote:
> Much as I'd love to get rid of traffic trombones, we might have to
> live with dedicated L4-7 appliances (physical or virtual) and
> associated trombones to satisfy the whims of security/compliance
> auditors (PCI comes to mind).

Or we might not.  In a scenario where a VM is being migrated to a
different data center because its current one is being taken offline,
tromboning isn't even an option.  And this, for better or worse, is
one of the basic use cases.

Melinda

---

From: <[email-masked]>
Date: Tue, 24 Jul 2012 12:28:06 -0400

Hi Linda,

I think this may be confusing physical provisioning (install equipment,
cable the network, etc.) with virtual provisioning (add a virtual machine
[VM] to already-running hardware infrastructure w/o changing any of the
hardware).

As I read it, Kireeti's draft is directed towards the latter (add a VM without
changing the hardware), and under that interpretation, all of the 1-8 steps
in Section 1 of the draft appear to be within the purview of a comprehensive
software-based VM provisioning system.

In another email, Maria commented:

> The document seems to be dictating provisioning process for a VM. What
> service/cloud computing providers need are the building blocks, i.e.
> standard APIs and protocols to create and manage virtual networks in
> cloud computing environment.

I agree with that goal, and I read the draft as "describing" as opposed
to "dictating" VM provisioning in order to tease out functional requirements
for those building blocks (e.g., they are going to have to deal with the races
that Kireeti identified - I'd suggest having a design that works correctly
regardless of whether the races occur, as opposed to trying to prevent the
races).

I don't read that draft as standardizing VM provisioning (e.g., I don't
see anything resembling an explicit or implicit "MUST do these 8 steps in
this order").  Rather, the draft describes some things that need to be done
for VM provisioning, and the value that I  see in the draft is that it uses
VM provisioning, termination and migration to explore the requirements on
the associate/dissociate protocol.  To the extent that the 8 steps aren't
completely accurate, I'd be happy to see them generalized/corrected + a
disclaimer added that they may not occur in that order, and I don't think
any of that detracts from the value of the draft.

One thing I do want to point out is that Kireeti has assumed (at least for
the sake of discussion) advance propagation/advertisement of the new
address at pre-associate time - it should also be fine to defer that
advertisement until the actual associate happens (e.g., eliminates need
to deal with provisional addresses sent in advance of a move).

Thanks,
--David

Sent: Wednesday, July 18, 2012 4:27 PM

Kireeti,

Hi,

M2a) tell S to pause X

M2b) tell S' to start X

----

M1a results in N1:

M2a results in N2a:

M2b results in N2b:

----

Race conditions:

--
Kireeti

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 24 Jul 2012 10:09:25 -0700

Hi David,

On Jul 24, 2012, at 9:28, <[email-masked]> wrote:

> I read the draft as “describing” as opposed to “dictating” VM provisioning in order to tease out functional requirements for those building blocks
> 
What you said exactly captures why I wrote this down. Actually, one further reason is to attract commentary like yours and Ivan's that correct details and help achieve a level of abstraction appropriate for discussing the network bits. 

Maybe I should preface that whole section with "In one embodiment of VM orchestration, ..." :-)
> (e.g., they are going to have to deal with the races that Kireeti identified
> 

... with help from the list. More help please!

Cheers,
Kireeti