Subject: [nvo3] draft-khasnabish-vmmi-problems-02

From: Richard Bin liu <[email-masked]>
Date: Fri, 19 Oct 2012 00:44:08 +0800

Greeting all,

We had published this draft and wanted to bring it to the NVO3 WG's
attention.  We plan to present/discuss this draft at the upcoming meeting
as well, consequently please discuss here on the maillist as well.

Thanks,

Bhumip Khasnabish, et al.

A new version of I-D, draft-khasnabish-vmmi-problems-02.txt
has been successfully submitted by Bin Liu and posted to the
IETF repository.
Filename:        draft-khasnabish-vmmi-problems
Revision:        02
Title:           Requirements for Mobility and Interconnection of Virtual
Machine and Virtual Network Elements
Creation date:   2012-10-15
WG ID:           Individual Submission
Number of pages: 46
URL:
http://www.ietf.org/internet-drafts/draft-khasnabish-vmmi-problems-02.txt
Status:
http://datatracker.ietf.org/doc/draft-khasnabish-vmmi-problems
Htmlized:
http://tools.ietf.org/html/draft-khasnabish-vmmi-problems-02
Diff:
http://www.ietf.org/rfcdiff?url2=draft-khasnabish-vmmi-problems-02
Abstract:
   In this draft, we discuss the challenges and requirements related to
   migration, mobility, and interconnection of Virtual Machines (VMs)and
   Virtual Network Elements (VNEs).  VM migration scheme across IP
   subnets is needed to implement virtual computing resources sharing
   across multiple network administrative domains.  For the seamless
   online migration in various scenarios, many problems are needed to be
   resolved on the control plane.  The VM migration process should be
   adapted to these aspects.  We also describe the limitations of
   various types of virtual local area networking (VLAN) and virtual
   private networking (VPN) techniques that are traditionally expected
   to support such migration, mobility, and interconnections.

---

From: "Pat Thaler" <[email-masked]>
Date: Tue, 23 Oct 2012 22:58:15 +0000

3 - migration from system to system that stays within the same data center and administrative domain isn't listed.

In 3 (C), IEEE 802.1Q PBB already provides for a larger (24-bit) ID when the 12-bit VLAN ID isn't large enough. VXLAN and NVGRE aren't needed to address that problem. What VXLAN and NVGRE provide that PBB doesn't is the ability to carry L2 over an underlying layer 3 network - so VXLAN and NVGRE should be in (D) but aren't.

There is a problem with NVGRE alluded to but not explained: And, NVGRE
   "[draft-sridharan-virtualization-nvgre-00] attempts to solve similar
   problems, but artificially causes interoperability problems between
   domains."
Why?

Also, 24 bits is to address the scaling needs within an administrative domain. It isn't clear that it provides enough space to allocate compatible assignments between administrative domains. Mapping may be needed at the boundary between administrative domains.

(E) Aren't A through D about mobility of VMs? I don't see what this adds.

3.1
"In addition, when re-configuring the VLAN in the traditional DC
   network, STP (MSTP) will lead to the VLAN isolation.  It is a very
   serious problem in the DC network, especially in the storage network,
   because the services that storage networks support demand
   uninterrupted operation.  In the Cloud computing and DC network, the
   problem with VM migration is fatal."
?? STP or MSTP don't cause VLAN isolation and IEEE 802.1Q defines MVRP to handle dynamic configuration and distribution of VLAN membership information. That there are times when a data center operator chooses to structure their data center(s) with multiple layer 3 subnets and make that transparent to VMs operating in their own IP address space doesn't need to be justified by attacking the capabilities of Layer 2.

"For example, in the process of VM migration in IDC, there are
   scenarios that VM in the traditional three-tier topological network
   is migrated through WAN to Fat-Tree topological network, or to a
   variety of other topological networks.  For the seamless online
   migration in these scenarios, a very intelligent seamless VM online
   migration is needed to be implemented on the control plane."

Lacks specificity - what problem is posed here and what features need to be provided in the control plane to solve it?

"If VM migration is only implemented in the L2 domain, its concern on

   the network is that the expansion of the number of VLANs or isolated

   domains, such as the 16,000,000 isolated domains in PBB."
24 bits is the same size as used by VXLAN and NVGRE so they also provide for 16 million tenant domains.

"How the source network environment adapts its configuration to the

   target environment?"
What do you mean by source network environment - "source" is used in the Framework and problem statement as the sending end of the data plane path; i.e. as source and destination, but that doesn't seem to be how you are using it here. Also explain "target environment".

3.1.2 Part of the purpose of NVO is to make VM addresses independent of the underlying network addressing or routing and we have examples that do so by using an outer IP address for routing. Furthermore, we want these addresses to remain unchanged on moves so location independence for them is a requirement. This section appears to be saying that they still should be location and underlying topology dependent and I disagree. The only constraint on them should be to be unique within the virtual network.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Richard Bin liu
Sent: Thursday, October 18, 2012 9:44 AM
To: [email-masked]
Subject: [nvo3] draft-khasnabish-vmmi-problems-02

Greeting all,

We had published this draft and wanted to bring it to the NVO3 WG's attention.  We plan to present/discuss this draft at the upcoming meeting as well, consequently please discuss here on the maillist as well.

Thanks,

A new version of I-D, draft-khasnabish-vmmi-problems-02.txt
has been successfully submitted by Bin Liu and posted to the
IETF repository.
Filename:        draft-khasnabish-vmmi-problems
Revision:        02
Title:           Requirements for Mobility and Interconnection of Virtual Machine and Virtual Network Elements
Creation date:   2012-10-15
WG ID:           Individual Submission
Number of pages: 46
URL:             http://www.ietf.org/internet-drafts/draft-khasnabish-vmmi-problems-02.txt
Status:          http://datatracker.ietf.org/doc/draft-khasnabish-vmmi-problems
Htmlized:        http://tools.ietf.org/html/draft-khasnabish-vmmi-problems-02
Diff:            http://www.ietf.org/rfcdiff?url2=draft-khasnabish-vmmi-problems-02
Abstract:
   In this draft, we discuss the challenges and requirements related to
   migration, mobility, and interconnection of Virtual Machines (VMs)and
   Virtual Network Elements (VNEs).  VM migration scheme across IP
   subnets is needed to implement virtual computing resources sharing
   across multiple network administrative domains.  For the seamless
   online migration in various scenarios, many problems are needed to be
   resolved on the control plane.  The VM migration process should be
   adapted to these aspects.  We also describe the limitations of
   various types of virtual local area networking (VLAN) and virtual
   private networking (VPN) techniques that are traditionally expected
   to support such migration, mobility, and interconnections.

---

From: "Pat Thaler" <[email-masked]>
Date: Tue, 23 Oct 2012 23:54:06 +0000

More comments:

In 3.4.2:
In this case, the VID used by the VM has a global significance.
   Various rules and usage range of the VID are required to be provided.
   The VLAN-ID used by a given VM is referred to the VLAN-ID carried by
   the traffic that is originated by that VM and within the same L2
   physical domain as the VM.

The VID used doesn't need to have global significance. It can be inserted by the virtual switch (in the hypervisor or NIC on the end system) and have been assigned to map to the VNI on the OBP in the ToR. In that case, it is stripped when the OBP does the encapsulation and doesn't go end to end.  In IEEE 802.1Qbg, we use that approach to map a VID to an I-CID on the a Provider Edge Bridge (e.g. at the ToR). VDP provides the communication between the End System and the Bridge so that the Bridge can assign the locally significant VID to map to the I-CID and so that the End System can signal that migration is occurring.

---

From: Richard Bin liu <[email-masked]>
Date: Wed, 24 Oct 2012 14:33:26 +0800

Hello Pat Thaler:

Thank you for your comments, I will feedback them in turn.

We can accomplish L3-migration by the suggested technologies listed in my
draft, combined with technologies like VXLAN and NVGRE. They are not
contradictory.

   “[draft-sridharan-virtualization-nvgre-00] attempts to solve similar

   domains.”

Why?

The method of a VXLAN connecting external devices is to use virtual L3
network equipments, such as vShield Edge and F5 load balancer. One vNIC is
required in one physical VLAN, however, one or more vNICs are required in
VXLAN.

Based on some premises, we can run VXLAN in any extension IP network:
Based on the premise that multicast is supported between data centers, a
remote VM migration can be implemented by Vxlan technology. But the
long-distance traffic delay cannot be handled by VXLAN. However, it must be
able to do this when VM migrates between multiple data centers in the same
logical subnet.

To realize seamless migration in wide range is one of our goals, but
currently VM migration tools is likely to require: the distance between two
places cannot be more than 50KM (or a larger value, but not too much). We
know that disaster recovery point should be as far as possible from the
local.

Are we going to accomplish remote VM migration through multiple relay
backup? I hope it will not be so much trouble.

On Wed, Oct 24, 2012 at 6:58 AM, Pat Thaler <[email-masked]> wrote:

>  3 – migration from system to system that stays within the same data
> center and administrative domain isn’t listed.****
>
> ** **
>
> In 3 (C), IEEE 802.1Q PBB already provides for a larger (24-bit) ID when
> the 12-bit VLAN ID isn’t large enough. VXLAN and NVGRE aren’t needed to
> address that problem. What VXLAN and NVGRE provide that PBB doesn’t is the
> ability to carry L2 over an underlying layer 3 network – so VXLAN and NVGRE
> should be in (D) but aren’t.****
>
> ** **
>
> There is a problem with NVGRE alluded to but not explained: And, NVGRE****
>
>    “[draft-sridharan-virtualization-nvgre-00] attempts to solve similar***
> *
>
>    problems, but artificially causes interoperability problems between****
>
>    domains.”****
>
> Why?****
>
> ** **
>
> Also, 24 bits is to address the scaling needs within an administrative
> domain. It isn’t clear that it provides enough space to allocate compatible
> assignments between administrative domains. Mapping may be needed at the
> boundary between administrative domains.****
>
> ** **
>
> (E) Aren’t A through D about mobility of VMs? I don’t see what this adds.*
> ***
>
> ** **
>
> 3.1****
>
> “In addition, when re-configuring the VLAN in the traditional DC****
>
>    network, STP (MSTP) will lead to the VLAN isolation.  It is a very****
>
>    serious problem in the DC network, especially in the storage network,**
> **
>
>    because the services that storage networks support demand****
>
>    uninterrupted operation.  In the Cloud computing and DC network, the***
> *
>
>    problem with VM migration is fatal.”****
>
> ?? STP or MSTP don’t cause VLAN isolation and IEEE 802.1Q defines MVRP to
> handle dynamic configuration and distribution of VLAN membership
> information. That there are times when a data center operator chooses to
> structure their data center(s) with multiple layer 3 subnets and make that
> transparent to VMs operating in their own IP address space doesn’t need to
> be justified by attacking the capabilities of Layer 2.****
>
> ** **
>
> “For example, in the process of VM migration in IDC, there are****
>
>    scenarios that VM in the traditional three-tier topological network****
>
>    is migrated through WAN to Fat-Tree topological network, or to a****
>
>    variety of other topological networks.  For the seamless online****
>
>    migration in these scenarios, a very intelligent seamless VM online****
>
>    migration is needed to be implemented on the control plane.”****
>
> ** **
>
> Lacks specificity – what problem is posed here and what features need to
> be provided in the control plane to solve it?****
>
> ** **
>
> “If VM migration is only implemented in the L2 domain, its concern on****
>
>    the network is that the expansion of the number of VLANs or isolated****
>
>    domains, such as the 16,000,000 isolated domains in PBB.”****
>
> 24 bits is the same size as used by VXLAN and NVGRE so they also provide
> for 16 million tenant domains.****
>
> ****
>
> “How the source network environment adapts its configuration to the****
>
>    target environment?”****
>
> What do you mean by source network environment – “source” is used in the
> Framework and problem statement as the sending end of the data plane path;
> i.e. as source and destination, but that doesn’t seem to be how you are
> using it here. Also explain “target environment”.****
>
> ** **
>
> 3.1.2 Part of the purpose of NVO is to make VM addresses independent of
> the underlying network addressing or routing and we have examples that do
> so by using an outer IP address for routing. Furthermore, we want these
> addresses to remain unchanged on moves so location independence for them is
> a requirement. This section appears to be saying that they still should be
> location and underlying topology dependent and I disagree. The only
> constraint on them should be to be unique within the virtual network.****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Richard Bin liu
> *Sent:* Thursday, October 18, 2012 9:44 AM
> *To:* [email-masked]
> *Subject:* [nvo3] draft-khasnabish-vmmi-problems-02****
>
> ** **
>
> Greeting all,
>
>
>    to support such migration, mobility, and interconnections.****
>
>

---

From: Jeff Wheeler <[email-masked]>
Date: Wed, 24 Oct 2012 17:12:43 -0400

> know that disaster recovery point should be as far as possible from the

Hello, Richard, I am new to these technologies and discussion, but we
are working out how to implement VXLAN and including the concern you
mention above.  I think it is not specifically relevant to VXLAN and
friends.

Our plan deals with Internet->VM traffic through VTEPs located at the
multiple datacenters, all of which may be aware of Internet->VXLAN
mappings by way of the provisioning system plus ARP/MAC
learning/snooping.  VM->Internet traffic flows natively (after passing
through a filter in the hypervisor's VTEP) and is not affected by
triangular routing, because we will manipulate ARP and NDP traffic to
trick the guest OS into doing what we want -- sending its traffic to a
VRRP gateway that is present "everywhere."  It will not tunnel back to
a hub VTEP or require state information in the native layer-3 network
used to carry the Internet-bound traffic.

VM<>VM traffic within a customer's private L2 or L3 domain will
certainly have high latency if the two VMs are located far apart, but
this is a limitation of physics, not VXLAN.  I would welcome an
internet-draft accelerating the speed of photons through glass if you
can demonstrate a way to implement it.

-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: "Pat Thaler" <[email-masked]>
Date: Wed, 24 Oct 2012 23:02:24 +0000

That should be I-SID below rather than I-CID. Pat

From: Pat Thaler
Sent: Tuesday, October 23, 2012 4:54 PM
To: Pat Thaler; Richard Bin liu; [email-masked]
Subject: RE: [nvo3] draft-khasnabish-vmmi-problems-02

More comments:

In 3.4.2:

---

From: "Martin, David" <[email-masked]>
Date: Thu, 25 Oct 2012 16:22:16 -0400

Hi Richard,

A few comments in blue text/[DWM].

Clause 3.5 The Evolution Problems...

"Note that although Ethernet and IP protocols are meant to support arbitrary topology, these Layer-2 and Layer-3 network protocols are not flexible enough for use in Data Center environments.  The lack of flexibility may result in lack of scalability, management difficulties, inflexible communications, and poor fault tolerance."

[DWM] This is a rather broad statement and should really be followed by a detailed critique of the issues - although I suppose that's the goal of the document. In any case, just so you're aware, besides the Equal Cost Trees outlined in SPB (which are Backbone Edge Bridge to BEB in extent), there is a project in-flight to add support for hop-by-hop ECMP (P802.1Qbp).

Clauses 3.6.1, 3.6.2 Requirements...

[DWM] I didn't find any actual equipment requirements, just brief descriptions of what the equipment does.

Clause 3.6.4  Fault Tolerance...

"After the VM migration, it is required to consider the impact on the switching network, such as whether the new network environment will have the problem of insufficient bandwidth."

[DWM] Good one, although it isn't really fault-related. The Virtual Station Interface Discovery and Configuration Protocol (VDP) that Pat mentioned from 802.1Qbg can be used by the target HV to register the migrating VM/VSI with the physical port on the attached bridge (e.g., ToR). That protocol exchange includes the VM/VSI Type which can be used by the switching network management system to allocate resources accordingly. All this can be performed at the destination prior to the VM migration using the 'pre-associate with resource reservation' VDP TLV type.

Clause 3.6.10.2 Requirements for Live Migration of Virtual Machines

[DWM] The list should include more than just the requirement to have an IP address. For example, the maximum latency between source DC and destination DC hypervisors is 10ms RTT for a VMware vSphere 5.0 vMotion.

Clause 3.6.11 Access and Migration of VMs without users' Perception

[DWM] There should be a target requirement for the application pause time (aka VM blackout time) experienced by the user, for example <4 seconds.

Clause 3.6.13 The East-West Traffic Problem

"In addition, TRILL..., nor it provides any solution to the requirement of supporting more than 4K VLANs."

[DWM] You should take a look at the draft-ietf-trill-fine-labeling-00 (December 8, 2011) which added the EX-Tag.

Clause 3.6.18 Routing Control - Multicast Processing

"There also exist additional overheads because each customer multicast group is associated with the forwarding tree network throughout the Ethernet switching network. Consequently, development and standardization of efficient Layer-2 multicast mechanism to support intra- and inter-DC VM mobility would be very useful."

[DWM] I don't follow these statements. Efficient multicast is an advantage of a bridged network, as opposed to source replication. What specific improvement would VM mobility need?

Clause 3.6.19 Problems and Requirement related to DMTF

[DWM] Why is this in an IETF draft?

Clauses 4.1.1 - 4.1.3 Migration...

[DWM] While the inspection, authentication and resource allocation steps are necessary, a HV can only do what's within its span of control (leveraging 802.1Qbg). Only the VM cluster manager and higher level DC management applications (e.g., vCloud Director) have a broad enough scope to do all this.

Clause 4.1.4 Standardization of Migration State

[DWM] Is the intent to standardize the steps of a VM migration? It's a very vendor specific operation.

Clauses 4.2 - 4.3 Mobility...

[DWM] Suggest deleting these as they repeat material from clause 3.

...Dave

From: [email-masked] [mailto:[email-masked]] On Behalf Of Richard Bin liu
Sent: Thursday, October 18, 2012 12:44 PM
To: [email-masked]
Subject: [nvo3] draft-khasnabish-vmmi-problems-02

Greeting all,

Thanks,

---

From: Richard Bin liu <[email-masked]>
Date: Fri, 26 Oct 2012 08:17:32 +0800

Hi Dave,
Thank you for your comments, I will put them as update file reference.
As your comment:

[DWM] Is the intent to standardize the steps of a VM migration? It’s a very
vendor specific operation.

I have read some standards, found that also provides state machine process,
although there will be different when people complete the actual function.
Could I  change it to optional reference suggest?

“In addition, TRILL…, nor it provides any solution to the requirement of
supporting more than 4K VLANs.”****

** **

[DWM] You should take a look at the draft-ietf-trill-fine-labeling-00
(December 8, 2011) which added the EX-Tag. ****

Yes, I had read the draft, so, the description should be deleted.

And other repeated place should be merged.

Many thanks for guidance.

Regard.

Richard
On Fri, Oct 26, 2012 at 4:22 AM, Martin, David <[email-masked]> wrote:

>  Hi Richard,****
>
> ** **
>
> A few comments in blue text/[DWM].****
>
> ** **
>
> Clause 3.5 The Evolution Problems…****
>
> ** **
>
> “Note that although Ethernet and IP protocols are meant to support
> arbitrary topology, these Layer-2 and Layer-3 network protocols are not
> flexible enough for use in Data Center environments.  The lack of
> flexibility may result in lack of scalability, management difficulties,
> inflexible communications, and poor fault tolerance.”****
>
> ** **
>
> [DWM] This is a rather broad statement and should really be followed by a
> detailed critique of the issues – although I suppose that’s the goal of the
> document. In any case, just so you’re aware, besides the Equal Cost Trees
> outlined in SPB (which are Backbone Edge Bridge to BEB in extent), there is
> a project in-flight to add support for hop-by-hop ECMP (P802.1Qbp).****
>
> ** **
>
> Clauses 3.6.1, 3.6.2 Requirements…****
>
> ** **
>
> [DWM] I didn’t find any actual equipment requirements, just brief
> descriptions of what the equipment does.****
>
> ** **
>
> Clause 3.6.4  Fault Tolerance…****
>
> ** **
>
> “After the VM migration, it is required to consider the impact on the
> switching network, such as whether the new network environment will have
> the problem of insufficient bandwidth.”****
>
> ** **
>
> [DWM] Good one, although it isn’t really fault-related. The Virtual
> Station Interface Discovery and Configuration Protocol (VDP) that Pat
> mentioned from 802.1Qbg can be used by the target HV to register the
> migrating VM/VSI with the physical port on the attached bridge (e.g., ToR).
> That protocol exchange includes the VM/VSI Type which can be used by the
> switching network management system to allocate resources accordingly. All
> this can be performed at the destination prior to the VM migration using
> the ‘pre-associate with resource reservation’ VDP TLV type.****
>
> ** **
>
> Clause 3.6.10.2 Requirements for Live Migration of Virtual Machines****
>
> ** **
>
> [DWM] The list should include more than just the requirement to have an IP
> address. For example, the maximum latency between source DC and destination
> DC hypervisors is 10ms RTT for a VMware vSphere 5.0 vMotion.****
>
> ** **
>
> Clause 3.6.11 Access and Migration of VMs without users' Perception****
>
> ** **
>
> [DWM] There should be a target requirement for the application pause time
> (aka VM blackout time) experienced by the user, for example <4 seconds.***
> *
>
> ** **
>
> Clause 3.6.13 The East-West Traffic Problem****
>
> ** **
>
> “In addition, TRILL…, nor it provides any solution to the requirement of
> supporting more than 4K VLANs.”****
>
> ** **
>
> [DWM] You should take a look at the draft-ietf-trill-fine-labeling-00
> (December 8, 2011) which added the EX-Tag. ****
>
> ** **
>
> Clause 3.6.18 Routing Control - Multicast Processing****
>
> ** **
>
> “There also exist additional overheads because each customer multicast
> group is associated with the forwarding tree network throughout the
> Ethernet switching network. Consequently, development and standardization
> of efficient Layer-2 multicast mechanism to support intra- and inter-DC VM
> mobility would be very useful.”****
>
> ** **
>
> [DWM] I don’t follow these statements. Efficient multicast is an advantage
> of a bridged network, as opposed to source replication. What specific
> improvement would VM mobility need?****
>
> ** **
>
> Clause 3.6.19 Problems and Requirement related to DMTF****
>
> ** **
>
> [DWM] Why is this in an IETF draft?****
>
> ** **
>
> Clauses 4.1.1 – 4.1.3 Migration…****
>
> ** **
>
> [DWM] While the inspection, authentication and resource allocation steps
> are necessary, a HV can only do what’s within its span of control
> (leveraging 802.1Qbg). Only the VM cluster manager and higher level DC
> management applications (e.g., vCloud Director) have a broad enough scope
> to do all this.****
>
> ** **
>
> Clause 4.1.4 Standardization of Migration State****
>
> ** **
>
> [DWM] Is the intent to standardize the steps of a VM migration? It’s a
> very vendor specific operation.****
>
> ** **
>
> Clauses 4.2 – 4.3 Mobility…****
>
> ** **
>
> [DWM] Suggest deleting these as they repeat material from clause 3.****
>
> ** **
>
> …Dave****
>
> ** **
>
> *Sent:* Thursday, October 18, 2012 12:44 PM
>
> *To:* [email-masked]