Subject: [nvo3] comment on rekhter-vm-mobility

From: Lucy yong <[email-masked]>
Date: Mon, 1 Oct 2012 19:16:56 +0000

Hi Yakov,

It is carefully written about L2-based CUG.

The scenarios that the draft describe are about that VMs in a CUG are exposed to a physical network, as it is referred to "L2 physical domain" in the doc.  To use NVO3 framework terms, it is the case when NVE on TOR and VM on servers. The connection between Servers and TOR is via a physical wire where VLAN is implemented. The draft describes the necessary rules of usage of VLAN-IDs to enable VM move in a L2-based CUG. I agree these rules.

Comment I want to give is that, one motivation to do NVO for DC is to create a true virtual network  environment for the VMs in a closed user group to communicate and to decouple the virtual environment from the physical DC networks which has a lot of constrain to support VM mobility.  You draft has specified these necessary constrains.  However, if we just want to implement in this way, we do not achieve our objective here: to create a true virtual network environment that is decoupled from a physical network.

It is clear that, in nvo3, we want to support that NVE is on Server. In this case, it will create a true virtual networking for VM communications. The traffic from a VM will not directly expose on a wire and DC switches any more. Thus, VM mobility does not face these VLAN ID issues and no need such constrains. It will be good for the draft to point out or clarify that.

Assuming an  NVE is on server,  a VM can be moved from one server to another if NVEs on both servers are members of the same VN or if NVEs on the both servers are member of different VNs but two VNs have interconnection. I think that the draft should mention this case and distinct it from when NVE is on ToRs.

In section 3.3, the scenarios should cover both when another server is not in the L2 physical domain or in the different  L2 physical domain that are connected to this L2 physical domain.  IMO, two have different. The first case need to let NVE on server to join the L3 physical domain first.

I think, this draft motivates us more to have NVE on server when DC operator want to a lot of vm mobility for resource and performance optimization.  Without NVE on server, you have these constrains (or rules) to deal with. IETF is to work out the solutions that allow these configuration and device still can interwork under different configurations. It is up to DC operator to chose which configuration is proper for their business.

I agree, regardless of NVE on server or on ToR, the issue in section 3.4 we need to solve in nvo3 solution.

Regards,
Lucy

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 3 Oct 2012 19:48:22 +0000

Lucy,

For VMs instantiated with applications which need to sit in multiple subnets, the VIDs are encoded in the data frames sent out from the VMs. For this scenario, even if the NVE is on server, VIDs are there between VMs and Virtual Switch.

Linda

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Monday, October 01, 2012 2:17 PM
To: Yakov Rekhter
Cc: [email-masked]
Subject: [nvo3] comment on rekhter-vm-mobility

Hi Yakov,

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Wed, 3 Oct 2012 19:56:59 +0000

Linda,

Regardless the frame from VM having VID or not, the frame is encapsulated at NVE and sent to a peer NVE.  If peer NVE is also on a server, it can pass the frame to VM directly based on the destination address. Therefore, the VID is not used at all.

I agree if the peer NVE is on ToR, it will send the frame to a VLAN identified by VID in order to reach that VM. Thus, the rules are caused by physical network configuration.  Is that right?

That is my point. The draft should clarify this. Do you agree?

Lucy

From: Linda Dunbar
Sent: Wednesday, October 03, 2012 2:48 PM
To: Lucy yong; Yakov Rekhter
Cc: [email-masked]
Subject: RE: comment on rekhter-vm-mobility

Lucy,

Linda

Hi Yakov,

Regards,
Lucy

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 3 Oct 2012 20:09:40 +0000

Lucy,

If the destination VMs also expect VIDs in the data frame, the target NVE (or egress NVE) should not take away the VID embedded in the data frames, regardless where NVE is.

The point is that the VIDs used by those VMs are globally significant.

LInda

From: Lucy yong
Sent: Wednesday, October 03, 2012 2:57 PM
To: Linda Dunbar; Yakov Rekhter
Cc: [email-masked]
Subject: RE: comment on rekhter-vm-mobility

Linda,

Lucy

From: Linda Dunbar

Lucy,

Linda

Hi Yakov,

Regards,
Lucy

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 3 Oct 2012 20:13:14 +0000

One more point, if destination VMs don't expect VIDs, then the NVE can remove the VID encoded in the data frame after the decapsulation, regardless where NVE is. Of course, if NVE is many hops away from the VMs, there is higher chance that the local VID is needed in the data frame after the decapsulation.

Linda

From: Linda Dunbar
Sent: Wednesday, October 03, 2012 3:10 PM
To: Lucy yong; Yakov Rekhter
Cc: [email-masked]
Subject: RE: comment on rekhter-vm-mobility

Lucy,

LInda

From: Lucy yong

Linda,

Lucy

From: Linda Dunbar

Lucy,

Linda

Hi Yakov,

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Wed, 3 Oct 2012 20:28:13 +0000

Linda,

NVE can transparently pass the VID. No problem at all. This should be default mode. If you do not want NVE pass a frame with VID to VM, you have to provision the NVE to peel off before sending.  In short, NVE on server does not use the VID on a frame to any purpose.

I think that we are agree that when all the NVEs are on servers, these rules are not necessary for VM mobility, is that right?

Lucy

From: Linda Dunbar

Lucy,

LInda

From: Lucy yong

Linda,

Lucy

From: Linda Dunbar

Lucy,

Linda

Hi Yakov,

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Thu, 4 Oct 2012 13:47:26 +0000

Hi Linda and Yakov,

Beside previous comment. I also want to add other two comments or say two suggestions.

1)      Suggest to add a new section (section 3.5)  to address policy impact. It can describe that although a virtual overlay network can be implemented  with a full mesh topology in general, it is possible to implement with different topologies by using policy. This is very often cases in IP/MPLS L3VPN, which allows tenant implementing firewall, security control, gateway function, and etc in certain places. However, the policy implemented topology may cause the performance change significantly when moving VMs. For example, when an NVO is implemented with hub-spoke topology, moving a VM from a Hub NVE place to a spoke NVE place will conduct  the communication from a VM in other spoke NVE places to go through the hub-NVE first, which may cause observable traffic delay. This performance change is caused by the policy, which is different from the performance change due to physical location.  In addition, another VM on the same spoke NVE now can directly talk to the moved VM without going through hub-NVE, which may impact the initial design goal.  I think this is fairly important to describe in this document, so DC operator has to pay attention on these factors when moving VMs.

2)      The doc. should mention that when a VM moves from one server to another server, if the destination virtual network is implemented with different address family, VM IP address has to change to match the destination network supported    address family. For example, a VM with IPv4 address is moved to an virtual network that supports IPv6 only. (I know this is obvious, but it should be the rule)

It is good we have one document to capture all VM mobility issues that DC operators need to know.

Regards,
Lucy

From: Linda Dunbar
Sent: Wednesday, October 03, 2012 3:13 PM
To: Linda Dunbar; Lucy yong; Yakov Rekhter
Cc: [email-masked]
Subject: RE: comment on rekhter-vm-mobility

Linda

From: Linda Dunbar

Lucy,

LInda

From: Lucy yong

Linda,

Lucy

From: Linda Dunbar

Lucy,

Linda

Hi Yakov,

Regards,
Lucy

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 4 Oct 2012 07:57:09 -0700

Lucy,

> The scenarios that the draft describe are about that VMs in a CUG are exposed
> to a physical network, as it is referred to "L2 physical domain" in the 
> doc.  To use NVO3 framework terms, it is the case when NVE on TOR and VM on
> servers. The connection between Servers and TOR is via a physical wire where
> VLAN is implemented. The draft describes the necessary rules of usage of
> VLAN-IDs to enable VM move in a L2-based CUG. I agree these rules.

The draft covers both the "NVE on ToR and VM on server" case, and
the case where both NVE and VM are on server. The former is described
as a "non-trivial L2 physical domain"; the latter as "trivial L2
physical domain".

>From 2.1 of the draft:

   A trivial L2 physical domain consists of just one server.
  
> Comment I want to give is that, one motivation to do NVO for DC is to create
> a true virtual network  environment for the VMs in a closed user group to
> communicate and to decouple the virtual environment from the physical DC 
> networks which has a lot of constrain to support VM mobility.  You draft has
> specified these necessary constrains.  However, if we just want to implement
> in this way, we do not achieve our objective here: to create a true virtual
> network environment that is decoupled from a physical network.
> 
> It is clear that, in nvo3, we want to support that NVE is on Server. In this
> case, it will create a true virtual networking for VM communications. The=
> traffic from a VM will not directly expose on a wire and DC switches any 
> more. Thus, VM mobility does not face these VLAN ID issues and no need such 
> constrains. It will be good for the draft to point out or clarify that.

The draft does this - from 3.1:

   This document assumes that within a trivial L2 physical domain
   traffic from/to VMs that are in this domain may not have VLAN-IDs at
   all.

> Assuming an  NVE is on server,  a VM can be moved from one server to another
> if NVEs on both servers are members of the same VN or if NVEs on the both
> servers are member of different VNs but two VNs have interconnection. I think
> that the draft should mention this case and distinct it from when NVE 
> is on ToRs.
> 
> In section 3.3, the scenarios should cover both when another server is not =
> in the L2 physical domain or in the different  L2 physical domain that are =
> connected to this L2 physical domain.  IMO, two have different. The first c=
> ase need to let NVE on server to join the L3 physical domain first.

Could you please propose the text that you think is necessary.

> I think, this draft motivates us more to have NVE on server when DC operator
> want to a lot of vm mobility for resource and performance optimization.  
> Without NVE on server, you have these constrains (or rules) to deal with. 
> IETF is to work out the solutions that allow these configuration and device 
> still can interwork under different configurations. It is up to DC operator
> to chose which configuration is proper for their business.
> 

Ok.

Yakov.

> 
> Regards,
> Lucy
> 
> 
> 
> 
> 
> 
> 
> 
> --_000_2691CE0099834E4A9C5044EEC662BB9D4481598Ddfweml505mbx_
> Content-Type: text/html; charset="us-ascii"
> Content-Transfer-Encoding: quoted-printable
> 
> <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
> osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
> xmlns:x=3D"urn:schemas-microsoft-com:office:excel" xmlns:p=3D"urn:schemas-m=
> icrosoft-com:office:powerpoint" xmlns:a=3D"urn:schemas-microsoft-com:office=
> :access" xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s=3D"=
> uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs=3D"urn:schemas-microsof=
> t-com:rowset" xmlns:z=3D"#RowsetSchema" xmlns:b=3D"urn:schemas-microsoft-co=
> m:office:publisher" xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadshee=
> t" xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" xmlns=
> :odc=3D"urn:schemas-microsoft-com:office:odc" xmlns:oa=3D"urn:schemas-micro=
> soft-com:office:activation" xmlns:html=3D"http://www.w3.org/TR/REC-html40" =
> xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc=3D"http://m=
> icrosoft.com/officenet/conferencing" xmlns:D=3D"DAV:" xmlns:Repl=3D"http://=
> schemas.microsoft.com/repl/" xmlns:mt=3D"http://schemas.microsoft.com/share=
> point/soap/meetings/" xmlns:x2=3D"http://schemas.microsoft.com/office/excel=
> /2003/xml" xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd" xmlns:ois=
> =3D"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir=3D"http://=
> schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds=3D"http://www.w3=
> .org/2000/09/xmldsig#" xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint=
> /dsp" xmlns:udc=3D"http://schemas.microsoft.com/data/udc" xmlns:xsd=3D"http=
> ://www.w3.org/2001/XMLSchema" xmlns:sub=3D"http://schemas.microsoft.com/sha=
> repoint/soap/2002/1/alerts/" xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"=
>  xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" xmlns:sps=3D"http://=
> schemas.microsoft.com/sharepoint/soap/" xmlns:xsi=3D"http://www.w3.org/2001=
> /XMLSchema-instance" xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/so=
> ap" xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udc=
> p2p=3D"http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf=3D"http:/=
> /schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss=3D"http://sche=
> mas.microsoft.com/office/2006/digsig-setup" xmlns:dssi=3D"http://schemas.mi=
> crosoft.com/office/2006/digsig" xmlns:mdssi=3D"http://schemas.openxmlformat=
> s.org/package/2006/digital-signature" xmlns:mver=3D"http://schemas.openxmlf=
> ormats.org/markup-compatibility/2006" xmlns:m=3D"http://schemas.microsoft.c=
> om/office/2004/12/omml" xmlns:mrels=3D"http://schemas.openxmlformats.org/pa=
> ckage/2006/relationships" xmlns:spwp=3D"http://microsoft.com/sharepoint/web=
> partpages" xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/20=
> 06/types" xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/200=
> 6/messages" xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/Sli=
> deLibrary/" xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortal=
> Server/PublishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" xmlns:=
> st=3D"&#1;" xmlns=3D"http://www.w3.org/TR/REC-html40">
> <head>
> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
> >
> <meta name=3D"Generator" content=3D"Microsoft Word 12 (filtered medium)">
> <style><!--
> /* Font Definitions */
> @font-face
> 	{font-family:SimSun;
> 	panose-1:2 1 6 0 3 1 1 1 1 1;}
> @font-face
> 	{font-family:"Cambria Math";
> 	panose-1:2 4 5 3 5 4 6 3 2 4;}
> @font-face
> 	{font-family:Calibri;
> 	panose-1:2 15 5 2 2 2 4 3 2 4;}
> @font-face
> 	{font-family:"\@SimSun";
> 	panose-1:2 1 6 0 3 1 1 1 1 1;}
> /* Style Definitions */
> p.MsoNormal, li.MsoNormal, div.MsoNormal
> 	{margin:0in;
> 	margin-bottom:.0001pt;
> 	font-size:11.0pt;
> 	font-family:"Calibri","sans-serif";}
> a:link, span.MsoHyperlink
> 	{mso-style-priority:99;
> 	color:blue;
> 	text-decoration:underline;}
> a:visited, span.MsoHyperlinkFollowed
> 	{mso-style-priority:99;
> 	color:purple;
> 	text-decoration:underline;}
> span.EmailStyle17
> 	{mso-style-type:personal-compose;
> 	font-family:"Calibri","sans-serif";
> 	color:windowtext;}
> .MsoChpDefault
	{mso-style-type:export-only;}
> @page WordSection1
> 	{size:8.5in 11.0in;
> 	margin:1.0in 1.0in 1.0in 1.0in;}
> div.WordSection1
> 	{page:WordSection1;}
> --></style><!--[if gte mso 9]><xml>
> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
> </xml><![endif]--><!--[if gte mso 9]><xml>
> <o:shapelayout v:ext=3D"edit">
> <o:idmap v:ext=3D"edit" data=3D"1" />
> </o:shapelayout></xml><![endif]-->
> </head>
> <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
> <div class=3D"WordSection1">
> <p class=3D"MsoNormal">Hi Yakov,<o:p></o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">It is carefully written about L2-based CUG. <o:p></o=
> :p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">The scenarios that the draft describe are about that=
>  VMs in a CUG are exposed to a physical network, as it is referred to &#822=
> 0;L2 physical domain&#8221; in the doc. &nbsp;To use NVO3 framework terms, =
> it is the case when NVE on TOR and VM on servers. The
>  connection between Servers and TOR is via a physical wire where VLAN is im=
> plemented. The draft describes the necessary rules of usage of VLAN-IDs to =
> enable VM move in a L2-based CUG. I agree these rules.<o:p></o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">Comment I want to give is that, one motivation to do=
>  NVO for DC is to create a true virtual network &nbsp;environment for the V=
> Ms in a closed user group to communicate and to decouple the virtual enviro=
> nment from the physical DC networks which
>  has a lot of constrain to support VM mobility. &nbsp;You draft has specifi=
> ed these necessary constrains.&nbsp; However, if we just want to implement =
> in this way, we do not achieve our objective here: to create a true virtual=
>  network environment that is decoupled from
>  a physical network.<o:p></o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">It is clear that, in nvo3, we want to support that N=
> VE is on Server. In this case, it will create a true virtual networking for=
>  VM communications. The traffic from a VM will not directly expose on a wir=
> e and DC switches any more. Thus,
>  VM mobility does not face these VLAN ID issues and no need such constrains=
> . It will be good for the draft to point out or clarify that.<o:p></o:p></p=
> >
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">Assuming an &nbsp;NVE is on server, &nbsp;a VM can b=
> e moved from one server to another if NVEs on both servers are members of t=
> he same VN or if NVEs on the both servers are member of different VNs but t=
> wo VNs have interconnection. I think that the
>  draft should mention this case and distinct it from when NVE is on ToRs.<o=
> :p></o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">In section 3.3, the scenarios should cover both when=
>  another server is not in the L2 physical domain or in the different&nbsp; =
> L2 physical domain that are connected to this L2 physical domain. &nbsp;IMO=
> , two have different. The first case need to
>  let NVE on server to join the L3 physical domain first.<o:p></o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">I think, this draft motivates us more to have NVE on=
>  server when DC operator want to a lot of vm mobility for resource and perf=
> ormance optimization. &nbsp;Without NVE on server, you have these constrain=
> s (or rules) to deal with. IETF is to work
>  out the solutions that allow these configuration and device still can inte=
> rwork under different configurations. It is up to DC operator to chose whic=
> h configuration is proper for their business.
> <o:p></o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">I agree, regardless of NVE on server or on ToR, the =
> issue in section 3.4 we need to solve in nvo3 solution.<o:p></o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">Regards,<o:p></o:p></p>
> <p class=3D"MsoNormal">Lucy<o:p></o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> </body>
> </html>
> 
> --_000_2691CE0099834E4A9C5044EEC662BB9D4481598Ddfweml505mbx_--

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 4 Oct 2012 08:31:44 -0700

Lucy,

> Hi Linda and Yakov,
> 
> Beside previous comment. I also want to add other two comments or say two s=
> uggestions.
> 
> 
> 1)      Suggest to add a new section (section 3.5)  to address policy impact.
> It can describe that although a virtual overlay network can be implemented
> with a full mesh topology in general, it is possible to implement with  
> different topologies by using policy. This is very often cases in IP/MPLS 
> L3VPN, which allows tenant implementing firewall, security control, gateway 
> function, and etc in certain places. However, the policy implemented topology
> may cause the performance change significantly when moving VMs. For example,
> when an NVO is implemented with hub-spoke topology, moving a VM from a 
> Hub NVE place to a spoke NVE place will conduct  the communication from a  
> VM in other spoke NVE places to go through the hub-NVE first, which may cause
> observable traffic delay. This performance change is caused by the policy,
> which is different from the performance change due to physical location.=

What you described in the above seems like changing VM from being a hub
to being a spoke. This has nothing to do with VM mobility, as changing
VM from being a hub to being a spoke could be accomplished without
moving VM at all, but by just changing import/export RTs of the EVI/VRF
associated with the VM.

>   In addition, another VM on the same spoke NVE now can directly talk to the
> moved VM without going through hub-NVE, which may impact the initial design
> goal.  I think this is fairly important to describe in this document, so
> DC operator has to pay attention on these factors when moving VMs.
> 

Two spokes should not allow to talk to each other directly, irrespective
of the location of these two spokes. And if they do, then there is
an implementation bug. As a side comment, in the context of
"usual" 2547 VPN  that means that even if two spokes are connected to
the same PE, they can not talk to each other directly (via this PE).

> 2)      The doc. should mention that when a VM moves from one server to ano=
> ther server, if the destination virtual network is implemented with differe=
> nt address family, VM IP address has to change to match the destination net=
> work supported    address family. For example, a VM with IPv4 address is mo=
> ved to an virtual network that supports IPv6 only. (I know this is obvious,=
>  but it should be the rule)

The scope of the draft is on seamless mobility, and seamless mobility,
by definition, means that VM's IP address does not change. From
section 2:

                                        ....  seamless mobility is
   defined as the ability to move a VM from one server in the data
   center to another server in the same or different data center, while
   retaining the IP and MAC address of the VM.

Changing VM's IP address family would certainly imply changing VM's
IP address, and thus is outside the scope of the draft.

> It is good we have one document to capture all VM mobility issues that DC o=
> perators need to know.

Yakov.

> One more point, if destination VMs don't expect VIDs, then the NVE can remo=
> ve the VID encoded in the data frame after the decapsulation, regardless wh=
> ere NVE is. Of course, if NVE is many hops away from the VMs, there is high=
> er chance that the local VID is needed in the data frame after the decapsul=
> ation.
> 
> Linda
> 
> From: Linda Dunbar
> If the destination VMs also expect VIDs in the data frame, the target NVE (=
> or egress NVE) should not take away the VID embedded in the data frames, re=
> gardless where NVE is.
> 
> Regardless the frame from VM having VID or not, the frame is encapsulated a=
> t NVE and sent to a peer NVE.  If peer NVE is also on a server, it can pass=
>  the frame to VM directly based on the destination address. Therefore, the =
> VID is not used at all.
> 
> I agree if the peer NVE is on ToR, it will send the frame to a VLAN identif=
> ied by VID in order to reach that VM. Thus, the rules are caused by physica=
> l network configuration.  Is that right?
> 
> For VMs instantiated with applications which need to sit in multiple subnet=
> s, the VIDs are encoded in the data frames sent out from the VMs. For this =
> scenario, even if the NVE is on server, VIDs are there between VMs and Virt=
> ual Switch.
> 
> Linda
> 
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Luc=
> y yong
> The scenarios that the draft describe are about that VMs in a CUG are expos=
> ed to a physical network, as it is referred to "L2 physical domain" in the =
> doc.  To use NVO3 framework terms, it is the case when NVE on TOR and VM on=
>  servers. The connection between Servers and TOR is via a physical wire whe=
> re VLAN is implemented. The draft describes the necessary rules of usage of=
> Comment I want to give is that, one motivation to do NVO for DC is to creat=
> e a true virtual network  environment for the VMs in a closed user group to=
>  communicate and to decouple the virtual environment from the physical DC n=
> etworks which has a lot of constrain to support VM mobility.  You draft has=
>  specified these necessary constrains.  However, if we just want to impleme=
> nt in this way, we do not achieve our objective here: to create a true virt=
> ual network environment that is decoupled from a physical network.
> 
> It is clear that, in nvo3, we want to support that NVE is on Server. In thi=
> s case, it will create a true virtual networking for VM communications. The=
>  traffic from a VM will not directly expose on a wire and DC switches any m=
> ore. Thus, VM mobility does not face these VLAN ID issues and no need such =
> Assuming an  NVE is on server,  a VM can be moved from one server to anothe=
> r if NVEs on both servers are members of the same VN or if NVEs on the both=
>  servers are member of different VNs but two VNs have interconnection. I th=
> ink that the draft should mention this case and distinct it from when NVE i=
> s on ToRs.
> 
> I think, this draft motivates us more to have NVE on server when DC operato=
> r want to a lot of vm mobility for resource and performance optimization.  =
> Without NVE on server, you have these constrains (or rules) to deal with. I=
> ETF is to work out the solutions that allow these configuration and device =
> still can interwork under different configurations. It is up to DC operator=
> I agree, regardless of NVE on server or on ToR, the issue in section 3.4 we=
>  need to solve in nvo3 solution.
> 
> Regards,
> Lucy
> 
> 
> 
> 
> 
> 
> 
> 
> --_000_2691CE0099834E4A9C5044EEC662BB9D44816464dfweml505mbx_
> 	{font-family:Tahoma;
> 	panose-1:2 11 6 4 3 5 4 4 2 4;}
> @font-face
> 	{font-family:"\@SimSun";
> p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
> 	{mso-style-priority:34;
> 	margin-top:0in;
> 	margin-right:0in;
> 	margin-bottom:0in;
> 	margin-left:.5in;
> 	margin-bottom:.0001pt;
> 	font-size:11.0pt;
> 	{mso-style-type:personal;
> span.EmailStyle18
> 	{mso-style-type:personal;
> 	color:#1F497D;}
> span.EmailStyle19
> 	{mso-style-type:personal;
> 	color:#1F497D;}
> span.EmailStyle20
> 	{mso-style-type:personal;
> 	color:#1F497D;}
> span.EmailStyle21
> 	{mso-style-type:personal;
> 	color:#1F497D;}
> span.EmailStyle22
> 	{mso-style-type:personal-reply;
> 	color:#1F497D;}
> 	{mso-style-type:export-only;
> 	font-size:10.0pt;}
> @page WordSection1
> 	{size:8.5in 11.0in;
> /* List Definitions */
> @list l0
> 	{mso-list-id:844978255;
> 	mso-list-type:hybrid;
> 	mso-list-template-ids:1301811554 67698705 67698713 67698715 67698703 67
698=
> 713 67698715 67698703 67698713 67698715;}
> @list l0:level1
> 	{mso-level-text:"%1\)";
> 	mso-level-tab-stop:none;
> 	mso-level-number-position:left;
> 	text-indent:-.25in;}
> @list l1
> 	{mso-list-id:1404599941;
> 	mso-list-type:hybrid;
> 	mso-list-template-ids:1484681866 67698705 67698713 67698715 67698703 67
698=
> 713 67698715 67698703 67698713 67698715;}
> @list l1:level1
> 	{mso-level-text:"%1\)";
> 	mso-level-tab-stop:none;
> 	mso-level-number-position:left;
> 	text-indent:-.25in;}
> @list l2
> 	{mso-list-id:1429733409;
> 	mso-list-type:hybrid;
> 	mso-list-template-ids:-509829154 67698705 67698713 67698715 67698703 67
698=
> 713 67698715 67698703 67698713 67698715;}
> @list l2:level1
> 	{mso-level-text:"%1\)";
> 	mso-level-tab-stop:none;
> 	mso-level-number-position:left;
> 	text-indent:-.25in;}
> ol
> 	{margin-bottom:0in;}
> ul
> 	{margin-bottom:0in;}
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Hi Linda and Yakov,<o:=
> p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Beside previous commen=
> t. I also want to add other two comments or say two suggestions.<o:p></o:p>=
> </span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoListParagraph" style=3D"text-indent:-.25in;mso-list:l1 level=
> 1 lfo2"><![if !supportLists]><span style=3D"color:#1F497D"><span style=3D"m=
> so-list:Ignore">1)<span style=3D"font:7.0pt &quot;Times New Roman&quot;">&n=
> bsp;&nbsp;&nbsp;&nbsp;&nbsp;
> </span></span></span><![endif]><span style=3D"color:#1F497D">Suggest to add=
>  a new section (section 3.5) &nbsp;to address policy impact. It can describ=
> e that although a virtual overlay network can be implemented &nbsp;with a f=
> ull mesh topology in general, it is possible
>  to implement with different topologies by using policy. This is very often=
>  cases in IP/MPLS L3VPN, which allows tenant implementing firewall, securit=
> y control, gateway function, and etc in certain places. However, the policy=
>  implemented topology may cause
>  the performance change significantly when moving VMs. For example, when an=
>  NVO is implemented with hub-spoke topology, moving a VM from a Hub NVE pla=
> ce to a spoke NVE place will conduct &nbsp;the communication from a VM in o=
> ther spoke NVE places to go through the
>  hub-NVE first, which may cause observable traffic delay. This performance =
> change is caused by the policy, which is different from the performance cha=
> nge due to physical location. &nbsp;In addition, another VM on the same spo=
> ke NVE now can directly talk to the moved
>  VM without going through hub-NVE, which may impact the initial design goal=
> .&nbsp; I think this is fairly important to describe in this document, so D=
> C operator has to pay attention on these factors when moving VMs.<o:p></o:p=
> ></span></p>
> <p class=3D"MsoListParagraph"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:=
> p></span></p>
> <p class=3D"MsoListParagraph" style=3D"text-indent:-.25in;mso-list:l1 level=
> 1 lfo2"><![if !supportLists]><span style=3D"color:#1F497D"><span style=3D"m=
> so-list:Ignore">2)<span style=3D"font:7.0pt &quot;Times New Roman&quot;">&n=
> bsp;&nbsp;&nbsp;&nbsp;&nbsp;
> </span></span></span><![endif]><span style=3D"color:#1F497D">The doc. shoul=
> d mention that when a VM moves from one server to another server, if the de=
> stination virtual network is implemented with different address family, VM =
> IP address has to change to match
>  the destination network supported &nbsp;&nbsp;&nbsp;address family. For ex=
> ample, a VM with IPv4 address is moved to an virtual network that supports =
> IPv6 only. (I know this is obvious, but it should be the rule)<o:p></o:p></=
> span></p>
> <p class=3D"MsoListParagraph"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:=
> p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">It is good we have one=
>  document to capture all VM mobility issues that DC operators need to know.=
> <o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Regards,<o:p></o:p></s=
> pan></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Lucy<o:p></o:p></span>=
> </p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <div>
> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
> 0in 0in">
> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:&quot=
> ;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style=3D"font-s=
> ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> Linda Du=
> nbar
> <br>
> <b>Sent:</b> Wednesday, October 03, 2012 3:13 PM<br>
> <b>To:</b> Linda Dunbar; Lucy yong; Yakov Rekhter<br>
> <b>Cc:</b> [email-masked]<br>
> <b>Subject:</b> RE: comment on rekhter-vm-mobility<o:p></o:p></span></p>
> </div>
> </div>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">One more point, if des=
> tination VMs don&#8217;t expect VIDs, then the NVE can remove the VID encod=
> ed in the data frame after the decapsulation, regardless where NVE is. Of c=
> ourse, if NVE is many hops away from the VMs,
>  there is higher chance that the local VID is needed in the data frame afte=
> r the decapsulation.
> <o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Linda<o:p></o:p></span=
> ></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <div style=3D"border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in =
> 4.0pt">
> <div>
> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
> 0in 0in">
> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:&quot=
> ;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style=3D"font-s=
> ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> Linda Du=
> nbar
> <br>
> <b>Sent:</b> Wednesday, October 03, 2012 3:10 PM<br>
> <b>To:</b> Lucy yong; Yakov Rekhter<br>
> <b>Cc:</b> [email-masked]<br>
> <b>Subject:</b> RE: comment on rekhter-vm-mobility<o:p></o:p></span></p>
> </div>
> </div>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Lucy, <o:p></o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">If the destination VMs=
>  also expect VIDs in the data frame, the target NVE (or egress NVE) should =
> not take away the VID embedded in the data frames, regardless where NVE is.
> <o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">&nbsp;<o:p></o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">The point is that the =
> VIDs used by those VMs are globally significant.
> <o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">LInda<o:p></o:p></span=
> ></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <div style=3D"border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in =
> 4.0pt">
> <div>
> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
> 0in 0in">
> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:&quot=
> ;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style=3D"font-s=
> ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> Lucy yon=
> g
> <br>
> <b>Sent:</b> Wednesday, October 03, 2012 2:57 PM<br>
> <b>To:</b> Linda Dunbar; Yakov Rekhter<br>
> <b>Cc:</b> [email-masked]<br>
> <b>Subject:</b> RE: comment on rekhter-vm-mobility<o:p></o:p></span></p>
> </div>
> </div>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Linda,<o:p></o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Regardless the frame f=
> rom VM having VID or not, the frame is encapsulated at NVE and sent to a pe=
> er NVE. &nbsp;If peer NVE is also on a server, it can pass the frame to VM =
> directly based on the destination address.
>  Therefore, the VID is not used at all. <o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">I agree if the peer NV=
> E is on ToR, it will send the frame to a VLAN identified by VID in order to=
>  reach that VM. Thus, the rules are caused by physical network configuratio=
> n. &nbsp;Is that right?<o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">That is my point. The =
> draft should clarify this. Do you agree?<o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Lucy<o:p></o:p></span>=
> </p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <div>
> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
> 0in 0in">
> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:&quot=
> ;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style=3D"font-s=
> ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> Linda Du=
> nbar
> <br>
> <b>Sent:</b> Wednesday, October 03, 2012 2:48 PM<br>
> <b>To:</b> Lucy yong; Yakov Rekhter<br>
> <b>Cc:</b> [email-masked]<br>
> <b>Subject:</b> RE: comment on rekhter-vm-mobility<o:p></o:p></span></p>
> </div>
> </div>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Lucy,<o:p></o:p></span=
> ></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">For VMs instantiated w=
> ith applications which need to sit in multiple subnets, the VIDs are encode=
> d in the data frames sent out from the VMs. For this scenario, even if the =
> NVE is on server, VIDs are there between
>  VMs and Virtual Switch. <o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Linda<o:p></o:p></span=
> ></p>
> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> n></p>
> <div style=3D"border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in =
> 4.0pt">
> <div>
> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
> 0in 0in">
> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:&quot=
> ;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style=3D"font-s=
> ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> nvo3-bou=
> [email-masked] [mailto:[email-masked]]
> <b>On Behalf Of </b>Lucy yong<br>
> <b>Sent:</b> Monday, October 01, 2012 2:17 PM<br>
> <b>To:</b> Yakov Rekhter<br>
> <b>Cc:</b> [email-masked]<br>
> <b>Subject:</b> [nvo3] comment on rekhter-vm-mobility<o:p></o:p></span></p>
> </div>
> </div>
> --_000_2691CE0099834E4A9C5044EEC662BB9D44816464dfweml505mbx_--

---

From: Lucy yong <[email-masked]>
Date: Thu, 4 Oct 2012 16:02:35 +0000

Hi Yakov,

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Yakov Rekhter
> Sent: Thursday, October 04, 2012 10:32 AM
> To: Lucy yong
> Cc: [email-masked]; Linda Dunbar
> Subject: Re: [nvo3] comment on rekhter-vm-mobility
> 
> Lucy,
> 
> > Hi Linda and Yakov,
> >
[Lucy] Let me make it clear. What I want to say is about moving a VM at a hub site to a spoke site. This is not about if an operator want to change a hub site into a spoke site. 
> the same PE, they can not talk to each other directly (via this PE).
[Lucy] good point. Setting a policy at spoke site can achieve that. However, in L3VPN case, SP VPN may not able to limit hosts at CE site to communicate each other in the CE network (without touch VPN at all). Thus, the main purpose of hub-spoke implementation is to prohibit the communication between two spoke sites directly in my opinion.
> IP address, and thus is outside the scope of the draft.
[Lucy] I see. I did not read this carefully. Agree that is outside the scope of the draft.

Lucy
> > schemas.microsoft.com/repl/"
> xmlns:mt=3D"http://schemas.microsoft.com/share=
> point/soap/meetings/"
> xmlns:x2=3D"http://schemas.microsoft.com/office/excel=
> > .org/2000/09/xmldsig#"
> xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint=
> > ://www.w3.org/2001/XMLSchema"
> xmlns:sub=3D"http://schemas.microsoft.com/sha=
> > repoint/soap/2002/1/alerts/"
> xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"=
> > /XMLSchema-instance"
> xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/so=
> > crosoft.com/office/2006/digsig"
> xmlns:mdssi=3D"http://schemas.openxmlformat=
> > om/office/2004/12/omml"
> xmlns:mrels=3D"http://schemas.openxmlformats.org/pa=
> > ckage/2006/relationships"
> xmlns:spwp=3D"http://microsoft.com/sharepoint/web=
> > partpages"
> xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/20=
> > 06/types"
> xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/200=
> > 6/messages"
> xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/Sli=
> > deLibrary/"
> xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortal=
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">Regards,<o:p></o:p></s=
> > pan></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">Lucy<o:p></o:p></span>=
> > </p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <div>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">Linda<o:p></o:p></span=
> > ></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">&nbsp;<o:p></o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">LInda<o:p></o:p></span=
> > ></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">Linda,<o:p></o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">Lucy<o:p></o:p></span>=
> > </p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <div>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">Lucy,<o:p></o:p></span=
> > ></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D">Linda<o:p></o:p></span=
> > ></p>
> > <p class=3D"MsoNormal"><span
> style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
> > n></p>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 4 Oct 2012 09:23:03 -0700

Lucy,

> [Lucy] Let me make it clear. What I want to say is about moving a VM at a hub
> site to a spoke site. This is not about if an operator want to change 
> a hub site into a spoke site. 

If a particular VM suppose to act as a spoke, then one can *not*
move this VM from an EVI/VRF provisioned as a spoke to an EVI/VRF
provisioned as a hub (as doing this would violate policies that
control inter-VM connectivity).

To cover this I would propose to add the following to the draft
(perhaps, as you suggested, in a separate section):

   Moving VM from one L2 physical domain to another means
   (among other things) that the NVE in the new domain that 
   provides connectivity between this VM and VMs in other L2 
   physical domains must be able to implement the policies
   that control connectivity between this VM and VMs in other
   L2 physical domains. In other words, the policies that
   control connectivity between a given VM and its peers
   MUST NOT change as the VM moves from one L2 physical domain
   to another. All of the above is irrespective of whether 
   the L2 physical domains are trivial or not. 

Yakov.

> [Lucy] good point. Setting a policy at spoke site can achieve that. However, 
in L3VPN case, SP VPN may not able to limit hosts at CE site to communicate eac
h other in the CE network (without touch VPN at all). Thus, the main purpose of
 hub-spoke implementation is to prohibit the communication between two spoke si
tes directly in my opinion.

---

From: John E Drake <[email-masked]>
Date: Thu, 4 Oct 2012 16:26:37 +0000

Good

Yours irrespectively,

John

> Sent: Thursday, October 04, 2012 9:23 AM
> To: Lucy yong
> > > From: [email-masked] [mailto:[email-masked]] On
> Behalf
> > > Of Yakov Rekhter
> > > > Beside previous comment. I also want to add other two comments or
> > > > say
> > > two s=
> > > > uggestions.
> > > >
> > > >
> > > > 1)      Suggest to add a new section (section 3.5)  to address
> policy
> > > impact.
> > > > L3VPN, which allows tenant implementing firewall, security
> > > > control,
> > > gateway
> > > > function, and etc in certain places. However, the policy
> > > > implemented
> > > topology
> > > > may cause the performance change significantly when moving VMs.
> > > > For
> > > example,
> > > > when an NVO is implemented with hub-spoke topology, moving a VM
> > > > from
> > > a
> > > > Hub NVE place to a spoke NVE place will conduct  the
> communication
> > > from a
> > > > VM in other spoke NVE places to go through the hub-NVE first,
> > > > which
> > > may cause
> > > > observable traffic delay. This performance change is caused by
> the
> > > policy,
> > > What you described in the above seems like changing VM from being a
> > > hub to being a spoke. This has nothing to do with VM mobility, as
> > > changing VM from being a hub to being a spoke could be accomplished
> > > without moving VM at all, but by just changing import/export RTs of
> > > the EVI/VRF associated with the VM.
> > [Lucy] Let me make it clear. What I want to say is about moving a VM
> > at a hub site to a spoke site. This is not about if an operator want
> > to change a hub site into a spoke site.
> 
> If a particular VM suppose to act as a spoke, then one can *not* move
> this VM from an EVI/VRF provisioned as a spoke to an EVI/VRF
> provisioned as a hub (as doing this would violate policies that control
> inter-VM connectivity).
> 
> > > >   In addition, another VM on the same spoke NVE now can directly
> > > > talk
> > > to the
> > > > moved VM without going through hub-NVE, which may impact the
> > > > initial
> > > design
> > > > goal.  I think this is fairly important to describe in this
> > > > document,
> > > so
> > > Two spokes should not allow to talk to each other directly,
> > > irrespective of the location of these two spokes. And if they do,
> > > then there is an implementation bug. As a side comment, in the
> > > context of "usual" 2547 VPN  that means that even if two spokes are
> > > connected to the same PE, they can not talk to each other directly
> > > (via this PE).
> in L3VPN case, SP VPN may not able to limit hosts at CE site to
> communicate eac h other in the CE network (without touch VPN at all).
> Thus, the main purpose of  hub-spoke implementation is to prohibit the
> communication between two spoke si tes directly in my opinion.
> > >
> > >
> > > > 2)      The doc. should mention that when a VM moves from one
> server
> > > to ano=
> > > > ther server, if the destination virtual network is implemented
> > > > with
> > > differe=
> > > > work supported    address family. For example, a VM with IPv4
> address
> > > is mo=
> > > > ved to an virtual network that supports IPv6 only. (I know this
> is
> > > obvious,=
> > > The scope of the draft is on seamless mobility, and seamless
> > > mobility, by definition, means that VM's IP address does not
> change.
> > > From section 2:
> > >
> > [Lucy] I see. I did not read this carefully. Agree that is outside
> the
> > scope
> of the draft.
> >
> > Lucy
> > >
> > > > One more point, if destination VMs don't expect VIDs, then the
> NVE
> > > can remo=
> > > > ere NVE is. Of course, if NVE is many hops away from the VMs,
> > > > there
> > > is high=
> > > > er chance that the local VID is needed in the data frame after
> the
> > > decapsul=
> > > > ation.
> > > >
> > > > Linda
> > > >
> > > > From: Linda Dunbar
> > > > If the destination VMs also expect VIDs in the data frame, the
> > > > target
> > > NVE (=
> > > > t NVE and sent to a peer NVE.  If peer NVE is also on a server,
> it
> > > can pass=
> > > >  the frame to VM directly based on the destination address.
> > > > Therefore,
> > > the =
> > > > VID is not used at all.
> > > >
> > > > I agree if the peer NVE is on ToR, it will send the frame to a
> > > > VLAN
> > > identif=
> > > > ied by VID in order to reach that VM. Thus, the rules are caused
> > > > by
> > > physica=
> > > > For VMs instantiated with applications which need to sit in
> > > > multiple
> > > subnet=
> > > > s, the VIDs are encoded in the data frames sent out from the VMs.
> > > > For
> > > this =
> > > > scenario, even if the NVE is on server, VIDs are there between
> VMs
> > > and Virt=
> > > > ual Switch.
> > > >
> > > > Linda
> > > >
> > > > From: [email-masked] [mailto:[email-masked]] On
> > > > Behalf
> > > Of Luc=
> > > > y yong
> > > > The scenarios that the draft describe are about that VMs in a CUG
> > > > are
> > > expos=
> > > > ed to a physical network, as it is referred to "L2 physical
> domain"
> > > in the =
> > > > doc.  To use NVO3 framework terms, it is the case when NVE on TOR
> > > > and
> > > VM on=
> > > >  servers. The connection between Servers and TOR is via a
> physical
> > > wire whe=
> > > > re VLAN is implemented. The draft describes the necessary rules
> of
> > > usage of=
> > > > Comment I want to give is that, one motivation to do NVO for DC
> is
> > > > to
> > > creat=
> > > > e a true virtual network  environment for the VMs in a closed
> user
> > > group to=
> > > >  specified these necessary constrains.  However, if we just want
> > > > to
> > > impleme=
> > > > It is clear that, in nvo3, we want to support that NVE is on
> Server.
> > > In thi=
> > > >  traffic from a VM will not directly expose on a wire and DC
> > > > switches
> > > any m=
> > > > ore. Thus, VM mobility does not face these VLAN ID issues and no
> > > > need
> > > such =
> > > > Assuming an  NVE is on server,  a VM can be moved from one server
> > > > to
> > > anothe=
> > > > r if NVEs on both servers are members of the same VN or if NVEs
> on
> > > the both=
> > > >  servers are member of different VNs but two VNs have
> interconnection.
> > > I th=
> > > > ink that the draft should mention this case and distinct it from
> > > > when
> > > NVE i=
> > > > s on ToRs.
> > > >
> > > > In section 3.3, the scenarios should cover both when another
> > > > server
> > > is not =
> > > > connected to this L2 physical domain.  IMO, two have different.
> > > > The
> > > first c=
> > > > I think, this draft motivates us more to have NVE on server when
> > > > DC
> > > operato=
> > > > Without NVE on server, you have these constrains (or rules) to
> > > > deal
> > > with. I=
> > > > ETF is to work out the solutions that allow these configuration
> > > > and
> > > device =
> > > > still can interwork under different configurations. It is up to
> DC
> > > operator=
> > > > I agree, regardless of NVE on server or on ToR, the issue in
> > > > section
> > > 3.4 we=
> > > > uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882"
> > > > xmlns:rs=3D"urn:schemas-
> > > microsof=
> > > > t" xmlns:c=3D"urn:schemas-microsoft-
> com:office:component:spreadsheet"
> > > xmlns=
> > > > soft-com:office:activation"
> > > > xmlns:html=3D"http://www.w3.org/TR/REC-
> > > html40" =
> > > > Server/PublishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-
> com:"
> > > xmlns:=
> > > > <meta http-equiv=3D"Content-Type" content=3D"text/html;
> > > > charset=3Dus-
> > > ascii"=
> > > > >
> > > > 713 67698715 67698703 67698713 67698715;} @list l0:level1
> > > > 	{mso-level-text:"%1\)";
> > > > 	mso-level-tab-stop:none;
> > > > 713 67698715 67698703 67698713 67698715;} @list l1:level1
> > > > 	{mso-level-text:"%1\)";
> > > > 	mso-level-tab-stop:none;
> > > > 713 67698715 67698703 67698713 67698715;} @list l2:level1
> > > > 	{mso-level-text:"%1\)";
> > > > 	mso-level-tab-stop:none;
> > > > </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout
> > > > v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" />
> > > > <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple"> <div
> > > > class=3D"WordSection1"> <p class=3D"MsoNormal"><span
> > > > style=3D"color:#1F497D">Hi Linda and
> > > Yakov,<o:=
> > > > p></o:p></span></p>
> > > > <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Beside
> > > > previous
> > > commen=
> > > > <p class=3D"MsoListParagraph"
> > > > style=3D"text-indent:-.25in;mso-list:l1
> > > level=
> > > > </span></span></span><![endif]><span
> > > > style=3D"color:#1F497D">Suggest
> > > to add=
> > > >  a new section (section 3.5) &nbsp;to address policy impact. It
> > > > can
> > > describ=
> > > > ull mesh topology in general, it is possible  to implement with
> > > > different topologies by using policy. This is very
> > > often=
> > > >  cases in IP/MPLS L3VPN, which allows tenant implementing
> > > > firewall,
> > > securit=
> > > > y control, gateway function, and etc in certain places. However,
> > > > the
> > > policy=
> > > >  the performance change significantly when moving VMs. For
> > > > example,
> > > when an=
> > > >  NVO is implemented with hub-spoke topology, moving a VM from a
> > > > Hub
> > > NVE pla=
> > > > ce to a spoke NVE place will conduct &nbsp;the communication from
> > > > a
> > > VM in o=
> > > > ther spoke NVE places to go through the  hub-NVE first, which may
> > > > cause observable traffic delay. This
> > > performance =
> > > > nge due to physical location. &nbsp;In addition, another VM on
> the
> > > same spo=
> > > > ke NVE now can directly talk to the moved  VM without going
> > > > through hub-NVE, which may impact the initial
> > > design goal=
> > > > .&nbsp; I think this is fairly important to describe in this
> > > > document,
> > > so D=
> > > > <p class=3D"MsoListParagraph"
> > > > style=3D"text-indent:-.25in;mso-list:l1
> > > level=
> > > > </span></span></span><![endif]><span style=3D"color:#1F497D">The
> doc.
> > > shoul=
> > > > d mention that when a VM moves from one server to another server,
> > > > if
> > > the de=
> > > >  the destination network supported &nbsp;&nbsp;&nbsp;address
> family.
> > > For ex=
> > > > <p class=3D"MsoNormal"><span style=3D"color:#1F497D">It is good
> we
> > > have one=
> > > >  document to capture all VM mobility issues that DC operators
> need
> > > > to
> > > know.=
> > > > <o:p></o:p></span></p>
> > > > ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-
> serif&quot;">
> > > Linda Du=
> > > > nbar
> > > > <br>
> > > > <b>Sent:</b> Wednesday, October 03, 2012 3:13 PM<br> <b>To:</b>
> > > > Linda Dunbar; Lucy yong; Yakov Rekhter<br> <b>Cc:</b>
> > > > [email-masked]<br> <b>Subject:</b> RE: comment on rekhter-vm-
> > > mobility<o:p></o:p></span></p>
> > > > </div>
> > > > </div>
> > > > <p class=3D"MsoNormal"><span style=3D"color:#1F497D">One more
> > > > point,
> > > if des=
> > > > tination VMs don&#8217;t expect VIDs, then the NVE can remove the
> > > > VID
> > > encod=
> > > > ed in the data frame after the decapsulation, regardless where
> NVE is.
> > > Of c=
> > > > ourse, if NVE is many hops away from the VMs,  there is higher
> > > > chance that the local VID is needed in the data
> > > frame afte=
> > > > r the decapsulation.
> > > > <o:p></o:p></span></p>
> > > > <div style=3D"border:none;border-left:solid blue
> 1.5pt;padding:0in
> > > 0in 0in =
> > > > 4.0pt">
> > > > <div>
> > > > ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-
> serif&quot;">
> > > Linda Du=
> > > > nbar
> > > > <br>
> > > > <b>Sent:</b> Wednesday, October 03, 2012 3:10 PM<br> <b>To:</b>
> > > > Lucy yong; Yakov Rekhter<br> <b>Cc:</b> [email-masked]<br>
> > > >  also expect VIDs in the data frame, the target NVE (or egress
> > > > NVE)
> > > should =
> > > > not take away the VID embedded in the data frames, regardless
> > > > where
> > > NVE is.
> > > > <o:p></o:p></span></p>
> > > > <div style=3D"border:none;border-left:solid blue
> 1.5pt;padding:0in
> > > 0in 0in =
> > > > 4.0pt">
> > > > <div>
> > > > ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-
> serif&quot;">
> > > Lucy yon=
> > > > g
> > > > <br>
> > > > <b>Sent:</b> Wednesday, October 03, 2012 2:57 PM<br> <b>To:</b>
> > > > Linda Dunbar; Yakov Rekhter<br> <b>Cc:</b> [email-masked]<br>
> > > > <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Regardless
> > > > the
> > > frame f=
> > > > rom VM having VID or not, the frame is encapsulated at NVE and
> > > > sent
> > > to a pe=
> > > > er NVE. &nbsp;If peer NVE is also on a server, it can pass the
> > > > frame
> > > to VM =
> > > >  Therefore, the VID is not used at all. <o:p></o:p></span></p> <p
> > > > class=3D"MsoNormal"><span
> > > > <p class=3D"MsoNormal"><span style=3D"color:#1F497D">I agree if
> > > > the
> > > peer NV=
> > > > E is on ToR, it will send the frame to a VLAN identified by VID
> in
> > > order to=
> > > > n. &nbsp;Is that right?<o:p></o:p></span></p> <p
> > > > class=3D"MsoNormal"><span
> > > > <p class=3D"MsoNormal"><span style=3D"color:#1F497D">That is my
> point.
> > > The =
> > > > draft should clarify this. Do you agree?<o:p></o:p></span></p> <p
> > > > class=3D"MsoNormal"><span
> > > > ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-
> serif&quot;">
> > > Linda Du=
> > > > nbar
> > > > <br>
> > > > <b>Sent:</b> Wednesday, October 03, 2012 2:48 PM<br> <b>To:</b>
> > > > Lucy yong; Yakov Rekhter<br> <b>Cc:</b> [email-masked]<br>
> > > > ith applications which need to sit in multiple subnets, the VIDs
> > > > are
> > > encode=
> > > > d in the data frames sent out from the VMs. For this scenario,
> > > > even
> > > if the =
> > > > NVE is on server, VIDs are there between  VMs and Virtual Switch.
> > > > <o:p></o:p></span></p> <p class=3D"MsoNormal"><span
> > > > <div style=3D"border:none;border-left:solid blue
> 1.5pt;padding:0in
> > > 0in 0in =
> > > > 4.0pt">
> > > > <div>
> > > > ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-
> serif&quot;">
> > > nvo3-bou=
> > > > [email-masked] [mailto:[email-masked]] <b>On Behalf Of
> > > > </b>Lucy yong<br> <b>Sent:</b> Monday, October 01, 2012 2:17
> > > > PM<br> <b>To:</b> Yakov Rekhter<br> <b>Cc:</b> [email-masked]<br>
> > > > <p class=3D"MsoNormal">Hi Yakov,<o:p></o:p></p> <p
> > > > class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> > > > <p class=3D"MsoNormal">It is carefully written about L2-based
> CUG.
> > > <o:p></o=
> > > > :p></p>
> > > >  VMs in a CUG are exposed to a physical network, as it is
> referred
> > > > to
> > > &#822=
> > > > 0;L2 physical domain&#8221; in the doc. &nbsp;To use NVO3
> > > > framework
> > > terms, =
> > > > it is the case when NVE on TOR and VM on servers. The  connection
> > > > between Servers and TOR is via a physical wire where VLAN
> > > is im=
> > > > plemented. The draft describes the necessary rules of usage of
> > > > VLAN-
> > > IDs to =
> > > > enable VM move in a L2-based CUG. I agree these
> > > > rules.<o:p></o:p></p> <p
> class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> > > > <p class=3D"MsoNormal">Comment I want to give is that, one
> > > > motivation
> > > to do=
> > > >  NVO for DC is to create a true virtual network &nbsp;environment
> > > > for
> > > the V=
> > > > Ms in a closed user group to communicate and to decouple the
> > > > virtual
> > > enviro=
> > > > nment from the physical DC networks which  has a lot of constrain
> > > > to support VM mobility. &nbsp;You draft has
> > > specifi=
> > > > in this way, we do not achieve our objective here: to create a
> > > > true
> > > virtual=
> > > >  network environment that is decoupled from  a physical
> > > > network.<o:p></o:p></p> <p
> > > > class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> > > > <p class=3D"MsoNormal">It is clear that, in nvo3, we want to
> > > > support
> > > that N=
> > > >  VM communications. The traffic from a VM will not directly
> expose
> > > > on
> > > a wir=
> > > > <p class=3D"MsoNormal">Assuming an &nbsp;NVE is on server,
> &nbsp;a
> > > > VM
> > > can b=
> > > > he same VN or if NVEs on the both servers are member of different
> > > > VNs
> > > but t=
> > > > wo VNs have interconnection. I think that the  draft should
> > > > mention this case and distinct it from when NVE is on
> > > ToRs.<o=
> > > > :p></o:p></p>
> > > > , two have different. The first case need to  let NVE on server
> to
> > > > join the L3 physical domain
> > > first.<o:p></o:p></p>
> > > > <p class=3D"MsoNormal">I think, this draft motivates us more to
> > > > have
> > > NVE on=
> > > >  server when DC operator want to a lot of vm mobility for
> resource
> > > and perf=
> > > > s (or rules) to deal with. IETF is to work  out the solutions
> that
> > > > allow these configuration and device still
> > > can inte=
> > > > <p class=3D"MsoNormal">I agree, regardless of NVE on server or on
> > > > ToR,
> > > the =
> > > > issue in section 3.4 we need to solve in nvo3
> > > > solution.<o:p></o:p></p> <p
> > > > class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>

---

From: Carter Bullard <[email-masked]>
Date: Thu, 4 Oct 2012 12:35:39 -0400

Gentle people,
These nodes generally won't know their role as a hub or spoke, and there probably isn't a formal connectivity policy.
Successfully moving them shouldn't change their roles, as their topology should generally be preserved as a part of their move ?

Hope all is most excellent,
Carter

Carter Bullard, QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax

On Oct 4, 2012, at 12:23 PM, Yakov Rekhter <[email-masked]> wrote:

>>> irrespective
>>>>    margin-top:0i

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 4 Oct 2012 17:32:22 +0000

Lucy, 

I think you are making it more complicated than it needs to. 

Traditional VPN hub-spoke is defined (or configured) by the network operators. 

But the "Hub-Spoke" relationship among a group of VMs (or hosts) are triggered by hosts (VMs) themselves, such as IGMP Join/Leave, instead of by network operators. 

When hosts (VMs) need to form their own Hub-Spoke relationship, they send out IGMP Join/Leave report. VMs migrated to the new location can send out a new IGMP Report (Join), then all the switches will learn the relationship.

VMware Hypervisor even triggers a fake IGMP Query after a new VM is instantiated on the local server, so the IGMP Report can be sent out quickly for all the switches to update the "hub-spoke" relationship. 

I don't think network operators need to configure the Hub-Spokes relationship for all the VMs groups. There will be too many, not scalable. Besides, VMs leased out to clients are totally under the control of clients, which is not under the control of network operators.

Linda   

> Sent: Thursday, October 04, 2012 11:03 AM
> To: Yakov Rekhter
> Subject: RE: [nvo3] comment on rekhter-vm-mobility
> 
> Hi Yakov,
> 
> hub
> [Lucy] Let me make it clear. What I want to say is about moving a VM at
> a hub site to a spoke site. This is not about if an operator want to
> change a hub site into a spoke site.
> >
> [Lucy] good point. Setting a policy at spoke site can achieve that.
> However, in L3VPN case, SP VPN may not able to limit hosts at CE site
> to communicate each other in the CE network (without touch VPN at all).
> communication between two spoke sites directly in my opinion.
> >
> >
> > > Comment I want to give is that, one motivation to do NVO for DC is
> to
> > creat=
> > > <p class=3D"MsoListParagraph" style=3D"text-indent:-.25in;mso-
> list:l1
> > level=
> > >  to implement with different topologies by using policy. This is
> very
> > often=
> > > <p class=3D"MsoListParagraph" style=3D"text-indent:-.25in;mso-
> list:l1
> > level=
> > >  document to capture all VM mobility issues that DC operators need
> to
> > know.=
> > > <o:p></o:p></span></p>
> > > ed in the data frame after the decapsulation, regardless where NVE
> is.
> > Of c=
> > >  VMs in a CUG are exposed to a physical network, as it is referred
> to
> > &#822=
> > >  connection between Servers and TOR is via a physical wire where
> VLAN
> > is im=
> rules.<o:p></o:p></p>
> > >  VM communications. The traffic from a VM will not directly expose
> on
> > a wir=
> > > <p class=3D"MsoNormal">Assuming an &nbsp;NVE is on server, &nbsp;a
> VM
> > can b=
> solution.<o:p></o:p></p>

---

From: Lucy yong <[email-masked]>
Date: Thu, 4 Oct 2012 18:17:30 +0000

Yakov,

> From: Yakov Rekhter [mailto:[email-masked]]
> Sent: Thursday, October 04, 2012 9:57 AM
> To: Lucy yong
> Cc: Yakov Rekhter; [email-masked]
> Subject: Re: comment on rekhter-vm-mobility
> 
> Lucy,
> 
> > Hi Yakov,
> >
[Lucy] I think using the term "trivial L2 physical domain" to reference the case where  both NVE and VM are on a server is a bit of confusing. In the context of NVO3, we distinguish the overlay network and underlay network. The latter is referred to as a physical network. The text:

   We say that an L2 physical domain contains a given VM (or that a
   given VM is in a given L2 physical domain), if the server presently
   hosting this VM is part of that domain, or the server is connected to
   a ToR that is part of that domain.

Do you say that this text covers the NVE and VM on a server? If NVE on a sever, server and ToR are not part of that domain in my opinion. The text:

   We say that a given L2-based CUG is present within a given data
   center if one or more VMs that are part of that CUG are presently
   hosted by the servers located in that data center.

   In the context of this document when we talk about VLAN-ID used by a
   given VM, we refer to the VLAN-ID carried by the traffic that is
   within the same L2 physical domain as the VM, and that is either
   originated or destined to that VM - e.g., VLAN-ID only has local
   significance within the L2 physical domain, unless it is stated
   otherwise.

This text applies more today situation when L2-basd CUG is not implemented by NVO3. If using NVO3 and all NVEes are on a server, there is no reason for VMs to use VLAN-ID any more. However, if NVE are on ToR switch and VLANs are used between Server and TORs, then you are back to the today's situation. This is why I feel that the document only cover the NVE on ToR case.
>    traffic from/to VMs that are in this domain may not have VLAN-IDs at
[Lucy] OK. Right Assumption. However, there is the text in 3.1:
  If a given VM's Guest OS sends packets that carry VLAN-ID, then when
   the VM moves from one L2 physical domain to another the VLAN-ID used
   by the Guest OS can not change (this is irrespective of whether L2
   physical domains are trivial or non-trivial).
Will this description controversial with the assumption? It is indicate that the VM moves from one L2 physical domain to another L2 physical domain but the VLAN ID used by the Guest OS can not change. If NVE is on a server, will this still relevant? If yes, please explain why? Section 3.3 also has the text that do not apply to when NVE on server.

> Could you please propose the text that you think is necessary.
[Lucy] how about this text for 3.3 first paragraph:
   Consider a scenario where a VM that is a member of a given L2-based
   CUG moves from one server to another. Three cases may exist. 1) another 
   server is in the same L2 physical domain; 2) another server is not in 
   "L2 physical domain"; 3) another server is in different L2 physical 
   domains, where these domains may be located in the same or different 
   data centers. In order to enable communication between this VM and other 
   VMs of that L2-based CUG, the case 1 works naturally; in case 2, 
   the server has to be configured as a member of the same L2-physical domain 
   or a member in a different L2-physical domain first. (the latter, in turn, 
   becomes case 3); in case 3, the new L2 physical domain must become interconnected 
   with the other L2 physical domain(s) that presently contain the rest of 
   the VMs of that CUG, and the interconnect must not violate the L2-based CUG 
   requirement to preserve source and destination MAC addresses in the Ethernet header
   of the packets exchange between this VM and other members of that
   CUG.

> > to chose which configuration is proper for their business.
[Lucy] Do you think if the draft should state out this? If all the NVE are on the servers, then it becomes the case that one "trivial L2 physical domain" spans across multiple servers, in which VM mobility is in a true virtual environment, i.e. no need to deal with these rules described in this document.

Regards,
Lucy
> >
> > I agree, regardless of NVE on server or on ToR, the issue in section
> 3.4 we

---

From: Lucy yong <[email-masked]>
Date: Thu, 4 Oct 2012 18:29:13 +0000

Yakov,

>    the L2 physical domains are trivial or not.

[Lucy] Great, we make some consent here. One comment: there may be policies applying to VMs and policies applying to L2 physical domains. We may want to clarify them and make clear both kinds of policies MUST NOT change as the VM moves from one L2 physical domain to another.

Thanks,
Lucy

> However,
> communicate eac
> mobility,
> the scope
> is to
> need to
> referred to
> > > >  has a lot of constrain to support VM mobility. &nbsp;You draft
> has
> > > specifi=
> expose on
> &nbsp;a VM
> > > >  draft should mention this case and distinct it from when NVE is
> on
> > > ToRs.<o=
> > > > :p></o:p></p>
> > > >  out the solutions that allow these configuration and device
> still
> > > can inte=

---

From: Lucy yong <[email-masked]>
Date: Thu, 4 Oct 2012 18:41:32 +0000

Linda,
> I think you are making it more complicated than it needs to.
[Lucy] I don't think so. Look at Aldrin POC case. Hub-Spoke is used.
> But the "Hub-Spoke" relationship among a group of VMs (or hosts) are
> triggered by hosts (VMs) themselves, such as IGMP Join/Leave, instead
> of by network operators.
[Lucy] my text does not apply to this case. The case I give is that DC may configure a NVO with hub-spoke topology instead of a full mesh topology.
> 
> When hosts (VMs) need to form their own Hub-Spoke relationship, they
> send out IGMP Join/Leave report. VMs migrated to the new location can
> send out a new IGMP Report (Join), then all the switches will learn the
> relationship.
> 
> VMware Hypervisor even triggers a fake IGMP Query after a new VM is
> instantiated on the local server, so the IGMP Report can be sent out
> quickly for all the switches to update the "hub-spoke" relationship.
> 
> 
> I don't think network operators need to configure the Hub-Spokes
> relationship for all the VMs groups. There will be too many, not
> scalable. Besides, VMs leased out to clients are totally under the
> control of clients, which is not under the control of network operators.
[Lucy] Again, look Aldrin's PoC case. I agree that there are a lot of applications that just need a full-mesh topology. However, other applications may need different topologies. That is why it is worth to point out it in the document, so both tenant operator and network operator are aware that vm moves may break the policy designed originally or may cause some unexpected problem.

Lucy
> > Of
> at
> > scope of the draft.

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 4 Oct 2012 12:44:19 -0700

Lucy,

> [Lucy] Great, we make some consent here. One comment: there may be
> policies applying to VMs and policies applying to L2 physical
> domains. We may want to clarify them and make clear both kinds of
> policies MUST NOT change as the VM moves from one L2 physical domain
> to another.

Policies that apply to VMs MUST NOT change. The text I proposed
above makes it clear.

Policies that apply to L2 physical domain may, or may not change. 
But what is important is that such policies should not preclude 
implementing policies that apply to VMs. With this in mind how about the
following variation on the above (the only change is the new penultimate
sentence):

    Moving VM from one L2 physical domain to another means
    (among other things) that the NVE in the new domain that
    provides connectivity between this VM and VMs in other L2
    physical domains must be able to implement the policies
    that control connectivity between this VM and VMs in other
    L2 physical domains. In other words, the policies that
    control connectivity between a given VM and its peers
    MUST NOT change as the VM moves from one L2 physical domain
    to another. Moreover, policies, if any, within the L2
    physical domain that contain a given VM MUST NOT preclude 
    realization of the policies that control connectivity between
    this VM and its peers. All of the above is irrespective of 
    whether the L2 physical domains are trivial or not.

Yakov.

---

From: Lucy yong <[email-masked]>
Date: Thu, 4 Oct 2012 20:25:57 +0000

Hi Yakov,

>     whether the L2 physical domains are trivial or not.
[Lucy] the text is clear and precise. But I read three times to understand it completely. It will be good to give an example to illustrate the statement, which make more intuitive for people to understand or interpret it.

Cheers,
Lucy

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 4 Oct 2012 21:01:53 +0000

For the NVO with hub-spoke topology configured by operator, like the scenario in Aldrin's PoC case, maybe it is better for those VMs not to move. 

Linda 

> Sent: Thursday, October 04, 2012 1:42 PM
> [Lucy] my text does not apply to this case. The case I give is that DC
> may configure a NVO with hub-spoke topology instead of a full mesh
> topology.
> >
> [Lucy] Again, look Aldrin's PoC case. I agree that there are a lot of
> applications that just need a full-mesh topology. However, other
> applications may need different topologies. That is why it is worth to
> point out it in the document, so both tenant operator and network
> operator are aware that vm moves may break the policy designed
> originally or may cause some unexpected problem.
> 
> Lucy
> >
> > Linda
> >
> > > > > with a full mesh topology in general, it is possible to
> implement
> > > > with
> > > > > different topologies by using policy. This is very often cases
> in
> > > > IP/MPLS
> > > > "usual" 2547 VPN  that means that even if two spokes are
> connected
> > to
> > > >                                         ....  seamless mobility
> > > > > It is good we have one document to capture all VM mobility
> issues
> > > > that DC o=
> > > > > perators need to know.
> > > >
> > > > Yakov.
> > > >
> > > > >
> > > > > Regards,
> > > > > Lucy
> > > > >
> > > > >
> > > > >
> > > > > From: Linda Dunbar
> > > > > or egress NVE) should not take away the VID embedded in the
> data
> > > > frames, re=
> > > > > gardless where NVE is.
> > > > >
> > > > > etworks which has a lot of constrain to support VM mobility.
> You
> > > > draft has=
> > > > > nt in this way, we do not achieve our objective here: to create
> a
> > > > true virt=
> > > > > constrains. It will be good for the draft to point out or
> clarify
> > > > that.
> > > > >
> > > > > in the L2 physical domain or in the different  L2 physical
> domain
> > > > that are =
> > > > > icrosoft-com:office:powerpoint" xmlns:a=3D"urn:schemas-
> microsoft-
> > > > com:office=
> > > > > ap"
> xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile"
> > > > xmlns:udc=
> > > > > <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Hi Linda
> and
> > > > Yakov,<o:=
> > > > > p></o:p></span></p>
> > > > > 1 lfo2"><![if !supportLists]><span
> style=3D"color:#1F497D"><span
> > > > style=3D"m=
> > > > > 1 lfo2"><![if !supportLists]><span
> style=3D"color:#1F497D"><span
> > > > style=3D"m=
> style=3D"color:#1F497D">The
> > > doc.
> > > > shoul=
> > > > > ample, a VM with IPv4 address is moved to an virtual network
> that
> > > > supports =
> > > > >  there is higher chance that the local VID is needed in the
> data
> > > > frame afte=
> > > > > r the decapsulation.
> > > > > <o:p></o:p></span></p>
> > > > > <p class=3D"MsoNormal"><span style=3D"color:#1F497D">The point
> is
> > > > that the =
> > > > > <p class=3D"MsoNormal">The scenarios that the draft describe
> are
> > > > about that=
> > > > > ed these necessary constrains.&nbsp; However, if we just want
> to
> > > > implement =
> > > > >  VM mobility does not face these VLAN ID issues and no need
> such
> > > > constrains=
> > > > > <p class=3D"MsoNormal">In section 3.3, the scenarios should
> cover
> > > > both when=
> > > > > L2 physical domain that are connected to this L2 physical
> domain.
> > > > &nbsp;IMO=
> > > > > ormance optimization. &nbsp;Without NVE on server, you have
> these
> > > > constrain=
> > > > > rwork under different configurations. It is up to DC operator
> to
> > > > chose whic=

---

From: Lucy yong <[email-masked]>
Date: Thu, 4 Oct 2012 21:04:34 +0000

Or say, only move a VM within hub site or within a spoke site.
Lucy

> Sent: Thursday, October 04, 2012 4:02 PM
> For the NVO with hub-spoke topology configured by operator, like the
> scenario in Aldrin's PoC case, maybe it is better for those VMs not to
> move.
> 
> Linda
> 
> > > > > > Beside previous comment. I also want to add other two
> comments
> > or
> > > > say
> > > > > two s=
> > > > > > uggestions.
> > > > > >
> > > > > >
> > > > > > It can describe that although a virtual overlay network can
> be
> > > > > implemented
> > > > > > which is different from the performance change due to
> physical
> > > > > location.=
> > > > >
> > > > > What you described in the above seems like changing VM from
> being
> > a
> > > > hub
> > > > >    center to another server in the same or different data
> center,
> > > > while
> > > > > > :access" xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-
> 00AA00C14882"
> > > > > xmlns:s=3D"=
> > > > > > t-com:rowset" xmlns:z=3D"#RowsetSchema"
> xmlns:b=3D"urn:schemas-
> > > > > microsoft-co=
> > > > > > /2003/xml"
> xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd"
> > > > > xmlns:ois=
> > > > > > <meta name=3D"Generator" content=3D"Microsoft Word 12
> (filtered
> > > > > medium)">
> > > > > > <style><!--
> > > > > > /* Font Definitions */
> > > > > > @font-face
> > > > > > 	{font-family:SimSun;
> > > > > >  hub-NVE first, which may cause observable traffic delay.
> This
> > > > > performance =
> > > > > >  VM without going through hub-NVE, which may impact the
> initial
> > > > > design goal=
> > > > > > stination virtual network is implemented with different
> address
> > > > > family, VM =
> > > > > > <p class=3D"MsoNormal"><b><span style=3D"font-
> size:10.0pt;font-
> > > > > family:&quot=
> > > > > > <p class=3D"MsoNormal"><b><span style=3D"font-
> size:10.0pt;font-
> > > > > family:&quot=
> > > > > > <p class=3D"MsoNormal"><b><span style=3D"font-
> size:10.0pt;font-
> > > > > family:&quot=
> style=3D"color:#1F497D">Regardless
> > > the
> > > > > frame f=
> > > > > >  reach that VM. Thus, the rules are caused by physical
> network
> > > > > configuratio=
> > > > > > <p class=3D"MsoNormal"><b><span style=3D"font-
> size:10.0pt;font-
> > > > > family:&quot=
> > > > > > <p class=3D"MsoNormal"><b><span style=3D"font-
> size:10.0pt;font-
> > > > > family:&quot=
> > > > > > e moved from one server to another if NVEs on both servers
> are
> > > > > members of t=
> > > > > > <p class=3D"MsoNormal">I agree, regardless of NVE on server
> or
> > on
> > > > ToR,
> > > > > the =

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 4 Oct 2012 19:25:26 -0400

Hub-spoke and other operator or customer defined virtual network are a
reality today.  However I do think the conversation may have gone off
track.

In regard to VMs, a VM is generally associated to a network service
profile.  In the simplest case the profile implements a basic VLAN.  In
another case, the profile defines the spoke aspect of a hub and spoke VPN.
 When a VM is moved within an administrative domain, it's profile remains
associated to it -- no matter where it moves in the administrative domain,
the nature of its network service remains the same.   It is very important
to understand that in the VM case the service profile is mapped to the VM,
not the NVE, physical port, or other thing.

On Thursday, October 4, 2012, Lucy yong wrote:

> [Lucy] my text does not apply to this case. The case I give is that DC may
> configure a NVO with hub-spoke topology instead of a full mesh topology.
> >
> point out it in the document, so both tenant operator and network operator
> are aware that vm moves may break the policy designed originally or may
> cause some unexpected problem.
>
> Lucy
> >
> > Linda
> >
> > > to communicate each other in the CE network (without touch VPN at
>

---

From: Lucy yong <[email-masked]>
Date: Fri, 5 Oct 2012 14:53:07 +0000

Hi Aldrin,

Thank you so much the explanation. It makes a lot of sense. I think that Yakov new text reflects that aspect.
Please see below.

From: Aldrin Isaac [mailto:[email-masked]]
Sent: Thursday, October 04, 2012 6:25 PM
To: Lucy yong
Cc: Linda Dunbar; Yakov Rekhter; [email-masked]
Subject: [nvo3] comment on rekhter-vm-mobility

Hub-spoke and other operator or customer defined virtual network are a reality today.  However I do think the conversation may have gone off track.

In regard to VMs, a VM is generally associated to a network service profile.  In the simplest case the profile implements a basic VLAN.  In another case, the profile defines the spoke aspect of a hub and spoke VPN.
[Lucy] This is equivalent to say that one is plain vanilla VPLS, another is like e-tree case.
 When a VM is moved within an administrative domain, it's profile remains associated to it -- no matter where it moves in the administrative domain, the nature of its network service remains the same.
[Lucy] fully agree.
 It is very important to understand that in the VM case the service profile is mapped to the VM, not the NVE, physical port, or other thing.
[Lucy] Agree. This is to say that the service features apply to VMs not NVE, or ect.

Thanks again.
Lucy

Lucy

---

From: Yakov Rekhter <[email-masked]>
Date: Sun, 7 Oct 2012 10:52:52 -0700

Lucy,

> [Lucy] I think using the term "trivial L2 physical domain" to reference the
> case where  both NVE and VM are on a server is a bit of confusing. In 
> the context of NVO3, we distinguish the overlay network and underlay 
> network. The latter is referred to as a physical network. The text:
> 
> Do you say that this text covers the NVE and VM on a server? 

Yes.

> If NVE on a sever, server and ToR are not part of that domain in 
> my opinion. 

If NVE is on a server, then the server is part of that domain, although
this is a "trivial L2 physical domain"; the ToR connected to the server
is *not* part of that domain.

> The text:
> This text applies more today situation when L2-basd CUG is not implemented 
> by NVO3. If using NVO3 and all NVEes are on a server, there is no reason 
> for VMs to use VLAN-ID any more. However, if NVE are on ToR switch and 
> VLANs are used between Server and TORs, then you are back to the 
> today's situation. This is why I feel that the document only cover 
> the NVE on ToR case.

The document covers both the case where the NVE is on a ToR, and
when NVE is on servers.

If NVEs are on servers, then we are dealing with trivial L2 physical
domain, in which case (quoting from 3.1 of the draft):

> Will this description controversial with the assumption? It is indicate 
> that the VM moves from one L2 physical domain to another L2 physical 
> domain but the VLAN ID used by the Guest OS can not change. If NVE 
> is on a server, will this still relevant? If yes, please explain why? 
> Section 3.3 also has the text that do not apply to when NVE on server.

I would let Linda comment on this, as the case where VM's Guest OS
sends packets that carry VLAN-ID was proposed by Linda.

>    of the packets exchange between this VM and other members of that

The above seems more like an outline of possible solutions. As such
this text is outside the scope of the draft (as the draft focuses
on listing the problems/issues).
 
Yakov.
 
> [Lucy] Do you think if the draft should state out this? If all the NVE are 
on the servers, then it becomes the case that one "trivial L2 physical domain
" spans across multiple servers, in which VM mobility is in a true virtual en
vironment, i.e. no need to deal with these rules described in this document.

---

From: Lucy yong <[email-masked]>
Date: Sun, 7 Oct 2012 23:19:12 +0000

Hi Yakov,

I read the draft once again to see why we think differently on the text meaning and am glad to find where I missed.

In this document, you define l2-based CUG, l2 physical domain, trivial L2 physical domain and non-trivial l2 physical domain, and the relationship between l2-based BGU and L2 physical domain. These definitions are clear in the introduction.

The first paragraph in 3.1 is to discuss l2 based CUG is implemented by a non-trivial l2 physical network, and specify the rules. This may apply to NVE in TOR and VMs on servers, where server and TOR are part of the non-trivial l2 physical network. 

The second paragraph in 3.1 state that a trail l2 physical domain may not have VLAN ID at all. this implies that it may use too.

Now I think that the problem is the description in the third paragraph, which assumes that Guess OS sends packets that carry VLAN-ID. We have to describe it in trivial l2 physical domain and non-trivial l2 physical domain separately. In a trivial L2 physical domain, the L2 domain identifier is not the VLAN-ID, regardless if the frames from guest OS have VLAN-ID or not; in the non-trivial l2 physical domain, if the VLAN-ID from the guest OS also identifies the domain, i.e. the device in the domain uses it to differ the forwarding tables, it has to follow the rule described in the 3.1 first paragraph; if the VLAN-ID from the guest OS is not used to identify the domain. For example, the former uses c-tag, the latter uses s-tag, this is equivalent to the trivial l2 physical domain case. In both cases, it requires that the domain maintains the Guest VLAN ID transparency. I think this paragraph description mixes these cases together.

The last two paragraphs in 3.1 cause the confusion too    

   This document assumes that within a given non-trivial L2 physical
   domain traffic from/to VMs that are in that domain, and belong to
   different L2-based CUG MUST have different VLAN-IDs.

   The above assumptions about VLAN-IDs are driven by (a) the assumption
   that within a given L2 physical domain VLANs are used to identify
   individual L2-based CUGs, and (b) the need to overcome the limitation
   on the number of different VLAN-IDs.

As I mention above, in the context of nvo3, for a trivial l2 physical domain, VLAN-ID is not used to identify the domain at all. So these statement is not right.

Regards,
Lucy  

> Sent: Sunday, October 07, 2012 12:53 PM
> To: Lucy yong
> Cc: [email-masked]
> > > exposed
> > > > to a physical network, as it is referred to "L2 physical domain"
> in
> > > the
> > > VM on
> > > > VLAN is implemented. The draft describes the necessary rules of
> usage
> > > of
> > > create
> > > > communicate and to decouple the virtual environment from the
> physical
> > > DC
> > > > networks which has a lot of constrain to support VM mobility.
> You
> > > draft has
> > > In this
> > > > case, it will create a true virtual networking for VM
> communications.
> > > The=
> > > any
> > > another
> > > > if NVEs on both servers are members of the same VN or if NVEs on
> the
> > > both
> > > I think
> >    the server has to be configured as a member of the same L2-
> physical doma
> in
> >    or a member in a different L2-physical domain first. (the latter,
> in tur
> n,
> >    becomes case 3); in case 3, the new L2 physical domain must become
> inter
> connected
> >    the VMs of that CUG, and the interconnect must not violate the L2-
> based
> CUG
> >    requirement to preserve source and destination MAC addresses in
> the Ethe
> rnet header
> > > operator
> > > with.
> > > > IETF is to work out the solutions that allow these configuration
> and
> > > device
> > > operator

---

From: Lucy yong <[email-masked]>
Date: Mon, 8 Oct 2012 20:37:14 +0000

Hi Yakov and Linda,

I was in hurry to finish the below e-mail yesterday and made one mistake.

Last two paragraphs in the section 3.1 are stated correctly.
>    on the number of different VLAN-IDs.

This is right for a non-trivial L2 physical domain traffic. It does not apply to a trivial L2 physical domain because the domain does not use the VLAN ID. 

Regards,
Lucy

> > > > wire where
> > >    hosting this VM is part of that domain, or the server is
> connected
> > to
> > >    In the context of this document when we talk about VLAN-ID used
> by
> > a
> > > > > a true virtual network  environment for the VMs in a closed
> user
> > > > group to
> > > > > more. Thus, VM mobility does not face these VLAN ID issues and
> no
> > > > need such
> > > > > that the draft should mention this case and distinct it from
> when
> > NVE
> > > > > is on ToRs.
> > > > >
> > >    server is in the same L2 physical domain; 2) another server is
> not
> > in
> > >    data centers. In order to enable communication between this VM
> and
> > other
> >
> > on the servers, then it becomes the case that one "trivial L2
> physical
> > domain

---

From: Lucy yong <[email-masked]>
Date: Tue, 9 Oct 2012 17:43:07 +0000

Hi Yakov,

I think: this is why the draft does not apply to NVE on server well although it defines that a trivial L2 physical domain consists of just one server.

IMO: when NVE on a server, VMs on a server and the server exist on two different L2 physical domain. Thus when VMs on a server belong to different L2-based CUGs, for L2-based CUGs boundaries among these VMs, it is necessary to use some (l2) mechanisms to segregate the L2 physical domain" that VMs belong, but not on the L2 physical domain that the server belong to. This is the key for the virtualization overlay, i.e. VMs L2 physical domain can exist in a true virtual environment that is completely decoupled from a physical network including the server where VM resides. Thus the text in the draft (copy below) does not apply to this case because it assumes that the server and VMs belong to the same L2 physical domain.

   A physical server connected to a given L2 physical domain may host
   VMs that belong to different L2-based CUGs (while each of these CUGs
   may span multiple L2 physical domains). If an L2 physical domain
   contains servers that host VMs belonging to different L2-based CUGs,
   then enforcing L2-based CUGs boundaries among these VMs within that
   domain is accomplished by relying on Layer 2 mechanisms (e.g.,
   VLANs).

The same issue for this text:

If my logic is correct, when NVE is on server, both server and ToR do not belong to the L2 physical domain that the VM belongs to. Do you agree on this analogy?  Should we differentiate the two l2 physical domains to address NVE on server case? IMO, yes.

I am on vacation for two weeks and off work for a while, which means I barely check e-mail from now.  

Regards,
Lucy