Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-framework-02

From: Qin Wu <[email-masked]>
Date: Fri, 1 Mar 2013 16:28:39 +0800

Dear authors:
I have read this draft and think is is well written and now in good shape.
Here is a few comments I like you to consider:
1. Section 2.1
 You use mangement entity inconsistently, e.g.,
a. management entity
b. management control entity
c. management plane entity
It is better to use consistent terminology.

2.Section 3.2
 Is end system same as tenant system? Are you saying "tenant end system"?

3. Section 5
What do you mean by "NVE managment system"?
Is NVE management system manage VM as well? How is NVE managment system
related to Oracle described in NVO3 problem statement draft?

4. Section 3.1.5.2
For resource, we have storage, compute and network connectivity
I am wondering whether each resource has its own managment system?
e.g., in section 3.1.5.2, you say compute management system? What does
that mean?

5.Section 4.1, 2nd bullet:
s/underkay network/underlay network

6.Section 4.1
What do yo mean by "Tenant flood containment", do you mean constraint flood traffic from tenant
in the local domain behind NVE?

7. In Section 3.3, it said:
"
With "hot" mobility, VM L2 and L3 addresses need to be 
preserved. With "cold" mobility, it may be desired to preserve VM L3
addresses.
"
For cold mobility, it seems you only need to perserve VM L3 addresses,
does it mean you don't move VM from old place to the new place and you just
destroy VM in the old place and create VM in the new place?

Also how does connectivity between tenant system and NVE can be maintained without service interruption?
Are you assuming simultaneous connectivity with old NVE and new NVE will be used?
Is it make before break? It is still not clear to me. But I am very curious to know.

Regards!
-Qin
----- Original Message ----- 
From: "Benson Schliesser" <[email-masked]>
To: <[email-masked]>
Sent: Saturday, February 23, 2013 8:28 AM
Subject: [nvo3] WG Last Call for draft-ietf-nvo3-framework-02

> This email begins a two week working group last call for
> draft-ietf-nvo3-framework-02.
> 
> Please review the draft and post any comments to the NVO3 list.
> 
> This working group last call will end on Friday 08-March-2013.
> 
> Cheers,
> -Benson & Matthew
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Lucy yong <[email-masked]>
Date: Mon, 4 Mar 2013 18:22:19 +0000

I read this version and glad to see adding a section for VM mobility.

Some comments:

1) The introduction states that the rationale for using overlay networks in DC is for building multi-tenant data center networks. It also mentions that a tenant network may consist of one or more virtual networks. However, the architecture and reference model is focusing on single virtual network overlay architecture only. It is lack of describing the architecture and model about multiple virtual networks forming one tenant network, i.e. virtual network overlay interconnections. 

Thus, a question to the WG chair, is this the scope for nvo3 WG? If yes, that is fine and the framework document should make it very clear that the framework document describes architecture model of a virtual network and only applies to a tenant network if it consists of one virtual network that may be implemented by either L2 overlay or L3 overlay. a tenant network that contain more than one virtual networks is outside scope of this document. If not, we can either have one framework draft to architect both, or we have two separate documents, one to intra virtual network overlay and another for inter virtual network overlay. 

Without clarifying this, I am not clear if the framework document is working on a virtual network over L3 infrastructure or a tenant network over L3 infrastructure. Two are not equivalents. 

2) Figure 2 illustrate a logical service connectivity for a single tenant. First of all, it should not show L3 infrastructure in the figure at all. Second, I interpret this figure as one tenant network consisting of four virtual networks, three L2 virtual networks and one L3 virtual network. It should point out that VMs on a L2 virtual network may reside on the same or different servers. The drawing and team LAN make easy to thing they are physical LANs. In fact they are not. VM on LAN11 and VM on LAN12 may reside on the same server. Thus it is very important to state them out clearly. Finally, based on my comment 1), it should clarify what this document address regarding to this figure here.

3) in Section 3.3, "In DC environments utilizing VM technologies, an important feature 
       is that VMs can move from one server to another server in the same 
       or different L2 physical domains (within or across DCs) in a 
       seamless manner." 

What does the L2 physical domain mean here? Why need to mention this?

Cheers,
Lucy

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Benson Schliesser
> Sent: Friday, February 22, 2013 6:28 PM
> To: [email-masked]

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 4 Mar 2013 22:04:50 +0000

I second Lucy's comments. 

Inter Virtual Network communication (or inter subnet communication) is a big part of data centers. Hosts in data center do communicate with external peers. Many tenants to DC need more than one Virtual Network Instances. And hosts in those virtual network instances do communicate with each other and some communicate with peers via public internet. 

IMHO, Inter-subnet communication set NV03 apart from other overlay done by IETF, such as L2VPN, TRILL, LISP, etc. 
Therefore, the framework should have a section on "inter Subnet Communication". 

There are some inter virtual network discussion in http://datatracker.ietf.org/doc/draft-yong-nvo3-frwk-dpreq-addition/. 
The content from this draft should be included in the general framework.

Linda 

> Lucy yong
> Sent: Monday, March 04, 2013 12:22 PM
> To: Benson Schliesser; [email-masked]
> Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-framework-02
> 
> 1) The introduction states that the rationale for using overlay
> networks in DC is for building multi-tenant data center networks. It
> also mentions that a tenant network may consist of one or more virtual
> networks. However, the architecture and reference model is focusing on
> single virtual network overlay architecture only. It is lack of
> describing the architecture and model about multiple virtual networks
> forming one tenant network, i.e. virtual network overlay
> interconnections.
> 
> Thus, a question to the WG chair, is this the scope for nvo3 WG? If yes,
> that is fine and the framework document should make it very clear that
> the framework document describes architecture model of a virtual
> network and only applies to a tenant network if it consists of one
> virtual network that may be implemented by either L2 overlay or L3
> overlay. a tenant network that contain more than one virtual networks
> is outside scope of this document. If not, we can either have one
> framework draft to architect both, or we have two separate documents,
> one to intra virtual network overlay and another for inter virtual
> network overlay.
> 
> Without clarifying this, I am not clear if the framework document is
> working on a virtual network over L3 infrastructure or a tenant network
> over L3 infrastructure. Two are not equivalents.
> 
> 2) Figure 2 illustrate a logical service connectivity for a single
> tenant. First of all, it should not show L3 infrastructure in the
> figure at all. Second, I interpret this figure as one tenant network
> consisting of four virtual networks, three L2 virtual networks and one
> L3 virtual network. It should point out that VMs on a L2 virtual
> network may reside on the same or different servers. The drawing and
> team LAN make easy to thing they are physical LANs. In fact they are
> not. VM on LAN11 and VM on LAN12 may reside on the same server. Thus it
> is very important to state them out clearly. Finally, based on my
> comment 1), it should clarify what this document address regarding to
> this figure here.
>

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 5 Mar 2013 11:36:25 +0100

Lucy,

See my comments below.

Thanks,
Marc 

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Lucy yong
> Sent: Monday, March 04, 2013 7:22 PM
> 1) The introduction states that the rationale for using 
> overlay networks in DC is for building multi-tenant data 
> center networks. It also mentions that a tenant network may 
> consist of one or more virtual networks. However, the 
> architecture and reference model is focusing on single 
> virtual network overlay architecture only. It is lack of 
> describing the architecture and model about multiple virtual 
> networks forming one tenant network, i.e. virtual network 
> overlay interconnections. 

The framework draft does not assume that a tenant network only consists of a single VN.
Multiple VNs can belong to a single tenant.

> 
> Thus, a question to the WG chair, is this the scope for nvo3 
> WG? If yes, that is fine and the framework document should 
> make it very clear that the framework document describes 
> architecture model of a virtual network and only applies to a 
> tenant network if it consists of one virtual network that may 
> be implemented by either L2 overlay or L3 overlay. a tenant 
> network that contain more than one virtual networks is 
> outside scope of this document. If not, we can either have 
> one framework draft to architect both, or we have two 
> separate documents, one to intra virtual network overlay and 
> another for inter virtual network overlay. 
> 
> Without clarifying this, I am not clear if the framework 
> document is working on a virtual network over L3 
> infrastructure or a tenant network over L3 infrastructure. 
> Two are not equivalents. 

Can you point to a specific paragraph where it is mentioned or implied that a tenant network can only consist of a single VN?

> 
> 2) Figure 2 illustrate a logical service connectivity for a 
> single tenant. First of all, it should not show L3 
> infrastructure in the figure at all. Second, I interpret this 

How can routing be avoided when VMs attached to different L2 networks have to communicate?

> figure as one tenant network consisting of four virtual 
> networks, three L2 virtual networks and one L3 virtual 
> network. It should point out that VMs on a L2 virtual network 
> may reside on the same or different servers. The drawing and 
> team LAN make easy to thing they are physical LANs. In fact 
> they are not. VM on LAN11 and VM on LAN12 may reside on the 
> same server. Thus it is very important to state them out 
> clearly. Finally, based on my comment 1), it should clarify 
> what this document address regarding to this figure here.

This is a *logical* and not a physical connectivity view.
Whether VMs are on the same server or not is irrelevant. 
Traffic between VMs in the same L2 network is still switched (by a virtual or physical switch).

> What does the L2 physical domain mean here? Why need to mention this?

I'm fine with changing the sentence to "In DC environments utilizing VM technologies, an important feature is that VMs can move from one server to another server (within or across DCs) in a seamless manner".

> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> > Of Benson Schliesser

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 5 Mar 2013 11:52:53 +0100

Linda,

I do not see why nvo3 differs from other technologies when it comes to switching and routing.

Obviously a tenant may have different VNs. Communication within an L2 VN is based on traditional switching,
Communication across L2 VNs is based on tradtional routing. IRB-like capabilities (as I think that this is what you are hinting) can be supported but this is implementation specific.

Hence, I'm not sure what a specific "inter-subnet communication" section would contain.

Marc

> Behalf Of Linda Dunbar
> Sent: Monday, March 04, 2013 11:05 PM
> To: Lucy yong; Benson Schliesser; [email-masked]
> Inter Virtual Network communication (or inter subnet 
> communication) is a big part of data centers. Hosts in data 
> center do communicate with external peers. Many tenants to DC 
> need more than one Virtual Network Instances. And hosts in 
> those virtual network instances do communicate with each 
> other and some communicate with peers via public internet. 
> 
> 
> IMHO, Inter-subnet communication set NV03 apart from other 
> overlay done by IETF, such as L2VPN, TRILL, LISP, etc. 
> There are some inter virtual network discussion in 
> http://datatracker.ietf.org/doc/draft-yong-nvo3-frwk-dpreq-addition/. 
> > Of Lucy yong
> > Thus, a question to the WG chair, is this the scope for nvo3 WG? If 
> > yes, that is fine and the framework document should make it 
> very clear 
> > that the framework document describes architecture model of 
> a virtual 
> document is 
> > working on a virtual network over L3 infrastructure or a tenant 
> > network over L3 infrastructure. Two are not equivalents.
> > 
> they are 
> > not. VM on LAN11 and VM on LAN12 may reside on the same 
> server. Thus 
> > it is very important to state them out clearly. Finally, 
> based on my 
> > 3) in Section 3.3, "In DC environments utilizing VM 
> technologies, an 
> > important feature
> >        is that VMs can move from one server to another 
> server in the 
> > same
> > > From: [email-masked] 
> [mailto:[email-masked]] On Behalf
> > Of
> > > Benson Schliesser

---

From: Lucy yong <[email-masked]>
Date: Tue, 5 Mar 2013 15:20:16 +0000

Hi Marc,

Please see inline.

> From: LASSERRE, MARC (MARC) [mailto:[email-masked]]
> Sent: Tuesday, March 05, 2013 4:36 AM
> Subject: RE: [nvo3] WG Last Call for draft-ietf-nvo3-framework-02
> 
> Lucy,
> 
> See my comments below.
> 
> Thanks,
> Marc
> 
[Lucy] OK, glad that you clarify that.
> Can you point to a specific paragraph where it is mentioned or implied
> that a tenant network can only consist of a single VN?
[Lucy] Figure 3 is the generic reference model and figure 4 is the model for NVE. Do you think these two reference models can represent the network virtual overlay for the tenant systems in the same and different VNs? If yes, please explain. I only see the former.
> How can routing be avoided when VMs attached to different L2 networks
[Lucy] This is not my point. My point is: this is the tenant network logical topology view. Why does it show underlying L3 infrastructure between Rtr1 and Rtr2? Tenant has one Routing domain and three L2 domain. That is what figure should show, right?
> virtual or physical switch).
[Lucy] OK, this is what I picture. Furthermore, the "logical" also indicate that VM in LAN11 communicate with VM in LAN12 MUST go through L3 overlay. 
> > technologies, an important feature
> I'm fine with changing the sentence to "In DC environments utilizing VM
> technologies, an important feature is that VMs can move from one server
> to another server (within or across DCs) in a seamless manner".
[Lucy] that is good.

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Tue, 5 Mar 2013 16:12:40 +0000

Marc,

I don't know what the technologies you hint to and assume that you mean L2VPN and L3VPN specified in IETF. L2VPN is for intra subnet. L3VPN can apply for both intra and inter subnet. SP typically offer either L2VPN or L3VPN service to customer, which is why IETF worked that way. In DC, DC provider need to offer customer a network that may contain multiple VNs, where some may be L2 overlay and some L3 overlay. IRB has been used in the network, but it is not offered as a service to customer nor in overlay mode. Further IETF has no specification to it too. Thus, we can't assume that is a standard technology referring to here. 

Regards,
Lucy

> Sent: Tuesday, March 05, 2013 4:53 AM
> To: Linda Dunbar; Lucy yong; Benson Schliesser; [email-masked]
> Communication across L2 VNs is based on tradtional routing. IRB-like
> capabilities (as I think that this is what you are hinting) can be
> supported but this is implementation specific.
>

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 5 Mar 2013 17:17:17 +0100

Hi Lucy,

See my answers inline.

Marc 

> From: Lucy yong [mailto:[email-masked]] 
> Sent: Tuesday, March 05, 2013 4:20 PM
> To: LASSERRE, MARC (MARC); Benson Schliesser; [email-masked]
> > From: LASSERRE, MARC (MARC) 
> [mailto:[email-masked]]
> > > also mentions that a tenant network may consist of one or more 
> > > virtual networks. However, the architecture and reference 
> model is 
> > > focusing on single virtual network overlay architecture 
> only. It is 
> > > lack of describing the architecture and model about 
> multiple virtual 
> > > networks forming one tenant network, i.e. virtual network overlay 
> > > interconnections.
> > 
> > > yes, that is fine and the framework document should make it very 
> > > clear that the framework document describes architecture 
> model of a 
> > > virtual network and only applies to a tenant network if 
> it consists 
> > > of one virtual network that may be implemented by either 
> L2 overlay 
> > > or L3 overlay. a tenant network that contain more than 
> one virtual 
> > > networks is outside scope of this document. If not, we can either 
> > > have one framework draft to architect both, or we have 
> two separate 
> > > documents, one to intra virtual network overlay and another for 
> > > inter virtual network overlay.
> > >
> [Lucy] Figure 3 is the generic reference model and figure 4 
> is the model for NVE. Do you think these two reference models 
> can represent the network virtual overlay for the tenant 
> systems in the same and different VNs? If yes, please 
> explain. I only see the former.

While these are generic pictures, figure 4 does show that tenant systems are attached via VAPs to specific VNIs. A set of VNIs with the same VN context (eg VNID) represents a VN. A Tenant system with multiple interfaces can attach to different VNIs, hence be part of different VNs. All this would be quite hard to represent in a single figure. Some add'l text might be needed for clarity.

> > > figure at all. Second, I interpret this
> > 
> > How can routing be avoided when VMs attached to different 
> L2 networks 
> > have to communicate?
> [Lucy] This is not my point. My point is: this is the tenant 
> network logical topology view. Why does it show underlying L3 
> infrastructure between Rtr1 and Rtr2? Tenant has one Routing 
> domain and three L2 domain. That is what figure should show, right?

VMs do not care what is behind the router boundary. But a tenant knows that routers run over an L3 infrastructure even if in its simplest form both rtr1 and rtr2 could be directly connected. Hence, this is logical connectvity as seen by a tenant.

> > 
> > > figure as one tenant network consisting of four virtual networks, 
> > > three L2 virtual networks and one L3 virtual network. It should 
> > > point out that VMs on a L2 virtual network may reside on 
> the same or 
> > > different servers. The drawing and team LAN make easy to 
> thing they 
> > > are physical LANs. In fact they are not. VM on LAN11 and 
> VM on LAN12 
> > > may reside on the same server. Thus it is very important to state 
> > > them out clearly. Finally, based on my comment 1), it 
> should clarify 
> [Lucy] OK, this is what I picture. Furthermore, the "logical" 
> also indicate that VM in LAN11 communicate with VM in LAN12 
> MUST go through L3 overlay. 
> > 
> > >
> > I'm fine with changing the sentence to "In DC environments 
> utilizing 
> > VM technologies, an important feature is that VMs can move from one 
> > server to another server (within or across DCs) in a 
> seamless manner".
> [Lucy] that is good.
> 
> Regards,
> Lucy
> > 
> > >
> > > Cheers,
> > > Lucy
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Tue, 5 Mar 2013 17:29:29 +0100

Lucy,

Indeed. This has been my point in a previous email thread.
You were the one asking for a new service type for a service that resembles IRB.
I, along with a lot more people, do not think that this is necessary as discussed extensively over the mailing list.

As I mentioned in my previous emails, a tenant can have multiple L2 VNs and L3 VNs. The framework draft allows for this.

Marc

> Sent: Tuesday, March 05, 2013 5:13 PM
> To: LASSERRE, MARC (MARC); Linda Dunbar; Benson Schliesser; 
> [email-masked]
> I don't know what the technologies you hint to and assume 
> that you mean L2VPN and L3VPN specified in IETF. L2VPN is for 
> intra subnet. L3VPN can apply for both intra and inter 
> subnet. SP typically offer either L2VPN or L3VPN service to 
> customer, which is why IETF worked that way. In DC, DC 
> provider need to offer customer a network that may contain 
> multiple VNs, where some may be L2 overlay and some L3 
> overlay. IRB has been used in the network, but it is not 
> offered as a service to customer nor in overlay mode. Further 
> IETF has no specification to it too. Thus, we can't assume 
> that is a standard technology referring to here. 
> 
> Regards,
> Lucy
> 
> > I do not see why nvo3 differs from other technologies when 
> it comes to 
> > switching and routing.
> > 
> > Obviously a tenant may have different VNs. Communication 
> within an L2 
> > VN is based on traditional switching, Communication across 
> L2 VNs is 
> > based on tradtional routing. IRB-like capabilities (as I think that 
> > this is what you are hinting) can be supported but this is 
> > implementation specific.
> > 
> > > Of Linda Dunbar
> > > communication) is a big part of data centers. Hosts in 
> data center 
> > > do communicate with external peers. Many tenants to DC need more 
> > > than one Virtual Network Instances. And hosts in those virtual 
> > > network instances do communicate with each other and some 
> > > communicate with peers via public internet.
> > >
> > >
> > > IMHO, Inter-subnet communication set NV03 apart from 
> other overlay 
> > > done by IETF, such as L2VPN, TRILL, LISP, etc.
> > > > Thus, a question to the WG chair, is this the scope for 
> nvo3 WG? 
> > > > If yes, that is fine and the framework document should make it
> > > very clear

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 5 Mar 2013 20:39:08 +0000

Marc, 

You are correct that the switching and routing for NV03 shouldn't differ from other.

However, Data Centers usually host many end stations, that might  have default GW configured. The NVE, even with the IRB function, might not be the GW configured. 

> Hence, I'm not sure what a specific "inter-subnet communication"

I can provide the text in a few days. 

Linda

---

From: Lucy yong <[email-masked]>
Date: Tue, 5 Mar 2013 20:59:54 +0000

Marc,

Please see inline.

> While these are generic pictures, figure 4 does show that tenant
> systems are attached via VAPs to specific VNIs. A set of VNIs with the
> same VN context (eg VNID) represents a VN. A Tenant system with
> multiple interfaces can attach to different VNIs, hence be part of
> different VNs. All this would be quite hard to represent in a single
> figure. Some add'l text might be needed for clarity.
> 
[Lucy] This is not my point. But agree that a tenant system may participate more than one VN.
My point is more about how two VNs interconnect each other. Are you saying that this is the way that two VN interconnecting each other, i.e. via a VM? 
> VMs do not care what is behind the router boundary. But a tenant knows
> that routers run over an L3 infrastructure even if in its simplest form
> both rtr1 and rtr2 could be directly connected. Hence, this is logical
> connectvity as seen by a tenant.
[Lucy] IMO, a tenant topology view should only include the domains, i.e. sub network. If that Rtr1 and Rtr2 represent two different administration domains, then we talk about nvo federation. Anyway, I still don't see why L3 infrastructure should be shown in this figure.

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Tue, 5 Mar 2013 20:59:54 +0000

Marc,
 
> 
> As I mentioned in my previous emails, a tenant can have multiple L2 VNs
> and L3 VNs. The framework draft allows for this.
[Lucy] Could you give an example how it will work by using the reference model? Just state that a tenant can have multiple L2VNs and L3VNs is not sufficient, IMO. How do you expect the address to be advertised among NVEs that may have the memberships of same and different VNs. How do you anticipate control plane work here.

Lucy