Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Mon, 2 Jul 2012 15:03:52 +0200

Maria,

What about the following rewording?

"A combination of data plane and control plane based learning may be applicable." 

Marc

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of NAPIERALA, MARIA H
> Sent: Saturday, June 30, 2012 2:18 AM
> To: Pedro Roque Marques; <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [nvo3] call for adoption: 
> draft-lasserre-nvo3-framework-02
> 
> > > As has been pointed out a number of times, pure data 
> plane learning
> > leads
> > > to a lot of BUM traffic flooding, so a combination of 
> data plane and
> > control
> > > plane can work better with existing systems and improve 
> scalability.
> > 
> > Data plane learning, l2 header transparency, bridging 
> interoperability 
> > are all very reasonable requirements for one type of 
> data-centers. But 
> > also an unacceptable burden for a different class of DC 
> designs. In my 
> > view you and Maria are just looking at different types of 
> DC designs.
> > They are different problems.
> 
> Precisely. And my comment was that the sentence in draft: 
> "Often a combination of data plane and control based learning 
> is necessary" seems to be assuming that pretty much all data 
> centers must use (some) data plane learning. I have suggested 
> to clarify it.
> 
> Maria
> 
> 
> >    Pedro.
> > 
> > >
> > > Thanks,
> > > --David
> > >
> > >
> > >> -----Original Message-----
> > >> From: [email-masked] [mailto:[email-masked]] On 
> > >> Behalf
> > Of
> > >> NAPIERALA, MARIA H
> > >> Sent: Friday, June 29, 2012 2:14 PM
> > >> To: [email-masked]
> > >> Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-
> > framework-02
> > >>
> > >> A comment on section 4.2.2 "Coordination between data plane and
> > control plane"
> > >>
> > >>   "Often a combination of data plane and control based 
> learning is
> > >>   necessary."
> > >>
> > >> I think this statement is too strong since in a solution 
> proposed 
> > >> in
> > draft-
> > >> marques-l3vpn-end-system, for example, there is no data plane
> > learning in a
> > >> virtual network. Maybe it should be explain when such 
> combination 
> > >> is necessary.
> > >>
> > >> Maria
> > >>
> > >>
> > >>> On 6/18/2012 11:51 PM, Benson Schliesser wrote:
> > >>> Dear NVO3 Participants -
> > >>>
> > >>> This message begins a two week Call for Adoption of
> > >>> http://tools.ietf.org/html/draft-lasserre-nvo3-framework-02
> > >>> by the NVO3 working group, ending on 02-July-2012.
> > >>>
> > >>> Please respond to the NVO3 mailing list with any statements of 
> > >>> approval or disapproval, along with any additional 
> comments that 
> > >>> might explain your position. Also, if any NVO3 participant is 
> > >>> aware of IPR associated with this draft, please inform 
> the mailing 
> > >>> list and/or the NVO3 chairs.
> > >>>
> > >>> Thanks,
> > >>> -Benson&  Matthew
> > >>>
> > >>>
> > >>
> > >> _______________________________________________
> > >> nvo3 mailing list
> > >> [email-masked]
> > >> https://www.ietf.org/mailman/listinfo/nvo3
> > >
> > > _______________________________________________
> > > nvo3 mailing list
> > > [email-masked]
> > > https://www.ietf.org/mailman/listinfo/nvo3
> > 
> > _______________________________________________
> > nvo3 mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/nvo3
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Mon, 2 Jul 2012 15:17:15 +0200

Janos,

I have used these terms in our email exchange but I did not plan to use them in the draft.
Thanks for pointing out the misleading text in the original introduction.

I will address your add'l comments in a separate mail.

Marc

> From: János Farkas [mailto:[email-masked]]
> Sent: Saturday, June 30, 2012 9:36 AM
> To: LASSERRE, MARC (MARC)
> Cc: [email-masked]
> Hi Marc,
>  > The limitations stated are still valid in most
> Ethernet-based (albeit
> traditional) DCs.
>
> I'm afraid the word "traditional" (and phrases alike) is not
> well defined, specific and exact. I would avoid using it in a
> standard specification.
>
>  > Understood but these technologies are not that common in
> data centers yet, even if available from some vendors.
>
> I'm afraid I also fail to translate this sentence to be
> applicable in a standard.
>
> As far as I understand these type of statements are not
> exact. Does not seem to mean much more than DCs built upon
> 20th century technology may have some issues, and one has to
> do something if aims to get rid of these issues. In order to
> achieve that, an upgrade is needed for such DCs, e.g. to the
> NVO3 solution, which may be not that common in data centers
> yet, even if it is available from some vendors.
>
> Please do not misunderstand me. All I'm trying to point on in
> my comments is to have exact, specific and correct statements
> in a standard as much as possible. I'm not arguing for any
> specific solution. This is a framework document; I think it
> should be quite generic and solution independent, and in-line
> with the specification of the charter.
>
> Regards,
> Janos
>
>
>
> On 6/29/2012 6:35 PM, LASSERRE, MARC (MARC) wrote:
> > Hi Janos,
> >
> > See my comments below.
> >
> > Thanks,
> > Marc
> >> From: János Farkas [mailto:[email-masked]]
> >> Sent: Friday, June 29, 2012 5:42 PM
> >> To: LASSERRE, MARC (MARC)
> >> Cc: [email-masked]
> >> Marc,
> >> On 6/29/2012 5:17 PM, LASSERRE, MARC (MARC) wrote:
> >>> Janos,
> >>>
> >>> It is well understood that PBB and SPBM address traditional
> >> Ethernet limitations.
> >>
> >> PBB and SPB are approved and deployed standards. This is Ethernet
> >> today.
> > Understood but these technologies are not that common in
> data centers yet, even if available from some vendors.
> >
> >
> >> With respect to Ethernet, all I suggest is to remove statements on
> >> Ethernet limitations that are not valid today.
> > The limitations stated are still valid in most
> Ethernet-based (albeit traditional) DCs.
> >
> >>> However, our intent is not to focus the nvo3 framework
> >> towards specific L2 technologies.
> >>> In fact, nvo3 targets IP and/or MPLS as the underlying
> >> transport and not Ethernet-based transport.
> >>
> >> As I wrote below:
> >> "I understand that the WG aims to provide a solution over L3.
> >> Nevertheless, it seems to me that it would be better to give other
> >> motivation for it than L2 has issues, given that no issue has been
> >> pointed out that is still valid for Ethernet today."
> >>
> >> I'm not against any L3 solution and clearly understand
> that the goal
> >> here is to provide one. I guess it has other motivation than L2 is
> >> poor.
> > True, it is also motivated by the fact that some DC
> operators prefer an IP-based solution.
> > I could make this clearer in the draft.
> >
> >
> >> All I pointed out in my comments to the introduction is
> that I think
> >> it would be better to explain that motivation in the introduction.
> >> Note further, that my comments to the document are not more
> >> L2 specific than the text itself in the draft.
> >>
> >> Best regards,
> >> Janos
> >>
> >>
> >>> Thanks,
> >>> Marc
> >>>
> >>>> From: [email-masked] [mailto:[email-masked]]
> >> On Behalf
> >>>> Of János Farkas
> >>>> Sent: Friday, June 29, 2012 3:35 PM
> >>>> To: [email-masked]
> >>>> Hi,
> >>>> I have some comments to the draft. Please find them below
> >> under the
> >>>> section header they belong to.
> >>>>
> >>>>
> >>>> 1. Introduction
> >>>>
> >>>> "Limited VLAN space"
> >>>> The 12-bit overlay network ID limitation of the Ethernet
> header is
> >>>> not there any more for several years. There are
> different types of
> >>>> VLANs.
> >>>> The I-SID can be considered as a 24-bit VLAN ID, which is
> >> sometimes
> >>>> called I-VLAN. Furthermore, there exist B-VLANs and
> >> S-VLANs aside the
> >>>> C-VLANs that were defined at the first place. The combination of
> >>>> these VLANs provides flexibility for network virtualization;
> >>>> altogether the Ethernet header today provides 60 bits
> for network
> >>>> virtualization.
> >>>> It would be better to remove this because the limitation
> >> is not there
> >>>> today.
> >>>>
> >>>> "FIB explosion due to handling of large number of MACs/IP
> >> addresses"
> >>>> FIB explosion might be there in a L2 network if the
> >> already available
> >>>> L2 network virtualization tools are not used. FIB
> explosion is not
> >>>> likely in a PBB network.
> >>>> It would be better to update the bullet accordingly or remove it.
> >>>>
> >>>> "Spanning Tree limitations"
> >>>> L2 networks are not limited to a single spanning tree
> for several
> >>>> years.
> >>>> Shortest Path Bridging (SPB) provides shortest path
> forwarding and
> >>>> uses physical links that might have been blocked by a
> >> single spanning
> >>>> tree instance. Moreover, MSTP also provides multiple
> spanning tree
> >>>> instances, which allow the use of the physical links.
> >>>> It would be better to remove the bullet.
> >>>>
> >>>> "Broadcast storms"
> >>>> Broadcast storms are not necessarily there in today's L2
> networks,
> >>>> e.g.
> >>>> there are no broadcast storms at all in an SPBM network.
> >> Furthermore,
> >>>> applying the existing L2 network virtualization techniques, the
> >>>> broadcast due to unknown destination is kept within the virtual
> >>>> network, thus it is not a broadcast storm.
> >>>> It would be better to update the bullet such that it exactly
> >>>> specifies the problem or remove it.
> >>>>
> >>>> "Inefficient Broadcast/Multicast handling"
> >>>> It is not clear what is meant here, it would be better
> to be more
> >>>> specific. (If it is about L2, then the statement is not valid.
> >>>> Handling of multicast is very efficient in L2, both
> shortest path
> >>>> trees and shared threes can be used today.)
> >>>>
> >>>> "Limited mobility/portability support"
> >>>> It is not clear what is meant here. Is it about VM
> >> migration? If so,
> >>>> then the statement is not exact for L2, VDP supports VM
> migration.
> >>>> (Furthermore, L2 networks typically automatically learn the new
> >>>> location of a station after its movement.)
> >>>>
> >>>> "Lack of service auto-discovery"
> >>>> If it is about L2, then that is not the case. SPB has in-built
> >>>> service auto-discovery.
> >>>> It would be better to be more specific in the bullet or
> remove it.
> >>>>
> >>>> The issue list is introduced as "Existing virtual network
> >> models used
> >>>> for data center networks have known limitations,
> >> specifically in the
> >>>> context of multiple tenants. These issues can be summarized as"
> >>>> As the comments to the bullets show the list is not
> accurate given
> >>>> the existing virtual network models available for data
> >> centers today.
> >>>> Therefore, it would be better to bring this issue list up
> >> to date to
> >>>> today's networking technologies.
> >>>> (I understand that the WG aims to provide a solution over L3.
> >>>> Nevertheless, it seems to me that it would be better to
> give other
> >>>> motivation for it than L2 has issues, given that no
> issue has been
> >>>> pointed out that is still valid for Ethernet today.)
> >>>>
> >>>>
> >>>> 4.1. Pros&   Cons
> >>>>
> >>>> "Unicast tunneling state management is handled at the
> edge of the
> >>>> network. Intermediate transport nodes are unaware of such
> >> state. Note
> >>>> that this is not the case when multicast is enabled in the core
> >>>> network."
> >>>> This statement may depend on the solution applied or the
> >> terms used
> >>>> here might not be entirely clear. Even if the core only provides
> >>>> point to point tunnels, then those tunnels have to be
> >> established in
> >>>> the core, hence maintenance of some states is required
> in the core
> >>>> nodes. If the core provides multipoint tunnels as well, then of
> >>>> course more state is required to maintain a multipoint
> >> tunnel in the
> >>>> core than a point to point tunnel. The type of connectivity that
> >>>> needs to be provided by the core depends on the actual
> location of
> >>>> VMs belonging to the same group/tenant. If VMs of a
> group are only
> >>>> behind two NVEs, then it is enough to provide a point to point
> >>>> connectivity/tunnel by the core independently of whether
> >> the traffic
> >>>> among the VMs is unicast or multicast. If VMs of a group
> >> are behind
> >>>> more than two NVEs, then the core has to provide a multipoint
> >>>> connectivity between those NVEs; the way it is provided is
> >> solution
> >>>> dependent.
> >>>>
> >>>> "Load balancing may not be optimal as the hash algorithm
> >> may not work
> >>>> well due to the limited number of combinations of tunnel
> >> source and
> >>>> destination addresses"
> >>>> There are other possibilities of load balancing besides hash.
> >>>> It would be better to update the sentence to "Hash-based load
> >>>> balancing may not be optimal as the hash algorithm may not
> >> work well
> >>>> due to the limited number of combinations of tunnel source and
> >>>> destination addresses"
> >>>>
> >>>>
> >>>> 4.2.1. Data plane vs Control plane driven
> >>>>
> >>>> "Multicasting in the core network for dynamic learning
> can lead to
> >>>> significant scalability limitations."
> >>>> Multicast should be kept within the virtual network even
> >> while it is
> >>>> transmitted in the core, which can be aided by service
> >>>> auto-discovery.
> >>>> Having these features, the scalability limitations should not be
> >>>> significant.
> >>>>
> >>>> "Specific forwarding rules must be enforced to prevent
> loops from
> >>>> happening. This can be achieved using a spanning tree
> >> protocol or a
> >>>> shortest path tree, or using a split-horizon mesh."
> >>>> "a spanning tree protocol" is not the same category as
> "a shortest
> >>>> path tree" or "split horizon mesh" here. The first one is
> >> a protocol,
> >>>> while the latter two are forwarding rules as said in the
> >> beginning of
> >>>> the sentence.
> >>>> It would be better to remove "protocol" from the sentence
> >> or resolve
> >>>> it another way.
> >>>>
> >>>> "the amount of state to be distributed is a function of
> >> the number of
> >>>> virtual machines."
> >>>> This 'function' is not that easy to draw, it depends on a lot of
> >>>> factors, furthermore it is most likely that the state to be
> >>>> maintained does not depend on the way the state is
> >> installed, i.e. it
> >>>> is the same both for the control and the data plane driven cases.
> >>>> For example, there is no need to maintain any state if the
> >>>> connectivity is point to point through the core, i.e. a
> >> group of VMs
> >>>> are only behind two NVEs.
> >>>> We can take a look on mapping tables e.g. by having an
> example of
> >>>> three
> >>>> NVEs: NVE-A, NVE-B and NVE-C providing the connectivity
> >> through the
> >>>> core for a group of VMs. If VM1 behind NVE-A
> communicates with VN2
> >>>> behind NVE-B, then the mapping table in NVE-A is VN2->NVE-B, i.e.
> >>>> NVE-A has to
> >>>> address the packets to NVE-B for the transmission through
> >> the core if
> >>>> any VM behind NVE-A sends them to VN-2. Similarly the
> >> mapping table
> >>>> is
> >>>> VN1->NVE-A in NVE-B. It does not matter how these mapping
> >> tables are
> >>>> maintained, the same states should be there. That is,
> >> states are the
> >>>> same both for data and control plane driven mapping tables.
> >>>> If state here means a mapping state, then it would be better to
> >>>> update the cited sentence. Moreover, maybe this section
> is not the
> >>>> best location for it. One way out is to remove the paragraph.
> >>>> Alternatively
> >>>> it can be updated, e.g. "the amount of state to be distributed
> >>>> depends on the network scenario, the grouping and the number of
> >>>> virtual machines."
> >>>> If the state means something else, then it would be useful
> >> to clarify
> >>>> what exactly meant here.
> >>>>
> >>>>
> >>>> 4.2.6. Interaction between network overlays and underlays
> >>>>
> >>>> Is it really the case that the DC provider wants to give
> >> visibility
> >>>> of its infrastructure to its Tenants? Isn't it that the
> >> Tenant gets
> >>>> is overlay and the underlay should be completely hidden?
> >> For example,
> >>>> if the Tenant buys L2aaS, does/should it bother with the
> >> fact that L2
> >>>> overlay happens to be provided by an L3 underlay? Maybe
> >> the direction
> >>>> of SLAs would be more appropriate here than providing
> >> visibility of
> >>>> the underlay.
> >>>>
> >>>>
> >>>> Best regards,
> >>>> János
> >>>>
> >>>>
> >>>>
> >>>>> Please respond to the NVO3 mailing list with any statements
> >>>> of approval or disapproval, along with any additional
> >> comments that
> >>>> might explain your position. Also, if any NVO3 participant
> >> is aware
> >>>> of IPR associated with this draft, please inform the
> mailing list
> >>>> and/or the NVO3 chairs.
> >>>>> Thanks,
> >>>>> -Benson&    Matthew
> >>>>>

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Mon, 2 Jul 2012 15:36:09 +0200

Janos,

See my comments below.

Thanks,
Marc 

> Behalf Of János Farkas
> under the section header they belong to.
> 

Snipped

> 
> 4.1. Pros & Cons
> 
> "Unicast tunneling state management is handled at the edge of the 
> network. Intermediate transport nodes are unaware of such state. Note 
> that this is not the case when multicast is enabled in the 
> core network."
> terms used here 
> might not be entirely clear. Even if the core only provides point to 
> point tunnels, then those tunnels have to be established in the core, 
> hence maintenance of some states is required in the core 
> nodes. If the 
> core provides multipoint tunnels as well, then of course more 
> state is 
> required to maintain a multipoint tunnel in the core than a point to 
> point tunnel. The type of connectivity that needs to be 
> provided by the 
> core depends on the actual location of VMs belonging to the same 
> group/tenant. If VMs of a group are only behind two NVEs, then it is 
> enough to provide a point to point connectivity/tunnel by the core 
> independently of whether the traffic among the VMs is unicast or 
> multicast. If VMs of a group are behind more than two NVEs, then the 
> core has to provide a multipoint connectivity between those NVEs; the 
> way it is provided is solution dependent.

This is a generic statement about the fact no tunnel state is mantained in core nodes for unicast tunnels - unlike mutlicast state.

> 
> "Load balancing may not be optimal as the hash algorithm may not work 
> well due to the limited number of combinations of tunnel source and 
> destination addresses"
> balancing 
> may not be optimal as the hash algorithm may not work well due to the 
> limited number of combinations of tunnel source and 
> destination addresses"

Fine.

> "Multicasting in the core network for dynamic learning can lead to 
> Multicast should be kept within the virtual network even while it is 

How about "may lead to " instead of "can lead to" then?
> 
> "Specific forwarding rules must be enforced to prevent loops from 
> happening. This can be achieved using a spanning tree protocol or a 
> "a spanning tree protocol" is not the same category as "a 
> shortest path 
> tree" or "split horizon mesh" here. The first one is a 
> protocol, while 
> the latter two are forwarding rules as said in the beginning of the 
> sentence.
> It would be better to remove "protocol" from the sentence or 
> resolve it 
> another way.

Fine

> 
> "the amount of state to be distributed is a function of the number of 
> virtual machines."
> maintained 
> does not depend on the way the state is installed, i.e. it is 
> the same 
> both for the control and the data plane driven cases.
> connectivity 
> is point to point through the core, i.e. a group of VMs are 
> only behind 
> two NVEs.
> example of three 
> through the core 
> for a group of VMs. If VM1 behind NVE-A communicates with VN2 behind 
> NVE-B, then the mapping table in NVE-A is VN2->NVE-B, i.e. 
> NVE-A has to 
> address the packets to NVE-B for the transmission through the core if 
> any VM behind NVE-A sends them to VN-2. Similarly the mapping 
> table is 
> VN1->NVE-A in NVE-B. It does not matter how these mapping tables are 
> maintained, the same states should be there. That is, states are the 
> If state here means a mapping state, then it would be better 
> to update 
> the cited sentence. Moreover, maybe this section is not the best 
> location for it. One way out is to remove the paragraph. 
> Alternatively 
> it can be updated, e.g. "the amount of state to be 
> distributed depends 
> on the network scenario, the grouping and the number of 
> virtual machines."
> If the state means something else, then it would be useful to clarify 
> what exactly meant here.
> 

Fine. I do not have a problem either removing this sentence or updating it to "the amount of state is dependent upon network topology and the number of VMs".

> its infrastructure to its Tenants? Isn't it that the Tenant gets is 
> overlay and the underlay should be completely hidden? For example, if 
> the Tenant buys L2aaS, does/should it bother with the fact that L2 
> overlay happens to be provided by an L3 underlay? Maybe the 
> direction of 
> SLAs would be more appropriate here than providing visibility of the 
> underlay.

This is not about providing visibility to tenants. It is about running multiple overlay services over a common underlay.

> comments that might explain your position. Also, if any NVO3 
> participant is aware of IPR associated with this draft, 
> please inform the mailing list and/or the NVO3 chairs.
> >
> > Thanks,
> > -Benson&  Matthew
> >

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Mon, 2 Jul 2012 15:58:07 +0000

Marc,

The sentence is correct, but I think you should clarify when data planning learning is applicable, when it is not, e.g. if all L3 solutions...
Luyuan

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> LASSERRE, MARC (MARC)
> Sent: Monday, July 02, 2012 9:04 AM
> To: NAPIERALA, MARIA H; Pedro Roque Marques; <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02
> 
> Maria,
> 
> > > >>> http://tools.ietf.org/html/c-02

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Mon, 2 Jul 2012 15:59:52 +0000

Sorry, typo.
Data plane learning.

> Luyuan Fang (lufang)
> Sent: Monday, July 02, 2012 11:58 AM
> To: LASSERRE, MARC (MARC); NAPIERALA, MARIA H; Pedro Roque Marques;
> <[email-masked]>
> Cc: [email-masked]
> The sentence is correct, but I think you should clarify when data
> planning learning is applicable, when it is not, e.g. if all L3
> solutions...
> Luyuan
> 
> of

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 2 Jul 2012 16:12:32 +0000

Marc,

Yes, this sentence is much more general and not excluding control-plane only solutions. You might clarify it more by stating that control plane only solutions with no data plane learning are also possible (as Luyuan suggested).

Maria

> From: Luyuan Fang (lufang) [mailto:[email-masked]]
> Subject: RE: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02
> 
> Marc,
>

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Mon, 2 Jul 2012 18:15:04 +0200

Luyuan,

True, DP learning is applicable to L2. This could be clarified in the previous section (4.2.1).
We could also highlight that CP driven methods are applicable to both L2 and L3.

Marc 

> Sent: Monday, July 02, 2012 5:58 PM
> To: LASSERRE, MARC (MARC); NAPIERALA, MARIA H; Pedro Roque 
> Marques; <[email-masked]>
> Cc: [email-masked]
> The sentence is correct, but I think you should clarify when 
> data planning learning is applicable, when it is not, e.g. if 
> all L3 solutions...
> Luyuan
> 
> > Of LASSERRE, MARC (MARC)
> > "A combination of data plane and control plane based 
> learning may be 
> > applicable."
> > 
> > Marc
> > 
> > > From: [email-masked] 
> [mailto:[email-masked]] On Behalf 
> > > Of NAPIERALA, MARIA H
> > > "Often a combination of data plane and control based learning is 
> > > necessary" seems to be assuming that pretty much all data centers 
> > > must use (some) data plane learning. I have suggested to 
> clarify it.
> > >
> > > Maria
> > >
> > >
> > > >    Pedro.
> > > >
> > > > >
> > > > > Thanks,
> > > > > --David
> > > > >
> > > > >
> > > > >>> http://tools.ietf.org/html/c-02 by the NVO3 working group, 
> > > > >>> ending on 02-July-2012.
> > > > >>>

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Mon, 2 Jul 2012 18:28:41 +0200

Maria, all,

I suggest adding the first sentence in section 4.2.1 (see below) for clarification.

Data plane learning is applicable to L2 whereas control plane learning is applicable to both L2 and L3.
Control plane learning does not require dynamic data plane learning.

Data plane learning implies that flooding of unknown destinations be supported and hence implies that broadcast and/or multicast be supported. Multicasting in the core network for dynamic learning may lead to significant scalability limitations. Specific forwarding rules must be enforced to prevent loops from happening. This can be achieved using a spanning tree, a shortest path tree, or a split-horizon mesh. 

It should be noted that the amount of state to be distributed is a function of the number of virtual machines. Different forms of caching can also be utilized to minimize state distribution between the various elements.

-Marc

> Sent: Monday, July 02, 2012 6:13 PM
> To: Luyuan Fang (lufang); LASSERRE, MARC (MARC); Pedro Roque 
> Yes, this sentence is much more general and not excluding 
> control-plane only solutions. You might clarify it more by 
> stating that control plane only solutions with no data plane 
> learning are also possible (as Luyuan suggested).
> 
> Maria
> 
> 
> > > > must use (some) data plane learning. I have suggested 
> to clarify 
> > > > it.
> > > >
> > > > Maria
> > > >
> > > >
> > > > >    Pedro.
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > --David
> > > > > >
> > > > > >
> > > > > >>> is aware of IPR associated with this draft, please inform
> > > > the mailing

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Mon, 2 Jul 2012 16:40:31 +0000

Sounds good, Marc.

> We could also highlight that CP driven methods are applicable to both L2 and L3.

It would be good if you could also describe the cases where only CP learning is needed, and many other cases which the combination of CP and DP learning would be applied, (I assume pure L2 is out of the scope in nvo3), where the demarcation is and the implications in each case, you could abstract the models/scenarios using the terms and architecture references defined in the doc, and I think any standardized IETF/IEEE technologies could be named out when they are mixed into the combination, or stand alone.

Thanks,
Luyuan

> From: LASSERRE, MARC (MARC) [mailto:[email-masked]]
> Sent: Monday, July 02, 2012 12:15 PM
> To: Luyuan Fang (lufang); NAPIERALA, MARIA H; Pedro Roque Marques;
> <[email-masked]>
> Cc: [email-masked]

---

From: David Allan I <[email-masked]>
Date: Mon, 2 Jul 2012 14:07:22 -0400

Hi:

This soundsa bit confused to me.

"learning" is required when the thing to be learned is not administered and there is some form of trust model to auto-configure the learner. The L2 example being MAC addresses. I'm really struggling to understand what CP learning for L3 is. It is usually the other way around, the host needs to learn it's L3 identity via DHCP or whatever...The CP knows.

???
Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of LASSERRE, MARC (MARC)
Sent: Monday, July 02, 2012 9:29 AM
To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Pedro Roque Marques; <[email-masked]>
Cc: [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Maria, all,

-Marc

> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of NAPIERALA, MARIA H
> To: Luyuan Fang (lufang); LASSERRE, MARC (MARC); Pedro Roque Marques; 
> <[email-masked]>
> Cc: [email-masked]
> control-plane only solutions. You might clarify it more by stating 
> that control plane only solutions with no data plane learning are also 
> possible (as Luyuan suggested).
> 
> Maria
> 
>

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 2 Jul 2012 18:22:27 +0000

Marc,

Regarding the first paragraph, what about saying the following:

"Data plane learning is applicable _only_ to L2 whereas control plane learning is applicable to both L2 and L3."

I would remove the second sentence "Control plane learning does not require dynamic data plane learning" because it's a tautology.

Also, it is still not clear whether you are assuming that a layer 2 solution can be solely based on control plane updates.

Maria

> Sent: Monday, July 02, 2012 12:29 PM
> Data plane learning implies that flooding of unknown destinations be
> supported and hence implies that broadcast and/or multicast be
> supported. Multicasting in the core network for dynamic learning may
> lead to significant scalability limitations. Specific forwarding rules
> must be enforced to prevent loops from happening. This can be achieved
> using a spanning tree, a shortest path tree, or a split-horizon mesh.
> 
> It should be noted that the amount of state to be distributed is a
> function of the number of virtual machines. Different forms of caching
> can also be utilized to minimize state distribution between the various
> elements.
> 
> -Marc
> 
> > > > > > >> A comment on section 4.2.2 "Coordination between data
> plane
> > > and
> > > > > > control plane"
> > > > > > >>

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 2 Jul 2012 22:22:01 -0400

(resending from my correct mailto email with minor edits)

Hi Marc,

My comments inline,

Best -- aldrin

On Jun 26, 2012, at 9:43 AM, LASSERRE, MARC (MARC) wrote:

> Hi Aldrin,
>  
> See my comments below.
>  
> Thanks,
> Marc
> 
> From: Aldrin Isaac [mailto:[email-masked]] 
> Sent: Tuesday, June 26, 2012 3:24 PM
> Cc: Aldrin Isaac; [email-masked]; [email-masked]; Lucy yong
> Thanks for the clarifications. 
> 
> Wrt my first question, should we assume that there isn't yet a consensus on whether a VNI can support more than one type of VN? 
> It does not preclude it. I'd like to hear how much interest there is to support this capability.

I'm not convinced mixing types is a good idea from a standardization perspective.

> 
> Regarding one-to-one mapping of VNI to VN, this implies that an end-station that needs to be a part of more than one VN will require a separate interface into each associated VNI.  This implies that the end-station will require a potentially complex routing table (depending on how aggregateable the address space within a VN is).  Unfortunately as an operator I don't find this an acceptable trade off.  Can we please address the issue of managing multiple interfaces and complex tables in end-systems? I believe the problem statement needs to capture this.   
> This is not any different than with existing VPN technologies. Supporting multiple interfaces to connect to different subnets and/or nvo3 domains does not make routing really complex.
> Have you had specific problems with past technologies?

If you are referring to VRFs and the like, then I don't see host-facing VNI as equivalent.  For one, most PE connect to a CE which generally tend to be a router, not a host -- the difference here is network versus system administration.

Even in cases where the PE connects to an access subnet (as opposed to a CE router), there are scenarios where the subnet-connecting VRF is a member of multiple VPNs.  For example: a regional extranet DMZ subnet connected to a VRF which is both a member of a customer-facing regional hub-and-spoke VPN as well as an internal DMZ mesh.  The advantage of this approach is that the servers do not need to have an interface on a subnet connected to an internal DMZ VRF which is a member of the internal mesh VPN and yet another separate interface on a subnet connected to another DMZ VRF which is a member of the customer-facing regional hub-and-spoke VPN.  In addition, the servers would only need a default route as opposed to the complex management of static routes on the servers which can become quite challenging to manage at scale.

Along similar lines, in a PE to CE-router scenario, there is also the concept of "shared VRF" model and "interface VRF" model.  In the shared VRF model multiple CE are connected to the same VRF and each VRF is connected to one VPN.  Here if a CE needs to be connected to another VPN you provision yet another interface to connect to the VRF on the PE associated to that VPN.  In this case there is also the additional configuration required on the CE and all the associated routing configuration and the overhead associated to managing additional interfaces and configuration.  OTOH, with the interface VRF model each VRF has only one CE connected.  In this model enabling an additional VPN for the CE only involves importing and exporting an additional RT pair at it's VRF -- no additional interfaces or routing configuration beyond that is generally required.

In a DC/overlay setting, a particular scenario where multiple VN per VNI are involved would be a DMZ subnet with public addressing provided by the ISP.  The DMZ would need to support different CUGs with a common gateway/firewall to efficiently share the ISP assigned prefix.  In this scenario, it is very common that "private vlans" are used (informational RFC 5517).  This would need to be implemented with at least two VNs per VNI -- a hub-and-spoke VN with the gateway/firewall as the hub and multiple "community" VNs where each community VN represents a CUG wherein members need to communicate with one another.

Furthermore, in an enterprise setting it is generally expected that the network bears the network-related complexity.  

All this to say that I am strongly of the opinion that one-to-one mapping of VN to VNI should not be an implicit or explicit requirement/mandate of nvo3.  Would like to see the framework draft address this since this is an important issue.

> 
> Best -- aldrin
> 
> On Tuesday, June 26, 2012, LASSERRE, MARC (MARC) wrote:
> Hi Aldrin,
> 
> See my comments below.
> 
> Thanks,
> Marc
> 
> > From: Aldrin Isaac [mailto:[email-masked]]
> > Sent: Monday, June 25, 2012 8:23 PM
> > Cc: Lucy yong; [email-masked]; [email-masked]
> > Looking for more clarity on the following...
> >
> > Should we assume that a VNI can only support L2VNs (VNI =
> > L2VNI) or L3VNs (VNI = L3VNI) but not both?
> >
> > draft-mity-nvo3-use-cases describes a "virtual network
> > instance interconnecting interface" (VNIF) to interconnect
> > VNI of different type.  An example of a VNIF would be an IRB
> > (integrated routing and bridging) interface that is common on
> > "L3 switches".  Is this something that draft-lasserre captures?
> 
> While the current draft focuses on L2VNs and L3VNs as distinct services, no assumptions have been made about the ability to support IRB.
> A couple of sentences could be added to highlight this possibility.
> 
> >
> > Are there any restriction to the topology of a VN or topology
> > created using a set of VNs.  I.e. can (1) a VNI be a member
> > of multiple VN and (2) can a VN be either hub-and-spoke or
> > full-mesh?  Does draft-lasserre support (1) and (2)?
> 
> A VN is represented by a VNI created within NVEs that participate in this VN. Hence there is a one-to-one mapping between a VN and VNI.
> 
> Hub-and-spoke is discussed briefly in this draft. More details are provided in draft-bl-nvo3-dataplane-requirements.
> 
> >
> > Best -- aldrin
> >
> >
> > On Jun 25, 2012, at 1:14 PM, <[email-masked]> wrote:
> >
> > > Extracting a couple of items for additional comment:
> > >
> > >> 2) section 2.3, why call it NVE service type? Should we call it VN
> > >> (or VNI) service type? One NVE may terminate both L2 VNI
> > and L3 VNI.
> > >> NVE is equivalent to PE in L2VPN/L3VPN. We did not have
> > service type
> > >> for PE and had service type for VPN.
> > >>
> > >> [ml] Like with L2 and L3 PEs, there are L2 and L3 NVEs.
> > >> [[LY]] Should we consider this as VN service type? I
> > understand the
> > >> description but have trouble with the title of NVE service type.
> > >
> > > I think VN service type makes sense, as an NVE could conceivably
> > > support both
> > > L2 and L3 services for the same end system.  OTOH, mixing L2 and L3
> > > service types on the same VN requires that the
> > encapsulation indicate
> > > the service type (for correct decap processing), and that
> > would also be useful to note.
> > >
> > >> 7) It is important to state that the major difference
> > between other
> > >> overlay network technology and NVO3 is that
> > >>   the client edges of the NVO3 network are individual virtualized
> > >> hosts and not network sites.
> > >>
> > >> [ml] NVO3 can cope with both virtualized and non-virtualized hosts.
> > >> [[LY]] Yeah, that is more precise. The point is that the
> > client edge
> > >> is not network sites like CEs in a VPN.
> > >
> > > Actually the NVE can be in a network node.  See Figure 3 and the
> > > discussion of "traditional physical server" in the last
> > paragraph on
> > > p.10, including the statement that the NVE can be in the
> > ToR.  FWIW,
> > > the Storage Systems example in that paragraph is fine, although one
> > > can envision storage systems that contain NVEs.
> > >
> > > Thanks,
> > > --David
> > >
> > >> Of Lucy yong
> > >> Sent: Monday, June 25, 2012 11:52 AM
> > >> To: LASSERRE, MARC (MARC); Benson Schliesser; [email-masked]
> > >> Cc: Bocci, Matthew (Matthew)
> > >> Please see inline below with [[LY]].
> > >>
> > >> Lucy
> > >>
> > >> From: LASSERRE, MARC (MARC)
> > [mailto:[email-masked]]
> > >> Sent: Monday, June 25, 2012 9:23 AM
> > >> To: Lucy yong; Benson Schliesser; [email-masked]
> > >> Cc: Bocci, Matthew (Matthew)
> > >> Hi Lucy,
> > >> Thanks for your comments.
> > >> See my comments below prefixed with [ml].
> > >>
> > >> Marc
> > >>
> > >> From: Lucy yong [mailto:[email-masked]]
> > >> Sent: Saturday, June 23, 2012 1:11 AM
> > >> To: LASSERRE, MARC (MARC); Benson Schliesser; [email-masked]
> > >> Cc: Bocci, Matthew (Matthew)
> > >> Here are some comments on this draft:
> > >>
> > >> 1) The VN (or VNI?) in the doc. is an overlay network that is
> > >> transported over the tunnels. These tunnels are provided by the

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 2 Jul 2012 22:24:32 -0400

Hi David,

I am also of the opinion that the we don't want to single a VNI to support both L2 and L3 service.  

The VNIF (VNI interconnecting InterFace) described in Lucy's draft-mity use-cases is used to interconnect an L2VNI with an L3VNI within the same NVE, similar to how an IRB logical interface would be used to connect a routing instance with a bridge instance on a routing switch (as you already seem to be intimately familiar with).  Here the L2VNI would be a member of L2VNs and the L3VNI a member of L3VNs.  My concern was that the framework draft doesn't seem capture the "VNIF" entity.  Are basic interworking entities such as VNIF not in scope?

Best -- aldrin

On Jun 26, 2012, at 9:51 AM, <[email-masked]> <[email-masked]> wrote:

> Hi Aldrin,
>  
> I observe that for IRB to work on a VNI, the VNI has to provide layer 2 service.  What
> I mean by this is that IRB assumes that the Ethernet header is present on all traffic -
> bridging uses that Ethernet header, whereas the routing portion of IRB (logically)
> strips the Ethernet header and operates on the IP header.  To apply IRB to nvo3 traffic,
> the inner headers (Ethernet and IP) are involved after decapsulation (again, logically).
>  
> The reason for “(logically)” is that there are optimization possibilities when the
> same headers will be present outbound (e.g., when routing between two VNIs that use
> the same encapsulation, an integrated implementation may modify the headers in place
> - as long as the outbound traffic is correctly formatted, it doesn’t matter whether
> the headers were stripped and reapplied internally vs. modified in place).
>  
> In order to mix layer 2 service (inner Ethernet header present) and layer 3 service
> (no inner Ethernet header present) on the same VNI, the encapsulation needs to indicate
> whether the first header after the encapsulation header is Ethernet or IP.  Obviously,
> there are ways to do that (GRE comes to mind), but I wonder whether there’s a strong
> requirement for multiple nvo3 encapsulation formats on a single VNI vs. forcing use of
> a separate VNI if there’s a desire for more than one encapsulation format.
>  
> Thanks,
> --David
>  
> Sent: Tuesday, June 26, 2012 9:24 AM
> Cc: Aldrin Isaac; Black, David; [email-masked]; Lucy yong

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 2 Jul 2012 22:29:04 -0400

Hi Marc/David,

I've attached a "proof-of-concept" design to express the points I have raised.  Hopefully a picture is worth a thousand words.  It is a PDF file since I simply don't have the time to draw it as ascii art.

The illustration tries to capture (1) multiple VN per VNI (and relatedly, single interface on TES) and (2) VNIF.  In this POC the L2-based subnet comprise of 3 VNs to force VMs to remain in their availability zone (DC) but allows them to communicate with other subnets or Internet via gateways in both DC (with preference for local DC).  This illustration could represent a single tenant of a cloud DC provider or infrastructure owned and operated by an enterprise.

The illustration also tries to capture a more mobile L3-based VN where routing information is in the form of host-routes with aggregation of those routes at the gateways.

Best -- aldrin

On Jun 26, 2012, at 11:51 PM, Aldrin Isaac wrote:

> Actually I'm not convinced mixing types is good.  Needed to clarify where things stood today.
> 
>> 
> Even in a PE to CE-router scenario, there is also the concept of "shared VRF" model and "interface VRF" model.  In the shared VRF model multiple CE are connected to the same VRF and each VRF is connected to one VPN.  Here if a CE needs to be connected to another VPN you provision yet another interface to connect to the associated VRF on the PE, the CE and all the associated routing configuration.  OTOH, with the interface VRF model each VRF has only one CE connected (sometimes each interface of the CE gets it's own VRF on the PE).  In this model enabling an additional VPN for the CE only involves importing and exporting an additional RT pair at it's VRF.
> 
> Furthermore, in an enterprise setting it is generally expected that the network bears the network-related complexity.  I suppose in outfits where the same folks deal with both sides (network and host) this may not matter -- but my understanding is that these folks are a minority.
> 
> 
>> 
>> Best -- aldrin
>>

---

From: <[email-masked]>
Date: Tue, 3 Jul 2012 07:48:23 +0000

Hi,

> "learning" is required when the thing to be learned is not administered and there is some form of trust model to auto-configure the learner. The L2 example being MAC addresses. I'm really strugglin…

I agree that we should try to avoid "learning" in the context of L3.

-Thomas

>>>>>>>>> list and/or the NVO3 chairs.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: <[email-masked]>
Date: Tue, 3 Jul 2012 11:51:10 +0000

Hello Maria, Dave, all,

Here is a proposed rewrite of section 4.2.2 aiming at avoiding the 
"learning" term for L3 and take into account the fact that, even for L2, 
dataplane learning is just one option among many.

----
4.2.2 Determining addresses behind VAPs

     For an L2 NVE, the NVE needs to be able to determine MAC 
address(es) of the end-systems present on a VAP (for instance, 
dataplane-learning may be relied upon for this purpose). For an L3 NVE, 
the NVE needs to be able to determine IP address(es) of the end-systems 
present on a VAP.

     In both cases, coordination with the NVE control protocol is needed 
such that when the NVE determines that the set of address(es) behind a 
VAP has changed, it triggers the local NVE control plane to distribute 
this information to its peers.
----

Comments ?

-Thomas

Current text is :

4.2.2. Coordination between data plane and control plane

     Often a combination of data plane and control based learning is 
necessary. Learning is applied towards end-user facing ports whereas 
distribution is used on the tunnel ports. Coordination between the 
learning engine and the control protocol is needed such
that when a new address gets learned or an old address is removed, it 
triggers the local control plane to distribute this information to its 
peers.

NAPIERALA, MARIA H a écrit :
 > "Data plane learning is applicable _only_ to L2 whereas control plane
 > learning is applicable to both L2 and L3."
 >
 > I would remove the second sentence "Control plane learning does not
 > require dynamic data plane learning" because it's a tautology.
 >
 > Also, it is still not clear whether you are assuming that a layer 2
 > solution can be solely based on control plane updates.
 >
 > Maria
 >
 >
 >> -----Original Message----- From: LASSERRE, MARC (MARC)
 >> [mailto:[email-masked]] Sent: Monday, July 02,
 >> 2012 12:29 PM To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Pedro
 >> Roque Marques; <[email-masked]> Cc: [email-masked] Subject: RE:
 >> [nvo3] call for adoption: draft-lasserre-nvo3-framework-02
 >>
 >> Maria, all,
 >>
 >> Data plane learning is applicable to L2 whereas control plane
 >> learning is applicable to both L2 and L3. Control plane learning
 >> does not require dynamic data plane learning.
 >>
 >> Data plane learning implies that flooding of unknown destinations
 >> be supported and hence implies that broadcast and/or multicast be
 >> supported. Multicasting in the core network for dynamic learning
 >> may lead to significant scalability limitations. Specific
 >> forwarding rules must be enforced to prevent loops from happening.
 >> This can be achieved using a spanning tree, a shortest path tree,
 >> or a split-horizon mesh.
 >>
 >> function of the number of virtual machines. Different forms of
 >> caching can also be utilized to minimize state distribution between
 >> the various elements.
 >>
 >> -Marc
 >>
 >>> -----Original Message----- From: [email-masked]
 >>> [mailto:[email-masked]] On Behalf Of NAPIERALA, MARIA H
 >>> Sent: Monday, July 02, 2012 6:13 PM To: Luyuan Fang (lufang);
 >>> LASSERRE, MARC (MARC); Pedro Roque Marques;
 >>> <[email-masked]> Cc: [email-masked] Subject: Re: [nvo3] call
 >>> for adoption: draft-lasserre-nvo3-framework-02
 >>>
 >>> Marc,
 >>>
 >>>> -----Original Message----- From: Luyuan Fang (lufang)
 >>>> [mailto:[email-masked]] Sent: Monday, July 02, 2012 11:58 AM
 >>>> Marques; <[email-masked]> Cc: [email-masked] Subject: RE:
 >>>> [nvo3] call for adoption: draft-lasserre-nvo3-framework-02
 >>>>
 >>>> Marc,
 >>>>
 >>>> all L3 solutions... Luyuan
 >>>>
 >>>>> -----Original Message----- From: [email-masked]
 >>>>> LASSERRE, MARC (MARC) Sent: Monday, July 02, 2012 9:04 AM To:
 >>>>> NAPIERALA, MARIA H; Pedro Roque Marques;
 >> <[email-masked]>
 >>>>> Cc: [email-masked] Subject: Re: [nvo3] call for adoption:
 >>>>>> -----Original Message----- From: [email-masked]
 >>>>>> [mailto:[email-masked]] On Behalf Of NAPIERALA,
 >>>>>> MARIA H Sent: Saturday, June 30, 2012 2:18 AM To: Pedro
 >>>>>> Roque Marques; <[email-masked]> Cc: [email-masked]
 >>>>>>>> Thanks, --David
 >>>>>>>>> -----Original Message----- From:
 >>>>>>>>> [email-masked]
 >>>>>>>>> NAPIERALA, MARIA H Sent: Friday, June 29, 2012 2:14
 >>>>>>>>> PM To: [email-masked] Subject: Re: [nvo3] call for
 >>>>>>>>> adoption: draft-lasserre-
 >> nvo3-
 >>>>>>> framework-02
 >>>>>>>>>
 >>>>>>>>>> This message begins a two week Call for Adoption
 >>>>>>>>>> of http://tools.ietf.org/html/c-02 by the NVO3
 >>> working group,
 >>>>>>>>>> ending on 02-July-2012.
 >>>>>>>>>>
 >>>>>>>>>> Thanks, -Benson& Matthew

---

From: David Allan I <[email-masked]>
Date: Tue, 3 Jul 2012 10:36:43 -0400

Hi Thomas:

It is a definite improvement.

Given that one of the reason for a change in the set of addresses is mobility, coordinating change as described does not sound like it goes far enough. What we really want is a registration system with minimal overhead, minimal latency, hence minimal possibilty of race conditions.

Without making a meal of it, perhaps...

"In both cases, coordination with the NVE control protocol is needed such that when the NVE determines that the set of address(es) behind a VAP has changed, it triggers the local NVE control plane to promptly synchronize this information to its peers."

Cheers
Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, July 03, 2012 4:51 AM
To: [email-masked]
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02 / section 4.2.2

Hello Maria, Dave, all,

Here is a proposed rewrite of section 4.2.2 aiming at avoiding the "learning" term for L3 and take into account the fact that, even for L2, dataplane learning is just one option among many.

----

     For an L2 NVE, the NVE needs to be able to determine MAC
address(es) of the end-systems present on a VAP (for instance, dataplane-learning may be relied upon for this purpose). For an L3 NVE, the NVE needs to be able to determine IP address(es) of the end-systems present on a VAP.

     In both cases, coordination with the NVE control protocol is needed such that when the NVE determines that the set of address(es) behind a VAP has changed, it triggers the local NVE control plane to distribute this information to its peers.
----

Comments ?

-Thomas

Current text is :

     Often a combination of data plane and control based learning is necessary. Learning is applied towards end-user facing ports whereas distribution is used on the tunnel ports. Coordination between the learning engine and the control protocol is needed such that when a new address gets learned or an old address is removed, it triggers the local control plane to distribute this information to its peers.

NAPIERALA, MARIA H a écrit :
 > "Data plane learning is applicable _only_ to L2 whereas control plane  > learning is applicable to both L2 and L3."
 >
 > I would remove the second sentence "Control plane learning does not  > require dynamic data plane learning" because it's a tautology.
 >
 > Also, it is still not clear whether you are assuming that a layer 2  > solution can be solely based on control plane updates.
 >
 > Maria
 >
 >
 >> -----Original Message----- From: LASSERRE, MARC (MARC)  >> [mailto:[email-masked]] Sent: Monday, July 02,  >> 2012 12:29 PM To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Pedro  >> Roque Marques; <[email-masked]> Cc: [email-masked] Subject: RE:
 >> [nvo3] call for adoption: draft-lasserre-nvo3-framework-02  >>  >> Maria, all,  >>  >> I suggest adding the first sentence in section 4.2.1 (see below)  >> for clarification.
 >>
 >> 4.2.1. Data plane vs Control plane driven  >>  >> Data plane learning is applicable to L2 whereas control plane  >> learning is applicable to both L2 and L3. Control plane learning  >> does not require dynamic data plane learning.
 >>
 >> Data plane learning implies that flooding of unknown destinations  >> be supported and hence implies that broadcast and/or multicast be  >> supported. Multicasting in the core network for dynamic learning  >> may lead to significant scalability limitations. Specific  >> forwarding rules must be enforced to prevent loops from happening.
 >> This can be achieved using a spanning tree, a shortest path tree,  >> or a split-horizon mesh.
 >>
 >> It should be noted that the amount of state to be distributed is a  >> function of the number of virtual machines. Different forms of  >> caching can also be utilized to minimize state distribution between  >> the various elements.
 >>
 >> -Marc
 >>
 >>> -----Original Message----- From: [email-masked]  >>> [mailto:[email-masked]] On Behalf Of NAPIERALA, MARIA H  >>> Sent: Monday, July 02, 2012 6:13 PM To: Luyuan Fang (lufang);  >>> LASSERRE, MARC (MARC); Pedro Roque Marques;  >>> <[email-masked]> Cc: [email-masked] Subject: Re: [nvo3] call  >>> for adoption: draft-lasserre-nvo3-framework-02  >>>  >>> Marc,  >>>  >>> Yes, this sentence is much more general and not excluding  >>> control-plane only solutions. You might clarify it more by  >>> stating that control plane only solutions with no data plane  >>> learning are also possible (as Luyuan suggested).
 >>>
 >>> Maria
 >>>
 >>>
 >>>> -----Original Message----- From: Luyuan Fang (lufang)  >>>> [mailto:[email-masked]] Sent: Monday, July 02, 2012 11:58 AM  >>>> To: LASSERRE, MARC (MARC); NAPIERALA, MARIA H; Pedro Roque  >>>> Marques; <[email-masked]> Cc: [email-masked] Subject: RE:
 >>>> [nvo3] call for adoption: draft-lasserre-nvo3-framework-02  >>>>  >>>> Marc,  >>>>  >>>> The sentence is correct, but I think you should clarify when  >>>> data planning learning is applicable, when it is not, e.g. if  >>>> all L3 solutions... Luyuan  >>>>  >>>>> -----Original Message----- From: [email-masked]  >>> [mailto:[email-masked]] On Behalf  >>>> Of  >>>>> LASSERRE, MARC (MARC) Sent: Monday, July 02, 2012 9:04 AM To:
 >>>>> NAPIERALA, MARIA H; Pedro Roque Marques;  >> <[email-masked]>  >>>>> Cc: [email-masked] Subject: Re: [nvo3] call for adoption:
 >>>>> "A combination of data plane and control plane based  >>> learning may be  >>>>> applicable."
 >>>>>
 >>>>> Marc
 >>>>>
 >>>>>> -----Original Message----- From: [email-masked]  >>>>>> [mailto:[email-masked]] On Behalf Of NAPIERALA,  >>>>>> MARIA H Sent: Saturday, June 30, 2012 2:18 AM To: Pedro  >>>>>> Roque Marques; <[email-masked]> Cc: [email-masked]  >>>>>> Subject: Re: [nvo3] call for adoption:
 >>>>>> draft-lasserre-nvo3-framework-02  >>>>>>  >>>>>>>> As has been pointed out a number of times, pure data  >>>>>> plane learning  >>>>>>> leads  >>>>>>>> to a lot of BUM traffic flooding, so a combination of  >>>>>> data plane and  >>>>>>> control  >>>>>>>> plane can work better with existing systems and  >>>>>>>> improve  >>>>>> scalability.
 >>>>>>>
 >>>>>>> Data plane learning, l2 header transparency, bridging  >>>>>> interoperability  >>>>>>> are all very reasonable requirements for one type of  >>>>>> data-centers. But  >>>>>>> also an unacceptable burden for a different class of DC  >>>>>> designs. In my  >>>>>>> view you and Maria are just looking at different types  >>>>>>> of  >>>>>> DC designs.
 >>>>>> "Often a combination of data plane and control based  >>> learning is  >>>>>> necessary" seems to be assuming that pretty much all  >>> data centers  >>>>>> must use (some) data plane learning. I have suggested  >>> to clarify  >>>>>> it.
 >>>>>>
 >>>>>> Maria
 >>>>>>
 >>>>>>
 >>>>>>> Pedro.
 >>>>>>>
 >>>>>>>>
 >>>>>>>> Thanks, --David
 >>>>>>>>
 >>>>>>>>
 >>>>>>>>> NAPIERALA, MARIA H Sent: Friday, June 29, 2012 2:14  >>>>>>>>> PM To: [email-masked] Subject: Re: [nvo3] call for  >>>>>>>>> adoption: draft-lasserre-  >> nvo3-  >>>>>>> framework-02  >>>>>>>>>  >>>>>>>>> A comment on section 4.2.2 "Coordination between  >>>>>>>>> data  >> plane  >>>> and  >>>>>>> control plane"
 >>>>>>>>>
 >>>>>>>>> "Often a combination of data plane and control based  >>>>>> learning is  >>>>>>>>> necessary."
 >>>>>>>>>
 >>>>>>>>> I think this statement is too strong since in a  >>>>>>>>> solution  >>>>>> proposed  >>>>>>>>> in  >>>>>>> draft-  >>>>>>>>> marques-l3vpn-end-system, for example, there is no  >>> data plane  >>>>>>> learning in a  >>>>>>>>> virtual network. Maybe it should be explain when  >>>>>>>>> such  >>>>>> combination  >>>>>>>>> is necessary.
 >>>>>>>>>
 >>>>>>>>> Maria
 >>>>>>>>>
 >>>>>>>>>
 >>>>>>>>>> This message begins a two week Call for Adoption  >>>>>>>>>> of http://tools.ietf.org/html/c-02 by the NVO3  >>> working group,  >>>>>>>>>> ending on 02-July-2012.
 >>>>>>>>>>
 >>>>>>>>>> Please respond to the NVO3 mailing list with any  >>> statements  >>>> of  >>>>>>>>>> approval or disapproval, along with any additional  >>>>>> comments that  >>>>>>>>>> might explain your position. Also, if any NVO3  >>> participant  >>>>>>>>>> is aware of IPR associated with this draft, please  >>>>>>>>>> inform  >>>>>> the mailing  >>>>>>>>>> list and/or the NVO3 chairs.
 >>>>>>>>>>
 >>>>>>>>>> Thanks, -Benson& Matthew
 >>>>>>>>>>
 >>>>>>>>>>
 >>>>>>>>>

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: <[email-masked]>
Date: Tue, 3 Jul 2012 14:42:26 +0000

Hi Dave,

> It is a definite improvement.

Good.

> Given that one of the reason for a change in the set of addresses is
> mobility, coordinating change as described does not sound like it
> goes far enough. What we really want is a registration system with
> minimal overhead, minimal latency, hence minimal possibilty of race
> conditions.
 >
> "In both cases, coordination with the NVE control protocol is needed
> such that when the NVE determines that the set of address(es) behind
> a VAP has changed, it triggers the local NVE control plane to
> promptly synchronize this information to its peers."

Let's not forget this is a framework document, and avoid going into what 
starts to be requirements on the NVE control plane protocol.

Thanks,

-Thomas

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 3 Jul 2012 15:06:04 +0000

Thomas,

What about the following:

> address(es) of the end-systems present on a VAP (typically
> dataplane-learning is relied upon for this purpose). For an L3 NVE,
> present on a VAP via a control plane. In the latter case, the data-plane learning is never involved.

BTW, I don’t think there is anything fundamentally wrong with "control plane learning" expression. In fact, it is used all over the place in many ietf documents I am familiar with.

Maria

> [email-masked]
> Sent: Tuesday, July 03, 2012 7:51 AM
> To: [email-masked]
> / section 4.2.2
> _______________________________________________________________________
> __________________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les
> messages electroniques etant susceptibles d'alteration,
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> As emails may be altered, France Telecom - Orange is not liable for
> messages that have been modified, changed or falsified.
> Thank you.
>

---

From: <[email-masked]>
Date: Tue, 3 Jul 2012 15:26:40 +0000

Hello Maria,

>> present on a VAP via a control plane. In the latter case, the data-plane learning is never involved.

This last sentence is not incorrect, but on the other hand if someone 
tomorrow would like to do consider doing dataplane learning for IP 
addresses, is it in the scope of the NVO3 framework document to say he 
should not ?

We can still hint that dataplane learning would be quite unexpected:

    In the latter case, data-plane learning is typically not involved.

?

-Thomas

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 3 Jul 2012 17:09:04 +0000

> addresses, 

I hope it never happens ;-)

is it in the scope of the NVO3 framework document to say he
>     In the latter case, data-plane learning is typically not involved.

I would rather use: "In the latter case, data-plane learning should be not involved.
> a
> >>   >> I suggest adding the first sentence in section 4.2.1 (see
> below)
> >>   >> for clarification.
> >>   >>

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 3 Jul 2012 19:58:52 +0000

Adrin,

Thanks for sharing the nice design figure.

A few comments to the figure:

1)      what about VMs (or TES) in DC which are connected by IPSec?  If you purchase Private Virtual Networks from Amazon's EC2, you can only connect to your VPC (VMs) via IPSec.

2)      What about the VMs under your "Overlay Module" which are not part of L3VPN or L2VPN? Are they terminated by the GW or terminated at the "Overlay Module"?

3)      In your picture, L3VNI/L2VNI are all terminated at the Overlay Modules. Do you mean that L3VPN/L2VPN encapsulations are actually done  by the "overlay module"? If yes, then existing L3VPN/L2VPN mechanisms  or solutions are already defined. What else is needed then?

Linda Dunbar

From: [email-masked] [mailto:[email-masked]] On Behalf Of Aldrin Isaac
Sent: Monday, July 02, 2012 9:29 PM
To: LASSERRE, MARC (MARC); [email-masked]
Cc: [email-masked]; Lucy yong; Aldrin Isaac
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Marc/David,

Best -- aldrin

---

From: Truman Boyes <[email-masked]>
Date: Tue, 3 Jul 2012 16:36:58 -0400

Hi,

Regarding IPSec to connect to VMs; are you referring to providing IPSec as
a method to provide secure connectivity to the VMs in a DC? Or are you
interested is describing connectivity between VMs in a private cloud and
VMs in EC2 via VPC?

Either can be accomplished by connecting an IPSec tunnel and associated
routes into the L3VPN Access Mesh or L3VPN Interconnect Mesh.

Kind regards,
Truman

On Tue, Jul 3, 2012 at 3:58 PM, Linda Dunbar <[email-masked]>wrote:

>  Adrin, ****
>
> ** **
>
> Thanks for sharing the nice design figure. ****
>
> ** **
>
> A few comments to the figure: ****
>
> **1)      **what about VMs (or TES) in DC which are connected by IPSec?
>  If you purchase Private Virtual Networks from Amazon’s EC2, you can only
> connect to your VPC (VMs) via IPSec. ****
>
> **2)      **What about the VMs under your “Overlay Module” which are not
> part of L3VPN or L2VPN? Are they terminated by the GW or terminated at the
> “Overlay Module”? ****
>
> **3)      **In your picture, L3VNI/L2VNI are all terminated at the
> Overlay Modules. Do you mean that L3VPN/L2VPN encapsulations are actually
> done  by the “overlay module”? If yes, then existing L3VPN/L2VPN
> mechanisms  or solutions are already defined. What else is needed then? **
> **
>
> ** **
>
> Linda Dunbar****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Aldrin Isaac
> *Sent:* Monday, July 02, 2012 9:29 PM
> *To:* LASSERRE, MARC (MARC); [email-masked]
> *Cc:* [email-masked]; Lucy yong; Aldrin Isaac
> *Subject:* Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02*
> ***
>
> ** **
>
> (resending from my correct mailto email with minor edits)****
>
> ** **
>
> ** **
>
> Hi Marc/David,****
>
> ** **
>
> I've attached a "proof-of-concept" design to express the points I have
> raised.  Hopefully a picture is worth a thousand words.  It is a PDF file
> since I simply don't have the time to draw it as ascii art.****
>
> ** **
>
> The illustration tries to capture (1) multiple VN per VNI (and relatedly,
> single interface on TES) and (2) VNIF.  In this POC the L2-based subnet
> comprise of 3 VNs to force VMs to remain in their availability zone (DC)
> but allows them to communicate with other subnets or Internet via gateways
> in both DC (with preference for local DC).  This illustration could
> represent a single tenant of a cloud DC provider or infrastructure owned
> and operated by an enterprise.****
>
> ** **
>
> The illustration also tries to capture a more mobile L3-based VN where
> routing information is in the form of host-routes with aggregation of those
> routes at the gateways.****
>
> ** **
>
> Best -- aldrin****
>
> ** **
>

---

From: János Farkas <[email-masked]>
Date: Thu, 5 Jul 2012 06:37:51 +0200

Hi Marc,

Thank you for your response!
Sorry for the non-prompt reply from my side, I'm traveling these weeks.

Please find some notes in-line.

Best regards,
Janos

On 7/2/2012 3:36 PM, LASSERRE, MARC (MARC) wrote:
>> From:[email-masked]  [mailto:[email-masked]] On
>> Behalf Of János Farkas
>> To:[email-masked]
It was just not entirely clear to me from the text itself. The reason 
might be somewhat different use of terminology.
I think, in general, a tunnel is either point-to-point or multipiont 
(including point-to-multipoint) and not unicast or multicast. A tunnel 
then may carry unicast and multicast VM (user) traffic. Tunnel states 
are maintained in the core, less states for a point-to-point tunnel, 
than for a multipoint, but both require some states. For VM traffic 
carried in a point-to-point tunnel, there is no additional forwarding 
state required, but the tunnel state is enough. In case of multipoint 
tunnels, however, forwarding state is also required for optimal 
forwarding aside the tunnel state, except for multicast traffic 
involving each end point (NVE) of the tunnel. (In some solutions tunnel 
and forwarding state may 'collapse', they may be handled together in a 
non-separable way.)
I'm not sure which terminology is the cleaner and easier to understand 
here, please go for that. (I just thought that a framework goes for the 
generic one.)

> How about "may lead to " instead of "can lead to" then?

Sounds good. Though, the sentence might be also extended to give some 
more details on the scalability limitations, e.g. s.g. like 
"Multicasting in the core network for dynamic learning may lead to 
scalability limitations, which can be avoided by optimized multicast."
What I mean under "optimized multicast" is that only those NVEs receive 
the multicast that really need to. Auto-dicsovery may help a lot here.

> Fine. I do not have a problem either removing this sentence or updating it to "the amount of state is dependent upon network topology and the number of VMs".
Removal seems simpler to me. It is not easy to incorporate in one 
sentence what the amount of state depends on. The updated version of the 
sentence looks better. Further update might be possible if you would 
like to keep the sentence, e.g. "the amount of state is dependent upon 
network topology and the number of VMs and the traffic among them."

> This is not about providing visibility to tenants. It is about running multiple overlay services over a common underlay.
I know that the main point here is to have several overlays on top of 
the same infrastructure.
Nonetheless, this section says "Better visibility between overlays and 
underlays or their controllers"
Clarification might be useful then if the goal is not providing 
visibility to the tenants.

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 5 Jul 2012 21:30:07 -0400

Hi Linda,  comments inline. Best regards -- aldrin

On Tuesday, July 3, 2012, Linda Dunbar wrote:

> This is as Truman mentioned in an earlier email.  The idea is to not
restrict the entry point of the IPSec connection.  As a matter of fact if
the tenant has multiple subnets/VN there is only need for a single IPSec
tunnel as the illustration depicts.

> The far left and far right NVE are NVE-only (no hypervisors) while the NVE
at the bottom are combined hypervisor/NVE.  All the VM in the illustration
are connected to a VNI through a "bump-in-the-virtual-wire" virtual
firewall. All the VNI are members of at least one VN. I'm not entirely
clear as to what you mean by "terminated by the GW".  From network
perspective I see end station as terminated at VNI.

> mechanisms  or solutions are already defined. What else is needed then?
>
> Here [I believe] I am using the models outlined in Marc's framework draft.
 The transport tunnel encapsulation is handled by the Overlay Module.  The
Ethernet/IP encapsulation is done by the end station.  The VNI is simply a
forwarding table that implements some match+action rules.

>
> Linda Dunbar****
>
> ** **
>
> *From:* [email-masked] <javascript:_e({}, 'cvml',
> [email-masked]');> [mailto:[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>]
> *On Behalf Of *Aldrin Isaac
> *To:* LASSERRE, MARC (MARC); [email-masked] <javascript:_e({},
> 'cvml', [email-masked]');>
> *Cc:* [email-masked] <javascript:_e({}, 'cvml', [email-masked]');>; Lucy
> yong; Aldrin Isaac
> raised.  Hopefully a picture is worth a thousand words.  It is a PDF f
>

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 5 Jul 2012 21:30:47 -0400

Adrin, ****

** **

** **

**1)      **what about VMs (or TES) in DC which are connected by IPSec?  If
you purchase Private Virtual Networks from Amazon’s EC2, you can only
connect to your VPC (VMs) via IPSec. ****

 **3)      **In your picture, L3VNI/L2VNI are all terminated at the Overlay
Modules. Do you mean that L3VPN/L2VPN encapsulations are actually done  by
the “overlay module”? If yes, then existing L3VPN/L2VPN mechanisms  or
solutions are already defined. What else is needed then?

Here [I believe] I am using the models outlined in Marc's framework draft.
 The transport tunnel encapsulation is handled by the Overlay Tunnel.  The
Ethernet/IP encapsulation is done by the end station.  The VNI is simply a
forwarding table that implements some match+action rules.

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 6 Jul 2012 23:34:35 +0000

Hi Aldrin,

My interpretation of the model in the framework document is that a VNI is the intersection of a single VN with a single NVE, with the VAP determining a connection to a single VNI.  Your picture seems to show a single VAP connecting to a single VNI where two VNs intersect at the same VNI.  Given that, how would the VAP differentiate which VN to associate ingress packets with?  For example, if one of the VMs at the bottom sent an ARP to the L2VNI, which VN would the broadcast be sent to?  Since each VN is allowed to support overlapping addresses, how would a unicast frame sent by that VM on the VAP be delivered to the correct overlapping destination address in the two VNs?

Thanks, Larry

From: Aldrin Isaac <[email-masked]<mailto:[email-masked]>>
Date: Monday, July 2, 2012 7:29 PM
To: "LASSERRE, MARC (MARC)" <[email-masked]<mailto:[email-masked]>>, David Black <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Aldrin Isaac <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Marc/David,

Best -- aldrin

Hi Marc,

My comments inline,

Hi Aldrin,

See my comments below.

Thanks,
Marc

________________________________
From: Aldrin Isaac [mailto:[email-masked]]
Sent: Tuesday, June 26, 2012 3:24 PM
To: LASSERRE, MARC (MARC)
Cc: Aldrin Isaac; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Lucy yong
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Marc,

Best -- aldrin

See my comments below.

Thanks,
Marc

---

From: Aldrin Isaac <[email-masked]>
Date: Sat, 7 Jul 2012 11:08:04 -0400

Hi Larry,

I view the VAP as just a virtualized host-facing network port, and the VNI as an instance of a forwarding table.  

Wrt multiple VNI on a single NVE that are members of the same VN:

Let me try the following example.  Let's assume you have 10 TES on a hypervisor/NVE that need to be on the same VN.  If you need a certain behavior implemented at the forwarding table (say an L2VNI equivalent of a "VACL") for 5 of those TES and another behavior for the other 5 TES then you can have two VNI on the same NVE that are members of the same VPN.

Another better example would be where the first 5 TES additionally need to be a part of another VN.

Wrt a single VNI being a member of more than one VN:

Consider a network overlay solution where at the center is a pub-sub system with VNI as clients of this system.  The VNI publish routes with one or more topic-IDs and also subscribe to one or more topic-IDs (and thereby receive routes that were published with those topic-IDs).  Let's assume each topic-ID can be considered a VN.  By publishing and subscribing with a topic-ID a VNI is effectively participating (becomes a member) in the topic (VN).

If a VNI is a member of (subscribes to) two VN (topic-IDs) it will receive routes from VNI that published routes with both VN (topic-IDs).  If a route is associated to both VN then the VNI (or the big brain in the sky) has to pick one based on some tie breaker.  However there are two cases (1) the route represents the same destination in which case it doesn't matter.  Also generally in the case of (1) the pub-sub system delivers to the VNI one route with two topic-ID on it (in case of BGP that would be two Route Targets).  Case (2) is where the route indeed represent different destinations -- this is a case of operator error.  The disambiguation will have to happen somewhere -- either in the network or in the host.

Furthermore, in L2VN a "VNI route" would probably be used for identifying where to send BUM, while unicast MAC routes would be used for forwarding known unicast.  In my illustration, the BUM traffic would be sent to all VNI represented by "VNI routes" in the ingress VNI regardless of how many VN the VNI is a member of.

I'm hoping this email will also address Dave's earlier email.

Best -- aldrin

On Jul 6, 2012, at 7:34 PM, Larry Kreeger (kreeger) wrote:

> From: Aldrin Isaac <[email-masked]>
> To: "LASSERRE, MARC (MARC)" <[email-masked]>, David Black <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>, Lucy yong <[email-masked]>, Aldrin Isaac <[email-masked]>

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Mon, 9 Jul 2012 11:20:39 +0200

Hi Larry,

See my comments below.

Thanks,
Marc

________________________________
From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: Saturday, July 07, 2012 1:35 AM
To: Aldrin Isaac; LASSERRE, MARC (MARC); [email-masked]
Cc: [email-masked]; Lucy yong; Aldrin Isaac
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Aldrin,

My interpretation of the model in the framework document is that a VNI is the intersection of a single VN with a single NVE, with the VAP determining a connection to a single VNI.
This is correct.
 Your picture seems to show a single VAP connecting to a single VNI where two VNs intersect at the same VNI.  Given that, how would the VAP differentiate which VN to associate ingress packets with?  For example, if one of the VMs at the bottom sent an ARP to the L2VNI, which VN would the broadcast be sent to?  Since each VN is allowed to support overlapping addresses, how would a unicast frame sent by that VM on the VAP be delivered to the correct overlapping destination address in the two VNs?
This is clearly a problem. Hence, the reason for having a VNI per VN in the draft. Unless we are talking about unqualified learning?

Thanks, Larry

Hi Marc/David,

Best -- aldrin

Hi Marc,

My comments inline,

Hi Aldrin,

See my comments below.

Thanks,
Marc

Hi Marc,

Best -- aldrin

See my comments below.

Thanks,
Marc

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Tue, 10 Jul 2012 03:48:48 +0000

Hi Aldrin,

I'm afraid this didn't help give me the answers to my questions – at least directly.

First, you discuss multiple VNIs on the same NVE connected to the same VN.  I did not see this illustrated in your diagram, so I didn't have questions about it.  From your description, you seem to be making an assumption that there is some kind of firewall policy associated with the VNI.  I don't see the benefit of applying firewall policies to VNIs.  It seems more straight forward to apply them to the VAPs.

Regarding the topic of my questions (one VNI connected to multiple VNs), I am not clear on the answers to my questions based on your example.  Based on your example, it seems like you are assuming that addresses cannot overlap between two VNs (or at least two that have been associated with the same VNI).  It also seems like you are assuming a VN doesn't necessarily represent a subnet (I'm thinking of an L2 VN), but perhaps more like a Closed User Group.  If a VN connects VNI connects to multiple VNs then it is a member of multiple CUGs.  Did I interpret that correctly?

Thanks, Larry

From: Aldrin Isaac <[email-masked]<mailto:[email-masked]>>
Date: Saturday, July 7, 2012 8:08 AM
To: Cisco Employee <[email-masked]<mailto:[email-masked]>>
Cc: "LASSERRE, MARC (MARC)" <[email-masked]<mailto:[email-masked]>>, David Black <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Rekhter Yakov <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02

Hi Larry,

Best -- aldrin

Hi Aldrin,

Thanks, Larry

Hi Marc/David,

Best -- aldrin

Hi Marc,

My comments inline,

Hi Aldrin,

See my comments below.

Thanks,
Marc

Hi Marc,

Best -- aldrin

See my comments below.

Thanks,
Marc

> From: Aldrin Isaac [mailto:[email-masked]<x-msg://5430/>]
> Cc: Lucy yong; [email-masked]<x-msg://5430/>; [email-masked]<x-msg://5430/>

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 10 Jul 2012 23:06:11 -0400

Larry, will try again as my attempt to give some background/perspective before I gave my answer to your question seems to have backfired.  See inline. -- aldrin

On Jul 9, 2012, at 11:48 PM, Larry Kreeger (kreeger) wrote:

> Hi Aldrin,
> 
> I'm afraid this didn't help give me the answers to my questions – at least directly

I think the problem is a matter of perspective/experience.  If the perspective of NVO3 is through the lens of VLAN/VXLAN/NVGRE/VNET/OTV where the VNID has global/network/domain-wide scope and is used as-is on the wire, then your questions are quite reasonable as they are based on the limitations of these solutions.  However if the perspective is through the lens of IPVPN and E-VPN then the restrictions of one VN-per-VNI and one VNI-per-VN-per-NVE are not reasonable since IPVPN/EVPN inherently have support for multiple VN-per-VNI and multiple VNI-per-VN-per-NVE.

> First, you discuss multiple VNIs on the same NVE connected to the same VN.  I did not see this illustrated in your diagram, so I didn't have questions about it.  From your description, you seem to…

This is indeed in my illustration.  I'm resending it.  If you look at NVE1 and NVE2 you will see that the 2 L2VNI on each of these NVE share a "DC<n> mesh" L2VN.  The illustration depicts both multiple VN per VNI as well as multiple VNI on a single NVE being members of the same VN.  

Regarding features such as filtering implemented at the forwarding table (such as VACL) -- L2 ACL on standard dot1q switches are usually implemented with VACL not per-port ACL.  But to be clear, VACL is only one possible example of a feature that can be implemented at the forwarding table.  Another thing that might be implemented at the forwarding table level might be features like PVLAN (although I prefer that pvlan solution be implemented using overlapping VN model as shown in my illustration).

> Regarding the topic of my questions (one VNI connected to multiple VNs), I am not clear on the answers to my questions based on your example.  Based on your example, it seems like you are assuming…

Overlapping address space is generally an inter-tenant issue, not an intra-tenant issue.  What reasonable example exists where a single tenant might choose to use the same IP to address different end stations?

Generally an L2VNI will be associated to a single subnet.  In the cases where pvlan is used, a single subnet might have multiple CUGs.  As you are more than likely well aware, in pvlan a "community" vlan represents a CUG yet it's end-station members are generally on the same subnet as end-station on another community VLAN that shares the same primary vlan.  The members of both community vlan will share a "primary" vlan where generally the gateway is present.  In EVPN this could be implemented by the DC network operator as multiple overlapping VPN, i.e two mesh VPN for the two community vlan CUGs and a hub-and-spoke VPN for the primary VLAN.  My illustration depicts this.  In this solution approach, a single VNI is a member of multiple VN where each VN represents a CUG.  How would full pvlan feature be implemented with VXLAN solution?  Would it be implemented at the VN, VNI or VAP level?

Hopefully I did a better job with my response this time :}

> To: Cisco Employee <[email-masked]>
> Cc: "LASSERRE, MARC (MARC)" <[email-masked]>, David Black <[email-masked]>, "[email-masked]" <[email-masked]>, Lucy yong <[email-masked]>, Rekhter Yakov <[email-masked]>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 11 Jul 2012 04:14:16 +0000

Date: Tuesday, July 10, 2012 8:06 PM

Hi Aldrin,

OK, now that you point it out, I see it.  I didn't recognize it for what it was.

Regarding features such as filtering implemented at the forwarding table (such as VACL) -- L2 ACL on standard dot1q switches are usually implemented with VACL not per-port ACL.

Yes, but a VACL would apply to all traffic on the VLAN.  If you want finer grain, then you would move down to a Port ACL.

 But to be clear, VACL is only one possible example of a feature that can be implemented at the forwarding table.  Another thing that might be implemented at the forwarding table level might be features like PVLAN (although I prefer that pvlan solution be implemented using overlapping VN model as shown in my illustration).

The example I am most aware of is supported by VMware's vCloud Director, in which vApps (groups of VMs that work together to provide an application) can be cloned multiple times from a template to deploy multiple instances of the application.  A tenant can deploy as many copies of the vApp as they like, and each copy retains its original MAC and IP addresses.  The VMs within a vApp communicate over an isolated layer 2 network.  A NAT device is used to interface these vApps to a network external to the vApp.

VXLAN does not support the concept of Private VLANs.  It provides the same services as a plain old VLAN.  If I wanted to implement a CUG with VXLANs, I would use something else (like a zone-based firewall) in conjunction with it.

Yes, thank you!  …and now that you have, I would question whether this functionality (Closed User Groups) needs to be built into the overlay connectivity itself, or can it be achieved independently (e.g. using a zone-based firewall).

Thanks, Larry

Hi Larry,

Best -- aldrin

Hi Aldrin,

Thanks, Larry

Hi Marc/David,

Best -- aldrin

Hi Marc,

My comments inline,

Hi Aldrin,

See my comments below.

Thanks,
Marc

Hi Marc,

Best -- aldrin

See my comments below.

Thanks,
Marc

---

From: Truman Boyes <[email-masked]>
Date: Wed, 11 Jul 2012 01:12:41 -0400

Hi, the statement around VM cloning and preservation of IP / MAC for each
VM is interesting:

*"The example I am most aware of is supported by VMware's vCloud Director,
in which vApps (groups of VMs that work together to provide an application)
can be cloned multiple times from a template to deploy multiple instances
of the application.  A tenant can deploy as many copies of the vApp as they
like, and each copy retains its original MAC and IP addresses.  The VMs
within a vApp communicate over an isolated layer 2 network.  A NAT device
is used to interface these vApps to a network external to the vApp."*

This is a novel concept, however I see large scale cloud environments doing
this very differently. I have yet to see an intra-tenant address overlap in
DCs, as dynamic environments involving cloning of images typically use DHCP
for address assignment to the VM, and service orchestration handles the
service IP to VM IP mapping (via NAT).  Cloning IP and MAC addresses even
for intra-tenant use is setting up issues for the future.

Kind regards,
Truman

On Wed, Jul 11, 2012 at 12:14 AM, Larry Kreeger (kreeger) <[email-masked]
> wrote:

> Cc: "LASSERRE, MARC (MARC)" <[email-masked]>, David
> Black <[email-masked]>, "[email-masked]" <[email-masked]>, Lucy yong <
> [email-masked]>, Rekhter Yakov <[email-masked]>
>   Larry, will try again as my attempt to give some background/perspective
> before I gave my answer to your question seems to have backfired.  See
> inline. -- aldrin
>
>  I think the problem is a matter of perspective/experience.  If the
> perspective of NVO3 is through the lens of VLAN/VXLAN/NVGRE/VNET/OTV where
> the VNID has global/network/domain-wide scope and is used as-is on the
> wire, then your questions are quite reasonable as they are based on the
> limitations of these solutions.  However if the perspective is through the
> lens of IPVPN and E-VPN then the restrictions of one VN-per-VNI and one
> VNI-per-VN-per-NVE are not reasonable since IPVPN/EVPN inherently have
> support for multiple VN-per-VNI and multiple VNI-per-VN-per-NVE.
>
>   First, you discuss multiple VNIs on the same NVE connected to the same
> VN.  I did not see this illustrated in your diagram, so I didn't have
> questions about it.  From your description, you seem to be making an
> assumption that there is some kind of firewall policy associated with the
> VNI.  I don't see the benefit of applying firewall policies to VNIs.  It
> seems more straight forward to apply them to the VAPs.
>
>
>  This is indeed in my illustration.  I'm resending it.  If you look at
> NVE1 and NVE2 you will see that the 2 L2VNI on each of these NVE share a
> "DC<n> mesh" L2VN.  The illustration depicts both multiple VN per VNI as
> well as multiple VNI on a single NVE being members of the same VN.
>
>
>  Regarding features such as filtering implemented at the forwarding table
> (such as VACL) -- L2 ACL on standard dot1q switches are usually implemented
> with VACL not per-port ACL.
>
>
>  Yes, but a VACL would apply to all traffic on the VLAN.  If you want
> finer grain, then you would move down to a Port ACL.
>
>      But to be clear, VACL is only one possible example of a feature that
> can be implemented at the forwarding table.  Another thing that might be
> implemented at the forwarding table level might be features like PVLAN
> (although I prefer that pvlan solution be implemented using overlapping VN
> model as shown in my illustration).
>
>
>   Regarding the topic of my questions (one VNI connected to multiple
> VNs), I am not clear on the answers to my questions based on your example.
>  Based on your example, it seems like you are assuming that addresses
> cannot overlap between two VNs (or at least two that have been associated
> with the same VNI).  It also seems like you are assuming a VN doesn't
> necessarily represent a subnet (I'm thinking of an L2 VN), but perhaps more
> like a Closed User Group.  If a VN connects VNI connects to multiple VNs
> then it is a member of multiple CUGs.  Did I interpret that correctly?
>
>
>  Overlapping address space is generally an inter-tenant issue, not an
> intra-tenant issue.  What reasonable example exists where a single tenant
> might choose to use the same IP to address different end stations?
>
>
>  The example I am most aware of is supported by VMware's vCloud Director,
> in which vApps (groups of VMs that work together to provide an application)
> can be cloned multiple times from a template to deploy multiple instances
> of the application.  A tenant can deploy as many copies of the vApp as they
> like, and each copy retains its original MAC and IP addresses.  The VMs
> within a vApp communicate over an isolated layer 2 network.  A NAT device
> is used to interface these vApps to a network external to the vApp.
>
>
>  Generally an L2VNI will be associated to a single subnet.  In the cases
> where pvlan is used, a single subnet might have multiple CUGs.  As you are
> more than likely well aware, in pvlan a "community" vlan represents a CUG
> yet it's end-station members are generally on the same subnet as
> end-station on another community VLAN that shares the same primary vlan.
>  The members of both community vlan will share a "primary" vlan where
> generally the gateway is present.  In EVPN this could be implemented by the
> DC network operator as multiple overlapping VPN, i.e two mesh VPN for the
> two community vlan CUGs and a hub-and-spoke VPN for the primary VLAN.  My
> illustration depicts this.  In this solution approach, a single VNI is a
> member of multiple VN where each VN represents a CUG.  How would full pvlan
> feature be implemented with VXLAN solution?  Would it be implemented at the
> VN, VNI or VAP level?
>
>
>  VXLAN does not support the concept of Private VLANs.  It provides the
> same services as a plain old VLAN.  If I wanted to implement a CUG with
> VXLANs, I would use something else (like a zone-based firewall) in
> conjunction with it.
>
>
>  Yes, thank you!  …and now that you have, I would question whether this
> functionality (Closed User Groups) needs to be built into the overlay
> connectivity itself, or can it be achieved independently (e.g. using a
> zone-based firewall).
>
>
>
>   Thanks, Larry
>
>
>
> Cc: "LASSERRE, MARC (MARC)" <[email-masked]>, David
> Black <[email-masked]>, "[email-masked]" <[email-masked]>, Lucy yong <
> [email-masked]>, Rekhter Yakov <[email-masked]>
>  I view the VAP as just a virtualized host-facing network port, and the
> VNI as an instance of a forwarding table.
>
>
>  Let me try the following example.  Let's assume you have 10 TES on a
> hypervisor/NVE that need to be on the same VN.  If you need a certain
> behavior implemented at the forwarding table (say an L2VNI equivalent of a
> "VACL") for 5 of those TES and another behavior for the other 5 TES then
> you can have two VNI on the same NVE that are members of the same VPN.
>
>  Consider a network overlay solution where at the center is a pub-sub
> system with VNI as clients of this system.  The VNI publish routes with one
> or more topic-IDs and also subscribe to one or more topic-IDs (and thereby
> receive routes that were published with those topic-IDs).  Let's assume
> each topic-ID can be considered a VN.  By publishing and subscribing with a
> topic-ID a VNI is effectively participating (becomes a member) in the topic
> (VN).
>
>  If a VNI is a member of (subscribes to) two VN (topic-IDs) it will
> receive routes from VNI that published routes with both VN (topic-IDs).  If
> a route is associated to both VN then the VNI (or the big brain in the sky)
> has to pick one based on some tie breaker.  However there are two cases (1)
> the route represents the same destination in which case it doesn't matter.
>  Also generally in the case of (1) the pub-sub system delivers to the VNI
> one route with two topic-ID on it (in case of BGP that would be two Route
> Targets).  Case (2) is where the route indeed represent different
> destinations -- this is a case of operator error.  The disambiguation will
> have to happen somewhere -- either in the network or in the host.
>
>  Furthermore, in L2VN a "VNI route" would probably be used for
> identifying where to send BUM, while unicast MAC routes would be used for
> forwarding known unicast.  In my illustration, the BUM traffic would be
> sent to all VNI represented by "VNI routes" in the ingress VNI regardless
> of how many VN the VNI is a member of.
>
>  My interpretation of the model in the framework document is that a VNI
> is the intersection of a single VN with a single NVE, with the VAP
> determining a connection to a single VNI.  Your picture seems to show a
> single VAP connecting to a single VNI where two VNs intersect at the same
> VNI.  Given that, how would the VAP differentiate which VN to associate
> ingress packets with?  For example, if one of the VMs at the bottom sent an
> ARP to the L2VNI, which VN would the broadcast be sent to?  Since each VN
> is allowed to support overlapping addresses, how would a unicast frame sent
> by that VM on the VAP be delivered to the correct overlapping destination
> address in the two VNs?
>
>  Thanks, Larry
>
> since I simply don't have the time to draw it as ascii art.
>
>  The illustration tries to capture (1) multiple VN per VNI (and
> relatedly, single interface on TES) and (2) VNIF.  In this POC the L2-based
> subnet comprise of 3 VNs to force VMs to remain in their availability zone
> (DC) but allows them to communicate with other subnets or Internet via
> gateways in both DC (with preference for local DC).  This illustration
> could represent a single tenant of a cloud DC provider or infrastructure
> owned and operated by an enterprise.
>
> routes at the gateways.
>  ------------------------------
> *From:* Aldrin Isaac [mailto:[email-masked]<[email-masked]>]
>
> *Sent:* Tuesday, June 26, 2012 3:24 PM
> *To:* LASSERRE, MARC (MARC)
> *Cc:* Aldrin Isaac; [email-masked]; [email-masked]; Lucy yong
> *Subject:* Re: [nvo3] call for adoption: draft-lasserre-nvo3-framework-02
>
>  Hi Marc,
>
>  Wrt my first question, should we assume that there isn't yet a consensus
> on whether a VNI can support more than one type of VN?
>
>  Regarding one-to-one mapping of VNI to VN, this implies that an
> end-station that needs to be a part of more than one VN will require a
> separate interface into each associated VNI.  This implies that the
> end-station will require a potentially complex routing table (depending on
> how aggregateable the address space within a VN is).  Unfortunately as an
> operator I don't find this an acceptable trade off.  Can we please address
> the issue of managing multiple interfaces and complex tables in
> end-systems? I believe the problem statement needs to capture this.
>
> This is not any different than with existing VPN technologies. Supporting
> multiple interfaces to connect to different subnets and/or nvo3 domains
> does not make routing really complex.
>  If you are referring to VRFs and the like, then I don't see host-facing
> VNI as equivalent.  For one, most PE connect to a CE which generally tend
> to be a router, not a host -- the difference here is network versus system
> administration.
>
>  Even in cases where the PE connects to an access subnet (as opposed to a
> CE router), there are scenarios where the subnet-connecting VRF is a member
> of multiple VPNs.  For example: a regional extranet DMZ subnet connected to
> a VRF which is both a member of a customer-facing regional hub-and-spoke
> VPN as well as an internal DMZ mesh.  The advantage of this approach is
> that the servers do not need to have an interface on a subnet connected to
> an internal DMZ VRF which is a member of the internal mesh VPN and yet
> another separate interface on a subnet connected to another DMZ VRF which
> is a member of the customer-facing regional hub-and-spoke VPN.  In
> addition, the servers would only need a default route as opposed to the
> complex management of static routes on the servers which can become quite
> challenging to manage at scale.
>
>  Even in a PE to CE-router scenario, there is also the concept of "shared
> VRF" model and "interface VRF" model.  In the shared VRF model multiple CE
> are connected to the same VRF and each VRF is connected to one VPN.  Here
> if a CE needs to be connected to another VPN you provision yet another
> interface to connect to the associated VRF on the PE, the CE and all the
> associated routing configuration.  OTOH, with the interface VRF model each
> VRF has only one CE connected (sometimes each interface of the CE gets it's
> own VRF on the PE).  In this model enabling an additional VPN for the CE
> only involves importing and exporting an additional RT pair at it's VRF.
>
>  In a DC/overlay setting, a particular scenario where multiple VN per VNI
> are involved would be a DMZ subnet with public addressing provided by the
> ISP.  The DMZ would need to support different CUGs with a common
> gateway/firewall to efficiently share the ISP assigned prefix.  In this
> scenario, it is very common that "private vlans" are used (informational
> RFC 5517).  This would need to be implemented with at least two VNs per VNI
> -- a hub-and-spoke VN with the gateway/firewall as the hub and multiple
> "community" VNs where each community VN represents a CUG wherein members
> need to communicate with one another.
>
>  Furthermore, in an enterprise setting it is generally expected that the
> network bears the network-related complexity.  I suppose in outfits where
> the same folks deal with both sides (network and host) this may not matter
> -- but my understanding is that these folks are a minority.
>
>
>
>  Best -- aldrin
>
> While the current draft focuses on L2VNs and L3VNs as distinct services,
> no assumptions have been made about the ability to support IRB.
> A VN is represented by a VNI created within NVEs that participate in this
> VN. Hence there is a one-to-one mapping between a VN and VNI.
>
> Hub-and-spoke is discussed briefly in this draft. More details are
> provided in draft-bl-nvo3-dataplane-requirements.
>
> >
> > Best -- aldrin
> >
> >

---

From: Thomas Narten <[email-masked]>
Date: Wed, 11 Jul 2012 08:36:44 -0400

Hi Aldrin.

> perspective of NVO3 is through the lens of VLAN/VXLAN/NVGRE/VNET/OTV
> where the VNID has global/network/domain-wide scope and is used
> as-is on the wire, then your questions are quite reasonable as they
> are based on the limitations of these solutions.  However if the
> perspective is through the lens of IPVPN and E-VPN then the
> restrictions of one VN-per-VNI and one VNI-per-VN-per-NVE are not
> reasonable since IPVPN/EVPN inherently have support for multiple
> VN-per-VNI and multiple VNI-per-VN-per-NVE.

Let me respin the above a bit differently. If you approach the problem
(as I did) from a DC centric view, where the basic connectivity model
is all nodes on a specific VN have connectivity to other nodes on the
VN, then the idea of some VNs not being able to communicate with other
VNs (by policy) doesn't seem like a particularly important feature to
have. Or, if useful, whether that can just be done outside of NVO3 or
whether it needs to be part of the NVO3 framework.

If you approach the problem from a more traditional VPN perspective
(e.g., BGP VPNs), and one that is heavily influenced by the notion
that VPNS are operating over point-to-point WAN links, existing VPN
solutions have a lot of features/functionality that was developed
(presumably) in response to requirements from that environment.

What I think may be happening here is an assumption that some (or all)
of those existing BGP VPN features apply and/or are also required in a
DC environment.  I don't think we should do that. For each feature, we
need to make the case that is useful and a requirement for NVO3, not
just assume that NVO3 inherits all requirements from existing VPN
solutions. We should not assume that DC based overlays have the same
requirements as WAN-oriented VPN solutions.

> > First, you discuss multiple VNIs on the same NVE connected to the
> > same VN.  I did not see this illustrated in your diagram, so I
> > didn't have questions about it.  From your description, you seem to
> > be making an assumption that there is some kind of firewall policy
> > associated with the VNI.  I don't see the benefit of applying
> > firewall policies to VNIs.  It seems more straight forward to apply
> > them to the VAPs.

> 
> This is indeed in my illustration.  I'm resending it.  If you look
> at NVE1 and NVE2 you will see that the 2 L2VNI on each of these NVE
> share a "DC<n> mesh" L2VN.  The illustration depicts both multiple
> VN per VNI as well as multiple VNI on a single NVE being members of
> the same VN.

I'm having a problem understanding the terminology being used here.

>From the framework doc:

        VN: Virtual Network. This is a virtual L2 or L3 domain that belongs 
        a tenant.
	
        VNI: Virtual Network Instance. This is one instance of a virtual 
        overlay network. Two Virtual Networks are isolated from one another 
        and may use overlapping addresses. 

A VNI is one specific VN instance.

I do not understand how one can have multiple VNIs associated with a
single VN, it sort of contradicts the defintion of a VNI.

What do you mean when you use the term VNI?

> Regarding features such as filtering implemented at the forwarding
> table (such as VACL) -- L2 ACL on standard dot1q switches are
> usually implemented with VACL not per-port ACL.  But to be clear,
> VACL is only one possible example of a feature that can be
> implemented at the forwarding table.  Another thing that might be
> implemented at the forwarding table level might be features like
> PVLAN (although I prefer that pvlan solution be implemented using
> overlapping VN model as shown in my illustration).

Without having a discussion about requirements, we should not (IMO)
assume that an NVE needs to provide the equivalent of VACLs. Or if it
does, how that directly impacts the NVO3 framework. ACLs can (of
course) be implemented on an NVE. Whether that (or many other
features) needs to be visible to and a part of the NVE framework is
unclear to me. It might be, but the case needs to be made first.

Thomas

---

From: Randy Bush <[email-masked]>
Date: Thu, 12 Jul 2012 19:57:51 +0900

> whether it needs to be part of the NVO3 framework.

colocation center vs data center?

randy

---

From: Aldrin Isaac <[email-masked]>
Date: Fri, 13 Jul 2012 00:20:19 -0400

Hi Thomas, 

> whether it needs to be part of the NVO3 framework.

Larry points out that VXLAN (a "one-VN-per-VNI" tunneling protocol) cannot support PVLAN which is quite common in data centers.  As you are also alluding to, Larry points out that the function of restricting connectivity between members of such a VN might belong to a firewall function rather than to NVO3 solutions.  What he suggests is possible if you push the firewall function to all VAP/VIF.  Interestingly we see this already happening with openflow and openvswitch which I suppose would be considered an NVE.  I happen to know other commercial solutions that can do this as well.  Aside from the arp issue, one could technically have one single giant subnet per tenant with edge virtual firewalls not only enforcing intended network level reachability (CUGs) but go further and enforce application-level reachability.  Sounds like the role of the NVO3 would boil down to only dealing with the overlapping address space issue between endpoints with firewalls doing the detail work.  Enterprise clouds could just use one giant subnet with edge virtual firewalls, which might indeed be better than multiple smaller VNs connected by gateways or multiple interfaces on endpoints.  This might very well be how it would be done in the simple one VN-per-VNI world.

> requirements as WAN-oriented VPN solutions.

I suppose in DC, as I point out above, we could use firewalls to do what we do with VPNs in the WAN -- certainly for where we need PVLAN.

> What do you mean when you use the term VNI?

A VNI is a table populated with match-action rules.  Protocols such openflow would be used to add/remove/modify these rules.  A Virtual Network (VN) allows a group of endpoints (VM, not NVE) to reach one another.  The VN is created by match-forward rules in the VNI to which those endpoints are attached.  Some might call this set of endpoints a CUG.  This is my view of VN and VNI.  

Best -- aldrin