Subject: Re: [nvo3] Support for multi-homed NVEs

From: Linda Dunbar <[email-masked]>
Date: Tue, 4 Sep 2012 16:08:33 +0000

My comments are inserted below:

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Ivan Pepelnjak
> Sent: Friday, August 31, 2012 2:23 AM
> To: 'Somesh Gupta'; [email-masked]
> Subject: Re: [nvo3] Support for multi-homed NVEs
> 
> This is definitely an interesting can of worms ;)

[Linda] Strongly agree. The WG should decide if this scenario should be included in the scope. 

> 
> While I don't think we should go down the path of IP-A/IP-B networks
> similar to some other DC technology, we will face the reality of some
> NVE elements (hypervisor soft switches) not being underlay IP routers.
> 
> We could either:
> 
> (A) ignore the issue and expect the network designer to solve it using
> any one of the existing NIC teaming/MLAG kludges while retaining a
> single encapsulation IP address per NVE;

[Linda] When NIC teaming/MLAG is used, you are assuming that the external switch is the two NICs on the server are connected by Layer 2, is it correct? If NVE is at the NIC, how to use NIC teaming or MLAG? 

> 
> (B) provide support for multiple encapsulation addresses per NVE so a
> multi-homed NVE could have one IP address per physical interface and
> send and receive nvo3-encapsulated frames using more than one address.

[Linda] Then the remote NVE will have two possible (IP) addresses to send data back. Who is responsible for balancing the inbound traffic? 

> 
> Option (A) is the easy way out similar to existing MPLS/VPN behavior
> and would fit well with existing DC deployments. It would also retain
> all the server-to-ToR multihoming complexity.

[Linda] I am not sure if Option (A) is simpler. 

> 
> Option (B) would reduce the complexity of the underlay DC network
> (which would become a simple L3 network with single-homed IP addresses),
> but we'd have to deal with a bunch of additional problems (peer IP
> address liveliness check).
> 
> Just speculating ...
> Ivan
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> Of
> > Somesh Gupta
> > Sent: Friday, August 31, 2012 6:58 AM
> > To: [email-masked]
> > Subject: [nvo3] Support for multi-homed NVEs
> >
> > I did not see any mention of multi-homed NVEs in draft-lasserre-nvo3-
> > framework-03.txt. NVEs are connected together by an L3 network - does
> that
> > mean only one?
> > Can it be multi-homes to two L3 networks?
> >
> > Somesh
> > _______________________________________________
> > nvo3 mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/nvo3
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 04 Sep 2012 18:25:20 +0200

>> While I don't think we should go down the path of IP-A/IP-B
>> networks similar to some other DC technology, we will face the
>> reality of some NVE elements (hypervisor soft switches) not being
>> underlay IP routers.
>>
>> We could either:
>>
>> (A) ignore the issue and expect the network designer to solve it
>> using any one of the existing NIC teaming/MLAG kludges while
>> retaining a single encapsulation IP address per NVE;
>
> [Linda] When NIC teaming/MLAG is used, you are assuming that the
> external switch is the two NICs on the server are connected by Layer
> 2, is it correct? If NVE is at the NIC, how to use NIC teaming or
> MLAG?

Did you actually mean NVE at the NIC, not hypervisor? Why would anyone 
do that?

>> (B) provide support for multiple encapsulation addresses per NVE so
>> a multi-homed NVE could have one IP address per physical interface
>> and send and receive nvo3-encapsulated frames using more than one
>> address.
>
> [Linda] Then the remote NVE will have two possible (IP) addresses to
> send data back. Who is responsible for balancing the inbound
> traffic?

Remote NVE, who else?

Ivan

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 4 Sep 2012 16:28:51 +0000

Ivan, 

See in line:

> From: Ivan Pepelnjak [mailto:[email-masked]]
> Sent: Tuesday, September 04, 2012 11:25 AM
> To: Linda Dunbar
> Cc: 'Somesh Gupta'; [email-masked]

[Linda] You could have NVE per NIC (I am not saying that anyone would). So if you have NVE at hypervisor, then the NIC teaming becoming IP layer NIC teaming. Do you know if any vendor support it? 

Most NIC teaming I've seen are at the MAC Layer.

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 04 Sep 2012 18:36:46 +0200

Every Linux distro for the last 10+ years?

http://www.kernel.org/doc/Documentation/networking/bonding.txt

Not sure what you mean by IP-layer NIC teaming, what we need is a single 
NVE IP address reachable through two NICs. That's a well-known and 
well-supported solution.

On 9/4/12 6:28 PM, Linda Dunbar wrote:
> So if you have NVE at hypervisor, then the NIC teaming becoming IP
> layer NIC teaming. Do you know if any vendor support it?
>

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Tue, 4 Sep 2012 11:45:32 -0500

I understand the discussion below is about the NVE multi-homing towards the IP core, on the tunnel side. 

We did not focus in the framework draft on the core redundancy as in our opinion there was no need to standardize anything here. There are no differences from what is available today in regular IP networks: if NVEs are multi-homed directly to the next IP router, regular routing will take care of it. If there is Ethernet switching in between NVE and the next IP hop, L2 resiliency mechanisms need to be employed. From what I read below it looks more of an implementation discussion than a standardization requirement. Am I right?

By Multi-homed NVEs one can also understand a set of NVEs multi-homed on the access side to other devices. That is a discussion we need to have in my opinion. An use case example: NVO3 network - NVE GWs multi-homed to external non-NVO3 networks. Handoff can be VLANs, VPLS PWs, or BGP EVPN labels... 

I think the latter is worth discussing although there are some mechanisms and some standardization initiatives in place already.

> Sent: Friday, August 31, 2012 12:23 AM
> (which would become a simple L3 network with single-homed IP
> addresses), but we'd have to deal with a bunch of additional problems
> (peer IP address liveliness check).
> 
> Just speculating ...
> Ivan
> 
> > Of Somesh Gupta
> > that mean only one?

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 04 Sep 2012 18:57:41 +0200

Ah, that other can of worms ;) Mine was simpler.

On the underlay side, we might decide that NVEs have a single IP address 
or multiple IP addresses (like some NVGRE load balancing proposals). If 
we decide NVEs have a single IP address (potential per virtual network 
segment), then the rest is implementation details (and we're back to 
MLAG/SMLT land for true redundancy). Alternatively we might implement 
the option of having multiple IP addresses per NVE, and the NVEs might 
use the IP-address-per-link option (thus no need for L2 or MLAG at all).

On the overlay side, the real problem (as you stated) is the 
multi-homing of NVO3-to-legacy gateways. I don't see any other need for 
overlay NVE multihoming.

BTW, Nicira has nicely solved the NVO3 gateway multihoming - the whole 
NVO3 network works exactly like VMware's vSwitch: split horizon bridging 
(thus no forwarding loops through NVO3), with every VM MAC address being 
dynamically assigned to one of the gateways, which also solves the 
return path issues (dynamic MAC learning in legacy network takes care of 
that). Maybe we should just use the wheel that has already been invented?

Kind regards,
Ivan

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Tue, 4 Sep 2012 12:31:29 -0500

We need to talk about it: the wheel you refer to below might not work on all the roads... :) 

For example when you go to external non-NVO3 domains you do not know all the external MAC/IP addresses so you can't program them from a centralized controller in the egress direction from NVO3. Also the non-NVO3 gateways are not controlled by the NVO3 control plane so you can't program their FIBs from the controller either. Broadcast and Multicast handling in general might require special consideration as well.
 
A number of mechanisms can be considered to address the external network use case - see BGP MH for VPLS and/or BGP EVPN work in IETF ... 
It looks like we need to add access multi-homing in the framework and follow up on use cases in data and control plane requirements so that NVO3 can do a proper solution gap analysis at the end.  

> Sent: Tuesday, September 04, 2012 9:58 AM
> To: Balus, Florin Stelian (Florin)
> On the underlay side, we might decide that NVEs have a single IP
> address or multiple IP addresses (like some NVGRE load balancing
> proposals). If we decide NVEs have a single IP address (potential per
> virtual network segment), then the rest is implementation details (and
> we're back to MLAG/SMLT land for true redundancy). Alternatively we
> might implement the option of having multiple IP addresses per NVE, and
> the NVEs might use the IP-address-per-link option (thus no need for L2
> or MLAG at all).
> 
> On the overlay side, the real problem (as you stated) is the multi-
> homing of NVO3-to-legacy gateways. I don't see any other need for
> overlay NVE multihoming.
> 
> NVO3 network works exactly like VMware's vSwitch: split horizon
> bridging (thus no forwarding loops through NVO3), with every VM MAC
> address being dynamically assigned to one of the gateways, which also
> solves the return path issues (dynamic MAC learning in legacy network
> takes care of that). Maybe we should just use the wheel that has
> already been invented?
> 
> Kind regards,
> Ivan
> 
> > We did not focus in the framework draft on the core redundancy as in
> our opinion there was no need to standardize anything here. There are
> no differences from what is available today in regular IP networks: if
> NVEs are multi-homed directly to the next IP router, regular routing
> will take care of it. If there is Ethernet switching in between NVE and
> the next IP hop, L2 resiliency mechanisms need to be employed. From
> what I read below it looks more of an implementation discussion than a
> standardization requirement. Am I right?
> >
> > By Multi-homed NVEs one can also understand a set of NVEs multi-homed
> on the access side to other devices. That is a discussion we need to
> have in my opinion. An use case example: NVO3 network - NVE GWs multi-
> homed to external non-NVO3 networks. Handoff can be VLANs, VPLS PWs, or
> BGP EVPN labels...
> >
> > I think the latter is worth discussing although there are some
> mechanisms and some standardization initiatives in place already.
> >
> >
> >> Of Ivan Pepelnjak
> a
> >>> I did not see any mention of multi-homed NVEs in
> >>> draft-lasserre-nvo3- framework-03.txt. NVEs are connected together
> >>> by an L3 network - does that mean only one?

---

From: Somesh Gupta <[email-masked]>
Date: Tue, 4 Sep 2012 10:50:32 -0700

Florin,

Regarding the multi-homing, my assumption is that the
NVE in the hypervisor would not (want to) run a routing
protocol. So as Ivan points out, the standard would
need to accommodate multiple IP addresses per NVE.

Somesh

> Cc: Somesh Gupta; [email-masked]
> address
> bridging
> using

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Tue, 4 Sep 2012 13:00:05 -0500

AFAIK the text in NVO3 framework and requirements drafts allows multiple IPs per NVE. We have multiple IPs per PE in current VPN implementations. This is an implementation matter though in my opinion.

> From: Somesh Gupta [mailto:[email-masked]]
> Sent: Tuesday, September 04, 2012 10:51 AM
> To: Ivan Pepelnjak; Balus, Florin Stelian (Florin)
> Cc: [email-masked]
> Subject: RE: [nvo3] Support for multi-homed NVEs
> 
> Florin,
> 
> Regarding the multi-homing, my assumption is that the NVE in the
> hypervisor would not (want to) run a routing protocol. So as Ivan
> points out, the standard would need to accommodate multiple IP
> addresses per NVE.
> 
> Somesh
> 
> > might implement the option of having multiple IP addresses per NVE,
> > and the NVEs might use the IP-address-per-link option (thus no need
> > for L2 or MLAG at all).
> >
> > multi-homing of NVO3-to-legacy gateways. I don't see any other need
> > for overlay NVE multihoming.
> >
> > will take care of it. If there is Ethernet switching in between NVE
> > and the next IP hop, L2 resiliency mechanisms need to be employed.
> > From what I read below it looks more of an implementation discussion
> > than a standardization requirement. Am I right?
> > >
> > homed to external non-NVO3 networks. Handoff can be VLANs, VPLS PWs,
> > or BGP EVPN labels...
> > >
> > >> reality of
> > some
> > >> multi-homed NVE could have one IP address per physical interface
> > >> Option (A) is the easy way out similar to existing MPLS/VPN
> > >> behavior and would fit well with existing DC deployments. It would
> > >> also
> > retain

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 04 Sep 2012 21:02:41 +0200

A) Not per prefix (at least not without serious amount of end-to-end BGP 
multipathing+BGP AddPath or multiple RDs)

B) There's no liveliness check in *VPN (apart from LISP)

In MPLS/VPN we rely on the host route to PE loopback and/or LSP to the 
/32 prefix to indicate next hop validity. That won't work for 
hypervisor-based NVEs

---

From: Ivan Pepelnjak <[email-masked]>
Date: Tue, 04 Sep 2012 21:10:56 +0200

Here's (very approximately) one way to make it work (assuming L2 S+D 
matching in hypervisor vSwitch, with multiple entries resulting in 
packet replication). Pretty easy to implement with OpenFlow (more so if 
you have OF 1.1+ with multiple lookup tables).

A) for every known destination MAC, install (*,D) entry pointing to 
destination NVE IP address
B) for multicast MAC addresses, install (*,MC) entries pointing to all 
NVEs in the same VN
C) for every source MAC (= VM), select one of the NVO3 GWs and install 
(S,*) entry pointing to that GW.

No need to know external MAC or IP addresses; just use the 
source-MAC-based "default route" toward one of the NVO3 gateways.

Nicira does this on L2, Midokura does it on L3. What's wrong with the 
shape of this wheel? ;)

Ivan

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Tue, 4 Sep 2012 16:46:22 -0500

Ivan,
See in-line...  

> Sent: Tuesday, September 04, 2012 12:03 PM
> A) Not per prefix (at least not without serious amount of end-to-end
> BGP multipathing+BGP AddPath or multiple RDs)
> 
[FB>] How are the VPN specifications (IP VPN, VPLS, VPWS or incoming EVPN) connected to LISP? Also can you be more specific on the absence of liveliness check in *VPN. Are you talking about a certain deployment/implementation environment?
Thanks,
Florin

> > AFAIK the text in NVO3 framework and requirements drafts allows
> multiple IPs per NVE. We have multiple IPs per PE in current VPN
> implementations. This is an implementation matter though in my opinion.
> >
> >>> proposals). If we decide NVEs have a single IP address (potential
> >>> per virtual network segment), then the rest is implementation
> >>> details
> >> (and
> >>> address being dynamically assigned to one of the gateways, which
> >>> also solves the return path issues (dynamic MAC learning in legacy
> >>> network takes care of that). Maybe we should just use the wheel
> that
> >>> has already been invented?
> >>>
> >>> Kind regards,
> >>> Ivan
> >>>
> >>> our opinion there was no need to standardize anything here. There
> >>> are no differences from what is available today in regular IP
> networks:
> >> if
> >>>  From what I read below it looks more of an implementation
> >>> discussion than a standardization requirement. Am I right?

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Tue, 4 Sep 2012 18:39:22 -0500

Ivan,
See in-line...

> Sent: Tuesday, September 04, 2012 12:11 PM
> Cc: [email-masked]; 'Somesh Gupta'
[FB>] Before jumping in the solution space, for NVE resiliency to an external network (NVE1, NVE2 gateways connected on their access side to GW1, GW2 in an external domain) it might be useful to have a look at previous IETF work on this - see http://datatracker.ietf.org/wg/l2vpn/, BGP MH and EVPN drafts.

> destination NVE IP address

[FB>] You are just talking about the ingress NVE... must consider also the L2 forwarding in the egress NVE towards the external network (say NVE1,2->GW1,2). Then the other direction GW1/GW2 -> NVE1/NVE2.

> B) for multicast MAC addresses, install (*,MC) entries pointing to all

[FB>] This is not enough - if packets are flooded to NVE1 and NVE2 in my example, duplication/loops may occur - see the inclusive mcast and split-horizon procedures described in EVPN draft. 

> No need to know external MAC or IP addresses; just use the source-MAC-
> based "default route" toward one of the NVO3 gateways.
> 

[FB>] difficult to discuss a solution based on email text... if you feel strongly about it you/them should submit a NVO3 proposal on this gateway resiliency... I believe though there is more to it than what you listed as A,B,C. 

Florin

> > For example when you go to external non-NVO3 domains you do not know
> all the external MAC/IP addresses so you can't program them from a
> centralized controller in the egress direction from NVO3. Also the non-
> NVO3 gateways are not controlled by the NVO3 control plane so you can't
> program their FIBs from the controller either. Broadcast and Multicast
> handling in general might require special consideration as well.
> >
> > A number of mechanisms can be considered to address the external
> network use case - see BGP MH for VPLS and/or BGP EVPN work in IETF ...
> > It looks like we need to add access multi-homing in the framework and
> follow up on use cases in data and control plane requirements so that
> NVO3 can do a proper solution gap analysis at the end.
> >
> per
> >> virtual network segment), then the rest is implementation details
> >> (and we're back to MLAG/SMLT land for true redundancy).
> Alternatively
> >> we might implement the option of having multiple IP addresses per
> >> NVE, and the NVEs might use the IP-address-per-link option (thus no
> >> need for L2 or MLAG at all).
> >>
> are
> >> no differences from what is available today in regular IP networks:
> >> if NVEs are multi-homed directly to the next IP router, regular
> >> routing will take care of it. If there is Ethernet switching in
> >> between NVE and the next IP hop, L2 resiliency mechanisms need to be
> >> employed. From what I read below it looks more of an implementation
> >> have in my opinion. An use case example: NVO3 network - NVE GWs
> >> multi- homed to external non-NVO3 networks. Handoff can be VLANs,
> >> VPLS PWs, or BGP EVPN labels...
> >>>> From: [email-masked] [mailto:[email-masked]] On
> >>>> Behalf Of Ivan Pepelnjak

---

From: Somesh Gupta <[email-masked]>
Date: Tue, 4 Sep 2012 16:44:33 -0700

I think multiple IPs per NVE will have to show up in the
solution - we can wait and see.

> From: Balus, Florin Stelian (Florin) [mailto:florin.balus@alcatel-
> lucent.com]
> Sent: Tuesday, September 04, 2012 11:00 AM
> To: Somesh Gupta; Ivan Pepelnjak
> Cc: [email-masked]
> discussion
> > multi-
> > > >> Behalf
> > > Behalf
> > > >>> I did not see any mention of multi-homed NVEs in draft-
> lasserre-
> > > nvo3-
> > > >>> framework-03.txt. NVEs are connected together by an L3 network
> -
> > > does
> > > >>> that mean only one?

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 5 Sep 2012 00:02:23 +0000

Agreed. It may be a practical matter for entropy over older devices (depending on the encap) which is likely a requirement.

Peter

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 5 Sep 2012 02:03:39 +0000

Totally agree. 

For those DCVPN approaches which use BGP as a control plane protocol (e.g., BGP/MPLS IP VPN), the mechanism defined in this draft ( http://tools.ietf.org/html/draft-xu-idr-tunnel-address-prefix-01) can be used to advertise multiple IP addresses per NVE/PE automatically. For those DCVPN approaches which use IS-IS as a control plane protocol, an IS-IS TLV extension can be used to advertise multiple IP addresses per NVE/PE as well, as has been mentioned in a previous email (http://www.ietf.org/mail-archive/web/nvo3/current/msg01346.html).

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> Somesh Gupta
> 发送时间: 2012年9月5日 7:45
> 收件人: Balus, Florin Stelian (Florin); Ivan Pepelnjak
> 抄送: [email-masked]
> 主题: Re: [nvo3] Support for multi-homed NVEs
>

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 5 Sep 2012 03:40:56 +0000

> 发送时间: 2012年9月5日 1:51
> 收件人: Ivan Pepelnjak; Balus, Florin Stelian (Florin)
> 抄送: [email-masked]

If a given multi-homed NVE/PE (e.g., a physical server) doesn't run a routing protocol with the P routers, it seems that different NICs could act as different virtual NVEs/PEs and VMs of a given VPN instance are therefore multi-homed to these virtual NVEs/PEs in either active-standby mode or in active-active mode. 

Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 5 Sep 2012 03:54:15 +0000

Sorry. Please neglect the following email since it just talks about how to facilitate the load-balancing of DCVPN traffic in the core by using multiple tunnel destination addresses in the case where NVEs/PEs run some routing protocol with P routers.

Xiaohu

> Xuxiaohu
> 发送时间: 2012年9月5日 10:04
> 收件人: Somesh Gupta; Balus, Florin Stelian (Florin); Ivan Pepelnjak
> 抄送: [email-masked]
> For those DCVPN approaches which use BGP as a control plane protocol (e.g.,
> BGP/MPLS IP VPN), the mechanism defined in this draft
> ( http://tools.ietf.org/html/draft-xu-idr-tunnel-address-prefix-01) can be used
> to advertise multiple IP addresses per NVE/PE automatically. For those DCVPN
> approaches which use IS-IS as a control plane protocol, an IS-IS TLV extension
> can be used to advertise multiple IP addresses per NVE/PE as well, as has been
> mentioned in a previous email
> (http://www.ietf.org/mail-archive/web/nvo3/current/msg01346.html).
> 
> Best regards,
> Xiaohu
> 
> > -----邮件原件-----

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Thu, 6 Sep 2012 20:46:29 +0200

Florin,

All the VPNs you've mentioned share the data-plane principles with LISP (that's why I mentioned them in the same sentence): IP packet is encapsulated into an envelope with a known destination endpoint (PE-router, ETR), and sent on its way. The big question is "is the destination endpoint reachable?"

In most MPLS-based implementations we assume the endpoint is reachable if we're getting VPN routing/signalling updates from it and if it's reachable through the IP routing table. Some implementations might be more cautious and use GRE encapsulation if there's no LSP to the endpoint, others might use MPLS OAM. Usually these techniques are glossed over and left as implementation details.

On the other hand, LISP defines several mechanisms that can be used to check the ETR liveliness.

Coming back to nvo3, we'll have serious problems with NVE in a hypervisor using more than one underlay IP address, more so if its control-plane session uses only one of them. We'll never know whether the other IP addresses are reachable (the problem becomes worse if you have a DC transport infrastructure that's a mixture of L2 and L3).

In situation where NVE has more than one IP address used by nvo3, we need (in my opinion) something that checks the liveliness of the remote NVE IP address ... and it's not an implementation detail, it's a mandatory requirement.

Hope this makes more sense
Ivan

> Sent: Tuesday, September 04, 2012 11:46 PM
> To: Ivan Pepelnjak
> [FB>] How are the VPN specifications (IP VPN, VPLS, VPWS or incoming EVPN)
> connected to LISP? Also can you be more specific on the absence of
> liveliness check in *VPN. Are you talking about a certain
> deployment/implementation environment?
> Thanks,
> Florin
> 
> >
> > >>> we're back to MLAG/SMLT land for true redundancy). Alternatively
> > >>> NVE, and the NVEs might use the IP-address-per-link option (thus
> > >>> no need for L2 or MLAG at all).
> > >>>
> > >>> multi-homing of NVO3-to-legacy gateways. I don't see any other
> > >>> need for overlay NVE multihoming.
> > >>>
> > >>> bridging (thus no forwarding loops through NVO3), with every VM
> > >>> MAC address being dynamically assigned to one of the gateways,
> > >>> which also solves the return path issues (dynamic MAC learning in
> > >>> legacy network takes care of that). Maybe we should just use the
> > >>> wheel
> > that
> > >>> will take care of it. If there is Ethernet switching in between
> > >>> NVE and the next IP hop, L2 resiliency mechanisms need to be
> employed.
> > >>>>> Option (B) would reduce the complexity of the underlay DC
> > >>>>> network (which would become a simple L3 network with
> > >>>>> single-homed IP addresses), but we'd have to deal with a bunch
> > >>>>> of additional
> > >>> problems
> > >>>>>> draft-lasserre-

---

From: AshwoodsmithPeter <[email-masked]>
Date: Thu, 6 Sep 2012 18:51:11 +0000

Ivan that is a good point. If you like we can capture that in the oam draft.

Peter

---

From: Lucy yong <[email-masked]>
Date: Thu, 6 Sep 2012 19:51:40 +0000

Hi Ivan,

Snip..

On the other hand, LISP defines several mechanisms that can be used to check the ETR liveliness.
[[LY]] LISP provides a control plane function. If an ETR is not live, it needs to update all the end systems attaching to that ETR as "deleted". So no packet should be forwarded to the end systems behind ETR.

Coming back to nvo3, we'll have serious problems with NVE in a hypervisor using more than one underlay IP address, more so if its control-plane session uses only one of them. We'll never know whether the other IP addresses are reachable (the problem becomes worse if you have a DC transport infrastructure that's a mixture of L2 and L3).
[[LY]] IMO: an overly network has to "believe" that underlying network is able to reach the remote NVE. Without such "believe", the architecture will not work. ETR live does not mean the intermediate nodes or links in underlying network being live. Overlay tunnel is to rely on the underlying network transport. Should underlying network report its transport status to an overlay? This is an very old problem in the layered transport network. It is option, not required because of the scalability.

In situation where NVE has more than one IP address used by nvo3, we need (in my opinion) something that checks the liveliness of the remote NVE IP address ... and it's not an implementation detail, it's a mandatory requirement.
[[LY]] I think this is different aspect. If NVE is dead, neither IP address can help to reach the end system behind it. In fact, a sender should not send a packet to a dead NVE.

Lucy

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 6 Sep 2012 15:46:54 -0700

> [[LY]] LISP provides a control plane function. If an ETR is not live, it needs to update all the end systems attaching to that ETR as "deleted". So no packet should be forwarded to the end systems…

There is a LISP mechanism to solve this problem already and it is not the way you describe. By just looking at one technique, called RLOC-probing, the ITRs can determine when one or more ETRs in their locator-set have gone unreachable, and when that happens, the ITR will encapsulate to the ETRs (from the locator-set) that are up and reachable (avoiding the down ETRs). 

When the overlay is deployed in the network, you have better opportunities for multi-homing, therefore you have more resiliency to failure.

Dino

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 6 Sep 2012 15:49:34 -0700

Ivan,

> Florin,
> 
> All the VPNs you've mentioned share the data-plane principles with
> LISP (that's why I mentioned them in the same sentence): IP packet
> is encapsulated into an envelope with a known destination endpoint
> (PE-router, ETR), and sent on its way. The big question is "is the
> destination endpoint reachable?"
> 
> In most MPLS-based implementations we assume the endpoint is reachable
> if we're getting VPN routing/signalling updates from it and if
> it's reachable through the IP routing table. Some implementations
> might be more cautious and use GRE encapsulation if there's no LSP
> to the endpoint, others might use MPLS OAM.  Usually these techniques
> are glossed over and left as implementation details.

In 2547 VPNs (or E-VPNs) it is the PE that originates routes to the
VPN sites connected to that PE. So, if the PE goes down, the routes
get withdrawn. This is "vanilla" BGP, and as such has be supported
by any non-broken BGP implementation. In addition, there are several
enhancements to speed up connectivity restoration after egress PE
failure, such as Next-Hop tracking, egress PE protection, etc...
that have been implemented by vendors.

> to check the ETR liveliness.

Yes, except that all these LISP mechanisms are broken in one way or another.

> Coming back to nvo3, we'll have serious problems with NVE in a
> hypervisor using more than one underlay IP address, more so if its
> control-plane session uses only one of them. We'll never know
> whether the other IP addresses are reachable (the problem becomes
> worse if you have a DC transport infrastructure that 's a mixture
> of L2 and L3).

> In situation where NVE has more than one IP address used by nvo3,
> we need (in my opinion) something that checks the liveliness of
> the remote NVE IP address ... and it's not an implementation detail,
> it's a mandatory requirement.

"check the liveness" is a bit underspecified, as it does not
say what should be the upper bound on the detection time. 
While discussing the upper bound, we should probably keep in
mind that the goal here is to minimize connectivity disruption
after a failure. Handling connectivity diruption could be
done via either global or local repair techniques (both of these
have been employed in 2547 VPNs to deal with PEs failures).

Yakov.

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 7 Sep 2012 01:26:58 +0000

If the DCVPN solution has a control plane for VPN membership auto-discovery, the concern of whether the destination endpoint (i.e., egress PE or ETR) is reachable can be addressed easily. Of course, if you need fast convergence, you can use the mechanism such as next-hop tracking, just as what Yakov has mentioned before. In addition, in the case where the egress PE is still reachable but its connectivity to the multi-homed CE is lost, if there is a control plane for MAC withdrawal or IP route withdrawal, there would not be any black-holing issue. Anyway, any DCVPN solution which lacks such a control plane seems not a good choice, especially in the multi-homing scenario.

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Ivan
> Pepelnjak
> 发送时间: 2012年9月7日 2:46
> 收件人: 'Balus, Florin Stelian (Florin)'
> 抄送: [email-masked]; 'Somesh Gupta'
> All the VPNs you've mentioned share the data-plane principles with LISP (that's
> why I mentioned them in the same sentence): IP packet is encapsulated into an
> envelope with a known destination endpoint (PE-router, ETR), and sent on its
> way. The big question is "is the destination endpoint reachable?"
> 
> In most MPLS-based implementations we assume the endpoint is reachable if
> we're getting VPN routing/signalling updates from it and if it's reachable
> through the IP routing table. Some implementations might be more cautious
> and use GRE encapsulation if there's no LSP to the endpoint, others might use
> MPLS OAM. Usually these techniques are glossed over and left as
> implementation details.
> 
> Coming back to nvo3, we'll have serious problems with NVE in a hypervisor
> using more than one underlay IP address, more so if its control-plane session
> uses only one of them. We'll never know whether the other IP addresses are
> reachable (the problem becomes worse if you have a DC transport
> infrastructure that's a mixture of L2 and L3).
> 
> In situation where NVE has more than one IP address used by nvo3, we need (in
> my opinion) something that checks the liveliness of the remote NVE IP
> address ... and it's not an implementation detail, it's a mandatory requirement.
>

---

From: "Garcia Del Rio, Diego (Diego)" <[email-masked]>
Date: Fri, 7 Sep 2012 04:50:18 +0200

The problem here is that if the nve is in the hypervisor, we will NOT see the exact prefix (/32) in the IGP, so NH-tracking is out. The main issue I see is if that if we have multiple IPs in the HV (thus multiple NHs announced for all the prefixes/MACs from that particular HV), its hard for the remote side to see which of those NHs can be reached.

The HV / nve could do a ping or bfd check to its default gw (I guess bfd in echo mode would be the simplest option) and if the connectivity is lost, then withdraw that next-hop.

As ivan mentioned, though, having multiple next-hops for the same prefix (and let it survive an RR) we would need something like add-path or multiple RT/RDs.

DIEGO GARCÍA DEL RIO
ALCATEL-LUCENT
IPD PRODUCT MANAGEMENT
777 E. Middlefield Rd
Mountain View CA 94043
Mobile: +1 (415) 439-9420
OnNet: 2852-2726
[email-masked]

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Thursday, 6 September 2012 6:27 PM
To: Ivan Pepelnjak; Balus, Florin Stelian (Florin)
Cc: [email-masked]; 'Somesh Gupta'
Subject: Re: [nvo3] Support for multi-homed NVEs

Best regards,
Xiaohu

> LISP (that's why I mentioned them in the same sentence): IP packet is 
> encapsulated into an envelope with a known destination endpoint 
> (PE-router, ETR), and sent on its way. The big question is "is the destination endpoint reachable?"
> 
> if we're getting VPN routing/signalling updates from it and if it's 
> reachable through the IP routing table. Some implementations might be 
> more cautious and use GRE encapsulation if there's no LSP to the 
> endpoint, others might use MPLS OAM. Usually these techniques are 
> glossed over and left as implementation details.
> 
> control-plane session uses only one of them. We'll never know whether 
> the other IP addresses are reachable (the problem becomes worse if you 
> have a DC transport infrastructure that's a mixture of L2 and L3).
> 
> In situation where NVE has more than one IP address used by nvo3, we 
> need (in my opinion) something that checks the liveliness of the 
> remote NVE IP address ... and it's not an implementation detail, it's a mandatory requirement.
> 
> > > A) Not per prefix (at least not without serious amount of 
> > > end-to-end BGP multipathing+BGP AddPath or multiple RDs)
> > >
> > [FB>] How are the VPN specifications (IP VPN, VPLS, VPWS or incoming 
> > EVPN) connected to LISP? Also can you be more specific on the 
> > absence of liveliness check in *VPN. Are you talking about a certain 
> > > >> Regarding the multi-homing, my assumption is that the NVE in 
> > > >> the hypervisor would not (want to) run a routing protocol. So 
> > > >> as Ivan points out, the standard would need to accommodate 
> > > >> multiple IP addresses per NVE.
> > > >>
> > > >> Somesh
> > > >>
> > > >>> On the underlay side, we might decide that NVEs have a single 
> > > >>> IP address or multiple IP addresses (like some NVGRE load 
> > > >>> balancing proposals). If we decide NVEs have a single IP 
> > > >>> address (potential per virtual network segment), then the rest 
> > > >>> is implementation details
> > > >> (and
> > > >>> we're back to MLAG/SMLT land for true redundancy). 
> > > >>> Alternatively we might implement the option of having multiple 
> > > >>> IP addresses per NVE, and the NVEs might use the 
> > > >>> IP-address-per-link option (thus no need for L2 or MLAG at all).
> > > >>>
> > > >>> BTW, Nicira has nicely solved the NVO3 gateway multihoming - 
> > > >>> the
> > > >> whole
> > > >>> NVO3 network works exactly like VMware's vSwitch: split 
> > > >>> horizon bridging (thus no forwarding loops through NVO3), with 
> > > >>> every VM MAC address being dynamically assigned to one of the 
> > > >>> gateways, which also solves the return path issues (dynamic 
> > > >>> MAC learning in legacy network takes care of that). Maybe we 
> > > >>> should just use the wheel
> > > that
> > > >>>> I understand the discussion below is about the NVE 
> > > >>>> multi-homing
> > > >>> towards the IP core, on the tunnel side.
> > > >>>> We did not focus in the framework draft on the core 
> > > >>>> redundancy as
> > > >> in
> > > >>> our opinion there was no need to standardize anything here. 
> > > >>> There are no differences from what is available today in 
> > > >>> regular IP
> > > networks:
> > > >> if
> > > >>> will take care of it. If there is Ethernet switching in 
> > > >>> on the access side to other devices. That is a discussion we 
> > > >>> need
> > > to
> > > >>> homed to external non-NVO3 networks. Handoff can be VLANs, 
> > > >>> VPLS
> > > PWs,
> > > >>> or BGP EVPN labels...
> > > >>>>> networks similar to some other DC technology, we will face 
> > > >>>>> the reality of
> > > >>> some
> > > >>>>> NVE elements (hypervisor soft switches) not being underlay 
> > > >>>>> IP
> > > >>> routers.
> > > >>>>> We could either:
> > > >>>>>
> > > >>>>> any one of the existing NIC teaming/MLAG kludges while 
> > > >>>>> retaining
> > > a
> > > >>>>> (B) provide support for multiple encapsulation addresses per 
> > > >>>>> NVE
> > > >> so
> > > >>> a
> > > >>>>> multi-homed NVE could have one IP address per physical 
> > > >>>>> interface and send and receive nvo3-encapsulated frames 
> > > >>>>> using more than one
> > > >>> address.
> > > >>>>> single-homed IP addresses), but we'd have to deal with a 
> > > >>>>> bunch of additional
> > > >>> problems

---

From: John E Drake <[email-masked]>
Date: Fri, 7 Sep 2012 04:56:09 -0700

Xuxiaohu,

Comment inline

Yours irrespectively,

John

> Sent: Thursday, September 06, 2012 6:27 PM
> To: Ivan Pepelnjak; 'Balus, Florin Stelian (Florin)'
> If the DCVPN solution has a control plane for VPN membership auto-
> discovery, the concern of whether the destination endpoint (i.e.,
> egress PE or ETR) is reachable can be addressed easily. Of course, if
> you need fast convergence, you can use the mechanism such as next-hop
> tracking, just as what Yakov has mentioned before. In addition, in the
> case where the egress PE is still reachable but its connectivity to the
> multi-homed CE is lost, if there is a control plane for MAC withdrawal
> or IP route withdrawal, there would not be any black-holing issue.

JD:  Block MAC withdrawal in E-VPN occurs in exactly this case.  The egress PE uses the withdrawal of a single route to indicate that all MAC addresses associated with the multi-homed CE are no longer reachable via it.  This route is the Per ESI Ethernet AD route which represents the connectivity between the egress PE and the multi-homed CE and it is associated with all VPNs of which the multi-homed CE is a member.

> Anyway, any DCVPN solution which lacks such a control plane seems not a
> good choice, especially in the multi-homing scenario.
> 
> Best regards,
> Xiaohu
>

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Fri, 7 Sep 2012 16:54:15 +0200

Yakov,

The scenario that worries me is a hypervisor-based NVE with more than one transport (underlay) IP address, with each IP address tied to a NIC (or multiple bonded NICs), and potentially being in a different IP subnet (because it's connected to a different ToR switch). I'm further assuming that the hypervisor is not running a routing protocol and thus behaves like an IP host locally multihomed to multiple logical networks.

Assuming we use something like MPLS/VPN for NVO3, the BGP sessions from NVE to its BGP neighbors (example: a set of route reflectors) would run from one of the underlay IP addresses. If another underlay IP address becomes unreachable, we cannot use BGP session drop to detect that.

Furthermore, we won't have /32 routes to BGP next hops any more (that would not scale to the several million endpoints from the NVO3 charter). The first L3 switch to which the server is connected (a ToR switch or a spine switch) will originate an IP prefix ==> we cannot use NH tracking.

Finally, the server might not experience a NIC link loss if it loses connection to first-hop IP gateway. Example: blade enclosure switch might be a L2-only switch ==> we cannot expect the hypervisor-based NVE to revoke prefixes or change BGP next hop if its transport address becomes unreachable.

Using BFD from the NVE to detect first-hop gateway reachability and using that information to set BGP next hops in BGP prefix origination process (as proposed by Diego Garcia Del Rio) sounds like the simplest option.

Does this make sense?
Ivan

> From: Yakov Rekhter [mailto:[email-masked]]
> Sent: Friday, September 07, 2012 12:50 AM
> To: Ivan Pepelnjak
> Cc: 'Balus, Florin Stelian (Florin)'; [email-masked]; 'Somesh Gupta';
> [email-masked]
> In 2547 VPNs (or E-VPNs) it is the PE that originates routes to the VPN
> sites connected to that PE. So, if the PE goes down, the routes get
> withdrawn. This is "vanilla" BGP, and as such has be supported by any non-
> broken BGP implementation. In addition, there are several enhancements to
> speed up connectivity restoration after egress PE failure, such as Next-
> Hop tracking, egress PE protection, etc...
> > have a DC transport infrastructure that 's a mixture of L2 and L3).
> 
> "check the liveness" is a bit underspecified, as it does not say what
> should be the upper bound on the detection time.
> While discussing the upper bound, we should probably keep in mind that the
> goal here is to minimize connectivity disruption after a failure. Handling
> connectivity diruption could be done via either global or local repair
> techniques (both of these have been employed in 2547 VPNs to deal with PEs
> failures).
> 
> Yakov.
> >

---

From: <[email-masked]>
Date: Fri, 7 Sep 2012 15:34:59 +0000

Hi Ivan,

> Furthermore, we won't have /32 routes to BGP next hops any more (that would not scale to the several million endpoints from the NVO3 charter). The first L3 switch to which the server is connected (…

The scale at which the underlay routing would need to scale for NH 
tracking to be usable, is the number of NVEs.
AFAIK, the target scale for the number of NVE is not millions, but 
rather multiples of 10ks, up to 100ks.
So, I would tend to agree with you that this kind of range would rule 
out IGP-based NH tracking, but the IGP is not the only tool we have to 
scale routing, is it ?

-Thomas

>>>>>>>>>>> Can it be multi-homes to two L3 networks?

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Fri, 7 Sep 2012 18:01:33 +0200

How about the number of IPvX prefixes supported by typical data center switches (particularly ToR ones)?

I'm positive there are people who could make a beast with 100K /32 prefixes work using data center gear (not high-end SP routers) ... I'm also positive almost everyone else would walk away.

Ivan

> [email-masked]
> Sent: Friday, September 07, 2012 5:35 PM
> To: [email-masked]
> > Furthermore, we won't have /32 routes to BGP next hops any more (that
> would not scale to the several million endpoints from the NVO3 charter).
> The first L3 switch to which the server is connected (a ToR switch or a
> spine switch) will originate an IP prefix ==> we cannot use NH tracking.
> 
> AFAIK, the target scale for the number of NVE is not millions, but rather
> multiples of 10ks, up to 100ks.
> So, I would tend to agree with you that this kind of range would rule out
> IGP-based NH tracking, but the IGP is not the only tool we have to scale
> routing, is it ?
> 
> -Thomas
> 
> 
> >>> In most MPLS-based implementations we assume the endpoint is
> >>> reachable if we're getting VPN routing/signalling updates from it
> >>> and if it's reachable through the IP routing table. Some
> >>> implementations might be more cautious and use GRE encapsulation if
> >>> there's no LSP to the endpoint, others might use MPLS OAM.  Usually
> >>> these techniques are glossed over and left as implementation details.
> >> get withdrawn. This is "vanilla" BGP, and as such has be supported by
> >> any non- broken BGP implementation. In addition, there are several
> >> failure, such as Next- Hop tracking, egress PE protection, etc...
> >>> worse if you have a DC transport infrastructure that 's a mixture of
> L2 and L3).
> >> While discussing the upper bound, we should probably keep in mind
> >> that the goal here is to minimize connectivity disruption after a
> >> failure. Handling connectivity diruption could be done via either
> >> global or local repair techniques (both of these have been employed
> >> in 2547 VPNs to deal with PEs failures).
> >>
> >> Yakov.
> >>>> absence of liveliness check in *VPN. Are you talking about a
> >>>> certain deployment/implementation environment?
> >>>> Thanks,
> >>>> Florin
> >>>>
> >>>>>>> hypervisor would not (want to) run a routing protocol. So as
> >>>>>>> Ivan points out, the standard would need to accommodate multiple
> >>>>>>> IP addresses per NVE.
> >>>>>>>
> >>>>>>> Somesh
> >>>>>>>
> >>>>>>>> which also solves the return path issues (dynamic MAC learning
> >>>>>>>> in legacy network takes care of that). Maybe we should just use
> >>>>>>>> the wheel
> >>>>> that
> >>>>>>>>> We did not focus in the framework draft on the core redundancy
> >>>>>>>>> as
> >>>>>>> in
> >>>>>>>>>> interface and send and receive nvo3-encapsulated frames using
> >>>>>>>>>> more than one
> >>>>>>>> address.
> __________________________________________________________________________
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message par
> erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les
> pieces jointes. Les messages electroniques etant susceptibles
> d'alteration, France Telecom - Orange decline toute responsabilite si ce
> message a ete altere, deforme ou falsifie. Merci.
> 
> This message and its attachments may contain confidential or privileged
> information that may be protected by law; they should not be distributed,
> used or copied without authorisation.
> As emails may be altered, France Telecom - Orange is not liable for
> messages that have been modified, changed or falsified.
> Thank you.
>

---

From: Lucy yong <[email-masked]>
Date: Fri, 7 Sep 2012 16:23:59 +0000

Ivan,

I am a bit confused with the configuration you pictured below. NVE is on server, the server has more than one nic so each has IP interface address. But there is no routing protocol running on server and use of MPLS/VPN.

To use MPLS/VPN, you have a IGP network as underlay network first. If NVE acts as PE. The server need to participate IGP first, the loopback address of server is advertised in IGP. It does not matter how many nics exist, the ecmp make it work.

If the server does not run IGP protocol and act as a IP host in underlay network. This seems not aligning with the nvo3 framework and MPLS/VPN architecture.

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Friday, September 07, 2012 9:54 AM
To: 'Yakov Rekhter'
Cc: 'Balus, Florin Stelian (Florin)'; 'Somesh Gupta'; [email-masked]
Subject: Re: [nvo3] Support for multi-homed NVEs

Yakov,

Does this make sense?
Ivan

---

From: Ivan Pepelnjak <[email-masked]>
Date: Fri, 07 Sep 2012 19:23:51 +0200

On 9/7/12 6:23 PM, Lucy yong wrote:
[...]
> To use MPLS/VPN, you have a IGP network as underlay network first. If
> NVE acts as PE. The server need to participate IGP first, the
> loopback address of server is advertised in IGP. It does not matter
> how many nics exist, the ecmp make it work.

Ah, that's how it works. Thank you! Now I get it.

> If the server does not run IGP protocol and act as a IP host in
> underlay network. This seems not aligning with the nvo3 framework and
> MPLS/VPN architecture.

Then I sincerely apologize for wasting everyone's time and bandwidth.

Thank you for your guidance,
Ivan

---

From: Yakov Rekhter <[email-masked]>
Date: Fri, 7 Sep 2012 11:00:15 -0700

Ivan,

> Yakov,
> 
> The scenario that worries me is a hypervisor-based NVE with more
> than one transport (underlay) IP address, with each IP address tied
> to a NIC (or multiple bonded NICs), and potentially being in a
> different IP subnet (because it's connected to a different ToR
> switch). I'm further assuming that the hypervisor is not running a
> routing protocol and thus behaves like an IP host locally multihomed
> to multiple logical networks.
> 
> Assuming we use something like MPLS/VPN for NVO3, the BGP sessions
> from NVE to its BGP neighbors (example: a set of route reflectors)
> would run from one of the underlay IP addresses. If another underlay
> IP address becomes unreachable, we cannot use BGP session drop to
> detect that.

In a scenario where an NVE has two transport addresses, the BGP
session from the NVE to its BGP neighbors will advertise two routes
for each VM (each with its own RD, and its own Next Hop).  This is
pretty much the same as how multi-homing is done today for 2547
VPNs.

If an underlay IP address, used as a Next Hop is no longer available,
the NVE will withdraw the routes that carry this Next Hop. As a
side comment, E-VPN has special machinery to speed up this process.

Yakov.

> Furthermore, we won't have /32 routes to BGP next hops any more
> (that would not scale to the several million endpoints from the
> NVO3 charter). The first L3 switch to which the server is connected
> (a ToR switch or a spine switch) will originate an IP prefix ==>
> we cannot use NH tracking.
> 
> Finally, the server might not experience a NIC link loss if it loses
> connect ion to first-hop IP gateway. Example: blade enclosure switch
> might be a L2-only switch ==> we cannot expect the hypervisor-based
> NVE to revoke prefixes or change BGP next hop if its transport
> address becomes unreachable.
> 
> Using BFD from the NVE to detect first-hop gateway reachability and using th
at information to set BGP next hops in BGP prefix origination process (as prop
osed by Diego Garcia Del Rio) sounds like the simplest option.

---

From: Patrick Frejborg <[email-masked]>
Date: Fri, 7 Sep 2012 21:09:03 +0300

Ivan,

I have been following this discussion for a while, think we could
avoid having IGP on the NVE in the hypervisor by changing the
transport protocol to a multipath enabled one such as SCTP/MPTCP.

By doing that no need to have an IGP/BGP on the NVE and you can
aggregate NVE IP addresses at the underlying network because the
multpath transport protocol would take care of liveness problem. If a
multi-homed NVE (e.g. two NICs to two separate ToRs) knows the remote
NVEs locators (two Nics, two ToRs) the multi-path protocol has the
possible to setup four paths to the remote NVE (Local locator
1->Remote Locator1, LL1-RL2, LL2->RL1 and LL2->RL2). The consequences
of this is that some of the traffic engineering is moved away from the
network to transport protocol, check out this study
http://www-malted.cs.ucl.ac.uk/staff/M.Handley/papers/multipath_dc_hotnets.pdf

The demarcation point between the hypervisor based NVE and the ToR
becomes simple, i.e. IP subnets and the underlying network doesn't
need to exchange much information to the NVE.

And when we get to the storage part we can maintain the air gap
between the two fabrics by leveraging SCTP/MPTCP - I'm thinking of
lossless iSCSI here and if the storage target supports multipath
transport protocol we can build SANs as we do today and maintain the
best practices.

Think it would be worthwhile to have a look on what SCTP/MPTCP can
offer, it seems that they might simplify things quite a lot.

Patrick

---

From: Ivan Pepelnjak <[email-masked]>
Date: Fri, 7 Sep 2012 20:42:32 +0200

And how will a NVE behind a summarization boundary detect the BGP next hop is unreachable? You think people will deploy NVE BGP route reflectors at every aggregation boundary?

=====
Mistyped and autocorrected on a clunky virtual keyboard

On 7. sep. 2012, at 19:43, Michael Richardson <[email-masked]> wrote:

> 
>>>>>> "Ivan" == Ivan Pepelnjak <[email-masked]> writes:
>    Ivan> I'm positive there are people who could make a beast with 100K
>    Ivan> /32 prefixes work using data center gear (not high-end SP
>    Ivan> routers) ... I'm also positive almost everyone else would walk
>    Ivan> away. 
> 
> So, we started with the problem of too many MAC addresses spreading all
> over the data centre.  We had millions of those.
> 
> If we had a flat layer-2 DC network for the "underlay" network, we'd
> have to manage 100K MAC addresses for the NVE boxes.  That won't work,
> so know that we have to go to a hierarchical model of address
> allocation, which means not using layer-2 addresses.
> 
> (/32 prefixes... don't you mean /128 prefixes?)
> 
> It seems to me that in the canonical case of a ToR with single
> trunks to NVEs, that the ToR should allocate addresses in topologically
> significant ways, and if the "route-reflector" is not co-located inside
> the ToR, that it will deal with a few racks only, and can deal with the
> number of /128 routes required.
> 
> In the non-canonical case where there are multiple trunks from the NVE
> to multiple ToR, that one of the ToR switch needs to allocate the /128
> addresses which the NVE uses on loopback (if not the ToR itself, then
> the DHCPv6 server that it relays to. DHCPv6PD might be appropriate for
> this).  Horizons (OSPF areas, etc.) would keep these /128 routes
> contained in the minimal pieces that need to know.  Since these
> addresses would form a hierarchy, the entire data centre won't need to
> know about all 100K NVE's /128.
> 
> 
> -- 
> Michael Richardson <[email-masked]>, Sandelman Software Works 
> 
>

---

From: Ivan Pepelnjak <[email-masked]>
Date: Fri, 7 Sep 2012 20:46:22 +0200

Are you talking about control or data plane? Data plane has to run over a datagram service or you're back to the TCP-over-SSH type of problems (not to mention performance issues).

=====

---

From: Somesh Gupta <[email-masked]>
Date: Fri, 7 Sep 2012 11:50:01 -0700

A typical multi-homed server does not run any routing protocol.
If the problem can be solved without inviting routing protocols
into the hypervisor based NVE, then that is the approach that
will get adopted.

> Sent: Friday, September 07, 2012 11:00 AM
> To: Ivan Pepelnjak
> Cc: 'Yakov Rekhter'; 'Balus, Florin Stelian (Florin)'; [email-masked];
> Somesh Gupta
> reachable
> it's
> VPN
> > > > On the other hand, LISP defines several mechanisms that can be
> used to
> > > > check the ETR liveliness.
> > >
> whether
> we
> > > > remote NVE IP address ... and it's not an implementation detail,
> it's
> > > > a mandatory requirement.
> > >
> that the
> > > > > > In MPLS/VPN we rely on the host route to PE loopback and/or
> LSP to
> > > > > > the
> > > > > > implementations. This is an implementation matter though in
> my
> > > opinion.
> > > > > > >
> > > > > > >>> NVEs are multi-homed directly to the next IP router,
> regular
> > > > > > routing
> > > > > > >>> between NVE and the next IP hop, L2 resiliency mechanisms
> need
> > > > > > >>> to be
> > > > > employed.
> > > > > > >>>> By Multi-homed NVEs one can also understand a set of
> NVEs
> > > > > > >>>> multi-homed
> > > > > > >>> have in my opinion. An use case example: NVO3 network -
> NVE
> > > > > > >>> GWs
> > > > > > >> multi-
> > > > > > >>>> I think the latter is worth discussing although there
> are
> > > > > > >>>> some
> > > > > > >>>>> From: [email-masked] [mailto:nvo3-
> [email-masked]]
> > > > > > >>>>> On Behalf
> > > > > > >>> Of
> > > > > > >>>>> Ivan Pepelnjak
> > > > > > >>>>> behavior and would fit well with existing DC
> deployments. It
> > > > > > would
> > > > > > >>>>> also
> > > > > > >>> retain
> > > > > > >>>>>> From: [email-masked] [mailto:nvo3-
> [email-masked]]
> > > > > > >>>>>> On
> > > > > > >>> Behalf
> > > > > > >>>>>> Of Somesh Gupta

---

From: Patrick Frejborg <[email-masked]>
Date: Fri, 7 Sep 2012 22:02:19 +0300

Dataplane, and yes, the issues (and benefits) would be the same as with STT
Need to look into the details of SCTP, as it is a streaming protocol,
it might perform better.

One thing that I forgot to mention is about backup traffic, VM
mobility and heartbeat cluster traffic. These applications are
performed on the private network and by using BCP for SAN you have two
fabrics for the private network (DCB enabled). With a singlepath
transport protocol I have to choose either fabric A or B for the
backup traffic, heartbeat etc, in order to keep the air gap between
the fabrics. With multipath support these applications could use both
fabrics concurrently.

Patrick

---

From: Lucy yong <[email-masked]>
Date: Fri, 7 Sep 2012 20:14:34 +0000

If this is the requirement, it looks like that the server is equivalent to CE site in MPLS/VPN, NVE is on the server as if vrf on CE (vrf-lite). ToR is the PE. In this case, you do not have to run routing protocol on server, and it is looks like multi-homing in E-VPN.

In this configuration, the server is decouple from DC physical network. The server only connects to a VPN configured on DC physical network. 

Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Somesh Gupta
Sent: Friday, September 07, 2012 1:50 PM
To: Yakov Rekhter; Ivan Pepelnjak
Cc: 'Balus, Florin Stelian (Florin)'; [email-masked]
Subject: Re: [nvo3] Support for multi-homed NVEs

> > > > a mandatory requirement.

---

From: Somesh Gupta <[email-masked]>
Date: Fri, 7 Sep 2012 13:18:16 -0700

I know there are a whole bunch of MPLS gurus on this
mailing list. Can we decouple the discussion from
MPLS to make it easier for those barely MPLS-literate?

> From: Lucy yong [mailto:[email-masked]]
> Sent: Friday, September 07, 2012 1:15 PM
> To: Somesh Gupta; Yakov Rekhter; Ivan Pepelnjak
> If this is the requirement, it looks like that the server is equivalent
> to CE site in MPLS/VPN, NVE is on the server as if vrf on CE (vrf-
> lite). ToR is the PE. In this case, you do not have to run routing
> protocol on server, and it is looks like multi-homing in E-VPN.
>
> In this configuration, the server is decouple from DC physical network.
> The server only connects to a VPN configured on DC physical network.
>
> Lucy
>
> > > > speed up connectivity restoration after egress PE failure, such
> as
> > Next-
> > > > Yes, except that all these LISP mechanisms are broken in one way
> or
> > > > another.
> > > >
> > > > > > > multiple IPs per NVE. We have multiple IPs per PE in
> current
> > VPN
> > > > > > > >> the hypervisor would not (want to) run a routing
> protocol.
> > So
> > > > > > > >>> address (potential per virtual network segment), then
> the
> > rest
> > > > > > > >>> On the overlay side, the real problem (as you stated)
> is
> > the
> > > > > > > >>> multi-homing of NVO3-to-legacy gateways. I don't see
> any
> > other
> > > > > > > >>> horizon bridging (thus no forwarding loops through
> NVO3),
> > with
> > > > > > > >>> every VM MAC address being dynamically assigned to one
> of
> > the
> > > > > > > >>> MAC learning in legacy network takes care of that).
> Maybe
> > we
> > > > > > > >>> on the access side to other devices. That is a
> discussion
> > we
> > > > > > > >>> need
> > > > > > > to
> > > > > > > >>> mechanisms and some standardization initiatives in
> place
> > > > already.
> > > > > > > >>>>
> > > > > > > >>>>> (A) ignore the issue and expect the network designer
> to
> > > > > > > >>>>> solve it
> > > > > > > >>> using
> > > > > > > >>>>> Option (B) would reduce the complexity of the
> underlay
> > DC
> > > > > > > >>>>> single-homed IP addresses), but we'd have to deal
> with
> > a
> > > > > > > >>>>> bunch of additional
> > > > > > > >>> problems
> > > > > > > >>>>>> framework-03.txt. NVEs are connected together by an
> L3
> > > > > > > >>>>>> network
> > > > > > > >>>>>> -
> > > > > > > >>> does
> > > > > > > >>>>>> that mean only one?

---

From: Lucy yong <[email-masked]>
Date: Fri, 7 Sep 2012 20:34:16 +0000

That is fine. We do not have to use a particular solution for the discussion. But we need to have common understanding on the framework. I do not picture current framework that has such assumption for the underlay network. If this is the requirement, we need to describe the underlay network component and make it clear too. 

Lucy

---

From: John E Drake <[email-masked]>
Date: Fri, 7 Sep 2012 13:41:34 -0700

Lucy,

I don't know if I mentioned it, but in E-VPN multi-homing between a CE and one or more PEs is modeled as a LAG, so there should only be a single IP address for the LAG.  From the E-VPN perspective, the LAG is modeled as an Ethernet Segment and is identified with the CE's MAC address.  A PE announces connectivity to a CE across an Ethernet Segment using the Per ESI Ethernet AD route, whose use in block MAC withdrawal I described in my email this morning.

Yours irrespectively,

John

> Lucy yong
> If the problem can be solved without inviting routing protocols into
> the hypervisor based NVE, then that is the approach that will get
> adopted.
>
> > pretty much the same as how multi-homing is done today for 2547 VPNs.
> >
> > the NVE will withdraw the routes that carry this Next Hop. As a side
> > comment, E-VPN has special machinery to speed up this process.
> >
> > Yakov.
> >
> > > (a ToR switch or a spine switch) will originate an IP prefix ==> we
> > > cannot use NH tracking.
> > >
> > at information to set BGP next hops in BGP prefix origination process
> > (as prop osed by Diego Garcia Del Rio) sounds like the simplest
> > option.
> > >
> > > Does this make sense?
> > > Ivan
> > >
> > > > sites connected to that PE. So, if the PE goes down, the routes
> > > > by
> > any non-
> > > > or another.
> > > > > > > /32 prefix to indicate next hop validity. That won't work
> > > > > > > for hypervisor-based NVEs
> > > > > > >
> > > > > > > >> as Ivan points out, the standard would need to
> > > > > > > >> accommodate multiple IP addresses per NVE.
> > > > > > > >>
> > > > > > > >> Somesh
> > > > > > > >>
> > > > > > > >>> IP address or multiple IP addresses (like some NVGRE
> > > > > > > >>> load balancing proposals). If we decide NVEs have a
> > > > > > > >>> single IP address (potential per virtual network
> > > > > > > >>> segment), then the
> > rest
> > > > > > > >>>>> to solve it
> > > > > > > >>>>> interface and send and receive nvo3-encapsulated
> > > > > > > >>>>> frames using more than one
> > > > > > > >>> address.
> > > > > > > >>>>>> L3 network

---

From: Lucy yong <[email-masked]>
Date: Fri, 7 Sep 2012 20:45:52 +0000

John,

I guess this is not matter for now. They do not want to use MPLS VPN solution for the discussion.

Lucy

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Friday, September 07, 2012 3:42 PM
To: Lucy yong; Somesh Gupta; Yakov Rekhter; Ivan Pepelnjak
Cc: 'Balus, Florin Stelian (Florin)'; [email-masked]
Subject: RE: [nvo3] Support for multi-homed NVEs

Lucy,

Yours irrespectively,

John

---

From: John E Drake <[email-masked]>
Date: Fri, 7 Sep 2012 13:49:25 -0700

Somesh,

MPLS or L3/L2 VPN?

The latter are NVO applications that use an MPLS label as the mux/demux technology at the VPN edges (much like VNI or Tenant ID except that the MPLS label has only local significance) and which may use a variety of tunneling technologies, including but not limited to MPLS, across the common physical infrastructure.

In this context, VXLAN and NVGRE might also be considered to be perfectly acceptable tunneling technologies.

Yours irrespectively,

John

> Sent: Friday, September 07, 2012 1:18 PM
> I know there are a whole bunch of MPLS gurus on this mailing list. Can
> we decouple the discussion from MPLS to make it easier for those barely
> MPLS-literate?
>
> > If this is the requirement, it looks like that the server is
> > equivalent to CE site in MPLS/VPN, NVE is on the server as if vrf on
> > CE (vrf- lite). ToR is the PE. In this case, you do not have to run
> > routing protocol on server, and it is looks like multi-homing in E-
> VPN.
> >
> > > Cc: 'Yakov Rekhter'; 'Balus, Florin Stelian (Florin)';
> > > [email-masked]; Somesh Gupta
> > > > than one transport (underlay) IP address, with each IP address
> > > > tied to a NIC (or multiple bonded NICs), and potentially being in
> > > > a different IP subnet (because it's connected to a different ToR
> > > If an underlay IP address, used as a Next Hop is no longer
> > > available, the NVE will withdraw the routes that carry this Next
> > > Hop. As a side comment, E-VPN has special machinery to speed up
> this process.
> > >
> > > Yakov.
> > >
> > > > NVO3 charter). The first L3 switch to which the server is
> > > > connected (a ToR switch or a spine switch) will originate an IP
> > > > prefix ==> we cannot use NH tracking.
> > > >
> > > > might be a L2-only switch ==> we cannot expect the
> > > > hypervisor-based NVE to revoke prefixes or change BGP next hop if
> > > > its transport address becomes unreachable.
> > > >
> > > > Using BFD from the NVE to detect first-hop gateway reachability
> > > > and
> > > using th
> > > at information to set BGP next hops in BGP prefix origination
> > > process (as prop osed by Diego Garcia Del Rio) sounds like the
> > > simplest option.
> > > >
> > > > Does this make sense?
> > > > Ivan
> > > >
> > > > > > encapsulated into an envelope with a known destination
> > > > > > endpoint (PE-router, ETR), and sent on its way. The big
> > > > > > question is "is
> > > the
> > > > > > endpoint, others might use MPLS OAM.  Usually these
> techniques
> > > are
> > get
> > > > > withdrawn. This is "vanilla" BGP, and as such has be supported
> > > > > by
> > > any non-
> > > > > > hypervisor using more than one underlay IP address, more so
> if
> > > its
> > > > > > the other IP addresses are reachable (the problem becomes
> > > > > > worse
> > > if you
> > > > > > have a DC transport infrastructure that 's a mixture of L2
> and
> > > L3).
> > > > >
> say
> > > what
> > > > > mind
> > > > > goal here is to minimize connectivity disruption after a
> failure.
> > > Handling
> > > > > techniques (both of these have been employed in 2547 VPNs to
> > > > > deal
> > > with PEs
> > > > > failures).
> > > > >
> > > > > Yakov.
> > > > > >
> > > > > > > EVPN) connected to LISP? Also can you be more specific on
> > > > > > > the absence of liveliness check in *VPN. Are you talking
> > > > > > > about a
> > > certain
> > for
> > > > > > > > > AFAIK the text in NVO3 framework and requirements
> drafts
> > > allows
> > > > > > > > >>> On the underlay side, we might decide that NVEs have
> a
> > > single
> > load
> > > > > > > > >>> IP-address-per-link option (thus no need for L2 or
> > > > > > > > >>> MLAG
> > > at all).
> > > > > > > > >>>
> > > > > > > > >>> NVO3 network works exactly like VMware's vSwitch:
> > > > > > > > >>> split horizon bridging (thus no forwarding loops
> > > > > > > > >>> through
> > NVO3),
> > > with
> > > > > > > > >>> will take care of it. If there is Ethernet switching
> > > > > > > > >>> in between NVE and the next IP hop, L2 resiliency
> > mechanisms
> > > need
> > > > > > > > >>> to be
> > > > > > > employed.
> > > > > > > > >>>>> While I don't think we should go down the path of
> > > > > > > > >>>>> IP-
> > > A/IP-B
> > > > > > > > >>>>> networks similar to some other DC technology, we
> > > > > > > > >>>>> will
> > > face
> > > > > > > > >>>>> the reality of
> > > > > > > > >>> some

---

From: John E Drake <[email-masked]>
Date: Fri, 7 Sep 2012 14:06:19 -0700

Lucy,

I don't feel there has been much polarization in the discussion, so I'm unsure who 'they' are.  Also, you mentioned 'E-VPN' in the email to which I was replying.

In any event, I was merely offering LAG/Ethernet Segment as one way of modeling this connectivity.

Yours irrespectively,

John

> Sent: Friday, September 07, 2012 1:46 PM
> To: John E Drake; Somesh Gupta; Yakov Rekhter; Ivan Pepelnjak
> I don't know if I mentioned it, but in E-VPN multi-homing between a CE
> and one or more PEs is modeled as a LAG, so there should only be a
> single IP address for the LAG.  From the E-VPN perspective, the LAG is
> modeled as an Ethernet Segment and is identified with the CE's MAC
> address.  A PE announces connectivity to a CE across an Ethernet
> Segment using the Per ESI Ethernet AD route, whose use in block MAC
> withdrawal I described in my email this morning.
>
> Yours irrespectively,
>
> John
>
>
> > Of Lucy yong
> > > > > > > > >>> through NVO3),

---

From: Somesh Gupta <[email-masked]>
Date: Fri, 7 Sep 2012 15:31:27 -0700

I am not opposed to using MPLS/VPLS as an example to discuss
how to addresses the problem. But if possible, please
make the relationship between the example and how it solves
somewhat explicit rather than cryptic - if I forgot something
I can always read up.

Essentially when the NVE is multi-homed (not LAGed) and not
running a routing protocol; It would be great if someone
could tell me the RFC# to read on how the problem is solved in MPLS.

I know it is a fine balance, and we have folks from different
backgrounds here - so accommodate to the extent possible.

Thanks
Somesh

> Sent: Friday, September 07, 2012 1:49 PM
> To: Somesh Gupta; Lucy yong; Yakov Rekhter; Ivan Pepelnjak
> The latter are NVO applications that use an MPLS label as the mux/demux
> technology at the VPN edges (much like VNI or Tenant ID except that the
> MPLS label has only local significance) and which may use a variety of
> tunneling technologies, including but not limited to MPLS, across the
> common physical infrastructure.
>
> In this context, VXLAN and NVGRE might also be considered to be
> perfectly acceptable tunneling technologies.
>
> Yours irrespectively,
>
> John
>
>
> into
> > > > > switch). I'm further assuming that the hypervisor is not
> running
> > a
> > > > > > > All the VPNs you've mentioned share the data-plane
> principles
> > > > with
> > > > > > > reachable through the IP routing table. Some
> implementations
> > > > might be
> > > > > > > more cautious and use GRE encapsulation if there's no LSP
> to
> > > the
> > > > > > In 2547 VPNs (or E-VPNs) it is the PE that originates routes
> to
> > > the
> > > > VPN
> > > > > > > Coming back to nvo3, we'll have serious problems with NVE
> in
> > a
> > > > > > > need (in my opinion) something that checks the liveliness
> of
> > > the
> > > > > > connectivity diruption could be done via either global or
> local
> > > > repair
> > > > > > > > [FB>] How are the VPN specifications (IP VPN, VPLS, VPWS
> or
> > > > incoming
> > > > > > > > > implementations. This is an implementation matter
> though
> > > > > > > > > in
> > > > my
> > > > > > opinion.
> > > > > > > > > >
> > > > > > > > > >>> balancing proposals). If we decide NVEs have a
> single
> > > IP
> > > > > > > > > >>> Alternatively we might implement the option of
> having
> > > > multiple
> > > > > > > > > >>> our opinion there was no need to standardize
> anything
> > > > here.
> > > > > > > > > >>> There are no differences from what is available
> today
> > > in
> > > > > > > > > >>> regular IP
> > > > > > > > > networks:
> > > > > > > > > >> if
> > > > > > > > > >>> have in my opinion. An use case example: NVO3
> network
> > > > > > > > > >>> -
> > > > NVE
> > > > > > > > > >>> GWs
> > > > > > > > > >> multi-
> > > > > > > > > >>>>> Option (A) is the easy way out similar to
> existing
> > > > MPLS/VPN

---

From: Lucy yong <[email-masked]>
Date: Fri, 7 Sep 2012 23:01:53 +0000

In addition, NVE is on a server and performs the overlay encapsulation.

Lucy

-----Original Message-----
From: Somesh Gupta [mailto:[email-masked]] 
Sent: Friday, September 07, 2012 5:31 PM
To: John E Drake; Lucy yong; Yakov Rekhter; Ivan Pepelnjak
Cc: 'Balus, Florin Stelian (Florin)'; [email-masked]
Subject: RE: [nvo3] Support for multi-homed NVEs

Thanks
Somesh

---

From: Aldrin Isaac <[email-masked]>
Date: Fri, 7 Sep 2012 19:33:44 -0400

Where a BGP-base solution is involved, IMO, it should be possible for the
default gateway of the NVE to also be its RR.  The transport tunnel could
be VXLAN, etc if MPLS is not an option.

On Friday, September 7, 2012, Ivan Pepelnjak wrote:

> And how will a NVE behind a summarization boundary detect the BGP next hop
> is unreachable? You think people will deploy NVE BGP route reflectors at
> every aggregation boundary?
>
> =====
> On 7. sep. 2012, at 19:43, Michael Richardson <[email-masked]<javascript:;>>
> wrote:
>
> >
> >>>>>> "Ivan" == Ivan Pepelnjak <[email-masked] <javascript:;>>
> writes:
> > Michael Richardson <[email-masked] <javascript:;>>, Sandelman
> Software Works
> >
> >
> [email-masked] <javascript:;>

---

From: Ivan Pepelnjak <[email-masked]>
Date: Sat, 8 Sep 2012 08:29:39 +0200

The DC Ops will love that idea - redesign and redeploy your IP routing protocols before deploying nvo3

=====

On 8. sep. 2012, at 01:33, Aldrin Isaac <[email-masked]> wrote:

> Where a BGP-base solution is involved, IMO, it should be possible for the default gateway of the NVE to also be its RR.  The transport tunnel could be VXLAN, etc if MPLS is not an option.  
>

---

From: Aldrin Isaac <[email-masked]>
Date: Sat, 8 Sep 2012 11:34:31 -0400

I am a DC Op, so with that hat on let me speak for my network and similar
network design:

Since my access routers exchange access/subnet routes over BGP (the IGP
only has to worry about router loopbacks and transit links), the only
change to these access routers should be to allow peering from adjacent NVE
RRC on its access subnets.

In this configuration if a default gateway is configured for passive
peering from local NVE subnets then I can see how, with some simple
scripting, an NVE can be made to automatically peer with it's default
gateways once it gets address from DHCP.  There are details that would need
to be worked out but I don't believe these are insurmountable.

On Saturday, September 8, 2012, Ivan Pepelnjak wrote:

> On 8. sep. 2012, at 01:33, Aldrin Isaac <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>>
> wrote:
>
>> And how will a NVE behind a summarization boundary detect the BGP next
>> hop is unreachable? You think people will deploy NVE BGP route reflectors
>> at every aggregation boundary?
>>
>> =====

---

From: Lucy yong <[email-masked]>
Date: Mon, 10 Sep 2012 16:00:14 +0000

Hi Ivan,

Thank you for pointing out this scenario. We updates the vpn gap analysis draft to capture this. Typically, BGP/MPLS VPN is configured at network edge device, not at a host. NVO3 requires NVE on a server and the server is a host, not network edge device.  However, BGP/MPLS [RFC4364] support Multi-ASes, in which Option B is similar to this case, i.e. PE may connect to multi-ASBRs.  Here is the draft. Any comment are welcome.

http://www.ietf.org/id/draft-hy-nvo3-vpn-protocol-gap-analysis-01.txt

Regards,
Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ivan Pepelnjak
Sent: Saturday, September 08, 2012 1:30 AM
To: Aldrin Isaac
Cc: Michael Richardson; [email-masked]; [email-masked]
Subject: Re: [nvo3] Support for multi-homed NVEs

On 8. sep. 2012, at 01:33, Aldrin Isaac <[email-masked]<mailto:[email-masked]>> wrote:
Where a BGP-base solution is involved, IMO, it should be possible for the default gateway of the NVE to also be its RR.  The transport tunnel could be VXLAN, etc if MPLS is not an option.

=====

On 7. sep. 2012, at 19:43, Michael Richardson <[email-masked]<javascript:;>> wrote:

>
>>>>>> "Ivan" == Ivan Pepelnjak <[email-masked]<javascript:;>> writes:
> Michael Richardson <[email-masked]<javascript:;>>, Sandelman Software Works
>
>
_______________________________________________
nvo3 mailing list
[email-masked]<javascript:;>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 11 Sep 2012 21:39:36 -0400

There is some concern about the use of MPLS because of the number of transport tunnels that the NVE would need to maintain.  In the architecture I describe below where the default gateway router is also the RR of the NVE, it is possible (if your favorite vendor is willing to add the feature) that the NVE can get the MPLS transport tunnel to remote NVE over labelled unicast BGP and automatically filtered down to only the subset of remote NVE from which a VPN route is also received.

On Sep 8, 2012, at 11:34 AM, Aldrin Isaac wrote:

> I am a DC Op, so with that hat on let me speak for my network and similar network design:
> 
> Since my access routers exchange access/subnet routes over BGP (the IGP only has to worry about router loopbacks and transit links), the only change to these access routers should be to allow peering from adjacent NVE RRC on its access subnets.
> 
> In this configuration if a default gateway is configured for passive peering from local NVE subnets then I can see how, with some simple scripting, an NVE can be made to automatically peer with it's default gateways once it gets address from DHCP.  There are details that would need to be worked out but I don't believe these are insurmountable.  
>