Subject: [nvo3] draft-drake-nvo3-evpn-control-plane

From: Thomas Nadeau <[email-masked]>
Date: Mon, 17 Sep 2012 08:55:25 -0700

A number of us just published this draft and wanted to bring it to the NVO3 WG's attention.  We will be presenting/discussing this draft at the interim meeting this week as well, but please discuss here on the list as well.

	Thanks,

	Tom, John, et al

A new version of I-D, draft-drake-nvo3-evpn-control-plane-00.txt
has been successfully submitted by Thomas D. Nadeau and posted to the
IETF repository.

Filename:	 draft-drake-nvo3-evpn-control-plane
Revision:	 00
Title:		 A Control Plane for Network Virtualized Overlays
Creation date:	 2012-09-16
WG ID:		 Individual Submission
Number of pages: 12
URL:             http://www.ietf.org/internet-drafts/draft-drake-nvo3-evpn-control-plane-00.txt
Status:          http://datatracker.ietf.org/doc/draft-drake-nvo3-evpn-control-plane
Htmlized:        http://tools.ietf.org/html/draft-drake-nvo3-evpn-control-plane-00

Abstract:
       The purpose of this document is to describe how Ethernet Virtual
       Private Network (E-VPN) can be used as the control plane for
       Network Virtual Overlays.  Currently this protocol is defined to
       act as the control plane for Virtual Extensible Local Area
       Network (VXLAN), Network Virtualization using Generic Routing
       Encapsulation (NVGRE), MPLS or VLANs while maintaining their
       existing data plane encapsulations. The intent is that this
       protocol will be capable of extensions in the future to handle
       additinal data plane encapsulations and functions as needed.

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Mon, 17 Sep 2012 16:11:57 +0000

May be I missing something here .. but does this suggest running BGP-EVPN
on the NVE 
that is located in the hypervisor?

Dimitri

On 9/17/12 8:55 AM, "Thomas Nadeau" <[email-masked]> wrote:

>
>	A number of us just published this draft and wanted to bring it to the
>NVO3 WG's attention.  We will be presenting/discussing this draft at the
>interim meeting this week as well, but please discuss here on the list as
>well.
>
>	Thanks,
>
>	Tom, John, et al
>
>
>URL:             
>http://www.ietf.org/internet-drafts/draft-drake-nvo3-evpn-control-plane-00
>.txt
>Status:          
>http://datatracker.ietf.org/doc/draft-drake-nvo3-evpn-control-plane
>Htmlized:        
>http://tools.ietf.org/html/draft-drake-nvo3-evpn-control-plane-00
>
>
>Abstract:
>_______________________________________________
>nvo3 mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/nvo3

---

From: Vishwas Manral <[email-masked]>
Date: Mon, 17 Sep 2012 10:57:50 -0700

Hi Dimitri,

Though I am not the author, in my view yes that is what this would suggest,
to run E-VPN control plane in the Hypervisor of other NVE located on the
hardware networking device.

Thanks,
Vishwas

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 17 Sep 2012 15:18:13 -0400

I'm not sure that the dust has fully settled on the matter.
http://tools.ietf.org/html/draft-marques-l3vpn-end-system-07 suggests
the use of XMPP.  The question is whether there is any sound technical
reason (versus preferences) why leveraging BGP is problematic.  I
personally haven't heard a convincing argument.

---

From: Lucy yong <[email-masked]>
Date: Mon, 17 Sep 2012 22:36:28 +0000

Read this draft.

RFC5512 applies a case where two BGP speakers are in a BGP free core. Using encapsulation tunnel between two speakers enables one speaker to send a packet to another speaker as the next-hop.

Using this approach in nvo3 may rise a high scalability concern because any pair of NVEs in an NVO will need to maintain a state for the tunnel encapsulation.  

If some NVEs support VXLAN and some support NVGRE, to build mcast tree for BUM, it has to build two distinct sub-trees for each, which is more complex.

   "This memo specifies that an egress PE must use the sender MAC 
   address to determine whether to send a received Broadcast or 
   Multicast packet to a given Ethernet Segment.  I.e., if the sender 
   MAC address is associated with a given Ethernet Segment, the egress
   PE must not send the packet to that Ethernet Segment."

Does it mean using BGP to exchange NVE MAC address that belong to an Ethernet segment first? How does this impact other evpn features?

This needs to be cooked more.

Cheers,
Lucy

---

From: Kireeti Kompella <[email-masked]>
Date: Mon, 17 Sep 2012 18:17:55 -0700

Hi Lucy,

> Using this approach in nvo3 may rise a high scalability concern because any pair of NVEs in an NVO will need to maintain a state for the tunnel encapsulation.  

They would have to in any case.  The tunnel encap is a couple of bits; the "tenant id" is also needed.

> Does it mean using BGP to exchange NVE MAC address that belong to an Ethernet segment first? How does this impact other evpn features?

Yes to the first question; not at all (imo) to the second.

> This needs to be cooked more.

I think it's pretty well cooked, although I must confess a predilection for sushi.  In effect, these very capable authors saved me the trouble of writing pretty much the same draft :-)

The only thing I would change is the draft name: I prefer "…-nvo3-l2-in-l3-control-plane".  Oh, and add a code point for STT :-)

Kireeti

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 18 Sep 2012 09:40:13 -0700

Cool, very happy to see this!

A couple of comments:
1) spell out that a tunnel attr with encap GRE means MPLS-in-GRE. 
2) sect 6 and first para of 6.1: make a note that BUM traffic may not need to be transported. In particular, unknown unicast is not needed, broadcast for ARP likewise.
3) why the funky four reserved bits for VNID? Leave it as 24 bit IDs ...
4) I like the idea of local VNIDs, but need to bounce that off the respective draft authors.
5) add an MPLS-in-UDP encap.

Kireeti

Sent from my iPad

---

From: Truman Boyes <[email-masked]>
Date: Tue, 18 Sep 2012 12:50:53 -0400

Agreed. However XMPP offers some interesting functionality if used on NVE. TLS, certificates, arbitrary data models and completely extendible. 

Kind regards,
Truman Boyes

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 18 Sep 2012 17:35:17 +0000

Kireeti > [...] I would change is the draft name: I prefer "...-nvo3-l2-
Kireeti > in-l3-control-plane".  

I have the same comment. The document describes control plane for layer 2 overlays (i.e., for transporting MAC/ethernet headers for IP packets). Throughout the document when "network virtualization overlay" is used it should be clarified that it is a layer 2 overlay.

Maria

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Kireeti Kompella
> Sent: Monday, September 17, 2012 9:18 PM
> To: Lucy yong
> Cc: [email-masked]
> > RFC5512 applies a case where two BGP speakers are in a BGP free core.
> Using encapsulation tunnel between two speakers enables one speaker to
> send a packet to another speaker as the next-hop.
> >
> > Using this approach in nvo3 may rise a high scalability concern
> because any pair of NVEs in an NVO will need to maintain a state for
> the tunnel encapsulation.
> 
> > If some NVEs support VXLAN and some support NVGRE, to build mcast
> tree for BUM, it has to build two distinct sub-trees for each, which is
> more complex.
> >
> > Does it mean using BGP to exchange NVE MAC address that belong to an
> Ethernet segment first? How does this impact other evpn features?
> 
> I think it's pretty well cooked, although I must confess a predilection
> for sushi.  In effect, these very capable authors saved me the trouble
> of writing pretty much the same draft :-)
> 
> The only thing I would change is the draft name: I prefer "...-nvo3-l2-
> in-l3-control-plane".  Oh, and add a code point for STT :-)
> 
> Kireeti
> 
> > Cheers,
> > Lucy
> >
> >
> >
> >
> >

---

From: Lucy yong <[email-masked]>
Date: Tue, 18 Sep 2012 19:05:43 +0000

Yes, add MPLS-in-UDP encap. It is described in
http://tools.ietf.org/html/draft-xu-mpls-in-udp-02

See below.

Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Tuesday, September 18, 2012 11:40 AM
To: Thomas Nadeau
Cc: [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

A couple of comments:

Kireeti

Sent from my iPad

---

From: Lucy yong <[email-masked]>
Date: Tue, 18 Sep 2012 20:18:33 +0000

Hi Kreeti,

Regarding interworking capability, Is "a given EVI can support multiple data plane encapsulation" equivalent to say "individual NVEs need to support multiple encapsulation schemas"? If one NVE only supports VXLAN and another NVE only supports MPLS-in-GRE, two will not able to work in a same EVI, is that right? Will this give more benefit than having one encapsulation in an EVI or make more complex?

Regards,
Lucy

-----Original Message-----
From: Kireeti Kompella [mailto:[email-masked]] 
Sent: Monday, September 17, 2012 8:18 PM
To: Lucy yong
Cc: [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Hi Lucy,

The only thing I would change is the draft name: I prefer "...-nvo3-l2-in-l3-control-plane".  Oh, and add a code point for STT :-)

Kireeti

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 18 Sep 2012 17:24:15 -0400

The imported E-VPN route will determine what the next hop entry in the
EVI will look like -- whether it will have encapsulation A or
encapsulation B.  That is determined by the sender of the E-VPN route.
 This is like having a PPP interface and an Ethernet interface
connected to the same VRF.

Ideally the other encapsulations would have included support for
multi-homing by including an additional field for a split-horizon ID
for use by control-planes and NVE that support multi-homing.  Maybe
it's not too late to add an SH field to these encapsulations since it
seems that there is some unused bits in nvGRE (maybe not enough) and
in VXLAN -- just a thought.

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 18 Sep 2012 18:26:06 -0700

Hi Lucy,

Sent from my iPad

On Sep 18, 2012, at 13:18, Lucy yong <[email-masked]> wrote:

> Hi Kreeti,
> 
> Regarding interworking capability, Is "a given EVI can support multiple data plane encapsulation" equivalent to say "individual NVEs need to support multiple encapsulation schemas"?

Not "need", but "nice to have", esp. from an interworking pov.

> If one NVE only supports VXLAN and another NVE only supports MPLS-in-GRE, two will not able to work in a same EVI, is that right?

Clearly so.

> Will this give more benefit than having one encapsulation in an EVI or make more complex?

I would guess that most DCs will use only one encap.  However, if more than one encap was used, and enough devices did multiple encaps, this might still work. 

Stepping back, though, this is not really about interworking, but rather, leveraging a single control across many encaps.  This is especially interesting as many encaps still don't have a signaling plane. 

A side benefit, yet to be explored and exploited, is making the VNID locally significant. This is a huge step, IMO, as this offloads the intelligence of assigning and managing VNIDs from a central entity to the nve. 

Kireeti

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 18 Sep 2012 16:09:35 -0700

There is definitely a requirement to do different encapsulations simultaneously. There is even support coming from NIC vendors to support multiple VMs with different encapsulations at the same time. 

	--Tom

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 18 Sep 2012 19:14:38 -0700

> Stepping back, though, this is not really about interworking, but rather, leveraging a single control across many encaps.  This is especially interesting as many encaps still don't have a signaling…

TOM: Precisely. 

> A side benefit, yet to be explored and exploited, is making the VNID locally significant. This is a huge step, IMO, as this offloads the intelligence of assigning and managing VNIDs from a central…

TOM: Yes.

	--Tom

---

From: Patrick Frejborg <[email-masked]>
Date: Wed, 19 Sep 2012 08:05:24 +0300

Hi Aldrin,

is BGP the best alternative as the control plane mechanism?

What NVO3 is trying to achieve is to setup and remove tunnels between
NVEs when VM/TES are added/moved on the NVEs. So what we really need
is "tunnel initiation protocol", right?
I believe that a SIP architecture is closer to that than BGP, which is
basically a routing protocol.

The NVE would be something similar as a SIP UA. When a VM/TES gets
added at the NVE the NVE sends an INVITE to the "conference group"
(the CUG) with its local members (MAC addresses) to the SIP signalling
system. Instead of having voice codes in the INVITE, it contain a list
of supported NVO3 "codecs", e.g. VXLAN, NVGRE; STT, MPLS, hopefully
some day PR-SCTP etc and the NVEs established tunnels between each
other, if not already established. Liveness detection of the path is
handled by the transport protocol.

Today, you can buy a device containing an advanced SIP UA stack for
less than 50$-100$, but a switch/router with an advanced BGP stack is
a lot more expensive. Thus it would interesting to visit an SIP
architecture to see what it can offer as an NVO3 control plane
solution before rushing into a BGP.

Patrick

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 19 Sep 2012 01:31:26 -0400

In the context of L2 and L3 VPNs BGP is engineered to be a pub-sub vector distribution protocol.  EVPN, IPVPN, BGP-VPLS, etc are distributed network applications that use/prefer BGP to distribute application-specific information to peers for the purpose of creating overlay virtual networks.  BGP isn't really what you pay for (you might even get it for free), it's the application that you pay for.  This is true whether you buy a centralized SDN controller running a network application or you choose a distributed solution using BGP, or applications that might use SIP.  I think people are getting too hung up on the simpler parts of the problem.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 19 Sep 2012 07:33:26 +0200

Hi Patrick,

 > is BGP the best alternative as the control plane mechanism?

I am not sure if it is the best, but clearly it is the one used widest 
today. And since data centers are not operating in vacuum it does make 
sense to use the same "signalling" mechanism as the rest of the networks 
are using for tenant isolation.

> is "tunnel initiation protocol", right?

Wrong. Tunneling has gone about 10 years ago in the industry. What it 
evolved to is encapsulation.

The significant difference is that you need to provide only sufficient 
information about encap header in order to make your application work .. 
this is opposed to establish any end to end state (what typically tunnel 
initiation protocols do).

Getting this subtle difference is the key to success.

> basically a routing protocol.

While I respect your believes years of deployment and operational 
experience with BGP based L3VPNs and L2VPNs indicate at least to me 
otherwise. Maybe I am biased but I have never head of service providers 
using SIP to provide VPNs for their customers.

> system.

While we could discuss details the NVE tenant will participate in many 
"conference groups". Those "groups" represent not only CUGs but also 
wide range of evolving services which will be real value add to the DC 
providers.

I am of the pretty strong opinion that revenue from network plumbing be 
it open pipe or VPN is over. The value add is in flexibility of service 
integration .. service being anything from appliance to big data/large 
job on demand processing. I do not see how SIP would help there.

> Thus it would interesting to visit an SIP

As said above I am not sure the "rushing" is right term. If you look at 
your requirements from operators point of view we have customers 
interconnected with L3 or L2 VPNs. Adding seamless DC to their services 
is a very important service.

SIP would be detached from reality and while it may even work in it's 
own juice I am afraid it would have zero chance for wide industry 
adoption in the data VPN space.

Best regards,
R.

---

From: Patrick Frejborg <[email-masked]>
Date: Wed, 19 Sep 2012 09:10:12 +0300

Hi Robert

On Wed, Sep 19, 2012 at 8:33 AM, Robert Raszuk <[email-masked]> wrote:
> today. And since data centers are not operating in vacuum it does make sense
> to use the same "signalling" mechanism as the rest of the networks are using
> for tenant isolation.

Somewhat true, it depends on the DC. Many enterprise DC and hosting
providers DC are disconnected from the WAN - the ISP offers the WAN
connections and the DC uses a different "signalling mechanism for
their network, such as STP ;.)
Not that many uses MPLS inside their DCs, a few though. Also the WAN
network and DC network are usually in different management domains.

> Getting this subtle difference is the key to success.

But the liveness detection of the path must be handled somehow by the
encapsulation scheme, right?
If that is the case, should that be taking care of the network or the
endpoints? By locating the NVE as far at the edge the tunneling scheme
will scale quite well depending on how many VMs you are allocating per
NVE (x86 platform) - 10, 50, 100?

> While I respect your believes years of deployment and operational experience
> with BGP based L3VPNs and L2VPNs indicate at least to me otherwise. Maybe I
> am biased but I have never head of service providers using SIP to provide
> VPNs for their customers.
>
>

My background is also from BGP/LDP based L3VPN and L2VPN and I have
some experience with SIP, trying to think out of the box

> "conference groups". Those "groups" represent not only CUGs but also wide
> range of evolving services which will be real value add to the DC providers.
>
> I am of the pretty strong opinion that revenue from network plumbing be it
> open pipe or VPN is over. The value add is in flexibility of service
> integration .. service being anything from appliance to big data/large job
> on demand processing. I do not see how SIP would help there.
>
Agree. SIP would only keep track of where the VMs are located. Turning
this around, what new services would a BGP based L3VPN and L2VPN offer
in the DC? We have been on this path for the last 10-15 years on the
WAN side and some experience with a few DCs.

> As said above I am not sure the "rushing" is right term. If you look at your
> requirements from operators point of view we have customers interconnected
> with L3 or L2 VPNs. Adding seamless DC to their services is a very important
> service.
>

Which operators do you refer to - enterprises, ISP, telcos, cloud,
hosting providers?

> SIP would be detached from reality and while it may even work in it's own
> juice I am afraid it would have zero chance for wide industry adoption in
> the data VPN space.

Maybe, but at least worth to have a discussion about.

Patrick

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 19 Sep 2012 08:46:42 +0200

Hi Patrick,

> their network, such as STP ;.)

To clarify ... by integration with WAN I meant a case of service 
provider who already offers VPN service for enterprise X to also offer 
DC services for the same enterprise X.

One could be very correct to ask if such enterprise X will not be better 
to use DC services outside of their VPN SP, but they are free to do that 
today anyway.

> Not that many uses MPLS inside their DCs, a few though.

There is huge not widely recognized difference between using MPLS for 
forwarding and using MPLS for applications as a demux tag.

While I can agree that the former is pretty much over .. too many 
architectural scaling and overhead protocol issues the latter is clearly 
IMHO as good as using any other value for demux .. be it GRE key in 
NVGRE or VXLAN.

> Also the WAN

Very true .. that's why the right interconnect model must consider such 
separation. And among most VPN interconnect models it seems that option 
B (with its A's flavor when needed) is gaining momentum today in DC 
interface to the WAN.

> encapsulation scheme, right?

Well .. not by the encapsulation itself. It could be control plane or 
data plane based. Encapsulation scheme is orthogonal.

For control plane there are number of solutions already today.

For data plane to be fair in practice there is only one - LISP.

> NVE (x86 platform) - 10, 50, 100?

Agree 100%. So while some still trying to see NVE in the network IMHO 
and maybe not so humble it seems clear that NVE should be on the host. 
Is 100 max number of VMs you can squeeze out of the host .. I think it 
really depends on your host.

> some experience with SIP, trying to think out of the box

That's always great. And while I admit I do not have any SIP experience 
other then playing with my SIP phone to get connected to the SIP gateway 
I think thinking out of the box is the most valuable feature.

> WAN side and some experience with a few DCs.

Ahhh you are touching very sensitive space on what new services model 
will be .. I think we do not know. We do not even know if new services 
will use traditional/legacy network model or new emerging one be it 
openflow based or forces based.

One thing is clear .. the NVO3 architecture should be compatible with 
any service virtualization mechanism. And again putting virtualization 
to the end hosts for tenants or for appliances seems like the easiest 
way to accomplish the compatibility with the future.

> Which operators do you refer to - enterprises, ISP, telcos, cloud,

I mean SPs/ISPs/Telcos with both VPN and internal cloud services.

Rgs,
R.

---

From: Jon Hudson <[email-masked]>
Date: Wed, 19 Sep 2012 00:21:20 -0700

Well said! Just a few anecdotal data points and an opinion or two....

On Sep 18, 2012, at 11:46 PM, Robert Raszuk <[email-masked]> wrote:

> To clarify ... by integration with WAN I meant a case of service provider who already offers VPN service for enterprise X to also offer DC services for the same enterprise X.
> 
> One could be very correct to ask if such enterprise X will not be better to use DC services outside of their VPN SP, but they are free to do that today anyway.
> 
> There is huge not widely recognized difference between using MPLS for forwarding and using MPLS for applications as a demux tag.
> 
> While I can agree that the former is pretty much over .. too many architectural scaling and overhead protocol issues the latter is clearly IMHO as good as using any other value for demux .. be it GRE key in NVGRE or VXLAN.
> 
>> Also the WAN
> Very true .. that's why the right interconnect model must consider such separation. And among most VPN interconnect models it seems that option B (with its A's flavor when needed) is gaining momentum today in DC interface to the WAN.

One result of the disconnect between the DC and WAN camps: requests from WAN customers to be able to detect and collect basic statistics on VXLAN/etc activity passing through is ever increasing.

> Well .. not by the encapsulation itself. It could be control plane or data plane based. Encapsulation scheme is orthogonal.
> 
> Agree 100%. So while some still trying to see NVE in the network IMHO and maybe not so humble it seems clear that NVE should be on the host. Is 100 max number of VMs you can squeeze out of the host .. I think it really depends on your host.

As long as the number of core per host increases, the number of VMs per host will also increase. No cap should be assumed.

> That's always great. And while I admit I do not have any SIP experience other then playing with my SIP phone to get connected to the SIP gateway I think thinking out of the box is the most valuable feature.
> 
> Ahhh you are touching very sensitive space on what new services model will be .. I think we do not know. We do not even know if new services will use traditional/legacy network model or new emerging one be it openflow based or forces based.
> 
> One thing is clear .. the NVO3 architecture should be compatible with any service virtualization mechanism. And again putting virtualization to the end hosts for tenants or for appliances seems like the easiest way to accomplish the compatibility with the future.

_yes_

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Wed, 19 Sep 2012 07:34:15 +0000

Nothing against leveraging BGP. Doing BGP at end-hosts is clearly a
problem.  XMPP is an
option, but not the only one.

Dimitri

---

From: John E Drake <[email-masked]>
Date: Wed, 19 Sep 2012 01:11:50 -0700

Lucy,

Why didn't you ask your question of the authors?  I had taken it as a given that the capability to have an EVI spanning MPLS, VXLAN, and NVGRE endpoints was a given.  If the network operator does not want to deploy this capability, that is their option.

Yours irrespectively,

John

> Lucy yong
> Sent: Tuesday, September 18, 2012 1:19 PM
> To: Kireeti Kompella
> Cc: [email-masked]
> Regarding interworking capability, Is "a given EVI can support multiple
> data plane encapsulation" equivalent to say "individual NVEs need to
> support multiple encapsulation schemas"? If one NVE only supports VXLAN
> and another NVE only supports MPLS-in-GRE, two will not able to work in
> a same EVI, is that right? Will this give more benefit than having one
> encapsulation in an EVI or make more complex?
> 
> Regards,
> Lucy
> 
> >> May be I missing something here .. but does this suggest running
> >> BGP-EVPN on the NVE that is located in the hypervisor?
> >>
> >> Dimitri
> >>
> >>>      A number of us just published this draft and wanted to bring
> it
> >>> to the
> >>> NVO3 WG's attention.  We will be presenting/discussing this draft
> at
> >>> the interim meeting this week as well, but please discuss here on
> >>> the list as well.
> >>>
> >>>      Thanks,
> >>>
> >>>      Tom, John, et al
> >>>
> >>>
> >>> has been successfully submitted by Thomas D. Nadeau and posted to
> >>> the IETF repository.
> >>>
> >>> http://www.ietf.org/internet-drafts/draft-drake-nvo3-evpn-control-
> pl
> >>> ane-00
> >>> .txt
> >>> Status:

---

From: John E Drake <[email-masked]>
Date: Wed, 19 Sep 2012 01:26:10 -0700

Precisely.

Yours irrespectively,

John

> Aldrin Isaac
> Sent: Tuesday, September 18, 2012 10:31 PM
> To: Patrick Frejborg
> Cc: Thomas Nadeau; [email-masked]; Stiliadis, Dimitrios (Dimitri)
> In the context of L2 and L3 VPNs BGP is engineered to be a pub-sub
> vector distribution protocol.  EVPN, IPVPN, BGP-VPLS, etc are
> distributed network applications that use/prefer BGP to distribute
> application-specific information to peers for the purpose of creating
> overlay virtual networks.  BGP isn't really what you pay for (you might
> even get it for free), it's the application that you pay for.  This is
> true whether you buy a centralized SDN controller running a network
> application or you choose a distributed solution using BGP, or
> applications that might use SIP.  I think people are getting too hung
> up on the simpler parts of the problem.
> 
> >> the use of XMPP.  The question is whether there is any sound
> >> technical reason (versus preferences) why leveraging BGP is
> >> problematic.  I personally haven't heard a convincing argument.
> >>
>

---

From: Patrick Frejborg <[email-masked]>
Date: Wed, 19 Sep 2012 13:24:13 +0300

Hi Robert

> forwarding and using MPLS for applications as a demux tag.

Aah, I see - that would need to compatible with the customer WAN VPN
and the DC VPN so the application tags matches. I find this useful but
it most likely would require an upgrade of switches/routers at the
branch offices.

> separation. And among most VPN interconnect models it seems that option B
> (with its A's flavor when needed) is gaining momentum today in DC interface
> to the WAN.
>

Option AB is interesting, have used it.

> Well .. not by the encapsulation itself. It could be control plane or data
> plane based. Encapsulation scheme is orthogonal.
>

Agree, LISP DDT as the control plane solution and LISP data plane as
the liveness detection mechanism. However, it might become quite
complicated

With BGP we could use the same liveness detection mechanism as in MPLS
L3VPN, which is to have the loopback of the PE (NVE) in the IGP of the
underlying network. How many many NVE /32 routes can we have in the
IGP of the underlying network, 8000, 16000? Aggregates at IGP ABRs is
not an option, AFAIK.
Or should you replace the IGP with Openflow and a centralized
controller in order to preserve FIB entries at the ToRs or simple
install ToRs with larger FIBs?
(I prefer to use IGP and IP FRR mechanisms, I'll avoid Openflow
solutions that are aiming to replace distributed routing protocols).

Can we simplify these approaches somehow?

Maybe, but it requires more research work and not sure if that work is
suitable for this WG, perhaps it should be done at IRTF instead?
The idea is to have a multipath transport protocol between the NVEs
and treat the underlying network as resource pool; Mark's presentation
changed my view on networking some time ago, see
http://www.ietf.org/proceedings/72/slides/RRG-2.pdf
Deploy SIP to keep track of where the VMs are attached to the NVO3 "fabric".
The drawback is that you have more state information at the NVE, not
for each tunnel but for each PR-SCTP association between the NVEs.
Here is scalability issue but I think SDN can help here, see my next
comment.

> Agree 100%. So while some still trying to see NVE in the network IMHO and
> maybe not so humble it seems clear that NVE should be on the host. Is 100
> max number of VMs you can squeeze out of the host .. I think it really
> depends on your host.
>

It depends on your host, your SLA and how much bandwidth you have on
your private network to move your VMs to another host when the primary
host fails. If you have a lot of DRAMs on your host it will take some
time before e.g. VMs on a 512 GB DRAM host is moved to another host.
The more VMs you put on your host to bigger pipe on your private
network you need in case of a host failure, depending of course on
your SLA.

If you have a fat DRAM host, best effort SLA (don't care how long the
VM migration takes) and a lot of VMs then you could end up with a lot
of PR-SCTP tunnels to adjacent NVEs. Here should SDN kick-in. The SDN
controller should keep track of VMs that are in the same CUG, optimize
the the placement of the VMs so that they are located within the same
rack. We might achieve the following:
- within the rack deploy legacy L2 switching, 4000 VLANs should be
enough inside the rack
- if customers deploy the common web-app-db application architecture
this traffic will stay within the rack, expensive backbone bandwidth
is preserved
- when the web-app-db traffic stays within the rack the end user
should gain a better experience since the latency is lower than having
the web-app-db resources scattered over the NVO3 backbone
- we can minimize the amount of PR-SCTP associations for the NVE3, so
having 100 VMs on the node do not necessarily mean 100 tunnels.
- NVE/PR-SCTP leverages the underlying network as a resource pool

This is an early idea, so there are a lot things to sort out.

Patrick

---

From: Lucy yong <[email-masked]>
Date: Wed, 19 Sep 2012 13:38:29 +0000

Hi Aldrin,

I understand what is proposed. 

I think supporting a multi data plane encapsulation in a mp2mp situation is much more complex than p2p situation. We can use a gateway (or say convertor) to achieve data plane interworking, which will be similar to p2p situation. This approach still allows working under one control plane. We can discuss more on this in the meeting.  

Regarding your second paragraph, I agree. To support multi-homing and prevent loop, it needs a split-horizon ID no matter which data plane to use.

Thanks,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Wed, 19 Sep 2012 13:48:51 +0000

Hi Kireeti,

Please see below.

-----Original Message-----
From: Kireeti Kompella [mailto:[email-masked]] 
Sent: Tuesday, September 18, 2012 8:26 PM
To: Lucy yong
Cc: [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Hi Lucy,

Sent from my iPad

Not "need", but "nice to have", esp. from an interworking pov.
[[LY]] using a gateway (or say a converter) can achieve this too under on a control plane.  

Clearly so.

I would guess that most DCs will use only one encap.  However, if more than one encap was used, and enough devices did multiple encaps, this might still work. 
[[LY]] If this will not be common cases, use a gateway is better. Only DC that needs to deal with multiple encaps configure a gateway. 

Stepping back, though, this is not really about interworking, but rather, leveraging a single control across many encaps.  This is especially interesting as many encaps still don't have a signaling plane. 
[[LY]] you change the aspect here. Section 4 is titled with interworking capability. However, I agree we can leveraging a single control across many encaps.  

A side benefit, yet to be explored and exploited, is making the VNID locally significant. This is a huge step, IMO, as this offloads the intelligence of assigning and managing VNIDs from a central entity to the nve. 
[[LY]] agree, this has great benefit.

Lucy

Kireeti

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 Sep 2012 09:51:33 -0400

Looking at the draft, there seems to be a very reasonable question about 
section 4.  The text starts by noting that the presence of the Tunnel 
Encapsulation attribute allows for supporting a range of tunnel 
encapsulations.  Sounds good.  It then asserts that this allows 
interoperability across the encapsualtions.  That does not seem to follow.

Normally, when we allow multiple encpsulations, we specify one as 
mandatory to implement in order to enable interoperability of the devices.
Communicating the encapsulation type does not magically enable a device 
that uses one encapsulation to communicate with a device that only 
supports some other encapsualtion.

I presume that there are steps missing in section 4.  Could you elaborate?

Yours,
Joel

On 9/19/2012 4:11 AM, John E Drake wrote:
>> technical

---

From: Lucy yong <[email-masked]>
Date: Wed, 19 Sep 2012 13:55:39 +0000

Tom,

I am not auguring if NVO3 should support different data encapsulations. I question that the proposed solution is proper in a mp2mp situation. Using a gateway for this case is much simpler, which can still be done in a single control plane.

However, in NVO3, encapsulation is done at NVE not end system, right? I don't know the intention of your second sentence.

Lucy

-----Original Message-----
From: Thomas Nadeau [mailto:[email-masked]] 
Sent: Tuesday, September 18, 2012 6:10 PM
To: Lucy yong
Cc: Kireeti Kompella; [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

	--Tom

---

From: Lucy yong <[email-masked]>
Date: Wed, 19 Sep 2012 13:58:44 +0000

John,

Sorry. My original mail is meant to authors. 

Kireeti is the first one to respond, so simple follow the discussion.

Lucy  

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Wednesday, September 19, 2012 3:12 AM
To: Lucy yong; Kireeti Kompella
Cc: [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Lucy,

Yours irrespectively,

John

---

From: Lucy yong <[email-masked]>
Date: Wed, 19 Sep 2012 14:02:57 +0000

John,

Thank you very much to help elaborate on this. My question is about text in the section 4.

Regards,
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Wednesday, September 19, 2012 8:52 AM
To: John E Drake
Cc: [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Yours,
Joel

---

From: John E Drake <[email-masked]>
Date: Wed, 19 Sep 2012 07:01:34 -0700

Joel,

>From section 4, the section you referenced in your note below:

"Note that an ingress PE must use the data plane encapsulation specified by a given egress PE in the subject MAC Advertisement or Per EVI Ethernet AD route when sending a packet to that PE.  Further, an ingress node that uses shared multicast trees for sending Broadcast and Multicast traffic must maintain distinct trees for each different encapsulation type."

Aldrin also recast this into English in his reply to Lucy:

"The imported E-VPN route will determine what the next hop entry in the EVI will look like -- whether it will have encapsulation A or encapsulation B.  That is determined by the sender of the E-VPN route.  This is like having a PPP interface and an Ethernet interface connected to the same VRF."

Yours irrespectively,

John

> From: Joel M. Halpern [mailto:[email-masked]]
> Sent: Wednesday, September 19, 2012 6:52 AM
> To: John E Drake
> Cc: [email-masked]
> Looking at the draft, there seems to be a very reasonable question
> about section 4.  The text starts by noting that the presence of the
> Tunnel Encapsulation attribute allows for supporting a range of tunnel
> > Why didn't you ask your question of the authors?  I had taken it as a
> given that the capability to have an EVI spanning MPLS, VXLAN, and
> NVGRE endpoints was a given.  If the network operator does not want to
> deploy this capability, that is their option.
> >
> > Yours irrespectively,
> >
> > John
> >
> >
> >> From: [email-masked] [mailto:[email-masked]] On Behalf
> >> Of Lucy yong
> >> Regarding interworking capability, Is "a given EVI can support
> >> multiple data plane encapsulation" equivalent to say "individual
> NVEs
> >> need to support multiple encapsulation schemas"? If one NVE only
> >> supports VXLAN and another NVE only supports MPLS-in-GRE, two will
> >> not able to work in a same EVI, is that right? Will this give more
> >> benefit than having one encapsulation in an EVI or make more
> complex?
> >>
> >> Regards,
> >> Lucy
> >>
> >> Using encapsulation tunnel between two speakers enables one speaker
> >> to send a packet to another speaker as the next-hop.
> >>>
> >> tree for BUM, it has to build two distinct sub-trees for each, which
> >> is more complex.
> >>>
> >> I think it's pretty well cooked, although I must confess a
> >> predilection for sushi.  In effect, these very capable authors saved
> >> me the trouble of writing pretty much the same draft :-)
> >>
> >> The only thing I would change is the draft name: I prefer
> >> "...-nvo3-l2- in-l3-control-plane".  Oh, and add a code point for
> STT
> >> :-)
> >>
> >> Kireeti
> >>
> >>> Cheers,
> >>> Lucy
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> From: [email-masked] [mailto:[email-masked]] On
> Behalf
> >>> Of Aldrin Isaac
> >>> http://tools.ietf.org/html/draft-marques-l3vpn-end-system-07
> >>> suggests the use of XMPP.  The question is whether there is any
> >>> sound
> >> technical
> >>>>> http://datatracker.ietf.org/doc/draft-drake-nvo3-evpn-control-
> plan
> >>>>> e
> >>>>> Htmlized:
> >>>>>       Network Virtual Overlays.  Currently this protocol is
> >>>>> defined
> >> to

---

From: John E Drake <[email-masked]>
Date: Wed, 19 Sep 2012 07:05:08 -0700

Lucy,

What exactly do you mean by 'mp2mp' and why do you think it is needed in this context? 

Yours irrespectively,

John

> Sent: Wednesday, September 19, 2012 6:56 AM
> To: Thomas Nadeau
> I am not auguring if NVO3 should support different data encapsulations.
> I question that the proposed solution is proper in a mp2mp situation.
> Using a gateway for this case is much simpler, which can still be done
> in a single control plane.
> 
> However, in NVO3, encapsulation is done at NVE not end system, right? I
> don't know the intention of your second sentence.
> 
> Lucy
> 
> 	There is definitely a requirement to do different encapsulations
> simultaneously. There is even support coming from NIC vendors to
> support multiple VMs with different encapsulations at the same time.
> 
> 	--Tom
> 
> multiple data plane encapsulation" equivalent to say "individual NVEs
> supports VXLAN and another NVE only supports MPLS-in-GRE, two will not
> able to work in a same EVI, is that right? Will this give more benefit
> than having one encapsulation in an EVI or make more complex?
> >
> > Regards,
> > Lucy
> >
> >> RFC5512 applies a case where two BGP speakers are in a BGP free
> core. Using encapsulation tunnel between two speakers enables one
> speaker to send a packet to another speaker as the next-hop.
> >>
> predilection for sushi.  In effect, these very capable authors saved me
> the trouble of writing pretty much the same draft :-)
> >
> > The only thing I would change is the draft name: I prefer "...-nvo3-
> l2-in-l3-control-plane".  Oh, and add a code point for STT :-)
> >
> > Kireeti
> >
> >> Cheers,
> >> Lucy
> >>
> >>
> >>
> >>
> >>
> suggests
> BGP-EVPN
> at the
> the
> plane-00
> plane

---

From: Lucy yong <[email-masked]>
Date: Wed, 19 Sep 2012 14:17:48 +0000

John,

First of all, I mean to say that the proposed solution is not proper in a mp2mp solution in the reply to Tom. This is my mistake.

If one egress PE supports NVGRE, one support VXLAN, and one support MPLS-GRE, then five ingress PEs that need to have tunnels to all three egress PEs will all need to support 3 encapsulation. 

Using a gateway interwork with different data plane is better on this.

Lucy

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Wednesday, September 19, 2012 9:05 AM
To: Lucy yong; Thomas Nadeau
Cc: Kireeti Kompella; [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Lucy,

Yours irrespectively,

John

---

From: John E Drake <[email-masked]>
Date: Wed, 19 Sep 2012 07:33:29 -0700

Lucy,

Or more precisely, all eight PEs each have to support all three encapsulations.

As Aldrin points out this is the same as having PPP and Ethernet encapsulations in the same VRF.  Also, as Tom points out, this is a capability that NIC vendors are designing into their products.

Yours irrespectively,

John

> From: Lucy yong [mailto:[email-masked]]
> Sent: Wednesday, September 19, 2012 7:18 AM
> To: John E Drake; Thomas Nadeau
> First of all, I mean to say that the proposed solution is not proper in
> a mp2mp solution in the reply to Tom. This is my mistake.
> 
> If one egress PE supports NVGRE, one support VXLAN, and one support
> MPLS-GRE, then five ingress PEs that need to have tunnels to all three
> egress PEs will all need to support 3 encapsulation.
> 
> > However, in NVO3, encapsulation is done at NVE not end system, right?
> > I don't know the intention of your second sentence.
> >
> > Lucy
> >
> not
> > >>>> interim meeting this week as well, but please discuss here on
> the
> > list as
> > >>>> well.
> > >>>>
> > >>>>    Thanks,
> > >>>>
> > >>>>    Tom, John, et al
> > >>>>
> > >>>>
> > >>>> http://www.ietf.org/internet-drafts/draft-drake-nvo3-evpn-
> control
> > >>>> -
> > plane-00
> > >>>> .txt
> > >>>> Status:

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 19 Sep 2012 14:38:39 +0000

John and Tom,

Regarding the title, may be more accurate to use something like "E-VPN Control Plane for Layer 2 Network Virtualized Overlays"?
Thx,
Luyuan

> NAPIERALA, MARIA H
> Sent: Tuesday, September 18, 2012 7:35 PM
> To: Kireeti Kompella
> Cc: [email-masked]; Lucy yong
> I have the same comment. The document describes control plane for layer
> 2 overlays (i.e., for transporting MAC/ethernet headers for IP
> packets). Throughout the document when "network virtualization overlay"
> is used it should be clarified that it is a layer 2 overlay.
> 
> Maria
> 
> 
> Of
> core.
> > They would have to in any case.  The tunnel encap is a couple of
> bits;
> > the "tenant id" is also needed.
> >
> is
> predilection
> l2-
> > it to the
> control-

---

From: John E Drake <[email-masked]>
Date: Wed, 19 Sep 2012 07:42:02 -0700

Fine with me.  I will add it to the list of updates for the next version.

Yours irrespectively,

John

> From: Luyuan Fang (lufang) [mailto:[email-masked]]
> Sent: Wednesday, September 19, 2012 7:39 AM
> To: NAPIERALA, MARIA H; Kireeti Kompella; John E Drake; Thomas Nadeau
> Regarding the title, may be more accurate to use something like "E-VPN
> Control Plane for Layer 2 Network Virtualized Overlays"?
> Thx,
> Luyuan
> 
> > Of NAPIERALA, MARIA H
> > > >>> http://tools.ietf.org/html/draft-drake-nvo3-evpn-control-plane-
> 0
> > > >>> 0
> > > >>>
> > > >>>
> > > >>> Abstract:

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 19 Sep 2012 14:53:55 +0000

Thx, John.

> Sent: Wednesday, September 19, 2012 4:42 PM
> To: Luyuan Fang (lufang); NAPIERALA, MARIA H; Kireeti Kompella; Thomas
> Nadeau
> > > Kireeti > [...] I would change is the draft name: I prefer "...-
> nvo3-
> > > l2-
> > > > > Does it mean using BGP to exchange NVE MAC address that belong
> to
> > > an
> > > > >>> has been successfully submitted by Thomas D. Nadeau and
> posted
> > > > >>> to
> > > > the
> > > > >>> IETF repository.
> > > > >>>

---

From: Thomas Nadeau <[email-masked]>
Date: Wed, 19 Sep 2012 11:11:01 -0400

I honestly don't care what we call it as long as the protocol inside the
document gets the job done. *)

	--Tom

---

From: Lou Berger <[email-masked]>
Date: Wed, 19 Sep 2012 11:44:41 -0400

Or just allow for & reference the IANA BGP Tunnel Encapsulation
Attribute Tunnel Types registry and get any assigned tunnel type...

Lou

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 19 Sep 2012 17:10:00 +0000

> option, but not the only one.

That's right.
Some providers are looking for a solution where network virtualization forwarding and control functions are fully decoupled, and where only the forwarding function (i.e., overlay encapsulation function) is implemented on, e.g., the application servers.

Maria

> >>>interim meeting this week as well, but please discuss here on the
> list
> >>>as
> >>>well.
> >>>
> >>>       Thanks,
> >>>
> >>>       Tom, John, et al
> >>>
> >>>
> plane-
> >>>00

---

From: Christian Martin <[email-masked]>
Date: Wed, 19 Sep 2012 10:45:02 -0400

While the N.V.O. ordering is apropos of the WG, it would be cleaner if it was called "E-VPN Control Plane for Virtualized Network Overlays."

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 19 Sep 2012 17:43:50 +0000

Presumably its possible to strip down to the bare minimum BGP required in the end hosts. Essentially one or a pair of TCP connections would be all that is required. A lot of the BGP machinery is not required in the end hosts.

Also just because its called BGP does not mean it is the same BGP instance that is used for the underlay routing. So full decoupling is not lost because the same protocol is used for both the underlay and the overlay.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of NAPIERALA, MARIA H
Sent: Wednesday, September 19, 2012 1:10 PM
To: Stiliadis, Dimitrios (Dimitri); Aldrin Isaac
Cc: Thomas Nadeau; [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

That's right.

Maria

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Wed, 19 Sep 2012 18:25:08 +0000

I would leave BGP as-is and use a simple RPC schema or Instant Messaging Protocol for switch forwarding state.

Maria

> From: AshwoodsmithPeter [mailto:[email-masked]]
> Sent: Wednesday, September 19, 2012 1:44 PM
> To: NAPIERALA, MARIA H; Stiliadis, Dimitrios (Dimitri); Aldrin Isaac
> Presumably its possible to strip down to the bare minimum BGP required
> in the end hosts. Essentially one or a pair of TCP connections would be
> all that is required. A lot of the BGP machinery is not required in the
> end hosts.
> 
> Also just because its called BGP does not mean it is the same BGP
> instance that is used for the underlay routing. So full decoupling is
> not lost because the same protocol is used for both the underlay and
> the overlay.
> 
> Peter
> 
> Some providers are looking for a solution where network virtualization
> forwarding and control functions are fully decoupled, and where only
> the forwarding function (i.e., overlay encapsulation function) is
> implemented on, e.g., the application servers.
> 
> Maria
> 
> 
> > Dimitri
> >

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Wed, 19 Sep 2012 13:28:22 -0500

John,
I think more details need to be added here. What happens if one PE advertises nvgre encap while the other advertises only vxlan? Do you allow asymmetric encapsulations? 
What if one NVE supports all 3 which one is chosen, advertised? Just a few examples....
Thanks,
Florin

On Sep 19, 2012, at 9:04 AM, John E Drake <[email-masked]> wrote:

>>>> bits; the "tenant id" is also needed.
>>>>

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Wed, 19 Sep 2012 13:33:19 -0500

This may be feasible but is it what will get deployed in the hypervisor operational environment?

We need to consider the protocols that garnered acceptance in the cloud networking space as well before deciding on which solution to push down their throat.

Florin

---

From: Shane Amante <[email-masked]>
Date: Wed, 19 Sep 2012 13:01:01 -0600

> I am not auguring if NVO3 should support different data encapsulations. I question that the proposed solution is proper in a mp2mp situation. Using a gateway for this case is much simpler, which ca…

Actually, I strongly disagree with the assertion that "using a gateway for this case is much simpler".  We (operators) build networks to scale by pushing as much functionality to the EDGE of the network, as possible.  If you mandate a GW be used as a converter that becomes a capacity planning headache, you have to worry about it's resiliency (fail-over), etc.  Do not go there.

-shane

---

From: Vishwas Manral <[email-masked]>
Date: Wed, 19 Sep 2012 12:03:53 -0700

Hi Florin,

I agree with the concerns you raise.

In my view in a DC there would be one or 2 encapsulations in the worst case
(Server-to-server and server to others) used at any point of time. This
could be as a parallel to how we have different networks in HPC.

Thanks,
Vishwas

On Wed, Sep 19, 2012 at 11:28 AM, Balus, Florin Stelian (Florin) <
[email-masked]> wrote:

> John,
> I think more details need to be added here. What happens if one PE
> advertises nvgre encap while the other advertises only vxlan? Do you allow
> asymmetric encapsulations?
> > "Note that an ingress PE must use the data plane encapsulation specified
> by a given egress PE in the subject MAC Advertisement or Per EVI Ethernet
> AD route when sending a packet to that PE.  Further, an ingress node that
> uses shared multicast trees for sending Broadcast and Multicast traffic
> must maintain distinct trees for each different encapsulation type."
> >
> > "The imported E-VPN route will determine what the next hop entry in the
> EVI will look like -- whether it will have encapsulation A or encapsulation
> B.  That is determined by the sender of the E-VPN route.  This is like
> having a PPP interface and an Ethernet interface connected to the same VRF."
> >
> > Yours irrespectively,
> >
> > John
> >

---

From: Vishwas Manral <[email-masked]>
Date: Wed, 19 Sep 2012 12:09:22 -0700

Hi Shane,

I agree.

Gateways to convert from one encaps to other may not be an ideal solution,
though the edge devices currently do some form of gateway functionality for
most purposes.

Thanks,
Vishwas

On Wed, Sep 19, 2012 at 12:01 PM, Shane Amante <[email-masked]>wrote:

> I question that the proposed solution is proper in a mp2mp situation. Using
> a gateway for this case is much simpler, which can still be done in a
> single control plane.
>
> Actually, I strongly disagree with the assertion that "using a gateway for
> this case is much simpler".  We (operators) build networks to scale by
> pushing as much functionality to the EDGE of the network, as possible.  If
> you mandate a GW be used as a converter that becomes a capacity planning
> headache, you have to worry about it's resiliency (fail-over), etc.  Do not
> go there.
>
> -shane
>
>
> simultaneously. There is even support coming from NIC vendors to support
> multiple VMs with different encapsulations at the same time.
> >
> >       --Tom
> >
> support multiple encapsulation schemas"? If one NVE only supports VXLAN and
> another NVE only supports MPLS-in-GRE, two will not able to work in a same
> EVI, is that right? Will this give more benefit than having one
> Using encapsulation tunnel between two speakers enables one speaker to send
> a packet to another speaker as the next-hop.
> >>>
> because any pair of NVEs in an NVO will need to maintain a state for the
> tunnel encapsulation.
> >>
> >>> If some NVEs support VXLAN and some support NVGRE, to build mcast tree
> for BUM, it has to build two distinct sub-trees for each, which is more
> complex.
> >>>
> for sushi.  In effect, these very capable authors saved me the trouble of
> writing pretty much the same draft :-)
> >>
> "...-nvo3-l2-in-l3-control-plane".  Oh, and add a code point for STT :-)
> >>
> >> Kireeti
> >>
> >>> Cheers,
> >>> Lucy
> >>>
> >>>
> >>>
> >>>
> >>>

---

From: Thomas Nadeau <[email-masked]>
Date: Wed, 19 Sep 2012 18:51:18 -0400

On 9/19/12 10:10 AM, "NAPIERALA, MARIA H" <[email-masked]> wrote:

>forwarding and control functions are fully decoupled, and where only the
>forwarding function (i.e., overlay encapsulation function) is implemented
>on, e.g., the application servers.

	I am confused. Why do you not think that control and data plane are not
separated in this solution?

	--Tom

---

From: Thomas Nadeau <[email-masked]>
Date: Wed, 19 Sep 2012 16:17:45 -0700

> What if one NVE supports all 3 which one is chosen, advertised? Just a few examples....

	That is just not how data centers are built today so that is unlikely to happen in the wild. With that in mind, this is an interesting corner case that we should handle just in case something is misconfigured or someone in the future decides to build such a DC.

	--Tom

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Wed, 19 Sep 2012 19:40:28 -0500

> Sent: Wednesday, September 19, 2012 4:18 PM
> To: Balus, Florin Stelian (Florin)
> Cc: John E Drake; Joel M. Halpern; [email-masked]
> advertises nvgre encap while the other advertises only vxlan? Do you
> allow asymmetric encapsulations?
> 	That is just not how data centers are built today 

[FB>] Agreed there are not too many NVO3 DCs today let alone with multiple encapsulations.

> so that is
> unlikely to happen in the wild. 

[FB>] If we talked about the future a combination of two encapsulations could be very likely in my opinion: regular EVPN and one of the VXLAN/NVGRE at the NVE GW. 

> With that in mind, this is an
> interesting corner case that we should handle just in case something is
> misconfigured or someone in the future decides to build such a DC.

[FB>] Definitely the case of multiple encaps needs to be addressed even if it is just for the mis-configuration case.
> >> "Note that an ingress PE must use the data plane encapsulation
> specified by a given egress PE in the subject MAC Advertisement or Per
> EVI Ethernet AD route when sending a packet to that PE.  Further, an
> ingress node that uses shared multicast trees for sending Broadcast and
> Multicast traffic must maintain distinct trees for each different
> encapsulation type."
> >>
> >> "The imported E-VPN route will determine what the next hop entry in
> the EVI will look like -- whether it will have encapsulation A or
> This is like having a PPP interface and an Ethernet interface connected
> to the same VRF."
> >>
> >> Yours irrespectively,
> >>
> >> John
> >>
> >>> Tunnel Encapsulation attribute allows for supporting a range of
> >>> tunnel encapsulations.  Sounds good.  It then asserts that this
> >>> allows interoperability across the encapsualtions.  That does not
> >>> seem to follow.
> >>>
> >>> Communicating the encapsulation type does not magically enable a
> >>> device that uses one encapsulation to communicate with a device
> that
> >>> only supports some other encapsualtion.
> >>>
> >>>> Why didn't you ask your question of the authors?  I had taken it
> as
> >>>> a
> >>> NVGRE endpoints was a given.  If the network operator does not want
> >>> to deploy this capability, that is their option.
> >>>>
> >>>> Yours irrespectively,
> >>>>
> >>>> John
> >>>>
> >>>>
> >>>>> Behalf Of Lucy yong
> >>>>> Using encapsulation tunnel between two speakers enables one
> >>>>> because any pair of NVEs in an NVO will need to maintain a state
> >>>>> for the tunnel encapsulation.
> >>>>>
> >>>>> tree for BUM, it has to build two distinct sub-trees for each,
> >>>>> which is more complex.
> >>>>>>
> >>>>> predilection for sushi.  In effect, these very capable authors
> >>>>> saved me the trouble of writing pretty much the same draft :-)
> >>>>>
> >>>>>>>> the interim meeting this week as well, but please discuss here
> >>>>>>>> on the list as well.
> >>>>>>>>
> >>>>>>>>     Thanks,
> >>>>>>>>
> >>>>>>>>     Tom, John, et al
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> A new version of I-D,
> >>>>>>>> draft-drake-nvo3-evpn-control-plane-00.txt
> >>>>>>>> has been successfully submitted by Thomas D. Nadeau and posted
> >>>>>>>> to the IETF repository.
> >>>>>>>>

---

From: Kireeti Kompella <[email-masked]>
Date: Wed, 19 Sep 2012 17:46:38 -0700

Hi Tom,

> 	That is just not how data centers are built today so that is unlikely to happen in the wild. With that in mind, this is an interesting corner case that we should handle just in case something is m…

As I've said, I like this draft.  However, "interworking" is fraught with misinterpretations and pitfalls, and perhaps at this stage distracts from other more pressing concerns.

Might I suggest the following reworking of Section 4:

4.  Multiple Encapsulations

    The Tunnel Encapsulation attribute enables a single control plane
    to oversee a number of different data plane encapsulations.  This can
    manifest itself in several ways:

    a) a data center may use a single common encapsulation for all EVIs, but
         different data centers may make independent choices.
    b) within a single data center, a given EVI may use a single
         encapsulation, but different EVIs may choose different encapsulations.
    c) a single EVI may use multiple encapsulations, one for each PE-PE pair,
         and maybe even use a different encapsulation in each direction.

    Going from (a) to (c ) increases generality, but also increases complexity.
    The initial focus will be on (a) and (b); further details for (c ) will be added if
    there is sufficient interest.

    In all cases, a PE within a given EVI knows which encapsulations other 
    PEs in that EVI support, and, when sending unicast traffic, it MUST choose
    one of the encapsulations advertised by the egress PE.

    For case (c ), an ingress PE that uses shared multicast trees for sending 
    Broadcast and Multicast traffic must maintain distinct trees for each
    different encapsulation type.  Further details will be given in a future version.

    The topic of interworking encapsulations and "gateway" functions will also be
    addressed in a future version.

Kireeti.

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 19 Sep 2012 21:33:49 -0400

Lucy, 

I'm not sure we want to go down the complicated road of segmented transport tunnels.  That seems to be a lot more work for the control plane and network designers.  I think that might also head us in the opposite direction of the pooling principle :).  Isn't it less expensive if the NVE supports different encaps versus adding complexity into the CP?

Best regards -- aldrin

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 19 Sep 2012 21:37:23 -0400

Yes. Exactly.  The RR could also only send BGP encaps routes to the NVE only for next hops associated to VPN routes that the NVE is importing.  This will reduce the amount of next-hop info that the NVE needs to maintain.  It is also not necessary for NVE to be part of the IGP in this model.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 Sep 2012 23:19:07 -0400

Thank you Kireeti.  That addresses my concerns.

Yours,

Joel

---

From: Jon Hudson <[email-masked]>
Date: Thu, 20 Sep 2012 00:12:03 -0400

However it is very common to see multiple hypervisors in use (esx, hyper-v, xen, kvm)

People will choose the hypervisors first, then choose from available supported encapsulation methods.

And while we could see those four end up using the same encapsulation method I think is more likely that at least two will remain from the pack of VXLAN/NVGRE/STT and now DOVE.

We must also keep in mind that while the "public cloud" providers may make more logical measure choices, all the "private cloud" shops will be wanting to build overlays between the two. Encapsulation choices may end up being more fragmented in the private space.

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Thu, 20 Sep 2012 07:39:52 +0000

+1 

Even when public clouds are build with only one encap, they will have to
support
interoperability with multiple enterprise clouds that will use their own
encaps
since they use different virtualization stacks. Translations are
unavoidable.

On 9/19/12 9:12 PM, "Jon Hudson" <[email-masked]> wrote:

>And while we could see those four end up using the same encapsulation
>method I think is more likely that at least two will remain from the pack
>of VXLAN/NVGRE/STT and now DOVE.
>
>We must also keep in mind that while the "public cloud" providers may
>make more logical measure choices, all the "private cloud" shops will be
>wanting to build overlays between the two. Encapsulation choices may end
>up being more fragmented in the private space.
>
>> [FB>] If we talked about the future a combination of two encapsulations
>>could be very likely in my opinion: regular EVPN and one of the
>>VXLAN/NVGRE at the NVE GW.
>> 
>> [FB>] Definitely the case of multiple encaps needs to be addressed even
>>if it is just for the mis-configuration case.
>>> 
>>>    --Tom
>>> 
>>> 
>>>> Thanks,
>>>> Florin
>>>>

---

From: John E Drake <[email-masked]>
Date: Thu, 20 Sep 2012 02:20:50 -0700

Florin,

The current text explicitly calls for asymmetric encapsulation.  If an egress PE advertises multiple encapsulations, the ingress PE is free to choose among them.  I would suggest that you send proposed text.

Yours irrespectively,

John

> From: Balus, Florin Stelian (Florin) [mailto:florin.balus@alcatel-
> lucent.com]
> Sent: Wednesday, September 19, 2012 11:28 AM
> To: John E Drake
> Cc: Joel M. Halpern; [email-masked]
> >> device that uses one encapsulation to communicate with a device that
> speaker

---

From: John E Drake <[email-masked]>
Date: Thu, 20 Sep 2012 02:22:34 -0700

I have seen emails suggesting that adding protocols to hypervisor implementations is considered to be business as usual.

Yours irrespectively,

John

> Balus, Florin Stelian (Florin)
> Sent: Wednesday, September 19, 2012 11:33 AM
> To: AshwoodsmithPeter
> Cc: Thomas Nadeau; Aldrin Isaac; Stiliadis, Dimitrios (Dimitri);
> NAPIERALA, MARIA H; [email-masked]
> We need to consider the protocols that garnered acceptance in the cloud
> networking space as well before deciding on which solution to push down
> their throat.
> 
> Florin
> 
> 
> > Presumably its possible to strip down to the bare minimum BGP
> required in the end hosts. Essentially one or a pair of TCP connections
> would be all that is required. A lot of the BGP machinery is not
> required in the end hosts.
> >
> >> problem.  XMPP is an option, but not the only one.
> >
> > That's right.
> > Some providers are looking for a solution where network
> virtualization forwarding and control functions are fully decoupled,
> and where only the forwarding function (i.e., overlay encapsulation
> function) is implemented on, e.g., the application servers.
> >
> > Maria
> >
> >
> >> Dimitri
> >>
> >>> sound technical reason (versus preferences) why leveraging BGP is

---

From: John E Drake <[email-masked]>
Date: Thu, 20 Sep 2012 03:12:02 -0700

I had an offline discussion with Joel and he suggests using the term 'encapsulation selection' rather than 'interworking'

Yours irrespectively,

John

> Sent: Wednesday, September 19, 2012 5:47 PM
> To: Thomas Nadeau
> Cc: Kireeti Kompella; Balus, Florin Stelian (Florin); John E Drake;
> Joel M. Halpern; [email-masked]
> > On Sep 19, 2012:11:28 AM, at 11:28 AM, "Balus, Florin Stelian
> (Florin)" <[email-masked]> wrote:
> >
> >> John,
> > 	That is just not how data centers are built today so that is
> unlikely to happen in the wild. With that in mind, this is an
> As I've said, I like this draft.  However, "interworking" is fraught
> with misinterpretations and pitfalls, and perhaps at this stage
> distracts from other more pressing concerns.
> 
> >>>> about section 4.  The text starts by noting that the presence of
> >>>> the Tunnel Encapsulation attribute allows for supporting a range
> of
> >>>> that only supports some other encapsualtion.
> >>>>
> >>>>> as a
> >>>>>> supports VXLAN and another NVE only supports MPLS-in-GRE, two
> >>>>>> will not able to work in a same EVI, is that right? Will this
> >>>>>> give more benefit than having one encapsulation in an EVI or
> make
> >>>>>> more
> >>>> complex?
> >>>>>>
> >>>>>> Regards,
> >>>>>> Lucy
> >>>>>>
> >>>>>>> reason (versus preferences) why leveraging BGP is problematic.
> >>>>>>> I personally haven't heard a convincing argument.
> >>>>>>>
> >>>>>>>> May be I missing something here .. but does this suggest
> >>>>>>>> running BGP-EVPN on the NVE that is located in the hypervisor?
> >>>>>>>>
> >>>>>>>> Dimitri
> >>>>>>>>
> >>>>>>>>> http://tools.ietf.org/html/draft-drake-nvo3-evpn-control-
> plane
> >>>>>>>>> -00
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Abstract:

---

From: Lucy yong <[email-masked]>
Date: Thu, 20 Sep 2012 12:38:10 +0000

John and Tom,

Yes, your answer is more precisely.

Then I question which following is the purpose of this draft? 

1) to allow evpn control plane supporting all data plane encapsulations that nvo3 has to support
2) because all NVEs can support a set of data plane encapsulations, an egress NVE can use evpn control plane to tell other ingress NVEs which encapsulation they should use to send packets.
3) Since some NVEs may support one type data plane encapsulation and others may support another only, evpn control plane can make them interwork together.

I know there are several proposals on data plane encapsulation for nvo3, is there a requirement to support all of them in nvo3?

Lucy

Please let me know your 

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Wednesday, September 19, 2012 9:33 AM
To: Lucy yong; Thomas Nadeau
Cc: Kireeti Kompella; [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Lucy,

Yours irrespectively,

John

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 20 Sep 2012 08:56:58 -0400

Generically when we discuss the need for different forms of NVE to
communicate, wouldnt we describe that as a need to interwork them?

On Thursday, September 20, 2012, John E Drake wrote:

> I had an offline discussion with Joel and he suggests using the term
> 'encapsulation selection' rather than 'interworking'
>
> Yours irrespectively,
>
> John
>
> > From: Kireeti Kompella [mailto:[email-masked] <javascript:;>
> ]
> > Joel M. Halpern; [email-masked] <javascript:;>
> > On Sep 19, 2012, at 4:17 PM, Thomas Nadeau <[email-masked]<javascript:;>
> >
> > wrote:
> >
> > >
> > (Florin)" <[email-masked] <javascript:;>> wrote:
> > >
> > >> John,
> > >> On Sep 19, 2012, at 9:04 AM, John E Drake <[email-masked]<javascript:;>
> >
> > wrote:
> > >>
> > >>> Joel,
> > >>>
> > >>>> From: Joel M. Halpern [mailto:[email-masked] <javascript:;>]
> > >>>> Cc: [email-masked] <javascript:;>
> > >>>>>> From: [email-masked] <javascript:;> [mailto:
> [email-masked] <javascript:;>] On
> > >>>>>> Behalf Of Lucy yong
> > >>>>>> Cc: [email-masked] <javascript:;>
> > >>>>>> From: Kireeti Kompella [mailto:[email-masked]<javascript:;>
> ]
> > >>>>>> Cc: [email-masked] <javascript:;>
> > >>>>>> saved me the trouble of writing pretty much th> >>>>>>>
> > >>>>>>> From: > >>>>>>> Of Aldrin Isaac
> >

---

From: John E Drake <[email-masked]>
Date: Thu, 20 Sep 2012 05:56:56 -0700

Lucy,

1) & 2) are correct.  3) does not apply because we are stipulating that that encapsulation selection (nee interworking) requires that every PE in a given EVI needs to support all of the encapsulations used by each of the PEs in that EVI.

As others have pointed out, in practice a given EVI will only use one or two encapsulations and supporting multiple encapsulations does not appear to be particularly onerous.

Yours irrespectively,

John

> Sent: Thursday, September 20, 2012 5:38 AM
> To: John E Drake; Lucy yong; Thomas Nadeau
> 2) because all NVEs can support a set of data plane encapsulations, an
> egress NVE can use evpn control plane to tell other ingress NVEs which
> encapsulation they should use to send packets.
> 3) Since some NVEs may support one type data plane encapsulation and
> others may support another only, evpn control plane can make them
> interwork together.
> 
> I know there are several proposals on data plane encapsulation for
> nvo3, is there a requirement to support all of them in nvo3?
> 
> Lucy
> 
> Please let me know your
> 
> As Aldrin points out this is the same as having PPP and Ethernet
> encapsulations in the same VRF.  Also, as Tom points out, this is a
> capability that NIC vendors are designing into their products.
> 
> Yours irrespectively,
> 
> John
> 
> 
> > First of all, I mean to say that the proposed solution is not proper
> > in a mp2mp solution in the reply to Tom. This is my mistake.
> >
> > > On Sep 18, 2012:1:18 PM, at 1:18 PM, Lucy yong
> > > <[email-masked]>
> > > wrote:
> > >
> > > > Hi Kreeti,
> > > >
> > > NVEs need to support multiple encapsulation schemas"? If one NVE
> > > only supports VXLAN and another NVE only supports MPLS-in-GRE, two
> > > will
> > not
> which
> > > >> Multicast packet to a given Ethernet Segment.  I.e., if the
> > > >> sender MAC address is associated with a given Ethernet Segment,
> > > >> the
> > egress
> saved
> I
> running

---

From: John E Drake <[email-masked]>
Date: Thu, 20 Sep 2012 05:58:52 -0700

Aldrin,

That's what I thought but Joel seemed adamant.  I am happy to use either term.

Yours irrespectively,

John

From: Aldrin Isaac [mailto:[email-masked]]
Sent: Thursday, September 20, 2012 5:57 AM
To: John E Drake
Cc: Kireeti Kompella; Thomas Nadeau; [email-masked]; Balus, Florin Stelian (Florin); Joel M. Halpern
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Generically when we discuss the need for different forms of NVE to communicate, wouldnt we describe that as a need to interwork them?

Yours irrespectively,

John

> From: Kireeti Kompella [mailto:[email-masked]<javascript:;>]
> Joel M. Halpern; [email-masked]<javascript:;>
> On Sep 19, 2012, at 4:17 PM, Thomas Nadeau <[email-masked]<javascript:;>>
> wrote:
>
> >
> (Florin)" <[email-masked]<javascript:;>> wrote:
> >
> >> John,
> >> On Sep 19, 2012, at 9:04 AM, John E Drake <[email-masked]<javascript:;>>
> wrote:
> >>
> >>> Joel,
> >>>
> >>>> From: Joel M. Halpern [mailto:[email-masked]<javascript:;>]
> >>>> Cc: [email-masked]<javascript:;>
> >>>>>> From: [email-masked]<javascript:;> [mailto:[email-masked]<javascript:;>] On
> >>>>>> Behalf Of Lucy yong
> >>>>>> Cc: [email-masked]<javascript:;>
> >>>>>> From: Kireeti Kompella [mailto:[email-masked]<javascript:;>]
> >>>>>> Cc: [email-masked]<javascript:;>
> >>>>>> On Sep 17, 2012, at 3:36 PM, Lucy yong <[email-masked]<mailto:[email-masked]>>
> wrote:
> >>>>>>
> >>>>>>> Read this draft.
> >>>>>>>
> >>>> sender
> >>>>>> saved me the trouble of writing pretty much th> >>>>>>> -----Original Message-----

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 20 Sep 2012 09:14:30 -0400

Ok.  But from a use case perspective it's inter working.  Plus I can't
change my use case slides at this point for today's use cases presentation.

On Thursday, September 20, 2012, John E Drake wrote:

> Aldrin,****
>
> ** **
>
> That’s what I thought but Joel seemed adamant.  I am happy to use either
> term.****
>
> ** **
>
> Yours irrespectively,****
>
> ** **
>
> John****
>
> ** **
>
> *From:* Aldrin Isaac [mailto:[email-masked] <javascript:_e({},
> 'cvml', [email-masked]');>]
> *Sent:* Thursday, September 20, 2012 5:57 AM
> *To:* John E Drake
> *Cc:* Kireeti Kompella; Thomas Nadeau; [email-masked] <javascript:_e({},
> 'cvml', [email-masked]');>; Balus, Florin Stelian (Florin); Joel M.
> Halpern
> *Subject:* Re: [nvo3] draft-drake-nvo3-evpn-control-plane****
>
> ** **
>
> On Thursday, September 20, 2012, John E Drake wrote:****
>
> > encapsulation B.  That is determined by the sender of t
>

---

From: John E Drake <[email-masked]>
Date: Thu, 20 Sep 2012 06:21:30 -0700

Why don't Joel and you have a Vulcan mind meld?

Yours irrespectively,

John

From: Aldrin Isaac [mailto:[email-masked]]
Sent: Thursday, September 20, 2012 6:15 AM
To: John E Drake
Cc: Kireeti Kompella; Thomas Nadeau; [email-masked]; Balus, Florin Stelian (Florin); Joel M. Halpern
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Ok.  But from a use case perspective it's inter working.  Plus I can't change my use case slides at this point for today's use cases presentation.

Yours irrespectively,

John

From: Aldrin Isaac [mailto:[email-masked]<javascript:_e(%7b%7d,%20'cvml',[email-masked]');>]
Sent: Thursday, September 20, 2012 5:57 AM
To: John E Drake
Cc: Kireeti Kompella; Thomas Nadeau; [email-masked]<javascript:_e(%7b%7d,%20'cvml',[email-masked]');>; Balus, Florin Stelian (Florin); Joel M. Halpern
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Yours irrespectively,

John

> Joel M. Halpern; [email-masked]<mailto:[email-masked]>
> On Sep 19, 2012, at 4:17 PM, Thomas Nadeau <[email-masked]<mailto:[email-masked]>>
> wrote:
>
> >
> (Florin)" <[email-masked]<mailto:[email-masked]>> wrote:
> >
> >> John,
> >> On Sep 19, 2012, at 9:04 AM, John E Drake <[email-masked]<mailto:[email-masked]>>
> wrote:
> >>
> >>> Joel,
> >>>

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 20 Sep 2012 06:35:29 -0700

Jon,

> method I think is more likely that at least two will remain from
> the pack of VXLAN/NVGRE/STT and now DOVE.
> 
> We must also keep in mind that while the "public cloud" providers
> may make more logical measure choices, all the "private cloud" shops
> will be wanting to build overlays between the two. Encapsulation
> choices may end up being more fragmented in the private space.

Does it mean that all of these encapsulation choices should become
IETF Standard ?

Yakov.

> > [FB>] If we talked about the future a combination of two encapsulations 
could be very likely in my opinion: regular EVPN and one of the VXLAN/NVGRE 
at the NVE GW.

---

From: Lucy yong <[email-masked]>
Date: Thu, 20 Sep 2012 14:22:00 +0000

John,

See below.

-----Original Message-----
From: John E Drake [mailto:[email-masked]] 
Sent: Thursday, September 20, 2012 7:57 AM
To: Lucy yong; Thomas Nadeau
Cc: Kireeti Kompella; [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Lucy,

As others have pointed out, in practice a given EVI will only use one or two encapsulations and supporting multiple encapsulations does not appear to be particularly onerous.
[[LY]] So some description in the draft need to be revised or clarified.

Yours irrespectively,

John

---

From: John E Drake <[email-masked]>
Date: Thu, 20 Sep 2012 07:23:57 -0700

It's on my list.

Yours irrespectively,

John

> Sent: Thursday, September 20, 2012 7:22 AM
> 1) & 2) are correct.  3) does not apply because we are stipulating that
> that encapsulation selection (nee interworking) requires that every PE
> in a given EVI needs to support all of the encapsulations used by each
> of the PEs in that EVI.
> 
> As others have pointed out, in practice a given EVI will only use one
> or two encapsulations and supporting multiple encapsulations does not
> appear to be particularly onerous.
> > > > >> On Mon, Sep 17, 2012 at 12:11 PM, Stiliadis, Dimitrios
> > > > >> (Dimitri) <[email-masked]> wrote:
> > > > >>>> http://datatracker.ietf.org/doc/draft-drake-nvo3-evpn-
> control
> > > > >>>> -
> > > > plane
> > > > >>>> Htmlized:

---

From: Jon Hudson <[email-masked]>
Date: Thu, 20 Sep 2012 10:50:42 -0400

UG, I sure hope not.

We already see some alignment in these already in so far as DOVE uses the VXLAN encap header.

My personal belief is that two will become obvious leaders. 

There is no easy answer

---

From: Lucy yong <[email-masked]>
Date: Thu, 20 Sep 2012 15:02:54 +0000

I like this proposed text.
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Wednesday, September 19, 2012 7:47 PM
To: Thomas Nadeau
Cc: Joel M. Halpern; Balus, Florin Stelian (Florin); [email-masked]; Kireeti Kompella; John E Drake
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Hi Tom,

Kireeti.

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 20 Sep 2012 15:08:39 +0000

Tom,

> separated in this solution?

Because the draft proposes that NVE implements both the control plane and data plane functions of E-VPN PE. 

Maria

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 20 Sep 2012 08:23:32 -0700

Maria,

> data plane functions of E-VPN PE. 

Do you think that an NVE that implements only XMPP has no control plane
at all ?

Yakov.

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 20 Sep 2012 15:41:44 +0000

> Do you think that an NVE that implements only XMPP has no control plane

It does not implement the (BGP) control plane of the overlay (e.g., route selection should be done on a controller and not on the NVE), and it should not directly participate in any other routing protocols.

Maria

---

From: Thomas Nadeau <[email-masked]>
Date: Thu, 20 Sep 2012 09:26:21 -0700

That seems to hit the nail on the head.  I don't think we want to deal with inter-working in the classic sense. 

	--Tom

On Sep 20, 2012:3:12 AM, at 3:12 AM, John E Drake <[email-masked]> wrote:

>> (Dimitri)

---

From: Christian Martin <[email-masked]>
Date: Thu, 20 Sep 2012 12:55:05 -0400

Agreed.  I would say, however, that interworking should be a topic of interest for this WG.

Cheers
Chris

---

From: Thomas Nadeau <[email-masked]>
Date: Thu, 20 Sep 2012 19:22:36 -0400

Maria,

	The only issue that is being raised seems to be one of which control
plane to run, not whether or not we need one. I think everyone agrees on
that.  In the way of BGP versus XMPP, perhaps you could elaborate why you
think BGP is a bad choice?

	--Tom

On 9/20/12 8:41 AM, "NAPIERALA, MARIA H" <[email-masked]> wrote:

>It does not implement the (BGP) control plane of the overlay (e.g., route
>selection should be done on a controller and not on the NVE), and it
>should not directly participate in any other routing protocols.
>
>Maria

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 21 Sep 2012 01:59:37 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Aldrin
> Isaac
> 发送时间: 2012年9月19日 5:24
> 收件人: Lucy yong
> 抄送: Kireeti Kompella; [email-masked]
> 主题: Re: [nvo3] draft-drake-nvo3-evpn-control-plane
> 

If the SH field is just intended for supporting active-active multi-homing on a per VPN instance basis, I suggest it'd better for us to seriously evaluate whether the cost for realizing that goal is worthwhile since that will make the solution itself, especially the data plane process, much more complex. In the multi-tenant cloud data center environment, isn't it good enough in practice to support active-standby multi-homing on a per VPN instance basis while supporting active-active multi-homing on a per physical interface basis? For instance, a physical server containing two VMs (say VM1 and VM2) which belong to different VPN instances respectively is dual-homed to two NVEs (say NVE1 and NVE2), NVE1 is the active NVE for VM1 and the standby NVE for VM2 while NVE2 is the active NVE for VM2 and the standby NVE for VM1.

Best regards,
Xiaohu

> > Regarding interworking capability, Is "a given EVI can support multiple data
> plane encapsulation" equivalent to say "individual NVEs need to support
> multiple encapsulation schemas"? If one NVE only supports VXLAN and another
> NVE only supports MPLS-in-GRE, two will not able to work in a same EVI, is that
> right? Will this give more benefit than having one encapsulation in an EVI or
> make more complex?
> >
> > Regards,
> > Lucy
> >
> >> Using this approach in nvo3 may rise a high scalability concern because any
> pair of NVEs in an NVO will need to maintain a state for the tunnel
> encapsulation.
> >
> for BUM, it has to build two distinct sub-trees for each, which is more complex.
> >>
> > I think it's pretty well cooked, although I must confess a predilection for sushi.
> In effect, these very capable authors saved me the trouble of writing pretty
> much the same draft :-)
> >

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 22 Sep 2012 09:02:28 +0000

Hi all,

I strongly believe this is an very important question which needs more attentions and discussions. Hence I redefine the subject of the original email so as to avoid this email from being missed.

Best regards,
Xiaohu 

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> Xuxiaohu
> 发送时间: 2012年9月21日 10:00
> 收件人: Aldrin Isaac; Lucy yong
> 主题: [nvo3] 答复: draft-drake-nvo3-evpn-control-plane
> 
> 
> 
> > -----邮件原件-----
> If the SH field is just intended for supporting active-active multi-homing on a per
> VPN instance basis, I suggest it'd better for us to seriously evaluate whether
> the cost for realizing that goal is worthwhile since that will make the solution
> itself, especially the data plane process, much more complex. In the
> multi-tenant cloud data center environment, isn't it good enough in practice to
> support active-standby multi-homing on a per VPN instance basis while
> supporting active-active multi-homing on a per physical interface basis? For
> instance, a physical server containing two VMs (say VM1 and VM2) which
> belong to different VPN instances respectively is dual-homed to two NVEs (say
> NVE1 and NVE2), NVE1 is the active NVE for VM1 and the standby NVE for VM2
> while NVE2 is the active NVE for VM2 and the standby NVE for VM1.
> 
> Best regards,
> Xiaohu
> 
> send
> tree

---

From: Jon Hudson <[email-masked]>
Date: Sat, 22 Sep 2012 02:15:39 -0700

If we don't do it as part of the standard, vendors will just implement
proprietary versions.

active/passive anything is not welcomed in the datacenter anymore

On Sat, Sep 22, 2012 at 2:02 AM, Xuxiaohu <[email-masked]> wrote:

> Hi all,
>
> I strongly believe this is an very important question which needs more
> attentions and discussions. Hence I redefine the subject of the original
> email so as to avoid this email from being missed.
>
> Best regards,
> Xiaohu
>
> > -----邮件原件-----
> > Aldrin
> multiple data
> > > NVE only supports MPLS-in-GRE, two will not able to work in a same
> EVI, is
> > that
> because
> > any
> predilection for
> > sushi.
> > > >> http://tools.ietf.org/html/draft-marques-l3vpn-end-system-07suggests
> it to
> defined to

-- 
"Do not lie. And do not do what you hate."

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 22 Sep 2012 10:10:55 +0000

Yes, if the active-active multi-homing on a per L2VPN instance basis was desired in some cases, vendors could offer their proprietary schemes like Cisco’s VSS to support that capability. In this way, there is no need to introduce the complexity into the standard, especially on the data plane perspective, and therefore other NVEs (i.e., PE routers) which are not within any active-active redundancy group don’t need to afford that complexity anymore.

As for active-passive multi-homing, I fully agree with you that the active/passive multi-homing on a per physical interface or trunk interface basis is not welcomed.  Hence we could achieve load-balancing among the physical interfaces or trunk interfaces which are connected to different NVEs of the same RG (redundancy group) on the basis of VLANs. For example, a dual-homed device (DHD) is multi-homed to NVE1 and NVE2 and it is configured with two VLANs (say VLAN x and y) on its physical NICs in a bond, NVE1’s corresponding interface is active for VLAN x while NVE2’s corresponding interface is active for VLAN y…

Best regards,
Xiaohu
-
发件人: Jon Hudson [mailto:[email-masked]]
发送时间: 2012年9月22日 17:16
收件人: Xuxiaohu
抄送: Aldrin Isaac; Lucy yong; Kireeti Kompella; [email-masked]
主题: Re: [nvo3] Is the cost of supporting active-active multi-homing on a per L2VPN instance basis sustainable?

If we don't do it as part of the standard, vendors will just implement proprietary versions.

On Sat, Sep 22, 2012 at 2:02 AM, Xuxiaohu <[email-masked]<mailto:[email-masked]>> wrote:
Hi all,

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] 代表
> Xuxiaohu
> 发送时间: 2012年9月21日 10:00
> 抄送: Kireeti Kompella; [email-masked]<mailto:[email-masked]>
> > 发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] 代表
> Aldrin
> > Isaac
> > 发送时间: 2012年9月19日 5:24
> > 收件人: Lucy yong
> > 抄送: Kireeti Kompella; [email-masked]<mailto:[email-masked]>
> > On Tue, Sep 18, 2012 at 4:18 PM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:
> > > Hi Kreeti,
> > >
> > > From: Kireeti Kompella [mailto:[email-masked]<mailto:[email-masked]>]
> > > Cc: [email-masked]<mailto:[email-masked]>
> > > On Sep 17, 2012, at 3:36 PM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:
> > >
> > >> Read this draft.
> > >>
> > >> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf
> Of
> > Aldrin Isaac
> > >> Cc: Thomas Nadeau; [email-masked]<mailto:[email-masked]>
> > >> <[email-masked]<mailto:[email-masked]>> wrote:
> > >>> On 9/17/12 8:55 AM, "Thomas Nadeau" <[email-masked]<mailto:[email-masked]>>
> > wrote:
> > >>>
> > >>>>
> > >>>> [email-masked]<mailto:[email-masked]>
> > >>> [email-masked]<mailto:[email-masked]>
> > >> [email-masked]<mailto:[email-masked]>
> > >> [email-masked]<mailto:[email-masked]>
> > > [email-masked]<mailto:[email-masked]>
> > [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>
_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Aldrin Isaac <[email-masked]>
Date: Sat, 22 Sep 2012 09:14:21 -0400

Cloud operators might require gateways to enter and exit from outside -- however, let's consider how operators (raising my hand here) can build internal networks without different closed "encapsulation domains" which would result in meet-me choke-points and the complexity of stitching these together.

---

From: Aldrin Isaac <[email-masked]>
Date: Sat, 22 Sep 2012 10:43:41 -0400

Xiaohu,

Assume a Virtual LAN is formed by the interconnection of an L2 VN in an
overlay domain with other LAN technology (dot1q, TRILL, PBB, etc) and/or
other L2 VN in other overlay domain.   In such a case would you mandate
that the interconnection points must be active/standby? What then if the VN
"bridges" with the VN in the other Overlay domain via multiple sites/DC?
 Would you pick one site to be the active interconnection point in order to
avoid data plane loops?  What then of optimizing ingress/egress?  What of
pooling?

Best regards. -- aldrin

On Saturday, September 22, 2012, Xuxiaohu wrote:

>  Yes, if the active-active multi-homing on a per L2VPN instance basis was
> desired in some cases, vendors could offer their proprietary schemes like
> Cisco’s VSS to support that capability. In this way, there is no need to
> introduce the complexity into the standard, especially on the data plane
> perspective, and therefore other NVEs (i.e., PE routers) which are not
> within any active-active redundancy group don’t need to afford that
> complexity anymore. ****
>
> ** **
>
> As for active-passive multi-homing, I fully agree with you that the
> active/passive multi-homing on a per physical interface or trunk interface
> basis is not welcomed.  Hence we could achieve load-balancing among the
> physical interfaces or trunk interfaces which are connected to different
> NVEs of the same RG (redundancy group) on the basis of VLANs. For example,
> a dual-homed device (DHD) is multi-homed to NVE1 and NVE2 and it is
> configured with two VLANs (say VLAN x and y) on its physical NICs in a
> bond, NVE1’s corresponding interface is active for VLAN x while NVE2’s
> corresponding interface is active for VLAN y…****
>
> ** **
>
> Best regards,****
>
> Xiaohu****
>
> -****
>
> *发件人:* Jon Hudson [mailto:[email-masked] <javascript:_e({}, 'cvml',
> [email-masked]');>]
> *发送时间:* 2012年9月22日 17:16
> *收件人:* Xuxiaohu
> *抄送:* Aldrin Isaac; Lucy yong; Kireeti Kompella; [email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> *主题:* Re: [nvo3] Is the cost of supporting active-active multi-homing on
> a per L2VPN instance basis sustainable?****
>
> ** **
>
> ** **
>
> proprietary versions.****
>
> ** **
>
> active/passive anything is not welcomed in the datacenter anymore ****
>
> ** **
>
> ** **
>
> On Sat, Sep 22, 2012 at 2:02 AM, Xuxiaohu <[email-masked]> wrote:****
>
> Hi all,
>
> > > > Regarding interworking

---

From: Aldrin Isaac <[email-masked]>
Date: Sat, 22 Sep 2012 11:16:21 -0400

Please see my email below.  This is what I sent as per my outbox.  What
shows up in the list is a partially edited message.  Seems like a gmail for
iPhone bug.... :(

On Saturday, September 22, 2012, Aldrin Isaac wrote:

> *发件人:* Jon Hudson [mailto:[email-masked]]
> *发送时间:* 2012年9月22日 17:16
> *收件人:* Xuxiaohu
> *抄送:* Aldrin Isaac; Lucy yong; Kireeti Kompella; [email-masked]
> > > -----邮件原件

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 24 Sep 2012 09:49:44 +0000

Hi Aldrin,

I can hardly understand your question. Would you please explain it in more details?

Best regards,
Xiaohu

发件人: Aldrin Isaac [mailto:[email-masked]]
发送时间: 2012年9月22日 22:44
收件人: Xuxiaohu
抄送: Jon Hudson; Lucy yong; Kireeti Kompella; [email-masked]
主题: Re: [nvo3] Is the cost of supporting active-active multi-homing on a per L2VPN instance basis sustainable?

Xiaohu,

Assume a Virtual LAN is formed by the interconnection of an L2 VN in an overlay domain with other LAN technology (dot1q, TRILL, PBB, etc) and/or other L2 VN in other overlay domain.   In such a case would you mandate that the interconnection points must be active/standby? What then if the VN "bridges" with the VN in the other Overlay domain via multiple sites/DC?  Would you pick one site to be the active interconnection point in order to avoid data plane loops?  What then of optimizing ingress/egress?  What of pooling?

Best regards. -- aldrin

Best regards,
Xiaohu
-
发件人: Jon Hudson [mailto:[email-masked]<javascript:_e(%7b%7d,%20'cvml',[email-masked]');>]
发送时间: 2012年9月22日 17:16
收件人: Xuxiaohu
抄送: Aldrin Isaac; Lucy yong; Kireeti Kompella; [email-masked]<javascript:_e(%7b%7d,%20'cvml',[email-masked]');>
主题: Re: [nvo3] Is the cost of supporting active-active multi-homing on a per L2VPN instance basis sustainable?

Hi all,

Best regards,
Xiaohu

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 24 Sep 2012 13:38:59 +0000

Tom,

decoupling PE control plane from the forwarding function has many advantages but mainly it substantially increases operational scale - PE/control element is able to control multiple (1000+) compute nodes spread across different servers and other devices. The software complexity (e.g., managing policy functions, gathering of operational information like stats, events, diagnostics, etc.) is implemented in the control plane elements only. These reduce overall cost of a data center deployment. 
In addition, having an open protocol between a control plane and a forwarding plane of a PE allows sending local forwarding rules to forwarding device(s).
XMPP is an open standard, light-weight, extendable (can carry various data objects), and flexible protocol known to application environment. 

Maria

> From: Thomas Nadeau [mailto:[email-masked]]
> Sent: Thursday, September 20, 2012 7:23 PM
> To: NAPIERALA, MARIA H; Yakov Rekhter
> Cc: [email-masked]; Stiliadis, Dimitrios (Dimitri); Aldrin Isaac; Thomas
> Nadeau

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 24 Sep 2012 14:09:28 +0000

I think we should try to clarify what problems or what type of data centers specific solutions are addressing.
Specifically, EVPN can only address traffic bridged in the same VLAN. In data centers where most traffic is inter-VLAN, i.e., packets are routed, the EVPN doesn't achieve much as an overall solution.
I tried to make this point on the webex during the nvo3 interim session when draft-drake-nvo3-evpn-control-plane was discussed but I am not sure if my message went through.

Maria

> Thomas Nadeau
> Sent: Monday, September 17, 2012 11:55 AM
> To: [email-masked]
> Subject: [nvo3] draft-drake-nvo3-evpn-control-plane
> 
> 
> 	A number of us just published this draft and wanted to bring it
> to the NVO3 WG's attention.  We will be presenting/discussing this
> draft at the interim meeting this week as well, but please discuss here
> on the list as well.
> 
> 	Thanks,
> 
> 	Tom, John, et al
> 
>

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Mon, 24 Sep 2012 14:45:42 +0000

Hi Maria

Perhaps I'm missing something but I thought EVPN was built using many of the same building blocks as RFC4364 so that it could be used in combination with EVPN.  In other words IP VPN + EVPN.  

Does that not address your point?

Thanks,
Don

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of NAPIERALA, MARIA H
Sent: Monday, September 24, 2012 10:09 AM
To: [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Maria

---

From: John E Drake <[email-masked]>
Date: Mon, 24 Sep 2012 07:45:46 -0700

Maria,

Do you have any basis for your assertion that "In data centers where most traffic is inter-VLAN..."?

In any event, the VM Mobility draft (http://tools.ietf.org/html/draft-raggarwa-data-center-mobility-03) describes how routing support of VM mobility can be achieved using E-VPN in conjunction with the judicious use of host routes and/or NHRP.  I.e., no new protocols other than E-VPN are required.

This material will be incorporated into a subsequent version of http://tools.ietf.org/html/draft-bitar-datacenter-vpn-applicability-01

Yours irrespectively,

John

> Sent: Monday, September 24, 2012 7:09 AM
> To: [email-masked]
> I think we should try to clarify what problems or what type of data
> centers specific solutions are addressing.
> Specifically, EVPN can only address traffic bridged in the same VLAN.
> In data centers where most traffic is inter-VLAN, i.e., packets are
> routed, the EVPN doesn't achieve much as an overall solution.
> I tried to make this point on the webex during the nvo3 interim session
> when draft-drake-nvo3-evpn-control-plane was discussed but I am not
> sure if my message went through.
> 
> Maria
> 
> > Of Thomas Nadeau
> > the NVO3 WG's attention.  We will be presenting/discussing this draft
> > at the interim meeting this week as well, but please discuss here on
> > the list as well.
> >
> > 	Thanks,
> >
> > 	Tom, John, et al
> >
> >
> > URL:             http://www.ietf.org/internet-drafts/draft-drake-
> nvo3-
> > evpn-control-plane-00.txt
> > Status:          http://datatracker.ietf.org/doc/draft-drake-nvo3-
> evpn-
> > control-plane

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 24 Sep 2012 07:51:42 -0700

Don,

> Hi Maria
> 
> Perhaps I'm missing something but I thought EVPN was built using many of the 
> same building blocks as RFC4364 so that it could be used in combination 
> with EVPN.  In other words IP VPN + EVPN.  

Moreover, section 6.1 of draft-raggarwa-data-center-mobility-03.txt
describes how EVPN can cover inter-VLAN traffic (by forming
EVPN extranets).

Yakov.

> Specifically, EVPN can only address traffic bridged in the same VLAN. In data
 centers where most traffic is inter-VLAN, i.e., packets are routed, the EVPN d
oesn't achieve much as an overall solution.
> I tried to make this point on the webex during the nvo3 interim session when 
draft-drake-nvo3-evpn-control-plane was discussed but I am not sure if my messa
ge went through.

---

From: John E Drake <[email-masked]>
Date: Mon, 24 Sep 2012 07:49:33 -0700

Precisely

Yours irrespectively,

John

> Fedyk, Donald (Don)
> Sent: Monday, September 24, 2012 7:46 AM
> To: NAPIERALA, MARIA H; [email-masked]
> Perhaps I'm missing something but I thought EVPN was built using many
> of the same building blocks as RFC4364 so that it could be used in
> combination with EVPN.  In other words IP VPN + EVPN.
>

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 24 Sep 2012 16:49:21 +0000

Don,

> combination with EVPN.  In other words IP VPN + EVPN.

It needs to be explained why/what are the requirements that you need both (IPVPN and EVPN) in the same solution because this is not a trivial complication. 
The complexity of the solution depends on the requirements.  If majority of traffic in a (significant class of) data center is inter-subnet, a solution should be optimized for such traffic and, hence, solving the forwarding of such traffic should be the starting point. 

Maria

---

From: Lucy yong <[email-masked]>
Date: Mon, 24 Sep 2012 17:11:20 +0000

Hi Aldrin,

Either a single standard encapsulation or all devices MUST supporting all the encapsulations will give what you want. The latter will be more cost for the vendors, which will convert to operator cost later. In addition, it makes an analysis tool more complex. Both ways need standard effort to make vendor devices interwork.

The basic question here is: is there any technical reason to require a device to support a set of standard encapsulations?  If not, could DC operators help push a single standard encapsulation in IETF standard instead of approaching a solution that requires a device to support all the encapsulations.

Regards,
Lucy

---

From: "Bitar, Nabil N" <[email-masked]>
Date: Mon, 24 Sep 2012 13:13:59 -0400

With snips.

Thanks,
Nabil
On 9/19/12 1:33 AM, "Robert Raszuk" <[email-masked]> wrote:

>>itiation protocol", right?
>

<NB> Glad you said. There is an important distinction between encap and
configuring a tunnel, as you stated, that is often missed.

---

From: Sunny Rajagopalan <[email-masked]>
Date: Mon, 24 Sep 2012 10:22:14 -0700

I strongly agree with this. XMPP is a query format, not a routing 
protocol, and is better suited for hypervisor implementations. NVE 
implementations in hypervisors will find supporting xmpp easy - its just a 
query format for them, without substantial compute or memory requirements. 

I prefer the approach in draft-marques-l3vpn-end-system, where NVEs talk 
to route servers using XMPP, and the router servers act as BGP route 
reflectors and are tasked with peering with other BGP instances. This 
approach has a couple of advantages:
1) It reduces the memory and compute requirements of hypervisor NVEs.
2) It makes it easier to replace NVE<->XMPP<->BGP with 
NVE<->XMPP<->Directory, if needed for certain implementations.

Note that BGP goes away only on the NVE; the rest of the network could 
still possibly run BGP to exchange end point information.

The way I read it, draft-drake-nvo3-evpn-control-plane will still work 
fine if you do NVE<->XMPP<->BGP[EVPN]. However, I would like to see this 
called out in the draft.
--
Sunny

From:   "NAPIERALA, MARIA H" <[email-masked]>
To:     Thomas Nadeau <[email-masked]>, Yakov Rekhter 
<[email-masked]>, 
Cc:     Thomas Nadeau <[email-masked]>, "[email-masked]" 
<[email-masked]>, "Stiliadis, Dimitrios \(Dimitri\)" 
<[email-masked]>, Aldrin Isaac 
<[email-masked]>
Date:   09/24/2012 06:42 AM
Subject:        Re: [nvo3] draft-drake-nvo3-evpn-control-plane
Sent by:        [email-masked]

Tom,

decoupling PE control plane from the forwarding function has many 
advantages but mainly it substantially increases operational scale - 
PE/control element is able to control multiple (1000+) compute nodes 
spread across different servers and other devices. The software complexity 
(e.g., managing policy functions, gathering of operational information 
like stats, events, diagnostics, etc.) is implemented in the control plane 
elements only. These reduce overall cost of a data center deployment. 
In addition, having an open protocol between a control plane and a 
forwarding plane of a PE allows sending local forwarding rules to 
forwarding device(s).
XMPP is an open standard, light-weight, extendable (can carry various data 
objects), and flexible protocol known to application environment. 

Maria

>                The only issue that is being raised seems to be one of 
which

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Mon, 24 Sep 2012 17:45:18 +0000

Hi Maria

I see the problem as data centers need both L2 and L3 connectivity. I think that the Requirements Documents do spell this out. Perhaps they need more. The complexity comes from the date center environment, but is seems better to provide both L2 and L3 capabilities and let deployments choose the mix. 

The nice property of BGP based IPVPN and EVPN is the ability to partition the information such that the both L2 and L3 VPNs can scale independently, each in context (for example VLANs, IP VPNs). In that sense IP VPNs and EVPNs are well suited as one solution for extending the L2 and L3 connectivity for large and physically disperse data centers.  

Don 

-----Original Message-----
From: NAPIERALA, MARIA H [mailto:[email-masked]] 
Sent: Monday, September 24, 2012 12:49 PM
To: Fedyk, Donald (Don); [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Don,

Maria

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 24 Sep 2012 18:01:29 +0000

Don,

> The complexity comes from the date center environment, but
> is seems better to provide both L2 and L3 capabilities and let
> deployments choose the mix.

I am afraid that once you mix those capabilities in one implementation there is no choice. As a result, inter-subnet IP forwarding which is trivial in L3VPN (as is intra-subnet forwarding, btw) becomes all of a sudden complex under such construct.

Maria
 
> The nice property of BGP based IPVPN and EVPN is the ability to
> partition the information such that the both L2 and L3 VPNs can scale
> independently, each in context (for example VLANs, IP VPNs). In that
> sense IP VPNs and EVPNs are well suited as one solution for extending
> the L2 and L3 connectivity for large and physically disperse data
> centers.
> 
> Don
> 
> 
> 
> It needs to be explained why/what are the requirements that you need
> both (IPVPN and EVPN) in the same solution because this is not a
> trivial complication.
> The complexity of the solution depends on the requirements.  If
> majority of traffic in a (significant class of) data center is inter-
> subnet, a solution should be optimized for such traffic and, hence,
> solving the forwarding of such traffic should be the starting point.
> 
> Maria
>

---

From: Patrick Frejborg <[email-masked]>
Date: Mon, 24 Sep 2012 21:03:15 +0300

+1

XMPP and SIP are somewhat related.

>From http://www.sygnalgroup.com/sip-101/
------------------------
There are five parts involved in the making and stopping of
communications that are supported by SIP. These are:

    Location of User: This part determines the location of the end
system that will be used when placing a call.
    Availability of User: This part determines the availability of the
end point to participate in a call.
    Capabilities of User: This part determines the parameters and
media that will be used in a call.
    Setup of Session: This part establishes the session parameters for
each party.
    Management of Session: This part invokes transfer, termination and
modification services.

It is important to note that SIP does not provide services. Instead,
it acts in cooperation with other protocols to provide services.
-----------------------
Replace "user" with "VM/TES/MAC address" and "call/session" with
tunnel - aren't these the characteristics that we are searching for
the control plane of NVO3??

Here is an example of the registration process for a SIP UA; replace
the SIP messages with needed NVO3 XMPP messages and we have a
location/mapping database for NVO3
http://www.siptutorial.net/SIP/registration.html

Each CUG could have their own NVO3 location databases; if they need to
exchange information (and allowed to do that) use INVITEs to setup
sessions between proxies, as shown in this diagram
http://www.siptutorial.net/SIP/example.html
(Of course, the INVITE/100/180/200 messages could be replaced with
NVO3 XMPP messages but it is worthwhile to replace it?)

A SIP proxy can collect CDR - a NVO3 location database should be able
to collect data so you can track where VMs have been located
(presence) and how many tunnels (TDR) a NVE has established.

The REFER method might be useful for VM mobilty....

Etc....

Patrick

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 24 Sep 2012 11:08:59 -0700

Patrick,

> > advantage s but mainly it substantially increases operational scale
> > - PE/control element is able to control multiple (1000+) compute
> > nodes spread across different serve rs and other devices. The
> > software complexity (e.g., managing policy functions, gathering of
> > operational information like stats, events, diagnostics, etc.) is
> > implemented in the control plane elements only. These reduce overall
> > cost of a data center deployment.
> > In addition, having an open protocol between a control plane and 
> > a forwarding plane of a PE allows sending local forwarding rules 
> > to forwarding device(s).  XMPP is an open standard, light-weight, 
> > extendable (can carry various data objects), and flexible protocol 
> > known to application environment.
> >
> > Maria
> 
> +1
> 

How is that information relevant to the choice of XMPP ?

Yakov.

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Mon, 24 Sep 2012 19:12:09 +0000

Don,

I'm ok with what you said people need both L2 and L3 capabilities... but have trouble with
> IP VPNs and EVPNs are well suited as one solution for extending the L2 and L3 connectivity for large and physically disperse data centers.

What do you mean by "one solution"? They are two solutions to me. I want to make sure that we are not redefining BGP L3VPN (RFC 4364, aka 2547).

Though E-VPN is leveraging many fundamental building blocks of BGP L3 VPN, it is essentially developed for people who must support certain existing L2 services and prefer L2 solutions with E-VPN. If one wants to have BGP L3 VPN solution only in the DC and inter-connecting DC to the existing network l3vpn, or inter-connecting DCs, they can use BGP L3VPN as its original form (if practical), or extend it to make more DC friendly (e.g. draft-marques-l3vpn-end-system, as many pointed to in their mails). 

L3 VPN has huge deployment base, mostly in SP network space over the last 14 years or so, and some recent development in DC space. I don't assume you are implying to ask operators to change their L3VPN from 4364 format into this so called "IP VPNs and EVPNs... as one solution", or ask them to inter-work between existing L3VPN with this new "IP VPN and EVPN ... as one solution" in DC, which would introduce complexity for nothing if they simply need the L3VPN from network to DC. I hope this is not the case.

My point is, leave BGP L3VPN and EVPN as two separate solutions as they are.

Thanks,
Luyuan

> Sent: Monday, September 24, 2012 8:01 PM
> I am afraid that once you mix those capabilities in one implementation
> there is no choice. As a result, inter-subnet IP forwarding which is
> trivial in L3VPN (as is intra-subnet forwarding, btw) becomes all of a
> sudden complex under such construct.
> 
> Maria
> 
> > > > draft at the interim meeting this week as well, but please
> discuss
> > > here
> > > > on the list as well.
> > > >
> > > > 	Thanks,
> > > >
> > > > 	Tom, John, et al
> > > >
> > > >
> > > > Htmlized:        http://tools.ietf.org/html/draft-drake-nvo3-
> evpn-
> > > > control-plane-00
> > > >
> > > >
> > > > Abstract:

---

From: Patrick Frejborg <[email-masked]>
Date: Mon, 24 Sep 2012 22:26:41 +0300

> How is that information relevant to the choice of XMPP ?

There has been some integration between SIP and XMPP solutions, how
well those works I don't know, but open source solutions are out
there.

For connections between hypervisor NVEs (hNVE)  you could use the SIP
stuff and PR-SCTP as the media (copying the part of the VXLAN header
and the inner Ethernet header in the SCTP data chunk), the NVO3 proxy
keeps track of the session and liveness detection&traffic engineering
is handled by the transport protocol.

To reach the external world (WAN) and regular standalone hosts we need
something else. Here I would go for the current VXLAN header since the
encap/decap is available in merchant silicon. The NVO3 proxy should
use XMPP signaling (or SIP messages but XMPP signaling seems to have
considered MPLS L3VPN parameters, thus XMPP is more appealing) to
setup the session between the hNVE and the asic based NVE (aNVE).
Something is needed for liveness detection - is xmpp ping good enough?
The NVO3 proxy leverages SIP towards the hNVE and XMPP signaling
towards the aNVE - the hNVE contains the modified SIP stack and the
aNVE contains the XMPP stack (which is further integrated to BGP).

Early ideas to play around with....

Patrick

---

From: "Fedyk, Donald (Don)" <[email-masked]>
Date: Mon, 24 Sep 2012 19:27:38 +0000

Hi Luyuan

Yes in that sense "one solution" is a poor choice of words. (I meant One option for control plane using BGP. NVO3 is discussing other control plane options.)   I agree with your wording two solutions with common building blocks. 

Thanks,
Don 

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]] 
Sent: Monday, September 24, 2012 3:12 PM
To: NAPIERALA, MARIA H; Fedyk, Donald (Don); [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Don,

Thanks,
Luyuan

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Mon, 24 Sep 2012 19:30:15 +0000

Don,

Thanks for the quick clarification!
Luyuan

> From: Fedyk, Donald (Don) [mailto:[email-masked]]
> Sent: Monday, September 24, 2012 9:28 PM
> To: Luyuan Fang (lufang); NAPIERALA, MARIA H; [email-masked]
> Yes in that sense "one solution" is a poor choice of words. (I meant
> One option for control plane using BGP. NVO3 is discussing other
> control plane options.)   I agree with your wording two solutions with
> common building blocks.
> 
> Thanks,
> Don
> 
> 
> 
> 
> > IP VPNs and EVPNs are well suited as one solution for extending the
> L2 and L3 connectivity for large and physically disperse data centers.
> 
> What do you mean by "one solution"? They are two solutions to me. I
> want to make sure that we are not redefining BGP L3VPN (RFC 4364, aka
> 2547).
> 
> Though E-VPN is leveraging many fundamental building blocks of BGP L3
> VPN, it is essentially developed for people who must support certain
> existing L2 services and prefer L2 solutions with E-VPN. If one wants
> to have BGP L3 VPN solution only in the DC and inter-connecting DC to
> the existing network l3vpn, or inter-connecting DCs, they can use BGP
> L3VPN as its original form (if practical), or extend it to make more DC
> friendly (e.g. draft-marques-l3vpn-end-system, as many pointed to in
> their mails).
> 
> L3 VPN has huge deployment base, mostly in SP network space over the
> last 14 years or so, and some recent development in DC space. I don't
> assume you are implying to ask operators to change their L3VPN from
> 4364 format into this so called "IP VPNs and EVPNs... as one solution",
> or ask them to inter-work between existing L3VPN with this new "IP VPN
> and EVPN ... as one solution" in DC, which would introduce complexity
> for nothing if they simply need the L3VPN from network to DC. I hope
> this is not the case.
>

---

From: "Bitar, Nabil N" <[email-masked]>
Date: Mon, 24 Sep 2012 15:44:39 -0400

Luyuan,
Well put. I am presuming that you are also saying that both need to
co-exist (E-VPN and RFC4364) and get interconnected in environments that
need that. Is that so? Interconnection may happen in different ways to
allow bridging when traffic is among endpoints on on same subnet and
routing when crossing subnets in the same vpn.

Thanks,
Nabil

On 9/24/12 3:12 PM, "Luyuan Fang (lufang)" <[email-masked]> wrote:

>> IP VPNs and EVPNs are well suited as one solution for extending the L2
>>and L3 connectivity for large and physically disperse data centers.
>
>What do you mean by "one solution"? They are two solutions to me. I want
>to make sure that we are not redefining BGP L3VPN (RFC 4364, aka 2547).
>
>existing L2 services and prefer L2 solutions with E-VPN. If one wants to
>have BGP L3 VPN solution only in the DC and inter-connecting DC to the
>existing network l3vpn, or inter-connecting DCs, they can use BGP L3VPN
>as its original form (if practical), or extend it to make more DC
>L3 VPN has huge deployment base, mostly in SP network space over the last
>14 years or so, and some recent development in DC space. I don't assume
>you are implying to ask operators to change their L3VPN from 4364 format
>into this so called "IP VPNs and EVPNs... as one solution", or ask them
>to inter-work between existing L3VPN with this new "IP VPN and EVPN ...
>as one solution" in DC, which would introduce complexity for nothing if
>they simply need the L3VPN from network to DC. I hope this is not the
>case.
>

---

From: Thomas Nadeau <[email-masked]>
Date: Mon, 24 Sep 2012 16:30:17 -0400

> decoupling PE control plane from the forwarding function has many advantages but mainly it substantially increases operational scale - PE/control element is able to control multiple (1000+) compute…

	Why do you not think that using EVPN as the control plane to signal VXlan and NVGRE is not an example of separated forwarding and control plane functions?  All of the advantages you listed above are available in the solution proposed.

> XMPP is an open standard, light-weight, extendable (can carry various data objects), and flexible protocol known to application environment. 

	Do you think that BGP or any of the other technologies described in the draft are not open standards? 

	--Tom

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Mon, 24 Sep 2012 20:57:06 +0000

Nabil,

See in-line.

> From: Bitar, Nabil N [mailto:[email-masked]]
> Sent: Monday, September 24, 2012 9:45 PM
> To: Luyuan Fang (lufang); NAPIERALA, MARIA H; Fedyk, Donald (Don);
> [email-masked]
> Well put. 

Thx.

I am presuming that you are also saying that both need to
> need that. 

Yes. Interconnected with the matching environment is a good description.

Personally, I see L3 (in general sense) is becoming the trend for scaling, the move from L2 to L3 happened already in some largest DCs deployed today. 
On the other hand, there are networks must support certain legacy L2 services, and some DC operators may be more familiar with L2 environment so prefer l2 solutions, or there is need to inter-connect to the matching environment as you mentioned. Besides, not everyone needs the scale like FB, MS, Amazon... 
So both L2 and L3 need to be supported.

>Is that so? Interconnection may happen in different ways to

Yes, particularly true as with today's typical l2 solutions.
In all L3 case, there would be no bridging.

Thanks,
Luyuan

> >I'm ok with what you said people need both L2 and L3 capabilities...
> but
> >have trouble with
> L2
> want
> last
> >My point is, leave BGP L3VPN and EVPN as two separate solutions as
> they
> >are.
> >
> >Thanks,
> >Luyuan
> >
> >> > > >        The purpose of this document is to describe how
> Ethernet
> >> > > Virtual
> >> > > >        Private Network (E-VPN) can be used as the control
> plane
> >> for

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Mon, 24 Sep 2012 23:08:36 +0200

Luyuan,

While I understand there are customers for L3, there are equally customers for L2 as you said, but also for a mix when application tiering is required.
We need to address all 3 use cases.

Cheers,
Wim

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Luyuan Fang (lufang)
Sent: maandag 24 september 2012 22:57
To: Bitar, Nabil N; NAPIERALA, MARIA H; Fedyk, Donald (Don); [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Nabil,

See in-line.

Thx.

Thanks,
Luyuan

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 24 Sep 2012 21:09:35 +0000

Tom,

> spread across different servers and other devices. The software
> complexity (e.g., managing policy functions, gathering of operational
> information like stats, events, diagnostics, etc.) is implemented in
> the control plane elements only. These reduce overall cost of a data
> center deployment.
> 
> 	Why do you not think that using EVPN as the control plane to
> signal VXlan and NVGRE is not an example of separated forwarding and
> control plane functions?  All of the advantages you listed above are
> available in the solution proposed.

If you run BGP (PE control plane function) on NVE then there is no separation of PE's control plane function from PE's forwarding function. As a result PE controls only the co-located (with it) forwarding plane and, hence, forwarding plane cannot be spread across multiple and physically distinct devices/servers. 

> > XMPP is an open standard, light-weight, extendable (can carry various
> data objects), and flexible protocol known to application environment.
> 

Is BGP known/used in application/server community?

Maria
> >> plane to run, not whether or not we need one. I think everyone
> agrees
> >> on
> >> that.  In the way of BGP versus XMPP, perhaps you could elaborate
> why
> >> you

---

From: Lucy yong <[email-masked]>
Date: Mon, 24 Sep 2012 21:14:45 +0000

Agree. The use case draft captures them. Draft-mity-nvo3-use-case-03
Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henderickx, Wim (Wim)
Sent: Monday, September 24, 2012 4:09 PM
To: Luyuan Fang (lufang); Bitar, Nabil N; NAPIERALA, MARIA H; Fedyk, Donald (Don); [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Luyuan,

Cheers,
Wim

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Luyuan Fang (lufang)
Sent: maandag 24 september 2012 22:57
To: Bitar, Nabil N; NAPIERALA, MARIA H; Fedyk, Donald (Don); [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Nabil,

See in-line.

Thx.

Thanks,
Luyuan

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Mon, 24 Sep 2012 21:40:51 +0000

Wim,

Agree. 
I tried not to go into the mix case :-). Need to better understand the real needs, its implication, and various (prefer simplified) potential solutions.

Mix could mean people deploy L2 and L3 separately in the shared infrastructure, or it could mean inter-working if you are thinking some sites are L2, some are L3 within one VPN, or inter-connecting LX DC with LY VPN network...

Technically, these can all be done. Practically, we need to ask operators what are the objectives, how much complication they are willing to get into, and the benefits, trade-offs. 

Do we see a lot of mixed case for L2/L3 VPNs in today's network VPN deployment after all these years? If not so, why do DC operators want to make it more complex?

In my past experience on the operator side, inter-working (e.g. FR/ATM, ATM/Ethernet, MPLS/Ethernet...) were good on papers, but always too complicated for operation, never that successful. 

I can see sometimes we are dealing with some existing and the new environment, the mix may or may not be avoidable. 
But I want to see if many operators really want to mix things, and to what level.

Thanks,
Luyuan

> From: Henderickx, Wim (Wim) [mailto:[email-masked]]
> Sent: Monday, September 24, 2012 11:09 PM
> While I understand there are customers for L3, there are equally
> customers for L2 as you said, but also for a mix when application
> tiering is required.
> Luyuan Fang (lufang)
> Personally, I see L3 (in general sense) is becoming the trend for
> scaling, the move from L2 to L3 happened already in some largest DCs
> deployed today.
> On the other hand, there are networks must support certain legacy L2
> services, and some DC operators may be more familiar with L2
> environment so prefer l2 solutions, or there is need to inter-connect
> to the matching environment as you mentioned. Besides, not everyone
> needs the scale like FB, MS, Amazon...
> > >> trivial in L3VPN (as is intra-subnet forwarding, btw) becomes all
> of
> > a
> > >> > > Perhaps I'm missing something but I thought EVPN was built
> using
> > >> many
> > >> > > of the same building blocks as RFC4364 so that it could be
> used
> > in
> > >> > > Specifically, EVPN can only address traffic bridged in the
> same
> > >> VLAN.
> > >> > > In data centers where most traffic is inter-VLAN, i.e.,
> packets
> > are
> > >> > > I tried to make this point on the webex during the nvo3
> interim
> > >> > session
> > >> > > when draft-drake-nvo3-evpn-control-plane was discussed but I
> am
> > not
> > >> > > > to the NVO3 WG's attention.  We will be
> presenting/discussing
> > >> this
> > >> > > > Status:          http://datatracker.ietf.org/doc/draft-
> drake-
> > >> nvo3-
> > >> > > evpn-
> > >> > > > control-plane
> > >> > > >        Encapsulation (NVGRE), MPLS or VLANs while
> maintaining
> > >> their
> > >> > > >        existing data plane encapsulations. The intent is
> that
> > >> this
> > >> > > >        protocol will be capable of extensions in the future
> to
> > >> > handle

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 24 Sep 2012 18:14:22 -0400

A subnet (L2 VPN) could communicate with end-systems on other subnets
via a default gateway IP interface that is in a VRF which is a member
of an L3 VPN.

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 24 Sep 2012 18:19:53 -0400

Not just BGP,... networking in general (other than the basic concept
of a subnet and a default gateway) is not well understood by most
folks in the application/server community.  Any advanced
network/services/security, etc are going to require participation from
people who understand those domains.  So if we're only dealing with
basic networking I would agree with the argument, but not for anything
advanced.

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 24 Sep 2012 21:05:36 -0400

Hi Xiaohu,

As an example, an important use case that needs to be addressed is how to
"stretch" a subnet across two or more NVO3 domain.  If two NVO3 domains
connect at two separate sites, then the subnet will be stretched across
these NVO3 domains at both sites.  Ideally the stretched subnet will make
the most optimal use of the interconnect points at both sites -- including
using the shortest path between end-stations on either side of the
stretched subnet.  This is not possible with active/standby solutions.
 Also the underutilization of resources comes at a significant cost to the
operator (inefficient use of resources also creates undesirable
costs/tradeoffs for the operator)

Best regards. -- aldrin

On Monday, September 24, 2012, Xuxiaohu wrote:

>  Hi Aldrin,****
>
> ** **
>
> I can hardly understand your question. Would you please explain it in more
> details? ****
>
> ** **
>
> Best regards,****
>
> Xiaohu****
>
> ** **
>
> *发件人:* Aldrin Isaac [mailto:[email-masked] <javascript:_e({},
> *发送时间:* 2012年9月22日 22:44
> *抄送:* Jon Hudson; Lucy yong; Kireeti Kompella; [email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> Xiaohu,****
> pooling?****
> Best regards. -- aldrin****
>
>
> On Saturday, September 22, 2012, Xuxiaohu wrote:****
>
>  ****
>  ****
>  ****
>  ****
>  ****

---

From: Jon Hudson <[email-masked]>
Date: Mon, 24 Sep 2012 18:17:01 -0700

Agreed.

We must remember that the use case for for many customers is to make 2+
datacenters appear as 1. So it must be as trasnparent as it can be with all
the redundancy that you would see in a single local site.

J

On Mon, Sep 24, 2012 at 6:05 PM, Aldrin Isaac <[email-masked]>wrote:

>> I can hardly understand your question. Would you please explain it in
>> more details? ****
>>
>> ** **
>>
>> Best regards,****
>>
>> Xiaohu****
>>
>> ** **
>>
>> *发件人:* Aldrin Isaac [mailto:[email-masked]]
>> *发送时间:* 2012年9月22日 22:44
>> *收件人:* Xuxiaohu
>> *抄送:* Jon Hudson; Lucy yong; Kireeti Kompella; [email-masked]

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Tue, 25 Sep 2012 05:12:22 +0200

Luyuan,

Current VPNs do either L2 or L3, a mix is rare, but in the Cloud/DC space we talk about a new service like IAAS. The customer of the operator will define what he would like to see and get. In this environment I am seeing a lot of people wanting a mix of L2/L3 connected to either a L2 VPN or L3 VPN in the WAN.

Cheers,
Wim

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]] 
Sent: maandag 24 september 2012 23:41
To: Henderickx, Wim (Wim); Bitar, Nabil N; NAPIERALA, MARIA H; Fedyk, Donald (Don); [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Wim,

Agree. 

Thanks,
Luyuan

---

From: John E Drake <[email-masked]>
Date: Tue, 25 Sep 2012 04:56:56 -0700

Sunny,

Your conclusion is correct.  Please provide us with your proposed text.  Yakov has told me that we need to include a section describing  the implications of the CE/PE being co-located in a single box and this text could be included in it.

Yours irrespectively,

John

From: [email-masked] [mailto:[email-masked]] On Behalf Of Sunny Rajagopalan
Sent: Monday, September 24, 2012 10:22 AM
To: NAPIERALA, MARIA H
Cc: Thomas Nadeau; Yakov Rekhter; [email-masked]; Aldrin Isaac; Thomas Nadeau; Stiliadis, Dimitrios (Dimitri); [email-masked]
Subject: [nvo3] BGP + XMPP (was Re: draft-drake-nvo3-evpn-control-plane)

I strongly agree with this. XMPP is a query format, not a routing protocol, and is better suited for hypervisor implementations. NVE implementations in hypervisors will find supporting xmpp easy - its just a query format for them, without substantial compute or memory requirements.
I prefer the approach in draft-marques-l3vpn-end-system, where NVEs talk to route servers using XMPP, and the router servers act as BGP route reflectors and are tasked with peering with other BGP instances. This approach has a couple of advantages:
1) It reduces the memory and compute requirements of hypervisor NVEs.
2) It makes it easier to replace NVE<->XMPP<->BGP with NVE<->XMPP<->Directory, if needed for certain implementations.

Note that BGP goes away only on the NVE; the rest of the network could still possibly run BGP to exchange end point information.

The way I read it, draft-drake-nvo3-evpn-control-plane will still work fine if you do NVE<->XMPP<->BGP[EVPN]. However, I would like to see this called out in the draft.
--
Sunny

From:        "NAPIERALA, MARIA H" <[email-masked]<mailto:[email-masked]>>
To:        Thomas Nadeau <[email-masked]<mailto:[email-masked]>>, Yakov Rekhter <[email-masked]<mailto:[email-masked]>>,
Cc:        Thomas Nadeau <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "Stiliadis, Dimitrios \(Dimitri\)" <[email-masked]<mailto:[email-masked]>>, Aldrin Isaac <[email-masked]<mailto:[email-masked]>>
Date:        09/24/2012 06:42 AM
Subject:        Re: [nvo3] draft-drake-nvo3-evpn-control-plane
Sent by:        [email-masked]<mailto:[email-masked]>
________________________________

Tom,

Maria

> Cc: [email-masked]<mailto:[email-masked]>; Stiliadis, Dimitrios (Dimitri); Aldrin Isaac; Thomas
> Nadeau
> On 9/20/12 8:41 AM, "NAPIERALA, MARIA H" <[email-masked]<mailto:[email-masked]>> wrote:
>
> >>

---

From: John E Drake <[email-masked]>
Date: Tue, 25 Sep 2012 04:58:11 -0700

Don,

Nicely put.

Yours irrespectively,

John

> Sent: Monday, September 24, 2012 10:45 AM
> I see the problem as data centers need both L2 and L3 connectivity. I
> think that the Requirements Documents do spell this out. Perhaps they
> need more. The complexity comes from the date center environment, but
> > I tried to make this point on the webex during the nvo3 interim
> > session when draft-drake-nvo3-evpn-control-plane was discussed but I
> > am not sure if my message went through.
> >
> > Maria
> >
> > > the NVO3 WG's attention.  We will be presenting/discussing this

---

From: John E Drake <[email-masked]>
Date: Tue, 25 Sep 2012 05:00:45 -0700

Maria,

I'm sorry, but do you have any information to support your assertion, below? 

Yours irrespectively,

John

> Sent: Monday, September 24, 2012 11:01 AM
> > The complexity comes from the date center environment, but is seems
> > better to provide both L2 and L3 capabilities and let deployments
> > choose the mix.
> 
> > > Perhaps I'm missing something but I thought EVPN was built using
> > > many of the same building blocks as RFC4364 so that it could be
> used
> > > in combination with EVPN.  In other words IP VPN + EVPN.
> >

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 25 Sep 2012 14:28:01 +0000

Agreed.

On 9/24/12 10:45 AM, "Fedyk, Donald (Don)"
<[email-masked]> wrote:

>Hi Maria
>
>Perhaps I'm missing something but I thought EVPN was built using many of
>the same building blocks as RFC4364 so that it could be used in
>Specifically, EVPN can only address traffic bridged in the same VLAN. In
>data centers where most traffic is inter-VLAN, i.e., packets are routed,
>the EVPN doesn't achieve much as an overall solution.
>when draft-drake-nvo3-evpn-control-plane was discussed but I am not sure
>if my message went through.
>
>Maria
>

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 25 Sep 2012 14:33:13 +0000

On 9/24/12 1:11 PM, "Lucy yong" <[email-masked]> wrote:

>Hi Aldrin,
>
>Either a single standard encapsulation or all devices MUST supporting all
>the encapsulations will give what you want. The latter will be more cost
>for the vendors, which will convert to operator cost later. In addition,
>it makes an analysis tool more complex. Both ways need standard effort to
>make vendor devices interwork.
>
>The basic question here is: is there any technical reason to require a
>device to support a set of standard encapsulations?  If not, could DC
>operators help push a single standard encapsulation in IETF standard
>instead of approaching a solution that requires a device to support all
>the encapsulations.

	Lucy,

	Need I point out that Aldrin IS one of those "DC operators" you are
referring to and IS providing this input?

	--Tom

>Cloud operators might require gateways to enter and exit from outside --
>however, let's consider how operators (raising my hand here) can build
>internal networks without different closed "encapsulation domains" which
>would result in meet-me choke-points and the complexity of stitching
>these together.
>
>>>>>connected

---

From: Lucy yong <[email-masked]>
Date: Tue, 25 Sep 2012 14:45:43 +0000

Tom,

I am point out the consequences and suggest if DC operators can help push a better choice.

Lucy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Nadeau
Sent: Tuesday, September 25, 2012 9:33 AM
To: Lucy yong; Aldrin Isaac; Stiliadis, Dimitrios (Dimitri)
Cc: Thomas Nadeau; John E Drake; [email-masked]; Balus, Florin Stelian (Florin); Joel M. Halpern; Jon Hudson
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

	Lucy,

	--Tom

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 25 Sep 2012 11:33:04 -0400

> I am point out the consequences and suggest if DC operators can help push a better choice.

	Are you asking for a truly better choice or just one that is different from a solution you do not prefer? I have frankly not seen really any factual technical arguments against our proposal other than "I'd prefer to use something else, especially if it is cool." There are already a number of real DC operators (i.e.: not just "in theory on a whiteboard") that have provided input on the draft and on the list who seem to clearly think it is a sound technical approach and have backed that up with sound, technical arguments.

	--Tom

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 25 Sep 2012 11:53:31 -0400

Hi Sunny,

Although I don't see any reason why BGP cannot be implemented on an
NVE (hypervisor, ToR, etc), I'm beginning to see why something other
than BGP might be an appropriate choice for transporting
network-control data to/from the NVE.  The idea of
NVE<->MESSAGING<->AUTHORITY where MESSAGING is standardized and
AUTHORITY can take different forms might indeed be a useful
decoupling.  It would probably be appropriate to spend a few cycles to
determine if XMPP is the best choice among available messaging options
(ex: AMQP).

Once we get past the choice of a common messaging transport, IMHO we
need to be talking a lot more (more sooner than later) about the
network as a set of one or more distributed applications (might be
centralized under one NVO3 domain, but distributed across multiple).
For a distributed application, the AUTHORITY is essentially the sum of
the application instances in that NVO3 domain.  This point is being
raised by others but doesn't seem to have "clicked".  In this view of
things, E-VPN, RFC 4364, etc would be considered "applications".  The
operator can choose one or more of these.  The difference between this
notion and what is proposed by others (SDN, openflow, etc) is that the
IETF standardizes applications as well in addition to data-plane
formats, etc.  This ensures that operators will have choice to select
a particular implementation of an application (versus a proprietary
application).

Taking this a little further using E-VPN as an example -- generally
E-VPN is thought of as being an application that is implemented in a
distributed manner at NVE/PE, however there is no rule that says it
can't be implemented in a centralized or hybrid (ex: semi-distributed)
manner (as Maria points out).  For example, an E-VPN application
running on a centralized server can implement all the procedures
specified in draft-ietf-l2vpn-evpn on behalf of NVE and either
distribute the resulting forwarding rules to NVE (ex: using openflow
over XMPP, etc) or make the results available for on-demand lookup
(directory mode) by slightly more intelligent NVE.  The rules of how
to construct E-VPN L2 VN, however, should remain the same regardless
of whether it's implemented in a distributed, centralized or hybrid
fashion.  In this model, it makes more sense to define a provisioning
schema per application (versus a generic all-encompassing schema) --
in E-VPN the provisioning schema would outline import/export RT and
the schema would need to be defined as part of the specification of
the application (as well as schema for other application specific
data).

At a data-plane level, NVE that support the E-VPN application should
support certain data-plane constructs as specified in
draft-ietf-l2vpn-evpn such that it is possible for E-VPN instances on
NVE under other AUTHORITY to send frames directly to it (ex: to
maximize network utilization) or otherwise the local AUTHORITY might
direct other AUTHORITY toward interconnecting gateways.  Likewise for
other applications.

Best regards -- aldrin

P.S.    I'm using AUTHORITY here as I am still struggling with the term ORACLE.

On Mon, Sep 24, 2012 at 1:22 PM, Sunny Rajagopalan
<[email-masked]> wrote:
> I strongly agree with this. XMPP is a query format, not a routing protocol,
> and is better suited for hypervisor implementations. NVE implementations in
> hypervisors will find supporting xmpp easy - its just a query format for
> them, without substantial compute or memory requirements.
> I prefer the approach in draft-marques-l3vpn-end-system, where NVEs talk to
> route servers using XMPP, and the router servers act as BGP route reflectors
> and are tasked with peering with other BGP instances. This approach has a
> couple of advantages:
> Note that BGP goes away only on the NVE; the rest of the network could still
> possibly run BGP to exchange end point information.
>
> The way I read it, draft-drake-nvo3-evpn-control-plane will still work fine
> if you do NVE<->XMPP<->BGP[EVPN]. However, I would like to see this called
> out in the draft.
> --
> Sunny
>
>
>
>
> decoupling PE control plane from the forwarding function has many advantages
> but mainly it substantially increases operational scale - PE/control element
> is able to control multiple (1000+) compute nodes spread across different
> servers and other devices. The software complexity (e.g., managing policy
> functions, gathering of operational information like stats, events,
> diagnostics, etc.) is implemented in the control plane elements only. These
> reduce overall cost of a data center deployment.
> forwarding plane of a PE allows sending local forwarding rules to forwarding
> device(s).

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 25 Sep 2012 12:06:44 -0400

Is it true that today's DC-grade ToR silicon support MPLS over UDP or
GRE?  Are there examples of silicon that does not support this?

---

From: Lucy yong <[email-masked]>
Date: Tue, 25 Sep 2012 16:07:03 +0000

Tom,

Indeed, I do not think requiring all the devices to support all encapsulations is a proper way and there is no technical reason either (that is why I am asking). Look at our reality world. There are several power plug types exist worldwide. For the people who do not travel at all, they just buy one type plug; for people who travel some places, they buy a power convert or adapter; for few people who travel all the places, they buy a convertor that support all the types. There is no standard to require all the devices supporting all the plug types so whole world people can benefit from it, right?

In fact, the third set people is just a very small group comparing to other two. The convertor that supports all the types is equivalent to a gateway entity in this context. If DC operator happens to deal with different encapsulations, using gateway is a right approach instead of an approach requiring all device to support all the encapsulations. As Aldrin points out, during DC migration, they need to use a gateway. If DC operators really do not want to use a gateway for this, they better strongly request standardize one encapsulation, which will make everyone life simple. This is my point.

In fact, a gateway has been used for many purposes and for many years. 

Lucy 

-----Original Message-----
From: Thomas Nadeau [mailto:[email-masked]] 
Sent: Tuesday, September 25, 2012 10:33 AM
To: Lucy yong
Cc: Thomas Nadeau; Aldrin Isaac; Stiliadis, Dimitrios (Dimitri); John E Drake; [email-masked]; Balus, Florin Stelian (Florin); Joel M. Halpern; Jon Hudson
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

	--Tom

---

From: Lucy yong <[email-masked]>
Date: Tue, 25 Sep 2012 16:18:30 +0000

Aldrin,

You are DC operator. It is best for you to answer the questions. IP/MPLS VPN in WANs mainly use LSP tunnels although GRE tunnel is standardized, but rarely deployed as far as I know. (I may be wrong, which some SP operator clarify it).

Lucy

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 25 Sep 2012 10:04:10 -0700

Hi Maria,

> I am afraid that once you mix those capabilities in one implementation there is no choice. As a result, inter-subnet IP forwarding which is trivial in L3VPN (as is intra-subnet forwarding, btw) bec…

I'll agree (sort of).  Let me dissect this a bit:

a) providing L2 and L3 with a common control plane is a Good Thing.

b) providing a choice of L2 and/or L3 virtual networks on a common platform is a Good Thing.

c) mixing L2 and L3 for a tenant (either as VNs or between DC and WAN) is fraught with complications (basically, IRB in a virtualized setting).

There is implementation and deployment proof that (a) and (b) are perfectly doable.  There is proof by scars that (c ) is painful.  However, I suspect that pragmatics realities will dictate that (c ) is necessary.

Kireeti.

---

From: Sunny Rajagopalan <[email-masked]>
Date: Tue, 25 Sep 2012 10:06:09 -0700

Will do. Within the CE/PE colocated case, this is actually a special case 
where the forwarding PE element is separate from the control PE element. 
XMPP would be the format that the control element would use to talk to the 
forwarding element. I had some additional comments/suggestions in my 
notes, I'll start another thread on them. I can submit a proposed text 
once we have some consensus on the other items as well.
--
Sunny

From:   John E Drake <[email-masked]>
To:     Sunny Rajagopalan/Santa Clara/IBM@IBMUS, "NAPIERALA, MARIA H" 
<[email-masked]>, 
Cc:     Thomas Nadeau <[email-masked]>, Yakov Rekhter 
<[email-masked]>, "[email-masked]" <[email-masked]>, Aldrin Isaac 
<[email-masked]>, Thomas Nadeau <[email-masked]>, "Stiliadis, 
Dimitrios (Dimitri)" <[email-masked]>, 
"[email-masked]" <[email-masked]>
Date:   09/25/2012 05:02 AM
Subject:        RE: [nvo3] BGP + XMPP (was Re: 
draft-drake-nvo3-evpn-control-plane)

Sunny,
 
Your conclusion is correct.  Please provide us with your proposed text. 
Yakov has told me that we need to include a section describing  the 
implications of the CE/PE being co-located in a single box and this text 
could be included in it.
 
Yours irrespectively,
 
John
 
From: [email-masked] [mailto:[email-masked]] On Behalf Of 
Sunny Rajagopalan
Sent: Monday, September 24, 2012 10:22 AM
To: NAPIERALA, MARIA H
Cc: Thomas Nadeau; Yakov Rekhter; [email-masked]; Aldrin Isaac; Thomas 
Nadeau; Stiliadis, Dimitrios (Dimitri); [email-masked]
Subject: [nvo3] BGP + XMPP (was Re: draft-drake-nvo3-evpn-control-plane)

From:        "NAPIERALA, MARIA H" <[email-masked]> 
To:        Thomas Nadeau <[email-masked]>, Yakov Rekhter <
[email-masked]>, 
Cc:        Thomas Nadeau <[email-masked]>, "[email-masked]" <
[email-masked]>, "Stiliadis, Dimitrios \(Dimitri\)" <
[email-masked]>, Aldrin Isaac <
[email-masked]> 
Date:        09/24/2012 06:42 AM 
Subject:        Re: [nvo3] draft-drake-nvo3-evpn-control-plane 
Sent by:        [email-masked] 

Tom,

Maria

which

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 25 Sep 2012 16:19:34 +0000

Aldrin, often you 'can' do more than the basic ASIC programming supports but with a performance cost. I.e. you have to bounce the packet off a dummy interface (like a loopback) and then bring it back in for the second layer processing. That can result in a large performance hit. I have not however seen MPLS over UDP/GRE supported in a single pass on the ASICs I've looked at but its easy to miss features out of a 1000 page document.... so take that with a grain of salt.

Peter

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 25 Sep 2012 10:15:30 -0700

> via a default gateway IP interface that is in a VRF which is a member

And, right on cue, Aldrin tells us why IRB is needed :-)

Totally agreed: necessary, whether one likes it or not.  To channel Maria, though, one could "simulate" this by IP routing both within and between subnets.  A uniform forwarding approach would make things a lot simpler.  So the question comes back: why consider a "subnet" a Layer 2 entity?

Kireeti.

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 25 Sep 2012 10:25:11 -0700

Hi Lucy,

On Sep 25, 2012, at 09:07 , Lucy yong <[email-masked]> wrote:

>  There are several power plug types exist worldwide. For the people who do not travel at all, they just buy one type plug; for people who travel some places, they buy a power convert or adapter; for few people who travel all the places, they buy a convertor that support all the types

Not to push your analogy too far: having several power plug standards means that many (but not all) power supplies eat the cost of supporting both 110V and 220V outlets.  The physical "encapsulation" gateway is relatively easy; the semantic translation (transformer from 110<->220) is harder.

What the draft is saying is, let's have a common voltage standard (BGP); we should decide later whether and how and where to do the physical conversion.

Kireeti.

---

From: Sunny Rajagopalan <[email-masked]>
Date: Tue, 25 Sep 2012 10:33:22 -0700

Hi, Aldrin
I agree & share your thoughts below. I picked XMPP just because it seems 
to be the one most talked about here, any messaging format that is open 
will do. 

Note that something similar happens today in PE routers - the control 
plane is usually running on a separate CPU from the forwarding plane, but 
the messaging between the two is proprietary. Making the messaging format 
open allows for the control plane to be implemented in different ways 
depending on the scenario. This is important because in some networks it 
might make sense to have a directory-based name resolution, and BGP based 
end location distribution would make sense in others. As you pointed out, 
these two could even co-exist, (for example if you run one application 
over the short haul and another over the long haul). In such a hybrid 
application scenario,  the directory could talk to both,  local NVEs and 
the BGP route reflectors using XMPP, with the directory being the entity 
that uploads endpoint information into the BGP route reflector.

To take this one step further, if you define the messaging [XMPP] format 
generically, you should be able to use the encapsulation method of your 
choice (MPLS, NVGRE, VXLAN) as well. This gives us a system with a 
completely decoupled control and forwarding plane.
--
Sunny

From:   Aldrin Isaac <[email-masked]>
To:     Sunny Rajagopalan/Santa Clara/IBM@IBMUS, 
Cc:     "NAPIERALA, MARIA H" <[email-masked]>, Thomas Nadeau 
<[email-masked]>, Yakov Rekhter <[email-masked]>, 
"[email-masked]" <[email-masked]>, Thomas Nadeau <[email-masked]>, 
"Stiliadis, Dimitrios (Dimitri)" <[email-masked]>, 
[email-masked]
Date:   09/25/2012 08:54 AM
Subject:        Re: [nvo3] BGP + XMPP (was Re: 
draft-drake-nvo3-evpn-control-plane)

Hi Sunny,

Best regards -- aldrin

P.S.    I'm using AUTHORITY here as I am still struggling with the term 
ORACLE.

> I strongly agree with this. XMPP is a query format, not a routing 
protocol,
to
> Note that BGP goes away only on the NVE; the rest of the network could 
still
> The way I read it, draft-drake-nvo3-evpn-control-plane will still work 
fine
> if you do NVE<->XMPP<->BGP[EVPN]. However, I would like to see this 
called
advantages
> forwarding plane of a PE allows sending local forwarding rules to 
forwarding
> XMPP is an open standard, light-weight, extendable (can carry various 
data

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Tue, 25 Sep 2012 17:29:37 +0000

> Totally agreed: necessary, whether one likes it or not.  To channel
> Maria, though, one could "simulate" this by IP routing both within and
> between subnets.  A uniform forwarding approach would make things a lot
> simpler.  So the question comes back: why consider a "subnet" a Layer 2
> entity?

Precisely. 
Hence, the only question that remains is how to address non-IP traffic, for those DCs that care about such traffic (where non-IP traffic is the actual application traffic, not DHCP or ARP generated by the guest OS). It would seem that the best way to answer that question would be to first to measure such traffic.

Maria

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 25 Sep 2012 10:36:45 -0700

> Hence, the only question that remains is how to address non-IP traffic, for those DCs that care about such traffic (where non-IP traffic is the actual application traffic, not DHCP or ARP generated…

With an electron microscope :)

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 25 Sep 2012 14:15:50 -0400

Resending last email without nvo3-bounces in the recipient list....

---

From: Lucy yong <[email-masked]>
Date: Tue, 25 Sep 2012 19:25:26 +0000

Hi Kireeti,

[[LY]] For the context of power voltage, there are several standard power voltage levels that are necessary for power transportation industry. They exist for technical reason. This is why I ask if there is a technical reason to standardize a set of encapsulations for nv03 in early email. In fact, that you (me too) want one control plane protocol to apply all nvo3 data planes, I think, is the same purpose: simple and avoid interworking. 

Since we don't disagree here, no need to continually make fun here. :)

Lucy

Kireeti.

---

From: John E Drake <[email-masked]>
Date: Tue, 25 Sep 2012 13:07:41 -0700

Lucy,

I was planning to start updating the draft tomorrow to reflect all of the comments that have been received on the list (thanks everyone!) over the past two weeks.

I haven't worked out the exact text but in spirit section 4 will state that within an EVI it is the operator of that EVI's responsibility to ensure that the set of encapsulations supported by the PEs in that EVI is consistent.  The advertisement of the encapsulations supported by each PE in the EVI assists the operator in obtaining this consistency. 

Yours irrespectively,

John

P.S.  I trimmed the recipient list because the first time I sent this email it bounced due to too many recipients.

> Sent: Tuesday, September 25, 2012 12:25 PM
> To: Kireeti Kompella
> Cc: Thomas Nadeau; John E Drake; [email-masked]; Aldrin Isaac; Thomas
> Nadeau; Balus, Florin Stelian (Florin); Stiliadis, Dimitrios (Dimitri);
> Joel M. Halpern; Jon Hudson
> Not to push your analogy too far: having several power plug standards
> means that many (but not all) power supplies eat the cost of supporting
> both 110V and 220V outlets.  The physical "encapsulation" gateway is
> relatively easy; the semantic translation (transformer from 110<->220)
> is harder.
> 
> What the draft is saying is, let's have a common voltage standard
> (BGP); we should decide later whether and how and where to do the
> physical conversion.
> 
> 
> [[LY]] For the context of power voltage, there are several standard
> power voltage levels that are necessary for power transportation
> industry. They exist for technical reason. This is why I ask if there
> is a technical reason to standardize a set of encapsulations for nv03
> in early email. In fact, that you (me too) want one control plane
> protocol to apply all nvo3 data planes, I think, is the same purpose:
> simple and avoid interworking.
>

---

From: John E Drake <[email-masked]>
Date: Tue, 25 Sep 2012 12:55:41 -0700

Lucy,

Yours irrespectively,

John

---

From: Ken Gray <[email-masked]>
Date: Tue, 25 Sep 2012 15:07:40 -0700

Been waiting for that moment of clarity. HOPEFULLY a single non-BGP
MESSAGING format (for those NVE that won't be implementing BGP) to achieve
this abstraction for all APPLICATIONS (would hope we can be economical
here from an implementation perspective).

"openflow over XMPP" ...interesting.

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 25 Sep 2012 22:51:55 -0400

Although a common messaging layer is useful, the messaging protocol is just
a means of transporting application specific data between the AUTHORITY and
the NVE.  There might be some room to define "objects" (say a common schema
for encapsulation with sub schema for a each specific encapsulation -- ex:
label stacking in MPLS) but it must be the application that should define
the overall schema/contents of the messages that are exchanged for it's
purpose.  Let's be careful not to call or treat the messaging part an
"application".

Additionally there are plenty of things that are available with BGP that
would need to be available as part of the "common capabilities" of a common
control-plane message layer including capabilities such as graceful restart
(purge or keep state if CP session is lost and for how long),
keep-alive/BFD, etc.  I hope folks on this list are careful not to
disregard (or even worse, to not know) what many years of control-plane
engineering have produced in the IETF.

Best regards -- aldrin

On Tuesday, September 25, 2012, Sunny Rajagopalan wrote:

> these two could even co-exist, (for example if you run one application over
> the short haul and another over the long haul). In such a hybrid
> Cc:        "NAPIERALA, MARIA H" <[email-masked]>, Thomas Nadeau <
> [email-masked]>, Yakov Rekhter <[email-masked]>, "
> [email-masked]" <[email-masked]>, Thomas Nadeau <[email-masked]>,
> ------------------------------
>
>
>
> Hi Sunny,
>

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Wed, 26 Sep 2012 04:59:24 +0000

Aldrin:

+1 on this and thanks for stating this so clearly.

>> determine if XMPP is the best choice among available messaging options

Exactly. This is the only thing needed. I should note though the
difference between verbose XML based communication (XMPP for example)
and more compact data transfer mechanisms that can match programming
structures (see binary). When we are talking about millions
of VMs and routes and fast communication for route update with minimum
overheads, I am not sure that the overheads of XML outweigh the benefits.
 Also, it looks to me that NVE->MESSAGE->AUTHORITY
is more of a point to point communication rather than a pub/sub interface.

We can also start the JSON vs XML war in this context though, and I am sure
lots of people will have lots of opinions ;)

Dimitri

---

From: Thomas Nadeau <[email-masked]>
Date: Wed, 26 Sep 2012 07:21:55 -0400

On Sep 25, 2012:10:51 PM, at 10:51 PM, Aldrin Isaac <[email-masked]> wrote:

> Although a common messaging layer is useful, the messaging protocol is just a means of transporting application specific data between the AUTHORITY and the NVE.  There might be some room to define "objects" (say a common schema for encapsulation with sub schema for a each specific encapsulation -- ex: label stacking in MPLS) but it must be the application that should define the overall schema/contents of the messages that are exchanged for it's purpose.  Let's be careful not to call or treat the messaging part an "application".
> 
> Additionally there are plenty of things that are available with BGP that would need to be available as part of the "common capabilities" of a common control-plane message layer including capabilities such as graceful restart (purge or keep state if CP session is lost and for how long), keep-alive/BFD, etc.  I hope folks on this list are careful not to disregard (or even worse, to not know) what many years of control-plane engineering have produced in the IETF.
> 
> Best regards -- aldrin

	And also to that end, we should note that we are working on such a "minimum capabilities" appendix section in the next revision of the draft. 

	--Tom

> I agree & share your thoughts below. I picked XMPP just because it seems to be the one most talked about here, any messaging format that is open will do. 
> 
> Note that something similar happens today in PE routers - the control plane is usually running on a separate CPU from the forwarding plane, but the messaging between the two is proprietary. Making the messaging format open allows for the control plane to be implemented in different ways depending on the scenario. This is important because in some networks it might make sense to have a directory-based name resolution, and BGP based end location distribution would make sense in others. As you pointed out, these two could even co-exist, (for example if you run one application over the short haul and another over the long haul). In such a hybrid application scenario,  the directory could talk to both,  local NVEs and  the BGP route reflectors using XMPP, with the directory being the entity that uploads endpoint information into the BGP route reflector. 
> 
> To take this one step further, if you define the messaging [XMPP] format generically, you should be able to use the encapsulation method of your choice (MPLS, NVGRE, VXLAN) as well. This gives us a system with a completely decoupled control and forwarding plane. 
> -- 
> Sunny 
> 
> 
> 
> 
> Cc:        "NAPIERALA, MARIA H" <[email-masked]>, Thomas Nadeau <[email-masked]>, Yakov Rekhter <[email-masked]>, "[email-masked]" <[email-masked]>, Thomas Nadeau <[email-masked]>, "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>, [email-masked] 
> Subject:        Re: [nvo3] BGP + XMPP (was Re: draft-drake-nvo3-evpn-control-plane) 
> 
> 
> 
> Hi Sunny,
>

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 26 Sep 2012 07:56:43 -0400

Hi Dimitrios,

Just to be clear, I'm not proposing that every implementation of say,
RFC 4364, replace BGP with XMPP (or other thing).  BGP will continue
to work great for many hardware-based NVE and even for implementations
of software-based NVE.  BGP-based NVE can co-exist with XMPP-based (as
an example) NVE that implement E-VPN (as an example).  The "backbone"
of the E-VPN control plane will remain BGP since things like route
distribution at scale, federating AS, etc come for free.  In the end
if all we end up doing with MESSAGING is what BGP is already used for,
then BGP should be the common messaging layer.

It is important for proponents of a common messaging layer to develop
a very good argument for it and provide clear evidence of it's value
(versus philosophical) and also discuss what may (or may not) be wrong
with more than one messaging protocols to co-exist on an NVE for
different roles (app-specific, counters, etc).  I can see the possible
value in the paradigm, but I personally need convincing proof not
rants.

I also agree that if we're talking about a common messaging layer, we
need to discuss what properties a common messaging layer would
require.  In the case of a basic software NVE (VM host) it's possible
that it may not require support for millions of routes if only the
ones that are needed for it are produced for it by the application,
however NVE on gateways (which may also be software-based in some
shops) may require handling of "millions of routes" and this might not
be a good place for XMPP and the like.

The most important point I'm trying to make, however, is that we can
agree on the idea of an APPLICATION such that the WG can discuss a
framework in the context of APPLICATIONS, and move forward to
standardize a useful and distinct set of these for the purpose of NVO.

Best regards -- aldrin

---

From: Eric Gray <[email-masked]>
Date: Wed, 26 Sep 2012 15:48:31 -0400

Maria,

	Given that we already have routing, I am not clear on what else it is that you're
saying needs to be done to produce an "overall solution."  If we use EVPN as one way
to address traffic "bridged in the same VLAN" and we use routing to handle traffic that
"is inter-VLAN, i.e., packets [that] are routed" do we have less than an overall solution?

	I am personally convinced that this is just one over-all solution, of possibly many
- but it certainly seems like a good candidate to consider for work here in the IETF.

--
Eric

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of NAPIERALA, MARIA H
Sent: Monday, September 24, 2012 10:09 AM
To: [email-masked]
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Maria

> 	A number of us just published this draft and wanted to bring it to

---

From: Eric Gray <[email-masked]>
Date: Wed, 26 Sep 2012 16:11:29 -0400

Maria,

	Not sure I follow (or fully agree with) your reasoning here.

	I agree that there is at least a small scale advantage in separating the control and 
forwarding planes.  However it is less than some might expect.

	Take the case where a single control element is able to control 1,000 nodes.  At
the same time, those 1,000 nodes are potentially at risk for a failure of that one control
element.  This can be addressed using redundant control elements, but this has related
complexity costs that will impact both CAPEX and OPEX.

	Management complexity should be pretty much the same, whether one does
configuration associated with 1,000 nodes in one, in ten, in a hundred, or in a thousand
control elements.

	In fact, management  complexity (among other forms of complexity) can become 
quite "interesting" in a number of scenarios - mostly related to failures in either control or 
data-plane connectivity that produce irregularities and inconsistencies in real vs. perceived
forwarding connectivity.

	In terms of software complexity, a component of complexity relates to the scale
in which the software operates.  Many experienced folks see distributed simplicity as a
bit better solution than concentrated complexity.  The complexity associated with keeping
control and data plane connectivity in alignment also has a substantial impact on software
complexity.

	In terms of having an "open protocol" between control and data plane elements,
we have done a lot of this kind of work in the IETF.  Do we have a clear understanding of
the additional work that may be required?

--
Eric

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of NAPIERALA, MARIA H
Sent: Monday, September 24, 2012 9:39 AM
To: Thomas Nadeau; Yakov Rekhter
Cc: Thomas Nadeau; [email-masked]; Stiliadis, Dimitrios (Dimitri); Aldrin Isaac
Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane

Tom,

Maria

> Cc: [email-masked]; Stiliadis, Dimitrios (Dimitri); Aldrin Isaac; 
> Thomas Nadeau
> plane to run, not whether or not we need one. I think everyone agrees 
> on that.  In the way of BGP versus XMPP, perhaps you could elaborate 
> why you think BGP is a bad choice?
> 
> 	--Tom
> 
>

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 27 Sep 2012 04:02:04 +0000

Eric,

(BTW, there was a follow up to this e-mail which you might want to check out also). 

A data center where all traffic (intra- and inter-subnet) is *IP* then a routing solution such as L3VPN is the most optimal for such data center. I would think everybody would agree with this assertion. 

The question is how to address *non-IP* traffic, in those DCs that care about such traffic. The best way to answer that question would be to measure such traffic. If a data center has a lot of *non-IP* traffic then it would make sense to use EVPN for intra-subnet forwarding. If non-IP traffic is localized to a small subset of VLANs perhaps it makes more sense to have a solution where EVPN is only used when necessary, rather than to design the entire solution around EVPN.

Maria

> From: Eric Gray [mailto:[email-masked]]
> Sent: Wednesday, September 26, 2012 3:49 PM

---

From: Somesh Gupta <[email-masked]>
Date: Wed, 26 Sep 2012 21:23:15 -0700

I agree that that IP will be dominant. I would also contend that
multicast/broadcast(non-arp) would also be used by only a small
subset (of course some of multicast and non-IP applications
are business critical to those who use them).

My recommendation is for solutions where multicast/broadcast is
optional i.e. the base functionality does not depend on that

> Sent: Wednesday, September 26, 2012 9:02 PM
> To: Eric Gray; [email-masked]
> A data center where all traffic (intra- and inter-subnet) is *IP* then
> a routing solution such as L3VPN is the most optimal for such data
> center. I would think everybody would agree with this assertion.
> 
> The question is how to address *non-IP* traffic, in those DCs that care
> about such traffic. The best way to answer that question would be to
> measure such traffic. If a data center has a lot of *non-IP* traffic
> then it would make sense to use EVPN for intra-subnet forwarding. If
> non-IP traffic is localized to a small subset of VLANs perhaps it makes
> more sense to have a solution where EVPN is only used when necessary,
> rather than to design the entire solution around EVPN.
> 
> Maria
> 
> draft

---

From: "Bitar, Nabil N" <[email-masked]>
Date: Thu, 27 Sep 2012 07:58:22 -0400

Do you mean by interworking a Layer2 service having access to an IP
routing service as commonly done today?  I wouldn't compare that to FR/ATM
interworking.

There are applications that require operation over Ethernet and have MAC
awareness. There are also dependencies on the other end of the
communication operations and capabilities.

There are other applications, legacy or not, that are not IP aware and
will require Layer2 connectivity. Storage, depending on the storage
networking technology, is an example and I think there was an email from
Dave Black about that some time back. There are probably other apps,
proprietary or commercial that rely on Ethernet only. I am raising that
not because of E-VPN or not but the general notion being debated by some
that we need to take a hard-line saying only IP service needs to be
supported, as much as I would like that to be the case.

While one can argue that if an application is not IP-based, it can be
migrated to one that is or you can tunnel to the application depending on
the application or ..,, it seems to me it is a hardline to say that an L2
service is not
needed in the DC at the functional level of NVE and gateways as it can be
disruptive in certain cases or create hurdles.

The net is that the DC fabric (underlay) can be IP only although for some
Ethernet can still be a choice. The service overlaid on that can be IP or
bridged Ethernet and that is a choice for people to do based on several
constraints including applications they want to support. Today Ethernet
and IP services are realities and need to be addressed. IP will be always
needed, no doubt. If one can live with IP only they would have the tool,
if they need a mix of IP of Ethernet bridge services, they should also
have the tools. Id people are only interested in IP services, they can
only work on that, and if people have the need for the mix or want to work
on the mix or or one or the other, they can also do that.

Thanks,
Nabil

On 9/24/12 5:40 PM, "Luyuan Fang (lufang)" <[email-masked]> wrote:

>Wim,
>
>Agree.
>I tried not to go into the mix case :-). Need to better understand the
>real needs, its implication, and various (prefer simplified) potential
>solutions.
>
>Mix could mean people deploy L2 and L3 separately in the shared
>infrastructure, or it could mean inter-working if you are thinking some
>sites are L2, some are L3 within one VPN, or inter-connecting LX DC with
>LY VPN network...
>
>Technically, these can all be done. Practically, we need to ask operators
>what are the objectives, how much complication they are willing to get
>into, and the benefits, trade-offs.
>
>Do we see a lot of mixed case for L2/L3 VPNs in today's network VPN
>deployment after all these years? If not so, why do DC operators want to
>make it more complex?
>
>In my past experience on the operator side, inter-working (e.g. FR/ATM,
>ATM/Ethernet, MPLS/Ethernet...) were good on papers, but always too
>complicated for operation, never that successful.
>
>I can see sometimes we are dealing with some existing and the new
>environment, the mix may or may not be avoidable.

---

From: Eric Gray <[email-masked]>
Date: Thu, 27 Sep 2012 10:52:37 -0400

Maria,

	Thanks for the explanation.

	Since there are already existing solutions for virtualized data centers, I think it is only too
obvious that we're conceding that a "one-size fits all" solution is pretty much out of the question.

	So, it seems possible (at least) that a solution built around EVPN may be applicable to at
least a subset of use cases.

	I am curious, however, about the suggestion that we need "to address *non-IP* traffic"
in the context of an IETF working group.  Why is that?

--
Eric

-----Original Message-----
From: NAPIERALA, MARIA H [mailto:[email-masked]] 
Sent: Thursday, September 27, 2012 12:02 AM
To: Eric Gray; [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane
Importance: High

Eric,

Maria

> 	Given that we already have routing, I am not clear on what else it is 
> that you're saying needs to be done to produce an "overall solution."  
> If we use EVPN as one way to address traffic "bridged in the same 
> VLAN" and we use routing to handle traffic that "is inter-VLAN, i.e., 
> packets [that] are routed" do we have less than an overall solution?
> 
> > draft at the interim meeting this week as well, but please discuss 
> > here on the list as well.
> >
> > 	Thanks,
> >
> > 	Tom, John, et al
> >
> >

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 27 Sep 2012 15:38:33 +0000

Eric,

> in the context of an IETF working group.  Why is that?

This is a good question. As I stated, if you assume that all traffic in a DC is IP then layer 3 overlay solution seems to be obvious one to me.

Maria

> > 	I am personally convinced that this is just one over-all
> solution, of
> > possibly many

---

From: "Black, David" <[email-masked]>
Date: Thu, 27 Sep 2012 11:59:03 -0400

> This is a good question. As I stated, if you assume that all traffic in a DC
> is IP then layer 3 overlay solution seems to be obvious one to me.

In addition, there is IP traffic that requires L2 adjacency based on needing
IP subnet multicast for traffic that does not use IP multicast.

Thanks,
--David

> Sent: Thursday, September 27, 2012 11:39 AM
> To: Eric Gray
> Cc: [email-masked]
> This is a good question. As I stated, if you assume that all traffic in a DC
> is IP then layer 3 overlay solution seems to be obvious one to me.
> 
> Maria
> 
> > --
> > Eric
> >

---

From: Eric Gray <[email-masked]>
Date: Thu, 27 Sep 2012 12:08:56 -0400

Maria,

	Even if all of the traffic in a Data Center is IP traffic, this IP traffic may use a
specific layer 2 technology for at least part of the deployment.  This may happen, for
example, if a company that uses data-center bridging to provide a SAN (or cloud)
storage solution merges with a company that uses some (as yet to be defined?) 
strictly L3 overlay solution.

	Or it may happen in cases where a combination of L2 and L3 virtualizations
are used in a common deployment (e.g. - to provide further scaling advantages).

	In a way, we (the IETF) have cooked our own goose and may have to stew in
the juices of the result.

	We have defined mechanisms for transport of (various types of) Layer 2 over
IP.  The result is that the mechanisms we've defined mimic Layer 2 constructs (e.g. - a 
link or a virtual bridged network) in ways that may be less than perfect.

	"Pseudo-wires" for example are not actual wires (or even fibers).  Layer 2
VPNs are not exactly the same thing as VLANs, either.

	And it matters.

	Since we have done this, we need to make sure that we deal with any mess
that arises out of these potential imperfections - particularly for the potential "killer 
applications" that we see (or have pointed out to us) on the horizon.

	We defined them in the IETF, therefore it is (potentially) our mess to clean up.

	For this reason, we need to consider at least the possibility that a strictly L3
overlay solution may not be enough.

	However, if we want to avoid making things worse, then we need to restrict 
our efforts to the bare-minimum "clean-up" required.  For this reason, we need to 
avoid thinking of the work we are doing as explicitly including support for any non-IP 
traffic.

	In some cases, I suspect that the "bare minimum" will comprise use of the
expression "just say no."  The trick will be getting agreement as to when this applies.

	:-)

--
Eric

-----Original Message-----
From: NAPIERALA, MARIA H [mailto:[email-masked]] 
Sent: Thursday, September 27, 2012 11:39 AM
To: Eric Gray
Cc: [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane
Importance: High

Eric,

> 	Since there are already existing solutions for virtualized data 
> centers, I think it is only too obvious that we're conceding that a 
> "one-size fits all" solution is pretty much out of the question.
> 
> 	So, it seems possible (at least) that a solution built around EVPN 
> may be applicable to at least a subset of use cases.
> 

Maria

> The question is how to address *non-IP* traffic, in those DCs that 
> care about such traffic. The best way to answer that question would be 
> to measure such traffic. If a data center has a lot of *non-IP* 
> traffic then it would make sense to use EVPN for intra-subnet 
> forwarding. If non-IP traffic is localized to a small subset of VLANs 
> perhaps it makes more sense to have a solution where EVPN is only used 
> when necessary, rather than to design the entire solution around EVPN.
> 
> Maria
> 
> > VLAN" and we use routing to handle traffic that "is inter-VLAN, 
> > i.e., packets [that] are routed" do we have less than an overall solution?
> >

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Thu, 27 Sep 2012 22:26:13 +0000

Nabil,

We talked this earlier this week while your mail was sent but "napping" in the system somewhere. :-). 

I did not mean that interworking a Layer 2 service having access to an IP routing service, that is done today...
I meant to say, e.g., today we don't establish a single VPN with some sites being l3vpn and others being vpls.

I think we are generally in agreement.

- both L2 and L3 support are needed
- both L2 and L3 can be supported in the same network and on the same device (it is done today)
- What we need to be more careful is wither we should mix different sites with l2 and l3 into the same vpn. It could create complication than its worth. Study/work need to be done if reality really calls for the case.

Luyuan

> Sent: Thursday, September 27, 2012 7:58 AM
> To: Luyuan Fang (lufang); Henderickx, Wim (Wim); Bitar, Nabil N;
> NAPIERALA, MARIA H; Fedyk, Donald (Don); [email-masked]
> >> > co-exist (E-VPN and RFC4364) and get interconnected in
> environments
> >> > that
> >> > need that.
> >>
> >> > >Though E-VPN is leveraging many fundamental building blocks of
> BGP
> >> L3
> >> > >have BGP L3 VPN solution only in the DC and inter-connecting DC
> to
> >> the
> >> > >into this so called "IP VPNs and EVPNs... as one solution", or
> ask
> >> > them
> >> > >to inter-work between existing L3VPN with this new "IP VPN and
> EVPN
> >> > ...
> >> > >as one solution" in DC, which would introduce complexity for
> nothing
> >> > if
> >> > >> there is no choice. As a result, inter-subnet IP forwarding
> which
> >> is
> >> > >> > partition the information such that the both L2 and L3 VPNs
> can
> >> > scale
> >> > >> > independently, each in context (for example VLANs, IP VPNs).
> In
> >> > that
> >> > >> > It needs to be explained why/what are the requirements that
> you
> >> > need
> >> > >> > majority of traffic in a (significant class of) data center
> is
> >> > inter-
> >> > >> > > I think we should try to clarify what problems or what type
> of
> >> > data
> >> > >> > > > From: [email-masked] [mailto:nvo3-
> [email-masked]]
> >> On
> >> > >> > Behalf
> >> > >> > > Of
> >> > >> > > > Thomas Nadeau
> >> > >> > > > A new version of I-D, draft-drake-nvo3-evpn-control-
> plane-
> >> > 00.txt
> >> > >> > > > Title:              A Control Plane for Network
> Virtualized
> >> > >> Overlays
> >> > >> > > > URL:             http://www.ietf.org/internet-
> drafts/draft-
> >> > drake-
> >> > >> > > nvo3-
> >> > >> > > >        act as the control plane for Virtual Extensible
> Local
> >> > Area
> >> > >> > > >        Network (VXLAN), Network Virtualization using
> Generic
> >> > >> > Routing
> >> > >> > > >        additinal data plane encapsulations and functions
> as
> >> > >> needed.
> >> > >> > > >
> >> > >> > > >

---

From: John E Drake <[email-masked]>
Date: Thu, 27 Sep 2012 22:58:10 +0000

Luyuan,

I think the correct model is to keep separate L2 & L3 VPNs but allow selective leaking of info between them. 

Yours irrespectively,

John

> Sent: Thursday, September 27, 2012 3:26 PM
> To: Bitar, Nabil N; Henderickx, Wim (Wim); NAPIERALA, MARIA H; Fedyk,
> Donald (Don); [email-masked]
> I did not mean that interworking a Layer 2 service having access to an
> IP routing service, that is done today...
> - What we need to be more careful is wither we should mix different
> sites with l2 and l3 into the same vpn. It could create complication
> than its worth. Study/work need to be done if reality really calls for
> the case.
> 
> Luyuan
> 
> > routing service as commonly done today?  I wouldn't compare that to
> > FR/ATM interworking.
> >
> > There are applications that require operation over Ethernet and have
> > MAC awareness. There are also dependencies on the other end of the
> > networking technology, is an example and I think there was an email
> > from Dave Black about that some time back. There are probably other
> > apps, proprietary or commercial that rely on Ethernet only. I am
> > raising that not because of E-VPN or not but the general notion being
> > debated by some that we need to take a hard-line saying only IP
> > service needs to be supported, as much as I would like that to be the
> > case.
> >
> > migrated to one that is or you can tunnel to the application
> depending
> > on the application or ..,, it seems to me it is a hardline to say
> that
> > an
> > L2
> > service is not
> > needed in the DC at the functional level of NVE and gateways as it
> can
> > be disruptive in certain cases or create hurdles.
> >
> > The net is that the DC fabric (underlay) can be IP only although for
> > some Ethernet can still be a choice. The service overlaid on that can
> > be IP or bridged Ethernet and that is a choice for people to do based
> > on several constraints including applications they want to support.
> > Today Ethernet and IP services are realities and need to be
> addressed.
> > IP will be always needed, no doubt. If one can live with IP only they
> > would have the tool, if they need a mix of IP of Ethernet bridge
> > services, they should also have the tools. Id people are only
> > interested in IP services, they can only work on that, and if people
> > have the need for the mix or want to work on the mix or or one or the
> > other, they can also do that.
> >
> > Thanks,
> > Nabil
> >
> >
> >
> > >I tried not to go into the mix case :-). Need to better understand
> > >the real needs, its implication, and various (prefer simplified)
> > >potential solutions.
> > >
> > >what are the objectives, how much complication they are willing to
> > >get into, and the benefits, trade-offs.
> > >
> > >deployment after all these years? If not so, why do DC operators
> want
> > to
> > >make it more complex?
> > >
> > >But I want to see if many operators really want to mix things, and
> to
> > >what level.
> > >
> > >Thanks,
> > >Luyuan
> > >
> > >> > To: Luyuan Fang (lufang); NAPIERALA, MARIA H; Fedyk, Donald
> > >> > (Don); [email-masked]
> > >> scaling, the move from L2 to L3 happened already in some largest
> > >> DCs deployed today.
> > >> On the other hand, there are networks must support certain legacy
> > >> L2 services, and some DC operators may be more familiar with L2
> allow
> > >> >bridging when traffic is among endpoints on on same subnet
> > and
> > >> > >> IP VPNs and EVPNs are well suited as one solution for
> > >> > >> extending
> > >> the
> > >> > L2
> > >> > >What do you mean by "one solution"? They are two solutions to
> me.
> > I
> > >> > want
> > >> > >to make sure that we are not redefining BGP L3VPN (RFC 4364,
> aka
> > >> > 2547).
> > >> > >
> > >> > >existing network l3vpn, or inter-connecting DCs, they can use
> > >> > >BGP
> > >> > L3VPN
> > >> > >as its original form (if practical), or extend it to make more
> > >> > >DC friendly (e.g. draft-marques-l3vpn-end-system, as many
> > >> > >pointed to
> > in
> > >> > >their mails).
> > >> > >
> > >> > >L3 VPN has huge deployment base, mostly in SP network space
> over
> > the
> > >> > last
> > >> > >14 years or so, and some recent development in DC space. I
> don't
> > >> > assume
> > >> > >you are implying to ask operators to change their L3VPN from
> > >> > >4364
> > >> > format
> > >> > >they simply need the L3VPN from network to DC. I hope this is
> > >> > >not
> > >> the
> > >> > >case.
> > >> > >
> > >> > >> > The nice property of BGP based IPVPN and EVPN is the
> ability
> > to
> > >> > >> > the L2 and L3 connectivity for large and physically
> disperse
> > >> data
> > >> > >> > centers.
> > >> > >> >
> > >> > >> > Don
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > subnet, a solution should be optimized for such traffic
> and,
> > >> > hence,
> > >> > >> > solving the forwarding of such traffic should be the
> > >> > >> > starting
> > >> > point.
> > >> > >> >
> > >> > >> > Maria
> > >> > >> >
> > >> > >> > > when draft-drake-nvo3-evpn-control-plane was discussed
> but
> > I
> > >> am
> > >> > not
> > >> > >> > > >    A number of us just published this draft and wanted
> > >> > >> > > > to
> > >> > >> bring it
> > >> > >> > > > Htmlized:        http://tools.ietf.org/html/draft-
> drake-
> > >> nvo3-
> > >> > >> evpn-
> > >> > >> > > > control-plane-00
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > Abstract:
> > >> > >> > > >        Network Virtual Overlays.  Currently this
> > >> > >> > > > protocol
> > is
> > >> > >> > defined
> > >> > >> > > to

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Fri, 28 Sep 2012 06:57:51 +0200

Luyuan, mixing L2 and L3 is very common for IAAS.
People will have different groups of networking:
- L2 domains connected via L3
- Something like this:

VM11 ---+          +-- VM21
        |          |
L2 CUG1 + -- L3 -- +   L2 CUG2
        |          |
VM12 ---+          +-- VM22

L2 CUG1 is a web tier
And L3 CUG2 is the business tier

A very common use case which I have seen so far.

-----Original Message-----
From: Luyuan Fang (lufang) [mailto:[email-masked]]
Sent: vrijdag 28 september 2012 0:26
To: Bitar, Nabil N; Henderickx, Wim (Wim); NAPIERALA, MARIA H; Fedyk, Donald (Don); [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Nabil,

Luyuan

> FR/ATM
> MAC
> from
> some

---

From: "Bitar, Nabil N" <[email-masked]>
Date: Fri, 28 Sep 2012 07:56:06 -0400

Or interconnection when needed, as you have L2 access to L3 as Wim's
example depicts.

Thanks,
Nabil

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 16:54:28 +0000

Since layer 3 can address both intra-subnet and inter-subnet IP communication by a uniform forwarding (i.e., there is no difference in forwarding between those two), the question is why to introduce a complication of mixing L2 and L3 forwarding in the *same* solution.
As Kireeti wrote, with which I agree: "mixing L2 and L3 for a tenant (either as VNs or between DC and WAN) is fraught with complications (basically, IRB in a virtualized setting)."
IRB was always intended to be used to solve specific and narrowly defined problems, not to build the entire network on IRB.

Maria

> Sent: Friday, September 28, 2012 12:58 AM
> > the application or ..,, it seems to me it is a hardline to say that
> an
> > L2
> > service is not
> > be
> > Ethernet can still be a choice. The service overlaid on that can be
> IP
> > or
> > >> To: Luyuan Fang (lufang); Bitar, Nabil N; NAPIERALA, MARIA H;
> Fedyk,
> (Don);
> DCs
> DC
> > >> > >friendly (e.g. draft-marques-l3vpn-end-system, as many pointed
> to
> > in
> > >> > >their mails).
> > >> > >

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Fri, 28 Sep 2012 19:56:08 +0200

Maria, there are applications in enterprise that require L2 communication between hosts, same IP subnet, etc. As such the solution needs to accommodate this. Don't get me wrong I also want to make it as simple as possible, but this is what people are looking to do and I believe we cannot ignore it. Enterprise customers will demand to enable this in the Public Cloud.

Had a discussion today with a SP who wants to burst/move capacity, enable disaster recovery in L2 or L3 or L2/L3/IRB constructions and support the model I highlighted below.

-----Original Message-----
From: NAPIERALA, MARIA H [mailto:[email-masked]]
Sent: vrijdag 28 september 2012 18:54
To: Henderickx, Wim (Wim); Luyuan Fang (lufang); Bitar, Nabil N; Fedyk, Donald (Don); [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Maria

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 18:12:22 +0000

Wim,

> Maria, there are applications in enterprise that require L2
> communication between hosts, same IP subnet, etc. As such the solution
> needs to accommodate this. Don't get me wrong I also want to make it as
> simple as possible, but this is what people are looking to do and I
> believe we cannot ignore it. Enterprise customers will demand to enable
> this in the Public Cloud.

Why would a customer care how a *Public* Cloud delivers IP traffic between VMs/appliances, as long as the traffic is delivered successfully? If this is just IP-subnet issue then this no issue since the traffic is handled the same way by a VM whether
one implements EVPN or L3VPN. There is no visible difference.

Maria

> Since layer 3 can address both intra-subnet and inter-subnet IP
> communication by a uniform forwarding (i.e., there is no difference in
> forwarding between those two), the question is why to introduce a
> complication of mixing L2 and L3 forwarding in the *same* solution.
> As Kireeti wrote, with which I agree: "mixing L2 and L3 for a tenant
> (either as VNs or between DC and WAN) is fraught with complications
> (basically, IRB in a virtualized setting)."
> IRB was always intended to be used to solve specific and narrowly
> defined problems, not to build the entire network on IRB.
> 
> Maria
> 
> 
> > We talked this earlier this week while your mail was sent but
> "napping"
> > in the system somewhere. :-).
> >
> > > only work on that, and if people have the need for the mix or want
> to
> > > work
> > > >sites are L2, some are L3 within one VPN, or inter-connecting LX
> DC
> > > with
> > > >LY VPN network...
> > > >
> > > >> > allow bridging when traffic is among endpoints on on same
> subnet
> > > and
> > > >> > >existing L2 services and prefer L2 solutions with E-VPN. If
> one
> > > >> wants
> > > >> > to
> > > >> > >My point is, leave BGP L3VPN and EVPN as two separate
> solutions
> > > as
> > > >> > they
> > > >> > >are.
> > > >> > >
> > > >> > >Thanks,
> > > >> > >Luyuan
> > > >> > >
> > > >> > >> > is seems better to provide both L2 and L3 capabilities
> and
> > > let
> > > >> > >> trivial in L3VPN (as is intra-subnet forwarding, btw)
> becomes
> > > all
> > > >> of
> > > >> > a
> > > >> > >> > sense IP VPNs and EVPNs are well suited as one solution
> for
> > > >> > extending
> > > >> > >> > both (IPVPN and EVPN) in the same solution because this
> is
> > > not a
> > > >> > >> > trivial complication.
> > > >> > >> > The complexity of the solution depends on the
> requirements.
> > > If
> http://datatracker.ietf.org/doc/draft-
> > > >> drake-
> > > >> > >> nvo3-
> > > >> > >> > > evpn-
> > > >> > >> > > > control-plane

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Fri, 28 Sep 2012 20:18:50 +0200

Maria, there is since a customer/enterprise will demand a L2 communication or L3 communication for certain Apps/VMs. As such the cloud provider needs to accommodate it. You cannot transport a broadcast packet via L3, neither can you route a packet in the same subnet as you would change the TTL and if it is 1 it will be dropped. As such they care since apps break.

EVPN allows to provide a L2/L3 communication.
L3VPN does not since you do not transport the MAC address.
You could also solve this by doing both EVPN and L3VPN.

These are the options you have so far.
WIm

-----Original Message-----
From: NAPIERALA, MARIA H [mailto:[email-masked]]
Sent: vrijdag 28 september 2012 20:12
To: Henderickx, Wim (Wim); Luyuan Fang (lufang); Bitar, Nabil N; Fedyk, Donald (Don); [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Wim,

Maria

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Fri, 28 Sep 2012 13:43:13 -0500

Maria,

Keep in mind the DC environment has and interacts with a lot of legacy L2 environment that does not have a control plane: i.e. VLANs, VPLS, IEEE new protocols or proprietary. So there is a lot of flooded traffic coming towards the new Cloud environment (NVO3/VPN4DC based). As you probably know it takes a really long time to get rid of the legacy stuff, unfortunately. :(

Even for the new cloud environment I always wondered whether a cloud provider can actually impose/guarantee that the Application running in the user space does not send any flooded traffic (Broadcast/multicast)? This is on top of the ARP broadcast that we can eliminate using EVPN/SDN controllers.

Florin

> Henderickx, Wim (Wim)
> Sent: Friday, September 28, 2012 11:19 AM
> To: NAPIERALA, MARIA H; Luyuan Fang (lufang); Bitar, Nabil N; Fedyk,
> Maria, there is since a customer/enterprise will demand a L2
> communication or L3 communication for certain Apps/VMs. As such the
> cloud provider needs to accommodate it. You cannot transport a
> broadcast packet via L3, neither can you route a packet in the same
> subnet as you would change the TTL and if it is 1 it will be dropped.
> As such they care since apps break.
>
> > needs to accommodate this. Don't get me wrong I also want to make it
> > as simple as possible, but this is what people are looking to do and
> I
> > believe we cannot ignore it. Enterprise customers will demand to
> > enable this in the Public Cloud.
>
> Why would a customer care how a *Public* Cloud delivers IP traffic
> between VMs/appliances, as long as the traffic is delivered
> successfully? If this is just IP-subnet issue then this no issue since
> the traffic is handled the same way by a VM whether one implements EVPN
> or L3VPN. There is no visible difference.
>
> Maria
>
> >
> > > I meant to say, e.g., today we don't establish a single VPN with
> > > some sites being l3vpn and others being vpls.
> > >
> > > > Do you mean by interworking a Layer2 service having access to an
> > > > IP routing service as commonly done today?  I wouldn't compare
> > > > that to FR/ATM interworking.
> > > >
> > > > There are other applications, legacy or not, that are not IP
> aware
> > > and
> > > > will require Layer2 connectivity. Storage, depending on the
> > > > storage networking technology, is an example and I think there
> was
> > > > an email from Dave Black about that some time back. There are
> > > > probably other
> > apps,
> > > > proprietary or commercial that rely on Ethernet only. I am
> raising
> > > that
> > > > not because of E-VPN or not but the general notion being debated
> > > > by some that we need to take a hard-line saying only IP service
> > > > needs to be supported, as much as I would like that to be the
> > > > case.
> > > >
> > > > While one can argue that if an application is not IP-based, it
> can
> > be
> > > > and IP services are realities and need to be addressed. IP will
> be
> > > > always needed, no doubt. If one can live with IP only they would
> > > > have the tool, if they need a mix of IP of Ethernet bridge
> > > > services, they should
> > > also
> > > > have the tools. Id people are only interested in IP services,
> they
> > > can
> > > > >infrastructure, or it could mean inter-working if you are
> > > > >thinking
> > > > some
> > > > >> While I understand there are customers for L3, there are
> > > > >> equally customers for L2 as you said, but also for a mix when
> > application
> > > > >> tiering is required.
> > > > >> > On 9/24/12 3:12 PM, "Luyuan Fang (lufang)"
> <[email-masked]>
> > > > wrote:
> > > > >> >
> > > > >> > >Don,
> > > > >> > >
> > > > >> > >VPN, it is essentially developed for people who must
> support
> > > > certain
> > > > >> > >have BGP L3 VPN solution only in the DC and inter-
> connecting
> > DC
> > > > to
> > > > >> the
> > > > >> > >> > > of the same building blocks as RFC4364 so that it
> > > > >> > >> > > could
> > > be
> > > > >> used
> > > > >> > in
> > > > >> > >> > > I think we should try to clarify what problems or
> what
> > > type
> > > > of
> > > > >> > data
> > > > >> > >> > > Specifically, EVPN can only address traffic bridged
> in
> > > the
> > > > >> same
> > > > >> > >> VLAN.
> > > > >> > >> > > >        existing data plane encapsulations. The
> > > > >> > >> > > > intent
> > > is
> > > > >> that
> > > > >> > >> this
> > > > >> > >> > > >        protocol will be capable of extensions in
> the
> > > > future
> > > > >> to
> > > > >> > >> > handle

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 19:04:41 +0000

Wim,
> broadcast packet via L3, 

Yes, you can.

> neither can you route a packet in the same

> L3VPN does not since you do not transport the MAC address.

Yes. 
But why would you need to transport MAC addresses? Applications do not send or receive Ethernet frames directly, they are restricted to IP services.
The requirement is to provide an "IP subnet" and traffic exchange between "IP subnets". IP service is: unicast, subnet broadcast, multicast.
Are you saying that the requirement for layer 2 service is to address the interaction with IEEE bridging?

> You could also solve this by doing both EVPN and L3VPN.

Yes. And IMO, it is better to keep layer 2 and layer 3 solutions separate - cleaner, less complex, no virtualized IRB.
I would rather design around the inter-subnet traffic exchange first (as it is majority of traffic), and then address intra-subnet, and whether interaction with bridging is required (because this is where complexity "lives"..)

Maria

> Sent: Friday, September 28, 2012 2:19 PM
> the traffic is handled the same way by a VM whether
> > To: Henderickx, Wim (Wim); Luyuan Fang (lufang); Bitar, Nabil N;
> Fedyk,
> > > sites being l3vpn and others being vpls.
> > >
> > > - both L2 and L3 can be supported in the same network and on the
> same
> > > device (it is done today)
> storage
> by
> > > > always
> > > > needed, no doubt. If one can live with IP only they would have
> the
> > > > tool,
> equally

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Fri, 28 Sep 2012 21:08:30 +0200

-----Original Message-----
From: NAPIERALA, MARIA H [mailto:[email-masked]]
Sent: vrijdag 28 september 2012 21:05
To: Henderickx, Wim (Wim); Luyuan Fang (lufang); Bitar, Nabil N; Fedyk, Donald (Don); [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

Wim,

Yes, you can.

WH> How?

Yes.

WH> yes you need to do IEEE bridging and this is what customers want

WH> you can but you need to do some interaction which can be a local implementation matter.

Maria

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 19:24:12 +0000

> > cloud provider needs to accommodate it. You cannot transport a

RFC 6513 - "Announcing the Presence of Unsolicited Flooded Data".

> But why would you need to transport MAC addresses? Applications do not
> send or receive Ethernet frames directly, they are restricted to IP
> services.
> The requirement is to provide an "IP subnet" and traffic exchange
> between "IP subnets". IP service is: unicast, subnet broadcast,
> multicast.

Are you referring to your customers?
 
> Yes. And IMO, it is better to keep layer 2 and layer 3 solutions
> separate - cleaner, less complex, no virtualized IRB.
> I would rather design around the inter-subnet traffic exchange first
> (as it is majority of traffic), and then address intra-subnet, and
> whether interaction with bridging is required (because this is where
> complexity "lives"..)
> 
> > > > I did not mean that interworking a Layer 2 service having access
> to
> > > an
> > > > than its worth. Study/work need to be done if reality really
> calls
> > > for
> > > > the case.
> > > >
> > > > Luyuan
> > > >
> > > > > routing service as commonly done today?  I wouldn't compare
> that
> > to
> > > > > FR/ATM
> > > > > interworking.
> > > > >
> > > > > that we need to take a hard-line saying only IP service needs
> to
> > be
> > > > > The net is that the DC fabric (underlay) can be IP only
> although
> > > for
> > > > > some
> > > > > Ethernet can still be a choice. The service overlaid on that
> can
> > be
> > > > IP
> > > > > or
> > > > > if they need a mix of IP of Ethernet bridge services, they
> should
> > > > also
> > > > > >Technically, these can all be done. Practically, we need to
> ask
> > > > > operators
> > > > > >what are the objectives, how much complication they are
> willing
> > to
> > > > get
> > > > > >Do we see a lot of mixed case for L2/L3 VPNs in today's
> network
> > > VPN
> > > > > >In my past experience on the operator side, inter-working
> (e.g.
> > > > > FR/ATM,
> > > > > >ATM/Ethernet, MPLS/Ethernet...) were good on papers, but
> always
> > > too
> > > > > >> Personally, I see L3 (in general sense) is becoming the
> trend
> > > for
> > > > > >> environment so prefer l2 solutions, or there is need to
> inter-
> > > > > connect
> > > > > >> > >>and L3 connectivity for large and physically disperse
> data
> > > > > centers.
> > > > > >> > >
> > > > > >> > >What do you mean by "one solution"? They are two
> solutions
> > to
> > > > me.
> > > > > I
> > > > > >> > want
> > > > > >> > >Though E-VPN is leveraging many fundamental building
> blocks
> > > of
> > > > > BGP
> > > > > >> L3
> > > > > >> > >into this so called "IP VPNs and EVPNs... as one
> solution",
> > > or
> > > > > ask
> > > > > >> > them
> > > > > >> > >to inter-work between existing L3VPN with this new "IP
> VPN
> > > and
> > > > > EVPN
> > > > > >> > ...
> > > > > >> > >they simply need the L3VPN from network to DC. I hope
> this
> > is
> > > > not
> > > > > >> the
> > > > > >> > >case.
> > > > > >> > >
> > > > > >> > >> > The complexity comes from the date center
> environment,
> > > but
> > > > > >> > >> > partition the information such that the both L2 and
> L3
> > > VPNs
> > > > > can
> > > > > >> > scale
> > > > > >> > >> > > Perhaps I'm missing something but I thought EVPN
> was
> > > > built
> > > > > >> using
> > > > > >> > >> many
> > > > > >> > >> > It needs to be explained why/what are the
> requirements
> > > that
> > > > > you
> > > > > >> > need
> > > > > >> > >> > > > draft at the interim meeting this week as well,
> but
> > > > > please
> > > > > >> > >> discuss
> > > > > >> > >> > > here
> > > > > >> > >> > > > on the list as well.
> > > > > >> > >> > > >
> > > > > >> > >> > > >    Thanks,
> > > > > >> > >> > > >
> > > > > >> > >> > > >    Tom, John, et al
> > > > > >> > >> > > >
> > > > > >> > >> > > >
> > > > > >> > >> > > > has been successfully submitted by Thomas D.
> Nadeau
> > > and
> > > > > >> posted
> > > > > >> > to
> > > > > >> > >> > the
> > > > > >> > >> > > > IETF repository.
> > > > > >> > >> > > >
> http://tools.ietf.org/html/draft-
> > > > drake-
> > > > > >> nvo3-
> > > > > >> > >> evpn-
> > > > > >> > >> > > > control-plane-00
> > > > > >> > >> > > >
> > > > > >> > >> > > >
> > > > > >> > >> > > > Abstract:
> > > > > >> > >> > > >        The purpose of this document is to
> describe
> > > how
> > > > > >> > Ethernet
> > > > > >> > >> > > Virtual

---

From: John E Drake <[email-masked]>
Date: Fri, 28 Sep 2012 19:38:10 +0000

Comment inline.

Yours irrespectively,

John

> Sent: Friday, September 28, 2012 9:54 AM

JD:  Maria, did you invent some new technology that you're not telling anyone about?

---

From: "Bitar, Nabil N" <[email-masked]>
Date: Fri, 28 Sep 2012 15:42:22 -0400

Hi,
Can you if a VM needs to know that far end VM MAC, aside from the other
cases? I believe your basic assumption is that the only need is to get the
IP packet from one VM to another and the applications running in the VMs
are communicating over IP only. That is aside from the other cases
mentioned.

Thanks,
Nabil

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 20:05:26 +0000

Let me refer you the following e-mail (attached) in case you missed it.

Maria

> Sent: Friday, September 28, 2012 3:38 PM
> To: NAPIERALA, MARIA H; Henderickx, Wim (Wim); Luyuan Fang (lufang);
> Bitar, Nabil N; Fedyk, Donald (Don); [email-masked]

Subject: Re: [nvo3] draft-drake-nvo3-evpn-control-plane
From: Yakov Rekhter <[email-masked]>
To: "Fedyk, Donald (Don)" <[email-masked]>
Cc: "NAPIERALA, MARIA H" <[email-masked]>, "[email-masked]" <[email-masked]>
Date: Mon, 24 Sep 2012 14:51:42 +0000

Don,

Yakov.

---

From: John E Drake <[email-masked]>
Date: Fri, 28 Sep 2012 20:23:03 +0000

Maria,

I had seen that email and I helped Yakov with the draft it references.

Because you think the section in question bolsters your argument, I think it is painfully clear that you haven't read the section, or the draft for that matter. 

Yours irrespectively,

John

> Sent: Friday, September 28, 2012 1:05 PM
> To: John E Drake; Henderickx, Wim (Wim); Luyuan Fang (lufang); Bitar,
> Nabil N; Fedyk, Donald (Don); [email-masked]
> > JD:  Maria, did you invent some new technology that you're not
> telling
> > anyone about?
> >

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 20:24:24 +0000

Florin

> Keep in mind the DC environment has and interacts with a lot of legacy
> L2 environment that does not have a control plane: i.e. VLANs, VPLS,
> IEEE new protocols or proprietary. So there is a lot of flooded traffic
> coming towards the new Cloud environment (NVO3/VPN4DC based). As you
> probably know it takes a really long time to get rid of the legacy
> stuff, unfortunately. :(
> 
> Even for the new cloud environment I always wondered whether a cloud
> provider can actually impose/guarantee that the Application running in
> the user space does not send any flooded traffic (Broadcast/multicast)?
> This is on top of the ARP broadcast that we can eliminate using
> EVPN/SDN controllers.

I did not say that flooded traffic (broadcast/multicast) does not need to be covered. It needs to be covered, for the reason you mentioned. L3VPN can also deliver such traffic.

> > > Fedyk, Donald (Don); [email-masked]
> > > > same device (it is done today)

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Fri, 28 Sep 2012 22:37:19 +0200

-----Original Message-----
From: NAPIERALA, MARIA H [mailto:[email-masked]]
Sent: vrijdag 28 september 2012 21:24
To: Henderickx, Wim (Wim); Luyuan Fang (lufang); Bitar, Nabil N; Fedyk, Donald (Don); [email-masked]
Subject: RE: [nvo3] draft-drake-nvo3-evpn-control-plane

WH2> this would not work for the use case I mentioned: broadcast flooding, TTL 1 apps

WH2> yes

---

From: John E Drake <[email-masked]>
Date: Fri, 28 Sep 2012 20:46:43 +0000

Maria,

Snipped, comment inline

Yours irrespectively,

John

> (as it is majority of traffic),

JD:  There is that pesky assertion again.

When we last left this discussion, you claimed to have data that supported this assertion but you weren't going to share it with us, presumably because it is held under double secret probation. 

> and then address intra-subnet, and

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 20:49:39 +0000

John,

Let me quote from the document (section 5):

  "MES(es) should be able to distinguish between intra-CUG/intra-subnet and
   inter-CUG/inter-subnet traffic originated by that VM (in order to
   apply MAC-based forwarding to the former and IP-based forwarding to
   the latter)."

(section 6.1):

  "When a MES imports an E-VPN route into a particular EVI, and this
   route is associated with a VM that is not part of the L2-based CUG
   associated with the E-VPN of that EVI, the MES creates IP forwarding
   state to forward traffic to the IP address present in the NLRI of the
   route towards the Next Hop, as specified in the route."

Maria

> John E Drake
> Sent: Friday, September 28, 2012 4:23 PM
> To: NAPIERALA, MARIA H; John E Drake; Henderickx, Wim (Wim); Luyuan
> Fang (lufang); Bitar, Nabil N; Fedyk, Donald (Don); [email-masked]
> Because you think the section in question bolsters your argument, I
> think it is painfully clear that you haven't read the section, or the
> draft for that matter.
> 
> Yours irrespectively,
> 
> John
> 
> 
> > > > communication by a uniform forwarding (i.e., there is no
> difference
> > > in

---

From: John E Drake <[email-masked]>
Date: Fri, 28 Sep 2012 21:06:02 +0000

Maria,

This is what you said in the email to which I initially responded:

" Since layer 3 can address both intra-subnet and inter-subnet IP communication by a uniform forwarding (i.e., there is no difference in forwarding between those two), the question is why to introduce a complication of mixing L2 and L3 forwarding in the *same* solution."

Since the text you quote, below, clearly contradicts your text, above, I would assert that you are arguing with yourself in public and that you lost.

Yours irrespectively,

John

> Sent: Friday, September 28, 2012 1:50 PM
> > Of John E Drake
> > > To: John E Drake; Henderickx, Wim (Wim); Luyuan Fang (lufang);
> > > > > forwarding between those two), the question is why to introduce
> > > > > a complication of mixing L2 and L3 forwarding in the *same*
> > solution.
> > > >

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Fri, 28 Sep 2012 21:16:44 +0000

In L3VPN there is only one type of forwarding and there is no distinction needed between intra- and inter-subnet traffic.

Maria

> Sent: Friday, September 28, 2012 5:06 PM
> " Since layer 3 can address both intra-subnet and inter-subnet IP
> complication of mixing L2 and L3 forwarding in the *same* solution."
> 
> Since the text you quote, below, clearly contradicts your text, above,
> I would assert that you are arguing with yourself in public and that
> you lost.
> 
> Yours irrespectively,
> 
> John
> 
> 
> >   "MES(es) should be able to distinguish between intra-CUG/intra-
> subnet
> > and
> > > > Let me refer you the following e-mail (attached) in case you
> missed
> > > it.
> > > >
> > > > Maria
> > > >
> > > >