Subject: [nvo3] FW: New Version Notification for draft-yong-nvo3-frwk-dpreq-addition-00.txt

From: Lucy yong <[email-masked]>
Date: Tue, 11 Dec 2012 22:18:54 +0000

Hello,

FYI. We just upload this draft. It provides additions to the nvo3 framework and data plane requirement beside current documents. The objective of this draft is to merge the suggested text into the framework and data plane requirement documents.

Your comments on this are welcome.

Regards,
Lucy

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, December 11, 2012 4:04 PM
> To: Lucy yong
> Cc: Linda Dunbar
> Subject: New Version Notification for draft-yong-nvo3-frwk-dpreq-
> addition-00.txt
> 
> 
> A new version of I-D, draft-yong-nvo3-frwk-dpreq-addition-00.txt
> has been successfully submitted by Lucy Yong and posted to the
> IETF repository.
> 
> Filename:	 draft-yong-nvo3-frwk-dpreq-addition
> Revision:	 00
> Title:		 NVO3 Framework and Data Plane Requirement Addition
> Creation date:	 2012-12-11
> WG ID:		 Individual Submission
> Number of pages: 9
> URL:             http://www.ietf.org/internet-drafts/draft-yong-nvo3-
> frwk-dpreq-addition-00.txt
> Status:          http://datatracker.ietf.org/doc/draft-yong-nvo3-frwk-
> dpreq-addition
> Htmlized:        http://tools.ietf.org/html/draft-yong-nvo3-frwk-dpreq-
> addition-00
> 
> 
> Abstract:
>    This document describes some additional functions and requirements
>    for NVO3 framework [NVO3FRWK] and data plane requirements [DPREQ].
>    These additions are necessary in supporting VM communication and
>    mobility.
> 
> 
> 
> 
> 
> The IETF Secretariat

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Thu, 13 Dec 2012 11:17:10 +0100

Lucy,

The statement "Neither of them naturally supports the communication among the VMs
When some VMs are on the same subnet and other on different" in the introduction of 
your draft is inaccurate.

Integrated routing and switching is configuration/implementation specific to a router.
Can you point to a specific RFC that describes IRB and a new interface type?
This capibility can be enabled as already mentioned in the dataplane requirements draft.
Hence, there is no need to specify a new NVE service type.

As far as mobility is concerned, I'm finishing up a new section for the framework draft.

Marc

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Lucy yong
> Sent: Tuesday, December 11, 2012 11:19 PM
> To: [email-masked]; [email-masked]; Bocci, Matthew (Matthew)
> Cc: [email-masked]; 
> [email-masked]
> Subject: [nvo3] FW: New Version Notification for 
> draft-yong-nvo3-frwk-dpreq-addition-00.txt
> 
> Hello,
> 
> FYI. We just upload this draft. It provides additions to the 
> nvo3 framework and data plane requirement beside current 
> documents. The objective of this draft is to merge the 
> suggested text into the framework and data plane requirement 
> documents.
> 
> > has been successfully submitted by Lucy Yong and posted to the IETF 
> > repository.
> > 
> > URL:             
> http://www.ietf.org/internet-drafts/draft-yong-nvo3-
> > Status:          
> http://datatracker.ietf.org/doc/draft-yong-nvo3-frwk-
> > dpreq-addition
> > Htmlized:        
> http://tools.ietf.org/html/draft-yong-nvo3-frwk-dpreq-
> > addition-00
> > 
> > 
> > Abstract:
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: Lucy yong <[email-masked]>
Date: Thu, 13 Dec 2012 17:16:09 +0000

Marc,

NVO3 work is not just to repeat what we have done before. I am not aware of a specific RFC describes IRB. But as you pointed out, IRB is widely deployed in the physical networks today, but it is not in VPN bases. Please look at Microsoft hyper-v, it also implements IRB for network virtualization. 

I know that the combination of L2 NVE and L3 NVE can serve some applications and it is useful, but it is not equivalent to IRB function. To support IRB function, the data plane has to perform MAC address translation and prevent the broadcast traffic in one subnet network from sending to another. There is no such requirement in current data plane requirement document.  

Hope this convince you a new service type of NVE.

Regards,
Lucy

> From: LASSERRE, MARC (MARC) [mailto:[email-masked]]
> Sent: Thursday, December 13, 2012 4:17 AM
> To: Lucy yong; [email-masked]; [email-masked]; Bocci, Matthew
> (Matthew)
> Cc: [email-masked]; draft-bl-nvo3-dataplane-
> [email-masked]
> Subject: RE: [nvo3] FW: New Version Notification for draft-yong-nvo3-

---

From: "Yves Hertoghs (yhertogh)" <[email-masked]>
Date: Thu, 13 Dec 2012 21:52:16 +0000

Lucy,

Why would IRB require MAC Address translation. IRB is nothing else than sticking an internal routed interface on top of a bridge domain, with L2 interfaces sticking out of the node.  Packets destined to the mac-address associated with the routed interface get routed (and would show up at the network side with a new mac-header), other ones get bridged.

Yves

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Lucy yong
> Sent: Thursday, December 13, 2012 18:16
> To: LASSERRE, MARC (MARC); [email-masked]; [email-masked]; Bocci,
> Matthew (Matthew)
> Subject: Re: [nvo3] FW: New Version Notification for draft-yong-nvo3-frwk-
> dpreq-addition-00.txt
> 
> Marc,
> 
> NVO3 work is not just to repeat what we have done before. I am not aware
> of a specific RFC describes IRB. But as you pointed out, IRB is widely deployed
> in the physical networks today, but it is not in VPN bases. Please look at
> Microsoft hyper-v, it also implements IRB for network virtualization.
> 
> I know that the combination of L2 NVE and L3 NVE can serve some
> applications and it is useful, but it is not equivalent to IRB function. To
> support IRB function, the data plane has to perform MAC address translation
> and prevent the broadcast traffic in one subnet network from sending to
> another. There is no such requirement in current data plane requirement
> document.
> 
> > The statement "Neither of them naturally supports the communication
> > among the VMs When some VMs are on the same subnet and other on
> > different" in the introduction of your draft is inaccurate.
> >
> > > From: [email-masked] [mailto:[email-masked]] On Behalf
> > > Of Lucy yong
> > > nvo3 framework and data plane requirement beside current documents.
> > > The objective of this draft is to merge the suggested text into the
> > > framework and data plane requirement documents.
> > >

---

From: Lucy yong <[email-masked]>
Date: Thu, 13 Dec 2012 22:28:55 +0000

Yves,

As the draft describes, TS uses ARP to discover the destination MAC that are on the same subnet and send Ethernet frame to other TS on the same subnet directly. TS sends the Ethernet frame to the gateway if the destination of the packets are on different subnet. L2-3 NVE acts as the gateway for a Tenant virtual network. When it gets the packets, it performs an IP look-up and find the corresponding MAC address, and replace the gateway MAC address with the MAC address before sending it out. TS uses ARP to find out the gateway MAC address and cache it. I call this as MAC address translation in data plane.

Hope this is clear.

Lucy

> From: Yves Hertoghs (yhertogh) [mailto:[email-masked]]
> Sent: Thursday, December 13, 2012 3:52 PM
> To: Lucy yong; LASSERRE, MARC (MARC); [email-masked];
> [email-masked]; Bocci, Matthew (Matthew)
> Why would IRB require MAC Address translation. IRB is nothing else than
> sticking an internal routed interface on top of a bridge domain, with
> L2 interfaces sticking out of the node.  Packets destined to the mac-
> address associated with the routed interface get routed (and would show
> up at the network side with a new mac-header), other ones get bridged.
> 
> Yves
> 
> 
> 
> Of
> > > Can you point to a specific RFC that describes IRB and a new
> interface
> > > type?
> > > This capibility can be enabled as already mentioned in the
> dataplane
> > > requirements draft.
> Behalf

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 14 Dec 2012 11:14:15 +0100

Lucy,

There is nothing new here. It is standard bridging and routing.
Hence IRB does not call for a new service type.

Marc

> Sent: Thursday, December 13, 2012 11:29 PM
> To: Yves Hertoghs (yhertogh); LASSERRE, MARC (MARC); 
> [email-masked]; [email-masked]; Bocci, Matthew (Matthew)
> Subject: Re: [nvo3] FW: New Version Notification for 
> As the draft describes, TS uses ARP to discover the 
> destination MAC that are on the same subnet and send Ethernet 
> frame to other TS on the same subnet directly. TS sends the 
> Ethernet frame to the gateway if the destination of the 
> packets are on different subnet. L2-3 NVE acts as the gateway 
> for a Tenant virtual network. When it gets the packets, it 
> performs an IP look-up and find the corresponding MAC 
> address, and replace the gateway MAC address with the MAC 
> address before sending it out. TS uses ARP to find out the 
> gateway MAC address and cache it. I call this as MAC address 
> translation in data plane.
> 
> Hope this is clear.
> 
> Lucy
> 
> > Why would IRB require MAC Address translation. IRB is nothing else 
> > than sticking an internal routed interface on top of a 
> bridge domain, 
> > with
> > address associated with the routed interface get routed (and would 
> > show up at the network side with a new mac-header), other 
> ones get bridged.
> > 
> > Yves
> > 
> > 
> > 
> > > Subject: Re: [nvo3] FW: New Version Notification for 
> > > draft-yong-nvo3-
> > frwk-
> > > dpreq-addition-00.txt
> > >
> > > Marc,
> > >
> > > and prevent the broadcast traffic in one subnet network 
> from sending
> > to
> > > > The statement "Neither of them naturally supports the 
> > > > communication among the VMs When some VMs are on the 
> same subnet 
> > > > and other on different" in the introduction of your 
> draft is inaccurate.
> > > >
> > > > Integrated routing and switching is 
> configuration/implementation 
> > > > specific to a router.
> > > > > The objective of this draft is to merge the suggested 
> text into
> > the
> > > > > >    These additions are necessary in supporting VM 
> > > > > > communication
> > and
> > > > > >    mobility.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The IETF Secretariat
> > > > >

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 14 Dec 2012 11:19:46 +0100

See below. 

> From: Lucy yong [mailto:[email-masked]] 
> Sent: Thursday, December 13, 2012 6:16 PM
> To: LASSERRE, MARC (MARC); [email-masked]; 
> Subject: RE: [nvo3] FW: New Version Notification for 
> NVO3 work is not just to repeat what we have done before. I 
> am not aware of a specific RFC describes IRB. But as you 
> pointed out, IRB is widely deployed in the physical networks 
> today, but it is not in VPN bases. Please look at Microsoft 
> hyper-v, it also implements IRB for network virtualization. 

And the reason why there is no IRB RFC is because it is *standard* bridging and routing integrated in one router.
IRB has nothing to do with network virtualization.

> 
> I know that the combination of L2 NVE and L3 NVE can serve 
> some applications and it is useful, but it is not equivalent 
> to IRB function. To support IRB function, the data plane has 
> to perform MAC address translation and prevent the broadcast 
> traffic in one subnet network from sending to another. There 
> is no such requirement in current data plane requirement document.  
> 
> > From: LASSERRE, MARC (MARC) 
> [mailto:[email-masked]]
> > Can you point to a specific RFC that describes IRB and a 
> new interface 
> > type?
> text into the 
> communication and

---

From: Lucy yong <[email-masked]>
Date: Fri, 14 Dec 2012 15:28:58 +0000

Marc,

I have to say there is nothing new there, L2 VPN and L3 VPN have been standardized long time ago and deployed widely. Why do we work on this now and standardize them again?

Let me also answer why we work on "here" and "there" now. L2VPN and L3VPN was developed for network service providers who can offer services to customers. The customers typically have either a switch or router to interface with service provider edge device that VPN function enables. IRB function has been used a lot in the enterprise networks, but not in service provider's. L2VPN provides Ethernet LAN like networking, where a customer can use it for the communication on one subnet. L3VPN provides IP routing across WAN network, where a customer can use it for the communication among one or different subnets at different sites, but CE at a site has to use layer 3 interface. 

A tenant virtual network in DC may contain one or more subnets. It requires bridging mechanism for the communication within one subnet and routing mechanism for the communication among one or different subnets with Ethernet interface only, i.e. CE has be L2 interface. The latter is similar to the IRB function.

Current IRB function on router is local function, so there is no need to be standardized. However, in a tenant virtual network, TSes on the same subnet can be on a same or different NVEs and/or moves to different NVEs, so it is no longer a local function. IMO: this is the reason why we need to define a new service type for this.

Regards,
Lucy  

> Sent: Friday, December 14, 2012 4:14 AM
> To: Lucy yong; Yves Hertoghs (yhertogh); [email-masked];
> > > > NVO3 work is not just to repeat what we have done before. I am
> not
> > > aware
> > > > in the physical networks today, but it is not in VPN bases.
> Please
> > > look at
> > > > > Subject: RE: [nvo3] FW: New Version Notification for draft-
> yong-
> > > nvo3-

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 14 Dec 2012 17:30:55 +0100

First of all, I do not understand why you keep refering to VPN work. This is not what we are discussing here.

Second of all, IRB is a useful feature that is applicable to NVO3 AND it has been specifically mentioned in the dataplane requirements draft.
But IRB does not call for a new service type.

Third of all, there is no need to repeat over and over again the same sentences. We all know what bridging, routing, and IRB is.
It is a waste of time and bandwidth.
 
End of this email thread on my side. Let's hear what others have to say about it.

-Marc

> Sent: Friday, December 14, 2012 4:29 PM
> To: LASSERRE, MARC (MARC); Yves Hertoghs (yhertogh); 
> I have to say there is nothing new there, L2 VPN and L3 VPN 
> have been standardized long time ago and deployed widely. Why 
> do we work on this now and standardize them again?
> 
> Let me also answer why we work on "here" and "there" now. 
> L2VPN and L3VPN was developed for network service providers 
> who can offer services to customers. The customers typically 
> have either a switch or router to interface with service 
> provider edge device that VPN function enables. IRB function 
> has been used a lot in the enterprise networks, but not in 
> service provider's. L2VPN provides Ethernet LAN like 
> networking, where a customer can use it for the communication 
> on one subnet. L3VPN provides IP routing across WAN network, 
> where a customer can use it for the communication among one 
> or different subnets at different sites, but CE at a site has 
> to use layer 3 interface. 
> 
> A tenant virtual network in DC may contain one or more 
> subnets. It requires bridging mechanism for the communication 
> within one subnet and routing mechanism for the communication 
> among one or different subnets with Ethernet interface only, 
> i.e. CE has be L2 interface. The latter is similar to the IRB 
> function.
> 
> Current IRB function on router is local function, so there is 
> no need to be standardized. However, in a tenant virtual 
> network, TSes on the same subnet can be on a same or 
> different NVEs and/or moves to different NVEs, so it is no 
> longer a local function. IMO: this is the reason why we need 
> to define a new service type for this.
> 
> Regards,
> Lucy  
> 
> [email-masked]; 
> destination MAC 
> > > that are on the same subnet and send Ethernet frame to 
> other TS on 
> > > the same subnet directly. TS sends the Ethernet frame to 
> the gateway 
> > > if the destination of the packets are on different 
> subnet. L2-3 NVE 
> > > acts as the gateway for a Tenant virtual network. When it 
> gets the 
> > > packets, it performs an IP look-up and find the corresponding MAC 
> > > address, and replace the gateway MAC address with the MAC address 
> > > before sending it out. TS uses ARP to find out the gateway MAC 
> > > address and cache it. I call this as MAC address 
> translation in data 
> > > plane.
> > >
> > > Hope this is clear.
> > >
> > > Lucy
> > >
> [email-masked];
> > > > Bocci,
> > > > > of a specific RFC describes IRB. But as you pointed 
> out, IRB is
> > > > widely deployed
> > > > > support IRB function, the data plane has to perform 
> MAC address
> > > > translation
> > > > > > > From: [email-masked] 
> [mailto:[email-masked]] 
> > > > > > > On
> > > > Behalf
> > > > > > > Of Lucy yong
> > > > > > > > Subject: New Version Notification for 
> > > > > > > > draft-yong-nvo3-frwk-
> > > > dpreq-
> > > > > > > > addition-00.txt
> > > > > > > >
> > > > > > > >
> > > > > > > > A new version of I-D,

---

From: Lucy yong <[email-masked]>
Date: Fri, 14 Dec 2012 17:55:53 +0000

Please see the below.

> Sent: Friday, December 14, 2012 10:31 AM
[Lucy] because I don't understand why you ask if there is RFC to describe the IRB. 
> 
> Second of all, IRB is a useful feature that is applicable to NVO3 AND
> it has been specifically mentioned in the dataplane requirements draft.
[Lucy] because, in NVO3, TSes on the same and different subnets can be on the same and different NVEs, and TS may be moved to different NVEes too. NVO3 architecture need to support it. I like to make it clear, This new type is like to IRB, but not exactly same. To take your suggestion, I better not refer it too much to cause confusion. 
> 
> Third of all, there is no need to repeat over and over again the same
> sentences. We all know what bridging, routing, and IRB is.
[Lucy] I have no intention to waste your and anyone time. However, as a draft author, you need to address individual feedbacks properly, please do not call it quit.  
> 
> End of this email thread on my side. Let's hear what others have to say
[Lucy] I like to hear other opinion on this too.
Lucy
> about it.
> > [mailto:[email-masked]] On Behalf
> > > > Of Lucy yong
> > > > [mailto:[email-masked]] On Behalf
> > > > > Of
> > > > > > Lucy yong
> am
> > > > > > applications and it is useful, but it is not equivalent to
> IRB
> > > > > function. To
> > > > > > > Cc: [email-masked]; draft-bl-
> nvo3-
> > > > > dataplane-
> new
> > > > > > > > >    This document describes some additional functions
> and
> > > > > > > > requirements

---

From: Thomas Narten <[email-masked]>
Date: Fri, 14 Dec 2012 12:58:15 -0500

Hi Lucy.

Lucy yong <[email-masked]> writes:

> IMO: this is the reason why we need to define a new service type for
> this.

I've tried following this thread and have also read the draft. I admit
to having difficulty understanding what the actual issue is or what
text is being asked for in the framework document.

It sounds to me like the crux of the issue is that for the case of
providing L3 (IP) service to Tenant Systems (TSs), there are two
sub-cases:

 - all the TSs are on the same subnet, so they all can reach each
   other without going through a router

 - TSs are on different IP subnets, with the collection of those
   subnets comprising a single NVO3 Virtual Network. 

Are you saying we need a different "service type" to distinguish those
two cases? If so, what do you mean by "service type"?

I think of NVO3 providing either an L2 or L3 service to TSes. I.e., 2
different types of service. Are you proposing that the L3 case be
split into two subcases?

If so, I agree that the two subcases are different, but I've always
thought of them as being the same service. I.e., that L3 service would
handle both cases. Why would we want to split them into different
"services"? And in any case, one could look at the first case (just
one subnet) as just being a special case of the second case, so why
handle it specially?

Do I understand correctly?

Thomas

---

From: Lucy yong <[email-masked]>
Date: Fri, 14 Dec 2012 18:15:25 +0000

Hi Tom,

Thank you for the comments. Please see inline.

> From: Thomas Narten [mailto:[email-masked]]
> Sent: Friday, December 14, 2012 11:58 AM
> To: Lucy yong
> Cc: [email-masked]
> Subject: Re: [nvo3] FW: New Version Notification for draft-yong-nvo3-
[Lucy] No, this is not what I mean. I mean we need a new service type to be able to support
a single NVo3 virtual network that contains both cases via a L2 interface.
> different types of service. Are you proposing that the L3 case be
[Lucy] L2 service assumes a L2 interface and L3 service assume a L3 interface. VM has L2 interface only.
This is about to address VM communication in a single nvo3 virtual network. assumption is to not change VM network function.  
> one subnet) as just being a special case of the second case, so why
[Lucy] But L3 service requires L3 interface between PE and CE. Do we want to change VM and vswitch to support that? 
> Do I understand correctly?
[Lucy] Hope it is not.

Regards,
Lucy

---

From: Thomas Narten <[email-masked]>
Date: Fri, 14 Dec 2012 13:50:59 -0500

Lucy,

> [Lucy] No, this is not what I mean. I mean we need a new service
> type to be able to support a single NVo3 virtual network that
> contains both cases via a L2 interface.

By "both cases", do you mean both L2 service and L3 service?

That is, do you mean a single Virtual Network, where TSs are
sending/receiving L2 frames, but in some cases the NVE treats them as
IP (forwarding them based on their TS IP header) while in other cases
they are treated as L2 (where the NVE forwards them based on the TS L2
header and doesn't look at the IP header (if there is one)?)

And can one TS send *both* types, or does a TS have to pick one format
(always) with different TSs on the same VN possibly choosing different
services?

If so, the WG has discussed this case before - and there are issues.
One issue is how does an NVE know whether to forward a packet received
from the TS using the packets L2 header vs. its IP header? For an IP
packet, it will have both headers, and depending on which choice is
made, you might get different forwarding behavior. That would probably
not be good.

> [Lucy] L2 service assumes a L2 interface and L3 service assume a L3
> interface. VM has L2 interface only.  This is about to address VM
> communication in a single nvo3 virtual network. assumption is to not
> change VM network function.

Regarding the first statement, I don't see it this way at all. I
assume that even for L3 service, where all tenant traffic is assumed
to be IP, the interface between the TS and the NVE will still be L2
Ethernet. However, the only allowed packets would be IP and ARP (and
maybe DHCP and small number of other IP-related protocols sent at
L2). Any others would simply be discarded.

To be honest, I'm not sure I understand what it means to have an "L3
interface" between the TS and NVE in the case where the TS is getting
L3 service only. Does that mean the TS is sending packets that DO NOT
have an L2 header in front of them? I.e., between the TS and NVE?
Because that would presumably require changes to VMs, it would seem to
be a non-starter. And what would be the point?

> [Lucy] But L3 service requires L3 interface between PE and CE. Do we
> want to change VM and vswitch to support that?

NVO3 doesn't have PE/CE terminology. :-)

Thomas

---

From: Lucy yong <[email-masked]>
Date: Fri, 14 Dec 2012 19:21:43 +0000

Tom,

> header and doesn't look at the IP header (if there is one)?)
[Lucy] Yes, this is what I mean. But I don't want to use two services to construct a tenant virtual network in some case. Can I just use one service to achieve it? 
> (always) with different TSs on the same VN possibly choosing different
[Lucy] Yes, one TS can send a frame to a TS that is on the same subnet, it can also send a frame to a TS that is on the different subnet.
> made, you might get different forwarding behavior. That would probably
[Lucy] You get it. But a frame from TS have both Ethernet and IP header. From TS perspective, the frame to the TS on the same subnet will be bridged, the frame to the
TS on the different subnet goes to the router. We need to a service to guarantee that.
> L2). Any others would simply be discarded.
[Lucy] you need to configure an IP interface first for the service.
> be a non-starter. And what would be the point?
[Lucy] This is not what I mean. I mean for L3 service, you need to configure IP interface. All the frames from/to VM are Ethernet frames. 
> NVO3 doesn't have PE/CE terminology. :-)
[Lucy] OK, you are right. Should be NVE/TS in nvo3.

Lucy

---

From: Thomas Narten <[email-masked]>
Date: Fri, 14 Dec 2012 14:57:52 -0500

Lucy,

> [Lucy] Yes, this is what I mean. But I don't want to use two
>  services to construct a tenant virtual network in some case. Can I
>  just use one service to achieve it?

I'm not at all sure it makes sense to mix and match L2 and IP service
on the same Virtual Network. Seems to me, that if you have some
traffic that needs to be forwarded at L2, just forward it all using
the VM's L2 addresses. This works. Doesn't matter if payload is IP or
not. Keep it simple.

If you forward *some* traffic using L2 information, but other TSs
expect only L3, what is supposed to happen? This isn't going to work
right. Why would you even allow such a model?

But before looking at the problems with such an approach, what I'd
really like to understand is what the use case is for such a mixed
model on a single VN. Not just "why not", but what is the real use
case. If there isn't one, I'd prefer that it just not be supported as
doing so will surely add complexity ... and if there is no real need
or benefit, then why?

> [Lucy] Yes, one TS can send a frame to a TS that is on the same
>  subnet, it can also send a frame to a TS that is on the different
>  subnet.

You didn't answer the question I'm trying to ask. The question I'm
asking is how does the NVE know (when it gets a packet) whether to
forward it using the L2 information or the L3. It has to pick one (for
each arriving packet). How does it know? Is the NVE supposed to know,
for example, what subnets look like from a VM's perspective (I would
argue that is not a good designe and has a bunch of issues).

And per above, if you just always forward on L2 info, you can make the
intra- and inter-subnet case work. So why not just do that?

> [Lucy] You get it. But a frame from TS have both Ethernet and IP
> header. From TS perspective, the frame to the TS on the same subnet
> will be bridged, the frame to the TS on the different subnet goes to
> the router. We need to a service to guarantee that.

I am assuming that NVO3 networks providing IP service to tenants will
have to embed at least limited routing capability. Enough to handle
the "inter-subnet" case. But I can also imagine this being done on the
ingress NVE, in which case, there is no penalty or "extra hop". I.e.,
the ingress NVE recognizes that a packet is being sent to a router,
and just forwards the packet directly to where its supposed to go. 

That is, although architecturally a Virtual Network MAY need to have
an internal router to handle inter-subnet forwarding with an VN, this
can be implemented at the ingress NVE and doesn't need to add much, if
any overhead. 

Is there a reason why that wouldn't be good enough?

> [Lucy] you need to configure an IP interface first for the service.

Who is configuring what interface on what device?

If you are referring to the VM and one of its interfaces, what VMs do
on their interfaces is outside of the scope of NVO3 -- I don't think
it matters to us. Right?

> [Lucy] This is not what I mean. I mean for L3 service, you need to
> configure IP interface. All the frames from/to VM are Ethernet
> frames.

Again, which IP interface? On what device?

Thomas

---

From: Lucy yong <[email-masked]>
Date: Fri, 14 Dec 2012 20:50:24 +0000

Tom,

The use case is simple. One tenant virtual network may contain more than one subnets. Each subnet presents a broadcast domain. The TS in the network can send traffic to another TSes that may be on the same or different network. A BD provides simple communication mechanism. Please look at Microsoft Hyper-v implementation. It is already implemented. NVo3 use case draft also describe it.

Regarding your question on how NVE to know if it should use L2 or L3 info. on the frame, this is back to the basic communication rule TS uses. TS send a frame to the destination directly if it is on the same subnet and send a frame to the gateway if the destination of the frame is not on the same subnet. In this new service, NVE plays the gateway rule. TS uses the arp to know the gateway address. When a frame is designated to the gateway, thus NVE knows that it should perform IP lookup on it, otherwise, it forwards based on MAC address. This is like IRB feature.  When NVE perform IP lookup and find the corresponding destination TS MAC as well as remote NVE IP address, NVE has to replace gateway MAC address with TS MAC address and put tunnel addresses before sending out.

You say to forward both intra and inter subnet traffic by using L2. Yes, it is simple. But TS does not behave this way. It forwards inter subnet traffic to the gateway first. This is my understanding. If we use the L2 service, where is the gateway?

The new service type is to allow ingress NVE performs some routing. It is the L2/3 gateway for the TSes. But from TS perspective, it attaches a LAN or VLAN, not router.

Regards,
Lucy

> Sent: Friday, December 14, 2012 1:58 PM
> To: Lucy yong
> Cc: [email-masked]

---

From: Aldrin Isaac <[email-masked]>
Date: Fri, 14 Dec 2012 22:16:45 -0500

Lucy, pardon me for using IPVPN and EVPN language, but can the bridging
with edge routing functionality that you are describing be implemented on
an access PE as a localized L3 VRF with interfaces to one or more local L2
EVI?  This would require that all L2VN to which edge routing
is desired would need corresponding local EVI (and all associated MACs) on
the PE even if these L2VN have no local VM.  Is this the functionality you
want to see represented?  Best -- aldrin

On Friday, December 14, 2012, Lucy yong wrote:

> Tom,
>
> The use case is simple. One tenant virtual network may contain more than
> one subnets. Each subnet presents a broadcast domain. The TS in the network
> can send traffic to another TSes that may be on the same or different
> network. A BD provides simple communication mechanism. Please look at
> Microsoft Hyper-v implementation. It is already implemented. NVo3 use case
> draft also describe it.
>
> Regarding your question on how NVE to know if it should use L2 or L3 info.
> on the frame, this is back to the basic communication rule TS uses. TS send
> a frame to the destination directly if it is on the same subnet and send a
> frame to the gateway if the destination of the frame is not on the same
> subnet. In this new service, NVE plays the gateway rule. TS uses the arp to
> know the gateway address. When a frame is designated to the gateway, thus
> NVE knows that it should perform IP lookup on it, otherwise, it forwards
> based on MAC address. This is like IRB feature.  When NVE perform IP lookup
> and find the corresponding destination TS MAC as well as remote NVE IP
> address, NVE has to replace gateway MAC address with TS MAC address and put
> tunnel addresses before sending out.
>
> You say to forward both intra and inter subnet traffic by using L2. Yes,
> it is simple. But TS does not behave this way. It forwards inter subnet
> traffic to the gateway first. This is my understanding. If we use the L2
> service, where is the gateway?
>
> The new service type is to allow ingress NVE performs some routing. It is
> the L2/3 gateway for the TSes. But from TS perspective, it attaches a LAN
> or VLAN, not router.
>
> Regards,
> Lucy
>
> > From: Thomas Narten [mailto:[email-masked] <javascript:;>]
> > Cc: [email-masked] <javascript:;>
> > > > > [Lucy] L2 service assumes a L2 interface and L3 service assume

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 17 Dec 2012 10:07:18 -0500

Here's some perspective from Yakov on another thread....

"In the context of DC each NVE could provide router functionality
(e.g. EVI+IRB). So, each NVE, in addition to acting as an EVI, can act
as a router for VMs behind this NVE."

Best regards -- aldrin

On Fri, Dec 14, 2012 at 10:16 PM, Aldrin Isaac <[email-masked]> wrote:
> Lucy, pardon me for using IPVPN and EVPN language, but can the bridging with
> edge routing functionality that you are describing be implemented on an
> access PE as a localized L3 VRF with interfaces to one or more local L2 EVI?
> This would require that all L2VN to which edge routing is desired would need
> corresponding local EVI (and all associated MACs) on the PE even if these
> L2VN have no local VM.  Is this the functionality you want to see
> represented?  Best -- aldrin
>
>
>> the L2/3 gateway for the TSes. But from TS perspective, it attaches a LAN or
>> VLAN, not router.
>>
>> Regards,
>> Lucy
>>

---

From: Lucy yong <[email-masked]>
Date: Mon, 17 Dec 2012 16:13:18 +0000

Hi Aldrin,

These (you and Yakov description) are exactly what I picture and think it is better to represent this as a new NVE service type because this NVE not only perform EVIes to attached TSes but also a router. 

L2 NVE service type is about EVI, L3 NVE is about VRF. Would be good for us to have a new type to differ from the two? From operator perspective, do you agree?

Thanks,
Lucy

> From: Aldrin Isaac [mailto:[email-masked]]
> Sent: Monday, December 17, 2012 9:07 AM
> To: Lucy yong
> Cc: [email-masked]
> Subject: Re: FW: New Version Notification for draft-yong-nvo3-frwk-
> dpreq-addition-00.txt
> 
> >> > > > sending/receiving L2 frames, but in some cases the NVE treats
> them
> >> > as
> >> > > > IP (forwarding them based on their TS IP header) while in
> other
> >> > cases
> >> > > > they are treated as L2 (where the NVE forwards them based on
> the TS
> >> > L2
> >> > > > And can one TS send *both* types, or does a TS have to pick
> one
> >> > format
> >> > > > from the TS using the packets L2 header vs. its IP header? For
> an
> >> > IP

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 17 Dec 2012 17:40:48 +0000

> These (you and Yakov description) are exactly what I picture and think
> it is better to represent this as a new NVE service type because this
> NVE not only perform EVIes to attached TSes but also a router.
> 
> L2 NVE service type is about EVI, L3 NVE is about VRF. Would be good
> for us to have a new type to differ from the two? From operator
> perspective, do you agree?

I assume that what is being discussed is a layer 2 overlay and a layer 3 overlay in a single solution, which is, I agree, different from a purely layer 2 overlay or purely layer 3 overlay.
IMO, a solution that concurrently addresses both, layer 2 and layer 3 overlays, will introduce a lot of unnecessary complexity. This is because a layer 2 overlay does not just consists of carrying Ethernet headers end-to-end (about which end-hosts/IP applications don't care) but, by definition, includes the interoperability with IEEE bridged networks.

Maria

> > On Fri, Dec 14, 2012 at 10:16 PM, Aldrin Isaac
> <[email-masked]>
> > wrote:
> > > edge routing functionality that you are describing be implemented
> on
> > an
> > > access PE as a localized L3 VRF with interfaces to one or more
> local
> > L2 EVI?
> > >> The use case is simple. One tenant virtual network may contain
> more
> > than
> > >> one subnets. Each subnet presents a broadcast domain. The TS in
> the
> > network
> > >> Regarding your question on how NVE to know if it should use L2 or
> L3
> > info.
> > >> on the frame, this is back to the basic communication rule TS
> uses.
> > TS send
> > >> a frame to the destination directly if it is on the same subnet
> and
> > send a
> > >> subnet. In this new service, NVE plays the gateway rule. TS uses
> the
> > arp to
> > >> know the gateway address. When a frame is designated to the
> gateway,
> > thus
> > >> based on MAC address. This is like IRB feature.  When NVE perform
> IP
> > lookup
> > >> and find the corresponding destination TS MAC as well as remote
> NVE
> > IP
> > >> address, NVE has to replace gateway MAC address with TS MAC
> address
> > and put
> > >> You say to forward both intra and inter subnet traffic by using
> L2.
> > Yes,
> > >> The new service type is to allow ingress NVE performs some
> routing.
> > It is
> > >> the L2/3 gateway for the TSes. But from TS perspective, it
> attaches
> > a LAN or
> > >> VLAN, not router.
> > >>
> > >> Regards,
> > >> Lucy
> > >>
> > >> > the VM's L2 addresses. This works. Doesn't matter if payload is
> IP
> > or
> > >> > not. Keep it simple.
> > >> >
> > >> > case. If there isn't one, I'd prefer that it just not be
> supported
> > as
> > >> > forward it using the L2 information or the L3. It has to pick
> one
> > (for
> > >> > And per above, if you just always forward on L2 info, you can
> make
> > the
> > >> > > > One issue is how does an NVE know whether to forward a
> packet
> > >> > received
> > >> > > > made, you might get different forwarding behavior. That
> would
> > >> > probably
> > >> > > > not be good.
> > >> >
> > >> > the "inter-subnet" case. But I can also imagine this being done
> on
> > the
> > >> > can be implemented at the ingress NVE and doesn't need to add
> much,
> > if
> > >> > any overhead.
> > >> >

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 17 Dec 2012 12:42:56 -0500

> L2 NVE service type is about EVI, L3 NVE is about VRF. Would be good for us to have a new type to differ from the two? From operator perspective, do you agree?

IMO, this edge routing functionality can be formed by interconnecting
an EVI (L2 forwarding instance) with a VRF (L3 forwarding instance) at
the NVE.  As others point out, this is just an implementation of IRB.

However I'm not sure the current framework draft can represent such an
IRB service since I don't see any mention of interfaces between
forwarding instances (of same or different types) within the same NVE,
but maybe I've missed something somewhere...

Best -- aldrin

---

From: Lucy yong <[email-masked]>
Date: Mon, 17 Dec 2012 18:12:38 +0000

Maria,

It seems that you agree this is necessary for a tenant virtual network, i.e. a layer 2 overlay and layer 3 overlay in a single solution. However, I don't get if you want to have a new service type for the solution or not.

Without it, it means that, for creating a tenant virtual network that contains multiple subnets, operators have to create individual layer 2 overlays and layer 3 overlay and specify the interfaces to interconnect them, etc. 

I like to know your preference as an operator.

Thanks,
Lucy

> From: NAPIERALA, MARIA H [mailto:[email-masked]]
> Sent: Monday, December 17, 2012 11:41 AM
> To: Lucy yong; Aldrin Isaac
> Cc: [email-masked]
> I assume that what is being discussed is a layer 2 overlay and a layer
> 3 overlay in a single solution, which is, I agree, different from a
> purely layer 2 overlay or purely layer 3 overlay.
> IMO, a solution that concurrently addresses both, layer 2 and layer 3
> overlays, will introduce a lot of unnecessary complexity. This is
> because a layer 2 overlay does not just consists of carrying Ethernet
> headers end-to-end (about which end-hosts/IP applications don't care)
> but, by definition, includes the interoperability with IEEE bridged
> networks.
> 
> Maria
> 
> >
> > Thanks,
> > Lucy
> >
> > > > corresponding local EVI (and all associated MACs) on the PE even
> if
> > > these
> > > >> network. A BD provides simple communication mechanism. Please
> look
> > > at
> > > >> Microsoft Hyper-v implementation. It is already implemented.
> NVo3
> > > use case
> > > >> draft also describe it.
> > > >>
> > > >> frame to the gateway if the destination of the frame is not on
> the
> > > same
> > > >> traffic to the gateway first. This is my understanding. If we
> use
> > > the L2
> > > >> > Subject: Re: [nvo3] FW: New Version Notification for draft-
> yong-
> > > nvo3-
> > > >> > expect only L3, what is supposed to happen? This isn't going
> to
> > > work
> > > >> > doing so will surely add complexity ... and if there is no
> real
> > > need
> > > >> > or benefit, then why?
> > > >> >
> > > >> > > > (always) with different TSs on the same VN possibly
> choosing
> > > >> > different
> > > >> > > > services?
> > > >> >
> > > >> > asking is how does the NVE know (when it gets a packet)
> whether
> > to
> > > >> > each arriving packet). How does it know? Is the NVE supposed
> to
> > > know,
> > > >> > > > If so, the WG has discussed this case before - and there
> are
> > > issues.
> > > >> > > [Lucy] You get it. But a frame from TS have both Ethernet
> and
> > IP
> > > >> > I am assuming that NVO3 networks providing IP service to
> tenants
> > > will
> > > >> > That is, although architecturally a Virtual Network MAY need
> to
> > > have
> > > >> > an internal router to handle inter-subnet forwarding with an
> VN,
> > > this

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Mon, 17 Dec 2012 18:18:37 +0000

Lucy,

> It seems that you agree this is necessary for a tenant virtual network,
> i.e. a layer 2 overlay and layer 3 overlay in a single solution.
> However, I don't get if you want to have a new service type for the
> solution or not.

No. I said that that this is unnecessary complication.

> Without it, it means that, for creating a tenant virtual network that
> contains multiple subnets, operators have to create individual layer 2
> overlays and layer 3 overlay and specify the interfaces to interconnect
> them, etc.

Intra-subnet traffic can be also handled by a layer 3 overlay.

Maria
 
> > > > (e.g. EVI+IRB). So, each NVE, in addition to acting as an EVI,
> can
> > > act
> > > > > This would require that all L2VN to which edge routing is
> desired
> > > > would need
> > > > >> NVE knows that it should perform IP lookup on it, otherwise,
> it
> > > > forwards
> > > > >> it is simple. But TS does not behave this way. It forwards
> inter
> > > > subnet
> > > > >> > > > > [Lucy] No, this is not what I mean. I mean we need a
> new
> > > > service
> > > > >> > > > > type to be able to support a single NVo3 virtual
> network
> > > > that
> > > > >> > > > they are treated as L2 (where the NVE forwards them
> based
> > on
> > > > the TS
> > > > >> > L2
> > > > >> > >  services to construct a tenant virtual network in some
> case.
> > > > Can I
> > > > >> > I'm not at all sure it makes sense to mix and match L2 and
> IP
> > > > service
> > > > >> > traffic that needs to be forwarded at L2, just forward it
> all
> > > > using
> > > > >> > If you forward *some* traffic using L2 information, but
> other
> > > TSs
> > > > >> > But before looking at the problems with such an approach,
> what
> > > I'd
> > > > >> > model on a single VN. Not just "why not", but what is the
> real
> > > use
> > > > >> > You didn't answer the question I'm trying to ask. The
> question
> > > I'm
> > > > >> > for example, what subnets look like from a VM's perspective
> (I
> > > > would
> > > > >> > > > from the TS using the packets L2 header vs. its IP
> header?
> > > For
> > > > an
> > > > >> > IP
> > > > >> > > > packet, it will have both headers, and depending on
> which
> > > > choice is
> > > > >> > > header. From TS perspective, the frame to the TS on the
> same
> > > > subnet
> > > > >> > > will be bridged, the frame to the TS on the different
> subnet
> > > > goes to
> > > > >> > ingress NVE, in which case, there is no penalty or "extra
> hop".
> > > > I.e.,
> > > > >> > and just forwards the packet directly to where its supposed
> to
> > > go.
> > > > >> >
> > > > >> > > > > [Lucy] L2 service assumes a L2 interface and L3
> service
> > > > assume

---

From: Kireeti Kompella <[email-masked]>
Date: Mon, 17 Dec 2012 10:57:10 -0800

> Intra-subnet traffic can be also handled by a layer 3 overlay.

Let me expand.

I see the need for E-VPN for non-IP traffic.  This is real, and is not met by IP VPNs (news flash!)

For IP traffic, whether intra and inter-subnet, IP VPNs suffice.

The solution is simple: route if IP, bridge if not.  Yes, one could do IRB, but why?  IRB brings in complications, especially for multicast.  I'm sure someone suggested this already, so put me down as supporting this view.

A NVE that supports both E-VPN and IP VPN for a given tenant simply sends IP traffic to the IP VPN and sends the rest to E-VPN.  How this happens is implementation specific.  Note that this assumes that the NVE intercepts ARPs and responds to them with the same MAC.  Does anyone see a problem with this?

If there is a case for _only_ intra-subnet traffic, one may create an E-VPN to handle both IP and non-IP; but I suspect this is a rare case.  

From that point of view, I would like to see E-VPNs in the data center *always* coupled with IP VPNs, and only dealing with non-IP traffic.

This may appear drastic, but I think operationally, this is will simplify things.  As always, I am open to alternate suggestions, provided they are presented without religion or politics.  I'm especially keen to hear from those deploying.

Kireeti.

---

From: Lucy yong <[email-masked]>
Date: Mon, 17 Dec 2012 19:48:14 +0000

Kireeti,

Comments inline.

> met by IP VPNs (news flash!)
[Lucy] Without IP, there is not subnet. So E-VPN works well.
> The solution is simple: route if IP, bridge if not.  Yes, one could do
> IRB, but why?  IRB brings in complications, especially for multicast.
> I'm sure someone suggested this already, so put me down as supporting
> this view.
> 
> A NVE that supports both E-VPN and IP VPN for a given tenant simply
> sends IP traffic to the IP VPN and sends the rest to E-VPN.  How this
> happens is implementation specific.  Note that this assumes that the
> NVE intercepts ARPs and responds to them with the same MAC.  Does
> anyone see a problem with this?
[Lucy] This may have an issue if NVE is remote. ARP message may reach both another TS and NVE. Do you want to mandate that each TS can not response any message or mandate that there is never a switch between TS and NVE.

Regards,
Lucy

---

From: Aldrin Isaac <[email-masked]>
Date: Mon, 17 Dec 2012 23:37:17 -0500

Kireeti,

How is the bridging and routing functionality you describe implemented on
say a hypervisor?  Would there be a routing instance (VRF) connected to one
or more bridging instances (EVI)?  I call this IRB, although I'm sure there
is a lot more involved with implementing it on dedicated hardware.

Also, given that (1) you want host specific routing because it gives
you the shortest path to anywhere, and (2) since IP is the Internet[work]
protocol (my networks, my partners networks, my services and the world in
general), it would seem to me that the NVE has a lot of /32 routing
information to bear.  What's the plan of aggregation, and then what of
shortest path?

Also, as I noted some time ago, currently IP routing is generally stable at
the expense of Ethernet since today router configuration is generally
static, whereas bridges have to deal with plug and play and lousy end
station software.  When the IP tables start churning, will IP still be the
good ole IP of yester year?

Best regards -- aldrin

On Monday, December 17, 2012, Kireeti Kompella wrote:

> On Dec 17, 2012, at 10:18 , "NAPIERALA, MARIA H" <[email-masked]<javascript:;>>
> wrote:
>
> IRB, but why?  IRB brings in complications, especially for multicast.  I'm
> sure someone suggested this already, so put me down as supporting this view.
>
> A NVE that supports both E-VPN and IP VPN for a given tenant simply sends
> IP traffic to the IP VPN and sends the rest to E-VPN.  How this happens is
> implementation specific.  Note that this assumes that the NVE intercepts
> ARPs and responds to them with the same MAC.  Does anyone see a problem
> with this?
>
> If there is a case for _only_ intra-subnet traffic, one may create an
> E-VPN to handle both IP and non-IP; but I suspect this is a rare case.
>
> From that point of view, I would like to see E-VPNs in the data center
> *always* coupled with IP VPNs, and only dealing with non-IP traffic.
>
> This may appear drastic, but I think operationally, this is will simplify
> things.  As always, I am open to alternate suggestions, provided they are
> presented without religion or politics.  I'm especially keen to hear from
> those deploying.
>
> Kireeti.
>
>

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 18 Dec 2012 06:04:00 +0000

>For IP traffic, whether intra and inter-subnet, IP VPNs suffice.

Yep.
>IRB, but why?  IRB brings in complications, especially for multicast.

Exactly, thank you for speaking out.

>I'm sure someone suggested this already, so put me down as supporting

I believe Pedro gave detailed explanations many times a few months ago. Of
course, Maria and I had been expressing such opinions all along on-line
and off-line, and brushed off good amount of bullets. :-).

>E-VPN to handle both IP and non-IP; but I suspect this is a rare case.

A solution just for some very narrow cases is not that too useful.

>*always* coupled with IP VPNs, and only dealing with non-IP traffic.

Do you mean the following? IP VPN alone is sufficient if all traffic is L3
or handling the majority traffic which is IP; but E-VPN is not in the same
case, E-VPN needs to leverage IP VPN for inter-subnet routing (unless you
never get out of a subnet), therefore it needs to "*always* coupled with
IP VPNs"?
If this is correct understanding, and we are talking about VPN cases only
(there are also other non-VPN options too), I agree with you.

The "*always* coupled" wording makes me a little nervous, better to ask
you to elaborate.

What I don't want to see is that the implementation ties IP VPN and E-VPN
together. If the traffic is IP (might well be 90+% cases), there should
not be any E-VPN involvement. If both L2 and L3 are needed, there should
be clear demarcation, not mixed together, not sharing the same VRF, not
make one VPN with some l2 sites and some l3 sites. Let's keep it clean and
simple. 

>things. 

Yes, operation simplicity is the key to success. IP VPN is a living proof.
Others are not at par.

I think Thomas Narten and Maria both expressed the point to avoid
complication, agree with them too.

I have a larger question, not to Kireeti or any particular person.
How come we only talk about MPLS VPNs here now?
What happened to other technologies? Were not VXLAN and NVGRE the original
motivation for nvo3?
I am a MPLS VPN person, but I also don't believe in one size fits all. IP
VPN or E-VPN are right for some operators, cannot say for everyone. Where
are the folks talking non MPLS topics?

Luyuan

---

From: Lucy yong <[email-masked]>
Date: Tue, 18 Dec 2012 15:50:46 +0000

Aldrin,

The solution Maria and Kireeti lean to is the L3 overlay only solution.  Pedro's end-system draft propose one solution for the L3 overlay only in DC. However the draft requires a substantial changes on the host OS system, and the architecture is very different from the physical enterprise network. In this solution, there is no L2 overlay at all. Each TS has to establish a tunnel with the overlay first hop router directly. That can be via a p2p or mp2p model.

I see the application space for this such as IP mobility but don't see it can fit for a tenant virtual network.  Because this architecture requires that individual TSes MUST use a security mechanism to join the VPN first.

For DC multi-tenancy, DC operator can create a tenant virtual network and assign VM and network appliance resource to associate with the tenant virtual network. Then tenant places and run the application on the resource. DC operator can move these VMs for server resource optimization while tenant application is running. This application is very different from IP mobility application, where TS can move on its own and need to request "join" the network again when it changes the network access point.

Thus, I do not see that this stone can hit two birds.

Kireeti, sorry to cut-in ahead.

Regards,
Lucy

From: Aldrin Isaac [mailto:[email-masked]]
Sent: Monday, December 17, 2012 10:37 PM
To: Kireeti Kompella
Cc: NAPIERALA, MARIA H; Lucy yong; [email-masked]
Subject: Re: FW: New Version Notification for draft-yong-nvo3-frwk-dpreq-addition-00.txt

Kireeti,

How is the bridging and routing functionality you describe implemented on say a hypervisor?  Would there be a routing instance (VRF) connected to one or more bridging instances (EVI)?  I call this IRB, although I'm sure there is a lot more involved with implementing it on dedicated hardware.

Also, given that (1) you want host specific routing because it gives you the shortest path to anywhere, and (2) since IP is the Internet[work] protocol (my networks, my partners networks, my services and the world in general), it would seem to me that the NVE has a lot of /32 routing information to bear.  What's the plan of aggregation, and then what of shortest path?

Also, as I noted some time ago, currently IP routing is generally stable at the expense of Ethernet since today router configuration is generally static, whereas bridges have to deal with plug and play and lousy end station software.  When the IP tables start churning, will IP still be the good ole IP of yester year?

Best regards -- aldrin

Let me expand.

Kireeti.

---

From: Thomas Narten <[email-masked]>
Date: Tue, 18 Dec 2012 12:03:33 -0500

Kireeti Kompella <[email-masked]> writes:

> The solution is simple: route if IP, bridge if not.  Yes, one could
>  do IRB, but why?  IRB brings in complications, especially for
>  multicast.  I'm sure someone suggested this already, so put me down
>  as supporting this view.

I'm not sure I understand the difference.

>From an *NVE* perspective, when it receives a packet (which will have
an L2 header), it can look at the Ethertype, and if its IP, it can
route it. Otherwise, it can provide normal L2 service. So, in this
sense, "route if IP, bridge if not" is straightforward. And more to
the point, I assume that if the packet gets L2 service, the entire VN
is treated as a *single* broadcast domain. All nodes can reach all
other nodes. Right?

Just so I understand, how is this different than IRB?  What does IRB
imply that the above does not?

But this is different than what (I believe) Lucy is arguing for. In
the case of a multi-subnet VN, you have one VN, but it contains
different subnets. Each subnet is intended to be one broadcast domain
(i.e., equivalent of a VLAN), so that when sending LL multicast and
the like on a specific subnet, such packets are *not* delivered to all
nodes in the VN, but only those that are part of subnet.

This is a more complex type of service to provide. And I'm not sure we
need this type of service to be provided by one VN. A (seemingly
simpler) alternative would be to put each subnet in its own VN and
allow inter-subnet traffic to be handed as inter-VN traffic. So long
as that case is optimized (i.e., the ingress NVE can tunnel directly
to the egress NVE without adding triangular routing), this would seem
to be a cleaner way to implement this.

Thomas

---

From: Lucy yong <[email-masked]>
Date: Tue, 18 Dec 2012 18:10:06 +0000

My comments are inline.

> From: Luyuan Fang (lufang) [mailto:[email-masked]]
> Sent: Tuesday, December 18, 2012 12:04 AM
> To: Kireeti Kompella; NAPIERALA, MARIA H
> Cc: Aldrin Isaac; Lucy yong; [email-masked]
[Lucy] not sure. We are working on the network virtualization. The one of goals is to virtualizes the enterprise physical networks that provides the networking for compute resource on a DC common infrastructure. If you look at today's enterprise data center networks, do they all use routers without any switch? Not at all. If routers can do all the functions, why it is not the case today in any physical network? You may argue that they are legacy. But I do not see a new built DC without a switch too. Why is that? How can we assume that IP VPN is sufficient for a tenant virtual network.
 
> Exactly, thank you for speaking out.
[Lucy] Again, IRB are widely used in enterprise data center. It is useful. How do we know that IP VPN is not complicated for a tenant virtual network? 
> and off-line, and brushed off good amount of bullets. :-).
[Lucy] IMO: Pedro's solution applies to certain application, but not other application. For example, his draft does not mention how to provide a broadcast for a subnet. Unless a tenant virtual network does not need this at all, we can not remove this requirement.
> A solution just for some very narrow cases is not that too useful.
[Lucy] Not at all. Another objective of NV03 is the ability to move TS among different servers regardless if they are on the same or different subnet. Do you see this is "very narrow cases"?

Sorry, we are back to use VPN terms again. I did not start this.

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Tue, 18 Dec 2012 19:34:25 +0000

> to be a cleaner way to implement this.

[Lucy] I see this as shifting the complexity to operator, which is worse than what I propose. I agree that this is more complex service than pure L2 overlay or pure L3 overlay. If we define it as a new service, we can have a solution for the operator. This will avoid an operator to construct individual L2 overlays for the subnets and an L3 overlay to interconnect them for a tenant virtual network.

Lucy

---

From: Thomas Narten <[email-masked]>
Date: Tue, 18 Dec 2012 14:53:05 -0500

> [Lucy] I see this as shifting the complexity to operator, which is
> worse than what I propose.

Not sure why that would be...

Operators will surely have the capability to provision multi-VN
services and the allowed interconnections of VNs. This will be needed
in any case. So one could also argue it would be simpler to leverage
the same functionality to handle the multi-subnet case.

And if you want VNs to know about subnets (so that each subnet is its
own broadcast domain), now the VN needs to know what subnets actually
are and who (which TS's) belongs to what subnet. That would seem to
add complexity to the L3 VN case...

> I agree that this is more complex service than pure L2 overlay or
> pure L3 overlay. If we define it as a new service, we can have a
> solution for the operator. This will avoid an operator to construct
> individual L2 overlays for the subnets and an L3 overlay to
> interconnect them for a tenant virtual network.

Whether this is a "new service" or just a variation of an existing
services seems less important than understanding what service needs to
be provided to a tenant and how best to do that.

Thomas

---

From: Lucy yong <[email-masked]>
Date: Tue, 18 Dec 2012 22:38:05 +0000

> be provided to a tenant and how best to do that.

[Lucy] Yes, could you give some aspects on what are a necessary for a tenant network? Any feedback from DC operators are welcome too.

Lucy

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 18 Dec 2012 15:06:38 -0800

Hi Thomas,

> is treated as a *single* broadcast domain. All nodes can reach all

Right.

> imply that the above does not?

IRB follows the principle of "bridge when you can, route otherwise".  So, an IP packet with dest IP in the same subnet actually gets bridged; the originator (e.g., the VM) is responsible for ARPing the IP address, slapping the right dest MAC on the packet and sending that to the NVE which simply forwards based on dest MAC address *without* decrementing the TTL.

If the dest IP is in another subnet, the packet is sent to the gateway (which for IRB would be the same NVE), which this time does an IP address lookup, decrements TTL and routes the packet.

For multicast, there are even more differences.

> nodes in the VN, but only those that are part of subnet.

If one were to configure multiple subnets on a VLAN, I wonder if LL traffic goes to all members of the VLAN, or just those in the same subnet as the sender.  I suspect the former (but don't know).

> need this type of service to be provided by one VN.

Agree.

> A (seemingly

Can be done.  However, we're on Lucy's topic; mine was "route if IP, bridge otherwise"; the goal was to rationalize the need for Layer 2 forwarding for non-IP traffic, and inter- and intra-subnet routing.

Kireeti.

---

From: Jeff Wheeler <[email-masked]>
Date: Tue, 18 Dec 2012 18:57:00 -0500

> If one were to configure multiple subnets on a VLAN, I wonder if LL traffic goes to all members of the VLAN, or just those in the same subnet as the sender.  I suspect the former (but don't know).

It won't go to "all members of the VLAN" except for traffic addressed
to the all-hosts address.  Link-local traffic has similar multicast
and unicasting capabilities to any other IPv6 address.

All multicast traffic can leak to uninterested receivers, including
link-local multicast.  This includes, for example, the solicited-node
addresses used by Neighbor Discovery.

Unicast link-local traffic won't be "leaked" because there will be an
L3 destination to L2 destination mapping.

This is not different than IPv4.

-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: Lucy yong <[email-masked]>
Date: Tue, 18 Dec 2012 23:57:02 +0000

Kireeti,

You sound like in favor to rationalize Layer 2 for the inter - and intra subnet routing. Is that right?

Regards,
Lucy
> Can be done.  However, we're on Lucy's topic; mine was "route if IP,
> bridge otherwise"; the goal was to rationalize the need for Layer 2
> forwarding for non-IP traffic, and inter- and intra-subnet routing.
> 
> Kireeti.
> 
> > Thomas
> >

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 19 Dec 2012 02:40:10 +0000

> Can be done.  However, we're on Lucy's topic; mine was "route if IP, bridge
> otherwise"; the goal was to rationalize the need for Layer 2 forwarding for
> non-IP traffic, and inter- and intra-subnet routing.

Agree. For non-IP traffic , Layer2 forwarding is the only choice. For inter-subnet traffic, Layer3 forwarding is the only choice. For intra-subnet IP traffic, Layer2 forwarding is the current mainstream choice, but Layer3 forwarding may be a better choice in some cases.

Best regards,
Xiaohu

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 18 Dec 2012 23:29:44 -0500

Kireeti,

I'm not clear what difference it makes whether a packet is unicast
forwarded using MAC address or IP address within a subnet as long as
it gets to the intended destination along the most optimal path,
particularly when the price to pay is non-standard behavior
(intra-subnet ARP manglers ;}, etc).  I understand the argument about
the sub-optimal routing from a third site, but when the primary sites
end up aggregating prefixes for scaling reasons that argument falls
off the table.  One way or other the piper gets paid.

In terms of the real world issue of getting there from here --
personally I haven't seen any vendor working towards a standards-based
solution that will allow intra-subnet routing for subnets over
HW/TOR-based PE, let alone intra-subnet routing for subnets that span
across both hypervisor-based PE and TOR-based PE.  This makes me leery
of solutions that can only take us half way there, particularly during
the transition phase.  So if we're talking about network
virtualization based purely on hypervisors, "route IP, bridge non-IP"
may be realistic if you're willing to accept the caveats, but does not
seem to be otherwise.

Btw, I understand how multicast may be less than efficient when
building both inter and intra subnet trees for the same IP mcast group
that end up overlapping links (maybe even more than twice) -- but I'd
like to hear your take on any other *insolvable* issues with regard to
multicast.

Best regards -- aldrin

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 19 Dec 2012 00:42:33 -0500

Btw, in regards to multicast efficiency on large bridged networks with
multiple subnets -- we use a more advanced version of MVR that allows
us to designated a single VLAN to be used for a multicast group -- at
the access switch, an extension to IGMP snooping maps receivers for a
group to that group's designated VLAN.  This prevents the creation of
multiple distribution trees in large multi-subnet bridge domains as
well as nicely solve the issue of multicast extranet with firewall
bypass.  Best regards -- aldrin

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 19 Dec 2012 05:45:01 +0000

Xiaohu,

Good sum up.

> For intra-subnet IP traffic, Layer2 forwarding is the current mainstream
>choice, but Layer3 forwarding may be a better choice in some cases.

I'd prefer to put this way - for IP traffic, l3 forwarding is a better
choice.
But you and/or others may have certain observation/reasoning on why it is
only in some cases, I'm willing to hear it out.

Thanks,
Luyuan

On 12/18/12 9:40 PM, "Xuxiaohu" <[email-masked]> wrote:

>>bridge
>Agree. For non-IP traffic , Layer2 forwarding is the only choice. For
>inter-subnet traffic, Layer3 forwarding is the only choice. For
>intra-subnet IP traffic, Layer2 forwarding is the current mainstream
>choice, but Layer3 forwarding may be a better choice in some cases.
>
>Best regards,
>Xiaohu
>
>> Kireeti.
>> 
>> > Thomas
>> >

---

From: Kireeti Kompella <[email-masked]>
Date: Tue, 18 Dec 2012 23:59:47 -0800

Hi Aldrin,

On Tue, Dec 18, 2012 at 8:29 PM, Aldrin Isaac <[email-masked]>wrote:

> forwarded using MAC address or IP address within a subnet

Two important differences:
a) you don't have to know the MAC address if you forward on IP.  I.e., you
don't have to propagate the ARP to the destination (flood), get the reply,
bind IP to MAC (ARP table), and maintain ARP binding (timeout, validate,
etc.).  The first is a real problem; the rest are annoyances that become
problems at scale.

(Note that the ARMD WG was created to address this issue, and you know
where that ended.)

(Note further that this may be hard to do in general, but in the case of an
orchestrated data center, you have the information about where a given IP
lives, and you have a control plane (ORACLE) to inform all relevant NVEs.
 And of course, an overlay to shield the infrastructure from poking its
nose into your forwarding behavior -- i.e., the infra doesn't care whether
you route or switch TS traffic.)

b) In the quite common case where all traffic from a TS is IP, you don't
have to maintain two tables and two forwarding paradigms at the NVE (one
for IPs and one for MACs).  This is common enough to warrant optimization.

A third difference is that if you have only unicast traffic, you don't have
to maintain a multicast tree (for flooding).  For some, this is a nice
bonus, but I know you have a multicast packet or two in your network :-)

as long as
> off the table.  One way or other the piper gets paid.

One way, the piper gets paid a fair bit more than the other!

> may be realistic if you're willing to accept the caveats, but does not

Good point.  Clearly, this is not a local decision: "route IP, bridge
non-IP" means that intra-subnet routes are propagated the same way as
inter-subnet routes, and thus every NVE, h/w or s/w, must be on the same
page.

To make this concrete using BGP VPNs, "route IP, bridge non-IP" means all
routes, intra- and inter-subnet, are propagated as IP VPN routes, and E-VPN
routes contain MACs without IPs.  "Bridge intra-subnet IP and non-IP, route
inter-subnet" means inter-subnet routes are propagated as IP VPN routes,
and intra-VPN routes as E-VPN MAC+IP routes.

We can have a chat off-list on h/w vendors working towards this.
 Hopefully, others will weigh the above arguments, and support this.
 Deployers (like you) have a say in this too :-)

> like to hear your take on any other *insolvable* issues with regard to

Isn't that enough?  :-)  I am not a multicast expert, but I can try to dig
up IRB multicast horror stories.

Cheers,
Kireeti.

Best regards -- aldrin
> > IRB follows the principle of "bridge when you can, route otherwise".
>  So, an IP packet with dest IP in the same subnet actually gets bridged;
> the originator (e.g., the VM) is responsible for ARPing the IP address,
> slapping the right dest MAC on the packet and sending that to the NVE which
> simply forwards based on dest MAC address *without* decrementing the TTL.
> >
> > If the dest IP is in another subnet, the packet is sent to the gateway
> (which for IRB would be the same NVE), which this time does an IP address
> lookup, decrements TTL and routes the packet.
> >
> > If one were to configure multiple subnets on a VLAN, I wonder if LL
> traffic goes to all members of the VLAN, or just those in the same subnet
> as the sender.  I suspect the former (but don't know).
> >

-- 
Kireeti

---

From: "Ali Sajassi (sajassi)" <[email-masked]>
Date: Wed, 19 Dec 2012 09:54:40 +0000

>I'm sure someone suggested this already, so put me down as supporting

IRB has been around and deployed for ages. What multicast complications
are you referring to?

>IP traffic to the IP VPN and sends the rest to E-VPN.  How this happens
>is implementation specific.  Note that this assumes that the NVE
>intercepts ARPs and responds to them with the same MAC.  Does anyone see
>a problem with this?

I don't think we should draw artificial lines around different solutions.
If a single IRB solution can do the job, then why do we need to mandate
two different solutions.

-Ali

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 19 Dec 2012 14:43:21 -0500

Hi Kireeti,

In E-VPN, ARP is only flooded when the MAC-IP binding is unknown in BGP.
 Once it is known, the local PE responds locally to the ARP request.   This
scales quite well so it's not the best reason to lean one way or other.

An alternative for edge routing using EVPN is for an NVE to localize the
VNs to which edge routing is desired and stand up a local IP forwarder
across these VN using the IP info in the EVPN routes.  If the DMAC on a
packet is not present in the EVI and if the payload is IP then pass to the
IP forwarder....

In regards to optimizing multicast, with EVPN this can be
done using VN dedicated to multicast distribution by using
the VLAN-based MVR model.  It works well and used today.

Another problem that is addressed in EVPN is that segments can be
multihomed using LAG.  With IP-only solutions, physical end station would
need to multihome by advertising loopback IP over multiple physical IP
interfaces.

We can have our TORs and use them too!! :)

Best regards -- aldrin

> (Note further that this may be hard to do in general, but in the case of
> an orchestrated data center, you have the information about where a given
> IP lives, and you have a control plane (ORACLE) to inform all relevant
> NVEs.  And of course, an overlay to shield the infrastructure from poking
> its nose into your forwarding behavior -- i.e., the infra doesn't care
> whether you route or switch TS traffic.)
>
> A third difference is that if you have only unicast traffic, you don't
> have to maintain a multicast tree (for flooding).  For some, this is a nice

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Thu, 20 Dec 2012 21:36:50 +0000

EVPN complexity lies in the interaction with bridging. For instance if one connects two EVPN access circuits with a physical wire (or bridges two VMs over a tunnel) you get a multihomed bridged site. Only one of the access ports can be active or otherwise loops will form.

But let's step back and look at the problem we are trying to solve. If majority (if not all) of traffic is IP and if majority of it is routed, wouldn't it be better to develop a networking solution that is optimized for this majority of traffic (and not the vice versa)?

The question is what problem does EVPN solve? In the context of DC, EVPN can only address packets bridged in the same VLAN. If most packets are routed then EVPN, even if all the complexity problems are addressed, doesn't achieve anything for the traffic that is routed. I believe it is the wrong tradeoff to design a solution around EVPN (i.e., around bridging).

Maria

From: [email-masked] [mailto:[email-masked]] On Behalf Of Aldrin Isaac
Sent: Wednesday, December 19, 2012 2:43 PM
To: Kireeti Kompella
Cc: Thomas Narten; [email-masked]
Subject: [nvo3] Multi-subnet VNs [was Re: FW: New Version Notification for draft-yong-nvo3-frwk-dpreq-addition-00.txt]

Hi Kireeti,

In E-VPN, ARP is only flooded when the MAC-IP binding is unknown in BGP.  Once it is known, the local PE responds locally to the ARP request.   This scales quite well so it's not the best reason to lean one way or other.

An alternative for edge routing using EVPN is for an NVE to localize the VNs to which edge routing is desired and stand up a local IP forwarder across these VN using the IP info in the EVPN routes.  If the DMAC on a packet is not present in the EVI and if the payload is IP then pass to the IP forwarder....

In regards to optimizing multicast, with EVPN this can be done using VN dedicated to multicast distribution by using the VLAN-based MVR model.  It works well and used today.

Another problem that is addressed in EVPN is that segments can be multihomed using LAG.  With IP-only solutions, physical end station would need to multihome by advertising loopback IP over multiple physical IP interfaces.

Best regards -- aldrin

On Wednesday, December 19, 2012, Kireeti Kompella wrote:
Hi Aldrin,
On Tue, Dec 18, 2012 at 8:29 PM, Aldrin Isaac <[email-masked]<mailto:[email-masked]>> wrote:
Kireeti,

Two important differences:
a) you don't have to know the MAC address if you forward on IP.  I.e., you don't have to propagate the ARP to the destination (flood), get the reply, bind IP to MAC (ARP table), and maintain ARP binding (timeout, validate, etc.).  The first is a real problem; the rest are annoyances that become problems at scale.

(Note that the ARMD WG was created to address this issue, and you know where that ended.)

(Note further that this may be hard to do in general, but in the case of an orchestrated data center, you have the information about where a given IP lives, and you have a control plane (ORACLE) to inform all relevant NVEs.  And of course, an overlay to shield the infrastructure from poking its nose into your forwarding behavior -- i.e., the infra doesn't care whether you route or switch TS traffic.)

b) In the quite common case where all traffic from a TS is IP, you don't have to maintain two tables and two forwarding paradigms at the NVE (one for IPs and one for MACs).  This is common enough to warrant optimization.

A third difference is that if you have only unicast traffic, you don't have to maintain a multicast tree (for flooding).  For some, this is a nice bonus, but I know you have a multicast packet or two in your network :-)

as long as

Good point.  Clearly, this is not a local decision: "route IP, bridge non-IP" means that intra-subnet routes are propagated the same way as inter-subnet routes, and thus every NVE, h/w or s/w, must be on the same page.

To make this concrete using BGP VPNs, "route IP, bridge non-IP" means all routes, intra- and inter-subnet, are propagated as IP VPN routes, and E-VPN routes contain MACs without IPs.  "Bridge intra-subnet IP and non-IP, route inter-subnet" means inter-subnet routes are propagated as IP VPN routes, and intra-VPN routes as E-VPN MAC+IP routes.

We can have a chat off-list on h/w vendors working towards this.  Hopefully, others will weigh the above arguments, and support this.  Deployers (like you) have a say in this too :-)

Isn't that enough?  :-)  I am not a multicast expert, but I can try to dig up IRB multicast horror stories.

Cheers,
Kireeti.

Best regards -- aldrin

On Tue, Dec 18, 2012 at 6:06 PM, Kireeti Kompella
<[email-masked]<mailto:[email-masked]>> wrote:
> Hi Thomas,
>
> On Dec 18, 2012, at 09:03 , Thomas Narten <[email-masked]<mailto:[email-masked]>> wrote:
>
>> Kireeti Kompella <[email-masked]<mailto:[email-masked]>> writes:
>>
>> [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>

--
Kireeti

---

From: Yakov Rekhter <[email-masked]>
Date: Fri, 21 Dec 2012 08:24:19 -0800

Maria,

> EVPN complexity lies in the interaction with bridging. For instance if one =
> connects two EVPN access circuits with a physical wire (or bridges two VMs =
> over a tunnel) you get a multihomed bridged site. Only one of the access po=
> rts can be active or otherwise loops will form.
> 
> But let's step back and look at the problem we are trying to solve. If majo=
> rity (if not all) of traffic is IP and if majority of it is routed, wouldn'=
> t it be better to develop a networking solution that is optimized for this =
> majority of traffic (and not the vice versa)?
> 
> The question is what problem does EVPN solve? In the context of DC, EVPN can
> only address packets bridged in the same VLAN. If most packets are routed
> then EVPN, even if all the complexity problems are addressed, doesn't achieve
> anything for the traffic that is routed. 

The claim you made in the last paragraph above is factually incorrect
- in the context of DC, EVPN can address *not* "only packets bridged
in the same VLAN", but *also* can be used to provide (optimal) routing
among VMs in *different* VLANs (different IP subnets). For more details 
please read section 6.1 of draft-raggarwa-data-center-mobility-04.txt
(please note that what is described in 6.1 is applicable both in the 
presence and in the absence of VM mobility).

> I believe it is the wrong tradeoff to design a solution around EVPN 
> (i.e., around bridging).

You are certainly entitled to have your own beliefs...

Yakov.

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ald=
> rin Isaac
> Subject: [nvo3] Multi-subnet VNs [was Re: FW: New Version Notification for =
> draft-yong-nvo3-frwk-dpreq-addition-00.txt]
> 
> Hi Kireeti,
> 
> In E-VPN, ARP is only flooded when the MAC-IP binding is unknown in BGP.  O=
> nce it is known, the local PE responds locally to the ARP request.   This s=
> cales quite well so it's not the best reason to lean one way or other.
> 
> An alternative for edge routing using EVPN is for an NVE to localize the VN=
> s to which edge routing is desired and stand up a local IP forwarder across=
>  these VN using the IP info in the EVPN routes.  If the DMAC on a packet is=
>  not present in the EVI and if the payload is IP then pass to the IP forwar=
> der....
> 
> In regards to optimizing multicast, with EVPN this can be done using VN ded=
> icated to multicast distribution by using the VLAN-based MVR model.  It wor=
> ks well and used today.
> 
> Another problem that is addressed in EVPN is that segments can be multihome=
> d using LAG.  With IP-only solutions, physical end station would need to mu=
> ltihome by advertising loopback IP over multiple physical IP interfaces.
> 
> On Tue, Dec 18, 2012 at 8:29 PM, Aldrin Isaac <[email-masked]<mailt=
> o:[email-masked]>> wrote:
> Kireeti,
> 
> a) you don't have to know the MAC address if you forward on IP.  I.e., you =
> don't have to propagate the ARP to the destination (flood), get the reply, =
> bind IP to MAC (ARP table), and maintain ARP binding (timeout, validate, et=
> c.).  The first is a real problem; the rest are annoyances that become prob=
> lems at scale.
> 
> (Note that the ARMD WG was created to address this issue, and you know wher=
> e that ended.)
> 
> (Note further that this may be hard to do in general, but in the case of an=
>  orchestrated data center, you have the information about where a given IP =
> lives, and you have a control plane (ORACLE) to inform all relevant NVEs.  =
> And of course, an overlay to shield the infrastructure from poking its nose=
>  into your forwarding behavior -- i.e., the infra doesn't care whether you =
> route or switch TS traffic.)
> 
> b) In the quite common case where all traffic from a TS is IP, you don't ha=
> ve to maintain two tables and two forwarding paradigms at the NVE (one for =
> IPs and one for MACs).  This is common enough to warrant optimization.
> 
> A third difference is that if you have only unicast traffic, you don't have=
>  to maintain a multicast tree (for flooding).  For some, this is a nice bon=
> us, but I know you have a multicast packet or two in your network :-)
> 
> as long as
> Good point.  Clearly, this is not a local decision: "route IP, bridge non-I=
> P" means that intra-subnet routes are propagated the same way as inter-subn=
> et routes, and thus every NVE, h/w or s/w, must be on the same page.
> 
> To make this concrete using BGP VPNs, "route IP, bridge non-IP" means all r=
> outes, intra- and inter-subnet, are propagated as IP VPN routes, and E-VPN =
> routes contain MACs without IPs.  "Bridge intra-subnet IP and non-IP, route=
>  inter-subnet" means inter-subnet routes are propagated as IP VPN routes, a=
> nd intra-VPN routes as E-VPN MAC+IP routes.
> 
> We can have a chat off-list on h/w vendors working towards this.  Hopefully=
> , others will weigh the above arguments, and support this.  Deployers (like=
>  you) have a say in this too :-)
> 
> Isn't that enough?  :-)  I am not a multicast expert, but I can try to dig =
> > On Dec 18, 2012, at 09:03 , Thomas Narten <[email-masked]<mailto:narte=
> [email-masked]>> wrote:
> >
> >> Kireeti Kompella <[email-masked]<mailto:kireeti.kompella@gma=
> il.com>> writes:
> >>
> > IRB follows the principle of "bridge when you can, route otherwise".  So,=
>  an IP packet with dest IP in the same subnet actually gets bridged; the or=
> iginator (e.g., the VM) is responsible for ARPing the IP address, slapping =
> the right dest MAC on the packet and sending that to the NVE which simply f=
> orwards based on dest MAC address *without* decrementing the TTL.
> >
> > If the dest IP is in another subnet, the packet is sent to the gateway (w=
> hich for IRB would be the same NVE), which this time does an IP address loo=
> kup, decrements TTL and routes the packet.
> >
> > If one were to configure multiple subnets on a VLAN, I wonder if LL traff=
> ic goes to all members of the VLAN, or just those in the same subnet as the=
>  sender.  I suspect the former (but don't know).
> >
> > Can be done.  However, we're on Lucy's topic; mine was "route if IP, brid=
> ge otherwise"; the goal was to rationalize the need for Layer 2 forwarding =
> for non-IP traffic, and inter- and intra-subnet routing.
> >
> > Kireeti.
> >
> >> Thomas
> >>
> --_000_1D70D757A2C9D54D83B4CBD7625FA80E010E66A5MISOUT7MSGUSR9I_
> Content-Type: text/html; charset="us-ascii"
> Content-Transfer-Encoding: quoted-printable
> 
> <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
> osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
> xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
> //www.w3.org/TR/REC-html40">
> <head>
> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
> >
> <meta name=3D"Generator" content=3D"Microsoft Word 12 (filtered medium)">
> <style><!--
> /* Font Definitions */
> @font-face
> 	{font-family:Calibri;
> 	panose-1:2 15 5 2 2 2 4 3 2 4;}
> @font-face
> 	{font-family:Tahoma;
> 	panose-1:2 11 6 4 3 5 4 4 2 4;}
> @font-face
> 	{font-family:Consolas;
> 	panose-1:2 11 6 9 2 2 4 3 2 4;}
> /* Style Definitions */
> p.MsoNormal, li.MsoNormal, div.MsoNormal
> 	{margin:0in;
> 	margin-bottom:.0001pt;
> 	font-size:12.0pt;
> 	font-family:"Times New Roman","serif";}
> a:link, span.MsoHyperlink
> 	{mso-style-priority:99;
> 	color:blue;
> 	text-decoration:underline;}
> a:visited, span.MsoHyperlinkFollowed
> 	{mso-style-priority:99;
> 	color:purple;
> 	text-decoration:underline;}
> p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
> 	{mso-style-priority:99;
> 	mso-style-link:"Plain Text Char";
> 	margin:0in;
> 	margin-bottom:.0001pt;
> 	font-size:10.5pt;
> 	font-family:Consolas;}
> span.EmailStyle17
> 	{mso-style-type:personal-reply;
> 	font-family:"Calibri","sans-serif";
> 	color:#1F497D;}
> span.PlainTextChar
> 	{mso-style-name:"Plain Text Char";
> 	mso-style-priority:99;
> 	mso-style-link:"Plain Text";
> 	font-family:Consolas;}
> .MsoChpDefault
> 	{mso-style-type:export-only;}
> @page WordSection1
> 	{size:8.5in 11.0in;
> 	margin:1.0in 1.0in 1.0in 1.0in;}
> div.WordSection1
> 	{page:WordSection1;}
> --></style><!--[if gte mso 9]><xml>
> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
> </xml><![endif]--><!--[if gte mso 9]><xml>
> <o:shapelayout v:ext=3D"edit">
> <o:idmap v:ext=3D"edit" data=3D"1" />
> </o:shapelayout></xml><![endif]-->
> </head>
> <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
> <div class=3D"WordSection1">
> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:Consolas=
> ">EVPN complexity lies in the interaction with bridging. For instance if on=
> e connects two EVPN access circuits with a physical wire (or bridges two VM=
> s over a tunnel) you get a multihomed
>  bridged site. Only one of the access ports can be active or otherwise loop=
> s will form.<o:p></o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:Consolas=
> "><o:p>&nbsp;</o:p></span></p>
> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:Consolas=
> ">But let&#8217;s step back and look at the problem we are trying to solve.=
>  If majority (if not all) of traffic is IP and if majority of it is routed,=
>  wouldn&#8217;t it be better to develop a networking
>  solution that is optimized for this majority of traffic (and not the vice =
> versa)?<o:p></o:p></span></p>
> <p class=3D"MsoPlainText"><span style=3D"font-size:11.0pt">The question is =
> what problem does EVPN solve? In the context of DC, EVPN can only address p=
> ackets bridged in the same VLAN. If most packets are routed then EVPN, even=
>  if all the complexity problems are
>  addressed, doesn't achieve anything for the traffic that is routed. I beli=
> eve it is the wrong tradeoff to design a solution around EVPN (i.e., around=
>  bridging).<o:p></o:p></span></p>
> <p class=3D"MsoPlainText"><span style=3D"font-size:11.0pt"><o:p>&nbsp;</o:p=
> ></span></p>
> <p class=3D"MsoPlainText"><span style=3D"font-size:11.0pt">Maria<o:p></o:p>=
> </span></p>
> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:&quot;Ca=
> libri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span><=
> /p>
> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:&quot;Ca=
> libri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span><=
> /p>
> <div style=3D"border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in =
> 4.0pt">
> <div>
> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
> 0in 0in">
> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:&quot=
> ;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style=3D"font-s=
> ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> nvo3-bou=
> [email-masked] [mailto:[email-masked]]
> <b>On Behalf Of </b>Aldrin Isaac<br>
> <b>Sent:</b> Wednesday, December 19, 2012 2:43 PM<br>
> <b>To:</b> Kireeti Kompella<br>
> <b>Cc:</b> Thomas Narten; [email-masked]<br>
> <b>Subject:</b> [nvo3] Multi-subnet VNs [was Re: FW: New Version Notificati=
> on for draft-yong-nvo3-frwk-dpreq-addition-00.txt]<o:p></o:p></span></p>
> </div>
> </div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> <p class=3D"MsoNormal">Hi Kireeti,<br>
> <br>
> In E-VPN, ARP is only flooded when the MAC-IP binding is unknown in BGP. &n=
> bsp;Once it is known, the local PE responds locally to the ARP request. &nb=
> sp;&nbsp;This scales quite well so it's not the best reason to lean one way=
>  or other.&nbsp;<o:p></o:p></p>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">An alternative for edge routing using EVPN&nbsp;is f=
> or an NVE to localize the VNs to which edge routing is desired and stand up=
>  a local IP&nbsp;forwarder across these VN using the IP info in the EVPN ro=
> utes. &nbsp;If the DMAC on a packet&nbsp;is not present
>  in the EVI and&nbsp;if the payload is IP then&nbsp;pass to the IP&nbsp;for=
> warder....<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">In regards to optimizing&nbsp;multicast,&nbsp;with&n=
> bsp;EVPN&nbsp;this can be done&nbsp;using&nbsp;VN&nbsp;dedicated to multica=
> st distribution by&nbsp;using the&nbsp;VLAN-based&nbsp;MVR model. &nbsp;It =
> works well and used today. &nbsp;<o:p></o:p></p>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">Another problem that is addressed in&nbsp;EVPN is th=
> at segments&nbsp;can be multihomed using LAG. &nbsp;With IP-only solutions,=
>  physical&nbsp;end station&nbsp;would need to multihome by&nbsp;advertising=
>  loopback IP over multiple physical&nbsp;IP interfaces.&nbsp;<o:p></o:p></p=
> >
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">We can have our TORs and use them too!! :)<o:p></o:p=
> ></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">Best regards -- aldrin<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><br>
> On Wednesday, December 19, 2012, Kireeti Kompella wrote:<o:p></o:p></p>
> <div>
> <div>
> <p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt">Hi Aldrin,<o:p></o:p>=
> </p>
> <div>
> <p class=3D"MsoNormal">On Tue, Dec 18, 2012 at 8:29 PM, Aldrin Isaac &lt;<a=
>  href=3D"mailto:[email-masked]">[email-masked]</a>&gt; wrot=
> e:<o:p></o:p></p>
> <p class=3D"MsoNormal">Kireeti,<br>
> <br>
> I'm not clear what difference it makes whether a packet is unicast<br>
> forwarded using MAC address or IP address within a subnet <o:p></o:p></p>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">Two important differences:<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">a) you don't have to know the MAC address if you for=
> ward on IP. &nbsp;I.e., you don't have to propagate the ARP to the destinat=
> ion (flood), get the reply, bind IP to MAC (ARP table), and maintain ARP bi=
> nding (timeout, validate, etc.). &nbsp;The first
>  is a real problem; the rest are annoyances that become problems at scale.<=
> o:p></o:p></p>
> </div>
> </div>
> </div>
> </div>
> <blockquote style=3D"border:none;border-left:solid #CCCCCC 1.0pt;padding:0i=
> n 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
> <div>
> <div>
> <div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">(Note that the ARMD WG was created to address this i=
> ssue, and you know where that ended.)<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">(Note further that this may be hard to do in general=
> , but in the case of an orchestrated data center, you have the information =
> about where a given IP lives, and you have a control plane (ORACLE) to info=
> rm all relevant NVEs. &nbsp;And of course,
>  an overlay to shield the infrastructure from poking its nose into your for=
> warding behavior -- i.e., the infra doesn't care whether you route or switc=
> h TS traffic.)<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">b) In the quite common case where all traffic from a=
>  TS is IP, you don't have to maintain two tables and two forwarding paradig=
> ms at the NVE (one for IPs and one for MACs). &nbsp;This is common enough t=
> o warrant optimization.<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">A third difference is that&nbsp;if you have only uni=
> cast traffic,&nbsp;you don't have to maintain a multicast tree (for floodin=
> g). &nbsp;For some, this is a nice bonus, but I know you have a multicast p=
> acket or two in your network :-)<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <blockquote style=3D"border:none;border-left:solid #CCCCCC 1.0pt;padding:0i=
> n 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
> <p class=3D"MsoNormal">as long as<br>
> it gets to the intended destination along the most optimal path,<br>
> particularly when the price to pay is non-standard behavior<br>
> (intra-subnet ARP manglers ;}, etc). &nbsp;I understand the argument about<=
> br>
> the sub-optimal routing from a third site, but when the primary sites<br>
> end up aggregating prefixes for scaling reasons that argument falls<br>
> off the table. &nbsp;One way or other the piper gets paid.<o:p></o:p></p>
> </blockquote>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">One way, the piper gets paid a fair bit more than th=
> e other!<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <blockquote style=3D"border:none;border-left:solid #CCCCCC 1.0pt;padding:0i=
> n 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
> <p class=3D"MsoNormal">In terms of the real world issue of getting there fr=
> om here --<br>
> personally I haven't seen any vendor working towards a standards-based<br>
> solution that will allow intra-subnet routing for subnets over<br>
> HW/TOR-based PE, let alone intra-subnet routing for subnets that span<br>
> across both hypervisor-based PE and TOR-based PE. &nbsp;This makes me leery=
> <br>
> of solutions that can only take us half way there, particularly during<br>
> the transition phase. &nbsp;So if we're talking about network<br>
> virtualization based purely on hypervisors, &quot;route IP, bridge non-IP&q=
> uot;<br>
> may be realistic if you're willing to accept the caveats, but does not<br>
> seem to be otherwise.<o:p></o:p></p>
> </blockquote>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">Good point. &nbsp;Clearly, this is not a local decis=
> ion: &quot;route IP, bridge non-IP&quot; means that intra-subnet routes are=
>  propagated the same way as inter-subnet routes, and thus every NVE, h/w or=
>  s/w, must be on the same page.<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">To make this concrete using BGP VPNs, &quot;route IP=
> , bridge non-IP&quot; means all routes, intra- and inter-subnet, are propag=
> ated as IP VPN routes, and E-VPN routes contain MACs without IPs. &nbsp;&qu=
> ot;Bridge intra-subnet IP and non-IP, route inter-subnet&quot;
>  means inter-subnet routes are propagated as IP VPN routes, and intra-VPN r=
> outes as E-VPN MAC&#43;IP routes.<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">We can have a chat off-list on h/w vendors working t=
> owards this. &nbsp;Hopefully, others will weigh the above arguments, and su=
> pport this. &nbsp;Deployers (like you) have a say in this too :-)<o:p></o:p=
> ></p>
> </div>
> <div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <blockquote style=3D"border:none;border-left:solid #CCCCCC 1.0pt;padding:0i=
> n 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
> <p class=3D"MsoNormal">Btw, I understand how multicast may be less than eff=
> icient when<br>
> building both inter and intra subnet trees for the same IP mcast group<br>
> that end up overlapping links (maybe even more than twice) -- but I'd<br>
> like to hear your take on any other *insolvable* issues with regard to<br>
> multicast.<o:p></o:p></p>
> </blockquote>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">Isn't that enough? &nbsp;:-) &nbsp;I am not a multic=
> ast expert, but I can try to dig up IRB multicast horror stories.<o:p></o:p=
> ></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">Cheers,<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal">Kireeti.<o:p></o:p></p>
> </div>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <blockquote style=3D"border:none;border-left:solid #CCCCCC 1.0pt;padding:0i=
> n 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
> <p class=3D"MsoNormal">Best regards -- aldrin<o:p></o:p></p>
> <div>
> <p class=3D"MsoNormal"><br>
> <br>
> <br>
> On Tue, Dec 18, 2012 at 6:06 PM, Kireeti Kompella<br>
> &lt;<a href=3D"mailto:[email-masked]">[email-masked]=
> m</a>&gt; wrote:<o:p></o:p></p>
> </div>
> <div>
> <div>
> <p class=3D"MsoNormal">&gt; Hi Thomas,<br>
> &gt;<br>
> &gt; On Dec 18, 2012, at 09:03 , Thomas Narten &lt;<a href=3D"mailto:narten=
> @us.ibm.com">[email-masked]</a>&gt; wrote:<br>
> &gt;<br>
> &gt;&gt; Kireeti Kompella &lt;<a href=3D"mailto:[email-masked]"=
> >[email-masked]</a>&gt; writes:<br>
> &gt;&gt;<br>
> &gt;&gt;&gt; The solution is simple: route if IP, bridge if not. &nbsp;Yes,=
>  one could<br>
> &gt;&gt;&gt; do IRB, but why? &nbsp;IRB brings in complications, especially=
>  for<br>
> &gt;&gt;&gt; multicast. &nbsp;I'm sure someone suggested this already, so p=
> ut me down<br>
> &gt;&gt;&gt; as supporting this view.<br>
> &gt;&gt;<br>
> &gt;&gt; I'm not sure I understand the difference.<br>
> &gt;&gt;<br>
> &gt;&gt; From an *NVE* perspective, when it receives a packet (which will h=
> ave<br>
> &gt;&gt; an L2 header), it can look at the Ethertype, and if its IP, it can=
> <br>
> &gt;&gt; route it. Otherwise, it can provide normal L2 service. So, in this=
> <br>
> &gt;&gt; sense, &quot;route if IP, bridge if not&quot; is straightforward. =
> And more to<br>
> &gt;&gt; the point, I assume that if the packet gets L2 service, the entire=
 VN<br>
> &gt;&gt; is treated as a *single* broadcast domain. All nodes can reach all=
> <br>
> &gt;&gt; other nodes. Right?<br>
> &gt;<br>
> &gt; Right.<br>
> &gt;<br>
> &gt;&gt; Just so I understand, how is this different than IRB? &nbsp;What d=
> oes IRB<br>
> &gt;&gt; imply that the above does not?<br>
> &gt;<br>
> &gt; IRB follows the principle of &quot;bridge when you can, route otherwis=
> e&quot;. &nbsp;So, an IP packet with dest IP in the same subnet actually ge=
> ts bridged; the originator (e.g., the VM) is responsible for ARPing the IP =
> address, slapping the right dest MAC on the packet
>  and sending that to the NVE which simply forwards based on dest MAC addres=
> s *without* decrementing the TTL.<br>
> &gt;<br>
> &gt; If the dest IP is in another subnet, the packet is sent to the gateway=
>  (which for IRB would be the same NVE), which this time does an IP address =
> lookup, decrements TTL and routes the packet.<br>
> &gt;<br>
> &gt; For multicast, there are even more differences.<br>
> &gt;<br>
> &gt;&gt; But this is different than what (I believe) Lucy is arguing for. I=
> n<br>
> &gt;&gt; the case of a multi-subnet VN, you have one VN, but it contains<br=
> >
> &gt;&gt; different subnets. Each subnet is intended to be one broadcast dom=
> ain<br>
> &gt;&gt; (i.e., equivalent of a VLAN), so that when sending LL multicast an=
> d<br>
> &gt;&gt; the like on a specific subnet, such packets are *not* delivered to=
>  all<br>
> &gt;&gt; nodes in the VN, but only those that are part of subnet.<br>
> &gt;<br>
> &gt; If one were to configure multiple subnets on a VLAN, I wonder if LL tr=
> affic goes to all members of the VLAN, or just those in the same subnet as =
> the sender. &nbsp;I suspect the former (but don't know).<br>
> &gt;<br>
> &gt;&gt; This is a more complex type of service to provide. And I'm not sur=
> e we<br>
> &gt;&gt; need this type of service to be provided by one VN.<br>
> &gt;<br>
> &gt; Agree.<br>
> &gt;<br>
> &gt;&gt; A (seemingly<br>
> &gt;&gt; simpler) alternative would be to put each subnet in its own VN and=
> <br>
> &gt;&gt; allow inter-subnet traffic to be handed as inter-VN traffic. So lo=
> ng<br>
> &gt;&gt; as that case is optimized (i.e., the ingress NVE can tunnel direct=
> ly<br>
> &gt;&gt; to the egress NVE without adding triangular routing), this would s=
> eem<br>
> &gt;&gt; to be a cleaner way to implement this.<br>
> &gt;<br>
> &gt; Can be done. &nbsp;However, we're on Lucy's topic; mine was &quot;rout=
> e if IP, bridge otherwise&quot;; the goal was to rationalize the need for L=
> ayer 2 forwarding for non-IP traffic, and inter- and intra-subnet routing.<=
> br>
> &gt;<br>
> &gt; Kireeti.<br>
> &gt;<br>
> &gt;&gt; Thomas<br>
> &gt;&gt;<br>
> &gt;&gt; _______________________________________________<br>
> &gt;&gt; nvo3 mailing list<br>
> &gt;&gt; <a href=3D"mailto:[email-masked]">[email-masked]</a><br>
> &gt;&gt; <a href=3D"https://www.ietf.org/mailman/listinfo/nvo3" target=3D"_=
> blank">https://www.ietf.org/mailman/listinfo/nvo3</a><br>
> &gt;<br>
> &gt; _______________________________________________<br>
> &gt; nvo3 mailing list<br>
> &gt; <a href=3D"mailto:[email-masked]">[email-masked]</a><br>
> &gt; <a href=3D"https://www.ietf.org/mailman/listinfo/nvo3" target=3D"_blan=
> k">https://www.ietf.org/mailman/listinfo/nvo3</a><o:p></o:p></p>
> </div>
> </div>
> </blockquote>
> </div>
> <p class=3D"MsoNormal"><br>
> <br clear=3D"all">
> <o:p></o:p></p>
> <div>
> <p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
> </div>
> <p class=3D"MsoNormal">-- <br>
> Kireeti<o:p></o:p></p>
> </div>
> </div>
> </blockquote>
> </div>
> </div>
> </div>
> </div>
> </body>
> </html>
> 
> --_000_1D70D757A2C9D54D83B4CBD7625FA80E010E66A5MISOUT7MSGUSR9I_--
> 
> --===============3861516006203949397==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
> 
> --===============3861516006203949397==--

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Dec 2012 17:52:06 +0100

Hi Yakov,

> among VMs in *different* VLANs (different IP subnets). For more details

Allow me to observe that the goal of DC design is not only to to
provide optimal routing within the DC various VLANs (different IP
subnets) and treat DCs are isolated islands.

For a fact I am currently working on integration of DC tenant VPNs
with *existing* deployed L3VPNs in the WAN network.

In this respect yr claim that EVPN solves the problem is factually
incorrect too as it requires to build EVPN (SAFI 70) to RFC4364 (SAFI
128) gateways or to transition my WAN to be now EVPN based. Both
options as you can imagine are not that operationally attractive.

Best wishes,
Robert.

---

From: Lucy yong <[email-masked]>
Date: Fri, 21 Dec 2012 17:51:58 +0000

Hi Robert,

Please see inline.

> Robert Raszuk
> Sent: Friday, December 21, 2012 10:52 AM
> To: Yakov Rekhter
> Cc: Thomas Narten; Kireeti Kompella; Aldrin Isaac; NAPIERALA, MARIA H;
> [email-masked]
> Subject: Re: [nvo3] Multi-subnet VNs [was Re: FW: New Version
> Notification for draft-yong-nvo3-frwk-dpreq-addition-00.txt]
> 
> Hi Yakov,
> 
[Lucy] yes, I see this use case.
> with *existing* deployed L3VPNs in the WAN network.
[Lucy] would you mind say which option in rfc4346 fit to your application?
> options as you can imagine are not that operationally attractive.
[Lucy] Then, what is the proper operational way for your application? 
Do you see that E-IP-VPN may bring operation attractive?

Lucy

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Fri, 21 Dec 2012 17:53:19 +0000

Route (IP traffic) within subnet is happening, regardless physical or
virtual, vpn or non-vpn, call it enhanced/advanced forwarding.
Luyuan

-----Original Message-----
From: Lucy yong <[email-masked]>
Date: Tuesday, December 18, 2012 10:10 AM
To: Luyuan Fang <[email-masked]>, Kireeti Kompella
<[email-masked]>, "NAPIERALA, MARIA H" <[email-masked]>
Cc: Aldrin Isaac <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: RE: [nvo3] FW: New Version Notification for
draft-yong-nvo3-frwk-dpreq-addition-00.txt

>[Lucy] not sure. We are working on the network virtualization. The one of
>goals is to virtualizes the enterprise physical networks that provides
>the networking for compute resource on a DC common infrastructure. If you
>look at today's enterprise data center networks, do they all use routers
>without any switch? Not at all. If routers can do all the functions, why
>it is not the case today in any physical network? You may argue that they
>are legacy. But I do not see a new built DC without a switch too. Why is
>that? How can we assume that IP VPN is sufficient for a tenant virtual
>network.
> 
>> >
>[Lucy] Again, IRB are widely used in enterprise data center. It is
>useful. How do we know that IP VPN is not complicated for a tenant
>virtual network? 
>> 
>> 
>[Lucy] IMO: Pedro's solution applies to certain application, but not
>other application. For example, his draft does not mention how to provide
>a broadcast for a subnet. Unless a tenant virtual network does not need
>this at all, we can not remove this requirement.
>> 
>> >
>[Lucy] Not at all. Another objective of NV03 is the ability to move TS
>among different servers regardless if they are on the same or different
>subnet. Do you see this is "very narrow cases"?
>
>

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Dec 2012 18:59:55 +0100

Hi Lucy,

> [Lucy] yes, I see this use case.

Great !

> [Lucy] would you mind say which option in rfc4346 fit to your application?

What options are you referring to ? I am talking about SAFI 128 vs
SAFI 70 - encoding of routes is the main difference

> [Lucy] Then, what is the proper operational way for your application?

To run RFC4364 within and outside of my DCs.

> Do you see that E-IP-VPN may bring operation attractive?

I am not claiming that there is no traffic which may not be IP within
a data center. However I can not accept that E-IP-VPN is the solution
for everything.

So perhaps we should come back to origin idea of running both RFC4364
and DC-optimized EVPN as control plane - however decoupled from each
other not hard-integrated.

Best regards,
R.

---

From: Yakov Rekhter <[email-masked]>
Date: Fri, 21 Dec 2012 10:11:25 -0800

Robert,

> options as you can imagine are not that operationally attractive.

Perhaps if you would bother to read my e-mail carefully you would
be able to notice that my claim was about the ability of E-VPN to
provide (optimal) routing among VMs in *different* VLANs - I made
no claims about interworking between VMs in DC and 2547 VPN sites.

So, you are yet to show that my claim "is factually incorrect".

Yakov.

---

From: Yakov Rekhter <[email-masked]>
Date: Fri, 21 Dec 2012 10:14:00 -0800

Robert,

> To run RFC4364 within and outside of my DCs.

Are you proposing that within a DC you run only RFC4364 ?

Yakov.

---

From: Lucy yong <[email-masked]>
Date: Fri, 21 Dec 2012 18:47:45 +0000

Robert,

> To run RFC4364 within and outside of my DCs.
[Lucy] In your DC, do VM or physical server support Pedro's solution? if not, how do you run rfc4364 in your DC?
 
> a data center. However I can not accept that E-IP-VPN is the solution
[Lucy] I do not say that E-IP-VPN for everything. we have EVPN and IP VPN. 
> other not hard-integrated.
[Lucy] where was the original idea? you mean to let operator deal with rfc4364 and evpn for their applications?

Lucy

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Dec 2012 21:08:55 +0100

> Are you proposing that within a DC you run only RFC4364 ?

Have you read my reply to Lucy ?

I think I state pretty clearly my opinion there. Quote:

"So perhaps we should come back to origin idea of running both RFC4364
and DC-optimized EVPN as control plane - however decoupled from each
other not hard-integrated."

Rgs,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Dec 2012 21:18:03 +0100

Hi,

> not, how do you run rfc4364 in your DC?

I am not sure what you are specifically referring to as "Pedro's solution"

L3vpn for end systems is useful in decoupling control and data plane -
so if you are asking for that - the answer is yes.

However there are many ways to accomplish the same decoupling or even
I am aware about work of some vendor to run BGP native to the hosts.

I think to organize the discussion we should be very specific and talk
SAFI here.

SAFI 128 is useful to interconnect IP subnets. SAFI 70 may be useful
to interconnect L2 segments.

> [Lucy] I do not say that E-IP-VPN for everything. we have EVPN and IP

As the name suggestes "E-IP-VPN" does integrate both EVPN and IPVPN.
What else it does not cover ? Token Ring or FDDI L2 ?

If you observe this thread carefully which apparently even Yakov have
missed the claim Kireeti has made is very clear what purpose E-IP-VPN
is to serve. Quote:

"b) In the quite common case where all traffic from a TS is IP, you
don't have to maintain two tables and two forwarding paradigms at the
NVE (one for IPs and one for MACs).  This is common enough to warrant
optimization."

> rfc4364 and evpn for their applications?

Why not ?

Cheers,
R.

---

From: Lucy yong <[email-masked]>
Date: Fri, 21 Dec 2012 22:32:43 +0000

Robert,

Please see inline.

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert
> Raszuk
> Sent: Friday, December 21, 2012 2:18 PM
> To: Lucy yong
> Cc: Yakov Rekhter; Thomas Narten; Kireeti Kompella; Aldrin Isaac;
> NAPIERALA, MARIA H; [email-masked]
> > [Lucy] In your DC, do VM or physical server support Pedro's solution?
> if
> > not, how do you run rfc4364 in your DC?
> 
[Lucy] something we have common reference. 
> so if you are asking for that - the answer is yes.
[Lucy] thanks.
> I am aware about work of some vendor to run BGP native to the hosts.
[Lucy] I can't reference this as rfc4364 that, imo, couple control and data plane.
> to interconnect L2 segments.
[Lucy] That is clear.
> What else it does not cover ? Token Ring or FDDI L2 ?
[Lucy] you have applications that fit better for EVPN or IPVPN. You also have the applications that 
require both. You can combine them as you want. However, some operator cost is expensive. For simple vanilla application, operation cost saving is beneficial.   
> NVE (one for IPs and one for MACs).  This is common enough to warrant
[Lucy] This is implementation specify. It is not necessary to maintain two tables.

Lucy 
> 
> > [Lucy] where was the original idea? you mean to let operator deal
> with
> > rfc4364 and evpn for their applications?
> 
> Why not ?
> 
> Cheers,
> R.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Dec 2012 23:45:23 +0100

Lucy,

>> NVE (one for IPs and one for MACs).  This is common enough to
>> warrant optimization."
>

Oh yes it is ... in the control plane. Kireeti was very clear and
correct separating control plane tables and forwarding paradigms in
his statement.

Implementation comes to optimize the latter not the former.

In other words you better have different tables for SAFI 128 and SAFI
70 in your BGP implementation if you are running both applications.

And to reiterate Maria's point if you already have SAFI 128 and the
major application is IP subnet interconnect adding SAFI 70 to address
marginal benefit of providing EVPN moreover trying to turn it also to
help with IP subnet interconnect seems to be clearly going wrong
direction.

Regards,
R.

---

From: "NAPIERALA, MARIA H" <[email-masked]>
Date: Sat, 22 Dec 2012 01:41:24 +0000

Yakov,

> presence and in the absence of VM mobility).

This re-invents l3vpn with using l2vpn route advertisements and introduces a possibility of extending the reach of a broadcast domain beyond a single subnet. The question is why to introduce a new paradigm for routing?

When traffic crosses between CUGs this traffic is IP routed (naturally and optimally addressed by L3VPNs). Within a CUG the traffic is handled the same way whether one implements EVPN or L3VPN (no visible difference). The only question remaining is how to address non-IP traffic (in DCs that carry or care about non-IP traffic). A solution should use EVPN only when necessary to handle non-IP traffic. As Kireeti described it: "route if IP, bridge otherwise".

In addition, the reference above (section 6.1) does not address one of the main requirements in a service provider's cloud computing environment which is a seamless integration with MPLS/BGP layer 3 VPNs in the WAN and the wireless access networks (pointed out by Robert). 

> You are certainly entitled to have your own beliefs...

It is not an isolated opinion or only a "personal" belief.

> Of Ald=
> multihome=
> non-I=
> Hopefully=
> > >> Kireeti Kompella
> <[email-masked]<mailto:kireeti.kompella@gma=
> > il.com>> writes:
> > >>
> traff=
> > libri&quot;,&quot;sans-
> serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span><=
> > /p>
> > libri&quot;,&quot;sans-
> serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span><=
> > /p>
> > &lt;<a
> href=3D"mailto:[email-masked]">[email-masked]=
> > &gt;&gt; Kireeti Kompella &lt;<a
> href=3D"mailto:[email-masked]"=

---

From: Kireeti Kompella <[email-masked]>
Date: Fri, 21 Dec 2012 20:20:37 -0800

Hi Maria,

On Dec 20, 2012, at 13:36, "NAPIERALA, MARIA H" <[email-masked]> wrote:
> The question is what problem does EVPN solve?
> 
Pure layer 2 traffic. Yes, it does exist, and needs to be dealt with properly. But just that. 
> In the context of DC, EVPN can only address packets bridged in the same VLAN. If most packets are routed then EVPN, even if all the complexity problems are addressed, doesn't achieve anything for the traffic that is routed. I believe it is the wrong tradeoff to design a solution around EVPN (i.e., around bridging).
> 
Agreed.

Kireeti.

---

From: Kireeti Kompella <[email-masked]>
Date: Fri, 21 Dec 2012 20:25:05 -0800

Luyuan,

> virtual, vpn or non-vpn, call it enhanced/advanced forwarding.

I would say, "will be happening" (today, default is bridging), but, yes, agree.

Kireeti.

---

From: Kireeti Kompella <[email-masked]>
Date: Fri, 21 Dec 2012 20:41:33 -0800

Hi Robert,

> To run RFC4364 within and outside of my DCs.

Perfect!

> other not hard-integrated.

Yes, but route if IP, bridge otherwise. :-)

Kireeti

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Sat, 22 Dec 2012 05:20:23 +0000

OK, will be happening (soon).:-).
Thx,
Luyuan

-----Original Message-----
From: Kireeti Kompella <[email-masked]>
Date: Friday, December 21, 2012 8:25 PM
To: Luyuan Fang <[email-masked]>
Cc: Aldrin Isaac <[email-masked]>, "NAPIERALA,	MARIA H"
<[email-masked]>, Lucy yong <[email-masked]>, "[email-masked]"
<[email-masked]>
Subject: Re: [nvo3] FW: New Version Notification
for	draft-yong-nvo3-frwk-dpreq-addition-00.txt

---

From: Yakov Rekhter <[email-masked]>
Date: Sat, 22 Dec 2012 15:28:03 -0800

Maria,

> This re-invents l3vpn with using l2vpn route advertisements and introduces a 
> possibility of extending the reach of a broadcast domain beyond a single 
> subnet. The question is why to introduce a new paradigm for routing?
> 
> When traffic crosses between CUGs this traffic is IP routed (naturally 
> and optimally addressed by L3VPNs). Within a CUG the traffic is handled 
> the same way whether one implements EVPN or L3VPN (no visible difference). 
> The only question remaining is how to address non-IP traffic (in DCs that 
> carry or care about non-IP traffic). A solution should use EVPN only 
> when necessary to handle non-IP traffic. As Kireeti described it: "route 
> if IP, bridge otherwise".

If we are to consider DC where we have a mix of IP and non-IP traffic,
then *within the DC* there are (at least) the following two options:

1. Service provider deploys both E-VPN and L3VPN; E-VPN is used solely
to implement L2-based CUGs for non-IP traffic, all the IP traffic is
handled by L3VPN 

2. Service provider deploys E-VPN, which handles both IP and non-IP
traffic.

> In addition, the reference above (section 6.1) does not address one
> of the main requirements in a service provider's cloud computing
> environment which is a seamless integration with MPLS/BGP layer 3
> VPNs in the WAN and the wireless acc ess networks (pointed out by
> Robert).

Section 6.2.2 describes integration between E-VPN and 2547 VPNs.

Yakov.

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 23 Dec 2012 00:56:51 +0100

> Section 6.2.2 describes integration between E-VPN and 2547 VPNs.

It says:

   A DCBR, in addition to implementing the E-VPN functionality, also
   implements functionality of a Provider Edge (PE) router, as specified
   in [RFC4364].

Question #1:

Why anyone would prefer to implement both: E-VPN and RFC4364 rather
then only RFC4364 option B on DCBR to solve his application and
customer's needs ?

Rgs,
R.

PS. EVPN spec says:

   The IP Address field is optional. By default, the IP Address Length
   field is set to 0 and the IP address field is omitted from the route.
   When a valid IP address is included, it is encoded as specified in
   section 12.

+

   The IP address field in the MAC advertisement route may optionally
   carry one of the IP addresses associated with the MAC address. This
   provides an option which can be used to minimize the flooding of ARP
   or Neighbor Discovery (ND) messages over the MPLS network and to
   remote CEs.

Question #2:

Why we would not just leave it as this and only use IP carried in EVPN
advertisements for proxy arp function ?

---

From: Lucy yong <[email-masked]>
Date: Wed, 26 Dec 2012 16:33:52 +0000

Kireeti,

It seems that you make EVPN and IPVPN orthogonal now: If IP, use IPVPN, if not, EVPN.

Do you also see that the end system can be distinguished this way?

Using IP VPN for all the IP applications is good in one way, but it requires the substantial changes on all the hosts/hypervisor and require the behavior changes on the VM/physical server. Giving millions VM/servers are there, will this realistic?   Why do we ask all the tenant systems to change behavior in order to use of IPVPN?

IMO, IPVPN is very useful for many applications and it is also  necessary to support multi-tenancy in DC without changing tenant system behavior.

Thanks,
Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Friday, December 21, 2012 10:21 PM
To: NAPIERALA, MARIA H
Cc: Thomas Narten; [email-masked]; Aldrin Isaac
Subject: Re: [nvo3] Multi-subnet VNs [was Re: FW: New Version Notification for draft-yong-nvo3-frwk-dpreq-addition-00.txt]

Hi Maria,

On Dec 20, 2012, at 13:36, "NAPIERALA, MARIA H" <[email-masked]<mailto:[email-masked]>> wrote:

Kireeti.

---

From: "Black, David" <[email-masked]>
Date: Wed, 26 Dec 2012 12:44:49 -0500

Hi Lucy,

I’m going back to the original question of whether we need a new service type, and starting from a couple of comments from you on how the proposed new L2-3 service works ...

> But a frame from TS have both Ethernet and IP header. From TS perspective,
> the frame to the TS on the same subnet will be bridged, the frame to the

> TS send a frame to the destination directly if it is on the same subnet and
> send a frame to the gateway if the destination of the frame is not on the same subnet.

I observe that these describe exactly how the currently-defined L2 service works - bridge the frame to the L2 destination, and route it beyond there if the L2 destination happens to be an L3 router.  Beyond that, the discussion around where the router is located and whether it’s distributed among multiple nodes, e.g., NVEs, is an implementation discussion, not a service definition discussion. To be specific, just spreading such a router (e.g., default gateway) across the NVEs need change the service that is provided; please see VRRP for a worked example of this sort of distribution, and VRRP does not change the service definition in the sense that nvo3 is using the term “service”.

I also think that the provisioning topic is a red herring:

> Without it [new service type], it means that, for creating a tenant virtual network that
> overlays and layer 3 overlay and specify the interfaces to interconnect them, etc.

That really does not require a new service type.  Rather, a single sentence ought to suffice to state that orchestration functionality (e.g., software) may provision multiple virtual networks as part of a single larger provisioning operation.

In contrast, there may be a new service type in the discussion of “route before bridge” because when traffic can be bridged at L2, routing that traffic at L3 may remove L2 adjacency information, and the discussion of use of EVPN for traffic that suffers from that removal then follows.  I would hope that this topic can be deferred in the same fashion that (I hope) non-transitive L2 connectivity (e.g., L2 stations A and B are in VN 1, stations B and C are in VN 2, A can talk to B, B can talk to C, but A can’t talk directly to C at L2) has been deferred.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Lucy yong <[email-masked]>
Date: Wed, 26 Dec 2012 18:52:12 +0000

Hi David,

Glad to see joining the discussion. I modify the subject a bit before we reach the consensus.
Please see inline.

From: Black, David [mailto:[email-masked]]
Sent: Wednesday, December 26, 2012 11:45 AM
To: Lucy yong
Cc: [email-masked]<mailto:[email-masked]>

Subject: Multi-subnet VNs - not a new service type

Hi Lucy,

> send a frame to the gateway if the destination of the frame is not on the same subnet.

I observe that these describe exactly how the currently-defined L2 service works - bridge the frame to the L2 destination, and route it beyond there if the L2 destination happens to be an L3 router.
[Lucy] Yes, you state correct. How does a tenant system to know the MAC of a router? It uses arp protocol and L2  service is transparent to arp protocol. What this lead to that even two TSes in the different subnets attaches the same NVE, the traffic between two has to be routed via an L3 router. If you choose to have a lot of routers and push the router function close to NVE for a tenant network, this problem is not a problem. However, this will make operator nightmare because you have to configure many L2 and L3 services. Second, this also require both L2 and L3 interworking mechanism on TS mobility.
 Beyond that, the discussion around where the router is located and whether it’s distributed among multiple nodes, e.g., NVEs, is an implementation discussion, not a service definition discussion. To be specific, just spreading such a router (e.g., default gateway) across the NVEs need change the service that is provided; please see VRRP for a worked example of this sort of distribution, and VRRP does not change the service definition in the sense that nvo3 is using the term “service”.
[Lucy] Do you say “need change” or “not need change” here? IMO: if using distributed routing, it is very different from designated routers design. Both are very useful for the operator, and it would be nice to distinguish two in overlay.

I also think that the provisioning topic is a red herring:
[Lucy] what do you think about SDN?

That really does not require a new service type.  Rather, a single sentence ought to suffice to state that orchestration functionality (e.g., software) may provision multiple virtual networks as part of a single larger provisioning operation.
[Lucy] optimizing operation is attractive for operators.

In contrast, there may be a new service type in the discussion of “route before bridge” because when traffic can be bridged at L2, routing that traffic at L3 may remove L2 adjacency information, and the discussion of use of EVPN for traffic that suffers from that removal then follows.  I would hope that this topic can be deferred in the same fashion that (I hope) non-transitive L2 connectivity (e.g., L2 stations A and B are in VN 1, stations B and C are in VN 2, A can talk to B, B can talk to C, but A can’t talk directly to C at L2) has been deferred.
[Lucy] this is the hub-spoke. Why do we need to defer this?

Lucy

Thanks,
--David

---

From: "Black, David" <[email-masked]>
Date: Wed, 26 Dec 2012 14:30:50 -0500

Hi Lucy,

More inline.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Wednesday, December 26, 2012 1:52 PM
To: Black, David
Cc: [email-masked]
Subject: Re: [nvo3] Multi-subnet VNs - should be a new service type?

Hi David,

Hi Lucy,

I observe that these describe exactly how the currently-defined L2 service works - bridge the frame to the L2 destination, and route it beyond there if the L2 destination happens to be an L3 router.
[Lucy] Yes, you state correct. How does a tenant system to know the MAC of a router? It uses arp protocol and L2  service is transparent to arp protocol. What this lead to that even two TSes in the different subnets attaches the same NVE, the traffic between two has to be routed via an L3 router.
[David] For clarity - it has to be routed via a *logical* L3 router, which could be part of an NVE.  That logical routing functionality has to be provisioned and configured independent of whether the two subnets are in different service instances or the same service instance in the architecture.

If you choose to have a lot of routers and push the router function close to NVE for a tenant network, this problem is not a problem. However, this will make operator nightmare because you have to configure many L2 and L3 services.
[David] I think you’ve missed the point.   If there are multiple subnets across which traffic has to be forwarded, the routing functionality to forward that traffic has to be configured.

Second, this also require both L2 and L3 interworking mechanism on TS mobility.
[David] What do you mean by “L3 interworking”?

 Beyond that, the discussion around where the router is located and whether it’s distributed among multiple nodes, e.g., NVEs, is an implementation discussion, not a service definition discussion. To be specific, just spreading such a router (e.g., default gateway) across the NVEs need change the service that is provided; please see VRRP for a worked example of this sort of distribution, and VRRP does not change the service definition in the sense that nvo3 is using the term “service”.
[Lucy] Do you say “need change” or “not need change” here? IMO: if using distributed routing, it is very different from designated routers design. Both are very useful for the operator, and it would be nice to distinguish two in overlay.
[David] “need not change” was intended, sorry for the typo.   I agree that solutions will have to spell out the router structure in detail, but I’m at a loss for why the service definition has to care.

I also think that the provisioning topic is a red herring:
[Lucy] what do you think about SDN?
[David] I think it’s orthogonal to this thread of discussion about whether provisioning provides a rationale for defining a new service type in the framework.

That really does not require a new service type.  Rather, a single sentence ought to suffice to state that orchestration functionality (e.g., software) may provision multiple virtual networks as part of a single larger provisioning operation.
[Lucy] optimizing operation is attractive for operators.
[David] and that optimization can be done via provisioning software that iterates over the virtual networks that need to be provisioned.

In contrast, there may be a new service type in the discussion of “route before bridge” because when traffic can be bridged at L2, routing that traffic at L3 may remove L2 adjacency information, and the discussion of use of EVPN for traffic that suffers from that removal then follows.  I would hope that this topic can be deferred in the same fashion that (I hope) non-transitive L2 connectivity (e.g., L2 stations A and B are in VN 1, stations B and C are in VN 2, A can talk to B, B can talk to C, but A can’t talk directly to C at L2) has been deferred.
[Lucy] this is the hub-spoke. Why do we need to defer this?
[David] I’m concerned that it will lead to a long discussion about redefining exactly what an L2 Ethernet service is and how it differs from various IEEE 802 definitions.

Lucy

Thanks,
--David

---

From: Kireeti Kompella <[email-masked]>
Date: Wed, 26 Dec 2012 18:48:59 -0800

Hi Lucy,

On Wed, Dec 26, 2012 at 8:33 AM, Lucy yong <[email-masked]> wrote:

>  Kireeti,****
>
> ** **
>
> It seems that you make EVPN and IPVPN orthogonal now: If IP, use IPVPN, if
> not, EVPN.****
>
> ** **
>
> Do you also see that the end system can be distinguished this way?****
>
> ** **
>
> Using IP VPN for all the IP applications is good in one way, but it
> requires the substantial changes on all the hosts/hypervisor and require
> the behavior changes on the VM/physical server. Giving millions VM/servers
> are there, will this realistic?   Why do we ask all the tenant systems to
> change behavior in order to use of IPVPN?
>

The only change needed is on the NVE.  If this resides in the
host/hypervisor, so be it.  The NVE has to change to implement IP VPN/EVPN.
 The additional change to "route if IP, bridge otherwise" is minor.

No change is needed on the VM/tenant system.

BTW, as an example, IRB (if my MAC, then route else bridge) is completely
transparent to the end host (VM, tenant system).

Kireeti.

IMO, IPVPN is very useful for many applications and it is also  necessary
> to support multi-tenancy in DC without changing tenant system behavior.***
> *
>
> ** **
>
> Thanks,****
>
> Lucy ****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Kireeti Kompella
> *Sent:* Friday, December 21, 2012 10:21 PM
> *To:* NAPIERALA, MARIA H
> *Cc:* Thomas Narten; [email-masked]; Aldrin Isaac
> *Subject:* Re: [nvo3] Multi-subnet VNs [was Re: FW: New Version
> Notification for draft-yong-nvo3-frwk-dpreq-addition-00.txt]****
>
> ** **
>
> Hi Maria,****
>
> ** **
>
> On Dec 20, 2012, at 13:36, "NAPIERALA, MARIA H" <[email-masked]> wrote:***
> *
>
> The question is what problem does EVPN solve? ****
>
>  Pure layer 2 traffic. Yes, it does exist, and needs to be dealt with
> properly. But just that. ****
>
> In the context of DC, EVPN can only address packets bridged in the same
> VLAN. If most packets are routed then EVPN, even if all the complexity
> problems are addressed, doesn't achieve anything for the traffic that is
> routed. I believe it is the wrong tradeoff to design a solution around EVPN
> (i.e., around bridging).****
>
>  Agreed.****
>
> ** **
>
> Kireeti. ****
>

-- 
Kireeti

---

From: Truman Boyes <[email-masked]>
Date: Thu, 27 Dec 2012 01:42:58 -0500

Hi, comments inline:

> Do you also see that the end system can be distinguished this way?

A NVE could support both services, IPVPN and EVPN. The guest/tenant should have no need to understand the method of communication chosen by the hypervisor. 

> Using IP VPN for all the IP applications is good in one way, but it requires the substantial changes on all the hosts/hypervisor and require the behavior changes on the VM/physical server. Giving m…

The changes necessary to support IPVPN or EVPN are located on the NVE. Likely this would be a kernel loadable module to the bridging an routing stack. I think the changes proposed are highly realistic given the current ecosystem of SDN/Cloud companies that are doing all sorts of interesting things in the networking stack of hypervisors. 

Best, 
Truman

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Dec 2012 16:55:28 +0000

David,

Please see in-line with [lucy1]

Hi Lucy,

I observe that these describe exactly how the currently-defined L2 service works - bridge the frame to the L2 destination, and route it beyond there if the L2 destination happens to be an L3 router.
[Lucy] Yes, you state correct. How does a tenant system to know the MAC of a router? It uses arp protocol and L2  service is transparent to arp protocol. What this lead to that even two TSes in the different subnets attaches the same NVE, the traffic between two has to be routed via an L3 router.
[David] For clarity - it has to be routed via a *logical* L3 router, which could be part of an NVE.  That logical routing functionality has to be provisioned and configured independent of whether the two subnets are in different service instances or the same service instance in the architecture.
[Lucy1]OK, I understand. For this service capability, do you see it as a L2 service or L3 service? IMO: neither of them fits.

If you choose to have a lot of routers and push the router function close to NVE for a tenant network, this problem is not a problem. However, this will make operator nightmare because you have to configure many L2 and L3 services.
[David] I think you’ve missed the point.   If there are multiple subnets across which traffic has to be forwarded, the routing functionality to forward that traffic has to be configured.
[Lucy1] If this is the case, you may want to choose few router locations to avoid massive configuration. This is the case where routing function is outside hypervisor and on some routers.

Second, this also require both L2 and L3 interworking mechanism on TS mobility.
[David] What do you mean by “L3 interworking”?
[Lucy1] If your case is that the L2/L3 are paired and co-exist on the same NVE, this does not apply to your case. do you think it would be more clear to express this case with a new NVE service type?

 Beyond that, the discussion around where the router is located and whether it’s distributed among multiple nodes, e.g., NVEs, is an implementation discussion, not a service definition discussion. To be specific, just spreading such a router (e.g., default gateway) across the NVEs need change the service that is provided; please see VRRP for a worked example of this sort of distribution, and VRRP does not change the service definition in the sense that nvo3 is using the term “service”.
[Lucy] Do you say “need change” or “not need change” here? IMO: if using distributed routing, it is very different from designated routers design. Both are very useful for the operator, and it would be nice to distinguish two in overlay.
[David] “need not change” was intended, sorry for the typo.   I agree that solutions will have to spell out the router structure in detail, but I’m at a loss for why the service definition has to care.
[Lucy1] IMO: we are working on overlay networking. We have either L2 overlay network or L3 overlay network. But sometime a tenant network needs both L2 and L3 functions to accomplish the needs, where neither L2 overlay nor L3 overlay fit it well.  We can let operators to combine two overlay networks to accomplish the task. But this is not help in positioning these functions in standard development and causing some confusion here and there. Furthermore, IMO: it brings a complexity for operators.

I also think that the provisioning topic is a red herring:
[Lucy] what do you think about SDN?
[David] I think it’s orthogonal to this thread of discussion about whether provisioning provides a rationale for defining a new service type in the framework.
[Lucy1] This is not what I mean. I agree that SDN is not the reason for the new service. Let’s stop here.

That really does not require a new service type.  Rather, a single sentence ought to suffice to state that orchestration functionality (e.g., software) may provision multiple virtual networks as part of a single larger provisioning operation.
[Lucy] optimizing operation is attractive for operators.
[David] and that optimization can be done via provisioning software that iterates over the virtual networks that need to be provisioned.
[Lucy1] you mean planning tool. I do not think this will work well for the dynamics in DC, i.e. TS mobility in real time.

In contrast, there may be a new service type in the discussion of “route before bridge” because when traffic can be bridged at L2, routing that traffic at L3 may remove L2 adjacency information, and the discussion of use of EVPN for traffic that suffers from that removal then follows.  I would hope that this topic can be deferred in the same fashion that (I hope) non-transitive L2 connectivity (e.g., L2 stations A and B are in VN 1, stations B and C are in VN 2, A can talk to B, B can talk to C, but A can’t talk directly to C at L2) has been deferred.
[Lucy] this is the hub-spoke. Why do we need to defer this?
[David] I’m concerned that it will lead to a long discussion about redefining exactly what an L2 Ethernet service is and how it differs from various IEEE 802 definitions.
[Lucy1]  Thank you to bring this point. If we have a new service type, that will be different from L2 Ethernet service. We can keep the L2 overlay service is exactly same as the L2 Ethernet service defined in IEEE. If not, for the scenarios you picture here, it does not belong to the L2 Ethernet service nor IETF defined L3VPN service. You either change these services or distinguish it from these two.

Thanks,
Lucy

Thanks,
--David

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Dec 2012 17:15:10 +0000

Hi Truman,

If NVE provides both IPVPN and EVPN to a TS, should we define a new service type for NVE?

Lucy

From: Truman Boyes [mailto:[email-masked]]
Sent: Thursday, December 27, 2012 12:43 AM
To: Lucy yong
Cc: Kireeti Kompella; NAPIERALA, MARIA H; Thomas Narten; [email-masked]; Aldrin Isaac
Subject: Re: [nvo3] Multi-subnet VNs [was Re: FW: New Version Notification for draft-yong-nvo3-frwk-dpreq-addition-00.txt]

Hi, comments inline:

Kireeti,

Best,
Truman

Thanks,
Lucy

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Kireeti Kompella
Sent: Friday, December 21, 2012 10:21 PM
To: NAPIERALA, MARIA H
Cc: Thomas Narten; [email-masked]<mailto:[email-masked]>; Aldrin Isaac
Subject: Re: [nvo3] Multi-subnet VNs [was Re: FW: New Version Notification for draft-yong-nvo3-frwk-dpreq-addition-00.txt]

Hi Maria,

Kireeti.

---

From: "Black, David" <[email-masked]>
Date: Thu, 27 Dec 2012 12:42:17 -0500

Lucy,

The core of our disagreement is here:

[Lucy] We have either L2 overlay network or L3 overlay network. But sometime a tenant network needs both L2 and L3 functions to accomplish the needs, where neither L2 overlay nor L3 overlay fit it well.

I strongly disagree with the statement that an L2 overlay is a poor fit for carrying tenant IP traffic that needs to cross L2 (virtual) network boundaries.  That sort of IP traffic that needs to cross L2 network boundaries works fine on VLANs and it works fine on L2 overlays.

I would agree that there are ways to implement routing support for traffic that exits an L2 virtual network that are poor fits to some situations (e.g., lengthy discussions here and elsewhere on trombone and triangle routing), but I do not see the need for a new hybrid L2-3 service definition to enable better implementations of routing integration with L2 networking.

And my answer to this question:

[Lucy1] For this service capability, do you see it as a L2 service or L3 service? IMO: neither of them fits.

is that it’s still an L2 service, in part because it makes no sense to me to restrict an L2 service (at least for nvo3) by forbidding the attachment of IP routing functionality to L2 (virtual) networks.

Thanks,
--David

From: Lucy yong [mailto:[email-masked]]
Sent: Thursday, December 27, 2012 11:55 AM
To: Black, David
Cc: [email-masked]
Subject: RE: Multi-subnet VNs - should be a new service type?

David,

Hi Lucy,

Thanks,
Lucy

Thanks,
--David

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Dec 2012 18:10:39 +0000

David,

From: Black, David [mailto:[email-masked]]
Sent: Thursday, December 27, 2012 11:42 AM
To: Lucy yong
Cc: [email-masked]
Subject: RE: Multi-subnet VNs - should be a new service type?

Lucy,

I strongly disagree with the statement that an L2 overlay is a poor fit for carrying tenant IP traffic that needs to cross L2 (virtual) network boundaries.  That sort of IP traffic that needs to cross L2 network boundaries works fine on VLANs and it works fine on L2 overlays.
[Lucy] I did not say it is poor fit, and just express if you integrate L2 function and L3 function on the same NVE, it is better to give a new NVE service type. Obviously, we have disagreement here.

is that it’s still an L2 service, in part because it makes no sense to me to restrict an L2 service (at least for nvo3) by forbidding the attachment of IP routing functionality to L2 (virtual) networks.
[Lucy] No, such restriction is not good. L2 switches have been used to bridge IP traffic between router and hosts for long time. We are on the same page here.

Thanks,
Lucy

Thanks,
--David

David,

Hi Lucy,

Thanks,
Lucy

Thanks,
--David

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 27 Dec 2012 18:58:01 +0000

The Default Gateway feature of EVPN helps to avoid tromboning.
http://tools.ietf.org/html/draft-ietf-l2vpn-evpn-02, Sec. 11.1.
Every PE responds to the ARP request for the default router
and every PE will route packets with the destination MAC of the
default router.

--
Jakob Heitz. x25475. 510-566-2901

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Thursday, December 27, 2012 10:11 AM
To: Black, David
Cc: [email-masked]
Subject: Re: [nvo3] Multi-subnet VNs - should be a new service type?

David,

Lucy,

is that it's still an L2 service, in part because it makes no sense to me to restrict an L2 service (at least for nvo3) by forbidding the attachment of IP routing functionality to L2 (virtual) networks.
[Lucy] No, such restriction is not good. L2 switches have been used to bridge IP traffic between router and hosts for long time. We are on the same page here.

Thanks,
Lucy

Thanks,
--David

David,

Hi Lucy,

I'm going back to the original question of whether we need a new service type, and starting from a couple of comments from you on how the proposed new L2-3 service works ...

If you choose to have a lot of routers and push the router function close to NVE for a tenant network, this problem is not a problem. However, this will make operator nightmare because you have to configure many L2 and L3 services.
[David] I think you've missed the point.   If there are multiple subnets across which traffic has to be forwarded, the routing functionality to forward that traffic has to be configured.
[Lucy1] If this is the case, you may want to choose few router locations to avoid massive configuration. This is the case where routing function is outside hypervisor and on some routers.

Second, this also require both L2 and L3 interworking mechanism on TS mobility.
[David] What do you mean by "L3 interworking"?
[Lucy1] If your case is that the L2/L3 are paired and co-exist on the same NVE, this does not apply to your case. do you think it would be more clear to express this case with a new NVE service type?

 Beyond that, the discussion around where the router is located and whether it's distributed among multiple nodes, e.g., NVEs, is an implementation discussion, not a service definition discussion. To be specific, just spreading such a router (e.g., default gateway) across the NVEs need change the service that is provided; please see VRRP for a worked example of this sort of distribution, and VRRP does not change the service definition in the sense that nvo3 is using the term "service".
[Lucy] Do you say "need change" or "not need change" here? IMO: if using distributed routing, it is very different from designated routers design. Both are very useful for the operator, and it would be nice to distinguish two in overlay.
[David] "need not change" was intended, sorry for the typo.   I agree that solutions will have to spell out the router structure in detail, but I'm at a loss for why the service definition has to care.
[Lucy1] IMO: we are working on overlay networking. We have either L2 overlay network or L3 overlay network. But sometime a tenant network needs both L2 and L3 functions to accomplish the needs, where neither L2 overlay nor L3 overlay fit it well.  We can let operators to combine two overlay networks to accomplish the task. But this is not help in positioning these functions in standard development and causing some confusion here and there. Furthermore, IMO: it brings a complexity for operators.

I also think that the provisioning topic is a red herring:
[Lucy] what do you think about SDN?
[David] I think it's orthogonal to this thread of discussion about whether provisioning provides a rationale for defining a new service type in the framework.
[Lucy1] This is not what I mean. I agree that SDN is not the reason for the new service. Let's stop here.

In contrast, there may be a new service type in the discussion of "route before bridge" because when traffic can be bridged at L2, routing that traffic at L3 may remove L2 adjacency information, and the discussion of use of EVPN for traffic that suffers from that removal then follows.  I would hope that this topic can be deferred in the same fashion that (I hope) non-transitive L2 connectivity (e.g., L2 stations A and B are in VN 1, stations B and C are in VN 2, A can talk to B, B can talk to C, but A can't talk directly to C at L2) has been deferred.
[Lucy] this is the hub-spoke. Why do we need to defer this?
[David] I'm concerned that it will lead to a long discussion about redefining exactly what an L2 Ethernet service is and how it differs from various IEEE 802 definitions.
[Lucy1]  Thank you to bring this point. If we have a new service type, that will be different from L2 Ethernet service. We can keep the L2 overlay service is exactly same as the L2 Ethernet service defined in IEEE. If not, for the scenarios you picture here, it does not belong to the L2 Ethernet service nor IETF defined L3VPN service. You either change these services or distinguish it from these two.

Thanks,
Lucy

Thanks,
--David

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Dec 2012 19:06:08 +0000

This will have an issue if VAP of EVPN are via a switched network. Need to modify the ARP response schema.

Lucy

From: Jakob Heitz [mailto:[email-masked]]
Sent: Thursday, December 27, 2012 12:58 PM
To: Lucy yong; Black, David
Cc: [email-masked]
Subject: RE: Multi-subnet VNs - should be a new service type?

Lucy,

Thanks,
Lucy

Thanks,
--David

David,

Hi Lucy,

Thanks,
Lucy

Thanks,
--David

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 27 Dec 2012 16:08:18 -0500

Lucy, I wonder if I am misunderstanding your point.  (Or maybe I am 
missing something basic about NVO3.)

If I take a host, and plug it into an Ethernet switch, te "service" it 
gets is a layer 2 interconnect service.  That is still the case even if 
there is also a router attached to that Ethernet switch.  And it remains 
the case even if the router happens to reside in the same device, and 
use the same physical ports, as the Ethernet switch.  (The router logic 
provides an L3 service.  The Ethernet switch provides an L2 service. 
Where the router resides is network and implementation dependent.)
I don't understand why NVO3 would change this logic.

I do not think that anyone who has discussed L2 service for datacenters 
(inside or outside NVO3) has ever asserted that an L2 service can not be 
interconnected at L3.

Yours,
Joel

On 12/27/2012 1:10 PM, Lucy yong wrote:
> David,
>
> *From:*Black, David [mailto:[email-masked]]
> *Sent:* Thursday, December 27, 2012 11:42 AM
> *To:* Lucy yong
> *Cc:* [email-masked]
> *Subject:* RE: Multi-subnet VNs - should be a new service type?
>
> Lucy,
>
> */[Lucy] We have either L2 overlay network or L3 overlay network. But
> sometime a tenant network needs both L2 and L3 functions to accomplish
> the needs, where neither L2 overlay nor L3 overlay fit it well. /*
>
> I strongly disagree with the statement that an L2 overlay is a poor fit
> for carrying tenant IP traffic that needs to cross L2 (virtual) network
> boundaries.  That sort of IP traffic that needs to cross L2 network
> boundaries works fine on VLANs and it works fine on L2 overlays.
>
> */[Lucy] I did not say it is poor fit, and just express if you integrate
> L2 function and L3 function on the same NVE, it is better to give a new
> NVE service type. Obviously, we have disagreement here. /*
>
> I would agree that there are ways to implement routing support for
> traffic that exits an L2 virtual network that are poor fits to some
> situations (e.g., lengthy discussions here and elsewhere on trombone and
> triangle routing), but I do not see the need for a new hybrid L2-3
> service definition to enable better implementations of routing
> integration with L2 networking.
>
> */[Lucy1] For this service capability, do you see it as a L2 service or
> L3 service? IMO: neither of them fits./*
>
> is that it’s still an L2 service, in part because it makes no sense to
> me to restrict an L2 service (at least for nvo3) by forbidding the
> attachment of IP routing functionality to L2 (virtual) networks.
>
> */[Lucy] No, such restriction is not good. L2 switches have been used to
> bridge IP traffic between router and hosts for long time. We are on the
> same page here./*
>
> *//*
>
> */Thanks,/*
>
> */Lucy/*
>
> Thanks,
> --David
>
> *From:*Lucy yong [mailto:[email-masked]]
> *Sent:* Thursday, December 27, 2012 11:55 AM
> *To:* Black, David
> *Cc:* [email-masked]
> *Subject:* RE: Multi-subnet VNs - should be a new service type?
>
> David,
>
> I’m going back to the original question of whether we need a new service
> type, and starting from a couple of comments from you on how the
> proposed new L2-3 service works ...
>
>> TS on the different subnet goes to the router. We need to a service to
>
>> guarantee that.
>
> I observe that these describe exactly how the currently-defined L2
> service works - bridge the frame to the L2 destination, and route it
> beyond there if the L2 destination happens to be an L3 router.
>
> */[Lucy] Yes, you state correct. How does a tenant system to know the
> MAC of a router? It uses arp protocol and L2  service is transparent to
> arp protocol. What this lead to that even two TSes in the different
> subnets attaches the same NVE, the traffic between two has to be routed
> via an L3 router./**//*
>
> [David] For clarity - it has to be routed via a **logical** L3 router,
> which could be part of an NVE.  That logical routing functionality has
> to be provisioned and configured independent of whether the two subnets
> are in different service instances or the same service instance in the
> architecture.
>
> */[Lucy1]OK, I understand. For this service capability, do you see it as
> a L2 service or L3 service? IMO: neither of them fits./*
>
> */If you choose to have a lot of routers and push the router function
> close to NVE for a tenant network, this problem is not a problem.
> However, this will make operator nightmare because you have to configure
> many L2 and L3 services. /**//*
>
> [David] I think you’ve missed the point.   If there are multiple subnets
> across which traffic has to be forwarded, the routing functionality to
> forward that traffic has to be configured.
>
> */[Lucy1] If this is the case, you may want to choose few router
> locations to avoid massive configuration. This is the case where routing
> function is outside hypervisor and on some routers. /*
>
> */Second, this also require both L2 and L3 interworking mechanism on TS
> mobility./*
>
> */[Lucy1] If your case is that the L2/L3 are paired and co-exist on the
> same NVE, this does not apply to your case. do you think it would be
> more clear to express this case with a new NVE service type? /*
>
>   Beyond that, the discussion around where the router is located and
> whether it’s distributed among multiple nodes, e.g., NVEs, is an
> implementation discussion, not a service definition discussion. To be
> specific, just spreading such a router (e.g., default gateway) across
> the NVEs need change the service that is provided; please see VRRP for a
> worked example of this sort of distribution, and VRRP does not change
> the service definition in the sense that nvo3 is using the term “service”.
>
> */[Lucy] Do you say “need change” or “not need change” here? IMO: if
> using distributed routing, it is very different from designated routers
> design. Both are very useful for the operator, and it would be nice to
> distinguish two in overlay./*
>
> [David] “need not change” was intended, sorry for the typo.   I agree
> that solutions will have to spell out the router structure in detail,
> but I’m at a loss for why the service definition has to care.
>
> */[Lucy1] IMO: we are working on overlay networking. We have either L2
> overlay network or L3 overlay network. But sometime a tenant network
> needs both L2 and L3 functions to accomplish the needs, where neither L2
> overlay nor L3 overlay fit it well.  We can let operators to combine two
> overlay networks to accomplish the task. But this is not help in
> positioning these functions in standard development and causing some
> confusion here and there. Furthermore, IMO: it brings a complexity for
> operators. /*
>
> */[Lucy] what do you think about SDN?/*
>
> [David] I think it’s orthogonal to this thread of discussion about
> whether provisioning provides a rationale for defining a new service
> type in the framework.
>
> */[Lucy1] This is not what I mean. I agree that SDN is not the reason
> for the new service. Let’s stop here./*
>
> That really does not require a new service type.  Rather, a single
> sentence ought to suffice to state that orchestration functionality
> (e.g., software) may provision multiple virtual networks as part of a
> single larger provisioning operation.
>
> */[Lucy] optimizing operation is attractive for operators. /*
>
> [David] and that optimization can be done via provisioning software that
> iterates over the virtual networks that need to be provisioned.
>
> */[Lucy1] you mean planning tool. I do not think this will work well for
> the dynamics in DC, i.e. TS mobility in real time./*
>
> In contrast, there may be a new service type in the discussion of “route
> before bridge” because when traffic can be bridged at L2, routing that
> traffic at L3 may remove L2 adjacency information, and the discussion of
> use of EVPN for traffic that suffers from that removal then follows.  I
> would hope that this topic can be deferred in the same fashion that (I
> hope) non-transitive L2 connectivity (e.g., L2 stations A and B are in
> VN 1, stations B and C are in VN 2, A can talk to B, B can talk to C,
> but A can’t talk directly to C at L2) has been deferred.
>
> */[Lucy] this is the hub-spoke. Why do we need to defer this? /*
>
> [David] I’m concerned that it will lead to a long discussion about
> redefining exactly what an L2 Ethernet service is and how it differs
> from various IEEE 802 definitions.
>
> */[Lucy1]  Thank you to bring this point. If we have a new service type,
> that will be different from L2 Ethernet service. We can keep the L2
> overlay service is exactly same as the L2 Ethernet service defined in
> IEEE. If not, for the scenarios you picture here, it does not belong to
> the L2 Ethernet service nor IETF defined L3VPN service. You either
> change these services or distinguish it from these two./*
>
> */Thanks,/*
>
> */Lucy/*
>
> Thanks,
> --David

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 27 Dec 2012 21:25:16 +0000

What is the issue?

Are you referring to the case where a network
is connected to multiple EVPN PEs in a multihoming
arrangement?

If that is the case, then I think the EVPN draft is
not clear. It should point out that an EVI acting as
standby in an active-standby topology can not act
as Default Gateway.

--
Jakob Heitz.

________________________________
From: Lucy yong [mailto:[email-masked]]
Sent: Thursday, December 27, 2012 11:06 AM
To: Jakob Heitz; Black, David
Cc: [email-masked]
Subject: RE: Multi-subnet VNs - should be a new service type?

Lucy

Lucy,

Thanks,
Lucy

Thanks,
--David

David,

Hi Lucy,

Thanks,
Lucy

Thanks,
--David

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Thu, 27 Dec 2012 21:51:21 +0000

Joe,

I share the same view, or confusion - what new service type is discussed
here?
Typically, service is determined at the service inter-connect point, - if
L2, it's L2; if L3, it's L3.
However you transport the packets beyond that point within the networks
does not change the service type.
I don't see a new service type in nvo3.

Thx,
Luyuan

-----Original Message-----
From: "Joel M. Halpern" <[email-masked]>
Date: Thursday, December 27, 2012 1:08 PM
To: Lucy yong <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [nvo3] Multi-subnet VNs - should be a new service type?

>> is that it¹s still an L2 service, in part because it makes no sense to
>> I¹m going back to the original question of whether we need a new service
>> [David] I think you¹ve missed the point.   If there are multiple subnets
>> [David] What do you mean by ³L3 interworking²?
>>
>> whether it¹s distributed among multiple nodes, e.g., NVEs, is an
>> the service definition in the sense that nvo3 is using the term
>>³service².
>>
>> */[Lucy] Do you say ³need change² or ³not need change² here? IMO: if
>> [David] ³need not change² was intended, sorry for the typo.   I agree
>> but I¹m at a loss for why the service definition has to care.
>>
>> [David] I think it¹s orthogonal to this thread of discussion about
>> for the new service. Let¹s stop here./*
>>
>> In contrast, there may be a new service type in the discussion of ³route
>> before bridge² because when traffic can be bridged at L2, routing that
>> but A can¹t talk directly to C at L2) has been deferred.
>>
>> [David] I¹m concerned that it will lead to a long discussion about

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Dec 2012 22:51:34 +0000

Hi Joe,

My original point is that NVO3 aims on the capability to provider multi tenant virtual networks on a common infrastructure, where a tenant virtual network may contain one or more subnets. To support a multi-subnet tenant virtual network, one way to do is to integrate both bridging and routing on every NVE that has the membership of the network virtual instance. Somehow, I don't see either L2 NVE service type or L3 NVE service type fit into this, so suggest a new type. 

What you describe is correct. If you plug a host into an Ethernet switch, the host connects a network via a layer 2 service and the host can reach a different subnet via a L3 service. In this case, the layer 2 service has local significant as if Ethernet switch. To construct a multi-subnet tenant virtual network, operator has to construct many layer 2 service instances and L3 service instance for the tenant. When both L2/L3 service functions resides on a NVE, if we just create a L2/3 virtual network instance with multi-subnets for a tenant, will it be much simpler for an operator?

Regards,
Lucy  

> From: Joel M. Halpern [mailto:[email-masked]]
> Sent: Thursday, December 27, 2012 3:08 PM
> To: Lucy yong
> Cc: [email-masked]
> > is that it's still an L2 service, in part because it makes no sense
> to
> > I'm going back to the original question of whether we need a new
> service
> > [David] I think you've missed the point.   If there are multiple
> subnets
> > whether it's distributed among multiple nodes, e.g., NVEs, is an
> "service".
> > */[Lucy] Do you say "need change" or "not need change" here? IMO: if
> > [David] "need not change" was intended, sorry for the typo.   I agree
> > but I'm at a loss for why the service definition has to care.
> >
> > [David] I think it's orthogonal to this thread of discussion about
> > for the new service. Let's stop here./*
> >
> > In contrast, there may be a new service type in the discussion of
> "route
> > before bridge" because when traffic can be bridged at L2, routing
> that
> > but A can't talk directly to C at L2) has been deferred.
> >
> > [David] I'm concerned that it will lead to a long discussion about

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Dec 2012 23:03:33 +0000

From: Jakob Heitz [mailto:[email-masked]]
Sent: Thursday, December 27, 2012 3:25 PM
To: Lucy yong; Black, David
Cc: [email-masked]
Subject: RE: Multi-subnet VNs - should be a new service type?

What is the issue?

Are you referring to the case where a network
is connected to multiple EVPN PEs in a multihoming
arrangement?
[Lucy] no, for multihoming case, there is designated forwarder to forward multicast traffic.

--
Jakob Heitz.

Lucy

Lucy,

Thanks,
Lucy

Thanks,
--David

David,

Hi Lucy,

Thanks,
Lucy

Thanks,
--David

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 27 Dec 2012 23:17:30 +0000

Multicast traffic is not at issue.
You said:
This will have an issue if VAP of EVPN are via a switched network
What is the issue?

Sent: Thursday, December 27, 2012 3:04 PM

What is the issue?

--
Jakob Heitz.

Lucy

Lucy,

Thanks,
Lucy

Thanks,
--David

David,

Hi Lucy,

Thanks,
Lucy

Thanks,
--David

---

From: Lucy yong <[email-masked]>
Date: Thu, 27 Dec 2012 23:19:54 +0000

A host may get an arp request via the switched network and reply with its MAC address if IP address matches.
This is current host be behavior.
Lucy

From: Jakob Heitz [mailto:[email-masked]]
Sent: Thursday, December 27, 2012 5:18 PM
To: Lucy yong; Black, David
Cc: [email-masked]
Subject: RE: Multi-subnet VNs - should be a new service type?

What is the issue?

--
Jakob Heitz.

Lucy

Lucy,

Thanks,
Lucy

Thanks,
--David

David,

Hi Lucy,

Thanks,
Lucy

Thanks,
--David

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 27 Dec 2012 23:24:10 +0000

Certainly. But how it that an issue?

--
Jakob Heitz.

Sent: Thursday, December 27, 2012 3:20 PM

--
Jakob Heitz

What is the issue?

--
Jakob Heitz.

Lucy

Lucy,

Thanks,
Lucy

Thanks,
--David

David,

Hi Lucy,

Thanks,
Lucy

Thanks,
--David