Subject: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN traffic over IP PSN using MPLS-in-UDP encapsulation// fwd: New Version Notification for draft-xu-mpls-in-udp-00.txt

From: Xuxiaohu <[email-masked]>
Date: Tue, 8 May 2012 03:20:36 +0000

Hi all,

This topic may be interesting to L2VPN, L3VPN and even NVo3 folks. 

Any comments are welcome.

Best regards,
Xiaohu/ Marshall/Lucy

> -----邮件原件-----
> 发件人: Xuxiaohu
> 发送时间: 2012年4月28日 10:55
> 收件人: [email-masked]'
> 抄送: [email-masked]'; Lucy yong
> 主题: fwd: New Version Notification for draft-xu-mpls-in-udp-00.txt
> 
> Hi all,
> 
> Equal Cost Multi-Path (ECMP) and Link Aggregation Group (LAG) are widely used
> in the core of IP-enabled Packet Switch Networks (PSN) for load-balancing
> purposes. Most core routers (i.e., P routers) in the IP-enabled PSN are capable of
> load-balancing IP traffic flows across ECMP paths and/or LAG based on the hash
> of the five-tuple of   UDP/TCP packets (i.e., source IP address, destination IP
> address, source port, destination port, and protocol) or some fields in the IP
> header of non-UDP/TCP packets (e.g., source IP address, destination IP address).
> 
> However, with existing IP-based encapsulation methods as defined in [RFC4023]
> such as MPLS-in-IP and MPLS-in-GRE, distinct customer traffic flows of various
> MPLS applications (e.g., MPLS-based L2VPN or L3VPN) between a given PE pair
> would be encapsulated with the same IP or GRE tunnel header prior to traversing
> the core. Since the encapsulating traffic is neither TCP nor UDP traffic, core
> routers could only perform hash calculation on the fields in the IP header of IP or
> GRE tunnels. As a result, core routers could not achieve an effective
> load-balancing for these traffic flows in the network due to the lack of adequate
> entropy information.
> 
> This document specifies one additional IP-based encapsulation technology for
> MPLS packets referred to as MPLS-in-UDP, which is intended to facilitate
> load-balancing the traffic of various MPLS applications such as MPLS-based L2VPN
> and L3VPN in the core of IP-enabled PSN.
> 
> Any comments are welcome.
> 
> Best regards,
> Xiaohu
> 
> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]]
> 发送时间: 2012年4月28日 10:18
> 收件人: Xuxiaohu
> 抄送: Lucy yong; [email-masked]
> 主题: New Version Notification for draft-xu-mpls-in-udp-00.txt
> 
> A new version of I-D, draft-xu-mpls-in-udp-00.txt has been successfully
> submitted by Xiaohu Xu and posted to the IETF repository.
> 
> Filename:	 draft-xu-mpls-in-udp
> Revision:	 00
> Title:		 Encapsulating MPLS in UDP
> Creation date:	 2012-04-28
> WG ID:		 Individual Submission
> Number of pages: 7
> 
> Abstract:
>    This document specifies one additional IP-based encapsulation
>    technology for MPLS packets referred to as MPLS-in-UDP, which is
>    intended to facilitate load-balancing the traffic of various MPLS
>    applications such as MPLS-based L2VPN and L3VPN in the core of IP-
>    enabled packet switch networks.
> 
> 
> 
> 
> 
> The IETF Secretariat

---

From: Keshav.A.K. <[email-masked]>
Date: Wed, 9 May 2012 10:15:28 +0530

Hi,

Yes,
Major drawback of  GRE kind of enacp, switch need to parse deeply into GRE packets for applying policies related to load distribution Port Channels that  are used to aggregate the bandwidth of multiple physical links into one logical link.  

If switches were to try to distribute GRE flows between two VTEPs that used a GRE encapsulation, all the traffic would be directed to use only one link within these Port Channels. However Parsing all the way to UDP (ex VxLan) source and destination port numbers, (may be configure the switches to use 5-tuple) can spread each UDP flow out to a different link of a Port Channel or ECMP route. 

Regards,
keshava

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Tuesday, May 08, 2012 8:51 AM
To: [email-masked]; [email-masked]; [email-masked]
Subject: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN traffic over IP PSN using MPLS-in-UDP encapsulation// fwd: New Version Notification for draft-xu-mpls-in-udp-00.txt

Hi all,

Best regards,
Xiaohu/ Marshall/Lucy

>    enabled packet switch networks.
_______________________________________________
nvo3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 09 May 2012 09:26:40 +0200

> If switches were to try to distribute GRE flows between two VTEPs
> that used a GRE encapsulation, all the traffic would be directed to
> use only one link within these Port Channels.

Not true. GRE encapsulation is not mandated to use the same src/dst 
addresses for all flows between given two end points.

You do not need to parse deep into packet to enable good load balancing 
hash across parallel links when you use GRE as an encapsulation technology.

And what is nice about IP encapsulation all GRE src/dst addresses can be 
naturally aggregated so from IGP point of view they still look like a 
single prefix.

Regards,
R.

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 09 May 2012 10:53:30 +0100

On 09/05/2012 08:26, Robert Raszuk wrote:
> You do not need to parse deep into packet to enable good load 
> balancing hash across parallel links when you use GRE as an 
> encapsulation technology.
>
> And what is nice about IP encapsulation all GRE src/dst addresses can 
> be naturally aggregated so from IGP point of view they still look like 
> a single prefix.
>
> Regards,
> R. 

Robert

I think that this needs some serious study rather than assertion, 
because it is not at all obvious what the LB properties of either method 
will be in a DC.

In the case of the use of multiple GRE addresses there are operational 
and route scaling issues that also need to be considered.

Stewart

---

From: "Ivan Pepelnjak" <[email-masked]>
Date: Wed, 9 May 2012 12:00:08 +0200

> addresses for all flows between given two end points.

True, but then you need a mechanism to discover all the available endpoints, whereas with single VTEP per host you can use existing mechanisms (either learning from source IP address or next-hop distribution with BGP-like protocols).

> hash across parallel links when you use GRE as an encapsulation

True, but many existing DC switches that do IP-based load balancing have no problems load-balancing on UDP 5-tuple. I am positive there's equipment out there that can do load balancing on GRE keys (or some other part of GRE header), I just haven't encountered it yet (or realized it does that), so please fix my ignorance.

> naturally aggregated so from IGP point of view they still look like a

... and you need a new endpoint prefix (or set-of-addresses) discovery mechanism.

Ivan

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 09 May 2012 13:01:24 +0200

Hi Ivan,

> True, but then you need a mechanism to discover all the available
> endpoints, whereas with single VTEP per host you can use existing
> mechanisms (either learning from source IP address or next-hop
> distribution with BGP-like protocols).

True.

I see few options:

- Signal different next hops in BGP for the application which runs across

- Treat next hop in BGP as prefix indicator not as host route. In
particular next hop resolves in IGP to a prefix which in turn could be
used as set of GRE destination addresses

- Signal such range explicitly. I think there will be a draft coming up
on just that soon.

> True, but many existing DC switches that do IP-based load balancing
> have no problems load-balancing on UDP 5-tuple. I am positive
> there's equipment out there that can do load balancing on GRE keys
> (or some other part of GRE header), I just haven't encountered it yet
> (or realized it does that), so please fix my ignorance.

Indeed. In that case we have nothing to worry about. I think this thread
was just about the case where the above does not apply.

In fact if we all agree that devices can build hash looking deeper and
that those which can not are just a corner case that would be great.
Problem solved :)

>> And what is nice about IP encapsulation all GRE src/dst addresses
>> can be naturally aggregated so from IGP point of view they still
>> look like a single prefix.
>

I don't need nothing on top on what I need today. I need to advertise
reachability to end point prefix anyway. The difference is that I
advertise for example /28 rather then /32.

Hi Stewart,

> because it is not at all obvious what the LB properties of either
> method will be in a DC.

The claim was much more general and my reply did not actually narrowed 
it to DC env. I was more talking about WAN.

> In the case of the use of multiple GRE addresses there are
> operational and route scaling issues that also need to be considered.

I do not see any.

Smart networking OS of ingress and egress point can do such 
encapsulation without any operational overhead. Please think in terms of 
mGRE.

Route scaling issues do not apply. You do not need to inject more then 
one prefix in any case.

Regards,
R.

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 9 May 2012 15:01:39 +0000

> True, but many existing DC switches that do IP-based load balancing have
> no problems load-balancing on UDP 5-tuple. I am positive there's equipment > out there that can do load balancing on GRE keys (or some other part of 
> GRE header), I just haven't encountered it yet (or realized it does that),
> so please fix my ignorance.
..
> Ivan

Hey Ivan,

Looking at the specs of one of the most popular chips there are generic CAM rules that allow you to override some of the 'default' hash behaviors. So I suspect you could program a number of CAM rules that match the NVGRE Ethertype and the inner src/dst IP low bits to give you the hash index .. but of course you'd burn CAM quite quicky but that's probably better than trying to use more src/dst Ips to get the entropy.

Realistically we can have LAG groups with many 10/40G links that need to be spread over. Especially between a monster core router and a monster core switch. So using a couple of extra IP addresses per hypervisor is not going to be adequate. I.e. does not provide enough entropy for such a wide LAG. So in that case and the vanilla ECMP case you'd probably be able to trade CAM space v.s. entropy but that's not a cheap solution.

Peter

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 11 May 2012 08:56:19 +0000

> -----邮件原件-----
> 发件人: Robert Raszuk [mailto:[email-masked]]
> 发送时间: 2012年5月9日 19:01
> 收件人: Ivan Pepelnjak
> 抄送: Keshava A K; [email-masked]; Xuxiaohu; [email-masked]; [email-masked]
> 主题: Re: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN traffic over IP
> PSN using MPLS-in-UDP encapsulation// fwd: New Version Notification for
> draft-xu-mpls-in-udp-00.txt
> 
> Hi Ivan,
> 

Hi all,

The draft that Robert mentioned is now available at http://tools.ietf.org/html/draft-xu-idr-tunnel-address-prefix-00.  Any comments and suggestions are welcome.

BTW, thanks Robert for your pre-announcement of this draft ;)

Best regards,
Xiaohu

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 15 May 2012 19:40:34 +0000

Robert, 

When GRE encapsulation uses different SRC/DST addresses for different flows, it means that same OverlayBoundaryPoint needs multiple addresses. Does it also mean an IP prefix can be given to a OverlayBoundaryPoint? I guess for IPv6, there shouldn't be any issues, should it? (well I am not an IPv6 expert). 

Linda

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Robert Raszuk
> Sent: Wednesday, May 09, 2012 2:27 AM
> To: Keshava A K
> Cc: [email-masked]; Xuxiaohu; [email-masked]; [email-masked]
> Subject: Re: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN
> traffic over IP PSN using MPLS-in-UDP encapsulation// fwd: New Version
> Notification for draft-xu-mpls-in-udp-00.txt
> 
> 
> be

---

From: Murari Sridharan <[email-masked]>
Date: Tue, 15 May 2012 20:12:24 +0000

Whichever entity assigns IP addresses to the host can assign one or many including 1:1 for every VM in that host. This isn't specific to IPv6 or IPv4. 

On a general note, devices/appliances etc that want to participate in NV03 will likely want to understand specific tenant information to provide richer services. No matter what the encapsulation format is devices/appliances will need to update if they want to do something interesting with the flow. I have seen several threads where folks seem to have a "let's keep switch-ECMP" view of the DC which I think is narrow. 

Thanks
Murari 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Tuesday, May 15, 2012 12:41 PM
To: [email-masked]; Keshava A K
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN traffic over IP PSN using MPLS-in-UDP encapsulation// fwd: New Version Notification for draft-xu-mpls-in-udp-00.txt

Robert, 

Linda

> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of Robert Raszuk

---

From: Murari Sridharan <[email-masked]>
Date: Tue, 15 May 2012 20:28:40 +0000

Linda, to clarify the address "assignment" could be setup as a SLAAC and hosts can generate address from a given prefix etc. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Murari Sridharan
Sent: Tuesday, May 15, 2012 1:12 PM
To: Linda Dunbar; [email-masked]; Keshava A K
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN traffic over IP PSN using MPLS-in-UDP encapsulation// fwd: New Version Notification for draft-xu-mpls-in-udp-00.txt

Thanks
Murari 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Tuesday, May 15, 2012 12:41 PM
To: [email-masked]; Keshava A K
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN traffic over IP PSN using MPLS-in-UDP encapsulation// fwd: New Version Notification for draft-xu-mpls-in-udp-00.txt

Robert, 

Linda

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 15 May 2012 20:59:48 +0000

Linda, just to clarify, its not the ultimate host addresses (VM) that need to be multiple, it's the Hyperisor/Vswitch that needs multiple IP addresses. The hash is on this *outer* IP address and is of course limited because there are insufficient outer addresses for good entropy.

You'd use a single IP address for the VM/host but then when encapsulating you'd pick one of say 32 or more Vswitch addresses for your SA based on your VM IPs and the UDP/TCP ports. Everywhere there was a dependency on /32 would have to be made flexible. Not sure if that's an issue but its possibly hard coded in some places. Routers ARP table for example? 

The number of outer addresses you use dictates how much entropy you get when you traverse a LAG which can have quite a few members, but 16 or 32 is likely not unreasonable. Also if you go through several hops, the number of bits of entropy puts a strict upper bound on the total number of different paths you can exercise. So if you have 4 bits you can only exercise 16 unique paths *end to end*. Since the number of paths grows exponentially in the path length a small number of bits does limit spead exponentially quickly.

This 'trick' is only needed of course if the LAG or ECMP hardware does not know to look at an NVGRE header or is not capable of generic hash extensions which some ASICs now can do... (even if they don't support NVGRE termination/encapsulation) so its likely more a concern between DC's than inside the DC.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Murari Sridharan
Sent: Tuesday, May 15, 2012 1:12 PM
To: Linda Dunbar; [email-masked]; Keshava A K
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN traffic over IP PSN using MPLS-in-UDP encapsulation// fwd: New Version Notification for draft-xu-mpls-in-udp-00.txt

Thanks
Murari 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Tuesday, May 15, 2012 12:41 PM
To: [email-masked]; Keshava A K
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN traffic over IP PSN using MPLS-in-UDP encapsulation// fwd: New Version Notification for draft-xu-mpls-in-udp-00.txt

Robert, 

Linda

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 16 May 2012 22:26:36 +0000

Peter, 

You are absolutely right. I am saying the same thing, i.e. the OverlayBounaryPoint has multiple addresses. So that some VMs can be encapsulated in one SR/DST address and other VMs can be encapsulated with other addresses. 

I agree with Murari that focusing on entropy values is too narrow. You can have all the entropy values, but you still will run into in-balanced distribution because flow of each VM is different. 
 
I have noticed that VMware (in their document) and Microsoft tend to use "host" to indicate "Server" which can support multiple VMs. V.s. in other environment "Host" is referring to "End-Station". 

Linda

> From: AshwoodsmithPeter
> Sent: Tuesday, May 15, 2012 4:00 PM
> Subject: RE: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN
> Linda, just to clarify, its not the ultimate host addresses (VM) that
> need to be multiple, it's the Hyperisor/Vswitch that needs multiple IP
> addresses. The hash is on this *outer* IP address and is of course
> limited because there are insufficient outer addresses for good entropy.
> 
> You'd use a single IP address for the VM/host but then when
> encapsulating you'd pick one of say 32 or more Vswitch addresses for
> your SA based on your VM IPs and the UDP/TCP ports. Everywhere there
> was a dependency on /32 would have to be made flexible. Not sure if
> that's an issue but its possibly hard coded in some places. Routers ARP
> table for example?
> 
> The number of outer addresses you use dictates how much entropy you get
> when you traverse a LAG which can have quite a few members, but 16 or
> 32 is likely not unreasonable. Also if you go through several hops, the
> number of bits of entropy puts a strict upper bound on the total number
> of different paths you can exercise. So if you have 4 bits you can only
> exercise 16 unique paths *end to end*. Since the number of paths grows
> exponentially in the path length a small number of bits does limit
> spead exponentially quickly.
> 
> This 'trick' is only needed of course if the LAG or ECMP hardware does
> not know to look at an NVGRE header or is not capable of generic hash
> extensions which some ASICs now can do... (even if they don't support
> NVGRE termination/encapsulation) so its likely more a concern between
> DC's than inside the DC.
> 
> Peter
> 
> Murari Sridharan
> Linda, to clarify the address "assignment" could be setup as a SLAAC
> and hosts can generate address from a given prefix etc.
> 
> Murari Sridharan
> Whichever entity assigns IP addresses to the host can assign one or
> many including 1:1 for every VM in that host. This isn't specific to
> IPv6 or IPv4.
> 
> On a general note, devices/appliances etc that want to participate in
> NV03 will likely want to understand specific tenant information to
> provide richer services. No matter what the encapsulation format is
> devices/appliances will need to update if they want to do something
> interesting with the flow. I have seen several threads where folks seem
> to have a "let's keep switch-ECMP" view of the DC which I think is
> narrow.
> 
> Thanks
> Murari
> 
> Subject: RE: [nvo3] Facilitating the load-balancing of L2VPN/L3VPN
> When GRE encapsulation uses different SRC/DST addresses for different
> flows, it means that same OverlayBoundaryPoint needs multiple addresses.
> Does it also mean an IP prefix can be given to a OverlayBoundaryPoint?
> I guess for IPv6, there shouldn't be any issues, should it? (well I am
> not an IPv6 expert).
> 
> Linda
>