Subject: [bess] Request discussion on Cumulative Link Bandwidth Draft

From: "Satya Mohanty (satyamoh)" <[email-masked]>
Date: Sat, 22 May 2021 15:38:25 +0000

Hi all,

On behalf of all the authors, we request a discussion of the draft https://datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03  and subsequent WG adoption.
This draft extends the usage of the DMZ link bandwidth to scenarios where the cumulative link bandwidth needs to be advertised to a BGP speaker.
Additionally, there is provision to send the link bandwidth extended community to EBGP speakers via configurable knobs. Please refer to section 3 and 4 for the use cases.

This feature has multiple-vendor implementations and has been deployed by several customers in their networks.

Best Regards,
--Satya

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 25 May 2021 03:23:35 -0400

Hi Satya

I read the draft and have a few questions.

IPv4 does not support per flow per packet load balancing as all packets
belonging to the same flow must hash to the same path to prevent out of
order packets and thus is subject to polarization of flows as high
bandwidth flows may hash to the same path and low bandwidth flows as well
to the same path resulting in very uneven load balancing.  Do to this issue
it does not make either iBGP or eBGP can really benefit from link bandwidth
extended community weight based load sharing.

IPV6 flow label RFC 6437 stateless locally significant 5-tuple header hash
generated 20 byte key input to hash function results in uniform 50/50 load
balancing over EGP or IGP ECMP paths.

I think it maybe a good idea to reference the IPv4 polarization issue with
flow based load balancing and that only with IPv6 flow label can true 50/50
uniform load balancing be achieved.

I noticed that the normative draft referenced was adopted but has not
progressed.

https://datatracker.ietf.org/doc/draft-ietf-idr-link-bandwidth/

Has the draft been implemented by Cisco or any other vendors ?

Kind Regards

Gyan

On Sat, May 22, 2021 at 11:38 AM Satya Mohanty (satyamoh) <satyamoh=
[email-masked]> wrote:

> Hi all,
>
>
>
> On behalf of all the authors, we request a discussion of the draft
> https://datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03  and
> subsequent WG adoption.
>
> This draft extends the usage of the DMZ link bandwidth to scenarios where
> the cumulative link bandwidth needs to be advertised to a BGP speaker.
>
> Additionally, there is provision to send the link bandwidth extended
> community to EBGP speakers via configurable knobs. Please refer to section
> 3 and 4 for the use cases.
>
>
>
> _______________________________________________
> BESS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bess
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Tue, 25 May 2021 16:48:18 +0000

The link bandwidth community has been implemented by Cisco and deployed by
our customers for several years.
Polarization of flows in multipath is a well known problem, but it hasn't deterred
people from using it.

Regards,
Jakob.

From: BESS <[email-masked]> On Behalf Of Gyan Mishra
Sent: Tuesday, May 25, 2021 12:24 AM
To: Satya Mohanty (satyamoh) <satyamoh=[email-masked]>
Cc: [email-masked]
Subject: Re: [bess] Request discussion on Cumulative Link Bandwidth Draft

Hi Satya

IPv4 does not support per flow per packet load balancing as all packets belonging to the same flow must hash to the same path to prevent out of order packets and thus is subject to polarization of flows as high bandwidth flows may hash to the same path and low bandwidth flows as well to the same path resulting in very uneven load balancing.  Do to this issue it does not make either iBGP or eBGP can really benefit from link bandwidth extended community weight based load sharing.

IPV6 flow label RFC 6437 stateless locally significant 5-tuple header hash generated 20 byte key input to hash function results in uniform 50/50 load balancing over EGP or IGP ECMP paths.

I think it maybe a good idea to reference the IPv4 polarization issue with flow based load balancing and that only with IPv6 flow label can true 50/50 uniform load balancing be achieved.

I noticed that the normative draft referenced was adopted but has not progressed.

Kind Regards

Gyan

On Sat, May 22, 2021 at 11:38 AM Satya Mohanty (satyamoh) <satyamoh=[email-masked]<mailto:[email-masked]>> wrote:
Hi all,

Best Regards,
--Satya
_______________________________________________
BESS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bess
--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<http://www.verizon.com/>

Gyan Mishra

Network Solutions Architect

Email [email-masked]<mailto:[email-masked]>

M 301 502-1347

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 25 May 2021 20:06:40 +0200

Gyan,

It is always helpful to an assessment into right scale.

Yes if you take few flows perhaps even a few big ones may suffer from
polarization. But the feature here is about hashing millions of micro
flows. With that in mind polarization effects are insignificant at
decent operational scale.

I support this draft.

Cheers,
Robert

PS. Also keep in mind that for handling elephant flows there are bunch of
TE like solutions in place which can be used which in turn give you very
fine control.

On Tue, May 25, 2021 at 9:23 AM Gyan Mishra <[email-masked]> wrote:

>> https://datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03
>>  and subsequent WG adoption.
>>

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 25 May 2021 14:52:25 -0400

That’s great  news that Cisco had implemented and customers have deployed
for years!

I see it’s supported in IOS and XR

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-16/irg-xe-16-book/bgp-link-bandwidth.html

https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5000/bgp/65x/b-bgp-cg-ncs5000-65x/b-bgp-cg-ncs5000-65x_chapter_010.html

Thanks!!

Gyan

On Tue, May 25, 2021 at 12:48 PM Jakob Heitz (jheitz) <[email-masked]>
wrote:

> *From:* BESS <[email-masked]> *On Behalf Of * Gyan Mishra
> *Sent:* Tuesday, May 25, 2021 12:24 AM
> *To:* Satya Mohanty (satyamoh) <satyamoh=[email-masked]>
> *Cc:* [email-masked]
> *Subject:* Re: [bess] Request discussion on Cumulative Link Bandwidth
> Draft
>
>
>
>
>
> Hi Satya
>
>
>
> *Network Solutions Architect *
>
-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 25 May 2021 14:57:17 -0400

Arie

Thanks for responding on the polarization question and I agree it can
enhance ECMP capability and maybe even counter or reduce effects of
polarization.

Thanks

Gyan

On Tue, May 25, 2021 at 1:26 PM Arie Vayner <[email-masked]> wrote:

> The flow polarization or elephant flow issues are well known industry
> items and they apply to both equal and unequal cost multi-path approaches.
>
> The objective of this proposal is to enhance ECMP, and enable unequal cost
> multi-pathing, which is very useful when a service is offered by a
> multitude of endpoints, which may or may not have the same capacity.
> This solution has been implemented in production, and offers a real option
> for traffic load management.
>
> Tnx
> Arie
>
> On Tue, May 25, 2021 at 9:48 AM Jakob Heitz (jheitz) <jheitz=
>> IPV6 flow label RFC 6437 stateless locally significant 5-tuple header
>> hash generated 20 byte key input to hash function results in uniform 50/50
>> load balancing over EGP or IGP ECMP paths.
>>
>>
>>
>> I think it maybe a good idea to reference the IPv4 polarization issue
>> with flow based load balancing and that only with IPv6 flow label can true
>> 50/50 uniform load balancing be achieved.
>>
>>
>>
>> *M 301 502-1347 <(301)%20502-1347>*
>>
>>

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 25 May 2021 15:01:07 -0400

Hi Robert

Very good point on millions of mice flows that can take advantage of
unequal cost lb,  versus a small number of elephant flows.

Good point on further granularity with  TE bandwidth management .

Gyan

On Tue, May 25, 2021 at 2:06 PM Robert Raszuk <[email-masked]> wrote:

>>> This draft extends the usage of the DMZ link bandwidth to scenarios
>>> where the cumulative link bandwidth needs to be advertised to a BGP
>>> speaker.
>>>

*Gyan Mishra*

*M 301 502-1347*

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 25 May 2021 12:55:01 -0700

Hi,

I support adoption of the draft as Informational, please note, that request to change transitivity characteristics of the community is requested in another draft.
Gyan  - please note, while pretty much every vendor has implemented the community and relevant data-plane constructs, initial draft defines the community as non transititive, some vendors have followed that while some other have implemented it a transitive (to support obvious use case - eBGP in DC).

Cheers,
Jeff

---

From: Arie Vayner <[email-masked]>
Date: Tue, 25 May 2021 13:16:14 -0700

Jeff,

Actually, the way this draft is written, and how the implementations I'm
aware of are implemented, this is not really a transitive community. It is
a new community that is being generated on the AS boundary.
The community value is not carried over, but is calculated based on an
cumulative value of other received communities, and then advertised as a
new value across the AS boundary.

Tnx,
Arie

On Tue, May 25, 2021 at 12:55 PM Jeff Tantsura <[email-masked]>
wrote:

> Hi,
>
> I support adoption of the draft as Informational, please note, that
> request to change transitivity characteristics of the community is
> requested in another draft.
> Gyan  - please note, while pretty much every vendor has implemented the
> community and relevant data-plane constructs, initial draft defines the
> community as non transititive, some vendors have followed that while some
> other have implemented it a transitive (to support obvious use case - eBGP
> in DC).
>
>
> Cheers,
> Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 26 May 2021 01:00:36 +0200

Hi Arie,

Draft  draft-ietf-idr-link-bandwidth talks about advertising towards IBGP.
It does not talk about advertising over EBGP.

While I do support your use case I think it would be much cleaner to just
ask for new ext. community type.

Reason being that as you illustrate you may want to accumulate BGP path's
bw across few EBGP hops in the DC. Today there is no way to do so unless
you want to completely hijack current lb ext community.

Also I see an analogy here to AIGP RFC although it clearly fits rather
poorly for those who use BGP as IGP :).

Best, R.

---

From: Gyan Mishra <[email-masked]>
Date: Wed, 26 May 2021 00:56:58 -0400

Across the DC space in general most providers use NVO3 and vxlan source
port entropy L2/L3/L4 hash which provides per packet uniform 50/50 load
balancing at the L2 VNI overlay layer, which translates into underlay load
balancing of flows and thus no polarization.

Across the DC space speaking from an operators perspective as under the
floor fiber is not at a premium compare to 100G facilities costs the net
addition of bandwidth can be done fairly quickly so you are ahead of the
congestion curve and can be proactive versus reactively upgrading bandwidth
piecemeal here and there ad hoc.

There still maybe cases that still arise as even if you have the fiber
infrastructure available, it’s not easy to upgrade and flash every link
simultaneously in the DC in one or multiple maintenance windows, so you
could still be left with some uneven bandwidth across the DC that could
utilize this feature.

DC comes into play for PE-CE “wan links”as well  use case for unequal cost
load balancing use of the cumulative link bandwidth community regenerated.

I think the use case where both the iBGP P core P-P links  or eBGP PE - PE
inter-as are all wan links where link upgrades tend to not get done in
unison uniformly, and in those cases both iBGP link bandwidth community can
be heavily utilized as well as eBGP cumulative regenerated link bandwidth
community for unequal cost  load balancing.  Across the core as well it is
hard to flash all links even under floor fiber to the same bandwidth all at
once you are left with the requirement for unequal coat load balancing.

As operators upgrade their DC as well as core infrastructure to IPv6
forwarding plane in the move towards SRv6, they can now take advantage of
flow label entropy stateless uniform load balancing and elimination of
polarization.  However, the wan link upgrades of core and DC PE-CE still
exists and thus may be done piecemeal, so then both of the drafts are an
extremely helpful tool for operators that much need the unequal cost load
balancing capability.

I support both drafts.

Have most vendors implemented this to support both 2 byte and 4 byte AS
extended community.  The drafts state 2 byte AS support.

Thanks

Gyan

On Tue, May 25, 2021 at 7:00 PM Robert Raszuk <[email-masked]> wrote:

> Hi Arie,
>
> Draft  draft-ietf-idr-link-bandwidth talks about advertising towards
> IBGP. It does not talk about advertising over EBGP.
>
>>> where the cumulative link bandwidth needs to be advertised to a BGP speaker.
>>>

*Gyan Mishra*

*M 301 502-1347*

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 26 May 2021 12:37:48 +0000

Does this work and Weighted Multi-Path Procedures for EVPN Multi-homing address the same field of use?

Thanks,
              Jim Uttaro

From: BESS <[email-masked]> On Behalf Of Gyan Mishra
Sent: Wednesday, May 26, 2021 12:57 AM
To: Robert Raszuk <[email-masked]>
Cc: Jeff Tantsura <[email-masked]>; Arie Vayner <[email-masked]>; [email-masked]; Satya Mohanty (satyamoh) <satyamoh=[email-masked]>
Subject: Re: [bess] Request discussion on Cumulative Link Bandwidth Draft

Across the DC space in general most providers use NVO3 and vxlan source port entropy L2/L3/L4 hash which provides per packet uniform 50/50 load balancing at the L2 VNI overlay layer, which translates into underlay load balancing of flows and thus no polarization.

Across the DC space speaking from an operators perspective as under the floor fiber is not at a premium compare to 100G facilities costs the net addition of bandwidth can be done fairly quickly so you are ahead of the congestion curve and can be proactive versus reactively upgrading bandwidth piecemeal here and there ad hoc.

There still maybe cases that still arise as even if you have the fiber infrastructure available, it’s not easy to upgrade and flash every link simultaneously in the DC in one or multiple maintenance windows, so you could still be left with some uneven bandwidth across the DC that could utilize this feature.

DC comes into play for PE-CE “wan links”as well  use case for unequal cost load balancing use of the cumulative link bandwidth community regenerated.

I think the use case where both the iBGP P core P-P links  or eBGP PE - PE inter-as are all wan links where link upgrades tend to not get done in unison uniformly, and in those cases both iBGP link bandwidth community can be heavily utilized as well as eBGP cumulative regenerated link bandwidth community for unequal cost  load balancing.  Across the core as well it is hard to flash all links even under floor fiber to the same bandwidth all at once you are left with the requirement for unequal coat load balancing.

As operators upgrade their DC as well as core infrastructure to IPv6 forwarding plane in the move towards SRv6, they can now take advantage of flow label entropy stateless uniform load balancing and elimination of polarization.  However, the wan link upgrades of core and DC PE-CE still exists and thus may be done piecemeal, so then both of the drafts are an extremely helpful tool for operators that much need the unequal cost load balancing capability.

I support both drafts.

Have most vendors implemented this to support both 2 byte and 4 byte AS extended community.  The drafts state 2 byte AS support.

Thanks

Gyan

On Tue, May 25, 2021 at 7:00 PM Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:
Hi Arie,

Draft  draft-ietf-idr-link-bandwidth talks about advertising towards IBGP. It does not talk about advertising over EBGP.

While I do support your use case I think it would be much cleaner to just ask for new ext. community type.

Reason being that as you illustrate you may want to accumulate BGP path's bw across few EBGP hops in the DC. Today there is no way to do so unless you want to completely hijack current lb ext community.

Also I see an analogy here to AIGP RFC although it clearly fits rather poorly for those who use BGP as IGP :).

Best, R.

On Wed, May 26, 2021 at 12:22 AM Arie Vayner <[email-masked]<mailto:[email-masked]>> wrote:
Jeff,

Actually, the way this draft is written, and how the implementations I'm aware of are implemented, this is not really a transitive community. It is a new community that is being generated on the AS boundary.
The community value is not carried over, but is calculated based on an cumulative value of other received communities, and then advertised as a new value across the AS boundary.

Tnx,
Arie

On Tue, May 25, 2021 at 12:55 PM Jeff Tantsura <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

Cheers,
Jeff
On May 22, 2021, 8:38 AM -0700, Satya Mohanty (satyamoh) <satyamoh=[email-masked]<mailto:[email-masked]>>, wrote:

Hi all,

On behalf of all the authors, we request a discussion of the draft https://datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03__;!!BhdT!0b982PpfH6BN3cZfnleSv0ex_IJjKEMUOXi42a_RhyGg6nB17aWOnm6zLkY$>  and subsequent WG adoption.
This draft extends the usage of the DMZ link bandwidth to scenarios where the cumulative link bandwidth needs to be advertised to a BGP speaker.
Additionally, there is provision to send the link bandwidth extended community to EBGP speakers via configurable knobs. Please refer to section 3 and 4 for the use cases.

Best Regards,
--Satya
_______________________________________________
BESS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bess<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bess__;!!BhdT!0b982PpfH6BN3cZfnleSv0ex_IJjKEMUOXi42a_RhyGg6nB17aWO_sLM9KM$>
_______________________________________________
BESS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bess<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bess__;!!BhdT!0b982PpfH6BN3cZfnleSv0ex_IJjKEMUOXi42a_RhyGg6nB17aWO_sLM9KM$>
_______________________________________________
BESS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bess<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/bess__;!!BhdT!0b982PpfH6BN3cZfnleSv0ex_IJjKEMUOXi42a_RhyGg6nB17aWO_sLM9KM$>
--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!BhdT!0b982PpfH6BN3cZfnleSv0ex_IJjKEMUOXi42a_RhyGg6nB17aWORYETg1w$>

Gyan Mishra

M 301 502-1347

---

From: "Satya Mohanty (satyamoh)" <[email-masked]>
Date: Wed, 26 May 2021 19:21:13 +0000

Hi Jim,

No, they do not.

This draft under discussion is a way to aggregate the link bandwidth in EBGP DCs and advertise it upstream.
It works well in multi-stage clos topology fabrics.
Traffic is demultiplexed (multi-path load balanced) when it arrives at a node of each stage (unless the sink).

The draft you are mentioning, https://tools.ietf.org/id/draft-ietf-bess-evpn-unequal-lb-06.html is really a way to communicate the link-bandwidth across EBGP boundaries.
It is mostly geared from an Inter-AS Option C viewpoint (next-hop unchanged) although it also applies to Option B deployments (next-hop-self).
There is no notion of aggregating bandwidth here.

HTH.

Best Regards,
--Satya

From: "UTTARO, JAMES" <[email-masked]>
Date: Wednesday, May 26, 2021 at 5:38 AM
To: Gyan Mishra <[email-masked]>, Robert Raszuk <[email-masked]>
Cc: Jeff Tantsura <[email-masked]>, Arie Vayner <[email-masked]>, "[email-masked]" <[email-masked]>, "Satya Mohanty (satyamoh)" <[email-masked]>
Subject: RE: [bess] Request discussion on Cumulative Link Bandwidth Draft

Thanks,
              Jim Uttaro

I support both drafts.

Thanks

Gyan

Best, R.

Tnx,
Arie

Cheers,
Jeff

Hi all,

Best Regards,
--Satya

[Image removed by sender.]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!BhdT!0b982PpfH6BN3cZfnleSv0ex_IJjKEMUOXi42a_RhyGg6nB17aWORYETg1w$>

Gyan Mishra

M 301 502-1347