Subject: Re: [Idr] I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Sat, 3 Jul 2021 09:38:43 +0000

Hi Robert,

Thank you for the questions and comments. Apologies - I somehow missed these emails with your questions, hence the delay in response.

Please see my response inline.. KV>

From: Robert Raszuk <[email-masked]>
Date: Friday, June 11, 2021 at 3:10 AM
To: Kaliraj Vairavakkalai <[email-masked]>, Minto Jeyananth <[email-masked]>
Cc: idr@ietf. org <[email-masked]>
Subject: Fwd: I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
[External Email. Be cautious of content]

Dear authors,

I have read yr draft with interest as some perhaps recall we have discussed this topic in the past number of times.

Cosmetics:

1. First nit - why do you say label must be 3107bis label ? (3.4.2.  Labeled IP nexthop) MPLS label is a label and I am not sure how one method of label distribution matter here.

KV> The intent was to say use the same label-encoding as specified in 3107bis (now rfc8277). But yes, I can just say it is an mpls-label, and specify how it is encoded.

2a. What is PNH ? It first occurs in section 3 as "PNH-Len" or "PNH-address", but it is never explained in the draft. Is this Path Next Hop ?

KV>  “Protocol Next Hop” The nexthop-ip-address received in a BGP-update. I will add it to the terminologies.

2b. Would it be better to call this new attribute MNH MultiNextHop ?

KV> sure, we can abbreviate it as MNH.

Tech:

3. In section 3.1.1 you describe how to assure that NH in MP_REACH is also part of MultiNext Hop Attribute. But I do not see any discussion on how to treat or ignore next hops in MP_REACH when a new attribute is present and is valid.

KV> Sure, I will add some text reg that. Basically, a new-speaker will use the information received in MNH attr to install forwarding state. The PNH received in attr-code 3 or 14 will be used only to validate the MNH, and not for installing forwarding state. If the MNH is not valid then the MNH will be ignored and the PNH received in attr-code 3,14 will be used for forwarding. An old-speaker that does not understand MNH will also use the PNH received in attr-code 3,14.

4. In section 3.1.2 you define behaviour of RR advertising paths from non MultiNexthop paths and those which carry new attributes ... But you should make it clear that this is only about 9.1.2.2 step in best path selection (or candidate selection). There can be other criteria before we even get to that step.

KV>  3.1.2 is actually not about path-selection. It is about add-path candidates selection. Multiple routes with same PNH are not advertised as add-path advertisements. Similarly when the routes contain MHN, we want to compare the MNH to decide whether the route is a candidate to be included in add-path advertisement or not; the decision should not be only based on PNH in attr-code 3, 14

5. Now the most important question - how do you plan to handle atomic withdraws ? I assume the plan is to readvertise the path with MNH - the removed next hop. So by implicit withdraw this next hop will be removed. The draft is silent on this.

KV> that is right. The bgp-update(implicit withdraw) will replace the MNH. I will add text describing it.

Now if the removed next hop was selected by some receivers as best (due to 9.1.2.2) and it was not arriving as part of MP_REACH this proposal require significant implementation changes on how BGP best path selection is triggered, how it runs, how it populates results to the RIB/FIB. I think a new section is needed in detailed discussing the withdraws.

KV> the path-selection selects a best-route. Not a best-nexthop. If the best-route contains MNH, section 3.1.3 specifies how to compute the effective IGP-cost of the route which will be used for best-route path-selection. All resolved(reachable) legs of the MNH will be installed to FIB.

KV> The path-selection will be re-run whenever the effective IGP-cost of a route changes. Any AIGP-attribute will also carry this effective-IGP-cost instead of the igp-cost to the PNH in attr 3,14.

6. How do you envision max-prefix safety knobs to work here ? On the surface it may seem orthogonal - but it is not. Today folks use this to protect infrastructure from for example operator's mistakes. Here one received path may fill the MNH attribute with 100s of next hops

KV> Let us say the max-num-ECMP supported by a router is 16. Then it will install the first 16 nexthops (as per the ‘Relative-preference’ field) received in the MNH. It will not install all the nexthops. So the same existing ecmp-limit safeguards can be used. The only difference is how the nexthops are received. Instead being received in 100 routes, it is received in 1 route.

and as being optional and transitive will be distributed to all routers all over the world.

KV> and I agree, I realize it may be better to make this attribute ‘optional non-transitive’.

7. Observe that metric to next hops is dynamic. So some implementations capable of next hop tracking register with RIB all next hops and each time metric changes they get a call back. Here we are effectively talking about exploding this 10x or 100x or more ...

KV> they could potentially get called-back when effective-igp-cost of the route changes. Pls note, this proposal does not increase number of unique PNHs in the system/network, for e.g. that is a function of how many PEs exist in the network. Just what changes is how those PNHs are conveyed in bgp-updates as ecmp/pic nexthops for a bgp-prefix. Number of routes required to convey that relationship is greatly reduced, while providing more control to the advertiser on signaling the ecmp/pic relationships. Thanks.

Many thx,
Robert

---------- Forwarded message ---------
From: <[email-masked]<mailto:[email-masked]>>
Date: Fri, Jun 11, 2021 at 10:56 AM
Subject: I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
To: <[email-masked]<mailto:[email-masked]>>

A New Internet-Draft is available from the on-line Internet-Drafts directories.

        Title           : BGP MultiNexthop attribute
        Authors         : Kaliraj Vairavakkalai
                          Minto Jeyananth
        Filename        : draft-kaliraj-idr-multinexthop-attribute-01.txt
        Pages           : 12
        Date            : 2021-06-11

Abstract:
   Today, a BGP speaker can advertise one nexthop for a set of NLRIs in
   an Update.  This nexthop can be encoded in either the BGP-Nexthop
   attribute (code 3), or inside the MP_REACH attribute (code 14).

   For cases where multiple nexthops need to be advertised, BGP-Addpath
   is used.  Though Addpath allows basic ability to advertise multiple-
   nexthops, it does not allow the sender to specify desired
   relationship between the multiple nexthops being advertised e.g.,
   relative-preference, type of load-balancing.  These are local
   decisions at the receiving speaker based on path-selection between
   the various additional-paths, which may tie-break on some arbitrary
   step like Router-Id.

   Some scenarios with a BGP-free core may benefit from having a
   mechanism, where egress-node can signal multiple-nexthops along with
   their relationship to ingress nodes.  This document defines a new BGP
   attribute "MultiNexthop" that can be used for this purpose.

   This attribute can be used for both labeled and unlabled BGP
   families.  For labeled-families, it is used for a different purpose
   in "downstream allocation" case than "upstream allocation" scenarios.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-kaliraj-idr-multinexthop-attribute/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-kaliraj-idr-multinexthop-attribute/__;!!NEt6yMaO-gk!SJ7yfFqQyEkq_YkuCGs7lDpAL_dkijADLgFM7qCDZlGE_sn2mBkcG3EgC7g463XU$>

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01__;!!NEt6yMaO-gk!SJ7yfFqQyEkq_YkuCGs7lDpAL_dkijADLgFM7qCDZlGE_sn2mBkcG3EgC-8LAzIb$>

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-kaliraj-idr-multinexthop-attribute-01<https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-kaliraj-idr-multinexthop-attribute-01__;!!NEt6yMaO-gk!SJ7yfFqQyEkq_YkuCGs7lDpAL_dkijADLgFM7qCDZlGE_sn2mBkcG3EgC91WiRjG$>

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/<https://urldefense.com/v3/__ftp:/ftp.ietf.org/internet-drafts/__;!!NEt6yMaO-gk!SJ7yfFqQyEkq_YkuCGs7lDpAL_dkijADLgFM7qCDZlGE_sn2mBkcG3EgC-RWSEME$>

_______________________________________________
I-D-Announce mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/i-d-announce<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/i-d-announce__;!!NEt6yMaO-gk!SJ7yfFqQyEkq_YkuCGs7lDpAL_dkijADLgFM7qCDZlGE_sn2mBkcG3EgC6XAC-1q$>
Internet-Draft directories: http://www.ietf.org/shadow.html<https://urldefense.com/v3/__http:/www.ietf.org/shadow.html__;!!NEt6yMaO-gk!SJ7yfFqQyEkq_YkuCGs7lDpAL_dkijADLgFM7qCDZlGE_sn2mBkcG3EgC35gi1-A$>
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt<https://urldefense.com/v3/__ftp:/ftp.ietf.org/ietf/1shadow-sites.txt__;!!NEt6yMaO-gk!SJ7yfFqQyEkq_YkuCGs7lDpAL_dkijADLgFM7qCDZlGE_sn2mBkcG3EgC8P5Eax-$>

Juniper Business Use Only

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Sat, 3 Jul 2021 09:38:55 +0000

Pls see inline.. KV>

From: Robert Raszuk <[email-masked]>
Date: Friday, June 11, 2021 at 6:09 AM
To: Kaliraj Vairavakkalai <[email-masked]>, Minto Jeyananth <[email-masked]>
Cc: idr@ietf. org <[email-masked]>
Subject: Re: I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
[External Email. Be cautious of content]

HI

Two more questions ...

8. I am not clear what are trying to describe in Section 8.

   Like any other optional transitive BGP attribute, it is possible that

   this attribute gets propagated thru speakers that don't understand

   this attribute and an error detected by a speaker multiple hops away.

   This is mitigated by requiring the receiving speaker to remove this

   attribute when doing nexthop-self.

First indeed, the attribute may be propagated by BGP speakers which do not understand it, but that in itself is not an error. In those cases partial bit is set but attribute is still valid.

KV> I agree, the attribute should be made ‘optional non-transitive’. Å receiving old-speaker cannot remove the attribute since he doesn’t understand it. I should correct this.

This is also completely orthogonal to setting the next hop self on the path when propagating for example across EBGP.

KV> a new-speaker that receives the MNH on a route should not propagate it as-is (received MNH must be stripped), if it readvertises with nexthop-self (new MNH can be added as required).  When the nexthop is left unchanged, the MNH should be propagated as-is.

9. You are providing a lot of analogy to Add-Paths. But in Add-Paths thx to capability negotiation it is mandatory for receiving speaker indicating support to act on it. Here you have chosen for some reason blind propagation as optional transitive which perhaps may be ok for networks which do end to end encapsulation, but I don't think it is going to work in pure IPv4/IPv6 hop by hop lookup - especially in the cases of mixed network elements some supporting the attribute and some not.

KV> I had chosen ‘optional transitive’ to be able to pass thru RRs that don’t support it. But we also need to scope it within an AS and not allow it to pass ebgp boundary. And I agree with you the only way to do it for old-speakers is to make it ‘optional non-transitive’. I will correct that.

KV> And yes, the proposal is intended for BGP-free cores that use tunneling. Use in hop-by-hop forwarding network scenario is out of scope of this document. Thanks.

Thx,
R.

On Fri, Jun 11, 2021 at 12:09 PM Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:
Dear authors,

Cosmetics:

Tech:

5. Now the most important question - how do you plan to handle atomic withdraws ? I assume the plan is to readvertise the path with MNH - the removed next hop. So by implicit withdraw this next hop will be removed. The draft is silent on this. Now if the removed next hop was selected by some receivers as best (due to 9.1.2.2) and it was not arriving as part of MP_REACH this proposal require significant implementation changes on how BGP best path selection is triggered, how it runs, how it populates results to the RIB/FIB. I think a new section is needed in detailed discussing the withdraws.

6. How do you envision max-prefix safety knobs to work here ? On the surface it may seem orthogonal - but it is not. Today folks use this to protect infrastructure from for example operator's mistakes. Here one received path may fill the MNH attribute with 100s of next hops and as being optional and transitive will be distributed to all routers all over the world.

Many thx,
Robert

---------- Forwarded message ---------
From: <[email-masked]<mailto:[email-masked]>>
Date: Fri, Jun 11, 2021 at 10:56 AM
Subject: I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
To: <[email-masked]<mailto:[email-masked]>>

Abstract:

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-kaliraj-idr-multinexthop-attribute/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-kaliraj-idr-multinexthop-attribute/__;!!NEt6yMaO-gk!V8NWgdDpCqZ3PhBo-vsbtyELsTm7DPfL_b0JOVvJlQSSVkgJ8InqCJdXLNQPUQl6$>

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01__;!!NEt6yMaO-gk!V8NWgdDpCqZ3PhBo-vsbtyELsTm7DPfL_b0JOVvJlQSSVkgJ8InqCJdXLNUvhMBP$>

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-kaliraj-idr-multinexthop-attribute-01<https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-kaliraj-idr-multinexthop-attribute-01__;!!NEt6yMaO-gk!V8NWgdDpCqZ3PhBo-vsbtyELsTm7DPfL_b0JOVvJlQSSVkgJ8InqCJdXLAMdHEi7$>

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/<https://urldefense.com/v3/__ftp:/ftp.ietf.org/internet-drafts/__;!!NEt6yMaO-gk!V8NWgdDpCqZ3PhBo-vsbtyELsTm7DPfL_b0JOVvJlQSSVkgJ8InqCJdXLKxw4CKE$>

_______________________________________________
I-D-Announce mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/i-d-announce<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/i-d-announce__;!!NEt6yMaO-gk!V8NWgdDpCqZ3PhBo-vsbtyELsTm7DPfL_b0JOVvJlQSSVkgJ8InqCJdXLO4SnRRL$>
Internet-Draft directories: http://www.ietf.org/shadow.html<https://urldefense.com/v3/__http:/www.ietf.org/shadow.html__;!!NEt6yMaO-gk!V8NWgdDpCqZ3PhBo-vsbtyELsTm7DPfL_b0JOVvJlQSSVkgJ8InqCJdXLGkuKZBP$>
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt<https://urldefense.com/v3/__ftp:/ftp.ietf.org/ietf/1shadow-sites.txt__;!!NEt6yMaO-gk!V8NWgdDpCqZ3PhBo-vsbtyELsTm7DPfL_b0JOVvJlQSSVkgJ8InqCJdXLOheoCZF$>

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Sat, 3 Jul 2021 09:39:13 +0000

Hi Gyan,

From: Gyan Mishra <[email-masked]>
Date: Friday, June 11, 2021 at 7:27 PM
To: Robert Raszuk <[email-masked]>
Cc: Kaliraj Vairavakkalai <[email-masked]>, idr@ietf. org <[email-masked]>, Minto Jeyananth <[email-masked]>
Subject: Re: [Idr] I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
[External Email. Be cautious of content]

Hi Kaliraj

I read through the draft and had a few questions.

This feature parallels add path but is different in the way it is deployed as it a BGP optional transitive path attribute as opposed to a add path capability code which is exchanged P2P between PE-RR peers in a controlled fashion.  In that manner the add paths is generally used within an AS for BGP PIC advertisement of backup paths as well as for BGP multipath load balancing within an AS.

The gap this draft fills is clear to be able to provide a more intelligent add path advertisement feature.  This also fills a very important gap of unequal cost load balancing as in a topology may not have equal cost cost metric for iBGP tie breaker, thus one path ends up being the best path even though multiple paths exist with add paths even with unique RD, which makes it difficult for iBGP load balancing to occur.

I support the development and progression of this draft as this is an import solution to a real world problem of VPN overlay iBGP unequal cost load balancing within the operator core as well as uniform predictable load balancing.

KV> Thank you.

MultiNextHop= MNH - referring to as

Is the intent to use within an AS and if so making the attribute non transitive I think  is a MUST as the next hop is changed for external peering the propagation of the multinexthop attribute is irrelevant.   I think you mentioned next-hop-self which is set on PE-RR peering at the domain ingress node can be used to stop the MNH from being propagated.  However, the next-hop-self is done in ingress inbound upon entering the domain and not leaving the domain so I think the MNH can still get propagated outside the domain.

Section 3.1.1 described interaction with top level mandatory transitive next hop attribute code 3 and the BGP Update MP_REACH next hop code 14.

Can you help explain section 3.1.1 interaction

   “When adding a MultiNexthop attribute to an advertised BGP route, the

   speaker MUST put the same next-hop address in the Advertising PNH

   field as it put in the Nexthop field inside NEXT_HOP attribute or

   MP_REACH_NLRI attribute.  Any speaker that recognizes this attribute

   and changes the PNH while re-advertising the route MUST remove the

   MultiNexthop-Attribute in the re-advertisement.  The speaker MAY

   however add a new MultiNexthop-Attribute to the re-advertisement;

   while doing so the speaker MUST record in the "Advertising-PNH" field

   the same next-hop address as used in NEXT_HOP field or MP_REACH_NLRI

   attribute.”

What is PNH?  primary next hop ??

KV> “Protocol Next Hop”. Nexthop ip-address carried in the attr 3, 14.

So the way I read this is the optional transitive MNH attribute is copied into code 3 and 14.  So if there are 10 NH in the PNH they all get written into top level code 3 and BGP update code 14.

KV> no. only the following advertising-pnh field has the same address as the attr 3, 14 nexthop field when advertising with nexthop-self.

   Advertising PNH     IPv4 or IPv6 PNH-address (Len = 32 or 128)
                          advertised in NEXT_HOP or MP_REACH_NLRI attr.
                          Used to sanity-check this attribute

KV> The formatting in 01 version of the draft is messed up, I’ll correct it. Pls refer to the 00 version which has proper formatting.

Since this is for “BGP free core” the P routers where Label swap forwarding semantics happens is not applicable as we don’t run BGP.  As the forwarding semantics is not applicable after the VPN label “push” label imposition CE to PE eBGP peering entering the core or “pop and forward” label disposition PE to CE eBGP peering leaving the core.
All the next hops have a VPN label imposed which is the next-hop-self from each PE, which I think would have only the “forwarding” semantics applicable, so I think that is the only one that would apply for MNH iBGP load balancing.

KV> For vpn-family routes we could continue to carry the label in rfc8277 bgp-nlri. And use MNH with 3.4.1.
        If label is used in both rfc8277 bgp-nlri and the MNH 3.4.2, then it would mean the bgp-nlri imposed mpls-label will be pushed on top of the 3.4.2 specified label.
        The main use of 3.4.2 is to be used with inet-unicast routes. That will equate inet-unicast routes with inet-vpn/inet-lu routes. Because using this now we can bind a mpls-label to a inet-unicast service route also.
        So that all these service families are treated equal. This helps in doing option-B for internet routes as-well, avoiding service-route scale on ASBR BNs.
        And yes, ECMP/PIC/WECMP will happen based on information received in “Nexthop Forwarding Semantic TLV”. Thanks.

Kind Regards

Gyan

On Fri, Jun 11, 2021 at 9:10 AM Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:
HI

Two more questions ...

Thx,
R.

Cosmetics:

Tech:

Many thx,
Robert

---------- Forwarded message ---------
From: <[email-masked]<mailto:[email-masked]>>
Date: Fri, Jun 11, 2021 at 10:56 AM
Subject: I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
To: <[email-masked]<mailto:[email-masked]>>

Abstract:

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-kaliraj-idr-multinexthop-attribute/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-kaliraj-idr-multinexthop-attribute/__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRLDTqKGj$>

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRLYOeXy-$>

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-kaliraj-idr-multinexthop-attribute-01<https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-kaliraj-idr-multinexthop-attribute-01__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRIE_FnU0$>

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/<https://urldefense.com/v3/__ftp:/ftp.ietf.org/internet-drafts/__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfROJyomj_$>

_______________________________________________
I-D-Announce mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/i-d-announce<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/i-d-announce__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfREhLtYSR$>
Internet-Draft directories: http://www.ietf.org/shadow.html<https://urldefense.com/v3/__http:/www.ietf.org/shadow.html__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRNd45oQu$>
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt<https://urldefense.com/v3/__ftp:/ftp.ietf.org/ietf/1shadow-sites.txt__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfROCmEIB8$>
_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRAjWelEr$>
--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRJoHyeCI$>

Gyan Mishra

Network Solutions Architect

Email [email-masked]<mailto:[email-masked]>

M 301 502-1347

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 3 Jul 2021 13:27:47 +0200

Hi,

Few follow up points ...

KV>  All resolved(reachable) legs of the MNH will be installed to FIB.

That is huge change BGP <--> RIB interaction. Note today unless you install
more next hops *only* if multipath is enabled.

Besides you can safely do it only when network is doing encapsulation end
to end. If you are doing hop by hop lookup and you do not even know if all
nodes support MNH loops can form.

KV> Let us say the max-num-ECMP supported by a router is 16. Then it will
> install the first 16 nexthops (as per the ‘Relative-preference’ field)
> received in the MNH. It will not install all the nexthops. So the same
> existing ecmp-limit safeguards can be
>

Nope that was not the point ... I am talking about control plane RAM when
the attacker is using MNH attribute fill to the top of the UPDATE MSG SIZE
to kill your box regardless if this will get down to RIB or FIB.

Today prefix limit saves you from this type of attacks. When one deployed
MNH with the peers it opens up his network ... I would love to see this
type of threat discussed in security section.

> used. The only difference is how the nexthops are received. Instead being
> received in 100 routes, it is received in 1 route.
>

Nope ... peers today sends you the single path as best unless you allow him
to do add-paths all.

KV> they could potentially get called-back when effective-igp-cost of the
> route changes. Pls note, this proposal does not increase number of unique
> PNHs in the system/network, for e.g. that is a function of how many PEs
> exist in the network. Just what changes is how those PNHs are conveyed in
> bgp-updates as ecmp/pic nexthops for a bgp-prefix. Number of routes
> required to convey that relationship is greatly reduced, while providing
> more control to the advertiser on signaling the ecmp/pic relationships.
>

If you allow this for EBGP you are changing the BGP game completely.
Especially if you allow this for SAFI 1.

- - -

Btw ... who would build this MNH attribute ? ASBR/PE if not doing next hop
self ?

Can it be build say in transit RR from paths received via add-paths from
PEs ?

If you talking labels ... considering each VPNv4/v6 route has a unique per
VRF RD such MNH would be not very useful anyway.

If not I am not sure what is the big gain here. Yes it could be done (with
lots of work) but I am a bit missing the need for it considering that you
can get the required functionality today already.

Is there anything it actually brings to the table other then ton of
implementation and new operational issues ?

Thx,
R.

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Sat, 3 Jul 2021 18:24:12 +0000

Pls see inline. KV2>

From: Robert Raszuk <[email-masked]>
Date: Saturday, July 3, 2021 at 4:28 AM
To: Kaliraj Vairavakkalai <[email-masked]>
Cc: Minto Jeyananth <[email-masked]>, idr@ietf. org <[email-masked]>
Subject: Re: I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
[External Email. Be cautious of content]

Hi,

Few follow up points ...

That is huge change BGP <--> RIB interaction. Note today unless you install more next hops *only* if multipath is enabled.

KV2> An implementation can chose to apply that kind of local policy knob to MNH as-well. I meant an implementation that intends to use the MNH treats the PNHs received in the MNH just like it would treat a PNH received in attr 3,14 and do nexthop-resolution (multihop) or interface-check (singlehop). And all useable legs will be installed to FIB, capped by the max-num-ECMP of the platform.

Besides you can safely do it only when network is doing encapsulation end to end. If you are doing hop by hop lookup and you do not even know if all nodes support MNH loops can form.

KV2> agree. This is intended for use in tunneled networks (bgp free core).
KV> Let us say the max-num-ECMP supported by a router is 16. Then it will install the first 16 nexthops (as per the ‘Relative-preference’ field) received in the MNH. It will not install all the nexthops. So the same existing ecmp-limit safeguards can be

Nope that was not the point ... I am talking about control plane RAM when the attacker is using MNH attribute fill to the top of the UPDATE MSG SIZE to kill your box regardless if this will get down to RIB or FIB.

KV2> That’s a good point. But the same max-num-ECMP checks can be applied during bgp update processing as-well to validate a MNH. An implementation can chose via local defaults/configs what is an acceptable max-limit for number of PNHs that can exist in a MNH. A route with an unacceptable MNH should be given ‘ignore the MNH-attribute exists’ treatment. And forwarding can happen based on PNH in attr 3, 14 – in those cases.

Today prefix limit saves you from this type of attacks. When one deployed MNH with the peers it opens up his network ... I would love to see this type of threat discussed in security section.

KV2> Sure. Will discuss it more.
used. The only difference is how the nexthops are received. Instead being received in 100 routes, it is received in 1 route.

Nope ... peers today sends you the single path as best unless you allow him to do add-paths all.

KV2> I see your point about the value of having capability-negotiation. It provides another level of  control on whether to accept MNH from a peer or not, especially the EBGP-case. I will add a capability for this MNH.

KV> they could potentially get called-back when effective-igp-cost of the route changes. Pls note, this proposal does not increase number of unique PNHs in the system/network, for e.g. that is a function of how many PEs exist in the network. Just what changes is how those PNHs are conveyed in bgp-updates as ecmp/pic nexthops for a bgp-prefix. Number of routes required to convey that relationship is greatly reduced, while providing more control to the advertiser on signaling the ecmp/pic relationships.

If you allow this for EBGP you are changing the BGP game completely. Especially if you allow this for SAFI 1.

KV2> EBGP is not discounted. One simple example where this can be useful for EBGP is the inter-AS parallel-links case. Even when using one bgp-session between loopbacks over the parallel-links, the MNH can be used to specify how the load-balancing should happen on the parallel links. And this can be done on a per route level. The mechanisms that exist today can at-best provide “per nexthop set” granularity load-balancing options (configuring a static-route to the lo0 PNH with the desired load-balancing)
- - -

Btw ... who would build this MNH attribute ? ASBR/PE if not doing next hop self ?

KV2> it can be PE/ASBR with doing nexthop-self.

Can it be build say in transit RR from paths received via add-paths from PEs ?

KV2> And it can be the RR as-well in some cases which may do a “nexthop-change to an anycast-address”. RR in essence acts like a controller. The MNH actually provides better granularity controls to a controller to install routes on receiving BGP speakers. I will be presenting one such usecase in IETF111, “Scaling Seamless-MPLS networks using BGP-MPLS-Namespaces”

If you talking labels ... considering each VPNv4/v6 route has a unique per VRF RD such MNH would be not very useful anyway.

KV2> there are some scenarios where it could be. The PE sends the RD:Pfx with MNH(IngressIP1, IngressIP2) thus attracting traffic to the more-optimal ingress-Forwarding-elements. Avoiding intra-fabric hops. The mechanism was invented to solve such a scenario only. Section 2 tries to explain the use-cases in brief. I will try to go over it in some detail during the above IETF111 presentation. Also, the EBGP parallel-links usecase is not currently recorded in section 2. I will record it.

If not I am not sure what is the big gain here. Yes it could be done (with lots of work) but I am a bit missing the need for it considering that you can get the required functionality today already.

Is there anything it actually brings to the table other then ton of implementation and new operational issues ?

KV2> I do think it brings in value of better scaling and finer-grained traffic engineering controls, with some moderate implementation complexity. Only time will tell for sure. :). Thanks.

Thx,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 3 Jul 2021 21:45:56 +0200

Hi,

> KV2> EBGP is not discounted. One simple example where this can be useful
for EBGP
> is the inter-AS parallel-links case. Even when using one bgp-session
between loopbacks
> over the parallel-links, the MNH can be used to specify how the
load-balancing should
> happen on the parallel links.

And how would you detect liveness of all of those parallel links ? Static +
BFD ?

What's wrong with today's solution of using loopback for next hop +
disable-connected-check knob and point static routes to resolve such next
hop over parallel links ? Sure you can use more then one loopback here as
well and that loopback is only significant locally over EBGP boundary.

KV2> there are some scenarios where it could be. The PE sends the RD:Pfx
> with MNH(IngressIP1, IngressIP2) thus attracting traffic to the
> more-optimal ingress-Forwarding-elements. Avoiding intra-fabric hops. The
> mechanism was invented to solve such a scenario only.
>

IMO this use case is a pretty bad idea. When such interface IngressIP1 or
IP2 goes down you need to repair the encapsulation all the way in the edge
instead of just performing local repair to get to the same loopback in
milliseconds. It almost smells like some solution to fix platform
deficiency in deployments of BGP as an IGP.

Tx,R.

---

From: Gyan Mishra <[email-masked]>
Date: Sat, 3 Jul 2021 16:55:40 -0400

Hi Kaliraj

Responses in-line.

There was a discussion recently on RFC 7911 add path related to the tie
breaker being missed in the RFC and possibly use path-id or Robert’s
fast-connection-restore path possible revival edge discriminator attribute.

https://datatracker.ietf.org/doc/html/draft-pmohapat-idr-fast-conn-restore

https://mailarchive.ietf.org/arch/msg/idr/nyFgEq6e2XLBganie_gVG70UDyM/

As your draft in the abstract mentions add path tie breaker in the second
paragraph may want to read through this thread.

Thank you

Gyan

On Sat, Jul 3, 2021 at 5:39 AM Kaliraj Vairavakkalai <[email-masked]>
wrote:

> Hi Gyan,
>
>
>
> Thank you for the questions and comments. Apologies - I somehow missed
> these emails with your questions, hence the delay in response.
>
>
>
> *From: *Gyan Mishra <[email-masked]>
> *Date: *Friday, June 11, 2021 at 7:27 PM
> *To: *Robert Raszuk <[email-masked]>
> *Cc: *Kaliraj Vairavakkalai <[email-masked]>, idr@ietf. org <
> [email-masked]>, Minto Jeyananth <[email-masked]>
> *Subject: *Re: [Idr] I-D Action:
> draft-kaliraj-idr-multinexthop-attribute-01.txt
>
> *[External Email. Be cautious of content]*
>
>
>
>
>
> Hi Kaliraj
>
>
>
> This feature parallels add path but is different in the way it is deployed
> as it a BGP optional transitive path attribute as opposed to a add path
> capability code which is exchanged P2P between PE-RR peers in a controlled
> fashion.  In that manner the add paths is generally used within an AS for
> BGP PIC advertisement of backup paths as well as for BGP multipath load
> balancing within an AS.
>
>
>
> The gap this draft fills is clear to be able to provide a more intelligent
> add path advertisement feature.  This also fills a very important gap of
> unequal cost load balancing as in a topology may not have equal cost cost
> metric for iBGP tie breaker, thus one path ends up being the best path even
> though multiple paths exist with add paths even with unique RD, which makes
> it difficult for iBGP load balancing to occur.
>
>
>
> I support the development and progression of this draft as this is an
> import solution to a real world problem of VPN overlay iBGP unequal cost
> load balancing within the operator core as well as uniform predictable load
> balancing.
>
>
>
> KV> Thank you.
>
>
>
> Is the intent to use within an AS and if so making the attribute non
> transitive I think  is a MUST as the next hop is changed for external
> peering the propagation of the multinexthop attribute is irrelevant.   I
> think you mentioned next-hop-self which is set on PE-RR peering at the
> domain ingress node can be used to stop the MNH from being propagated.
> However, the next-hop-self is done in ingress inbound upon entering the
> domain and not leaving the domain so I think the MNH can still get
> propagated outside the domain.
>
>
>
> KV> I had chosen ‘optional transitive’ to be able to pass thru RRs that
> don’t support it. But we also need to scope it within an AS and not allow
> it to pass ebgp boundary. And I agree with you the only way to do it for
> old-speakers is to make it ‘optional non-transitive’. I will correct that.
>
>  Gyan> Perfect
>
> Section 3.1.1 described interaction with top level mandatory transitive
> next hop attribute code 3 and the BGP Update MP_REACH next hop code 14.
>
>
>
>  Gyan> Ack
> So the way I read this is the optional transitive MNH attribute is copied
> into code 3 and 14.  So if there are 10 NH in the PNH they all get written
> into top level code 3 and BGP update code 14.
>
>
>
> KV> no. only the following advertising-pnh field has the same address as
> the attr 3, 14 nexthop field when advertising with nexthop-self.
>
>  Gyan> Understood
>
> KV> The formatting in 01 version of the draft is messed up, I’ll correct
> it. Pls refer to the 00 version which has proper formatting.
>
>
>
> Since this is for “BGP free core” the P routers where Label swap
> forwarding semantics happens is not applicable as we don’t run BGP.  As the
> forwarding semantics is not applicable after the VPN label “push” label
> imposition CE to PE eBGP peering entering the core or “pop and forward”
> label disposition PE to CE eBGP peering leaving the core.
>
> All the next hops have a VPN label imposed which is the next-hop-self from
> each PE, which I think would have only the “forwarding” semantics
> applicable, so I think that is the only one that would apply for MNH iBGP
> load balancing.
>
>
>
>         If label is used in both rfc8277 bgp-nlri and the MNH 3.4.2, then
> it would mean the bgp-nlri imposed mpls-label will be pushed on top of the
> 3.4.2 specified label.
>
>         The main use of 3.4.2 is to be used with inet-unicast routes. That
> will equate inet-unicast routes with inet-vpn/inet-lu routes. Because using
> this now we can bind a mpls-label to a inet-unicast service route also.
>
>         So that all these service families are treated equal. This helps
> in doing option-B for internet routes as-well, avoiding service-route scale
> on ASBR BNs.
>
>         And yes, ECMP/PIC/WECMP will happen based on information received
> in “Nexthop Forwarding Semantic TLV”. Thanks.
>
>  Gyan> Ack.  As another possible use case with RSVP-TE could be the per
> VRF to TE mapping feature which required multiple loopbacks for the next
> hop rewrite for steering over different next hop paths which is not
> scalable.  SR filled the gap nicely with SR-TE per VRF steering and per
> micro and macro flow steering capabilities.  This MNH draft could make per
> VRF mapping to TE tunnel more scalable as it now does not require a
> discrete loopback per next hop and static routing over the tunnel making
> the solution more scalable.  Also with MNH RSVP-TE maybe able to provide
> SR-TE like per flow traffic steering.  This could help bridge the gap
> further for operators not able to migrate yet to SR they can now reap some
> of the SR steering benefits and can extend the life of RSVP-TE or decide to
> stay with RSVP-TE.
>
Have you thought about how this draft can work together or  complement the
> CT draft?

   Also maybe how this draft for load balancing use cases can work with BGP
link bandwidth extended community and other load balancing drafts below.

https://datatracker.ietf.org/doc/html/draft-ietf-idr-link-bandwidth-07

https://datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03

https://datatracker.ietf.org/doc/html/draft-ietf-bess-evpn-unequal-lb

Kind Regards
>
>
>
> Gyan
>
>
>
> On Fri, Jun 11, 2021 at 9:10 AM Robert Raszuk <[email-masked]> wrote:
>
> HI
>
>
>
> Two more questions ...
>
>
>
> First indeed, the attribute may be propagated by BGP speakers which do not
> understand it, but that in itself is not an error. In those cases partial
> bit is set but attribute is still valid.
>
>
>
> This is also completely orthogonal to setting the next hop self on the
> path when propagating for example across EBGP.
>
>
>
>
>
> 9. You are providing a lot of analogy to Add-Paths. But in Add-Paths thx
> to capability negotiation it is mandatory for receiving speaker indicating
> support to act on it. Here you have chosen for some reason blind
> propagation as optional transitive which perhaps may be ok for networks
> which do end to end encapsulation, but I don't think it is going to work in
> pure IPv4/IPv6 hop by hop lookup - especially in the cases of mixed network
> elements some supporting the attribute and some not.
>
>
>
> Thx,
> R.
>
>
>
>
>
>
>
>
>
>
>
> On Fri, Jun 11, 2021 at 12:09 PM Robert Raszuk <[email-masked]> wrote:
>
> Dear authors,
>
>
>
> I have read yr draft with interest as some perhaps recall we have
> discussed this topic in the past number of times.
>
>
>
> Cosmetics:
>
>
>
> 1. First nit - why do you say label must be 3107bis label ? (3.4.2.
> Labeled IP nexthop) MPLS label is a label and I am not sure how one method
> of label distribution matter here.
>
>
>
> 2a. What is PNH ? It first occurs in section 3 as "PNH-Len" or
> "PNH-address", but it is never explained in the draft. Is this Path Next
> Hop ?
>
>
>
> 3. In section 3.1.1 you describe how to assure that NH in MP_REACH is also
> part of MultiNext Hop Attribute. But I do not see any discussion on how to
> treat or ignore next hops in MP_REACH when a new attribute is present and
> is valid.
>
>
>
> 4. In section 3.1.2 you define behaviour of RR advertising paths from non
> MultiNexthop paths and those which carry new attributes ... But you should
> make it clear that this is only about 9.1.2.2 step in best path selection
> (or candidate selection). There can be other criteria before we even get to
> that step.
>
>
>
> 5. Now the most important question - how do you plan to handle atomic
> withdraws ? I assume the plan is to readvertise the path with MNH - the
> removed next hop. So by implicit withdraw this next hop will be removed.
> The draft is silent on this. Now if the removed next hop was selected by
> some receivers as best (due to 9.1.2.2) and it was not arriving as part of
> MP_REACH this proposal require significant implementation changes on how
> BGP best path selection is triggered, how it runs, how it populates results
> to the RIB/FIB. I think a new section is needed in detailed discussing the
> withdraws.
>
>
>
> 6. How do you envision max-prefix safety knobs to work here ? On the
> surface it may seem orthogonal - but it is not. Today folks use this to
> protect infrastructure from for example operator's mistakes. Here one
> received path may fill the MNH attribute with 100s of next hops and as
> being optional and transitive will be distributed to all routers all over
> the world.
>
>
>
> 7. Observe that metric to next hops is dynamic. So some implementations
> capable of next hop tracking register with RIB all next hops and each time
> metric changes they get a call back. Here we are effectively talking about
> exploding this 10x or 100x or more ...
>
>
>
> Many thx,
>
> Robert
>
>
>
>
>
> From: <[email-masked]>
> To: <[email-masked]>
>
>
>
>
> https://datatracker.ietf.org/doc/draft-kaliraj-idr-multinexthop-attribute/
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-kaliraj-idr-multinexthop-attribute/__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRLDTqKGj$>
>
> https://datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRLYOeXy-$>
>
> https://www.ietf.org/rfcdiff?url2=draft-kaliraj-idr-multinexthop-attribute-01
> <https://urldefense.com/v3/__https:/www.ietf.org/rfcdiff?url2=draft-kaliraj-idr-multinexthop-attribute-01__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRIE_FnU0$>
>
>
> ftp://ftp.ietf.org/internet-drafts/
> <https://urldefense.com/v3/__ftp:/ftp.ietf.org/internet-drafts/__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfROJyomj_$>
>
>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i-d-announce
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/i-d-announce__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfREhLtYSR$>
> Internet-Draft directories: http://www.ietf.org/shadow.html
> <https://urldefense.com/v3/__http:/www.ietf.org/shadow.html__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRNd45oQu$>
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
> <https://urldefense.com/v3/__ftp:/ftp.ietf.org/ietf/1shadow-sites.txt__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfROCmEIB8$>
>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRAjWelEr$>
>
> --
>
>
> <https://urldefense.com/v3/__http:/www.verizon.com/__;!!NEt6yMaO-gk!Tp_AgObe4LEZZKd8evxc9YHtDdlil74QLaAsjA-TqTnhbmWFCxhJPUlfRJoHyeCI$>
>
> *Gyan Mishra*
>
> *Network Solutions Architect *
>
> *Email [email-masked]* <[email-masked]>
>
> *M 301 502-1347*
>
>
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Tue, 6 Jul 2021 18:00:20 +0000

> And how would you detect liveness of all of those parallel links ? Static + BFD ?

BFD to the BGP nexthop elements. (instead of the bgp-peer address).

> What's wrong with today's solution of using loopback for next hop + disable-connected-check knob and point static routes to resolve such next hop over parallel links ? Sure you can use more then one loopback here as well and that loopback is only significant locally over EBGP boundary.

Firstly, when not using ebgp-multihop and doing N parallel-links and N ebgp-sessions over them to be able to do ECMP/WECMP. In this model (call it model A), the RIB-size is increased N times, where N is the number of parallel links. With N=4 or 8 this may be tolerable today. But when N goes higher 128, 512, this model is un-manageable. So we need to go ebgp-multihop between the loopbacks (model B). But then we need an intermediate static route to a) provide connectivity and b) to resolve the BGP PNH over. And in that scenario, following problems exist for the “nexthop resolution (b) part”:

  *   Different service-routes cannot take different set of parallel-links. This would have been possible earlier in model-A.
  *   All service-routes need to share the same load-balancing treatment that is given to the static-route to the lo0. IOW, the per-service route different WECMP treatment that was earlier possible with model-A can no longer be guaranteed.

So  in essence, we don’t get the same functionality as model-A. Further,

  *   The EBGP routes use nexthop-resolution, because now the BGP PNHs are not directly connected. Which is not bad, but is additional load on the system. May be desirable to avoid.
  *   Traffic restoration on link-failure is no longer “a local-repair event confined to Forwarding-plane” because the PNH are not directly connected interface-addresses. The control-plane has to assist in detecting some unreachability events.
  *   Some platforms may not support preserving nexthop chain/hierarchy in forwarding plane, in which case convergence will take longer. All these problems are avoided if we just use a ECMP(Unlist nexthop) of directly-connected nexthops in forwarding-plane.

When using MNH (model C), we retain the nexthop-formation properties of model A, along with the RIB-size reduction properties of model-B. Use one ebgp-multihop session, and advertise N (e.g. 512) nexthops in the MNH attribute for a route, and 8 nexthops for another route. And each route can have its own ECMP/WECMP/Relative-Preference specified. So operator has better controls on which service routes to give what treatment on the ASBR boundary, thus can continue to monetize (like in model-A) those services being offered with reduced RIB-size consumption.

2> When such interface IngressIP1 or IP2 goes down you need to repair the encapsulation all the way in the edge instead of just performing local repair to get to the same loopback in milliseconds.

No. The repair happens at the nearest BN called the Forwarding-Helper. So the repair happens in sub-second. And actually this is orthogonal to whether we use IngressIP or lo0. In this case the IngressIP used as PNH are not visible outside this region at-all. So there is no way a convergence event propagates until the other edge-PE. The BN does a label-swap to the ingressIP to make forwarding happen. So it absorbs any failure event within this region.

And the BN needs more optimal forwarding information, such that intra-fabric hops can be avoided. This was a virtualization solution, where a physical box was split into multiple Virtual-Network-Functions. But the same model is followed when multiple physical boxes are involved too (e.g. N PEs and a pair of BNs in a region), and they are represented by one Anycast-nexthop to the rest of the network. The real PE-loopbacks are not visible outside their home region, thus reducing network-wide transport-layer scale.

Thanks
Kaliraj
Date: Saturday, July 3, 2021 at 12:46 PM

Hi,

> KV2> EBGP is not discounted. One simple example where this can be useful for EBGP
> is the inter-AS parallel-links case. Even when using one bgp-session between loopbacks
> over the parallel-links, the MNH can be used to specify how the load-balancing should

And how would you detect liveness of all of those parallel links ? Static + BFD ?

What's wrong with today's solution of using loopback for next hop + disable-connected-check knob and point static routes to resolve such next hop over parallel links ? Sure you can use more then one loopback here as well and that loopback is only significant locally over EBGP boundary.

KV2> there are some scenarios where it could be. The PE sends the RD:Pfx with MNH(IngressIP1, IngressIP2) thus attracting traffic to the more-optimal ingress-Forwarding-elements. Avoiding intra-fabric hops. The mechanism was invented to solve such a scenario only.

IMO this use case is a pretty bad idea. When such interface IngressIP1 or IP2 goes down you need to repair the encapsulation all the way in the edge instead of just performing local repair to get to the same loopback in milliseconds. It almost smells like some solution to fix platform deficiency in deployments of BGP as an IGP.

Tx,R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 6 Jul 2021 20:32:29 +0200

> BFD to the BGP nexthop elements. (instead of the bgp-peer address).

Is there any BFD draft describing this ?

it model A), the RIB-size is increased N times, where N is the number of
> parallel links. With N=4 or 8 this may be tolerable today. But when N goes
> higher 128, 512, this model is un-manageable.
>

Few emails back you said: "KV> Let us say the max-num-ECMP supported by a
router is 16." Now you are bringing a case where we have 512 ECMP paths
.... Is this realistic deployment where anyone including DC fabric would
connect a pair of routers with 512 parallel links for ECMP ???

So we need to go ebgp-multihop between the loopbacks (model B). But then we
> need an intermediate static route to a) provide connectivity and b) to
> resolve the BGP PNH over. And in that scenario, following problems exist
> for the “nexthop resolution (b) part”:
>
>
>
>    - Different service-routes cannot take different set of
>    parallel-links. This would have been possible earlier in model-A.
>
> No it is possible in model B. Just use few groups each with different
loopback.

>    - All service-routes need to share the same load-balancing treatment
>    that is given to the static-route to the lo0. IOW, the per-service route
>    different WECMP treatment that was earlier possible with model-A can no
>    longer be guaranteed.
>
> It is possible in model B. Just use few groups each with different
loopback.

2> When such interface IngressIP1 or IP2 goes down you need to repair the
> encapsulation all the way in the edge instead of just performing local
> repair to get to the same loopback in milliseconds.
>
>
>
> No. The repair happens at the nearest BN called the Forwarding-Helper. So
> the repair happens in sub-second. And actually this is orthogonal to
> whether we use IngressIP or lo0. In this case the IngressIP used as PNH are
> not visible outside this region at-all. So there is no way a convergence
> event propagates until the other edge-PE. The BN does a label-swap to the
> ingressIP to make forwarding happen. So it absorbs any failure event within
> this region.
>

I am talking about single flat IGP - single global region. We have two LSPs
one to IP1 and another to IP2. For simplicity assume no VPNs - just IPvX
routing over say MPLS.

So IP1 interface goes down.

How can any P node (not even running BGP) just swap labels from the one
going to IP1 to go to IP2 ? Based on what source of information ?

To conclude - In my view this draft requires few clearly spelled out  use
case(s).

In general the proposed encoding can be done. But IMO the cost exceeds the
practical benefit.

Thx,
R.

---

From: Kaliraj Vairavakkalai <[email-masked]>
Date: Thu, 8 Jul 2021 21:41:50 +0000

Thanks Gyan, noted.

The below thread seems to be discussing about tie-breaker during path-selection of add-path received routes.
Yes MNH abstract para 2 mentions tie-breaking on Router-ID (or originator-id) just as an example, it could tie-break on something else too, like BGP nexthop.
About this thread – yes the edge-discriminator can be useful to distinguish between two “EBGP-routes advertised in add-path”, especially in nexthop-self scenarios.
But I am not clear on what diversity the two paths will provide, if the nexthops are the same (nexthop-self) for ipv4-unicast routes. Anyway, that may be a separate discussion.

There is also another section in MNH draft that mentions add-path interaction. (https://datatracker.ietf.org/doc/html/draft-kaliraj-idr-multinexthop-attribute-01#section-3.1.2 ).
That is about the following aspect of add-path:

https://datatracker.ietf.org/doc/html/draft-ietf-idr-add-paths-guidelines-08#section-2

  Diverse path: A BGP path associated with a different BGP next-hop and

   BGP router than some other set of paths. The BGP router associated
   with a path is inferred from the ORIGINATOR_ID attribute or, if there
   is none, the BGP Identifier of the peer that advertised the path.

So, the MNH draft suggests to consider the MNH-attribute also, when doing the “different BGP next-hop” decision, to provide path-diversity. This is inline to how implementations today consider “mpls-label” also besides bgp-nexthop, for a labeled-bgp-family, when doing add-path.

May be I should clarify this section with a reference to addpath-guidelines draft.

Thanks
Kaliraj
From: Gyan Mishra <[email-masked]>
Date: Saturday, July 3, 2021 at 1:56 PM
To: Kaliraj Vairavakkalai <[email-masked]>
Cc: Minto Jeyananth <[email-masked]>, Robert Raszuk <[email-masked]>, idr@ietf. org <[email-masked]>
Subject: Re: [Idr] I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
[External Email. Be cautious of content]

Hi Kaliraj

Responses in-line.

There was a discussion recently on RFC 7911 add path related to the tie breaker being missed in the RFC and possibly use path-id or Robert’s fast-connection-restore path possible revival edge discriminator attribute.

https://datatracker.ietf.org/doc/html/draft-pmohapat-idr-fast-conn-restore<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-pmohapat-idr-fast-conn-restore__;!!NEt6yMaO-gk!QVII_XLxGBlX69rN38hKLDLicegdhAE_AwlEABNNzEWJxo9VYk5qeMZgNTIBpvUe$>

https://mailarchive.ietf.org/arch/msg/idr/nyFgEq6e2XLBganie_gVG70UDyM/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/idr/nyFgEq6e2XLBganie_gVG70UDyM/__;!!NEt6yMaO-gk!QVII_XLxGBlX69rN38hKLDLicegdhAE_AwlEABNNzEWJxo9VYk5qeMZgNZN3lqbU$>

As your draft in the abstract mentions add path tie breaker in the second paragraph may want to read through this thread.

Thank you

Gyan

On Sat, Jul 3, 2021 at 5:39 AM Kaliraj Vairavakkalai <[email-masked]<mailto:[email-masked]>> wrote:
Hi Gyan,

From: Gyan Mishra <[email-masked]<mailto:[email-masked]>>
Date: Friday, June 11, 2021 at 7:27 PM
To: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Cc: Kaliraj Vairavakkalai <[email-masked]<mailto:[email-masked]>>, idr@ietf. org <[email-masked]<mailto:[email-masked]>>, Minto Jeyananth <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Idr] I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
[External Email. Be cautious of content]

Hi Kaliraj

KV> Thank you.

   attribute.”

KV> For vpn-family routes we could continue to carry the label in rfc8277 bgp-nlri. And use MNH with 3.4.1.
        If label is used in both rfc8277 bgp-nlri and the MNH 3.4.2, then it would mean the bgp-nlri imposed mpls-label will be pushed on top of the 3.4.2 specified label.
        The main use of 3.4.2 is to be used with inet-unicast routes. That will equate inet-unicast routes with inet-vpn/inet-lu routes. Because using this now we can bind a mpls-label to a inet-unicast service route also.
        So that all these service families are treated equal. This helps in doing option-B for internet routes as-well, avoiding service-route scale on ASBR BNs.
        And yes, ECMP/PIC/WECMP will happen based on information received in “Nexthop Forwarding Semantic TLV”. Thanks.
 Gyan> Ack.  As another possible use case with RSVP-TE could be the per VRF to TE mapping feature which required multiple loopbacks for the next hop rewrite for steering over different next hop paths which is not scalable.  SR filled the gap nicely with SR-TE per VRF steering and per micro and macro flow steering capabilities.  This MNH draft could make per VRF mapping to TE tunnel more scalable as it now does not require a discrete loopback per next hop and static routing over the tunnel making the solution more scalable.  Also with MNH RSVP-TE maybe able to provide SR-TE like per flow traffic steering.  This could help bridge the gap further for operators not able to migrate yet to SR they can now reap some of the SR steering benefits and can extend the life of RSVP-TE or decide to stay with RSVP-TE.
Have you thought about how this draft can work together or  complement the CT draft?

   Also maybe how this draft for load balancing use cases can work with BGP link bandwidth extended community and other load balancing drafts below.

https://datatracker.ietf.org/doc/html/draft-ietf-idr-link-bandwidth-07<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-idr-link-bandwidth-07__;!!NEt6yMaO-gk!QVII_XLxGBlX69rN38hKLDLicegdhAE_AwlEABNNzEWJxo9VYk5qeMZgNX2D8fnY$>

https://datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-mohanty-bess-ebgp-dmz-03__;!!NEt6yMaO-gk!QVII_XLxGBlX69rN38hKLDLicegdhAE_AwlEABNNzEWJxo9VYk5qeMZgNedxy3FA$>

https://datatracker.ietf.org/doc/html/draft-ietf-bess-evpn-unequal-lb<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-bess-evpn-unequal-lb__;!!NEt6yMaO-gk!QVII_XLxGBlX69rN38hKLDLicegdhAE_AwlEABNNzEWJxo9VYk5qeMZgNbsQ2ON4$>

Kind Regards

Gyan

Two more questions ...

Thx,
R.

Cosmetics:

Tech:

Many thx,
Robert

---------- Forwarded message ---------
From: <[email-masked]<mailto:[email-masked]>>
Date: Fri, Jun 11, 2021 at 10:56 AM
Subject: I-D Action: draft-kaliraj-idr-multinexthop-attribute-01.txt
To: <[email-masked]<mailto:[email-masked]>>

Abstract:

Gyan Mishra

M 301 502-1347

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!NEt6yMaO-gk!QVII_XLxGBlX69rN38hKLDLicegdhAE_AwlEABNNzEWJxo9VYk5qeMZgNed4hVOs$>

Gyan Mishra

M 301 502-1347