Subject: [Idr] Adoption of draft-varlashkin-bgp-nh-cost-02 as IDR WG document?

From: John Scudder <[email-masked]>
Date: Wed, 16 Nov 2011 01:06:19 -0800

Folks,

We have received a request from the authors to adopt draft-varlashkin-bgp-nh-cost-02 as an IDR WG document.  Please send your comments to the list.  The deadline for comments is December 5, 2011 at noon EST.

Thanks,

--John

---

From: Keyur Patel <[email-masked]>
Date: Wed, 16 Nov 2011 10:27:46 -0800

Support.

-Keyur

On 11/16/11 1:06 AM, "John Scudder" <[email-masked]> wrote:

> Folks,
> 
> We have received a request from the authors to adopt
> draft-varlashkin-bgp-nh-cost-02 as an IDR WG document.  Please send your
> comments to the list.  The deadline for comments is December 5, 2011 at noon
> EST.
> 
> Thanks,
> 
> --John
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: Bhavani Parise <[email-masked]>
Date: Thu, 17 Nov 2011 00:56:34 -0800

support

-Bhavani

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Thu, 17 Nov 2011 03:11:41 -0600

Support.

Cheers,
Rajiv

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
Bhavani
> Parise (bhavani)
> Sent: Thursday, November 17, 2011 3:57 AM
> To: [email-masked]
> Subject: Re: [Idr] Adoption of draft-varlashkin-bgp-nh-cost-02 as IDR
> WGdocument?
> 
> 
> support
> 
> -Bhavani
> 
draft-varlashkin-bgp-
> nh-cost-02 as an IDR WG document.  Please send your comments to the
list.  The
> deadline for comments is December 5, 2011 at noon EST.
> >
> > Thanks,
> >
> > --John

---

From: Jeff Tantsura <[email-masked]>
Date: Thu, 17 Nov 2011 04:51:28 -0500

Yes/support

Regards,
Jeff

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 17 Nov 2011 21:43:40 -0500

Support

--
Jakob Heitz.

---

From: Shane Amante <[email-masked]>
Date: Sun, 20 Nov 2011 21:37:44 -0700

If this draft is adopted as a WG doc, I would like to see some refinements wrt the following:
1)  IIUC, the draft alludes to a situation where one can either send regular IGP Cost in the NH_SAFI attribute -or- "(not necessarily IGP cost, but whatever has been chosen to be carried in NH SAFI)".  See also Appendix A.2.  Although I see the "simplicity" of wanting to go with this latter direction (non-IGP cost in NH_SAFI), I think it has the potential to create some significant operational (and/or, implementation) hazards and thus should be considered for removal.  Unless, someone can convince us that it is, after all, "safe".  More specifically, I think:
    a)  If a node -- let's say it's another RR client in the same cluster -- does not understand (hasn't implemented) NH_SAFI, then how is it supposed to receive the NH value + "special cost", which presumably is needed at a bare minimum for routes to pass BGP's NEXT_HOP reachability check?  
        i)  At the very least, this makes an assumption and puts a burden on the RR client to have, at the very least, a less specific route for the NH in it's IGP (or traditional BGP) for that purpose, which should get documented in a Operational Considerations section.
        ii) What's potentially more concerning is that if RR clients in the same cluster are configured to be fully iBGP meshed, (not using client-to-client reflection on their RR), then if one of those RR clients can't see the NH_SAFI, then it could fall back on the IGP metric and make an inconsistent path selection and lead to forwarding loops.
    b)  Can an operator stick any value he/she wants for NEXT_HOP in that field of the NH_SAFI?  Or, must the implementation only send a NEXT_HOP in NH_SAFI, when it has a valid match (IPv4 /32 or IPv6 /128?) for that NEXT_HOP in their IGP routing table?
    c)  Is the NH_SAFI + "special cost" only supposed to be configured on the RR or on the RR-clients and sent to the RR?  (I think it's the latter).  Regardless, it should be recommended that the RR-client must always send a consistent NH_SAFI + "special cost" to it's redundant set of (upstream) RR's -- or, the RR's must be configured identically with the same NH_SAFI + "special cost" -- in order to ensure consistent path selection at the redundant set of RR's.

2)  I think the document could benefit from an "Applicability Statement" section, which:
    a)  More clearly articulates that this solution is primarily intended to be used in networks whose RR's don't have full visibility to IGP costs from the perspective of each of its clients due to well-known issues with lack of IGP visibility due to Inter-Area or Inter-Level summarization.
    b)  I don't think this draft is aimed at Inter-AS use, since presumably MED's are still sufficient in that case?  Either way, that should get clarified.

3)  I also think the document could benefit from an "Operational Considerations" section, which:
    a)  Discusses whether changes to "Administrative Distance" are needed at the RR's, specifically for the NH_SAFI addresses.  Are these going to be required (manually) by Operators or automatically through the implementators of NH_SAFI so that NH_SAFI are consistently selected for NH resolution compared to traditional IGP metric costs.  For example, on Cisco iBGP AD is 200 vs. IGP's such as OSPF or ISIS that are at 110 and 115, respectively.  (There is a very brief discussion already in Section 3 that talks about using NH_SAFI instead of IGP cost, but it's not clear if this is enforced through the implementation of NH_SAFI or if it will be required of the operator to manually tune AD's to get this 'right').
    b)  Recommendations that it would probably be advisable for operators to enable and tune SPF throttling in their networks and, more importantly, that implementations of NH_SAFI only flood "new" NH_SAFI information _after_ an SPF has occurred.  This would hopefully mitigate a vicious feedback loop from the IGP cranking through lots of SPF's and shoving that info into BGP toward an, already very busy, RR.

Thanks,

-shane

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 21 Nov 2011 09:32:47 +0100

Hi Shane,

> If this draft is adopted as a WG doc, I would like to see some
> refinements wrt the following: 1)  IIUC, the draft alludes to a
> situation where one can either send regular IGP Cost in the NH_SAFI
> attribute -or- "(not necessarily IGP cost, but whatever has been
> chosen to be carried in NH SAFI)".  See also Appendix A.2.  Although
> I see the "simplicity" of wanting to go with this latter direction
> (non-IGP cost in NH_SAFI), I think it has the potential to create
> some significant operational (and/or, implementation) hazards and
> thus should be considered for removal.  Unless, someone can convince
> us that it is, after all, "safe".

The addition of any arbitrary metric dates back to the discussion on the 
list where some folks expressed the need to have ability for manually 
configured "provider's policy exit selection" from any possible PE/ASBR.

With tunneled networks (which are prerequisite for such policy) and 
minimum operational carefulness I see that much harm in allowing the 
next hop "metric" from a given PE to be derived by other then IGP metric 
means. One could be static configuration, the other could be RTT to the 
next hop, one could be geographic distances etc ...

> More specifically, I think: a)  If
> a node -- let's say it's another RR client in the same cluster --
> does not understand (hasn't implemented) NH_SAFI, then how is it
> supposed to receive the NH value + "special cost", which presumably
> is needed at a bare minimum for routes to pass BGP's NEXT_HOP
> reachability check?

I would envision this SAFI to be used between RR and the client and be 
rather applicable to inter-cluster routes.

> i)  At the very least, this makes an assumption
> and puts a burden on the RR client to have, at the very least, a less
> specific route for the NH in it's IGP (or traditional BGP) for that
> purpose, which should get documented in a Operational Considerations
> section.

I don't think this is required.

> ii) What's potentially more concerning is that if RR clients
> in the same cluster are configured to be fully iBGP meshed, (not
> using client-to-client reflection on their RR), then if one of those
> RR clients can't see the NH_SAFI, then it could fall back on the IGP
> metric and make an inconsistent path selection and lead to forwarding
> loops.

Again this tool is not so much applicable to client to client. It is RR 
to client as optimal best path selection is a feature one would enable 
on RRs only.

>  b)  Can an operator stick any value he/she wants for NEXT_HOP
> in that field of the NH_SAFI?  Or, must the implementation only send
> a NEXT_HOP in NH_SAFI, when it has a valid match (IPv4 /32 or IPv6
> /128?) for that NEXT_HOP in their IGP routing table?

The next hop must be reachable. I do not see why /32 or /128 would be 
required. While with MPLS encap those may already be there anyway with 
IP encap this is not required.

> c)  Is the
> NH_SAFI + "special cost" only supposed to be configured on the RR or
> on the RR-clients and sent to the RR?  (I think it's the latter).

Yes.

> Regardless, it should be recommended that the RR-client must always
> send a consistent NH_SAFI + "special cost" to it's redundant set of
> (upstream) RR's -- or, the RR's must be configured identically with
> the same NH_SAFI + "special cost" -- in order to ensure consistent
> path selection at the redundant set of RR's.

I agree.

> 2)  I think the document could benefit from an "Applicability
> Statement" section, which: a)  More clearly articulates that this
> solution is primarily intended to be used in networks whose RR's
> don't have full visibility to IGP costs from the perspective of each
> of its clients due to well-known issues with lack of IGP visibility
> due to Inter-Area or Inter-Level summarization.

If you consider the pure metric propagation them I would agree. However 
some may argue that even in such cases we would like to send pure metric 
with NH_SAFI as only PEs/ASBRs are in position to determine if such NHs 
are for example labels switch paths really reachable from their location 
of the network.

> b)  I don't think
> this draft is aimed at Inter-AS use, since presumably MED's are still
> sufficient in that case?  Either way, that should get clarified.

I agree.

> 3)  I also think the document could benefit from an "Operational
> Considerations" section, which: a)  Discusses whether changes to
> "Administrative Distance" are needed at the RR's, specifically for
> the NH_SAFI addresses.  Are these going to be required (manually) by
> Operators or automatically through the implementators of NH_SAFI so
> that NH_SAFI are consistently selected for NH resolution compared to
> traditional IGP metric costs.

The assumption was (perhaps too implicit) that NH_SAFI replaces the 
value which nh_metric RIB check would return for a given NH.

>  For example, on Cisco iBGP AD is 200
> vs. IGP's such as OSPF or ISIS that are at 110 and 115, respectively.
> (There is a very brief discussion already in Section 3 that talks
> about using NH_SAFI instead of IGP cost, but it's not clear if this
> is enforced through the implementation of NH_SAFI or if it will be
> required of the operator to manually tune AD's to get this 'right').

IMHO it should be automated.

> b)  Recommendations that it would probably be advisable for operators
> to enable and tune SPF throttling in their networks and, more
> importantly, that implementations of NH_SAFI only flood "new" NH_SAFI
> information _after_ an SPF has occurred.  This would hopefully
> mitigate a vicious feedback loop from the IGP cranking through lots
> of SPF's and shoving that info into BGP toward an, already very busy,
> RR.

True. NH_SAFI would flood only new information as well as it would flood 
them only on "significant" threshold change.

Many thx for your review,
R.

>> draft-varlashkin-bgp-nh-cost-02 as an IDR WG document.  Please send
>> your comments to the list.  The deadline for comments is December
>> 5, 2011 at noon EST.
>>
>> Thanks,
>>
>> --John _______________________________________________ Idr mailing
>> list [email-masked] https://www.ietf.org/mailman/listinfo/idr
>
> _______________________________________________ Idr mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/idr
>
>

---

From: Anton Elita <[email-masked]>
Date: Wed, 23 Nov 2011 17:04:43 +0100

Support

-- 
 Anton Elita
 Tinet

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Sat, 26 Nov 2011 14:33:45 +0000

Do not support.

Jim Uttaro

Ilya,

	Apologies for not responding sooner.. I do have some questions in re the draft..

1. Motivation

"ADDPATH solves this problem by letting
   route-reflector to advertise multiple paths for given prefix.  If
   number of advertised paths sufficiently big, route-reflector clients
   can choose same route as they would in case of full-mesh.  This
   approach however places additional burden on the control plane."

First of I do not anticipate having to send more than two paths for almost all cases. Secondly this is applied on a per prefix basis so there is not a doubling of routing state.. So I do not think this concern is warranted.. There are tools and guidelines for setting the number of paths see http://tools.ietf.org/html/draft-ietf-idr-add-paths-guidelines-02 for a complete description which examines a number of use cases..

"For example, if next-hop information itself has been learned via BGP then
   simple SPF run on link-state database won't be sufficient to obtain
   cost information."

The AIGP metric is used to carry the "cost" to the NH of the path.. This draft was developed to address the exact issue of carrying NH info in bgp and losing the notion of an accumulated cost.. Why is that not sufficient? 

I also do not see how you del with this except for the creation of a table of NHs and associated costs.. But how is that created? Does the new AFI/SAFI accumulate costs across multiple IPG domains?

I would like to understand what scenarios require this solution above and beyond a correct addpaths setting and/or AIGP..

2. Next-Hop Information Base

"  NHIB can be populated from various sources both static and dynamic.
   This document focuses on populating NHIB using BGP.  However it is
   possible that protocols other than BGP could be also used to populate
   NHIB."

This is a bit vague.. if you are learning NHs in BGP without the accumulated cost then the only way to populate is to statically define costs to NHs.. this seems burdensome and is not easily change to reflect the dynamic of cost due to maintenance or failure.. putting aside the difficulty in creating and maintaining this NHIB it seems that it cannot respond to changes in the topology..

3. BGP BEST PATH SELECTION MODIFICATION

Lots of chatter about the sanctity of the BGP Best path selection.. Have you spoken to folks who seem to be vehemently opposed?

4.

A general note about introducing AFI/SAFIs.. This is always not trivial as it introduces scale, control plane complexity and vulnerability.. What happens if the AFI/SAFI becomes compromised? It would be good if you could describe that here.. My assumption would be that you would fall back to igp cost?

In summary, I do not understand how addpaths and AIGP do not solve this problem. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anton Elita
Sent: Wednesday, November 23, 2011 11:05 AM
To: [email-masked] List
Subject: Re: [Idr] Adoption of draft-varlashkin-bgp-nh-cost-02 as IDR WG document?

Support

-- 
 Anton Elita
 Tinet

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 26 Nov 2011 18:21:25 +0100

James,

 > First of I do not anticipate having to send more than two paths for
 > almost all cases.

Great that you said this. As tier 1 I have 25 paths for each b_net. I 
have a path for each of my peering point with other tier1s or tier2s

How do I figure which ones are optimal for the client or group of 
clients located in the same POP from the route reflector pov ?

With add-paths or without add-paths the problem to be solved remains 
exactly the same.

Problem goes away only in the following cases:

- if you have only 1 path for all prefixes,
- if you have only 2 paths max for all prefixes,
- if you decide to send _all_ my domain external paths to all clients .. 
in the above case 25 paths ! Sorry ... bad idea.

 > The AIGP metric is used to carry the "cost" to the NH of the path..

AIGP is completely irrelevant here. In fact it is a pretty poor design 
choise to choose a path for all clients based on AIGP metric or IGP 
metric from the point of view of control plane route reflector in a 
given AS.

Keep in mind that AIGP is relevant only when number of ASes are under 
the same administration. Not that common case in the Internet :)Usually 
folks do know how to run single AS globally without any issues.

The problem which needs to be solved is to provide information to the RR 
in order for RR client to be able to receive optimal paths - optimal for 
example to follow hot potato routing requirements in a given AS.

If one would follow AIGP metric some control plane RR clients may 
clearly get not optimal exit paths from the AS they reside point of view.

Best,
R.

> "ADDPATH solves this problem by letting route-reflector to advertise
> multiple paths for given prefix.  If number of advertised paths
> sufficiently big, route-reflector clients can choose same route as
> they would in case of full-mesh.  This approach however places
> additional burden on the control plane."
>
> First of I do not anticipate having to send more than two paths for
> almost all cases. Secondly this is applied on a per prefix basis so
> there is not a doubling of routing state.. So I do not think this
> concern is warranted.. There are tools and guidelines for setting the
> number of paths see
> http://tools.ietf.org/html/draft-ietf-idr-add-paths-guidelines-02 for
> a complete description which examines a number of use cases..
>
> "For example, if next-hop information itself has been learned via BGP
> then simple SPF run on link-state database won't be sufficient to
> obtain cost information."
>
> The AIGP metric is used to carry the "cost" to the NH of the path..
> This draft was developed to address the exact issue of carrying NH
> info in bgp and losing the notion of an accumulated cost.. Why is
> that not sufficient?
>
> I also do not see how you del with this except for the creation of a
> table of NHs and associated costs.. But how is that created? Does the
> new AFI/SAFI accumulate costs across multiple IPG domains?
>
> I would like to understand what scenarios require this solution above
> and beyond a correct addpaths setting and/or AIGP..
>
> "  NHIB can be populated from various sources both static and
> dynamic. This document focuses on populating NHIB using BGP.  However
> it is possible that protocols other than BGP could be also used to
> populate NHIB."
>
> This is a bit vague.. if you are learning NHs in BGP without the
> accumulated cost then the only way to populate is to statically
> define costs to NHs.. this seems burdensome and is not easily change
> to reflect the dynamic of cost due to maintenance or failure..
> putting aside the difficulty in creating and maintaining this NHIB it
> seems that it cannot respond to changes in the topology..
>
> Lots of chatter about the sanctity of the BGP Best path selection..
> Have you spoken to folks who seem to be vehemently opposed?
>
> 4.
>
> A general note about introducing AFI/SAFIs.. This is always not
> trivial as it introduces scale, control plane complexity and
> vulnerability.. What happens if the AFI/SAFI becomes compromised? It
> would be good if you could describe that here.. My assumption would
> be that you would fall back to igp cost?
>
> -----Original Message----- From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Anton Elita Sent:
> Wednesday, November 23, 2011 11:05 AM To: [email-masked] List Subject:
> Re: [Idr] Adoption of draft-varlashkin-bgp-nh-cost-02 as IDR WG
> document?
>
> Support
>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Sun, 27 Nov 2011 01:21:20 +0000

Comments In-Line..

Jim Uttaro

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Saturday, November 26, 2011 12:21 PM
To: UTTARO, JAMES
Cc: 'Anton Elita'; [email-masked] List
Subject: Re: [Idr] Adoption of draft-varlashkin-bgp-nh-cost-02 as IDR WG document?

James,

How do I figure which ones are optimal for the client or group of 
clients located in the same POP from the route reflector pov ?
[Jim U>] I believe that there should be some determination made as to what to flood.. I wasn't actually thinking of it from a topology sense but from a perspective of which prefixes should get the enhanced service definition above and beyond a traditional internet definition.. 
[Jim U>] Hmm I guess I am not understanding this. I am under the impression that path P1 is learned from let's say 25 different peering points.. first let's examine let's examine P1 from Peer A and Peer B where these two peers are close together.. The RR serving these two POPs would forward either P1:NH=A and P1:NH=B to RR1 based on its local BGP Table that has been statically configured (Ouch) .. If RR1 also learns P1:NH=C then you want the RR1 to evaluate the BGP "Cost" and select the NH with the best IGP cost.. is that right? Ok So RR1 has statically accumulated the cost from itself to RR and to the egress peer?  Is it really your intention to create unique tables at every point in the network which would essentially be an IGP graph of the topology using the local node as the root.. That is an operational nightmare and probably a catastrophe.. Again as I stated before there is no way to adapt to a changing topology due to failure or maintenance.. can you address that.. 

It is not clear how sub-optimal the selection of P1 with addpaths=2 would be. Certainly there are other metrics i.e AS-PATH length which will immediately paths Again to our ex.. with addpaths=2 in the RR topology you would always send the two best IGP paths based on cost.. If you do this at every point how sub-optimal could it be?  Have you done an analysis of this... That would be a good addition to the draft as it would clear up any confusion in terms of addpaths applicability and the need for this draft..

If one would follow AIGP metric some control plane RR clients may 
clearly get not optimal exit paths from the AS they reside point of view.
[Jim U>] I thought you wanted an RR-client ( PE ) to select the best egress NH for a given path..Assuming a BGP 3107 design with AIGP ( pseudo IGP )

Best,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 27 Nov 2011 16:57:13 +0100

> Great that you said this. As tier 1 I have 25 paths for each b_net.
> I have a path for each of my peering point with other tier1s or
> tier2s
>

> [Jim U>] I believe that there should be some determination made as to
> what to flood.. I wasn't actually thinking of it from a topology
> sense but from a perspective of which prefixes should get the
> enhanced service definition above and beyond a traditional internet
> definition..

Since we are talking about Internet routing what is your definition of 
"enhanced service definition" vs "traditional internet definition" ? Are 
you trying to say that those who pay less do not deserve closest exit ? 
In other words if I am a customer and I did not pay premium my packets 
will end up traveling around ATT's autonomous system miles before they 
exit towards the destination ?

I think most providers today do value sending customer packets to their 
closest exit point from the point of view of PE which is receiving those 
packets rather then from the point of view of some control plane RR 
sitting somewhere in the middle or maybe on the other side of their AS. 
That's why for Internet routing RRs are still in the data path most 
often at the core to pop boundaries and POPs are fully meshed.

> [Jim U>] Hmm I guess I am not understanding this. I am
> under the impression that path P1 is learned from let's say 25
> different peering points.. first let's examine let's examine P1 from
> Peer A and Peer B where these two peers are close together.. The RR
> serving these two POPs would forward either P1:NH=A and P1:NH=B to
> RR1 based on its local BGP Table that has been statically configured
> (Ouch) ..

Ouch x 2 :)

1. If you read the IDR WG draft RRs are fully meshed and RRs exchange 
all externals paths.

2. The draft which is the subject of this thread or just IGP itself 
avoid to have any static configuration on RRs.

> If RR1 also learns P1:NH=C then you want the RR1 to
> evaluate the BGP "Cost" and select the NH with the best IGP cost.. is
> that right? Ok So RR1 has statically accumulated the cost from itself
> to RR and to the egress peer?  Is it really your intention to create
> unique tables at every point in the network which would essentially
> be an IGP graph of the topology using the local node as the root..
> That is an operational nightmare and probably a catastrophe..

LOL ... therefor I guess you will never deploy LFA as this would be a 
catastrophe too.

> Again
> as I stated before there is no way to adapt to a changing topology
> due to failure or maintenance.. can you address that..

There is no static configuration involved at all so adjusting to 
topology changes is automatic and build in. For both auto IGP 
calculation as well as for NH-SAFI there are thresholds of information 
distribution.

> It is not clear how sub-optimal the selection of P1 with addpaths=2
> would be. Certainly there are other metrics i.e AS-PATH length which
> will immediately paths Again to our ex.. with addpaths=2 in the RR
> topology you would always send the two best IGP paths based on cost..

That would be the cost from the point of view of RR. That is the 
problem. If you distribute RRs to be in each POP .. there is no problem. 
However more and more operators are moving towards control plane RRs .. 
even control plane RR in the virtual environments.

> If you do this at every point how sub-optimal could it be?  Have you
> done an analysis of this... That would be a good addition to the
> draft as it would clear up any confusion in terms of addpaths
> applicability and the need for this draft..

See above. Analysis is obvious.

> view. [Jim U>] I thought you wanted an RR-client ( PE ) to select the
> best egress NH for a given path..Assuming a BGP 3107 design with AIGP
> ( pseudo IGP )

AIGP can still be used but applied as additive metric from the point of 
view of the client not the RR.

Best,
R.

---

From: "iLya" <[email-masked]>
Date: Mon, 28 Nov 2011 12:28:22 +0100

Jim,

thanks for your review. Before I go into addressing your points, please bear 
in mind that NH SAFI is not a competing but complementing technology to 
ADD-PATH. Neither of these two can solve certain problems alone, but 
together they provide comprehensive toolkit for wide range of scenarios.

--------------------------------------------------
> > First of I do not anticipate having to send more than two paths for

To amplify Robert's response - I also have widely distributed set of ASBR's 
that peer with foreign networks in Asia, Europe and America. Currently 
they're fully meshed because ADD-PATH alone would require me either to 
deploy almost as many RR's as I have ASBR's today or to send 4 to 8 various 
path's per prefix, else I might endup sending traffic from Hong Kong to West 
Coast US via Europe or from Netherlands to Germany via East Coast US. With 
NH SAFI _plus_ ADD-PATH solution for optimum redundant routeing is to obtain 
next-hop cost from each RR-client via NH SAFI and to send best and next-best 
route using ADD-PATH.

> [Jim U>] I believe that there should be some determination made as to what 
> to flood.. I wasn't actually thinking of it from a topology sense but from 
> a perspective of which prefixes should get the enhanced service definition 
> above and beyond a traditional internet definition..

Not sure what "enhanced" means in this context, but I as service provider do 
want to send any traffic to an optimum exit points regardless how much 
customer pays because such traffic management allows me better resource 
utilisation. And with that in mind I'd like my route reflectors to advertise 
(flood) what's better from each RR-client's perspective.

> [Jim U>] Hmm I guess I am not understanding this. I am under the 
> impression that path P1 is learned from let's say 25 different peering 
> points.. first let's examine let's examine P1 from Peer A and Peer B where 
> these two peers are close together.. The RR serving these two POPs would 
> forward either P1:NH=A and P1:NH=B to RR1 based on its local BGP Table 
> that has been statically configured (Ouch) .. If RR1 also learns P1:NH=C 
> then you want the RR1 to evaluate the BGP "Cost" and

Where does static configuration come from? NH SAFI allows RR to dynamically 
obtain cost info from each of its clients, and this information may (though 
doesn't have to) be updated when network topology changes. Also note that RR 
learns new info only when client something new to tell, which is normally 
rather infrequent. It's important to realise that RR does not query RR-c 
before sending each prefix and does not wait for client reply (it starts 
using client's perspective for best path selection when it has collected all 
necessary info), respectively client does not need to wait for request from 
RR but can guesstimate which /32 or /128 prefixes in its routeing table are 
potential exit points and send that info to RR.

> select the NH with the best IGP cost.. is that right? Ok So RR1 has 
> statically accumulated the cost from itself to RR and to the egress peer? 
> Is it really your intention to create unique tables at every point in the 
> network which would essentially be an IGP graph of the topology using the 
> local node as the root.. That is an operational nightmare and probably a 
> catastrophe.. Again as I stated before there is no way to adapt to a 
> changing topology due to failure or maintenance.. can you address that..
>

Since RR-c can send info to RR at any time, clearly NH SAFI has built-in 
mechanism to adapt to changing topology. Yes, RR needs to maintain 
per-client cost-to-NH tables, but even for large networks number of entries 
in such tables is very small (<1000 for huge networks, usually less than 50 
for tier-1 class networks). On the other hand number of various best path's 
is even smaller because you'll most likely have groups of clients that share 
same preference for particular next-hops (though their absolute costs to 
such NH may vary). Considering that NH SAFI info is stored only in RAM but 
not in TCAM, and even software-based platforms today have plenty of RAM 
available, I do not foresee NH SAFI to have significant impact on RR 
capacity.

> It is not clear how sub-optimal the selection of P1 with addpaths=2 would 
> be. Certainly there are other metrics i.e AS-PATH length which will 
> immediately paths Again to our ex.. with addpaths=2 in the RR topology you 
> would always send the two best IGP paths based on cost.. If you do this at 
> every point how sub-optimal could it be?  Have you done an analysis of 
> this... That would be a good addition to the draft as it would clear up 
> any confusion in terms of addpaths applicability and the need for this 
> draft..
>

If I peer with ISP1 in 10 locations across the globe, I can have up 10 
potential next-hops. With addpath=2 I get two best paths for RR-c to choose 
from. If one my RR is located in Amsterdam and another in LA, an RR-c in 
Switzerland will get two best paths - one via LA and one via Amsterdam, but 
path to AMS goes via Germany where traffic could exit. With NH SAFI added 
RR-c in Switzerland can tell RR that Frankfurt and Amsterdam are closest so 
both AMS and LA route reflectors will furnish Swiss client again with two 
paths (using ADD-PATH) but this time via Frankfurt and via Amsterdam.

> > The AIGP metric is used to carry the "cost" to the NH of the path..

Suppose AIGP info has arrived to RR. Now RR knows in fine details what's 
cost to Prefix A, but sadly that's still from RR perspective. From RR-c 
perspective AIGP has not added any improvement.

> [Jim U>] I thought you wanted an RR-client ( PE ) to select the best 
> egress NH for a given path..Assuming a BGP 3107 design with AIGP ( pseudo 
> IGP )
>

Note that best egress NH is not only for a given path, but for a given path 
from given client perspective. In situation where you have comprehensive 
number of peerings you're still back into square one regarding selection of 
best egress point.

I hope I have addressed your concerns and we could progress with the draft 
adoption.

Kind regards,
iLya