Subject: [Idr] draft-chen-bgp-redist - A Juniper Perspective

From: Jeffrey Haas <[email-masked]>
Date: Fri, 20 Aug 2021 15:22:01 -0400

Enke & Jenny,

While I'm inclined to say that the matters described in
draft-chen-bgp-redist aren't a problem, it's probably more reasonable to say
"this isn't a problem for everyone".

I'm not going to debate what other implementations do.  Instead, I'll spend
the much of my message discussing what the implementation I work on does.
This does leave us in an interesting headache as a working group whether
there's more general work to do here that merits discussion as an RFC.

----

One thing I will contribute toward some possible need for discussion is the
sloppy ground that RFC 4271 had with regards to interactions with non-BGP
protocols.  We had quite a difficult time getting the text for this "right"
and there was multiple pushes to try to NOT discuss non-BGP protocols at
all.  Such discussions got very deep in the internal details of various
implementations.  What we did end up with in RFC 4271 was the following:

The BGP RIBs are a distinct entity from the "Routing Table".

Section 9.1.2 places the best BGP route in the Loc-RIB.
It also says, "Whether the new BGP route replaces an existing non-BGP route
in the Routing Table depends on the policy configured on the BGP speaker."

Section 9.1.3 for Route Dissemination then says the following:
 :    The Phase 3 decision function is invoked on completion of Phase 2, or
 :    when any of the following events occur:
 : [...]
 :       b) when locally generated routes learned by means outside of BGP
 :          have changed
 : [...]
 : 
 :    All routes in the Loc-RIB are processed into Adj-RIBs-Out according
 :    to configured policy. 

So... there's not a lot of normative text there.  The last sentence above,
"configured policy", is the wiggle room the RFC has to say "I'm originating
a route".  

The general call in the RFC to not advertise things you can't forward to
also provides some level of wiggle room that the best route in the Routing
Table is what should be redistributed:
 :    A route SHALL NOT
 :    be installed in the Adj-Rib-Out unless the destination, and NEXT_HOP
 :    described by this route, may be forwarded appropriately by the
 :    Routing Table.

Anyway, the text is sloppy and discussions about how redistribution is done
and whether that is shown in the Loc-Rib view or as an override to the
rib-out manifests in discussions in things like BMP or the Yang modules.
It's still not as cleanly settled as it should be.

Offering my opinion on this abstract model, I've tended to think about a
better route in the Routing Table (e.g. a static route with a better admin
distance/preference) as being selected into the Loc-Rib by considering it a
route injected via a virtual Adj-Rib-In.  But that's just my personal
justification.

----

Juniper's implementation roughly works like the following.  Since it's
derived from GateD heritage, many similar implementations will behave in a
similar fashion.

In a single routing table, there is a total ordering on all contributors to
a given destination.

The highest level of ordering is "preference", which roughly corresponds to
admin-distance.

When routes are BGP routes, the BGP routes are ordered based largely on
standard RFC rules.  The deviations from those rules are the usual vendor
deviations from standards based on when the later standards were published,
and the mix of features the operator wants.  Examples of deviations are
whether RFC 5004 is used for temporal vs. router-id based deterministic
tie-breaking.

BGP routes vs. non-BGP routes are not directly comparable, even when they
have the same preference, but we have criteria that permits them to be
selected deterministically.  

For non-BGP routes vs. non-BGP routes of two different protocols, we
similarly will select things deterministically, but may be willing to use
particular properties of the routes that may be comparable; e.g. metric.

Much of this process is documented here:
https://www.juniper.net/documentation/en_US/junos/topics/reference/general/routing-protocols-address-representation.html

For the described scenarios in draft-chen-bgp-redist, JUNOS doesn't have
determinism issues.

In our more recent multi-threaded BGP mode, we at one point in the design
had an issue somewhat related to the one described in the draft.  In our
implementation, we address the issue by always making the Routing Table and
the BGP thread always exchange the best route in the local total-ordering.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 20 Aug 2021 22:48:43 +0200

Hi Jeff,

I think you have very well described and summarized both 4271 loose
language as well as local behaviour of Junos/GateD based BGP
implementations.

But my reading and some off-list discussions with Enke about this draft
rather indicate that the crux of the issue is how to *signal* the
preference across BGP speakers to be consistent with intended network wide
behaviour.

I am not able to find your opinion on such signalling (say in the form of
LOCAL PREF) or alternative options which would allow to work network wide.

Many thx,
R.

On Fri, Aug 20, 2021 at 9:22 PM Jeffrey Haas <[email-masked]> wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Enke Chen <[email-masked]>
Date: Fri, 20 Aug 2021 14:27:40 -0700

Hi, Jeff:

Thanks for the pointer to JUNOS's BGP path selection. It's great that the
Gated/JUNOS already has some consideration of the "admin-distance" /
"preference" in BGP. That should take care of the case described in Sect.
2.1 ("On a single router").  It's not clear to me, though, whether and how
the cases described in Sec. 2.2 ("Network-wide behavior") are handled.

This is just a quick comment. I am still trying to digest your email.

-- Enke

On Fri, Aug 20, 2021 at 12:22 PM Jeffrey Haas <[email-masked]> wrote:

> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.juniper.net_documentation_en-5FUS_junos_topics_reference_general_routing-2Dprotocols-2Daddress-2Drepresentation.html&d=DwIBAg&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=OPLTTSu-451-QhDoSINhI2xYdwiMmfF5A2l8luvN11E&m=-dwyFCMbZS-tuTtu3oL1HNFBH-2mpqRlUm9F7wDkv6g&s=R7AEen-1wMEbbOC4rK4EjfOnMRVPTfQpMnYKMWKGmAA&e=
>

---

From: Jeff Tantsura <[email-masked]>
Date: Fri, 20 Aug 2021 15:59:38 -0700

Jeff,

Many thanks for the analysis and detailed explanation of Junos behavior.
It would be great if other vendors would do a similar work and post to the list.

Perhaps RFC5004/router-id (in densely meshed eBGP topologies) behavior  deserves a good discussion as well.

Cheers,
Jeff

> On Aug 20, 2021, at 14:28, Enke Chen <[email-masked]> wrote:
> 
> 
> Hi, Jeff:
> 
> Thanks for the pointer to JUNOS's BGP path selection. It's great that the Gated/JUNOS already has some consideration of the "admin-distance" / "preference" in BGP. That should take care of the case described in Sect. 2.1 ("On a single router").  It's not clear to me, though, whether and how the cases described in Sec. 2.2 ("Network-wide behavior") are handled.
>

---

From: Jeffrey Haas <[email-masked]>
Date: Sat, 21 Aug 2021 12:26:29 -0400

Enke,

> the cases described in Sec. 2.2 ("Network-wide behavior") are handled.

In terms of route selection, the behavior remains consistent.  If you've set
your route's preference to be better from BGP rather than than from your
static route (your section 2.2), the better preference will win.

However, I think what you're actually asking is implied in your draft but
perhaps in need of some additional text: You want admin distance to have
domain-wide meaning.

I find this problematic.

My two high order concerns with the solution space discussed in the draft are:

1. Section 3.3, consistent admin distance across the deployment.  (At least
   on a per-prefix basis.)  
2. LOCAL_PREF is the best way to de-pref some classes of backup route.

The values various implementations use for their admin-distance style
behavior are DEEPLY embedded in the software.  While those implementations
usually give some level of room to adjust the values in question, doing so
in a multi-vendor fashion is likely to be challenging at best.  Given such
challenges, it's almost an argument that there should be a separate value
used for this tuning that is vendor-neutral.  I don't think this is likely
to be a good answer.  Let's come back to this point in a moment.

In current BGP route selection, LOCAL_PREF is a very heavy hammer to use on
route preferencing.  For all practical purposes, the minute one of your
contributing routes has a worse local preference than other otherwise
comparable routes, it's out of the game.  If this is your intent, as I think
your draft scenarios are largely tailored for, this works fine.  But I
suspect you'll find most operators usually have 2, maybe 3 values of
LOCAL_PREF they want to use because of this property.

In situations where you may want individual routers to make specific choices
for de-preferencing a route, that decision may be local.  Even so, that
local decision's result may want to be otherwise comparable for the rest of
the deployment.  

I think your answer to this point is, "No adjustment to local-preference is
necessary".  But if that local decision is made based on some form of
admin-distance and the procedures you're proposing are in place, you now
have exception cases to write code or policy for.

Where these two things lead us is what I think your not-quite-stated desire
is in this draft: You want a level of total-ordering of route selection
across the deployment, and you want admin-distance to be one of your inputs
to that.

This is doable with LOCAL_PREF, so I agree with that observation from your
proposal.

This should theoretically be implementable with nothing more than policy.
And this is the kind of thing that operators do on a regular basis.

In the implementations you discuss in your draft, the lack of local
determinism in the redistribution mechanisms is giving you an interesting
headache.  Figuring out a useful remediation for that is worth doing.  In my
opinion, it'd be better to fix the issue of local determinism without
forcing that into the local-pref - but I realize that changing existing
implementations with other non-obvious consequences is challenging at best.

-----

If you accept the above discussion, what do I think you should do about my
comments?

- Update the draft to focus a bit more on discussion of total ordering of
  routes across a deployment.  
  + Reach out to the grow WG, NANOG, etc. for their comment on this goal and
    how this intersects with their own operational practices.
- Update your procedures such that you discuss the end-goal of achieving a
  set of ordered LOCAL_PREF that matches the intent of the total-ordering
  for the different classes of routes by your metric-of-intent.
  (admin-distance, as an example).
  + Discuss how this mapping operation may or may not be able to be
    accomplished.  In some cases, math on a metric may work, but in a
    multi-vendor environment a local mapping operation may be auto-magic or
    potentially hand-coded policy may be best.
- Make the draft informational status.
  + Depending on how discussion proceeds, this is perhaps a better fit for
    grow than idr?

Interesting optional discussion: Do we finally tighten up our RFC 4271
language to discuss what redistribution actually looks like?  We end up in
interesting situations from this ambiguity every few years or so.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Sat, 21 Aug 2021 12:28:40 -0400

Jeff,

> Perhaps RFC5004/router-id (in densely meshed eBGP topologies) behavior  deserves a good discussion as well.

A topic I've discussed Sue in the past is we're in need of some documents
discussing some BGP protocol intrinsics that really aren't well documented.

The one you mention is an example of "your domain really needs to implement
consistent route selection procedures".

This is also a reminder that I have errata for RFC 5004 to file and keep
forgetting to do so.

-- Jeff

---

From: Jeff Tantsura <[email-masked]>
Date: Sun, 22 Aug 2021 14:13:17 -0700

Jeff

Thanks again!
In the draft-lapukhov-bgp-ecmp-considerations we have touched upon RFC5004 and some implications in ECMP fabrics, in the future versions we will expand the topic and issues associated (this is quite focused on DC use cases though - I believe is summary - we observe > 50% BGP deployments).
I’d encourage the vendors among us to share some data(p2p is ok) wrt implementations, potential interactions with router-id tie-breaking and if could be shared publicly - implications wrt HW resources (ECMP groups and similar).

Cheers,
Jeff

> On Aug 21, 2021, at 09:28, Jeffrey Haas <[email-masked]> wrote:
> 
> Jeff,
>