Subject: [Idr] comments on draft-ietf-idr-rs-bfd

From: Nick Hilliard <[email-masked]>
Date: Tue, 01 Aug 2017 14:07:42 +0100

>    2.  Client routers must have a means of communicating the knowledge
>        of the failure (and restoration) back to the route server.

I don't see why this is necessary.

It would be useful to have a mechanism for a RS to broker bfd
connectivity between rsclients, but that's about as far as it should go.
 Having this information relayed back to and processed by the RS itself
is both unnecessary and unwise.

It's unnecessary because if there is a mechanism available to broker bfd
sessions between rsclients, then this allows each rsclient to make their
own decision about whether they want to use this and make their own
decisions about NH reachability.  If this is in place, the RS doesn't
need to augment this information because the client has all the tools
they need to be able to do this.

It's unwise because:

-  the proposal allows rsclient A to influence reachability for their
competitor, rsclient B.  This is a show-stopper in my books but other
people may have different opinions.

- it puts the ixp in the position of making pronouncements about the
reachability of its rsclient base.  From a layer 9 point of view, it
could be argued that there might be liability problems if it gets this
wrong due to e.g. poisoned / late / incorrect data from rsclients.

- the draft makes an implicit assumption that reachability between each
rsclient is a ternary state, which it is not.  E.g. one of the most
common ixp commutativity failure situations is where you have an
ecmp/lag bundle between rsclient A and B, and a single member link dies,
leading to partial failure between A and B.  If BFD and BGP work fine,
then the IXP has just made a pronouncement that NH reachability is good,
which it is not.  Awkward.

- maintaining state, particularly derived state, is generally a poor
idea, particularly when based on slightly vague data inputs, and where
the consequences of either misinformation or misinterpretation of this
state would be dropped packets.

- this proposal adds a good deal of complication to the interaction
between RS, rsclients and the routing decision engine functionality.
This stuff is complicated enough already, and there would have to be
unusually clear and compelling arguments to make it worth adding in the
extra failure modes that complication like this is going to create.

- route server connectivity brokerage is generally considered to be the
poor first cousin of bilateral connectivity over an IXP and the poor
second cousin of a private network interconnection.  If an rsclient
cares enough about its data flows that reachability is sufficiently
important to justify being protected by BFD, then I'd argue that they
should just do this rather than outsourcing the problem to someone else.
 There are plenty of automation tools out there and it's not that hard
to do.

Please let's keep rs bfd brokerage as simple as possible. I don't want
front-line ixp support issues to have to be dealt with by 3rd line
engineering.

Nick

---

From: Job Snijders <[email-masked]>
Date: Tue, 1 Aug 2017 15:31:09 +0200

Hi Nick,

I'll cherry pick a bit in responding.

> people may have different opinions.

I am under the impression that the NHIB (Next Hop Information Base) is
per-client, like the Loc-RIB and Adj-RIB-Out are, so what you describe
shouldn't be an issue. See section 4.4 "The route server maintains a
per-client Next Hop Information Base".

Kind regards,

Job

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 1 Aug 2017 16:50:27 +0200

Hi Job,

Not quite ... Nick is correct - next hop reachability are global in RS.
Sure each client may have his path in his view but this is not what will
limit the next hop liveness decision.

See messages and suggestions from John on how to treat failures reported by
N clients as something more important then failures of NH reachability
reported by one or two clients. In both cases they would affect entire RS
not only those clients views.

Cheers,
R.

On Tue, Aug 1, 2017 at 3:31 PM, Job Snijders <[email-masked]> wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 1 Aug 2017 13:19:54 -0400

Robert and Job answered some of the points already.  I'll skip those.

> they need to be able to do this.

In the absence of multiple paths from the RS, the only choice the client
gets to make is use/don't use.

> wrong due to e.g. poisoned / late / incorrect data from rsclients.

Vs. announce stuff that is blackholing?  Recall we're having this
conversation for a reason.

> which it is not.  Awkward.

Already an existing issue.  Perfect? No.  Not trying to be.

> between RS, rsclients and the routing decision engine functionality.

Route selection using proxied reachability state in a RS view is hard?  No,
same as existing RS cost.

The one valid criticism is what to do when a RS view contains multiple
peers.  In such cases, this proposal is less useful except in the clear case
where all peers in the view are unable to reach the relevant nexthop.

> should just do this rather than outsourcing the problem to someone else.

The issue is fundamental to BGP third-party nexthops.  It's particularly the
case when such nexthops are available across a fundamentally NBMA
environment.

>  There are plenty of automation tools out there and it's not that hard

It's not clear what you're suggesting.

-- Jeff

---

From: Job Snijders <[email-masked]>
Date: Tue, 1 Aug 2017 20:32:39 +0200

Hi Robert,

On Tue, Aug 01, 2017 at 04:50:27PM +0200, Robert Raszuk wrote:
> Not quite ... Nick is correct - next hop reachability are global in
> RS.  Sure each client may have his path in his view but this is not
> what will limit the next hop liveness decision.
> 
> See messages and suggestions from John on how to treat failures
> reported by N clients as something more important then failures of NH
> reachability reported by one or two clients. In both cases they would
> affect entire RS not only those clients views.

This surprises me. If clients A & B report that they can't reach Z, that
doesn't imply anything about whether C can reach Z. In such a scenario
would be somewhat counter-intuitive if the RS prunes all paths with
next-hop Z from C's Loc-RIB, based on a message from client A or B.

I think I would benefit from further elaboration on what the draft's
intentions are in this regard.

Kind regards,

Job

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 1 Aug 2017 20:38:41 +0200

Hi Job,

Indeed intentions of authors of this draft in this respect would be very
useful.

I think the notion was that if few clients report that they can not reach Z
then it means that Z is down for all other clients.

So if A, B tell RS that Z is down that could trigger declaring Z as
unreachable for all especially if C does not even support the new SAFI.

The other interesting question is if Z is the only next hop for a prefix P
for a client does it get withdrawn or is left as the only resort to reach P
?

Thx,
R.

---

From: Gert Doering <[email-masked]>
Date: Tue, 1 Aug 2017 20:47:36 +0200

Hi,

> Not quite ... Nick is correct - next hop reachability are global in RS.

This would be a possible but fairly silly implementation choice.

The whole point is that reachability issues on a too-large fabric are
not universal - otherwise, the RS would lose reachability to the client
in question as well, and the whole point would be moot.

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Gert Doering <[email-masked]>
Date: Tue, 1 Aug 2017 20:46:10 +0200

Hi,

> people may have different opinions.

How so?  A can communicate "*I* can not reach Z" (or "B").  

How's that interfering with B's reachability towards and from X, Y, Z?

[..]
> which it is not.  Awkward.

This situation is neither made better nor worse by this draft.

[..]
>  There are plenty of automation tools out there and it's not that hard

Speaking as an operator and user of an IXP RS,  I value the service that
these RS bring to us.  "Plenty of automation tools" do not replace "not
having to deal with 100s of other IXP participants that cannot even get
e-mail right".

Right now, indirect reachability issues are an issue - not frequently,
but if they are, annoying and hard to see nonetheless.  Sending information
about these in an automated fashion to the IXP operator is a good thing
in my book.

> front-line ixp support issues to have to be dealt with by 3rd line

If your IXP does not want to offer this feature, then don't.

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 1 Aug 2017 21:21:50 +0200

That's the quote from John's mail on this topic:

"One observation regarding the "RR won't scale if it has to do per-client
route selection" concern -- it seems to me it would be entirely feasible
for an implementation to associate some figure-of-merit with each next hop,
a function of its reported reachability by each of its indirect peers. That
figure-of-merit could be used during traditional (not per-client) route
selection. The general idea would be "if one other client can't reach you,
that's their problem, if dozens of other clients can't reach you, maybe we
should choose a different next hop if we've got one" [*]. Think of it as
making the binary resolvability condition of RFC 4271 9.1.2.1 fuzzy.

I think specifying the function would be beyond the scope of this (or
really, any) draft since it's "just policy". But it demonstrates there's
some potential benefit from the proposal without going all the way down to
per-client route selection."

---

From: Gert Doering <[email-masked]>
Date: Tue, 1 Aug 2017 21:53:13 +0200

Hi,

> It's not clear what you're suggesting.

Nick was proposing "just do not bother using route-servers and always
set up direct peerings, using automatization to solve all the problems
caused by non-responsive / non-caring peers".

Gert Doering
        -- NetMaster

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 01 Aug 2017 21:29:28 +0100

> caused by non-responsive / non-caring peers".

here's what I'm proposing:

- it would be a great idea for route servers to broker BFD sessions

- the gain provided by implementation of proposals like this will be
measured in minutes per year.

- the NH reachability feedback mechanism defined in
draft-ietf-idr-rs-bfd is too complicated for its own good and goes well
past the point of diminishing returns

- draft-chen-bfd-unsolicited is slightly too naive an approach to this
problem and ixp participants will end up being unhappy with its results

- if the NH reachability mechanism is cut out of draft-ietf-idr-rs-bfd
and replaced with a BFD candidacy mechanism, it would provide an very
good all-round fit for most peoples' bfd requirements

- the remaining small segment of organisations whose requirements cannot
be satisfied by this should use bilateral peering sessions

- organisations who feel strongly about fast convergence and
connectivity commutativity problems at IXPs need to think very carefully
about whether route servers are the best approach for their requirements
in the first place

- these last two sets of organisations are probably clued in enough to
be able to handle their peering requirements via automation (i.e.
hooking euro-ix json schema / peeringdb / etc into tools like napalm).

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 01 Aug 2017 22:14:32 +0100

> gets to make is use/don't use.

that's an acceptable compromise.  IXPs provide alternative / nearer
paths, but never the only path.  Add-path is orthogonal to this problem:
 it would be a useful addition for providing alternatives, but the
nature of IXPs is such that it's not necessary.

> conversation for a reason.

this is a policy / optics point: the difference between the two things
is that a baseline RS announces a NH without making any pronouncement
about the NH's reachability. draft-ietf-idr-rs-bfd is making a statement
about the NH's reachability, but based on heuristics which we know will
not take into account all failure modes of an IXP fabric, particularly
some of the more common ones (i.e. partial reachability).

The baseline RS performance will not perform as well from a technical
point of view.

The draft-ietf-idr-rs-bfd could potentially open the IXP operator up to
service loss liability if a statement is made about NH reachability
which is not true from the point of view of any pair of RS clients.

[...]

> It's not clear what you're suggesting.

see my earlier reply to Gert.

Nick

---

From: Randy Bush <[email-masked]>
Date: Wed, 02 Aug 2017 15:43:52 +0900

> Not quite ... Nick is correct - next hop reachability are global in RS.

nope

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 2 Aug 2017 11:07:45 +0200

Interesting ...

Look at section 5 of yr draft:

"The next hop field is the key for the NH-Reach NLRI type".

As defined today your new NLRI encoding in MP_REACH does not have per
client distinguisher. And if it is the NLRI key as the draft states good
luck to have the next hop information it carries scoped to be per client.

Best,
R.

---

From: Gert Doering <[email-masked]>
Date: Wed, 2 Aug 2017 11:32:33 +0200

Hi,

> luck to have the next hop information it carries scoped to be per client.

It's an implemention decision on whether the RS sees this as global 
or per-client thing.

Since RSs already today do many things in ways the original BGP specs
did not envision (like, per-client bestpath calculation) doing this 
particular one globally does not sound (as said before) like the most
reasonable implementation decision.

OTOH, if the draft had a few more words on this (like, "It is RECOMMENDED
that next-hop reachability is considered a per-client value.  A route-server
MAY decide that a given next-hop should not be considered anymore for
any client if more than a configurable threshold of clients report 
unreachability.") it would help to de-confuse readers.

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 2 Aug 2017 12:49:40 +0200

Hi Gert,

Sure you can say that this is implementation detail ... but if you design
protocol right implementation becomes much easier.

Q:

Let's consider a scenario that RS client has a view with some bgp policy.
Now this client is using 1500 mtu lan and 9k mtu lan for ipv4 connectivity
via ix.

As reachability to next hops may be different for each lan are we going to
have two sessions of new safi between RS and said client ? Or are we in
this case going to put such client into two different contexts on RS ?

Cheers
R.

On Aug 2, 2017 5:32 AM, "Gert Doering" <[email-masked]> wrote:

Hi,

Gert Doering
        -- NetMaster

---

From: Gert Doering <[email-masked]>
Date: Wed, 2 Aug 2017 13:05:49 +0200

Hi,

> this case going to put such client into two different contexts on RS ?

If you have two independent LANs, I can not see primary IPv4 for two
different set of peers (aka "subnets of next-hops") go over a single 
eBGP session to a RS located outside the subnet for at least one of them.

OTOH I've never been at an IXP that has different-mtu VLANs and "joint" RS,
so I can't speak from experience here.

Seems the draft needs to document edge cases like this.

Gert Doering
        -- NetMaster