Subject: Re: [Idr] [GROW] Potential route flap with bgp-diverse-path

From: Jakob Heitz <[email-masked]>
Date: Sun, 27 Mar 2011 15:33:42 -0400

If there are 3 paths: P1, P2 and P3
and 2 RR planes.
Plane1 advertises best, Plan2 advertizes 2nd best.

When the bestpath P1 goes away, the client can
flap to the 3rd best path, P3, before coming
back to the 2nd bset path P2.

Here is the initial condition:
P1 is best, P2 2nd best, P3 3rd best.
Plane1 advertises P1, Plane2 advertises P2

Because the advertisements from the RR planes are
not synchronized, the order of events could be:

1. P1 goes down: P2 becomes best, P3 becomes 2nd best.
2. Client receives IGP message of P1 down: Client chooses P2.
3. Client receives P3 from Plane2:         Client chooses P3.
4. Client receives P2 from Plane1:         Client chooses P2.

Possible fix: Recommend at least as many RR planes
as possible paths for any destination.

--
Jakob Heitz.

> -----Original Message-----
> From: Robert Raszuk [mailto:[email-masked]] 
> Sent: Saturday, March 26, 2011 1:48 AM
> To: Jakob Heitz
> Cc: [email-masked]
> Subject: Re: [GROW] Potential route flap with bgp-diverse-path
> 
> Hi Jakob,
> 
>  > Suppose 2 RR planes, plane1 advertises the best path
>  > and plane2 advertises the second best.
>  >
>  > If there is no second best, does plane2 advertise nothing?
> 
> Yes it would advertise overall best if backup is missing.
> 
>  > Suppose it does.
> 
> Ok.
> 
>  > Suppose there are 2 paths to a destination.
>  > P1 is best and P2 second best.
>  > Now P1 goes down.
>  > Plane1 will change its advertisement to P2.
>  > Plane2 will withdraw P2.
> 
> No.
> 
> The overall assumption is that all RRs keep all paths. So 
> plane 1 will 
> advertise P2 and plane 2 will do nothing if P1 goes down as 
> it already 
> advertised P2 to the client.
> 
>  > If the RR client receives and processes the withdrawal
>  > first, it will flap.
> 
> No withdraw will happen.
> 
> Cheers,
> R.
> 
>  > Note, the messages are asynchronous. Even if plane1
>  > sends his message first, the client can still process
>  > the message from plane2 first.
>  >
>  > A solution would be to delay any advertisements from
>  > the second-best plane by an interval greater than the MRAI
>  > timer. Not good, but I can't think of anything better.
>  >
>  > --
>  > Jakob Heitz.
>  >
>  > _______________________________________________
>  > GROW mailing list
>  > [email-masked]
>  > https://www.ietf.org/mailman/listinfo/grow
>  >
> 
>

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 27 Mar 2011 15:49:08 -0400

If every RR plane must send a path, then the clients
will have N paths for every destination, even if there
is only one path.

Given N RR planes,
that means clients will need N times the BGP memory
and N times the BGP processing power for destinations
with only one path.

Please make a recommendation on how many RR planes
are optimal.

--
Jakob Heitz.

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 27 Mar 2011 22:46:13 +0200

Hello Jakob,

Ok.

> back to the 2nd bset path P2.

Nope it will not .. see below on comments to your description.

> Plane1 advertises P1, Plane2 advertises P2

Perfect !

Ok.

Ok.

> 3. Client receives P3 from Plane2:         Client chooses P3.

This is only when client would not have P2 at all. IMHO corner case (see 
below why)

> 4. Client receives P2 from Plane1:         Client chooses P2.

P2 should not go away from client.

> as possible paths for any destination.

Nope.

There are two simple solution here:

- Presence of an extra session from client to either RR to make sure 
that your second best is not removed before overall best is advertised

- Small delay on the RR when advertising 2nd best to clients to make 
sure overall best is there first in some implementations just 
configuring per session mrai may help here.

Cheers,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 27 Mar 2011 22:47:36 +0200

> will have N paths for every destination, even if there

Nope that is not a requirement as discussed in the previous email.

 > Please make a recommendation on how many RR planes

For all applications of today I see 2 as sufficient and optimal.

Thx,
R.

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 27 Mar 2011 16:50:27 -0400

> Nope that is not a requirement as discussed in the previous email.

In the previous email, you said the exact opposite:
"Yes it would advertise overall best if backup is missing."

You require that to stop the flap.

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 27 Mar 2011 22:53:56 +0200

> "Yes it would advertise overall best if backup is missing."

That is true but this is not applicable to the case here.

Here RRs as you have indicated would have P1, P2 and P3 and when P1 goes 
down there is backup on both RRs.

Thx,
R.

> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 27 Mar 2011 17:05:48 -0400

The "corner" case happens 1/2 the time.

Initial:
Plane1 advertises P1.
Plane2 advertises P2.

P1 goes down:
Plane1 advertises P2.
Plane2 advertises P3.

The case that the client receives Plane2 message first
is 50% likely. During the flap, it has:

Plane1: P1
Plane2: P3

P1 is unreachable, so it chooses P3.

Originally, I proposed a delay on Plane2 advertisements,
but you didn't like it. How "small" should the delay be?
I think longer than MRAI, since Plane1 may be running
that timer for its P2 advertisement.

What is the "extra session"?
Is it a 3rd RR plane?

--
Jakob Heitz.

> Sent: Sunday, March 27, 2011 1:46 PM
> To: Jakob Heitz
> Cc: [email-masked]; IETF IDR

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 27 Mar 2011 17:09:05 -0400

So what is the rule to make it applicable?

If there is no second path,
When does the 2nd best RR plane advertise the best
and when does it advertise nothing?

--
Jakob Heitz.

> Sent: Sunday, March 27, 2011 1:54 PM
> To: Jakob Heitz
> Cc: IETF IDR; [email-masked]
> Subject: Re: [Idr] [GROW] Potential route flap with bgp-diverse-path
> 
>

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 27 Mar 2011 23:24:17 +0200

Hi Jakob,

> The "corner" case happens 1/2 the time.

If you insist let's focus on it then ...

> P1 is unreachable, so it chooses P3.

There is no "flap" in the above. Let's see what client does in this case ...

Initially you have net X via P1 and P2.

P1 goes down so P2 becomes active in prefix independent convergence way 
no packet drops other then IGP notification about P1 down bad event.

Now RR2 sends implicit withdraw and advertises P3 over the session so 
client does RIB update for net X and in place modification for it's next 
hop (assuming that P2 and P3 have even different next hops).

In decent implementation this is not removal of the path, drop then it's 
new installation. In decent bgp implementation switching one active and 
valid next hop with a given net with another valid next hop is not 
traffic impacting (if that is what you mean by route flap).

So with the light of this what problem are you trying to address in the 
first place ?

> but you didn't like it. How "small" should the delay be?

In non decent implementation the delay of advertising 2nd best may cover 
for it. I would not make any recommendation for the value at this point 
and leave it as an configuration option.

> that timer for its P2 advertisement.

Well this now depends how MRAI is implemented ... does it allow always 
first transition to go or not. Besides default MRAI for IBGP and this is 
IBGP we are discussing has been for a long time recommended to be of zero.

> What is the "extra session"?

Additional session allows you to receive from the same RR both best and 
second best paths one on each session. Hence non of the issues you are 
describing do apply then.

Cheers,
R.

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 27 Mar 2011 17:37:41 -0400

> traffic impacting (if that is what you mean by route flap).

I mean it switches to P2, then to P3, then
back to P2. This does not happen when using add-path.
It causes churn if this router readvertises to external
peers and P2 has different atributes to P3.

The issue is not necessarily a problem. However, it needs
to be stated in the draft, so that operators can be
aware and avoid the problems.

> describing do apply then.

That makes it a 3rd RR plane.
It means you are recommending at least as many RR planes
as possible paths for any destination.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 28 Mar 2011 00:05:40 +0200

Hi Jakob,

> back to P2. This does not happen when using add-path.

No one is trying to compare it to add-paths.

> peers and P2 has different atributes to P3.

What it needs to be compared against is current BGP deployments. You 
best path from both RRs and when it fails you blackhole traffic till BGP 
provides you a new path.

The external re-advertisement is also not really valid observation. One 
main reason why EBGP MRAI is longer then IBGP MRAI is exactly to 
mitigate those short lived BGP transitions from being visible in the 
entire Internet.

> aware and avoid the problems.

If you have some text to propose which would first correctly describe 
and clarify the issue I would be happy to incorporate it.

Just keep in mind that diverse-path is a very simple way, without need 
to upgrade the client side to provide much better path switchover 
results with no packet drop as compared to what is today.

> as possible paths for any destination.

Nope. I think you are drawing conclusions too quickly.

Second client-RR session would allow to advertise best and second best 
from any given RR. That means that you would never transition to 3rd 
best even for fraction of time as your R2 would still be there on the 
client.

Keep in mind that some deployment proposal models from customers are 
even today focused on sharing RRs and configuring some clients to get 
best and some to get second best from a given RR even if RRs are in 
pair. And this is just configuration as the diverse-path as will be 
discussed in GROW meeting this week during the implementation report is 
a per peer knob.

Many thx,
R.

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 27 Mar 2011 18:26:48 -0400

I propose to add this text to the draft:

Advertisements of multiple paths by multiple
planes of advertisers are received asynchronously.
It is not possible to order the messages reliably
from different planes. This may cause churn in
route selection when paths change.

To address this issue, a speaker that advertises the
Nth best path, where N>1, SHOULD delay its advertisement
by MRAI plus (N-1) times X. X SHOULD be a small
value that is greater than the message delivery
time between BGP speakers. Such a speaker should not
add its own MRAI timer to this delay.

A speaker that advertises the Nth best path shall
initially advertise nothing if an Nth best path
does not exist. Once the Nth best path changes,
it should apply the above delay before sending
the changed advertisement. The change may be a
withdrawal.

--
Jakob Heitz.

> Sent: Sunday, March 27, 2011 3:06 PM
> To: Jakob Heitz

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 29 Mar 2011 10:03:24 +0200

Hi Jakob,

I have discussed this proposed text with other diverse-path co-authors 
and we concluded that such additional constrain is not required.

The reason is that BGP implementations deployed today can easily delay 
reaction by BGP to IGP bad events flooding while in the same time 
switchover of the data plane to a pre-installed backup path will happen 
instantaneously in FIB. This will result in no data plane traffic drop 
as well as no BGP churn of any kind.

Many thx,
R.

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 30 Mar 2011 07:31:00 -0400

Aww, c'mon Robert, I think it's a great idea.
You just need to clean up a few loose ends.

--
Jakob Heitz. x25475. 510-566-2901

> Sent: Tuesday, March 29, 2011 1:03 AM
> To: Jakob Heitz