Subject: [Idr] Possible weakness in long-lived graceful restart

From: Paul Mattes <[email-masked]>
Date: Fri, 4 Apr 2014 19:45:55 +0000

I have been doing some work on long-lived graceful restart (LLGR) and a plausible failure scenario has come up in discussions here, which does not appear to be covered, though it certainly seems to be the sort of failure that should be addressed by LLGR.

The scenario happens when we have LLGR configured on both sides of a number of peering sessions, and BGP speaker (e.g., a Route Reflector) loses contact with a number of its peers simultaneously. The likely cause would be that the paths to the peers share a common element that has failed. (And note that the common element could be part of the BGP speaker in question - something that can be repaired without restarting the system.)

When contact with this set of peers is re-established, the scenario has many of the same semantics of a restart. In particular, it would be best to delay sending End-of-RIB indications to any of the members of this set of peers until End-Of-RIB to any of them, or until a reasonable timeout expires after the first comes back. Without this delay, the one BGP speaker would likely send each of the peers a large number of advertisements with the LLGR_STALE attribute (for the stale routes held for the other disconnected peers), followed by a large number of advertisements of the those same routes without the LLGR_STALE attribute (as those routes are refreshed by the other peers). There would likely be no change in reachability throughout the network, but there could be a significant amount of FIB thrashing and transient spikes in forwarding traffic, as well as a significant BGP-related I/O and processing load on each of the speakers.

While it seems like a restart, it is of course not a restart. And as far as the peers which remain connected are concerned, it does not appear anything like a restart at all.

While it is possible to imagine an implementation of LLGR that addresses this issue outside of the specification, I think it would be best if the specification handled this scenario better, whether through a change in the protocol or additional implementation advice.

--
        pdm

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 4 Apr 2014 22:32:20 +0200

Hi Paul,

I think your observation is valid and possible in real deployments however
the fix you are asking for may be slightly.

First let's observe that "Restart Time" and "Long lived Stale Time" is per
session and defined procedures are also per session.

To apply any solution it would be required to find a way to dynamically
group peers which may belong to the same "peer failure risk group", but
that is hard as IBGP sessions to those peers naturally are designed to work
around single network failures.

An option could be perhaps to treat all peers running in under "Long lived
Stale Time" state as one group and then define a "Long Lived Read Only
Timer" which would start from the first session previously in such group to
get re-established and till it expires suppress any best path computation
based on the newly received paths from those peers.

Literally what you are pointing out could be fixed by the following text
addition to "draft-uttaro-idr-bgp-persistence-03" section 4.2:

Current text:

Once the session is re-established, the procedures specified in
[RFC4724] apply for the stale routes irrespective of whether the stale
routes are retained during the"Restart Time" period or the "Long-lived
Stale Time" period.

However, in the case of consecutive restarts (i.e,the session goes
down before the EoR is received) the previously marked stale routes
MUST NOT be deleted before the timer for the "Long-lived Stale Time"
expires.

New text:

All sessions in "Long-lived Stale Time" are grouped into "Long-lived
Sessions Group". New timer called "Long-lived Session Read Only Timer" is
defined.

Once the session is re-established, the procedures specified in [RFC4724]

apply for the stale routes irrespective of whether the stale routes
are retained during the"Restart Time" period or the "Long-lived Stale
Time" period except that such procedures are delayed for the peers
belonging to "Long-lived

Sessions Group" till all members of the group send EOR message or the

"Long-lived Session Read Only Timer" expires.

The "Long-lived Session Read Only Timer" is started upon
reestablishment ofthe first session belonging to "Long-lived Sessions
Group".

However, in the case of consecutive restarts (i.e, the session goes down

before the EoR is received) the previously marked stale routes MUST NOT

be deleted before the timer for the "Long-lived Stale Time" expires.

Best regards,
R.

On Fri, Apr 4, 2014 at 9:45 PM, Paul Mattes <[email-masked]> wrote:

>  I have been doing some work on long-lived graceful restart (LLGR) and a
> plausible failure scenario has come up in discussions here, which does not
> appear to be covered, though it certainly seems to be the sort of failure
> that should be addressed by LLGR.
>
> The scenario happens when we have LLGR configured on both sides of a
> number of peering sessions, and BGP speaker (e.g., a Route Reflector) loses
> contact with a number of its peers simultaneously. The likely cause would
> be that the paths to the peers share a common element that has failed. (And
> note that the common element could be part of the BGP speaker in question -
> something that can be repaired without restarting the system.)
>
> When contact with this set of peers is re-established, the scenario has
> many of the same semantics of a restart. In particular, it would be best to
> delay sending End-of-RIB indications to any of the members of this set of
> peers until End-Of-RIB to any of them, or until a reasonable timeout
> expires after the first comes back. Without this delay, the one BGP speaker
> would likely send each of the peers a large number of advertisements with
> the LLGR_STALE attribute (for the stale routes held for the other
> disconnected peers), followed by a large number of advertisements of the
> those same routes without the LLGR_STALE attribute (as those routes are
> refreshed by the other peers). There would likely be no change in
> reachability throughout the network, but there could be a significant
> amount of FIB thrashing and transient spikes in forwarding traffic, as well
> as a significant BGP-related I/O and processing load on each of the
> speakers.
>
> While it seems like a restart, it is of course not a restart. And as far
> as the peers which remain connected are concerned, it does not appear
> anything like a restart at all.
>
> While it is possible to imagine an implementation of LLGR that addresses
> this issue outside of the specification, I think it would be best if the
> specification handled this scenario better, whether through a change in the
> protocol or additional implementation advice.
>
> --
>         *pdm*
>
>
>
>
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>
>

---

From: Paul Mattes <[email-masked]>
Date: Fri, 4 Apr 2014 22:26:00 +0000

I agree that my original suggestion of just suppressing End-of-RIB wouldn't help much with the thrashing. So this appears less like a partial-restart scenario than I had thought.

I see that your solution could avoid a lot thrashing. The router implementing this procedure would still initially advertise its stale RIB (some routes LLGR-stale, some not) to each of the newly-re-established peers, and send End-of-RIB. The new timer would prevent it from advertising intermediate states for routes from the group, until each of the peers has sent End-of-RIB. Then it would re-advertise whichever routes are now best, looking like something close to the last state they were advertised in before the disaster. The worst case for this is advertising every route from the group twice, which is much better than the existing worst case, which is (I think) advertising every route N times, where N is the number of peers that disconnected.

But the best case, and I think the typical case with the read-only timer, would also be double advertisements of each route - once with LLGR_STALE attached, and once without it.

Can we do better still? My hope is to avoid sending out that first wave of LLGR_STALE routes, too, if we can. Or am I misunderstanding how your mechanism would work?

--
        pdm

From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Friday, April 04, 2014 3:32 PM
To: Paul Mattes
Cc: [email-masked]
Subject: Re: [Idr] Possible weakness in long-lived graceful restart

Hi Paul,

I think your observation is valid and possible in real deployments however the fix you are asking for may be slightly.

First let's observe that "Restart Time" and "Long lived Stale Time" is per session and defined procedures are also per session.

To apply any solution it would be required to find a way to dynamically group peers which may belong to the same "peer failure risk group", but that is hard as IBGP sessions to those peers naturally are designed to work around single network failures.

An option could be perhaps to treat all peers running in under "Long lived Stale Time" state as one group and then define a "Long Lived Read Only Timer" which would start from the first session previously in such group to get re-established and till it expires suppress any best path computation based on the newly received paths from those peers.

Literally what you are pointing out could be fixed by the following text addition to "draft-uttaro-idr-bgp-persistence-03" section 4.2:

Current text:

Once the session is re-established, the procedures specified in [RFC4724] apply for the stale routes irrespective of whether the stale routes are retained during the"Restart Time" period or the "Long-lived Stale Time" period.

However, in the case of consecutive restarts (i.e,the session goes down before the EoR is received) the previously marked stale routes MUST NOT be deleted before the timer for the "Long-lived Stale Time" expires.

New text:

All sessions in "Long-lived Stale Time" are grouped into "Long-lived Sessions Group". New timer called "Long-lived Session Read Only Timer" is defined.

apply for the stale routes irrespective of whether the stale routes are retained during the"Restart Time" period or the "Long-lived Stale Time" period except that such procedures are delayed for the peers belonging to "Long-lived

The "Long-lived Session Read Only Timer" is started upon reestablishment ofthe first session belonging to "Long-lived Sessions Group".

Best regards,

R.

On Fri, Apr 4, 2014 at 9:45 PM, Paul Mattes <[email-masked]<mailto:[email-masked]>> wrote:
I have been doing some work on long-lived graceful restart (LLGR) and a plausible failure scenario has come up in discussions here, which does not appear to be covered, though it certainly seems to be the sort of failure that should be addressed by LLGR.

--
        pdm

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 5 Apr 2014 00:47:19 +0200

Hi Paul,

I actually do not see why you envision double advertisements.

The amount of advertisement depends on two factors:

- Duration of "Long-lived Session Read Only Timer" - if this is long
enough and all peers in the LLGR-stale group would manage to restart -
overall best path would be send. We do need this timer not to wait
forever for EOR from each member of this group.

- The fact that some peers will restart later then "Long-lived Session
Read Only Timer" does not really mean that you will see full table
again in all peers. Only if such peers paths would become best you
will see the paths re-advertised .. so likely it would be small delta
of overall number of paths.

> But the best case, and I think the typical case with the read-only timer, would
> also be double advertisements of each route - once with LLGR_STALE attached,
> and once without it.
>
> Can we do better still? My hope is to avoid sending out that first wave of
> LLGR_STALE routes, too, if we can. Or am I misunderstanding how your
> mechanism would work?

If you are talking after restart then not really. Those restarted and
selected as best would be advertised when "Long-lived Session Read
Only Timer" without the STALE mark.

If you are referring to original advertisement when sessions go down
then again .. unless there is no other best only then they will be
advertised with such MARK, but this is what this entire persistence is
all about after all :)

Please let me know if you have any questions ...

Cheers,
R.

---

From: Paul Mattes <[email-masked]>
Date: Fri, 4 Apr 2014 23:07:04 +0000

Then I'm definitely missing something. But it wouldn't be the first time. ;-)

Let's call the router implementing this new timer R, and the other routers A, B and C.

(1) R loses contact with A, B and C.
(2) The outage lasts long enough for R to go into LLGR helper mode for each.
(3) R puts A, B and C into a group.

At this point, the best paths for some number of routes in R's RIB came from A, B or C. So it marks them with LLGR_STALE.

(4) A reconnects.
(5) R starts the Read-Only timer.
(6) B and C reconnect.

At this point, A, B and C have not re-advertised anything to R.

(7) R advertises what it has to A, B and C. That includes the LLGR_STALE routes mentioned above.

(8) A, B and C re-advertise their routes. The Read-Only timer prevents these routes from being use for best path computations, so nothing is re-advertised.
(9) A, B and C send End-of-RIB.
(10) R runs best path on any routes that A, B or C refreshed.
(11) R re-advertises any routes whose best path has now changed, or that now have the LLGR_STALE community removed because they were refreshed.

So I think the point of confusion is step (7), where R sends its current RIB to the peers, which would include stale routes from B and C sent to A, stale routes from A and C sent to B, and stale routes from A and B sent to C. Or is something keeping R from doing that?

--
        pdm

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Friday, April 04, 2014 5:47 PM
To: Paul Mattes
Cc: [email-masked]
Subject: Re: [Idr] Possible weakness in long-lived graceful restart

Hi Paul,

- Duration of "Long-lived Session Read Only Timer" - if this is long enough and all peers in the LLGR-stale group would manage to restart - overall best path would be send. We do need this timer not to wait forever for EOR from each member of this group.

- The fact that some peers will restart later then "Long-lived Session Read Only Timer" does not really mean that you will see full table again in all peers. Only if such peers paths would become best you will see the paths re-advertised .. so likely it would be small delta of overall number of paths.

> But the best case, and I think the typical case with the read-only 
> timer, would also be double advertisements of each route - once with 
> LLGR_STALE attached, and once without it.
>
> Can we do better still? My hope is to avoid sending out that first 
> wave of LLGR_STALE routes, too, if we can. Or am I misunderstanding 
> how your mechanism would work?

If you are talking after restart then not really. Those restarted and selected as best would be advertised when "Long-lived Session Read Only Timer" without the STALE mark.

If you are referring to original advertisement when sessions go down then again .. unless there is no other best only then they will be advertised with such MARK, but this is what this entire persistence is all about after all :)

Cheers,
R.

On Sat, Apr 5, 2014 at 12:26 AM, Paul Mattes <[email-masked]> wrote:
> [RFC4724]
> number of peering sessions, and BGP speaker (e.g., a Route Reflector) 
> loses contact with a number of its peers simultaneously. The likely 
> cause would be that the paths to the peers share a common element that 
> has failed. (And note that the common element could be part of the BGP 
> speaker in question - something that can be repaired without 
> restarting the system.)
>
>
>

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 5 Apr 2014 01:20:02 +0200

Hi Paul,

If A, B & C have not readvisrtised their non stale (any longer) paths
to R then R would not compute its best path (no trigger) and would not
advertised anything to them.

Keep in mind that Read-Only mode is called such for a reason .. there
is no advertisement done during that time to peers in such RO mode.
Here we are just creating a new group and applying new timer .. but
the basic behavior stays as it is today.

Cheers,
R.

---

From: Paul Mattes <[email-masked]>
Date: Mon, 7 Apr 2014 19:59:53 +0000

Then what I missed was that the R would not advertise anything *to* A, B or C until the Read-Only timer expired. Then I agree that A, B and C are likely to receive only one set of updates of one another's routes. Also routers D and E (other peers of R, not affected by the outage) would receive only one set of updates for routes that originated from A, B and C. All good -- thank you for your feedback and patience.

--
        pdm

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Friday, April 04, 2014 6:20 PM
To: Paul Mattes
Cc: [email-masked]
Subject: Re: [Idr] Possible weakness in long-lived graceful restart

Hi Paul,

If A, B & C have not readvisrtised their non stale (any longer) paths to R then R would not compute its best path (no trigger) and would not advertised anything to them.

Keep in mind that Read-Only mode is called such for a reason .. there is no advertisement done during that time to peers in such RO mode.
Here we are just creating a new group and applying new timer .. but the basic behavior stays as it is today.

Cheers,
R.

On Sat, Apr 5, 2014 at 1:07 AM, Paul Mattes <[email-masked]> wrote:
>> cause would be that the paths to the peers share a common element 
>> that has failed. (And note that the common element could be part of 
>> the BGP speaker in question - something that can be repaired without 
>> restarting the system.)
>>
>>
>>

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 7 Apr 2014 22:03:50 +0200

Hi Paul,

Yes you are correct.

Now I guess if others agree that this is a valid problem to be solved
we would need to update the draft with this additional procedure.

As this is local optimization we can clearly make it optional to
implement so existing implementations do not need to worry about being
not compliant ;)

Thx,
R.

---

From: Paul Mattes <[email-masked]>
Date: Mon, 7 Apr 2014 20:43:53 +0000

I agree that it should be optional. Implementing it would make you more neighborly, but nothing in someone else's LLGR implementation would break if you didn't.

Now that we seem to have a viable solution, I will speak with John Scudder about the possibility/likelihood/wisdom of getting it into a draft.

--
        pdm

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Monday, April 07, 2014 3:04 PM
To: Paul Mattes
Cc: [email-masked]
Subject: Re: [Idr] Possible weakness in long-lived graceful restart

Hi Paul,

Yes you are correct.

Now I guess if others agree that this is a valid problem to be solved we would need to update the draft with this additional procedure.

As this is local optimization we can clearly make it optional to implement so existing implementations do not need to worry about being not compliant ;)

Thx,
R.

On Mon, Apr 7, 2014 at 9:59 PM, Paul Mattes <[email-masked]> wrote:
>> advertised with such MARK, but this is what this entire persistence 
>> is all about after all :)
>>
>>> are retained during the"Restart Time" period or the "Long-lived 
>>> Stale Time" period except that such procedures are delayed for the 
>>> peers belonging to "Long-lived
>>>
>>> number of peering sessions, and BGP speaker (e.g., a Route 
>>> Reflector) loses contact with a number of its peers simultaneously. 
>>> The likely cause would be that the paths to the peers share a common 
>>> element that has failed. (And note that the common element could be 
>>> part of the BGP speaker in question - something that can be repaired 
>>> without restarting the system.)
>>>
>>>
>>>