Subject: On minimizing SPF backoff induced blackouts

From: Hannes Gredler <[email-masked]>
Date: Thu, 24 Jul 2014 14:35:45 +0200

hi RTGWG,

some old presentation from Dave Katz on backoff and timing.
https://www.nanog.org/meetings/nanog25/presentations/katz.ppt
    [courtesy Chris Bowers for digging it up ...]

slide 15 is of particular interest - says that if
an implementation and the system around it is properly
designed then no backoff should be required.

/hannes

---

From: <[email-masked]>
Date: Thu, 24 Jul 2014 16:59:05 +0000

Hannes,

" if an implementation and the system around it is properly designed"
-> This is exactly what SP are trying to be protected against (bad implementations) :)

Can every vendor sign with blood that their implementation and system are properly designed for all situations ... unfortunately (for us) I don't think so :)

Stephane

-----Original Message-----
From: rtgwg [mailto:[email-masked]] On Behalf Of Hannes Gredler
Sent: Thursday, July 24, 2014 08:36
To: [email-masked]
Subject: On minimizing SPF backoff induced blackouts

hi RTGWG,

slide 15 is of particular interest - says that if an implementation and the system around it is properly designed then no backoff should be required.

/hannes

_______________________________________________
rtgwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtgwg

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: "Russ White" <[email-masked]>
Date: Thu, 24 Jul 2014 13:39:28 -0400

> Can every vendor sign with blood that their implementation and system are
> properly designed for all situations ... unfortunately (for us) I don't
think so :)

Which is why this would be a good BCP -- both for vendors and providers.

:-)

Russ

---

From: <[email-masked]>
Date: Thu, 24 Jul 2014 17:40:22 +0000

Hi Hannes,

Many thanks for the slides: quite interesting and to the point. Very useful.

More inline.

> slide 15 is of particular interest - says that if an implementation and the
> system around it is properly designed then no backoff should be required.

- so why did this particular implementation included a SPF backoff algorithm? ;-)
- I will not comment about "properly designed" but (probably) all systems have bugs and/or sub-optimal design parts
- Sorry, but I would argue that if existing SPF backoff algorithm were "properly designed", they would be standardized to allow for inter-vendor consistency
- If after 10 consecutive SPFs runs in 1 second, the IGP still haven't solved the problem, then keep doing the same thing again and again may not help, at least urgency will probably not help/be needed. Eventually the IGP SPF is part of the problem e.g. by creating system & network churn, micro-loops... IMHO, trying to calm down the IGP reaction seems a valid option to try. At least this reduces the number of micro-loops in the network.
- slide 15 seems to only consider intra-system impacts (looks even intra-IGP only). From a network wide perspective, the more SPF/FIB change, the more micro-loops. 

Bruno

---

From: <[email-masked]>
Date: Thu, 24 Jul 2014 17:53:05 +0000

Russ,

> > Can every vendor sign with blood that their implementation and system
> > are properly designed for all situations ... unfortunately (for us) I
> > don't  think so :)
> 

I'm not sure to exactly see your point. So let me expand into more details:

We don't need consistency between different AS/providers.
We need consistency within a given AS, across vendors.
BCP for existing _parameters_ do not solve the problem if we have different algorithm computing different values.
What we need is to standardize an algorithm. Is your point is that such document should be labeled as "IETF BCP" rather than "IETF standard track"? If so, I would tend to disagree as IMO this is required for interop.

Thanks,
Bruno

---

From: "Russ White" <[email-masked]>
Date: Thu, 24 Jul 2014 14:02:21 -0400

> We need consistency within a given AS, across vendors.

Correct -- but a BCP isn't just for inter-AS, it's also for intervendor.

> BCP for existing _parameters_ do not solve the problem if we have
different
> algorithm computing different values.
> What we need is to standardize an algorithm. Is your point is that such
> document should be labeled as "IETF BCP" rather than "IETF standard
track"?
> If so, I would tend to disagree as IMO this is required for interop.

The problem is with the word "interop." Typically, algorithms have only been
specified to the point that all instances of a given routing protocol are
eventually consistent (how the best path is calculated, which events should
or should not trigger updates, etc.) -- what you're asking is for this to be
extended to timing and the actual process used in specific situations. This
would analogous to asking for a spec around the actual way in which BGP
tables are stored (rather than the theoretical model proposed in the BGP
specs). I see a couple of problems with going down this path -- the most
prominent of which is that it kills off the ability to think of better ways
to implement the functionality under discussion. 

Given all of this, the problem we have is that different implementations
back off in different ways, which causes microloops and microblackholes. If
we set upper and lower bounds on the time to react in each given situation,
then we're solving that problem without telling people how to get to those
upper and lower bounds. If we propose an actual algorithm, then we're
telling people not only what must happen, but how it must happen. I suspect
we're crossing a line in there someplace that reaches outside the IETF's
scope.

Hence, I would argue for a BCP that says, "this is what must happen and
when," which would be similar to telling BGP implementors the way dampening
should look from another implementation's point of view... The
implementation is still (pretty much) a block box. I don't know if we should
slip over the line into telling people how to implement.

I'm willing to be persuaded, I just don't see the argument for specifying an
algorithm with what's been put on the table to this point.

:-)

Russ

---

From: <[email-masked]>
Date: Thu, 24 Jul 2014 18:49:20 +0000

> Correct -- but a BCP isn't just for inter-AS, it's also for intervendor.

Good.
#Ref A

> > What we need is to standardize an algorithm. Is your point is that
> > such document should be labeled as "IETF BCP" rather than "IETF
> > standard
> track"?
> or should not trigger updates, etc.)

we also need consistency, between nodes in a given AS, on the delay before running the SPF.
We agreed on this. cf above #Ref A

> -- what you're asking is for this to be
> specs).

I disagree with your analogy.
IMO, the BGP analogy is, within an AS, following an internal route change, to trigger BGP route re-computation at the same time across vendors. e.g. agree on the mechanism (e.g. BGP scan time vs triggered computation).
(but it's still a bit different as for BGP you could use tunnels between ASBR to avoid hitting RIB inconsistencies)

> I see a couple of problems with going down this path -- the most
> prominent of which is that it kills off the ability to think of better ways to
> implement the functionality under discussion.

I disagree. cf slide 7 http://tools.ietf.org/agenda/90/slides/slides-90-rtgwg-2.pdf

> Given all of this, the problem we have is that different implementations back
> off in different ways, which causes microloops and microblackholes.

+1

> If we set
> upper and lower bounds on the time to react in each given situation, then
> we're solving that problem without telling people how to get to those upper
> and lower bounds. If we propose an actual algorithm, then we're telling
> people not only what must happen, but how it must happen. I suspect we're
> crossing a line in there someplace that reaches outside the IETF's scope.

We need "close"/identical values, across all nodes, for _all_ the steps of the back-off algorithm/spec. From the first event of the set of consecutive events, up to the last events.
If we have a 2 steps back-off algorithm (fast - slow) I guess setting the upper and lower bound is enough. But this assumes that we all agree on this 2 steps mechanism/algo. (while we/vendors don't today)
We also need to define how we determine this set of consecutive events. And make sure we are consistent about which are those events (e.g. refresh LSA/LSP are not).
 
> when," which would be similar to telling BGP implementors the way
> dampening should look from another implementation's point of view... The

Ok. So I guess my presentation/draft was not clear enough. 
Can we try to sync face to face during this week?

Thanks,
Bruno

---

From: <[email-masked]>
Date: Thu, 24 Jul 2014 18:56:37 +0000

> > I'm willing to be persuaded, I just don't see the argument for
> > specifying an algorithm with what's been put on the table to this point.
> 

Let me try again.
In slide 5: http://tools.ietf.org/agenda/90/slides/slides-90-rtgwg-2.pdf
- do we agree that it would be better if node A and B both schedule their SPF at roughly the same time? i.e. wait for the same duration?
- if so, what would be your proposition?

Bruno

---

From: Hannes Gredler <[email-masked]>
Date: Thu, 24 Jul 2014 19:03:19 +0000

hi bruno,

IMO the problem that you're trying to solve is to have the routers
still being responsive when multiple events are happening in the
network.

wouldn't then a simple fix be to kick the SPF pacing logic at a much more
later point in time ? - i.e. rather than slowing down things
down on event #2 or #3, do it at event #<N> (please insert you're
favorite number for <N>)

/hannes

________________________________________
From: rtgwg <[email-masked]> on behalf of [email-masked] <[email-masked]>
Sent: Thursday, July 24, 2014 20:56
To: Russ White
Cc: [email-masked]
Subject: RE: On minimizing SPF backoff induced blackouts

Let me try again.

Bruno

---

From: <[email-masked]>
Date: Thu, 24 Jul 2014 19:13:50 +0000

Hi Hannes,

> hi bruno,
> 
> IMO the problem that you're trying to solve is to have the routers still being
> responsive when multiple events are happening in the network.
> 
> wouldn't then a simple fix be to kick the SPF pacing logic at a much more later
> point in time ? - i.e. rather than slowing down things down on event #2 or #3,
> do it at event #<N> (please insert you're favorite number for <N>)

IMO that's a valid option.
Would work for me if the WG agrees on that one. (but there are many options and different opinions, so next step is to build consensus on this)
We'll try to propose something, with your proposition in mind.

Thanks for your contribution.
Bruno

> From: rtgwg <[email-masked]> on behalf of
> [email-masked] <[email-masked]>
> - do we agree that it would be better if node A and B both schedule their SPF
> at roughly the same time? i.e. wait for the same duration?
> __________________________________________________________
> __________________________________________________________
> _____
> 
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites
> ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez
> le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les
> messages electroniques etant susceptibles d'alteration, Orange decline toute
> responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
> 
> This message and its attachments may contain confidential or privileged
> information that may be protected by law; they should not be distributed,
> used or copied without authorisation.

---

From: Russ White <[email-masked]>
Date: Thu, 24 Jul 2014 15:49:07 -0400

Replying from mobile, forgive the top post... :-(

I was thinking of a bcp that would say something like:

- these are the different points that matter in convergence timing
- vendors are encouraged to provide timers between x and y for each of these
- vendors are further encouraged to use x, y, ... As defaults
- operators are cautioned that this is the set of problems they will face if these timers are not consistent
- operators are encouraged to use consistent timers, rather than waiting on vendors to use a common set of defaults

Would that cover what needs to be done without specifying an actual algorithm?

:-)

Russ

---

From: <[email-masked]>
Date: Fri, 25 Jul 2014 16:43:15 +0000

Hannes,

I agree with you at now routers are able to handle multiple events in a short time but anyway :
- delaying is always necessary (I can't really see how we can go under 100/150ms) :
	If you go to something like 0 or 5 msec, we may fall into implementations issues with constant READ/WRITE contentions in FIB that often cause corner cases where the router is completely lost in what it must do
	Moreover in case of complex outage (node failure or SRLG failure) where multiple LSPs are sent. We must ensure that all LSPs are received before computing SPF to ensure that the target topology is a good one.

- Increasing delay is also necessary. By experience there are situations where you need absolutely to calm down computation to prevent churn amplification (even today with stronger CPUs)
- we need to ensure that increasing delay steps are not so big, so in case of router unsynchronization (in delay value), the difference would be small.
- as agreed , we can run more fast scheduled SPFs

Both current implementations (two steps and backoff) do no fill these "requirements".

I would be in favor of standardizing something really simple with no mathematic underway. Just extend the two steps to an N-Step tunable system that the operator can manage.
Something like defining a WRED profile but without interpolation.
Example :

Initial : 100 ms, 4 runs
2nd : 200ms, 2 runs
3rd : 300ms, 2 runs
4th : 500ms, 2 runs
5th : 700ms, 2 runs
6th : 800ms, 2 runs
7th : 1s, all subsequent runs

-----Original Message-----
From: rtgwg [mailto:[email-masked]] On Behalf Of Hannes Gredler
Sent: Thursday, July 24, 2014 15:03
To: DECRAENE Bruno IMT/OLN; Russ White
Cc: [email-masked]
Subject: RE: On minimizing SPF backoff induced blackouts

hi bruno,

IMO the problem that you're trying to solve is to have the routers still being responsive when multiple events are happening in the network.

wouldn't then a simple fix be to kick the SPF pacing logic at a much more later point in time ? - i.e. rather than slowing down things down on event #2 or #3, do it at event #<N> (please insert you're favorite number for <N>)

/hannes

Let me try again.

Bruno

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Vitkovský Adam <[email-masked]>
Date: Mon, 28 Jul 2014 09:36:07 +0000

Stephane,

My opinion is that majority of the changes in backbones can be described by a single or two LSPs. 
So modern routers the initial wait is unnecessary in most of the cases or is roughly equal to network jitter. 
If there's a major failure you might want to wait for roughly the full or 1/2 of the time it takes the LSPs to be propagated from one end of your network to the other end + the delay variation (this of course would be different for each network).
That is when you would like the SPF to converge asap.
However with FRR at hand we can go with whatever the common default timers are gonna be. 

adam
> From: rtgwg [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Friday, July 25, 2014 6:43 PM
> To: Hannes Gredler; DECRAENE Bruno IMT/OLN; Russ White
> Cc: [email-masked]
> 	If you go to something like 0 or 5 msec, we may fall into
> implementations issues with constant READ/WRITE contentions in FIB that
> often cause corner cases where the router is completely lost in what it must
> do
> 	Moreover in case of complex outage (node failure or SRLG failure)
> where multiple LSPs are sent. We must ensure that all LSPs are received
> before computing SPF to ensure that the target topology is a good one.
> 
> - Increasing delay is also necessary. By experience there are situations where
> you need absolutely to calm down computation to prevent churn
> amplification (even today with stronger CPUs)
> - we need to ensure that increasing delay steps are not so big, so in case of
> router unsynchronization (in delay value), the difference would be small.
> I would be in favor of standardizing something really simple with no
> mathematic underway. Just extend the two steps to an N-Step tunable
> system that the operator can manage.

---

From: Hannes Gredler <[email-masked]>
Date: Mon, 28 Jul 2014 17:07:18 +0200

On Fri, Jul 25, 2014 at 04:43:15PM +0000, [email-masked] wrote:
| I agree with you at now routers are able to handle multiple events in a short time but anyway :
| - delaying is always necessary (I can't really see how we can go under 100/150ms) :

the only reason it is necessary is that some implementations (ours included)
do some state compression, such that there is a collection window of 20ms of
local-events. strictly speaking thats just a efficiency/responsiveness tradeoff
and we got some requests in the past to lower that window.

| 	If you go to something like 0 or 5 msec, we may fall into implementations issues with constant READ/WRITE contentions in FIB that often cause corner cases where the router is completely lost in what it must do

again thats implementation dependent. if you have now state-compression on your RIB/FIB
path then you're doomed anyway. state of the art is that you do not de-queue
route 'N(t)' if route 'N(t+1)' is available, but cancel N(t) update.

| 	Moreover in case of complex outage (node failure or SRLG failure) where multiple LSPs are sent. We must ensure that all LSPs are received before computing SPF to ensure that the target topology is a good one.

see and this i do not believe is possible.
whatever <N> SPF initial wait will be, there will be
always incidents where the LSP triggers arrives
after <N>, so there will be always corner cases for loops.

| - Increasing delay is also necessary. By experience there are situations where you need absolutely to calm down computation to prevent churn amplification (even today with stronger CPUs)

no doubt - the question is when do we need to kick
in that self-protection layer ... is it after
reception of the 2nd LSP or after the 10th ?

| - we need to ensure that increasing delay steps are not so big, so in case of router unsynchronization (in delay value), the difference would be small.
| - as agreed , we can run more fast scheduled SPFs
| 
| Both current implementations (two steps and backoff) do no fill these "requirements".

rapid-run in a certain way does because it is widening
your window of 'concurrent, unrelated network events'
without going into backup mode.

| I would be in favor of standardizing something really simple with no mathematic underway. Just extend the two steps to an N-Step tunable system that the operator can manage.
| Something like defining a WRED profile but without interpolation.
| Example :
| 
| Initial : 100 ms, 4 runs
| 2nd : 200ms, 2 runs
| 3rd : 300ms, 2 runs
| 4th : 500ms, 2 runs
| 5th : 700ms, 2 runs
| 6th : 800ms, 2 runs
| 7th : 1s, all subsequent runs

as tony (and others) have pointed out ... it will only work
during nice-weather conditions ... as soon as you have the
perfect-storm, you will have micro loops (and all the pain
you try to protect yourself) - therefore i'd
be much more in favour of 'something else' which gets us to
zero uloops, and for that i am afraid we need to have the notions
of independent forwarding planes (ala make before break) -

trying to synchronize RIB/FIB update across different
CPUs/routing-stacks/vendors/load conditions is
a battle that can never be won.

/hannes

| -----Original Message-----
| From: rtgwg [mailto:[email-masked]] On Behalf Of Hannes Gredler
| Sent: Thursday, July 24, 2014 15:03
| To: DECRAENE Bruno IMT/OLN; Russ White
| Cc: [email-masked]
| Subject: RE: On minimizing SPF backoff induced blackouts
| 
| hi bruno,
| 
| IMO the problem that you're trying to solve is to have the routers still being responsive when multiple events are happening in the network.
| 
| wouldn't then a simple fix be to kick the SPF pacing logic at a much more later point in time ? - i.e. rather than slowing down things down on event #2 or #3, do it at event #<N> (please insert you're favorite number for <N>)
| 
| /hannes
| 
| ________________________________________
| From: rtgwg <[email-masked]> on behalf of [email-masked] <[email-masked]>
| Sent: Thursday, July 24, 2014 20:56
| To: Russ White
| Cc: [email-masked]
| Subject: RE: On minimizing SPF backoff induced blackouts
| 
| > > I'm willing to be persuaded, I just don't see the argument for 
| > > specifying an algorithm with what's been put on the table to this point.
| >
| > Ok. So I guess my presentation/draft was not clear enough.
| 
| Let me try again.
| In slide 5: http://tools.ietf.org/agenda/90/slides/slides-90-rtgwg-2.pdf
| - do we agree that it would be better if node A and B both schedule their SPF at roughly the same time? i.e. wait for the same duration?
| - if so, what would be your proposition?
| 
| Bruno
| 
| 
| _________________________________________________________________________________________________________________________
| 
| Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
| 
| This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
| If you have received this email in error, please notify the sender and delete this message and its attachments.
| As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
| Thank you.
| 
| _______________________________________________
| rtgwg mailing list
| [email-masked]
| https://www.ietf.org/mailman/listinfo/rtgwg
| 
| _______________________________________________
| rtgwg mailing list
| [email-masked]
| https://www.ietf.org/mailman/listinfo/rtgwg
| 
| _________________________________________________________________________________________________________________________
| 
| Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
| pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
| a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
| Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
| 
| This message and its attachments may contain confidential or privileged information that may be protected by law;
| they should not be distributed, used or copied without authorisation.
| If you have received this email in error, please notify the sender and delete this message and its attachments.
| As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
| Thank you.
|

---

From: <[email-masked]>
Date: Mon, 28 Jul 2014 15:47:33 +0000

Hannes, all

> From: Hannes Gredler [mailto:[email-masked]] > Sent: Monday, July 28, 2014 5:07 PM

[...]

> trying to synchronize RIB/FIB update across different CPUs/routing-
> stacks/vendors/load conditions is a battle that can never be won.

The thread has somewhat diverge.

For the record, this is not what draft-decraene-rtgwg-backoff-algo is about. So the comment does not apply to this draft.

The draft is about not arbitrarily introducing a delay between SPF computations runs on different routers/implementations. Such delay is bug, not a feature. Let's correct it.
We can either correct it via an IETF standard that we can all discuss and improve together, or by declaring that the Cisco choice is the de facto standard. My preference is the first option since I believe in standard. I guess that this is also your preference.

> as soon as you have the perfect-storm, you will have micro loops (and all the pain you try to protect yourself) - therefore i'd be much more in favour of 'something else' which gets us to zero uloops, and for that i am afraid we need to have the notions of independent forwarding planes (ala make before break)

As stated in the draft and the slides, we have never claimed that the proposition remove all micro-loops. It removes/reduces some.
There is a trade-off between the effectiveness of the solution and its cost.
Over the last 8 years, the wg has worked on micro-loops avoidance, proposing different solutions. The problem still exists. So may be a simple small proposition may bring some benefits (at least more than a complete non deployed/implemented one).

BTW, most solutions have considered the simple case of a single event IGP. I'd be curious to have the analysis of how they behave in case of multiple events when different routers compute their SPF at different times on different LSDB hence with more than 2 forwarding states in the network. Independent forwarding planes may require multiple (N) forwarding planes (and without knowing how routers delay/group their SPF that may be not that simple. I would assume that draft-decraene-rtgwg-backoff-algo would be beneficial)). oFIB does not apply.

 Bruno

---

From: <[email-masked]>
Date: Tue, 29 Jul 2014 11:44:38 +0000

More inline ...

-----Original Message-----
From: Hannes Gredler [mailto:[email-masked]] 
Sent: Monday, July 28, 2014 17:07
To: LITKOWSKI Stephane SCE/IBNF
Cc: DECRAENE Bruno IMT/OLN; Russ White; [email-masked]
Subject: Re: On minimizing SPF backoff induced blackouts

the only reason it is necessary is that some implementations (ours included) do some state compression, such that there is a collection window of 20ms of local-events. strictly speaking thats just a efficiency/responsiveness tradeoff and we got some requests in the past to lower that window.

| 	If you go to something like 0 or 5 msec, we may fall into 
| implementations issues with constant READ/WRITE contentions in FIB 
| that often cause corner cases where the router is completely lost in 
| what it must do

again thats implementation dependent. if you have now state-compression on your RIB/FIB path then you're doomed anyway. state of the art is that you do not de-queue route 'N(t)' if route 'N(t+1)' is available, but cancel N(t) update.

see and this i do not believe is possible.
whatever <N> SPF initial wait will be, there will be always incidents where the LSP triggers arrives after <N>, so there will be always corner cases for loops.

[SLI] Right, there may always be some corner case, but I think the number of occurences is low ... I can only think because we do not have statistics yet from routers :)

| - Increasing delay is also necessary. By experience there are 
| situations where you need absolutely to calm down computation to 
| prevent churn amplification (even today with stronger CPUs)

no doubt - the question is when do we need to kick in that self-protection layer ... is it after reception of the 2nd LSP or after the 10th ?

[SLI] That's the good question :) We are doing such tuning by experience for years now.
The only point is that the behavior must be aligned between all nodes and trying to figure out worst case (slowest routers)

rapid-run in a certain way does because it is widening your window of 'concurrent, unrelated network events'
without going into backup mode.

[SLI] Right, but the gap between rapid-run and backup-mode is really high.

as tony (and others) have pointed out ... it will only work during nice-weather conditions ... as soon as you have the perfect-storm, you will have micro loops (and all the pain you try to protect yourself) - therefore i'd be much more in favour of 'something else' which gets us to zero uloops, and for that i am afraid we need to have the notions of independent forwarding planes (ala make before break) -

trying to synchronize RIB/FIB update across different CPUs/routing-stacks/vendors/load conditions is a battle that can never be won.

[SLI] This is not a definitive solution for microloops, just a simple quick win to remove somes.
I'm still dreaming on a simple definitive solution ... but I don't see any ... oFIB or synchronized FIBs sounds to not really have a good of support for implementations ...
In the meantime, I do think there are small and fast areas of improvments even if not definitive solutions. 

Stephane

| -----Original Message-----
| From: rtgwg [mailto:[email-masked]] On Behalf Of Hannes 
| Gredler
| Sent: Thursday, July 24, 2014 15:03
| To: DECRAENE Bruno IMT/OLN; Russ White
| Cc: [email-masked]
| Subject: RE: On minimizing SPF backoff induced blackouts
| 
| hi bruno,
| 
| IMO the problem that you're trying to solve is to have the routers still being responsive when multiple events are happening in the network.
| 
| wouldn't then a simple fix be to kick the SPF pacing logic at a much 
| more later point in time ? - i.e. rather than slowing down things down 
| on event #2 or #3, do it at event #<N> (please insert you're favorite 
| number for <N>)
| 
| /hannes
| 
| ________________________________________
| From: rtgwg <[email-masked]> on behalf of 
| [email-masked] <[email-masked]>
| Sent: Thursday, July 24, 2014 20:56
| To: Russ White
| Cc: [email-masked]
| Subject: RE: On minimizing SPF backoff induced blackouts
| 
| > > I'm willing to be persuaded, I just don't see the argument for 
| > > specifying an algorithm with what's been put on the table to this point.
| >
| > Ok. So I guess my presentation/draft was not clear enough.
| 
| Let me try again.
| In slide 5: 
| http://tools.ietf.org/agenda/90/slides/slides-90-rtgwg-2.pdf
| - do we agree that it would be better if node A and B both schedule their SPF at roughly the same time? i.e. wait for the same duration?
| - if so, what would be your proposition?
| 
| Bruno
| 
| 
| ______________________________________________________________________
| ___________________________________________________
| 
| Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
| 
| This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
| If you have received this email in error, please notify the sender and delete this message and its attachments.
| As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
| Thank you.
| 
| _______________________________________________
| rtgwg mailing list
| [email-masked]
| https://www.ietf.org/mailman/listinfo/rtgwg
| 
| _______________________________________________
| rtgwg mailing list
| [email-masked]
| https://www.ietf.org/mailman/listinfo/rtgwg
| 
| ______________________________________________________________________
| ___________________________________________________
| 
| Ce message et ses pieces jointes peuvent contenir des informations 
| confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
| exploites ou copies sans autorisation. Si vous avez recu ce message 
| par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
| 
| This message and its attachments may contain confidential or 
| privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
| If you have received this email in error, please notify the sender and delete this message and its attachments.
| As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
| Thank you.
|

---

From: Antoni Przygienda <[email-masked]>
Date: Tue, 29 Jul 2014 15:04:19 +0000

Well informed discussion by now ;-) just one inline comment 

> as tony (and others) have pointed out ... it will only work during nice-weather
> conditions ... as soon as you have the perfect-storm, you will have micro
> loops (and all the pain you try to protect yourself) - therefore i'd be much
> more in favour of 'something else' which gets us to zero uloops, and for that i
> am afraid we need to have the notions of independent forwarding planes
> (ala make before break) -
> 
> I'm still dreaming on a simple definitive solution ... but I don't see any ... oFIB
> or synchronized FIBs sounds to not really have a good of support for
> implementations ...

[Tony said]  The battle of synchronizing the nodes (and the network in a sense) cannot be won since beside dealing with asynchronous networks (ah, the halcyon days of TDM ;-) we are dealing here with hugely asynchronous systems within networks (by now). The days of same single-core processor doing everything on the side while running fast-path written in assembly by a pizza-fed wizard are long, long past (but then, 200 msec before flooding something & other guy being able to look @ it was considered quite normal ;-).  Looking @ e.g. our architecture I am having fun discussions along the lines "I gave you ACK but you know, it only means I processed some of the stuff you gave me & then I have to do all those other things until it's in FIB & fwd'ing but you surely don't want me to wait until it's done before I ACK you. So, what does an 'ACK' really mean here ?" ;-)   We can get 'some' amount of better synchronization (and again, we better not get too good @ it, a perfectly sync'ed network on LSA refreshes or HELLOs is _not_ a fun thing to debug ;-) and that's what this work will need to settle for.  The important things in the work are IMO

	* make sure you don't aim for _perfect_ synchronization but some small jitter to avoid network-wide Dirac pulses on your control planes. Even if you don't, I'm pretty sure the asynchronous nature of today's architectures will confound you. All kind of hysteresis like Hannes said is built into all the large systems today (packing, pipelining of async comms, state batching, reordering of state updates to preserve FIB integrity, chip restrictions & so on). 
	* allow for 'how many new LSAs and/or a timebound' is my 'normal failure'  knob for the operator.  Those numbers may shift dramatically. If I'm running large numbers of IGP shortcuts over a link, I may end up with tons of stuff on my plate on a single phy link failure before I want my computation to jump-start.  Yes, normal will be probably, one link - jump @ it if you're @ cool temperature but everybody is doing that today already pretty much so it's more 'what's bad enough to start to back-off'. 

Looking fwd' to what will emerge as practical proposed backoff algorithm here

--- tony