Subject: Re: About OSPF route flapping

From: Erblichs <[email-masked]>
Date: Tue, 20 Dec 2005 15:02:53 -0800

Yasu,

	I think I read this paper before and wanted to ask
	you and the group a few questions. Why are you suggesting
	black-holing routes and not suggesting a SIMPLER approach?

	My suggested simpler approach is #4.

	0) Are't you effectively black-holing the route
	   and the routers behind the "flapping link"?

	1) If a link, route is flapping and it is a low traveled
	   data link (almost a demand circuit) what is the harm
	   if the link is periodicly down for short periods of
	   time?
	
	2) If the link is unstable because it is a "wireless"
	   type link with large packet losses, won't you
	   effectively isolate the wireless links if their
	   are no alternative routes?

	3) If the link has random periods of instability isn't
	   your dampening increasing the period that the link
	   CAN'T be used?

	3b) If the link has become stable after a period of
	    unstability, won't you have an extended period of 
	    down time?

	Maybe an upcoming Informational RFC if I FIND the time..
	4) WHY don't you just increment the cost of the link
	   during periods of instability? Maybe, a demand
	   type link that doesn't have to be up 100% of the time.
	
	   Yes, you have to increment the cost as the time of
	   instability increases or assume an initial level of
	   instability and as the link ages, decrease the link
	   cost.
	   
	   * Shouldn't this minimize any effects and supply
	     as much activeness as possible?

	   * As a followup, if hellos are lost and no data
	     is pending to be sent out (must be implimented
	     on all sides of the link), delay dropping the
	     adj. I am assuming that the period of hellos
	     could be too short for short periods of high
	     levels of traffic.

	    * Followup, if just a few LSAs have been sent out
	      this questionable link and no hellos or acks
	      have been returned, identify whether their are
	      any alternatives for the queued pkts. If their
	      is drop the link, else delay dropping the link?

	Thus, did BGP do the right thing? It is difficult to
	make a quick decision because TCP will rexmit lost or
	non acked pkts/segments within time. And TCP will force
	an initial handshake before data can be sent. Plus
	we are on a OSPF mail alias!

	Mitchell Erblich
	---------------------------

Yasuhiro Ohara wrote:
> 
> We once checked the relation between link flapping and
> OSPF fixed timer. In the paper we solved that by applying
> dampening function similar with BGP's.
> 
> http://www.sfc.wide.ad.jp/~yasu/saint2003.pdf
> 
> The experiment was done by Zebra ospf6d.
> Unfortunately I removed the dampening code since then,
> because there seemed no interest.
> 
> Let me know if you want further info.
> 
> regards,
> yasu
> 
> From: Kouzengjie <[email-masked]>
> Subject: About OSPF route flapping
> Date: Thu, 24 Nov 2005 15:02:18 +0800
> 
> > hi,all
> >     How does OSPF route flapping affect?
> >     Is it important?
> >     what is the method for solving it?
> >
> > thanks,
> > Kou
> >

---

From: Manav Bhatia <[email-masked]>
Date: Wed, 21 Dec 2005 12:04:25 +0530

Hi  Erblichs,

Each time an IGP route/link changes (could be because of a flap) there are a
large number of BGP routes that get affected because of this. If the number
of routes that get affected is small then it isn't a problem. However, given
the current Internet routing table size its likely that a change in IGP will
lead to a change in a *large* number of BGP routes. Processing this and
updating the FIB on the line cards, etc. can take up quite some time and it
may be long enough to prevent time sensitive packets (HELLOs , etc) from
being sent, which may in the worst case cause adjacencies to drop.

Its not as bad as it sounds and a clever implementation will do it nicely,
however my point is to emphasize that an IGP change can cause a LOT of
problems when BGP is around.

The above would explain, which I am sure you already understand, why a
flapping link needs to be damped - or looked into.

Coming back to damping scheme. One could apply a less aggressive suppression
scheme to the case where no alternate route exists. In the simplest case, a
more aggressive suppression could be applied if some alternate route exists.

If the only penalty to a flapping link/route is an increase in the cost then
this still does not solve our BGP problem. Given no other IGP route, BGP
will continue using the unstable IGP route (even if its cost has increased).

!> 
!> 	I think I read this paper before and wanted to ask
!> 	you and the group a few questions. Why are you suggesting
!> 	black-holing routes and not suggesting a SIMPLER approach?
!> 
!> 	My suggested simpler approach is #4.
!> 
!> 	0) Are't you effectively black-holing the route
!> 	   and the routers behind the "flapping link"?
!> 
!> 	1) If a link, route is flapping and it is a low traveled
!> 	   data link (almost a demand circuit) what is the harm
!> 	   if the link is periodicly down for short periods of
!> 	   time?

For a demand link one could configure different sets of thresholds. Such
routes/links are expected to change the state somewhat more often, but
should be suppressed if continuous state changes indicate instability.

!> 	
!> 	2) If the link is unstable because it is a "wireless"
!> 	   type link with large packet losses, won't you
!> 	   effectively isolate the wireless links if their
!> 	   are no alternative routes?
!> 
!> 	3) If the link has random periods of instability isn't
!> 	   your dampening increasing the period that the link
!> 	   CAN'T be used?
!> 
!> 	3b) If the link has become stable after a period of
!> 	    unstability, won't you have an extended period of 
!> 	    down time?

I'll answer all the 3 together:

Its not that a single flap will cause the link/route to be removed. The
amount of instability that you are willing to accept before you damp the
link/route is totally configurable. Also the amount of time you want this
link/route to be suppressed is completely configurable. Thus by changing the
initial set of values given to the algorithm you could have a (i) highly
sensitive algorithm that quickly recognizes an unstable entity and
suppresses it and keeps is suppressed for a long time (ii) a flexible
algorithm that accepts a number of flaps before it decides to suppress the
route/link and quickly announces it if it remains active for a little time.

Lets go back to the basic question. How can one accurately predict the
future stability of a link/route? The recent history of how stable the
route/link has been is generally regarded as a good basis for estimating the
likelihood of future stability. The criterion that is used to distinguish
stable routes/links from the unstable ones is therefore based on the recent
history of stability of the route/link. So, a link/route is damped only if
the past history shows that its likely for the link/route to again flap in
future.

!> 
!> 	Maybe an upcoming Informational RFC if I FIND the time..
!> 	4) WHY don't you just increment the cost of the link
!> 	   during periods of instability? Maybe, a demand
!> 	   type link that doesn't have to be up 100% of the time.
!> 	
[clipped]

!> 
!> 	   * As a followup, if hellos are lost and no data
!> 	     is pending to be sent out (must be implimented
!> 	     on all sides of the link), delay dropping the
!> 	     adj. I am assuming that the period of hellos
!> 	     could be too short for short periods of high
!> 	     levels of traffic.

I am not sure if I understand this.

!> 
!> 	    * Followup, if just a few LSAs have been sent out
!> 	      this questionable link and no hellos or acks
!> 	      have been returned, identify whether their are
!> 	      any alternatives for the queued pkts. If their
!> 	      is drop the link, else delay dropping the link?

I don't see a reason why you would want to delay dropping such a link. In
case of BGP, all the BGP routes will get blackholed. 

Regards,
Manav

!> 
!> 	Thus, did BGP do the right thing? It is difficult to
!> 	make a quick decision because TCP will rexmit lost or
!> 	non acked pkts/segments within time. And TCP will force
!> 	an initial handshake before data can be sent. Plus
!> 	we are on a OSPF mail alias!
!> 
!> 	Mitchell Erblich
!> 	---------------------------
!> 
!> 	
!> 
!> 	
!> 
!> Yasuhiro Ohara wrote:
!> > 
!> > We once checked the relation between link flapping and
!> > OSPF fixed timer. In the paper we solved that by applying
!> > dampening function similar with BGP's.
!> > 
!> > http://www.sfc.wide.ad.jp/~yasu/saint2003.pdf
!> > 
!> > The experiment was done by Zebra ospf6d.
!> > Unfortunately I removed the dampening code since then,
!> > because there seemed no interest.
!> > 
!> > Let me know if you want further info.
!> > 
!> > regards,
!> > yasu
!> > 
!> > From: Kouzengjie <[email-masked]>
!> > Subject: About OSPF route flapping
!> > Date: Thu, 24 Nov 2005 15:02:18 +0800
!> > 
!> > > hi,all
!> > >     How does OSPF route flapping affect?
!> > >     Is it important?
!> > >     what is the method for solving it?
!> > >
!> > > thanks,
!> > > Kou
!> > >

___________________________________________________________ 
Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com

---

From: Yasuhiro Ohara <[email-masked]>
Date: Sat, 24 Dec 2005 16:20:14 +0900

Hi Erblichs,

Here I try to respond your requests by describing my opinion
beside what manav have told already.

Please see inline.

From: Erblichs <[email-masked]>
Subject: Re: About OSPF route flapping
Date: Tue, 20 Dec 2005 15:02:53 -0800
Message-ID: <[email-masked]>

erblichs> Yasu,
erblichs> 
erblichs> 	I think I read this paper before and wanted to ask
erblichs> 	you and the group a few questions. Why are you suggesting
erblichs> 	black-holing routes and not suggesting a SIMPLER approach?

We were not suggesting black-holing, we suggest to apply BGP flap
dampening also to OSPF events. Black-holing in the experiment
just happened because of time lag between result of OSPF computation
and the actual link state, and it is regarded as a bad phenomenon.

Black-holing is bad because it keep us from using alternative routes.
We applied flap dampening so that there would be no black-holing.

erblichs> 	My suggested simpler approach is #4.
erblichs> 
erblichs> 	0) Are't you effectively black-holing the route
erblichs> 	   and the routers behind the "flapping link"?

Please see above. Black-holing was the bad situation resulted from
the flapping route.

erblichs> 	1) If a link, route is flapping and it is a low traveled
erblichs> 	   data link (almost a demand circuit) what is the harm
erblichs> 	   if the link is periodicly down for short periods of
erblichs> 	   time?
erblichs> 	
erblichs> 	2) If the link is unstable because it is a "wireless"
erblichs> 	   type link with large packet losses, won't you
erblichs> 	   effectively isolate the wireless links if their
erblichs> 	   are no alternative routes?

As manav have told, we can configure the threshold of how unstable
for the link event to be dampend, and how quickly it starts to damp.
What types of link should be dampened or not is beyond the scope
of the paper, but I agree that on demand circuit we would not want
to dampen the link event. On wireless the dampening feature might be
useful, I guess.

Regardless of whether it is wireless or not, it is important
if we have another alternative route to the same destination other
than the flapping route. If we don't have any alternative, then
the dampening should leave the link "up" against ordinary
dampening method which would leave the link "down".
In the paper we only investigate the case when we have a alternative
route, so the dampening is made to leave the link as down.

I think I don't understand what you've meant by "isolate",
so please tell me if I am missing your point.

erblichs> 	3) If the link has random periods of instability isn't
erblichs> 	   your dampening increasing the period that the link
erblichs> 	   CAN'T be used?

Hmm I don't understand 'random periods of instability' ... What model
are you mentioning precisely ?

If the duration of previous up/down state is short enough compared to
configured threshold settings, then the dampening function will
increase the period that the link can't be used. Otherwise,
when the previous down duration is long enough, and when the link event
just occured is up, the link would be used immediately (dampening function
does not effect).

erblichs> 	3b) If the link has become stable after a period of
erblichs> 	    unstability, won't you have an extended period of 
erblichs> 	    down time?

Yes, we will have an extended period of down time just after the
unstability. That is from the nature of dampening algorithm.
We will leave the link as down even if the link is up
until the penalty of the link decreases as time passes.

erblichs> 	Maybe an upcoming Informational RFC if I FIND the time..
erblichs> 	4) WHY don't you just increment the cost of the link
erblichs> 	   during periods of instability? Maybe, a demand
erblichs> 	   type link that doesn't have to be up 100% of the time.
erblichs> 	
erblichs> 	   Yes, you have to increment the cost as the time of
erblichs> 	   instability increases or assume an initial level of
erblichs> 	   instability and as the link ages, decrease the link
erblichs> 	   cost.
erblichs> 	   
erblichs> 	   * Shouldn't this minimize any effects and supply
erblichs> 	     as much activeness as possible?

I think increasing the link cost as the link's instability increases
is not a good thing, because the OSPF cost of the link have totally
different meaning in essence.

The OSPF's link cost have different measure compared to instability
(flapping rate), so intuitively we can't explain what does it mean
to reflect flapping rate to the OSPF cost. Reflecting flapping rate
to the OSPF cost will result that, say, we will use the flapping link
if the flapping duration is longer than every 3 second. Actual flapping
rate (or duration) that we will shift to alternative route is decided
by the alternative route's total OSPF link cost. Is that making any
sense ? I don't think so.

The only benefit we get from manipulating OSPF cost would be
in the discussion of "leave up or leave down". If we make OSPF cost
as LsInfinity, it means that the link is up but almost down.
Using LsInfinity is good, but about reflecting flapping rate to OSPF cost,
I would disagree.

erblichs> 	   * As a followup, if hellos are lost and no data
erblichs> 	     is pending to be sent out (must be implimented
erblichs> 	     on all sides of the link), delay dropping the
erblichs> 	     adj. I am assuming that the period of hellos
erblichs> 	     could be too short for short periods of high
erblichs> 	     levels of traffic.
erblichs> 
erblichs> 	    * Followup, if just a few LSAs have been sent out
erblichs> 	      this questionable link and no hellos or acks
erblichs> 	      have been returned, identify whether their are
erblichs> 	      any alternatives for the queued pkts. If their
erblichs> 	      is drop the link, else delay dropping the link?

Why do you consider hello events ? We should consider link up/down events
or route appearing/disappearing events, not hello events.
We might postpone/defer link or route events, but we will not postpone
hello events. Further, how do you detect that a Hello packet has been
lost ? "if hellos are lost" can be detected only by timer, and it
should mean the link down event. So we don't need to consider about
hello event at all.

erblichs> 	Thus, did BGP do the right thing? It is difficult to
erblichs> 	make a quick decision because TCP will rexmit lost or
erblichs> 	non acked pkts/segments within time. And TCP will force
erblichs> 	an initial handshake before data can be sent. Plus
erblichs> 	we are on a OSPF mail alias!
erblichs> 
erblichs> 	Mitchell Erblich
erblichs> 	---------------------------

Detecting if the packet is lost is difficult, almost impossible.
but BGP's flap dampening does not need to detect them.
It detect only receiving update or withdrawal, and only dampening
if the duration is too short or rate is too high.
TCP is not involved if the BGP packet have already been processed.

regards,
yasu

erblichs> 	
erblichs> 
erblichs> 	
erblichs> 
erblichs> Yasuhiro Ohara wrote:
erblichs> > 
erblichs> > We once checked the relation between link flapping and
erblichs> > OSPF fixed timer. In the paper we solved that by applying
erblichs> > dampening function similar with BGP's.
erblichs> > 
erblichs> > http://www.sfc.wide.ad.jp/~yasu/saint2003.pdf
erblichs> > 
erblichs> > The experiment was done by Zebra ospf6d.
erblichs> > Unfortunately I removed the dampening code since then,
erblichs> > because there seemed no interest.
erblichs> > 
erblichs> > Let me know if you want further info.
erblichs> > 
erblichs> > regards,
erblichs> > yasu
erblichs> > 
erblichs> > From: Kouzengjie <[email-masked]>
erblichs> > Subject: About OSPF route flapping
erblichs> > Date: Thu, 24 Nov 2005 15:02:18 +0800
erblichs> > 
erblichs> > > hi,all
erblichs> > >     How does OSPF route flapping affect?
erblichs> > >     Is it important?
erblichs> > >     what is the method for solving it?
erblichs> > >
erblichs> > > thanks,
erblichs> > > Kou
erblichs> > >
erblichs>