Subject: [bmwg] Request for Opinions on Benchmarking the BFD protocol.

From: "Esa Salahuddin \(esalahud\)" <[email-masked]>
Date: Mon, 5 Mar 2007 14:26:58 -0500

Hi folks,
 
Jay and I would like to hear your opinion on the idea of benchmarking
the BFD protocol in the BMWG. 
 
The BFD protocol is becoming an increasingly popular alternative to a
variety of existing methods to detect forwarding failures on links
between network elements. It is being proposed as the forwarding failure
detection mechanism for a host of routing protocols, virtual links and
p2mp topologies. It is a very useful in the situations where underlying
physical media doesn't lend itself well to fast detection of failures.
You can find extensive information on the BFD protocol itself at the BFD
WG weblink http://tools.ietf.org/wg/bfd/
 
Please see
http://www.ietf.org/internet-drafts/draft-salahuddin-bmwg-bfd-motivation
-00.txt for the deails on the topic and our position and please let us
know what do you think.
 
thx, esa
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 05 Mar 2007 16:30:41 -0500

In message <[email-masked]>
"Esa Salahuddin \(esalahud\)" writes:
> http://www.ietf.org/internet-drafts/draft-salahuddin-bmwg-bfd-motivation-00.txt
> for the deails on the topic and our position and please let us 
> know what do you think.
>  
> thx, esa

BFD is part of a bigger picture.  If you want to do useful
benchmarking you would also want to test against what BFD is intended
for, namely control plane failure.  You need a way to fail a control
plane in an uncontroled manner.  In practice yanking a card will do
the trick for most routers.  Its hard to get the exact time of the
card yank.  In routers which switch to a backup route processor, the
change may not be externally visible with or without BFD.

It would not be useful to benchmark BFD as described without more
thoroughly benchmarking control plane failure, including control plane
failure in routers that support some form of fallback to another
procesor and routers that support graceful restart of protocols.

In a given test topology the amount of data loss resulting from the
yank of a control processor could be measured, with and without BFD.
This would involve yanking a card for the test or otherwise making the
control plane fail.  Loss should be zero with many modern routers
assuming there is no loss due to IGP convergence (not applicable if
MPLS/TE is used).

For routers that fall back to another processor, two measures would be
relevant, apparent control plane dead time and actual control plane
dead time.  The apparent control plane dead time is the time after
failure during which it does not pass protocol changes.  For some
routers and protocols this might be zero (forwarding cards handle
adjacencies) but is likely to be non-zero if protocol changes are sent
through the route processor.  For example some routers may take OSPF
changes and populate a BGP Adj-In but not readvertise this out other
interfaces.  The actual dead time would be the time in which the route
is passing protocol changes but is not installing the route changes.
These times might be quite long for some routers.  Note that these
only matter if the network is not quiescent.

There are two ways the a router could deal with fall back to another
processor and BFD.  The router could fake BFD by having forwarding
cards handle BFD which may to some extent defeat the purpose.  They
could also do graceful restart, allow BFD to allow the failure to be
detected, and just come back to life fairly quickly.

For routers that do not fall back to another processor but do support
graceful restart, the same apparent control plane dead time and actual
control plane dead time metrics can be applied.  If only one control
processor is supported, then yanking a card could not be used in the
test, however some other technique such as driving a non-maskable
interrupt pin on the processor should be possible.

For routers that support neither fall back to another processor or
graceful restart there will be a route disruption in which other
routers must route around it.

The protocol related response times could be measured by providing a
fairly constant but slow flow of artificial protocol changes, /30
route changes for example.  Route install can be detected using a low
bandwidth flow of traffic with packets from each of the routes being
changed.  For example you could use up to 4 million /30 routes within
net 10 (use 192.168 for the testbed itself in this case).  Changing
one route every msec you could run a test covering a 4,000 second
interval (so you wouldn't need 4 million if the router recovers in
under a minute and the test can be run in a few minutes considering it
will take time after starting the test to pull a card).  

You would have to change one route every 10 msec or longer on most
routers (burst of lots of change are fine on these but no break in
route changes can cause problems in itself) and you'd normally see
some delay readvertising and installing routes.  Of course changing
millions of MPLS/TE LSP as part of this test is probably not feasible.
Very large numbers of LDP FEC should work.

Note that fall back support may depend on what protocols are running
and the same is true of graceful restart.

BFD improves detection time on a route processor failure but its only
part of what happens when a route processor fails.  IMHO benchmarking
BFD in isolation is not very useful.  Benchmarking the whole control
plane failure response would be very useful but its a much bigger job.

Curtis

---

From: "Aamer Akhter \(aakhter\)" <[email-masked]>
Date: Mon, 5 Mar 2007 18:35:40 -0500

Curtis,

It seems to me what you're really asking for is NSF/graceful restart benchmarking. 

Keep in mind that BFD is for forwarding plane failures, if the control-plane fails (ie just OSPF drops dead, BFD will probably not even catch this). The intersection of GR+BFD+control-plane that you're looking at is a very specific case. In fact, there are architectures where BFD will keep on running even if the RP dies.

Apologies if I'm sticking words into your mouth :-)

Regards,

-- 
Aamer Akhter / [email-masked]
Ent & Commercial Systems, cisco Systems

> -----Original Message-----
> From: Curtis Villamizar [mailto:[email-masked]]
> Sent: Monday, March 05, 2007 4:31 PM
> To: Esa Salahuddin (esalahud)
> Cc: [email-masked]
> Subject: Re: [bmwg] Request for Opinions on Benchmarking the BFD
> protocol.
> 
>

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Mar 2007 10:52:54 -0500

In message <[email-masked]>
"Aamer Akhter \(aakhter\)" writes:
> Keep in mind that BFD is for forwarding plane failures, if the
> control-plane fails (ie just OSPF drops dead, BFD will probably not
> even catch this). The intersection of GR+BFD+control-plane that you're
> looking at is a very specific case. In fact, there are architectures
> where BFD will keep on running even if the RP dies.
>  

I just looked at the latest (expired) BFD draft and you are right in
that BFD is now divorced from the route processor.

You are corrent that it would then make more sense to separately
benchmark the speed of recovery from route processor failure.

Sorry for the distraction.

Curtis

---

From: "Esa Salahuddin \(esalahud\)" <[email-masked]>
Date: Tue, 6 Mar 2007 23:12:10 -0500

Curtis/Aamer,

It is true that the BFD protocol is designed to detected forwarding
plane failures. Infact, one of the benchmarks we are suggesting is this
specific behavior where BFD implementation doesn't share the fate of the
control plane failure.

I think Curtis suggestion of control plane failures benchmarking does
make sense though, and if anything, the BFD can be used to trigger the
(client) control plane failures at a precise instance. Also,
benchmarking forwarding plane failures using a uniform detection
mechanism like BFD could be the first step towards benchmarking of the
overall system response to failures in control plane. 

Thx, esa

-----Original Message-----
From: Curtis Villamizar [mailto:[email-masked]] 
Sent: Tuesday, March 06, 2007 10:53 AM
To: Aamer Akhter (aakhter)
Cc: [email-masked]; Esa Salahuddin (esalahud); [email-masked]
Subject: Re: [bmwg] Request for Opinions on Benchmarking the BFD
protocol. 

In message
<[email-masked]>

Curtis

---

From: "Aamer Akhter \(aakhter\)" <[email-masked]>
Date: Tue, 6 Mar 2007 23:30:27 -0500

Esa,

Couldn't agree more. I think we have to keep the end goal/metric of 'time to recovery' in mind. Traditionally this has been broken down into:

Detection (where BFD sits)
Propagation (flooding, path calculations etc)
Repair (route install, lsp install etc)

What we want to be able to do is measure (as much as possible) each one of these (for the purposes of interchangeable technologies) as well as look at the end to end measurement (due to lack of granular vision into each implementation, eg total packet loss). I think for that, the existing TE-FRR draft would be good work to follow.

> From: Esa Salahuddin (esalahud)
> Sent: Tuesday, March 06, 2007 11:12 PM
> To: [email-masked]; Aamer Akhter (aakhter)
> Cc: [email-masked]
> Subject: RE: [bmwg] Request for Opinions on Benchmarking the BFD
> protocol.
> 
> Curtis/Aamer,
> 
> specific behavior where BFD implementation doesn't share the fate of
> the control plane failure.
>

---

From: "Arun Gandhi" <[email-masked]>
Date: Wed, 7 Mar 2007 09:18:19 -0500

Hi Esa, Jay:

I support this as a working group item and propose that we break it down
into two drafts or two parts of the same document: (i) Motivation,
definitions & key areas to benchmark - single & multi-hop (ii) Benchmarking
methodologies for characterizing the BFD protocol. In nutshell, the main
motivation of BFD is the "time to recovery" via fast failure detection
whether it's in forwarding plane (main objective though) or control plane.

I'd be happy to affectively contribute to these efforts (more specifically
item 2 above) as have been working on it. 

Regards,

Arun Gandhi

Juniper Networks

  _____  

From: Esa Salahuddin (esalahud) [mailto:[email-masked]] 
Sent: Monday, March 05, 2007 2:27 PM
To: [email-masked]
Subject: [bmwg] Request for Opinions on Benchmarking the BFD protocol.

Hi folks,

Jay and I would like to hear your opinion on the idea of benchmarking the
BFD protocol in the BMWG. 

The BFD protocol is becoming an increasingly popular alternative to a
variety of existing methods to detect forwarding failures on links between
network elements. It is being proposed as the forwarding failure detection
mechanism for a host of routing protocols, virtual links and p2mp
topologies. It is a very useful in the situations where underlying physical
media doesn't lend itself well to fast detection of failures. You can find
extensive information on the BFD protocol itself at the BFD WG weblink
http://tools.ietf.org/wg/bfd/

Please see
http://www.ietf.org/internet-drafts/draft-salahuddin-bmwg-bfd-motivation-00.
txt for the details on the topic and our position and please let us know
what you think.

thx, esa

---

From: "Jay Karthik \(jakarthi\)" <[email-masked]>
Date: Wed, 7 Mar 2007 23:32:47 -0500

Hi Arun,
 
Thank you for the remarks.
 
We actually were thinking along similar lines, but didn't want to come
out with a methodology draft till we determine whether (or not) there is
enough interest around in the area of benchmarking the BFD technology.
Initial reaction of the group seems positive, but we would like to give
others ample time to express their opinion and to build consensus.
 
Once the idea of benchmarking BFD gathers enough momentum in the group,
we'll like to team up those who are interested.
 
Cheers, Jay

________________________________

	From: Arun Gandhi [mailto:[email-masked]] 
	Sent: Wednesday, March 07, 2007 9:18 AM
	To: Esa Salahuddin (esalahud); [email-masked]
	Cc: [email-masked]
	Subject: RE: [bmwg] Request for Opinions on Benchmarking the BFD
protocol.

	Hi Esa, Jay:

	I support this as a working group item and propose that we break
it down into two drafts or two parts of the same document: (i)
Motivation, definitions & key areas to benchmark - single & multi-hop
(ii) Benchmarking methodologies for characterizing the BFD protocol. In
nutshell, the main motivation of BFD is the "time to recovery" via fast
failure detection whether it's in forwarding plane (main objective
though) or control plane.

	I'd be happy to affectively contribute to these efforts (more
specifically item 2 above) as have been working on it. 

	Regards,

	Arun Gandhi

	Juniper Networks

________________________________

	From: Esa Salahuddin (esalahud) [mailto:[email-masked]] 
	Sent: Monday, March 05, 2007 2:27 PM
	To: [email-masked]
	Subject: [bmwg] Request for Opinions on Benchmarking the BFD
protocol.

	Hi folks,

	Jay and I would like to hear your opinion on the idea of
benchmarking the BFD protocol in the BMWG. 

	The BFD protocol is becoming an increasingly popular alternative
to a variety of existing methods to detect forwarding failures on links
between network elements. It is being proposed as the forwarding failure
detection mechanism for a host of routing protocols, virtual links and
p2mp topologies. It is a very useful in the situations where underlying
physical media doesn't lend itself well to fast detection of failures.
You can find extensive information on the BFD protocol itself at the BFD
WG weblink http://tools.ietf.org/wg/bfd/

	Please see
http://www.ietf.org/internet-drafts/draft-salahuddin-bmwg-bfd-motivation
-00.txt for the details on the topic and our position and please let us
know what you think.

	thx, esa

---

From: "Andrey Kiselev \(ankisele\)" <[email-masked]>
Date: Thu, 8 Mar 2007 19:20:33 -0500

Hi Esa/Jay,
 
I think there is a need in BFD benchmarking standardization, so your
document and subsequent drafts coming later will be useful. 
 
I have a few questions/comments on the document itself.
 
Key BFD benchmarks section, 3.1. Detection Time: 
" Shortest detection time supported for a certain scale deployment" 
I have a question here. How scale of deployment affects BFD detection
time for a given link? Or are you referring to a multihop scenario here?
 
Items 3.2 (Mode of operation), 3.3 (Echo function capability) 3.4.
(Control Plane Independence)  3.9. (Multi-Hop BFD sessions) look more
like features of BFD implementation to me, rather than benchmarks.
Benchmarks is something you can actually measure in numbers, in my
opinion. Correct me, if I am wrong here.

Section 4 Key BFD Use Scenarios:
If you mentioned AToM, should L2VPNs like VPLS and VPWS also be in the
list?
 
Thanks,
 
Andrey 

From: 

Hi folks,

 
Please see
 
thx, esa

---

From: "Esa Salahuddin \(esalahud\)" <[email-masked]>
Date: Mon, 12 Mar 2007 11:08:50 -0400

Hi Andrey,
 
Sorry about the late response and thank you for supporting the effort.
You have raised some very good point points. Please, see my response
inline.
 
thx, esa

	From: Andrey Kiselev (ankisele) 
	Sent: Thursday, March 08, 2007 7:21 PM
	To: [email-masked]
	Cc: Esa Salahuddin (esalahud); Jay Karthik (jakarthi)
	Subject: RE: [bmwg] Request for Opinions on Benchmarking the BFD
protocol.

	Hi Esa/Jay,
	 
	I think there is a need in BFD benchmarking standardization, so
your document and subsequent drafts coming later will be useful. 

	Key BFD benchmarks section, 3.1. Detection Time: 
	" Shortest detection time supported for a certain scale
deployment" 
	I have a question here. How scale of deployment affects BFD
detection time for a given link? Or are you referring to a multihop
scenario here?
	
	ESA::: Ideally it shouldn't, practically it does and that is why
it is a good bench mark to see how good an implementation is when
supporting say 100,000 sessions compared to supporting 100 or so. For me
personally, this bench mark alone tells me a lot about the scalability
and quality of a typical BFD implementation. 
	 
	Items 3.2 (Mode of operation), 3.3 (Echo function capability)
3.4. (Control Plane Independence)  3.9. (Multi-Hop BFD sessions) look
more like features of BFD implementation to me, rather than benchmarks.
Benchmarks is something you can actually measure in numbers, in my
opinion. Correct me, if I am wrong here.
	
	ESA::: I think you are right in the sense these benchmarks
referenced as such have more of a yes/no answer than a number that you
can measure and compare. However, consider the detection time variance
in benchmarking single hop vs. multi hop OR the detection times
measurement when BFD is operating in one mode vs. the other OR a
benchmark that just can't be measured through a specific test method
where control plane should have gone down.  So, i guess these questions
will have to be answered while benchmarking the BFD technology. You are
right in that they may not qualify as independent benchmarks in
themselves and that should be addressed in next revision of the draft.
	
	Section 4 Key BFD Use Scenarios:
	If you mentioned AToM, should L2VPNs like VPLS and VPWS also be
in the list?
	
	ESA::: I would say "No!". The target should be measuring the BFD
performance on the links single-hop/multi-hop/physical/virtual, and let
any control plane application be the customer of that performance. To
me, benchmarking BFD for VCs, both mpls/l2tpv3, would be the target.
Whether these VCs are part of a VPLS or VPWS network would be out of
scope for this effort. Let us see if anybody else has anything to say on
the topic.
	 
	Thanks,
	 
	Andrey 

	From: 

	Hi folks,

	 
	Please see
	 
	thx, esa