Subject: Re: [bmwg] Request for Opinions on Benchmarking the BFD protocol.

From: Jeffrey Haas <[email-masked]>
Date: Thu, 15 Mar 2007 14:19:23 -0400

On Thu, Mar 15, 2007 at 10:43:31AM -0700, Dave Katz wrote:
> The answer is *always* "It depends."  These systems are simply far  
> too complex to characterize in a meaningful way, and characterizing  
> them in a meaningless way seems like a waste of time.

Having been at the QA and Interop end of this and been bludgeoned by
vague specs, I'll agree wholeheartedly.  When benchmarks for heavily
intertwined systems are used as marketing mauls, they're not very
useful.  However, we can hope that what comes out of this are tools that
help implementors model scalability in their implementations.  From that
standpoint, there's some benefit.

> --Dave

-- Jeff

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Dave Katz <[email-masked]>
Date: Thu, 15 Mar 2007 10:43:31 -0700

I may as well pile on.  ;-)

The problem is that none of the benchmarks stand in isolation (they  
interact deeply) and thus are meaningless by themselves.  I suppose  
one could attempt to characterize combinations of parameters (how  
many sessions can an implementation seem to support at interval X)  
but the results are combinatoric, and even that is essentially  
meaningless without taking the larger system and network state into  
account.  Is the network stable?  What is the fan-in?  Are there a  
lot of route calculations taking place?  Are you initiating all of  
the sessions simultaneously?  Is the box terminating a lot of legacy  
SNA connections?

Furthermore, particularly as the intervals are made shorter, all  
kinds of accelerating nonlinearities in performance take place that  
will make the measurements highly variable (and thus unusable).

This is the oldest question in the industry ("How many OSPF routers  
can you support in an area?"  "It depends.") and has never been  
answered adequately, nor can it be.  Ultimately, all vendors can do  
is to rely on existing deployments ("FooNet has 500 routers and 1700  
links and it seems to work"), and quote very conservative deployment  
numbers, and anybody that says they can do anything more than this in  
any practical sense is lying.

IMHO.  ;-)

I guess I don't understand, at the meta-level, what the point of the  
benchmarking exercise is.  Is it to provide a framework for vendors  
to attempt to lie about, I mean present, performance data to the  
public?  It's hard for me to see why any vendor would be motivated to  
do so.  Reading the WG description, which explicitly says that the  
results have nothing to do with operational reality, makes me wonder  
what the point is.  But that's just me being old and grumpy and  
cynical and tired.

--Dave

On Mar 15, 2007, at 9:27 AM, David Ward wrote:

> Also not that testing a protocol w/ adaptive timers is also hard. The
> adaptive timers can be modified automagically in an implementation  
> according
> to spec w/ forewarning.
>
> -DWard
>
>
> On 3/15/07 10:19 AM, "Jeffrey Haas" <[email-masked]> wrote:
>
>> Jay (& bmwg-list),
>>
>> Please note that I'm not on the bmwg mailing list.
>>
>> Echo functionality: Benchmarking this functionality as part of BFD is
>> somewhat difficult in the sense that you cease benchmarking the BFD
>> mechanism and start benchmarking the forwarding mechanism.  The  
>> Min Echo
>> RX Interval, beyond signaling that the echo functionality is  
>> available,
>> is largely about "send packets no faster than this"; i.e. prevent the
>> loopback mechanism from being unnecessarily overwhelmed.
>>
>> Multi-hop BFD: I'd strongly suggest restricting the original
>> benchmarking effort to directly attached links.  Multi-hop BFD  
>> involves
>> a number of elements that would perturb measuring BFD performance  
>> such
>> as forwarding queueing and rates, the implementation of BFD in
>> individual forwarding elements vs. system-wide implementation, etc.
>>
>> Authentication support: I'd suggest including hardware assist for
>> authentication as an axis of measurement.
>>
>> -- Jeff
>>
>>
>> On Tue, Mar 06, 2007 at 01:41:28PM -0500, Jay Karthik (jakarthi)  
>> wrote:
>>> Hello Dave/Jeff,
>>>
>>> We are soliciting comments on the BFD benchmarking motivation  
>>> draft and
>>> appreciate your feedback on this ID. Could you please share your
>>> thoughts on this at [email-masked] <mailto:[email-masked]>
>>>
>>> Thanks,
>>> Jay
>>>
>>>
>>> ________________________________
>>>
>>> From: Esa Salahuddin (esalahud)
>>> Sent: Monday, March 05, 2007 2:27 PM
>>> To: [email-masked]
>>> Subject: [bmwg] Request for Opinions on Benchmarking the BFD
>>> protocol.
>>>
>>>
>>>
>>> Hi folks,
>>>
>>> Jay and I would like to hear your opinion on the idea of
>>> benchmarking the BFD protocol in the BMWG.
>>>
>>> The BFD protocol is becoming an increasingly popular alternative
>>> to a variety of existing methods to detect forwarding failures on  
>>> links
>>> between network elements. It is being proposed as the forwarding  
>>> failure
>>> detection mechanism for a host of routing protocols, virtual  
>>> links and
>>> p2mp topologies. It is a very useful in the situations where  
>>> underlying
>>> physical media doesn't lend itself well to fast detection of  
>>> failures.
>>> You can find extensive information on the BFD protocol itself at  
>>> the BFD
>>> WG weblink http://tools.ietf.org/wg/bfd/
>>>
>>> Please see
>>> http://www.ietf.org/internet-drafts/draft-salahuddin-bmwg-bfd- 
>>> motivation
>>> -00.txt for the deails on the topic and our position and please  
>>> let us
>>> know what do you think.
>>>
>>> thx, esa
>>>
>

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 15 Mar 2007 11:19:32 -0400

Jay (& bmwg-list),

Echo functionality: Benchmarking this functionality as part of BFD is
somewhat difficult in the sense that you cease benchmarking the BFD
mechanism and start benchmarking the forwarding mechanism.  The Min Echo
RX Interval, beyond signaling that the echo functionality is available,
is largely about "send packets no faster than this"; i.e. prevent the
loopback mechanism from being unnecessarily overwhelmed.

Multi-hop BFD: I'd strongly suggest restricting the original
benchmarking effort to directly attached links.  Multi-hop BFD involves
a number of elements that would perturb measuring BFD performance such
as forwarding queueing and rates, the implementation of BFD in
individual forwarding elements vs. system-wide implementation, etc.

-- Jeff

On Tue, Mar 06, 2007 at 01:41:28PM -0500, Jay Karthik (jakarthi) wrote:
> Hello Dave/Jeff,
>  
> We are soliciting comments on the BFD benchmarking motivation draft and
> to a variety of existing methods to detect forwarding failures on links
> between network elements. It is being proposed as the forwarding failure
> detection mechanism for a host of routing protocols, virtual links and
> p2mp topologies. It is a very useful in the situations where underlying
> physical media doesn't lend itself well to fast detection of failures.
> You can find extensive information on the BFD protocol itself at the BFD
> http://www.ietf.org/internet-drafts/draft-salahuddin-bmwg-bfd-motivation
> -00.txt for the deails on the topic and our position and please let us
> know what do you think.
> 	 
> 	thx, esa
>

---

From: David Ward <[email-masked]>
Date: Thu, 15 Mar 2007 11:27:01 -0500

Also not that testing a protocol w/ adaptive timers is also hard. The
adaptive timers can be modified automagically in an implementation according
to spec w/ forewarning.

-DWard

---

From: Dave Katz <[email-masked]>
Date: Thu, 15 Mar 2007 15:06:41 -0700

On Mar 15, 2007, at 11:19 AM, Jeffrey Haas wrote:

> However, we can hope that what comes out of this are tools that
> help implementors model scalability in their implementations.

I guess I have no idea what this means.  The kinds of things that  
should scale well ought to be obvious (anything there can be a lot  
of, or that wants to go fast, or both), and this does little to guide  
an implementor toward *how* he might achieve that scalability.   
Furthermore, the hardest scaling problems, which are typically due to  
interactions, memory allocators, schedulers, queueing strategies,  
etc., are far removed from the basic parameters.

--Dave

---

From: "Esa Salahuddin \(esalahud\)" <[email-masked]>
Date: Fri, 16 Mar 2007 00:42:24 -0400

Hi Dave,

You did pile it on!!! :) 

I think you have a very interesting perspective and a practical
approach. However, getting to a day where we can characterize and
benchmark systems completely is not going to happen automagically from
today's "It depends" state. If at all, it will be a series of steps that
will get us there and I believe benchmarking parts of the whole system
in isolation is a means to get there.

I think the problem lies with those who take these benchmarks as a
measure of whole system. Within the limitations described by any
benchmarking/characterizing effort, the benchmarks stand and do
represent useful data points.

Thx, Esa

-----Original Message-----
From: Dave Katz [mailto:[email-masked]] 
Sent: Thursday, March 15, 2007 1:44 PM
To: David Ward
Cc: Jeffrey Haas; [email-masked]; Jay Karthik (jakarthi); David Ward
(wardd); Esa Salahuddin (esalahud)
Subject: Re: [bmwg] Request for Opinions on Benchmarking the BFD
protocol.

IMHO.  ;-)

--Dave

---

From: "Jay Karthik \(jakarthi\)" <[email-masked]>
Date: Fri, 16 Mar 2007 08:27:29 -0400

Hi David, Jeff & Dave,

First of all, thanks for your cross working-group comments. Your
suggestions are greatly appreciated.

Please see my responses inline.

Thanks,
Jay 

> From: David Ward [mailto:[email-masked]] 
> Sent: Thursday, March 15, 2007 12:27 PM
> To: Jeffrey Haas; [email-masked]; Jay Karthik (jakarthi); Dave Katz
> Cc: David Ward (wardd); Esa Salahuddin (esalahud)
> Subject: Re: [bmwg] Request for Opinions on Benchmarking the 
> BFD protocol.
> 
> Also not that testing a protocol w/ adaptive timers is also 
> hard. The adaptive timers can be modified automagically in an 
> implementation according to spec w/ forewarning.
> 

Jay: While this could get tricky especially when this is not a
configurable timer, it may be possible by simulating the adjacent node
of the DUT (test tool) to vary it's timer in-flight and characterize the
effect of it. We will approach you with questions we may have then.

> > mechanism and start benchmarking the forwarding mechanism.  The Min 
> > Echo RX Interval, beyond signaling that the echo functionality is 
> > available, is largely about "send packets no faster than 
> this"; i.e. 
> > prevent the loopback mechanism from being unnecessarily overwhelmed.

Jay: I see what you mean and agree that it would not be a value add to
benchmark as a standalone entity. We should have this combined with
other scaling parameters and the ability of the DUT to simultaneously
loopback.

> > involves a number of elements that would perturb measuring BFD 
> > performance such as forwarding queueing and rates, the 
> implementation 
> > of BFD in individual forwarding elements vs. system-wide 
> implementation, etc.

Jay: We should be able to have a phased approach and declare Multi-hop
as out-of-scope (atleast for the initial versions of Benchmarking
Methodology).

> > authentication as an axis of measurement.

Jay: Most certainly. We shall propose benchmarking with (and without)
support offered by the hardware.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 16 Mar 2007 12:14:49 -0400

On Thu, Mar 15, 2007 at 03:06:41PM -0700, Dave Katz wrote:
> should scale well ought to be obvious 

I think each of us can pick their favorite Internet technology and an
example of how obvious scaling issues weren't obvious to someone.

> and this does little to guide  

I would agree with that statement.  At best, this allows some level of
apples to apples comparison of the externally visible results of a
relatively isolated part of the system.  IMO, that's the useful output
(for some definition of useful) of these benchmarking exercises.

> etc., are far removed from the basic parameters.

This sort of benchmarking, by its nature, is going to be blackbox.
Solutions dive in a bit deeper than that.

FWIW, I don't dispute anything you've said.  Your perspective is from
the standpoint of an implementor rather than someond doing benchmarking.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 16 Mar 2007 12:22:39 -0400

> support offered by the hardware.

In particular, I'd recommend your baselines be with authentication
disabled.

-- Jeff

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 16 Mar 2007 13:14:18 -0400

Esa, Dave,

Sorry to top post.  Just following the precedence on this thread.

I kind of agree with both of you.

Dave makes a good point in that we don't have a useful benchmark for
the performance of complex things like routing protocols, and BFD
might be tough to get a useful performance number or a useful measure
of scalability limits.

Esa is also right that an imperfect benchmark can be better than none.
The exception to this is when the marketing people get a hold of a
benchmark and then start advertising some useless figure (BGP giga
updates per second or BFD nanoseconds to respond or something
meaningless like that).

Benchmarks are generally most useful, but often not very repeatable,
when the entire problem space is explored.  For example, if a router
can sustain some high rate of BFD but the test uses only on one
interface, that's not useful.

To be truly useful, testing has to enable all protocols that are
intended to be used and stress a set of protocols and other inputs
(ie: framing errors, intermittent interfaces, SNMP/NMS load, etc).
If BFD performance were to degrade under such stress, then overall
network instability might result.

That doesn't mean that individual protocols can't be benchmarked in
isolation.  Its just that such tests don't indicate scalability and
may give an overstated measure of performance capabilities.

We are a bit off topic, but maybe BMWG should look at a modular
approach to stress testing in which the stress test component that
goes with the addition of any particular protocol was covered in a
separate RFC for that protocol.

This modular stress testing would reflect what the better providers
and vendors are doing but standardize the testing methodology.  We are
past the day where every provider can rely entirely on inhouse
developed benchmarking and stress testing and all of the vendors rely
more heavily than ever on test equipment developed by others.

Sorry to top post.  Sorry also to go a bit off topic.

Curtis

btw - I support doing BFD benchmarking but perhaps it is only useful
to test BFD in a stress test involving the full set of protocols.

In message <[email-masked]>
"Esa Salahuddin \(esalahud\)" writes:

Hi Dave,

You did pile it on!!! :) 

Thx, Esa

-----Original Message-----
From: Dave Katz [mailto:[email-masked]] 
Sent: Thursday, March 15, 2007 1:44 PM
To: David Ward
Cc: Jeffrey Haas; [email-masked]; Jay Karthik (jakarthi); David Ward

IMHO.  ;-)

--Dave

---

From: Al Morton <[email-masked]>
Date: Fri, 16 Mar 2007 13:47:11 -0400

At 01:14 PM 3/16/2007, Curtis Villamizar wrote:
>...To be truly useful, testing has to enable all protocols that are

Or, should we look at ways to make our Accelerated Stress Benchmarking
drafts more modular, and then consider the BFD topic as a possible module?

In any case, more folks should read the Accelerated Stress Benchmarking
drafts, they are overdue milestone-wise.

Al
bmwg co-chair who couldn't resist making a connection
in an effort to keep the existing work moving!

---

From: David Ward <[email-masked]>
Date: Fri, 16 Mar 2007 14:24:44 -0500

With respect to BFD, the performance metric of interest is the rate that a
router's OS can take the BFD trigger and tell interested parties about it.
Stress testing this protocol, given it's architecture, is different than
most others. The protocol is going to do everything in it's power to avoid
being stressed and give any false notifications. See the adaptive timers
point earlier.

-DWard

On 3/16/07 12:14 PM, "Curtis Villamizar" <[email-masked]> wrote:

> In message 
> <[email-masked]>

---

From: Dave Katz <[email-masked]>
Date: Fri, 16 Mar 2007 12:46:56 -0700

Without belaboring this too much, I'll throw one more bomb and then  
be quiet.  ;-)

There is a continuum for data, IMHO, that goes something like  
interesting < meaningful < useful.  "Interesting" is a pretty low  
bar;  any data can be interesting if you're a big enough geek.  ;-)   
"Meaningful" is tougher;  this requires pulling together a collection  
of data in order to characterize some larger truth about a system.   
"Useful" is a very high bar;  it implies that something predictive or  
otherwise operational can be pulled from the data that has some  
application in the real world.

I think it is fundamentally impossible to ever "characterize and  
benchmark systems completely" for the simple reason that their  
behavior is at once very complex and subject to the vagaries of  
implementation at many levels, and the systems are only becoming more  
complex.  I'm reminded of the early 90s, when academic folks wanted  
to attempt to model the behavior of routing protocols in large  
networks.  What they didn't seem to understand was that aspects of  
the particular implementation (at a very deep level down to things  
like internal data path congestion) *completely* overwhelmed the  
relatively simple bit of modeling messages passing over links, thus  
rendering the models not very useful, in the sense that they would be  
predictive in only a very limited way.

I guess I continue to wonder how this work will be useful in a real  
operational sense (that being my definition of "useful.")

--Dave

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 16 Mar 2007 16:31:56 -0400

In message <[email-masked]>
David Ward writes:
>  
> With respect to BFD, the performance metric of interest is the rate
> that a router's OS can take the BFD trigger and tell interested
> parties about it.  Stress testing this protocol, given it's
> architecture, is different than most others. The protocol is going to
> do everything in it's power to avoid being stressed and give any false
> notifications. See the adaptive timers point earlier.
>  
> -DWard

If under some instability condition a particular implementation had
lapses in returning echo packets that violate its advertised rate,
this would be nice to know.

Benchmarking without stress might yield values of Required Min RX
Interval and Required Min Echo RX Interval that are too low to use in
practice.  If past bugs over the last decade or more are any
indication, the difference between non-stress sustainable rates and
stress condition sustainable rates could be orders of magnitude.

Even if BFD is decoupled from the control plane and is handled by a
microprocessor on a forwarding card that microprocessor could have a
lapse due to a long running high priority task.  If echo is handled in
the forwarding hardware, used only as a link level fault detection,
then lapse in response should not be an issue.  That only applies to
the echo function.

So the first issue is related to false detections.

A second issue is how long it takes to detect a fault and propogate a
detected fault.

Curtis

---

From: "Poretsky, Scott" <[email-masked]>
Date: Mon, 19 Mar 2007 23:39:06 -0400

Accelerated Stress Benchmarking documents are at
 
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-11.tx
t
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-07.tx
t

This work item is "modular".  The working group decided to breakout
Layer 3 technology specific methodologies into separate documents and
work those after the Terminology and General Methodology Guidelines are
completed.  These were written for EBGP and Operational Security
(consistent with OpSec WG work) and the intention of the work item is to
be modular for support of more methodologies.  The now expired
methodology documents are draft-ietf-bmwg-acc-bench-meth-ebgp-00.txt and
draft-ietf-bmwg-acc-bench-meth-opsec-00.txt.  Work will continue on
those after the Terminology and Methodology Guidelines docs are
complete.  BFD could certainly be addressed in this work item. 

Scott

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Friday, March 16, 2007 1:47 PM
To: [email-masked]; Esa Salahuddin (esalahud)
Cc: Jeffrey Haas; Dave Katz; [email-masked]; David Ward
Subject: Re: [bmwg] useful testing (was: Opinions on Benchmarking theBFD
protocol)

At 01:14 PM 3/16/2007, Curtis Villamizar wrote:
>isolation.  Its just that such tests don't indicate scalability and may

>give an overstated measure of performance capabilities.
>
>approach to stress testing in which the stress test component that goes

>with the addition of any particular protocol was covered in a separate 
>RFC for that protocol.

Or, should we look at ways to make our Accelerated Stress Benchmarking
drafts more modular, and then consider the BFD topic as a possible
module?

Al
bmwg co-chair who couldn't resist making a connection in an effort to
keep the existing work moving!