Subject: [bmwg] Device Convergence Benchmarking

From: Kevin Dubray <[email-masked]>
Date: Wed, 06 Mar 2002 10:46:46 -0500

Currently, we have two efforts in the BMWG regarding
"convergence":  one WG item on BGP device convergence and
a proposed work item on OSPF device convergence. (The last call
on the new work proposal recently closed.)

The WG chose a device-based division of labor in our first attempt to 
characterize convergence because of the complexity and breadth
of the subject.  It's generally held in the WG that characterizing 
device *GP convergence is a "good thing."

What we're learning is while the collection methodology is usually
straightforward, specifying the collection context is not.
This is significant because test context (i.e., the simulated network
scenario and ensuing test stimuli applied to the DUT) helps gives 
meaning to the test results.

And implied meaning appears central to the some of the critique 
regarding the effort.  Some questions are: 
   
   Is the current division of labor (singular device characterization)
   so constrained as to have little utility? 
   Will these tests yield insight into operating networks?
   Are the suggested methodologies applicable to characterizing 
   "convergence" in live networks?
   Is lowest "convergence time" always best, or are there
   qualitative classes of responses?
   What's the significance of these classes of metrics when operators
   often adjust protocol timers to trade-off convergence time for
   network stability? 
   Are these results applicable to a single device only or should
   the results form a basis for comparison across different devices
   of the same type?
   How should the reader of these benchmarks apply these results?
   What significance should the network operator put on these 
   results? Is this the same for the enterprise network admin?

While sometimes implied, these issues don't appear to be directly 
addressed by the related work.

The BMWG has generally not attempted to define goodness for its 
benchmarks. That said, it might be useful given the network-wide
implication of the term "convergence,"  for the BMWG to 
provide stronger guidance as to the applicability, use, and limits 
of these particular benchmarks.

Thoughts?

-Kevin

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 06 Mar 2002 13:53:38 -0500

>meaning to the test results.

Let us, my friends and colleague, gather around the tribal campfire, 
or its equivalent, the IETF bar, and let me explain how this work got 
started, and then put it into a broader context.

One fine day, while working as a router product manager at Nortel, a 
wild-eyed salesman ran in and said "How many BGP neighbors can we 
support?"

Appropriately enough, I answered, "that depends. What are they doing?"

He screamed back, "But Brand X says they can do 256! We have to do better!"

Of course, the number of neighbors is meaningless when we don't know 
the number and rate of updates per neighbor, etc. I began some 
informal conversations both with internal developers and friends at 
our competitors, and we reached a consensus that it would be valuable 
to have an objective set of criteria for benchmarking the control 
plane performance of a router.

So there were two motivations. The first, shared by engineers of 
several companies, was getting the salesdroids under control.  The 
second is a purer, more research/technology orientation.

That second orientation is an observation that people such as Ahuja 
and Labovits had done useful work about Internet-wide performance, 
but we still didn't have a way to characterize the individual routers 
(and potentially AS) that led to that performance.  All these kinds 
of convergence performance are complementary.  To draw a medical 
analogy, you can speak of a disease of the entire body (Internet), 
but you may have to understand organ-level (AS) or cellular (router) 
problems before you know what to treat.  Internet-level convergence 
is more environmental and public health medicine, while the 
finer-grained analysis is pathology. [It has been observed that the 
study of routing is path-ology, anyway].

>    so constrained as to have little utility?

Just like current BMWG works allow switches, etc., to be compared on 
a box basis, this will allow the controlled comparison of control 
plane devices.

>    Will these tests yield insight into operating networks?

Yes, but that's not the primary goal -- one can argue that's an IPPM goal.

>    "convergence" in live networks?

No. Nor, IMHO, should they be.  The test mix should be representative 
of what we see in real networks, but the essence of testing is that 
it be repeatable.

>    qualitative classes of responses?

We make no judgement about the goodness of convergence time, simply 
what it is under a particular set of parameters.

>    often adjust protocol timers to trade-off convergence time for

We started with a set of performance tests based on defaults. 
Certainly, additional sets of tests could be defined with the sort of 
alternate settings you describe.

>    the results form a basis for comparison across different devices

I'm not sure I understand. For the allegedly same class of Crisco and 
Pinetree and Forge routers, as well as Hippety-Hoppity and Striped 
Horse code on a reference platform, with the same number of neighbors 
with the same rates, yes, they should be a basis for comparison.

>    results? Is this the same for the enterprise network admin?

A different problem than we are addressing.  We provide data that can 
be used in such more global analyses. In particular, if the global 
analysis shows a control processing bottleneck, we can give objective 
criteria for specifying a device that does not have such limits.

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 06 Mar 2002 13:53:38 -0500