Subject: [bmwg] IGP Data Plane Convergence

From: Scott Poretsky <[email-masked]>
Date: Wed, 09 Jul 2003 12:13:05 -0400

> BMWG-ers,
 >
 > We have had a couple of issues raised for the IGP Data Plane Convergence
 > benchmarking. The first is the Packet Sampling Interval of the test
 > equipment and its impact to results and the second is the choice of values
 > for the timers that impact the results. We would like to address these in
 > the next revision of the documents as follow:
 >
 > 1. Packet Sampling Interval -
 > With some test equipment vendors you must use the Average Convergence
 > Time and not the preferred Full (Peak-to-Peak) Convergence Time. This is 
because the
 > minimum configurable packet sample interval is as high as 1 second. 
Being this high, the
 > Convergence Recovery Slope is grossly exaggerated for fast convergence 
times, causing
 > the Full Convergence Time to measure greater than it actually is. With the
 > other test equipment it is still appropriate to use the more accurate Full
 > Convergence Time instead of the Average Convergence Time, because 
100msec packet
 > sample intervals are used. This is particularly important as the 
industry is
 > discussing millisecond convergence time.
 >
 > The authors intend to add to the Terminology draft a new term - Convergence
 > Sampling Rate - with a maximum value of 100msec. The Methodology draft will
 > recommend in the "Test Considerations" section will include this discussion
 > and state that when the Convergence Sampling Rate >100msec then the Average
 > Convergence Time must be used.
 >
 > 2. Convergence Timers
 > The Methodology draft "Test Considerations" section currently states to set
 > Convergence Timers to minimum value. It has been requested that we include
 > suggested values. We propose that the values appear as follow:
 >
 > SONET Failure Indication Delay (<10ms, almost immediate)
 > IGP Hello Timer (1 sec)
 > IGP Dead-Interval (3 sec)
 > LSA Generation Delay (0, immediate)
 > LSA Flood Packet Pacing (0, immediate)
 > LSA Retransmission Packet Pacing (0, immediate)
 > SPF Delay (0, immediate)
 >
 > Is there agreement, concern, or questions for these changes to be made for
 > the 01 versions of the drafts? Are there additional issues that should be
 > addressed?
 >
 > Scott/Brent

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Al Morton <[email-masked]>
Date: Wed, 09 Jul 2003 17:55:40 -0400

At 12:13 PM 07/09/2003 -0400, Scott Poretsky wrote:
> > benchmarking.
> >...
> > Time and not the preferred Full (Peak-to-Peak) Convergence Time. This 
> is because the

I think it's fair for the benchmarks we compose to encourage
performance improvements in *both* DUTs and test equipment.

I agree you should add the new term on sampling rate, but the text on
test considerations reveals other issues:

1. The Methodology draft uses the undefined term "Peak-to-Peak
Convergence Time," which you appear to be treating as equivalent to
"Full Convergence Time" above.

2. If your proposal is to allow either "Full" or "Average"
convergence time measurements in section 4,
then make that option clear so we can discuss it further.

> > the 01 versions of the drafts? Are there additional issues that should be

Yes.  I agree that the timer values should be reported with
the results.  But, if we require very low specific values,
we run the risk of precluding benchmarks for some products.
(I have *not* surveyed the configuration ranges for all manufacturers.)
If a configuration interface places limits on timers, the designers
may have had a good reason to do so.  Do we require them to remove
these safety limits in order to be benchmarked?

The proposal to report timer values raises another issue:
The Methodology Draft has no Reporting Formats specified.
Perhaps we can do something creative with the reports,
where we normalize-out relevant timer values and simply report
the additional convergence time? (thinking out loud here)
Then we don't encourage designers to allow configurations
that might be unstable just to perform well on the benchmarks.
This is worth some investigation, IMO.

thanks for starting discussions, more comments to follow,
Al

---

From: Al Morton <[email-masked]>
Date: Wed, 09 Jul 2003 17:55:40 -0400

---

From: Scott Poretsky <[email-masked]>
Date: Wed, 09 Jul 2003 18:13:09 -0400

Al,

Thanks for the insightful comments.  See replies below.

Scott

>performance improvements in *both* DUTs and test equipment.

Absolutely!

>"Full Convergence Time" above.

Yes.  Good catch.  The Terminology draft will be updated to use the Term 
"Full Convergence Time" instead, as defined by the measured Peak-to-Peak 
time in the graph.

>then make that option clear so we can discuss it further.

Got it.  Our proposal is to use Full Convergence Time, which provides the 
better result.  If however the test equipment Convergence Sampling Rate 
is >100msec, then the Average Convergence Time  may be used with indication 
that the result is the Average Convergence Time  .

>(I have *not* surveyed the configuration ranges for all manufacturers.)

This is true and certainly a concern.

>these safety limits in order to be benchmarked?

We should not.

>This is worth some investigation, IMO.

I agree.  This is a good idea.  The co-authors will propose a reporting 
format that facilitates the calculations in section 5.2.  The "recommended" 
values can still be provided for guidance.

---

From: Scott Poretsky <[email-masked]>
Date: Wed, 09 Jul 2003 18:13:09 -0400

Al,

Scott

Absolutely!

We should not.