Subject: [bmwg] Objections to Methodology draft: OSPF convergence benchmarking

From: "Manral, Vishwas" <[email-masked]>
Date: Sat, 15 Feb 2003 03:33:03 -0500

Hi Scott,
 
A lot of factors regarding the particular protocol define the convergence
time for the protocol, the overall dataplane measurements would not depend
on the protocol at all, not just IGP. 
 
I would like to clarify two statements you have made.
 
1. "White box benchmarks" - which is wrong. Please check the methodology and
you will see we haven't used any white box measurements. Let me know if any
test cases involve White-Box measurements. Kevin, did raise an issue
regarding the measurements for SPF, when we were using measurements given by
the device, which we have changed and been verified, also included in the "
Experience in  <http://www.cse.ucsc.edu/~aman/imw02-talk.ppt> Black-box OSPF
Measurement"
 
2.  The measurements are not LSA propagation measurements. If you check the
document named above you will be able to understand what the exact
measurements are for. If you read the methodology draft propagation delays
infact have been considered, so that they do not make a differece in
measurements, although if you read " Experience in
<http://www.cse.ucsc.edu/~aman/imw02-talk.ppt> Black-box OSPF Measurement",
you will know they do not make too much of a difference.
 
3. I have clearly stated my stand about convergence measurements.
 
4. Regarding Bullet 2, I agree we should strike off the point as it does not
make a difference for our measurements at all. Our methodology does take
that into consideration already.
 
5. I saw the defnition of "IGP routes", in your draft, however I am not sure
you can make a general statement about effecting convergence. Intra-Area
LSA's are dealt with differently than any External LSA(as they are treated
seperately in SPF), and measurements will vary greatly. Please check "Tree
Shape and the SPF Algorithm" section in the terminology draft to clarify the
point.
 
6. I am sure someone aware of the OSPF protocol will know that any change of
cost measurements are similar to the LSA origination/propagation and
processing times. Besides I am not sure I agree with the fact that cost
change happens regularly in the internet. 
 
If however you feel strongly about this, we can easily put in a test case
for that.
 
7. You have repeatedly stated "implementation specific" it would be nice if
you clarify, which tests you exactly consider.
 
I will again state such measurements have been used to compare protocol
convergence between various vendor devices.
 
Thanks,
Vishwas

-----Original Message-----
From: Scott Poretsky [mailto:[email-masked]]
Sent: Friday, February 14, 2003 11:28 PM
To: [email-masked]
Subject: Re: Fwd: Re: [bmwg] WG Last Call: OSPF convergence benchmarking

Folks,

While I think that IGP Convergence testing is very worthwhile work for the
BMWG to undertake, these drafts do not address IGP Convergence.  These
proposed drafts focus on LSA propagation benchmarking.  Due to the extreme
interest in industry for IGP convergence measurements, I was working on IGP
Convergence terminology and benchmarking methodology drafts when these
proposed drafts were submitted.  I have provided the IGP Convergence drafts
to Kevin and they should soon be posted.  My comments to the proposed LSA
Propagation benchmarking drafts are below.  

Scott
----------------------------------------------------------------------------
-
Terminology
Internal Measurements and External Measurements are already defined as White
Box and Black Box.

Point-to-Point, Broadcast, SPF, Hello Interval, and Router Dead Interval are
already defined in IETF RFCs

Incremental-SPF is a vendor-specific feature.

Convergence is stated as " A network is termed to be converged when all of
the devices within the network have a loop free path to each possible
destination. "   The definition is then modified for  a single DUT as follow
"Since we are not testing network convergence, but performance for a
particular device within a network, however, this definition needs to be
narrowed somewhat to fit within a single device view.  In this case,
convergence will mean the point in time when the DUT has performed all
actions needed to react to the change in topology represented by the test
condition ".   This must include updating the FIB and hardware as externally
verified with data traffic.  This draft contradicts its own definition by
verifying convergence through measurement of only LSA propagation.  In fact,
updating the FIB is explicitly excluded from measurement.

Methodology
Section 5. Overview and Scope
Control Plane measurements are not convergence measurements.  Convergence is
benchmarked by time to reroute traffic.

Section 6. Test Conditions
Bullet 2  It cannot be assumed that there is zero delay to execute the SPF
or  LSA processing.  Reference [5] actually _recommends_ router vendors
implement zero delay to achieve low convergence.  The benchmarking
methodology should allow convergence measurements to be made regardless of
implementation.

Bullet 3  Data traffic is actually required to make a deterministic
measurement of route convergence.
Unlisted: Number of IGP routes is a critical parameter for convergence time.

Sections 7. through 9.
The biggest problem with the methodology is that not all components for
convergence time are measured.  Focus is on LSA Processing, which is only 1
of 4 major components.

Route change due to cost or next-hop change is not considered.  This is the
most common reason for route convergence.

Well -defined Convergence methodology can be applied to ISIS or OSPF.  The
Link-State IGP does not matter.  In fact the methodology should be able for
use to benchmark one IGP against the other given the same DUT.

All procedures meet one or more of the following flaws:
White box measurements are part of the procedure.
Time to install route (update FIB), update hardware, and reroute traffic are
not considered.
Test case is Implementation-specific.
Test case has nothing to do with convergence - more OSPF performance test. 

Applicability
The applicability document explains that convergence testing is needed using
1. White box benchmarks
2. DUT intrusive measurements
3. Knowledge of DUT implementation
4. Absence of traffic results in impractical benchmarks.  Convergence time
is benchmarked by traffic loss.
1 through 3 violate basic test methodology requirements.  4 makes invalid
convergence tests.
This work is less about convergence benchmarking and more about LSA
propagation benchmarking.