Subject: Re: Fwd: Re: [bmwg] WG Last Call: OSPF convergence benchmarking

From: Scott Poretsky <[email-masked]>
Date: Fri, 14 Feb 2003 12:57:53 -0500

Folks,

While I think that IGP Convergence testing is very worthwhile work for the 
BMWG to undertake, these drafts do not address IGP Convergence.  These 
proposed drafts focus on LSA propagation benchmarking.  Due to the extreme 
interest in industry for IGP convergence measurements, I was working on IGP 
Convergence terminology and benchmarking methodology drafts when these 
proposed drafts were submitted.  I have provided the IGP Convergence drafts 
to Kevin and they should soon be posted.  My comments to the proposed LSA 
Propagation benchmarking drafts are below.

Scott
-----------------------------------------------------------------------------
Terminology
Internal Measurements and External Measurements are already defined as 
White Box and Black Box.

Point-to-Point, Broadcast, SPF, Hello Interval, and Router Dead Interval 
are already defined in IETF RFCs

Incremental-SPF is a vendor-specific feature.

Convergence is stated as " A network is termed to be converged when all of 
the devices within the network have a loop free path to each possible 
destination. "   The definition is then modified for  a single DUT as 
follow "Since we are not testing network convergence, but performance for a 
particular device within a network, however, this definition needs to be 
narrowed somewhat to fit within a single device view.  In this case, 
convergence will mean the point in time when the DUT has performed all 
actions needed to react to the change in topology represented by the test 
condition ".   This must include updating the FIB and hardware as 
externally verified with data traffic.  This draft contradicts its own 
definition by verifying convergence through measurement of only LSA 
propagation.  In fact, updating the FIB is explicitly excluded from 
measurement.

Methodology
Section 5. Overview and Scope
Control Plane measurements are not convergence measurements.  Convergence 
is benchmarked by time to reroute traffic.

Section 6. Test Conditions
Bullet 2  It cannot be assumed that there is zero delay to execute the SPF 
or  LSA processing.  Reference [5] actually _recommends_ router vendors 
implement zero delay to achieve low convergence.  The benchmarking 
methodology should allow convergence measurements to be made regardless of 
implementation.

Bullet 3  Data traffic is actually required to make a deterministic 
measurement of route convergence.
Unlisted: Number of IGP routes is a critical parameter for convergence time.

Sections 7. through 9.
The biggest problem with the methodology is that not all components for 
convergence time are measured.  Focus is on LSA Processing, which is only 1 
of 4 major components.

Route change due to cost or next-hop change is not considered.  This is the 
most common reason for route convergence.

Well -defined Convergence methodology can be applied to ISIS or OSPF.  The 
Link-State IGP does not matter.  In fact the methodology should be able for 
use to benchmark one IGP against the other given the same DUT.

All procedures meet one or more of the following flaws:
White box measurements are part of the procedure.
Time to install route (update FIB), update hardware, and reroute traffic 
are not considered.
Test case is Implementation-specific.
Test case has nothing to do with convergence - more OSPF performance test.

Applicability
The applicability document explains that convergence testing is needed using
1. White box benchmarks
2. DUT intrusive measurements
3. Knowledge of DUT implementation
4. Absence of traffic results in impractical benchmarks.  Convergence time 
is benchmarked by traffic loss.
1 through 3 violate basic test methodology requirements.  4 makes invalid 
convergence tests.
This work is less about convergence benchmarking and more about LSA 
propagation benchmarking.

>>>>>At 08:42 PM 2/7/03 -0500, you wrote:
>>>>>
>>>>>>BMWG'ers:
>>>>>>
>>>>>>A WG Last Call period for the Internet-Drafts regarding
>>>>>>OSPF convergence benchmarking terminology, methodology, and
>>>>>>benchmark applicability:
>>>>>>
>>>>>>    <draft-ietf-bmwg-ospfconv-term-02.txt>,
>>>>>>    <draft-ietf-bmwg-ospfconv-intraarea-03.txt>,
>>>>>>    <draft-ietf-bmwg-ospfconv-applicability-01.txt>
>>>>>>
>>>>>>will be open from 7 February 2003 until 28 February 2003.
>>>>>>
>>>>>>Please weigh in on whether or not you feel each individual draft
>>>>>>should not be given to the Area Directors for consideration in 
>>>>>>progressing the draft to an Informational RFC.  Send your comments
>>>>>>to this list or [email-masked].
>>>>>>
>>>>>>
>>>>>>URLs for the Internet-Drafts are:
>>>>>>
>>>>>>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ospfconv-term-02.txt
>>>>>>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ospfconv-intraarea-03.txt 
>>>>>>
>>>>>>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-ospfconv-applicability-01.txt 
>>>>>>
>>>>>>
>>>>>>-Kevin
>>>>>>
>>>>>>
>>>>>>_______________________________________________
>>>>>>bmwg mailing list
>>>>>>[email-masked]
>>>>>>https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Scott Poretsky <[email-masked]>
Date: Fri, 14 Feb 2003 12:57:53 -0500

Folks,

Scott

Methodology

Sections 7. through 9.

Applicability

---

From: Russ White <[email-masked]>
Date: Sun, 16 Feb 2003 12:12:40 -0500 (Eastern Standard Time)

> While I think that IGP Convergence testing is very worthwhile work for
> the BMWG to undertake, these drafts do not address IGP Convergence.
> These proposed drafts focus on LSA propagation benchmarking.

Actually, I disagree. I think that these drafts attempt to seperate the
concept of convergence within the protocol from convergence within the
network. For instance, we specifically state that we are attempting to
measure SPF times, and give possible tests for doing so, not just the
propogation of LSA's. I think it's valuable to be able to split the entire
convergence process into seperate steps, and measure each one seperately,
since this will help you in various situations, such as comparing one
release of software against another, and trying to determine _where_ in the
convergence process a time differential has occurred.

Measuring the time it takes for a network to converge is a distinctly
different topic, and I actually have a draft under way to discuss network
convergence in various terms.

> are already defined in IETF RFCs

Yes, but we are attempting to cover things which these drafts may use in a
way which is specific, etc. I'm not partial to having specific definitions
in or out, but I thought these made sense.

> Incremental-SPF is a vendor-specific feature.

No, it's not. It's actually described in various IS-IS docs, and un various
documents on improvements in the SPF algorithm. All of the problems are not
solved in many of these references, but the idea is common enough. That
various vendors may or may not have implemented this is immaterial, except
that it may impact measurements in certain ways, and the way in which it
impacts measurements is important.

> Convergence is stated as " A network is termed to be converged when all
> of the devices within the network have a loop free path to each possible
> follow "Since we are not testing network convergence, but performance for
> a particular device within a network, however, this definition needs to
> be narrowed somewhat to fit within a single device view.  In this case,

No, it must not. I suspect this is the point where we disagree. Again, it's
very important to be able to "pull apart" convergence, and understand what
each element of it is, etc.

> Section 5. Overview and Scope Control Plane measurements are not
> convergence measurements.  Convergence is benchmarked by time to reroute
> traffic.

I don't agree with this--it depends on your definition of "convergence."
The dictionary handy by my desk states convergence is when "several people
meet at the same place, or one person arrives at the same place others are
already." In this case, it's when several routers agree on the best path.
By explicitely including the time it takes to push the information that
OSPF has discovered to the forwarding tables, you are explicitely not
allowing the testing of convergence within OSPF as a protocol.

> methodology should allow convergence measurements to be made regardless
> of implementation.

No, you cannot assume this, and thus this is just one method. You'll note
that we've suggested that testers compare the results of various methods to
determine the true time, rather than relying on one test.

> measurement of route convergence.

No, it isn't, unless you are explicitely trying to measure network
convergence, rather than OSPF convergence (see above for my arguments
here).

> Unlisted: Number of IGP routes is a critical parameter for convergence time.

I think this is discussed, at some point, although it might not be at this
point.

> convergence time are measured.  Focus is on LSA Processing, which is only 1

Actually, focus is also on SPF time. There are other elements, but they can
be calssified as _outside_ OSPF's control, and thus not a part of OSPF's
convergence. It's valid to attempt to measure those other elements, but not
to call them a part of OSPF.

> Route change due to cost or next-hop change is not considered.  This is
> the most common reason for route convergence.

Hmmm... I see very few changes due to cost change in a link state protocol
within a single flooding domain.  In a distance vector protocol, route
metric changes are common, but not in link state. People don't typically
change the bandwidth of their links without simply taking the link down,
and bringing another link up. Beyond an area boundary, in OSPF, a change in
path due to a link failure would show up as a change in metric, but this
draft does not address interarea issues, only intrarea issues.

> use to benchmark one IGP against the other given the same DUT.

Unless you are attempting to measure the convergence of the protocol
itself, between variations in a given impementation, etc. See above.

So, in short, I do agree with your premise--that convergence measurements
must include the time it takes to change the forarding plane, or to
discover the failure that causes the change. There are different pieces,
and this draft looks specifically at some of those pieces.

:-)

Russ

--
[email-masked] CCIE <>< Grace Alone
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Russ White <[email-masked]>
Date: Sun, 16 Feb 2003 12:12:40 -0500 (Eastern Standard Time)

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Mon, 17 Feb 2003 09:52:02 -0500 (Eastern Standard Time)

> So, in short, I do agree with your premise--that convergence measurements
                    ^not

Typo....

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Mon, 17 Feb 2003 09:52:02 -0500 (Eastern Standard Time)

^not

Typo....

:-)

Russ

---

From: Kevin Dubray <[email-masked]>
Date: Wed, 19 Feb 2003 18:11:02 -0500

Scott Poretsky wrote:
> I have provided the 
> IGP Convergence drafts to Kevin and they should soon be posted.  

The individually-submitted I-Ds to which Scott alludes can be
found here:

http://www.ietf.org/internet-drafts/draft-poretsky-igp-convergence-term-00.txt
http://www.ietf.org/internet-drafts/draft-poretsky-igp-convergence-meth-00.txt
http://www.ietf.org/internet-drafts/draft-poretsky-igp-convergence-app-00.txt

-Kevin

---

From: Kevin Dubray <[email-masked]>
Date: Wed, 19 Feb 2003 18:11:02 -0500

-Kevin