Subject: [bmwg] Comments for OSPF Convergence Benchmarking Terminology

From: Scott Poretsky <[email-masked]>
Date: Mon, 10 Mar 2003 16:58:36 -0500

Folks,

Below are comments to each section of the Terminology Document.  I 
previously provided comments to the authors about 3 weeks ago.  Those 
comments were around the merits of Control Plane Convergence measurements 
versus Data Plane Convergence measurements.  There was good discussion 
between Russ and I that ultimately led us to mutually agree that documents 
for both approaches should exist, as both approaches are mutually exclusive 
for terminology and methodology.  This is a follow-up with response to the 
specific content.  Please reply with comments.

Thanks,
Scott
########################################
1. Terminology Section
The terminology section has 10 terms.  7 of the 10 are existing 
terms.  These are Internal Measurement, External Measurement, 
Point-to-Point Link, Broadcast Link, SPF Time, Hello Interval, and Router 
Dead Interval.  Of the other 3, Multi-Device Measurement serves no useful 
purpose, Incremental-SPF is an optional internal algorithm, and convergence 
is incorrectly defined.  Here are detailed remarks for each one.

Internal Measurements - This is a redefinition of White Box 
measurement.  Recommend it be removed.

External Measurements - This is a redefinition of Black Box 
measurement.    Recommend it be removed.  In the Discussion section for 
External Measurements it states:  "when a downstream device receives 
complete routing information from the DUT, it can be inferred that the DUT 
has transmitted all of the routing information available".
How can tests be built so that it is not "inferred", but instead "known"?

Multi-Device Measurement - This is a definition describing a test with 
multiple DUTs.  This term serves no purpose.  In fact, it does not even 
appear in the companion methodology.  Recommend it be removed.

Point-to-Point Link - Term was already defined by the Point-to-Point 
Protocol Working Group.  See RFC 1171, The Point-to-Point Protocol for the 
Transmission of Multi-Protocol Datagrams over Point-to-Point 
Links.  Recommend it be removed.

Broadcast Link - While this term is not defined, but frequently used, in 
IETF documents, it is a well-known term defined in IEEE 802 
documents.  Recommend it be removed.

SPF Time - This is a white box parameter that is not externally 
observable.  RFC 2328 already defines SPF.  Recommend it be removed.

Hello Interval - This is a term already defined in RFC 2328, OSPFv2, 
Section 9, The Interface
Data Structure and its use described in detail RFC 2328, OSPFv2, in Section 
9.5, Sending Hello
Packets and Section 10 Neighbor Data Structure.  Recommend it be removed.

Router Dead Interval -  This is a term already defined in RFC 2328, OSPFv2, 
Section 9, The Interface
Data Structure and its use described in detail RFC 2328, OSPFv2, Section 10 
Neighbor Data Structure.  Recommend it be removed.

Incremental-SPF - This is a white box component of OSPF. Implementation of 
this feature is optional.
Superior Convergence times can be obtained without Incremental-SPF (as two 
vendors have demonstrated).  Black box measurements cannot indicate if 
Incremental-SPF is internally implemented in the DUT.  RFC 2328 makes one 
reference to Incremental-SPF as an example algorithm to improve SPF 
run-time.  It does not
require it nor indicate it to be the _most_ efficient algorithm.  If we 
were to standardize internal SPF execution algorithms then we should also 
include PRC, RTC, and others.  There would be no reason to choose just 
Incremental-SPF.  Recommend it be removed.

Convergence - Measured by Time to Flood, Run SPF, and Update RIB.  This is 
a partial measurement of router
components that contribute to Convergence time.  Time to update FIB is 
certainly a missing component.  Also, network operators speak of 
"Convergence" as the time it takes for all nodes in the network to 
converge.  I propose that this term be renamed "Protocol Convergence" or 
"Control-Plane Convergence".

In summary for this section, I see Convergence as the only term that is 
original, but it must still be renamed (two alternates suggested 
above).  If it is decided to keep other terms, please add them to a section 
"Existing Terms" with references for each one.

2. Measuring Traffic Convergence Section
Three measurements are listed.  Here are the concerns:
A. LSA Propagation
B. SPF Execution - This is a White Box Measurement
C. RIB Update - How can this be measured in isolation from 1 and 2 without 
it being a white-box test?

3. Type of Network Events Section
Next-Hop change (e.g. due to metric change) is a unique event because no 
packet loss is expected, so this should also be listed.  The other events 
listed - Link Up/Down, Session Initialization, and Adjacency Down
produce packet loss.

4. LSA and Destination Mix
No new terminology.  Methodology related information.  Recommend move to 
methodology document.

5. Tree Shape and SPF algorithm
These are White Box items.  Neither is externally observable.  Tree shape 
is already described in RFC 2328.  I recommend that this section be 
removed.  It could be added as an appendix to the methodology document, but 
that is not worth it since it is already covered in RFC 2328.

---

From: "Manral, Vishwas" <[email-masked]>
Date: Tue, 11 Mar 2003 11:15:55 -0500

Hi Scott,

Sorry to budge in, but I would like to comment on this.

We initially had only a white-box test for SPF capculations, which used show
commands(which are readily available in most routers). However as in the
working group we use black-box measurements, we added another test to
measure the SPF time thru black-box methods(which is the same test as in the
paper I have referred to in previous mails).

In the 53rd meeting of IETF in MNP, it was decided(if I understood it right)
that we keep the test more from the perspective of actually checking
white-box measurements against the actual black-box measurements. We have
used both these methods in our measurements, and have left both the methods
in the draft.

Thanks,
Vishwas 

-----Original Message-----
From: Scott Poretsky [mailto:[email-masked]]
Sent: Tuesday, March 11, 2003 9:08 PM
To: Russ White
Cc: [email-masked]
Subject: Re: [bmwg] Comments for OSPF Convergence Benchmarking
Terminology

Kevin,

I thought BMWG drafts are not permitted to have White Box test 
coverage.  Is this correct?  I also thought that existing terminology 
should either be excluded or referenced in the Existing Terms section.  Is 
this correct?

Scott

At 06:24 PM 3/10/03 -0500, Russ White wrote:

> > Internal Measurements - This is a redefinition of White Box measurement.
> > Recommend it be removed.
>
>The definition is not the major point of this defintion, but rather the
>discussion about internal measurements, that they can impact the
>performance of the box being measured, etc. I don't know if there is a
>"generic" draft which covers these sorts of things, or if one is planned,
>but it could go there just as well as here. In the meantime, it seems like
>a useful discussion to have on paper.
>
> > External Measurements - This is a redefinition of Black Box measurement.
> > Recommend it be removed.  In the Discussion section for External
> > Measurements it states:  "when a downstream device receives complete
> > routing information from the DUT, it can be inferred that the DUT has
> > transmitted all of the routing information available". How can tests be
> > built so that it is not "inferred", but instead "known"?
>
>You can't do more than infer, in most cases. You can state that it is
>"known" that the DUT has finished sending information, but you don't know
>if you are going to, for instance, drop the last ack, and force the DUT to
>retransmit.
>
>Actually, in our definition, a multi-device test requires measurement of
>the test results on several devices, not necessarily several DUTs. For
>instance, you may need to measure the timestamp on both the route generator
>and the DUT, which means it's a multi-device mesasurement (not a
>multi-device test, which would imply something different). I don't argue
>that we didn't use the term in the methodology, but we certainly described
>some multi-device measurements in some of the tests.
>
>Again, the discussion is important here, in how some of these issues
>surrounding multi-device measurements.
>
> > Protocol Working Group.  See RFC 1171, The Point-to-Point Protocol for
> > the Transmission of Multi-Protocol Datagrams over Point-to-Point Links.
> > Recommend it be removed.
>
>First, the concept of a point-to-point link encompasses more link types
>that the PPP working group covers. For instance, there is a draft in the
>IS-IS and OSPF working groups which discusses point-to-point ethernet
>links, and how these protocols should treat these links different than a
>normal ethernet link. In this case, I'm not certain the duplicate defintion
>is a bad thing, an the discussion about convergence times in OSPF vis-a-vis
>point-to-point links needs a home.
>
> > IETF documents, it is a well-known term defined in IEEE 802 documents.
> > Recommend it be removed.
>
>Note the definition is in terms of OSPF, and the benchmark draft. Again, it
>may be a commonly known term, but the concept of a DR being elected on the
>link is something which is specific to OSPF.
>
>There are tests included which discuss how to measure SPF time from outside
>the box, and inside the box. I'm fairly certain the methodology draft
>includes white box measurements, so I'm not certain why white box
>measurements shouldn't be included in the terminology draft?
>
> > Section 9, The Interface Data Structure and its use described in detail
> > RFC 2328, OSPFv2, in Section 9.5, Sending Hello Packets and Section 10
>The discussion here is just as important as the definition, and applies to
>all tests discussed in the methodology draft. Suggestions on where else we
>would put this discussion?
>
> > Router Dead Interval - This is a term already defined in RFC 2328,
> > OSPFv2, Section 9, The Interface Data Structure and its use described in
> > detail RFC 2328, OSPFv2, Section 10 Neighbor Data Structure.  Recommend
> > it be removed.
>
>See above, on hello interval.
>
> > Incremental-SPF - This is a white box component of OSPF. Implementation
> > of this feature is optional. Superior Convergence times can be obtained
> > without Incremental-SPF (as two vendors have demonstrated).  Black box
> > measurements cannot indicate if Incremental-SPF is internally
implemented
> > in the DUT.  RFC 2328 makes one reference to Incremental-SPF as an
> > example algorithm to improve SPF run-time.  It does not require it nor
> > indicate it to be the _most_ efficient algorithm.  If we were to
> > standardize internal SPF execution algorithms then we should also
include
> > PRC, RTC, and others.  There would be no reason to choose just
>I don't see it used in either draft, so this is fine.
>
> > Convergence - Measured by Time to Flood, Run SPF, and Update RIB.  This
> > is a partial measurement of router components that contribute to
> > Convergence time.  Time to update FIB is certainly a missing component.
> > Also, network operators speak of "Convergence" as the time it takes for
> > all nodes in the network to converge.  I propose that this term be
> > renamed "Protocol Convergence" or "Control-Plane Convergence".
>
>Either way is fine with me.... Of course, this definition draft only sets
>the meaning of words for the accompanying methods and applicability drafts,
>and not in general. Others have defined convergence differently, so.... I
>suppose that we, as a working group, can go one of two directions on this:
>
>-- We come up with a new name for convergence, prepending it with some
>other word, to imply different things in different places. This could
>probably lead to large and protracted arguments, and perhaps
>overspecification of the meaning of words.
>
>(for instance, we've already added "SR" to the front of convergence for a
>similar objection, so now we'll have:
>
>-- SR-CP-Convergence, Single Router Control Plane
>-- N-CP-Convergence, Network Control Plane
>-- SR-DP-Convergence, Single Router Data Plane
>-- N-DP-Convergence, Network Data Plane
>
>as a base set of variants, with, I assume, many many more to come).
>
>-- We allow each draft to use the word "convergence" within it's bounding
>limits (the dictionary definition), and just make certain that each draft
>discusses its definition of convergence at some point, to clarify for the
>readers of that draft. always remembering to have a flexible mind, and seek
>out the meaning of convergence before reading any given BMWG draft.
>
>I'll defer to the working group for that discussion and decsion, though I
>prefer the latter.
>
>A test is included in the methods draft, so I'm not certain why this
>shouldn't be here?
>
> > C. RIB Update - How can this be measured in isolation from 1 and 2
> > without it being a white-box test?
>
>We don't include a test for this, so whether it is white box or black box
>makes no difference? It's just noted as a part of overal convergence times,
>to be complete in the discussion of the elements of convergence time.
>
>We didn't discuss packet loss in any section, since we are concerned with
>contorl plane convergence, rather than data plane convergence (?). We can
>add next-hop change, although I'm not certain we base any tests on it, and
>making this list exhasutive would, well, unless we want to do that as a
>generic document, we probably don't want to go there. :-)
>
>This could also be moved to the applicability document, but we tried to
>move these sorts of discussions outside either one of those docs, since
>they didn't seem to match the flow of the documents themselves. I don't
>know, actually, where we would put them in the methdology draft--it would
>have to either go in the front, in a general considerations section of some
>type, or it would have to go in the individual tests, repeated as needed,
>or at the end, where it likely wouldn't get read.
>
>I'll defer to the working group on this one--I see no compelling reason to
>move them or to leave them.
>
>First, I'm not certain where we are defining these drafts as black box
>only? We refer to a paper on black box measurements, but I don't think
>this methodology draft is confined to black box measurements. There are
>specific warnings in various discussions about using white box
>measurements, and the problems you can have with them.
>
>Second, the concept of tree depth and tree breadth might be considered in
>RFC2328, but I'm not certain it's not worth discussing it again, from the
>point of view of the impact the shape of the topology you feed to the
>routers can have on the test results.
>
>As for where it should be, again, I see no reason to move it or not move
>it.... I'll defer to the chair and the rest of the working group on this
>one.
>
>:-)
>
>Russ
>
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Manral, Vishwas" <[email-masked]>
Date: Tue, 11 Mar 2003 11:15:55 -0500

Hi Scott,

Thanks,
Vishwas 

-----Original Message-----
From: Scott Poretsky [mailto:[email-masked]]
Sent: Tuesday, March 11, 2003 9:08 PM
To: Russ White
Cc: [email-masked]
Subject: Re: [bmwg] Comments for OSPF Convergence Benchmarking
Terminology

Kevin,

Scott

implemented
include