Subject: [bmwg] Comments for OSPF Convergence Benchmarking Terminology

From: Scott Poretsky <[email-masked]>
Date: Mon, 10 Mar 2003 16:58:36 -0500

Folks,

Below are comments to each section of the Terminology Document.  I 
previously provided comments to the authors about 3 weeks ago.  Those 
comments were around the merits of Control Plane Convergence measurements 
versus Data Plane Convergence measurements.  There was good discussion 
between Russ and I that ultimately led us to mutually agree that documents 
for both approaches should exist, as both approaches are mutually exclusive 
for terminology and methodology.  This is a follow-up with response to the 
specific content.  Please reply with comments.

Thanks,
Scott
########################################
1. Terminology Section
The terminology section has 10 terms.  7 of the 10 are existing 
terms.  These are Internal Measurement, External Measurement, 
Point-to-Point Link, Broadcast Link, SPF Time, Hello Interval, and Router 
Dead Interval.  Of the other 3, Multi-Device Measurement serves no useful 
purpose, Incremental-SPF is an optional internal algorithm, and convergence 
is incorrectly defined.  Here are detailed remarks for each one.

Internal Measurements - This is a redefinition of White Box 
measurement.  Recommend it be removed.

External Measurements - This is a redefinition of Black Box 
measurement.    Recommend it be removed.  In the Discussion section for 
External Measurements it states:  "when a downstream device receives 
complete routing information from the DUT, it can be inferred that the DUT 
has transmitted all of the routing information available".
How can tests be built so that it is not "inferred", but instead "known"?

Multi-Device Measurement - This is a definition describing a test with 
multiple DUTs.  This term serves no purpose.  In fact, it does not even 
appear in the companion methodology.  Recommend it be removed.

Point-to-Point Link - Term was already defined by the Point-to-Point 
Protocol Working Group.  See RFC 1171, The Point-to-Point Protocol for the 
Transmission of Multi-Protocol Datagrams over Point-to-Point 
Links.  Recommend it be removed.

Broadcast Link - While this term is not defined, but frequently used, in 
IETF documents, it is a well-known term defined in IEEE 802 
documents.  Recommend it be removed.

SPF Time - This is a white box parameter that is not externally 
observable.  RFC 2328 already defines SPF.  Recommend it be removed.

Hello Interval - This is a term already defined in RFC 2328, OSPFv2, 
Section 9, The Interface
Data Structure and its use described in detail RFC 2328, OSPFv2, in Section 
9.5, Sending Hello
Packets and Section 10 Neighbor Data Structure.  Recommend it be removed.

Router Dead Interval -  This is a term already defined in RFC 2328, OSPFv2, 
Section 9, The Interface
Data Structure and its use described in detail RFC 2328, OSPFv2, Section 10 
Neighbor Data Structure.  Recommend it be removed.

Incremental-SPF - This is a white box component of OSPF. Implementation of 
this feature is optional.
Superior Convergence times can be obtained without Incremental-SPF (as two 
vendors have demonstrated).  Black box measurements cannot indicate if 
Incremental-SPF is internally implemented in the DUT.  RFC 2328 makes one 
reference to Incremental-SPF as an example algorithm to improve SPF 
run-time.  It does not
require it nor indicate it to be the _most_ efficient algorithm.  If we 
were to standardize internal SPF execution algorithms then we should also 
include PRC, RTC, and others.  There would be no reason to choose just 
Incremental-SPF.  Recommend it be removed.

Convergence - Measured by Time to Flood, Run SPF, and Update RIB.  This is 
a partial measurement of router
components that contribute to Convergence time.  Time to update FIB is 
certainly a missing component.  Also, network operators speak of 
"Convergence" as the time it takes for all nodes in the network to 
converge.  I propose that this term be renamed "Protocol Convergence" or 
"Control-Plane Convergence".

In summary for this section, I see Convergence as the only term that is 
original, but it must still be renamed (two alternates suggested 
above).  If it is decided to keep other terms, please add them to a section 
"Existing Terms" with references for each one.

2. Measuring Traffic Convergence Section
Three measurements are listed.  Here are the concerns:
A. LSA Propagation
B. SPF Execution - This is a White Box Measurement
C. RIB Update - How can this be measured in isolation from 1 and 2 without 
it being a white-box test?

3. Type of Network Events Section
Next-Hop change (e.g. due to metric change) is a unique event because no 
packet loss is expected, so this should also be listed.  The other events 
listed - Link Up/Down, Session Initialization, and Adjacency Down
produce packet loss.

4. LSA and Destination Mix
No new terminology.  Methodology related information.  Recommend move to 
methodology document.

5. Tree Shape and SPF algorithm
These are White Box items.  Neither is externally observable.  Tree shape 
is already described in RFC 2328.  I recommend that this section be 
removed.  It could be added as an appendix to the methodology document, but 
that is not worth it since it is already covered in RFC 2328.

---

From: Russ White <[email-masked]>
Date: Mon, 10 Mar 2003 18:24:28 -0500 (Eastern Standard Time)

> Internal Measurements - This is a redefinition of White Box measurement.
> Recommend it be removed.

The definition is not the major point of this defintion, but rather the
discussion about internal measurements, that they can impact the
performance of the box being measured, etc. I don't know if there is a
"generic" draft which covers these sorts of things, or if one is planned,
but it could go there just as well as here. In the meantime, it seems like
a useful discussion to have on paper.

> External Measurements - This is a redefinition of Black Box measurement.
> Recommend it be removed.  In the Discussion section for External
> Measurements it states:  "when a downstream device receives complete
> routing information from the DUT, it can be inferred that the DUT has
> transmitted all of the routing information available". How can tests be
> built so that it is not "inferred", but instead "known"?

You can't do more than infer, in most cases. You can state that it is
"known" that the DUT has finished sending information, but you don't know
if you are going to, for instance, drop the last ack, and force the DUT to
retransmit.

> appear in the companion methodology.  Recommend it be removed.

Actually, in our definition, a multi-device test requires measurement of
the test results on several devices, not necessarily several DUTs. For
instance, you may need to measure the timestamp on both the route generator
and the DUT, which means it's a multi-device mesasurement (not a
multi-device test, which would imply something different). I don't argue
that we didn't use the term in the methodology, but we certainly described
some multi-device measurements in some of the tests.

Again, the discussion is important here, in how some of these issues
surrounding multi-device measurements.

> Protocol Working Group.  See RFC 1171, The Point-to-Point Protocol for
> the Transmission of Multi-Protocol Datagrams over Point-to-Point Links.
> Recommend it be removed.

First, the concept of a point-to-point link encompasses more link types
that the PPP working group covers. For instance, there is a draft in the
IS-IS and OSPF working groups which discusses point-to-point ethernet
links, and how these protocols should treat these links different than a
normal ethernet link. In this case, I'm not certain the duplicate defintion
is a bad thing, an the discussion about convergence times in OSPF vis-a-vis
point-to-point links needs a home.

> IETF documents, it is a well-known term defined in IEEE 802 documents.
> Recommend it be removed.

Note the definition is in terms of OSPF, and the benchmark draft. Again, it
may be a commonly known term, but the concept of a DR being elected on the
link is something which is specific to OSPF.

> observable.  RFC 2328 already defines SPF.  Recommend it be removed.

There are tests included which discuss how to measure SPF time from outside
the box, and inside the box. I'm fairly certain the methodology draft
includes white box measurements, so I'm not certain why white box
measurements shouldn't be included in the terminology draft?

> Section 9, The Interface Data Structure and its use described in detail
> RFC 2328, OSPFv2, in Section 9.5, Sending Hello Packets and Section 10

The discussion here is just as important as the definition, and applies to
all tests discussed in the methodology draft. Suggestions on where else we
would put this discussion?

> Router Dead Interval - This is a term already defined in RFC 2328,
> OSPFv2, Section 9, The Interface Data Structure and its use described in
> detail RFC 2328, OSPFv2, Section 10 Neighbor Data Structure.  Recommend
> it be removed.

See above, on hello interval.

> Incremental-SPF - This is a white box component of OSPF. Implementation
> of this feature is optional. Superior Convergence times can be obtained
> without Incremental-SPF (as two vendors have demonstrated).  Black box
> measurements cannot indicate if Incremental-SPF is internally implemented
> in the DUT.  RFC 2328 makes one reference to Incremental-SPF as an
> example algorithm to improve SPF run-time.  It does not require it nor
> indicate it to be the _most_ efficient algorithm.  If we were to
> standardize internal SPF execution algorithms then we should also include
> PRC, RTC, and others.  There would be no reason to choose just

I don't see it used in either draft, so this is fine.

> Convergence - Measured by Time to Flood, Run SPF, and Update RIB.  This
> is a partial measurement of router components that contribute to
> Convergence time.  Time to update FIB is certainly a missing component.
> Also, network operators speak of "Convergence" as the time it takes for
> all nodes in the network to converge.  I propose that this term be
> renamed "Protocol Convergence" or "Control-Plane Convergence".

Either way is fine with me.... Of course, this definition draft only sets
the meaning of words for the accompanying methods and applicability drafts,
and not in general. Others have defined convergence differently, so.... I
suppose that we, as a working group, can go one of two directions on this:

-- We come up with a new name for convergence, prepending it with some
other word, to imply different things in different places. This could
probably lead to large and protracted arguments, and perhaps
overspecification of the meaning of words.

(for instance, we've already added "SR" to the front of convergence for a
similar objection, so now we'll have:

-- SR-CP-Convergence, Single Router Control Plane
-- N-CP-Convergence, Network Control Plane
-- SR-DP-Convergence, Single Router Data Plane
-- N-DP-Convergence, Network Data Plane

as a base set of variants, with, I assume, many many more to come).

-- We allow each draft to use the word "convergence" within it's bounding
limits (the dictionary definition), and just make certain that each draft
discusses its definition of convergence at some point, to clarify for the
readers of that draft. always remembering to have a flexible mind, and seek
out the meaning of convergence before reading any given BMWG draft.

I'll defer to the working group for that discussion and decsion, though I
prefer the latter.

> B. SPF Execution - This is a White Box Measurement

A test is included in the methods draft, so I'm not certain why this
shouldn't be here?

> C. RIB Update - How can this be measured in isolation from 1 and 2
> without it being a white-box test?

We don't include a test for this, so whether it is white box or black box
makes no difference? It's just noted as a part of overal convergence times,
to be complete in the discussion of the elements of convergence time.

> listed - Link Up/Down, Session Initialization, and Adjacency Down

We didn't discuss packet loss in any section, since we are concerned with
contorl plane convergence, rather than data plane convergence (?). We can
add next-hop change, although I'm not certain we base any tests on it, and
making this list exhasutive would, well, unless we want to do that as a
generic document, we probably don't want to go there. :-)

> No new terminology.  Methodology related information.  Recommend move to

This could also be moved to the applicability document, but we tried to
move these sorts of discussions outside either one of those docs, since
they didn't seem to match the flow of the documents themselves. I don't
know, actually, where we would put them in the methdology draft--it would
have to either go in the front, in a general considerations section of some
type, or it would have to go in the individual tests, repeated as needed,
or at the end, where it likely wouldn't get read.

I'll defer to the working group on this one--I see no compelling reason to
move them or to leave them.

> that is not worth it since it is already covered in RFC 2328.

First, I'm not certain where we are defining these drafts as black box
only? We refer to a paper on black box measurements, but I don't think
this methodology draft is confined to black box measurements. There are
specific warnings in various discussions about using white box
measurements, and the problems you can have with them.

Second, the concept of tree depth and tree breadth might be considered in
RFC2328, but I'm not certain it's not worth discussing it again, from the
point of view of the impact the shape of the topology you feed to the
routers can have on the test results.

As for where it should be, again, I see no reason to move it or not move
it.... I'll defer to the chair and the rest of the working group on this
one.

:-)

Russ

-- 
[email-masked] CCIE <>< Grace Alone
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Russ White <[email-masked]>
Date: Mon, 10 Mar 2003 18:24:28 -0500 (Eastern Standard Time)

:-)

Russ

---

From: Scott Poretsky <[email-masked]>
Date: Tue, 11 Mar 2003 10:38:19 -0500

Kevin,

I thought BMWG drafts are not permitted to have White Box test 
coverage.  Is this correct?  I also thought that existing terminology 
should either be excluded or referenced in the Existing Terms section.  Is 
this correct?

Scott

---

From: Scott Poretsky <[email-masked]>
Date: Tue, 11 Mar 2003 10:38:19 -0500

Kevin,

Scott

---

From: Kevin Dubray <[email-masked]>
Date: Tue, 11 Mar 2003 11:42:10 -0500

Scott Poretsky wrote:
> coverage.  Is this correct? 

The focus of BMWG is providing tests that have general applicability.
Tests relying on internal (vendor-specific) mechanisms may be
dissimlar in what they report or how they report it. This
variance is generally the cause for us to steer clear of
"white box" measurements..

That said, if my read of <draft-ietf-bmwg-ospfconv-applicability-01>
is on, use of the internal mechanism, while not good for
vendor comparison, may be good for building understanding
of an implementation. For example, knowing how a device's CLI-reported
telmetry varies from the externally measured results may be
valuable insight.  (And insight is what we're all about here
in the BMWG. :-))

A suggestion might be to relegate all internal measurements and
their methodologies to an appendix with all the appropriate
warnings and caveats.  (Not "official" BMWG metrics; applicable to
a single implementation; use to reconcile internal measurements to 
external measurements, not to be cited for multivendor comparison,
blah, blah, blah.)

  I also thought that existing terminology
> should either be excluded or referenced in the Existing Terms section.  
> Is this correct?

Historically, "existing terms" was used to draw relevance to
other BMWG docs to ensure consistency.  This section, alas, has
not been used consistantly.

But to Ross' point: drawing relevance of the terms to use in
associated benchmarks is important.

Let's look at an example: Hello Interval.  Here the OSPF convergence
terminology I-D doesn't recast otherwise contradict references
in RFC 2328, a standard. So, I don't really have issue with
it's inclusion in the I-D.  (In point of fact, I find it useful
to have it inline with the draft.)  The discussion on timers
is important to this class of benchmarks.  Calling out related timers
is appropriate. (And required, if you ask me.)

For the lawyers, an alternativate might be:

      Hello Interval-

      Definition: see RFC 2328, sec. 9, sec. 9.5, sec. 10.

and move everything else to the "Discussion" section.

FWIW,
Kevin

>> measurement.

---

From: Kevin Dubray <[email-masked]>
Date: Tue, 11 Mar 2003 11:42:10 -0500

Hello Interval-

FWIW,
Kevin

---

From: Russ White <[email-masked]>
Date: Tue, 11 Mar 2003 15:26:29 -0500 (Eastern Standard Time)

> Tests relying on internal (vendor-specific) mechanisms may be dissimlar
> in what they report or how they report it. This variance is generally the
> cause for us to steer clear of "white box" measurements..

Right--I think we tried to warn readers of this in multiple places,
including the strengthening of the statement in the applicability draft
somewhat.

> A suggestion might be to relegate all internal measurements and their
> methodologies to an appendix with all the appropriate warnings and
> caveats.  (Not "official" BMWG metrics; applicable to a single
> implementation; use to reconcile internal measurements to external
> measurements, not to be cited for multivendor comparison, blah, blah,
> blah.)

Well, there's a balance there, and I'm not certain which side should play
in. The first part of the balance is that you have tests which are for the
same thing, and it's nice to have them categorized along those lines. There
are also tests which build on another test--subtract the results of test 1
from the results of test 2 to find x.

If we push all the white box tests into an appendix, then we mess up this
sort of flow in the doc, and make it just a collection of tests, with no
bearing or relationship on each other.

My alternate suggestion would be to refer the reader specifically to the
applicability doc for each white box test included, and make certain that
there's a section specifically for white box tests in the applicability
doc, that clearly explains what you've said above. Flow is, in my mind,
more important in being able to use something than getting the format
right. :-)

> But to Russ' point: drawing relevance of the terms to use in

Actually, that's a good idea, as well; take the text of the definitions
out, where there's discussion to be had, and simply put in a reference to
the specific draft where the definition is made.

I would be glad to do that, and add the warnings above, if everyone agrees
this is the way to proceed.

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Tue, 11 Mar 2003 15:26:29 -0500 (Eastern Standard Time)

:-)

Russ

---

From: Kevin Dubray <[email-masked]>
Date: Wed, 12 Mar 2003 08:30:29 -0500

> more important in being able to use something than getting the format

I'm all for flow - which is making me rethink my previous comment.
If we can't effectively segregate the mandatory, black box test set
from informative-but-difficult-to-standardize white box test set in
the user's mind, I think we've missed the mark. There's
no guarantee that the applicability doc will get read.

While I'm all for illluminating users as to possible areas
of exploration, the WG must focus on specifying the external tests.
The internal measurements can't be interpreted as external
measurement equivalents.

That said, for bmwg-ospfconv-intraarea-03, aren't we really talking
about only one internal measurement? (Section 8.3 SPF comput Time,
Internal Test)

An auxillary, alternate suggestion :-) would be to remove section
8.3 and mention the "interesting relationship," why it's interesting,
and it's associated caveats in the Applicability I-D, but leave the
user to derive their own methodology.  This would sow the seeds
of possibly insightful and non-mandated areas of exploration, but
leave the methodological details as an exercise to the reader.

A easier, (and more benchmark-centric?) alternative would be to
delete section 8.3. :-(

Apologies for waffling-
Kevin

---

From: Kevin Dubray <[email-masked]>
Date: Wed, 12 Mar 2003 08:30:29 -0500

Apologies for waffling-
Kevin

---

From: Russ White <[email-masked]>
Date: Wed, 12 Mar 2003 10:37:41 -0500 (EST)

> I'm all for flow - which is making me rethink my previous comment. If we
> can't effectively segregate the mandatory, black box test set from
> informative-but-difficult-to-standardize white box test set in the user's
> mind, I think we've missed the mark. There's no guarantee that the
> applicability doc will get read.
>
> While I'm all for illluminating users as to possible areas of
> exploration, the WG must focus on specifying the external tests. The
> internal measurements can't be interpreted as external measurement
> equivalents.
>
> That said, for bmwg-ospfconv-intraarea-03, aren't we really talking about
> only one internal measurement? (Section 8.3 SPF comput Time, Internal
> Test)

I think that's correct....

> An auxillary, alternate suggestion :-) would be to remove section 8.3 and
> mention the "interesting relationship," why it's interesting, and it's
> associated caveats in the Applicability I-D, but leave the user to derive
> their own methodology.  This would sow the seeds of possibly insightful
> and non-mandated areas of exploration, but leave the methodological
> details as an exercise to the reader.

Okay--I can change that section to a note explaining there are also white
box ways of testing spf time, with one exmaple being simply taking the
output of the vendor supplied commands. I can then point the reader to the
applicability document for "further explaination," perhaps, where there is
a section on the problems with white box tests, and also note the
relationship between the white and black box testing. (?).

I'll try to repost the entire draft in a bit, and also post just the
impacted paragraphs as well.

:-)

Russ

__________________________________
[email-masked] CCIE <>< Grace Alone

---

From: Russ White <[email-masked]>
Date: Wed, 12 Mar 2003 10:37:41 -0500 (EST)

:-)

Russ

---

From: David Newman <[email-masked]>
Date: Wed, 12 Mar 2003 09:20:59 -0800 (PST)

> output of the vendor supplied commands. 

A wise man and sometime denizen of this list, Mr. John Dawson, taught me 
long ago that it is dangerous to assume correct behavior on the part of 
the DUT. In this case, OSPF could be totally broken but the CLI or net 
management system could report everything is fine, or vice-versa. 

IOW, the internal reporting mechanism may be (and in my experience,
probably is) a different variable than the one you're actually studying.
When both variables are internal, it's not possible to say where one ends
and the other begins.

I'm sure you are well aware of this. My concern would be any MAYs or 
SHOULDs about internal observations.

I can then point the reader to the
> relationship between the white and black box testing. (?).

Oh, that's easy. With black box, all we have to do is characterize the
various ways in which the test methodology and/or test instrument are
broken, and compensate accordingly. With white box, forget it...

:-)

dn

---

From: David Newman <[email-masked]>
Date: Wed, 12 Mar 2003 09:20:59 -0800 (PST)

:-)

dn

---

From: Russ White <[email-masked]>
Date: Mon, 17 Mar 2003 14:43:36 -0500 (EST)

> > the Transmission of Multi-Protocol Datagrams over Point-to-Point Links.

Actually, 1171 has been obsoleted by 1331, and there doesn't appear to be
any definition of point-to-point links there?

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Mon, 17 Mar 2003 14:43:36 -0500 (EST)

:-)

Russ