Subject: RE: [bmwg] Is the BMWG a proper home for this I-D?

From: Jim McQuaid <[email-masked]>
Date: Wed, 7 Apr 2004 10:56:35 -0000

<sorry for the blank reply>

At the time the IPPM was "spun off" of the BMWG, the general understanding
was that BMWG would look at "things" more or less in isolation, although
that concept is somewhat elastic.  IPPM would look at end to end network
kinds of performance issues.

It seems clear to me that convergence is a performance characteristic of
complete networks and primarily an end to end or at least, core element to
core element characteristic.

My thought would be that this is interesting and useful work but not
necessarily for BMWG.

Jim McQuaid

-----Original Message-----
From: Howard C. Berkowitz [mailto:[email-masked]]
Sent: Tuesday, April 06, 2004 7:43 PM
To: [email-masked]
Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

>Supporters of the BMWG:
>
>A [long] while back, Russ White approached the BMWG with an 
>individually submitted I-D
>that presented considerations when measuring network convergence:
>
>http://www.ietf.org/internet-drafts/draft-white-network-benchmark-00.txt
>
>The draft seeks to highlight: ...considerations that testers should 
>be aware of when
>attempting to measure network convergence using various methods. 
>(Adapted from the I-D's
>abstract.)
>
>Not being a pure terminology or methodological statement, or 
>constraining itself to
>laboratory evaluations, the topic might not be a perfect fit as BMWG 
>output.  On the
>other hand, the community might benefit from memo on this topic.
>
>Please offer your comments in two dimensions: a) does the topic 
>merit future effort, and
>b) is the BMWG the best home for this effort (as opposed to another 
>WG or progressing the
>work as an Individually submitted I-D)?  (You might have to read the 
>WG Charter to determine
>part B. :-)

I'm not quite sure that I'm ready to answer in the specific 
dimensions, but I can make some comments. I'll add a dimension that I 
have always believed that it would be highly desirable for the IETF 
to generate more frameworks, tutorials, etc., that help introduce 
newbies to a technology, and also help control the salesdroids from 
running amuck.

>
>
>Network Working Group                                           R. White
>Internet Draft                                                 V. Manral
>Expiration Date: August 2003                                    R. Adams
>File Name: draft-white-network-benchmark-00.txt            February 2003
>
>   Considerations in Benchmarking Routing Protocol Network Convergence
>                   draft-white-network-benchmark-00.txt
>
>1. Motivation
>
>    As the ability to benchmark components within a network appears to be
>    coming under greater scrutiny, and specifications are being written
>    to standardize ways to measure the performance of individual
>    components within given frameworks, the next level of benchmarking
>    has not been approached, that of measuring the perfomance of
>    networks. But what is meant when we say the performance of a network,
>    from the perspective of routing protocols? Various tests have been
>    used in the past to measure the convergence of a network, some of
>    which actually measure completely different things than others.
>
>    It's important to attempt to examine the measurement of network
>    convergence in a way that exposes these differences, and helps
>    vendors, end users, and those in the research community have some
>    common ground when discussing network convergence.
>
>
>2. A Problem of Definitions
>
>    As we examine the issues and concepts surrounding the measurement of
>    network performance in terms of convergence, we find that most of the
>    basic problems we face surround defining the terms in use. For
>    instance, what is convergence, exactly? What is a network? In the
>    following sections, we discuss each of these concepts, and attempt to
>    address each one.
>
>
>2.1. Networks
>
>    In its most nominal form, a network is composed of a group of devices
>    interconnected in some way, which send data over these
>    interconnections for various purposes. But, when we discuss the
>    concept of routing protocol convergence within a network, the
>    definition needs to be more precise. For instance, since hosts do
>    not, generally, participate in routing, should they be considered a
>    part of the network when benchmarking the performance of a routing
>    protocol?  The obvious answer appears to be a resounding no, but, in
>    some possible tests types, hosts which do not participate in routing
>    play a large part in the test itself.
>
>    When considering tests in which hosts participate as traffic or route
>    generators, then, we must consider the impact these hosts have on the
>    test results, although we may not consider them a part of the network
>    we are measuring the performance of.
>
>
>2.2. Convergence
>
>    Convergence is probably one of the hardest words in networking to
>    define.

:-) you mean it DOESN'T mean running voice, data, and video on the 
same network?

>Just about everyone who has worked on networks for a period
>    of time knows what it means, but no-one can explain it sufficently to
>    someone who doesn't understand how a network works for it to be
>    understood. In fact, this is because there are several different
>    meanings attributed to convergence, and which meaning is intended
>    depends on the context in which the word is set. Convergence can
>    mean:
>
>
>    o    The time at which all the routing protocol processes running on
>         devices which participate in routing in the network agree on the
>         best path to each reachable destination in the network.
>
>    o    The time at which the best path to each reachable destination in
>         the network has been loaded into some local table which may then
>         be used to forward packets (the routing information base, or
>         RIB).
>
>    o    The time at which each router in the network has built the
>         tables necessary to actually forward packets through the net-
>         work, so that a packet transmitted from one part of the network
>         would actually reach any given reachable destination within the
>         network.
>
>    For instance, on a Cisco router, show ip ospf stats would allow the
>    tester to see the time of the last completed SPF, show ip route would
>    allow the tester to see what routes are installed in the RIB, and
>    show ip cef would allow the tester to see the forwarding information
>    which has been built from the RIB. Each test designed to measure the
>    performance of routing protocols within a network must determine
>    which type of convergence is being measured, if that measurement is
>    acceptable to the information being gathered, and which test will
>    actually measure the desired type of convergence.

Not to criticize, because we consciously restricted our scope to 
single router in draft-ietf-bmwg-conterm-05.txt, leaving single-AS as 
a future and Internet-wide as outside our scope, is this definition 
not specific to single routers?

Should the paper include any definitions that apply to groups of 
routers?  Can the convergence of a group of routers, perhaps under 
the constraint that they start from a known state, receive some set 
of updates, then run to convergence, be described adequately in terms 
of the convergence of the last router to meet single-router 
convergence criteria?

>
>3. Polling Devices in a Network

By "the network", are you now suggesting a routing domain rather than 
a single router?

>
>    One common way to measure network convergence is to poll the devices
>    in the network, using some command suppplied within the routing
>    software, to determine when particular events have occured, or par-
>    ticular pieces of information have reached all the routers in the
>    network. Polling elimiates the need for the clock of each device
>    within the network to be synchronized for the test to have meaningful
>    results. However, there are some issues with the rate of polling dev-
>    ices within the network which need to be addressed in any test which
>    polls devices for this information; the first is the rate at which
>    polling takes place.
>
>    If, in a test, you are attempting to measure some parameter to within
>    one second of its occurance, then you would need to poll at a rate
>    much higher than once per second.
>
>     test starts here
>      |
>      |                   event occurs here
>      |                   |
>      v                   v
>    -+----------+----------+----------+---
>     ^          ^          ^          ^
>     |          |          |          |
>     0 seconds  1 second   2 seconds  3 seconds
>
>    For instance, in this time line, suppose a polling event is set up
>    which takes place every second. An event is started just after some
>    polling event takes place, but the polling process doesn't recognize
>    the test as starting until the 1 second poll. An event occurs just
>    before the 2 second poll, and the polling process detects this at the
>    2 second poll. The polling process would indicate that from the time
>    the event started until the time the event has finished, one second
>    has elapsed. In reality, closer to two seconds has elapsed.
>
>    The interval of the polling process can be reduced until the measure-
>    ment is felt to be accurate, but it should be at least half of the
>    desired accuracy. Common practice actually shows that it should be
>    about one tenth of the desired accuracy.
>
>    A second consideration when polling for network events is the prefor-
>    mance of the device running the polling process. If the process can-
>    not poll each device at the scheduled interval, or the polling is
>    "jittered," the time between each actual poll varies by some amount,
>    the accuracy of the tests will be called into question. The amount of
>    jitter introduced by the polling device, and the rate at which the
>    device can effectively poll, should be measured in some way, and this
>    measurement should be taken into account when designing tests which
>    rely on polling.

But are there not cases where some jittering is desirable to avoid 
unexpected weak synchronization effects (assuming the measurement 
itself has some Heisenbergian flavor), is it not wise to lengthen the 
test interval, or do multiple measurement runs with different seeds 
to a pseudorandom number generator, so the effect of jitter nulls out 
statistically?

>    Finally, when polling devices to determine when a network event
>    occurs, issues with serialization must be considered. Most devices
>    which would be used for polling will not be able to poll several dev-
>    ices within the network at once, and will thus serialize the polling
>    of devices.
>
>     p1   p3  p5  p7  p9
>      | p2| p4| p6| p8| p10
>      | | | | | | | | | |
>      v v v v v v v v v v
>    -+----------+----------+----------+---
>     ^          ^          ^          ^
>     |          |          |          |
>     0 seconds  1 second   2 seconds  3 seconds
>
>    Suppose, for instance, that a single device is polling ten devices in
>    the network. If it can poll five devices per second, it will take a
>    full two seconds for it to detect any event on all ten devices, giv-
>    ing an effective accuracy of about four seconds. The amount of time
>    required for a polling device to serialize through all the devices it
>    is polling needs to be considered when polling a very large number of
>    devices.

Or if there is a point where realistic measurement requires multiple 
polling devices.

>
>
>4. Tests to Measure Routing Protocols Convergence
>
>    In this section, we will outline some of the various tests which have
>    been used in the past to measure routing protocols convergence within
>    a network, and discuss some aspects of these tests.

It occurs to me that there is something of value here that probably 
isn't benchmarking, but certainly is relevant to the Operations Area. 
For want of a better term, I'll call it walking the reachability 
tables in a domain.

I'm currently doing some work with automated evaluation of student 
configurations, and part of that involves generating a meaningful 
script to check the control plane on multiple routers, at the point 
where a step of a scenario is complete. We ask ourselves the question 
of whether it is truly necessary to do the equivalent of 
ping-every-nonlocal-interface from each router in the domain, or if 
there is some meaningful way to define a smaller and more practical 
subset.

>
>
>4.1. Determining When Each Device has Received Information About All
>    Reachable Destinations
>
>    In link state protocols, information is flooded throughout the net-
>    work; discovering when each router in the network has received this
>    information is an important consideration in network convergence.
>    Slower flooding times will, of course, mean slower network conver-
>    gence overall, thus flooding performance directly impacts overall
>    routing protocols performance in the network.
>
>    There are three methods which can be used to determine when the
>    flooding of information has been completed.

A clarification here:  as I read this, I have a sense that you think of
black box testing as done by external stimulus/response devices, while
external packet monitoring is done by inference from live traffic streams.
Another way of putting this is that black box is active while monitoring
is passive.

Do I misconstrue what you mean?

>
>
>4.1.1. Black Box Polling
>
>[snip]
>4.1.2. White Box Output
>
>[snip]
>
>4.1.3. External Packet Monitoring

[snip]

>
>4.2. Determining When Each Device has Finished Finding the Best Path to
>    Each Reachable Destination
>
>    This is, probably, the most difficult measurement to take in a net-
>    work, since there are no known black box ways of determining when a
>    device has finished computing the best path to each destination in
>    the network. The only possible way of measuring this time is to use
>    output from the devices in the network to provide this information.
>
>    It's possible to poll each device periodically, examining output pro-
>    vided by the devices, to determine when each device has calculated
>    the best path to each destination in the network. This method is sub-
>    ject to the limitations described in the section on polling devices,
>    above.
>
>    It's also possible to rely on some event driven output of each device
>    in the network. For this to yield accurate results, the time clocks
>    of all the devices in the network must be closely synchronized.

While this may be "ideal" convergence, it does imply what I'll 
loosely call best exit routing policy.  There may be a perfectly 
reasonable earlier point where a router can reach all destinations 
and do some, although not ideally performing, work, as opposed to 
when it has the best path to all destinations.

[snip]
>
>
>4.3.1. The Various Elements of Performance Cannot Be Seperated
>
>    Using this sort of testing, there is no way to seperate the perfor-
>    mance of a routing protocol from the performance the interaction
>    between the routing protocol and the forwarding engine, nor from the
>    performance of the forwarding engine itself.

Here I disagree, if I take a broad look at "routing". Your statement 
is probably true if we limit ourselves to packet forwarding, even 
though it doesn't quite fit systems with a single control plane but 
multiple forwarders.

It is probably reasonable to talk of IP control plane in some 
isolation, if the control information is then used to set up (G)MPLS.

What I am not seeing is conditions of workload and test -- issues 
such as convergence of N routes in topologies where the router under 
test has M or 2M peers.

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 7 Apr 2004 09:28:57 -0400

>My thought would be that this is interesting and useful work but not

I don't completely disagree with you.  I do feel that single-platform 
performance definition and benchmarking is definitely within the 
scope of BMWG, and some of my comments relating to clarifying test 
methods (e.g., black box vs. white box vs. passive analysis) 
definitely can be applied to one box.  BMWG has always looked at 
forwarding plane performance of single boxes, and I really don't 
think we are straying far to look at there control plane performance.

You are correct that IPPM, as well as some of the internet 
scalability issues both in the IETF and IRTF, are probably better 
places for end-to-end convergence.

At least in the case of BGP, however, there is a meaningful 
intermediate level that isn't single box and isn't end-to-end: 
convergence within an AS, and conceivably subsets thereof such as 
route reflector clusters.  Similar points can be made about 
intra-area convergence in OSPF and ISIS.  Where do these fit?

I'm inclined to say BMWG, and let me suggest what might be a very 
crude rule of thumb in saying BMWG or not-BMWG.  Many of the 
Internet-wide experiments have to rely on sampling live data, with 
perhaps some selective probes.  Single-box, or small-system-of-boxes 
measurement, plausibly can be done with an artificial benchmark in a 
testing lab.

Given the name of this group, perhaps, then, the decision if 
something belongs here depends on asking the question early, "can the 
measurement be made with a synthetic benchmark and test equipment," 
or does some of it really have to come from sampling and probing live 
networks?  If the former, it's BMWG eligible. If the latter, it's 
interesting, might even warrant BMWG liaison and/or help in placing 
it elsewhere, but shouldn't stay in BMWG.

---

From: "Robert Holley" <[email-masked]>
Date: Wed, 7 Apr 2004 09:36:45 -0400

I think the concept of "things in isolation" can be
broadened to encompass things not necessarily contained
on a single element.

For example, routing protocol convergence would impact
multiple components, 1) single node reaching steady state
2) all routing nodes in AS reaching steady state.

However, routing protocol convergence may be one
of many factors that would make up the
"overall end-to-end" performance picture.

A complex network may consist of many things that
can be initially considered in isolation, but each
having subtle relationships, which could be
"reconstituted" into an end-to-end picture.

For example, a network carrying MPLS tunnels,
Multicast, some VPN technologies etc. and
in addition running a routing protocol.

At first each one of the technologies could be
studied independently and "benchmarked". Then the
overall description of the network performance would
be a combination of each of the independent benchmarks.

So, it may be that the benchmarking group could develop a
benchmark description for each of the various routing protocols,
then some group could develop benchmarks for the other technologies
(e.g. MPLS, Multicast etc.) and finally one group could tie
them all together for an end-to-end comprehensive description
of network performance.

Thats my view on the problem, but I can't really help decide
which group should be doing the work.

regards
Bob

> From: [email-masked] [mailto:[email-masked]]On Behalf Of Jim
> McQuaid
> Sent: Wednesday, April 07, 2004 6:57 AM
> To: Howard C. Berkowitz; [email-masked]
> Subject: RE: [bmwg] Is the BMWG a proper home for this I-D?
>
>

---

From: David Newman <[email-masked]>
Date: Wed, 7 Apr 2004 09:19:42 -0700 (PDT)

On Wed, 7 Apr 2004, Howard C. Berkowitz wrote:

> definitely can be applied to one box.

What about 2547bis et al, where one physical element contains multiple
logical elements? In effect a "one box only" rule would let slide virtual
technologies like MPLS VPNs (which IMO shouldn't be getting any slack...)
but not more basic stuff like straight routing.

Rather than getting hung up on one vs. not one box, it might be more
constructive to draw a distinction between measurements made in isolation
and those made end-to-end on live networks (for some definition of
"live").

The bmwg has always worked to define the measurement of devices or systems
(multiple devices) in a controlled environment. This is the "bench" in
benchmarking and while it's conceptually 180 degrees the opposite of live
network testing it doesn't necessarily rule out multiple-box tests either.

dn

---

From: David Newman <[email-masked]>
Date: Wed, 7 Apr 2004 09:19:42 -0700 (PDT)

dn

---

From: "Robert Holley" <[email-masked]>
Date: Wed, 7 Apr 2004 14:04:43 -0400

To a certain extent Howard's example of multiple components
of a single routing protocol re-enforces my earlier concept,
by diving down in a hierachy.

He referred to route reflectors, intra-as etc, as all being
sub-components of a problem (focusing on routing protocols).

And the next level UP in the hierarchy would be routing
protocols themselves as a part of a bigger picture
(as I eluded to earlier).

As far as the mentioned rule of thumb.... on "synthetic lab" vs
"field measurement";

I'm not familar with your history on this criteria, but I would
assume that if standard interfaces (SNMP) are used to define the "model",
then it could be evalutated in the lab with generators and monitoring tools,
and also deployed in the field (at least the monitoring part).

Of course there will most likely be the issues of MIB development to
support the requirements for the newly defined benchmark interfaces.

I assume MIB development is another group?

regards
Bob

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Howard C. Berkowitz
> Sent: Wednesday, April 07, 2004 9:29 AM
> To: [email-masked]

---

From: "Robert Holley" <[email-masked]>
Date: Wed, 7 Apr 2004 14:04:43 -0400

regards
Bob

---

From: Kevin Dubray <[email-masked]>
Date: Wed, 07 Apr 2004 15:17:51 -0400

> network testing it doesn't necessarily rule out multiple-box tests either.

Hear, hear.

This is reinforced by language in the charter:

"To better distinguish the BMWG from other measurement initiatives
in the IETF, the scope of the BMWG is limited to technology
characterization using simulated stimuli in a laboratory environment.
Said differently, the BMWG does not attempt to produce benchmarks for
live, operational networks."

No mention of number of nodes. :-)

-Kevin

---

From: Kevin Dubray <[email-masked]>
Date: Wed, 07 Apr 2004 15:17:51 -0400

Hear, hear.

-Kevin

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 7 Apr 2004 17:23:49 -0400

>but not more basic stuff like straight routing.

Let me clarify. Single box is clearly in scope, but I have absolutely 
nothing against multiple box as long as those boxes can be bounded 
and the bounded system tested with isolated loads. Are we in violent 
agreement?

What I was trying to exclude was "Internet-wide" scope and 
uncontrolled, non-repeatable loads.

>and those made end-to-end on live networks (for some definition of

That's valid.  I'd also ask Russ et al if this distinction takes 
passive traffic monitoring out of the scope here, and brings us back 
to white vs. black box only.

>network testing it doesn't necessarily rule out multiple-box tests either.

Agreed.

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 7 Apr 2004 17:23:49 -0400

Agreed.

---

From: Russ White <[email-masked]>
Date: Fri, 9 Apr 2004 10:31:47 -0400 (Eastern Daylight Time)

> nothing against multiple box as long as those boxes can be bounded and
> the bounded system tested with isolated loads. Are we in violent
> agreement?
>
> What I was trying to exclude was "Internet-wide" scope and uncontrolled,
> non-repeatable loads.

Exactly, but that's where the draft is so troublesome. I actually worked on
it in terms of experience in lab environments. But, I can see where it
would overlap with testing in live networks, so I'm not certain. I'm just
afraid the ippm wg is going to say it doesn't belong there, either, since
it's lab, rather than real world, in their eyes.

> That's valid.  I'd also ask Russ et al if this distinction takes passive
> traffic monitoring out of the scope here, and brings us back to white vs.
> black box only.

I don't think so.... With passive monitoring, you're not necessarily
talking about passively monitoring user traffic, you could be passively
monitoring rp traffic, or even bpdu's, or arp, or injected test traffic, or
a number of other things.

So, in that sense, I don't think passive monitoring is out of scope.

:-)

Russ

__________________________________
[email-masked] CCIE <>< Grace Alone

---

From: Russ White <[email-masked]>
Date: Fri, 9 Apr 2004 10:31:47 -0400 (Eastern Daylight Time)

:-)

Russ

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Fri, 9 Apr 2004 11:37:11 -0400

>So, in that sense, I don't think passive monitoring is out of scope.

So the key difference between black box and passive is that black box 
actually specifies the generator and receiver, where passive is a set 
of rules for analyzing and correlating streams conformant with the 
protocol definition?

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Fri, 9 Apr 2004 11:37:11 -0400

---

From: Russ White <[email-masked]>
Date: Wed, 14 Apr 2004 15:51:02 -0400 (Eastern Daylight Time)

> actually specifies the generator and receiver, where passive is a set of
> rules for analyzing and correlating streams conformant with the protocol
> definition?

I think that sounds like a fair definition.

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Wed, 14 Apr 2004 15:51:02 -0400 (Eastern Daylight Time)

:-)

Russ

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 14 Apr 2004 20:51:01 -0400

>I think that sounds like a fair definition.

The former is nice for test engine vendors and the latter for 
protocol analyzer vendors?

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 14 Apr 2004 20:51:01 -0400