Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

From: "Howard C. Berkowitz" <[email-masked]>
Date: Tue, 6 Apr 2004 19:43:26 -0400

>Supporters of the BMWG:
>
>A [long] while back, Russ White approached the BMWG with an 
>individually submitted I-D
>that presented considerations when measuring network convergence:
>
>http://www.ietf.org/internet-drafts/draft-white-network-benchmark-00.txt
>
>The draft seeks to highlight: ...considerations that testers should 
>be aware of when
>attempting to measure network convergence using various methods. 
>(Adapted from the I-D's
>abstract.)
>
>Not being a pure terminology or methodological statement, or 
>constraining itself to
>laboratory evaluations, the topic might not be a perfect fit as BMWG 
>output.  On the
>other hand, the community might benefit from memo on this topic.
>
>Please offer your comments in two dimensions: a) does the topic 
>merit future effort, and
>b) is the BMWG the best home for this effort (as opposed to another 
>WG or progressing the
>work as an Individually submitted I-D)?  (You might have to read the 
>WG Charter to determine
>part B. :-)

I'm not quite sure that I'm ready to answer in the specific 
dimensions, but I can make some comments. I'll add a dimension that I 
have always believed that it would be highly desirable for the IETF 
to generate more frameworks, tutorials, etc., that help introduce 
newbies to a technology, and also help control the salesdroids from 
running amuck.

>
>
>Network Working Group                                           R. White
>Internet Draft                                                 V. Manral
>Expiration Date: August 2003                                    R. Adams
>File Name: draft-white-network-benchmark-00.txt            February 2003
>
>   Considerations in Benchmarking Routing Protocol Network Convergence
>                   draft-white-network-benchmark-00.txt
>
>1. Motivation
>
>    As the ability to benchmark components within a network appears to be
>    coming under greater scrutiny, and specifications are being written
>    to standardize ways to measure the performance of individual
>    components within given frameworks, the next level of benchmarking
>    has not been approached, that of measuring the perfomance of
>    networks. But what is meant when we say the performance of a network,
>    from the perspective of routing protocols? Various tests have been
>    used in the past to measure the convergence of a network, some of
>    which actually measure completely different things than others.
>
>    It's important to attempt to examine the measurement of network
>    convergence in a way that exposes these differences, and helps
>    vendors, end users, and those in the research community have some
>    common ground when discussing network convergence.
>
>
>2. A Problem of Definitions
>
>    As we examine the issues and concepts surrounding the measurement of
>    network performance in terms of convergence, we find that most of the
>    basic problems we face surround defining the terms in use. For
>    instance, what is convergence, exactly? What is a network? In the
>    following sections, we discuss each of these concepts, and attempt to
>    address each one.
>
>
>2.1. Networks
>
>    In its most nominal form, a network is composed of a group of devices
>    interconnected in some way, which send data over these
>    interconnections for various purposes. But, when we discuss the
>    concept of routing protocol convergence within a network, the
>    definition needs to be more precise. For instance, since hosts do
>    not, generally, participate in routing, should they be considered a
>    part of the network when benchmarking the performance of a routing
>    protocol?  The obvious answer appears to be a resounding no, but, in
>    some possible tests types, hosts which do not participate in routing
>    play a large part in the test itself.
>
>    When considering tests in which hosts participate as traffic or route
>    generators, then, we must consider the impact these hosts have on the
>    test results, although we may not consider them a part of the network
>    we are measuring the performance of.
>
>
>2.2. Convergence
>
>    Convergence is probably one of the hardest words in networking to
>    define.

:-) you mean it DOESN'T mean running voice, data, and video on the 
same network?

>Just about everyone who has worked on networks for a period
>    of time knows what it means, but no-one can explain it sufficently to
>    someone who doesn't understand how a network works for it to be
>    understood. In fact, this is because there are several different
>    meanings attributed to convergence, and which meaning is intended
>    depends on the context in which the word is set. Convergence can
>    mean:
>
>
>    o    The time at which all the routing protocol processes running on
>         devices which participate in routing in the network agree on the
>         best path to each reachable destination in the network.
>
>    o    The time at which the best path to each reachable destination in
>         the network has been loaded into some local table which may then
>         be used to forward packets (the routing information base, or
>         RIB).
>
>    o    The time at which each router in the network has built the
>         tables necessary to actually forward packets through the net-
>         work, so that a packet transmitted from one part of the network
>         would actually reach any given reachable destination within the
>         network.
>
>    For instance, on a Cisco router, show ip ospf stats would allow the
>    tester to see the time of the last completed SPF, show ip route would
>    allow the tester to see what routes are installed in the RIB, and
>    show ip cef would allow the tester to see the forwarding information
>    which has been built from the RIB. Each test designed to measure the
>    performance of routing protocols within a network must determine
>    which type of convergence is being measured, if that measurement is
>    acceptable to the information being gathered, and which test will
>    actually measure the desired type of convergence.

Not to criticize, because we consciously restricted our scope to 
single router in draft-ietf-bmwg-conterm-05.txt, leaving single-AS as 
a future and Internet-wide as outside our scope, is this definition 
not specific to single routers?

Should the paper include any definitions that apply to groups of 
routers?  Can the convergence of a group of routers, perhaps under 
the constraint that they start from a known state, receive some set 
of updates, then run to convergence, be described adequately in terms 
of the convergence of the last router to meet single-router 
convergence criteria?

>
>3. Polling Devices in a Network

By "the network", are you now suggesting a routing domain rather than 
a single router?

>
>    One common way to measure network convergence is to poll the devices
>    in the network, using some command suppplied within the routing
>    software, to determine when particular events have occured, or par-
>    ticular pieces of information have reached all the routers in the
>    network. Polling elimiates the need for the clock of each device
>    within the network to be synchronized for the test to have meaningful
>    results. However, there are some issues with the rate of polling dev-
>    ices within the network which need to be addressed in any test which
>    polls devices for this information; the first is the rate at which
>    polling takes place.
>
>    If, in a test, you are attempting to measure some parameter to within
>    one second of its occurance, then you would need to poll at a rate
>    much higher than once per second.
>
>     test starts here
>      |
>      |                   event occurs here
>      |                   |
>      v                   v
>    -+----------+----------+----------+---
>     ^          ^          ^          ^
>     |          |          |          |
>     0 seconds  1 second   2 seconds  3 seconds
>
>    For instance, in this time line, suppose a polling event is set up
>    which takes place every second. An event is started just after some
>    polling event takes place, but the polling process doesn't recognize
>    the test as starting until the 1 second poll. An event occurs just
>    before the 2 second poll, and the polling process detects this at the
>    2 second poll. The polling process would indicate that from the time
>    the event started until the time the event has finished, one second
>    has elapsed. In reality, closer to two seconds has elapsed.
>
>    The interval of the polling process can be reduced until the measure-
>    ment is felt to be accurate, but it should be at least half of the
>    desired accuracy. Common practice actually shows that it should be
>    about one tenth of the desired accuracy.
>
>    A second consideration when polling for network events is the prefor-
>    mance of the device running the polling process. If the process can-
>    not poll each device at the scheduled interval, or the polling is
>    "jittered," the time between each actual poll varies by some amount,
>    the accuracy of the tests will be called into question. The amount of
>    jitter introduced by the polling device, and the rate at which the
>    device can effectively poll, should be measured in some way, and this
>    measurement should be taken into account when designing tests which
>    rely on polling.

But are there not cases where some jittering is desirable to avoid 
unexpected weak synchronization effects (assuming the measurement 
itself has some Heisenbergian flavor), is it not wise to lengthen the 
test interval, or do multiple measurement runs with different seeds 
to a pseudorandom number generator, so the effect of jitter nulls out 
statistically?

>    Finally, when polling devices to determine when a network event
>    occurs, issues with serialization must be considered. Most devices
>    which would be used for polling will not be able to poll several dev-
>    ices within the network at once, and will thus serialize the polling
>    of devices.
>
>     p1   p3  p5  p7  p9
>      | p2| p4| p6| p8| p10
>      | | | | | | | | | |
>      v v v v v v v v v v
>    -+----------+----------+----------+---
>     ^          ^          ^          ^
>     |          |          |          |
>     0 seconds  1 second   2 seconds  3 seconds
>
>    Suppose, for instance, that a single device is polling ten devices in
>    the network. If it can poll five devices per second, it will take a
>    full two seconds for it to detect any event on all ten devices, giv-
>    ing an effective accuracy of about four seconds. The amount of time
>    required for a polling device to serialize through all the devices it
>    is polling needs to be considered when polling a very large number of
>    devices.

Or if there is a point where realistic measurement requires multiple 
polling devices.

>
>
>4. Tests to Measure Routing Protocols Convergence
>
>    In this section, we will outline some of the various tests which have
>    been used in the past to measure routing protocols convergence within
>    a network, and discuss some aspects of these tests.

It occurs to me that there is something of value here that probably 
isn't benchmarking, but certainly is relevant to the Operations Area. 
For want of a better term, I'll call it walking the reachability 
tables in a domain.

I'm currently doing some work with automated evaluation of student 
configurations, and part of that involves generating a meaningful 
script to check the control plane on multiple routers, at the point 
where a step of a scenario is complete. We ask ourselves the question 
of whether it is truly necessary to do the equivalent of 
ping-every-nonlocal-interface from each router in the domain, or if 
there is some meaningful way to define a smaller and more practical 
subset.

>
>
>4.1. Determining When Each Device has Received Information About All
>    Reachable Destinations
>
>    In link state protocols, information is flooded throughout the net-
>    work; discovering when each router in the network has received this
>    information is an important consideration in network convergence.
>    Slower flooding times will, of course, mean slower network conver-
>    gence overall, thus flooding performance directly impacts overall
>    routing protocols performance in the network.
>
>    There are three methods which can be used to determine when the
>    flooding of information has been completed.

A clarification here:  as I read this, I have a sense that you think of
black box testing as done by external stimulus/response devices, while
external packet monitoring is done by inference from live traffic streams.
Another way of putting this is that black box is active while monitoring
is passive.

Do I misconstrue what you mean?

>
>
>4.1.1. Black Box Polling
>
>[snip]
>4.1.2. White Box Output
>
>[snip]
>
>4.1.3. External Packet Monitoring

[snip]

>
>4.2. Determining When Each Device has Finished Finding the Best Path to
>    Each Reachable Destination
>
>    This is, probably, the most difficult measurement to take in a net-
>    work, since there are no known black box ways of determining when a
>    device has finished computing the best path to each destination in
>    the network. The only possible way of measuring this time is to use
>    output from the devices in the network to provide this information.
>
>    It's possible to poll each device periodically, examining output pro-
>    vided by the devices, to determine when each device has calculated
>    the best path to each destination in the network. This method is sub-
>    ject to the limitations described in the section on polling devices,
>    above.
>
>    It's also possible to rely on some event driven output of each device
>    in the network. For this to yield accurate results, the time clocks
>    of all the devices in the network must be closely synchronized.

While this may be "ideal" convergence, it does imply what I'll 
loosely call best exit routing policy.  There may be a perfectly 
reasonable earlier point where a router can reach all destinations 
and do some, although not ideally performing, work, as opposed to 
when it has the best path to all destinations.

[snip]
>
>
>4.3.1. The Various Elements of Performance Cannot Be Seperated
>
>    Using this sort of testing, there is no way to seperate the perfor-
>    mance of a routing protocol from the performance the interaction
>    between the routing protocol and the forwarding engine, nor from the
>    performance of the forwarding engine itself.

Here I disagree, if I take a broad look at "routing". Your statement 
is probably true if we limit ourselves to packet forwarding, even 
though it doesn't quite fit systems with a single control plane but 
multiple forwarders.

It is probably reasonable to talk of IP control plane in some 
isolation, if the control information is then used to set up (G)MPLS.

What I am not seeing is conditions of workload and test -- issues 
such as convergence of N routes in topologies where the router under 
test has M or 2M peers.

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Tue, 6 Apr 2004 19:43:26 -0400

[snip]

[snip]

---

From: Russ White <[email-masked]>
Date: Fri, 9 Apr 2004 10:28:12 -0400 (Eastern Daylight Time)

> Not to criticize, because we consciously restricted our scope to single
> router in draft-ietf-bmwg-conterm-05.txt, leaving single-AS as a future
> and Internet-wide as outside our scope, is this definition not specific
> to single routers?
>
> Should the paper include any definitions that apply to groups of routers?
> Can the convergence of a group of routers, perhaps under the constraint
> that they start from a known state, receive some set of updates, then run
> to convergence, be described adequately in terms of the convergence of
> the last router to meet single-router convergence criteria?

Hmm.... I wonder if that's not one of the basic problems here--what is
network convergence verses single device convergence? So, suppose we
consider the network converged when every router in the network has
completed some stage of processing--whether you consider that recalculating
the protocol tables, installing the routes in the rib, or even building the
forwarding tables as needed--then you could, of course, poll every router
in the network (finding some clever mechanism to do so) to discover when
the network has converged.

On the other hand, does anyone really care if a "backup path" has
converged or not? Perhaps... That just all falls into the definition of the
word "converged," maybe (?).

> By "the network", are you now suggesting a routing domain rather than

Yes....

> unexpected weak synchronization effects (assuming the measurement itself
> has some Heisenbergian flavor), is it not wise to lengthen the test
> interval, or do multiple measurement runs with different seeds to a
> pseudorandom number generator, so the effect of jitter nulls out
> statistically?

Yes, you could do that.... It would be a good addition to the draft to
discuss ways around this particular problem, most likely.

> Or if there is a point where realistic measurement requires multiple

Yes, correct. You could, in theory, place a polling device behind each
polled device to get the most accurate measurements, and the correlation
the data afterwords. You'd need to put the measurement devices on some
seperate network infrastructure, so theire traffic doesn't impact the
traffic on the testbed network, of course.... But then there come in issue
of scale. Could you really build out a network of, say, 2000 devices, 1000
to test, and the other 1000 to measure the test, so you get "real world"
numbers in a test environment? That's the hard part....

> It occurs to me that there is something of value here that probably isn't
> benchmarking, but certainly is relevant to the Operations Area.  For want
> of a better term, I'll call it walking the reachability tables in a
> domain.
>
> configurations, and part of that involves generating a meaningful script
> to check the control plane on multiple routers, at the point where a step
> of a scenario is complete. We ask ourselves the question of whether it is
> truly necessary to do the equivalent of ping-every-nonlocal-interface
> from each router in the domain, or if there is some meaningful way to
> define a smaller and more practical subset.

Yes, that's part of the problem. Within the Scaling Team in Cisco, we've
seen the effects of each of these tests outlined (in a lab environment),
and we build new tests to avoid each of these. I would like to say you can
avoid all such problems in a lab environment, but maybe you can't--maybe
you just have to watch for them, and realize how they are impacting your
results (?).

> external packet monitoring is done by inference from live traffic
> streams. Another way of putting this is that black box is active while
> monitoring is passive.
>

Yes, although passive also falls within black box tests--I think it might
be useful to actually spearate the definitions of the two, though, for
clarity of discussion (?).... Although we risk becoming like doctors, with
a separate language, in our quest for precision, losing the concept of
"context" in interpreting words.... There's real danger in both directions,
inmprecision verses intelligability. :-)

> While this may be "ideal" convergence, it does imply what I'll loosely
> call best exit routing policy.  There may be a perfectly reasonable
> earlier point where a router can reach all destinations and do some,
> although not ideally performing, work, as opposed to when it has the best
> path to all destinations.

Perhaps.... Or perhaps you're seeing the output of a transient loop that
was formed during convergence, and is now resolving. Perhaps one thing that
should be added to these tests is checking the TTL of packets received on
the other side of the network. If you know the "right" amount of hops, then
you can determine if you're seeing the output from a real route through the
network, or from a transient or microloop, or something else.

> Here I disagree, if I take a broad look at "routing". Your statement is
> probably true if we limit ourselves to packet forwarding, even though it
> doesn't quite fit systems with a single control plane but multiple
> forwarders.
>
> It is probably reasonable to talk of IP control plane in some isolation,
> if the control information is then used to set up (G)MPLS.

Hmmm.... In the ping sorts of tests, there isn't any way to separate the
two. You have to do either black or whitebox monitoring of some sort along
with the pinging or traffic test to really understand the two together.

> What I am not seeing is conditions of workload and test -- issues such as
> convergence of N routes in topologies where the router under test has M
> or 2M peers.

Yes, well, this was a first cut, and I'm not certain if we want to try to
cover that here, or put it in some separate draft called something like:
"Considerations in load and other factors in benchamarking networks,"
perhaps, or something else. I prefer smaller, more focused drafts, in
general, since it's so hard to get concensus on things any more.... :-)

:-)

Russ

__________________________________
[email-masked] CCIE <>< Grace Alone

---

From: Russ White <[email-masked]>
Date: Fri, 9 Apr 2004 10:28:12 -0400 (Eastern Daylight Time)

Yes....

:-)

Russ

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Fri, 9 Apr 2004 11:20:11 -0400

>inmprecision verses intelligability. :-)

[Comic relief preceding substantive comment]

I'm not sure about comparing the intelligibility to outsiders of:

    I admitted a LOL C/O SOB 2ary COPD. Hx mentions "asthma" but w/o
    FEV1 measurement or beta-agonist reponse. CXR, ABG, chem-7 and
    CBC pending. Pt is in ICU on 10L O2 by nonrebreathing mask, with
    pulse ox. ? to insert ABG catheter. Current TX neb with albuterol
    and ipratropium bromide

and

      In the test scenarios in the methodology discussion that will
       follow this draft, peers send BGP advertisements to the DUT and
       receive DUT-originated advertisements.  We recommend that the
       peering relation be established before tests are begun.  It might
       also be interesting to measure the time required to reach the
       established state.

       This is a protocol-specific definition, not to be confused with
       another frequent usage, which refers to the business/economic
       definition for the exchange of routes without financial
       compensation.

       It is worth noting that a BGP peer, by this definition is
       associated with a BGP peering session, and there may be more than
       one such active session on a router or on a tester.  The peering
       sessions referred to here may exist between various classes of BGP
       routers (see Section 4.2).

:-) Howard <--can speak "doctor" with native fluency

PS--if you REALLY want separate language, I am also fluent in OSI-speak.

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Fri, 9 Apr 2004 11:35:41 -0400

>the network has converged.

In the context of the BGP work, my informal criterion was that the 
DUT should be able to ping every destination for which it receives an 
update, assuming those destinations are physically reachable and 
operating.

>word "converged," maybe (?).

There is a valid separation, I believe, between initial convergence, 
and reconvergence after a route withdrawal.  If the backup path has 
converged before the withdrawal happens, reconvergence is fast. If 
the DUT has to learn about the route, or do a major recomputation, 
reconvergence is slow.

Reconvergence becomes more complex when you move outside the pure 
control plane, and have to deal with issues like cache flush and 
rebuild, obviously implementation specific.

>discuss ways around this particular problem, most likely.

A better statistician than I should speak up here...

>the data afterwords.

I'm not sure I understand "behind". Are you perhaps talking about 
time-synchronized transmitters and receivers?

>You'd need to put the measurement devices on some

And I'm afraid it doesn't lend itself to extrapolation. Given 
real-world data structures both in internal tables and the update 
stream, convergence time probably increases nonlinarily with the 
number of prefixes (to say nothing of policies).

>you just have to watch for them, and realize how they are impacting your

I've been developing some training scenarios for which I'm playing 
with writing assertions about reachability in some formal logic 
notation. Again, this is just whiteboard stuff for now. It would be 
interesting to see if there is a WG that would be interested in it.

>inmprecision verses intelligability. :-)

OK, so an action item is refining black box versus passive.

>network, or from a transient or microloop, or something else.

The TTL check is reasonable.

Best exit vs. closest exit might be more of an IPPM issue, at least 
with BGP. For OSPF, we could, I suppose, specify policy with respect 
to area stubbiness, external type/metric, etc.

>with the pinging or traffic test to really understand the two together.

I'd argue that you do see a way to separate them in the control 
plane. The "ping request" is the arrival of an update, while the 
response is the readvertising of that NLRI.

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Fri, 9 Apr 2004 11:35:41 -0400

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Fri, 9 Apr 2004 11:20:11 -0400

and

---

From: Russ White <[email-masked]>
Date: Wed, 14 Apr 2004 15:50:17 -0400 (Eastern Daylight Time)

> >completed some stage of processing--whether you consider that
> >recalculating the protocol tables, installing the routes in the rib, or
> >even building the forwarding tables as needed--then you could, of
> >course, poll every router in the network (finding some clever mechanism
> >to do so) to discover when the network has converged.
>
> In the context of the BGP work, my informal criterion was that the DUT
> should be able to ping every destination for which it receives an update,
> assuming those destinations are physically reachable and operating.

But reachability doesn't always imply convergence (?). In other words, you
may reachability along some suboptimal path, and the routing protocol could
still be churning away finding the best path through some other direction.
So, that's why it's important to define convergence as "when all the
routers in the network have found (and agree on) the best path through the
network," or "when the destination is reachable."

I'm okay with both definitions, but it's important to define which meaning
you're using when you discuss a specific testing procedure.

> There is a valid separation, I believe, between initial convergence, and
> reconvergence after a route withdrawal.  If the backup path has converged
> before the withdrawal happens, reconvergence is fast. If the DUT has to
> learn about the route, or do a major recomputation, reconvergence is
> slow.
>
> Reconvergence becomes more complex when you move outside the pure control
> plane, and have to deal with issues like cache flush and rebuild,
> obviously implementation specific.

Well, yes, but also: If the primary path has been found, does anyone care
about when the alternate paths are found? Suppose you have a protocol set
up so you can compute a single loop free path, and then put of determining
all other loop free paths (for load sharing/etc) until all possible
destinations have at least one loop free path. Now, I don't know of any
such beast, but it seems possible. Would you care when the load shared
paths were available, or only when the destination is absolutely reachable?
Either way can be valid, it depends on what you are trying to test, I
think.

> A better statistician than I should speak up here...

So, then we're looking for volunteers.... :-)

> time-synchronized transmitters and receivers?

In other words, pair each device in the network with a measurement device
"outside" the network, and the measurement devices would be time
sunchronized, and simply log what they see, for later
collation/evaluation.

> And I'm afraid it doesn't lend itself to extrapolation. Given real-world
> data structures both in internal tables and the update stream,
> convergence time probably increases nonlinarily with the number of
> prefixes (to say nothing of policies).

I was hoping to actually have some testing on this to publish at some
point. I'm still working on it. :-)

> I've been developing some training scenarios for which I'm playing with
> writing assertions about reachability in some formal logic notation.
> Again, this is just whiteboard stuff for now. It would be interesting to
> see if there is a WG that would be interested in it.

I think it could be.

> OK, so an action item is refining black box versus passive.

I think so, yes.... I can add those definitions in this draft, but I wonder
if they wouldn't be better off in some more generic draft (?).

> The TTL check is reasonable.

I think I can add this to the list of things to add to this draft with no
problems.

> Best exit vs. closest exit might be more of an IPPM issue, at least with
> BGP. For OSPF, we could, I suppose, specify policy with respect to area
> stubbiness, external type/metric, etc.

Well, it depends more, I think, on if it's a lab test, or a real test. In a
lab test, I still care about when I find the bext exit, even in a small lab
built bgp test. I'm getting the idea that the two groups focus not on
different tests, but on different places where tests are done--the real
world, and the lab.

This draft was definitely written with the lab in mind....

> I'd argue that you do see a way to separate them in the control plane.
> The "ping request" is the arrival of an update, while the response is the
> readvertising of that NLRI.

I don't know that it's that clean. You can't assume that just because
you've received the advertisement for something, all the microloops have
actually sorted themselves out.

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Wed, 14 Apr 2004 15:50:17 -0400 (Eastern Daylight Time)

I think it could be.

:-)

Russ

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 14 Apr 2004 21:01:26 -0400

>you're using when you discuss a specific testing procedure.

Good point. In single-router convergence, they MAY be close enough, 
when there is a small number of peers, to be roughly equivalent. When 
you get to the scope of an IGP area or of a BGP AS, it may be 
appropriate to measure and report both.

Do distinguish between the cases where the path determination 
algorithm produces a preliminary suboptimal result from a fixed set 
of topological information, and  when the algorithm gets new 
information (advertised/withdrawn route) that causes it to rethink 
either interim or final routes.

>  > >word "converged," maybe (?).

I'd say we don't necessarily care today, but if we start getting into 
some of the newer SPF, etc., algorithms that keep apparently 
suboptimal routes for use in fast restoral without full 
recomputation, this might become a meaningful measurement.

>Either way can be valid, it depends on what you are trying to test, I

Load sharing, of course, can be for fault tolerance, bandwidth, or 
both. It gets interesting indeed when you consider 
more-or-less-parallel-but-not-a-shared-risk-group multiple paths 
where there can be graceful failover within a bundle, and failover to 
another inactive path.

>So, then we're looking for volunteers.... :-)

My distinguished feline advisor looked up at that, yawned, observed 
that his skills are more in licking faces, and went back to sleep. He 
isn't volunteering. A wise cat, Mr. Clark.

>This draft was definitely written with the lab in mind....

Avoiding 800-pound gorilla jokes, I can't help but wonder "how big is 
a lab"?  Single router, even with multiple black box transmitters, is 
still reasonably different from "real world".  AS or area tests may 
start getting fairly large, and there may be a point where lab isn't 
economic.  We then go either to simulation or passive measurements.

>actually sorted themselves out.

Further evidence that more thought about degrees/flavors of 
convergence are important.

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Wed, 14 Apr 2004 21:01:26 -0400

---

From: Russ White <[email-masked]>
Date: Wed, 14 Apr 2004 21:32:28 -0400 (Eastern Daylight Time)

> >But reachability doesn't always imply convergence (?). In other words,
> >you may reachability along some suboptimal path, and the routing
> >protocol could still be churning away finding the best path through some
> >other direction. So, that's why it's important to define convergence as
> >"when all the routers in the network have found (and agree on) the best
> >path through the network," or "when the destination is reachable."
> >
> >I'm okay with both definitions, but it's important to define which
> >meaning you're using when you discuss a specific testing procedure.
>
> Good point. In single-router convergence, they MAY be close enough, when
> there is a small number of peers, to be roughly equivalent. When you get
> to the scope of an IGP area or of a BGP AS, it may be appropriate to
> measure and report both.

Yes, possibly, or possibly not. This becomes even more interesting with BGP
tests. For instance, with four routers, you can set up a simple test where
the IGP converges quite regularly, but pings don't fail at all. It's all a
game of next hops and such, and this sort of thing is particular to BGP, I
think, but may also apply to overlay routing sorts of things in general
(?).

> Do distinguish between the cases where the path determination algorithm
> produces a preliminary suboptimal result from a fixed set of topological
> information, and when the algorithm gets new information
> (advertised/withdrawn route) that causes it to rethink either interim or
> final routes.

Yes....

> some of the newer SPF, etc., algorithms that keep apparently suboptimal
> routes for use in fast restoral without full recomputation, this might
> become a meaningful measurement.

Yes, that's true. Becuase the performance of a two event sequence might be
completely different depending on the spacing of those events, which is
another entire area of research and thought.

> My distinguished feline advisor looked up at that, yawned, observed that
> his skills are more in licking faces, and went back to sleep. He isn't
> volunteering. A wise cat, Mr. Clark.

:-) That doesn't stop the quest for volunteers. I assume we have at least
one good statistician on list that could write such a section.

> Avoiding 800-pound gorilla jokes, I can't help but wonder "how big is a
> lab"?  Single router, even with multiple black box transmitters, is still
> reasonably different from "real world".  AS or area tests may start
> getting fairly large, and there may be a point where lab isn't economic.
> We then go either to simulation or passive measurements.

And this is an interesting question. My own lab is nary 7 routers and a
couple of other boxes. But it lives in the context of several 200+ router
testbeds, and I've seen/worked bigger ones. I suppose that's why I wonder
about these sorts of things, because I know people are actually building
larger and larger testbeds, and using them to try and find what the
performance of a possible design would be in the "real world," as well as
to improve implementations of protocols. Thus, anything that helps define
the types of measurements, and gets people to think about the types of
things you can do here, seems to be of some worth.

> convergence are important.

Yes, exactly. And they are contextual. I'm not certain we can provide a
definition for convergence that will actually cover all situations.

But.... Back on the original question....

I've not heard too many people say if they think this specific draft (the
network benchmarking considerations draft) belongs here or not. I'd like it
to, and to find some volunteers to add sections and clean things up as
needed. The section on statistics, for instance, and other possible things
people think of in this sort of testing.

Thoughts (anyone on list, not just Howard!)?? :-)

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Wed, 14 Apr 2004 21:32:28 -0400 (Eastern Daylight Time)

Yes....

:-)

Russ

---

From: "Robert Holley" <[email-masked]>
Date: Thu, 15 Apr 2004 05:17:42 -0400

> Thoughts (anyone on list, not just Howard!)?? :-)

Obviously there is a question (or subject) that needs further
exploration. Also it appears that the qualified resources to
address that question are located within (if not in the close proximity)
of this group.

Charters can be modified with the appropriate debate
and due process. Working groups come and go according to
need and interests.

I believe the industry is now increasing it's interest
on a larger question for which routing protocol convergence
is a significant corner stone.

As an example, I would submit the Notice of Public
Rule Making (NPRM) recently published by the Federal
Communications Commissions (FCC). Talk about your 800
pound gorillas....

In short, that NPRM is opening a debate on outage reporting
requirements for public communication networks within the US.
An obvious question that is being circulated is "well,
how do you do that for converged and next generation networks".

The FCC has not specifically targeted the IP protocol portion of
converged networks (this time). Probably due to the sensitive
political nature of such a direct question.

But it seems to me that it is only a matter of time. There is
too much at stake and IP is too critical a component to
be given the allowances it has in the past for very much longer.

As a relatively new outsider, I would like to humbly encourage this
work and volunteer to assist where possible.

regards
Bob

---

From: "Robert Holley" <[email-masked]>
Date: Thu, 15 Apr 2004 05:17:42 -0400

regards
Bob

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Thu, 15 Apr 2004 11:17:22 -0400

>is a significant corner stone.

You're talking about an important issue that doesn't really fall into 
the scope of any group -- and it should. I find out more about this 
from some industry policy & law lists like cybertel (Telecom 
Regulation & the Internet, <[email-masked]>  for 
anyone interested) than anywhere else.  Yet that sort of mailing 
list, with a few significant exceptions, is rather limited in 
technical experts.

For those of you who read the NANOG and RIPE lists, there again has 
been a suggestion that NANOG should publish operational best practice 
documents rather than just archive presentations. There are RIPE 
groups that actually do document such things.

I don't have a real answer. There is a twilight zone where things 
start getting out of the traditional scope of the IETF Operations 
Area, but don't really have other places to go. I wish somebody had 
an idea that would catch fire. The work is valuable, but there is no 
scope for it.

This is really off topic, but I wonder if there might be the remotest 
chance of something like an "Internet Operations Task Force" spawned 
by ISOC.

All I can say, Bob, is that I wish I had a better answer. For that 
matter, I wish we could even formulate a better question.

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Thu, 15 Apr 2004 11:17:22 -0400

---

From: "Robert Holley" <[email-masked]>
Date: Thu, 15 Apr 2004 12:42:49 -0400

> the scope of any group -- and it should. I find out more about this 

agreed, I believe the scope is large enough where
it would have to be divided into areas of expertise.
Hence, the IETF, being IP experts, could provide 
technical leadership on parts of the IP portion.
Perhaps the first and foremost of importance would 
be routing protocol convergence (imho).

> list, with a few significant exceptions, is rather limited in 

I've been participating in an industry organization called
the "Network Reliability and Interoperability Council"
NRIC (www.nric.org) for some time now. NRIC is a D.C. based
organization that is chartered by the FCC to investigate
many different issues. One being network reliability and 
outage reports. In fact, NRIC is the organization that oversees
the national outage reporting for the telephony industry today.

> an idea that would catch fire. The work is valuable, but there is no 

I don't think there is a real answer at this time.

But there needs to be a accepted, uniform method developed that can be used
to quantify the reliability of an IP network.

It seems like routing protocol convergence is a good place to 
start.

My concern is that if the IETF does not address the question,
(at least the portion where it has expertise),
then other organizations will attempt as best possible
to provide guidance. Which would be a less than optimal approach.

> chance of something like an "Internet Operations Task Force" spawned 

Well, if that is what it would take....
I am not familar with the flexibility of the ISOC on such matters.

regards
Bob

---

From: "Robert Holley" <[email-masked]>
Date: Thu, 15 Apr 2004 12:42:49 -0400

regards
Bob

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Thu, 15 Apr 2004 13:14:40 -0400

>the national outage reporting for the telephony industry today.

I know of NRIC, but haven't directly worked with them. Maybe I 
should, if they have any staff, as I'm local to DC.

Not wanting to sound like an old Official Standards Organization type 
(correctly, I'm Howard, and I'm an ISOholic who has been sober for 
quite some years), I wonder if there is enough potential interest in 
NRIC that it might send a couple of letters to the IETF (IAB? IESG? 
even IRTF?) and NANOG to suggest cooperation.  In the case of NANOG, 
it would be really more effective to send to the list, as it's fairly 
clear that the Merit people don't see NANOG getting into the document 
business, in spite of volunteers.

>to quantify the reliability of an IP network.

Agreed.

>It seems like routing protocol convergence is a good place to

Certainly one has to start somewhere both on technical and 
organizational terms. My question to you: do you think NRIC would be 
willing at least to get involved to some extent, which could be 
people that belong to both?  I see that involvement as simply 
cooperating to see whether we can start on a joint opinion that a 
need exists.

>to provide guidance. Which would be a less than optimal approach.

This is getting closer and closer as VoIP providers and ILECs start 
facing off against one another, and potentially moving to the Field 
of Honor, dusting off dueling pistols -- I mean, petition the FCC.

>I am not familar with the flexibility of the ISOC on such matters.

I don't know either. AD's, chairs, etc., please chip in, but I think 
the target would be the IAB, possibly through the IESG, possibly not. 
Is there any sentiment for BMWG to draft a memo and start it up the 
IETF food chain?

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Thu, 15 Apr 2004 13:14:40 -0400

Agreed.

---

From: "Robert Holley" <[email-masked]>
Date: Thu, 15 Apr 2004 14:05:53 -0400

> business, in spite of volunteers.

I believe there was some discussion concerning presentations
to NANOG. I'm not sure what the status of that task is
at the moment.

> cooperating to see whether we can start on a joint opinion that a 

Good suggestions. I'll look into it.

regards
Bob

---

From: "Robert Holley" <[email-masked]>
Date: Thu, 15 Apr 2004 14:05:53 -0400

regards
Bob

---

From: Russ White <[email-masked]>
Date: Thu, 15 Apr 2004 19:51:46 -0400 (Eastern Daylight Time)

> My concern is that if the IETF does not address the question, (at least
> the portion where it has expertise), then other organizations will
> attempt as best possible to provide guidance. Which would be a less than
> optimal approach.

I think what we would need here is a specific draft that addresses the
concept of what a down network means... This might be better off as a
collusion between IPPM and BMWG, perhaps, since it involves real world
networks, but it also involves specific measurement points and etc.

> > chance of something like an "Internet Operations Task Force" spawned by
> > ISOC.
>
> Well, if that is what it would take.... I am not familar with the
> flexibility of the ISOC on such matters.

We could discuss this with the AD's, and see what their opinion is.

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Thu, 15 Apr 2004 19:51:46 -0400 (Eastern Daylight Time)

:-)

Russ