Subject: RE: [bmwg] Is the BMWG a proper home for this I-D?

From: Jim McQuaid <[email-masked]>
Date: Wed, 7 Apr 2004 10:56:34 -0000

-----Original Message-----
From: Howard C. Berkowitz [mailto:[email-masked]]
Sent: Tuesday, April 06, 2004 7:43 PM
To: [email-masked]
Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

>Supporters of the BMWG:
>
>A [long] while back, Russ White approached the BMWG with an 
>individually submitted I-D
>that presented considerations when measuring network convergence:
>
>http://www.ietf.org/internet-drafts/draft-white-network-benchmark-00.txt
>
>The draft seeks to highlight: ...considerations that testers should 
>be aware of when
>attempting to measure network convergence using various methods. 
>(Adapted from the I-D's
>abstract.)
>
>Not being a pure terminology or methodological statement, or 
>constraining itself to
>laboratory evaluations, the topic might not be a perfect fit as BMWG 
>output.  On the
>other hand, the community might benefit from memo on this topic.
>
>Please offer your comments in two dimensions: a) does the topic 
>merit future effort, and
>b) is the BMWG the best home for this effort (as opposed to another 
>WG or progressing the
>work as an Individually submitted I-D)?  (You might have to read the 
>WG Charter to determine
>part B. :-)

I'm not quite sure that I'm ready to answer in the specific 
dimensions, but I can make some comments. I'll add a dimension that I 
have always believed that it would be highly desirable for the IETF 
to generate more frameworks, tutorials, etc., that help introduce 
newbies to a technology, and also help control the salesdroids from 
running amuck.

>
>
>Network Working Group                                           R. White
>Internet Draft                                                 V. Manral
>Expiration Date: August 2003                                    R. Adams
>File Name: draft-white-network-benchmark-00.txt            February 2003
>
>   Considerations in Benchmarking Routing Protocol Network Convergence
>                   draft-white-network-benchmark-00.txt
>
>1. Motivation
>
>    As the ability to benchmark components within a network appears to be
>    coming under greater scrutiny, and specifications are being written
>    to standardize ways to measure the performance of individual
>    components within given frameworks, the next level of benchmarking
>    has not been approached, that of measuring the perfomance of
>    networks. But what is meant when we say the performance of a network,
>    from the perspective of routing protocols? Various tests have been
>    used in the past to measure the convergence of a network, some of
>    which actually measure completely different things than others.
>
>    It's important to attempt to examine the measurement of network
>    convergence in a way that exposes these differences, and helps
>    vendors, end users, and those in the research community have some
>    common ground when discussing network convergence.
>
>
>2. A Problem of Definitions
>
>    As we examine the issues and concepts surrounding the measurement of
>    network performance in terms of convergence, we find that most of the
>    basic problems we face surround defining the terms in use. For
>    instance, what is convergence, exactly? What is a network? In the
>    following sections, we discuss each of these concepts, and attempt to
>    address each one.
>
>
>2.1. Networks
>
>    In its most nominal form, a network is composed of a group of devices
>    interconnected in some way, which send data over these
>    interconnections for various purposes. But, when we discuss the
>    concept of routing protocol convergence within a network, the
>    definition needs to be more precise. For instance, since hosts do
>    not, generally, participate in routing, should they be considered a
>    part of the network when benchmarking the performance of a routing
>    protocol?  The obvious answer appears to be a resounding no, but, in
>    some possible tests types, hosts which do not participate in routing
>    play a large part in the test itself.
>
>    When considering tests in which hosts participate as traffic or route
>    generators, then, we must consider the impact these hosts have on the
>    test results, although we may not consider them a part of the network
>    we are measuring the performance of.
>
>
>2.2. Convergence
>
>    Convergence is probably one of the hardest words in networking to
>    define.

:-) you mean it DOESN'T mean running voice, data, and video on the 
same network?

>Just about everyone who has worked on networks for a period
>    of time knows what it means, but no-one can explain it sufficently to
>    someone who doesn't understand how a network works for it to be
>    understood. In fact, this is because there are several different
>    meanings attributed to convergence, and which meaning is intended
>    depends on the context in which the word is set. Convergence can
>    mean:
>
>
>    o    The time at which all the routing protocol processes running on
>         devices which participate in routing in the network agree on the
>         best path to each reachable destination in the network.
>
>    o    The time at which the best path to each reachable destination in
>         the network has been loaded into some local table which may then
>         be used to forward packets (the routing information base, or
>         RIB).
>
>    o    The time at which each router in the network has built the
>         tables necessary to actually forward packets through the net-
>         work, so that a packet transmitted from one part of the network
>         would actually reach any given reachable destination within the
>         network.
>
>    For instance, on a Cisco router, show ip ospf stats would allow the
>    tester to see the time of the last completed SPF, show ip route would
>    allow the tester to see what routes are installed in the RIB, and
>    show ip cef would allow the tester to see the forwarding information
>    which has been built from the RIB. Each test designed to measure the
>    performance of routing protocols within a network must determine
>    which type of convergence is being measured, if that measurement is
>    acceptable to the information being gathered, and which test will
>    actually measure the desired type of convergence.

Not to criticize, because we consciously restricted our scope to 
single router in draft-ietf-bmwg-conterm-05.txt, leaving single-AS as 
a future and Internet-wide as outside our scope, is this definition 
not specific to single routers?

Should the paper include any definitions that apply to groups of 
routers?  Can the convergence of a group of routers, perhaps under 
the constraint that they start from a known state, receive some set 
of updates, then run to convergence, be described adequately in terms 
of the convergence of the last router to meet single-router 
convergence criteria?

>
>3. Polling Devices in a Network

By "the network", are you now suggesting a routing domain rather than 
a single router?

>
>    One common way to measure network convergence is to poll the devices
>    in the network, using some command suppplied within the routing
>    software, to determine when particular events have occured, or par-
>    ticular pieces of information have reached all the routers in the
>    network. Polling elimiates the need for the clock of each device
>    within the network to be synchronized for the test to have meaningful
>    results. However, there are some issues with the rate of polling dev-
>    ices within the network which need to be addressed in any test which
>    polls devices for this information; the first is the rate at which
>    polling takes place.
>
>    If, in a test, you are attempting to measure some parameter to within
>    one second of its occurance, then you would need to poll at a rate
>    much higher than once per second.
>
>     test starts here
>      |
>      |                   event occurs here
>      |                   |
>      v                   v
>    -+----------+----------+----------+---
>     ^          ^          ^          ^
>     |          |          |          |
>     0 seconds  1 second   2 seconds  3 seconds
>
>    For instance, in this time line, suppose a polling event is set up
>    which takes place every second. An event is started just after some
>    polling event takes place, but the polling process doesn't recognize
>    the test as starting until the 1 second poll. An event occurs just
>    before the 2 second poll, and the polling process detects this at the
>    2 second poll. The polling process would indicate that from the time
>    the event started until the time the event has finished, one second
>    has elapsed. In reality, closer to two seconds has elapsed.
>
>    The interval of the polling process can be reduced until the measure-
>    ment is felt to be accurate, but it should be at least half of the
>    desired accuracy. Common practice actually shows that it should be
>    about one tenth of the desired accuracy.
>
>    A second consideration when polling for network events is the prefor-
>    mance of the device running the polling process. If the process can-
>    not poll each device at the scheduled interval, or the polling is
>    "jittered," the time between each actual poll varies by some amount,
>    the accuracy of the tests will be called into question. The amount of
>    jitter introduced by the polling device, and the rate at which the
>    device can effectively poll, should be measured in some way, and this
>    measurement should be taken into account when designing tests which
>    rely on polling.

But are there not cases where some jittering is desirable to avoid 
unexpected weak synchronization effects (assuming the measurement 
itself has some Heisenbergian flavor), is it not wise to lengthen the 
test interval, or do multiple measurement runs with different seeds 
to a pseudorandom number generator, so the effect of jitter nulls out 
statistically?

>    Finally, when polling devices to determine when a network event
>    occurs, issues with serialization must be considered. Most devices
>    which would be used for polling will not be able to poll several dev-
>    ices within the network at once, and will thus serialize the polling
>    of devices.
>
>     p1   p3  p5  p7  p9
>      | p2| p4| p6| p8| p10
>      | | | | | | | | | |
>      v v v v v v v v v v
>    -+----------+----------+----------+---
>     ^          ^          ^          ^
>     |          |          |          |
>     0 seconds  1 second   2 seconds  3 seconds
>
>    Suppose, for instance, that a single device is polling ten devices in
>    the network. If it can poll five devices per second, it will take a
>    full two seconds for it to detect any event on all ten devices, giv-
>    ing an effective accuracy of about four seconds. The amount of time
>    required for a polling device to serialize through all the devices it
>    is polling needs to be considered when polling a very large number of
>    devices.

Or if there is a point where realistic measurement requires multiple 
polling devices.

>
>
>4. Tests to Measure Routing Protocols Convergence
>
>    In this section, we will outline some of the various tests which have
>    been used in the past to measure routing protocols convergence within
>    a network, and discuss some aspects of these tests.

It occurs to me that there is something of value here that probably 
isn't benchmarking, but certainly is relevant to the Operations Area. 
For want of a better term, I'll call it walking the reachability 
tables in a domain.

I'm currently doing some work with automated evaluation of student 
configurations, and part of that involves generating a meaningful 
script to check the control plane on multiple routers, at the point 
where a step of a scenario is complete. We ask ourselves the question 
of whether it is truly necessary to do the equivalent of 
ping-every-nonlocal-interface from each router in the domain, or if 
there is some meaningful way to define a smaller and more practical 
subset.

>
>
>4.1. Determining When Each Device has Received Information About All
>    Reachable Destinations
>
>    In link state protocols, information is flooded throughout the net-
>    work; discovering when each router in the network has received this
>    information is an important consideration in network convergence.
>    Slower flooding times will, of course, mean slower network conver-
>    gence overall, thus flooding performance directly impacts overall
>    routing protocols performance in the network.
>
>    There are three methods which can be used to determine when the
>    flooding of information has been completed.

A clarification here:  as I read this, I have a sense that you think of
black box testing as done by external stimulus/response devices, while
external packet monitoring is done by inference from live traffic streams.
Another way of putting this is that black box is active while monitoring
is passive.

Do I misconstrue what you mean?

>
>
>4.1.1. Black Box Polling
>
>[snip]
>4.1.2. White Box Output
>
>[snip]
>
>4.1.3. External Packet Monitoring

[snip]

>
>4.2. Determining When Each Device has Finished Finding the Best Path to
>    Each Reachable Destination
>
>    This is, probably, the most difficult measurement to take in a net-
>    work, since there are no known black box ways of determining when a
>    device has finished computing the best path to each destination in
>    the network. The only possible way of measuring this time is to use
>    output from the devices in the network to provide this information.
>
>    It's possible to poll each device periodically, examining output pro-
>    vided by the devices, to determine when each device has calculated
>    the best path to each destination in the network. This method is sub-
>    ject to the limitations described in the section on polling devices,
>    above.
>
>    It's also possible to rely on some event driven output of each device
>    in the network. For this to yield accurate results, the time clocks
>    of all the devices in the network must be closely synchronized.

While this may be "ideal" convergence, it does imply what I'll 
loosely call best exit routing policy.  There may be a perfectly 
reasonable earlier point where a router can reach all destinations 
and do some, although not ideally performing, work, as opposed to 
when it has the best path to all destinations.

[snip]
>
>
>4.3.1. The Various Elements of Performance Cannot Be Seperated
>
>    Using this sort of testing, there is no way to seperate the perfor-
>    mance of a routing protocol from the performance the interaction
>    between the routing protocol and the forwarding engine, nor from the
>    performance of the forwarding engine itself.

Here I disagree, if I take a broad look at "routing". Your statement 
is probably true if we limit ourselves to packet forwarding, even 
though it doesn't quite fit systems with a single control plane but 
multiple forwarders.

It is probably reasonable to talk of IP control plane in some 
isolation, if the control information is then used to set up (G)MPLS.

What I am not seeing is conditions of workload and test -- issues 
such as convergence of N routes in topologies where the router under 
test has M or 2M peers.

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Jim McQuaid <[email-masked]>
Date: Wed, 7 Apr 2004 10:56:34 -0000

-----Original Message-----
From: Howard C. Berkowitz [mailto:[email-masked]]
Sent: Tuesday, April 06, 2004 7:43 PM
To: [email-masked]
Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

[snip]

[snip]

---

From: Jim McQuaid <[email-masked]>
Date: Wed, 7 Apr 2004 10:56:35 -0000

<sorry for the blank reply>

At the time the IPPM was "spun off" of the BMWG, the general understanding
was that BMWG would look at "things" more or less in isolation, although
that concept is somewhat elastic.  IPPM would look at end to end network
kinds of performance issues.

It seems clear to me that convergence is a performance characteristic of
complete networks and primarily an end to end or at least, core element to
core element characteristic.

My thought would be that this is interesting and useful work but not
necessarily for BMWG.

Jim McQuaid

-----Original Message-----
From: Howard C. Berkowitz [mailto:[email-masked]]
Sent: Tuesday, April 06, 2004 7:43 PM
To: [email-masked]
Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

[snip]

[snip]

---

From: [email-masked]
Date: Wed, 14 Apr 2004 18:28:31 -0400

Both of these proposed definitions are incomplete.  Convergence does not
occur until the traffic is correctly routed out the correct interface, which
means in today's routers the FIB must be updated in an ASIC at the line
card.  Remember - SP customers do not observe route table changes due to
cvonvergence, they observe traffic loss.

Scott

-----Original Message-----
From: Russ White [mailto:[email-masked]]
Sent: Wednesday, April 14, 2004 3:50 PM
To: Howard C. Berkowitz
Cc: [email-masked]
Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

But reachability doesn't always imply convergence (?). In other words, you
may reachability along some suboptimal path, and the routing protocol could
still be churning away finding the best path through some other direction.
So, that's why it's important to define convergence as "when all the
routers in the network have found (and agree on) the best path through the
network," or "when the destination is reachable."

I'm okay with both definitions, but it's important to define which meaning
you're using when you discuss a specific testing procedure.

---

From: [email-masked]
Date: Wed, 14 Apr 2004 18:39:32 -0400

Which would you consider the "lab test" - Convergence using Control Plane or
Convergence using Data Plane?  I think it is much easier to create a black
box test to measure convergence based upon traffic flow measured in data
plane.   No "dissecting" (White Box) required.

Scott

-----Original Message-----
From: Russ White [mailto:[email-masked]]
Sent: Wednesday, April 14, 2004 6:34 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [bmwg] Is the BMWG a proper home for this I-D?

> occur until the traffic is correctly routed out the correct interface,
> which means in today's routers the FIB must be updated in an ASIC at the
> line card.  Remember - SP customers do not observe route table changes
> due to cvonvergence, they observe traffic loss.

Well, that's the difference between a benchmark in a lab and in the real
world, I'd think. In a lab, when evaluating a particular implementation, I
can care very much more about actual total rp convergence than about
dropped packets. You have dissect a frog as well as observe it to
understand it, you can't just ask "how far does it jump," or "how fast."

IMHO, the concept of convergence is very flexible, and we should define
within the context of the document at hand, rather than trying to nail down
a specific definition of the word.

:-)

Russ

__________________________________
[email-masked] CCIE <>< Grace Alone

---

From: [email-masked]
Date: Thu, 15 Apr 2004 08:16:06 -0400

There is currently a proposed BMWG work item for benchmarking MPLS Fast
Reroute.  

Scott

-----Original Message-----
From: Howard C. Berkowitz [mailto:[email-masked]]
Sent: Wednesday, April 14, 2004 8:50 PM
To: [email-masked]
Subject: RE: [bmwg] Is the BMWG a proper home for this I-D?

At 6:28 PM -0400 4/14/04, [email-masked] wrote:
there are some interesting further 
complications introduced by MPLS.

---

From: [email-masked]
Date: Thu, 15 Apr 2004 08:10:13 -0400

I think this is "interesting work".  When you make mention of a test bed
with 200 nodes then I seriously think that this work is inappropriate for
the BMWG.  One important criteris for BMWG work items is reproducibility -
so we would need to agree on # of nodes and their interconnection if we were
to proceed.

Scott
-----Original Message-----
From: Russ White [mailto:[email-masked]]
Sent: Wednesday, April 14, 2004 9:32 PM
To: Howard C. Berkowitz
Cc: [email-masked]
Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

But.... Back on the original question....

I've not heard too many people say if they think this specific draft (the
network benchmarking considerations draft) belongs here or not. I'd like it
to, and to find some volunteers to add sections and clean things up as
needed. The section on statistics, for instance, and other possible things
people think of in this sort of testing.

Thoughts (anyone on list, not just Howard!)?? :-)

:-)

Russ

---

From: [email-masked]
Date: Thu, 15 Apr 2004 08:16:06 -0400

Scott

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Thu, 15 Apr 2004 10:18:45 -0400

At 8:12 AM -0400 4/15/04, [email-masked] wrote:
>Howard,
>
>How about for Service Providers - the users of the routers?
>
>Scott

Are you suggesting the service providers will develop the testing 
techniques? In the past, this might have been true. Right now, it 
seems as if the only people willing to invest in development are 
firms that can see an ROI on what they develop -- and that means that 
someone benefits from a measurement product.

My reference to vendors was that I could see those types of 
organizations making the investment to develop the test tools, which, 
hypothetically, the SPs would buy.

There is a strategic benefit to router vendors participating in tool 
development, so there is some quantitative basis for SPs to judge 
their products.

There is also a benefit to router vendor engineers, if not router 
salespeople. I know our motivation, in starting on the BGP 
convergence effort, was to calm down salespeople who kept 
interrupting development with demands for meaningless numbers that 
would prove their product was better than the competition's.  The 
number itself became the objective in selling to the service provider 
-- if a router could do more of something than another, it seemed 
that the service provider automatically assumed it was better.

The specific case that caused me to start the project, while I was at 
Nortel, was a salesman charging in to say that some SP customer -- I 
honestly don't remember who it was -- had come back to say "Juniper 
says it can do 250 BGP peers -- we have to do more to be competitive."

Of course, there was no information on the workload emanating from 
any of those peers (were they customer BGP routers that just needed a 
default?), or what the customer saw about the economic value of doing 
more peers -- would they pay twice as much for a machine optimized to 
have 500 peers?  I found colleagues in other vendor organizations 
equally frustrated about the same problem, and we at least agreed 
that it was useful to have a set of common parameters, rather than 
constantly dropping direct support and development to measure an 
essentially random parameter.

Maybe I'm especially cynical today, from being unemployed to 
underemployed for far too long a time. I've done a lot of the BMWG 
work because I cared about it, not because anyone was paying for my 
time.

>Sent: Wednesday, April 14, 2004 8:51 PM
>To: [email-masked]
>The former is nice for test engine vendors and the latter for
>protocol analyzer vendors?
>

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Thu, 15 Apr 2004 10:18:45 -0400