Subject: Re: Router Benchmarking Framework and Metric

From: "Howard C. Berkowitz" <[email-masked]>
Date: Tue, 25 Apr 2000 10:33:34 -0400

(apologies to Kevin for having missed this earlier thread).

The metrics proposed seem to focus on router throughput, but not 
convergence and reconvergence.  Recently, I've been working on a set 
of definitions for BGP convergence, which certainly is applicable to 
other routing protocols.

My methodology has been to start with a single router's convergence 
before going to routing-domain-wide or Internet-wide convergence. 
Even there, there definitely appear to be several useful meanings of 
convergence, from initialization, to time to reconverge on a 
previously known less preferred route after a withdraw, to time to 
reconverge on a withdraw and update, to time to begin propagating a 
received route.

Does this fit into the proposed document scope?  Would another 
document be preferable?

Howard Berkowitz
Carrier Packet Solutions
Nortel Networks

(I work remotely.  [email-masked] is far more reliable than 
[email-masked].

(voice (703)998-5819   ESN 451-5819

>There are two documents that are to be developed this year.  We believe
>that the first "Framework for Router Metrics and Benchmarking" is a
>document that MUST be written before any other router documents are
>submitted. This framework document will be format of RFC 2330 "Framework
>for IP Performance Metrics", May 1998.  We also believe that router
>documents can be submitted when most of the "Framework for Router
>Benchmarking" is at completed, there is no need to wait to have this
>published as an RFC before other work can begin.
>
>Cynthia Martin
>Jeff Dunn
>ANC, Inc.
>-----------------------------------------------------------------------
>
>Work Plan for "Framework for Router Metrics and Benchmarking"
>
>I. Introduction
>
>The purpose of this memo is to define a general framework for particular
>metrics to be developed by the Benchmarking Methodology Working Group
>(BMWG) of the Operational Requirements Area. This memo extends existing
>work, specifically RFC 2330 "Framework for IP Performance Metrics", May
>1998.
>
>This document will lay out several criteria for the routing metrics to be
>developed.  These criteria are designed to promote a BMWG effort that will
>maximize an accurate common understanding by Internet users and Internet
>providers of the performance and reliability both of end-to-end paths
>through the Internet and of specific 'IP clouds' that comprise portions of
>those paths.
>
>This document will reference Internet vocabulary, clearly describing
>Internet components such as routers, routing protocols, and router MIB
>element definitions.  Any additional router related vocabulary necessary to
>develop router metrics will be defined in this document. Measurement
>uncertainties and errors will be described, including how they relate to
>the analytical framework shared by many aspects of the Internet engineering
>discipline.
>
>The remainder of the document will provide the definitions of configuration
>and state parameter sets which effect IP packet forwarding performance in
>routers.  Specifically, the following four classes of configuration and
>state parameter sets will be defined:
>1)   static configuration parameters
>2)   dynamic configuration parameters
>3)   static state parameters
>4)   dynamic state parameters
>
>
>II. Framework Document Milestones
>
>April 2000 - First draft of framework document submitted to working group
>
>July 2000 - Rough consensus as to framework
>
>December 2000 - Final thrashing out of document and last call
>
>
>
>
>
>
>Work Plan for "Metrics for IPv4 Router Benchmarking"
>
>I. Introduction
>
>This memo will discuss and define the basic metrics associated with
>performance benchmarking tests and the results of these tests in the
>context of router devices supporting IPv4. Terminology for router and
>routing protocols is already defined in various IETF RFC's and ID's and
>will be not discussed in the memo.
>
>This document assumes that necessary services are available and active to
>provide routing services. This document covers only exterior routing
>protocols, interior routing protocols, and their interaction.  The metrics
>are generic and can be applied to all IETF defined routing protocols for
>IPv4.  It may be necessary to define routing protocol specific metrics in
>separate documents.  This document presents only the metrics associated
>with benchmarking IP performance in routers; therefore, it does not
>represent a total compilation of router test metrics.
>
>II. Objective
>
>The goal of this effort is to produce a set of metrics, from which a
>methodology can be derived, to characterize the effects of a router's
>configuration and state on IP packet forwarding performance as defined in
>RFC 2544 "Benchmarking Methodology for Network Interconnect Devices".
>"Framework for Router Metrics and Benchmarking" ID should be consulted
>before attempting to make use of this document.  Metrics will be defined in
>accordance with the framework specified in RFC 1242 "Benchmarking
>Terminology for Network Interconnect Devices".  Specifically, this effort
>will focus on the effects of the following configuration and states
>parameter sets on IP packet forwarding performance:
>
>1. Static configuration parameters, e.g., route cache vs. total available
>memory size
>
>2. Dynamic configuration parameters, e.g., BGP4
>MinRouteAdvertisementInterval
>
>3. Static states, e.g., response to BGP4 NLRI updates
>
>4. Dynamics states, e.g., response to conflicting BGP4 NLRI updates (route
>flapping)
>
>Metrics will be defined which characterize both the impact on IP packet
>forwarding performance and router response to route updates based on these
>states.  Packet forwarding performance assessment will be based on metrics
>described in RFCs 1242, 2285 and 2761.  The assessment of router response
>will be based on the values of MIB objects described in RFC 1850 and the
>upcoming BGP MIB document.
>
>Specific metrics will include:
>
>1. IP packet: Delay Variation, Error Ratio, Loss Rate, Latency,
>Throughput and many more
>
>2. Router: Interface and Interface Metric Table state, MIB Alignment
>Time, Neighbor and Virtual Neighbor Table state, Route Update Response
>Time and many more
>
>III. Metric Document Milestones
>
>April 2000 - First draft of metric document submitted to working group
>
>July 2000 - Rough consensus as to metric definitions
>
>December 2000 - Final thrashing out of document and last call
>
>IV. Future developments
>
>December 2000 - First draft of methodology document based on rough
>consensus on metrics
>
>March 2001 - Final draft of methodology document and last call