Subject: Comments to draft benchmarking methodology

From: [email-masked] (John Dawson)
Date: Wed, 12 Aug 1998 14:36:48 -0700

My Comments relate to the methodology proposed for Address caching capacity
and address learning rate measurements.

I have two concerns with the proposed methodology.  The first concern is
that the test is designed to only test a single port at a time, and the
second concern is that the learning rate measurement is too focused on
perfect learning characteristics (i.e. the switch is not allowed to miss an
offered address).

The address caching methodology is only valid for a single port.  It does
not consider that each port in a switch may have a combination of separate
and/or pooled address caches.  Additionally, it is not appropriate to run
this test against one port at a time and simply add the individual results
together - this would generate an incorrect result for a switch with a
pooled address cache architecture.  I believe that the methodology must be
extended to test a quantity of ports at the same time (i.e. 12, 16, 24, or
32 ports at once).

The address learning methodology tries to determine the loss-less address
learning rate, rather than the typical address learning rate.  The current
algorithm assumes that even if a single address is not-learned then all
other address that are learned are meaningless.  This has caused problems
in the past, at my company, interpreting results for switches that employ
adaptive algorithms (algorithms that can fail to learn every address in a
perfect fashion).  I would suggest that a more appropriate address learning
methodology would test a quantity of ports at the same time and be
structured in a fashion similar to forwarding rates - in this case
documenting learning rate at a specific offered load.

For instance if a switch is offered 1000 packets in 1 second and learns 900
of these addresses, then the learning rate is 900 packets per second (not
test inconclusive invoke a binary search).

Extending the learning rate test method to a multi-port scenario could be
accomplished using an algorithm similar to the following:

Step 1:  Each port populates a single known address.
Step 2:  Each port then offers N addresses per port destined to 
         some combination of ports learned in step 1.  The time interval
         for which the N addresses are offered defines the offered load.
Step 3:  Each port is set up to count the number of packets 
         that are destined to one of its neighboring ports (not itself).  
Step 4:  Addresses in step 1 are used to transmit back to N addresses
         for verification purposeses.
Step 5:  Packets counted on an incorrect port are considered flooded 
         (rather than mis-delivered).  Since each port monitors flooding
         associated with a different port, the total flooded would 
         be equal to the sum of packets counted on all ports.
Step 6:  Calculated address learning rate = 
         (addresses offered - addresses flooded) / time_interval_offered.
Step 7:  Express learning rate as a function of offered load.
Step 8:  Repeat the test at a variety of loads.

Do other people share my concerns with respect to multi-port test
methodology vs. single-port test methodology?

John Dawson

---

From: "Bob Mandeville" <[email-masked]>
Date: Thu, 13 Aug 1998 20:49:56 +0200

All,
It is important to include gigabit Ethernet in the methodology draft.
Fortunately this should be straightforward. It does raise another issue
however which is the distinction between testing in half and full duplex
modes. Perhaps there is a need to refer to each mode in a very explicit
manner whenever the distinction applies.
As for referring to real world traffic patterns when describing benchmarking
test methodologies - I do not think that it is or ever has been the role of
the bmwg to determine what specific traffic patterns correspond to real
world traffic. It strikes me as an almost impossible task. Benchmarking has
to do with fully exercizing the functions of DUTs/SUTs at the layers they
are meant to run on. Fully meshed tests as described do just that for the
forwarding function of switching devices. In RFC 2285 and the to be done
sections of the intial draft there is plenty of room for other test patterns
like one to many, many to one and multiple simultaneous instances thereof.
Any help on such is welcome.
Bob

---

From: "Bob Mandeville" <[email-masked]>
Date: Thu, 13 Aug 1998 20:33:05 +0200

John,
I take your first comment on the address learning capacity test - the
methodology should extend to testing learning on more than a single port. In
fact I am tempted to sugest a 2 port minimum since this would allow the user
to distinguish devices with per port or shared caches.
The second point - how many 'misses' should the test tolerate when
determining the address learning capacity is a more difficult one. How about
the following suggestion: run an algorithm which determines how many
'misses' occur before an address is learned and make it a MUST to report the
number of such misses. This may be pretty tricky to implement.
Bob

---

From: [email-masked] (John Dawson)
Date: Thu, 13 Aug 1998 12:36:57 -0700

Hi Bob,

>  On the subject of how many misses should be tolerated in an address
capacity test.

Since the test is trying to determine address capacity and not learning
rates, does it really matter how many times an address is missed?  Why not
just send the addresses over and over allowing the switch some reasonable
number of learning opportunities for each address over some reasonable time
period.  After a sufficient number of passes through the learning phase,
then execute the verification process.

John
--------------------------------------

---

From: "Bob Mandeville" <[email-masked]>
Date: Thu, 13 Aug 1998 22:40:19 +0200

Hi John,
I suppose we agree that there is some kind of trade-off between address
learning capacity and rate. A rate test with only 2 addresses to be learned
would not be very interesting, nor a capacity test which tolerated hundreds
of'misses. We could avoid arbitrary choices of what learning rate is
reasonable in the capacity test and what number of addresses should be
learned in the rate test in two different ways: plotting out all values in
what would probably be an intolerably long test or try a few reasonable
values but report at which fixed rate or capacity the capacity or rate tests
were run. So what I am suggesting is doing both: your reasonable dose
together with a report of the fixed values used.
Bob