Subject: App Performance, Statistics, etc.

From: Nevil Brownlee <[email-masked]>
Date: Tue, 21 Dec 1999 17:05:43 +1300 (New Zealand Daylight Time)

Hello all:

There has been interesting discussion on the RMON list about Application
Performance Measurement.  Essentially this requires measuring the time
between two packets on the wire, which requires ways to specify the two
packets, and to match them up. 

Something which hasn't been discussed much is 'how do we collect and
present the resulting measurements?' The notion of 'statistics' has been
mentioned, and various descriptive measures - min, max, mean, standard
deviation - have been mentioned.  Such statistics provide a way of
describing in simple numbers an underlying distribution, i.e. the
distribution from which the observed times are drawn. 

One needs to remember, however, that such measures often rely on
assumptions about the underlying distribution, e.g. standard deviation
commonly assumes a normal distribution, or at least one which has a
finite variance.  This may not be the case for our response-time
distributions. 

To get round this one can use 'non-parametric' statistics, which are
constructed so as not to rely on the distribution.  The commonest
examples are percentiles.  It would be very convenient to collect a
specified set of percentiles in a MIB, say 5, 25, 50, 75 and 95 percent,
but the usual algorithms for computing percentiles require one to build
the distribution, sort it and search it! 

If one has to collect a distribution, it makes sense to do it by
specifying an array of 'buckets' to hold it.  The RTFM Working Group has
proposed a straightforward way to specify distributions; this is
documented in RFC 2724.  I have implemented distrbutions like this in
NeTraMet, handling them in SNMP as a BER-encoded sequence of counters.
This allows one to retrieve distributions at regular intervals, then
subtract them so as to produce a series of distributions for each of the
intervals. 

I suggest that - unless we can devise a way of computing percentiles 
directly - RTFM-style distributions offer the cleanest way to gather 
statistical data for RMON performance measurements.

Cheers, Nevil

+---------------------------------------------------------------------+
| Nevil Brownlee                     Director, Technology Development |
| Phone: +64 9 373 7599 x8941        ITSS, The University of Auckland |
|   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand |
+---------------------------------------------------------------------P