Subject: [ippm] New version of draft-ietf-ippm-reporting

From: Martin Swany <[email-masked]>
Date: Mon, 10 Mar 2008 11:50:48 -0400

Hi all,

I'm sorry for the short lead time on this, but there is a new
version of the "Reporting IP Performance Metrics to Users"
draft.  It will soon be officially submitted.  In the meantime, it
is here:

http://www.cis.udel.edu/~swany/ippm/draft-ietf-ippm-reporting-02.txt

martin
_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm

---

From: "Stanislav Shalunov" <[email-masked]>
Date: Tue, 11 Mar 2008 14:44:32 -0800

Further optimization -- only do alpha and alpha+epsilon.

The epsilon value should not matter very much for continuous
underlying PDF (I expect this is a prereq for the result anyway), so
using something like epsilon = C/sample_size where C is a small
integer value like 5 should work, right?  This would require to know
sample_size in advance, which contradicts the "in one pass"
requirement.  Hmm.  OK, can bound sample_size at the top -- never make
epsilon less than 1%, for example.  Samples that are smaller than
100/C can just be kept in memory and considered part of the fixed
memory cost.  Yeah.

This would double the computation cost for a large sample.  Is it
worth the ability to flag statistically suspect values?  For some
applications, definitely.  I would still make this a MAY -- it's not
an absolute necessity and it allows you to make your implementation of
the whole error estimate a differentiating feature.

-- Stas

P.S. Please CC any responses from the IPPM list to me.  Thanks.

On 3/11/08, David Scheibelhut <[email-masked]> wrote:
> Stas,
>
>  I will resend to [email-masked] with your response.
>
>  Your comment on computational complexity is correct.  One just needs an
>  algorithm for calculating epsilon.
>
>  As far as the 99.999 percentile goes, at least it has lower standard
>  error than the 100th percentile.  (It would be interesting to produce a
>  histogram of 100 percentile values and see if it matches an extreme
>  value distribution).
>
>  Dave
>
>
>  Stanislav Shalunov wrote:
>  > David,
>  >
>  > Great points.  Thank you for writing, and doing the work and digging
>  > up the references.  We don't and should not rely on stddev of delay
>  > being finite, but the percentile-based estimate is the right thing.
>  >
>  > This message should be resent to [email-masked] also.  Would it be OK
>  > with you?  (Along with, perhaps, this response.)  If it's OK, I hope
>  > Martin, who is on the list, might resend?
>  >
>  > Estimating the PDF is only roughly twice as computationally expensive
>  > as computing the percentile, no?  Just compute percentiles alpha +/-
>  > espison and divide?
>  >
>  > Doing some sort of sanity check is useful.  People often will insist
>  > on producing statistics that can't be derived (see my own reports at
>  > http://netflow.internet2.edu/weekly/ -- that 99.999 percentile is
>  > garbage, but I spent enough time explaining this to the management who
>  > really wanted a bigger number at the end of the table that I just put
>  > it in).  But yeah, it'd be useful to flag results that are suspect.
>  > This way, the number of reported metrics does not increase in the
>  > general case.  It could flag the value as suspect if the expected
>  > relative error is more than 10%, for example.
>  >
>  > I agree about the needless difficulty of understanding C code.  I
>  > still think that advantages of C outweigh this disadvantage: C is
>  > known to every reader who knows a programming language; fast; likely
>  > to be the actual implementation language; not likely to become the
>  > focus of discussion.  Put R code in, and readers will become
>  > distracted by "weird language" or think that the reporting draft is
>  > only for those who know/use/understand R or statistics.  Yet its whole
>  > point is to make good statistics easily reportable by every programmer
>  > who is working on an applicable problem.  Have a game that reports
>  > "ping" to the user?  We can tell you how to compute that so that it
>  > doesn't jump all over, so that "ping" from one game is the same as
>  > "ping" in another, and it's easy -- just grab this code.  Replace C by
>  > R, and the game programmer will be put off.
>  >
>  > -- Stas
>  >
>  > On 3/11/08, David Scheibelhut <[email-masked]> wrote:
>  >> All,
>  >>
>  >>  I found the previous draft quite helpful in the development of
>  >>  requirements and tests for my own work.  The focus on robust and
>  >>  meaningful metrics cuts through a lot of clutter.  The only area which
>  >>  caused confusion is the topic of errors.
>  >>
>  >>  Most people, when presented with a statistic such as the mean, median,
>  >>  or 90th percentile, are inclined to presume that these metrics are
>  >>  exact.  But all of these metrics are based on statistical sampling and
>  >>  thus are thus subject to sampling error.  The owamp program indeed
>  >>  reports an error metric, but it is the error associated with the clocks
>  >>  in the two hosts and has nothing to do with sampling error.  This clock
>  >>  error is reported by the owamp program as being the error associated
>  >>  with its other metrics.  But its not called clock error, just error.
>  >>  Its left as an exercise for the user to determine exactly what sort of
>  >>  error is being reported.  After concluding that it could not possibly be
>  >>  sampling error, I read the code.
>  >>
>  >>  I think, at minimum, all error should be labeled according to its type.
>  >>   Tools should, by default, include enough samples to create reasonably
>  >>  bounded error.  Tools should flag as suspect metrics with high error.
>  >>
>  >>  Any introductory statistics course will teach that the standard error of
>  >>  the mean, SEM, is the standard deviation of the sample divided by the
>  >>  square root of the sample size:
>  >>
>  >>         SEM = s / sqrt(n)
>  >>
>  >>  But having expressed a preference for the median, we are obligated to
>  >>  report its standard error.  It can be shown [W. Edwards Deming, Some
>  >>  Theory of Sampling, p434] that
>  >>
>  >>  Given percentile p such that 0 < p < 1
>  >>  Given n samples
>  >>  Given xp, the estimated value of the sample for percentile p
>  >>  Given yp, the estimated value of the density function for the sample at
>  >>  percentile p (the estimated PDF(xp))
>  >>  then the standard error of the estimate for xp is
>  >>
>  >>         SEP = sqrt (p * (1 - p) / n) / yp
>  >>
>  >>  This formula requires us to estimate the value of the sample's histogram
>  >>  (its density) at the desired percentile.  Unfortunately, such a
>  >>  calculation is much more complicated than just deriving the percentile's
>  >>  value.  Yet estimating the standard error is the only way to be
>  >>  confident with its value.  Furthermore, use of this formula illustrates
>  >>  the huge number of samples necessary for a good estimate of higher
>  >>  percentiles.  On request, the owamp tool will happily report the 99%
>  >>  value based on just a few samples, irregardless of whether the reported
>  >>  value has much precision.
>  >>
>  >>  ======
>  >>
>  >>  I know that C is has a long history of appearing in RFCs.  But perhaps a
>  >>  statistical language would make more sense for producing sample metric
>  >>  code.  The code in appendix A has more data manipulation than
>  >>  statistics.  From open source consider R (syntax close to C), Octave
>  >>  (syntax close to MATLIB), J (APL!) or Lisp-Stat (!!!).  R would be a
>  >>  good choice.
>  >>
>  >>  David Scheibelhut
>  >>
>  >>  Alcatel-Lucent
>  >>  Wireless Performance
>  >>
>  >
>  >
>

-- 
Stanislav Shalunov
http://shlang.com/

---

From: David Scheibelhut <[email-masked]>
Date: Wed, 12 Mar 2008 12:45:14 -0500

All,

Most people, when presented with a statistic such as the mean, median, 
or 90th percentile, are inclined to presume that these metrics are 
exact.  But all of these metrics are based on statistical sampling and 
thus are thus subject to sampling error.  The owamp program indeed 
reports an error metric, but it is the error associated with the clocks 
in the two hosts and has nothing to do with sampling error.  This clock 
error is reported by the owamp program as being the error associated 
with its other metrics.  But its not called clock error, just error. Its 
left as an exercise for the user to determine exactly what sort of error 
is being reported.  After concluding that it could not possibly be 
sampling error, I read the code.

     SEM = s / sqrt(n)

======

David Scheibelhut

Alcatel-Lucent
Wireless Performance

=====

Response from [email-masked]

> as computing the percentile, no?  Just compute percentiles alpha ±
> espison and divide?
>