Subject: Re: [ippm] Comparism of metrics

From: Henk Uijterwaal <[email-masked]>
Date: Tue, 13 May 2008 15:06:52 +0200

Hi Rudiger,

> there has been no discussion on the list since while and I assume 
> comparing metrics still is an open issue.

Yes.

> A point I was concerned 
> about is how to compare probes of different probe sizes (say one 
> of 100 000 probes measured with system 1 as compared to one of 
> 10 000 probes measured by system 2 during the same time interval 
> T). A possibility is to create 10 probe sets of 10 000 probes of 
> the first set and then compare the statistical properties of 
> samples of the same size only. One obviously has to pick the 10 
> probe sets with a reasonable function, as each set must be 
> represenative for the entire time interval T.  

Sorry, this got stuck in my mailbox, mainly because I don't quite
understand what the problem is here.  I still don't :-) so I'm
probably missing something...

Now, suppose I measure something with probes.  I get 10,000 events,
then bin them to get an idea of the distribution.  This gives me
a median and percentiles.  If I then repeat this by sampling 100,000
times, then I simply expect 10x as many events in each bin, but I
don't expect the distribution and thus the median to change.   In
other words, my curves will be a bit smoother, but the overall
result won't change much.

But this is probably not what you meant :-)

Henk
-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

Is one of the choices leaving the office open?
                                       Alan Greenspan on the next elections

_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm

---

From: "Geib, Ruediger" <[email-masked]>
Date: Wed, 14 May 2008 10:10:58 +0200

Hi Henk, 

as you mention, if the samples aren't of the same size, then you

| don't expect the distribution and thus the median to [basically] 
| change. In other words, my curves will be a bit smoother, but the 
| overall result won't change much.

The last sentence adresses the point, that there is a deviation - 
so the metric comparism standard would have to define 
how to compare samples of different sizes (I guess statisticians 
can provide literature on this). If all sample sizes are the same 
and if the sample function is the same, then you directly can 
compare results.

In other words: samples of different size produce results with 
different errors. This error should be visible in an evaluation to 
indicate, whether results are compatible or not.

I've done some of these comparisms with regard to packet loss, 
starting from a CBR probe distance of 200 ms and ending at a probe 
distance of one per minute, always using the same basic set of 
measurement data. The experiment had the aim to figure out, which 
minimum sample size is required to measure a loss period correctly.
And there are sample sizes, when the means start to deviate 
significantly.

So I repeat my point of view: don't ignore the impact of the 
sample size if two measurement systems to be compared don't work 
with the same. My proposal: Always try to make the sample sizes 
the same (which should be possible with or requires respectively
access to raw data of the measurement results). 

Regards,

Rudiger

| -----Original Message-----
| From: Henk Uijterwaal [mailto:[email-masked]] 
| Sent: Tuesday, May 13, 2008 3:07 PM
| To: Geib, Rüdiger
| Cc: [email-masked]
| Subject: Re: Comparism of metrics
| 
| Hi Rudiger,
| 
| > there has been no discussion on the list since while and I assume 
| > comparing metrics still is an open issue.
| 
| Yes.
| 
| > A point I was concerned
| > about is how to compare probes of different probe sizes (say one of 
| > 100 000 probes measured with system 1 as compared to one of 10 000 
| > probes measured by system 2 during the same time interval T). A 
| > possibility is to create 10 probe sets of 10 000 probes of 
| the first 
| > set and then compare the statistical properties of samples 
| of the same 
| > size only. One obviously has to pick the 10 probe sets with a 
| > reasonable function, as each set must be represenative for 
| the entire 
| > time interval T.
| 
| Sorry, this got stuck in my mailbox, mainly because I don't 
| quite understand what the problem is here.  I still don't :-) 
| so I'm probably missing something...
| 
| Now, suppose I measure something with probes.  I get 10,000 
| events, then bin them to get an idea of the distribution.  
| This gives me a median and percentiles.  If I then repeat 
| this by sampling 100,000 times, then I simply expect 10x as 
| many events in each bin, but I
| don't expect the distribution and thus the median to change.   In
| other words, my curves will be a bit smoother, but the 
| overall result won't change much.
| 
| But this is probably not what you meant :-)
| 
| Henk
| --
| --------------------------------------------------------------
| ----------------
| Henk Uijterwaal                           Email: 
| henk.uijterwaal(at)ripe.net
| RIPE Network Coordination Centre          
| http://www.amsterdamned.org/~henk
| P.O.Box 10096          Singel 258         Phone: +31.20.5354414
| 1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
| The Netherlands        The Netherlands    Mobile: +31.6.55861746
| --------------------------------------------------------------
| ----------------
| 
| Is one of the choices leaving the office open?
|                                        Alan Greenspan on the 
| next elections
| 
| 
_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm