Subject: [ippm] Old work on comparing metrics

From: Henk Uijterwaal <[email-masked]>
Date: Tue, 12 Feb 2008 12:29:38 +0100

IPPM Group,

In Vancouver, I promised to dig up work on comparing implementations
of metrics that I (together with Rene, Sunil and Matt) did and presented
at PAM2000.  Here are the slides:

    http://amsterdamned.org/~henk/PAM2000.pdf

Question to the list: is a simple visual comparison sufficient for our
purposes (see slide 28), or should we do something more (see slide 33).

Henk

-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

Is one of the choices leaving the office open?
                                       Alan Greenspan on the next elections
_______________________________________________
ippm mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ippm

---

From: "Geib, Ruediger" <[email-masked]>
Date: Wed, 13 Feb 2008 09:01:02 +0100

Hello Henk,

I'd like to learn about the calibration or measurement error, when I measure. I can't tell you exactly how to obtain this for IPPM metrics. IPPM should look for a way to characterise this error, rather than just compare implementations. Consider the following: System 1 measures with 0,5% error, System 2 with 10% and System 3 with 0,5% more error then System 2 but showing otherwise similar results as System 2 (distribution etc.). I doubt that it is a useful result of standardisation, if System 2 and 3 of the example are deemed standards conformant whereas System 1 isn't.

If Surveyour produces 2 packets per second and the TTM 3 per minute, the size of the Surveyor sample is 40 times that of the TTM. This causes a different measurement error. Further you discovered a problem with different packet sizes and you further dealt with clock issues (another source of differing errors). I'm not sure whether "visual comparism" is the best way to go. Proposing any other method isn't simmple too (Bradner metric test is one approach, but also not perfect). Does the benchmarking WG have any useful methodology?

Regards,

Ruediger

| -----Original Message-----
| From: [email-masked] [mailto:[email-masked]] On 
| Behalf Of Henk Uijterwaal
| Sent: Tuesday, February 12, 2008 12:30 PM
| To: IETF IPPM WG
| Subject: [ippm] Old work on comparing metrics
| 
| IPPM Group,
| 
| In Vancouver, I promised to dig up work on comparing 
| implementations of metrics that I (together with Rene, Sunil 
| and Matt) did and presented at PAM2000.  Here are the slides:
| 
|     http://amsterdamned.org/~henk/PAM2000.pdf
| 
| Question to the list: is a simple visual comparison 
| sufficient for our purposes (see slide 28), or should we do 
| something more (see slide 33).
| 
| Henk
| 
| --
| --------------------------------------------------------------
| ----------------
| Henk Uijterwaal                           Email: 
| henk.uijterwaal(at)ripe.net
| RIPE Network Coordination Centre          
| http://www.amsterdamned.org/~henk
| P.O.Box 10096          Singel 258         Phone: +31.20.5354414
| 1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
| The Netherlands        The Netherlands    Mobile: +31.6.55861746
| --------------------------------------------------------------
| ----------------
| 
| Is one of the choices leaving the office open?
|                                        Alan Greenspan on the 
| next elections _______________________________________________
| ippm mailing list
| [email-masked]
| http://www.ietf.org/mailman/listinfo/ippm
| 
_______________________________________________
ippm mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ippm

---

From: Al Morton <[email-masked]>
Date: Wed, 13 Feb 2008 08:50:14 -0500

Henk, Ruediger, and all.

I've been thinking about Henk's question since yesterday,
another phrasing might be:
"When we make comparisons between systems in a side-by-side trial,
what's the best way to quantify and report the results?"
Both methods in Henk's slides seem reasonable for Delay,
though we probably need something different for Loss, and other metrics.

Ruediger is pointing-out a weakness of the practical method using
two or more measurement systems and live networks. Without a
trusted or proven measurement made along-side the others, majority rules.

And Ruediger's question,
At 03:01 AM 2/13/2008, Geib, Ruediger wrote:
...Does the benchmarking WG have any useful methodology?

BMWG has the advantage of working in the lab, with conditions
that should be reproducible. So that's one kind of recommendation:
part of the comparison could be based on measurements of
simulated network conditions, in the lab, where repeatability
can be easily quantified. And where specific path conditions should
be easier to create (with network simulators).

There's probably more BMWG ideas to bring to bear on this, too
(still thinking about it).

My main point is this:
I think the goal of the this testing is to show that
the metrics in the RFCs are unambiguously specified.
The methodology used here compares the results
of two or more independent implementations, and that's fine.
But the test conditions need to focus on aspects of the
specification - Does the Poisson stream have the desired
distribution of inter-packet sending times?
Is the Delay singleton definition really First-bit to Last Bit?

IMO, this is about comparing implementations,
in order to test the spec.

regards,
Al

At 03:01 AM 2/13/2008, Geib, Ruediger wrote:
>Hello Henk,
>
>I'd like to learn about the calibration or measurement error, when I 
>measure. I can't tell you exactly how to obtain this for IPPM 
>metrics. IPPM should look for a way to characterise this error, 
>rather than just compare implementations. Consider the following: 
>System 1 measures with 0,5% error, System 2 with 10% and System 3 
>with 0,5% more error then System 2 but showing otherwise similar 
>results as System 2 (distribution etc.). I doubt that it is a useful 
>result of standardisation, if System 2 and 3 of the example are 
>deemed standards conformant whereas System 1 isn't.
>
>If Surveyour produces 2 packets per second and the TTM 3 per minute, 
>the size of the Surveyor sample is 40 times that of the TTM. This 
>causes a different measurement error. Further you discovered a 
>problem with different packet sizes and you further dealt with clock 
>issues (another source of differing errors). I'm not sure whether 
>"visual comparism" is the best way to go. Proposing any other method 
>isn't simmple too (Bradner metric test is one approach, but also not 
>perfect). Does the benchmarking WG have any useful methodology?
>
>Regards,
>
>Ruediger
>
>
>

---

From: Henk Uijterwaal <[email-masked]>
Date: Wed, 13 Feb 2008 16:23:19 +0100

Geib, Ruediger wrote:
> Hello Henk,
> 
> I'd like to learn about the calibration or measurement error, when I measure.
> I can't tell you exactly how to obtain this for IPPM metrics. IPPM should
> look for a way to characterise this error, rather than just compare
> implementations. 

This is what RFC 2679 says one should do: give an estimate of the
measurement error.

 > Consider the following: System 1 measures with 0,5% error,
> System 2 with 10% and System 3 with 0,5% more error then System 2 but showing
> otherwise similar results as System 2 (distribution etc.). I doubt that it is
> a useful result of standardisation, if System 2 and 3 of the example are

I'm not quite sure how this can happen:

   1 measures 100 ms with 0.5 ms error.
   2 measures 105 ms with 10 ms error.
   3 measures 110 ms with 0.5 ms error.

A comparison will show that results of (1) and (2) are consistent, and that
(2) and (3) are consistent, but the group (1, 2, 3) produces inconsistent
results.  How can one say that (1) is the bad guy in this sample?

> If Surveyour produces 2 packets per second and the TTM 3 per minute, the size
> of the Surveyor sample is 40 times that of the TTM. This causes a different
> measurement error.

I doubt it.  2 packets/second is 80 bytes/second and this is only a small
fraction of the capacity of the link.

> Further you discovered a problem with different packet
> sizes and you further dealt with clock issues (another source of differing
> errors).

Correct.

  I'm not sure whether "visual comparism" is the best way to go.
> Proposing any other method isn't simmple too (Bradner metric test is one
> approach, but also not perfect). Does the benchmarking WG have any useful
> methodology?

What the visual method (and R^2 method) try to do is, is to include the
overall shape of the distribution: i.e. both start measuring at 9am and
find (say) 20 ms, at 10am something changes and delays go up to 30 ms,
at 11 am, things change back and delays go down again.  If both tools
show this, this is a good indication that the implementations measure
the same thing.  We somehow have to quantify this and that his what
the 2 methods try to accomplish.

Henk

---

From: Henk Uijterwaal <[email-masked]>
Date: Wed, 13 Feb 2008 16:28:52 +0100

> be easier to create (with network simulators).

This sort-of what we tried to do back in 2000.  Install equipment
at both ends of a connection and send packets along the path at the
same time, then compare the result.  We could have done this in
the lab with just a piece of cable.

> Is the Delay singleton definition really First-bit to Last Bit?

This looks sounds like an audit of the code.  (Generate a poisson
distribution and see if it really is poisson, check if one
measures 1st to last bit, ...).   This would require a checklist
and somebody checking things off.

Henk

---

From: "Geib, Ruediger" <[email-masked]>
Date: Wed, 13 Feb 2008 16:57:43 +0100

Hello Henk, 

| > Consider the following: System 1 measures with 0,5% error,
| > System 2 with 10% and System 3 with 0,5% more error then 
| > System 2 but showing otherwise similar results as System 2 
| > (distribution etc.). I doubt that it is a useful result 
| > of standardisation, if System 2 and 3 of the example are 
| > deemed standards conformant whereas System 1 isn't.
| 
| I'm not quite sure how this can happen:
| 
|    1 measures 100 ms with 0.5 ms error.
|    2 measures 105 ms with 10 ms error.
|    3 measures 110 ms with 0.5 ms error.

To clarify my point: Lets pick averages 100, 110 and 110,5 
with similar variations for each one. 

| > If Surveyour produces 2 packets per second and the TTM 3 
| > per minute, the size of the Surveyor sample is 40 times 
| > that of the TTM. This causes a different measurement error.
| 
| I doubt it.  2 packets/second is 80 bytes/second and this is 
| only a small fraction of the capacity of the link.

The sample size of all packets travelling from A to B within 
one minute is 120 for Surveyor and 3 for TTM.

| What the visual method (and R^2 method) try to do is, is to 
| include the overall shape of the distribution: i.e. both 
| start measuring at 9am and find (say) 20 ms, at 10am 
| something changes and delays go up to 30 ms, at 11 am, things 
| change back and delays go down again.  If both tools show 
| this, this is a good indication that the implementations 
| measure the same thing.  We somehow have to quantify this and 
| that his what the 2 methods try to accomplish.

The sample size impacts the measurement error and the average 
temporal distance influences the kind of events you can measure. 
This again may influence the results (loss is an easily 
understandable metric for this) To recognise a temporal event, 
you must measure at least with twice the frequency of that 
event. Looking at the above, Surveyor may capture more fine 
grained delay events than TTM. If present, they may cause 
different results (queuing comes to my mind).

While I agree that a look at the results shows that the 
distributions are similar, I'm not sure, whether the method 
can be generalised unconditionally. The conditions for which 
this method may be applied should be mentioned.

Regards,

Ruediger