Subject: Re: [IMRG] Measurement TCPs

From: [email-masked] (P.F. Chimento)
Date: Fri, 13 Sep 2002 08:13:20 -0400

I agree with Rolf and Henk; if we have pretensions of being 
true scientists, then you have to have repeatability of results
and for that you need to have exact procedures and the raw data.
Yes, I know that not everyone has access to the same links/systems, 
but the results can be checked. Not a bad thing when you realize how 
subtly numerical errors can creep into computations. 

Regards, Phil Chimento

__________________________________________________________________
The NEW Netscape 7.0 browser is now available. Upgrade now! http://channels.netscape.com/ns/browsers/download.jsp 

Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/
_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: Mark Allman <[email-masked]>
Date: Fri, 13 Sep 2002 11:30:38 -0400

Interesting discussion, thanks to everyone for participating thus
far!

(Terminating back-slapping chair mode.)

So, please forget for a moment that I wrote a paper on the
importance of making raw data available in a standard way for a
moment. 

> I agree with Rolf and Henk; if we have pretensions of being true
> scientists, then you have to have repeatability of results and for
> that you need to have exact procedures and the raw data.

Let me agree with everything but the last two words.  And, I'll
assert that measurement should really be about insight, not numbers.
So...

Are we counting on the release of raw data only to cure all our
ills?

That is, do we really need raw data to confirm results?  If someone
gives me their exact procedure (python scripts, say) and their raw
data and I re-run the whole mess and end up with the same numbers is
that confirmation?  Is that repeatability?  To some degree, yes (if
we can't even do that -- and, sometimes we can't, BTW -- we're in a
tough spot).  But, to a large extent I don't see that as
confirmation.  (Of course, if I can look at their python code that
is helpful in verifying there is no small bug/oversight/whatever.)

If you tell me how to run an experiment I should be able to run the
experiment again and get the same results.  Where "the same" means
the same insights, not the same numbers.  If I cannot do that then
we look further -- maybe the network changed, or maybe the
heterogeneous nature of the network bit us, or maybe the results
were wrong or about a thousand other things.  But, if I can repeat
the experiment and gather the same insights then I think we have
found repeatability through independent trials -- and then we know
we have sound results.  In my opinion, that -- moreso than just
letting everyone hack over the same data -- is good science.

This all doesn't mean that I think repositories of data, tools,
etc. are bad.  I think they could be very nice.  They give everyone
more vantage points.  And, we can do things like trace driven
simulations of various algorithms across the same data.  And...

However, I am not convinced that measurement repositories are an
absolute necessaity for good science.

allman

--
Mark Allman -- BBN/NASA GRC -- http://roland.grc.nasa.gov/~mallman/
_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: Ratul Mahajan <[email-masked]>
Date: Sat, 14 Sep 2002 16:45:37 -0700 (PDT)

> absolute necessaity for good science.

forget the good science war cry. 

so why do you think people would be interested in the raw data i used for
my work. 99.9% of the times it is not to do what i did with it - that way
they cannot publish unless they prove me wrong. its not repeatability the
world (at least not our community, i feel)  cares about.

i think the real value of raw data lies in the fact that different people
would be able to do different things with it. data ends up being used in
various ways that original collectors never imagined. look at route-views
- who would have imagined all the uses it is being put to now. we are
beginning to experience some of that with rocketfuel.

cheers,
	-- ratul

---

From: Mark Allman <[email-masked]>
Date: Sun, 15 Sep 2002 23:15:20 -0400

Ratul-

> forget the good science war cry. 

OK.  

[Flipping your topic around.]

> i think the real value of raw data lies in the fact that different
> people would be able to do different things with it. data ends up
> being used in various ways that original collectors never
> imagined. look at route-views - who would have imagined all the
> uses it is being put to now. we are beginning to experience some
> of that with rocketfuel.

Yep.  I completely agree.  And, the other thing that sharing data is
quite useful for is the case when you want to compare some algorithm
you designed with some algorithm I published.  You can sometimes
look at the data I used and then run some trace-driven simulation
across it to compare apples to apples.  Also, making data available
allows for trending down the road.  For instance, storing network
maps (or traceroute measurements) is useful for looking for
invariants in topology over time.

So, I agree completely that storing raw data is very useful to the
community for quite a lot of reasons.

> so why do you think people would be interested in the raw data i
> used for my work. 99.9% of the times it is not to do what i did
> with it - that way they cannot publish unless they prove me
> wrong. its not repeatability the world (at least not our
> community, i feel) cares about.

Yeah.  I agree with this statement too.

But, some questions for the group: Is Ratul's assertion the way
things ouught to be?  That is, do we not verify each other's results
because we cannot publish that work if the original research is
right?  Or, do we not verify the original research because we do not
think it is useful?  Or, some other reason (e.g., not as sexy)?  If
there were a place to publish the "mundane" work of confirmation
studies would that encourage folks to do them?  

For instance, I could see this being a sort of first grad school
project.  Something to get student's fingers dirty, while serving
the community (and, giving a professor a chance to see the student's
skills and abilities in action before committing to something
bigger).  And, if there was a place to publish short verification
papers that would seem nice to me.  Opinions?

allman

--
Mark Allman -- NASA GRC/BBN -- http://roland.grc.nasa.gov/~mallman/

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Mon, 16 Sep 2002 12:55:37 +0200 (CEST)

> Are we counting on the release of raw data only to cure all our

Thinking about this a bit more: No, I think that they are the second best
thing, and very useful when the best approach is not feasible, which it
often isn't.

> that confirmation?  Is that repeatability?  To some degree, yes (if we
> can't even do that -- and, sometimes we can't, BTW -- we're in a tough
> spot).  But, to a large extent I don't see that as confirmation.

I don't think that rerunning the same code on the same data is going to
add much, except perhaps in the case where one uses different hardware
platforms and gets different results.

However, back in my HEP days, it was quite common that 2 or 3 groups tried
to extract the same numbers from the same raw data set, in order to make
sure that there were no trivial mistakes in the (often 1,000's of lines
of) code.  In fact, some of the experiments that we worked on required
that there were 2 completely independent analyses with different code that
agreed on the outcome, before any result could be published.  Now, these
were experiments with 100's of collaborators, so it was easy to find 2 or
3 groups of people.  In Network measurements, the groups are much smaller
so this is impossible.

If we want repeatability, then having the raw data can be very useful.
Any group that wants to do something related, can re-use the old data to
calibrate their code and verify the old result.  If the result is the
same, then this adds to the credibility of the new results, if the old
result cannot be reproduced, there is a problem which then has to be
investigated.

> letting everyone hack over the same data -- is good science.

I agree, if one _can_ repeat the experiment from scratch, then this is
much better.  It excludes the possibility that something went wrong with
the experiment itself. resulting in buggy raw data. Unfortunately, it
often won't be possible to rerun an experiment for all kinds of reasons,
such as access to equipment, equipment changes, privacy/security, company
policies, etc, etc. In that case, having the raw data available is the
next best thing.

Henk

------------------------------------------------------------------------------
Henk Uijterwaal                    Email: [email-masked]
RIPE Network Coordination Centre     WWW: http://www.ripe.net/home/henk
Singel 258                         Phone: +31.20.5354414
1016 AB Amsterdam                    Fax: +31.20.5354445
The Netherlands                   Mobile: +31.6.55861746
------------------------------------------------------------------------------

That problem that we weren't having yesterday, is it better? (Big ISP NOC)

NOTE: My email address (and a hole in our mailing list software) is being
      abused by a spammer.  We are working on fixing this hole and tracking
      the spammer down.  If you receive mail from "[email-masked]" that is
      obviously spam, please send me a copy of the mail including ALL headers.
      I'm sorry for any inconvenience caused by this.

---

From: Mark Allman <[email-masked]>
Date: Mon, 16 Sep 2002 12:06:31 -0400

Henk-

Nice thoughts, thanks!

> I agree, if one _can_ repeat the experiment from scratch, then
> this is much better.  It excludes the possibility that something
> went wrong with the experiment itself. resulting in buggy raw
> data. Unfortunately, it often won't be possible to rerun an
> experiment for all kinds of reasons, such as access to equipment,
> equipment changes, privacy/security, company policies, etc,
> etc. In that case, having the raw data available is the next best
> thing.

I agree with this to an extent.  But, I'd like to offer the opinion
that while an exact replica of the experiment is likely impossible
repeating an experiment to confirm the key insights of the original
work should be quite possible.  Even though the numbers will not
line-up the insights should hold.  It should not matter if I am
measuring one-way delay on a sun machine or a freebsd box.  The
numbers might be different, for sure (due to more or less overhead
or the scheduler or whatever).  But, some insight (e.g., we see
network induced delay spikes (for definition of "delay spike") every
N seconds) should remain.  If it does not then we dig.

And, note: I am not saying we should be careless in repeating
experiments.  We certainly should try to do our best to reproduce
the experiment accurately.  But, in the inevitable cases where you
cannot exactly replicate some aspect of the experiment (as you note
there will be) we should still be able to assume that a reasonable
substitution should yield us the key insights the original
experiment gave us.  And, if we do not get the same insight the
first place to look for a reason should be the differences between
the two experiments.

(And, not to argue against making raw data available.  Just that
verifying previous results on raw data only goes so far.)

Thanks!

allman