Subject: Re: [IMRG] Measurement data and scientific process

From: Ratul Mahajan <[email-masked]>
Date: Thu, 3 Oct 2002 15:00:28 -0700 (PDT)

> Alas, it seems to me that this is not likely to happen in the
> networking community ... not because some don't fudge experiments,
> but because I doubt the rest care enough ... :(

i don't think this is the case. whenever anyone has made startling claims
about the internet, there have been many people who have verified those. i
can think of power-law stuff for router outdegrees, zipf nature of web
files accesses, and self-similarity. maybe the non-proprietary nature of
most internet data helps.

	-- ratul

On Mon, 30 Sep 2002, Mark Allman wrote:

> 
> Henning-
> 
> Thanks for sending the pointer.
> 
> > http://www.lucent.com/news_events/researchreview.html is, I
> > believe, of some interest to this community, even though the topic
> > of the investigation is far removed from networking. The report
> > stresses the importance of maintaining primary data and
> > reproducibility:
> > 
> > "At a minimum, Hendrik Schön showed reckless disregard for the
> > sanctity of data in the value system of science. His failure to
> > retain primary data files compounds the problem."
> 
> Alas, it seems to me that this is not likely to happen in the
> networking community ... not because some don't fudge experiments,
> but because I doubt the rest care enough ... :(
> 
> allman
> 
> 
> --
> Mark Allman -- NASA GRC/BBN -- http://roland.grc.nasa.gov/~mallman/
> _______________________________________________
> IMRG mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/imrg
> 

_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: Mark Allman <[email-masked]>
Date: Thu, 03 Oct 2002 23:07:45 -0400

Ratul-

Thanks for the comments!

> i don't think this is the case. whenever anyone has made startling
> claims about the internet, there have been many people who have
> verified those. i can think of power-law stuff for router
> outdegrees, zipf nature of web files accesses, and
> self-similarity. 

I agree, there are cases where we have done OK.  As you suggest,
controversial topics are vetted by the community -- but, some of
that is fueled by folks who disbelieve the research in the
literature.  And, that is fine, don't get me wrong.  

And, we end up doing some verification in the process of new work.
I just finished reading your SIGCOMM rocketfuel paper and you vetted
out the mercator alias resolution scheme because you found that it
didn't work as well as you had hoped.  That is nice, too.

(And, BTW, I found the rocketfuel study to be a very nice paper!)

But, what of the less startling results?  The ones we look at and
say "oh yeah -- that makes good sense", rather than the ones we look
at and say "but, wait a minute... I'm not so sure...  huh?".  

Allow me to pick on myself...  Vern Paxson and I have a 1999 SIGCOMM
paper that shows the performance of the standard TCP RTO scheme when
tweaked in a lot of different ways (varying gains, varying K, using
a minimum, varying the clock granularity, etc.).  As far as I know
nobody has verified our results.  The results were not a huge leap.
I think we added to the understanding of the TCP RTO and what is
important and what is not.  But, man, if we screwed a line in a perl
script and are off by an order of magnitude our results might be way
different.  And, this paper has been cited pretty widely as Good
Stuff.  It influenced the RFC written on the RTO (RFC 2988).  It may
very well influence any update to RFC 1323 that recommends timing
every segment/ACK pair rather than one per RTT.  Is this a Good
Thing?  Is this the Right Path?

I don't mean to suggest that we (royal sense) never do anything
right.  My point is that it might be useful to start thinking about
what sorts of things we can do better -- i.e., what are the
challenges to network measurement going forward?

allman

---

From: k claffy <[email-masked]>
Date: Thu, 17 Oct 2002 11:13:55 -0700

in the interest of full disclosure,
andre actually found non-confirmation
for `power-law stuff for router outdegrees'
http://www.caida.org/outreach/papers/2001/OSD/
(see section VIII; figs 10,11of paper)

but the fact that it's easier to verify/refute/calibrate
publically available data is not really a point of
contention. it's more a lack of (vern's words, slide 39/40:)
`community culture for/ interest in cultivating 
measurement as science and discpline'

(sorry to repeat the URL but the talk is brilliant;
	http://www.icir.org/vern/talks/vp-nrdm01.ps.gz 
i wish vern's audio were up with it)

it should be required reading/memorization 
before subscribing to this list.
(mark you should link to it from imrg home page
if vern doesn't mind)

not that we can achieve slide 40
(and i do seriously lack knowledge of 
`what can we learn here from other fields'
so if anyone has pointers, please offer,
since we're rather on hook here) 

last week james cowie said he and andy tried
to do something similar for the simulation field
(at least with the ssfnet project) at he felt
it was an abject failure.  discouraging.
k
  
  	-- ratul