Subject: [manet] [Fwd: [e2e] simulation, sanity, citation, and storage]

From: Erik Nordström <[email-masked]>
Date: Thu, 17 Nov 2005 14:16:50 +0100

Below is a forwarded message posted on the e2e list by Jon Crowcroft. It
should be mandatory read for every MANET researcher, especially those
doing research based on simulation. It is pretty frightening reading,
but raises issues that have gone undebated for too long in the
community.

Erik Nordström

-------- Forwarded Message --------
From: Jon Crowcroft <[email-masked]>
To: [email-masked]
Subject: [e2e] simulation, sanity, citation, and storage
Date: Wed, 16 Nov 2005 16:37:40 +0000

How perhaps not to do mobile nets research :-

1. The random waypoint mobility model
(probably
the most commonly used model for how devices move that is used in
100s of papers for simulation of how good my new mobile ad hoc routing
protocol is compared to yours) -
has non stationary distribution of velocity in pretty much any
plausible 2D space, so you
don't know when to stop/start simulators using the random waypoint
mobility model:

This paper is whats wrong, and how to fix it:
"Perfect Simulation and Stationarity of a Class of Mobility Models,"
J.-Y. Le Boudec and M. Vojnovic, IEEE Infocom 2005, Miami, FL, 2005
(Infocom 2005 Best Paper Award)
http://ica1www.epfl.ch/RandomTrip.

2. Many simulation papers have not only got non reproduceable results,
but the quality of simulation papers is apparently
getting worse over time:

This paper is about how this trend has progressed, and should be
essential reading for all 1st year grad students:)

"{MANET} Simulation Studies: The Incredibles",
by
Stuart Kurkowski and Tracy Camp and Michael Colagrosso,
mc2r, 2005, volume 9, 4, pages "50--61",
"http://doi.acm.org/10.1145/1096166.1096174"

Its pretty "interesting" how bad things are out there:)

Two possible social solutions:-

a) on the one hand,
so how about we introduce the idea (e..g via Vint Cerf, now at google)
to Google Scholar, of _negative_ citations. it ought to be easy to add
to the pagerank algorithm:)

Given we find that a numnber of papers have been written that
are rooted on some baseline reference (e.g. the original
NS code for mobile IP, or the original random waypoint)
when we then show that the NS code is wrong, or the random waypoint
model cannot be used to compare results in simulation experiments due
to lack of stationarity, those papers all suddenly have their
citation count reduced:)

b) on the other hand,
if people post their _code_, then bugfixes to the  code would accrue
_positive_ citations. we could use
google base for this 
http://base.google.com/base/default
could be a repository for simulation
code, configurations and draft papers
(A bit like the science preprints idea, but for _supporting_ evidence.

cheers

jon

_______________________________________________
manet mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/manet

---

From: Andrés Lagar Cavilla <[email-masked]>
Date: Thu, 17 Nov 2005 09:25:27 -0500

I wouldn't say "undebated". There is a growing number of publications 
regarding the weaknesses of many simulation models/techniques for MANET 
evaluation.  I mean, a lot of researchers are alert and have been 
publishing "Reconsider your techniques!!" papers and articles in the 
last two years.
Consider this self-promotion as yet one more example: "Simplified 
Simulation Models for Indoor MANET Evaluation are not Robust", IEEE 
SECON 2004, 
http://www.cs.toronto.edu/~andreslc/publications/LagarCavillaSECON04.pdf
Maybe the problem is that people do not listen, or are too comfortable 
tweaking their routing protocols in the freely available ns2 (and 
assuming all the other issues have been magically solved forever).
But I feel the momentum of the community is shifting in the right direction.
Andres

---

From: Erik Nordström <[email-masked]>
Date: Thu, 17 Nov 2005 16:06:46 +0100

By debate, I mean an actual discussion. Publications are valuable to
back up the arguments, but provide little in terms of debate.
Considering this list's central position in this matter, it has to my
knowledge been rather silent when it comes to debating this topic.

Erik

---

From: Einar Vollset <[email-masked]>
Date: Thu, 17 Nov 2005 15:12:47 +0000

Ok well...

To add to Jon's suggestions; here's another one:

Make everything work through the web! It should be quite simple:

1. Someone make a simulator (or even better, a real world testbed)  
available
to one and all, with documented and configurable stack (you could  
even make money by selling subscriptions to it, standard, gold,  
platinum versions :-)?)
2. Make a "default" set of test scenarios available,  say"busy-city"  
and "battlefield" +++
3. To test your protocol, you simply submit everything to the  
webservice; the protocol, config files,
chosen scenarios, etc, etc, and everything gets logged and and all  
data gets stored.
4. When you publish, rather than having to have a loooong tedious  
list of what you did and how (and
face it, that's never going to be fully reproducible), you simply  
reference an entry in the www.manetsimulator.com
where anyone and everyone can look at the data, get the protocol,  
everything.
5. To compare and contrast your protocol, either simply use the  
logged data, or if you're trying to outperform in some
new field, re-run the protocol.

Hey presto, problem solved.. ;-)

Cheers,

Einar

---

From: Christophe Jelger <[email-masked]>
Date: Thu, 17 Nov 2005 16:28:51 +0100

Hi all,

Well the "problems" of the RWP model have been highlighted since quite a 
long time and there are available "solutions" out there. For example 
some interesting papers are:

- J. Yoon, M. Liu , B. Noble, "Random Waypoint Considered Harmful", 
Infocom'03
- W. Navidi, T. Camp, N. Bauer, "Improving the Accuracy of Random 
Waypoint Simulations Through Steady-State Initialization", in Conf. on 
Modeling and Simulation '04 --> these people also provide a tool to 
generate ns-compatible mobility patterns that reach steady-state.

The key to avoid poor simulations being published is twofold:

1) publish simulation "best practices" in order to make people aware of 
new findings
2) have a more strict and thorough reviewing process

However these two issues also introduce new problems:

1) what is being today thought as current "best practice" might be 
invalidated in a few years. For example, 5 years ago nobody was 
complaining about the RWP model until somebody realized there was a 
steady-state problem.

1bis) Unfortunately more complex cases sometimes end-up in contradicting 
conclusions (e.g. the debates of power-laws in Internet topology or 
self-similarity in Internet traffic). So who's right and who's not ?

2) reviewing process is hard to control, and there is REALLY a lot of 
conferences out there.

My collegue C. Tschudin in Basel once proposed that we create a 
"simulation depot" where people could make their ns-code available so 
that others can re-use it, verify stuff and so on ... that sounds good 
but would also introduce quite many problems (copyright, "secrecy 
concerns", etc).

It's clear that there is a need for better simulation practices, but 
this issue is rather complex and is not restricted to computer science 
(see the testbed issues around sono(cold)fusion in physics ...).

Regards,
Christophe

---

From: Wenjun Hu <[email-masked]>
Date: Thu, 17 Nov 2005 16:14:47 +0000

There are several separate issues:

1. Whether the models are sound for a simulation from a statistical point of 
view, e.g., concerns of stationarity.

2. Whether the models match the real-world scenarios they're supposed to 
simulate, so realism issue.

3. Given a model, how to run simulations using that so that the simulation 
results are reproducible - e.g., making sure the simulation run is long enough 
for the results to converge (if that's possible).

The INFOCOM paper Jon cited mostly addresses the first point, and shows that 
it is not possible to perform repeatable simulations using RWP, due to the way 
the model works. The MC2R paper has quite some emphasis on documenting the 
_exact_ simulation settings, so that others can use the same parameters to 
_try_ to reproduce/validate the published results (if that's possible).

Wenjun

> 1. Someone make a simulator (or even better, a real world testbed) =20
> available
> to one and all, with documented and configurable stack (you could =20
> even make money by selling subscriptions to it, standard, gold, =20
> platinum versions :-)?)
> 2. Make a "default" set of test scenarios available,  say"busy-city" =20
> and "battlefield" +++
> 3. To test your protocol, you simply submit everything to the =20
> chosen scenarios, etc, etc, and everything gets logged and and all =20
> data gets stored.
> 4. When you publish, rather than having to have a loooong tedious =20
> face it, that's never going to be fully reproducible), you simply =20
> where anyone and everyone can look at the data, get the protocol, =20
> everything.
> 5. To compare and contrast your protocol, either simply use the =20

> >>> 100s of papers for simulation of how good my new mobile ad hoc =20
> >>> routing
> >>> 2. Many simulation papers have not only got non reproduceable =20
> >>> results,
> >>> so how about we introduce the idea (e..g via Vint Cerf, now at =20
> >>> google)
> >>> to Google Scholar, of _negative_ citations. it ought to be easy =20
> >>> to add
> >>> model cannot be used to compare results in simulation experiments =20=
> 
> >>> due
> >>> (A bit like the science preprints idea, but for _supporting_ =20
> >>> evidence.
> >>>
> >>> cheers
> >>>
> >>> jon
> >>>
> >>>
> >>>

---

From: Einar Vollset <[email-masked]>
Date: Thu, 17 Nov 2005 16:20:54 +0000

All agreed!

However,

> The MC2R paper has quite some emphasis on documenting the

...my contention is that this is near on impossible unless the  
protocol being
simulated is designed for "publication" in the first place.

Cheers,

Einar

---

From: Erik Nordström <[email-masked]>
Date: Thu, 17 Nov 2005 17:34:35 +0100

> 1. Someone make a simulator (or even better, a real world testbed)  

For a testbed, please look here:

http://apetestbed.sourceforge.net/

Has been available for some time and comes with predefined scenarios...

Erik

---

From: Einar Vollset <[email-masked]>
Date: Thu, 17 Nov 2005 16:44:54 +0000

Ah, yes, good, excellent, except that it doesn't (appear to) do what  
I was suggesting.
Whomever is developing the protocol shouldn't have to install and run  
the actual simulator/testbed.
It should only (mainly?) be available to run remotely. You register  
your stuff, upload the config/protocol descriptions
and press the big green button. All logged and readily available/ 
reproducible/ha-ha-look-how-crap-that-is-able.. ;-)

Cheers,

Einar

---

From: "Matthew Bygo" <[email-masked]>
Date: Thu, 17 Nov 2005 17:01:26 +0000

it would be useful to have a repository, where researchers can log both 
successful reproduction of published results and discrepencies between 
published and reproduced results. Of course, this should lead to (and log) 
discussion between authors and "reproducers" to understand the 
discrepencies, as the "fault" may not lie with the authors !
It would allow the community to check how much scrutiny particular results 
have had before using them.
Matt

>From: Erik Nordstr�m <[email-masked]>
>To: [email-masked]
>Subject: [manet] [Fwd: [e2e] simulation, sanity, citation, and storage]
>Date: Thu, 17 Nov 2005 14:16:50 +0100
>

_________________________________________________________________
The new MSN Search Toolbar now includes Desktop search! 
http://toolbar.msn.co.uk/

---

From: Mirco Musolesi <[email-masked]>
Date: Thu, 17 Nov 2005 18:25:22 +0000

Hi all,

> Ah, yes, good, excellent, except that it doesn't (appear to) do  
> what I was suggesting.
> Whomever is developing the protocol shouldn't have to install and  
> run the actual simulator/testbed.

At the same time, I think that the actual implementation is quite  
important and useful too. I think that a "black box" approach may be  
sufficient only in first instance.  A repository of simulation code  
can be extremely useful for comparisons.

The problem of reproducing realistic scenarios is still very open,  
since by analysing the available traces (see http:// 
crawdad.cs.dartmouth.edu/), existing mobility models (like RWP) are  
not able to capture a large set of observable real movement patterns  
(in particular, among communities in built environments, especially  
in case of non-dense networks).

Mirco

--
Mirco Musolesi
Department of Computer Science, University College London
Gower Street London WC1E 6BT United Kingdom
Phone: +44 20 7679 0391 Fax: +44 20 7387 1397
Web: http://www.cs.ucl.ac.uk/staff/m.musolesi

---

From: Mirco Musolesi <[email-masked]>
Date: Thu, 17 Nov 2005 18:41:19 +0000

Hi all,

I forgot to say that we are currently studying possible more  
realistic mobility models. If you are interested, the preliminary  
results of our work are published here:

http://www.cs.ucl.ac.uk/staff/m.musolesi/mobilitymodelns2.html

[More coming soon...]

Mirco

---

From: Kwan-Wu Chin <[email-masked]>
Date: Fri, 18 Nov 2005 08:02:59 +1100

To add to this,

  Recently, a mobility model inspire by networked games such
as Quake has emerged.   This effectively allows one to use the
floorplan of a building (say the last IETF meeting), have MANET
 researchers roam through the building (shooting each other, maybe),
and have their movement then logged or modelled.   This "realistic"
mobility model can then be used in MANET simulations.

  The paper,

  http://www.research.ibm.com/netgames2005/program.html

Networked Game Mobility Model for First-Person-Shooter Games 
<http://www.research.ibm.com/netgames2005/papers/tan.pdf>

Swee Ann Tan, W. Lau, Alan Loh

cheers,
kwan-wu