Subject: RE: [manet] [Fwd: [e2e] simulation, sanity, citation, and storage]

From: "Anders Nilsson" <[email-masked]>
Date: Thu, 17 Nov 2005 20:21:33 +0100 (CET)

Hi all,

Having a repository like this sounds to me, to be a very good idea. By
having others reproduce your results, your results may not only gain
higher credibility, but your results may also receive more attention. A
win-win situation. If they are not validated, you may always learn
something useful, maybe even important, from the other tester.

As for having a common MANET simulator on the net, I think that sounds
like a BIG, but exciting project. But for such a simulator to be
successful, it needs to have a deeply rooted support within our community.
Which it very well may receive, if enough people put an effort into making
it as good as it possibly can get. As it is now, a lot of people seem to
be using the wireless models within NS, although not that many seems to be
satisfied with it. It seems to me that NS are receiving acceptance within
the MANET community because of extensive use, rather than technical
accuracy. I think a MANET simulator made by the community, for the
community, have a bigger potential. But it is a big project, and it needs
a lot effort as well as support from a number of people.

Anders Nilsson

> it would be useful to have a repository, where researchers can log both
> successful reproduction of published results and discrepencies between
> published and reproduced results. Of course, this should lead to (and log)
> discussion between authors and "reproducers" to understand the
> discrepencies, as the "fault" may not lie with the authors !
> It would allow the community to check how much scrutiny particular results
> have had before using them.
> Matt
>
>>From: Erik Nordström <[email-masked]>
>>To: [email-masked]
>>Subject: [manet] [Fwd: [e2e] simulation, sanity, citation, and storage]
>>Date: Thu, 17 Nov 2005 14:16:50 +0100
>>
>>Below is a forwarded message posted on the e2e list by Jon Crowcroft. It
>>should be mandatory read for every MANET researcher, especially those
>>doing research based on simulation. It is pretty frightening reading,
>>but raises issues that have gone undebated for too long in the
>>community.
>>
>>Erik NordstrÃ¶m
>>
>>-------- Forwarded Message --------
>>From: Jon Crowcroft <[email-masked]>
>>To: [email-masked]
>>Subject: [e2e] simulation, sanity, citation, and storage
>>Date: Wed, 16 Nov 2005 16:37:40 +0000
>>
>>How perhaps not to do mobile nets research :-
>>
>>1. The random waypoint mobility model
>>(probably
>>the most commonly used model for how devices move that is used in
>>100s of papers for simulation of how good my new mobile ad hoc routing
>>protocol is compared to yours) -
>>has non stationary distribution of velocity in pretty much any
>>plausible 2D space, so you
>>don't know when to stop/start simulators using the random waypoint
>>mobility model:
>>
>>This paper is whats wrong, and how to fix it:
>>"Perfect Simulation and Stationarity of a Class of Mobility Models,"
>>J.-Y. Le Boudec and M. Vojnovic, IEEE Infocom 2005, Miami, FL, 2005
>>(Infocom 2005 Best Paper Award)
>>http://ica1www.epfl.ch/RandomTrip.
>>
>>2. Many simulation papers have not only got non reproduceable results,
>>but the quality of simulation papers is apparently
>>getting worse over time:
>>
>>This paper is about how this trend has progressed, and should be
>>essential reading for all 1st year grad students:)
>>
>>"{MANET} Simulation Studies: The Incredibles",
>>by
>>Stuart Kurkowski and Tracy Camp and Michael Colagrosso,
>>mc2r, 2005, volume 9, 4, pages "50--61",
>>"http://doi.acm.org/10.1145/1096166.1096174"
>>
>>Its pretty "interesting" how bad things are out there:)
>>
>>
>>
>>Two possible social solutions:-
>>
>>a) on the one hand,
>>so how about we introduce the idea (e..g via Vint Cerf, now at google)
>>to Google Scholar, of _negative_ citations. it ought to be easy to add
>>to the pagerank algorithm:)
>>
>>Given we find that a numnber of papers have been written that
>>are rooted on some baseline reference (e.g. the original
>>NS code for mobile IP, or the original random waypoint)
>>when we then show that the NS code is wrong, or the random waypoint
>>model cannot be used to compare results in simulation experiments due
>>to lack of stationarity, those papers all suddenly have their
>>citation count reduced:)
>>
>>b) on the other hand,
>>if people post their _code_, then bugfixes to the  code would accrue
>>_positive_ citations. we could use
>>google base for this
>>http://base.google.com/base/default
>>could be a repository for simulation
>>code, configurations and draft papers
>>(A bit like the science preprints idea, but for _supporting_ evidence.
>>
>>cheers
>>
>>jon
>>
>>
>>
>>_______________________________________________
>>manet mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/manet
>
> _________________________________________________________________
> The new MSN Search Toolbar now includes Desktop search!
> http://toolbar.msn.co.uk/
>
>
> _______________________________________________
> manet mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/manet
>

_______________________________________________
manet mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/manet

---

From: "Richard Gwee" <[email-masked]>
Date: Fri, 18 Nov 2005 09:35:06 +0800

Hi everyone,

I do like the idea of having repository. In a way, we can be more sure that our simulation results are credible enough to reflect practical situations as close as possible.

As for the common MANET simulator, it does seem that it requires a lot of effort. But my gut feeling is that no matter what kind of simulation software we develop, we will still face the common issue of technical accuracy. I will have preferred to think that we should not overhaul the current simulation systems we have but rather try to improve on the technical accuracy of such systems. I believe such work will be greatly appreciated and recognized by the community.

Just some thoughts.

Thanks and regards
Richard Gwee

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anders Nilsson
Sent: Friday, November 18, 2005 3:22 AM
To: Matthew Bygo
Cc: [email-masked]
Subject: RE: [manet] [Fwd: [e2e] simulation, sanity, citation, and storage]

Hi all,

Anders Nilsson

Republic Polytechnic, Tanglin Campus, 1 Kay Siang Road, Singapore 248922 . Website: www.rp.sg . Fax: +65 6415-1310 .

>From April 2006, we will be located in our new home at 9 Woodlands Avenue 9,Singapore 738964 
Republic Polytechnic, the first Institute of Higher Learning to fully adopt the Problem-Based Learning approach in Singapore, continues to strive towards best practices and maintain excellence in service standards with the following certifications: Singapore Quality Class (SQC) certification, People Developer Standards and QEHS standards (ISO 9001, 14001 and OHSAS 18001 certifications).
---------------------------------------------------------------------
CONFIDENTIALITY CAUTION: This message is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged and confidential. If you, the reader of this message, are not the intended recipient, you should not disseminate, distribute or copy this communication. If you have received this communication in error, please notify us immediately by return email and delete the original message. Thank you.

---

From: Andrea Passarella <[email-masked]>
Date: Fri, 18 Nov 2005 08:13:10 +0000

Hi all,

I agree that having common places (and practices!) to share simulation 
code/setup/results would be nice. And it would also lead to have 
higher-quality simulation code maybe.

BTW. here are some efforts on building real testbed eventually 
(hopefully) open to the community I'm aware of (maybe you already heard 
about them). I think they are in the direction Einar was suggesting. 
Maybe it would be worth discussing about the approaches/approximations 
etc. to have consensus on testbeds to use?

ORBIT - WinLab @ Rutgers http://www.orbit-lab.org/
iWWT - ECE @ Urbana-Champaign (in the first stages?)
both presented at E-WIND this year
http://www.acm.org/sigs/sigcomm/sigcomm2005/w1-e-wind.html

MiNT - CS @ Stony Brook http://www.ecsl.cs.sunysb.edu/mint/
presented at Infocom this year

cheers,

andrea

 >Ah, yes, good, excellent, except that it doesn't (appear to) do what
 >I was suggesting.
 >Whomever is developing the protocol shouldn't have to install and run
 >the actual simulator/testbed.
 >It should only (mainly?) be available to run remotely. You register
 >your stuff, upload the config/protocol descriptions
 >and press the big green button. All logged and readily available/
 >reproducible/ha-ha-look-how-crap-that-is-able..  ;-)

 >Cheers,

 >Einar

 >On 17 Nov 2005, at 16:34, Erik Nordström wrote:

 >> On Thu, 2005-11-17 at 15:12 +0000, Einar Vollset wrote:
 >
 >>>> Ok well...
 >>>>
 >>>> To add to Jon's suggestions; here's another one:
 >>>>
 >>>> Make everything work through the web! It should be quite simple:
 >>>>
 >>>> 1. Someone make a simulator (or even better, a real world testbed)
 >>>> available
 >
 >>
 >> For a testbed, please look here:
 >>
 >> http://apetestbed.sourceforge.net/
 >>
 >> Has been available for some time and comes with predefined
 >> scenarios...
 >>
 >> Erik
 >>

-- 
Andrea Passarella
=========================================================
		University of Cambridge
The Computer Laboratory	|| voice:+44 (0)1223 763658
15 JJ Thomson Avenue	|| fax: +44 (0)1223 334678
Cambridge CB3 0FD, UK	|| web: www.cl.cam.ac.uk/~ap458/
	email: [email-masked]
=========================================================
	Pervasive Computing & Networking Lab.
			(PerLab)