Subject: [p2prg] P2P simulator survey

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 07 Mar 2011 10:29:54 -0600

Folks: To the best of my knowledge, there isn't a large-scale
survey of p2p simulators today.  There are some papers written
on this subject --- most notably [1] and [2] --- but [1] is
fairly dated and [2] scratches the surface.

Hardware (multicore machines) and software (parallel programming
frameworks) advances have allowed greater flexibility in
simulating large-scale systems today.

I think that having p2prg look deeper into a survey of p2p
simulators may be of interest to both academics as well as
practitioners.

I have started the ball rolling by submitting a document [3] that
gathers some of my thoughts on how to proceed.  The document is
not complete, nor do I claim that it is comprehensive.  To
impart these properties, I would need help from the p2prg
community, especially those of you who are currently working with
simulation frameworks.

Can you please read [3] --- it is a short draft --- and see if
this is of interest to you?  I have requested an agenda slot in
Prague, but it'll be great if we could have a larger discussion
on this (not to mention some volunteer editors) before the Prague
meeting.

Thanks,

[1] Naicken, S., Basu, A., Livingston, B., and S. Rodhetbhai,
     "A Survey of Peer-to-Peer Network Simulators",
     Proceedings of the Seventh Annual Postgraduate Symposium,
     Liverpool, UK, 2006.

[2] Zhang, D., Chen, X., and H. Yang, "State of the Art and
     Challenges on Peer-to-Peer Simulators," in 5th International
     Conference on Wireless Communications, Networking and Mobile
     Computing (Wicom), Beijing, September 2009.

[3] http://www.ietf.org/id/draft-gurbani-p2prg-simulation-survey-00.txt

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{bell-labs.com,acm.org} / [email-masked]
Web:   http://ect.bell-labs.com/who/vkg/

---

From: Matthias Waehlisch <[email-masked]>
Date: Mon, 7 Mar 2011 17:36:38 +0100

Hi Vijay,

  just as side remark: There has been an internet draft on reviewing 
"Tools for Peer-to-Peer Network Simulation", 2006:

  * http://tools.ietf.org/html/draft-irtf-p2prg-core-simulators-00 

Best regards
  matthias

-- 
Matthias Waehlisch
.  Freie Universitaet Berlin, Inst. fuer Informatik, AG CST
.  Takustr. 9, D-14195 Berlin, Germany
.. mailto:[email-masked] .. http://www.inf.fu-berlin.de/~waehl
:. Also: http://inet.cpt.haw-hamburg.de .. http://www.link-lab.net

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 07 Mar 2011 10:46:43 -0600

>    * http://tools.ietf.org/html/draft-irtf-p2prg-core-simulators-00

Matthias: Excellent, thanks for the pointer.  This work is also
dated and looks like it fizzled out.

I will get in touch with the authors and see if they are
interested in contributing to the current work.  Hopefully, we will
have enough interest to finish the current work.

Ciao,

- vijay

---

From: Matthias Waehlisch <[email-masked]>
Date: Mon, 7 Mar 2011 23:09:04 +0100

Hi Vijay,

  I had a quick look on your document: First of all, I agree with you 
that the RG should work further on it.

  It would probably be helpful to add a section (e.g., before Section 3) 
that discusses the problem space: What are important requirements for 
P2P simulators?

  Some minor comments:

  *Introduction*:

  The draft may also be a guidance for people, who want to develop their 
own P2P simulators.

  *Criteria for evaluating simulation frameworks* 

  Support for other models (Churn, ALTO, etc.)

  *List of simulation frameworks*

  There is also PeerfactSim 
(http://peerfact.kom.e-technik.tu-darmstadt.de/), which provides a good 
model for the Internet delay space. This is important for 
proximity-based peer selection.

Cheers
  matthias

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 07 Mar 2011 16:46:05 -0600

> that the RG should work further on it.

Matthias: Great.  Thanks!

> that discusses the problem space: What are important requirements for

I agree; I suspect that as the work progresses, this will become more
clear and we can enunciate it crisply.

>    The draft may also be a guidance for people, who want to develop their

Sure; added to my working copy.

>    Support for other models (Churn, ALTO, etc.)

Ah, interesting.  To which extent are properties like churn a function
of the model itself versus an intrinsic service that the simulator
framework should provide?  In other words, if I look at ns-3, I can
develop a Chord application model and imbue it with a churn rate of
my choosing.  NS-3 does not have to do anything special.  Thoughts?

> proximity-based peer selection.

Cool; added in my working copy.

Before Prague, I will make my working copy available at a public site
if the comments pore in (which I hope they do).  However, I suspect that
having submitted -00 today, the datatracker tool will not allow me to
submit a -01 on the same day, or even during the next week.

Thanks,

- vijay

---

From: Riccardo Bernardini <[email-masked]>
Date: Tue, 08 Mar 2011 10:26:49 +0100

Dear all,
triggered by this draft, I would like to throw in the discussion a  
couple of random thoughts, not directly related with the specific  
draft, but still in the area of influence of P2P network simulations.

I am currently involved in a national project about P2P streaming.  In  
the last meeting we agreed that a problem in this field is that there  
is no "standard" (in a weak sense) methodology to compare performances  
of P2P systems and this makes it difficult to compare different  
solutions.  Therefore, we are trying to find one or more standard "use  
cases" to be used to compare different P2P systems.

The problem is quite complex, both because of the complexity of the   
"boundary conditions" of your system (e.g., number of nodes, churn  
level, network congestion status, ...) and the "performance value" you  
want to measure: what do you measure?  Overall SNR at the receiver?   
Probability of "video freezing"? Variance of SNR?  Start-up times?  
Delay from source to receiver? (clearly those measures are related  
with streaming applications, other applications could use different  
quality figures) .  Moreover, there are some "parameters" that could  
be considered as part of the "boundary conditions" or as part of the  
system to be measured (e.g., network topology).

I am _not_ saying that this type of consideration should be included  
in the draft (they could be included, they could be part of a  
different draft or they could be discarded right away), but maybe the  
two aspects (what p2p simulation systems are available and how they  
should be used) are related.

Just my 0.02 thrown in the discussion.

Greetings,
Riccardo

Quoting "Vijay K. Gurbani" <[email-masked]>:

> _______________________________________________
> p2prg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/p2prg
>

-- 
Riccardo Bernardini
DIEGM -- University of Udine
via delle Scienze 208
33100 Udine
Tel: +39-0432-55-8271
Fax: +39-0432-55-8251

----------------------------------------------------------------------
SEMEL (SErvizio di Messaging ELettronico) - CSIT -Universita' di Udine

---

From: Dr Mario Kolberg <[email-masked]>
Date: Tue, 08 Mar 2011 13:52:07 +0000

Dear Vijay,

I very much support your effort in creating a document surveying P2P 
simulators. At Stirling we have been doing some work on Oversim 
validating overlay models (Chord, Kademlia, EpiChord, Pastry) against 
results from the original papers proposing these overlays.

We would be very happy contributing to your efforts. We agree this would 
be a useful document for the community - the lack of such a document 
this was our initial motivation too.

Best regards,
Mario Kolberg
University of Stirling, UK

-- 
The Sunday Times Scottish University of the Year 2009/2010
The University of Stirling is a charity registered in Scotland, 
 number SC 011159.

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Tue, 08 Mar 2011 09:43:39 -0600

On 03/08/2011 03:26 AM, Riccardo Bernardini wrote:
> couple of random thoughts, not directly related with the specific draft,
> but still in the area of influence of P2P network simulations.

Riccardo: Thank you for your input.  More inline.

[...]
> I am _not_ saying that this type of consideration should be included in
> the draft (they could be included, they could be part of a different
> draft or they could be discarded right away), but maybe the two aspects
> (what p2p simulation systems are available and how they should be used)
> are related.

To a certain extent, p2p file-sharing systems are well understood since
the academic community has looked at them for a good while now.  So, it
is easier to extrapolate the requirements of a simulator for a p2p file-
sharing type of application.  P2P streaming applications, obviously,
have different demands than do static file-sharing ones.  I would love
to include some considerations for a simulator framework to support p2p
streaming-type of applications.  However, as usual it will boil down
to interest (*many* people have lots of this) versus time (*all* people
have much less of this) and expertise (*some* people have lots of this).

If your interest is high, and you have the time and expertise to
contribute insights on what p2p streaming applications would require
of a simulator framework, I would much appreciate working with you to
get some text into the draft (and again, I am not saying we need to do
this before Prague, but by the time the p2prg meets next, we should
take a reasonable stab at this).

Thanks,

- vijay

---

From: Matthias Waehlisch <[email-masked]>
Date: Tue, 8 Mar 2011 16:49:40 +0100

Hi Vijay,

On Mon, 7 Mar 2011, Vijay K. Gurbani wrote:

> develop a Chord application model and imbue it with a churn rate of my 
> choosing.  NS-3 does not have to do anything special.  Thoughts?
> 
  I don't know if I understand it correctly. NS-3 is a discrete event 
simulator, which provides a basic set of functions (mainly handling 
events etc.) to implement different models. To implement churn, NS-3 
does not have to do anything special.

  There are properties that require their own model and that makes only 
sense in a narrowed scenario. For example, in a dedicated P2P simualtor 
we would not expect an extensive layer 2 model, but may expect a 
detailed underlay topology support.

  Btw: Churn is good example that properties may be intricate. Churn may 
result from application/user behaviour or node behaviour (e.g., due to 
node movement and lossy WLAN). This results more or less in the question 
of abstraction: Is it necessary to account for the latter in P2P?

> if the comments pore in (which I hope they do).  However, I suspect 
> that having submitted -00 today, the datatracker tool will not allow 
> me to submit a -01 on the same day, or even during the next week.
> 
  It should be possible to submit a 01 version.

Cheers
  matthias

---

From: Matthias Waehlisch <[email-masked]>
Date: Tue, 8 Mar 2011 16:49:44 +0100

Hi Riccardo,

  finding a standard model and a standard simulation scenario to compare 
different P2P systems is a continuous challenge. And it even gets more 
complicated if you want to establish it as reference model ... Analyzing 
the general "boundary conditions" and "performance measures" for P2P 
systems could be part of a problem statement as it is necessary to 
evaluate the simulators.

  General side remark: A "standard methodology" to compare different 
solutions is not only missing in the context of P2P, but also in the 
broader network community. In contrast to the video coding people, for 
example, we have no real reference models/scenarios. This problem is 
mainly related to researchers, but it could be an interesting symbiosis 
of IETF and IRTF to develop such reference models.

Best regards
  matthias

On Tue, 8 Mar 2011, Riccardo Bernardini wrote:

> Dear all, triggered by this draft, I would like to throw in the 
> discussion a couple of random thoughts, not directly related with the 
> specific draft, but still in the area of influence of P2P network 
> simulations.
> 
> I am currently involved in a national project about P2P streaming.  
> In the last meeting we agreed that a problem in this field is that 
> there is no "standard" (in a weak sense) methodology to compare 
> performances of P2P systems and this makes it difficult to compare 
> different solutions. Therefore, we are trying to find one or more 
> standard "use cases" to be used to compare different P2P systems.
> 
> > [3]
> > http://www.ietf.org/id/draft-gurbani-p2prg-simulation-survey-00.txt
> > 
> > - vijay
> > --

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Tue, 08 Mar 2011 10:13:55 -0600

> does not have to do anything special.

Right, exactly.  The application model running on top of ns-3
will have the required intelligence.

I was merely trying to point out that as we look at the simulation
frameworks, one way to characterize them could be what sort of p2p-like
services they provide intrinsically (i.e., the user does not have to add
the behaviour to the simulator explicitly).

Churn is one such service.  NS-3 will not provide it natively, but
other simulators may (does anyone know if dedicated p2p simulators
provide knobs to control the churn rate?)

Replication is another service that may be provided by a dedicated
p2p simulator but would need to be explicitly put into a model
running over ns-3.

Thanks,

- vijay

---

From: Matthias Waehlisch <[email-masked]>
Date: Tue, 8 Mar 2011 17:19:23 +0100

Hi Vijay,

On Tue, 8 Mar 2011, Vijay K. Gurbani wrote:

> frameworks, one way to characterize them could be what sort of 
> p2p-like services they provide intrinsically (i.e., the user does not 
> have to add the behaviour to the simulator explicitly).
> 
> Churn is one such service.  NS-3 will not provide it natively,
>
  hm, partly: If NS-3 provides a mobility model for the nodes, you can 
evaluate churn conditions (P2P & MANETs).

> but other simulators may (does anyone know if dedicated p2p simulators 
  OverSim, which is P2P framework for OMNeT++, provides several churn 
models. You can control the churn rate for each generator: 
http://oversim.org/wiki/OverSimChurn

Cheers
  matthias

---

From: Dr Mario Kolberg <[email-masked]>
Date: Tue, 08 Mar 2011 16:27:12 +0000

Oversim has a churn generator which can be parametrised in the 
simulation configuration. It supports NoChurn (after all the nodes have 
been added to the network, no nodes leave), LifetimeChurn (a node's 
lifetime is selected randomly based on a given probability), ParetoChurn 
(using heavily tailed Pareto distribution of node lifetimes), 
RandomChurn (selecting a node at random after a fixed interval).

Mario

---

From: "zhangyunfei" <[email-masked]>
Date: Wed, 9 Mar 2011 09:19:03 +0800

Hi Vijay and all,
   I am not familiar with ns-3.I don't know what's new in ns-3 compared with ns-2. So my comments are basically based on ns-2: To my understanding, ns-2 like simulator is not suitable for large-scale user activity simulation. Is there any remedy for this in the research field?

BR
Yunfei

zhangyunfei
2011-03-09

发件人： Vijay K. Gurbani
发送时间： 2011-03-09 00:12:35
收件人： Matthias Waehlisch
抄送： [email-masked]
主题： Re: [p2prg] P2P simulator survey

Thanks,

-  vijay

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Wed, 09 Mar 2011 14:23:54 -0600

On 03/08/2011 07:19 PM, zhangyunfei wrote:
> Hi Vijay and all,
> I am not familiar with ns-3.I don't know what's new in ns-3 compared 
> with ns-2.

Yunfei: See http://www.nsnam.org/docs/ns-3-overview.pdf.

> So my comments are basically based on ns-2: To my 
> understanding, ns-2 like simulator is not suitable for large-scale user 
> activity simulation. Is there any remedy for this in the research field?

Flow level or message level simulators may be more scalable
in the sense that they generate less number of events.  Distributed
simulation is another way to scale.  One of the outputs of the
work under consideration will be to look at the scalability
aspects of the most commonly used simulators.

Thanks,

- vijay

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Wed, 09 Mar 2011 14:34:23 -0600

On 03/08/2011 10:19 AM, Matthias Waehlisch wrote:
> evaluate churn conditions (P2P&  MANETs).

Ah, OK.  That is one way to look at it --- while ns-3 does not
provide the notion of "churn" itself, it does provide the ability to
bring hosts up and down thereby *causing* churn at the DHT layer.
The DHT application model itself can schedule events to bring a
host up or down, thereby inducing churn into the system.

> http://oversim.org/wiki/OverSimChurn

I have not used OverSim, but from what you and Mario describe, it looks
like churn is modeled as a specific attribute in the simulator
itself.

The same can be approximated in ns-3 if we had a Chord model in
ns-3 and the model had a "churn" attribute.  BTW, we have publicly
released a CAN DHT model in ns-3 [1], in case anyone is interested.
We are working to make it scale to 1 million nodes.  We have not
introduced churn into our CAN DHT model yet.

[1] 
http://groups.google.com/group/ns-3-users/browse_thread/thread/b287940942f6998b/0bf00740d43700ee?lnk=gst&q=CAN+DHT#0bf00740d43700ee

Thanks,

- vijay

---

From: Matthias Waehlisch <[email-masked]>
Date: Thu, 10 Mar 2011 00:07:17 +0100

Hi Vijay,

  just for clarification:

On Wed, 9 Mar 2011, Vijay K. Gurbani wrote:

> >    OverSim, which is P2P framework for OMNeT++, provides several 
> > churn models. You can control the churn rate for each generator: 
> I have not used OverSim, but from what you and Mario describe, it 
> looks like churn is modeled as a specific attribute in the simulator 
> itself.
> 
  In OverSim, churn is implemented as a separate model, independent of 
the P2P algorithm. There are different churn generators available. Churn 
generators will be associated with the scenario that is simulated, and 
can be parameterized. In 
this sense churn is a specific attribute of the scenario.

> The same can be approximated in ns-3 if we had a Chord model in ns-3 
> and the model had a "churn" attribute.  BTW, we have publicly released 
> a CAN DHT model in ns-3 [1], in case anyone is interested. We are 
> working to make it scale to 1 million nodes.  We have not introduced 
> churn into our CAN DHT model yet.
> 
  1 million nodes sounds ambitious. OverSim allows for 100,000 nodes, 
which does not hold in all cases, but is quite good from experiences. It 
obviously depends on the complexity of the overall model in use (which 
type of underlying topology, etc.).

Cheers
  matthias

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Wed, 09 Mar 2011 17:22:34 -0600

On 03/09/2011 05:07 PM, Matthias Waehlisch wrote:
> the P2P algorithm. There are different churn generators available.
> Churn generators will be associated with the scenario that is
> simulated, and can be parameterized. In this sense churn is a
> specific attribute of the scenario.

Ah, OK.

> 1 million nodes sounds ambitious.

I agree; we'll see how close we can get.  We have got to 2^14
nodes and have stretched it a bit beyond that.  However, the
time it takes to create the DHT topology increases non-linearly
beyond a certain point.  We are looking into why this is the
case.

Thanks,

- vijay