Subject: [manet] Re: performance comparaison between Ad Hoc routing protocols

From: [email-masked]
Date: Wed, 13 Mar 2002 15:50:25 -0800

Thomas,

I checked very thoroughly to make sure we are extracting
information correctly from the ns2 trace files, and we are
definitely doing this correctly (as you described).

I am very curious to know why our simulations differ from yours.
Maybe we are not even running the same scenario, which is why
I sent you our cbr and scenario script files last Friday.
Did you run that scenario yet?  If so, did you get results
that are similar to ours?

Richard

------- Forwarded Message

Message-Id: <[email-masked]>
To: [email-masked], [email-masked]
cc: [email-masked]
Reply-To: [email-masked]
From: [email-masked]
Subject: Re: [manet] Re: performance comparaison between Ad Hoc routing protocols 
In-reply-to: Your message of "Fri, 08 Mar 2002 08:09:06 PST."
             <[email-masked]> 
Date: Fri, 08 Mar 2002 17:01:20 -0800
Sender: [email-masked]
Errors-To: [email-masked]
X-Mailman-Version: 1.0
Precedence: bulk
List-Id: Mobile Ad-hoc Networks  <manet.ietf.org>
X-BeenThere: [email-masked]
Content-Type: text
Content-Length: 7048

Thomas,

So that we can determine why our results differ, I will
provide additional details and scripts, so that you can run
the same experiment I ran.  I will give you the cbr 
and scenario scripts in a separate email.

I will refer to the case with 100 nodes, 20 sources, 670x670 area, 
no mobility, and a packet rate of 6 packets/second/source.  
Each stream begins at a random time between 10 and 40 seconds, and 
changes its destination to a randomly selected node every 30 seconds. 
The size of each data packet is 512 bytes. 
The total number of packets generated in 500 sec is 57261,
which is slightly less than 60000 = 6 * 20 * 500.
Is this similar to what you ran?

Richard

> Thomas,
> 
> > The way in which I extract overhead information from the ns2 tracefiles is
> > to look at all "send" events on the "MAC" layer from the agent
> > "OLSR". This should yield both the number of packets being transmitted as
> > well as the number of bytes, actually reaching the MAC layer and not
> > being dropped somewhere else.
> > 
> > However, when I tried to disregard the layer (MAC, RTR, AGT etc), and just
> > look at all the lines in the tracefile wish an "s" event from "OLSR", it
> > was possible to end up with overhead figures very close those that you
> > cite. I do believe, though, that this is a wrong way of accounting, since
> > (1) control packets, dropped in the protocol stack (and thus not
> > contributing to diffusion of topology information) are accounted as
> > protocol overhead and, (2) control packets are accounted multiple times
> > (each time a layer is sending the packet "downwards" in the stack.)
> 
> We counted packets exactly the same way you did, i.e., 
> only at the MAC layer.  
> 
> To double check, I did the following:
> 
> grep "s " out.tr | grep MAC | grep OLSR   
> 
> and got exactly the same number of OLSR packets
> as our perl script did.
> 
> I did this for the following case:
> 100 nodes, 670x670 area, no mobility, 20 sources, packet rate = 6
> For 500 seconds, there were 154744 OLSR packets totaling 20684584, 
> so the average size of an OLSR packet (including the IP and MAC 
> headers) is 133 bytes. 
> 
> I also would like to find out why our results differ.
> If you have any more ideas, please let me know.
> 
> Regarding the 90-120% increase of overhead for the bug fix, 
> that was for 100 nodes.  I later tried 50 nodes and found
> the increase to be roughly 75-90%.
> 
> Richard
> 
> 
> > Richard,
> > 
> > I suspect serious problems with your control overhead measures. Please see
> > below.
> > 
> > On Wed, 20 Feb 2002 [email-masked] wrote:
> > 
> > > 
> > > Thomas,
> > > 
> > > > Richard,
> > > > 
> > > > I am not quite sure what to make from your simulations. It seems to me
> > > > that they are from a network in overload condition - in which case buff
er
> > > > management has a critical impact on performance. We've been simulating
> > > > OLSR in conditions, similar to those you've been showing, and our resul
ts
> > > > differ significantly. We experience that a large abmount of (data and
> > > > control) traffic in these situations are rejected from the various
> > > > buffers.
> > > 
> > 
> > <SNIP> 
> > 
> > > To ensure that the protocols were compared on a fair basis,
> > > since AODV and TBRPF gave control traffic higher priority than
> > > data traffic, we did the same for OLSR.  It has long been considered
> > > a good idea to give control traffic higher priority.  This could 
> > > explain why your results differ from mine.
> > > You can do this simply by adding the line
> > >                 case PT_OLSR:
> > > to the function PriQueue::recv() in priqueue.cc.
> > > 
> > > > Also, I notice that we obtain much less overhead for OLSR when counting
> > > > MAC/IP overhead. Have you alterated the ns2 code to make it output more
> > > > detailed information (e.g. physical overhead etc.) ?
> > > 
> > > No we did not.  In addition to the 20-byte IP header, ns2 adds 52 bytes 
> > > at the MAC layer.  
> > > 
> > 
> > <SNIP> 
> > 
> > I've tried to reproduce your published results for OLSR, using a large set
> > of scenario files, corresponding to the parameters you've used.
> > 
> > Unfortunately I fail to achieve the same results as those you present
> > (both with and without control traffic priority): in particulary in the
> > overload condition (packet rate of 6-8 packets/second) I find the overhead
> > of OLSR to be significantly lower (up to 25-30%) than what your
> > simulations show.
> > 
> > Also, as we were discussing in another mail, my simulations showed that
> > the "control packet size" bug resulted in approximately 50% more overhead,
> > whereas you cited your simulations as showing in the orders of 90-120%
> > more overhead. I have not had success in reproducing the 90-120% figures 
> > either.
> > 
> > I am basically looking for an explanation on why we have such different
> > results.
> > 
> > I am well aware that a paticulary "bad" or "good" choise of scenario (even
> > corresponding to the same general parameters such as mobility and traffic
> > patterns) may trigger particulary good or bad results, so I've run a
> > number of different scenarios through, all with roughly the same results.
> > This left me wondering if our different results were just a matter of
> > computing the overhead differently?
> > 
> > The way in which I extract overhead information from the ns2 tracefiles is
> > to look at all "send" events on the "MAC" layer from the agent
> > "OLSR". This should yield both the number of packets being transmitted as
> > well as the number of bytes, actually reaching the MAC layer and not
> > being dropped somewhere else.
> > 
> > However, when I tried to disregard the layer (MAC, RTR, AGT etc), and just
> > look at all the lines in the tracefile wish an "s" event from "OLSR", it
> > was possible to end up with overhead figures very close those that you
> > cite. I do believe, though, that this is a wrong way of accounting, since
> > (1) control packets, dropped in the protocol stack (and thus not
> > contributing to diffusion of topology information) are accounted as
> > protocol overhead and, (2) control packets are accounted multiple times
> > (each time a layer is sending the packet "downwards" in the stack.)
> 
> 
> > 
> > -- 
> > 
> > -------------------------------------------
> >   Thomas Heide Clausen
> >   Civilingeniør i Datateknik (cand.polyt)
> >   M.Sc in Computer Engineering
> > 
> >   E-Mail: [email-masked]
> >   WWW:    http://www.cs.auc.dk/~voop
> > -------------------------------------------
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > manet mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/manet
> 
> _______________________________________________
> manet mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/manet

_______________________________________________
manet mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/manet

------- End of Forwarded Message

_______________________________________________
manet mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/manet

---

From: [email-masked]
Date: Fri, 15 Mar 2002 12:59:58 +0100 (CET)

Dear Richard,

I've been running 30 different scenarios, all corresponding to the
parameters you specify and at all data rates between 1 and 10. Over
those simulations, I consistently find that, at the sample point of 6 pps,
the routing load is roughly half that of the simulations you present.
Interrestingly enough, I also get a higher delivery rate: above 90%,
rather than the (roughly) 85% you present.

--thomas

--

---

From: [email-masked]
Date: Fri, 15 Mar 2002 12:59:58 +0100 (CET)

Dear Richard,

--thomas

--

---

From: [email-masked]
Date: Sat, 16 Mar 2002 18:11:25 -0800

Thomas,

> > > > I am basically looking for an explanation on why we have such different

I am also trying to find such an explanation.  I hope that you will
cooperate with me in this effort.

I thoroughly checked to make sure that I am extracting information correctly 
from the ns2 trace files, and I assume you have also.
Also, I am using exactly the OLSR code from your website (with the previously
mentioned bug fix).

The next logical step is to make sure we are using exactly the same scenario 
models.  I ran the experiment I mentioned (without mobility) 
again with a different randomly generated topology, and got approximately 
the same results for overhead and delivery percentage.

(Interestingly, without the bug fix, I *did* get a routing load that
is roughly half as much, and a delivery rate near 92%, but I assume
you have fixed the bug.)

Just because you are using the parameters I specified does not imply
that we are using the same model for cbr traffic.  For example, we could 
be using different methods for selecting the source-destination pairs 
(e.g., the 20 sources are fixed in my simulations),
or we could be starting the cbr traffic at different times,
or maybe one of us made an error in generating the scenario.
Or maybe we are defining the mobile node differently, e.g., using
different MAC parameters.

So as a start, if you will not run the cbr and scenario scripts that
I sent you, please send me one cbr and scenario script that you ran
so that I can run it for comparison.  

In any case, even if we are not using exactly the same scenario models,
I am using the same scenario for the different protocols,
so it is still a fair comparison.

Also, you are welcome to obtain TBRPF source code and run your own
comparisons.  We have already released the code to about 20 people,
including someone from INRIA.

Richard

---

From: [email-masked]
Date: Sat, 16 Mar 2002 18:11:25 -0800

Thomas,

Richard