Subject: Re: performance comparaison between Ad Hoc routing protocols

From: [email-masked]
Date: Wed, 13 Feb 2002 11:28:04 -0800

On Thu, 7 Feb 2002 Mahesh Marina wrote:
 
> Your simulation results for AODV somehow were way off from what I have
> observed in the past. I was one of the people involved in the development
> of AODV code for ns2. 
> 
> You mention that the link-layer notification was turned off in your
> evaluations. I assumed that HELLOs were instead used for link
> detection/failure in AODV as well and looked at the piece of code for
> handling HELLOs. 

> I found that neighbor deletion (and therefore, link failure
> detection) was commented out inadvertently, perhaps during
> debugging. As most of our performance studies used link-layer feedback,
> this bug didn't show up. So, I was wondering whether AODV results would be
> same without this bug.

I reran the ns-2 simulations for AODV with the above fix,
which improved the performance of AODV substantially in the 
mobile scenarios (but not in the non-mobile scenarios).
To ensure fairness with TBRPF and OLSR, I also commented out
the following code in AODV so that no protocol is allowed
to retrieve packets from the interface queue (ifqueue):

       // Not sure whether this is the right thing to do
       Packet *pkt;
       while((pkt = ifqueue->filter(ih->saddr()))) {
			drop(pkt, DROP_RTR_MAC_CALLBACK);

(The above comment is by one of the AODV programmers, not me.)

The simulation results can be found at the TBRPF web site: 
http://www.erg.sri.com/projects/tbrpf/

The results are summarized below.  Comments are welcome.
Note that TBRPF code for ns-2.1b8 is now available.

Protocols compared:

   - AODV code for ns-2.1b8
     Available from http://www.cs.ucsb.edu/~eroyer/aodv.html

   - TBRPF code for ns-2.1b8
     Available from http://www.erg.sri.com/projects/tbrpf/sourcecode.html
     Uses the same 'C' code as our Linux implementation

   - OLSR code for ns-2.1b7
     Available from http://hipercom.inria.fr/olsr/
     Bug affecting packet size was fixed.

   - In all protocols, link-layer failure notification was NOT used, and 
     packets could not be retrieved from the link layer (interface queue). 

Simulation model:

  - ns-2 version 2.1b8
  - WaveLAN IEEE 802.11 MAC with rate 2Mb/s and range 250m
  - 50 and 100 nodes
  - Two area sizes: 670m x 670m and 1500m x 300m
  - Mobility model: No mobility, and random waypoint model with 0 pause 
    time and maximum speed 20 m/s.
  - 20 simultaneous CBR traffic streams:
    - Source and destination of each stream are selected randomly 
    - Duration of each stream is 30 seconds
    - Size of each data packet is 512 bytes
    - Packet generation rate per stream is increased from 2 to 8 packets/s
  - Each simulation was run for 500 simulated seconds.

Summary of results:

  - In every scenario, TBRPF achieved a higher delivery percentage (up to 
    15% higher) than OLSR.  TBRPF also achieved a higher delivery percentage 
    (up to 15% higher) than AODV in all scenarios with no mobility, and in 
    all scenarios using the square area (670x670) with the lower packet rates 
    (2 and 4 packets/s).  For the long rectangular area (1500x300), AODV 
    achieved a higher delivery percentage (up to 5% higher) than TBRPF.

  - In every scenario, TBRPF generated less routing control traffic
    than the other protocols: up to 60% less than OLSR and up to 48% 
    less than AODV.  This is despite the fact that TBRPF sends
    HELLOs twice as frequently as OLSR.

  - In every scenario, TBRPF used the shortest paths (except nearly shortest 
    in some cases with the higher packet rates).  In every scenario, AODV 
    used paths that were 12-20% longer on average than TBRPF.

  - TBRPF usually had the best or nearly best delay.  

Remarks:

  - This study did not use link-layer failure notification.  It is important 
    to also compare the protocols with link-layer notification, since this 
    allows one to achieve a delivery fraction close to 100%.

  - A comparison of TBRPF with AODV using link-layer notification can
    also be found at the TBRPF web site.  That comparison also compares
    the protocols with 40 sources (which gives TBRPF more advantage).

  - A routing protocol that generates less control traffic and uses 
    shorter paths:

    - uses less energy,
    - is more likely to scale to a larger number of nodes,
    - is more likely to perform well in lower bandwidth radios such
      as those used by the military.

-----------------------
Richard Ogier
Sr. Research Engineer
SRI International
333 Ravenswood Ave.
Menlo Park, CA 94025
Tel: 650-859-4216
Fax: 650-859-4812
Email: [email-masked]
------------------------

---

From: [email-masked]
Date: Wed, 20 Feb 2002 11:20:08 +0100 (CET)

Richard,

I am not quite sure what to make from your simulations. It seems to me
that they are from a network in overload condition - in which case buffer
management has a critical impact on performance. We've been simulating
OLSR in conditions, similar to those you've been showing, and our results
differ significantly. We experience that a large abmount of (data and
control) traffic in these situations are rejected from the various
buffers.

Dropping control-traffic in the various buffers due to overload should in
fact result in (and, indeed, does in our simulations) *less* control
traffic being sent on the medium, not more - as suggested by your results.
A solution, obviously, is to use priority queues and to prioritize control
traffic. However this is approaching an implementation issue, and is not
something included in our code. So I am puzzled how you get the figures
showing an increasing control traffic overhead when increasing the
overload condition.

Also, I notice that we obtain much less overhead for OLSR when counting
MAC/IP overhead. Have you alterated the ns2 code to make it output more
detailed information (e.g. physical overhead etc.) ?

Please keep in mind that the ns2-code (version 3) of OLSR does not include
features such as link hysteresis and loss link management for mobility
conditions. I thins this is the case for AODV (version 2) as well
(features not fully implemented, new features not present).

Finally, I've been looking at your graphs, and I think that achieving
optimal performance (for either of the manet protocols) is largely a
matter of tuning protocol parameters. I noticed, for example, that the
paraeters you used for tbrpf in the simulations differ from those
specifyed in the draft.

--thomas

-- 

-------------------------------------------
  Thomas Heide Clausen
  Civilingeniør i Datateknik (cand.polyt)
  M.Sc in Computer Engineering

  E-Mail: [email-masked]
  WWW:    http://www.cs.auc.dk/~voop
-------------------------------------------

---

From: [email-masked]
Date: Wed, 20 Feb 2002 11:09:50 -0800

Thomas,

> control) traffic in these situations are rejected from the various

I considered four packet rates: 2, 4, 6, and 8 packets/s.  
The latter two are indeed usually an overload condition 
(except that TBRPF sometimes did very well with 6 packets/s),
but 2 packets/s is certainly NOT.

> showing an increasing control traffic overhead when increasing the

To ensure that the protocols were compared on a fair basis,
since AODV and TBRPF gave control traffic higher priority than
data traffic, we did the same for OLSR.  It has long been considered
a good idea to give control traffic higher priority.  This could 
explain why your results differ from mine.
You can do this simply by adding the line
                case PT_OLSR:
to the function PriQueue::recv() in priqueue.cc.

> detailed information (e.g. physical overhead etc.) ?

No we did not.  In addition to the 20-byte IP header, ns2 adds 52 bytes 
at the MAC layer.  

> (features not fully implemented, new features not present).

Yes. Please let me know when OLSR code is available that has these 
new features.

Regarding hysteresis, this is a feature that can be used by either
OLSR or TBRPF.  For example, TBRPF can use the same EWMA hysteresis 
technique used by OLSR if it helps performance.

> paraeters you used for tbrpf in the simulations differ from those

We also tried using the parameter values in the draft, but the difference 
in performance was only slight.  Note that we used the same parameter
values for all scenarios, so we did not tune the parameters to each
scenario.  

Richard