Subject: Simulations & Comparisons

From: "M. Scott Corson" <[email-masked]>
Date: Mon, 21 Sep 1998 18:08:23 -0400 (EDT)

CMU folks,

First of all...

Thanks for contributing the ns-2 manet simulation models to the group,
and for taking a first cut at simulating several of the proposed manet 
protocols---it took a lot of effort.

Regarding your mobicom paper...

It appears likely that there are problems with the IMEP/TORA simulation
models, but that is somewhat understandable as the respective I-D's were 
and are still incomplete.

In terms of the performance metrics comparing the algorithms, 
after reading the paper, I find myself wondering about two metrics:

1) Route acquisition latency

2) Data delivery latency

The issue of route acquisition latency has already been raised on 
the list (and subsequently incorporated into the MANET routing protocol
performance issues draft) as a potential negative consequence of 
on-demand (reactive) routing algorithms.  This metric would be very 
useful in comparing various reactive routing algorithms, as well as 
comparing reactive and proactive approaches.

Data delivery latency is one of the most important performance metrics
as it applies to all routing algorithms, and it is not included in the paper.
Along with data throughput (which you have computed in percentage form in
the packet delivery ratio), it is fundamental to comparing routing algorithm
performance.  Without this metric, I do not know how to realistically 
assess routing algorithm performance. 

For the working group effort, I think a protocol performance comparison 
should include these metrics.

An issue for the group...

Once end-to-end data throughput and delay are known, it is still an 
interesting problem to assess which protocol is better for some environment.

For instance, if one has two protocols with the following performances:

Protocol       Throughput      Avg. Pkt. Delay
   A              95%             10 sec.
   B              90%              9 sec.

Which is better?  
What else (other than (1) above) should we consider?

The metric of ``routing power" has been devised as a means of quantifying
the Throughput-Delay relationship by creating a ratio of the two.

                Routing Power = Throughput / Delay

It normalizes throughput with respect to delay.

Here, 

Protocol       Throughput      Avg. Pkt. Delay   Routing Power
   A              95%             10 sec.            9.5%/sec.
   B              90%              9 sec.           10.0%/sec.

Routing Power rates B more highly than A due to B's relatively lesser delay.

Routing Power is not the ultimate measure, but it should be considered 
along with other factors.  For example, from the perspective of assessing 
a routing protocol's effect on transport layer performance, 
routing power probably underpenalizes the effect of
missed network-layer throughput on reliable end-to-end delay (e.g. TCP), 
as missing packets must be retransmitted end-to-end at the higher layer.
But it's probably a reasonable measure for predicting a protocol's effect
on best-effort, end-to-end data delivery performance.

-Scott

*******************************************************************************

M. Scott Corson                                 Institute for Systems Research
[email-masked]                              A.V. Williams Bldg. (115)
Phone: 301-405-6630 (Rm. 2205 AVW)              University of Maryland     
Fax:   301-314-8586                             College Park, MD  20742

*******************************************************************************

---

From: "M. Scott Corson" <[email-masked]>
Date: Mon, 21 Sep 1998 23:01:50 -0400 (EDT)

> Routing Power rates B more highly than A due to B's relatively lesser delay.

On second reading, RP is not all that useful either...sigh... ;-(, 
as it tries to quantify a trade-off which is inherently application-specific.

Being in a good throughput-delay trade-off cannot necessarily be indicated by 
their ratio.  These two algorithms with T/D metrics

100kbps / 1 sec = 100 (measured in Rout. Power)
10kbps / .1 sec = 100 (measured in Rout. Power)

give the same RP, but either might be better-suited for a given application
depending on its throughput and delay requirements.

So...as a group, what do we want to target in terms of initial performance 
trade-offs?  

Do we want to primarily maximize network-layer throughput whenever possible 
(to minimize the need for end-to-end retransmissions), and secondly minimize
latency?  This would seem to be better-suited for supporting 
file transfer and the like.  Is that a reasonable near-term goal?  
If not...what?  Should such be added to the applicability statement 
for each protcoll...indicating perhaps the types of applications that 
might be favored by a given approach?  

There was significant discussion concerning an applicability statment
during the meeting, but very little since I posted that initial cut.
Is everyone happy with the version I sent a week or so ago?

-Scott