Subject: [ippm] EVALUATING TOOLS

From: Victor Reijs <[email-masked]>
Date: Thu, 30 Aug 2001 16:40:08 +0100

Hello all of you,

Some time ago a great overview of evaluating items for capacity tools
was presented by Constantinos. I have added a few and made it more
general. So if people have comments/additions, please let me know. Below
it is.

Thanks.

All the best,

Victor

-----
6. ANNEX I
This overview presents issues important for evaluating measurements
tools, based on the initial list of Constantinos Dovrolis, from the
Internet2 IPPM working group.
1. What is the name and version of the tool?
2. What does the tool attempt to measure (capacity, available bandwidth,
BTC, RTT, OWD, etc.)?
3. What protocols can be supported (such IPv4, IPv6, etc.)?
4. On what measurement principle is the measurement based? E.g. with
bandwidth tools: Is it based on the dispersion of packet pairs/trains,
on the one-way delay variations of variable-sized probing packets, on
actual TCP transfers (e.g., BTC measurements) or on emulated TCP flows?
- Is it using active or passive measurements?
- Does it use information from the routers (i.e., SNMP-based tools like
MRTG), or is it based on end-point measurements?
- In case of active measurements: Does the tool require access (and
software) at both ends of the path? 
- Does it measure estimates for each circuit in the path, or at an
end-to-end basis?
- How does the tool handle parallel-circuits in the path (such as
channelized ethernet, OC-3c)?
- Can the reverse path affect the measurements?
- Does it require ICMP replies from systems in the path?
- For some tools: Is it robust to cross traffic? In other words, can it
make accurate measurements even when the path is heavily loaded?
- Can the tool utilize QoS bits (like TOS/DSCP)?
- Is the tool loading the path with the traffic over the duration of the
measurements? And at what level? What is the duration of the
measurement?
- What is the influence of the run-time parameters on the overall
measurement output (configuration parameters such as: packet size,
number of probes, etc.)?
- Does it take user-level or kernel-level timestamps? 
Time stamping accuracy and resolution are major issues in measuring
high-speed paths. Resolution is not a problem these days (1 msec is
typical). Accuracy, though, can be a major problem in both kernel and
user-level timestamps (more in the later), and especially when the
measuring host is not totally idle.
- Does it require access at the raw-IP socket (i.e., need special user
privileges)?
5. What is the granularity of the tool? 
Network managers need very granular measurement (like msec for OWD), but
users need only msec granularity.
6. What is its accuracy? 
Obviously, the background can influence the results in a statistical
way. This means that there is no guarantee that they will give you the
right value every time you run them. The possibility of error is always
there in every area of measurements and experimentation. However, do
they give the right estimate "most of the time"?
7. Does the tool calculate (correct) experimental errors?
8. Do hidden Layer-2 switches and/or other store-and-forward devices
affect the measurements?
9. On which operating systems does the tool run?
10. Are there known bugs and limitations?
11. Is the tool actively maintained/developed?
_______________________________________________
ippm mailing list
[email-masked]
http://mailhost.advanced.org/mailman/listinfo/ippm