Subject: Re: [IMRG] New "AvailableBandwidth" Measurement tool]]

From: "Jin Guojun [NCS]" <[email-masked]>
Date: Fri, 22 Aug 2003 12:21:17 -0700

A few points I would like to clear for my previous message.
(1) The high-speed network I mentioned is not 1 Gb/s, but 10 Gb/s
to 1 Tb/s. Current backbone speed is 10Gb/s. 100 Gb/s testbed has
been seen in a couple of years ago.
We have no problem to measure 1 Gb/s paths, but most tools do.

(2) Generic measurement environment vs. specific configuration.
Generic environment is what I think for measurement and development.
That is, buy a $500 PC to run Linux/FreeBSD etc. The measurement
tools should run on any such environment including to use $500,000 machines. 

Using onboard timestamp and Jumbo MTU is not realistic approach
for generic development. This is because:
<1>
Onboard timestamp NIC is very rare. I only know SysKonnect GigE
NICs (v 1.x) has such feature, and they seems to remove this
feature on their new NICs (v 2.0). Also, to use this feature
requires modifying device driver plus either kernel BPF or
L4/IP stack to pass onboard timestamp up to the user layer
for all O.S.s if you want your tool to work everywhere.
I am working with some new vendors to convince them to add
timestamp feature on the new 10Gb NICs, but the final decision
is theirs.

Key: if a tool only works for a specific environment with
very high development (modify kernel) and maintenance cost,
why do not we just buy a dedicate NIC such as DAG board for the 
measurement?

<2> Jumbo Frame will not help. It because that Jumbo has not
standardized yet. Let's assume that 9K MTU is accepted by
everyone in 5 years, it increases single packet size 600%
and it will probably stay at this size for another 15 years.
How much the network bandwidth will increase in this period?
Maybe 1000,000% or more.

The paper that I point to has discussed all these issues and 
issues you mentioned below.

So, we need to differ the generic and specific tool development.

As to the capability for available bandwidth measurement in
the future, it may not be a serious problem because if available
bandwidth is higher than the maximum throughput that a machine can
make, then just send out packets at full NIC speed. If we can detect
that available is lower than the system throughput, then slow it
down.

Depend on the view point, the system issues may or may not be
applied.

	-Jin

-------- Original Message --------
  Date: Fri, 22 Aug 2003 11:50:50 -0500 (CDT)
  From: Vinay Joseph Ribeiro <[email-masked]>
  Subject: [Imrg-dist] Re: [IMRG] New "Available Bandwidth" Measurement tool
  To: <[email-masked]>
  
  Hi all,
  
  	While Jin has made some excellent points, I would beg to differ
  with him on some issues.
  
  The point that "network bandwidth has exceeded system I/O bandwidth" is
  misleading. It gives one the impression that machines connected to Gigabit
  networks cannot read data at 1Gbps (or close to it) which is not quite
  true. The real issue with high-speed NIC cards is that they "coalesce"
  packet arrival interrupts. For example if a NIC receives three 1000byte
  packets spaced 10 micro-sec apart it may send just 1 interrupt to the
  higher layer at the end of the 30 micro-sec interval rather than bother
  the cpu with 3 interrupts one for each packet. This improves efficiency
  and actually IMPROVES the total system I/O bandwidth.
  
  Coalescence clusters packets and so affects packet dispersion techniques
  that expect only cross-traffic to disperse packets.
  
  Several solutions suggest themselves that would allow the use of current
  packet dispersion techniques.
  
  1. Make use of the larger MTU size. While 1000byte packets can be spaced
  as close as 8micro-sec on a Giga network, 8000byte packets will be spaced
  64micro-sec apart. From my preliminary experiments on a Gigabit testbed I
  observed that the NIC cards did not coalesce packets spaced this far
  apart. This solution will require the path MTU to be larger than
  1500 bytes.
  
  A strong case has been made by Phil Dyksta for the use of jumbo frames:
  http://sd.wareonearth.com/~phil/jumbo.html
  
  Another link by Mathis:
  http://www.psc.edu/~mathis/MTU/
  
  2. Modify NIC cards to provide time stamps of when packets arrive at the
  NIC. If this is achieved then it does not matter for how long packets are
  delayed in the NIC due to coalescence. With this solution it will not
  matter if the network speed is 1Gbps or 1Tbps for that matter.
  
  I take this opportunity to advertise "pathChirp", a tool that uses special
  exponentially spaced packet trains to estimate available bandwidth.
  
  http://www.spin.rice.edu/Software/pathChirp/
  
  Please feel free to comment/correct any of my opinions. Thanks.
  
  Vinay Ribeiro
  
  Rice University | [email-masked] | www.ece.rice.edu/~vinay | 713.348.2285

  On Thu, 21 Aug 2003, Jin Guojun [NCS] wrote:
  
  > Since I am not on the mailing list, I do not know how many replies
  > you may already get on it. Here is the link:
  >
  > 	http://www-2.cs.cmu.edu/~hnn/igi/
  >
  > It is a nice paper because it has systematically discussed more
  > details of the technology used IGI/PTR, which is modified Pathload
  > algorithm for efficiency. The problem is the accuracy is lower.
  >
  > Most current bandwidth measurement research has ignored the system
  > capability to handle high-speed network I/O; or the designers have
  > no idea about system capability at all because system issues belong
  > to the engineer. This creates a significant problem on network
  > bandwidth measurement tool development:
  > 10 years ago, we have been able to measure the 100 Mb/s network;
  > however, most new tools are only able to measure 100 Mb/s path now.
  > IT IS USELESS regardless how wonderful the algorithms are.
  >
  > The new problem is toward to two directions in the future --
  >
  > 	accuracy and capability
  >
  > Accuracy is the old problem and capability is the new one
  > because the network speed has exceeded the system I/O bandwidth.
  > All methods except algorithm used in pathchar are requiring the
  > system I/O is higher than the network bandwidth in order to make
  > measurement.
  > Due to this issue, we need to find a very different algorithm to use
  > low-speed packets to measure high-speed network, such as pathchar.
  > Research on the packet-pair-dispersion-based algorithm that requires
  > packets' speed higher than the network available bandwidth and very
  > high resolution system timer has no more valuable to the real world,
  > but the pure research. More information can be found at:
  >
  > 	http://dsd.lbl.gov/DIDC/papers/imc-2003.pdf
  >
  > 100 Gb/s network will be available very soon. If new tools still
  > work only for O(100 Mb/s) network, then, the memory is wasted.
  > I hope to see some new algorithms for 1 Tb/s, but no more tool for
  > O(100 Mb/s) measurement.
  >
  > 	-Jin
  >
  > -------- Original Message --------
  > >>> From: Yeggy Easwaran <[email-masked]>
  > >>> Date: Tue Aug 12, 2003  1:41:43 PM America/Los_Angeles
  > >>> To: [email-masked]
  > >>> Subject: [IMRG] New "Available Bandwidth" Measurement tool
  > >>>
  > >>> Ningning Hu, Peter Steenkiste. Evaluation and Characterization of
  > >>> Available Bandwidth Probing Techniques. In the IEEE JSAC Special
  > >> Issue
  > >>> in Internet and WWW Measurement, Mapping, and Modeling, Vol. 21(6),
  > >>> Aug. 2003
  > >>>
  > >>> talks about a technique called IGI/PTR for measuring available
  > >>> bandwidth. This technique is supposed to measure available bandwidth
  > >>> in a network path faster than Pathload
  > >>>
  > >>>  (``End-to-End Available Bandwidth: Measurement methodology,
  > >> Dynamics,
  > >>> and Relation with TCP Throughput'', in Proceedings of ACM SIGCOMM,
  > >>> August 2002 (also look at an extended version, to appear in the
  > >>> ACM/IEEE Transactions on Networking, 2003).
  > >>>
  > >>> Does anybody have more information/insights on this?
  >
  > _______________________________________________
  > IMRG mailing list
  > [email-masked]
  > https://www1.ietf.org/mailman/listinfo/imrg
  >

_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: Michael Welzl <[email-masked]>
Date: 27 Aug 2003 10:35:36 +0200

To GHPN'ers: the relevant part of this message is the
last, "controversial" comment - what do you think
of this? Should I write something down on using PTP for
Grid-specific network measurements?

(If this mail is too long for you, the quick way is:
 take a look at http://www.welzl.at/ptp and consider its
 usage for network measurements in Grids.)

--------------------------------------------

Hi all,

I have a comment, a question and an additional, probably very
controversial comment.

> standardized yet. Let's assume that 9K MTU is accepted by

... another problem I see with Jumbo Frames is that the
packet size limits your measurement granularity - the larger
your packets, the less precise the result.

Question:

If the limit is not the link capacity but processing abilities
at high speeds (BTW, Jin, I skipped over your paper "system
capability effects on ... measurement", it looks very
interesting - in fact, I have been wondering myself why
nobody ever seemed to care of these things?!), then I assume
that packet header processing can become the main problem.

So, if you send large packets at a high rate, and then
send small packets at the same rate (assuming no background
traffic for simplification), you are perhaps going to see
reduced throughput provided that processing power is the
actual bottleneck, right? My question would be:
has anybody ever tried this (if it makes sense at all)?

Finally, here's the controversial comment:

ANY kind of end-to-end avail-bandwidth measurement method
needs to cause at least mild congestion in order to be
able to measure anything (assuming no knowledge about the
network interior, such as the queuing in use), be it in a
separate tool or used within congestion control as in TCP.
That is, a queue must grow before you can tell that it did.
That's better (earlier) than waiting for packet loss to occur
(thus it has been done in TCP enhancements - ECN, or using
delay: FAST TCP and Vegas), but still, it means that you
need to cause a bit of congestion. I think that claims
of non-intrusiveness for end-to-end avail-bw measurement
tools are false - they could at best be MILDLY intrusive.

>   > low-speed packets to measure high-speed network, such as pathchar.

...although, as Jin rightly pointed out, it may not
be serious because the max. bandwidth a machine can
reach (by any means possible) may indeed be the max.
bandwidth it is interested in.

My controversial opinion is:

I believe that it may be beneficial to consider QUERYING
for performance (traffic) information instead of probing
the network. I devised the "Performance Transparency
Protocol (PTP)" as a means to do efficiently retrieve
performance information from a path a few years ago, and
I showed that it can be very useful for congestion control in
my ph.d. thesis, a revised version of which is now available
from Kluwer ("Scalable Performance Signalling and Congestion
Avoidance").

How does it work?

It retrieves the capacity (link capacity in the case of
droptail queuing and no traffic class separation), a
timestamp, a traffic counter (ifMIB-ifOutOctets) and
the address from each router. After two such packets
(and with each consecutive one), you get an idea of
the available bandwidth at each router (and thus, also
the bottleneck) for the interval between the two packets.

Obvious problems:

* Overhead -> well, is it really more overhead than sending
  enough extra packets to make a queue grow?

* Requires support from ALL routers -> right, but why
  not start to use it SOMEWHERE ... e.g. in support of
  one single specific Grid that is hosted by ISPs a, b and c?

* Share my information??? Never!!! -> well, the capacity
  is just what you would get with packet pair, the traffic
  is just what you could measure by probing the network,
  a timestamp is not that secret, and the address may at
  least not be such a problem in a dedicated, specific
  scenario (Grid example above). It could perhaps also be
  taken care of by introducing other unique keys, like MPLS
  labels - something like that

Obvious advantages:

* A single packet does it all

* No problem if the source system throughput is limited

* No misinterpretations of, e.g., packet loss (corruption
  or congestion?), delay (link layer ARQ or congestion?)

So, is it really such a bad idea? I think not!
I believe that the controlled environment of single,
well-defined Grid scenarios would be a very good
starting point for this.

For more information about PTP (papers, ns code etc.),
please take a look at
http://www.welzl.at/ptp

Cheers,
Michael