Subject: [Tmrg-interest] new versions of TMRG internet-drafts submitted
From: [email-masked] (Sally Floyd)
Date: Fri, 14 Oct 2005 13:20:48 -0700

I have submitted new versions of 
  Metrics for the Evaluation of Congestion Control Mechanisms
and 
  Metrics for the Evaluation of Congestion Control Mechanisms,
with only small changes from the most recent versions
that had been posted on the TMRG web site.

They are available at:
http://www.icir.org/tmrg/draft-irtf-tmrg-metrics-01.txt
http://www.icir.org/tmrg/draft-irtf-tmrg-tools-01.txt

Because I will be on vacation for the next two weeks, I will
probably not be responding to email about them until
November.

Regards,

- Sally
http://www.icir.org/tmrg/

From: [email-masked] (Sally Floyd)
Date: Thu, 13 Oct 2005 16:53:37 -0700
Subject: [Tmrg-interest] I found a volunteer for the tmrg mailing list!
Message-ID: <[email-masked]>

Yeah!

Sushant Rewaskar from UNC Chapel Hill has volunteered to take care
of this while I am gone.

- Sally

From: [email-masked] (Sally Floyd)
Date: Thu, 13 Oct 2005 16:27:25 -0700
Subject: [Tmrg-interest] Looking for a volunteer to help handle the tmrg mailing list
Message-ID: <[email-masked]>

I will be out of town for the next two weeks (in South Africa on
vacation!), and am looking for a volunteer to help handle the
tmrg mailing list while I am gone.  

This is a very low-overhead job, and means logging on to the mailing
list web site once a day as administrator.  So far, it has been
sufficient to say yes to all the requests to join the mailing list,
and to say no to all email sent to the mailing list from non-members,
so it is not very hard.  The rate of requests is low.  If someone
could do this for the next two weeks (and occasionally afterwards,
if they were so inclined), I would appreciate it.

Thanks.

- Sally

From: [email-masked] (Sally Floyd)
Date: Thu, 13 Oct 2005 16:21:12 -0700
Subject: [Tmrg-interest] metrics for packet drop rates
Message-ID: <[email-masked]>

>There seem to exist metrics at different levels.
>a) user perceived metrics
>b) measurable per-flow metrics
>c) flow statistics
>d) network statistics (that are not per-flow)
>
>It seems the current "metrics" Internet-draft talks about a combination
>of (b), (c) and (d).  Perhaps it is helpful to clarify their type as we
>define them.

I will try to do that.

...
>I find the current "metrics" ID too brief in discussing the methodology
>for evaluation.  I think we need to feed the following parameters into
>the "methodology":
>- network topology
>- workload (flows, their application type and their routes, and traffic
>  intensity if dynamic load)
>- which traffic control used for which flow
>The "methodology" can be a simulation model.  For simplified situations,
>it would be good if we can have some analytical/numerical solutions too.

I agree with you completely that all of those parameters have to be
considered in methodology.  However, the "metrics" ID is not supposed
to be about methodology at all.  It is only a high-level discussion
of the set of metrics to be considered.  I am assuming that 
separate documents will be written on scenarios, e.g.,

* Scenarios for the general evaluation of transport protocols;
* Scenarios for the evaluation of high-bandwidth protocols;
* Scenarios for the evaluation of AQM mechanisms;
* Scenarios for the evaluation of transport protocols over wireless links;
* Etc.

Where each document is not a final word, but an attempt to capture
best current practice as it exists so far.

And these scenarios will use some of the tools from the "tools" ID,
e.g., about the range of RTTS, of connection sizes, two-way traffic,
and the like.  The scenarios will include topology, workload, traffic
control, and the like.

Then *with* or *after* the scenario documents, some more methodology
issues will need to be considered.  E.g., that each simulation
should be run a number of times; statistical tools; interpreting
results; and the like.

And issues that are contentious or unresolved can have their own
series of short drafts by people expressing opinions, e.g., on the
use of utility functions or user-based metrics for the evaluation
of congestion control mechanisms, perhaps.

Does that seem reasonable?

- Sally
http://www.icir.org/tmrg/

From: [email-masked] (Dah Ming Chiu)
Date: Thu, 13 Oct 2005 00:05:38 +0800
Subject: [Tmrg-interest] metrics for packet drop rates
References: <[email-masked]>
Message-ID: <[email-masked]>

Hi,

There seem to exist metrics at different levels.
a) user perceived metrics
b) measurable per-flow metrics
c) flow statistics
d) network statistics (that are not per-flow)

It seems the current "metrics" Internet-draft talks about a combination
of (b), (c) and (d).  Perhaps it is helpful to clarify their type as we
define them.

If you measure TCP (from the socket interface), you don't
see packet drop rates.  The per-flow metric you can measure is "time to
transfer N bytes", and this is a perfectably good and adequate per-flow
metric for classic elastic applications such as ftp.  Packet drop rate,
number
of retransmissions etc are flow statistics you have to measure from a
simulation model, or instrumented stack.

If the application is a multimedia one, then other (per-flow) metrics
become of interest:
- packet loss rate
- delay
- interpacket arrival distribution (jitter)
- connection blocking probability (if admission control used)
- packet blocking/queueing
when you send packets at a constant rate.

You can measure these metrics for a given traffic control in use. If you
used TCP, you would not see packet loss rate, or connection blocking;
but the effects are rolled into jitter and packet blocking rates.
If you use UDP, or UDP+RTP, you will see some of the other metrics
such as packet loss rate.

So how can we compare different traffic controls if we have multiple
metrics? That's when (a) comes in.  A user level metric combines
multiple measurable metrics into an user level index. The MOS and
eModel are such user level indexes.  There is some research aimed
at coming up with formulas to map flow-level metrics into such user
level metrics (e.g. work by Rubino et al at INRIA). It is probably quite
hairy as Sally feared.  As consider various utility functions in analytical
studies another (abstract) form of user level metric (e.g. concex vs
non-convex utility functions).

Ultimately, if we are interested in comparing different traffic controls
for different applications in different scenarios, then we need to consider
the aggregate (sum) of user level metrics for all flows.  If we care about
"fairness", we may also need to look at the distribution of the user
level metrics for all flows.  In this sense, I agree with Dado - if our
interest is in evaluating different traffic controls, then we are going to
use (a) and (b) type of metrics.  The type (c) and (d) statistics can be
useful for improving the design of traffic controls, but not for comparing
them.

I find the current "metrics" ID too brief in discussing the methodology
for evaluation.  I think we need to feed the following parameters into
the "methodology":
- network topology
- workload (flows, their application type and their routes, and traffic
  intensity if dynamic load)
- which traffic control used for which flow
The "methodology" can be a simulation model.  For simplified situations,
it would be good if we can have some analytical/numerical solutions too.

I hope this long-winded email help clarify some points, or at least stur
some more discussion.

Regards
Dah Ming Chiu

ps some of these ideas are discussed in a recent paper (by Chiu and Tam)
    on my webpage.

----- Original Message ----- 
From: "Sally Floyd" <[email-masked]>
To: "Dado Colussi" <[email-masked]>
Cc: "Janardhan Iyengar" <[email-masked]>;
<[email-masked]>
Sent: Wednesday, October 05, 2005 2:58 AM
Subject: Re: [Tmrg-interest] metrics for packet drop rates

> >ultimately, only the effect of packet drops on the application matters.
> >Or even further, only the effect on user perception and satisfaction
> >matters. In this regard, the whole discussion of packet drops, as a
> >metric on transport level, is questionable.
>
> Well, I guess I disagree.
>
> If the *only* way to evaluate congestion control mechanisms was in
> terms of user perception and satisfaction for the end-user, this
> would be pretty unwieldly approach.   And I don't think that approach
> would lend much insight into the performance of different congestion
> control mechanisms.  My assumption is that the effect of congestion
> control mechanisms on user perception and satisfaction are usually
> mediated by the per-flow throughput, delay, and packet drops.
>
> (Not to mention the fact that in some cases, for congestion control
> mechanisms such as AQM mechanisms at the router, the immediate user
> might be the person buying VPN services from the ISP, and in this case
> that user cares about numbers about throughput, delay, and drop
> rates, I believe.)
>
> - Sallly
> _______________________________________________
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>

From: [email-masked] (Damon Wischik)
Date: Fri, 7 Oct 2005 10:05:00 +0100 (BST)
Subject: [Tmrg-interest] metrics for packet drop rates
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

> For TCP-mediated applications flows, one user-perception metric is user
> wait time (e.g., waiting for a web page to download or a file to
> transfer).  Back during my e-commerce days, our chief architect gave us
> an engineering criterion that a web page should download in no more than
> 10 seconds (preferably 1-2 seconds), and he based this on some study
> (sorry I don't know the reference) which said that most users abandon a
> page download after 11 seconds.
> So let's establish 10 seconds, or more generally X seconds, as a
> user-perception metric, the upper bound on user wait time.

But a web page is made up of many pieces. Do you want the wait time for 
the main html file to be 10 seconds, or do you want the wait time for the 
entire web page to be 10 seconds? (I'm thinking of cartoon web pages, 
where the cartoon is a gif which appears seconds after the frame.)

I remember some work from Cambridge, the gist of which was (by my fading 
memory): "Most web pages consist of multiple small items (just a few 
packets long); most web browsers allow four concurrent connections; 
therefore what matters most is whether four of these small items have gone 
into timeout".

"The Effect of Early Packet Loss on Web Page Download Times"
James Hall, Ian Pratt, Ian Leslie and Andrew Moore 
http://www.cl.cam.ac.uk/Research/SRG/netos/nprobe/publications/pam2003hall.html

This view says that web page performance is dominated by the timeout 
behaviour of short flows. That's a function of packet loss probability. So 
I think the proper user-centric metric here may just be packet loss 
probability! (also, it matters of course whether short flows see the same 
packet loss probability as do large flows.)

Damon.

From: [email-masked] (Moore, Sean (Sean))
Date: Thu, 6 Oct 2005 20:47:49 -0400
Subject: [Tmrg-interest] metrics for packet drop rates
Message-ID: <[email-masked]>

I made an attempt at looking at this user-perception metric
constructively - maybe my heuristic analysis of the issue will be useful
to the group.

The general approach is to define a user perception/satisfaction metric
or metrics and then propagate the metric down to evaluation metrics for
congestion control.  

For TCP-mediated applications flows, one user-perception metric is user
wait time (e.g., waiting for a web page to download or a file to
transfer).  Back during my e-commerce days, our chief architect gave us
an engineering criterion that a web page should download in no more than
10 seconds (preferably 1-2 seconds), and he based this on some study
(sorry I don't know the reference) which said that most users abandon a
page download after 11 seconds.  

So let's establish 10 seconds, or more generally X seconds, as a
user-perception metric, the upper bound on user wait time.  The metric
is independent of the size of the page - users don't know and/or don't
care about the volume of traffic associated with a page, they just want
it to download quickly.  The underlying TCP connections either have
enough available/effective bandwidth to transfer the information within
X seconds or not.  If not, then there are two methods I can think of to
solve the problem.  

One method is to distribute, or load balance, the TCP flows across
multiple different paths in such a way that the flows don't all share
the same bottleneck and have enough bandwidth to meet the metric.  This
method is not practical using generally available networking technology,
and even if it were, the solution is independent of the underlying
congestion congestion control mechanisms.  

The other method is to create a tunnel that is composed of multiple TCP
connections into which are multiplexed the TCP flows that transport the
page data.  The number of TCP connections composing the tunnel can be
dynamically adjusted to maintain a constant goodput, the value of which
may be computed from the volume of data and the metric value.  Such
tunnelling technology has been described as "elastic TCP" by a Boston
University team and has also been reported as a solution for
transporting streaming video over 802.11 WLAN.  But again, this solution
is independent of the underlying congestion control mechanism.  The
congestion control mechanism may have a small effect on the number of
tunnelled TCP connections needed to maintain constant gooduput, but in
general the number of TCP connections is not of any concern (unless
there are so many that host resources could be oversubscribed - but I
consider that to be an exceptional situation).

Hence, I can't see a straightforward method for mapping this user
perception metric to the evaluation of congestion control mechanisms.
Maybe someone else can see a map or can define a different user
perception metric that does map directly to evaluation of congestion
control mechanisms.

- Sean Moore

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Dado Colussi
Sent: Wednesday, October 05, 2005 5:30 AM
To: Sally Floyd
Cc: [email-masked]
Subject: Re: [Tmrg-interest] metrics for packet drop rates

Hi Sally,

I had cut my point quite short, so let me try to explain it with broader
words.

Sally Floyd wrote:
>>ultimately, only the effect of packet drops on the application
matters. 
>>Or even further, only the effect on user perception and satisfaction 
>>matters. In this regard, the whole discussion of packet drops, as a 
>>metric on transport level, is questionable.
> 
> 
> Well, I guess I disagree.
> 
> If the *only* way to evaluate congestion control mechanisms was in
> terms of user perception and satisfaction for the end-user, this
> would be pretty unwieldly approach.   And I don't think that approach
> would lend much insight into the performance of different congestion
> control mechanisms.  My assumption is that the effect of congestion
> control mechanisms on user perception and satisfaction are usually
> mediated by the per-flow throughput, delay, and packet drops.

Ok, this is how I see the matter. I understand performance as the degree

to which a transport protocol delivers a service. The service can be 
mapped to the well-being of an application, e.g., through an application

specific utility function. There are many different applications and, 
thus, many different utility functions. Events in the transport layer 
impact the application in an application specific way defined by the 
utility function. The well-being of an application is in close relation 
to user satisfaction, I believe. This relation, however, is probaly 
extremely hard to model and depends on each user. But the relation does 
exist.

Application well-being is indeed affected by per-flow throughput, delay 
and packet drops, but not all applications are affecteb by all of them. 
For example, TCP makes packet drops unperceivable. An application using 
TCP perceives a packet loss as increased delay due to recovery and 
retransmission. The impact of a packet drop to such an application 
depends on the speed of recovery (assuming the application cares for low

delay). Then, the packet drop itself has little value as a metric, I
think.

I would argue that an elephant flow application (e.g. FTP/TCP) depends 
on throughput but not delay or packet drops. A mouse application (e.g. 
HTTP/TCP) depends on throughput and delay but not packet drops. A 
real-time application (e.g. VoIP/UDP) depends on throughput, delay and 
packet drops. With UDP, the packet drops are perceivable and the impact 
depends on the way the application can handle data loss.

When evaluating transport protocols, we should, I think, assume a set of

applications being served and study how the transport protocol delivers 
the service. A packet drop is not an absolute metric but depends on the 
assumed set of applications being served. This is why I question the 
approach of using a fixed transport layer packet drop metric.

You probably still disagree. ;)

> (Not to mention the fact that in some cases, for congestion control
> mechanisms such as AQM mechanisms at the router, the immediate user
> might be the person buying VPN services from the ISP, and in this case
> that user cares about numbers about throughput, delay, and drop
> rates, I believe.)

Can't that person buying the VPN service be modelled as a utility
function?

Cheers,
Dado

_______________________________________________
Tmrg-interest mailing list
[email-masked]
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest

From: [email-masked] (Sally Floyd)
Date: Wed, 05 Oct 2005 14:21:46 -0700
Subject: [Tmrg-interest] Interesting Draft Document on "TCP Benchmarking"
Message-ID: <[email-masked]>

David -

>Our understanding is that the benchmark suite is to answer three 
>questions for different groups of people, in three corresponding steps:

>1. "How robust is the design in various situations?" or "Did I miss anything 
>in my new design?"
>-- from new algorithm designers

>We want to help the designers to verify their design and implementation. For 
>this purpose, we need a (sub)set of clean scenarios that focus on each 
>individual property of a protocol. 

I would agree that for a designer to verify their own implementation,
very simple scenarios can be needed.  And that this is something that
TMRG is not attempting to do.

However, this is different than answering "How robust is the design in
various situations?".   That is something that TMRG is hoping to
address, as is mentioned in the paragraph below.  

...
>2. "How does this algorithm perform in the most common scenarios?"
>-- from researchers in the network community who want to evaluate the 
>protocols

>I guess this is probably what we currently emphasize in TMRG. 

My hope would be that the best current practice scenarios that
are put together for TMRG would include a wide range of scenarios
needed to evaluate the performance of congestion control mechanisms.
This would include not only common scenarios, but also edge
cases, to illustrate the limits of different congestion control
mechanisms.  (Edge cases could include very high congestion; many
transient changes in load; more UDP traffic than usual; challenging
wireless scenarios; and the like.)

...
>We agree that 
>heavy-tail file size distribution and two-way traffic are probably more 
>common in Internet. 

I would go one step further, and say that I haven't seen any data
yet from a network that doesn't have *some* degree of two-way data
traffic, and *some* medium-size transfers that are capable of causing
the burstiness that comes from slow-start.  But there could easily
be measurement data out there that I haven't seen yet.  I would be
interested.  E.g., do networks with HEP traffic really have only
long-lived flows, and web mice of ten packets or less?

>3. "Which algorithm shall I use for my environment/operation?"
>-- from our end-users: system administrators and engineers in various 
>scientific or industrial groups that heavily use TCP, such as High Energy 
>Physic groups, distributed storage operators and etc

...
>Probably, we should also get information from 
>the end users and feed back their usage patterns to step 2 and generate some 
>other "realistic scenarios". 

Yep, that would be very good, I think.

- Sally

From: [email-masked] (Dado Colussi)
Date: Wed, 05 Oct 2005 12:30:18 +0300
Subject: [Tmrg-interest] metrics for packet drop rates
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Hi Sally,

I had cut my point quite short, so let me try to explain it with broader 
words.

Sally Floyd wrote:
>>ultimately, only the effect of packet drops on the application matters. 
>>Or even further, only the effect on user perception and satisfaction 
>>matters. In this regard, the whole discussion of packet drops, as a 
>>metric on transport level, is questionable.
> 
> 
> Well, I guess I disagree.
> 
> If the *only* way to evaluate congestion control mechanisms was in
> terms of user perception and satisfaction for the end-user, this
> would be pretty unwieldly approach.   And I don't think that approach
> would lend much insight into the performance of different congestion
> control mechanisms.  My assumption is that the effect of congestion
> control mechanisms on user perception and satisfaction are usually
> mediated by the per-flow throughput, delay, and packet drops.

Ok, this is how I see the matter. I understand performance as the degree 
to which a transport protocol delivers a service. The service can be 
mapped to the well-being of an application, e.g., through an application 
specific utility function. There are many different applications and, 
thus, many different utility functions. Events in the transport layer 
impact the application in an application specific way defined by the 
utility function. The well-being of an application is in close relation 
to user satisfaction, I believe. This relation, however, is probaly 
extremely hard to model and depends on each user. But the relation does 
exist.

Application well-being is indeed affected by per-flow throughput, delay 
and packet drops, but not all applications are affecteb by all of them. 
For example, TCP makes packet drops unperceivable. An application using 
TCP perceives a packet loss as increased delay due to recovery and 
retransmission. The impact of a packet drop to such an application 
depends on the speed of recovery (assuming the application cares for low 
delay). Then, the packet drop itself has little value as a metric, I think.

I would argue that an elephant flow application (e.g. FTP/TCP) depends 
on throughput but not delay or packet drops. A mouse application (e.g. 
HTTP/TCP) depends on throughput and delay but not packet drops. A 
real-time application (e.g. VoIP/UDP) depends on throughput, delay and 
packet drops. With UDP, the packet drops are perceivable and the impact 
depends on the way the application can handle data loss.

When evaluating transport protocols, we should, I think, assume a set of 
applications being served and study how the transport protocol delivers 
the service. A packet drop is not an absolute metric but depends on the 
assumed set of applications being served. This is why I question the 
approach of using a fixed transport layer packet drop metric.

You probably still disagree. ;)

> (Not to mention the fact that in some cases, for congestion control
> mechanisms such as AQM mechanisms at the router, the immediate user
> might be the person buying VPN services from the ISP, and in this case
> that user cares about numbers about throughput, delay, and drop
> rates, I believe.)

Can't that person buying the VPN service be modelled as a utility function?

Cheers,
Dado

From: [email-masked] (Sally Floyd)
Date: Tue, 04 Oct 2005 11:58:10 -0700
Subject: [Tmrg-interest] metrics for packet drop rates
Message-ID: <[email-masked]>

>ultimately, only the effect of packet drops on the application matters. 
>Or even further, only the effect on user perception and satisfaction 
>matters. In this regard, the whole discussion of packet drops, as a 
>metric on transport level, is questionable.

Well, I guess I disagree.

If the *only* way to evaluate congestion control mechanisms was in
terms of user perception and satisfaction for the end-user, this
would be pretty unwieldly approach.   And I don't think that approach
would lend much insight into the performance of different congestion
control mechanisms.  My assumption is that the effect of congestion
control mechanisms on user perception and satisfaction are usually
mediated by the per-flow throughput, delay, and packet drops.

(Not to mention the fact that in some cases, for congestion control
mechanisms such as AQM mechanisms at the router, the immediate user
might be the person buying VPN services from the ISP, and in this case
that user cares about numbers about throughput, delay, and drop
rates, I believe.)

- Sallly

From: [email-masked] (Paulo Loureiro)
Date: Mon, 3 Oct 2005 17:06:02 +0100
Subject: [Tmrg-interest] [email-masked]
Message-ID: <[email-masked]>

This is a multi-part message in MIME format.

------=_NextPart_000_0029_01C5C83C.BC3E2730
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

[email-masked]

------=_NextPart_000_0029_01C5C83C.BC3E2730
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<style>
<!--
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin-top:6.0pt;
	margin-right:0cm;
	margin-bottom:0cm;
	margin-left:0cm;
	margin-bottom:.0001pt;
	text-align:justify;
	line-height:150%;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:Arial;
	color:windowtext;}
@page Section1
	{size:595.3pt 841.9pt;
	margin:70.85pt 3.0cm 70.85pt 3.0cm;}
div.Section1
	{page:Section1;}
-->
</style>

</head>

<body lang=3DPT link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
line-height:150%;font-family:Arial'>[email-masked]<o:p></o:p><=
/span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
line-height:150%;font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

</div>

</body>

</html>

------=_NextPart_000_0029_01C5C83C.BC3E2730--

From: [email-masked] (Xiaoliang (David) Wei)
Date: Mon, 3 Oct 2005 00:51:58 -0700
Subject: [Tmrg-interest] Interesting Draft Document on "TCP Benchmarking"
References: <[email-masked]>
Message-ID: <018601c5c7ef$5575c5a0$f5f2010a@seashadow>

> I like the metrics in your paper, and the dumbbell topology with a
> range of round-trip times.  My main feedback on your paper is that
> I don't think it is sufficient to have only one scenario with two-way
> traffic, only one separate scenario for heavy-tailed file size
> distributions, as two among many scenarios tested.  (Or perhaps I
> mis-understood the paper?)

Thank you very much, Sally:)
We are happy to work with Hamilton group if possible. I will contact Dr. 
Leith or Y-T soon. (We actually discussed briefly in PFLDNet2005.)

Your comments are very helpful. We completely agree that we need to add more 
realistic scenarios in the benchmark. The current benchmark is a 
work-in-progress and I'd like to explain the whole plan for the benchmark 
suite. Our understanding is that the benchmark suite is to answer three 
questions for different groups of people, in three corresponding steps:

1. "How robust is the design in various situations?" or "Did I miss anything 
in my new design?"
-- from new algorithm designers

We want to help the designers to verify their design and implementation. For 
this purpose, we need a (sub)set of clean scenarios that focus on each 
individual property of a protocol. In this set, we probably want each 
scenario to focus on one specific property. In our understanding, heavy-tail 
and two-way traffic will introduce some effects that are hard for the 
designers to analyze or understand. Specifically, for heavy-tail traffic, we 
probably need to have a large number of flows to have a distribution close 
to heavy-tail. Looking at a figure with so many flows, it is very hard for a 
designer to tell if a performance problem is due to the file distribution, 
or due to something else in their design. Similarly, for two-way traffic, 
even a single reverse traffic can cause the queues in the two directions 
swapping back-and-forth. It is harder for people to discover other problems 
if two-way traffic is presented in all the scenarios. (See Table V in our 
summary for some observation on the effect of reverse traffic with different 
congestion control algorithms)

2. "How does this algorithm perform in the most common scenarios?"
-- from researchers in the network community who want to evaluate the 
protocols

I guess this is probably what we currently emphasize in TMRG. We agree that 
heavy-tail file size distribution and two-way traffic are probably more 
common in Internet. We should add more scenarios with such effects as we 
understand more about what the real world looks like and figure out the way 
to simulate/emulate these effects. "Tools for the Evaluation of Simulation 
and Testbed Scenarios" is a very informative discussion on this case. I 
actually have some comments on the draft later.:) For researchers in this 
group, we also expect them to give a much heavier weight in scenarios with 
these effects. e.g. People can even ignore all the scenarios without 2-way 
traffic if they only want to see how an algorithm performs in cases with 
2-way traffic.

3. "Which algorithm shall I use for my environment/operation?"
-- from our end-users: system administrators and engineers in various 
scientific or industrial groups that heavily use TCP, such as High Energy 
Physic groups, distributed storage operators and etc

Several congestion control algorithms have been hooked up to Linux. It is up 
to the end-users' decision on which algorithm they should use. We feel very 
necessary to let the end users -- network engineers and system 
administrators and etc -- to understand which algorithm is good for which 
environments. Starting from step 1 and step 2, we want to translate our 
network-friendly metrics to application-friendly metrics. e.g. how does TCP 
throughput relate to job latency? How does delay and loss rate relate to 
jitters in telnet terminal? Probably, we should also get information from 
the end users and feed back their usage patterns to step 2 and generate some 
other "realistic scenarios". These are the application-driven scenarios, 
different from the statistic-driven scenarios that we focused in step 2. For 
example, HEP groups may only have long flows (data files) and very small 
mice (emails) in their research networks.

Currently, we believe that our benchmark suite is doing reasonably well for 
step 1, with some important elements in step 2 and 3. It tests a protocol in 
many different aspects. People of different interests can always find some 
scenarios that they really care. Also, the current suite can act as a set of 
criteria on a new protocol and its implementation: a new protocol is 
expected do well in handling these scenarios, before advancing to handling 
more complex scenarios.

To design a good and complete benchmark, we still have a long way to go to 
fulfil all the three purposes. We also need to have common consent among the 
community to make a benchmark successful. We will include your advice and 
others' comments to the work and try to achieve the goals.

Thanks, again:)

-David

Xiaoliang (David) Wei             Graduate Student in CS@Caltech
http://www.cs.caltech.edu/~weixl/
==================================================== 

From: [email-masked] (Dado Colussi)
Date: Sun, 02 Oct 2005 15:06:56 +0300
Subject: [Tmrg-interest] metrics for packet drop rates
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Sally Floyd wrote:
> My own guess would be that in terms of the effect of packet drops
> on congestion control performance, the packet drop rate and the
> congestion event rate are generally sufficient.  But they don't
> necessarily suffice as metrics for evaluating the effect of packet
> drops on the application.
> 
> - Sally

Sally,

ultimately, only the effect of packet drops on the application matters. 
Or even further, only the effect on user perception and satisfaction 
matters. In this regard, the whole discussion of packet drops, as a 
metric on transport level, is questionable.

There are applications with different interests regarding the service 
provided by the network (mice prefer low delay while elephants generally 
don't care about delay). Zero packet drop rate (meaning data loss, not 
packet marking) and zero queuing delay in the network is the only regime 
where applications with heterogeneous interests can be served in the 
optimal way. Under such regime, optimality reduces to bandwidth allocation.

Cheers,
Dado