Subject: [Tmrg-interest] PFLDnet panel on transport modeling
From: [email-masked] (Aaron Falk)
Date: Tue, 21 Feb 2006 11:08:03 -0800

I recently ran a panel on transport modeling at PFLDnet 2006.  The  
slides are available at http://www.hpcc.jp/pfldnet2006/ 
technical.html.  The panel members were
     Lachlan Andrew, Caltech
     Richard Hughes-Jones, University of Manchester
     Doug Leith, Hamilton Institute in Dublin
     Injong Rhee, NC State

I asked each panel member to write up a few paragraphs on their  
perspective to send to this list.  Several perspectives are  
reflected.  I hope it stimulates some interesting thoughts and  
discussion.

--aaron

PS.  My talk summarized the ICCRG and TMRG charters and gave an  
overview of the two TMRG drafts.

Begin forwarded message:

> From: Lachlan Andrew <[email-masked]>
> Date: January 20, 2006 9:57:46 AM PST
> To: Aaron Falk <[email-masked]>
> Subject: Re: PFLDnet panel on Transport Benchmarking
> Reply-To: [email-masked]
>
> Greetings Aaron,
>
> Here are some rough thoughts on what I'd like to say:
>
> 1. We should not optimise for today's traffic mix, but for mix we
> expect over the life of the standard (say 10-20 years).
>
> 2. As a result of 1, TCP friendliness is only a medium term
> requirement. We should test how protocols work when all the background
> traffic is using the new protocol, as well as when the background
> traffic is using Reno/SACK.
>
> 3. As well as large BDP networks, wireless networks suffer from the
> current TCP. Wireless Gigabit is already possible, so networks will
> soon suffer from both.  Higher channel error rates and/or random
> delays due to link layer retransmissions should be considered.
>
> 4. We should not focus on single-bottleneck scenarios.  They
> over-emphasise synchronised loss, and give the illusion that buffer
> overflow corresponds to the maximum observed RTT.
>
> 5. We need to test for both the "typical" case and as many "corner
> cases" as we can think of.  That means not just the ones for which the
> protocols are designed (like bulk transfer), but resilience to DoS,
> router misconfiguration, wireless and satellite links, interaction
> with router QoS mechanisms.  The protocols should be optimised for the
> typical case, and "safe" in the other cases.
>
> I'll also probably spend a slide talking about our WAN-in-Lab testbed
> which aims to avoid artefacts of using only Dummynet.
>
> I hope this summary isn't too long to be useful...
>
> Cheers,
> Lachlan
> --
> Lachlan Andrew  Dept of Computer Science, Caltech
> Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603

Begin forwarded message:

> From: "R. Hughes-Jones" <[email-masked]>
> Date: February 21, 2006 10:13:09 AM PST
> To: "'Aaron Falk'" <[email-masked]>, "'Lachlan Andrew'"  
> <[email-masked]>
> Cc: "'Injong Rhee'" <[email-masked]>, "'Douglas Leith'"  
> <[email-masked]>
> Subject: RE: pfldnet panel summaries
>
> Dear all
>
> Here is (at last) a summary of my panel talk and input from the  
> discussions
> - hope its useful:
>
> The approach taken was to show slides of some TCP transport tests to
> illustrate some of the characteristics (sometimes called metrics)  
> that we
> might or should consider when evaluating different stacks. This was  
> followed
> by a brief discussion of how "Lightpaths" were being used in  
> practice to
> connect high performance end systems and laboratories, suggesting some
> topologies for testing. Finally a list of possible characteristics,
> operational conditions under which the tests were conducted, and test
> methodologies were shown for discussion.
>
> I felt that many speakers at the PFLDnet meeting showed similar  
> measurements
> and there was general consensus amongst the delegates about what
> characteristics might be useful. An initial list of characteristics is
> suggested below; we should take care to establish a common  
> definition of
> their meaning possibly based on the GGF Network Measurements  
> Working Group
> Characteristic Hierarchy Proposed Recommendation and the various  
> IETF IPPM
> RFCs.
>
> .	The value of Achievable Throughput that can be obtained
> .	Ability to share link Capacity (OK what is sharing?)
> .	Time to converge to fair or equal Achievable Throughput
> .	Responsiveness - ability to increase the Achievable Throughput when
>            there is spare link capacity
> .	rtt fairness (OK what is fairness?)
> .	mtu fairness
> .	TCP friendliness - how does this transport protocol interact with
>            other TCP flows (is this "sharing link Capacity" ?)
> .	Link utilisation (by this flow and all flows)
> .	Stability of Achievable Throughput
> .	Burst behaviour
> .	Packet loss behaviour (deliberate and in real-life networks)
> .	Packet re-ordering behaviour
> .	The latency in sending a message over the TCP stack - could be a
>            relative one-way observation or a round trip observation.
> .	Measurement of the CPU load on the end systems
>
> Many of these might be measured as some form of time-series, and  
> there are
> several possible derived characteristics eg Jain's fairness index.
>
> Some possible topologies:
> .	Simple end-to-end - eg back to back
> .	Dumbbell with end-to-end flows passing over a bottleneck.
>           Independent many-to-many, one-to-many, many-to-one.
> .	Background and or cross traffic - how realistic is needed? - what
>           protocol mix UDP, TCP packet sizes?
> .	Reverse traffic flows
> .	Layer 2 paths only
> .	Inclusion of Layer 3 routers - with a set of buffer sizes and queue
>           management techniques
> .	Long and short  rtt - e.g. 1ms 20 ms 100 ms 250ms
>
> Methodologies
> .	Mathematical models
> .	Simulation
> .	Hardware emulation
> .	Real testbed links
> .	Live "fibre" or "Copper" networks
> .	Mobile networks
>
> It is clear that a basic set of characteristics, baseline tests (eg  
> standard
> New Reno TCP/IP), test topologies, experimental conditions, and  
> methods
> would be very helpful and we should try to document these so that  
> different
> researchers can repeat the experiments made. Scientific  
> understanding and
> advances are based on reproducible results! However the framework  
> for making
> stack comparisons should be extensible to include technology  
> advances etc.
>
> Regards
> Richard Hughes-Jones
>
> Tel:  +44 (0) 161 275 4117
> Fax: +44 (0) 161 273 5867

Begin forwarded message:

> From: "Douglas Leith" <[email-masked]>
> Date: February 3, 2006 6:32:17 AM PST
> To: "Aaron Falk" <[email-masked]>, "Injong Rhee" <[email-masked]>,  
> "Lachlan Andrew" <[email-masked]>, "Douglas Leith"  
> <[email-masked]>, "R. Hughes-Jones" <R.Hughes- 
> [email-masked]>
> Subject: RE: panel summary
>
> Perhaps this captures the guts of the discussion Aaron, at least  
> from my perspective ...
>
> 1. Lets try to start by defining a *small* subset of tests that we  
> can agree on.  The aim is for consensus, which I felt was pretty  
> much already there in the common component of test results  
> presented at pfldnet.   In fact I would advocate just lifting the  
> common subset of tests already being used, documenting them and  
> taking things from there.  The plan would of course be to extend  
> this by including new tests over time, but I think we really need  
> to focus on making progress on a limited front just now or we run  
> the danger of making no progress at all.  Such an approach is not  
> intended to in any way limiting - anyone is of course free to run  
> any additional test they like (and indeed I would encourage this)  
> and we can include them downstream if desired.
>
> 2. Lets define what key measurements to record.  Again, I think  
> there was a de facto consensus on a subset of measurements - cwnd  
> time history, throughput, goodput, connection completion time  
> (others ?).  I argue that we need measurements that are agnostic  
> and so not pre-judge decision-making.  For example, metrics such as  
> Jain's fairness index can be derived from this physical data if  
> desired, but if we just record Jain's index and through away the  
> physical data the reverse is not true.
>
> 3. Tests should always include a baseline case e.g. standard tcp,  
> that can be used as a sanity check and to confirm the equivalence  
> of tests run by different people.  This is a real issue as Injong  
> and myself cannot even get our standard tcp throughput measurements  
> to agree under what seem like identical conditions but evidently  
> are not.
>
> 4.  In terms of facilitating testing, it seemed good to try to  
> share scripting code for automating tests.  This would hopefully  
> help improve consistently as well as avoiding duplication of effort.
>
> 5. I feel that an effective way to make concrete progress would be  
> to organise a hands-on workshop where people actively taking  
> measurements get together and run tests/share measurements.  Hoped  
> for outcomes would include closer agreement on test results  
> obtained under the same nominal conditions but using different  
> testbeds, plus sharing of best practice.
>
> Doug
>

Begin forwarded message:

> From: [email-masked]
> Date: February 3, 2006 12:10:53 PM PST
> To: "Douglas Leith" <[email-masked]>
> Cc: "Aaron Falk" <[email-masked]>, "Injong Rhee" <[email-masked]>,  
> "Lachlan Andrew" <[email-masked]>, "Douglas Leith"  
> <[email-masked]>, "R. Hughes-Jones" <r.hughes- 
> [email-masked]>
> Subject: RE: panel summary
>
> This is the summary of my position.
>
> As a community, we need a common testbed and a user-friendly way to
> setup, schedule and run various tests.
>
> 1. There is a difference between Emulab and the tool we need. We  
> need a "congestion control" research tool that involves setting up  
> specific testing environments for congestion control testing.  
> Testing environments are just more than hardware or network  
> topology. They are workloads, scenarios, access to specific  
> performance variables in the kernel, and specific ways of  
> reporting. Emulab is rather too general purpose for CC and  
> availability of such CC tools would reduce a lot of wasted time in  
> designing and setting up the test cases.  At least, our team has  
> spent a lot of time in running the tests and we would like to share  
> our knowhow and want other researchers to engage more in research  
> rather than nitty-details of test setups.
>
> 2. The tool must very user-friendly. Setup, scenario generation,  
> and reporting must be done at a few clicks of mouse through a web  
> page. Uploading a testing protocol or TCP stack or images should be  
> allowed and scheduling a test should be very simple.
>
> 3. Having a common testbed (now I am talking about hardware) is  
> also useful. It is useful for comparing test results. In many  
> cases, even if two researchers claim to have the exact same setup  
> in their individual labs, we often find it hard to reproduce exact  
> same results. Thus, it is hard to agree on the results because we  
> often get different results because of some hardware dependency  
> such as NIC, CPU, MEMORY, etc.
>
> 4. Sharing tool, script and testing results is great only if proper  
> acknowledgment is made to recognize the developer of the scripts.  
> It is little courtesy we pay to the original developer. Even bug  
> fixes and any additional intuitions that result from the testing  
> report of other folks must also be acknowledged. We often forget to  
> do this important basics. Even development of new metrics (e.g.,  
> RTT-fairness) and discovery of some protocol idiosyncrasy, and some  
> test setup, and metric all need proper citation and references. I  
> often find many of us forgetting this basic as a researcher.
>
> 5. Another point that I did not make in the panel discussion  
> (because we all looked a little tired in the end, so i skipped).  
> Many testing metrics and scenarios are presented and actually  
> commonly used among us. But no one talks about stability. I know it  
> is hard to measure relative stability. I am not talking about  
> stability in Nyquist model (or whatever we use in a fluid model)  
> which tells just a binary answer to stability -- whether a protocol  
> is stable or now. In many cases, most protocols pass this type of  
> stability test. But we need a better metric than the fluid model  
> sense of stability. We need a metric that can tell one protocol is  
> more stable than the others. Some discussion on this and consensus  
> on this important metric would be very important and useful.

From: [email-masked] (Xiaoliang (David) Wei)
Date: Sat, 4 Feb 2006 08:37:01 -0800
Subject: [Tmrg-interest] Re: [Iccrg] Proposal for ICCRG operations
References: <[email-masked]> <[email-masked]> <[email-masked]> <[email-masked]> <[email-masked]>
Message-ID: <07e101c629a9$3a4e4e20$f5f2010a@seashadow>

Thanks, all. About the benchmark, I have been thinking about a few questions 
that might be interesting in the process of designing a benchmark:

1. Who is expected to use the benchmark?
    My understanding is that there may be different groups of users that are 
interested in different properties of a congestion control algorithm.
    For example, distributed computation applications would prefer a low 
latency of the worst flow (this is max-min fairness). Data transfer users 
may emphasize on a higher average throughput. Media servers would like to 
have stable delay... As Linux allows users to select their preferred 
congestion control algorithm, network administrator is capable to select 
different algorithms in their private network, based on the major 
applications they have. In this case, I think it make sense to provide a few 
different metrics for different users.
    If the benchmark is to be used to identify a TCP candidate for 
standardization, we probably need to consider even more: incremental 
deployability (interaction with Reno), the impact of the algorithm to the 
network (link loss rate, queue fluctuation) and etc. In this case, we may 
also want to measure the realistic workload in Internet and feed it back to 
the benchmark.

2. Who will perform the benchmark tests, and how?
    As the e2e mailing list talking on some strange behaviors in NCSU's 
performance evaluation due to some buggy implementation, it brings up the 
problem that the same congestion control algorithm may perform very 
different because of a bug in the implementation, or even a hardware defect 
in the testbed.
    I think it will be better if we can have a standard hardware testbed 
that can repeatedly evaluate an implementation of congestion control 
algorithm. So, the protocol designers can iterate the process of 
design->implementation->testing and make sure that their algorithms are 
correctly evaluated. In this issue, I think WAN-in-Lab may help (at least 
partially) as a standard testing platform, if we can build a automatic 
testing system based on it.

just my 2 cents.

-David

Xiaoliang (David) Wei             Graduate Student in CS@Caltech
http://www.davidwei.org
====================================================
----- Original Message ----- 
From: "Lachlan Andrew" <[email-masked]>
To: "Michael Welzl" <[email-masked]>
Cc: "Aaron Falk" <[email-masked]>; "Steven Low" <[email-masked]>; "Lachlan 
Andrew" <[email-masked]>; "iccrg" <[email-masked]>; "Pei Cao" 
<[email-masked]>; "David Wei" <[email-masked]>; 
<[email-masked]>
Sent: Friday, February 03, 2006 3:57 PM
Subject: Re: [Iccrg] Proposal for ICCRG operations

Greetings all,

On 03 Feb 2006 09:32:27 +0100, Michael Welzl <[email-masked]> 
wrote:
> So, the index should contain:
>
> * normalized throughput
> * loss
> * fairness (let's use the Jain Fairness Index)
>
> It could then look something like this:
>
> WI = (T*t + L*l + F*f) / 3
>
> where WI is "Welzl Index" (tm)   ;-)
> T is normalized throughput
> L is normalized loss
> F is the Jain fairness index (which is already a value
> in the 0..1 range)

A good way to summarise throughput and fairness is just to take the
*harmonic* mean of the flows' throughputs instead of the arithmetic
mean (or equivalently, the aggregate througput).  That corresponds to
measuring the mean time to transfer a given number of bytes, which is
the quantity actually observed by an application.

This metric avoids the arbitraryness both of using a sum or product,
with the problem of  selecting weightings, and also of Jain's index.
(If one flow out of many is starved, Jain's index can still report
high "fairness", but the harmonic mean of rates goes to zero
highlighting the problem.)  Similarly, if one flow (or protocol) is
unable to achieve a high rate, Jain's index says it is "unfair" for
another to out-perfom it.  The harmonic mean correctly indicates that
rates  0.5 and 1  are better than rates 0.5 and 0.5 (but less good
than 0.75 and 0.75).

(Trying to maximise the harmonic mean of the rates is just a special
case of "utility maximisation", using  alpha=2 in Mo and Walrand's
alpha-fairness framework.  It could be argued that this utility
function is itself arbitrary, but it seems less so than summing
thoughput and "fairness".)

Combining throughput and fairness of throughput can be done
systematically, since they both just reflect the average rates.  A
bigger challenge is incorporating loss/delay and the other metrics
Wesley Eddy mentioned (stability, convergence time etc).  For file
transfers, it could be argued that these only matter insofar as they
affect the total transfer times.  This applies to both long- and
short-lived flows (but not to real-time flows).  However, there is
still a problem of how to weight the times:  The transfer time of each
short flow should get less weight than a long flow, but weighting them
in proportion to the amount of data may put too little weight on them.

Cheers,
Lachlan

--
Lachlan Andrew  Dept of Computer Science, Caltech
Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603

From: [email-masked] (Lachlan Andrew)
Date: Fri, 3 Feb 2006 15:57:08 -0800
Subject: [Tmrg-interest] Re: [Iccrg] Proposal for ICCRG operations
In-Reply-To: <[email-masked]>
References: <[email-masked]> <[email-masked]> <[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

Greetings all,

On 03 Feb 2006 09:32:27 +0100, Michael Welzl <[email-masked]> wrote:
> So, the index should contain:
>
> * normalized throughput
> * loss
> * fairness (let's use the Jain Fairness Index)
>
> It could then look something like this:
>
> WI = (T*t + L*l + F*f) / 3
>
> where WI is "Welzl Index" (tm)   ;-)
> T is normalized throughput
> L is normalized loss
> F is the Jain fairness index (which is already a value
> in the 0..1 range)

A good way to summarise throughput and fairness is just to take the
*harmonic* mean of the flows' throughputs instead of the arithmetic
mean (or equivalently, the aggregate througput).  That corresponds to
measuring the mean time to transfer a given number of bytes, which is
the quantity actually observed by an application.

This metric avoids the arbitraryness both of using a sum or product,
with the problem of  selecting weightings, and also of Jain's index. 
(If one flow out of many is starved, Jain's index can still report
high "fairness", but the harmonic mean of rates goes to zero
highlighting the problem.)  Similarly, if one flow (or protocol) is
unable to achieve a high rate, Jain's index says it is "unfair" for
another to out-perfom it.  The harmonic mean correctly indicates that
rates  0.5 and 1  are better than rates 0.5 and 0.5 (but less good
than 0.75 and 0.75).

(Trying to maximise the harmonic mean of the rates is just a special
case of "utility maximisation", using  alpha=2 in Mo and Walrand's
alpha-fairness framework.  It could be argued that this utility
function is itself arbitrary, but it seems less so than summing
thoughput and "fairness".)

Combining throughput and fairness of throughput can be done
systematically, since they both just reflect the average rates.  A
bigger challenge is incorporating loss/delay and the other metrics
Wesley Eddy mentioned (stability, convergence time etc).  For file
transfers, it could be argued that these only matter insofar as they
affect the total transfer times.  This applies to both long- and
short-lived flows (but not to real-time flows).  However, there is
still a problem of how to weight the times:  The transfer time of each
short flow should get less weight than a long flow, but weighting them
in proportion to the amount of data may put too little weight on them.

Cheers,
Lachlan

--
Lachlan Andrew  Dept of Computer Science, Caltech
Phone: +1 (626) 395-8820    Fax: +1 (626) 568-3603

From: [email-masked] (Wesley Eddy)
Date: Fri, 3 Feb 2006 09:26:58 -0500
Subject: [Tmrg-interest] Re: [Iccrg] Proposal for ICCRG operations
In-Reply-To: <[email-masked]>
References: <[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

--9zSXsLTf0vkW971A
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Feb 03, 2006 at 09:32:27AM +0100, Michael Welzl wrote:
> So, the index should contain:
>=20
> * normalized throughput
> * loss
> * fairness (let's use the Jain Fairness Index)
>=20
> It could then look something like this:
>=20
> WI =3D (T*t + L*l + F*f) / 3
>=20
> where WI is "Welzl Index" (tm)   ;-)
> T is normalized throughput
> L is normalized loss
> F is the Jain fairness index (which is already a value
> in the 0..1 range)
>=20
> and t, l, f are weighting factors which decide how
> important the parameters are to us.
>
> ...

It would be nice to have a simple scheme, but I don't think that a simple
summation-based scheme is going to suffice.  In addition to the problems
Michael described with this simple scheme, here are a couple more:

1) no notion of how stability factors in
2) no notion of convergence rate
3) no notion of what happens to a mix of different types of flows
4) ranking is ambiguous in that it gives identical results for very differnt
   inputs

There are ways to cobble in or correct for these, but I don't know how to
meaningfully do it all at once.  In previous work on RED, Mark Allman and
I started with an index based on summation like the one you propose, and
we found problem 4 irritating.  We were only looking at two inputs to the
rating function -- utilization (U) and Jain's index (F), both from 0 to
1.  We decided that an index computed using  U^alpha * F^(1-alpha)
worked much better for our purposes, with alpha used to weight preferences.
This is described in a pretty obscure paper called "A Comparison of RED's
Byte and Packet Modes" from Computer Networks volume 42, issue 2, in June
2003.

-Wes

--=20
Wesley M. Eddy
Verizon Federal Network Systems

--9zSXsLTf0vkW971A
Content-Type: application/pgp-signature
Content-Disposition: inline

--9zSXsLTf0vkW971A--

From: [email-masked] (Filipe Abrantes)
Date: Fri, 03 Feb 2006 14:16:27 +0000
Subject: [Tmrg-interest] Re: [Iccrg] Proposal for ICCRG operations
In-Reply-To: <[email-masked]>
References: <[email-masked]>	 <[email-masked]>	 <[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

Hi Michael,

before I agree with you, let me disagree for a bit :) .

Michael Welzl wrote:
> [cc'ing TMRG]
> 
> 
> Hi all,
> 
> I'm very happy to see some movement here. Clearly, many of us
> would want something concrete to come out of ICCRG, but how to
> reach agreement is the tricky bit.
> 
> 
> 
>>>I think a TCP benchmarking suite will be a very helpful tool to
>>>guide us in this process.  It will be useful especially in your
>>>Phase 3 to evaluate various proposals, but just thinking through
>>>the design and implementation of a good benchmark lends clarity
>>>to issues of Internet congestion control and metrics to optimize
>>>against.
>>>
>>>Aaron Falk is chairing a panel at a conference to discuss this
>>>very issue.  Hopefully, we wll hear from the participants soon
>>>on their thoughts.
>>
>>So, the panel isn't for several hours (at PFLDnet 2006) but one thing  
>>that has become clear to me so far is that this community has not  
>>converged on metrics and scenarios for evaluation.  The TMRG work is  
>>really needed to make progress.
> 
> 
> I have an overly simplistic proposal ... this is something I had
> in my mind for a long time, but I kept dismissing it again and
> again because it probably isn't feasible. Nevertheless, I'm now
> bringing it up as food for thought rather than a concrete
> suggestion:
> 
> Wouldn't it be great if we had a single "congestion control
> performance index" ? (or wait, before anybody agrees with me,
> let me call it "Welzl index" instead   :)   )
> 
> After all, it's not that much that we demand from a
> congestion control mechanism: it must avoid congestion
> while efficiently and fairly utilizing the available
> bandwidth, right? Congestion means that queues grow, leading
> to increased delay and eventually loss beyond a certain
> length (or depending on some AQM function); should the
> index then contain loss _AND_ delay? For simplicity, I'd
> suggest to ignore delay. Let's say we only have loss
> in the index, and someone comes up with a result that
> shows how good a mechanism is doing in a certain scenario,
> while in fact queues are growing - well, then it's a
> matter of changing the scenario just a little bit (more
> flows and/or shorter queues), and we have a case of
> "inadequate performance evaluation" (which we can always
> have).
> 

I have thought for a while why do we talk about "congestion control" 
when what all these mechanisms are responsible for, is a bit more than 
solely avoid congestion. Rate control would be a bit more accurate from 
my point of view. That said, I just wanted to disagree about leaving 
delay out of the metric. I didn't quite understood your explanation, but 
certainly a mechanisms that maintains a lower queue is better than one 
that maintains an higher queue (and note that there are mechanisms that 
maintain stable queues avoiding most of the packet drops).

Other than that I agree with you about the need of nice little index for 
uniform performance measurement.

br

-filipe

> So, the index should contain:
> 
> * normalized throughput
> * loss
> * fairness (let's use the Jain Fairness Index)
> 
> It could then look something like this:
> 
> WI = (T*t + L*l + F*f) / 3
> 
> where WI is "Welzl Index" (tm)   ;-)
> T is normalized throughput
> L is normalized loss
> F is the Jain fairness index (which is already a value
> in the 0..1 range)
> 
> and t, l, f are weighting factors which decide how
> important the parameters are to us.
> 
> Now the question would be how to appropriately set these
> factors... we'd have to decide on fixed values, I
> guess, or otherwise the simplification obtained with the
> index isn't really a simplification after all. Should
> t, l and f all be 1?
> 
> While I don't seriously believe that we can all just
> agree on the Welzl index, I believe that thinking in
> this direction of simplifying performance evaluations
> is worthwhile.
> 
> Just imagine what congestion control papers could look
> like if we had such an index... any comparisons could be
> more focused. Sure, all the usual scenarios would still
> have to be checked (dumbbell and beyond  :-)  ), but
> we'd reduce the problem space by one dimension - instead
> of showing loss, throughput, fairness etc., the Welzl index
> would be enough.
> 
> Cheers,
> Michael
> 
> _______________________________________________
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
> 

-- 
Filipe Lameiro Abrantes
INESC Porto
Campus da FEUP
Rua Dr. Roberto Frias, 378
4200-465 Porto
Portugal

Phone: +351 22 209 4266
E-mail: [email-masked]

From: [email-masked] (Michael Welzl)
Date: 03 Feb 2006 09:32:27 +0100
Subject: [Tmrg-interest] Re: [Iccrg] Proposal for ICCRG operations
In-Reply-To: <[email-masked]>
References: <[email-masked]> <[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

[cc'ing TMRG]

Hi all,

I'm very happy to see some movement here. Clearly, many of us
would want something concrete to come out of ICCRG, but how to
reach agreement is the tricky bit.

> > I think a TCP benchmarking suite will be a very helpful tool to
> > guide us in this process.  It will be useful especially in your
> > Phase 3 to evaluate various proposals, but just thinking through
> > the design and implementation of a good benchmark lends clarity
> > to issues of Internet congestion control and metrics to optimize
> > against.
> >
> > Aaron Falk is chairing a panel at a conference to discuss this
> > very issue.  Hopefully, we wll hear from the participants soon
> > on their thoughts.
> 
> So, the panel isn't for several hours (at PFLDnet 2006) but one thing  
> that has become clear to me so far is that this community has not  
> converged on metrics and scenarios for evaluation.  The TMRG work is  
> really needed to make progress.

I have an overly simplistic proposal ... this is something I had
in my mind for a long time, but I kept dismissing it again and
again because it probably isn't feasible. Nevertheless, I'm now
bringing it up as food for thought rather than a concrete
suggestion:

Wouldn't it be great if we had a single "congestion control
performance index" ? (or wait, before anybody agrees with me,
let me call it "Welzl index" instead   :)   )

After all, it's not that much that we demand from a
congestion control mechanism: it must avoid congestion
while efficiently and fairly utilizing the available
bandwidth, right? Congestion means that queues grow, leading
to increased delay and eventually loss beyond a certain
length (or depending on some AQM function); should the
index then contain loss _AND_ delay? For simplicity, I'd
suggest to ignore delay. Let's say we only have loss
in the index, and someone comes up with a result that
shows how good a mechanism is doing in a certain scenario,
while in fact queues are growing - well, then it's a
matter of changing the scenario just a little bit (more
flows and/or shorter queues), and we have a case of
"inadequate performance evaluation" (which we can always
have).

So, the index should contain:

* normalized throughput
* loss
* fairness (let's use the Jain Fairness Index)

It could then look something like this:

WI = (T*t + L*l + F*f) / 3

where WI is "Welzl Index" (tm)   ;-)
T is normalized throughput
L is normalized loss
F is the Jain fairness index (which is already a value
in the 0..1 range)

and t, l, f are weighting factors which decide how
important the parameters are to us.

Now the question would be how to appropriately set these
factors... we'd have to decide on fixed values, I
guess, or otherwise the simplification obtained with the
index isn't really a simplification after all. Should
t, l and f all be 1?

While I don't seriously believe that we can all just
agree on the Welzl index, I believe that thinking in
this direction of simplifying performance evaluations
is worthwhile.

Just imagine what congestion control papers could look
like if we had such an index... any comparisons could be
more focused. Sure, all the usual scenarios would still
have to be checked (dumbbell and beyond  :-)  ), but
we'd reduce the problem space by one dimension - instead
of showing loss, throughput, fairness etc., the Welzl index
would be enough.

Cheers,
Michael