Subject: [Tsvwg] Test environments for congestion response

From: Yogesh Prem Swami <[email-masked]>
Date: 05 Mar 2003 13:03:49 -0600

Hi,

I asked this in last IETF, but let me ask again. Since now we have many
congestion response algorithms, will it make sense to have a well
defined test environment against which authors can run their tests and
get results? The test environment could be an information RFC that can
be used in future too.

In the past, I have seen various drafts and papers where authors want to
evaluate the congestion behavior of their algorithm, but they only use a
single TCP connection in the entire system. This approach sometimes
gives hundreds of percents of improvement and might suit the author
needs, but when these tests are run with multiple connections the
performance goes down significantly--sometimes below the performance of
existing algorithms.

Having spent time trying to model networks, I understand that it's not
always possible to model  a 100% realistic network, but leaving it
totally un-addressed is even worse. Probably it will be useful to have a
set of guidelines: have 10 TCP connections, 5 TFRC based connections
with 5k, 10K, 100K, ... file size, travesing 10 routers with certain
buffer spaces on the path, etc., etc. These numbers could be used from
the Internet measurements (BGP statistics ... ) 

I think whenever we have multiple schemes we must run the tests along
these guidelines and pick the scheme which comes up with the best
compromise in terms of congestion or whaever the working group deems
important. 

--yogesh

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Thu, 6 Mar 2003 00:10:27 -0500 (EST)

> the Internet measurements (BGP statistics ... )

Wouldn't an aggregation of Pareto ON-OFF sources be enough? The
cross-traffic should be self-similar. A few TCP and TFRC connections
probably will not give you the arrival pattern that a router actually
experiences.

I think the idea of an Information RFC as you suggest would be a good
idea, but I'd be really interested in seeing a large group agree on
something like this. :-)

~armando

0--                                                  --0
| Armando L. Caro Jr.      |  Protocol Engineering Lab |
| www.cis.udel.edu/~acaro  |    University of Delaware |
0--                                                  --0

---

From: Yogesh Prem Swami <[email-masked]>
Date: 06 Mar 2003 10:59:11 -0600

Hello Armando,

> probably will not give you the arrival pattern that a router actually

I agree. What is needed is a realistic scenario which can mimic the
network behaviour in a certain way. I am appalled by results which show
300% improvent in TCP performance, but of course the authors were using
loopback addresses for testing their improved schemes. [Just
exaggerating ...]

> idea, but I'd be really interested in seeing a large group agree on

Yes, but not defining anything is probably worse. 

--yogesh

---

From: Aaron Falk <[email-masked]>
Date: Fri, 7 Mar 2003 11:44:43 -0800

Yogesh-

You might find this of interest:

http://www.acm.org/sigcomm/ccr/archive/1999/oct99/allman2.pdf

--aaron

---

From: "Andrei Gurtov" <[email-masked]>
Date: Mon, 10 Mar 2003 15:44:36 +0200

> loopback addresses for testing their improved schemes. [Just

The scenario we've used in the INFOCOM'03 paper is targeted at a specific
environment, namely UMTS-kind of cellular links. Such links have dedicated
per-user buffers and are not likely to have many concurrent TCP or real-time
flows.
Because of the page limit on the paper we have included only the most
important scenarios: uncongested (a single TCP flow) and congested
(TCP+CBR).
However, I agree that it is useful to explore other traffic mix and I'll
provide results later on.

Still, if we assume that our paper demonstrates the best scenario for Eifel
(+300% throughput) and your DCLOR the worst (-10% throughput), I would vote
for deploying Eifel.  Actuallly, since the difference between all schemes,
including the standard TCP, in your tests is below 10%, the scenario does
not look interesting to me at all. Furthermore, your tests are impossible to
verify as the tools you've used are not publicly available.

In the INFOCOM'03 paper we argue why congestion control should be undone
after a spurious timeout. If not, ACKs are ignored after a spurious timeout
which is a poor idea because it drops the thoughput up to ten times and at
the same time *decreases* goodput as more timeouts and hence retransmissions
occur.

We've also made some measurement of TCP&TFRC over vertical handovers
("subnet changes" in your terms) with significant variation of link
characteristics. http://www.cs.helsinki.fi/u/gurtov/papers/vho.html

Andrei

---

From: Yogesh Prem Swami <[email-masked]>
Date: 10 Mar 2003 10:48:59 -0600

Andrei,

> per-user buffers and are not likely to have many concurrent TCP or real-time

I don't know much about UMTS networks, but I am told that this is left
to implementation. I will confirm this and comment later on ...

> (+300% throughput) and your DCLOR the worst (-10% throughput), I would vote

I was not talking about your paper in this example. The example I was
giving was an exaggeration to emphasize the fact that results could be
deceiving. 

(I explicitly remember making the point that it was an exaggeration. I
apologize if I gave a different impression--it was not intentional)

> verify as the tools you've used are not publicly available.

My tests were run on *real networks*, not on NS-2 simulator (I don't
trust NS-2). DCLOR is implemented in Linux. I will make it publically
available by tomorrow end of business.

I also encourage you to run DCLOR in a real whatever network (UMTS/
wireless lan ...) if you have access to it. It won't give a -10%
improvement even in the worst case. But please don't take my word for
it, verify it for yourself.

> the same time *decreases* goodput as more timeouts and hence retransmissions

That happens if you blindly reduce the data rate to half. A clever
implementaion can handle it properly.

> We've also made some measurement of TCP&TFRC over vertical handovers

Please use "subnet change." Not everyone (read Yogesh) knows about fancy
wireless network terminology.
 
> characteristics. http://www.cs.helsinki.fi/u/gurtov/papers/vho.html

Thanks. This certainly helps.

--yogesh