Subject: [Tmrg-interest] Interesting Draft Document on "TCP Benchmarking"
From: [email-masked] (Saverio Mascolo)
Date: Fri, 30 Sep 2005 14:38:49 +0200

>So my strong recommendation
> would be that *all* of the scenarios contain *some* two-way traffic.
> Except perhaps for a single scenario exploring the corner case
> of one-way traffic.

I agree totally with this strong recommendation! Our experience is that it
is of primary importance to include reverse traffic because of the effect on
the ACK packets (TCP is ack cloked!). In some cases, such as with TCP Vegas,
there is also an effect of reverse traffic on the algorithm (with vegas
forward connections are heavily affected by reverse traffic because of the
effect on rtt due to queueing on reverse path). We have reported results  in
the ACM CCR, issue of  April 2004.
A similar behaviour was revealed when   Fast TCP was tested in the presence
of reverse traffic by Les Cottrell at SLAC.

Saverio

From: [email-masked] (Sally Floyd)
Date: Thu, 29 Sep 2005 13:55:03 -0700
Subject: [Tmrg-interest] Interesting Draft Document on "TCP Benchmarking"
Message-ID: <[email-masked]>

>Oops. First, I'd thank Cesar for looking at and introducing our work in 
>progress. We surely appreciate any comments.

Many thanks.  I have added a pointer to your papers to the TRMG web
site.  I listed your paper on "Time for a TCP Benchmark Suite?"
next to the 2005 paper by Li, Leith, and Shorten on "Experimental
Evaluation of TCP Protocols for High-Speed Networks", which has the
following abstract:

  "In this paper we present experimental results evaluating the
  performance of the Scalable-TCP, HS-TCP, BIC-TCP, FAST-TCP and H-TCP
  proposals in a series of benchmark tests."

Since both groups are proposing benchmark test suites for TCP
protocols in high-speed networks, it might be useful if the two
groups could collaborate.  (One possibility would be to produce an
internet-draft together that incorporates feedback from TMRG.)

I like the metrics in your paper, and the dumbbell topology with a
range of round-trip times.  My main feedback on your paper is that
I don't think it is sufficient to have only one scenario with two-way
traffic, only one separate scenario for heavy-tailed file size
distributions, as two among many scenarios tested.  (Or perhaps I
mis-understood the paper?)

For example, it seems pretty clear that *any* scenario with only
one-way traffic is unrealistic.  Even with asymmetric routing, I
don't think there are any realistic scenarios with congested traffic
on a link, where none of the packets in those connections ever have
to contend with the occasional other packet on the reverse path.
(E.g., requests to the web server, the other half of VoIP traffic,
the mix of peer-to-peer traffic, etc.)  So my strong recommendation
would be that *all* of the scenarios contain *some* two-way traffic.
Except perhaps for a single scenario exploring the corner case
of one-way traffic.

Similarly, I doubt that there are any realistic scenarios with only
long-lived traffic and very short traffic, and no medium-size
traffic.  (I haven't ever seen any measurement data from such a
scenario, and I don't have any reason to expect such scenarios to
be more likely to emerge in the future.)  Thus, my strong recommendation
would be that *all* scenarios contain *some* traffic generated from
Poisson arrivals with a heavy-tailed distribution of file sizes.
With perhaps a single scenario with only long-lived traffic and very
small traffic, to illustrate the corner case of behaviors that can
occur in such scenarios.

(These issues about traffic apply both for research in done in
simulations or in testbeds.)

You could look at the internet-draft on "Tools for the Evaluation
of Simulation and Testbed Scenarios", on the TMRG web site, for
more discussion of these issues.  

The main reason that I started TMRG was to try to create a consensus
in the research community about the need for realistic scenarios
with two-way traffic, a range of round-trip times, a range of file
sizes, and the like, so I would love to have feedback on this (and
arguments from others if I am wrong).

Many thanks,

- Sally
http://www.icir.org/tmrg/

From: [email-masked] (Xiaoliang (David) Wei)
Date: Tue, 27 Sep 2005 18:01:41 -0700
Subject: [Tmrg-interest] Interesting Draft Document on "TCP Benchmarking"
References: <[email-masked]>
Message-ID: <01b201c5c3c8$32c27f10$f5f2010a@seashadow>

Oops. First, I'd thank Cesar for looking at and introducing our work in 
progress. We surely appreciate any comments.

For readers who are interested in the work, there is a slightly better 
organized summary at:
http://www.cs.caltech.edu/~weixl/research/technicals/benchmark/summary.ps
It includes our motivation, work-in-progress and our plan to utilize 
Wan-in-Lab to help the community to run benchmark suites in the future.

The draft pointed out by Cesar contains all the details of our scenarios, 
which are ignored in the summary.:
> http://www.cs.caltech.edu/%7Eweixl/research/technicals/benchmark/draft3.pdf 
> (unfinished draft)
> http://www.cs.caltech.edu/~hegdesan/sigcomtest/   (Some results)

We have been working on this since early 2004. After writing down all the 
scenario details in the original draft, we have been iteratively running 
experiments and simulations on the several high speed variants of TCP, and 
modify the scenario details according to our experiment experience. Since we 
haven't understood some of the results, we haven't published the work yet --  
to minimize the possibility that some of our results may be misleading. 
(e.g. problems of slow loss recoveries in some experiments are probably not 
from TCP algorithms, but from Linux implementation. And some other results 
may be "artefacts" of bugs in certain version of NIC hardware...)

We have a plan to combine these resources into a better organized technical 
report, after we understand the problems carefully. Thanks.

-David

Xiaoliang (David) Wei             Graduate Student in CS@Caltech
http://www.cs.caltech.edu/~weixl
==================================================== 

From: [email-masked] (Cesar Marcondes)
Date: Mon, 26 Sep 2005 11:01:34 -0700
Subject: [Tmrg-interest] Interesting Draft Document on "TCP Benchmarking"
Message-ID: <[email-masked]>

Dear all,

   I'm not sure if you are aware about it. There is an unfinished draft 
document on "TCP Benchmark" available in the following website (S. Low 
group).

http://www.cs.caltech.edu/%7Eweixl/research/technicals/benchmark/draft3.pdf 
(unfinished draft)
http://www.cs.caltech.edu/~hegdesan/sigcomtest/   (Some results)

Best regards,
Cesar Marcondes
UCLA

From: [email-masked] (Sally Floyd)
Date: Thu, 22 Sep 2005 16:04:00 -0700
Subject: [Tmrg-interest] Goodput x throughput
Message-ID: <[email-masked]>

Nelson -

>after reading the draft-irtf-tmrg-metrics I would like to suggest
>to distinguish between "throughput" and "goodput"

Good idea, I will do that.

...
>As a general comment, I feel that precise defintion of the metric of
>interest would help the readership to understad what is meant by the
>metrics.

I will see what I can do, but in some cases there isn't necessarily
a consensus in the community on precise definitions for the metrics,
e.g., for fairness, response to transient changes, minimizing
oscillations, and the like.  And some of the metrics, such as
deployability, might never be susceptible to a particularly precise
definition...

Many thanks for the feedback.

- Sally
http://www.icir.org/tmrg/

From: [email-masked] (Sally Floyd)
Date: Wed, 21 Sep 2005 16:18:39 -0700
Subject: [Tmrg-interest] info about packet drop rates as a function of packet size?
Message-ID: <[email-masked]>

I am in search of measuremenet information from real-world routers
or real-world networks about how the packet drop rate varies as a
function of packet size.  E.g., I assume that for some paths and
for some configured routers, large (data) and small (voip, TCP ack)
packets are dropped with the same probability, and that for other
paths and other configured routers, small packets are much less
likely to be dropped than large packets.  This depends on whether
the queue is in byte mode or packet mode, on whether the AQM mechanism
is in byte or packet mode, whether the congested router uses per-flow
or FIFO scheduling, etc.

Does anyone know of any data about this?

This is explained more in Section 10 on "Drop or Mark Rates as a
Function of Packet Size" in "Tools for the Evaluation of Simulation
and Testbed Scenarios", available from "http://www.icir.org/tmrg/".

(I also need to know this for a draft of mine on congestion control
for VoIP traffic, "TCP Friendly Rate Control (TFRC) for Voice: VoIP
Variant and Faster Restart"
("http://www.ietf.org/internet-drafts/draft-ietf-dccp-tfrc-voip-02.txt").)

Many thanks.

- Sally
"http://www.icir.org/tmrg/

From: [email-masked] ([email-masked])
Date: Tue, 20 Sep 2005 23:16:53 -0300 (BRT)
Subject: [Tmrg-interest] Goodput x throughput
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Dear Sally Floyd,

after reading the draft-irtf-tmrg-metrics I would like to suggest
to distinguish between "throughput" and "goodput"

Throughput is normally used as the rate of bytes/packet transmitted

whereas "goodput" is normally used for the rate of useful bytes/packet
transmitted.

As a general comment, I feel that precise defintion of the metric of
interest would help the readership to understad what is meant by the
metrics.

nelson Fonseca

> Send Tmrg-interest mailing list submissions to
> 	[email-masked]
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
> or, via email, send a message with subject or body 'help' to
> 	[email-masked]
>
> You can reach the person managing the list at
> 	[email-masked]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Tmrg-interest digest..."
>
>
> Today's Topics:
>
>    1. Re: the most recent version of draft-irtf-tmrg-metrics (Janardhan
> Iyengar)
>    2. RE: the most recent version of draft-irtf-tmrg-metrics (Moore, Sean
> (Sean))
>    3. Re: the most recent version of draft-irtf-tmrg-metrics (Armando L.
> Caro, Jr.)
>    4. RE: the most recent version of draft-irtf-tmrg-metrics (Moore, Sean
> (Sean))
>
> --__--__--
>
> Message: 1
> Date: Mon, 19 Sep 2005 18:02:45 -0400 (EDT)
> From: Janardhan Iyengar <[email-masked]>
> To: Sally Floyd <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
>
> Sally,
>
> This is a very meaningful task, and long in waiting.
>
> I have a suggestion regarding the metric "packet drop rates". IMHO, it is
> more meaningful to have a metric, if possible, a probability distribution,
> that is rich enough to capture more than just a "rate of packet drops".
> I'm open to what this metric should be, but I do feel that this definition
> should be richer.
>
> At the least, a metric such as a "packet drop event rate" which captures
> the burstiness of drops could be included, in addition to "packet drop
> rate".
>
> I'm interested in hearing what others have to say.
>
> regards,
> jana
>
>
> On Mon, 29 Aug 2005, Sally Floyd wrote:
>
>> I have put a copy of draft-irtf-tmrg-metrics-01a.txt,
>> under construction for draft-irtf-tmrg-metrics-01, on the tmrg
>> web site.  It contains the modifications that I made today.
>> People giving feedback, or suggesting references to add,
>> might want to refer to this as the most version (even
>> though it won't be submitted to the internet-drafts editor
>> until it has stabilized a bit).
>>
>> http://www.icir.org/tmrg/draft-irtf-tmrg-metrics-01a.txt
>>
>> - Sally
>> http://www.icir.org/tmrg/
>>
>> _______________________________________________
>> Tmrg-interest mailing list
>> [email-masked]
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>>
>
> ---------------------------------------------------------------
> Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
> Protocol Engineering Lab  --   CIS   --  University Of Delaware
> ---------------------------------------------------------------
>
>
> --__--__--
>
> Message: 2
> Subject: RE: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
> Date: Mon, 19 Sep 2005 19:59:15 -0400
> From: "Moore, Sean \(Sean\)" <[email-masked]>
> To: "Janardhan Iyengar" <[email-masked]>,
>         "Sally Floyd" <[email-masked]>
> Cc: <[email-masked]>
>
> I have a suggestion.
>
> When computing quality measures for VoIP flows, a similar issue arises
> in which the average packet loss rate doesn't capture the quality
> problems that arise from bursty packet loss.  RFC 3611 describes a VoIP
> performance reporting standard called RTCP XR (an extension of RTCP).
> RTCP XR has defined packet loss burst metrics.  Maybe these are a good
> starting point from which to derive packet drop burst metrics.
>
> - Sean Moore
>
> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Janardhan
> Iyengar
> Sent: Monday, September 19, 2005 6:03 PM
> To: Sally Floyd
> Cc: [email-masked]
> Subject: Re: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
>
> Sally,
>
> This is a very meaningful task, and long in waiting.
>
> I have a suggestion regarding the metric "packet drop rates". IMHO, it
> is more meaningful to have a metric, if possible, a probability
> distribution, that is rich enough to capture more than just a "rate of
> packet drops".
> I'm open to what this metric should be, but I do feel that this
> definition should be richer.
>
> At the least, a metric such as a "packet drop event rate" which captures
> the burstiness of drops could be included, in addition to "packet drop
> rate".
>
> I'm interested in hearing what others have to say.
>
> regards,
> jana
>
>
> On Mon, 29 Aug 2005, Sally Floyd wrote:
>
>> I have put a copy of draft-irtf-tmrg-metrics-01a.txt, under
>> construction for draft-irtf-tmrg-metrics-01, on the tmrg web site.  It
>
>> contains the modifications that I made today.
>> People giving feedback, or suggesting references to add, might want to
>
>> refer to this as the most version (even though it won't be submitted
>> to the internet-drafts editor until it has stabilized a bit).
>>
>> http://www.icir.org/tmrg/draft-irtf-tmrg-metrics-01a.txt
>>
>> - Sally
>> http://www.icir.org/tmrg/
>>
>> _______________________________________________
>> Tmrg-interest mailing list
>> [email-masked]
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>>
>
> ---------------------------------------------------------------
> Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
> Protocol Engineering Lab  --   CIS   --  University Of Delaware
> ---------------------------------------------------------------
>
> _______________________________________________
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>
>
>
>
> --__--__--
>
> Message: 3
> Date: Tue, 20 Sep 2005 09:28:51 -0400
> From: "Armando L. Caro, Jr." <[email-masked]>
> To: "Moore, Sean \(Sean\)" <[email-masked]>
> CC: [email-masked]
> Subject: Re: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
>
> Moore, Sean (Sean) wrote:
>>
>> When computing quality measures for VoIP flows, a similar issue arises
>> in which the average packet loss rate doesn't capture the quality
>> problems that arise from bursty packet loss.  RFC 3611 describes a VoIP
>> performance reporting standard called RTCP XR (an extension of RTCP).
>> RTCP XR has defined packet loss burst metrics.  Maybe these are a good
>> starting point from which to derive packet drop burst metrics.
>
> Those metrics seem very useful. However, I wonder if for TCP/SCTP
> traffic, burst duration and gap duration should be measured in the
> number of packets instead of absolute time. Also, burst density and gap
> density (as specified in that RFC) are calculated since the beginning of
> a flow. For long lived flows, this metric will not respond quickly
> enough to more recent burst events. I think the densities should be
> calculated over some moving window of time, but I don't know how large
> or dynamic the window should be.
>
> --
> Armando
> www.armandocaro.net
>
> --__--__--
>
> Message: 4
> Subject: RE: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
> Date: Tue, 20 Sep 2005 10:02:50 -0400
> From: "Moore, Sean \(Sean\)" <[email-masked]>
> To: "Armando L. Caro, Jr." <[email-masked]>
> Cc: <[email-masked]>
>
> This makes sense.
>
> I should probably be more clear about my suggestion that "Maybe these
> are a good starting point from which to derive packet drop burst
> metrics."  Overall, the burst metrics for UDP flows sourced by inelastic
> applications such as voice, which is the target of RTCP XR, and the
> burst-metrics for TCP-mediated flows will be different.  But we should
> look for areas of overlap and then be consistent with RFC 3611 in the
> overlapping areas, and derive the appropriate extensions to cover the
> non-overlapping areas.  It looks like Armando has already identified
> some possible extensions.
>
> Armando touched on another point that I think we need to explore
> further.  I believe much of the congestion control research and
> literature is focused on environments in which a large share of the
> traffic is composed of long-lived flows.  My personal definition of a
> long-lived flow is one that transitions out of the slow-start phase into
> the congestion avoidance phase.  My practical definition of a long-lived
> flow is that the associated file needs to be about 40Kbytes in size
> (this heuristic is based on  default Windows wmax values of 12 packets,
> an Ethernet MTU of 1500 bytes, and T1 access).  A lot of (most?)
> TCP-mediated traffic today is web-based, and since a typical web page
> results in multiple TCP connections, a typical underlying file size is
> less than 40Kbyte.  The inferrence is that it is likely that a large
> share (most) of the TCP-mediated traffic in a network is sourced by
> flows that never make it out of slow-start.
>
> In such an environment, do our typical assumptions and solutions for
> fair sharing hold up?  Does anyone know of any literature on this
> subject?  Does this have any impact on our proposed evaluation metrics?
>
> - Sean
>
> -----Original Message-----
> From: Armando L. Caro, Jr. [mailto:[email-masked]]
> Sent: Tuesday, September 20, 2005 9:29 AM
> To: Moore, Sean (Sean)
> Cc: [email-masked]
> Subject: Re: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
>
> Moore, Sean (Sean) wrote:
>>
>> When computing quality measures for VoIP flows, a similar issue arises
>
>> in which the average packet loss rate doesn't capture the quality
>> problems that arise from bursty packet loss.  RFC 3611 describes a
>> VoIP performance reporting standard called RTCP XR (an extension of
> RTCP).
>> RTCP XR has defined packet loss burst metrics.  Maybe these are a good
>
>> starting point from which to derive packet drop burst metrics.
>
> Those metrics seem very useful. However, I wonder if for TCP/SCTP
> traffic, burst duration and gap duration should be measured in the
> number of packets instead of absolute time. Also, burst density and gap
> density (as specified in that RFC) are calculated since the beginning of
> a flow. For long lived flows, this metric will not respond quickly
> enough to more recent burst events. I think the densities should be
> calculated over some moving window of time, but I don't know how large
> or dynamic the window should be.
>
> --
> Armando
> www.armandocaro.net
>
>
>
>
>
> --__--__--
>
> _______________________________________________
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>
>
> End of Tmrg-interest Digest
>

From: [email-masked] (Sally Floyd)
Date: Tue, 20 Sep 2005 14:38:04 -0700
Subject: [Tmrg-interest] the most recent version of draft-irtf-tmrg-metrics
Message-ID: <[email-masked]>

>Armando touched on another point that I think we need to explore
>further.  I believe much of the congestion control research and
>literature is focused on environments in which a large share of the
>traffic is composed of long-lived flows.  
...
>Does this have any impact on our proposed evaluation metrics?

Throughput metrics are one of the metrics that can be used to 
evaluate the performance of congestion control mechanisms for
short flows.

As an example, Quick-Start is a proposal to allow connections to
start up faster than slow-start
("http://www.icir.org/floyd/quickstart.html"), and the Quick-Start
paper evaluates Quick-Start partly in terms of the transfer time
for a range of transfer sizes.

- Sally

From: [email-masked] (Sally Floyd)
Date: Tue, 20 Sep 2005 14:27:06 -0700
Subject: [Tmrg-interest] metrics for packet drop rates
Message-ID: <[email-masked]>

Here is the current (just-modified) text from draft-irtf-tmrg-metrics
on packet drop rates:

>3.1.3.  Packet Drop Rates
>       
>    Packet drop rates can be measured as a network-based or as a user-  
>    based metric.
>       
>    When evaluating the effect of packet drops or ECN marks (Explicit   
>    Congestion Notification, RFC 3168) on the performance of a
>    congestion control mechanism for an individual flow, researchers
>    often use both the packet loss/mark rate for that connection, and   
>    the congestion event rate (also called the loss event rate), where a
>    congestion event or loss event consists of one or more lost or
>    marked packets in one round-trip time [RFC 3448].
>    
>    Some users might care about packet drop rates only in so far as they
>    affect per-connection transfer times, while other users might care  
>    about packet drop rates directly.  RFC 3611, RTP Control Protocol   
>    Extended Reports, describes a VoIP performance-reporting standard   
>    called RTCP XR, which includes a set of burst metrics.  In RFC 3611,
>    a burst is defined as the maximal sequence starting and ending with
>    a lost packet, and not including a sequence of Gmin or more packets
>    that are not lost [RFC3611].  The burst metrics in RFC 3611 consist
>    of the burst density (the fraction of packets in bursts), gap
>    density (the fraction of packets in the gaps between bursts), burst
>    duration (the mean duration of bursts in seconds), and gap duration
>    (the mean duration of gaps in seconds).
>
>    One network-related reason to avoid high steady-state packet drop
>    rates is to avoid congestion collapse in environments containing
>    paths with multiple congested links.  In such environments, high
>    packet drop rates could result in congested links wasting scarce
>    bandwidth by carrying packets that will only be dropped downstream,
>    before being delivered to the receiver.

(with names added to the Acknowlegments section...)

- Sally

From: [email-masked] (Sally Floyd)
Date: Tue, 20 Sep 2005 13:40:25 -0700
Subject: [Tmrg-interest] metrics for packet drop rates
Message-ID: <[email-masked]>

>From Sean Moore:
>RFC 3611 describes a VoIP
>performance reporting standard called RTCP XR (an extension of RTCP).
>RTCP XR has defined packet loss burst metrics.  Maybe these are a good
>starting point from which to derive packet drop burst metrics.

Great, I will add a pointer to that.  

And I agree with Armando that the packet drop rate metrics useful
for VoIP are not necessarily the same as the general packet drop
rate metrics useful for TCP and SCTP applications.

For TCP and TFRC, the two most common packet drop rate metrics are
simply the overall packet drop rate (for the link or for the flow),
and the loss event rate/congestion event rate (for an individual
connection, where a congestion event, is one or more congestion
events (lost or marked packets) in a window of data).  So I will add
a discussion of these two metrics, and see what else emerges from
the mailing list in terms of metrics for loss bursts. 

My own guess would be that in terms of the effect of packet drops
on congestion control performance, the packet drop rate and the
congestion event rate are generally sufficient.  But they don't
necessarily suffice as metrics for evaluating the effect of packet
drops on the application.

- Sally

From: [email-masked] (Sally Floyd)
Date: Tue, 20 Sep 2005 13:14:56 -0700
Subject: [Tmrg-interest] Please don't use "Tmrg-interest digest" subject headings...
Message-ID: <[email-masked]>

Some people in tmrg-interest receive each email message separately,
and others receive daily digests, with a subject heading of
"Tmrg-interest digest".  If you receive daily digests, please use
a subject heading for your mail that is more content-related - e.g.,
the subject heading of one of the email messages in that daily
digest.  

This should make it easier in the long run for people to scan mail
on the mailing list, when there might be discussions at the same
time about several different topics, all appearing in the same daily
digest...

- Sally
acting as list moderator...

From: [email-masked] ([email-masked])
Date: Tue, 20 Sep 2005 16:30:27 -0300 (BRT)
Subject: [Tmrg-interest] Re: Tmrg-interest digest, Vol 1 #47 - 4 msgs
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Dear All,

Packet drop rate should be included as a mean value

metrics to describe loss bursts can also be included.
Mean and variance of the size of the loss burst can be useful
to better characterize the loss pattern

In ATM specification, there was a metric called rate of block of
cells severely damaged which gives the mean rate of loss bursts

nelson fonseca

> Send Tmrg-interest mailing list submissions to
> 	[email-masked]
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
> or, via email, send a message with subject or body 'help' to
> 	[email-masked]
>
> You can reach the person managing the list at
> 	[email-masked]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Tmrg-interest digest..."
>
>
> Today's Topics:
>
>    1. Re: the most recent version of draft-irtf-tmrg-metrics (Janardhan
> Iyengar)
>    2. RE: the most recent version of draft-irtf-tmrg-metrics (Moore, Sean
> (Sean))
>    3. Re: the most recent version of draft-irtf-tmrg-metrics (Armando L.
> Caro, Jr.)
>    4. RE: the most recent version of draft-irtf-tmrg-metrics (Moore, Sean
> (Sean))
>
> --__--__--
>
> Message: 1
> Date: Mon, 19 Sep 2005 18:02:45 -0400 (EDT)
> From: Janardhan Iyengar <[email-masked]>
> To: Sally Floyd <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
>
> Sally,
>
> This is a very meaningful task, and long in waiting.
>
> I have a suggestion regarding the metric "packet drop rates". IMHO, it is
> more meaningful to have a metric, if possible, a probability distribution,
> that is rich enough to capture more than just a "rate of packet drops".
> I'm open to what this metric should be, but I do feel that this definition
> should be richer.
>
> At the least, a metric such as a "packet drop event rate" which captures
> the burstiness of drops could be included, in addition to "packet drop
> rate".
>
> I'm interested in hearing what others have to say.
>
> regards,
> jana
>
>
> On Mon, 29 Aug 2005, Sally Floyd wrote:
>
>> I have put a copy of draft-irtf-tmrg-metrics-01a.txt,
>> under construction for draft-irtf-tmrg-metrics-01, on the tmrg
>> web site.  It contains the modifications that I made today.
>> People giving feedback, or suggesting references to add,
>> might want to refer to this as the most version (even
>> though it won't be submitted to the internet-drafts editor
>> until it has stabilized a bit).
>>
>> http://www.icir.org/tmrg/draft-irtf-tmrg-metrics-01a.txt
>>
>> - Sally
>> http://www.icir.org/tmrg/
>>
>> _______________________________________________
>> Tmrg-interest mailing list
>> [email-masked]
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>>
>
> ---------------------------------------------------------------
> Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
> Protocol Engineering Lab  --   CIS   --  University Of Delaware
> ---------------------------------------------------------------
>
>
> --__--__--
>
> Message: 2
> Subject: RE: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
> Date: Mon, 19 Sep 2005 19:59:15 -0400
> From: "Moore, Sean \(Sean\)" <[email-masked]>
> To: "Janardhan Iyengar" <[email-masked]>,
>         "Sally Floyd" <[email-masked]>
> Cc: <[email-masked]>
>
> I have a suggestion.
>
> When computing quality measures for VoIP flows, a similar issue arises
> in which the average packet loss rate doesn't capture the quality
> problems that arise from bursty packet loss.  RFC 3611 describes a VoIP
> performance reporting standard called RTCP XR (an extension of RTCP).
> RTCP XR has defined packet loss burst metrics.  Maybe these are a good
> starting point from which to derive packet drop burst metrics.
>
> - Sean Moore
>
> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Janardhan
> Iyengar
> Sent: Monday, September 19, 2005 6:03 PM
> To: Sally Floyd
> Cc: [email-masked]
> Subject: Re: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
>
> Sally,
>
> This is a very meaningful task, and long in waiting.
>
> I have a suggestion regarding the metric "packet drop rates". IMHO, it
> is more meaningful to have a metric, if possible, a probability
> distribution, that is rich enough to capture more than just a "rate of
> packet drops".
> I'm open to what this metric should be, but I do feel that this
> definition should be richer.
>
> At the least, a metric such as a "packet drop event rate" which captures
> the burstiness of drops could be included, in addition to "packet drop
> rate".
>
> I'm interested in hearing what others have to say.
>
> regards,
> jana
>
>
> On Mon, 29 Aug 2005, Sally Floyd wrote:
>
>> I have put a copy of draft-irtf-tmrg-metrics-01a.txt, under
>> construction for draft-irtf-tmrg-metrics-01, on the tmrg web site.  It
>
>> contains the modifications that I made today.
>> People giving feedback, or suggesting references to add, might want to
>
>> refer to this as the most version (even though it won't be submitted
>> to the internet-drafts editor until it has stabilized a bit).
>>
>> http://www.icir.org/tmrg/draft-irtf-tmrg-metrics-01a.txt
>>
>> - Sally
>> http://www.icir.org/tmrg/
>>
>> _______________________________________________
>> Tmrg-interest mailing list
>> [email-masked]
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>>
>
> ---------------------------------------------------------------
> Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
> Protocol Engineering Lab  --   CIS   --  University Of Delaware
> ---------------------------------------------------------------
>
> _______________________________________________
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>
>
>
>
> --__--__--
>
> Message: 3
> Date: Tue, 20 Sep 2005 09:28:51 -0400
> From: "Armando L. Caro, Jr." <[email-masked]>
> To: "Moore, Sean \(Sean\)" <[email-masked]>
> CC: [email-masked]
> Subject: Re: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
>
> Moore, Sean (Sean) wrote:
>>
>> When computing quality measures for VoIP flows, a similar issue arises
>> in which the average packet loss rate doesn't capture the quality
>> problems that arise from bursty packet loss.  RFC 3611 describes a VoIP
>> performance reporting standard called RTCP XR (an extension of RTCP).
>> RTCP XR has defined packet loss burst metrics.  Maybe these are a good
>> starting point from which to derive packet drop burst metrics.
>
> Those metrics seem very useful. However, I wonder if for TCP/SCTP
> traffic, burst duration and gap duration should be measured in the
> number of packets instead of absolute time. Also, burst density and gap
> density (as specified in that RFC) are calculated since the beginning of
> a flow. For long lived flows, this metric will not respond quickly
> enough to more recent burst events. I think the densities should be
> calculated over some moving window of time, but I don't know how large
> or dynamic the window should be.
>
> --
> Armando
> www.armandocaro.net
>
> --__--__--
>
> Message: 4
> Subject: RE: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
> Date: Tue, 20 Sep 2005 10:02:50 -0400
> From: "Moore, Sean \(Sean\)" <[email-masked]>
> To: "Armando L. Caro, Jr." <[email-masked]>
> Cc: <[email-masked]>
>
> This makes sense.
>
> I should probably be more clear about my suggestion that "Maybe these
> are a good starting point from which to derive packet drop burst
> metrics."  Overall, the burst metrics for UDP flows sourced by inelastic
> applications such as voice, which is the target of RTCP XR, and the
> burst-metrics for TCP-mediated flows will be different.  But we should
> look for areas of overlap and then be consistent with RFC 3611 in the
> overlapping areas, and derive the appropriate extensions to cover the
> non-overlapping areas.  It looks like Armando has already identified
> some possible extensions.
>
> Armando touched on another point that I think we need to explore
> further.  I believe much of the congestion control research and
> literature is focused on environments in which a large share of the
> traffic is composed of long-lived flows.  My personal definition of a
> long-lived flow is one that transitions out of the slow-start phase into
> the congestion avoidance phase.  My practical definition of a long-lived
> flow is that the associated file needs to be about 40Kbytes in size
> (this heuristic is based on  default Windows wmax values of 12 packets,
> an Ethernet MTU of 1500 bytes, and T1 access).  A lot of (most?)
> TCP-mediated traffic today is web-based, and since a typical web page
> results in multiple TCP connections, a typical underlying file size is
> less than 40Kbyte.  The inferrence is that it is likely that a large
> share (most) of the TCP-mediated traffic in a network is sourced by
> flows that never make it out of slow-start.
>
> In such an environment, do our typical assumptions and solutions for
> fair sharing hold up?  Does anyone know of any literature on this
> subject?  Does this have any impact on our proposed evaluation metrics?
>
> - Sean
>
> -----Original Message-----
> From: Armando L. Caro, Jr. [mailto:[email-masked]]
> Sent: Tuesday, September 20, 2005 9:29 AM
> To: Moore, Sean (Sean)
> Cc: [email-masked]
> Subject: Re: [Tmrg-interest] the most recent version of
> draft-irtf-tmrg-metrics
>
> Moore, Sean (Sean) wrote:
>>
>> When computing quality measures for VoIP flows, a similar issue arises
>
>> in which the average packet loss rate doesn't capture the quality
>> problems that arise from bursty packet loss.  RFC 3611 describes a
>> VoIP performance reporting standard called RTCP XR (an extension of
> RTCP).
>> RTCP XR has defined packet loss burst metrics.  Maybe these are a good
>
>> starting point from which to derive packet drop burst metrics.
>
> Those metrics seem very useful. However, I wonder if for TCP/SCTP
> traffic, burst duration and gap duration should be measured in the
> number of packets instead of absolute time. Also, burst density and gap
> density (as specified in that RFC) are calculated since the beginning of
> a flow. For long lived flows, this metric will not respond quickly
> enough to more recent burst events. I think the densities should be
> calculated over some moving window of time, but I don't know how large
> or dynamic the window should be.
>
> --
> Armando
> www.armandocaro.net
>
>
>
>
>
> --__--__--
>
> _______________________________________________
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>
>
> End of Tmrg-interest Digest
>

From: [email-masked] (Moore, Sean (Sean))
Date: Tue, 20 Sep 2005 10:02:50 -0400
Subject: [Tmrg-interest] the most recent version of draft-irtf-tmrg-metrics
Message-ID: <[email-masked]>

This makes sense.

I should probably be more clear about my suggestion that "Maybe these
are a good starting point from which to derive packet drop burst
metrics."  Overall, the burst metrics for UDP flows sourced by inelastic
applications such as voice, which is the target of RTCP XR, and the
burst-metrics for TCP-mediated flows will be different.  But we should
look for areas of overlap and then be consistent with RFC 3611 in the
overlapping areas, and derive the appropriate extensions to cover the
non-overlapping areas.  It looks like Armando has already identified
some possible extensions.

Armando touched on another point that I think we need to explore
further.  I believe much of the congestion control research and
literature is focused on environments in which a large share of the
traffic is composed of long-lived flows.  My personal definition of a
long-lived flow is one that transitions out of the slow-start phase into
the congestion avoidance phase.  My practical definition of a long-lived
flow is that the associated file needs to be about 40Kbytes in size
(this heuristic is based on  default Windows wmax values of 12 packets,
an Ethernet MTU of 1500 bytes, and T1 access).  A lot of (most?)
TCP-mediated traffic today is web-based, and since a typical web page
results in multiple TCP connections, a typical underlying file size is
less than 40Kbyte.  The inferrence is that it is likely that a large
share (most) of the TCP-mediated traffic in a network is sourced by
flows that never make it out of slow-start.

In such an environment, do our typical assumptions and solutions for
fair sharing hold up?  Does anyone know of any literature on this
subject?  Does this have any impact on our proposed evaluation metrics?

- Sean

-----Original Message-----
From: Armando L. Caro, Jr. [mailto:[email-masked]] 
Sent: Tuesday, September 20, 2005 9:29 AM
To: Moore, Sean (Sean)
Cc: [email-masked]
Subject: Re: [Tmrg-interest] the most recent version of
draft-irtf-tmrg-metrics

Moore, Sean (Sean) wrote:
> 
> When computing quality measures for VoIP flows, a similar issue arises

> in which the average packet loss rate doesn't capture the quality 
> problems that arise from bursty packet loss.  RFC 3611 describes a 
> VoIP performance reporting standard called RTCP XR (an extension of
RTCP).
> RTCP XR has defined packet loss burst metrics.  Maybe these are a good

> starting point from which to derive packet drop burst metrics.

Those metrics seem very useful. However, I wonder if for TCP/SCTP
traffic, burst duration and gap duration should be measured in the
number of packets instead of absolute time. Also, burst density and gap
density (as specified in that RFC) are calculated since the beginning of
a flow. For long lived flows, this metric will not respond quickly
enough to more recent burst events. I think the densities should be
calculated over some moving window of time, but I don't know how large
or dynamic the window should be.

--
Armando
www.armandocaro.net

From: [email-masked] (Armando L. Caro, Jr.)
Date: Tue, 20 Sep 2005 09:28:51 -0400
Subject: [Tmrg-interest] the most recent version of draft-irtf-tmrg-metrics
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Moore, Sean (Sean) wrote:
> 
> When computing quality measures for VoIP flows, a similar issue arises
> in which the average packet loss rate doesn't capture the quality
> problems that arise from bursty packet loss.  RFC 3611 describes a VoIP
> performance reporting standard called RTCP XR (an extension of RTCP).
> RTCP XR has defined packet loss burst metrics.  Maybe these are a good
> starting point from which to derive packet drop burst metrics.

Those metrics seem very useful. However, I wonder if for TCP/SCTP
traffic, burst duration and gap duration should be measured in the
number of packets instead of absolute time. Also, burst density and gap
density (as specified in that RFC) are calculated since the beginning of
a flow. For long lived flows, this metric will not respond quickly
enough to more recent burst events. I think the densities should be
calculated over some moving window of time, but I don't know how large
or dynamic the window should be.

-- 
Armando
www.armandocaro.net

From: [email-masked] (Moore, Sean (Sean))
Date: Mon, 19 Sep 2005 19:59:15 -0400
Subject: [Tmrg-interest] the most recent version of draft-irtf-tmrg-metrics
Message-ID: <[email-masked]>

I have a suggestion.

When computing quality measures for VoIP flows, a similar issue arises
in which the average packet loss rate doesn't capture the quality
problems that arise from bursty packet loss.  RFC 3611 describes a VoIP
performance reporting standard called RTCP XR (an extension of RTCP).
RTCP XR has defined packet loss burst metrics.  Maybe these are a good
starting point from which to derive packet drop burst metrics.

- Sean Moore

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Janardhan
Iyengar
Sent: Monday, September 19, 2005 6:03 PM
To: Sally Floyd
Cc: [email-masked]
Subject: Re: [Tmrg-interest] the most recent version of
draft-irtf-tmrg-metrics

Sally,

This is a very meaningful task, and long in waiting.

I have a suggestion regarding the metric "packet drop rates". IMHO, it
is more meaningful to have a metric, if possible, a probability
distribution, that is rich enough to capture more than just a "rate of
packet drops". 
I'm open to what this metric should be, but I do feel that this
definition should be richer.

At the least, a metric such as a "packet drop event rate" which captures
the burstiness of drops could be included, in addition to "packet drop
rate".

I'm interested in hearing what others have to say.

regards,
jana

On Mon, 29 Aug 2005, Sally Floyd wrote:

> I have put a copy of draft-irtf-tmrg-metrics-01a.txt, under 
> construction for draft-irtf-tmrg-metrics-01, on the tmrg web site.  It

> contains the modifications that I made today.
> People giving feedback, or suggesting references to add, might want to

> refer to this as the most version (even though it won't be submitted 
> to the internet-drafts editor until it has stabilized a bit).
>
> http://www.icir.org/tmrg/draft-irtf-tmrg-metrics-01a.txt
>
> - Sally
> http://www.icir.org/tmrg/
>
> _______________________________________________
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>

---------------------------------------------------------------
Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
Protocol Engineering Lab  --   CIS   --  University Of Delaware
---------------------------------------------------------------

_______________________________________________
Tmrg-interest mailing list
[email-masked]
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest

From: [email-masked] (Janardhan Iyengar)
Date: Mon, 19 Sep 2005 18:02:45 -0400 (EDT)
Subject: [Tmrg-interest] the most recent version of draft-irtf-tmrg-metrics
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Sally,

This is a very meaningful task, and long in waiting.

I have a suggestion regarding the metric "packet drop rates". IMHO, it is 
more meaningful to have a metric, if possible, a probability distribution, 
that is rich enough to capture more than just a "rate of packet drops". 
I'm open to what this metric should be, but I do feel that this definition 
should be richer.

At the least, a metric such as a "packet drop event rate" which captures 
the burstiness of drops could be included, in addition to "packet drop 
rate".

I'm interested in hearing what others have to say.

regards,
jana

On Mon, 29 Aug 2005, Sally Floyd wrote:

> I have put a copy of draft-irtf-tmrg-metrics-01a.txt,
> under construction for draft-irtf-tmrg-metrics-01, on the tmrg
> web site.  It contains the modifications that I made today.
> People giving feedback, or suggesting references to add,
> might want to refer to this as the most version (even
> though it won't be submitted to the internet-drafts editor
> until it has stabilized a bit).
>
> http://www.icir.org/tmrg/draft-irtf-tmrg-metrics-01a.txt
>
> - Sally
> http://www.icir.org/tmrg/
>
> _______________________________________________
> Tmrg-interest mailing list
> [email-masked]
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/tmrg-interest
>

---------------------------------------------------------------
Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
Protocol Engineering Lab  --   CIS   --  University Of Delaware
---------------------------------------------------------------

From: [email-masked] (Sally Floyd)
Date: Mon, 12 Sep 2005 16:44:53 -0700
Subject: [Tmrg-interest] Tools for the Evaluation of Simulation and Testbed Scenarios
Message-ID: <[email-masked]>

We have submitted a *very* rough draft of a new TMRG internet-draft,
Tools for the Evaluation of Simulation and Testbed Scenarios, and
there is a local copy at the URL below:

 http://www.icir.org/tmrg/draft-irtf-tmrg-tools-00.txt
 http://www.icir.org/tmrg/draft-irtf-tmrg-tools-00.ps

The abstract is as follows:

    This document describes tools for the evaluation of simulation and
    testbed scenarios used in research on Internet congestion control
    mechanisms.  We believe that research in congestion control
    mechanisms has been seriously hampered by the lack of good models
    underpinning analysis, simulation, and testbed experiments, and that
    tools for the evaluation of simulation and testbed scenarios can
    help in the construction of better scenarios, based on better
    underlying models.  One use of the tools described in this document
    is in comparing key characteristics of test scenarios with known
    characteristics from the diverse and ever-changing real world.
    Tools characterizing the aggregate traffic on a link include the
    distribution of per-packet round-trip times, the distribution of
    packet sequence numbers, and the like.  Tools characterizing end-to-
    end paths include drop rates as a function of packet size and of
    burst size, the synchronization ratio between two end-to-end TCP
    flows, and the like.  For each characteristic, we describe what
    aspects of the scenario determine this characteristic, how the
    characteristic can affect the results of simulations and experiments
    for the evaluation of congestion control mechanisms, and what is
    known about this characteristic in the real world.  We also explain
    why the use of such tools can add considerable power to our
    understanding and evaluation of simulation and testbed scenarios.

I am sending the draft to the mailing list at this very early state
to solicit volunteers to provide text, and/or citations to the most
recent literature, for the various sections in the document.  One
goal is to clarify what measurements are already largely in place
(e.g., the distribution of connection sizes, or of round-trip times)
and what measurements are desperately needed (e.g., the packet drop
rate as a function of packet size, or the synchronization ratio for
high-bandwidth protocols over realistic high-bandwidth networks.)

Any general feedback about the overall direction of the draft would
also be useful.

My hope is for the group to use the information in this draft, as
it progresses, to guide the collection and creation of
best-current-practice simulation and testbed scenarios for evaulating
congestion control mechanisms.

Many thanks.

- Sally
http://www.icir.org/tmrg/