Subject: [rmcat] Fwd: New Version Notification for draft-singh-rmcat-cc-eval-00.txt

From: Varun Singh <[email-masked]>
Date: Mon, 15 Oct 2012 23:33:37 +0300

Greetings,

We have submitted an initial draft outlining the criteria for evaluating
congestion control algorithms. The metrics section is a bit light
on details and will to expand on that in the next revision.

We hope that the draft is a good starting point to get the discussion
started about metrics and scenarios for simulation and testbed experiments.

Cheers,
Varun

Begin forwarded message:

A new version of I-D, draft-singh-rmcat-cc-eval-00.txt
has been successfully submitted by Varun Singh and posted to the
IETF repository.

Filename:	 draft-singh-rmcat-cc-eval
Revision:	 00
Title:		 Evaluating Congestion Control for Interactive Real-time Media.
Creation date:	 2012-10-15
WG ID:		 Individual Submission
Number of pages: 9
URL:
http://www.ietf.org/internet-drafts/draft-singh-rmcat-cc-eval-00.txt
Status:          http://datatracker.ietf.org/doc/draft-singh-rmcat-cc-eval
Htmlized:        http://tools.ietf.org/html/draft-singh-rmcat-cc-eval-00

Abstract:
  The Real-time Transport Protocol (RTP) is used to transmit media in
  telephony and video conferencing applications.  This document
  describes the guidelines to evaluate new congestion control
  algorithms for interactive point-to-point real-time media.

The IETF Secretariat

---

From: Luca De Cicco <[email-masked]>
Date: Tue, 16 Oct 2012 11:01:21 +0200

Dear Varun,

I've read the draft and I have a couple of comments.

1) regarding the definition of stability which is given in the draft:

Stability: is the period of time when the endpoint's encoding
         rate is relatively stable, i.e., the bandwidth utilization is
         constant.

I would use "steady-state" instead of "stability" since the latter has a
formal definition which has a precise meaning. Moreover, it has to be
noticed that "steady-state" can be reached if and only if the
available bandwidth is constant.

2) for what concerns the video quality indices, it is well known
that PSNR is not well related to the QoE. (See for instance
http://www.pscr.gov/about_pscr/press/video/video_quality_experts_interview_sigmm-122009.pdf)
Are we sure we want to cite PSNR?

Cheers,
--
Luca De Cicco, PhD, Eng.
Politecnico di Bari
Dipartimento di Elettrotecnica ed Elettronica
Via Re David, 200 - Bari - ITALY
Office: +39 080 596 3851

---

From: Varun Singh <[email-masked]>
Date: Tue, 16 Oct 2012 13:57:12 +0300

Hi Luca,

Thanks for your comments, responses inline.

On Tue, Oct 16, 2012 at 12:01 PM, Luca De Cicco <[email-masked]> wrote:
[snip]
> formal definition which has a precise meaning. Moreover, it has to be

True, will change stability to steady-state.

> available bandwidth is constant.

I defined steady-state as a period when bandwidth utilization is
constant (not close to 1).
Apart from the situation when the congestion control is operating
close to the bottleneck capacity, it can also have constant utilization
if it is under-utilizing the link or detects cross-traffic and
attempts to be fair.

The metrics section has intentionally been left a bit open for discussion
because I want to tune the definitions to match the exact requirements
of the congestion control.

> Are we sure we want to cite PSNR?

I believe we need some objective metric for video either
PSNR or something else. I am happy to include or
update the document based on that discussion.
(Note: I have not read the above document but
will go through it.)

However, when choosing these quality metrics we should be aware
of their defined operating range (some models are limited to specific
range of video resolutions or bit rates, etc.).

Cheers,
Varun

-- 
http://www.netlab.tkk.fi/~varun/

---

From: Stefan Holmer <[email-masked]>
Date: Tue, 16 Oct 2012 13:29:14 +0200

> range of video resolutions or bit rates, etc.).
I agree that PSNR isn't likely to be a good choice, I would assume that we
at least need a metric which takes both the video quality and the delay
into account? The experienced video quality depends on several aspects
other than the congestion control, such as encoder settings, transmitted
spatial and temporal resolution, video jitter buffer algorithm at the
receiver, etc. Therefore I'm not sure it's a good idea to evaluate our
congestion control algorithms based on video quality. If possible I would
rather see us concentrate on simpler metrics as those you described earlier
in your draft, which are more related to how the algorithm actually adapts
to changes in bandwidth.

> >> Status:
> http://datatracker.ietf.org/doc/draft-singh-rmcat-cc-eval
> >> Htmlized:
> http://tools.ietf.org/html/draft-singh-rmcat-cc-eval-00
> >>
> >>
> >> Abstract:

---

From: Piers O'Hanlon <[email-masked]>
Date: Wed, 17 Oct 2012 12:25:08 +0100

Hi Varun,

The draft looks like a good start.

I think it would good to add evaluation of idle and data-limited behaviours - as these are important as regards media transport - they were the reasons the original TFRC (RFC3448) was updated to RFC5348. See: http://dx.doi.org/10.1016%2fj.comcom.2011.05.003
"TCP-Friendly Rate Control (TFRC) for bursty media flows", Arjuna Sathiaseelan, Gorry Fairhurst, Comcom 2011

Furthermore it would also be useful to evaluate the algorithms with respect to their operation in terms of competing against a small number of flows vs a large number - see: 
On the Effectiveness of Delay-Based Congestion Avoidance, Ravi S. Prasad, Manish Jain Georgia, Constantinos Dovrolis, PFLDnet 2004

Cheers,

Piers

---

From: Varun Singh <[email-masked]>
Date: Fri, 19 Oct 2012 13:11:49 +0300

Hi Piers,

Comments inline,

> "TCP-Friendly Rate Control (TFRC) for bursty media flows", Arjuna Sathiaseelan, Gorry Fairhurst, Comcom 2011

Do you mean the audio media traffic should be modelled as having idle
periods? and video because of I and P frames will have bursty
behaviour?

> On the Effectiveness of Delay-Based Congestion Avoidance, Ravi S. Prasad, Manish Jain Georgia, Constantinos Dovrolis, PFLDnet 2004
In [S5], I have an editors note:
[many and few short TCP flows may depend on the bottleneck link capacity.]

I can add a line to both cross traffic and self fairness guideline
(Section 4) to evaluate the performance with small number and large
number of competing flows.

Cheers,
Varun

---

From: Piers O'Hanlon <[email-masked]>
Date: Fri, 19 Oct 2012 12:58:44 +0100

Hi Varun,

> periods? and video because of I and P frames will have bursty
Yes both media can have idle and data-limited periods due to silence or low motion (depending upon codec type etc) - If we're trying stay away from adding full QoE evaluation these terms describe more generic behaviours that should be addressed when testing a congestion control scheme. These were some of the original complaints about TFRC operation that lead to the formation of this group.

These behaviours are also linked to the corresponding TCP Congestion Window Validation techniques (e.g. RFC2861)

> number of competing flows.
That sounds good. This issue is more of a concern as most of the proposed schemes are likely to be primarily based upon some sort of delay measurement which can become more problematic in certain scenarios as mentioned in the short paper (and elsewhere).

Cheers,

Piers

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 22 Oct 2012 12:50:10 +0000

Hi,

you may also want to take a look at the following paper, which attempted to define some community-agreed evaluation scenarios for proposed TCP extensions. Maybe there are some aspects in there that are applicable to your draft.

Towards a Common TCP Evaluation Suite. Lachlan Andrew, Cesar Marcondes, Sally Floyd, Lawrence Dunn, Wang Gang, Lars Eggert, Sangtae Ha and Injong Rhee. Proc. International Workshop on Protocols for Fast Long-Distance Networks (PFLDnet),Manchester, UK, March 5-7, 2008. http://eggert.org/papers/2008-pfldnet.pdf

Lars