Subject: [rmcat] draft-vanderauwera-rmcat-video-quality

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 14 Oct 2013 09:42:07 +0000

Hi WG,

please see http://trac.tools.ietf.org/html/draft-vanderauwera-rmcat-video-quality-00 for a new ID describing quality test scenarios.

Please discuss.

Lars

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 15 Oct 2013 09:50:12 +0000

Hi

I have a question as regards to the size of the tail drop queue (32 packets). Given the bandwidth you end up with one way delays up to ~400ms when the link is loaded with an FTP download. Is the 32 packet value given by some typical hardware considerations ?. 
Given the bandwidth delay product (1Mbps, RTT=2*10ms) it should be sufficient with a much lower queue size ?

/Ingemar
> -----Original Message-----
> From: Eggert, Lars [mailto:[email-masked]]
> Sent: den 14 oktober 2013 11:42
> To: rmcat WG
> Subject: [rmcat] draft-vanderauwera-rmcat-video-quality
> 
> Hi WG,
> 
> please see http://trac.tools.ietf.org/html/draft-vanderauwera-rmcat-video-
> quality-00 for a new ID describing quality test scenarios.
> 
> Please discuss.
> 
> Lars

---

From: "Van Der Auwera, Geert" <[email-masked]>
Date: Tue, 15 Oct 2013 18:50:28 +0000

Dear Ingemar,

Thank you for your feedback. The queue size parameter in the test scenarios can be defined smaller as you suggest to reduce delay. I based my estimate on 1200 byte packets and 1Mbps, which results in a delay of ~300ms. There are no particular hardware considerations for us, but I verified that for some typical router the queue length can be configured between 0 and 4096 packets (default seems 75), for example.

I would also like to use this opportunity to introduce the document by providing the introduction here and hopefully have some discussions in the RMCAT WG:

"The impact of congestion control on media streams is important in real-world deployments. Therefore, the purpose of this evaluation is to verify or inspect the video quality of RMCAT congestion control solutions under different test scenarios. RMCAT based rate adaptations should result in expected or desired video quality when implemented in a state-of-the-art video coding and communication framework. In other words, the purpose is to verify that there are no unexpected or undesired video quality problems caused by congestion control behaviors, while the main purpose is not to directly compare RMCAT solutions against each other. It is proposed to perform such video quality evaluation of candidate RMCAT solutions in addition to network traffic evaluations.

An additional test scenario is proposed to evaluate the rate allocation behavior of RMCAT solutions when asymmetric flow conditions exist. In this test the video quality is not evaluated."

Best regards,

Geert

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Tuesday, October 15, 2013 2:50 AM
To: Van Der Auwera, Geert
Cc: Eggert, Lars; rmcat WG
Subject: RE: [rmcat] draft-vanderauwera-rmcat-video-quality

Hi

/Ingemar

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 16 Oct 2013 06:42:02 +0000

Hi

Thanks for the reply.
I assumed 1500byte packeys which gives almost 400ms delay. 
As I see it it seems a bit pessimistic with a ratio 30-40 between latency under load and min RTT. 
I am fully aware of the bufferbloat problem but isn't there a risk that this becomes a corner case and that it might drive designs in a direction that makes them suboptimal for the main operating conditions.

To take an example: In order for the RMCAT flow to be able to compete with an FTP using TCP Reno or Cubic over this bottleneck, it becomes necessary that the RMCAT flow contributes to the bufferbloat for instance by increasing a target delay or whatever. Unless the RMCAT flow gets some magic information from the network, it needs to infer information about the existence of competing flows, it can work more or less well and I see before be a lot of special code to do this, some of it actually becomes engineered to solve this particular test case. 
The problem with this is that this added intelligence can actually increase the risk that the RMCAT flow it self creates self-inflicted congestion due to faulty decisions, meaning that the RMCAT flow it self locks it self in 400ms RTT even though there is no competing traffic.

I leave to others to comment, perhaps I outline a corner problem :-).

/Ingemar

> From: Van Der Auwera, Geert [mailto:[email-masked]]
> Sent: den 15 oktober 2013 20:50
> To: rmcat WG
> Cc: Ingemar Johansson S; Eggert, Lars
> Thank you for your feedback. The queue size parameter in the test scenarios
> can be defined smaller as you suggest to reduce delay. I based my estimate
> on 1200 byte packets and 1Mbps, which results in a delay of ~300ms. There
> are no particular hardware considerations for us, but I verified that for some
> typical router the queue length can be configured between 0 and 4096
> packets (default seems 75), for example.
> 
> I would also like to use this opportunity to introduce the document by
> providing the introduction here and hopefully have some discussions in the
> RMCAT WG:
> 
> "The impact of congestion control on media streams is important in real-
> world deployments. Therefore, the purpose of this evaluation is to verify or
> inspect the video quality of RMCAT congestion control solutions under
> different test scenarios. RMCAT based rate adaptations should result in
> expected or desired video quality when implemented in a state-of-the-art
> video coding and communication framework. In other words, the purpose is
> to verify that there are no unexpected or undesired video quality problems
> caused by congestion control behaviors, while the main purpose is not to
> directly compare RMCAT solutions against each other. It is proposed to
> perform such video quality evaluation of candidate RMCAT solutions in
> addition to network traffic evaluations.
> 
> An additional test scenario is proposed to evaluate the rate allocation
> behavior of RMCAT solutions when asymmetric flow conditions exist. In this
> test the video quality is not evaluated."
> 
> Best regards,
> 
> Geert
> 
> I have a question as regards to the size of the tail drop queue (32 packets).
> Given the bandwidth you end up with one way delays up to ~400ms when
> the link is loaded with an FTP download. Is the 32 packet value given by some
> typical hardware considerations ?.
> Given the bandwidth delay product (1Mbps, RTT=2*10ms) it should be
> sufficient with a much lower queue size ?
> 
> /Ingemar

---

From: Michael Welzl <[email-masked]>
Date: Wed, 16 Oct 2013 09:06:32 +0200

> To take an example: In order for the RMCAT flow to be able to compete with an FTP using TCP Reno or Cubic over this bottleneck, it becomes necessary that the RMCAT flow contributes to the bufferblo…

Measured delay can serve as this "magic information"...

> The problem with this is that this added intelligence can actually increase the risk that the RMCAT flow it self creates self-inflicted congestion due to faulty decisions, meaning that the RMCAT fl…

That would be the result of a badly designed congestion controller. I don't think that this naturally follows from trying to minimize delay, yet compete with TCP when it is present.

> /Ingemar

Cheers,
Michael

---

From: Manuel Gorius <[email-masked]>
Date: Wed, 16 Oct 2013 11:05:27 +0200

> That would be the result of a badly designed congestion controller. I don't think that this naturally follows from trying to minimize delay, yet compete with TCP when it is present.
Designing a sophisticated congestion controller in order to address this problem is still challenging, though (are there any such solutions available so far?). We performed experiments with delay-based congestion control where we ran into exactly the same issue: The intensity of the delay-based flows had to be increased in order to compete with loss-based TCP-flows in environments with huge buffers. At the same time it is probably desirable that the queues aren't filled excessively in case there is only delay-controlled traffic. I think for a delay-based decision, the controller has to learn the range of the delay variation in such environments (maybe incorporating loss-based feedback as well) and must derive a suitable weighting for the congestion feedback in order to keep the flow competitive under an increasing number of loss-based flows. Having a mixture of loss- and delay-based feedback could solve the problem of self-inflicted congestion in my opinion.

Best regards,
Manuel

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 17 Oct 2013 09:14:02 +0000

Hi

I believe that the closest to some kind of an adaptive  delay-based to loss-based congestion control is the Hamilton Delay TCP (thanks for the heads up Mirja!). 
http://eprints.nuim.ie/1745/1/HamiltonDelayAIMD.pdf 
I have not experimented with it though so I can't comment on the usefulness. 

/Ingemar

> From: Manuel Gorius [mailto:[email-masked]]
> Sent: den 16 oktober 2013 11:05
> To: Michael Welzl
> Cc: Ingemar Johansson S; Van Der Auwera, Geert; rmcat WG; Eggert, Lars
> Subject: Re: [rmcat] draft-vanderauwera-rmcat-video-quality
> 
> 
> >> I am fully aware of the bufferbloat problem but isn't there a risk that this
> becomes a corner case and that it might drive designs in a direction that
> makes them suboptimal for the main operating conditions.
> >>
> >> To take an example: In order for the RMCAT flow to be able to compete
> with an FTP using TCP Reno or Cubic over this bottleneck, it becomes
> necessary that the RMCAT flow contributes to the bufferbloat for instance by
> increasing a target delay or whatever. Unless the RMCAT flow gets some
> magic information from the network, it needs to infer information about the
> existence of competing flows, it can work more or less well and I see before
> be a lot of special code to do this, some of it actually becomes engineered to
> solve this particular test case.
> >
> >> The problem with this is that this added intelligence can actually increase
> the risk that the RMCAT flow it self creates self-inflicted congestion due to
> faulty decisions, meaning that the RMCAT flow it self locks it self in 400ms RTT
> even though there is no competing traffic.
> >
> > That would be the result of a badly designed congestion controller. I don't
> think that this naturally follows from trying to minimize delay, yet compete
> with TCP when it is present.
> >
> Designing a sophisticated congestion controller in order to address this
> problem is still challenging, though (are there any such solutions available so
> far?). We performed experiments with delay-based congestion control
> where we ran into exactly the same issue: The intensity of the delay-based
> flows had to be increased in order to compete with loss-based TCP-flows in
> environments with huge buffers. At the same time it is probably desirable
> that the queues aren't filled excessively in case there is only delay-controlled
> traffic. I think for a delay-based decision, the controller has to learn the range
> of the delay variation in such environments (maybe incorporating loss-based
> feedback as well) and must derive a suitable weighting for the congestion
> feedback in order to keep the flow competitive under an increasing number
> of loss-based flows. Having a mixture of loss- and delay-based feedback
> could solve the problem of self-inflicted congestion in my opinion.
> 
> Best regards,
> Manuel

---

From: "Van Der Auwera, Geert" <[email-masked]>
Date: Fri, 18 Oct 2013 17:45:41 +0000

Thank you for the paper; it also has an interesting reference list. Buffer size is an important parameter and it should indeed be chosen carefully in the test scenarios. There seems to be the need to evaluate congestion control under several buffer size conditions. On the other hand, it is better not to have an excessive amount of test cases. What's the feeling on two buffer sizes in the tests and are there any suggestions, for example, 10 and 32 packets (1Mbps link)?

Best regards,

Geert

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Thursday, October 17, 2013 2:14 AM
To: Manuel Gorius; Michael Welzl
Cc: Van Der Auwera, Geert; rmcat WG; Eggert, Lars
Subject: RE: [rmcat] draft-vanderauwera-rmcat-video-quality

Hi

/Ingemar

> >> As I see it it seems a bit pessimistic with a ratio 30-40 between 
> >> latency
> under load and min RTT.
> becomes a corner case and that it might drive designs in a direction 
> that makes them suboptimal for the main operating conditions.
> >>
> necessary that the RMCAT flow contributes to the bufferbloat for 
> instance by increasing a target delay or whatever. Unless the RMCAT 
> flow gets some magic information from the network, it needs to infer 
> information about the existence of competing flows, it can work more 
> or less well and I see before be a lot of special code to do this, 
> some of it actually becomes engineered to solve this particular test case.
> >
> the risk that the RMCAT flow it self creates self-inflicted congestion 
> due to faulty decisions, meaning that the RMCAT flow it self locks it 
> self in 400ms RTT even though there is no competing traffic.
> >
> think that this naturally follows from trying to minimize delay, yet 
> compete with TCP when it is present.
> >
> Designing a sophisticated congestion controller in order to address 
> this problem is still challenging, though (are there any such 
> solutions available so far?). We performed experiments with 
> delay-based congestion control where we ran into exactly the same 
> issue: The intensity of the delay-based flows had to be increased in 
> order to compete with loss-based TCP-flows in environments with huge 
> buffers. At the same time it is probably desirable that the queues 
> aren't filled excessively in case there is only delay-controlled 
> traffic. I think for a delay-based decision, the controller has to 
> learn the range of the delay variation in such environments (maybe 
> incorporating loss-based feedback as well) and must derive a suitable 
> weighting for the congestion feedback in order to keep the flow 
> competitive under an increasing number of loss-based flows. Having a mixture of loss- and delay-based feedback could solve the problem of self-inflicted congestion in my opinion.
> 
> Best regards,
> Manuel