Subject: Re: [rmcat] How to handle an increasing send buffer

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Tue, 11 Nov 2014 00:31:46 +0000

Adding to the list of mechanisms for confining local sending buffer depth/delay:

In NADA the further the video codec is notified of a target rates well below the rate calculated from congestion control (R_n) ; likewise the sending rate goes above that specified by congestion control (R_n) by a fair amount. This helps to deplete the queue over time, at the expense of slightly dislodging the congestion control feedback loop over the network.

Note that one can apple this mechanism in conjunction of the other mechanisms (esp. dropping RTP packets or skipping frames).

Before we decide how important this issue is, I think we need to gather some good survey data on how fast a typical modern codec — designed/configured to support live conferencing — reacts to abrupt changes in target rates.

I also fully agree that interaction between the congestion control feedback look over the network and the local encoder rate control is a) tricky business to get right; b) important for the overall performance of the video conferencing application. Just that I think we should decouple this discussion from the core congestion control algorithm (what rate to send given a certain set of network observations).

-Xiaoqing

From: Ingemar Johansson S <[email-masked]<mailto:[email-masked]>>
Date: Sunday, November 9, 2014 at 10:12 PM
To: "rmcat WG ([email-masked]<mailto:[email-masked]>)" <[email-masked]<mailto:[email-masked]>>
Subject: [rmcat] How to handle an increasing send buffer

Hi

Both SCReAM and NADA argues for buffering on the sender side to handle the the difference between the video coder output bitrate and the actual (or estimated) throughput of the transmission path. This is particularly useful esp. in cases where the link thorughput drops dramatically.
The problem with this of course that, while this is very helpful in order to keep the network delay well behaved, the additional queueing of Video RTP packets can in principle cause very high video frame delays unless handled properly.
There are, as I understand it a number of ways to keep the video frame bounded.  Parts of this was aAddnlready discussed at the interim but it could be good to get this discussion on the list as well.

1)      Drop RTP packets in the send buffer: Easy to implement , requires that the codec state is repaired. The good part here is that this is known already on the sender side so there is no need to way an RTT (+ a little extra) for e.g an RFC4585 generic NACK or something. A possible problem is that the codec state may be costly to repair as Full Intra Refresh may be needed.

2)      Frame skipping : Raw video frames are dropped, i.e not forwarded to the video coder. The frame rate is therefore temporarly reduced. The video codec state is preserved but the prediction can may suffer because of a larger difference between the frame. SCReAM limits the number of consequtive frame skips to avoid too large differences. It is however not known hos different video coders will react to this. Another problem is API related

3)      Sender detects a too large buffer, discards entire (or part of) send buffer and generates a Full Intra Refresh or similar, pretty similar to 1)

4)      Receiver sends an RFC4585 PLI, the sender discards entire (or part of) send buffer and generates a Full Intra Refresh or similar.

5)      Other ?

6)      Other 2 ?

7)      Do nothing, the consequence is that it may take time then for the video frame delay to become reasonably low

Perhaps there are more alternatives to the above, the SCReAM draft discuss alternative 1 and 2. This is a somewhat tricky area and one have every reason to argue whether or not this should be a part of the RMCAT charter. As I said at the interim it can be difficult to treat the network congestion control and the video rate control (including the sender buffer buffer control) as two black boxes, at least if one wish to create an optimal solution, so I guess some discussion on API is needed, not sure if RMCAT is the correct venue for this though.

Comments welcome
/Ingemar

=================================
Ingemar Johansson  M.Sc.
Senior Researcher

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked]<mailto:[email-masked]>
www.ericsson.com

“The difference between stupidity and
genius is that genius has its limits."
Albert Einstein
=================================

---

From: Randell Jesup <[email-masked]>
Date: Tue, 11 Nov 2014 00:53:17 -0500

On 11/10/2014 7:31 PM, Xiaoqing Zhu (xiaoqzhu) wrote:
> Before we decide how important this issue is, I think we need to 
> gather some good survey data on how fast a typical modern codec — 
> designed/configured to support live conferencing — reacts to abrupt 
> changes in target rates.

Well, all I can say is that the codecs I've been exposed to have 
generally responded with something on the order of the target size in 
1-2, maybe 3 frames (but I haven't looked in great detail at any 
recently - those numbers were from much older (back to 2003) pure DSP 
and DSP with HW acceleration mostly).   I have seen some that were 
designed for streaming or local camera-like recording that are slower.  
(I've also seen some that change only on an iframe, but I consider that 
a bug and not suitable for interactive use.)

Thus the main determinator of delay in response was "where in the cycle" 
the encoder is - has it submitted a frame for encoding already?  If so, 
you may wait up to encode-time + interframe-time + encode-time.  Typical 
would be ~ 1/2 that.

I think expected/average encoder-response delay is reasonable tuning 
parameter for an algorithm.  I would suggest we assume a low delay 
(circa 33ms max, effectively next-frame) in our evaluations to avoid 
apples-and-oranges comparisons.  It might be faster than a given 
real-world codec will respond, of course.  It's probably a good idea for 
any algorithm proposer to check for algorithm stability for all tuning 
values in any case.

Others with a wider hardware exposure may chime in, but I think right 
now the main questions/targets would be OpenH264 and libvpx, and 
comments from Cisco/Ericsson/etc.  I'll admit I have not deeply examined 
the behavior of libvpx/openh264 beyond seeing that their response time 
seems acceptable given the current controls we're using.  Android/etc 
targeted chipsets would be interesting, but I'll note that actually 
using such hardware codecs in an unregulated context (eg Android) is 
fraught with landmines, as many of these were created and tested for 
non-interactive use.  (Witness the "change bitrates on iframe" issue I 
mentioned.)

-- 
Randell Jesup -- rjesup a t mozilla d o t com

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Tue, 11 Nov 2014 06:43:00 +0000

I like the suggestion of considering a low-delay (e.g., bounded by a single frame interval) case as the baseline scenario for our evaluations. It is reasonable to assume that a codec designed for interactive applications will respond to rate change requests as early as it can.

However, a distinction needs to be made between when it responds to a change, vs. when the output rate converges to the new target rate.  I am guessing that the later will take a bit more time, maybe converging over a couple of frames.  I will also ask my colleagues working on OpenH264 to regarding its behavior and report back later.

Best,
Xiaoqing

From: Randell Jesup <[email-masked]<mailto:[email-masked]>>
Date: Monday, November 10, 2014 at 7:53 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [rmcat] How to handle an increasing send buffer

Well, all I can say is that the codecs I've been exposed to have generally responded with something on the order of the target size in 1-2, maybe 3 frames (but I haven't looked in great detail at any recently - those numbers were from much older (back to 2003) pure DSP and DSP with HW acceleration mostly).   I have seen some that were designed for streaming or local camera-like recording that are slower.  (I've also seen some that change only on an iframe, but I consider that a bug and not suitable for interactive use.)

Thus the main determinator of delay in response was "where in the cycle" the encoder is - has it submitted a frame for encoding already?  If so, you may wait up to encode-time + interframe-time + encode-time.  Typical would be ~ 1/2 that.

I think expected/average encoder-response delay is reasonable tuning parameter for an algorithm.  I would suggest we assume a low delay (circa 33ms max, effectively next-frame) in our evaluations to avoid apples-and-oranges comparisons.  It might be faster than a given real-world codec will respond, of course.  It's probably a good idea for any algorithm proposer to check for algorithm stability for all tuning values in any case.

Others with a wider hardware exposure may chime in, but I think right now the main questions/targets would be OpenH264 and libvpx, and comments from Cisco/Ericsson/etc.  I'll admit I have not deeply examined the behavior of libvpx/openh264 beyond seeing that their response time seems acceptable given the current controls we're using.  Android/etc targeted chipsets would be interesting, but I'll note that actually using such hardware codecs in an unregulated context (eg Android) is fraught with landmines, as many of these were created and tested for non-interactive use.  (Witness the "change bitrates on iframe" issue I mentioned.)

---

From: Piers O'Hanlon <[email-masked]>
Date: Tue, 11 Nov 2014 10:59:14 +0000

It would seem that there's a number of key codecs parameters that could be relevant to rmcat. A good starting point might be to base them upon a common group of functions offered by some well known codec APIs such as OpenMAX, Apple's CoreVideo, and Android's MediaCodec. After all even if a new codec comes along it's going to have to fit into one of these APIs to be usable - I guess the soon-to-reconvene netvc BoF may also have some views on this...

Just to add - in my experience I've also seen non-realtime bias in code libraries such as libx264 which - when I was trying to use it (about 4-5yrs ago) - required a codec restart to change things like target bitrate. I would hope it's changed but as Randell points out alot of codecs have traditionally been streaming oriented.

Cheers,

Piers

On 11 Nov 2014, at 06:43, Xiaoqing Zhu (xiaoqzhu) wrote:

> From: Randell Jesup <[email-masked]>
> To: "[email-masked]" <[email-masked]>

---

From: Dave Taht <[email-masked]>
Date: Tue, 11 Nov 2014 16:13:03 -0800

On Nov 10, 2014 9:54 PM, "Randell Jesup" <[email-masked]> wrote:
>> Before we decide how important this issue is, I think we need to gather
some good survey data on how fast a typical modern codec —
designed/configured to support live conferencing — reacts to abrupt changes
in target rates.
> Well, all I can say is that the codecs I've been exposed to have
generally responded with something on the order of the target size in 1-2,
maybe 3 frames (but I haven't looked in great detail at any recently -
those numbers were from much older (back to 2003) pure DSP and DSP with HW
acceleration mostly).   I have seen some that were designed for streaming
or local camera-like recording that are slower.  (I've also seen some that
change only on an iframe, but I consider that a bug and not suitable for
interactive use.)

Well, call me old fashioned, but in my day we did it on a scan line basis,
with nearly no encoding, and got rid of all that latency at the expense of
using up way more bandwidth.

> Thus the main determinator of delay in response was "where in the cycle"
the encoder is - has it submitted a frame for encoding already?  If so, you
may wait up to encode-time + interframe-time + encode-time.  Typical would
be ~ 1/2 that.
> I think expected/average encoder-response delay is reasonable tuning
parameter for an algorithm.  I would suggest we assume a low delay (circa
33ms max, effectively next-frame) in our evaluations to avoid
apples-and-oranges comparisons.  It might be faster than a given real-world
codec will respond, of course.  It's probably a good idea for any algorithm
proposer to check for algorithm stability for all tuning values in any case.

I would kind of like to avoid baking a hard 33ms figure into everything,
given that higher frame rates are feasible, (consider games that run today
at higher rates already, and that an ideal face to face conversation in the
real world is less than 1ms...)

But it seems a reasonable major figure to circle around.

>
> Others with a wider hardware exposure may chime in, but I think right now
the main questions/targets would be OpenH264 and libvpx, and comments from
Cisco/Ericsson/etc.  I'll admit I have not deeply examined the behavior of
libvpx/openh264 beyond seeing that their response time seems acceptable
given the current controls we're using.  Android/etc targeted chipsets
would be interesting, but I'll note that actually using such hardware
codecs in an unregulated context (eg Android) is fraught with landmines, as
many of these were created and tested for non-interactive use.  (Witness
the "change bitrates on iframe" issue I mentioned.)