Subject: Re: [rmcat] How to handle an increasing send buffer

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Tue, 11 Nov 2014 23:53:02 +0000

Regarding what’s the best way to model the encoder behavior …

We submitted a new draft on synthetic video traffic modeling for RMCAT evaluation, which contains some discussions what behaviors we want to model for such a codec (Sec 3):

https://tools.ietf.org/html/draft-zhu-rmcat-video-traffic-source-00

Thought I forwarded the new draft submission to the mailing list before but turns out I never did (Thanks to Varun for pointing this out).

I’ll give a short presentation on this draft in tomorrow’s meeting. In the meantime, comments are welcome on the draft.

Thanks,
Xiaoqing

From: Ingemar Johansson S <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, November 11, 2014 at 3:17 AM
To: "Michael Ramalho (mramalho)" <[email-masked]<mailto:[email-masked]>>, "rmcat WG ([email-masked]<mailto:[email-masked]>)" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [rmcat] How to handle an increasing send buffer

Thanks all, for the response.
As I understand from what I read there is a certain resistance against modeling more exotic things like frame skipping. Good thing is that it probably makes the SCReAM draft simpler as it is possible to remove some parts.

Besides this I am not sure what is the best approach here when it comes to the best way to model the video coder.
We have seen in our experiment implementation that there is a sluggishness in the rate response, so I don’t know how good it is to model a video coder with an instant (the next frame) response, there is after all a risk that one start to over-optimize against unrealistic models.

/Ingemar

From: Michael Ramalho (mramalho) [mailto:[email-masked]]
Sent: den 10 november 2014 19:36
To: Ingemar Johansson S; rmcat WG ([email-masked]<mailto:[email-masked]>)
Subject: RE: How to handle an increasing send buffer

Ingemar,

We discussed a lot of these points at the interim meeting yesterday that you attended. In the end, I think Karen and Mirja had proposed a correct way to proceed – essentially to involve video encoding experts.

With that context, let me add my $0.02 with “MAR:” below. Thanks for your email.

Re: Perhaps there are more alternatives to the above, the SCReAM draft discuss alternative 1 and 2. This is a somewhat tricky area and one have every reason to argue whether or not this should be a part of the RMCAT charter.

MAR: There are a significant number of possibilities outside those you mention. It seems like the only encoder tricks you mention are to drop frames, drop packets and create “Full Intra Refresh or similar.” As mentioned in the meeting yesterday, many encoders will have information beyond what the transport knows about – and if it knew what the transport is requiring of it (e.g., what the transport would prioritize or preferentially send if it cannot send everything), it could employ the many concealment techniques at its disposal. So while I agree the issue of designing a good overall QoE is a joint optimization problem between transport and encoding, neither side (encoder or source transport side) has deep knowledge of the other. Nor should they; as we would desire RMCAT candidates to operate with future, yet-to-be-developed video standards.

Re: As I said at the interim it can be difficult to treat the network congestion control and the video rate control (including the sender buffer buffer control) as two black boxes, at least if one wish to create an optimal solution, so I guess some discussion on API is needed, not sure if RMCAT is the correct venue for this though.

MAR: In the end it is up to the vendor to create a differentiated solution, one that links encoder behaviors (that are allowed/consistent with the video standard being used) resulting in “video encoder output” within the envelope that the transport communicates to the encoder that it is prepared to accommodate. This is similar, although obviously more complicated, than flow control of other transports (e.g., send windows and the like). I do NOT believe that it is the job or role of the RMCAT WG to ensure the best possible solution with a given set of fixed encoder assumption – rather only to provide a transport that allows for such joint optimization (in this case of QoE) across the transport/encoding layers. Note, this is not new – Karen has lots of experience with SCTP (and it’s PR variants) that are essentially going through “API creation” for specific application needs (e.g. “priority policy” in https://tools.ietf.org/html/draft-ietf-tsvwg-sctp-prpolicies-05 ).

Michael Ramalho

From: rmcat [mailto:[email-masked]] On Behalf Of Ingemar Johansson S
Sent: Sunday, November 09, 2014 10:12 PM
To: rmcat WG ([email-masked]<mailto:[email-masked]>)
Subject: [rmcat] How to handle an increasing send buffer

Hi

Both SCReAM and NADA argues for buffering on the sender side to handle the the difference between the video coder output bitrate and the actual (or estimated) throughput of the transmission path. This is particularly useful esp. in cases where the link thorughput drops dramatically.
The problem with this of course that, while this is very helpful in order to keep the network delay well behaved, the additional queueing of Video RTP packets can in principle cause very high video frame delays unless handled properly.
There are, as I understand it a number of ways to keep the video frame bounded.  Parts of this was already discussed at the interim but it could be good to get this discussion on the list as well.

1)      Drop RTP packets in the send buffer: Easy to implement , requires that the codec state is repaired. The good part here is that this is known already on the sender side so there is no need to way an RTT (+ a little extra) for e.g an RFC4585 generic NACK or something. A possible problem is that the codec state may be costly to repair as Full Intra Refresh may be needed.

2)      Frame skipping : Raw video frames are dropped, i.e not forwarded to the video coder. The frame rate is therefore temporarly reduced. The video codec state is preserved but the prediction can may suffer because of a larger difference between the frame. SCReAM limits the number of consequtive frame skips to avoid too large differences. It is however not known hos different video coders will react to this. Another problem is API related

3)      Sender detects a too large buffer, discards entire (or part of) send buffer and generates a Full Intra Refresh or similar, pretty similar to 1)

4)      Receiver sends an RFC4585 PLI, the sender discards entire (or part of) send buffer and generates a Full Intra Refresh or similar.

5)      Other ?

6)      Other 2 ?

7)      Do nothing, the consequence is that it may take time then for the video frame delay to become reasonably low

Perhaps there are more alternatives to the above, the SCReAM draft discuss alternative 1 and 2. This is a somewhat tricky area and one have every reason to argue whether or not this should be a part of the RMCAT charter. As I said at the interim it can be difficult to treat the network congestion control and the video rate control (including the sender buffer buffer control) as two black boxes, at least if one wish to create an optimal solution, so I guess some discussion on API is needed, not sure if RMCAT is the correct venue for this though.

Comments welcome
/Ingemar

=================================
Ingemar Johansson  M.Sc.
Senior Researcher

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked]<mailto:[email-masked]>
www.ericsson.com

“The difference between stupidity and
genius is that genius has its limits."
Albert Einstein
=================================

---

From: Randell Jesup <[email-masked]>
Date: Tue, 11 Nov 2014 21:10:10 -0500

On 11/11/2014 6:53 PM, Xiaoqing Zhu (xiaoqzhu) wrote:
> We submitted a new draft on synthetic video traffic modeling for RMCAT 
> evaluation, which contains some discussions what behaviors we want to 
> model for such a codec (Sec 3):
>
> Thought I forwarded the new draft submission to the mailing list 
> before but turns out I never did (Thanks to Varun for pointing this out).

I would NOT have the congestion controller send "drop frame" commands.  
It should indicate:
* Estimated bitrate share
* Target bitrate (assuming continuous generation)
* Estimated additional one-way-delay above base estimate
and/or estimated number of buffered bits at bottleneck
* Loss indication/level (not for direct recovery - that's handled by 
other RTCP messages - but to tune the generated traffic

For the synthetic algorithm, I suggest that the fake encoder drop 
limited number of frames when the overhang (bits-behind-the-8-ball, 
number of bits buffered at bottleneck, or additional OWD*bitrate - all 
ways to say the same thing) is on the order of multiple encoded frames.  
This moves the frame skipping out where it belongs, in the app/encoder 
(which we're abstracting here).

Errors when reported back to the sender would invoke one of several 
recovery mechanisms:
* retransmit - congestion-invoked loss means retransmit will need to 
steal N bits from your available total in the next encoding cycle, 
perhaps plus more.
* FEC - means loss may not cause an error result, at the code of 
"burning" some of your bandwidth.   Adaptive FEC will use more bits if 
errors are higher (or are anticipated to, see Varun's paper), so 
differences between algorithms in resultant error rates will affect 
quality (reduced by using more FEC bits) and number of recoveries-by-IDR 
(when FEC fails).
* IDR request - PLI/etc - classic recovery, causes a burst of traffic 
and in many (not all) cases may be best followed by a frame drop or two 
or three.  Obviously the size of the burst will affect the delay, but 
also affects quality.  This can be compensated for without waiting for a 
delay signal.  Note that dropping  frames is NOT good if we're not 
causing or near a bottleneck.

Since errors caused by the congestion mechanism affect the sending codec 
rates, they're important.

-- 
Randell Jesup -- rjesup a t mozilla d o t com

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 12 Nov 2014 08:43:48 +0000

I encourage folks to read and give feedback on the app interaction draft which covers all of this.

The interaction between the congestion control and encoder is limited to things like Allowed Rate (CC to codec), Media Elasticity (codec to CC), etc.

Codec decisions like changing frame rate, frame type, quantization, etc. are not part of the CC-codec interaction, so no solution candidate should include such things.

Simulated codecs should have the exact same interactions with CC as real ones, and their internal behavior should model real ones. For example, simulate changes in frame rate, frame type, quantization, etc. with proportional changes in synthetic output size and frequency.

Mahalo!
Mo

On 11/11/14, 9:10 PM, Randell Jesup <[email-masked]<mailto:[email-masked]>> wrote:

I would NOT have the congestion controller send "drop frame" commands.  It should indicate:
* Estimated bitrate share
* Target bitrate (assuming continuous generation)
* Estimated additional one-way-delay above base estimate
and/or estimated number of buffered bits at bottleneck
* Loss indication/level (not for direct recovery - that's handled by other RTCP messages - but to tune the generated traffic

For the synthetic algorithm, I suggest that the fake encoder drop limited number of frames when the overhang (bits-behind-the-8-ball, number of bits buffered at bottleneck, or additional OWD*bitrate - all ways to say the same thing) is on the order of multiple encoded frames.  This moves the frame skipping out where it belongs, in the app/encoder (which we're abstracting here).

Errors when reported back to the sender would invoke one of several recovery mechanisms:
* retransmit - congestion-invoked loss means retransmit will need to steal N bits from your available total in the next encoding cycle, perhaps plus more.
* FEC - means loss may not cause an error result, at the code of "burning" some of your bandwidth.   Adaptive FEC will use more bits if errors are higher (or are anticipated to, see Varun's paper), so differences between algorithms in resultant error rates will affect quality (reduced by using more FEC bits) and number of recoveries-by-IDR (when FEC fails).
* IDR request - PLI/etc - classic recovery, causes a burst of traffic and in many (not all) cases may be best followed by a frame drop or two or three.  Obviously the size of the burst will affect the delay, but also affects quality.  This can be compensated for without waiting for a delay signal.  Note that dropping  frames is NOT good if we're not causing or near a bottleneck.

Since errors caused by the congestion mechanism affect the sending codec rates, they're important.