Subject: [rmcat] How to handle an increasing send buffer

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 10 Nov 2014 08:12:13 +0000

Hi

Both SCReAM and NADA argues for buffering on the sender side to handle the the difference between the video coder output bitrate and the actual (or estimated) throughput of the transmission path. This is particularly useful esp. in cases where the link thorughput drops dramatically.
The problem with this of course that, while this is very helpful in order to keep the network delay well behaved, the additional queueing of Video RTP packets can in principle cause very high video frame delays unless handled properly.
There are, as I understand it a number of ways to keep the video frame bounded.  Parts of this was already discussed at the interim but it could be good to get this discussion on the list as well.

1)      Drop RTP packets in the send buffer: Easy to implement , requires that the codec state is repaired. The good part here is that this is known already on the sender side so there is no need to way an RTT (+ a little extra) for e.g an RFC4585 generic NACK or something. A possible problem is that the codec state may be costly to repair as Full Intra Refresh may be needed.

2)      Frame skipping : Raw video frames are dropped, i.e not forwarded to the video coder. The frame rate is therefore temporarly reduced. The video codec state is preserved but the prediction can may suffer because of a larger difference between the frame. SCReAM limits the number of consequtive frame skips to avoid too large differences. It is however not known hos different video coders will react to this. Another problem is API related

3)      Sender detects a too large buffer, discards entire (or part of) send buffer and generates a Full Intra Refresh or similar, pretty similar to 1)

4)      Receiver sends an RFC4585 PLI, the sender discards entire (or part of) send buffer and generates a Full Intra Refresh or similar.

5)      Other ?

6)      Other 2 ?

7)      Do nothing, the consequence is that it may take time then for the video frame delay to become reasonably low

Perhaps there are more alternatives to the above, the SCReAM draft discuss alternative 1 and 2. This is a somewhat tricky area and one have every reason to argue whether or not this should be a part of the RMCAT charter. As I said at the interim it can be difficult to treat the network congestion control and the video rate control (including the sender buffer buffer control) as two black boxes, at least if one wish to create an optimal solution, so I guess some discussion on API is needed, not sure if RMCAT is the correct venue for this though.

Comments welcome
/Ingemar

=================================
Ingemar Johansson  M.Sc.
Senior Researcher

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked]<mailto:[email-masked]>
www.ericsson.com

"The difference between stupidity and
genius is that genius has its limits."
Albert Einstein
=================================

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Mon, 10 Nov 2014 18:36:00 +0000

Ingemar,

We discussed a lot of these points at the interim meeting yesterday that you attended. In the end, I think Karen and Mirja had proposed a correct way to proceed - essentially to involve video encoding experts.

With that context, let me add my $0.02 with "MAR:" below. Thanks for your email.

Re: Perhaps there are more alternatives to the above, the SCReAM draft discuss alternative 1 and 2. This is a somewhat tricky area and one have every reason to argue whether or not this should be a part of the RMCAT charter.

MAR: There are a significant number of possibilities outside those you mention. It seems like the only encoder tricks you mention are to drop frames, drop packets and create "Full Intra Refresh or similar." As mentioned in the meeting yesterday, many encoders will have information beyond what the transport knows about - and if it knew what the transport is requiring of it (e.g., what the transport would prioritize or preferentially send if it cannot send everything), it could employ the many concealment techniques at its disposal. So while I agree the issue of designing a good overall QoE is a joint optimization problem between transport and encoding, neither side (encoder or source transport side) has deep knowledge of the other. Nor should they; as we would desire RMCAT candidates to operate with future, yet-to-be-developed video standards.

Re: As I said at the interim it can be difficult to treat the network congestion control and the video rate control (including the sender buffer buffer control) as two black boxes, at least if one wish to create an optimal solution, so I guess some discussion on API is needed, not sure if RMCAT is the correct venue for this though.

MAR: In the end it is up to the vendor to create a differentiated solution, one that links encoder behaviors (that are allowed/consistent with the video standard being used) resulting in "video encoder output" within the envelope that the transport communicates to the encoder that it is prepared to accommodate. This is similar, although obviously more complicated, than flow control of other transports (e.g., send windows and the like). I do NOT believe that it is the job or role of the RMCAT WG to ensure the best possible solution with a given set of fixed encoder assumption - rather only to provide a transport that allows for such joint optimization (in this case of QoE) across the transport/encoding layers. Note, this is not new - Karen has lots of experience with SCTP (and it's PR variants) that are essentially going through "API creation" for specific application needs (e.g. "priority policy" in https://tools.ietf.org/html/draft-ietf-tsvwg-sctp-prpolicies-05 ).

Michael Ramalho

From: rmcat [mailto:[email-masked]] On Behalf Of Ingemar Johansson S
Sent: Sunday, November 09, 2014 10:12 PM
To: rmcat WG ([email-masked])
Subject: [rmcat] How to handle an increasing send buffer

Hi

5)      Other ?

6)      Other 2 ?

Comments welcome
/Ingemar

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289

---

From: Randell Jesup <[email-masked]>
Date: Mon, 10 Nov 2014 15:45:54 -0500

On 11/10/2014 3:12 AM, Ingemar Johansson S wrote:
>
> Hi
>
> Both SCReAM and NADA argues for buffering on the sender side to handle 
> the the difference between the video coder output bitrate and the 
> actual (or estimated) throughput of the transmission path. This is 
> particularly useful esp. in cases where the link thorughput drops 
> dramatically.
>

I think there's also a strong dependence on the encoding-feedback-loop 
time - 1/2 second (as mentioned in the NADA paper) means there's a lot 
of inertia (much like RTT for feedback) to changing the rate.  If the 
encoder loop delay is basically just the framerate or less, then 
normally the send buffer should drain before the next frame is ready, 
modulo jitter in encode size (and key/iframe sizes).

For example, do you "allow" a keyframe to burst the bitrate well above 
normal, and if it does do so, do we (or the encoder application) skip 
frames to compensate for the burst?  That's not an easy question: if 
we're near bottleneck or worse link rate, the burst will take N frames 
to be received and decoded (i.e. you'll skip frames *before* the 
keyframe at the receiver).  An obvious solution is to drop N-1 frames 
after a keyframe - but that may not be the best user experience.  Also 
the encoder has the ability to control the burst size via quantization 
(i.e. N), and the answers about this may vary with resolution, frame 
rate and bitrate.  And if we're below the expected bandwidth (never 
found a top, or are still well below the expected bandwidth) it may be 
best to just let it burst and keep sending at full rate.  For example, 
my FiOS link can handle any reasonable upstream burst (90Mbps!), as can 
my LAN.  You don't want to drop frames there, and output 
pacing/buffering will hurt final quality.  These are complex, 
user-perception-interactive issues we shouldn't rathole into in the 
algorithm design.

There are many strategies an encoder framework can use to best leverage 
the transport stream.   You don't want the transport too tied to 
specific strategies (like frame drops), but instead provide the inputs 
to the encoder to make use of this.  For example, don't only produce a 
single number of current encoder rate target, but provide information 
about the expected RTT, current estimated bandwidth (which may not be 
the same as current suggested encoder target), current estimated 
added-one-way-delay above observed minimum and estimated number of bits 
of overhang (Bits Behind the 8-Ball (BB8B) ;-) ).    There's no 
requirement that the transport provide any of this other than an encoder 
target: but providing it allows for better end-to-end quality.

Because of this, I think it would be easy to too-closely tie the 
transport design to the application/encoder design.

I think output pacing should be disabled or nearly so when we haven't 
found a max rate or believe we're well below it.  And I think it should 
be considerably limited even in other cases (and allow packets to drain 
out faster than the line estimate, at least for a while).  The main 
advantage of pacing would be allowing feedback packets to "jump" the 
queue and avoid being stuck behind a keyframe burst, for example.

P.S.
Side note, minor: output pacing may cause some added display delay, even 
if we're at bottleneck rates and sharing it with other streams.

First: paced output, one stream, estimate is 1Mbps, but link actually 
can handle 2Mbps and is running no extra buffer - last packet to go out 
for a frame is almost a full frame-time delayed compared to the first 
packet, and frequently would be received 1/2 frame later than it could 
be if sent as a burst, which has a 50% chance of pushing you to 1 frame 
later in compositing.

Second, with pacing with 4 streams sharing: each frame is 4 packets, for 
example 1a, 1b, 1c, 1d.  The frame can't be decoded and displayed until 
all packets are received.  Frame rate is (let's say) 30fps.

With "perfect" pacing, perfectly aligned with vsync/compositing ('|' is 
where a compositing operation occurs)
| 1a 2a 3a 4a 1b 2b 3b 4b 1c 2c 3c 4c 1d 2d 3d 4d |
Random alignment (normal, and it may drift):
1a 2a 3a 4a 1b 2b 3b 4b 1c 2c 3c | 4c 1d 2d 3d 4d ...                   
      |
Note that no frame of any of the video streams will be rendered until 
the second | (and I'm ignoring that if the receivers are all different, 
then each will have it's own randomly-places composite opertation).
With burst-send instead of pacing, "perfect" alignment:
| 1a 1b 1c 1d 2a 2b 2c 2d 3a 3b 3c 3d 4a 4b 4c 4d |
With random alignment:
| 1a 1b 1c 1d 2a 2b 2c 2d 3a 3b | 3c 3d 4a 4b 4c 4d 
...                                   |
Note that 1 and 2 will get rendered to the screen one frame earlier than 
with pacing; 3 & 4 end up the same as before.

-- 
Randell Jesup -- rjesup a t mozilla d o t com

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 11 Nov 2014 13:17:44 +0000

Thanks all, for the response.
As I understand from what I read there is a certain resistance against modeling more exotic things like frame skipping. Good thing is that it probably makes the SCReAM draft simpler as it is possible to remove some parts.

Besides this I am not sure what is the best approach here when it comes to the best way to model the video coder.
We have seen in our experiment implementation that there is a sluggishness in the rate response, so I don't know how good it is to model a video coder with an instant (the next frame) response, there is after all a risk that one start to over-optimize against unrealistic models.

/Ingemar

From: Michael Ramalho (mramalho) [mailto:[email-masked]]
Sent: den 10 november 2014 19:36
To: Ingemar Johansson S; rmcat WG ([email-masked])
Subject: RE: How to handle an increasing send buffer

Ingemar,

Michael Ramalho

From: rmcat [mailto:[email-masked]] On Behalf Of Ingemar Johansson S
Sent: Sunday, November 09, 2014 10:12 PM
To: rmcat WG ([email-masked]<mailto:[email-masked]>)
Subject: [rmcat] How to handle an increasing send buffer

Hi

5)      Other ?

6)      Other 2 ?

Comments welcome
/Ingemar

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289