Subject: [Moq] Latency strategy

From: Luke Curley <[email-masked]>
Date: Mon, 15 Nov 2021 12:36:18 -0800

It's been great seeing the discussion about congestion control. I'd like to
start the conversation about something more video-specific but just as
important.

Bitrate estimation involves predicting the future. Occasionally, you will
overestimate the bitrate due to congestion and the network will not be able
to deliver the encoded media in a timely manner. How a protocol responds to
this situation is its primary means to reduce latency, while I'm calling
its "latency strategy".

I think we should be talking about these tangible approaches for reducing
latency, rather than trying to lump protocols into vague marketing terms
like "low" and "ultra-low" latency. I've done my best to simplify the many
approaches into a few different categories:

-----

The first available option is to configure the encoder.

1. *Encoder *f*eedback*. Protocols like WebRTC and RTMP work by adjusting
the encoder bitrate based on network conditions. This does not impact any
media that has already been encoded and flushed to the network layer.

2. *Produce multiple renditions*. Protocols like HLS and DASH work by
generating multiple renditions at different bitrates. The client has the
ability to switch between renditions at the next keyframe to adjust the
bitrate.

In my opinion, this is the fundamental difference between contribution and
distribution protocols (+CDN support). Encoder feedback is not feasible
when there are multiple consumers as it is expensive to encode and transmit
a unique bitstream per consumer.

-----

If you want to reduce latency, you need to go even further and drop encoded
media. This effectively reduces the bitrate faster than an encoder could
react.

a. *Drop individual frames.* Protocols like WebRTC work by deciding not to
(re)transmit frames. This will cause decoding errors if the frame is used
as a reference for a later frame. Note that TCP-based protocols like RTMP
can still drop frames, but only before they are flushed to the network
layer.

b. *Request a new keyframe*. If too many reference frames/slices have been
dropped, there will be persistent decoding errors until the next keyframe.
Protocols like WebRTC will request the  generation of a new keyframe to
exit this state sooner than waiting for the next keyframe.

c. *Drop remainder of GoP*. A protocol can drop all frames until the next
keyframe without causing decode errors. Protocols like HLS and DASH can
cancel a pending segment download, be it to switch to another rendition or
to jump ahead.

d. *Drop parts of a frame*. SVC
<https://en.wikipedia.org/wiki/Scalable_Video_Coding> gives you the ability
to drop parts of the frame to downgrade the quality or resolution. I don't
know how well this is actually supported or adopted.

The ability to drop data requires support from the network layer. This has
been the main selling point for building your own protocol on top of UDP,
but I believe all of these approaches are now possible with QUIC streams.

-----

Finally, if these approaches are not enough to account for congestion.

i. *Buffer*. Protocols like HLS and DASH will pause playback and wait until
the network has caught back up. The larger the video buffer, the larger the
latency.

ii. *Fall behind*. Protocols like RTMP will deliver media late, lagging
behind when it was supposed to be received. This is acceptable when it's
being fed to a decoder/encoder that can work faster than real-time, or it's
ultimately going to get buffered anyway.

iii. *Abandon stragglers*. Protocols like WebRTC refuse to increase latency
and will instead drop any media that arrives late.

-----

There's no approach that best supports every use-case and ultimately it
comes down to the target user experience.

That being said, there are too many video transport protocols and they all
seem to focus on just one niche. There is ample space for a protocol that
spans the latency spectrum without sacrificing simplicity.

---

From: Justin Uberti <[email-masked]>
Date: Mon, 15 Nov 2021 13:31:15 -0800

This is a great summary, Luke. I agree it would be nice to have a single
media protocol that would work across multiple classes of application.
Ultimately I think this would involve something like WebRTC's congestion
control and media recovery options (retransmit, FEC, new keyframe) in a
transport that was simpler than the current ICE/DTLS/RTP/SCTP protocol
sandwich, and from which it was easy to derive multiple renditions with
minimal latency cost (likely simulcast or layered coding).

On Mon, Nov 15, 2021 at 12:36 PM Luke Curley <[email-masked]> wrote:

> It's been great seeing the discussion about congestion control. I'd like
> to start the conversation about something more video-specific but just as
> important.
>

There's also the science of how one detects congestion is occurring, which
is linked to the actual rate prediction. Congestion controllers that are
monitoring packet arrival delays (e.g., GoogleCC) can detect that queuing
is increasing and back off before any losses occur. This seems like a
critical feature for any realtime scenario.

> a unique bitstream per consumer.

Note that WebRTC can also produce multiple renditions, either a) at the
client, e.g., simulcast or layered coding, or b), at the server, by
controlling which streams and quality levels to propagate. I tend to think
that both techniques are probably necessary to cover the full range of
potential applications (including deriving different representations from a
single inbound bitstream).

>
> -----
>
> If you want to reduce latency, you need to go even further and drop
> encoded media. This effectively reduces the bitrate faster than an encoder
> could react.
>
> a. *Drop individual frames.* Protocols like WebRTC work by deciding not
> to (re)transmit frames. This will cause decoding errors if the frame is
> used as a reference for a later frame. Note that TCP-based protocols like
> RTMP can still drop frames, but only before they are flushed to the network
> layer.
>

If RTT is low enough, WebRTC will retransmit frames, and this works well at
scale even in sub-100ms-latency applications (e.g. gaming).

> <https://en.wikipedia.org/wiki/Scalable_Video_Coding> gives you the
> ability to drop parts of the frame to downgrade the quality or resolution.
> I don't know how well this is actually supported or adopted.
>

This is supported at scale in most conferencing applications, sometimes
through simpler mechanisms like simulcast. Sometimes, the critical frames
can also be reconstructed via Forward Error Correction.

> i. *Buffer*. Protocols like HLS and DASH will pause playback and wait
> until the network has caught back up. The larger the video buffer, the
> larger the latency.
>
> iii. *Abandon stragglers*. Protocols like WebRTC refuse to increase
> latency and will instead drop any media that arrives late.
>

WebRTC has configurable latency. If the jitter buffer depth is high enough,
media that arrives 100s of ms late can still be played out. I have seen
this used in view-only cases where the desired latency was ~1s, rather than
the typical 150ms for interactive applications.

> --
> Moq mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/moq
>

---

From: "Ali C. Begen" <[email-masked]>
Date: Tue, 16 Nov 2021 12:38:53 +0300

Luke, Justin: Some comments and questions inline.

> On Nov 16, 2021, at 12:31 AM, Justin Uberti <[email-masked]> wrote:
> 
> This is a great summary, Luke. I agree it would be nice to have a single media protocol that would work across multiple classes of application. Ultimately I think this would involve something like WebRTC's congestion control and media recovery options (retransmit, FEC, new keyframe) in a transport that was simpler than the current ICE/DTLS/RTP/SCTP protocol sandwich, and from which it was easy to derive multiple renditions with minimal latency cost (likely simulcast or layered coding).

I am quite a bit concerned about creating an overly complicated protocol for the sake of stuffing every possible/desirable feature into one “grand” protocol. Obviously, folks on this list have a number of different scenarios and there won’t be one size that fits all. Such a grand protocol will likely have too many knobs to configure according to the specific environment/use case and that is a bigger issue for operational reasons. Most simply won’t know how to configure them and will stick with the default values.

> It's been great seeing the discussion about congestion control. I'd like to start the conversation about something more video-specific but just as important.
> 
> Bitrate estimation involves predicting the future. Occasionally, you will overestimate the bitrate due to congestion and the network will not be able to deliver the encoded media in a timely manner. How a protocol responds to this situation is its primary means to reduce latency, while I'm calling its "latency strategy".
> 
> There's also the science of how one detects congestion is occurring, which is linked to the actual rate prediction. Congestion controllers that are monitoring packet arrival delays (e.g., GoogleCC) can detect that queuing is increasing and back off before any losses occur. This seems like a critical feature for any realtime scenario.

One nit. We *measure* the bandwidth we can get at the moment and we *predict* what bandwidth we will likely get in the near future. Measurement is not easy: the data samples are often too noisy. Prediction is even worse but thankfully we only need prediction for the near short term. 

> I think we should be talking about these tangible approaches for reducing latency, rather than trying to lump protocols into vague marketing terms like "low" and "ultra-low" latency. I've done my best to simplify the many approaches into a few different categories:

Agreed.

> 1. Encoder feedback. Protocols like WebRTC and RTMP work by adjusting the encoder bitrate based on network conditions. This does not impact any media that has already been encoded and flushed to the network layer.
> 
> 2. Produce multiple renditions. Protocols like HLS and DASH work by generating multiple renditions at different bitrates. The client has the ability to switch between renditions at the next keyframe to adjust the bitrate.
> 
> In my opinion, this is the fundamental difference between contribution and distribution protocols (+CDN support). Encoder feedback is not feasible when there are multiple consumers as it is expensive to encode and transmit a unique bitstream per consumer.

Lets simplify this. Assume we have a sender, an origin and a receiver. The sender should “contribute” the best quality it can to the origin and origin can produce (transcodes) multiple renditions and the receiver picks up the best one it can (distribution). The same workflow works just fine if you have a million receivers instead of one. Everybody gets whatever they can with the help of a CDN. 

If the sender’s connection to the origin is not good enough, the sender needs to adjust accordingly (like when one is doing a live broadcast over social media from a phone). For professional content (like from a studio or stadium), there will be plenty of bandwidth, redundant paths, etc. so no issues here. 

Overall, the sender is only responsible for the contribution part and there is a single receiver here - a closed loop rate control is appropriate here. For the distribution side, there are N receivers, so each receiver must have their own rate control. If this is a 1:1 videoconference, then obviously there is no need for an origin and two endpoints can run a closed loop between them.

> Note that WebRTC can also produce multiple renditions, either a) at the client,

I am not sure what you mean by "the client can produce multiple renditions.” I suppose you mean the client decides what it should get but that still requires the server producing multiple streams in the first place.

> e.g., simulcast or layered coding, or b), at the server, by controlling which streams and quality levels to propagate. I tend to think that both techniques are probably necessary to cover the full range of potential applications (including deriving different representations from a single inbound bitstream).

Sending simulcast streams or layered streams from the sender (so that we can avoid encoding/transcoding at the origin I mentioned above) might be a good option for real-time conferencing but I argue that it is a poor choice for streaming media. Simulcast requires substantial encoding power and bandwidth. Layered encoding has little benefits for streaming video at high resolutions. I am not a codec guy but I know that if you have multiple renditions at say 5 different resolutions, you will likely need 5 base layers anyway. 
> 
> -----
> 
> If you want to reduce latency, you need to go even further and drop encoded media. This effectively reduces the bitrate faster than an encoder could react.

Discarding frames sounds easy but the size of the discardable frames is usually quite small so won’t give you much. Discarding non-discardable frames will get you errors and will make the viewers upset. This might be ok for your personal broadcast to some family members or friends but you cannot expect people to miss frames (content) for a paid service.

> a. Drop individual frames. Protocols like WebRTC work by deciding not to (re)transmit frames. This will cause decoding errors if the frame is used as a reference for a later frame. Note that TCP-based protocols like RTMP can still drop frames, but only before they are flushed to the network layer.
> 
> If RTT is low enough, WebRTC will retransmit frames, and this works well at scale even in sub-100ms-latency applications (e.g. gaming). 

I doubt the retransmitted frame will reach the destination earlier than the discarded frame would if it had not been discarded. Discarding something assuming that retransmitting it can recover it sounds illogical (I can envision cases when this might work but in most cases, it won’t).

> b. Request a new keyframe. If too many reference frames/slices have been dropped, there will be persistent decoding errors until the next keyframe. Protocols like WebRTC will request the  generation of a new keyframe to exit this state sooner than waiting for the next keyframe.

This new key frame will be substantially larger than the other frames. If the bandwidth is already lower than what you need, this might not help you as much.

> 
> c. Drop remainder of GoP. A protocol can drop all frames until the next keyframe without causing decode errors. Protocols like HLS and DASH can cancel a pending segment download, be it to switch to another rendition or to jump ahead.

Canceling an outstanding segment request is too costly - it means killing the TCP connection in http/1.1. There may be scenarios where it is still beneficial to give up and switch to a lower bitrate rendition (like when the segment is long and you are relatively closer to the beginning). I need to remind you, though, that DASH doesn’t like variable segment sizes too much (complicates the MPD, requires MPD updates, events, etc.). My comments from above about the missing content still apply here - nobody likes missing frames when watching sports for example.

>  d. Drop parts of a frame. SVC gives you the ability to drop parts of the frame to downgrade the quality or resolution. I don't know how well this is actually supported or adopted.
> 
> This is supported at scale in most conferencing applications, sometimes through simpler mechanisms like simulcast. Sometimes, the critical frames can also be reconstructed via Forward Error Correction. 

Ditto.

> The ability to drop data requires support from the network layer. This has been the main selling point for building your own protocol on top of UDP, but I believe all of these approaches are now possible with QUIC streams.

Overall, discarding stuff should not be our primary tool in meeting real-time requirements. I totally get that it works ok for gaming or conferencing but argue that we have better tools for streaming media and view-only conferencing/gaming.

> i. Buffer. Protocols like HLS and DASH will pause playback and wait until the network has caught back up. The larger the video buffer, the larger the latency.

Indeed but for live streaming, you can always use adaptive media playback so that you can catch up with the live edge once the network has caught up.

> 
> ii. Fall behind. Protocols like RTMP will deliver media late, lagging behind when it was supposed to be received. This is acceptable when it's being fed to a decoder/encoder that can work faster than real-time, or it's ultimately going to get buffered anyway.
> 
> iii. Abandon stragglers. Protocols like WebRTC refuse to increase latency and will instead drop any media that arrives late.
> 
> WebRTC has configurable latency. If the jitter buffer depth is high enough, media that arrives 100s of ms late can still be played out. I have seen this used in view-only cases where the desired latency was ~1s, rather than the typical 150ms for interactive applications.
> 
> -----
> 
> There's no approach that best supports every use-case and ultimately it comes down to the target user experience.
> 
> That being said, there are too many video transport protocols and they all seem to focus on just one niche. There is ample space for a protocol that spans the latency spectrum without sacrificing simplicity.
> 

-acbegen

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 16 Nov 2021 08:25:26 -0500

Ali said:

"Sending simulcast streams or layered streams from the sender (so that we
can avoid encoding/transcoding at the origin I mentioned above) might be a
good option for real-time conferencing but I argue that it is a poor choice
for streaming media."

[BA] We are now seeing use cases such as webinars and very large meetings
which require both large scale (100K+ participants) and low latencies (<
600 ms).  These use cases often combine aspects of RTC and streaming
technologies, so they do not clearly lie in one category or the other.

"Simulcast requires substantial encoding power and bandwidth. "

[BA] One of the recent trends has been towards enabling video ingestion
from mass market devices (e.g. mobile devices).  In these devices hardware
acceleration may be a practical requirement due to thermal or battery
constraints. In my experience, hardware encoder support for SVC remains
rare, but simulcast is possible assuming that hardware resources are
available.  Bandwidth does remain a constraint though.

"Layered encoding has little benefits for streaming video at high
resolutions."

[BA]  The potential benefits of SVC in video ingestion are:
1. Ability to bypass the transcoding step, since multiple
resolutions/framerates can be obtained by selectively dropping frames.
2. Improved resilience.  The base layer can be protected (by retransmission
and/or FEC) while upper layers need not be.  So the need to retransmit
keyframes can be reduced.

"I am not a codec guy but I know that if you have multiple renditions at
say 5 different resolutions, you will likely need 5 base layers anyway."

[BA] That would be true for simulcast, but not for spatial scalability,
where you could have a single base layer.  5 spatial layers is quite a lot
though, so you'd probably need more than 1 base layer.

On Tue, Nov 16, 2021 at 4:39 AM Ali C. Begen <ali.begen=
[email-masked]> wrote:

> I am quite a bit concerned about creating an overly complicated protocol
> for the sake of stuffing every possible/desirable feature into one “grand”
> protocol. Obviously, folks on this list have a number of different
> scenarios and there won’t be one size that fits all. Such a grand protocol
> will likely have too many knobs to configure according to the specific
> environment/use case and that is a bigger issue for operational reasons.
> Most simply won’t know how to configure them and will stick with the
> default values.
>
> > Bitrate estimation involves predicting the future. Occasionally, you
> will overestimate the bitrate due to congestion and the network will not be
> able to deliver the encoded media in a timely manner. How a protocol
> responds to this situation is its primary means to reduce latency, while
> I'm calling its "latency strategy".
> >
> > There's also the science of how one detects congestion is occurring,
> which is linked to the actual rate prediction. Congestion controllers that
> are monitoring packet arrival delays (e.g., GoogleCC) can detect that
> queuing is increasing and back off before any losses occur. This seems like
> a critical feature for any realtime scenario.
>
> One nit. We *measure* the bandwidth we can get at the moment and we
> *predict* what bandwidth we will likely get in the near future. Measurement
> is not easy: the data samples are often too noisy. Prediction is even worse
> but thankfully we only need prediction for the near short term.
>
> > I think we should be talking about these tangible approaches for
> reducing latency, rather than trying to lump protocols into vague marketing
> terms like "low" and "ultra-low" latency. I've done my best to simplify the
> many approaches into a few different categories:
>
> Agreed.
>
> > -----
> >
> > 1. Encoder feedback. Protocols like WebRTC and RTMP work by adjusting
> > 2. Produce multiple renditions. Protocols like HLS and DASH work by
> > In my opinion, this is the fundamental difference between contribution
> and distribution protocols (+CDN support). Encoder feedback is not feasible
> Lets simplify this. Assume we have a sender, an origin and a receiver. The
> sender should “contribute” the best quality it can to the origin and origin
> can produce (transcodes) multiple renditions and the receiver picks up the
> best one it can (distribution). The same workflow works just fine if you
> have a million receivers instead of one. Everybody gets whatever they can
> with the help of a CDN.
>
> If the sender’s connection to the origin is not good enough, the sender
> needs to adjust accordingly (like when one is doing a live broadcast over
> social media from a phone). For professional content (like from a studio or
> stadium), there will be plenty of bandwidth, redundant paths, etc. so no
> issues here.
>
> Overall, the sender is only responsible for the contribution part and
> there is a single receiver here - a closed loop rate control is appropriate
> here. For the distribution side, there are N receivers, so each receiver
> must have their own rate control. If this is a 1:1 videoconference, then
> obviously there is no need for an origin and two endpoints can run a closed
> loop between them.
>
> client,
> I am not sure what you mean by "the client can produce multiple
> renditions.” I suppose you mean the client decides what it should get but
> that still requires the server producing multiple streams in the first
> place.
>
> > e.g., simulcast or layered coding, or b), at the server, by controlling
> which streams and quality levels to propagate. I tend to think that both
> techniques are probably necessary to cover the full range of potential
> applications (including deriving different representations from a single
> inbound bitstream).
>
> Sending simulcast streams or layered streams from the sender (so that we
> can avoid encoding/transcoding at the origin I mentioned above) might be a
> good option for real-time conferencing but I argue that it is a poor choice
> for streaming media. Simulcast requires substantial encoding power and
> bandwidth. Layered encoding has little benefits for streaming video at high
> resolutions. I am not a codec guy but I know that if you have multiple
> renditions at say 5 different resolutions, you will likely need 5 base
> layers anyway.
> >
> > -----
> >
> Discarding frames sounds easy but the size of the discardable frames is
> usually quite small so won’t give you much. Discarding non-discardable
> frames will get you errors and will make the viewers upset. This might be
> ok for your personal broadcast to some family members or friends but you
> cannot expect people to miss frames (content) for a paid service.
>
> > a. Drop individual frames. Protocols like WebRTC work by deciding not to
> > If RTT is low enough, WebRTC will retransmit frames, and this works well
> at scale even in sub-100ms-latency applications (e.g. gaming).
>
> I doubt the retransmitted frame will reach the destination earlier than
> the discarded frame would if it had not been discarded. Discarding
> something assuming that retransmitting it can recover it sounds illogical
> (I can envision cases when this might work but in most cases, it won’t).
>
> > b. Request a new keyframe. If too many reference frames/slices have been
> This new key frame will be substantially larger than the other frames. If
> the bandwidth is already lower than what you need, this might not help you
> as much.
>
> >
> > c. Drop remainder of GoP. A protocol can drop all frames until the next
> Canceling an outstanding segment request is too costly - it means killing
> the TCP connection in http/1.1. There may be scenarios where it is still
> beneficial to give up and switch to a lower bitrate rendition (like when
> the segment is long and you are relatively closer to the beginning). I need
> to remind you, though, that DASH doesn’t like variable segment sizes too
> much (complicates the MPD, requires MPD updates, events, etc.). My comments
> from above about the missing content still apply here - nobody likes
> missing frames when watching sports for example.
>
> >  d. Drop parts of a frame. SVC gives you the ability to drop parts of
> the frame to downgrade the quality or resolution. I don't know how well
> this is actually supported or adopted.
> >
> > The ability to drop data requires support from the network layer. This
> has been the main selling point for building your own protocol on top of
> UDP, but I believe all of these approaches are now possible with QUIC
> streams.
>
> Overall, discarding stuff should not be our primary tool in meeting
> real-time requirements. I totally get that it works ok for gaming or
> conferencing but argue that we have better tools for streaming media and
> view-only conferencing/gaming.
>
> > -----
> >
> > i. Buffer. Protocols like HLS and DASH will pause playback and wait
> Indeed but for live streaming, you can always use adaptive media playback
> so that you can catch up with the live edge once the network has caught up.
>
> >
> > ii. Fall behind. Protocols like RTMP will deliver media late, lagging
> > iii. Abandon stragglers. Protocols like WebRTC refuse to increase
> > WebRTC has configurable latency. If the jitter buffer depth is high
> enough, media that arrives 100s of ms late can still be played out. I have
> seen this used in view-only cases where the desired latency was ~1s, rather
> than the typical 150ms for interactive applications.
> >
> > -----
> >
> > That being said, there are too many video transport protocols and they
> all seem to focus on just one niche. There is ample space for a protocol
> that spans the latency spectrum without sacrificing simplicity.
> >
>
> -acbegen
> --
> Moq mailing list
> [email-masked]

---

From: "Ali C. Begen" <[email-masked]>
Date: Tue, 16 Nov 2021 16:41:01 +0300

> On Nov 16, 2021, at 4:25 PM, Bernard Aboba <[email-masked]> wrote:
> 
> Ali said: 
> 
> "Sending simulcast streams or layered streams from the sender (so that we can avoid encoding/transcoding at the origin I mentioned above) might be a good option for real-time conferencing but I argue that it is a poor choice for streaming media."
> 
> [BA] We are now seeing use cases such as webinars and very large meetings which require both large scale (100K+ participants) and low latencies (< 600 ms).  These use cases often combine aspects of RTC and streaming technologies, so they do not clearly lie in one category or the other.  

If there are 100K participants on an event, I am sure 99.99% of them are just listening - cannot be even chatting in a meaningful way. The obvious question then is why would you need 600ms of latency in that case. Are they taking an action based on what they hear/see right away? If you need a latency of few hundreds of milliseconds, that is way more like RTC than streaming.

> [BA] One of the recent trends has been towards enabling video ingestion from mass market devices (e.g. mobile devices).  In these devices hardware acceleration may be a practical requirement due to thermal or battery constraints. In my experience, hardware encoder support for SVC remains rare, but simulcast is possible assuming that hardware resources are available.  Bandwidth does remain a constraint though. 

Even if you get the best device today, it won’t do more than a few encodes at high resolution in real time. Then, why not use your resources for the best quality you can for the current network conditions and let someone else do the transcoding for you. You will likely need transcoding anyway for reaching different platforms/receivers.

> "Layered encoding has little benefits for streaming video at high resolutions."
> 
> 1. Ability to bypass the transcoding step, since multiple resolutions/framerates can be obtained by selectively dropping frames.

If you only consider dropping frames (for which you actually don’t need SVC), you cannot get multiple resolutions. You can only get variable frame rate and as I said before, dropping frames will save you little in the bandwidth for a given encode. for multiple resolutions, you will still need multiple base layers.

> 2. Improved resilience.  The base layer can be protected (by retransmission and/or FEC) while upper layers need not be.  So the need to retransmit keyframes can be reduced.

You are ignoring the redundancy needed by SVC. Unequal error protection is a 30-year old story, it looks nice on the paper but not as much in practice.

> "I am not a codec guy but I know that if you have multiple renditions at say 5 different resolutions, you will likely need 5 base layers anyway."
> 
> [BA] That would be true for simulcast, but not for spatial scalability, where you could have a single base layer.  5 spatial layers is quite a lot though, so you'd probably need more than 1 base layer.

Amazon prime content sometimes has more than 20 renditions. Obviously there are renditions for different codecs but most such services support more than 5 spatial resolutions from 360p to 2160p. Spatial scalability is quite limited. One base layer in 360p will not let you scale up to 720p or higher. You cannot scale up from 720p to 1080p as I was told by some HEVC gurus. If I have wrong info, please correct me but spatial scalability (or scalability in general) sounds more hype than anything else (needless to mention the added encoding complexity).

-acbegen

> 
> On Tue, Nov 16, 2021 at 4:39 AM Ali C. Begen <ali.begen=[email-masked]> wrote:

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 16 Nov 2021 09:31:36 -0500

"If there are 100K participants on an event, I am sure 99.99% of them are
just listening - cannot be even chatting in a meaningful way. The obvious
question then is why would you need 600ms of latency in that case. Are they
taking an action based on what they hear/see right away? If you need a
latency of few hundreds of milliseconds, that is way more like RTC than
streaming."

[BA] In a large meeting or webinar that is mostly one-way, there can be
Question/Answer periods where the Questioner will be enabled for
audio/video transmission.  So you want to be able to enable the questioner
and then have them start talking without too long a delay.

There are also use cases where audio/video from conferencing participants
is combined with streaming content.  This would include things like
"co-watching" (a separated family watching streamed content together) or
"together mode" (virtual fans being superimposed onto the empty seats in a
venue which is hosting a sporting event).  In these use cases, you want the
reaction of the conference participants to not lag the streaming content
too much.

On Tue, Nov 16, 2021 at 8:41 AM Ali C. Begen <[email-masked]>
wrote:

> > "Sending simulcast streams or layered streams from the sender (so that
> we can avoid encoding/transcoding at the origin I mentioned above) might be
> a good option for real-time conferencing but I argue that it is a poor
> choice for streaming media."
> >
> > [BA] We are now seeing use cases such as webinars and very large
> meetings which require both large scale (100K+ participants) and low
> latencies (< 600 ms).  These use cases often combine aspects of RTC and
> streaming technologies, so they do not clearly lie in one category or the
> other.
>
> If there are 100K participants on an event, I am sure 99.99% of them are
> just listening - cannot be even chatting in a meaningful way. The obvious
> question then is why would you need 600ms of latency in that case. Are they
> taking an action based on what they hear/see right away? If you need a
> latency of few hundreds of milliseconds, that is way more like RTC than
> streaming.
>
> Even if you get the best device today, it won’t do more than a few encodes
> at high resolution in real time. Then, why not use your resources for the
> best quality you can for the current network conditions and let someone
> else do the transcoding for you. You will likely need transcoding anyway
> for reaching different platforms/receivers.
>
> If you only consider dropping frames (for which you actually don’t need
> SVC), you cannot get multiple resolutions. You can only get variable frame
> rate and as I said before, dropping frames will save you little in the
> bandwidth for a given encode. for multiple resolutions, you will still need
> multiple base layers.
>
> > 2. Improved resilience.  The base layer can be protected (by
> retransmission and/or FEC) while upper layers need not be.  So the need to
> retransmit keyframes can be reduced.
>
> You are ignoring the redundancy needed by SVC. Unequal error protection is
> a 30-year old story, it looks nice on the paper but not as much in practice.
>
> Amazon prime content sometimes has more than 20 renditions. Obviously
> there are renditions for different codecs but most such services support
> more than 5 spatial resolutions from 360p to 2160p. Spatial scalability is
> quite limited. One base layer in 360p will not let you scale up to 720p or
> higher. You cannot scale up from 720p to 1080p as I was told by some HEVC
> gurus. If I have wrong info, please correct me but spatial scalability (or
> scalability in general) sounds more hype than anything else (needless to
> mention the added encoding complexity).
>
> -acbegen
>
> >
> > > This is a great summary, Luke. I agree it would be nice to have a
> single media protocol that would work across multiple classes of
> application. Ultimately I think this would involve something like WebRTC's
> congestion control and media recovery options (retransmit, FEC, new
> keyframe) in a transport that was simpler than the current
> ICE/DTLS/RTP/SCTP protocol sandwich, and from which it was easy to derive
> multiple renditions with minimal latency cost (likely simulcast or layered
> coding).
> >
> > > It's been great seeing the discussion about congestion control. I'd
> like to start the conversation about something more video-specific but just
> as important.
> > >
> > Lets simplify this. Assume we have a sender, an origin and a receiver.
> The sender should “contribute” the best quality it can to the origin and
> origin can produce (transcodes) multiple renditions and the receiver picks
> up the best one it can (distribution). The same workflow works just fine if
> you have a million receivers instead of one. Everybody gets whatever they
> can with the help of a CDN.
> >
> > > e.g., simulcast or layered coding, or b), at the server, by
> > > a. Drop individual frames. Protocols like WebRTC work by deciding not
> > > If RTT is low enough, WebRTC will retransmit frames, and this works
> well at scale even in sub-100ms-latency applications (e.g. gaming).
> >
> > > b. Request a new keyframe. If too many reference frames/slices have
> been dropped, there will be persistent decoding errors until the next
> keyframe. Protocols like WebRTC will request the  generation of a new
> keyframe to exit this state sooner than waiting for the next keyframe.
> >
> > This new key frame will be substantially larger than the other frames.
> If the bandwidth is already lower than what you need, this might not help
> you as much.
> >
> > >
> > > c. Drop remainder of GoP. A protocol can drop all frames until the
> next keyframe without causing decode errors. Protocols like HLS and DASH
> can cancel a pending segment download, be it to switch to another rendition
> or to jump ahead.
> >
> > Canceling an outstanding segment request is too costly - it means
> killing the TCP connection in http/1.1. There may be scenarios where it is
> still beneficial to give up and switch to a lower bitrate rendition (like
> when the segment is long and you are relatively closer to the beginning). I
> need to remind you, though, that DASH doesn’t like variable segment sizes
> too much (complicates the MPD, requires MPD updates, events, etc.). My
> comments from above about the missing content still apply here - nobody
> likes missing frames when watching sports for example.
> >
> > > This is supported at scale in most conferencing applications,
> sometimes through simpler mechanisms like simulcast. Sometimes, the
> critical frames can also be reconstructed via Forward Error Correction.
> >
> > Ditto.
> >
> > Indeed but for live streaming, you can always use adaptive media
> playback so that you can catch up with the live edge once the network has
> caught up.
> >
> > >
> > > There's no approach that best supports every use-case and ultimately
> it comes down to the target user experience.
> > >

---

From: "Ali C. Begen" <[email-masked]>
Date: Tue, 16 Nov 2021 17:48:52 +0300

> On Nov 16, 2021, at 5:31 PM, Bernard Aboba <[email-masked]> wrote:
> 
> "If there are 100K participants on an event, I am sure 99.99% of them are just listening - cannot be even chatting in a meaningful way. The obvious question then is why would you need 600ms of latency in that case. Are they taking an action based on what they hear/see right away? If you need a latency of few hundreds of milliseconds, that is way more like RTC than streaming."
> 
> [BA] In a large meeting or webinar that is mostly one-way, there can be Question/Answer periods where the Questioner will be enabled for audio/video transmission.  So you want to be able to enable the questioner and then have them start talking without too long a delay. 

I get the envisioned use cases but are they really practical or are they really what we are after? Look at the ietf sessions (e.g., avtcore) where you would consider 99% of the participants quite knowledgeable about how videoconferencing works at a very low level and we are often 30-40 people not even 100. And when we queue up for Q&A, switching time from one speaker to another is often several seconds even if you do everything right. So, saying I want 100x ms latency in a 100K attendee meeting sounds questionable. The only practical way would be typing the questions in the chat window or submitting a recorded audio/video question, not asking the question right at that moment. 

> There are also use cases where audio/video from conferencing participants is combined with streaming content.  This would include things like "co-watching" (a separated family watching streamed content together) or "together mode" (virtual fans being superimposed onto the empty seats in a venue which is hosting a sporting event).  In these use cases, you want the reaction of the conference participants to not lag the streaming content too much.

Co-watching requires synchronization at multiple destinations, does not require low latency. But yes, “together mode” requires low latency both on the streamed content and the RTC part. Yet, this still does not mean layered encoding is the way to go here.

---

From: Justin Uberti <[email-masked]>
Date: Tue, 16 Nov 2021 09:26:14 -0800

On Tue, Nov 16, 2021 at 6:49 AM Ali C. Begen <ali.begen=
[email-masked]> wrote:

> > "If there are 100K participants on an event, I am sure 99.99% of them
> are just listening - cannot be even chatting in a meaningful way. The
> obvious question then is why would you need 600ms of latency in that case.
> Are they taking an action based on what they hear/see right away? If you
> need a latency of few hundreds of milliseconds, that is way more like RTC
> than streaming."
> >
> I get the envisioned use cases but are they really practical or are they
> really what we are after? Look at the ietf sessions (e.g., avtcore) where
> you would consider 99% of the participants quite knowledgeable about how
> videoconferencing works at a very low level and we are often 30-40 people
> not even 100. And when we queue up for Q&A, switching time from one speaker
> to another is often several seconds even if you do everything right. So,
> saying I want 100x ms latency in a 100K attendee meeting sounds
> questionable. The only practical way would be typing the questions in the
> chat window or submitting a recorded audio/video question, not asking the
> question right at that moment.
>
> There are many products that allow the audience to instantly react to the
live feed (applause, laughter, thumbs-up, etc). It's desirable for these
reactions to be tightly synced (at least sub-second latency) with the
content, and several products support such functionality at scales of XXK
users.

Also, when you want to actually speak, if you are seconds behind there is
no great way to quickly catch up to realtime. Thus maintaining low latency
has benefits in this situation as well.

> Co-watching requires synchronization at multiple destinations, does not
> require low latency. But yes, “together mode” requires low latency both on
> the streamed content and the RTC part. Yet, this still does not mean
> layered encoding is the way to go here.
> --
> Moq mailing list
> [email-masked]

---

From: "Ali C. Begen" <[email-masked]>
Date: Tue, 16 Nov 2021 21:09:31 +0300

Justin, just to be clear: If there are such services at the SLAs you mention, that is fine. I just believe that many participants cannot show such reactions at sub-second latency and the said latency requirements are unnecessarily harsh. 

Having said that, my main point was regarding the use of layered codecs and simulcast not the specific use case or scenario.

-acbegen

> On Nov 16, 2021, at 8:26 PM, Justin Uberti <[email-masked]> wrote:
> 
> 
> 
> On Tue, Nov 16, 2021 at 6:49 AM Ali C. Begen <ali.begen=[email-masked]> wrote:
> 
> 
> There are many products that allow the audience to instantly react to the live feed (applause, laughter, thumbs-up, etc). It's desirable for these reactions to be tightly synced (at least sub-second latency) with the content, and several products support such functionality at scales of XXK users.
> 
> Also, when you want to actually speak, if you are seconds behind there is no great way to quickly catch up to realtime. Thus maintaining low latency has benefits in this situation as well.
>

---

From: Luke Curley <[email-masked]>
Date: Tue, 16 Nov 2021 12:53:57 -0800

There's a constant trade-off between quality and latency. All of the
latency strategies that I've mentioned have varying amounts of user impact
roughly relative to their effectiveness at reducing latency.

In the case of Twitch, we want to enable interactive use-cases. We want to
allow events like the NFL to broadcast at a high quality (~5s latency). We
want broadcasters to be able to respond to their viewers (communicating via
chat) without unnecessary delay (~2s latency). We want to let viewers
influence the contents of the broadcast, like Twitch Plays Pokemon (~500ms
latency). We want viewers to be able to join a broadcast in progress,
like @bernard mentioned (~150ms latency). And most importantly and yet
often overlooked, we want to have a higher latency (to offset the poor
quality) for viewers in remote parts of the world with poor internet
infrastructure.

In all of these cases, the quality of the stream will suffer as latency is
reduced; that's unavoidable. Ideally we want both the broadcaster and
viewer to configure their desired level of interactivity versus quality.
However, we cannot and should not run a separate video stack for every
use-case and latency target.

On Tue, Nov 16, 2021 at 10:09 AM Ali C. Begen <ali.begen=
[email-masked]> wrote:

> Justin, just to be clear: If there are such services at the SLAs you
> mention, that is fine. I just believe that many participants cannot show
> such reactions at sub-second latency and the said latency requirements are
> unnecessarily harsh.
>
> Having said that, my main point was regarding the use of layered codecs
> and simulcast not the specific use case or scenario.
>
> -acbegen
>
> > > [BA] In a large meeting or webinar that is mostly one-way, there can
> be Question/Answer periods where the Questioner will be enabled for
> > There are many products that allow the audience to instantly react to
> the live feed (applause, laughter, thumbs-up, etc). It's desirable for
> these reactions to be tightly synced (at least sub-second latency) with the
> > Also, when you want to actually speak, if you are seconds behind there
> is no great way to quickly catch up to realtime. Thus maintaining low
> latency has benefits in this situation as well.
> >
> > > There are also use cases where audio/video from conferencing
> participants is combined with streaming content.  This would include things
> like "co-watching" (a separated family watching streamed content together)
> or "together mode" (virtual fans being superimposed onto the empty seats in
> a venue which is hosting a sporting event).  In these use cases, you want
> the reaction of the conference participants to not lag the streaming
> content too much.
> >

---

From: Luke Curley <[email-masked]>
Date: Tue, 16 Nov 2021 13:44:05 -0800

I wanted to break down video protocols into the underlying latency
strategies to ask two questions:

1.) Why do UDP-based protocols have lower latency than TCP-based protocols?
2.) How does QUIC change this?

My realization is that QUIC streams can be used to employ virtually all of
the WebRTC latency reduction strategies. For example, if you want to drop
frames, create a QUIC stream per frame which can be closed by either
endpoint (ex. RUSH).

My argument is that QUIC should be more than a proxy for RTP packets. We
can employ the same latency reduction strategies but using QUIC streams
instead of RTP headers. That's what I've done (very similar to Facebook's
RUSH) and I'll have a draft to share in the next few days.

There are missing features like Justin mentioned, namely FEC and per-packet
timestamps. It's not clear if these are critical, but at the very least
WebRTC would have a slight latency advantage until these are added to QUIC.

On Tue, Nov 16, 2021 at 12:53 PM Luke Curley <[email-masked]> wrote:

>> > I get the envisioned use cases but are they really practical or are
>> they really what we are after? Look at the ietf sessions (e.g., avtcore)
>> where you would consider 99% of the participants quite knowledgeable about
>> how videoconferencing works at a very low level and we are often 30-40
>> people not even 100. And when we queue up for Q&A, switching time from one
>> speaker to another is often several seconds even if you do everything
>> right. So, saying I want 100x ms latency in a 100K attendee meeting sounds

---

From: Justin Uberti <[email-masked]>
Date: Tue, 16 Nov 2021 14:16:08 -0800

On Tue, Nov 16, 2021 at 1:44 PM Luke Curley <[email-masked]> wrote:

> There are missing features like Justin mentioned, namely FEC and
> per-packet timestamps. It's not clear if these are critical, but at the
> very least WebRTC would have a slight latency advantage until these are
> added to QUIC.
>

But isn't this rebuilding WebRTC on top of QUIC? And if so, what benefit
does this actually have versus using WebRTC directly? If there is a
benefit, could it be realized simply by tunnelling WebRTC in QUIC?

>> In the case of Twitch, we want to enable interactive use-cases. We want
>> to allow events like the NFL to broadcast at a high quality (~5s latency).
>> We want broadcasters to be able to respond to their viewers (communicating
>> via chat) without unnecessary delay (~2s latency). We want to let viewers
>> In all of these cases, the quality of the stream will suffer as latency
>> is reduced; that's unavoidable. Ideally we want both the broadcaster and
>>> > > "If there are 100K participants on an event, I am sure 99.99% of
>>> them are just listening - cannot be even chatting in a meaningful way. The
>>> > Co-watching requires synchronization at multiple destinations, does
>>> not require low latency. But yes, “together mode” requires low latency both
>>> on the streamed content and the RTC part. Yet, this still does not mean

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 16 Nov 2021 17:22:12 -0500

> 2.) How does QUIC change this?

There are multiple effects here. 

One is the “head of line blocking” issue in TCP, wherein a non-discardable frame may be blocked pending retransmission of a discardable frame. Sending frames on separate QUIC streams addresses this.

QUIC is more effective than TCP at handling variable RTT and/or high loss so the frame/stream model can be quite useable for frames you need to get through (e.g. non-discardable). 

For discardable frames there is the latency vs. reliability tradeoff. With UDP, you can stop retransmitting if your latency budget is exhausted. QUIC datagrams doesn’t change this. Partially reliable QUIC streams can be used here but you will not have as much control as with datagrams. For example, you can end up consuming bandwidth with retransmissions that you would prefer to allocate elsewhere.

---

From: Luke Curley <[email-masked]>
Date: Tue, 16 Nov 2021 15:27:14 -0800

That's a good discussion in itself, Justin. What's wrong with WebRTC? What
does using QUIC actually accomplish?

I know Twitch and other companies are approaching QUIC as an improvement
for the higher latency protocols like RTMP and HLS/DASH. But it begs the
question, why aren't we using WebRTC in the first place? I think it comes
down to a lack of flexibility; WebRTC is a black-box wired for real-time
latency. Replacing ICE/SRTP/SCTP with QUIC would be an improvement, but it
doesn't push the needle.

But I understand what you mean; there's no point redesigning WebRTC just
because it's possible. I think we should focus our efforts on the higher
latency/quality use-cases that currently use RTMP or SRT. I just wanted to
point out that lower latency strategies are possible with QUIC streams and
not just datagrams.

On Tue, Nov 16, 2021 at 2:16 PM Justin Uberti <
[email-masked]> wrote:

>> My realization is that QUIC streams can be used to employ virtually all
>> of the WebRTC latency reduction strategies. For example, if you want to
>> drop frames, create a QUIC stream per frame which can be closed by either
>> endpoint (ex. RUSH).
>>
>>>> > > [BA] In a large meeting or webinar that is mostly one-way, there
>>>> can be Question/Answer periods where the Questioner will be enabled for
>>>> > Also, when you want to actually speak, if you are seconds behind
>>>> there is no great way to quickly catch up to realtime. Thus maintaining low

---

From: Kirill Pugin <[email-masked]>
Date: Tue, 16 Nov 2021 23:30:57 +0000

For one, it’s not easy to run WebRTC, there is signaling part, there is media part. It’s also not straight-forward to be able to get higher quality form WebRTC – given it’s mostly used for Video Call type of application, it has been “tuned” for latency.
It’s also not easy to “cache” WebRTC

Best,
Kirill

From: Justin Uberti <[email-masked]>
Date: Tuesday, November 16, 2021 at 2:16 PM
To: Luke Curley <[email-masked]>
Cc: "Ali C. Begen" <ali.begen=[email-masked]>, MOQ Mailing List <[email-masked]>, Bernard Aboba <[email-masked]>
Subject: Re: [Moq] Latency strategy

On Tue, Nov 16, 2021 at 1:44 PM Luke Curley <[email-masked]<mailto:[email-masked]>> wrote:
I wanted to break down video protocols into the underlying latency strategies to ask two questions:

My realization is that QUIC streams can be used to employ virtually all of the WebRTC latency reduction strategies. For example, if you want to drop frames, create a QUIC stream per frame which can be closed by either endpoint (ex. RUSH).

My argument is that QUIC should be more than a proxy for RTP packets. We can employ the same latency reduction strategies but using QUIC streams instead of RTP headers. That's what I've done (very similar to Facebook's RUSH) and I'll have a draft to share in the next few days.

There are missing features like Justin mentioned, namely FEC and per-packet timestamps. It's not clear if these are critical, but at the very least WebRTC would have a slight latency advantage until these are added to QUIC.

But isn't this rebuilding WebRTC on top of QUIC? And if so, what benefit does this actually have versus using WebRTC directly? If there is a benefit, could it be realized simply by tunnelling WebRTC in QUIC?

On Tue, Nov 16, 2021 at 12:53 PM Luke Curley <[email-masked]<mailto:[email-masked]>> wrote:
There's a constant trade-off between quality and latency. All of the latency strategies that I've mentioned have varying amounts of user impact roughly relative to their effectiveness at reducing latency.

In the case of Twitch, we want to enable interactive use-cases. We want to allow events like the NFL to broadcast at a high quality (~5s latency). We want broadcasters to be able to respond to their viewers (communicating via chat) without unnecessary delay (~2s latency). We want to let viewers influence the contents of the broadcast, like Twitch Plays Pokemon (~500ms latency). We want viewers to be able to join a broadcast in progress, like @bernard mentioned (~150ms latency). And most importantly and yet often overlooked, we want to have a higher latency (to offset the poor quality) for viewers in remote parts of the world with poor internet infrastructure.

In all of these cases, the quality of the stream will suffer as latency is reduced; that's unavoidable. Ideally we want both the broadcaster and viewer to configure their desired level of interactivity versus quality. However, we cannot and should not run a separate video stack for every use-case and latency target.

On Tue, Nov 16, 2021 at 10:09 AM Ali C. Begen <ali.begen=[email-masked]<mailto:[email-masked]>> wrote:
Justin, just to be clear: If there are such services at the SLAs you mention, that is fine. I just believe that many participants cannot show such reactions at sub-second latency and the said latency requirements are unnecessarily harsh.

-acbegen

> On Nov 16, 2021, at 8:26 PM, Justin Uberti <[email-masked]<mailto:[email-masked]>> wrote:
>
>
>
> On Tue, Nov 16, 2021 at 6:49 AM Ali C. Begen <ali.begen=[email-masked]<mailto:[email-masked]>> wrote:
>
>
> > On Nov 16, 2021, at 5:31 PM, Bernard Aboba <[email-masked]<mailto:[email-masked]>> wrote:
> >
> [email-masked]<mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/moq<https://www.ietf.org/mailman/listinfo/moq>

--
Moq mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/moq<https://www.ietf.org/mailman/listinfo/moq>

---

From: Justin Uberti <[email-masked]>
Date: Tue, 16 Nov 2021 16:48:48 -0800

This is good input and I think starts to identify the problems we should be
trying to address:
a) can we make WebRTC run more easily (combined media/signaling, thinner
protocol sandwich, etc) or, barring that, come up with an easier-to-use
replacement?
b) can we come up with a "cacheable" protocol that has lower latency than
HLS?

I disagree with the notion that WebRTC is a black box wired for real-time
latency (as noted in my prior comments about current broadcast customers),
unless the definition of real-time is "uncacheable".

On Tue, Nov 16, 2021 at 3:31 PM Kirill Pugin <[email-masked]> wrote:

> For one, it’s not easy to run WebRTC, there is signaling part, there is
> media part. It’s also not straight-forward to be able to get higher quality
> form WebRTC – given it’s mostly used for Video Call type of application, it
> has been “tuned” for latency.
>
> *From: *Justin Uberti <[email-masked]>
> *Date: *Tuesday, November 16, 2021 at 2:16 PM
> *To: *Luke Curley <[email-masked]>
> *Cc: *"Ali C. Begen" <ali.begen=[email-masked]>, MOQ
> Mailing List <[email-masked]>, Bernard Aboba <[email-masked]>
> *Subject: *Re: [Moq] Latency strategy
>
>
>
>
>
>
>

---

From: Alan Frindell <[email-masked]>
Date: Wed, 17 Nov 2021 01:02:23 +0000

> Partially reliable QUIC streams can be used here but you will not have as much control as with datagrams. For example, you can end up consuming bandwidth with retransmissions that you would prefer to allocate elsewhere. 

I think you can have as much control over stream retransmission policy as your QUIC API is willing to give you.

-Alan

---

From: Victor Vasiliev <[email-masked]>
Date: Tue, 16 Nov 2021 23:13:24 -0500

> 2.) How does QUIC change this?

I'll note that at least some of this is due to the fact that when
implementing UDP protocols, people tune the transport logic for their
specific application use case, and not caused by the fundamental properties
of the transport protocol used.

> frames, create a QUIC stream per frame which can be closed by either

Note that in addition to using stream resets for partial reliability, you
can read partially received data on any stream since all of it is in
userspace (this is something you could theoretically do in TCP if the API
allowed).  This is useful if you haven't received data by the deadline, but
still want to output something at the cost of perceivable artifacts.

> RUSH) and I'll have a draft to share in the next few days.

I wholeheartedly agree.  From my observation, QUIC as-is is an
excellent media transport for basically any use case that is not real-time.

> very least WebRTC would have a slight latency advantage until these are

FEC is in general only useful when the latency target is at most 2x-3x RTT;
the reason for that is that in all other cases, you're always better off
retransmitting the exact packets that were lost.  Per-packet timestamps are
useful for some CCs, but less so for others.

I think we stand to gain a lot of simplicity by focusing on a well-defined
set of core use cases, because there are some specific cases (e.g.
low-bandwidth real time scenarios like "voice call over 2G") that would
make things unnecessarily complicated.