Subject: [Moq] Latency @ Twitch

From: Luke Curley <[email-masked]>
Date: Mon, 1 Nov 2021 17:39:08 -0700

Hey folks, I wanted to quickly summarize the problems we've run into at
Twitch that have led us to QUIC.

Twitch is a live one-to-many product. We primarily focus on video quality
due to the graphical fidelity of video games. Viewers can participate in a
chat room, which the broadcaster reads and can respond to via video. This
means that latency is also somewhat important to facilitate this social
interaction.

A looong time ago we were using RTMP for both ingest and distribution
(Flash player). We switched to HLS for distribution to gain the benefit of
3rd party CDNs, at the cost of dramatically increasing latency. A later
project lowered the latency of HLS using chunked-transfer delivery, very
similar to LL-DASH (and not LL-HLS). We're still using RTMP for
contribution.

To summarize the issues with our current distribution system:

*1. HLS suffers from head-of-line blocking.*
During congestion, the current segment stalls and is delivered slower than
the encoded bitrate. The player has no recourse than to wait for the
segment to finish downloading, risking depleting the buffer. It can switch
down to a lower rendition at segment boundaries, but these boundaries occur
too infrequently (every 2s) to handle sudden congestion. Trying to switch
earlier, either by canceling the current segment or downloading the lower
rendition in parallel, only exacerbates the issue.

*2. HLS has poor "auto" quality (ABR).*
The player is responsible for choosing the rendition to download. This is a
problem when media is delivered frame-by-frame (ie. HTTP chunked-transfer),
as we're effectively application-limited by the encoder bitrate. The player
can only measure the arrival timestamp of data and does not know when the
network can sustain a higher bitrate without just trying it. We hosted an ACM
challenge
<https://blog.twitch.tv/en/2020/01/15/twitch-invites-you-to-take-on-our-acm-mmsys-2020-grand-challenge/>
for this issue in particular. I believe this is why LL-HLS opts to burst
small chunks of data (sub-segments) at the cost of higher latency.

Both of these necessitate a larger player buffer, which increases latency.
The contribution system it's own problems, but let me sync up with that
team first before I try to enumerate them.

---

From: "Ali C. Begen" <[email-masked]>
Date: Tue, 2 Nov 2021 23:39:47 +0300

> On Nov 2, 2021, at 3:39 AM, Luke Curley <[email-masked]> wrote:
> 
> Hey folks, I wanted to quickly summarize the problems we've run into at Twitch that have led us to QUIC.
> 
> 
> Twitch is a live one-to-many product. We primarily focus on video quality due to the graphical fidelity of video games. Viewers can participate in a chat room, which the broadcaster reads and can respond to via video. This means that latency is also somewhat important to facilitate this social interaction.
> 
> A looong time ago we were using RTMP for both ingest and distribution (Flash player). We switched to HLS for distribution to gain the benefit of 3rd party CDNs, at the cost of dramatically increasing latency. A later project lowered the latency of HLS using chunked-transfer delivery, very similar to LL-DASH (and not LL-HLS). We're still using RTMP for contribution.
> 
> 
> 1. HLS suffers from head-of-line blocking.
> During congestion, the current segment stalls and is delivered slower than the encoded bitrate. The player has no recourse than to wait for the segment to finish downloading, risking depleting the buffer. It can switch down to a lower rendition at segment boundaries, but these boundaries occur too infrequently (every 2s) to handle sudden congestion. Trying to switch earlier, either by canceling the current segment or downloading the lower rendition in parallel, only exacerbates the issue.

DASH has the concept of Resync points that were designed exactly for this purpose (allowing you to emergency downshift in the middle of a segment).

> 2. HLS has poor "auto" quality (ABR).
> The player is responsible for choosing the rendition to download. This is a problem when media is delivered frame-by-frame (ie. HTTP chunked-transfer), as we're effectively application-limited by the encoder bitrate. The player can only measure the arrival timestamp of data and does not know when the network can sustain a higher bitrate without just trying it. We hosted an ACM challenge for this issue in particular.

That exact challenge had three competing solutions, two of which are now part of the official dash.js code. And yes, the player can figure what the network can sustain *without* trying higher bitrate renditions.
https://github.com/Dash-Industry-Forum/dash.js/wiki/Low-Latency-streaming
Or read the paper that even had “twitch” in its title here: https://ieeexplore.ieee.org/document/9429986

> I believe this is why LL-HLS opts to burst small chunks of data (sub-segments) at the cost of higher latency.
> 
> 
> Both of these necessitate a larger player buffer, which increases latency. The contribution system it's own problems, but let me sync up with that team first before I try to enumerate them.
> -- 
> Moq mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/moq

---

From: "Piers O'Hanlon" <[email-masked]>
Date: Wed, 3 Nov 2021 15:50:06 +0000

On 2 Nov 2021, at 20:39, Ali C. Begen <ali.begen=[email-masked]<mailto:ali.begen=[email-masked]>> wrote:

On Nov 2, 2021, at 3:39 AM, Luke Curley <[email-masked]<mailto:[email-masked]>> wrote:

I guess Apple do also have their BYTERANGE/CTE mode for LL-HLS which is pretty similar to LL-DASH.

Isn't the HoL limitation more down to the use of HTTP/1.1?

I was curious if there are any studies or experience of how resync points perform in practice?

The limitation here may also be down to the lack of access to sufficiently accurate timing information about data arrivals in the browser - unfortunately the Streams API, which provides data from the fetch API, doesn’t directly timestamp the data arrivals so the JS app has to timestamp it which can suffer from noise such as scheduling etc - especially a problem for small/fast data arrivals. I guess another issue could be that if the system is only sending single frames then the network transport may be operating in application limited mode so the cwnd doesn’t grow sufficiently to take advantage of the available capacity.

There was a recent study that seems to show that none of the current algorithms are that great for low latency, and the two new dash.js ones appear to lead to much higher levels of rebuffering:
https://dl.acm.org/doi/pdf/10.1145/3458305.3478442

Piers

Both of these necessitate a larger player buffer, which increases latency. The contribution system it's own problems, but let me sync up with that team first before I try to enumerate them.
--
Moq mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/moq

--
Moq mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/moq

---

From: "Ali C. Begen" <[email-masked]>
Date: Wed, 3 Nov 2021 20:05:06 +0300

> On Nov 3, 2021, at 6:50 PM, Piers O'Hanlon <[email-masked]> wrote:
> 
> 
> 
>> On 2 Nov 2021, at 20:39, Ali C. Begen <ali.begen=[email-masked]> wrote:
>> 
>> 
>> 

Yes, Apple can list the parts (chunks in LL-DASH) as byteranges in the playlist but the frequent playlist refresh and part retrieval process is inevitable in LL-HLS, which is one of the main differences from LL-DASH (no need for manifest refresh and request per segment not chunk).

> I was curious if there are any studies or experience of how resync points perform in practice?

Resync points are pretty fresh out of the oven. dash.js has it in the roadmap but not yet implemented (and we also need to generate test streams). So, there is no data available yet with the real clients. But, I suppose you can imagine how in-segment switching can help in sudden bw drops especially for long segments.

> The limitation here may also be down to the lack of access to sufficiently accurate timing information about data arrivals in the browser - unfortunately the Streams API, which provides data from the fetch API, doesn’t directly timestamp the data arrivals so the JS app has to timestamp it which can suffer from noise such as scheduling etc - especially a problem for small/fast data arrivals.

Yes, you need to get rid of that noise (see LoL+).

> I guess another issue could be that if the system is only sending single frames then the network transport may be operating in application limited mode so the cwnd doesn’t grow sufficiently to take advantage of the available capacity.

Unless the video bitrate is too low, this should not be an issue most of the time. 

> https://dl.acm.org/doi/pdf/10.1145/3458305.3478442

Brightcove’s paper uses the LoL and L2A algorithms from the challenge where low latency was the primary goal. For Twitch’s own evaluation, I suggest you watch:
https://www.youtube.com/watch?v=rcXFVDotpy4
We later addressed the rebuffering issue, developed LoL+, which is the version included in dash.js now and explained at the ieeexplore link I gave above.

Copying the authors in case they want to add anything for the paper you cited.

-acbegen

---

From: Luke Curley <[email-masked]>
Date: Wed, 3 Nov 2021 13:40:07 -0700

I think resync points are an interesting idea although we haven't evaluated
them. Twitch did push for S-frames in AV1 which will be another option in
the future instead of encoding a full IDR frame at these resync boundaries.

An issue is you have to make the hard decision to abort the current
download and frantically try to pick up the pieces before the buffer
depletes. It's a one-way door (maybe your algorithm overreacted) and you're
going to be throwing out some media just to redownload it at a lower
bitrate.

Ideally, you could download segments in parallel without causing
contention. The idea is to spend any available bandwidth on the new segment
to fix the problem, and any excess bandwidth on the old segment in
the event it arrives before the player buffer actually depletes. That's
more or less the core concept for what we've built using QUIC, and it's
compatible with resync points if we later go down that route.

And you're exactly right Piers. The fundamental issue is that a web player
lacks the low level timing information required to infer the delivery rate.
You would want something like BBR's rate estimation
<https://datatracker.ietf.org/doc/html/draft-cheng-iccrg-delivery-rate-estimation>
which
inspects the time delta between packets to determine the send rate. That
gets really difficult when the OS and browser delay flushing data to the
application, be it for performance reasons or due to packet loss (to
maintain head-of-line blocking).

I did run into CUBIC/Reno not being able to grow the congestion window when
frames are sent one at a time (application limited). I don't believe BBR
suffers from the same problem though due to the aforementioned rate
estimator.

On Wed, Nov 3, 2021 at 10:05 AM Ali C. Begen <[email-masked]>
wrote:

> >>> Twitch is a live one-to-many product. We primarily focus on video
> quality due to the graphical fidelity of video games. Viewers can
> participate in a chat room, which the broadcaster reads and can respond to
> via video. This means that latency is also somewhat important to facilitate
> this social interaction.
> >>>
> Yes, Apple can list the parts (chunks in LL-DASH) as byteranges in the
> playlist but the frequent playlist refresh and part retrieval process is
> inevitable in LL-HLS, which is one of the main differences from LL-DASH (no
> need for manifest refresh and request per segment not chunk).
>
> >
> >>>
> >>> During congestion, the current segment stalls and is delivered slower
> than the encoded bitrate. The player has no recourse than to wait for the
> >> DASH has the concept of Resync points that were designed exactly for
> this purpose (allowing you to emergency downshift in the middle of a
> segment).
> >>
> Resync points are pretty fresh out of the oven. dash.js has it in the
> roadmap but not yet implemented (and we also need to generate test
> streams). So, there is no data available yet with the real clients. But, I
> suppose you can imagine how in-segment switching can help in sudden bw
> drops especially for long segments.
>
> >
> >>> The player is responsible for choosing the rendition to download. This
> is a problem when media is delivered frame-by-frame (ie. HTTP
> chunked-transfer), as we're effectively application-limited by the encoder
> bitrate. The player can only measure the arrival timestamp of data and does
> not know when the network can sustain a higher bitrate without just trying
> it. We hosted an ACM challenge for this issue in particular.
> >>
> > The limitation here may also be down to the lack of access to
> sufficiently accurate timing information about data arrivals in the browser
> - unfortunately the Streams API, which provides data from the fetch API,
> doesn’t directly timestamp the data arrivals so the JS app has to timestamp
> it which can suffer from noise such as scheduling etc - especially a
> problem for small/fast data arrivals.
>
> > I guess another issue could be that if the system is only sending single
> frames then the network transport may be operating in application limited
> mode so the cwnd doesn’t grow sufficiently to take advantage of the
> available capacity.
>
> >> That exact challenge had three competing solutions, two of which are
> now part of the official dash.js code. And yes, the player can figure what
> the network can sustain *without* trying higher bitrate renditions.
> >>
> >> Or read the paper that even had “twitch” in its title here:
> https://ieeexplore.ieee.org/document/9429986
> >>
> > There was a recent study that seems to show that none of the current
> algorithms are that great for low latency, and the two new dash.js ones
> appear to lead to much higher levels of rebuffering:
> Brightcove’s paper uses the LoL and L2A algorithms from the challenge
> where low latency was the primary goal. For Twitch’s own evaluation, I
> suggest you watch:
> We later addressed the rebuffering issue, developed LoL+, which is the
> version included in dash.js now and explained at the ieeexplore link I gave
> above.
>
> >>> I believe this is why LL-HLS opts to burst small chunks of data
> (sub-segments) at the cost of higher latency.
> >>>
> >>>
> >>> Both of these necessitate a larger player buffer, which increases
> latency. The contribution system it's own problems, but let me sync up with
> that team first before I try to enumerate them.
> >>> --
> >>> Moq mailing list
> >>> [email-masked]

---

From: Ian Swett <[email-masked]>
Date: Wed, 3 Nov 2021 17:12:42 -0400

>From personal experience, BBR has some issues with application limited
behavior, but it is still able to grow the congestion window, at least
slightly, so it's likely an improvement over Cubic or Reno.

On Wed, Nov 3, 2021 at 4:40 PM Luke Curley <[email-masked]> wrote:

> I think resync points are an interesting idea although we haven't
> evaluated them. Twitch did push for S-frames in AV1 which will be another
> option in the future instead of encoding a full IDR frame at these resync
> boundaries.
>
> <https://datatracker.ietf.org/doc/html/draft-cheng-iccrg-delivery-rate-estimation> which
> I did run into CUBIC/Reno not being able to grow the congestion window
> when frames are sent one at a time (application limited). I don't believe
> BBR suffers from the same problem though due to the aforementioned rate
> estimator.
>
>> >>> The player is responsible for choosing the rendition to download.
>> This is a problem when media is delivered frame-by-frame (ie. HTTP
>> > I guess another issue could be that if the system is only sending
>> single frames then the network transport may be operating in application
>> limited mode so the cwnd doesn’t grow sufficiently to take advantage of the
>> available capacity.
>>

---

From: Luke Curley <[email-masked]>
Date: Wed, 3 Nov 2021 14:46:40 -0700

Yeah, there's definitely some funky behavior in BBR when application
limited but it's nowhere near as bad as Cubic/Reno. With those
algorithms you need to burst enough packets to fully utilize the congestion
window before it can be grown. With BBR I believe you need to burst just
enough to fully utilize the pacer, and even then this condition
<https://source.chromium.org/chromium/chromium/src/+/master:net/third_party/quiche/src/quic/core/congestion_control/bbr_sender.cc;l=393>
lets
you use application-limited samples if they would increase the send rate.

I started with BBR first because it's simpler, but I'm going to try out
BBR2 at some point because of the aforementioned PROBE_RTT issue. I don't
follow the congestion control space closely enough; are there any notable
algorithms that would better fit the live video use-case?

On Wed, Nov 3, 2021 at 2:12 PM Ian Swett <[email-masked]> wrote:

>> And you're exactly right Piers. The fundamental issue is that a web
>> player lacks the low level timing information required to infer
>> the delivery rate. You would want something like BBR's rate estimation
>>> >>> A looong time ago we were using RTMP for both ingest and
>>> distribution (Flash player). We switched to HLS for distribution to gain
>>> the benefit of 3rd party CDNs, at the cost of dramatically increasing
>>> latency. A later project lowered the latency of HLS using chunked-transfer
>>> delivery, very similar to LL-DASH (and not LL-HLS). We're still using RTMP
>>> for contribution.
>>> >>>
>>> >>> During congestion, the current segment stalls and is delivered
>>> slower than the encoded bitrate. The player has no recourse than to wait
>>> for the segment to finish downloading, risking depleting the buffer. It can
>>> switch down to a lower rendition at segment boundaries, but these
>>> boundaries occur too infrequently (every 2s) to handle sudden congestion.
>>> Trying to switch earlier, either by canceling the current segment or
>>> downloading the lower rendition in parallel, only exacerbates the issue.
>>> >>

---

From: "Piers O'Hanlon" <[email-masked]>
Date: Thu, 4 Nov 2021 10:58:31 +0000

On 3 Nov 2021, at 21:46, Luke Curley <[email-masked]<mailto:[email-masked]>> wrote:

Yeah, there's definitely some funky behavior in BBR when application limited but it's nowhere near as bad as Cubic/Reno. With those algorithms you need to burst enough packets to fully utilize the congestion window before it can be grown. With BBR I believe you need to burst just enough to fully utilize the pacer, and even then this condition<https://source.chromium.org/chromium/chromium/src/+/master:net/third_party/quiche/src/quic/core/congestion_control/bbr_sender.cc;l=393> lets you use application-limited samples if they would increase the send rate.

And there’s also the idle cwnd collapse/reset behaviour to consider if you’re sending a number of frames together and their inter-data gap exceeds the RTO - I’m not quite sure how the various QUIC stacks have translated RFC2861/7661 advice on this…?

I started with BBR first because it's simpler, but I'm going to try out BBR2 at some point because of the aforementioned PROBE_RTT issue. I don't follow the congestion control space closely enough; are there any notable algorithms that would better fit the live video use-case?

I guess Google’s Goog_CC appears to be well used in the WebRTC space (e.g. WEBRTC<https://webrtc.googlesource.com/src/+/refs/heads/main/modules/congestion_controller/goog_cc> and aiortc<https://github.com/aiortc/aiortc/blob/1a192386b721861f27b0476dae23686f8f9bb2bc/src/aiortc/rate.py#L271>) despite the draft<https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc> never making it to RFC status… There's also SCREAM<https://datatracker.ietf.org/doc/rfc8298/> which has an open source implementation<https://github.com/EricssonResearch/scream> but not sure how widely deployed it is.

On Wed, Nov 3, 2021 at 2:12 PM Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:
From personal experience, BBR has some issues with application limited behavior, but it is still able to grow the congestion window, at least slightly, so it's likely an improvement over Cubic or Reno.

On Wed, Nov 3, 2021 at 4:40 PM Luke Curley <[email-masked]<mailto:[email-masked]>> wrote:
I think resync points are an interesting idea although we haven't evaluated them. Twitch did push for S-frames in AV1 which will be another option in the future instead of encoding a full IDR frame at these resync boundaries.

An issue is you have to make the hard decision to abort the current download and frantically try to pick up the pieces before the buffer depletes. It's a one-way door (maybe your algorithm overreacted) and you're going to be throwing out some media just to redownload it at a lower bitrate.

Ideally, you could download segments in parallel without causing contention. The idea is to spend any available bandwidth on the new segment to fix the problem, and any excess bandwidth on the old segment in the event it arrives before the player buffer actually depletes. That's more or less the core concept for what we've built using QUIC, and it's compatible with resync points if we later go down that route.

And you're exactly right Piers. The fundamental issue is that a web player lacks the low level timing information required to infer the delivery rate. You would want something like BBR's rate estimation<https://datatracker.ietf.org/doc/html/draft-cheng-iccrg-delivery-rate-estimation> which inspects the time delta between packets to determine the send rate. That gets really difficult when the OS and browser delay flushing data to the application, be it for performance reasons or due to packet loss (to maintain head-of-line blocking).

I did run into CUBIC/Reno not being able to grow the congestion window when frames are sent one at a time (application limited). I don't believe BBR suffers from the same problem though due to the aforementioned rate estimator.

On Wed, Nov 3, 2021 at 10:05 AM Ali C. Begen <[email-masked]<mailto:[email-masked]>> wrote:

> On Nov 3, 2021, at 6:50 PM, Piers O'Hanlon <[email-masked]<mailto:[email-masked]>> wrote:
>
>
>
>> On 2 Nov 2021, at 20:39, Ali C. Begen <ali.begen=[email-masked]<mailto:[email-masked]>> wrote:
>>
>>
>>

-acbegen

---

From: Justin Uberti <[email-masked]>
Date: Tue, 9 Nov 2021 09:46:28 -0800

Ultimately we found that it wasn't necessary to standardize the CC as long
as the behavior needed from the remote side (e.g., feedback messaging)
could be standardized.

As others have noted, BBR does not work great out of the box for realtime
scenarios. The last time this was discussed, the prevailing idea was to
allow the QUIC CC to be used as a sort of circuit-breaker, but within that
envelope the application could use whatever realtime algorithm it preferred
(e.g, goog-cc).

On Thu, Nov 4, 2021 at 3:58 AM Piers O'Hanlon <[email-masked]>
wrote:

>
> On 3 Nov 2021, at 21:46, Luke Curley <[email-masked]> wrote:
>
> <https://source.chromium.org/chromium/chromium/src/+/master:net/third_party/quiche/src/quic/core/congestion_control/bbr_sender.cc;l=393> lets
> And there’s also the idle cwnd collapse/reset behaviour to consider if
> you’re sending a number of frames together and their inter-data gap exceeds
> the RTO - I’m not quite sure how the various QUIC stacks have translated
> RFC2861/7661 advice on this…?
>
> I guess Google’s Goog_CC appears to be well used in the WebRTC space (e.g.
> WEBRTC
> <https://webrtc.googlesource.com/src/+/refs/heads/main/modules/congestion_controller/goog_cc>
>  and aiortc
> <https://github.com/aiortc/aiortc/blob/1a192386b721861f27b0476dae23686f8f9bb2bc/src/aiortc/rate.py#L271>)
> despite the draft
> <https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc> never making
> it to RFC status… There's also SCREAM
> <https://datatracker.ietf.org/doc/rfc8298/> which has an open
> source implementation <https://github.com/EricssonResearch/scream> but
> not sure how widely deployed it is.
>
>

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 9 Nov 2021 13:12:43 -0500

Justin said:

"As others have noted, BBR does not work great out of the box for realtime
scenarios."

[BA] At the ICCRG meeting on Monday, there was an update on BBR2:
https://datatracker.ietf.org/meeting/112/materials/slides-112-iccrg-bbrv2-update-00.pdf

While there are some improvements, issues such as "PROBE_RTT" and rapid
rampup after loss remain, and overall, it doesn't seem like BBR2 is going
to help much with realtime scenarios.  Is that fair?

On Tue, Nov 9, 2021 at 12:46 PM Justin Uberti <
[email-masked]> wrote:

>> I guess Google’s Goog_CC appears to be well used in the WebRTC space
>> (e.g. WEBRTC
>> <https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc> never
>> making it to RFC status… There's also SCREAM
>>>>> >> DASH has the concept of Resync points that were designed exactly
>>>>> for this purpose (allowing you to emergency downshift in the middle of a
>>>>> segment).
>>>>> >>
>>>>> >> That exact challenge had three competing solutions, two of which
>>>>> are now part of the official dash.js code. And yes, the player can figure
>>>>> what the network can sustain *without* trying higher bitrate renditions.
>>>>> >>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 9 Nov 2021 10:25:06 -0800

> to help much with realtime scenarios.  Is that fair?

Yes, but...

Firstly, QUIC implementations are "in app", including the BBR 
implementation. So it would be really easy to offer an API that disables 
"probe RTT", or limit its impact on the stream of datagrams.

Secondly, most congestion is self inflicted -- sending faster than the 
uplink sustains, for example. There is value in detecting that quickly 
and reacting, and doing that in the transport.

I wish we had a set of specific issues to fix in the CC algorithms, so 
we could build profiles of BBR (for example) that fixes those issues.

-- Christian Huitema

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 9 Nov 2021 18:26:33 +0000

All current QUIC CCs (BBRv1/2, CUBIC, NewReno, etc.) are not well suited for real-time media, even for a rough “envelope” or “circuit-breaker”. RMCAT CCs are explicitly designed for real-time media, but, of course, rely on RTCP feedback, so must be adapted to QUIC feedback.

Mo

On 11/9/21, 1:13 PM, "Bernard Aboba" <[email-masked]<mailto:[email-masked]>> wrote:

Justin said:

"As others have noted, BBR does not work great out of the box for realtime scenarios."

While there are some improvements, issues such as "PROBE_RTT" and rapid rampup after loss remain, and overall, it doesn't seem like BBR2 is going to help much with realtime scenarios.  Is that fair?

On Tue, Nov 9, 2021 at 12:46 PM Justin Uberti <[email-masked]<mailto:[email-masked]>> wrote:
Ultimately we found that it wasn't necessary to standardize the CC as long as the behavior needed from the remote side (e.g., feedback messaging) could be standardized.

As others have noted, BBR does not work great out of the box for realtime scenarios. The last time this was discussed, the prevailing idea was to allow the QUIC CC to be used as a sort of circuit-breaker, but within that envelope the application could use whatever realtime algorithm it preferred (e.g, goog-cc).

On Thu, Nov 4, 2021 at 3:58 AM Piers O'Hanlon <[email-masked]<mailto:[email-masked]>> wrote:

-acbegen

---

From: Luke Curley <[email-masked]>
Date: Tue, 9 Nov 2021 14:58:30 -0800

Maybe a dumb thought, but is the PROBE_RTT phase required when sufficiently
application limited, as is primarily the case for live video? If I
understand correctly, it's meant to drain the queue to remeasure the
minimum RTT, but that doesn't seem necessary when the queue is constantly
being drained due to a lack of data to send.

Either way, the issue is that existing TCP algorithms don't care about the
live video use-case, and those are the ones that have been ported to QUIC
thus far. But like Justin mentioned, this doesn't actually matter for the
sake of standardizing a video over QUIC protocol provided the building
blocks are in place.

The real question is: do QUIC ACKs contain enough signal to implement an
adequate live video congestion control algorithm? If not, how can we
increase that signal, potentially taking cues from RMCAT (ex. RTT on a
per-packet basis)?

On Tue, Nov 9, 2021, 10:27 AM Mo Zanaty (mzanaty) <mzanaty=
[email-masked]> wrote:

> All current QUIC CCs (BBRv1/2, CUBIC, NewReno, etc.) are not well suited
> for real-time media, even for a rough “envelope” or “circuit-breaker”.
> RMCAT CCs are explicitly designed for real-time media, but, of course, rely
> on RTCP feedback, so must be adapted to QUIC feedback.
>
>
>
> Mo
>
>
>
>
>
> On 11/9/21, 1:13 PM, "Bernard Aboba" <[email-masked]> wrote:
>
>
>
> Justin said:
>
>
>

---

From: Kirill Pugin <[email-masked]>
Date: Tue, 9 Nov 2021 23:01:56 +0000

Fwiw, for RUSH/QUIC we tested RMCAT as CC 😃
We did have to “re-implement” RTCP feedback, but that was relatively straightforward and we could re-use RMCAT implementation that was used for video calling applications.

Best,
Kirill

From: Luke Curley <[email-masked]>
Date: Tuesday, November 9, 2021 at 2:58 PM
To: "Mo Zanaty (mzanaty)" <mzanaty=[email-masked]>
Cc: Bernard Aboba <[email-masked]>, Justin Uberti <[email-masked]>, Ian Swett <[email-masked]>, "Ali C. Begen" <[email-masked]>, MOQ Mailing List <[email-masked]>
Subject: Re: [Moq] Latency @ Twitch

Maybe a dumb thought, but is the PROBE_RTT phase required when sufficiently application limited, as is primarily the case for live video? If I understand correctly, it's meant to drain the queue to remeasure the minimum RTT, but that doesn't seem necessary when the queue is constantly being drained due to a lack of data to send.

Either way, the issue is that existing TCP algorithms don't care about the live video use-case, and those are the ones that have been ported to QUIC thus far. But like Justin mentioned, this doesn't actually matter for the sake of standardizing a video over QUIC protocol provided the building blocks are in place.

The real question is: do QUIC ACKs contain enough signal to implement an adequate live video congestion control algorithm? If not, how can we increase that signal, potentially taking cues from RMCAT (ex. RTT on a per-packet basis)?

On Tue, Nov 9, 2021, 10:27 AM Mo Zanaty (mzanaty) <mzanaty=[email-masked]<mailto:[email-masked]>> wrote:
All current QUIC CCs (BBRv1/2, CUBIC, NewReno, etc.) are not well suited for real-time media, even for a rough “envelope” or “circuit-breaker”. RMCAT CCs are explicitly designed for real-time media, but, of course, rely on RTCP feedback, so must be adapted to QUIC feedback.

Mo

Justin said:

[BA] At the ICCRG meeting on Monday, there was an update on BBR2:
https://datatracker.ietf.org/meeting/112/materials/slides-112-iccrg-bbrv2-update-00.pdf<https://datatracker.ietf.org/meeting/112/materials/slides-112-iccrg-bbrv2-update-00.pdf>

>> Or read the paper that even had “twitch” in its title here: https://ieeexplore.ieee.org/document/9429986<https://ieeexplore.ieee.org/document/9429986>
>>
> https://dl.acm.org/doi/pdf/10.1145/3458305.3478442<https://dl.acm.org/doi/pdf/10.1145/3458305.3478442>

-acbegen

>>> https://www.ietf.org/mailman/listinfo/moq<https://www.ietf.org/mailman/listinfo/moq>
>>
>> --
>> Moq mailing list
>> https://www.ietf.org/mailman/listinfo/moq<https://www.ietf.org/mailman/listinfo/moq>
--
Moq mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/moq<https://www.ietf.org/mailman/listinfo/moq>

--
Moq mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/moq<https://www.ietf.org/mailman/listinfo/moq>
--
Moq mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/moq<https://www.ietf.org/mailman/listinfo/moq>
--
Moq mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/moq<https://www.ietf.org/mailman/listinfo/moq>

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 9 Nov 2021 23:04:47 +0000

Hey

On Tue, 9 Nov 2021, 22:58 Luke Curley, <[email-masked]> wrote:

> Maybe a dumb thought, but is the PROBE_RTT phase required when
> sufficiently application limited, as is primarily the case for live video?
> If I understand correctly, it's meant to drain the queue to remeasure the

Folks might be interested in
https://datatracker.ietf.org/doc/draft-smith-quic-receive-ts/, which is on
the agenda at tomorrow's QUIC session.

Slides at
https://github.com/quicwg/wg-materials/blob/main/ietf112/receive-timestamp.pdf

---

From: Justin Uberti <[email-masked]>
Date: Tue, 9 Nov 2021 15:18:51 -0800

My recollection from QRTP is that QUIC ACKs contain enough of the info from
https://datatracker.ietf.org/doc/html/draft-holmer-rmcat-transport-wide-cc-extensions-01
to make GCC work. (Which is essentially packet arrival time deltas.)

Accordingly, you can essentially separate the transport layer into the base
QUIC protocol, which handles encryption and delivery feedback (coupled with
an application-injected congestion controller), and RTP, which
provides packetization and higher-level semantics (e.g., FIR).

On Tue, Nov 9, 2021 at 2:58 PM Luke Curley <[email-masked]> wrote:

>> Ultimately we found that it wasn't necessary to standardize the CC as
>> long as the behavior needed from the remote side (e.g., feedback messaging)
>> could be standardized.
>>
>>
>>

---

From: Justin Uberti <[email-masked]>
Date: Tue, 9 Nov 2021 15:20:53 -0800

Reading Lucas' note, I may have misremembered the modifications that we
needed to apply to QUIC to make QRTP work. But
https://datatracker.ietf.org/doc/draft-smith-quic-receive-ts/ seems like it
might have exactly the right info to make GCC work.

On Tue, Nov 9, 2021 at 3:18 PM Justin Uberti <[email-masked]>
wrote:

> Accordingly, you can essentially separate the transport layer into the
> base QUIC protocol, which handles encryption and delivery feedback (coupled
> with an application-injected congestion controller), and RTP, which
>> Either way, the issue is that existing TCP algorithms don't care about
>> the live video use-case, and those are the ones that have been ported to
>> QUIC thus far. But like Justin mentioned, this doesn't actually matter for
>> the sake of standardizing a video over QUIC protocol provided the building
>> blocks are in place.
>>
>>> As others have noted, BBR does not work great out of the box for
>>> realtime scenarios. The last time this was discussed, the prevailing idea
>>> was to allow the QUIC CC to be used as a sort of circuit-breaker, but
>>> within that envelope the application could use whatever realtime algorithm
>>> it preferred (e.g, goog-cc).
>>>
>>>
>>>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 9 Nov 2021 16:05:28 -0800

> being drained due to a lack of data to send.

The probe RTT mechanism is not engaged if there was at least one 
measurement at or below the stated RTT min in the last 10 seconds. So, 
yes, if there are "natural silences" there will be a clean RTT min 
measurement and no need to probe. But if there is competition with 
another data sources, that may well not happen.

I could trivially provide an API to my Quic stack and set the Probe RTT 
delay to 5 minutes or 1 hour, and look at the consequences. Most of the 
times it will probably work just fine, but I can see scenarios in which 
not resetting the RTT is actually harmful. For example, what if the RTT 
actually increased? So maybe we need to think about something smarter 
than either "stop everything every 10 seconds" or "keep the same min RTT 
for an hour."

If you are interested, there is quite a bit of literature about the RTT 
min issues and the "latecomer advantage" in the context of LEDBAT. Some 
of the LEDBAT issues apply almost directly to BBR. They may very well 
apply just the same to other CC algorithms based on delays.

> sake of standardizing a video over QUIC protocol provided the building
Once QUIC developers start being interested in the use case, they will 
most probably find ways to tune their stacks and make them usable. It 
might be as simple as adding a dozen or so test cases in the validation 
suites...
> increase that signal, potentially taking cues from RMCAT (ex. RTT on a

We could have an interesting debate about whether RTT measurements on a 
per packet basis provides more signal or more noise. The feedback loop 
has a delay of 1 RTT, it is not obvious that performing control actions 
more often than 4 or 8 times per RTT brings you all that much. In any 
case, the QUIC ACK carry a lot of information -- ACK value, delay 
between arrival of highest sequence number packet and departure of ACK, 
list of packet ranges being acked with this ACK.

Standard QUIC acks every two packets. Many QUIC stack support the "ACK 
Frequency" extension, usually to tune down the frequency of ACKs for 
performance reasons. But the ACK frequency extension could also be used 
to increase the ACK frequency if that makes more sense for the scenario.

-- Christian Huitema

>> On 11/9/21, 1:13 PM, "Bernard Aboba"<[email-masked]>  wrote:
>>
>>
>>
>> Justin said:
>>
>>
>>
>> On Thu, Nov 4, 2021 at 3:58 AM Piers O'Hanlon<[email-masked]>
>> wrote:
>>
>>
>>
>> On 3 Nov 2021, at 21:46, Luke Curley<[email-masked]>  wrote:
>>
>>
>>
>> source implementation<https://github.com/EricssonResearch/scream>  but
>> On Wed, Nov 3, 2021 at 2:12 PM Ian Swett<[email-masked]>  wrote:
>>
>> On Wed, Nov 3, 2021 at 4:40 PM Luke Curley<[email-masked]>  wrote:
>>
>> On Wed, Nov 3, 2021 at 10:05 AM Ali C. Begen<[email-masked]>
>> wrote:
>>
>>
>>
>>> On Nov 3, 2021, at 6:50 PM, Piers O'Hanlon<[email-masked]>
>> wrote:
>>>
>>>
>>>>> On Nov 2, 2021, at 3:39 AM, Luke Curley<[email-masked]>  wrote:
>>>>>

---

From: Joerg Ott <[email-masked]>
Date: Wed, 10 Nov 2021 16:28:48 +0100

If you can get to QUIC's readily available stats it seems you can
roughly approximate what you need for know for building some of the
RTP CC on top without extra RTCP feedback packets.

Per-packet timestamps mentioned by Justin would still be a nice to have.

Best,
Jörg

On 10.11.21 00:01, Kirill Pugin wrote:
> We did have to “re-implement” RTCP feedback, but that was relatively 
> straightforward and we could re-use RMCAT implementation that was used 
> for video calling applications.
> 
> Best,
> 
> Kirill
> 
> *From: *Luke Curley <[email-masked]>
> *Date: *Tuesday, November 9, 2021 at 2:58 PM
> *To: *"Mo Zanaty (mzanaty)" <mzanaty=[email-masked]>
> *Cc: *Bernard Aboba <[email-masked]>, Justin Uberti 
> <[email-masked]>, Ian Swett <[email-masked]>, "Ali 
> C. Begen" <[email-masked]>, MOQ Mailing List <[email-masked]>
> *Subject: *Re: [Moq] Latency @ Twitch
> 
> sufficiently application limited, as is primarily the case for live 
> video? If I understand correctly, it's meant to drain the queue to 
> remeasure the minimum RTT, but that doesn't seem necessary when the 
> queue is constantly being drained due to a lack of data to send.
> 
> QUIC thus far. But like Justin mentioned, this doesn't actually matter 
> for the sake of standardizing a video over QUIC protocol provided the 
> building blocks are in place.
> 
> On Tue, Nov 9, 2021, 10:27 AM Mo Zanaty (mzanaty) 
> <mzanaty=[email-masked] <mailto:[email-masked]>> 
> wrote:
> 
>     All current QUIC CCs (BBRv1/2, CUBIC, NewReno, etc.) are not well
>     suited for real-time media, even for a rough “envelope” or
>     “circuit-breaker”. RMCAT CCs are explicitly designed for real-time
>     media, but, of course, rely on RTCP feedback, so must be adapted to
>     QUIC feedback.
> 
>     Mo
> 
>     <https://datatracker.ietf.org/meeting/112/materials/slides-112-iccrg-bbrv2-update-00.pdf>
> 
>     While there are some improvements, issues such as "PROBE_RTT" and
>     rapid rampup after loss remain, and overall, it doesn't seem like
>     BBR2 is going to help much with realtime scenarios. Is that fair?
> 
>     On Tue, Nov 9, 2021 at 12:46 PM Justin Uberti
>     <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>         Ultimately we found that it wasn't necessary to standardize the
>         CC as long as the behavior needed from the remote side (e.g.,
>         feedback messaging) could be standardized.
> 
>         realtime scenarios. The last time this was discussed, the
>         prevailing idea was to allow the QUIC CC to be used as a sort of
>         circuit-breaker, but within that envelope the application could
>         use whatever realtime algorithm it preferred (e.g, goog-cc).
> 
>         On Thu, Nov 4, 2021 at 3:58 AM Piers O'Hanlon
>         <[email-masked] <mailto:[email-masked]>> wrote:
> 
>                 On 3 Nov 2021, at 21:46, Luke Curley
>                 <[email-masked] <mailto:[email-masked]>> wrote:
> 
>                 Yeah, there's definitely some funky behavior in BBR when
>                 application limited but it's nowhere near as bad as
>                 Cubic/Reno. With those algorithms you need to burst
>                 enough packets to fully utilize the congestion window
>                 before it can be grown. With BBR I believe you need to
>                 burst just enough to fully utilize the pacer, and even
>                 then this condition
>             And there’s also the idle cwnd collapse/reset behaviour to
>             consider if you’re sending a number of frames together and
>             their inter-data gap exceeds the RTO - I’m not quite sure
>             how the various QUIC stacks have translated RFC2861/7661
>             advice on this…?
> 
>                 I started with BBR first because it's simpler, but I'm
>                 going to try out BBR2 at some point because of the
>                 aforementioned PROBE_RTT issue. I don't follow the
>                 congestion control space closely enough; are there any
>                 notable algorithms that would better fit the live video
>                 use-case?
> 
>             I guess Google’s Goog_CC appears to be well used in the
>             WebRTC space (e.g. WEBRTC
>             <https://webrtc.googlesource.com/src/+/refs/heads/main/modules/congestion_controller/goog_cc> and
>             aiortc
>             <https://datatracker.ietf.org/doc/rfc8298/> which has an
>             open source implementation
>             <https://github.com/EricssonResearch/scream> but not sure
>             how widely deployed it is.
> 
>                 On Wed, Nov 3, 2021 at 2:12 PM Ian Swett
>                 <[email-masked] <mailto:[email-masked]>> wrote:
> 
>                      From personal experience, BBR has some issues with
>                     application limited behavior, but it is still able
>                     to grow the congestion window, at least slightly, so
>                     it's likely an improvement over Cubic or Reno.
> 
>                     On Wed, Nov 3, 2021 at 4:40 PM Luke Curley
>                     <[email-masked] <mailto:[email-masked]>>
>                     wrote:
> 
>                         I think resync points are an interesting idea
>                         although we haven't evaluated them. Twitch did
>                         push for S-frames in AV1 which will be another
>                         option in the future instead of encoding a full
>                         IDR frame at these resync boundaries.
> 
>                         An issue is you have to make the hard decision
>                         to abort the current download and frantically
>                         try to pick up the pieces before the buffer
>                         depletes. It's a one-way door (maybe your
>                         algorithm overreacted) and you're going to
>                         be throwing out some media just to redownload it
>                         at a lower bitrate.
> 
>                         Ideally, you could download segments in parallel
>                         without causing contention. The idea is to spend
>                         any available bandwidth on the new segment to
>                         fix the problem, and any excess bandwidth on the
>                         old segment in the event it arrives before the
>                         player buffer actually depletes. That's more or
>                         less the core concept for what we've built using
>                         QUIC, and it's compatible with resync points if
>                         we later go down that route.
> 
>                         And you're exactly right Piers. The fundamental
>                         issue is that a web player lacks the low level
>                         timing information required to infer
>                         inspects the time delta between packets to
>                         determine the send rate. That gets really
>                         difficult when the OS and browser delay flushing
>                         data to the application, be it for performance
>                         reasons or due to packet loss (to maintain
>                         head-of-line blocking).
> 
>                         I did run into CUBIC/Reno not being able to grow
>                         the congestion window when frames are sent one
>                         at a time (application limited). I don't believe
>                         BBR suffers from the same problem though due to
>                         the aforementioned rate estimator.
> 
>                         On Wed, Nov 3, 2021 at 10:05 AM Ali C. Begen
>                         <[email-masked]
>                         <mailto:[email-masked]>> wrote:
> 
> 
> 
>                              > On Nov 3, 2021, at 6:50 PM, Piers
>                             O'Hanlon <[email-masked]
>                             <mailto:[email-masked]>> wrote:
>                              >
>                              >
>                              >
>                              >> On 2 Nov 2021, at 20:39, Ali C. Begen
>                             <ali.begen=[email-masked]
>                             <mailto:[email-masked]>>
>                             wrote:
>                              >>
>                              >>
>                              >>
>                              >>> On Nov 2, 2021, at 3:39 AM, Luke Curley
>                             <[email-masked]
>                             <mailto:[email-masked]>> wrote:
>                              >>>
>                              >>> Twitch is a live one-to-many product.
>                             We primarily focus on video quality due to
>                             the graphical fidelity of video games.
>                             Viewers can participate in a chat room,
>                             which the broadcaster reads and can respond
>                             to via video. This means that latency is
>                             also somewhat important to facilitate this
>                             social interaction.
>                              >>>
>                              >>> A looong time ago we were using RTMP
>                             for both ingest and distribution (Flash
>                             player). We switched to HLS for distribution
>                             to gain the benefit of 3rd party CDNs, at
>                             the cost of dramatically increasing latency.
>                             A later project lowered the latency of HLS
>                             using chunked-transfer delivery, very
>                             similar to LL-DASH (and not LL-HLS). We're
>                             still using RTMP for contribution.
>                              >>>
>                              > I guess Apple do also have their
>                             BYTERANGE/CTE mode for LL-HLS which is
>                             pretty similar to LL-DASH.
> 
>                             Yes, Apple can list the parts (chunks in
>                             LL-DASH) as byteranges in the playlist but
>                             the frequent playlist refresh and part
>                             retrieval process is inevitable in LL-HLS,
>                             which is one of the main differences from
>                             LL-DASH (no need for manifest refresh and
>                             request per segment not chunk).
> 
>                              >
>                              >>>
>                              >>> During congestion, the current segment
>                             stalls and is delivered slower than the
>                             encoded bitrate. The player has no recourse
>                             than to wait for the segment to finish
>                             downloading, risking depleting the buffer.
>                             It can switch down to a lower rendition at
>                             segment boundaries, but these boundaries
>                             occur too infrequently (every 2s) to handle
>                             sudden congestion. Trying to switch earlier,
>                             either by canceling the current segment or
>                              >> DASH has the concept of Resync points
>                             that were designed exactly for this purpose
>                             (allowing you to emergency downshift in the
>                             middle of a segment).
>                              >>
>                              > I was curious if there are any studies or
>                             experience of how resync points perform in
>                             practice?
> 
>                             Resync points are pretty fresh out of the
>                             oven. dash.js has it in the roadmap but not
>                             yet implemented (and we also need to
>                             generate test streams). So, there is no data
>                             available yet with the real clients. But, I
>                             suppose you can imagine how in-segment
>                             switching can help in sudden bw drops
>                             especially for long segments.
> 
>                              >
>                              >>> The player is responsible for choosing
>                             the rendition to download. This is a problem
>                             when media is delivered frame-by-frame (ie.
>                             HTTP chunked-transfer), as we're effectively
>                             application-limited by the encoder bitrate.
>                             The player can only measure the arrival
>                             timestamp of data and does not know when the
>                             network can sustain a higher bitrate without
>                             just trying it. We hosted an ACM challenge
>                             for this issue in particular.
>                              >>
>                              > The limitation here may also be down to
>                             the lack of access to sufficiently accurate
>                             timing information about data arrivals in
>                             the browser - unfortunately the Streams API,
>                             which provides data from the fetch API,
>                             doesn’t directly timestamp the data arrivals
>                             so the JS app has to timestamp it which can
>                             suffer from noise such as scheduling etc -
>                             especially a problem for small/fast data
>                             arrivals.
> 
>                              > I guess another issue could be that if
>                             the system is only sending single frames
>                             then the network transport may be operating
>                             in application limited mode so the cwnd
>                             doesn’t grow sufficiently to take advantage
>                             of the available capacity.
> 
>                              >> That exact challenge had three competing
>                             solutions, two of which are now part of the
>                             official dash.js code. And yes, the player
>                             can figure what the network can sustain
>                             *without* trying higher bitrate renditions.
>                              >>
>                             <https://github.com/Dash-Industry-Forum/dash.js/wiki/Low-Latency-streaming>
>                             <https://ieeexplore.ieee.org/document/9429986>
>                              >>
>                              > There was a recent study that seems to
>                             show that none of the current algorithms are
>                             that great for low latency, and the two new
>                             dash.js ones appear to lead to much higher
>                             levels of rebuffering:
>                              >
>                             <https://dl.acm.org/doi/pdf/10.1145/3458305.3478442>
> 
>                             Brightcove’s paper uses the LoL and L2A
>                             algorithms from the challenge where low
>                             latency was the primary goal. For Twitch’s
>                             own evaluation, I suggest you watch:
>                             <https://www.youtube.com/watch?v=rcXFVDotpy4>
>                             We later addressed the rebuffering issue,
>                             developed LoL+, which is the version
>                             included in dash.js now and explained at the
>                             ieeexplore link I gave above.
> 
>                              >>> I believe this is why LL-HLS opts to
>                             burst small chunks of data (sub-segments) at
>                             the cost of higher latency.
>                              >>>
>                              >>>
>                              >>> Both of these necessitate a larger
>                             player buffer, which increases latency. The
>                             contribution system it's own problems, but
>                             let me sync up with that team first before I
>                             try to enumerate them.
>                              >>> --
>                              >>> Moq mailing list
>                              >>> [email-masked] <mailto:[email-masked]>
>                              >>>
>                             <https://www.ietf.org/mailman/listinfo/moq>
>                              >>
>                              >> --
>                              >> Moq mailing list
>                              >> [email-masked] <mailto:[email-masked]>
>                              >>
>                             <https://www.ietf.org/mailman/listinfo/moq>
> 
>                         -- 
>                         Moq mailing list
>                         [email-masked] <mailto:[email-masked]>
>                         <https://www.ietf.org/mailman/listinfo/moq>
> 
>             -- 
>             Moq mailing list
>             [email-masked] <mailto:[email-masked]>
>             <https://www.ietf.org/mailman/listinfo/moq>
> 
>         -- 
>         Moq mailing list
>         [email-masked] <mailto:[email-masked]>
>         <https://www.ietf.org/mailman/listinfo/moq>
> 
>     -- 
>     Moq mailing list
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/moq>
> 
>

---

From: "Piers O'Hanlon" <[email-masked]>
Date: Wed, 10 Nov 2021 15:44:41 +0000

I guess it depends whether a QUIC API would allow for obtaining per flow stats or just info for the entire connection? Using information from ACKs just for your flow may provide a clearer picture…

Best,

Piers

On 10 Nov 2021, at 15:28, Joerg Ott <[email-masked]<mailto:[email-masked]>> wrote:

Best,
Jörg

*From: *Luke Curley <[email-masked]<mailto:[email-masked]>>
*To: *"Mo Zanaty (mzanaty)" <mzanaty=[email-masked]<mailto:mzanaty=[email-masked]>>
*Cc: *Bernard Aboba <[email-masked]<mailto:[email-masked]>>, Justin Uberti <[email-masked]<mailto:[email-masked]>>, Ian Swett <[email-masked]<mailto:[email-masked]>>, "Ali C. Begen" <[email-masked]<mailto:[email-masked]>>, MOQ Mailing List <[email-masked]<mailto:[email-masked]>>
On Tue, Nov 9, 2021, 10:27 AM Mo Zanaty (mzanaty) <mzanaty=[email-masked]<mailto:mzanaty=[email-masked]> <mailto:[email-masked]>> wrote:
   On 11/9/21, 1:13 PM, "Bernard Aboba" <[email-masked]<mailto:[email-masked]>
   <mailto:[email-masked]>> wrote:
   "As others have noted, BBR does not work great out of the box for
   realtime scenarios."
   <[email-masked]<mailto:[email-masked]>
       <[email-masked]<mailto:[email-masked]> <mailto:[email-masked]>> wrote:
               <[email-masked]<mailto:[email-masked]> <mailto:[email-masked]>> wrote:
               you use application-limited samples if they would
               increase the send rate.
               <[email-masked]<mailto:[email-masked]> <mailto:[email-masked]>> wrote:
                   <[email-masked]<mailto:[email-masked]> <mailto:[email-masked]>>
                       the delivery rate. You would want something like
                       BBR's rate estimation
                       <[email-masked]<mailto:[email-masked]>
                            > On Nov 3, 2021, at 6:50 PM, Piers
                           O'Hanlon <[email-masked]<mailto:[email-masked]>
                            >> On 2 Nov 2021, at 20:39, Ali C. Begen
                           <ali.begen=[email-masked]<mailto:ali.begen=[email-masked]>
                           <mailto:[email-masked]>>
                            >>> On Nov 2, 2021, at 3:39 AM, Luke Curley
                           <[email-masked]<mailto:[email-masked]>
                           downloading the lower rendition in parallel,
                           only exacerbates the issue.
                           Yes, you need to get rid of that noise (see
                           LoL+).
                           the system is only sending single frames
                           then the network transport may be operating
                           in application limited mode so the cwnd
                           doesn’t grow sufficiently to take advantage
                           of the available capacity.
                           Unless the video bitrate is too low, this
                           should not be an issue most of the time.
                           levels of rebuffering:
                           Copying the authors in case they want to add
                           anything for the paper you cited.
                           -acbegen
                            >>> --
                            >>> Moq mailing list
                            >>> [email-masked]<mailto:[email-masked]> <mailto:[email-masked]>
                            >>>
                            >>
                            >> --
                            >> Moq mailing list
                            >> [email-masked]<mailto:[email-masked]> <mailto:[email-masked]>
                            >>
                           https://www.ietf.org/mailman/listinfo/moq
                           <https://www.ietf.org/mailman/listinfo/moq>
                       --                         Moq mailing list
                       [email-masked]<mailto:[email-masked]> <mailto:[email-masked]>
                       https://www.ietf.org/mailman/listinfo/moq
                       <https://www.ietf.org/mailman/listinfo/moq>
           --             Moq mailing list
           [email-masked]<mailto:[email-masked]> <mailto:[email-masked]>
           https://www.ietf.org/mailman/listinfo/moq
           <https://www.ietf.org/mailman/listinfo/moq>
       --         Moq mailing list
       [email-masked]<mailto:[email-masked]> <mailto:[email-masked]>
       https://www.ietf.org/mailman/listinfo/moq
       <https://www.ietf.org/mailman/listinfo/moq>
   --     Moq mailing list
   [email-masked]<mailto:[email-masked]> <mailto:[email-masked]>
   https://www.ietf.org/mailman/listinfo/moq
   <https://www.ietf.org/mailman/listinfo/moq>

---

From: Justin Uberti <[email-masked]>
Date: Wed, 10 Nov 2021 08:17:33 -0800

I think you want these stats for the entire connection; if the packets for
a flow have large interarrival deltas because of other packets that are
interleaved, it will appear that more queuing is occurring and therefore
throw off the CC algorithm.

See related commentary at
https://datatracker.ietf.org/doc/html/draft-holmer-rmcat-transport-wide-cc-extensions-01#section-2.1
.

On Wed, Nov 10, 2021 at 7:44 AM Piers O'Hanlon <[email-masked]>
wrote:

> I guess it depends whether a QUIC API would allow for obtaining per flow
> stats or just info for the entire connection? Using information from ACKs
> just for your flow may provide a clearer picture…
>
> Best,
>
> Piers
>
> On 10 Nov 2021, at 15:28, Joerg Ott <[email-masked]> wrote:
>
> straightforward and we could re-use RMCAT implementation that was used for
> video calling applications.
> Best,
> Kirill
> *Cc: *Bernard Aboba <[email-masked]>, Justin Uberti <
> [email-masked]>, Ian Swett <[email-masked]>, "Ali C.
> Begen" <[email-masked]>, MOQ Mailing List <[email-masked]>
> On Tue, Nov 9, 2021, 10:27 AM Mo Zanaty (mzanaty) <
> mzanaty=[email-masked] <mailto:[email-masked]
> <[email-masked]>>> wrote:
>    On 11/9/21, 1:13 PM, "Bernard Aboba" <[email-masked]
>    <mailto:[email-masked] <[email-masked]>>> wrote:
>    Justin said:
>    <
>    <mailto:[email-masked] <[email-masked]>>>
> wrote:
>        <[email-masked] <mailto:[email-masked]
> <[email-masked]>>> wrote:
>                <[email-masked] <mailto:[email-masked]
> <[email-masked]>>> wrote:
>                <
> https://source.chromium.org/chromium/chromium/src/+/master:net/third_party/quiche/src/quic/core/congestion_control/bbr_sender.cc;l=393
> > lets
>            <
> https://webrtc.googlesource.com/src/+/refs/heads/main/modules/congestion_controller/goog_cc
> > and
>            aiortc
>            <
> https://github.com/aiortc/aiortc/blob/1a192386b721861f27b0476dae23686f8f9bb2bc/src/aiortc/rate.py#L271
> >)
>            despite the draft
>            <https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc
> > never
>                <[email-masked] <mailto:[email-masked]
> <[email-masked]>>> wrote:
>                    <[email-masked] <mailto:[email-masked]
> <[email-masked]>>>
>                    wrote:
>                        <
> https://datatracker.ietf.org/doc/html/draft-cheng-iccrg-delivery-rate-estimation
> > which
>                        <mailto:[email-masked]
> <[email-masked]>>> wrote:
>                            <mailto:[email-masked]
> <[email-masked]>>> wrote:
>                             >
>                             >
>                             >
>                            <mailto:[email-masked]
> <[email-masked]>>>
>                            wrote:
>                             >>
>                             >>
>                             >>
>                            <mailto:[email-masked]
> <[email-masked]>>> wrote:
>                             >>>
>                            <
> https://github.com/Dash-Industry-Forum/dash.js/wiki/Low-Latency-streaming>
>                            <
> https://dl.acm.org/doi/pdf/10.1145/3458305.3478442>
>                             >>> [email-masked] <mailto:[email-masked]
> <[email-masked]>>
>                             >>>
>                             >> [email-masked] <mailto:[email-masked]
> <[email-masked]>>
>                             >>
>                        [email-masked] <mailto:[email-masked] <[email-masked]>>
>            [email-masked] <mailto:[email-masked] <[email-masked]>>
>        [email-masked] <mailto:[email-masked] <[email-masked]>>
>    [email-masked] <mailto:[email-masked] <[email-masked]>>