Subject: Some notes about WebRTC and QUIC

From: Ian Swett <[email-masked]>
Date: Wed, 17 Apr 2019 11:42:20 -0400

When in Prague, I organized a small WebRTC over QUIC lunch to discuss what
it might look like and what the goals are.

Attached are some notes from the conversation.

Thanks, Ian

---

From: Roberto Peon <[email-masked]>
Date: Wed, 17 Apr 2019 15:48:22 +0000

Thanks Ian!

Were the “Challenges” in any particular order?

It would be really quite interesting to get a good feel for the severity/impact of the lack-of-features/challenges!

-=R

From: QUIC <[email-masked]> on behalf of Ian Swett <ianswett=[email-masked]>
Date: Wednesday, April 17, 2019 at 8:43 AM
To: IETF QUIC WG <[email-masked]>
Subject: Some notes about WebRTC and QUIC

When in Prague, I organized a small WebRTC over QUIC lunch to discuss what it might look like and what the goals are.

Thanks, Ian

---

From: Ian Swett <[email-masked]>
Date: Wed, 17 Apr 2019 11:52:38 -0400

The challenges were not in any particular order.

Based on my experience, it's absolutely necessary to tune QUIC's ACK
behavior for realtime traffic, because they send lots of smaller packets,
so it's easy to send as many bytes of ACKs as data with a bit of loss or
reordering.  Conveniently, that's not overly difficult and I believe i
should be possible to create a fairly simple extension to add the necessary
features.

On Wed, Apr 17, 2019 at 11:48 AM Roberto Peon <[email-masked]> wrote:

> It would be really quite interesting to get a good feel for the
> severity/impact of the lack-of-features/challenges!
>
> -=R
>
>
>
> *From: *QUIC <[email-masked]> on behalf of Ian Swett <ianswett=
> [email-masked]>
> *Date: *Wednesday, April 17, 2019 at 8:43 AM
> *To: *IETF QUIC WG <[email-masked]>
> *Subject: *Some notes about WebRTC and QUIC
>
>
>

---

From: Roberto Peon <[email-masked]>
Date: Wed, 17 Apr 2019 16:06:42 +0000

Makes sense.

I’d imagine that I’d want my characters ack’d nearly immediately for things like ssh even when the ratio of bytes/acks is poor, but my VoD video can bundle acks on longer timeframes, and can reasonably do higher acks/bytes proportional to the client-buffer up-to, I guess, BDP.

Were you thinking about per-stream hinting or per-connection or something else?
-=R

From: Ian Swett <[email-masked]>
Date: Wednesday, April 17, 2019 at 8:53 AM
To: Roberto Peon <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Some notes about WebRTC and QUIC

Based on my experience, it's absolutely necessary to tune QUIC's ACK behavior for realtime traffic, because they send lots of smaller packets, so it's easy to send as many bytes of ACKs as data with a bit of loss or reordering.  Conveniently, that's not overly difficult and I believe i should be possible to create a fairly simple extension to add the necessary features.

On Wed, Apr 17, 2019 at 11:48 AM Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
Thanks Ian!

-=R

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Ian Swett <ianswett=[email-masked]<mailto:[email-masked]>>
Date: Wednesday, April 17, 2019 at 8:43 AM
To: IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Some notes about WebRTC and QUIC

Thanks, Ian

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 18 Apr 2019 07:18:47 +0000

Hi

And thanks for sharing the meeting notes. I believe that there are benefits with WebRTC over QUIC, especially the possibility to run a unified congestion control has some merit given simulation studies that I have run. 

For what it’s worth.. During the development of SCReAM (https://tools.ietf.org/html/rfc8298 ) I have collected some experience in how SCReAM works and what it requires in terms of feedback for optimal function. SCReAM is congestion control for RTP media, SCReAM has built in support for prioritization between media streams. The observant reader may make the observation that SCReAM resembles LEDBAT quite a lot and that is quite true, SCReAM however also adds the interaction with the media coders, stream queues and ECN/L4S support. Mixing small and large packets (voice, video) does not seem to be a problem in simulations, but sofar SCReAM has only been used for the transmission of video at bitrates ranging from ~200kbps to 50+Mbps.

I don’t believe that it is possible to take SCReAM and put it as is into a WebRTC over QUIC solution but perhaps some ideas can be stolen.

A few observations:

Timestamps : A receive timestamp for each packet is not needed for the function of SCReAM. It is sufficient with a receive timestamp of the last received RTP packet.  The current implementation ( <https://github.com/EricssonResearch/scream> https://github.com/EricssonResearch/scream) implements  <https://tools.ietf.org/html/draft-ietf-avtcore-cc-feedback-message-02> https://tools.ietf.org/html/draft-ietf-avtcore-cc-feedback-message-02 which reports timestamps per packet, the issue is that it increases the feedback size quite a lot. An earlier version of the SCReAM code generated feedback packets that were just 48bytes, the size with the new format is 128 bytes or even more.

ACK rates: A low bitrates it is possible with feedback as seldom as every 50-100ms, at higher bitrate more frequent feedback is required. For instance, at 50Mbps or beyond, feedback is transmitted roughly every 2ms.

Stream scheduling and prioritization: SCReAM implements an RTP queue per media stream, something that is best referred to as a credit based scheduler picks RTP packets from the queues in a priority fashion ( <https://tools.ietf.org/html/rfc8298#page-24> https://tools.ietf.org/html/rfc8298#page-24). This approach seems to work well according to tests we have done. The RTP queues are mostly empty but serves as shock absorbers for instance when link throughput drops quickly.

Regards

Ingemar

 

From: Ian Swett <[email-masked]> 
Sent: den 17 april 2019 17:42
To: IETF QUIC WG <[email-masked]>
Subject: Some notes about WebRTC and QUIC

Thanks, Ian

---

From: Ian Swett <[email-masked]>
Date: Fri, 19 Apr 2019 11:10:18 -0400

Roberto, I was thinking per-connection ACK policies.

Thanks Ingemar, that's great information.

On Thu, Apr 18, 2019 at 3:18 AM Ingemar Johansson S <
[email-masked]> wrote:

> Hi
>
>
>
> And thanks for sharing the meeting notes. I believe that there are
> benefits with WebRTC over QUIC, especially the possibility to run a unified
> congestion control has some merit given simulation studies that I have run.
>
>
>
> For what it’s worth.. During the development of SCReAM (
> https://tools.ietf.org/html/rfc8298 ) I have collected some experience in
> how SCReAM works and what it requires in terms of feedback for optimal
> function. SCReAM is congestion control for RTP media, SCReAM has built in
> support for prioritization between media streams. The observant reader may
> make the observation that SCReAM resembles LEDBAT quite a lot and that is
> quite true, SCReAM however also adds the interaction with the media coders,
> stream queues and ECN/L4S support. Mixing small and large packets (voice,
> video) does not seem to be a problem in simulations, but sofar SCReAM has
> only been used for the transmission of video at bitrates ranging from
> ~200kbps to 50+Mbps.
>
> I don’t believe that it is possible to take SCReAM and put it as is into a
> WebRTC over QUIC solution but perhaps some ideas can be stolen.
>
>
>
>
>
> A few observations:
>
>
>
> Timestamps : A receive timestamp for each packet is not needed for the
> function of SCReAM. It is sufficient with a receive timestamp of the last
> received RTP packet.  The current implementation (
> https://github.com/EricssonResearch/scream) implements
> https://tools.ietf.org/html/draft-ietf-avtcore-cc-feedback-message-02 which
> reports timestamps per packet, the issue is that it increases the feedback
> size quite a lot. An earlier version of the SCReAM code generated feedback
> packets that were just 48bytes, the size with the new format is 128 bytes
> or even more.
>

It's very nice to only need a receive timestamp from the last received RTP
pacekt.  Currently, GoogCC is designed to require one timestamp for every
RTP packet, which as you said greatly increases the size.

>
>
> ACK rates: A low bitrates it is possible with feedback as seldom as every
> 50-100ms, at higher bitrate more frequent feedback is required. For
> instance, at 50Mbps or beyond, feedback is transmitted roughly every 2ms.
>
>
>
> Stream scheduling and prioritization: SCReAM implements an RTP queue per
> media stream, something that is best referred to as a credit based
> scheduler picks RTP packets from the queues in a priority fashion (
> https://tools.ietf.org/html/rfc8298#page-24). This approach seems to work
> well according to tests we have done. The RTP queues are mostly empty but
> serves as shock absorbers for instance when link throughput drops quickly.
>
>
>
> Regards
>
> Ingemar
>
>
>
>
>
>
>
>
>
>
>
> *From:* Ian Swett <[email-masked]>
> *Sent:* den 17 april 2019 17:42
> *To:* IETF QUIC WG <[email-masked]>
> *Subject:* Some notes about WebRTC and QUIC
>
>
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 23 Apr 2019 08:45:05 +0000

Hi

As additional input. 

I understand that GoogCC need these per packet timestamps as it estimates the link bandwidth based in interpacket arrival.

SCReAM does however not benefit from this information as it is more “traditional” ACK-clocked in its design. However, being ACK-clocked it requires at least an ACK every RTT to keep a healthy self-clocking loop at high bitrates and that at-least-one-ACK-per-RTT  is actually to be seen as the absolute pain limit.

As far as I can see there could be congestion control work that benefits from a more dense time stamp.

BBR(v2) : I guess it is best if Neil or Yuchung outline what is needed in terms of timestamp for good BBR performance

TCP Prague (paced chirp) : The paced chirp method can benefit from more dense timestamps. I guess Bob is the best to provide with input on this and how much time stamp information that is needed for good performance.

Other work on congestion control ?

/Ingemar

From: Ian Swett <[email-masked]> 
Sent: den 19 april 2019 17:10
To: Ingemar Johansson S <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Some notes about WebRTC and QUIC

 

On Thu, Apr 18, 2019 at 3:18 AM Ingemar Johansson S <[email-masked] <mailto:[email-masked]> > wrote:

Hi

 

A few observations:

Timestamps : A receive timestamp for each packet is not needed for the function of SCReAM. It is sufficient with a receive timestamp of the last received RTP packet.  The current implementation ( <https://protect2.fireeye.com/url?k=593f7b92-05b470a8-593f3b09-86925ec6fd56-7ff6d8b56b340c6e&u=https://github.com/EricssonResearch/scream> https://github.com/EricssonResearch/scream) implements  <https://tools.ietf.org/html/draft-ietf-avtcore-cc-feedback-message-02> https://tools.ietf.org/html/draft-ietf-avtcore-cc-feedback-message-02 which reports timestamps per packet, the issue is that it increases the feedback size quite a lot. An earlier version of the SCReAM code generated feedback packets that were just 48bytes, the size with the new format is 128 bytes or even more.

It's very nice to only need a receive timestamp from the last received RTP pacekt.  Currently, GoogCC is designed to require one timestamp for every RTP packet, which as you said greatly increases the size.

Regards

Ingemar

 

From: Ian Swett <[email-masked] <mailto:[email-masked]> > 
Sent: den 17 april 2019 17:42
To: IETF QUIC WG <[email-masked] <mailto:[email-masked]> >
Subject: Some notes about WebRTC and QUIC

Thanks, Ian

---

From: Bjorn Mellem <[email-masked]>
Date: Tue, 23 Apr 2019 13:12:20 -0700

SCReAM could be interesting to experiment with in WebRTC-over-QUIC.  The
aspect of managing RTP queues and prioritizing through weighted credit
scheduling could be helpful.  Thanks for raising this!

We had some problems with the ACK-clocked part of BBR in early experiments:
bumping up against the CWND caused sending freezes and delay spikes.
However, it might be solvable by keeping changes to media rate control in
sync with changes to the CWND and applying back-pressure based on local
queues (as SCReAM does).

More inline.

Bjorn

On Tue, Apr 23, 2019 at 1:45 AM Ingemar Johansson S <
[email-masked]> wrote:

> Hi
>
>
>
> As additional input.
>
>
>
> I understand that GoogCC need these per packet timestamps as it estimates
> the link bandwidth based in interpacket arrival.
>
> SCReAM does however not benefit from this information as it is more
> “traditional” ACK-clocked in its design. However, being ACK-clocked it
> requires at least an ACK every RTT to keep a healthy self-clocking loop at
> high bitrates and that at-least-one-ACK-per-RTT  is actually to be seen as
> the absolute pain limit.
>

Interesting--so it needs more frequent ACKs but less data per ACK.  As
Ian's notes mentioned, one of the challenges we've had in WebRTC-over-QUIC
is tuning the ACK policy to control overhead (especially at low
bandwidth).  Since we're using GoogCC, we've mostly tuned it for less
frequent ACKs.

The real killer has been small ACK frames that aren't bundled with other
frames, as they contribute especially high UDP/IP overhead.

Scaling the frequency with bandwidth should help.  This is something that
WebRTC does with GoogCC as well.  We don't currently have that capability
in gQUIC, but it's the kind of ACK policy we're likely to end up wanting
from QUIC.

> BBR(v2) : I guess it is best if Neil or Yuchung outline what is needed in
> terms of timestamp for good BBR performance
>
> TCP Prague (paced chirp) : The paced chirp method can benefit from more
> dense timestamps. I guess Bob is the best to provide with input on this and
> how much time stamp information that is needed for good performance.
>

> *Sent:* den 19 april 2019 17:10
> *To:* Ingemar Johansson S <[email-masked]>
> *Cc:* IETF QUIC WG <[email-masked]>
> *Subject:* Re: Some notes about WebRTC and QUIC
>
>
>
> https://github.com/EricssonResearch/scream
> <https://protect2.fireeye.com/url?k=593f7b92-05b470a8-593f3b09-86925ec6fd56-7ff6d8b56b340c6e&u=https://github.com/EricssonResearch/scream>)
> implements