Subject: [rmcat] NADA Implementation in Mozilla

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Tue, 8 Dec 2020 15:24:27 +0000

Hi all,

A while back we presented some initial evaluation results of the NADA implementation in the open-source Mozilla browser under various real-world settings at IETF-106<https://www.ietf.org/proceedings/106/slides/slides-106-rmcat-nada-update-01>. After further tinkering, we are happy to share a cleaned-up version of our implementation rebased to a recent version of the Mozilla code:

https://github.com/sergio-mena/gecko-dev/tree/nada

It supports the use of either one-way-delay (OWD) or round-trip-time (RTT) as the congestion signal for NADA, and includes all algorithmic features for congestion control as described in the corresponding RFC<https://tools.ietf.org/html/rfc8698>. Some illustrative results from running the modified Firefox Nightly browser using the default algorithm, or OWD/RTT-based NADA can be found here<https://www.dropbox.com/s/g3idujywj573gax/2020-12-07-nada-eval-in-mozilla.pdf?dl=0>.

For your convenience, this branch also contains a folder. It contains brief instructions and a utility python script for post-processing the logs and plotting the graphs (as shown in the link above):

https://github.com/sergio-mena/gecko-dev/tree/nada/nada_eval

Since this effort has spanned over multiple years, we’ve rebased our code twice along the way to catch up with major changes in the Mozilla base code. Those rebases required us to squash our commit history. So we are also sharing two branches below for archiving the original commit history:

https://github.com/sergio-mena/gecko-dev/tree/nada_rebase_0
https://github.com/sergio-mena/gecko-dev/tree/nada_rebase_1

Please feel free to check out the above. Let us know if you are interested in integrating it in your WebRTC-based app or testing it out in the real-world – always happy to jump on a call for that 😊

Thanks,
Sergio and Xiaoqing

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Tue, 8 Dec 2020 17:03:18 +0100

Hi Xiaoqing and Sergio,

Great to see you are still working on it.

I proposed a while back the use of a common logging format for being 
able to create tools for measuring the performance and comparing the 
behavior of the different bwe algorithms:

https://github.com/medooze/bwe-stats-viewer

Would you consider adding support to it on your source code? If you are 
still using the transport wide cc feedback messages the information 
dumped would be quite straight forward.

Also, it would be interesting knowing how did you perform the testing 
scenarios and if we could create a set of scripts to replicate them 
easily, what do you think?

Best regards
Sergio

On 08/12/2020 16:24, Xiaoqing Zhu (xiaoqzhu) wrote:
>
> Hi all,
>
> A while back we presented some initial evaluation results of the NADA 
> implementation in the open-source Mozilla browser under various 
> real-world settings at IETF-106 
> <https://www.ietf.org/proceedings/106/slides/slides-106-rmcat-nada-update-01>. 
> After further tinkering, we are happy to share a cleaned-up version of 
> our implementation rebased to a recent version of the Mozilla code:
>
> <https://github.com/sergio-mena/gecko-dev/tree/nada>
>
> It supports the use of either one-way-delay (OWD) or round-trip-time 
> (RTT) as the congestion signal for NADA, and includes all algorithmic 
> features for congestion control as described in the corresponding RFC 
> <https://tools.ietf.org/html/rfc8698>. Some illustrative results from 
> running the modified Firefox Nightly browser using the default 
> algorithm, or OWD/RTT-based NADA can be found here 
> <https://www.dropbox.com/s/g3idujywj573gax/2020-12-07-nada-eval-in-mozilla.pdf?dl=0>. 
>
>
> For your convenience, this branch also contains a folder. It contains 
> brief instructions and a utility python script for post-processing the 
> logsand plotting the graphs (as shown in the link above):
>
> <https://github.com/sergio-mena/gecko-dev/tree/nada/nada_eval>
>
> Since this effort has spanned over multiple years, we’ve rebased our 
> code twice along the way to catch up with major changes in the Mozilla 
> base code.Those rebases required us to squash our commit history. So 
> we are also sharing two branches below for archiving the original 
> commit history:
>
> <https://github.com/sergio-mena/gecko-dev/tree/nada_rebase_0>
>
> <https://github.com/sergio-mena/gecko-dev/tree/nada_rebase_1>
>
> Please feel free to check out the above. Let us know if you are 
> interested inintegrating it in your WebRTC-based app ortesting it out 
> in the real-world – always happy to jump on a call for that😊
>
> Thanks,
>
> Sergio and Xiaoqing
>

---

From: Sergio Mena <[email-masked]>
Date: Tue, 8 Dec 2020 17:43:25 +0100

Hi Sergio,

Thanks a lot for your interest.

It’s a great idea to be able to compare common statistics across 
different bandwidth estimation algorithms.  In fact, in our code we had 
to modify the logging process in the default algorithm to support that, 
but the plotting part is less sophisticated than what you've been 
sharing :-)

Would you be willing to hook (and modify as needed) the logging 
mechanism in our Firefox branch to your repo? We can explain where all 
logs are located.

In one of our algorithm's modes -- NADA-OWD -- we are indeed still using 
transport-wide cc feedback. We had also implemented the new CCFB format 
(in past hackathons) in another branch, but we never came round to 
merging both branches.

About how we carried out the testing scenarios, they were mostly manual, 
i.e., having actual calls between two of us, then collecting and 
processing the logs from the sender. For more details on this, please 
check our update at IETF-106 
(https://www.ietf.org/proceedings/106/slides/slides-106-rmcat-nada-update-01). 
If someone could write scripts to automate this, that would be of course 
awesome!

Thanks,

Xiaoqing & Sergio

On 08/12/2020 17:03, Sergio Garcia Murillo wrote:
> Would you consider adding support to it on your source code? If you 
> are still using the transport wide cc feedback messages the 
> information dumped would be quite straight forward.
>
>> After further tinkering, we are happy to share a cleaned-up version 
>> of our implementation rebased to a recent version of the Mozilla code:
>>
>> brief instructions and a utility python script for post-processing 
>> the logsand plotting the graphs (as shown in the link above):
>>
>> code twice along the way to catch up with major changes in the 
>> Mozilla base code.Those rebases required us to squash our commit 
>> history. So we are also sharing two branches below for archiving the 
>> original commit history:
>>
> -->

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Tue, 8 Dec 2020 23:02:04 +0100

On 08/12/2020 17:43, Sergio Mena wrote:
> different bandwidth estimation algorithms.  In fact, in our code we 
> had to modify the logging process in the default algorithm to support 
> that, but the plotting part is less sophisticated than what you've 
> been sharing :-)
>
I am out on vacations on December, but may try to get something done 
when I get back to work on January ;)

> In one of our algorithm's modes -- NADA-OWD -- we are indeed still 
> using transport-wide cc feedback. We had also implemented the new CCFB 
> format (in past hackathons) in another branch, but we never came round 
> to merging both branches.
>

Transport wide cc would be fine, if we could try to hool the logging in 
the correct place we could get also some data from the default bwe 
algorithm and compare both.

>
> About how we carried out the testing scenarios, they were mostly 
> manual, i.e., having actual calls between two of us, then collecting 
> and processing the logs from the sender. For more details on this, 
> please check our update at IETF-106 
> If someone could write scripts to automate this, that would be of 
> course awesome!
>
I did some basic test with tc on the past, but it would be great if we 
could find the appropriate parameters to cover the test cases defined by 
the group.

Best regards

Sergio

---

From: Xiaoqing Zhu <[email-masked]>
Date: Wed, 9 Dec 2020 15:59:51 -0600

Hi Sergio,

Hope you enjoy your December vacation. Perhaps after that if you are
interested in taking a look at more automated testing and comparisons
between BWE algorithms, feel free to let us know and we can get on a call
go over the current logging mechanisms as a start.

It's a good idea to leverage tc to perform controlled tests (e.g., to cover
the test cases in the two drafts developed by this WG).  On top of that, if
we could figure out a way to perform evaluations and comparisons in the
wild in an automated fashion that would be awesome.

Cheers,
Xiaoqing

On Tue, Dec 8, 2020 at 4:01 PM Sergio Garcia Murillo <
[email-masked]> wrote:

> > (
> https://www.ietf.org/proceedings/106/slides/slides-106-rmcat-nada-update-01).
>