Subject: Re: [rmcat] Hackathon: RMCAT (and CCFB) in Jitsi (fwd)

From: Jonathan Lennox <[email-masked]>
Date: Fri, 1 Nov 2019 16:32:45 -0400

> On Oct 22, 2019, at 10:37 AM,Sergio Garcia Murillo <[email-masked]> wrote:
> 
> Hi Jonathan,
> 
> My C++ interface is slightly different, although I have it implemented for google's transport wide feedback message:
> https://github.com/medooze/media-server/blob/master/include/SendSideBandwidthEstimation.h <https://github.com/medooze/media-server/blob/master/include/SendSideBandwidthEstimation.h>
> 
> Main difference is that I have split your packet arrival processing in two calls, one for notifying when one rtp packet is sent and one to notify when a feedback packet is received with a map of the received stats. If you prefer to have an individual call per feedbacked rtp packet, I think it would be important to notify when the last packet of the feedback has been received as for example I do the final bwe calculations there.

I had a single call in hopes of using the same algorithm interface to implement REMB (receiver-side) feedback with the abs-send-time extension; in that case you only have information at the time of packet receipt, not packet transmission, and each piece of “feedback” information comes in separately.

I agree that some ability to tell the algorithm which feedback came in at once would be useful.

> I miss also to be able to control the congestion control part of the SFU, in my implementation which is based on BBR, I rely heavily on continuously sending RTX as probing bitrate as described on Varum's FRACTAL algorithm. 

How tightly coupled do you think probing needs to be with the algorithm?  I’ve generally considered probing to be a separable piece — the prober’s responsibility is to consume the output of the bwe algorithm, and to generate traffic when appropriate (i.e. when the overall application has some larger discrete quantum of bandwidth that it wants to be able to start sending).

> The main problem when implementing the bwe was about how to test it and troubleshoot it. I tried to use the ns3-rmcat repo, but the differences in behavior with my sfu are so huge that it was not practical. I tried also to use the ns3 direct code execution and reuse the ns3-rmcat scenarios, but the ns3-dce does not work at all with latest linux versions. 
> 
> I ended up testing with custom linux tc scripts, but I am not really sure that the ones I tested are meaningful or how to map them to the rmcat test scenarios. If we could create a common set of tc script modeling the rmcat test cases I think that would be very useful for all of us.

Yes — if there are transport / Linux people who are knowledgable about tc and its parameters, it would be very helpful to have some guidance about which settings best reflect the rmcat test cases in particular, and realistic router and network scenarios in general.

One tool I’ve found is tcconfig — https://github.com/thombashi/tcconfig <https://github.com/thombashi/tcconfig> — which is a front-end to tc with parameters comprehensible by mere mortals who aren’t deeply knowledgeable about Linux kernel queueing.  However, I still don’t know precisely how it should be mapped to rmcat tests.

> I won't be attending the Hackaton in Singapore, but Cosmo's Kite team will, so we could integrate those tc scripts into Kite's network instrumentation tests and have a run test with them for the Hackaton.

That sounds good, I’ll look for them.

> Last topic is troubleshooting, I have created a tool for viewing bwe stats dumps from my SFU: 
> https://github.com/medooze/bwe-stats-viewer <https://github.com/medooze/bwe-stats-viewer>
> 
> If we can agree on a common format for the logs, we could enhance it and use it for troubleshooting and comparing behaviors.

That does look useful.  Do you have documentation for the log format you’re using?

> Best regards
> Sergio
> On 21/10/2019 20:34, Jonathan Lennox wrote:
>> Hi, all —
>> 
>> As I’m now at 8x8 working on the open-source Jitsi project (https://jitsi.org/ <https://jitsi.org/>), I’ve been working on modifying the Jitsi Videobridge to make it easy to adapt it for alternate bandwidth estimation algorithms.  I hope this could be a good environment to make running-code RMCAT testing and comparisons possible.
>> 
>> The API I’ve created is hopefully clean and simple, modeled on the one in the ns3-rmcat repo with some changes.  The code is written in Kotlin, so Kotlin and Java should work with it immediately, and native (C or C++) should be able to work through JNI.
>> 
>> The interface can be seen at https://github.com/jitsi/jitsi-media-transform/blob/master/src/main/kotlin/org/jitsi/nlj/rtp/bandwidthestimation/BandwidthEstimator.kt <https://github.com/jitsi/jitsi-media-transform/blob/master/src/main/kotlin/org/jitsi/nlj/rtp/bandwidthestimation/BandwidthEstimator.kt>.
>> 
>> I welcome any comments, and I’d love it if people could give it a try — I’d be able to help out to let people get a Jitsi meet environment working.
>> 
>> Furthermore, I’ll be attending the Hackathon in Singapore.  Is anyone else planning to attend, in person or remotely, to work on RMCAT or CCFB work?  I’d love to work with folks to try to get interop or testing.
>> 
>> — 
>> Jonathan Lennox
>> [email-masked] <mailto:[email-masked]>
>> 
> 
> 
> 
> 
> -- 
> Jonathan Lennox
> [email-masked]

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 4 Nov 2019 12:00:44 +0100

On 01/11/2019 21:32, Jonathan Lennox wrote:
>
>
>> On Oct 22, 2019, at 10:37 AM,Sergio Garcia Murillo 
>> <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Hi Jonathan,
>>
>> My C++ interface is slightly different, although I have it 
>> implemented for google's transport wide feedback message:
>> https://github.com/medooze/media-server/blob/master/include/SendSideBandwidthEstimation.h
>>
>> Main difference is that I have split your packet arrival processing 
>> in two calls, one for notifying when one rtp packet is sent and one 
>> to notify when a feedback packet is received with a map of the 
>> received stats. If you prefer to have an individual call per 
>> feedbacked rtp packet, I think it would be important to notify when 
>> the last packet of the feedback has been received as for example I do 
>> the final bwe calculations there.
>
> I had a single call in hopes of using the same algorithm interface to 
> implement REMB (receiver-side) feedback with the abs-send-time 
> extension; in that case you only have information at the time of 
> packet receipt, not packet transmission, and each piece of “feedback” 
> information comes in separately.

I tried that, but mixing REMB and transport wide cc feedback on same API 
didn't worked for me.

>> I miss also to be able to control the congestion control part of the 
>> SFU, in my implementation which is based on BBR, I rely heavily on 
>> continuously sending RTX as probing bitrate as described on Varum's 
>> FRACTAL algorithm.
>
> How tightly coupled do you think probing needs to be with the 
> algorithm?  I’ve generally considered probing to be a separable piece 
> — the prober’s responsibility is to consume the output of the bwe 
> algorithm, and to generate traffic when appropriate (i.e. when the 
> overall application has some larger discrete quantum of bandwidth that 
> it wants to be able to start sending).

Most probably is a requirement given my design choices. As I said 
before, my implementation is a delay based algorithm inspired on BBR and 
I always send RTX as probing bitrate when not sending the upper layer 
available on the SFU. So when I detect a increase on the delay when the 
bloat bitrate is hit, I update the bwe estimation, but set the probing 
bitrate to a proportionate amount lower than the bloat bitrate to 
decrease the delay back to best case scenario

[...]

>> https://github.com/medooze/bwe-stats-viewer
>>
>> If we can agree on a common format for the logs, we could enhance it 
>> and use it for troubleshooting and comparing behaviors.
>

I have updated the readme with that file 
(https://github.com/medooze/bwe-stats-viewer/blob/master/README.md). 
Basically it is a csv file with a per packet stats (so should be quite 
easy to add with your API

Best regards

Sergio