Subject: [AVTCORE] RTP circuit breakers practical experience feedback

From: Simon Perreault <[email-masked]>
Date: Fri, 07 Aug 2015 10:29:04 -0400

All,

We implemented draft-ietf-avtcore-rtp-circuit-breakers-10 and put it in
production here at Jive. It is exposed to all kinds of traffic:
intra-DC, inter-DC, SIP peer to SIP peer, and wild public Internet
traffic. I obviously can't discuss traffic volume. Here are some lessons
learned...

- The "media timeout" circuit breaker is unusable in practice because at
least one popular B2BUA implementation we routinely talk to (not going
to name names) lies in its RTCP reports. In one particular instance, it
sends RTCP with non-increasing maximum sequence numbers even though it
is receiving the media we send it. We have found no way to work around
this in general apart from fingerprinting the implementation of the
remote endpoint (which we are *not* considering, for obvious reasons).

- The "RTCP timeout" circuit breaker is unusable in practice because it
is common to stop receiving RTCP when an RTP relay we are sending to
changes its destination from an RTCP-sending receiver to a
non-RTCP-sending receiver. There is no indication at the signalling
level that this is happening because what happens beyond the RTP relay
is invisible to us. We have found no way to work around this in general.

- The "congestion" circuit breaker has never triggered. I don't really
know how to interpret this: am I supposed to rejoice or is the circuit
breaker simply useless in practice?

In addition, the text describing the algorithm leaves much to the
programmer's interpretation. I've discussed this with the authors and I
expect a revision with tighter text.

- We have not implemented the "media usability" circuit breaker as we
were unable to find a good practical criteria for usability. I'd be
surprised if anyone else did.

Simon

---

From: Colin Perkins <[email-masked]>
Date: Mon, 10 Aug 2015 22:32:46 +0100

Simon, 

Many thanks for implementing this, and reporting back to the group.

> is invisible to us. We have found no way to work around this in general.

These are clearly bugs in the devices you're communicating with. I agree that if such devices are common in some environments, then an RTP circuit breaker that checks connectivity using RTCP will be difficult to deploy in those environments. I'm not sure there's much we can do about this, other than report the bugs and hope the broken devices eventually get fixed. I will make sure the circuit breaker draft notes the issue. 

> breaker simply useless in practice?

It perhaps addresses the concern that the circuit breaker is overly sensitive... 

(If you have data you're able to share privately, I'd be interested in learning more about the performance you're seeing, and how the circuit breaker is behaving in your deployment.)

> expect a revision with tighter text.

We're working on this.

> surprised if anyone else did.

I think it depends very much on the application. If I were implementing an interactive video conferencing tool, I might treat a series of consecutive RTCP reports showing an RTT in excess of a couple of seconds as not usable media, for example. It's a deliberately vague fallback: if what you're receiving is clearly unusable for your application, consider stopping sending.

Cheers,
Colin

-- 
Colin Perkins
https://csperkins.org/

---

From: Simon Perreault <[email-masked]>
Date: Tue, 11 Aug 2015 07:52:40 -0400

Le 2015-08-10 17:32, Colin Perkins a écrit :
> These are clearly bugs in the devices you're communicating with. I
> agree that if such devices are common in some environments, then an
> RTP circuit breaker that checks connectivity using RTCP will be
> difficult to deploy in those environments. I'm not sure there's much
> we can do about this, other than report the bugs and hope the broken
> devices eventually get fixed. I will make sure the circuit breaker
> draft notes the issue.

Agreed. In our situation we have to deal with lots of different
implementations. I suppose in a pure WebRTC world things could be different.

>> - The "congestion" circuit breaker has never triggered. I don't
>> really know how to interpret this: am I supposed to rejoice or is
>> the circuit breaker simply useless in practice?
> 

Perhaps, although our data is exclusively from the server side, where we
have excellent connectivity. We have no circuit breaker implementation
running client-side on the wild public Internet. I guess the results
there could be different, especially for peer-to-peer, wild public
Internet to wild public Internet connections.

Simon