Subject: [rmcat] How we should handle feedback, and where the congestion should run

From: Randell Jesup <[email-masked]>
Date: Thu, 5 Nov 2015 19:43:47 -0500

The line in the room was cut, but a few comments:

Even if RMCAT yields a single algorithm, having a standardized feedback 
message will help future work to improve on the algorithm, or to develop 
or deploy future algorithms.  This will not be the last work done in 
this area, and I'm sure improvements will be worked on, and people will 
trial alternative algorithms.  Getting support for the "other end" is a 
huge step forward in trialing (or deploying) an algorithm.

Also it appears there are multiple implementers dealing with the 
bandwidth and frequency for RTCP traffic if the control is on the sender 
side.  Almost all algorithms *can* be moved to run entirely on one side 
or on the other.  Running it on the receiver side can dramatically 
reduce the feedback required (as Michael said in the room).  Given use 
of the RTP send-time header extension, there's very little info that the 
sender has that the receiver doesn't (mostly just the size and timing of 
any lost packets).  The bandwidth for sending CC results would be far 
less than any (compressed or not) arrival time feedback to the sender.

Running on the receiver also means that no "merging" of feedback data 
needs to be done to run CC across multiple streams/5-tuples/etc.

We can build a generic arrival-time feedback message, or we can build a 
generic results-of-receiver-CC message.

-- 
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email [email-masked]!  Way too much spam

---

From: Michael Welzl <[email-masked]>
Date: Fri, 6 Nov 2015 08:52:00 +0100

> On 06 Nov 2015, at 01:43, Randell Jesup <[email-masked]> wrote:
> 
> Even if RMCAT yields a single algorithm, having a standardized feedback message will help future work to improve on the algorithm, or to develop or deploy future algorithms.  This will not be the last work done in this area, and I'm sure improvements will be worked on, and people will trial alternative algorithms.  Getting support for the "other end" is a huge step forward in trialing (or deploying) an algorithm.
> 
> Also it appears there are multiple implementers dealing with the bandwidth and frequency for RTCP traffic if the control is on the sender side.  Almost all algorithms *can* be moved to run entirely on one side or on the other.  Running it on the receiver side can dramatically reduce the feedback required (as Michael said in the room).  Given use of the RTP send-time header extension, there's very little info that the sender has that the receiver doesn't (mostly just the size and timing of any lost packets).  The bandwidth for sending CC results would be far less than any (compressed or not) arrival time feedback to the sender.
> 
> Running on the receiver also means that no "merging" of feedback data needs to be done to run CC across multiple streams/5-tuples/etc.
> 
> We can build a generic arrival-time feedback message, or we can build a generic results-of-receiver-CC message.

The big problem I see with receiver-side schemes is that you don't only need to standardize 1) the feedback message, you also need to standardize the sender behavior in the absence of feedback.
2) If feedback is missing for a while, maybe that's a good thing, as a result of feedback suppression?  So we just gradually and blindly increase the rate?  See early versions of the GCC scheme.
3) If feedback is missing for a long time, at some point you HAVE to have a timeout and react accordingly.

Can we agree on all these things? At least 1) AND 3) are necessary.

To me, this seems like creating difficulties for a problem that MAY also be a non-problem. Note that TCP sends tons of feedback, maybe including SACK blocks, across the same thin uplinks that we're considering. I don't think there is conclusive evidence of that being a problem.

=> I tend to think that moving everything to the sender side is an easy way out.

Cheers,
Michael

---

From: Randell Jesup <[email-masked]>
Date: Fri, 6 Nov 2015 11:09:34 -0500

> Can we agree on all these things? At least 1) AND 3) are necessary.

I agree that we'd need to standardize that when using a receiver-side 
algorithm with a generic message (i.e. not a matched sender with a 
custom/private message), the sender side must be prepared to deal with 
1) and 3) and likely 2), and we would need to specify fairly completely 
the sender-side behavior in response to the feedback (though that 
behavior can be simple).  I.e. the send will attempt to send at the rate 
requested in a feedback message; will inform codecs of the new rate as 
fast as possible; will optionally pace data entering the network to not 
exceed either the congestion rate, or a separate pacing rate supplied.  
Time periods for calculating both rates shall also be provided, though 
the sender may not be able to use an arbitrary time for calculating rate 
(i.e. encoders may not be flexible on that - easier to do for pacing).

In terms of 1/2/3 above: if we do this, the receive-side algorithm could 
configure the sender via these messages and/or negotiation. The feedback 
messages can (optionally) have control information that tells the sender 
what to do on missing feedback, and when.  For example, it could have a 
feedback-timeout field, or timeout and what-to-do-at-timeout fields.  We 
could also define a "are you still there" RTCP that can be sent when we 
see feedback timeouts or are getting close to them, asking the receiver 
to resend (and exactly how to respond to those can be part of the 
receive-side CC algorithm).  It could include updated RS/RR stats for 
any RTP flows, letting the receiver-side update their idea of the 
return-path congestion.

This brings up a more general point for coupling congestion control (I 
forget if this has been discussed) - not only can it make sense to 
couple multiple CC's running in the same direction, but also to inform 
CC's of return-path congestion (and expected network delay!) that 
feedback (RTCP) will contend with.  This applies no matter which end the 
algorithms run at.

I'm sure more specification of a standard receive-side feedback message 
would needed if we decide to go that way; the above would be a start.  I 
do think it's *very* worthwhile defining a standard for feedback 
(whichever end we end up with) such that new single-ended algorithms can 
be trialed or deployed (and coincidentally provides a way for the CC 
algorithm to probe to see if the other side supports something 
compatible, without mandating it be part of a higher-level negotiation).

> To me, this seems like creating difficulties for a problem that MAY also be a non-problem. Note that TCP sends tons of feedback, maybe including SACK blocks, across the same thin uplinks that we're…

True - but the constraints on TCP are very different, forward delay is 
encouraged to maximize throughput, traffic is often primarily 
unidirectional (so the feedback path is uncontended and often close to 
loss-free especially where the bottleneck link is the local last-mile or 
Wifi), while for us we typically have saturated links in both directions 
while trying to keep delays low, and each packet of feedback takes away 
from payload.

> => I tend to think that moving everything to the sender side is an easy way out.

This still may be so, since in many ways (see above) it's easier to 
feedback raw data and keep *all* decisions at the sender side. (Though a 
sender-side standard feedback packet may still need rules on how to 
operate in reverse-path contention/loss that will need to be added.)  
However, as several of the candidates mentioned, there are issues with 
reverse-path bandwidth use - and I don't think they've even really 
gotten much into two-way tests; all the graphs I see are for one-way.  
Two-way will complicate things - and it's a critical part of the use-cases.

A send-side setup with feedback from the receiver for each packet 
(generic or not!) will almost certainly require careful compression 
decisions (and feedback-rate decisions, perhaps semi-controlled by the 
sender - see above similar to the receive-side discussion).  For this 
reason alone, I'm not sure if RTCP-XR will work, even with modifications 
or additions - but I haven't looked much at RTCP-XR other than to note 
it has a LOT of stuff and isn't very thin.  But take that with a grain 
of salt.

---

From: Michael Welzl <[email-masked]>
Date: Fri, 6 Nov 2015 17:54:41 +0100

> On 6. nov. 2015, at 17.09, Randell Jesup <[email-masked]> wrote:
> 
> I agree that we'd need to standardize that when using a receiver-side algorithm with a generic message (i.e. not a matched sender with a custom/private message), the sender side must be prepared to deal with 1) and 3) and likely 2), and we would need to specify fairly completely the sender-side behavior in response to the feedback (though that behavior can be simple).  I.e. the send will attempt to send at the rate requested in a feedback message; will inform codecs of the new rate as fast as possible; will optionally pace data entering the network to not exceed either the congestion rate, or a separate pacing rate supplied.  Time periods for calculating both rates shall also be provided, though the sender may not be able to use an arbitrary time for calculating rate (i.e. encoders may not be flexible on that - easier to do for pacing).
> 
> In terms of 1/2/3 above: if we do this, the receive-side algorithm could configure the sender via these messages and/or negotiation. The feedback messages can (optionally) have control information that tells the sender what to do on missing feedback, and when.  For example, it could have a feedback-timeout field, or timeout and what-to-do-at-timeout fields.  We could also define a "are you still there" RTCP that can be sent when we see feedback timeouts or are getting close to them, asking the receiver to resend (and exactly how to respond to those can be part of the receive-side CC algorithm).  It could include updated RS/RR stats for any RTP flows, letting the receiver-side update their idea of the return-path congestion.

I agree about all these “could”s - but I’m trying to be pragmatic about what will happen…. I see an everything-on-the-sender variant as a more realistic target for the group.

> This brings up a more general point for coupling congestion control (I forget if this has been discussed) - not only can it make sense to couple multiple CC's running in the same direction, but also to inform CC's of return-path congestion (and expected network delay!) that feedback (RTCP) will contend with.  This applies no matter which end the algorithms run at.

No that hasn’t been discussed….

> I'm sure more specification of a standard receive-side feedback message would needed if we decide to go that way; the above would be a start.  I do think it's *very* worthwhile defining a standard for feedback (whichever end we end up with) such that new single-ended algorithms can be trialed or deployed (and coincidentally provides a way for the CC algorithm to probe to see if the other side supports something compatible, without mandating it be part of a higher-level negotiation).

I completely agree.

> True - but the constraints on TCP are very different, forward delay is encouraged to maximize throughput, traffic is often primarily unidirectional (so the feedback path is uncontended and often close to loss-free especially where the bottleneck link is the local last-mile or Wifi), while for us we typically have saturated links in both directions while trying to keep delays low, and each packet of feedback takes away from payload.

So consider a TCP download going on while you have upload video… not an unlikely scenario and the ACKs won’t give way. Is it a huge problem? Shouldn’t be….

> This still may be so, since in many ways (see above) it's easier to feedback raw data and keep *all* decisions at the sender side. (Though a sender-side standard feedback packet may still need rules on how to operate in reverse-path contention/loss that will need to be added.)  However, as several of the candidates mentioned, there are issues with reverse-path bandwidth use - and I don't think they've even really gotten much into two-way tests; all the graphs I see are for one-way.  Two-way will complicate things - and it's a critical part of the use-cases.
> 
> A send-side setup with feedback from the receiver for each packet (generic or not!) will almost certainly require careful compression decisions (and feedback-rate decisions, perhaps semi-controlled by the sender - see above similar to the receive-side discussion).  For this reason alone, I'm not sure if RTCP-XR will work, even with modifications or additions - but I haven't looked much at RTCP-XR other than to note it has a LOT of stuff and isn't very thin.  But take that with a grain of salt.

So again, my main reason to argue for sender-side-only is that I think this is a goal the group can achieve. Yes it may not be the absolutely best technical solution, but it’s probably better than discussing complicated things that we won’t be able to agree on before all the other work in RMCAT is finished. I’m just trying to be realistic… I have trouble imagining all the CC authors agreeing on a common reacting to 2) and 3) above… at least 3) would be needed.

I’d be happy to be proven wrong of course!  But let’s try to agree on SOMETHING soon.

Cheers,
Michael

---

From: John Leslie <[email-masked]>
Date: Fri, 6 Nov 2015 13:43:29 -0500

Michael Welzl <[email-masked]> wrote:
> 
> The big problem I see with receiver-side schemes is that you don't only
> need to standardize
> 1) the feedback message, you also need to standardize the sender behavior
>    in the absence of feedback.

   The sender must take responsibility for the congestion it causes.
(Nobody else _can_!)

> 2) If feedback is missing for a while, maybe that's a good thing, as a
>    result of feedback suppression?  So we just gradually and blindly
>    increase the rate?  See early versions of the GCC scheme.

   Absent feedback proving a reasonably uncongested path, the sender
MUST NOT increase its sending rate.

>    have a timeout and react accordingly.

   It's not quite a "MUST", but the sender SHOULD decay its sending
rate in the absence of positive feedback that the path is uncongested.

> Can we agree on all these things? At least 1) AND 3) are necessary.

   I'd say both 1) and 2) are necessary, while 3) is a strong SHOULD.

> To me, this seems like creating difficulties for a problem that MAY
> also be a non-problem. Note that TCP sends tons of feedback, maybe
> including SACK blocks, across the same thin uplinks that we're
> considering. I don't think there is conclusive evidence of that being
> a problem.

   Nonetheless, both cable DSLAMs and wireless can suffer packet
congestion; and don't forget that both exist (in series) for many of
the customers we wish to serve.

   That said, it's a _feature_ to work around, not a total barrier.

> => I tend to think that moving everything to the sender side is an

   I entirely agree.

   Also, the _sender_ knows what data-rate he was sending when the packet
which is marked or dropped was sent. It's _much_ simpler for the sender
to keep this data, and moderate its sending rate to "slightly less" than
that rate when it learns the packet was CE-marked.

   IMHO, RMCAT should be designing to maintain a uniform sending rate
whenever possible. Sawtooth sending-rate is our enemy!

--
John Leslie <[email-masked]>

---

From: Randell Jesup <[email-masked]>
Date: Fri, 6 Nov 2015 14:47:46 -0500

>     The sender must take responsibility for the congestion it causes.

Yes, though that doesn't say where various processing has to happen.

I do see that it's likely simpler to standardize on 
feedback-for-sender-side-CC; I am concerned that we can compress that 
data enough to avoid practical problems in two-way communication at 
low-delay.

> MUST NOT increase its sending rate.

Sure.

In uncongested state (at or near BW cap rate for the flow), we *could* 
pace out feedback (and freeze the sending rate).  If we're bouncing off 
the channel limits, we'll generally want to be conservative about 
increases and react quickly to decreases (increases in latency); this 
either requires constant low-delay feedback OR requires a more complex 
algorithm that provides some support on the receiver side to identify 
decreases and send feedback earlier/more-frequently.   You can also 
design things to allow the sender to control the receiver's feedback 
over RTCP (or a header extension!) (ask for more-frequent reports or a 
report 'now' because it wants to bump the rate to test the barrier).

I used to run MUCH slower than RTT feedback except when I saw a strong 
change in delay or loss; slow changes in delay would cause me to wait to 
react (some) and see if it's transitory, and avoid using as much return 
bandwidth (or over-correcting).  (Note: I was running CC on the receive 
side, tracking delay slope, and generally running on 100-250Kbps rates - 
single packet-per-frame.  Also I wasn't using any fancy filters on the 
delay tracking.)  But decisions I made 10+ years ago for a different 
environment aren't (totally) applicable today - though some aspects may 
still be, especially in lower-bandwidth networks.

> rate in the absence of positive feedback that the path is uncongested.

Yes.  Or start asking for feedback (see above) and reducing rates pretty 
fast if feedback doesn't come (i.e. more extreme version of 'decay').  
Such requests could also inform the far side of a bad bottleneck in the 
return path, serving as a secondary signal to couple-CC at the far 
(receiver) end to reduce RTP send rates as well.

> that rate when it learns the packet was CE-marked.

The receiver should be able to recover the sender's approximate sending 
rate from the timestamps/header-extensions from the sending side; any 
receive-side algorithm would be doing this already anyways.  It is 
simpler to know about dropped packet sizes, but I think in practice this 
would have little impact.

> whenever possible. Sawtooth sending-rate is our enemy!

Yes! (See above about stability at a) cap-rate for the flow, and b) when 
we've discovered a (stable!) path limit.   (i.e. we should probe more if 
the limit seems dynamic, and less if the limit seems relatively static.  
My old algorithm also had a concept for the highest stable limit found 
for the entire flow, and got VERY conservative going over that limit.  
This worked very well for adapting to last-mile link limits and avoiding 
sawtooth; perhaps less so for WiFi but still not bad.

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Sat, 7 Nov 2015 04:33:44 +0000

Hi Randell, 

I noticed your mention of ³Š I don't think they've even really gotten much
into two-way tests; all the graphs I see are for one-way. Two-way will
complicate things - and it's a critical part of the use-cases.²  Fully
agree on the importance of two-way calls as use cases.

In the eval-test-case draft, there is currently one test case dedicated
for exploring impact of two-way traffic (Sec. 5.3.  Congested Feedback
Link with Bi-directional RMCAT flows). Some corresponding graphs can be
found below (they may not reflect the most up-to-date algorithm
performance). 

* NADA:http://www.ietf.org/proceedings/92/slides/slides-92-rmcat-4.pdf
(page #9 and #10)
* SCReAM: 
http://www.ietf.org/proceedings/interim/2014/11/09/rmcat/slides/slides-inte
rim-2014-rmcat-1-1.pdf (page #9)
(Sorry, I was not able to find one with GCC yet.).
  
Would like to know whether you think that current design of the test case
has captured gist of the issue encountered by two-way calls, or any
suggestions on additional test scenarios in that regard?

Thanks,
Xiaoqing

On 11/6/15, 8:09 AM, "rmcat on behalf of Randell Jesup"
<[email-masked] on behalf of [email-masked]> wrote:

>>need to standardize 1) the feedback message, you also need to
>>standardize the sender behavior in the absence of feedback.
>>considering. I don't think there is conclusive evidence of that being a
>>problem.
>

---

From: Randell Jesup <[email-masked]>
Date: Sat, 7 Nov 2015 00:43:38 -0500

> (Sorry, I was not able to find one with GCC yet.).

Thanks!  Those don't look too bad, though it can be hard to drill down 
into the details due to the PDF and thickness of the lines (and 
dense-ness of the graph); I found the closeups at the end of your pdf 
especially interesting (though not focused on this case).  One question 
that will be interesting as we develop a feedback mechanism is going to 
be the requirements from the algorithms on it, and the bandwidth it uses.

I note NADA seems to be slower to respond to delay, though it also seems 
to do much better against competing TCP flows (and likely the two items 
are linked, which will make for some interesting tradeoffs and tuning to 
do).

> suggestions on additional test scenarios in that regard?

I think showing a graph similar to Scream's, with congested and 
non-congested feedback paths is a useful comparison - much easier to see 
the impact.  Feedback bandwidth would be useful, though that might be 
fairly fixed right now (multiple of packet rate), so unless it's 
surprising it may not need graphing (just noting on the graph).

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 09 Nov 2015 07:57:11 +0000

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 09 Nov 2015 08:01:30 +0000

Sorry, accidentally sent too quickly...

>> > (Sorry, I was not able to find one with GCC yet.).

Simulations at slides 33-35
Emulations at slides 42-43
https://www.ietf.org/proceedings/interim/2015/07/19/rmcat/slides/slides-interim-2015-rmcat-1-2.pdf

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 09 Nov 2015 08:22:18 +0000

On Fri, Nov 6, 2015 at 8:52 AM Michael Welzl <[email-masked]> wrote:

> message will help future work to improve on the algorithm, or to develop or
> deploy future algorithms.  This will not be the last work done in this
> area, and I'm sure improvements will be worked on, and people will trial
> alternative algorithms.  Getting support for the "other end" is a huge step
> forward in trialing (or deploying) an algorithm.
> >
> side.  Almost all algorithms *can* be moved to run entirely on one side or
> on the other.  Running it on the receiver side can dramatically reduce the
> feedback required (as Michael said in the room).  Given use of the RTP
> send-time header extension, there's very little info that the sender has
> that the receiver doesn't (mostly just the size and timing of any lost
> packets).  The bandwidth for sending CC results would be far less than any
> (compressed or not) arrival time feedback to the sender.
>

I also think the sender has useful information about the type of stream
it's encoding. It knows if the stream is likely to produce an average
bitrate which can match the target (e.g., regular video from a camera) or
if it's likely to produce something that is very bursty and not likely to
produce an average bitrate matching the target (e.g., screencasts). It's
not obvious to me that the CC should behave in the same way for these two
cases, and therefore it's also not obvious to me that the right CC result
to feedback is a target bitrate.

Another thing that the receiver doesn't know, is if the sender decided to
send the packets in some particular pattern to probe the link. It can try
to infer the packet based on the send-time extension, but it's much easier
to do at the sender where you actually know what you tried to do.

> needs to be done to run CC across multiple streams/5-tuples/etc.

That is true, but I think it will be questioned if the RTP architecture
allows you to send an RTCP message containing CC results covering multiple
RTP streams. And if that's not allowed then you'd still have to feed back
CC results for each stream separately and have the sender combine them...

> need to standardize 1) the feedback message, you also need to standardize
> the sender behavior in the absence of feedback.
> result of feedback suppression?  So we just gradually and blindly increase
> the rate?  See early versions of the GCC scheme.

> To me, this seems like creating difficulties for a problem that MAY also
> be a non-problem. Note that TCP sends tons of feedback, maybe including
> SACK blocks, across the same thin uplinks that we're considering. I don't
> think there is conclusive evidence of that being a problem.
>

---

From: Colin Perkins <[email-masked]>
Date: Mon, 9 Nov 2015 11:22:10 +0000

> On 9 Nov 2015, at 08:22, Stefan Holmer <[email-masked]> wrote:
> 
> On Fri, Nov 6, 2015 at 8:52 AM Michael Welzl <[email-masked] <mailto:[email-masked]>> wrote:
> 
> > On 06 Nov 2015, at 01:43, Randell Jesup <[email-masked] <mailto:[email-masked]>> wrote:
> >
> I also think the sender has useful information about the type of stream it's encoding. It knows if the stream is likely to produce an average bitrate which can match the target (e.g., regular video from a camera) or if it's likely to produce something that is very bursty and not likely to produce an average bitrate matching the target (e.g., screencasts). It's not obvious to me that the CC should behave in the same way for these two cases, and therefore it's also not obvious to me that the right CC result to feedback is a target bitrate.
> 
> Another thing that the receiver doesn't know, is if the sender decided to send the packets in some particular pattern to probe the link. It can try to infer the packet based on the send-time extension, but it's much easier to do at the sender where you actually know what you tried to do.
>  
> >
> That is true, but I think it will be questioned if the RTP architecture allows you to send an RTCP message containing CC results covering multiple RTP streams. And if that’s not allowed then you'd still have to feed back CC results for each stream separately and have the sender combine them...

If the congestion control RTCP feedback message is supposed to be independent of the congestion control algorithm, then it has to provide per-SSRC reports, since the congestion control might be operating on a per-SSRC basis, rather than on the aggregate.

-- 
Colin Perkins
https://csperkins.org/

---

From: John Leslie <[email-masked]>
Date: Mon, 9 Nov 2015 12:22:33 -0500

Stefan Holmer <[email-masked]> wrote:
>... 

   +1

   There are a number of tricks available here, the sender serving different
levels of time-sensitivity. Also, the sender has the best idea of RTT, and
may wish to hold a sending rate constant _only_ for the portion of the RTT
which is likely to hit a congested forwarder (for a particular packet) --
in order to optimize the reduction-to-sending-rate-when-the-packet-was-sent.

   (I am happy to have receiver-side experiments along with sender-side
experiments: I just have a strong prejudice about which will win. ;^)

---

From: "Xiaoqing Zhu (xiaoqzhu)" <[email-masked]>
Date: Mon, 9 Nov 2015 23:47:20 +0000

Thanks, Randell, for your comments regarding the bidirectional test case
results and also what performance metrics to show. Agree that it will be
good to have a side-by-side comparison between scenarios with congested
vs. uncongested feedback path. Will add such graphs in future.

Back to the more general discussions on sender-based vs. receiver-based
CC: the discussions here are helpful so we are aware of the advantages and
limitations of either approach.    To me, one additional advantage of the
sender is that it¹s right at the ³place of action², and may have better
knowledge on the limitations and current status of the live encoder (e.g.,
how fast the encoder is reacting to the ramp-up rate request). So, even if
the CC rate is recommended by a receiver, the sender will have the final
say on what rate to encode at and send.

That said, it¹s probably still possible to combine the best of both
worlds. The original GCC proposal has both a sender-based and a
receiver-based component, whereas their current design has moved to
sender-based. Maybe some good lessons to be learned there from that
³migration². 

Currently, since all three candidate schemes are sender-based, one obvious
first step is for us to figure out how often and how much feedback from
the receiver is needed as feedback to support this mode of operation, and
whether the amount of required overhead is within reason.

For the scenario with bidirectional calls (A<=>B), there should also be
the opportunity to piggy-back feedback for A->B along with payload for
B->A, which typically saves on feedback bandwidth. We may either want to
consider that optional now, or wait till after we have a good
understanding on the feedback needed for the basic case with
one-directional calls.

- Xiaoqing

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Nov 2015 08:25:07 +0000

Hi
First I think that there biggest reason to have congestion control on either sender "XOR" receiver side is that it removes some of the issues with synchronization between sender and receiver (as pointed out by Stefan). It is for instance quite likely that some users update their browsers on a regular basis, while others do it less often. This of course means that one need to make the feedback generic and as future proof as possible. draft-holmer-rmcat-transport-wide-cc-extensions is a very good start in that direction, one can always argue on the details and an AVT approved version will probably look different, but for experimental purposes I welcome the generic feedback draft and hopefully we will have it implemented in OpenWebRTC before the next IETF meeting.

Sender XOR receiver (a +1 on Stefan´s and John´s comment below)
It is no secret that I favor sender side rate adaptation, in fact I find it hard to understand how SCReAM would be implemented on the receiver side. 
To take one example, the stream prioritization in SCReAM is quite straightforward, and that is because of the sender side implementation, for instance in the cases where the throughput drops and the video coder rate reduction lags behind, the stream prioritization can prioritize a smaller audio stream and enable audio without (too noticeable) interruptions, even though video freezes.  There are also a few necessities with video such as FIR that can potentially bloat a network bottleneck, the sender side algorithm here has a better capability to decide when and how to send the packets with the objective to reach the best possible aggregate media quality.

/Ingemar

> -----Original Message-----
> From: John Leslie [mailto:[email-masked]]
> Sent: den 9 november 2015 18:23
> To: Stefan Holmer
> Cc: Randell Jesup; rmcat WG; Michael Welzl
> Subject: Re: [rmcat] How we should handle feedback, and where the
> congestion should run
> 
> > Another thing that the receiver doesn't know, is if the sender decided
> > to send the packets in some particular pattern to probe the link. It
> > can try to infer the packet based on the send-time extension, but it's
> > much easier to do at the sender where you actually know what you tried to
> do.
> 
>    +1
> 
> levels of time-sensitivity. Also, the sender has the best idea of RTT, and may
> wish to hold a sending rate constant _only_ for the portion of the RTT which
> is likely to hit a congested forwarder (for a particular packet) -- in order to
> optimize the reduction-to-sending-rate-when-the-packet-was-sent.
>

---

From: Michael Welzl <[email-masked]>
Date: Tue, 10 Nov 2015 09:35:06 +0100

> On 10 Nov 2015, at 09:25, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi

+1, and:

> It is no secret that I favor sender side rate adaptation,

+1

> in fact I find it hard to understand how SCReAM would be implemented on the receiver side. 

More generally, it's not only the network that dictates the right behavior - there is information about the data stream available on the sender side that may feed into the rate decision, and then splitting the logic between sender really gets awkward.

Cheers,
Michael

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Tue, 10 Nov 2015 15:33:59 +0000

+1 to what Xiaoqing said below.

In my discussion time at the microphone I mentioned that all the information needed in the feedback for a per-packet feedback reporting scheme (defined as information on every packet reception sent in one or more the reverse direction packets but at a rate much less than the send packet rate) is highly compressible.

If I assume for the purpose of this discussion that we won't use a transport-wide sequence number (because you can account for all sequence numbers IF you know all the SSRCs you want in the aggregate), then you need to compress the identifier for the equivalent information of 1: {sequence numbers + SSRC} mapping (to know gaps in the aggregate) and 2: receive timestamps for packets in the immediate past.

If I further assume an RTCP XR like reporting which has an overlap in packet reception data (i.e., some portion of the information in older packets were also included in one or more previous reports), I mentioned that one could do an experiment on how efficiently one could compress the data with a "trivial/simple" compressor. Perhaps a delta on sequence numbers, a delta on timestamps and some form of entropy or run-length encoding?

If we can come to agreement on the rough syntax of such information, I'd like to see a back-of-the-envelope calculation for how many bits (on average) the result of this compression would look like with reasonable assumptions on receive time granularity (~100 microseconds) and a reasonable number of SSRCs (for simulcast situations) in the aggregate.

Then we could re-run the experiment with a "transport wide sequence number" in each sent packet (we know exactly how much bandwidth this takes). Then the report need not have the SSRC mapping part of what I describe above (because we have sequence numbers on the aggregate) and the resulting information may compress better (I really don't have a good feeling for this - perhaps Stefan does).

If the result of this thought experiment is that the number of bits per RTCP XR feedback isn't too great  (i.e., the bandwidth penalty relative to the feedback each proposal has now isn't too great) ... then we can think of developing this per-packet feedback approach some more.

Thoughts? Anyone want to take a stab at: 1) the format, 2) the RTCP XR reporting interval (e.g., every 100ms), 3) the overlap in packet reporting (1/3 new packets, 2/3 old packets similar to the 1-out-of-3 RTCP loss assumption for DTMF?).

Michael Ramalho

-----Original Message-----
From: rmcat [mailto:[email-masked]] On Behalf Of Xiaoqing Zhu (xiaoqzhu)
Sent: Monday, November 09, 2015 6:47 PM
To: Randell Jesup
Cc: [email-masked]
Subject: Re: [rmcat] How we should handle feedback, and where the congestion should run

Thanks, Randell, for your comments regarding the bidirectional test case results and also what performance metrics to show. Agree that it will be good to have a side-by-side comparison between scenarios with congested vs. uncongested feedback path. Will add such graphs in future.

Back to the more general discussions on sender-based vs. receiver-based
CC: the discussions here are helpful so we are aware of the advantages and
limitations of either approach.    To me, one additional advantage of the
sender is that it¹s right at the ³place of action², and may have better knowledge on the limitations and current status of the live encoder (e.g., how fast the encoder is reacting to the ramp-up rate request). So, even if the CC rate is recommended by a receiver, the sender will have the final say on what rate to encode at and send.

That said, it¹s probably still possible to combine the best of both worlds. The original GCC proposal has both a sender-based and a receiver-based component, whereas their current design has moved to sender-based. Maybe some good lessons to be learned there from that ³migration². 

Currently, since all three candidate schemes are sender-based, one obvious first step is for us to figure out how often and how much feedback from the receiver is needed as feedback to support this mode of operation, and whether the amount of required overhead is within reason.

For the scenario with bidirectional calls (A<=>B), there should also be the opportunity to piggy-back feedback for A->B along with payload for
B->A, which typically saves on feedback bandwidth. We may either want to
consider that optional now, or wait till after we have a good understanding on the feedback needed for the basic case with one-directional calls.

- Xiaoqing

On 11/6/15, 11:43 PM, "rmcat on behalf of Randell Jesup"
<[email-masked] on behalf of [email-masked]> wrote:

>> In the eval-test-case draft, there is currently one test case 
>> dedicated for exploring impact of two-way traffic (Sec. 5.3.  
>> Congested Feedback Link with Bi-directional RMCAT flows). Some 
>> corresponding graphs can be found below (they may not reflect the 
>> most up-to-date algorithm performance).
>>
>> * 
>> NADA:http://www.ietf.org/proceedings/92/slides/slides-92-rmcat-4.pdf
>> (page #9 and #10)
>> * SCReAM:
>> 
>>http://www.ietf.org/proceedings/interim/2014/11/09/rmcat/slides/slides
>>-in
>>te
>I note NADA seems to be slower to respond to delay, though it also 
>seems to do much better against competing TCP flows (and likely the two 
>items are linked, which will make for some interesting tradeoffs and 
>tuning to do).
>
>> Would like to know whether you think that current design of the test 
>>case  has captured gist of the issue encountered by two-way calls, or 
>>any  suggestions on additional test scenarios in that regard?
>
>non-congested feedback paths is a useful comparison - much easier to 
>see the impact.  Feedback bandwidth would be useful, though that might 
>be fairly fixed right now (multiple of packet rate), so unless it's 
>Randell Jesup -- rjesup a t mozilla d o t com Please please please 
>don't email [email-masked]!  Way too much spam
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 10 Nov 2015 16:04:44 +0000

I’d guess you’ll get better compression reporting on a per-SSRC basis, since the timing of packets sent by an individual SSRC is more predictable than the timing of the aggregate. Then, combine reports into a single compound RTCP packet. 

Colin

> On 10 Nov 2015, at 15:33, Michael Ramalho (mramalho) <[email-masked]> wrote:
> 

-- 
Colin Perkins
https://csperkins.org/

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Sun, 15 Nov 2015 12:21:26 +0100

Just one comment to this realm:

In some application domains (e.g., signaling environment) TCP is running
bi-directional steady.
One thing to note is that the run of data with SACK  (SACK is feedback
path) actually gives additional robustness to drop
of feedback as data retransmission algorithms help recover cc feedback
information.

I don't know how the RTP feedback is designed and whether the above
comment is relevant in this context, but
thought that I would give it anyway.

BR, Karen

> Sent: 7. november 2015 05:34
> To: Randell Jesup <[email-masked]>
> Cc: rmcat WG <[email-masked]>
agree on the
> importance of two-way calls as use cases.
>
for
> exploring impact of two-way traffic (Sec. 5.3.  Congested Feedback Link
with
> Bi-directional RMCAT flows). Some corresponding graphs can be found
below
> (they may not reflect the most up-to-date algorithm performance).
>
case
> >> The big problem I see with receiver-side schemes is that you don't
> >>only need to standardize 1) the feedback message, you also need to
> >> 2) If feedback is missing for a while, maybe that's a good thing, as
> >>a result of feedback suppression?  So we just gradually and blindly
> >behavior can be simple).  I.e. the send will attempt to send at the
> >rate requested in a feedback message; will inform codecs of the new
> >rate as fast as possible; will optionally pace data entering the
> >network to not exceed either the congestion rate, or a separate pacing
rate
> supplied.
> >the sender may not be able to use an arbitrary time for calculating
> >rate (i.e. encoders may not be flexible on that - easier to do for
pacing).
> >
> >In terms of 1/2/3 above: if we do this, the receive-side algorithm
> >could configure the sender via these messages and/or negotiation. The
> >feedback messages can (optionally) have control information that tells
> >the sender what to do on missing feedback, and when.  For example, it
> >could have a feedback-timeout field, or timeout and
> >what-to-do-at-timeout fields.  We could also define a "are you still
> >there" RTCP that can be sent when we see feedback timeouts or are
> >getting close to them, asking the receiver to resend (and exactly how
> >to respond to those can be part of the receive-side CC algorithm).  It
> >could include updated RS/RR stats for any RTP flows, letting the
> >receiver-side update their idea of the return-path congestion.
> >
> >feedback (RTCP) will contend with.  This applies no matter which end
> >the algorithms run at.
> >
> >would needed if we decide to go that way; the above would be a start.
> >I do think it's *very* worthwhile defining a standard for feedback
> >(whichever end we end up with) such that new single-ended algorithms
> >can be trialed or deployed (and coincidentally provides a way for the
> >CC algorithm to probe to see if the other side supports something
> >loss-free especially where the bottleneck link is the local last-mile
> >or Wifi), while for us we typically have saturated links in both
> >directions while trying to keep delays low, and each packet of feedback
> >takes away from payload.
> >
> >feedback raw data and keep *all* decisions at the sender side. (Though
> >a sender-side standard feedback packet may still need rules on how to
> >reason alone, I'm not sure if RTCP-XR will work, even with
> >modifications or additions - but I haven't looked much at RTCP-XR other
> >than to note it has a LOT of stuff and isn't very thin.  But take that
> >with a grain of salt.
> >
> >--