Subject: Re: [rmcat] [R-C] Sender- vs. receiver-based, yet again...

From: Michael Welzl <[email-masked]>
Date: Mon, 24 Sep 2012 10:40:01 +0200

Hi,

Sorry, this answer is almost ridiculously late...
I compensate by being the first one to send a technical discussion message to the new list  :-)

On 21. sep. 2012, at 16:20, Stefan Holmer wrote:

> 
> On Wed, Sep 19, 2012 at 2:26 PM, Michael Welzl <[email-masked]> wrote:
> Hi,
> 
> Sorry for bringing this up yet again - but my thoughts just keep returning to the question of sender-vs.-receiver based congestion control, because it's a seemingly simple problem for which we should be able to agree on a clear answer?!
> 
> As most of you probably know by now, I'm critical of a receiver-based approach as in RRTCC. (In fact, as I will explain further below, I argue against a *combined* sender+receiver approach). Here is a plus/minus list as I see it:
> 
> 
> PLUS no. 1: it helps to reduce the amount of feedback. That's really the ONLY benefit.
> 
> It also has a small benefit in that it can more easily make use of new signals as they are introduced, without having to modify the feedback message, but I guess that argument is a bit hypothetical.

Yes, I'd say that's hypothetical  :-)   because at the end of the day, there's only so much you can measure... has a packet made it or not, and how long did it take. I argue for giving all of that information to the sender if we go for a sender-based approach.

> 
> MINUS no. 1: it requires the same mechanism on the sender and receiver side, which means that, if RTP becomes a "framework" for congestion control mechanisms, we must signal which mechanism is used to the other side. In contrast, we don't have to do this with TCP's (implemented but not standardized) pluggable congestion control. This is, to me, the biggest problem, as it requires code updates to happen on both sides for a new mechanism to be deployed.
> 
> MINUS no. 2: it could make combined congestion control between multiple flows via the FSE more complicated. Anyway it seems we'll need an FSE on the sender AND receiver side, but with receiver-based congestion control, we'll face the question: for two flows between the same two hosts, sharing the same bottleneck, should the receivers exchange information and coordinate their feedback, or should the senders exchange information and coordinate their behavior?  => when congestion control mechanisms are split across sender and receiver, the right answer to this question becomes a per-mechanism decision.
> 
> 
> Please amend this list if I'm missing something. Regarding the one PLUS above, here's a critical look at the benefits of reducing feedback **via receiver-side congestion control**:
> 
> - less feedback means a greater chance for the feedback to be dropped, which translates into a greater chance for undesirable sender behavior
> - we deal with interactive traffic, in which case a lot of the feedback could hopefully be piggybacked onto payload packets
> - in some cases there will be other parallel flows (e.g. the data channel in rtcweb) that already give us all the feedback that we need, this is another chance for feedback reduction
> - receiver-side-congestion-control-based feedback reduction as in RRTCC means to reduce the feedback whenever feedback isn't urgent for the sender. I'd argue that, in fact, we should reduce feedback only when the channel carrying it (the backwards channel) is congested, and that is a generic function, not bound to the sender-side congestion control behavior.
> 
> 
> So how urgent is it to reduce feedback, anyway? Some time ago, Bill Ver Steeg wrote:
> 
> ***
> 1- The upstream/return traffic is (generally) assumed to be a precious resource by the RTP community. This is partially due to multicast considerations (which we can mostly ignore here) and partially due to asynchronous networks. Note that the CableTV community has found it necessary to play games with TCP acks in Cable Modems because of upstream bandwidth limitations, even for unicast TCP traffic. This is an ugly hack that I will not defend here, but it is indicative of the problem space. The net-net is that we have to be somewhat conservative in our sending of acks, or we will end up in similar trouble.
> ***
> 
> As he states, multicast is out of scope. Plus, I think we should care more about technical reasons than what the RTP community considers to be a precious resource. As for the CableTV example, one can always find examples where people started doing useless things  :-)   but consider the following: many parallel TCP connections, each sending lots of feedback, work okay for maaaany end users that 1) surf the Internet or (bringing it closer to rtcweb thinking)  2) use P2P applications. And TCP doesn't even do any form of congestion control on ACKs (an RFC exists but noone uses it (so far)).
> 
> I'm not saying ACKs never cause problems. I'm saying it's probably rare that they do, and reducing their number WHEN they cause trouble is good enough (and, I would argue, a better decision).
> 
> 
> Let's take a step back and summarize the situation:
> 1) congestion-control-based-feedback-reduction has at least one significant MINUS (no. 1), probably two
> 2) the benefits of doing it seem minimal
> 3) not being based on congestion along the backwards path, it misses the point somewhat...
> 
> 
> As a result of the above, I would suggest to move all congestion control functionality to the sender side and keep the receiver side dumb, as in TCP, but amended with some generic (i.e. not mechanism-specific) feedback-reduction functions (ACK congestion control; piggybacking; omitting feedback when it is known (e.g. via a signal from the sender) that we get feedback from another flow).
> 
> Alternatively, we could still remove the MINUS no. 1 above by agreeing on a generic sender behavior. IIRC, in RRTCC, the sender is rather simple - it follows the rate dictated by the receiver and has a timeout behavior. If all congestion control mechanisms are installed on the receiver side only, with the same agreed generic sender behavior for everyone, we can also have pluggable congestion control without a need for signalling.
> 
> I agree that if we go for a receiver-based approach, the sender must be dumb. And the other way around if we go for a sender-based approach.

Great! Having one side dumb is the most important thing here IMO.

>  
> 
> It's just the idea of multiple proposals for congestion control involving newly described sender AND receiver behavior that strikes me as bad design.
> 
> Thoughts?
> 
> One benefit I see with having the intelligence at the sender is if you want to do something more clever when probing the channel, which would require coordination between the sender and the one who's supposed to come to some conclusion from the incoming packets. I think Bill Ver Steeg touched upon something along those lines:
> 
> "Basic senders will simply tell the receiver to gather the data every N seconds and send it every N seconds (plus when there are problems, as defined by the specification). More sophisticated senders could drive more complex relationships, asking the receiver to tabulate/send the data on demand in semi-real time. This would allow a sophisticated sender to send a varying data patterns and get the feedback they require, yet not require too much shared state between the sender and receiver. As long as the receiver is a simple state machine, I think we would be OK."

That method actually just becomes another entry in the list of feedback-reduction mechanisms that we already have.

Here's an updated summary:

1) I have argued that reducing feedback really isn't all that important.
2) Say, we want to do it (still a good thing, I think).
.. a) Then we can and should, first of all, do it based on congestion on the backward path.
.. b) Additionally, it might be possible to reduce feedback because it can be piggybacked onto payload (as most scenarios would have traffic flowing both ways).
.. c) Additionally, it might be possible to exploit feedback from other parallel flows (e.g., the data channel in rtcweb)
.. d) Additionally, we can do what is written above, let the sender tell the receiver how much feedback it needs.

All of this is possible with having the logic on the sender side only, and having a dumb receiver.

Cheers,
Michael

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 24 Sep 2012 14:06:16 +0200

On Mon, Sep 24, 2012 at 10:40 AM, Michael Welzl <[email-masked]> wrote:

>> Sorry for bringing this up yet again - but my thoughts just keep
>> returning to the question of sender-vs.-receiver based congestion control,
>> because it's a seemingly simple problem for which we should be able to
>> agree on a clear answer?!
>>
>> As most of you probably know by now, I'm critical of a receiver-based
>> approach as in RRTCC. (In fact, as I will explain further below, I argue
>> against a *combined* sender+receiver approach). Here is a plus/minus list
>> as I see it:
>>
>>
> It also has a small benefit in that it can more easily make use of new
> signals as they are introduced, without having to modify the feedback
> message, but I guess that argument is a bit hypothetical.
>
>
> Yes, I'd say that's hypothetical  :-)   because at the end of the day,
> there's only so much you can measure... has a packet made it or not, and
> how long did it take. I argue for giving all of that information to the
> sender if we go for a sender-based approach.
>
>
>
>
>> MINUS no. 1: it requires the same mechanism on the sender and receiver
>> side, which means that, if RTP becomes a "framework" for congestion control
>> mechanisms, we must signal which mechanism is used to the other side. In
>> contrast, we don't have to do this with TCP's (implemented but not
>> standardized) pluggable congestion control. This is, to me, the biggest
>> problem, as it requires code updates to happen on both sides for a new
>> mechanism to be deployed.
>>
>> MINUS no. 2: it could make combined congestion control between multiple
>> flows via the FSE more complicated. Anyway it seems we'll need an FSE on
>> the sender AND receiver side, but with receiver-based congestion control,
>> we'll face the question: for two flows between the same two hosts, sharing
>> the same bottleneck, should the receivers exchange information and
>> coordinate their feedback, or should the senders exchange information and
>> coordinate their behavior?  => when congestion control mechanisms are split
>> across sender and receiver, the right answer to this question becomes a
>> per-mechanism decision.
>>
>>
>> Please amend this list if I'm missing something. Regarding the one PLUS
>> above, here's a critical look at the benefits of reducing feedback **via
>> receiver-side congestion control**:
>>
>> - less feedback means a greater chance for the feedback to be dropped,
>> which translates into a greater chance for undesirable sender behavior
>> - we deal with interactive traffic, in which case a lot of the feedback
>> could hopefully be piggybacked onto payload packets
>> - in some cases there will be other parallel flows (e.g. the data channel
>> in rtcweb) that already give us all the feedback that we need, this is
>> another chance for feedback reduction
>> - receiver-side-congestion-control-based feedback reduction as in RRTCC
>> means to reduce the feedback whenever feedback isn't urgent for the sender.
>> I'd argue that, in fact, we should reduce feedback only when the channel
>> carrying it (the backwards channel) is congested, and that is a generic
>> function, not bound to the sender-side congestion control behavior.
>>
>>
>> 1- The upstream/return traffic is (generally) assumed to be a precious
>> resource by the RTP community. This is partially due to multicast
>> considerations (which we can mostly ignore here) and partially due to
>> asynchronous networks. Note that the CableTV community has found it
>> necessary to play games with TCP acks in Cable Modems because of upstream
>> bandwidth limitations, even for unicast TCP traffic. This is an ugly hack
>> that I will not defend here, but it is indicative of the problem space. The
>> net-net is that we have to be somewhat conservative in our sending of acks,
>> or we will end up in similar trouble.
>> ***
>>
>> As he states, multicast is out of scope. Plus, I think we should care
>> more about technical reasons than what the RTP community considers to be a
>> precious resource. As for the CableTV example, one can always find examples
>> where people started doing useless things  :-)   but consider the
>> following: many parallel TCP connections, each sending lots of feedback,
>> work okay for maaaany end users that 1) surf the Internet or (bringing it
>> closer to rtcweb thinking)  2) use P2P applications. And TCP doesn't even
>> do any form of congestion control on ACKs (an RFC exists but noone uses it
>> (so far)).
>>
>> I'm not saying ACKs never cause problems. I'm saying it's probably rare
>> that they do, and reducing their number WHEN they cause trouble is good
>> enough (and, I would argue, a better decision).
>>
>>
>> As a result of the above, I would suggest to move all congestion control
>> functionality to the sender side and keep the receiver side dumb, as in
>> TCP, but amended with some generic (i.e. not mechanism-specific)
>> feedback-reduction functions (ACK congestion control; piggybacking;
>> omitting feedback when it is known (e.g. via a signal from the sender) that
>> we get feedback from another flow).
>>
>> Alternatively, we could still remove the MINUS no. 1 above by agreeing on
>> a generic sender behavior. IIRC, in RRTCC, the sender is rather simple - it
>> follows the rate dictated by the receiver and has a timeout behavior. If
>> all congestion control mechanisms are installed on the receiver side only,
>> with the same agreed generic sender behavior for everyone, we can also have
>> pluggable congestion control without a need for signalling.
>>
>
> I agree that if we go for a receiver-based approach, the sender must be
> dumb. And the other way around if we go for a sender-based approach.
>
>
>> It's just the idea of multiple proposals for congestion control involving
>> newly described sender AND receiver behavior that strikes me as bad design.
>>
>> Thoughts?
>>
>
> One benefit I see with having the intelligence at the sender is if you
> want to do something more clever when probing the channel, which would
> require coordination between the sender and the one who's supposed to come
> to some conclusion from the incoming packets. I think Bill Ver Steeg
> touched upon something along those lines:
>
> "Basic senders will simply tell the receiver to gather the data every N
> seconds and send it every N seconds (plus when there are problems, as
> defined by the specification). More sophisticated senders could drive more
> complex relationships, asking the receiver to tabulate/send the data on
> demand in semi-real time. This would allow a sophisticated sender to send a
> varying data patterns and get the feedback they require, yet not require
> too much shared state between the sender and receiver. As long as the
> receiver is a simple state machine, I think we would be OK."
>
>
> That method actually just becomes another entry in the list of
> feedback-reduction mechanisms that we already have.
>
> .. b) Additionally, it might be possible to reduce feedback because it can
> be piggybacked onto payload (as most scenarios would have traffic flowing
> both ways).
> .. c) Additionally, it might be possible to exploit feedback from other
> parallel flows (e.g., the data channel in rtcweb)

Yes, however, if we have a feedback message per packet the sender will
always have all the feedback it needs. What I was thinking of was that if
the implementation wants to do something clever by sending packets in some
pattern, knowledge about that clever thing must be available both at the
sender and the receiver if we limit the feedback to, for instance, a
bitrate.

If we add a way for the sender to ask for more feedback then that would
allow for the sender to do this kind of cleverness also in the
receiver-based approach. And it doesn't have to be decided in advance what
that cleverness is.

---

From: Michael Welzl <[email-masked]>
Date: Mon, 24 Sep 2012 14:56:18 +0200

On 24. sep. 2012, at 14:06, Stefan Holmer wrote:

> Yes, however, if we have a feedback message per packet the sender will always have all the feedback it needs. What I was thinking of was that if the implementation wants to do something clever by sending packets in some pattern, knowledge about that clever thing must be available both at the sender and the receiver if we limit the feedback to, for instance, a bitrate.
> 
> If we add a way for the sender to ask for more feedback then that would allow for the sender to do this kind of cleverness also in the receiver-based approach. And it doesn't have to be decided in advance what that cleverness is.

Maybe I'm not fully getting your idea, but somehow this all sounds like making things unnecessarily complicated. Knowing whether the congestion control mechanism requires more feedback or not seems to me to miss the problem that we're trying to solve.

What's the point of reducing feedback when there's no congestion on the backward path anyway? The less feedback we send, the more risk of a message being lost due to whatever reason (congestion appearing, link noise, ...), and the more fragile the whole thing becomes. I would therefore argue for sending tons of feedback when we can, and only reduce when we must. And then, in the face of backwards congestion, we really MUST - the data congestion control mechanism (I say "data" to distinguish from ACK congestion control) will just have to cope with it. So it's probably not even useful to know how much feedback we ideally need...

Cheers,
Michael

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 24 Sep 2012 15:05:20 +0200

On Mon, Sep 24, 2012 at 2:56 PM, Michael Welzl <[email-masked]> wrote:

> On Mon, Sep 24, 2012 at 10:40 AM, Michael Welzl <[email-masked]>wrote:
>
>> Hi,
>>
>> On Wed, Sep 19, 2012 at 2:26 PM, Michael Welzl <[email-masked]>wrote:
>>
>>> Hi,
>>>
>>> - in some cases there will be other parallel flows (e.g. the data
>>> channel in rtcweb) that already give us all the feedback that we need, this
>>> is another chance for feedback reduction
>>> Alternatively, we could still remove the MINUS no. 1 above by agreeing
>>> on a generic sender behavior. IIRC, in RRTCC, the sender is rather simple -
>>> it follows the rate dictated by the receiver and has a timeout behavior. If
>>> It's just the idea of multiple proposals for congestion control
>>> involving newly described sender AND receiver behavior that strikes me as
>>> bad design.
>>>
>>> Thoughts?
>>>
>>
>> .. b) Additionally, it might be possible to reduce feedback because it
>> can be piggybacked onto payload (as most scenarios would have traffic
>> flowing both ways).
> Maybe I'm not fully getting your idea, but somehow this all sounds like
> making things unnecessarily complicated. Knowing whether the congestion
> control mechanism requires more feedback or not seems to me to miss the
> problem that we're trying to solve.
>

What I was trying to say is that doing something clever at the sender is
easy as long as the sender has per-packet feedback. It gets more involved
if you have limited feedback, but it would still be possible if you could
increase the amount of feedback for the packets sent in a clever way.

>
> What's the point of reducing feedback when there's no congestion on the
> backward path anyway? The less feedback we send, the more risk of a message
> being lost due to whatever reason (congestion appearing, link noise, ...),
> and the more fragile the whole thing becomes. I would therefore argue for
> sending tons of feedback when we can, and only reduce when we must. And
> then, in the face of backwards congestion, we really MUST - the data
> congestion control mechanism (I say "data" to distinguish from ACK
> congestion control) will just have to cope with it. So it's probably not
> even useful to know how much feedback we ideally need...
>
> Cheers,
> Michael
>
>

---

From: Michael Welzl <[email-masked]>
Date: Mon, 24 Sep 2012 15:15:20 +0200

On 24. sep. 2012, at 15:05, Stefan Holmer wrote:

> What I was trying to say is that doing something clever at the sender is easy as long as the sender has per-packet feedback. It gets more involved if you have limited feedback, but it would still be possible if you could increase the amount of feedback for the packets sent in a clever way.

Ah. Sure. Agreed

---

From: Randell Jesup <[email-masked]>
Date: Mon, 24 Sep 2012 11:56:00 -0400

Moving this sub-thread of discussion to the new list...

On 9/24/2012 10:46 AM, Harald Alvestrand wrote:
> On 09/19/2012 02:26 PM, Michael Welzl wrote:
>> Hi,
>>
>> returning to the question of sender-vs.-receiver based congestion 
>> control, because it's a seemingly simple problem for which we should 
>> be able to agree on a clear answer?!
>>
>> approach as in RRTCC. (In fact, as I will explain further below, I 
>> argue against a *combined* sender+receiver approach). Here is a 
>> plus/minus list as I see it:
>>
>>
>> MINUS no. 1: it requires the same mechanism on the sender and 
>> receiver side, which means that, if RTP becomes a "framework" for 
>> congestion control mechanisms, we must signal which mechanism is used 
>> to the other side. In contrast, we don't have to do this with TCP's 
>> (implemented but not standardized) pluggable congestion control. This 
>> is, to me, the biggest problem, as it requires code updates to happen 
>> on both sides for a new mechanism to be deployed.
> I don't see the logic here. The requirement is that both sides send 
> signals that are responded to appropriately by the other side. So the 
> two sides' mechanisms have to be compatible (in the RRTCC case, the 
> demand is that the sender responds to REMB or TMMBR messages - this is 
> hardly the "same mechanism" as the Kalman filter implemented on the 
> receiver side).
>
> In the TCP case, the feedback mechanism is the ack, which is needed 
> for reasons beyond congestion control. In the RTP case, there isn't 
> such a single, simple mechanism needed for other reasons.

Agreed, though Michael indicated that we can piggyback congestion info 
on reverse traffic.

This may be possible and I've thought about it.  Part of the problem is 
that if you want to signal data back quickly, you have to wait up to 
20ms and in some cases longer to piggyback (and on 
low-upstream-bandwidth links, the uplink transit time if you're 
piggybacking on a 1500-byte video packet might be 20+ms).  If you do 
piggyback, it has to be optional on a instance-by-instance basis 
(perhaps even estimating when the next chance will be, and the priority 
of the data - but that speaks against a dumb receiver).

TCP can "get away" with high packet count feedback because flows are 
typically largely unidirectional or alternating flows, or on 
well-connected devices, not edge nodes.  The PacketCable/etc issue with 
shared upstreams and slot allocation is an example of reacting to this - 
there even with the typically largely downstream flows, the upstream 
acks become a limiting factor.

>>
>> MINUS no. 2: it could make combined congestion control between 
>> multiple flows via the FSE more complicated. Anyway it seems we'll 
>> need an FSE on the sender AND receiver side, but with receiver-based 
>> congestion control, we'll face the question: for two flows between 
>> the same two hosts, sharing the same bottleneck, should the receivers 
>> exchange information and coordinate their feedback, or should the 
>> senders exchange information and coordinate their behavior?  => when 
>> congestion control mechanisms are split across sender and receiver, 
>> the right answer to this question becomes a per-mechanism decision.
> FSE?
> I don't see the logic here either. Either the two flows are under a 
> common controller on one side, they're under a common controlller on 
> the other side, they're not under any common controller, or they are 
> under common controllers on both sides. All four cases can happen with 
> both sender side and receiver side implementations of the BWE 
> computation.
>>
>>
>> Please amend this list if I'm missing something. Regarding the one 
>> PLUS above, here's a critical look at the benefits of reducing 
>> feedback **via receiver-side congestion control**:
>>
>> - less feedback means a greater chance for the feedback to be 
>> dropped, which translates into a greater chance for undesirable 
>> sender behavior
> Stickler: Amount of feedback has little correlation with packet drop. 
> If feedback is lost due to congestion, more feedback will lead to more 
> packet drops. What is true is that when feedback occurs more rarely, 
> losing one feedback packet will lead to a greater gap between feedback 
> packets.

I'd say less feedback means the odds that a drop will hit a feedback 
packet are *lower* (as routers typically either tail-drop or 
random-drop, but in either case in a static evaluation a 
lower-bandwidth/packet flow (i.e. the feedback packets) will get hit 
less often), but as Harald says the impact is higher.  And in a dynamic 
evaluation this may shift some.

>> - we deal with interactive traffic, in which case a lot of the 
>> feedback could hopefully be piggybacked onto payload packets

See above.  This may not be that useful in practice, or may degrade 
congestion response time.

>> channel in rtcweb) that already give us all the feedback that we 
>> need, this is another chance for feedback reduction

Do not count on the data channel being active or substituting for 
media-channel feedback even if active.  It may provide another source of 
feedback data.

>> - receiver-side-congestion-control-based feedback reduction as in 
>> RRTCC means to reduce the feedback whenever feedback isn't urgent for 
>> the sender. I'd argue that, in fact, we should reduce feedback only 
>> when the channel carrying it (the backwards channel) is congested, 
>> and that is a generic function, not bound to the sender-side 
>> congestion control behavior.

Personally, I assume both channels are always congested, or will be soon 
(if we're doing our job right and aren't on a huge link), so bandwidth 
reduction for feedback always allows more bandwidth for useful data.  In 
fact, this is one of the better arguments for piggybacking - less 
bandwidth required to provide the same bandwidth, but at the cost of the 
feedback possibly being delayed (and in some cases perhaps more likely 
to be impacted if the drops are packet-size aware, not just 
packet-aware, or if they get piggybacked on a packet with a lower DSCP 
marking).

>> 1- The upstream/return traffic is (generally) assumed to be a 
>> precious resource by the RTP community. This is partially due to 
>> multicast considerations (which we can mostly ignore here) and 
>> partially due to asynchronous networks. Note that the CableTV 
>> community has found it necessary to play games with TCP acks in Cable 
>> Modems because of upstream bandwidth limitations, even for unicast 
>> TCP traffic. This is an ugly hack that I will not defend here, but it 
>> is indicative of the problem space. The net-net is that we have to be 
>> somewhat conservative in our sending of acks, or we will end up in 
>> similar trouble.
>> ***
> I asusme "asynchronous" is "asymmetric" above?
>>
>> more about technical reasons than what the RTP community considers to 
>> be a precious resource. As for the CableTV example, one can always 
>> find examples where people started doing useless things  :-)   but 
>> consider the following: many parallel TCP connections, each sending 
>> lots of feedback, work okay for maaaany end users that 1) surf the 
>> Internet or (bringing it closer to rtcweb thinking)  2) use P2P 
>> applications. And TCP doesn't even do any form of congestion control 
>> on ACKs (an RFC exists but noone uses it (so far)).
>>
>> I'm not saying ACKs never cause problems. I'm saying it's probably 
>> rare that they do, and reducing their number WHEN they cause trouble 
>> is good enough (and, I would argue, a better decision).
>>

And my position is that any *good* two-way communication will run 
near-but-not-quite-over the bandwidth limit as much of the time as 
possible, so ACKs always have the potential to cause problems (and 
always will force you a little further down the quality curve).

> Both are debatable.
> What that bullet was meant to say totally escaped me. RRTCC is all 
> about congestion on the forward path, not the backwards path.

and in addition, per above, one should assume back-path congestion anyways.

>>
>> As a result of the above, I would suggest to move all congestion 
>> control functionality to the sender side and keep the receiver side 
>> dumb, as in TCP, but amended with some generic (i.e. not 
>> mechanism-specific) feedback-reduction functions (ACK congestion 
>> control; piggybacking; omitting feedback when it is known (e.g. via a 
>> signal from the sender) that we get feedback from another flow).
>>
>> Alternatively, we could still remove the MINUS no. 1 above by 
>> agreeing on a generic sender behavior. IIRC, in RRTCC, the sender is 
>> rather simple - it follows the rate dictated by the receiver and has 
>> a timeout behavior. If all congestion control mechanisms are 
>> installed on the receiver side only, with the same agreed generic 
>> sender behavior for everyone, we can also have pluggable congestion 
>> control without a need for signalling.

Correct.

>> involving newly described sender AND receiver behavior that strikes 
>> me as bad design.

I think if the sender (or receiver) have a simple design that can be 
interacted with by the (more complex) main algorithm would allow for 
improvement without requiring negotiation (and allow for asymmetric 
behaviors by different implementations/revisions or even asymmetry due 
to local network knowledge (wifi, etc).

>>
>> Thoughts?
> As you can see, I tend to disagree with the validity of your arguments.
> You may still be right, but you're not convincing me with this set of 
> arguments.
> (Another conversation I had last week was with one guy who argued that 
> we should use a traceroute-like functionality to directly probe queues 
> near the sender, rather than depending on whole-RTT signalling - that 
> kind of mechanism would rely on packets that didn't even reach the 
> receiver. Now, if we explore *that*, it's a real argument for putting 
> control at the sender....)

Such probing may be problematic... if you can determine local and nearby 
network configuration, that might be useful.  In my old algorithm, I 
used a heuristic of establishing max-known-good-bandwidth early in the 
call, and then being very cautious at "dipping my toes over" the (likely 
upstream) limit. (If it did remain delay-stable for long enough after 
going over, I'd update the limit.) Obviously this heuristic  is more 
useful in leaf-node cases, but this provided some of what I think that 
person may have been thinking about.

-- 
Randell Jesup
[email-masked]

---

From: Michael Welzl <[email-masked]>
Date: Wed, 26 Sep 2012 13:28:46 +0200

Hi,

(new list only)

On Sep 24, 2012, at 5:56 PM, Randell Jesup wrote:

>>> control, because it's a seemingly simple problem for which we  
>>> should be able to agree on a clear answer?!
>>>
>>> As most of you probably know by now, I'm critical of a receiver- 
>>> based approach as in RRTCC. (In fact, as I will explain further  
>>> below, I argue against a *combined* sender+receiver approach).  
>>> Here is a plus/minus list as I see it:
>>>
>>>
>>> congestion control mechanisms, we must signal which mechanism is  
>>> used to the other side. In contrast, we don't have to do this with  
>>> TCP's (implemented but not standardized) pluggable congestion  
>>> control. This is, to me, the biggest problem, as it requires code  
>>> updates to happen on both sides for a new mechanism to be deployed.
>> signals that are responded to appropriately by the other side. So  
>> the two sides' mechanisms have to be compatible (in the RRTCC case,  
>> the demand is that the sender responds to REMB or TMMBR messages -  
>> this is hardly the "same mechanism" as the Kalman filter  
>> implemented on the receiver side).
>>

What are these other reasons? I see why TCP's ACKs wouldn't do, but I  
don't see why a similarly simple generic feedback mechanism wouldn't.
(sure, that won't work if we have RRTCC with a Kalman filter vs. ABC  
with a CDE filter vs. ....   but my argument is that this is bad  
design for exactly that reason)

> Agreed, though Michael indicated that we can piggyback congestion  
> info on reverse traffic.
>
> This may be possible and I've thought about it.  Part of the problem  
> is that if you want to signal data back quickly, you have to wait up  
> to 20ms and in some cases longer to piggyback (and on low-upstream- 
> bandwidth links, the uplink transit time if you're piggybacking on a  
> 1500-byte video packet might be 20+ms).  If you do piggyback, it has  
> to be optional on a instance-by-instance basis (perhaps even  
> estimating when the next chance will be, and the priority of the  
> data - but that speaks against a dumb receiver).

Ok, I agree with that.

> typically largely unidirectional or alternating flows, or on well- 
> connected devices, not edge nodes.  The PacketCable/etc issue with  
> shared upstreams and slot allocation is an example of reacting to  
> this - there even with the typically largely downstream flows, the  
> upstream acks become a limiting factor.

I can see that, too.

>>> need an FSE on the sender AND receiver side, but with receiver- 
>>> based congestion control, we'll face the question: for two flows  
>>> between the same two hosts, sharing the same bottleneck, should  
>>> the receivers exchange information and coordinate their feedback,  
>>> or should the senders exchange information and coordinate their  
>>> behavior?  => when congestion control mechanisms are split across  
>>> sender and receiver, the right answer to this question becomes a  
>>> per-mechanism decision.
>> FSE?

Sorry - I meant Flow State Exchange, my idea for how combined  
congestion control should be done, as described here:
http://www.alvestrand.no/pipermail/rtp-congestion/2012-July/000436.html
It doesn't matter. For what we're discussing here, it's fine to think  
of a single common controller.

>> common controller on one side, they're under a common controlller  
>> on the other side, they're not under any common controller, or they  
>> are under common controllers on both sides. All four cases can  
>> happen with both sender side and receiver side implementations of  
>> the BWE computation.

So, I think we had some agreement already that there need to be common  
controllers on both sides. ( On the sender side, N out of M flows may  
share a bottleneck, and on the receiver side, X out of Y flows may  
share a bottleneck. )
The common control has to interact with, as you say, the BWE  
computation, which one mechanism could do on the sender side and  
another could do on the receiver side, and hence I said it's mechanism- 
specific. I just think we should agree to have it on one side only -  
ok with me if it's the receiver, dealing with a relatively dumb  
generic sender.

>> Stickler: Amount of feedback has little correlation with packet  
>> drop. If feedback is lost due to congestion, more feedback will  
>> lead to more packet drops. What is true is that when feedback  
>> occurs more rarely, losing one feedback packet will lead to a  
>> greater gap between feedback packets.
>
> packet are *lower* (as routers typically either tail-drop or random- 
> drop, but in either case in a static evaluation a lower-bandwidth/ 
> packet flow (i.e. the feedback packets) will get hit less often),  
> but as Harald says the impact is higher.  And in a dynamic  

Sorry sorry! I was thinking TCP, where there is some redundancy in  
ACKs. Indeed I meant the same as what Harald wrote - IF something is  
lost, the impact is higher. To correct my own phrasing, I meant: less  
feedback means a greater chances that some important piece of  
information never reaches the sender.

> congestion response time.

Ok, but as you say it depends. I think we should allow that, optionally.

> media-channel feedback even if active.  It may provide another  
> source of feedback data.

Sure, that's all I meant. It might be there, and might then help  
reduce the feedback further.

>>> RRTCC means to reduce the feedback whenever feedback isn't urgent  
>>> for the sender. I'd argue that, in fact, we should reduce feedback  
>>> only when the channel carrying it (the backwards channel) is  
>>> congested, and that is a generic function, not bound to the sender- 
>>> side congestion control behavior.
>
> Personally, I assume both channels are always congested, or will be  
> soon (if we're doing our job right and aren't on a huge link), so  
> bandwidth reduction for feedback always allows more bandwidth for  
> useful data.  In fact, this is one of the better arguments for  
> piggybacking - less bandwidth required to provide the same  
> bandwidth, but at the cost of the feedback possibly being delayed  
> (and in some cases perhaps more likely to be impacted if the drops  
> are packet-size aware, not just packet-aware, or if they get  
> piggybacked on a packet with a lower DSCP marking).

Ok, I think you got me convinced. The way I see it, there's a choice  
between:
1) sending a lot of feedback, and only reducing it when there is  
backwards congestion, by doing congestion control on the backwards  
channel like ACK-CC
2) simply always sending as little feedback as possible, based on the  
(forward) congestion control mechanism's requirements

If we think of e.g. a videoconference, indeed congestion from all the  
feedback on the uplink is quite realistic. 1) can then become rather  
complicated... you may get congestion on your uplink, and it might be  
necessary to reduce the rate of your data stream, or it might suffice  
to reduce it on some of the feedback streams, but which? And then we  
have to apply priorities between feedback packets and your payload...

2) seems simpler, but it has the problem of making the mechanism more  
fragile, and perhaps unnecessarily so (in a situation where it might  
be fine to send much more feedback). I say "unnecessarily" because I'm  
not convinced about "both channels are always congested, or will be  
soon" because codecs may not be greedy, and streams such as VoIP may  
simply not generate enough traffic for that.

Anyway, putting these pro's and con's together, yes, I think I'm now  
convinced that receiver-side congestion control is the better  
solution, mainly because it makes things easier, and the disadvantages  
aren't huge. I still think that we should consider making things less  
fragile by sending more feedback only when there is very clearly no  
backwards congestion, which could be done by backing off immediately  
at the slightest indication of increasing delay on the backward path  
(up to the point of what the minimum rate is, as determined by the  
receiver). But this could be an optional later add-on to a receiver- 
side mechanism such as RRTCC.

>>> community has found it necessary to play games with TCP acks in  
>>> Cable Modems because of upstream bandwidth limitations, even for  
>>> unicast TCP traffic. This is an ugly hack that I will not defend  
>>> here, but it is indicative of the problem space. The net-net is  
>>> that we have to be somewhat conservative in our sending of acks,  

Yes, I assume the same.

>>> As he states, multicast is out of scope. Plus, I think we should  
>>> care more about technical reasons than what the RTP community  
>>> considers to be a precious resource. As for the CableTV example,  
>>> one can always find examples where people started doing useless  
>>> things  :-)   but consider the following: many parallel TCP  
>>> connections, each sending lots of feedback, work okay for maaaany  
>>> end users that 1) surf the Internet or (bringing it closer to  
>>> rtcweb thinking)  2) use P2P applications. And TCP doesn't even do  
>>> any form of congestion control on ACKs (an RFC exists but noone  
>>> uses it (so far)).
>>>
>>> rare that they do, and reducing their number WHEN they cause  
>>> trouble is good enough (and, I would argue, a better decision).
>>>
>

See above: I don't buy that. You can have very good VoIP traffic never  
doing that.
But it doesn't matter, as I did buy your argument about the TCP  
scenario usually being different. Indeed, if we e.g. think of a video  
conference, there may be tons of feedback packets from multiple  
streams on the uplink, while we desperately want to get our video  
across that same link too.

>>> significant MINUS (no. 1), probably two

I agree they are - and indeed, as I wrote above, I see the pro/con  
weights differently now.

> and in addition, per above, one should assume back-path congestion  

Yes, and I don't see what was not understandable about that bullet. If  
you're not concerned about congestion on the backwards path, why  
reduce the amount of feedback? What else is your concern - processing?

>>> control functionality to the sender side and keep the receiver  
>>> side dumb, as in TCP, but amended with some generic (i.e. not  
>>> control; piggybacking; omitting feedback when it is known (e.g.  
>>> via a signal from the sender) that we get feedback from another  
>>> flow).
>>>
>>> agreeing on a generic sender behavior. IIRC, in RRTCC, the sender  
>>> is rather simple - it follows the rate dictated by the receiver  
>>> and has a timeout behavior. If all congestion control mechanisms  
>>> are installed on the receiver side only, with the same agreed  
>>> generic sender behavior for everyone, we can also have pluggable  
>>> congestion control without a need for signalling.
>
> Correct.

Stefan said that too. So I think we already have some agreement here  
on having a relatively dumb generic sender behavior, and I'm now  
joining you people in thinking that the per-mechanism intelligence is  
better done on the receiver side.

>>> involving newly described sender AND receiver behavior that  
>>> strikes me as bad design.
>
> behaviors by different implementations/revisions or even asymmetry  
> due to local network knowledge (wifi, etc).

Yes. The sender, it seems.

>> You may still be right, but you're not convincing me with this set  
>> of arguments.
>> (Another conversation I had last week was with one guy who argued  
>> that we should use a traceroute-like functionality to directly  
>> probe queues near the sender, rather than depending on whole-RTT  
>> signalling - that kind of mechanism would rely on packets that  
>> didn't even reach the receiver. Now, if we explore *that*, it's a  
>> real argument for putting control at the sender....)
>
> Such probing may be problematic... if you can determine local and  
> nearby network configuration, that might be useful.  In my old  
> algorithm, I used a heuristic of establishing max-known-good- 
> bandwidth early in the call, and then being very cautious at  
> "dipping my toes over" the (likely upstream) limit. (If it did  
> remain delay-stable for long enough after going over, I'd update the  
> limit.) Obviously this heuristic  is more useful in leaf-node cases,  
> but this provided some of what I think that person may have been  
> thinking about.

I like the idea, and I guess the guy was John Leslie, as this idea was  
a part of his CC-workshop paper:
http://www.tschofenig.priv.at/cc-workshop/irtf_iab-ccirtcpaper14.pdf
...but this strikes me as VERY researchy - I'd definitely want to see  
some solid data before believing that this can work (and hence basing  
rmcat's design decisions on it). There are some unknowns here,  
including the possible impact of strange behavior of the equipment  
sending you an ICMP message back. I'd first have to believe that one  
can trust the timing of these ICMP enough to make any sense of it for  
a congestion control mechanism. I'd love to see some experimental data  
presented in ICCRG!

Cheers,
Michael