Subject: [rmcat] Multihop congestion control - two words of caution

From: Harald Alvestrand <[email-masked]>
Date: Thu, 18 Oct 2012 15:25:56 +0200

I got into a discussion in Vancouver with someone about multihop 
congestion control, why it was hard, and why it should be ruled out of 
scope - and promised to say something about it later.
But at this point, I no longer remember who it was with, so I'm posting 
it here instead.

One interesting thing about congestion control is a tendency to 
oscillation. If there is a feedback signal, that's always based on past 
history; the decisions based on that signal will affect traffic in the 
future.

So there's always a delay between the events we take action on and the 
action we take.
If, for instance, the delay is 1 second (the argument's the same for 1 
millisecond, but a second is easier to think about), the feedback signal 
is "congestion", and the decision is "reduce traffic" .. the signal an 
instant after the decision is made is based on the *previous* second's 
worth of traffic, and will still say "congestion"; if we take action on 
that, we'll reduce traffic even more - perhaps too much.

Once we detect "no congestion", the same thing happens in reverse. If 
the delays and the size of action is just right, we can get into 
oscillation: jumping between inefficient use of bandwidth (sending too 
little) and serious congestion (sending too much).

The mitigation against this kind of behaviour is that one designs the 
strength of the action to be commensurate with the delay so as to avoid 
damping; in TCP, this is obtained by scaling reaction time by the RTT; 
we don't react faster than 1 unit of reaction per RTT.

Now consider what happens if we have two hops after each other, possibly 
with different delays, each running their own congestion control 
algorithm - and a linkage that tries to feed congestion control 
information from the second hop back to the congestion control algorithm 
running on the first hop.

The overall system will have a delay - but that delay won't be visible 
to the congestion control algorithm on the first hop, so the reaction 
time and size can't be scaled appropriately.

The result is that the overall system is very likely to exhibit 
oscillatory behaviour - simply because the time constants we need to 
observe for a proper design are invisible to the observer.

We can imagine ways of working around this limitation - but for the 
purposes of RMCAT, I think we should just stick with two principles:

- We don't design for the two-hop case.
- We advise designers of two-hop systems (like relay nodes) that if they 
want to feed capacity information from a second hop back to the first 
hop, they should reflect those changes SLOWLY.

Experts, did this sound reasonable?

                      Harald

---

From: "Roy, Radhika R CIV (US)" <[email-masked]>
Date: Thu, 18 Oct 2012 14:38:36 +0000

Hi, Herald:

In general, your observation is OK. There are a couple of things are as
follows:

1. We have to look into aggregate statistical model of traffic congestions.
2. Contributions to congestions are different for different types of
traffic.
3. Behavior of each kind of traffic is also different. 
4. Performance requirements of each kind of traffic are also different.
4. High bandwidth-intensive video vs. low bandwidth audio vs. low-high
bandwidth with very busty data with different performance requirements over
the same IP network will need to have very different requirements for taking
actions when congestions are detected.

The point that I am making is this: IETF is being the technical organization
needs to try to solve, if not complete solutions for congestions, but at
least to look into in addressing the problems especially when video is
becoming (or has become) the most dominant traffic. As we gain more and more
insights of this complex problem, we will be developing the technical
standards accordingly as time goes by.

BR/Radhika

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Harald Alvestrand
Sent: Thursday, October 18, 2012 9:26 AM
To: [email-masked]
Subject: [rmcat] Multihop congestion control - two words of caution

I got into a discussion in Vancouver with someone about multihop congestion
control, why it was hard, and why it should be ruled out of scope - and
promised to say something about it later.
But at this point, I no longer remember who it was with, so I'm posting it
here instead.

One interesting thing about congestion control is a tendency to oscillation.
If there is a feedback signal, that's always based on past history; the
decisions based on that signal will affect traffic in the future.

So there's always a delay between the events we take action on and the
action we take.
If, for instance, the delay is 1 second (the argument's the same for 1
millisecond, but a second is easier to think about), the feedback signal is
"congestion", and the decision is "reduce traffic" .. the signal an instant
after the decision is made is based on the *previous* second's worth of
traffic, and will still say "congestion"; if we take action on that, we'll
reduce traffic even more - perhaps too much.

Once we detect "no congestion", the same thing happens in reverse. If the
delays and the size of action is just right, we can get into
oscillation: jumping between inefficient use of bandwidth (sending too
little) and serious congestion (sending too much).

The mitigation against this kind of behaviour is that one designs the
strength of the action to be commensurate with the delay so as to avoid
damping; in TCP, this is obtained by scaling reaction time by the RTT; we
don't react faster than 1 unit of reaction per RTT.

Now consider what happens if we have two hops after each other, possibly
with different delays, each running their own congestion control algorithm -
and a linkage that tries to feed congestion control information from the
second hop back to the congestion control algorithm running on the first
hop.

The overall system will have a delay - but that delay won't be visible to
the congestion control algorithm on the first hop, so the reaction time and
size can't be scaled appropriately.

The result is that the overall system is very likely to exhibit oscillatory
behaviour - simply because the time constants we need to observe for a
proper design are invisible to the observer.

We can imagine ways of working around this limitation - but for the purposes
of RMCAT, I think we should just stick with two principles:

- We don't design for the two-hop case.
- We advise designers of two-hop systems (like relay nodes) that if they
want to feed capacity information from a second hop back to the first hop,
they should reflect those changes SLOWLY.

                      Harald

---

From: Matthew Ford <[email-masked]>
Date: Thu, 18 Oct 2012 15:56:05 +0100

> - We don't design for the two-hop case.

For me at least, visualisation helps: http://www.youtube.com/watch?v=QXf95_EKS6E

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 18 Oct 2012 17:58:44 +0200

Hi Harald,

> running on the first hop.

Are we talking of end-to-end congestion control here? Then if we have 
congestion feedback from different nodes on one path, this feedback signal 
will have the same delay. And usually there is only one bottleneck at one 
point of time as there is always on link with the smallest available 
bandwidth. Is this the scenario your are talking about or something else?

Mirja

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: Michael Welzl <[email-masked]>
Date: Fri, 19 Oct 2012 09:52:03 +0200

Hi all,

I feel in a bit of a dilemma answering Harald's email, because this requires that I call myself an "expert"... yet I'm tempted to answer... tsk!   :-)     Anyway, I'll try to reply the emails from Harald, Radhika and Mirja in one go here, in the hope that I can clarify things a little:

Harald: all of what you have written sounds 100% reasonable to me.

Radhika: your answer appears (to me) to miss the point, and maybe (sorry if I'm wrong!) it's because you misunderstood the scenario? So maybe my answer to Mirja below helps.

Mirja: RTP/RTCP can let end systems take the role of an intermediate node, to take care of data stream specific things before traffic reaches the receiver (synchronizing, mixing, ...). Essentially, this is an overlay network. I think Harald refers to such a case with multiple RTP end systems, i.e. a path like this:

Node1 [ RTP1 sender ] => Node 2 [RTP1 receiver + RTP2 sender] => Node 3 [RTP2 receiver]

...and his description of problems goes into the behavior of node 2 (which is, in principle, quite similar to the behavior of some PEPs (RFC3135), I guess, e.g. TCP splitters. The decision at the BOF was not *yet* to define what is needed to make Node 2 work correctly, which is what we referred to as multi-hop congestion control. The exact phrasing in the charter is: "implications on multi-hop connections will be considered at a later stage". The reason for writing this is that Magnus Westerlund (quite correctly, I think) stated that defining this will at some point be needed.

I hope my interpretation of Harald's scenario is correct, and I hope that this was helpful!

Cheers,
Michael

---

From: "Roy, Radhika R CIV (US)" <[email-masked]>
Date: Fri, 19 Oct 2012 14:16:19 +0000

Hi, Michael:

Yes, in one way, I did misunderstood Harald's scenarios. The key point that
I have been pointing out that we have not got any mathematical models to
quantify the situations of congestions for multimedia traffic with competing
goals of bandwidth and performance requirements over UDP transport.

In absence of the above, we are trying something that is based on some
heuristic things hoping to work. Well, I am not against this approach. All I
am trying to point to the long-term approach in addressing the traffic
congestions problems.

BR/Radhika

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Michael Welzl
Sent: Friday, October 19, 2012 3:52 AM
To: Mirja Kuehlewind
Cc: [email-masked]; Harald Alvestrand
Subject: Re: [rmcat] Multihop congestion control - two words of caution

...

Radhika: your answer appears (to me) to miss the point, and maybe (sorry if
I'm wrong!) it's because you misunderstood the scenario? So maybe my answer
to Mirja below helps.

...

I hope my interpretation of Harald's scenario is correct, and I hope that
this was helpful!

Cheers,
Michael

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 22 Oct 2012 18:55:33 +0000

Do we consider a TURN server or other UDP relay to be multi-hop? By relay, I mean something which never alters or processes the RTP/RTCP at all. It is certainly another hop from the network and congestion perspectives, but not the RTP perspective.

The multi-hop case is certainly more complex, but complexity alone should not determine whether we tackle it or not. The importance of the problem and the need for a solution should be considered along with complexity. I suspect many major deployments of RMCAT will have multi-hop cases, and they may perhaps even be the dominant use cases. If we punt based on complexity, there will likely be wildly different behavior across these cases. (Even wilder than double pendulums...)

Mo

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Welzl
Sent: Friday, October 19, 2012 3:52 AM
To: Mirja Kuehlewind
Cc: [email-masked]; Harald Alvestrand
Subject: Re: [rmcat] Multihop congestion control - two words of caution

Hi all,

Cheers,
Michael

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 23 Oct 2012 13:16:29 +0200

> The multi-hop case is certainly more complex, but complexity alone should not determine whether we tackle it or not. The importance of the problem and the need for a solution should be considered a…

I don't think of a link via a TURN server as multihop; the TURN server 
should have the same characteristics as a switch or router in the path 
(lose packets if congested, but don't attempt to shape the stream).

I'd like to see guidelines (such as "go slower than X when you're a 
destination changing the desirable bandwidth based on something other 
than source/destination congestion") that avoid the double-pendulum 
style chaotic behaviour. I don't want to start off trying to write a 
controller for efficient management of double pendulums.

---

From: Michael Welzl <[email-masked]>
Date: Tue, 23 Oct 2012 13:23:30 +0200

On 23. okt. 2012, at 13:16, Harald Alvestrand wrote:

> I don't think of a link via a TURN server as multihop; the TURN server should have the same characteristics as a switch or router in the path (lose packets if congested, but don't attempt to shape the stream).
> 
> I'd like to see guidelines (such as "go slower than X when you're a destination changing the desirable bandwidth based on something other than source/destination congestion") that avoid the double-pendulum style chaotic behaviour. I don't want to start off trying to write a controller for efficient management of double pendulums.

Things don't have to get THAT bad... after all, TCP splitters do pretty much the same job that an intermediate hop in our setup would be doing, probably without getting into the fun of a double pendulum. Now, that's always easy to say, with little proof out there... but we could dig through the early literature on PEPs, I suppose, for a start...

Cheers,
Michael

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 23 Oct 2012 13:51:31 +0200

Hi Micheal,

thanks for the explanation. I believe I understood the scenario now but I'm 
still not sure about the problem. In TCP you have separate congestion control 
on each hop. If the second hop is slower than the first one, you also have 
flow control to not overload the receiver (by announcing a certain receive 
window). Is there a benefit in having a common congestion control over both 
hops?

Mirja

On Friday 19 October 2012 09:52:03 Michael Welzl wrote:
> Hi all,
>
> I feel in a bit of a dilemma answering Harald's email, because this
> requires that I call myself an "expert"... yet I'm tempted to answer...
> tsk!   :-)     Anyway, I'll try to reply the emails from Harald, Radhika
> and Mirja in one go here, in the hope that I can clarify things a little:
>
> Mirja: RTP/RTCP can let end systems take the role of an intermediate node,
> to take care of data stream specific things before traffic reaches the
> receiver (synchronizing, mixing, ...). Essentially, this is an overlay
> network. I think Harald refers to such a case with multiple RTP end
> systems, i.e. a path like this:
>
> ...and his description of problems goes into the behavior of node 2 (which
> is, in principle, quite similar to the behavior of some PEPs (RFC3135), I
> guess, e.g. TCP splitters. The decision at the BOF was not *yet* to define
> what is needed to make Node 2 work correctly, which is what we referred to
> as multi-hop congestion control. The exact phrasing in the charter is:
> "implications on multi-hop connections will be considered at a later
> stage". The reason for writing this is that Magnus Westerlund (quite
> correctly, I think) stated that defining this will at some point be needed.
>
> > congestion feedback from different nodes on one path, this feedback
> > signal will have the same delay. And usually there is only one bottleneck
> > at one point of time as there is always on link with the smallest
> > available bandwidth. Is this the scenario your are talking about or
> > something else?
> >
> > Mirja
> >

tel: +49(0)711/685-67973

---

From: Michael Welzl <[email-masked]>
Date: Tue, 23 Oct 2012 14:22:48 +0200

Hi,

> flow control to not overload the receiver (by announcing a certain receive 

Right;

> window). Is there a benefit in having a common congestion control over both 

I really don't know. Maybe? Not sure if someone suggested that?
Anyway, all I was trying to explain is that this is the scenario that, according to the charter, we're not looking at (and we keep it for the future).

This being said, personally, I'm interested in it and don't have a problem with discussions  :)   I just wanted to explain/clarify.

Cheers,
Michael