Subject: [iccrg] BBR and other congestion control mechanisms

From: Michael Welzl <[email-masked]>
Date: Wed, 16 Nov 2016 17:14:41 +0900

Dear all,

Thanks very much to the authors of BBR for an interesting presentation yesterday!
(see https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-01.pdf if you missed it).

I’d be interested to know how BBR compares to some other TCP variants. For example, TIMELY:
http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p537.pdf

Any idea, anyone?

( I tried, honestly, I did - but I just can’t manage to hold back a tiny bit of sarcasm here:  ...is BBR also 133 times faster than TIMELY?  :-)  )

Cheers,
Michael

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 17 Nov 2016 08:22:46 +0900

On Wed, Nov 16, 2016 at 5:14 PM, Michael Welzl <[email-masked]> wrote:
> (see
https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-01.pdf
if you missed it).
> ( I tried, honestly, I did - but I just can’t manage to hold back a tiny
bit of sarcasm here:  ...is BBR also 133 times faster than TIMELY?  :-)  )

I'd like to explain further about 133x result of BBR vs Cubic first.

This is a spot-test. It is not a general quantification of BBR's
improvement on our B4 network (we'll publish that data in a separate
report). This is a 8MB RPC every 30s on a warmed connection on east-US <->
west-EU using the lowest QoS (best effort). That particular path has quite
a bit of burst induced losses. In this case, Cubic just died due to
1/sqrt(p). Notice the network path is fairly different from the typical
Internet access: it's +10G from start to end, with little buffers (compared
to BDP) on the way to build persistent large queues. The losses are caused
by burst traffic (of same or higher QoS). What the number 133x highlighted
was when loss does not indicate a persistent queue, using it as a sole
congestion signal is brittle. It is not to say loss signal is completely
useless, but it does not always indicate self-induced queues.

Comparing TIMELY and BBR are at one level comparing apples to oranges.
TIMELY is designed for a very specific hw/transport (RDMA/kernel-bypass)
and networks (intra-DC) with the goal to reduce tail latency. Hence it uses
specific NIC features like HW timestamps to perform its control.

BBR is designed to be a generic good transport to achieve high tput with
adequate queue. It faces an unknown network and sole feedback is TCP-acks
hence it must first perform a exponential search to get a basic sense of
the network. It also needs to deal with all sorts of adverse issues in the
wild Internet as we'v described.

A more interesting comparison would be vs other Internet-CC like Vegas
(which we are working on).

>
> Cheers,
> Michael
>
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: Michael Welzl <[email-masked]>
Date: Thu, 17 Nov 2016 09:57:43 +0900

Hi Yuchung,

> On Nov 17, 2016, at 8:22 AM, Yuchung Cheng <[email-masked]> wrote:
> 
> 
> 
> On Wed, Nov 16, 2016 at 5:14 PM, Michael Welzl <[email-masked] <mailto:[email-masked]>> wrote:
> >
> > Dear all,
> >
> > (see https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-01.pdf <https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-01.pdf> if you missed it).
> >
> > http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p537.pdf <http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p537.pdf>
> >
> > Any idea, anyone?
> >
> This is a spot-test. It is not a general quantification of BBR's improvement on our B4 network (we'll publish that data in a separate report). This is a 8MB RPC every 30s on a warmed connection on east-US <-> west-EU using the lowest QoS (best effort). That particular path has quite a bit of burst induced losses. In this case, Cubic just died due to 1/sqrt(p). Notice the network path is fairly different from the typical Internet access: it's +10G from start to end, with little buffers (compared to BDP) on the way to build persistent large queues. The losses are caused by burst traffic (of same or higher QoS). What the number 133x highlighted was when loss does not indicate a persistent queue, using it as a sole congestion signal is brittle. It is not to say loss signal is completely useless, but it does not always indicate self-induced queues.

Thanks for this information!

> Comparing TIMELY and BBR are at one level comparing apples to oranges. TIMELY is designed for a very specific hw/transport (RDMA/kernel-bypass) and networks (intra-DC) with the goal to reduce tail latency. Hence it uses specific NIC features like HW timestamps to perform its control. 

Well … I understand it was made for intra-DC communication, and it uses these features to get the necessary timer granularity. I can’t see why this wouldn’t make it applicable to networks with larger RTTs?

> BBR is designed to be a generic good transport to achieve high tput with adequate queue. It faces an unknown network and sole feedback is TCP-acks hence it must first perform a exponential search to get a basic sense of the network. It also needs to deal with all sorts of adverse issues in the wild Internet as we'v described. 

Right… E.g. I guess Timely wouldn’t work so well when facing losses.

> A more interesting comparison would be vs other Internet-CC like Vegas (which we are working on).  

No, I think that’s a terribly uninteresting comparison to do. I’d even call it ridiculous, as many many MANY mechanisms work better than Vegas, after all it's now 16 years old.
May I propose comparing against the state of the art, not the state of 2000?

Some of the direct follow-ups on Vegas for example, like FAST? Or delay-based algorithms that also use a bit more information than just delay?

E.g. Westwood comes to mind, as a mechanism that also looks at the rate at which ACKs arrive. AFAIK this one is readily available in Linux for comparison.

Also, some TCP variants try to distinguish between random packet loss and congestion-induced packet loss - so similar to BBR, you wouldn’t overreact to random losses with them. Veno is an example, and CTCP (Coded TCP, not Compound TCP) too.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Thu, 17 Nov 2016 10:49:06 +0900

Hi,

An add-on to my previous email below:

Mechanisms like Veno and CTCP, if I get this right, only decide that a loss is due to congestion when it also comes with delay growth.
I just witnessed your presentation in MAPRG about policing, and how BBR interacts with it:
https://www.ietf.org/proceedings/97/slides/slides-97-maprg-traffic-policing-in-the-internet-yuchung-cheng-and-neal-cardwell-00.pdf <https://www.ietf.org/proceedings/97/slides/slides-97-maprg-traffic-policing-in-the-internet-yuchung-cheng-and-neal-cardwell-00.pdf>
( Thanks for giving this presentation, this was very interesting. )

So this would mean that Veno, CTCP etc. would get it wrong in the face of a token bucket - there would be loss without delay growth, yet it IS due to a rate limit.
Given the prevalence of policing that you have shown, and how BBR nicely interacts with it, I find this very interesting (and good); I’m not sure if this has been considered in the design of any congestion control before?  Does anyone know of any other examples?

Cheers,
Michael

> On Nov 17, 2016, at 9:57 AM, Michael Welzl <[email-masked]> wrote:
> 
> Hi Yuchung,
> 
> 
>> On Nov 17, 2016, at 8:22 AM, Yuchung Cheng <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>>

---

From: "Bob Briscoe" <[email-masked]>
Date: Thu, 17 Nov 2016 07:34:38 -0000

Michael,

On Thu, November 17, 2016 1:49 am, Michael Welzl wrote:
> Mechanisms like Veno and CTCP, if I get this right, only decide that a
> loss is due to congestion when it also comes with delay growth. I just
> witnessed your presentation in MAPRG about policing, and how BBR
> interacts with it:
> https://www.ietf.org/proceedings/97/slides/slides-97-maprg-traffic-polici
> ng-in-the-internet-yuchung-cheng-and-neal-cardwell-00.pdf
> <https://www.ietf.org/proceedings/97/slides/slides-97-maprg-traffic-polic
> ing-in-the-internet-yuchung-cheng-and-neal-cardwell-00.pdf> ( Thanks for
> giving this presentation, this was very interesting. )
>
> So this would mean that Veno, CTCP etc. would get it wrong in the face of
> a token bucket - there would be loss without delay growth, yet it IS due
> to a rate limit. Given the prevalence of policing that you have shown, and
> how BBR nicely interacts with it, I find this very interesting (and
> good); I’m not sure if this has been considered in the design of any
> congestion control before?  Does anyone know of any other examples?

1/ I would be interested if there is enough data in the BBR dataset to
zoom in on clients in specific ISPs. I know at least one traffic
management vendor uses a congestion policer that limits the rate of heavy
users, but the rate at which it limits varies all the time, because it is
the outcome congestion limit, not a rate limit. More precisely, it
enforces a "congestion rate token bucket" limit on the per-user
contribution to shared congestion.

This would look similar to the varying capacity of a scheduler, but
distinguished from 3G/LTE or WiFi by only tiny additional delay, because
the measurement of shared congestion in the policer is based on an AQM on
a very high bandwidth link (typically 10G or 40Gb/s), so queuing delay is
very low before loss appears.

2/ HULL (High throughput, utra-low latency) uses DCTCP at the sender. In
the network it marks ECN on the real packets based on the length of a
'phantom' queue, which is a number incremented by the size of every
arriving packet, and decremented slightly more slowly than the drain rate
of the real queue. This keeps the real queue extremely short.

This is an example where there is zero delay build-up before the intended
operating point. Because of the lack of delay, it also uses hardware
pacing on the sender NIC, because there is no queuing delay to space out
TCP's ACK clock otherwise.

I don't know whether any DC operators are using HULL (does anyone?).

You would not want to put Not-ECT packets (like current BBR) into such a
queue structure.

Bob

>>>> Thanks very much to the authors of BBR for an interesting
>>>> presentation yesterday! (see
>>>> <https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-con
>>>> gestion-control-01.pdf> if you missed it).
>>>>
>>>> <http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p537.pdf>
>>>>
>>>>
>>>> Any idea, anyone?
>>>>
>>>>
>>>> ( I tried, honestly, I did - but I just can’t manage to hold back a
>>>> tiny bit of sarcasm here:  ...is BBR also 133 times faster than
>>>> TIMELY?  :-)  )
>>>>
>>>
>>> report). This is a 8MB RPC every 30s on a warmed connection on
>>> east-US <-> west-EU using the lowest QoS (best effort). That
>>> particular path has quite a bit of burst induced losses. In this
>>> case, Cubic just died due to 1/sqrt(p). Notice the network path is
>>> fairly different from the typical Internet access: it's +10G from
>>> start to end, with little buffers (compared to BDP) on the way to
>>> build persistent large queues. The losses are caused by burst traffic
>>> (of same or higher QoS). What the number 133x highlighted was when
>>> loss does not indicate a persistent queue, using it as a sole
>>> congestion signal is brittle. It is not to say loss signal is
>>> completely useless, but it does not always indicate self-induced
>>> queues.
>>
>>> Comparing TIMELY and BBR are at one level comparing apples to
>>> oranges. TIMELY is designed for a very specific hw/transport
>>> (RDMA/kernel-bypass) and networks (intra-DC) with the goal to reduce
>>> tail latency. Hence it uses specific NIC features like HW timestamps
>>> to perform its control.
>>
>> Well … I understand it was made for intra-DC communication, and it uses
>> these features to get the necessary timer granularity. I can’t see why
>> this wouldn’t make it applicable to networks with larger RTTs?
>>
>>
>>> BBR is designed to be a generic good transport to achieve high tput
>>> with adequate queue. It faces an unknown network and sole feedback is
>>> TCP-acks hence it must first perform a exponential search to get a
>>> basic sense of the network. It also needs to deal with all sorts of
>>> adverse issues in the wild Internet as we'v described.
>>
>> No, I think that’s a terribly uninteresting comparison to do. I’d even
>> call it ridiculous, as many many MANY mechanisms work better than
>> Vegas, after all it's now 16 years old.
>> Some of the direct follow-ups on Vegas for example, like FAST? Or
>> delay-based algorithms that also use a bit more information than just
>> delay?
>>
>> E.g. Westwood comes to mind, as a mechanism that also looks at the rate
>> at which ACKs arrive. AFAIK this one is readily available in Linux for
>> comparison.
>>
>> Also, some TCP variants try to distinguish between random packet loss
>> and congestion-induced packet loss - so similar to BBR, you wouldn’t
>> overreact to random losses with them. Veno is an example, and CTCP
>> (Coded TCP, not Compound TCP) too.
>>
>>
>> Cheers,
>> Michael
>>
>>
>> iccrg mailing list [email-masked]
> iccrg mailing list [email-masked]

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 17 Nov 2016 17:11:01 +0900

On Thu, Nov 17, 2016 at 9:57 AM, Michael Welzl <[email-masked]> wrote:
> these features to get the necessary timer granularity. I can’t see why this
> wouldn’t make it applicable to networks with larger RTTs?
>

Our team's experience suggests that with today's last-mile and WAN
networks there would be a number of challenges in applying congestion
control algorithms that use RTT increases as a direct signal to reduce
the sending rate. Some that come to mind:

- many last-mile technologies can have significant delays that are not
  necessarily signals that a bottleneck's bandwidth has been
  saturated:

  (1) link-layer retransmissions to recover from radio noise in
  cellular and wifi links

  (2) TCP delayed ACKs (often up to 200ms of delay)

  (3) packet aggregation and/or delays waiting for time slots in
  shared or half-duplex media in wifi/cellular/DOCSIS links

  (4) ACK decimation/suppression (e.g. in DOCSIS uplinks or
  middleboxes)

- policers: a bandwidth limitation without any delay signal

- WANs with shallow buffers: a 200ms two-way propagation delay with
  <=2ms of buffering at the bottleneck means you only ever get about 1%
  variations in RTT from congestion; then you have to somehow separate
  this signal from the noise, which may also be on the order of 1% of
  the RTT, due to various delays on either end host or in the network;
  this probably makes delay-based schemes a challenge in such
  environments

- loss-based congestion control sharing the bottleneck: if loss-based
  congestion control is sharing the bottleneck, then goodput for
  purely RTT-based algorithms will tend to drop toward zero; to
  survive, it seems one would need two separate algorithms (RTT-based
  and something else), plus a mechanism for switching between the two
  (CDG takes roughly this approach, AFAIK)

That's not to say that one couldn't design a congestion control
algorithm for WAN or last-mile links that used RTT as *one* of the
signals it considers. It's just that IMHO it would probably be tricky in
those environments to use RTT as the central signal.

cheers,
neal

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 18 Nov 2016 06:59:39 +0900

> > congestion control before?  Does anyone know of any other examples?
It'd be an interesting research for sure. We can share some of the tc
script we use to for BBR to test policers.

> contribution to shared congestion.
We've definitely seen dynamic policer limit than static, meaning the
end users may receive multiple rate-limit depends on his usage.
Although in BBR's policer module always measure that dynamically.

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 17 Nov 2016 23:10:15 -0500

> the sending rate. Some that come to mind:
Of course the other tricky business with using RTT increases as a direct
signal to reduce the sending rate, when the path is over a WAN, is that
routing changes can and do vastly increase the RTT, when often the
available bandwidth does not change, or can even increase.

neal