Subject: Bufferbloat and Congestion Control

From: "Bless, Roland (TM)" <[email-masked]>
Date: Fri, 31 Mar 2017 15:41:04 +0200

Hi,

thanks to the session recordings I just listened to the very
interesting ICCRG related discussion in the tsvarea meeting from
Monday.

Lars Eggert said something like:
"Bufferbloat is something that a congestion controller
can't really do anything about anyway...
it's not necessarily a congestion control problem."

I believe that this statement is not correct.
A large buffer will not be filled completely by a delay-based
congestion control (e.g., Vegas was a very early
one). Thus even if the buffer is large, the queuing
delay doesn't increase dramatically and we wouldn't
see that bufferbloat effect so prominently (as in slide 3 of
https://www.ietf.org/proceedings/98/slides/slides-98-tsvarea-sessb-reflections-on-congestion-control-praveen-balasubramanian-00.pdf).

So if _everyone_ would use such a delay-based variant, we wouldn't
have the bufferbloat problem.

However, since we have more loss-based congestion control variants
deployed, which will fill a tail-drop buffer completely, such
delay-based variants get suppressed and cannot get a reasonable share
of the bottleneck bandwidth. Therefore, several variants have a
fallback to a more aggressive mode in order to get a reasonable
share (I think Christian Huitema also referred to this by the different
modes). What could help is some kind of co-existence mechanism that
separates queue-filling TCP CC variants from delay-based variants
(e.g., see http://ieeexplore.ieee.org/document/7796842/ for details).
So IMHO getting rid of the queue filling variants would be useful,
but is probably hard in practice, i.e., replacing them isn't easy
due to the co-existence problem.

Consequently, fighting against bufferbloat could be done in the network
using AQMs (enforcing shorter queues) or by using a different
congestion control algorithm (trying to limit inflicted queuing delay),
or even by a combination of both. However, what delay and throughput
performance can be achieved with AQMs is limited, so better performance
in both metrics requires eventually a better congestion control algorithm.

Regards,
 Roland

---

From: Michael Welzl <[email-masked]>
Date: Fri, 31 Mar 2017 09:02:00 -0500

Hi Roland,

I agree with everything, except in your email below, everywhere:

s/delay/delay or ECN

I state this because it’s my opinion but also to hopefully prevent a flood of long emails related to ECN…

Cheers,
Michael

> On Mar 31, 2017, at 8:41 AM, Bless, Roland (TM) <[email-masked]> wrote:
> 
> Hi,
> 
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 31 Mar 2017 10:24:45 -0400

I would agree the bufferbloat is susceptible to mitigation by
congestion control at the sender.

One relevant alternative I have not seen mentioned in this thread is
BBR congestion control. (For BBR info, including the CACM paper and
links to the BBR talks at the last 2 ICCRG sessions, see:
https://groups.google.com/d/forum/bbr-dev )

Defeating bufferbloat was one of the core goals of BBR, and our
experience deploying it for YouTube shows it achieves this goal. BBR
flows sharing with other BBR flows avoid filling bloated buffers, and
instead maintain a reasonable-sized queue. But when BBR flows share a
bottleneck with a loss-based CC like Reno or CUBIC, the model of the
path used by the BBR flows adapts to the standing queue created by the
loss-based flows, scaling up cwnd so that the BBR flows are not
starved (as delay-based schemes like Vegas are starved in such
scenarios). If the loss-based flows leave the bottleneck, BBR will
quickly converge back to its preferred operating regime of small
queues.

BBR is deployed for YouTube and Google TCP traffic, and available in
Linux TCP and QUIC, and work is under way at Netflix to implement it
for FreeBSD TCP. Folks concerned about bufferbloat should keep it in
mind as an available alternative.

neal

ps: AQMs are also good.

---

From: Michael Welzl <[email-masked]>
Date: Fri, 31 Mar 2017 09:46:18 -0500

Hi,

As a researcher with possible interest in evaluating BBR, my question would be: what is “it” ?

My understanding is that we have:
- 2 IETF presentations
- A paper in ACM Queue
- Linux code

and none of these three things are really equal.  ( Sorry if I’m wrong!  Maybe it’s only the ACM Queue paper that doesn’t quite match the rest?  )
If this is true, it would be good to try to write the whole thing up in a draft so we could have a reference description of what BBR really is.

Else, what would you consider the most important reference among the three items above?  The Linux code, I guess?

Cheers,
Michael

> On Mar 31, 2017, at 9:24 AM, Neal Cardwell <[email-masked]> wrote:
>

---

From: Jim Gettys <[email-masked]>
Date: Fri, 31 Mar 2017 10:47:23 -0400

> mind as an available alternative.

There are other issues that people need to keep in mind:

Packet scheduling is *really* important.  You really don't want to
have a burst of packets monopolize your bottleneck link, particularly
when that bottleneck may be small bandwidth.

For example, my brother suffers on a DSL line, with something like
384K up and a megabit or two down.

One full size packet @ 1mbps == 13 milliseconds.  Two of those packets
is already human perceptible latency for some applications (and gamers
and stock traders will tell you that even a single 13ms delay to their
application is a significant disadvantage). On his line, even one
packet upbound is of order 40ms.

So while congestion avoidance is key to keeping TCP's behavior sane,
it is *not* sufficient to keep latency to where it needs to be for
many applications that share the bottleneck. Wonderful as BBR is (and
I'm a great fan), it still can induce more than a BDP of queue: over
continental paths, that's of order 100ms and more over
intercontinental paths.

And even one flow of something other than BBR can ruin your whole day
completely.  Getting everyone to convert to a delay based congestion
system ain't ever going to happen as they don't compete properly, so
finally having an algorithm like BBR that can be deployed is
wonderful.

In short, BBR will reduce the fraction of time latencies are out
completely of control, but no TCP congestion control system by itself
can get latencies to where they need to be for many applications.

Jim

On Fri, Mar 31, 20

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 31 Mar 2017 10:59:45 -0400

> Else, what would you consider the most important reference among the three items above?  The Linux code, I guess?

Yes, the Linux TCP code would be the most important reference, since
that's the most precise, and it's what's actually controlling YouTube
and google.com TCP.

That said, the Linux TCP code should very closely match the CACM/Queue
paper. I can think of only one or two details in the gain cycling code
that were added to the code in the middle of the paper editing
process, and thus were not captured in the paper. And in turn, the
ICCRG presentations should also very closely match the code and paper
(except for the parts marked "experiments", eg the IETF 98 ICCRG
slides 14-19). Please let us know if there are specific places where
there are confusing divergences.

We are working on writing a draft or similar prose spec for those who
would prefer to use that as a reference.

thanks,
neal

---

From: Michael Welzl <[email-masked]>
Date: Fri, 31 Mar 2017 10:43:57 -0500

Hi,

In line:

> On Mar 31, 2017, at 9:59 AM, Neal Cardwell <[email-masked]> wrote:
> 

That’s great to hear - my impression of differences was a mix of “impression from looking at things” and “impression from what people say” - so this confirmation that things don’t really diverge so much is very useful, thanks!

> would prefer to use that as a reference.

Wonderful ! Thanks for that too!

Cheers,
Michael

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 31 Mar 2017 13:20:14 -0400

> packet upbound is of order 40ms.

Yes, this is a great point. In the BBR team we are very conscious of
the latency issues in low-bandwidth links, and this is something we
focus on. For example, this consideration you mention is exactly why
Linux TCP BBR chooses a TSO burst size of 1 packet (instead of the
Linux TCP default of 2 packets) for bandwidths below 1.2Mbps.

> continental paths, that's of order 100ms and more over

The queue levels you mention can indeed be seen when there are
multiple flows sharing a bottleneck and using the current version of
BBR. There is nothing inherent about the BBR approach or framework
that requires that particular level of queue, so one of our big
focuses right now is testing some tweaks to the algorithm to reduce
the degree of queuing in such scenarios. We discussed some of the
experiments in pages 14-19 of this week's BBR talk:

  https://www.ietf.org/proceedings/98/slides/slides-98-iccrg-an-update-on-bbr-congestion-control-00.pdf

> finally having an algorithm like BBR that can be deployed is

Thanks. It is also our sense that BBR provides performance and
behavior that makes it deployable on today's Internet.

> can get latencies to where they need to be for many applications.

Indeed, some very demanding latency-sensitive apps may need changes in
the network to work well. Until those changes are widely deployed, and
even afterward, my sense is that it's worth pushing congestion control
to be as good as it can be.

thanks,
neal

---

From: Jim Gettys <[email-masked]>
Date: Fri, 31 Mar 2017 13:36:30 -0400

> to be as good as it can be.

I think we have to push from all directions to drain this global swamp.

We're getting some fq_codel uptake in commercial consumer devices at
this date, and finally have running code to deal with WiFi for the
first time (out the door in the new LEDE release: try it, you'll like
it!). It's time to put market forces to work, as it can be a major
competitive advantage, and playing one vendor off against the other is
an effective way to get motion in the market.

Very early in tilting at the bufferbloat windmill, Dave Clark told me:
"Don't yell fire in the theater until the exits are marked".  It think
it is time
for us all to yell "fire".

"FIRE!"
                                           - Jim

---

From: Joe Touch <[email-masked]>
Date: Fri, 31 Mar 2017 15:18:11 -0700

On 3/31/2017 7:47 AM, Jim Gettys wrote:
> One full size packet @ 1mbps == 13 milliseconds. 
That sort of blocking is its own problem and the only solution is to use
smaller packets.

It can be solved at TCP by pushing the MSS down, but for UDP that's
another good reason for considering UDP options as a TSVWG WG item!

Joe

---

From: Jim Gettys <[email-masked]>
Date: Fri, 31 Mar 2017 19:30:55 -0400

> another good reason for considering UDP options as a TSVWG WG item!

​Yes, but without better packet scheduling such as fq_codel at the
bottleneck, it won't help.  But I'm decently optimistic that ​that
fq_codel/cake/other things will be deploying widely, since we're seeing
fq_codel appear in commercial products at this date.
                                     - Jim

---

From: Jim Gettys <[email-masked]>
Date: Fri, 31 Mar 2017 19:41:45 -0400

> fq_codel appear in commercial products at this date.

​Oh, I forgot: I wrote this piece recently. It may be useful to people, and
let me know if there are products you think should be added to the list.

https://gettys.wordpress.com/2017/02/02/home-products-that-fixmitigate-bufferbloat/

The Evenroute router is particularly good if you need a "plug and play"
home router, and/or one that can deal with ISP's that under provision their
network and deliver variable bandwidth, such as my brother's DSL line. I
need to check if the latest Evenroute supports the new WiFi code; that is
stunningly good.  My Ubiquiti Picostation running LEDE keeps latency around
40ms or less even under load :-).

I've just installed a DOCSIS 3.1 modem yesterday, and will probably run
some tests on it versus what SQM does in the next week.  That should be
interesting.

It's particularly frustrating that major vendors "bait and switch" their
customers​ with trade names.  Said vendor(s) will get no endorsement from
me, despite the fact it's enough of the market to be "interesting".

Would be nice to get pacing into the new WiFi code and fq_codel so we can
use BBR with it on clients (Linux/Android).  Any volunteers for an
adventure?
                                             - Jim

---

From: Joe Touch <[email-masked]>
Date: Fri, 31 Mar 2017 19:51:17 -0700

On 3/31/2017 4:30 PM, Jim Gettys wrote:
>
>
> On Fri, Mar 31, 2017 at 6:18 PM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>
> fq_codel/cake/other things will be deploying widely, since we're
> seeing fq_codel appear in commercial products at this date.

No amount of scheduling overcomes a limit on the unit of scheduling -
the packet size, which has increasing impact as link bandwidth decreases.

Better scheduling is important necessary too, but by itself insufficient.

Joe