Subject: Re: [iccrg] [bbr-dev] Re: Evaluation of BBR congestion control

From: Roland Bless <[email-masked]>
Date: Tue, 5 Sep 2017 16:50:51 +0200

Hi Neal,

please, see comments inline.

On 30.08.2017 at 18:14 Neal Cardwell wrote:
> Thanks, Roland, for posting this paper. It does a nice job of
> illustrating the known areas where BBR's behavior can be significantly
> improved. We have discussed these areas previously on the bbr-dev
> list, and in presentations at the IETF, but this paper also does a
> nice job of illustrating them.

I don't think that the observed problems have been laid on the table so
clearly and the paper explains in section III why BBR's model leads to
the observed problems.

> BBR is under active development, and we are working on the areas
> mentioned in this paper. My sense is that two of the BBR efforts we

As also mentioned in our conclusion, we are fully aware that BBR is
under active development. However, I think that the observed problems
are not easy to fix.

> are working on are most relevant for the scenarios examined in this
> paper:
> 
> (1) Improving the bandwidth estimator.
> 
> There are classes of scenarios, discussed on the bbr-dev list, our
> IETF presentations, and this paper, where the BBR bandwidth estimator
> can be greatly improved. In particular, our group is working on
> refining BBR's response to packet loss, integrating the packet loss
> signal in with the delivery rate signal to ensure the overall behavior
> is well-adapted both when:
> 
>   (a) loss corresponds to the bottleneck being persistently full, and
> 
>   (b) when loss is likely just due to random/short burst losses.

Integrating packet loss as signal sounds necessary and differentiating
between (a) and (b) may also have solved the throughput problems of
shallow buffers in B4.

> The initial BBR release prioritized simplicity, but we believe there
> is more signal there to be extracted and filtered and applied.
> 
> (2) More frequently drawing down the amount of in-flight data to match
> the target.

It's probably not easy to determine the "target". IMHO it would be good
if BBR defines congestion signals and a response to congestion.
Moreover, controlling the amount of inflight data by a rate-based
approach is IMHO more difficult than with a congestion window-based
approach: a small mismatch in the rate may lead to an increasing amount
of inflight data (over several RTTs) whereas a too large congestion
window just leads to a fixed amount of excess data. The current window
filter keeps the sending rate probably too high for 11 RTTs (the
win_minmax implementation will update to a new max _after_ 10 RTTs if
win=10).

> This can reduce queues, which in turn reduces delays and losses, and
> improves fairness. An example mechanism would be predominantly pacing> at a sub-unity gain while in-flight is above the estimated BDP.
> Randall mentions this aspect, and we posted a patch in this vein back
> in December on the bbr-dev list (and Brian Tierney kindly took it for
> a spin):
>    https://groups.google.com/forum/#!msg/bbr-dev/Ps4zyM5AD9g/HjvkwK2qEAAJ
> 
> This mechanism is tricky to get right, for a variety of reasons, some
> of which Randall mentions. Aspects that make this effort tricky
> include:
> 
> (a) being robust to link-layer aggregation effects that can cause the
> ACK stream to disappear for long periods, potentially leading to
> cruising too long at a sub-unity gain with no ACK feedback,
> 
> (b) ensuring that even when using a prolonged sub-unity gain we ensure
> we continue to regularly refresh out bandwidth estimate with a
> super-unity probing phase,
> 
> (c) being careful about which packets are considered to be "in
> flight", since some can be in the pacing layer rather than the
> network.
> 
> But IMHO it seems that each of (1) and (2) is a necessary, but not
> sufficient, part of a complete solution.

At least from our experience, several aspects are intertwined, e.g.,
changing one property may also change the fairness behavior and vice
versa. However, we look forward to hear news about BBR's development.

Regards,
 Roland

> thanks,
> neal
> 
> 
> On Wed, Aug 30, 2017 at 10:32 AM, 'Randall Stewart' via BBR
> Development <[email-masked]> wrote:
>> It is an interesting read. I have been working on the FreeBSD implementation
>> of
>> BBR for quite some time and it surely explains some of the things I am
>> seeing :)
>>
>> One thing I do that Neal and I have chatted about that can help a small
>> amount with this is I do not exit the 3/4 drain until it reaches the target
>> cwnd (no matter what the time). The paper (and I-D) discuss staying in
>> at most a RTT -Prop (IIRR).. This at times causes extended drain periods
>> but on the other hand if you are in such an extended drain then most likely
>> you are overshooting on the b/w.
>>
>> Another thing I am just starting to work on is some modifications that will
>> go along with this. In using this for video I find that probeRTT can also
>> be a killer.. I have seen extended periods of probeRTT and during
>> such an extended period our clients switch away (and rebuffer).
>>
>> Some things I am working on include:
>>  - Allow probeRTT to only last a max of 2 seconds
>>  - Any time probeRTT is interrupted schedule it to be re-entered
>>    in 2 seconds. Also reduce the b/w estimate by a configurable amount (I am
>>    starting with 25%)
>> - When I see that I have extended my 3/4 drain, I will also reduce the b/w
>> estimate
>>   by 1% for each rtt it extended by capping at a max of 10% (again
>> configurable).
>> - Prevent probeRTT from being entered if we are in any form of recovery.
>>
>> I will experiment with this in our transport lab and if it improves some of
>> the
>> issues we are seeing I will give it a try it a bit more widely
>>
>>
>> On Tuesday, August 29, 2017 at 1:26:33 PM UTC-4, Roland Bless wrote:
>>>
>>> Hi,
>>>
>>> those who are interested in BBR congestion control
>>> might be also interested in this forthcoming paper:
>>> Mario Hock, Roland Bless, Martina Zitterbart, "Experimental Evaluation
>>> of BBR Congestion Control", IEEE ICNP 2017, October 2017
>>>
>>> A pre-published version is available here:
>>> http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
>>>
>>> Best regards,
>>>  Roland
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "BBR Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email-masked].
>> For more options, visit https://groups.google.com/d/optout.

---

From: Neal Cardwell <[email-masked]>
Date: Sat, 16 Sep 2017 10:05:49 -0400

Hi Roland,

Thanks for your very thoughtful remarks.

> clearly and the paper explains in section III why BBR's model leads to

Yes, your paper does a nice job of illustrating some of the biggest
remaining issues with BBR.

> under active development. However, I think that the observed problems

I agree it's not easy, but we are working on it. :-)

> between (a) and (b) may also have solved the throughput problems of

Yes, BBR does use packet loss as a signal (see
bbr_set_cwnd_to_recover_or_restore()) but we agree that the response
to loss can be improved, which we're actively working on.

> It's probably not easy to determine the "target".

BBR already has a target amount of in-flight data; from my perspective
part of the question is how to best improve that target. I agree it's
not easy to improve, since there are a number of competing concerns to
trade off.

> IMHO it would be good

BBR already has several (implicit) congestion signals that it responds
to: (a) a decrease in a flow's delivery rate, and (b) packet loss. I
think the main question is exactly how to refine or extend the set of
signals and responses.

> window just leads to a fixed amount of excess data.

For best behavior, a congestion control algorithm needs both a good
limit on the sending rate (pacing rate), and a good limit on the
volume of inflight data (cwnd). Both are needed, as laid out in the
"Characterizing the Bottleneck" section of the BBR CACM paper. I agree
a tight bound on the amount of data in flight can be particularly
important at times; and indeed, when there is packet loss, BBR uses a
tighter cwnd.

> The current window

Yes, we have experimented with much shorter windows for bandwidth
filtering in the face of apparent congestion. I agree this seems like
a good direction. The trick is to make this robust to cases where
apparent congestion turns out to just be random noise.

> versa. However, we look forward to hear news about BBR's development.

Yes, I agree that this is tricky, since all these goals are
intertwined: low queuing and packet loss, high throughput, fairness
with other BBR flows, and avoiding starvation when coexisting with
loss-based congestion control algorithms, etc. Again, it's tough to
trade off these competing concerns, all while keeping the code simple
enough and robust enough for the real world. But that's part of why
it's a fun challenge. :-)

thanks,
neal