Subject: [iccrg] BBR v2 questions (was Re: [bbr-dev] BBR updates at the IETF 102 ICCRG session)

From: "Bless, Roland (TM)" <[email-masked]>
Date: Mon, 13 Aug 2018 14:39:57 +0200

Hi Neal,

thanks for the update @IETF102.
While reviewing the presented slides some clarification questions
came up:

* When is inflight_lo set?

* Is the max delivery_rate filter still in place?
  Slide 15 mentions: "Bandwidth estimator filter window
  now simply covers last 2PROBE_BW cycles". Is "bandwidth
  estimator filter window" the same as the max delivery_rate filter?
  If so this window period could be as long as 10s?

* estimated_bdp still corresponds to the current bottleneck
  share of the flow and is calculated by (filtered estimates
  of) max_delivery_rate * RTT_min?

* Is BBR v2 now window-based, or, is it still rate-based
  with a cwnd cap as in version 1? (e.g., Slide 5 mentions
  "start probing at 1 extra packet": per what time period?).

* How is the ACK aggregation compensation (cf. ICCRG@IETF101
  presentation) integrated? Will this extra amount be added to
  inflight values?

* "hard ceiling" means "hit inflight_hi loss or ECN ceiling"
  or something else?

* What is meant by "flow balance"?

* There is no explicit decrease action except for the
  DRAIN and PROBE_BW DOWN phases (slide 30 mentions
  multiplicative decrease)? How low is the mentioned
  pacing rate at PROBE_BW DOWN (slide 30)?

Regards,
 Roland

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 13 Aug 2018 12:33:40 -0400

> While reviewing the presented slides some clarification questions

Hi Roland,

Thanks for the excellent questions. Responses in-line.

Any page numbers I mention will be in reference to the IETF 102 slides
under discussion:

https://datatracker.ietf.org/meeting/102/materials/slides-102-iccrg-an-update-on-bbr-work-at-google-00

The inflight_lo parameter is set/updated (a) in fast recovery, or (b) when
receiving ECN signals (page 10).

>   If so this window period could be as long as 10s?

Yes, the max delivery rate filter is still in place. Yes, the "bandwidth
estimator filter window" is the same as the max delivery_rate filter (aka
BtlBw or bbr_bw(sk) from the Linux TCP BBR code).

As you note, in the version of the design described in the slides, yes, the
filter window could stretch to 2 * 5secs, or 10secs. In the current
prototype design, the max bandwidth probing interval is 3 seconds, so it
could stretch to 2 * 3sec = 6 secs. Note that it would only be that long
for WAN flows with long RTTs. For flows with shorter RTTs the Reno probing
time scale would usually be much shorter, so the BBR v2 bw probing time
scale would be governed by the Reno calculation (eg for 10G datacenter
environments the bandwidth probing time scale would tend to be less than
150 * 100usec ~= 15msec).

The 6 sec band width filter timescale for WAN flows is longer than ideal,
but seemed like the simplest way to meet the design constraints:

  (1) bandwidth probing can cause packet loss, so the time in between
bandwidth probes should be adapted to longer time-scales for high-RTT flows
for better coexistence with Reno/CUBIC

  (2) to maintain a robust bandwidth estimate, the bandwidth estimator
filter window needs to cover at least the most recent bandwidth probing
period, and also needs to cover some amount of history before that if the
most recent bandwidth probe was very recent

If anyone sees a better way to harmonize these design constraints, please
let us know.

>   of) max_delivery_rate * RTT_min?

Yes, exactly.

>   with a cwnd cap as in version 1?

The latter. It is still mostly rate-based but with a cwnd cap, as in
version 1. As with BBR v1, the BBR v2 code tries to smoothly control
inflight with the pacing rate in the common case, but there is still a cwnd
that places a hard upper bound on the amount of inflight data, and the cwnd
becomes the limiting factor if (a) ACKs are delayed beyond the expected
degree, or (b) the BBR sender cuts inflight to adapt to signals like packet
loss (or ECN in v2).

> (e.g., Slide 5 mentions

In the current BBR v2 prototype code, the time scale for increasing the
magnitude of probing is per round trip. This is described in more detail in
slide 11: "inflight_probe grows exponentially per round trip: - 1, 2, 4,
8... packets". I think there are interesting trade-offs to be explored
between making this probing time scale (a) per round trip (to allow rapid
utilization of free bandwidth), or (b) per fixed wall clock interval (to
improve RTT fairness). The current prototype opts for rapid utilization
over RTT fairness. Partly this is based on the history of
users/applications just opening more connections if TCP does not do a good
job of utilizing free bandwidth. The hope is to make the congestion control
efficient enough at quickly utilizing free bandwidth that
users/applications do not have to resort to opening multiple connections
just to make good use of newly-available bandwidth.

>   presentation) integrated? Will this extra amount be added to

The aggregation compensation discussed in the IETF 101 slides
<https://datatracker.ietf.org/meeting/101/materials/slides-101-iccrg-an-update-on-bbr-work-at-google-00>
is still implemented using the same code posted in the patches in this
bbr-dev thread
<https://groups.google.com/d/msg/bbr-dev/8pgyOyUavvY/aaGGR4_eBgAJ>, early
on in the computation of the target cwnd. Then in BBR v2, as the final step
of the bbr_set_cwnd() function the BBR v2 code makes sure that the cwnd is
at or below any upper bound imposed by the BBR v2 model of what is an
appropriate inflight level to maintain at this point in the state machine
(based on inflight_lo and inflight_hi). So the inflight_lo/inflight_hi take
precedence over the aggregation estimator. In particular this means that if
BBR tries allowing extra data in flight to compensate for aggregation and
then packet loss or ECN signals provide evidence that this higher level
inflight data is causing excessive queues, then BBR adapts by trimming cwnd
and inflight.

> * "hard ceiling" means "hit inflight_hi loss or ECN ceiling"

Yes, the "hard ceiling" phrase on slide 10 means the case where inflight_hi
was set due to loss or ECN signals.

> * What is meant by "flow balance"?

By "flow balance" the slides mean a condition where the sender's current
sending rate matches the current delivery rate for the flow.

>   multiplicative decrease)?

There are two other explicit decrease actions in addition to the two you
note (DRAIN and PROBE_BW DOWN). These two new mechanisms are at the core of
the improvements in BBR v2. The slides try to outline these decrease
actions on slide 10:

First, if the last bandwidth probe saw a "hard ceiling" where loss or ECN
signals suggested there was an excessive queue, this is reflected in
inflight_hi. After this happens, we try to maintain an inflight level that
leaves unutilized headroom below that inflight level, by maintaining an
inflight level respecting the constraint:

       inflight <= (1 - headroom) * inflight_hi     (where headroom=0.15)

Note that the 0.85 factor here is the same constant used by CUBIC's
somewhat analogous "fast convergence" mechanism for having bigger flows
make way for smaller/new flows.

Second, upon loss or ECN signals the sender cuts inflight_lo to ensure that
the sender does not increase inflight beyond the current level (until the
next time it probes for bandwidth). This is somewhat similar to the way
Linux TCP PRR respects packet conservation in the first round trip of fast
recovery.

> How low is the mentioned

The pacing rate in PROBE_BW DOWN is still 0.75 * estimated_bandwidth (as in
BBR v1).

I hope that helps clarify some of these points.

best,
Neal