BBR parameter tuning

[Usama Naseer]

Hi,

I am working on a study that explores BBR configuration parameters tuning. Till now I have compiled a list of constants to be:

- BtlBwFilterLen — The length of the BBR.BtlBw max filter window is BtlBwFilterLen = 10 round trips.
- RTpropFilterLen — BBR estimates BBR.RTprop using the minimum recent RTT sample seen by the connection over that past RTpropFilterLen seconds.
- BBRHighGain -- A constant specifying the minimum gain value that will allow the sending rate to double each round (2/ln(2) ~= 2.89), used in Startup mode for both BBR.pacing_gain and BBR.cwnd_gain.

- ProbeRTTInterval -- A constant specifying the minimum time interval between ProbeRTT states: 10 secs.

- ProbeRTTDuration -- A constant specifying the minimum duration for which ProbeRTT state holds inflight to BBRMinPipeCwnd or fewer packets: 200 ms.

I wanted to ask a few questions regarding the space of parameters to tune for optimal BBR performance.

(i) Is this the complete list or are there any other parameters that I am missing?

(ii) Have you observed BBR performance to be sensitive to these (or a subset) parameters?

(iii) Imagine that you have an oracle that can tune BBR for a specific network case, what different parameters will you advise for the oracle to tune?

Thanks!

Usama

[Dave Taht]

Take a look at bbrv2. There's more.

If I had an oracle, I'd tune the default 200ms probe duration to
something that actually matched the observed
RTT periodically, particularly when the observed RTT is larger than,
say, 250ms (which is one trip around the planet) Ridiculous RTTs
exist from overbuffering, and BBRv1 at least suffers from the
latecomer problem.

>
> Thanks!
> Usama
>
> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/9fc50e63-3897-49cb-b52c-af22b1d7607b%40googlegroups.com.



--

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

[Neal Cardwell]

On Mon, Sep 2, 2019 at 12:50 PM Dave Taht <dave...@gmail.com> wrote:

> (i) Is this the complete list or are there any other parameters that I am missing?
> (ii) Have you observed BBR performance to be sensitive to these (or a subset) parameters?
> (iii) Imagine that you have an oracle that can tune BBR for a specific network case, what different parameters will you advise for the oracle to tune?

If I had an oracle, I'd tune the default 200ms probe duration to
something that actually matched the observed
RTT periodically, particularly when the observed RTT is larger than,
say, 250ms (which is one trip around the planet)  Ridiculous RTTs
exist from overbuffering, and BBRv1 at least suffers from the
latecomer problem.

Please note that the duration of PROBE_RTT is not merely 200ms. The duration of PROBE_RTT is already always at least as long as the maximum of 200ms and the current round-trip time. In the Linux TCP BBR code you can see that in the parts of the code that reference "probe_rtt_round_done". The basic idea is that once the flows pull down the queue, they need to wait at least one round-trip to obtain an RTT sample that reflects the lower queue.

neal

[Dave Taht]

On Mon, Sep 2, 2019 at 10:54 AM Neal Cardwell <ncar...@google.com> wrote:
>
>
>
> On Mon, Sep 2, 2019 at 12:50 PM Dave Taht <dave...@gmail.com> wrote:
>>
>> > (i) Is this the complete list or are there any other parameters that I am missing?
>> > (ii) Have you observed BBR performance to be sensitive to these (or a subset) parameters?
>> > (iii) Imagine that you have an oracle that can tune BBR for a specific network case, what different parameters will you advise for the oracle to tune?
>>
>> If I had an oracle, I'd tune the default 200ms probe duration to
>> something that actually matched the observed
>> RTT periodically, particularly when the observed RTT is larger than,
>> say, 250ms (which is one trip around the planet) Ridiculous RTTs
>> exist from overbuffering, and BBRv1 at least suffers from the
>> latecomer problem.
>
>
> Please note that the duration of PROBE_RTT is not merely 200ms. The duration of PROBE_RTT is already always at least as long as the maximum of 200ms and the current round-trip time.

Sorry. What I'd meant to say was:

>> If I had an oracle, I'd tune the default 200ms probe duration to

^^^^^
>> something that actually matched the physical
^^^^^^^

>> RTT periodically, particularly when the observed RTT is larger than,
>> say, 250ms (which is one trip around the planet)

But my subconcious lost the joke when I typed it up.

"In the Linux TCP BBR code you can see that in the parts of the code
that reference "probe_rtt_round_done". The basic idea is that once the
flows pull down the queue, they need to wait at least one round-trip
to obtain an RTT sample that reflects the lower queue.
>
> neal
>

[Dave Taht]

On Mon, Sep 2, 2019 at 11:00 AM Dave Taht <dave...@gmail.com> wrote:
>
> On Mon, Sep 2, 2019 at 10:54 AM Neal Cardwell <ncar...@google.com> wrote:
> >
> >
> >
> > On Mon, Sep 2, 2019 at 12:50 PM Dave Taht <dave...@gmail.com> wrote:
> >>
> >> > (i) Is this the complete list or are there any other parameters that I am missing?
> >> > (ii) Have you observed BBR performance to be sensitive to these (or a subset) parameters?
> >> > (iii) Imagine that you have an oracle that can tune BBR for a specific network case, what different parameters will you advise for the oracle to tune?
> >>
> >> If I had an oracle, I'd tune the default 200ms probe duration to
> >> something that actually matched the observed
> >> RTT periodically, particularly when the observed RTT is larger than,
> >> say, 250ms (which is one trip around the planet) Ridiculous RTTs
> >> exist from overbuffering, and BBRv1 at least suffers from the
> >> latecomer problem.
> >
> >
> > Please note that the duration of PROBE_RTT is not merely 200ms. The duration of PROBE_RTT is already always at least as long as the maximum of 200ms and the current round-trip time.
>
> Sorry. What I'd meant to say was:
>
> >> If I had an oracle, I'd tune the default 200ms probe duration to
> ^^^^^
> >> something that actually matched the physical
> ^^^^^^^
> >> RTT periodically, particularly when the observed RTT is larger than,
> >> say, 250ms (which is one trip around the planet)
>
> But my subconcious lost the joke when I typed it up.

In fact it would have worked better had I said:

If I had an oracle I'd tune for one way delay, rather than RTT.

There's a lot of jokes I could make that start off with "If I had an
oracle..." - but it was precoffee, and happy monday,
to all.

[Neal Cardwell]

On Mon, Sep 2, 2019 at 12:43 PM Usama Naseer <usama_...@brown.edu> wrote:
>
> Hi,
>
> I am working on a study that explores BBR configuration parameters tuning.

Sounds great! Looking forward to hearing updates about your findings.

> Till now I have compiled a list of constants to be:
>
> - BtlBwFilterLen — The length of the BBR.BtlBw max filter window is BtlBwFilterLen = 10 round trips.
> - RTpropFilterLen — BBR estimates BBR.RTprop using the minimum recent RTT sample seen by the connection over that past RTpropFilterLen seconds.
> - BBRHighGain -- A constant specifying the minimum gain value that will allow the sending rate to double each round (2/ln(2) ~= 2.89), used in Startup mode for both BBR.pacing_gain and BBR.cwnd_gain.
> - ProbeRTTInterval -- A constant specifying the minimum time interval between ProbeRTT states: 10 secs.
> - ProbeRTTDuration -- A constant specifying the minimum duration for which ProbeRTT state holds inflight to BBRMinPipeCwnd or fewer packets: 200 ms.
>
> I wanted to ask a few questions regarding the space of parameters to tune for optimal BBR performance.
>
>
> (i) Is this the complete list or are there any other parameters that I am missing?

In Linux TCP BBRv1, you can see the full list by looking at all the
lines that start with "static const" in the source file:
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/net/ipv4/tcp_bbr.c

In Linux TCP BBRv2 alpha, you can see the full list by looking at all
the module_param_named() lines:
https://github.com/google/bbr/blob/v2alpha/net/ipv4/tcp_bbr2.c

> (ii) Have you observed BBR performance to be sensitive to these (or
a subset) parameters?

My impression is that BBR performance is most sensitive to the
following (using the Linux TCP BBRv2 alpha algorithm and
nomenclature):

o high_gain, startup_cwnd_gain - these specify the rate of exponential
growth for BBR flows in STARTUP

o cwnd_gain - this specifies how much delay variation to
expect/tolerate, by default

o loss_thresh - this specifies what loss rate to take as an indication
to stop bandwidth probing

o bw_probe_pif_gain - this specifies the level of inflight data
(relative to the estimated BDP) that is taken as a signal that
periodic bandwidth probing may have built up sufficient queue for it
to make sense to exit bandwidth probing

o bbr_probe_rtt_win_ms - this specifies the typical interval between
entering PROBE_RTT mode

> (iii) Imagine that you have an oracle that can tune BBR for a specific network case, what different parameters will you advise for the oracle to tune?

My sense is that a good starting point would be the parameters to
which performance is most sensitive. :-)

best,
neal

[Usama Naseer]

This is great! This is exactly what I was looking for. Thanks for pointing to the code where these constants are defined. 

> To unsubscribe from this group and stop receiving emails from it, send an email to bbr...@googlegroups.com.