Any way to use BBRv3 with newer kernels?

[Viraj Alankar]

Hello,

I've been using the BBRv3 kernel source at https://github.com/google/bbr/tree/v3 and the results are pretty amazing for a high latency network. Great work!

I'm wondering if it's possible to use this with a newer kernel, such as with the latest stable v6.13.7? Does it involve just copying in tcp_bbr.c? It sounds like there are other patches as well.

Also, it is unclear to me whether the davem/net-next kernel tree contains BBRv3 or not. Comparing the source it seems not so. Is that correct?

Basically, what's the best way to use the latest stable kernel tree with BBRv3?

Thanks,

Viraj.

[Neal Cardwell]

Hi,

Thanks for the post!

On Sat, Mar 15, 2025 at 2:37 PM Viraj Alankar <vala...@gmail.com> wrote:

Hello,

I've been using the BBRv3 kernel source at https://github.com/google/bbr/tree/v3 and the results are pretty amazing for a high latency network. Great work!

Thanks! We are glad it worked well for you!

 

I'm wondering if it's possible to use this with a newer kernel, such as with the latest stable v6.13.7? Does it involve just copying in tcp_bbr.c? It sounds like there are other patches as well.

As you note, there are multiple patches involved, so unfortunately it's not as simple as copying over the tcp_bbr.c file. Generally, it requires using either "git rebase" or "git cherry-pick" to port over the entire patch series to the new source tree, resolving conflicts as necessary in spots where the Linux source tree and iproute2 source tree have changed.

I have rebased the Linux TCP BBRv3 code from net-next at ba2274dcfda859b8a27193e68ad37bfe4da28ddc (July 2023, shortly after Linux 6.4) to the v6.13.7 you mention, since v6.13.7 is the latest stable release right now (v6.13.7 is from Mar 13, 2025).

Please give it a try and let me know how it goes!

Also, it is unclear to me whether the davem/net-next kernel tree contains BBRv3 or not. Comparing the source it seems not so. Is that correct?

That's correct. The net-next and mainline Linux trees still have BBR 1.x.

 

Basically, what's the best way to use the latest stable kernel tree with BBRv3?

After today's rebase you can use the "v3" branch from https://github.com/google/bbr.git using the same instructions as before:

  https://github.com/google/bbr/blob/v3/README.md

best regards,

neal

 

Thanks,

Viraj.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bbr-dev/6f1e3420-e99e-46cb-9f25-0302dbed02f2n%40googlegroups.com.

[Viraj Alankar]

I just tried the latest rebase and it works great. Thank you!

[Neal Cardwell]

Great to hear. Thanks for testing it!

neal

To view this discussion visit https://groups.google.com/d/msgid/bbr-dev/3b871e9f-5822-4c44-9ad8-19435c830f11n%40googlegroups.com.

[Oleksandr Natalenko]

Hello.

Just to note, I've been forward-porting BBRv3 for every kernel release here: https://codeberg.org/pf-kernel/linux. I've been running BBRv3 on 6 hosts 24/7 for a long time with no issues.

Neal, with regard to this report I posted a while ago: https://lore.kernel.org/lkml/4616579.L...@natalenko.name/T/#u Could you please check my "fix" here https://codeberg.org/pf-kernel/linux/commit/e8f5232a9e6c556efa21c8dd6a9c0f96423ec5e7 and share your thoughts please?

Thank you.

[Neal Cardwell]

On Mon, Mar 17, 2025 at 8:45 AM Oleksandr Natalenko <pfa...@gmail.com> wrote:

Hello.

Just to note, I've been forward-porting BBRv3 for every kernel release here: https://codeberg.org/pf-kernel/linux. I've been running BBRv3 on 6 hosts 24/7 for a long time with no issues.

Great to hear. Thanks for sharing!

 

Neal, with regard to this report I posted a while ago: https://lore.kernel.org/lkml/4616579.L...@natalenko.name/T/#u Could you please check my "fix" here https://codeberg.org/pf-kernel/linux/commit/e8f5232a9e6c556efa21c8dd6a9c0f96423ec5e7 and share your thoughts please?

Yes, those fixes look fine. I'm surprised clang does not complain about this line in bbr_check_ecn_too_high_in_startup() as well:

     !bbr_param(sk, full_ecn_cnt) || !bbr_param(sk, ecn_thresh))

That check is also OK to remove.

A note about the rationale: in our internal version of BBR those parameters are per-socket variables that can be dynamically changed for A/B experiments. The bbr_param() macro is an attempt to get sensible behavior both (a) upstream, where these are constants, and (b) in our version internally, where these are per-socket variables. This works for gcc, which we use internally, but not for clang.

neal

 

Thank you.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/bbr-dev/32b0eec5-98cd-4cb8-8b65-d7e2f3bcfbfan%40googlegroups.com.

[Oleksandr Natalenko]

Hello.

On Monday, 17 March 2025 at 15:13:57 UTC+1 Neal Cardwell wrote:

Yes, those fixes look fine. I'm surprised clang does not complain about this line in bbr_check_ecn_too_high_in_startup() as well:

     !bbr_param(sk, full_ecn_cnt) || !bbr_param(sk, ecn_thresh))

That check is also OK to remove.

Does https://codeberg.org/pf-kernel/linux/commit/16f8ea549e2d93fe165713587546925d42f8bdaf look fine?

 

A note about the rationale: in our internal version of BBR those parameters are per-socket variables that can be dynamically changed for A/B experiments. The bbr_param() macro is an attempt to get sensible behavior both (a) upstream, where these are constants, and (b) in our version internally, where these are per-socket variables. This works for gcc, which we use internally, but not for clang.

I see, thank you for the explanation. I suspected something like this.

[Oleksandr Natalenko]

Hello.

On Monday, 17 March 2025 at 15:13:57 UTC+1 Neal Cardwell wrote:

I'm surprised clang does not complain about this line in bbr_check_ecn_too_high_in_startup() as well:

     !bbr_param(sk, full_ecn_cnt) || !bbr_param(sk, ecn_thresh))

I think the difference that makes clang stay silent here is negation. I've found out that if places that trigger the warning are prepended with double negation, the warnings disappear too.

Double negation effectively converts an integer to a boolean. So something like this may also work:

diff --git a/net/ipv4/tcp_bbr.c b/net/ipv4/tcp_bbr.c
index 9b31b4e5943ef..a12b1f79d0355 100644
--- a/net/ipv4/tcp_bbr.c
+++ b/net/ipv4/tcp_bbr.c
@@ -1078,7 +1078,7 @@ static int bbr_update_ecn_alpha(struct sock *sk)
 
  /* See if we should use ECN sender logic for this connection. */
  if (!bbr->ecn_eligible && bbr_can_use_ecn(sk) &&
-     bbr_param(sk, ecn_factor) &&
+     !!bbr_param(sk, ecn_factor) &&
      (bbr->min_rtt_us <= bbr_ecn_max_rtt_us ||
       !bbr_ecn_max_rtt_us))
  bbr->ecn_eligible = 1;
@@ -1185,7 +1185,7 @@ static bool bbr_is_inflight_too_high(const struct sock *sk,
  }
 
  if (rs->delivered_ce > 0 && rs->delivered > 0 &&
-     bbr->ecn_eligible && bbr_param(sk, ecn_thresh)) {
+     bbr->ecn_eligible && !!bbr_param(sk, ecn_thresh)) {
  ecn_thresh = (u64)rs->delivered * bbr_param(sk, ecn_thresh) >>
  BBR_SCALE;
  if (rs->delivered_ce > ecn_thresh) {
@@ -1383,7 +1383,7 @@ static void bbr_adapt_lower_bounds(struct sock *sk,
  return;
 
  /* ECN response. */
- if (bbr->ecn_in_round && bbr_param(sk, ecn_factor)) {
+ if (bbr->ecn_in_round && !!bbr_param(sk, ecn_factor)) {
  bbr_init_lower_bounds(sk, false);
  bbr_ecn_lower_bounds(sk, &ecn_inflight_lo);
  }

What do you think?

Thanks.

[Neal Cardwell]

On Mon, Mar 17, 2025 at 12:54 PM Oleksandr Natalenko <pfa...@gmail.com> wrote:

Hello.

On Monday, 17 March 2025 at 15:13:57 UTC+1 Neal Cardwell wrote:

Yes, those fixes look fine. I'm surprised clang does not complain about this line in bbr_check_ecn_too_high_in_startup() as well:

     !bbr_param(sk, full_ecn_cnt) || !bbr_param(sk, ecn_thresh))

That check is also OK to remove.

Does https://codeberg.org/pf-kernel/linux/commit/16f8ea549e2d93fe165713587546925d42f8bdaf look fine?

Yes, that looks fine.

thanks,

neal

 

 

A note about the rationale: in our internal version of BBR those parameters are per-socket variables that can be dynamically changed for A/B experiments. The bbr_param() macro is an attempt to get sensible behavior both (a) upstream, where these are constants, and (b) in our version internally, where these are per-socket variables. This works for gcc, which we use internally, but not for clang.

I see, thank you for the explanation. I suspected something like this.

--

You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/bbr-dev/9fe3a325-7f1e-4a13-82fc-611628facf48n%40googlegroups.com.

[Neal Cardwell]

Sounds good. Thanks! I have incorporated your fix into tbe v3 code on github. Seems to compile with both gcc and clang (make LLVM=1 net/ipv4/tcp_bbr.o) now.

Thanks,

neal

 

Thanks.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/bbr-dev/9335db3d-6baa-4f9f-954f-f958228382adn%40googlegroups.com.

[X Y]

Hello,

Perhaps some silly questions, but I'd still ask for some clarifications on BBRv3's integration into Linux.

Does kernel 6.13.7 (https://kernel.ubuntu.com/mainline/v6.13.7/) have BBRv3 automatically included? I'm asking because we had trouble following your procedure on Ubuntu 24.04 and we did a "normal" kernel update. Additionally, I'm interested in how to check which version of BBR is on the system?

Is it possible to install BBRv3 on kernel 6.8 as well? If so, is there a procedure described anywhere?

Regards,
Xandy

[Neal Cardwell]

On Mon, Mar 24, 2025 at 9:22 AM X Y <stro...@gmail.com> wrote:

Hello,

Perhaps some silly questions, but I'd still ask for some clarifications on BBRv3's integration into Linux.

Does kernel 6.13.7 (https://kernel.ubuntu.com/mainline/v6.13.7/) have BBRv3 automatically included?

No. :-)

 

I'm asking because we had trouble following your procedure on Ubuntu 24.04 and we did a "normal" kernel update. Additionally, I'm interested in how to check which version of BBR is on the system?

If you are using BBR from mainline Linux it will be BBRv1. If you are using BBR from https://github.com/google/bbr/blob/v3/README.md it will be BBRv3.

 

Is it possible to install BBRv3 on kernel 6.8 as well? If so, is there a procedure described anywhere?

It's possible to rebase/cherry-pick BBRv3 onto 6.8, with some effort. There's no procedure described anywhere for that AFAIK.

You may want to check with Oleksandr Natalenko <pfa...@gmail.com>, who mentioned earlier in this thread: "I've been forward-porting BBRv3 for every kernel release here: https://codeberg.org/pf-kernel/linux".

neal

 

To view this discussion visit https://groups.google.com/d/msgid/bbr-dev/4f94cfb8-854d-49c0-90ff-eaf41b556314n%40googlegroups.com.

[Viraj Alankar]

I recommend compiling the kernel from source. However, FWIW I've created kernel packages for my own Debian VM/cloud installations at:

https://github.com/valankar/ansible/tree/master/kernel

Keep in mind you'd be installing random binaries I created, and I don't recommend using them for anything other than testing. They are using the Debian 12 cloud configs which are pretty barebones and meant for VMs. I created them just for convenience in my installs and ansible.

Viraj.