In Which Cases BBR is not Recommended?

[Nahum Nir]

Hi,

While trying to fine tune my server I started reading about BBR. Is there any topology or use cases where other TCP algorithm are better then BBR?

Are there any measurements about BBR's CPU/RAM usage?

Thanks,

Nir 

[Roland Bless]

Hi,

That depends. If BBR is used in a scenario where your bottleneck buffer
is smaller than a "BDP", several BBR flows may cause massive packet loss
whereas a single BBR flow may behave fine. Other traffic, e.g., CUBIC
TCP flows may be completely suppressed in this case. However, if your
sender is application-limited the situation can be better. Since BBR
ignores loss as congestion signal, it can keep throughput high even
if packet loss occurs. That helps in situations where loss in not caused
by congestion. More answers can be found here:
http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
In some cases delay-based TCP variants may keep queuing delay
lower than BBR.

However, BBR is under development so its behavior may change.
Neal mentioned that BBR will probably not ignore loss as congestion
signal in the future.

Regards,
Roland

[Nahum Nir]

Hi Roland,

Thank you for detailed answer.

I was able to create a simple test were CUBIC was better (16MB files, LFN, client is on WiFi).

Are there any tools or measurements about servers' CPU/RAM usage?

Best,

Nir 

[Roland Bless]

Hi,

On 12.11.2017 at 18:24 Nahum Nir wrote:
> Thank you for detailed answer.
> I was able to create a simple test were CUBIC was better (16MB files,
> LFN, client is on WiFi).
>
> Are there any tools or measurements about servers' CPU/RAM usage?

I don't think that RAM usage is an issue, but we have a tool
that we use for measuring (interface) throughput and CPU usage:
https://git.scc.kit.edu/CPUnetLOG/CPUnetLOG

However, it is used mostly as sanity check
to see whether the server is operating under severe overload conditions.

Best regards,
Roland

[Neal Cardwell]

On Sat, Nov 11, 2017 at 2:23 PM, Nahum Nir <hello.s...@gmail.com> wrote:

Hi,

While trying to fine tune my server I started reading about BBR. Is there any topology or use cases where other TCP algorithm are better then BBR?

 

The initial release of BBR (BBR v1.0, if you will) had, in our experience, excellent performance in most cases. However, it  caused high loss with multiple bulk BBR flows in shallow buffers (examined by Roland and his co-authors in their paper), and low throughput on wifi LANs (where the min RTT is ~1ms but the typical RTT is 10x that). These issues have been discussed elsewhere on the bbr-dev list.

We've been actively working on both issues, and we'll talk about our recent and upcoming work for BBR v2.0 at the ICCRG session at the IETF. You can watch the talk tonight/tomorrow (depending on your time zone) at https://ietf100.conf.meetecho.com/  or check out the slides and video after the talk.

Are there any measurements about BBR's CPU/RAM usage?

For Linux TCP, the difference in CPU usage vs CUBIC is small, in our experience. To give some quick examples, here are some CPU usage numbers from LInux systems with 10Gbps NICs. The CPU utilization difference (reported by "netperf -c") tends to be small, with BBR being less efficient with a single TCP flow and more efficient with multiple concurrent flows (the more typical case for production servers). Here are some numbers, with four trials for each CC:

1 netperf stream transfer:

BBR:   1.65%  1.18%  0.45%  1.00%

CUBIC: 0.91%  0.54%  0.52%  1.22%

1000 concurrent netperf stream transfers:

BBR:    1.6%  1.4%  1.7%  1.7%

CUBIC:  2.1%  1.8%  2.0%  2.2%

As for RAM usage, Linux TCP BBR"s RAM usage should not be higher than that of CUBIC, since BBR allocates no extra memory. Connections using BBR use the exact same slab-allocated tcp_sock structure used by CUBIC connections.

neal

[Nahum Nir]

Thank you Neal and thanks to Roland as well.

>> "BBR will probably not ignore loss as congestion signal in the future."

Do you think that one day BBR will be able to handle all topology/use-cases or that for fine tuning other CC algorithms might be better?

Regards,

Nahum

[Neal Cardwell]

On Sun, Nov 12, 2017 at 11:27 AM, Nahum Nir <hello.s...@gmail.com> wrote:

Thank you Neal and thanks to Roland as well.

>> "BBR will probably not ignore loss as congestion signal in the future."

Do you think that one day BBR will be able to handle all topology/use-cases or that for fine tuning other CC algorithms might be better?

Yes, based on how our team's research is proceeding, my sense is that we'll be able to get BBR to work as well or better than Reno and CUBIC in the common scenarios for which those algorithms are used today.

neal

[Klatsky, Carl]

Neal,

Does any of the current "BBR v2.0" work items address the issues being raised in the paper from Roland's team? Have those finding's been replicated in Google labs?

Regards,
Carl Klatsky

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Neal Cardwell]

On Thu, Nov 16, 2017 at 9:39 AM, Klatsky, Carl <Carl_K...@comcast.com> wrote:

Neal,

Does any of the current "BBR v2.0" work items address the issues being raised in the paper from Roland's team?  Have those finding's been replicated in Google labs?

Yes, most of the work in BBR v2.0 is specifically targeted at the (already-known) issues described in Roland's team's paper. We are able to verify in lab settings that the approaches in the pre-release BBR v2.0 significantly improve the situation. For more details, please check out the slides and video for the BBR talk at the ICCRG session at IETF 100:

 - [YouTube] (BBR starts at 44:00)

 - [slides]

cheers,

neal

 

-----Original Message-----
From: bbr...@googlegroups.com [mailto:bbr-dev@googlegroups.com] On Behalf Of Roland Bless
Sent: Saturday, November 11, 2017 11:31 PM
To: Nahum Nir <hello.s...@gmail.com>; BBR Development <bbr...@googlegroups.com>
Subject: Re: [bbr-dev] In Which Cases BBR is not Recommended?

Hi,

On 12.11.2017 at 03:23 Nahum Nir wrote:
> While trying to fine tune my server I started reading about BBR. Is
> there any topology or use cases where other TCP algorithm are better
> then BBR?
> Are there any measurements about BBR's CPU/RAM usage?

That depends. If BBR is used in a scenario where your bottleneck buffer is smaller than a "BDP", several BBR flows may cause massive packet loss whereas a single BBR flow may behave fine. Other traffic, e.g., CUBIC TCP flows may be completely suppressed in this case. However, if your sender is application-limited the situation can be better. Since BBR ignores loss as congestion signal, it can keep throughput high even if packet loss occurs. That helps in situations where loss in not caused by congestion. More answers can be found here:
http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
In some cases delay-based TCP variants may keep queuing delay lower than BBR.

However, BBR is under development so its behavior may change.
Neal mentioned that BBR will probably not ignore loss as congestion signal in the future.

Regards,
 Roland

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Nahum Nir]

Can you please share your lab setup? Other people may also want to reproduce the tests.

[Neal Cardwell]

On Thu, Nov 16, 2017 at 10:01 AM, Nahum Nir <hello.s...@gmail.com> wrote:

Can you please share your lab setup? Other people may also want to reproduce the tests.

For testing BBR our team uses a set of internal scripts that we have not  yet had time to open-source.

IMHO the simplest way to investigate this kind of behavior is using the Linux netem qdisc. The main trick is to remember to set the netem "limit" to be big enough for the BDP of the emulated pipe plus the buffer you want it to emulate.

Hope that helps,

neal

[Nahum Nir]

Cool, thanks! 

[Jim Gettys]

On Thu, Nov 16, 2017 at 12:49 PM, Nahum Nir <hello.s...@gmail.com> wrote:

Cool, thanks! 

On Nov 16, 2017 7:45 PM, "Neal Cardwell" <ncar...@google.com> wrote:

On Thu, Nov 16, 2017 at 10:01 AM, Nahum Nir <hello.s...@gmail.com> wrote:

Can you please share your lab setup? Other people may also want to reproduce the tests.

For testing BBR our team uses a set of internal scripts that we have not  yet had time to open-source.

IMHO the simplest way to investigate this kind of behavior is using the Linux netem qdisc. The main trick is to remember to set the netem "limit" to be big enough for the BDP of the emulated pipe plus the buffer you want it to emulate.

Hope that helps,

neal

​Be careful with netem; it can hurt you if you are not careful.

Some of the "hard knocks" we've had with netem and other knobs in Linux are documented here:

https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#the-netem-qdisc-does-not-work-in-conjunction-with-other-qdiscs

                                       - Jim

                                                   - Jim

[James]

Hi Neal,

On Thu, Nov 16, 2017 at 9:39 AM, Klatsky, Carl <Carl_K...@comcast.com> wrote:

Neal,

Does any of the current "BBR v2.0" work items address the issues being raised in the paper from Roland's team?  Have those finding's been replicated in Google labs?

Yes, most of the work in BBR v2.0 is specifically targeted at the (already-known) issues described in Roland's team's paper. We are able to verify in lab settings that the approaches in the pre-release BBR v2.0 significantly improve the situation. For more details, please check out the slides and video for the BBR talk at the ICCRG session at IETF 100:

 - [YouTube] (BBR starts at 44:00)

 - [slides]

Is there any plan to upstream BBR v2.0?

Regards

[Neal Cardwell]

Yes, absolutely. When BBR v2.0 is ready we'll send it upstream to the open source community. For the moment we are still refining and testing it.

thanks,

neal

 

Regards