Can I use BBR with fq_codel?

[naner...@gmail.com]

I know in the README, BBR team suggests fq, but does using fq_codel has any negative impact?

Thanks,

Eric

[Neal Cardwell]

BBR requires pacing. The Linux fq_codel qdisc does not implement pacing, so fq_codel would not be sufficient.

So the note in the tcp_bbr.c code is still current and complete:

   NOTE: BBR *must* be used with the fq qdisc ("man tc-fq") with pacing enabled,
   since pacing is integral to the BBR design and implementation.
   BBR without pacing would not function properly, and may incur unnecessary
   high packet loss rates.

neal

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[naner...@gmail.com]

Thank you, Neal.

Another question:

How does ECN impact BBR? Should I disable ECN?

在 2017年1月16日星期一 UTC+8下午10:43:08，Neal Cardwell写道：

BBR requires pacing. The Linux fq_codel qdisc does not implement pacing, so fq_codel would not be sufficient.

So the note in the tcp_bbr.c code is still current and complete:

   NOTE: BBR *must* be used with the fq qdisc ("man tc-fq") with pacing enabled,
   since pacing is integral to the BBR design and implementation.
   BBR without pacing would not function properly, and may incur unnecessary
   high packet loss rates.

neal

On Mon, Jan 16, 2017 at 2:18 AM, <naner...@gmail.com> wrote:

I know in the README, BBR team suggests fq, but does using fq_codel has any negative impact?

Thanks,

Eric

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

[Jonathan Morton]

> On 17 Jan, 2017, at 07:05, naner...@gmail.com wrote:
>
> How does ECN impact BBR? Should I disable ECN?

I believe the current Linux implementation of BBR ignores ECN. Given that, it would be more RFC-compliant to set "sysctl net.ipv4.tcp_ecn=0”. You can (and should) leave it on in any qdiscs you’re running.

- Jonathan Morton

[Jim Gettys]

On Mon, Jan 16, 2017 at 9:42 AM, 'Neal Cardwell' via BBR Development <bbr...@googlegroups.com> wrote:

BBR requires pacing. The Linux fq_codel qdisc does not implement pacing, so fq_codel would not be sufficient.

​Patches gratefully accepted.  Also, there is a new qdisc based on fq_codel called "cake" that could use pacing, that is particularly useful at the edge of the network.  And the new code for WiFi that is showing stunning results could also use pacing magic, I expect.

The network you may improve may be close to home ;-).

                                                           - Jim

[Eric Dumazet]

Few observations, just in case these points are not already known.

1) fq_codel purpose is to be used at the bottleneck point. Typically a router, an access point.

2) fq_codel is based on a stochastic hash. Multiple flows can collide in a hash bucket.

3) fq is implementing pacing, and is meant for end hosts. It was designed with performance in mind.

   Codel was not added there, because it was not needed (see point 1)), and only adding wasting cpu cycles.

4) Most of end hosts can usually send much faster than the bottlenecks their TCP flow might reach.

5) BBR just reacts fine in the rare case the bottleneck is right at the host itself.

In other words, you can install fq on your laptop or typical linux device, then you can use BBR right now.

For the few people that absolutely want their Access Point being able to initiate its own TCP flows,

and these TCP flows might saturate the bottleneck,

we will add a fallback to TCP stack.

TCP stack will implement its own pacing when it detects FQ is not installed/used.

At a performance cost, but who cares really.

 

On Tuesday, January 17, 2017 at 7:22:48 AM UTC-8, Jim Gettys wrote:

On Mon, Jan 16, 2017 at 9:42 AM, 'Neal Cardwell' via BBR Development <bbr...@googlegroups.com> wrote:

BBR requires pacing. The Linux fq_codel qdisc does not implement pacing, so fq_codel would not be sufficient.

​Patches gratefully accepted.  Also, there is a new qdisc based on fq_codel called "cake" that could use pacing, that is particularly useful at the edge of the network.  And the new code for WiFi that is showing stunning results could also use pacing magic, I expect.

The network you may improve may be close to home ;-).

                                                           - Jim

​

 

So the note in the tcp_bbr.c code is still current and complete:

   NOTE: BBR *must* be used with the fq qdisc ("man tc-fq") with pacing enabled,
   since pacing is integral to the BBR design and implementation.
   BBR without pacing would not function properly, and may incur unnecessary
   high packet loss rates.

neal

On Mon, Jan 16, 2017 at 2:18 AM, <naner...@gmail.com> wrote:

I know in the README, BBR team suggests fq, but does using fq_codel has any negative impact?

Thanks,

Eric

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

[naner...@gmail.com]

BBR+FQ:

BBR+FQ_CODEL:

在 2017年1月17日星期二 UTC+8下午11:42:49，Eric Dumazet写道：

[Eric Dumazet]

Please, do not post graphs or anything with BBR+<whatever qdisc that is not FQ>.

If you want to impress your friends, just blast packets with UDP and no congestion control.

You will get line rate.

Repeat : BBR without proper pacing (which currently is only ensured by FQ) is not supported.
You are likely to have serious problems if you plan doing that at scale.

 https://www.ietf.org/rfc/rfc2119.txt is a good resource in case the words "MUST"  or "REQUIRED" are not familiar.

[Zagfai Kwong]

I don't understand.

It seems that BBR with fq_codel runs well.

[Eric Dumazet]

BBR will run well with fq_codel only after linux-4.13 (not yet released)

Before linux-4.13, BBR _needs_  fq packet scheduler.

fq is not fq_codel.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Zagfai Kwong]

But in fact it runs well with fq_codel in linux 4.9.

I use BBR for a high loss rate network which high up to 15%, and it solve the problem using cubie.

[Eric Dumazet]

This is not a supported configuration.

If many people/hosts were using this configuration, danger of hurting Internet traffic would be high.

If you want to blast packets, just use UDP and be happy.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Neal Cardwell]

It depends on what one means by "runs well". :-)

It can be said that BBR "runs well" with fq_codel in linux 4.9, in the sense of achieving high throughput. But, as Eric points out, the problem is that without some sort of pacing BBR will inject too many packets into the network too quickly. In that sense, BBR without pacing does not run well.

See section 4.3.4.4. "Restarting From Idle" in the BBR Internet Draft for a discussion of related issues:

  https://tools.ietf.org/html/draft-cardwell-iccrg-bbr-congestion-control-00#section-4.3.4

Without pacing in place, BBR would behave like the "BDP-scale line-rate burst" approach mentioned in that section.

And beyond that, BBR without pacing keeps more packets in flight than the algorithm intends (because the algorithm periodically uses lower pacing rates to drain packets out of the bottleneck queue).

neal

[Zagfai Kwong]

Thank you for your explanation.

That is to say, it would affect other users seriously in some case that bbr+fq_codel.

Thank you.

[martin]

as of 2023, end-host-context, kernel 6.1.50-0-lts: is it OK now to use bbr+fq_codel (+sysctl net.ipv4.tcp_ecn=2 (default))?

[Neal Cardwell]

On Sat, Sep 2, 2023 at 4:09 PM martin <llig...@gmail.com> wrote:

as of 2023, end-host-context, kernel 6.1.50-0-lts: is it OK now to use bbr+fq_codel (+sysctl net.ipv4.tcp_ecn=2 (default))?

Yes, it is fine to use BBR with fq_codel on recent kernels.

For kernels v4.20 and later, BBR will use the Linux TCP-layer pacing if the connection notices that there is no qdisc on the sending host implementing pacing. So one reasonable configuration for v4.20 and later is BBR for congestion control and the fq_codel qdisc for queuing (though hopefully BBR would not typically create queues long enough for fq_codel to start dropping...). 

best regards,

neal

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/78a6ec25-055d-410f-ae12-aa76b23e2618n%40googlegroups.com.

[MUHAMMAD AHSAN]

I dont find any "pacing" option in CAKE qdisc ( as per trailing email says it has ) , can anyone share if it has?

https://man7.org/linux/man-pages/man8/tc-cake.8.html

rgds,

Ahsan

On Tuesday, January 17, 2017 at 8:22:48 PM UTC+5 j...@freedesktop.org wrote:

On Mon, Jan 16, 2017 at 9:42 AM, 'Neal Cardwell' via BBR Development <bbr...@googlegroups.com> wrote:

BBR requires pacing. The Linux fq_codel qdisc does not implement pacing, so fq_codel would not be sufficient.

​Patches gratefully accepted.  Also, there is a new qdisc based on fq_codel called "cake" that could use pacing, that is particularly useful at the edge of the network.  And the new code for WiFi that is showing stunning results could also use pacing magic, I expect.

The network you may improve may be close to home ;-).

                                                           - Jim

​

 

So the note in the tcp_bbr.c code is still current and complete:

   NOTE: BBR *must* be used with the fq qdisc ("man tc-fq") with pacing enabled,
   since pacing is integral to the BBR design and implementation.
   BBR without pacing would not function properly, and may incur unnecessary
   high packet loss rates.
neal

On Mon, Jan 16, 2017 at 2:18 AM, <naner...@gmail.com> wrote:

I know in the README, BBR team suggests fq, but does using fq_codel has any negative impact?

Thanks,

Eric

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

[Eric Dumazet]

For any qdisc other than sch_fq (the one which has pacing), TCP stack
falls back to internal pacing,

Internal pacing should mostly work, but for high demand workloads, or
some hosts/architectures with slow hrtimers.

> To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/eb5ca4c7-a8fc-4a98-869e-368c752db73bn%40googlegroups.com.