How to comfirm BBRv1 works with HTB+FQ?

[Tao Tse]

Hi,

As I know, BBRv1 ONLY works well with FQ (tc qdisc add dev eth0 fq), for the pacing machanism.

Now, I need more traffic control strategies on NICs, the classful queueing discipline HTB is needed. So I did the configuration as follow:

tc qdisc add dev eth0 root handle 1:  htb default 2

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 100mbit ceil 150mbit prio 2

tc qdisc add dev eth0 parent 1:3 fq

tc filter add dev eth0 protocol ip parent 1:0 u32 match ip src x.x.x.x flowid 1:3

According to the test result, the throughput on eth0 behaves as the class described above. But how could I know whether this works fine with the BBR congestion control ?

--

xtao

[Neal Cardwell]

On Tue, Jun 1, 2021 at 4:07 AM Tao Tse <g.xi...@gmail.com> wrote:

Hi,

As I know, BBRv1 ONLY works well with FQ (tc qdisc add dev eth0 fq), for the pacing machanism.

That information is out of date. :-)

Thanks to code by Eric Dumazet, since Linux kernel 4.20 (2018), for BBR flows the Linux TCP stack will use a pacing mechanism internal to TCP if the network stack detects that there is no fq qdisc handling pacing.

This means that you can use any qdisc setup you want with BBR.

This is documented in two spots:

(1) in the quick-start guide at:

  https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md

...which says:

"TCP BBR is in Linux v4.9 and beyond. However, we recommend compiling from the latest sources, from the networking development branch. In particular, the davem/net-next networking development branch (and Linux v4.20 and beyond) support TCP-level pacing. This means that there is no longer a strict requirement to install the "fq" qdisc to use BBR. Any qdisc will work, though "fq" performs better for highly-loaded servers. (Note that TCP-level pacing was added in v4.13-rc1 but did not work well for BBR until a fix was added in 4.20.)"

(2) in the tcp_bbr.c code in a comment near the top:

 * NOTE: BBR might be used with the fq qdisc ("man tc-fq") with pacing enabled,
 * otherwise TCP stack falls back to an internal pacing using one high
 * resolution timer per TCP socket and may use more resources.

 

Now, I need more traffic control strategies on NICs, the classful queueing discipline HTB is needed. So I did the configuration as follow:

tc qdisc add dev eth0 root handle 1:  htb default 2

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 100mbit ceil 150mbit prio 2

tc qdisc add dev eth0 parent 1:3 fq

tc filter add dev eth0 protocol ip parent 1:0 u32 match ip src x.x.x.x flowid 1:3

According to the test result, the throughput on eth0 behaves as the class described above. But how could I know whether this works fine with the BBR congestion control ?

I cannot vouch for your particular qdisc setup. However, in general, if your machine is running a 4.20 kernel or newer, any qdisc setup that works for any congestion control should also work for BBR.

best,

neal

[Tao Tse]

neal,

Thanks for your detailed infomation!

I've really missed a lot.

xtao

[Tao Tse]

Hi,

Further more question here.

As the internal-pacing commit described, FQ consumes less interrupts. So, what's FQ's approach to achieve this? Why not the TCP internal pacing goes in the same way?

---

xtao

On Tuesday, June 1, 2021 at 11:59:58 PM UTC+8 Neal Cardwell wrote:

[Eric Dumazet]

On Mon, Jun 7, 2021 at 5:51 AM Tao Tse <g.xi...@gmail.com> wrote:
>
> Hi,
>

> Further more question here.
> As the internal-pacing commit described, FQ consumes less interrupts. So, what's FQ's approach to achieve this? Why not the TCP internal pacing goes in the same way?

Because FQ has the assumption of being used in MQ+FQ setups, with a
strong affinity of flows per TX queue (per CPU)

TCP has no such knowledge/assumptions.

Sharing a timer with multiple TCP flows, possibly handled by different
CPUS would be a performance killer.

HR timers scale reasonably well, you do not have to worry,

> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/7cc73d37-80ab-4a8d-936a-4002e70dcf35n%40googlegroups.com.

[Neal Cardwell]

To expand on Eric's (excellent and authoritative) answer and add some more details for other parts of the question:

> So, what's FQ's approach to achieve this?

The fq qdisc has low overhead partly because it has a single (high resolution) pacing timer per fq qdisc instance, and there is generally a single fq qdisc per NIC tx queue.

For the fq disc, by the time packets reach the qdisc the locking has already happened to allow the shared fq pacing timer manipulation to happen without any added overhead.

To tie this back to Eric's point, for TCP, as Eric notes, there is no equivalent locking in place, and adding such locking to coordinate sharing a pacing timer between TCP connections would have a prohibitive performance penalty.

best,

neal

To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CANn89iKD5pPGQEvFa2NatTbxPDzv7Z9gzDyiGF8C2%2BkLvM54yA%40mail.gmail.com.

[Tao Tse]

Thanks to Neal & Eric for your explanation!

In other words:

 * For fq qdisc, it has fix number of pacing timers in general, so the frequency of interrupt triggered is limited.

 * For internal pacing, there's a pacing timer for every single TCP instance, as there are more connections, there will be more timers, interrupts comes more.

right?

---

xtao

[Neal Cardwell]

On Tue, Jun 8, 2021 at 7:55 AM Tao Tse <g.xi...@gmail.com> wrote:

Thanks to Neal & Eric for your explanation!

In other words:

 * For fq qdisc, it has fix number of pacing timers in general, so the frequency of interrupt triggered is limited.

 * For internal pacing, there's a pacing timer for every single TCP instance, as there are more connections, there will be more timers, interrupts comes more.

right?

Yes. :-)

neal

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/bd336d29-3f5e-4986-b539-5ee2d7cb6283n%40googlegroups.com.