BBR vs Cubic on Wifi network

[Francis Rammeloo]

Hi,

I have run some Wifi tests where I make a Linux laptop compete with a Samsung phone. The Linux laptop runs Ubuntu 17.04 with Linux 4.10 kernel and has BBR+FQ enabled. The setup was created in a mostly isolated location (the cellar of the company building) where both devices connect to the same Wifi access point.

I've run a few tests where sometimes the laptop was configured with BBR and sometimes with Cubic.

After running multiple tests my main findings are:

• BBR never seems to reach full line rate. It peaks at around 90-95%. (It reaches 450Mbit/s, while Cubic reaches 500Mbit/s.)
• When competing with another device the throughput drops to ~5Mbit/s (coming from from ~450Mbit/s)

To illustrate this I've attached a TCP graph which shows the two competing flows. Each flow runs for two minutes and there's one minute of overlap in the middle. The Samsung flow clearly drowns out the BBR flow.

Is there any way I can improve the performance of the TCP BBR flow in this situation? Are these problems that are inherent to BBR or can they be improved with bugfixing or parametrization?

Best regards,

Francis

[Francis Rammeloo]

For completeness I've also attached the results where the laptop is configured with cubic.

[Eric Dumazet]

Hi Francis

Looks like the Samsung kills the laptop even when laptop is using Cubic ?

Better use the same hardware for your BBR competes with Cubic tests maybe ?

Who knows, maybe the wifi chip on your laptop was tuned assuming only
UDP and TCP cubic would be used ;)

> --
> You received this message because you are subscribed to the Google Groups
> "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbr-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Francis Rammeloo]

Hi Eric,

But at least with Cubic we still get 150Mbps while the BBR flow is really starved.  Isn't starvation something that should be avoided?

Francis

[Neal Cardwell]

Hi Francis,

Thanks for your test results!

These problems are not inherent to the BBR algorithm. I strongly
suspect that these wifi tests are running into the known issue where
the current upstream BBR parameters for provisioning cwnd need to be
more generous for paths like wifi LAN paths, which have ACKs that are
often delayed much longer than the minimum RTT observed over the path.
This issue is discussed in the "BBR test on wireless path" thread from
Jan 11:

https://groups.google.com/d/msg/bbr-dev/zUrcENm9rZI/Ea28juVoFAAJ

We are actively working on tuning this aspect of the BBR code.

A few quick questions:

+ What's the model of Linux laptop?

+ What's the destination of the TCP transfer? (A remote host on the
Internet somewhere? Or locally connected to the wifi access point via
wifi or ethernet?)

+ Would you be able to post a tcpdump trace of a few seconds of the
TCP BBR flow from the Linux laptop while it is competing with the flow
from the Samsung phone? We'd like to verify that the behavior you're
seeing here matches the known behavior we're working on. Perhaps
something like:

tcpdump -w /tmp/test.pcap -s 120 -i $DEVICE -c 100000 port $PORT

Thanks!
neal


On Tue, Apr 25, 2017 at 8:59 AM, Francis Rammeloo

[Francis Rammeloo]

Hi Neal,

To answer your questions:

• The Linux laptop is a Latitude E5570 which contains an Intel Wireless 8260 chip.
• The destination is a machine (traffic generator) that is locally connected to the access point via Ethernet.
• The pcap is attached. Strangely enough the performance was slightly better this time (~50Mbit/s).

I hope this helps.

Best regards,

Francis

[Eric Dumazet]

When you say Cubic, is this with FQ as the packet scheduler ?

We know that TCP small queues + FQ + wifi is not a good combination
today, regardless of BBR being used or not.

( This is a well known problem, that might soon have a fix )


On Tue, Apr 25, 2017 at 8:02 AM, Francis Rammeloo

[Francis Rammeloo]

On Tuesday, April 25, 2017 at 5:31:07 PM UTC+2, Eric Dumazet wrote:

When you say Cubic, is this with FQ as the packet scheduler ?

Yes. I'm using FQ both for Cubic and BBR.

 

We know that TCP small queues + FQ + wifi is not a good combination
today, regardless of BBR being used or not.

( This is a well known problem, that might soon have a fix )

 Ok. I'll give it a try without FQ.

[Francis Rammeloo]

 

We know that TCP small queues + FQ + wifi is not a good combination
today, regardless of BBR being used or not.

( This is a well known problem, that might soon have a fix )

 Ok. I'll give it a try without FQ.

Ok that seems to have made no difference for  BBR or Cubic :)

[Neal Cardwell]

Hi Francis,

Thanks for the BBR packet trace! Yes, this seems to be this known
issue ("where the current upstream BBR parameters for provisioning

cwnd need to be more generous for paths like wifi LAN paths, which
have ACKs that are often delayed much longer than the minimum RTT

observed over the path").

The attached graph shows an excerpt of a time/sequence plot of your
trace generated by tcptrace and xplot. The white vertical line
segments represent transmitted packets, and the green stair case
represents the sequence numbers that have been cumulatively
acknowledged. You can see that the RTT is often less than 2ms but then
sometimes a *lot* higher; here there is a highlighted example where
the RTT is 20ms, or more than 10x higher than the min_rtt.

The current cwnd_gain parameter for BBR allows a cwnd of 2 * BtlBw *
RTProp; given the 10x divergence between min RTT and typical RTT, this
particular path might require an in-flight as high as 10 * BtlBw *
RTProp.

We are working on a scheme to dynamically scale the amount of
in-flight data allowed for situations like this.

Thanks!
neal

On Tue, Apr 25, 2017 at 11:39 AM, Francis Rammeloo

[Francis Rammeloo]

Thanks for the informative reply :)

Francis

[Francis Rammeloo]

Might this also explain why the performance was slightly better when taking the capture? I assume tcpdump overhead could increase the min-rtt so there's less divergence.

Francis

[Neal Cardwell]

On Thu, May 4, 2017 at 12:20 PM, Francis Rammeloo
<francis....@gmail.com> wrote:
> Might this also explain why the performance was slightly better when taking
> the capture? I assume tcpdump overhead could increase the min-rtt so there's
> less divergence.

Yes, it seems plausible that there could be an effect like that.

cheers,
neal