wireless/ath10k TCP throughput and TSQ, reno/cubic/bbr

[Andrew W Elble]

I came across the BBR implementation while we are in the midst of a
wireless refresh project here and tested it a bit. Was surprised at
the lack of throughput on the transmit side and came across the thread
where Eric Dumazet wrote:

"This TSQ behavior has been discussed during some netdev linux
conferences in Toronto last year, and I suggested that wifi drivers
experiment in varying the 1ms TSQ budget to 5 ms, or 4 ms, to help
aggregation logic.

So far I am not aware of someone completing the experiment, so the
code in Linux is unchanged."

...and I seem to remember this thread:

https://lkml.org/lkml/2016/4/12/763

"Guessing is nice, but investigating and fixing is better.
Do not assume that nothing can be done, please ?"

In the interest of moving these discussions forward,
I've given this a shot. I hope it's useful.

Kernel: 4.9.11+-ish
Wireless card: ath10k / Compex WLE900VX / FW: 10.2.4-1.0-00016
Access point: unnamed enterprise-class 802.11ac wave2,
80MHz channel (SNR such that MCS9 rates are possible)
Patch applied to test is below results:

FQ is in place on interface:
tc qdisc add dev wlp10s0 root fq

test command:

iperf3 -c <servername> -t 60 -p 2112

RENO:

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=reno
net.ipv4.tcp_congestion_control = reno
[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=5
net.ipv4.tcp_tsq_limit_output_interval = 1

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 1.59 GBytes 228 Mbits/sec 62 sender
[ 4] 0.00-60.00 sec 1.59 GBytes 228 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=reno
net.ipv4.tcp_congestion_control = reno
[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=5
net.ipv4.tcp_tsq_limit_output_interval = 5

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 2.57 GBytes 368 Mbits/sec 50 sender
[ 4] 0.00-60.00 sec 2.57 GBytes 368 Mbits/sec receiver

CUBIC:

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_congestion_control = cubic
[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=1
net.ipv4.tcp_tsq_limit_output_interval = 1

NOTE: stays stuck at lower throughput for the first 15 seconds or so:
e.g.
[ 4] 7.00-8.00 sec 8.15 MBytes 68.4 Mbits/sec 0 69.8 KBytes)

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 1.40 GBytes 201 Mbits/sec 59 sender
[ 4] 0.00-60.00 sec 1.40 GBytes 201 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_congestion_control = cubic
[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=5
net.ipv4.tcp_tsq_limit_output_interval = 5

NOTE: jumps up to full BW almost immediately:
e.g.
[ 4] 1.00-2.00 sec 44.3 MBytes 372 Mbits/sec 0 369 KBytes

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 2.56 GBytes 367 Mbits/sec 158 sender
[ 4] 0.00-60.00 sec 2.56 GBytes 367 Mbits/sec receiver

BBR:

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=1
net.ipv4.tcp_tsq_limit_output_interval = 1

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 104 MBytes 14.5 Mbits/sec 0 sender
[ 4] 0.00-60.00 sec 103 MBytes 14.4 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=5
net.ipv4.tcp_tsq_limit_output_interval = 5

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 2.32 GBytes 333 Mbits/sec 0 sender
[ 4] 0.00-60.00 sec 2.32 GBytes 332 Mbits/sec receiver

diff --git a/include/net/tcp.h b/include/net/tcp.h
index 123979f..02642aa 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -264,6 +264,7 @@
extern int sysctl_tcp_thin_dupack;
extern int sysctl_tcp_early_retrans;
extern int sysctl_tcp_limit_output_bytes;
+extern int sysctl_tcp_tsq_limit_output_interval;
extern int sysctl_tcp_challenge_ack_limit;
extern int sysctl_tcp_min_tso_segs;
extern int sysctl_tcp_min_rtt_wlen;
diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c
index 80bc36b..75cd823 100644
--- a/net/ipv4/sysctl_net_ipv4.c
+++ b/net/ipv4/sysctl_net_ipv4.c
@@ -502,6 +502,13 @@ static int proc_tcp_fastopen_key(struct ctl_table *ctl, int write,
.proc_handler = proc_dointvec
},
{
+ .procname = "tcp_tsq_limit_output_interval",
+ .data = &sysctl_tcp_tsq_limit_output_interval,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec
+ },
+ {
.procname = "tcp_challenge_ack_limit",
.data = &sysctl_tcp_challenge_ack_limit,
.maxlen = sizeof(int),
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 65d6189..9494836 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -53,6 +53,9 @@
/* Default TSQ limit of four TSO segments */
int sysctl_tcp_limit_output_bytes __read_mostly = 262144;

+/* Limit on number of packets in qdisc/devices in units of ms */
+int sysctl_tcp_tsq_limit_output_interval __read_mostly = 1;
+
/* This limits the percentage of the congestion window which we
* will allow a single TSO frame to consume. Building TSO frames
* which are too large can cause TCP streams to be bursty.
@@ -2064,7 +2067,7 @@ static bool tcp_small_queue_check(struct sock *sk, const struct sk_buff *skb,
{
unsigned int limit;

- limit = max(2 * skb->truesize, sk->sk_pacing_rate >> 10);
+ limit = max_t(u32, 2 * skb->truesize, sysctl_tcp_tsq_limit_output_interval * (sk->sk_pacing_rate >> 10));
limit = min_t(u32, limit, sysctl_tcp_limit_output_bytes);
limit <<= factor;

--
Andrew W. Elble
awe...@discipline.rit.edu
Infrastructure Engineer, Communications Technical Lead
Rochester Institute of Technology
PGP: BFAD 8461 4CCF DC95 DA2C B0EB 965B 082E 863E C912

[Eric Dumazet]

Hi Andrew. Thanks a lot for doing these experiments.

Given that sysctl_tcp_limit_output_bytes should guard against
excessive bloat in a qdisc for a TCP flow,
we might simply change the 1ms to 4ms [1]. The only risk is that rtx
packets would have a bigger queue in front of them,
so some repairs would take 3 additional ms on the receiver,
potentially adding stalls.

([1] use sk->sk_pacing_rate >> 8 instead of sk->sk_pacing_rate >> 10)

Could you run your throughput experiments for sysctl
net.ipv4.tcp_tsq_limit_output_interval=4 on your kernel ?

Thanks !

[Andrew W Elble]

Eric Dumazet <edum...@google.com> writes:

> Hi Andrew. Thanks a lot for doing these experiments.
>
> Given that sysctl_tcp_limit_output_bytes should guard against
> excessive bloat in a qdisc for a TCP flow,
> we might simply change the 1ms to 4ms [1]. The only risk is that rtx
> packets would have a bigger queue in front of them,
> so some repairs would take 3 additional ms on the receiver,
> potentially adding stalls.
>
> ([1] use sk->sk_pacing_rate >> 8 instead of sk->sk_pacing_rate >> 10)
>
> Could you run your throughput experiments for sysctl
> net.ipv4.tcp_tsq_limit_output_interval=4 on your kernel ?
>
> Thanks !

NOTE: all tests went to full bandwidth almost immediately.

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=reno
net.ipv4.tcp_congestion_control = reno

[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=4
net.ipv4.tcp_tsq_limit_output_interval = 4

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 2.46 GBytes 352 Mbits/sec 151 sender
[ 4] 0.00-60.00 sec 2.46 GBytes 352 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_congestion_control = cubic

[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=4
net.ipv4.tcp_tsq_limit_output_interval = 4

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 2.47 GBytes 354 Mbits/sec 31 sender
[ 4] 0.00-60.00 sec 2.47 GBytes 354 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr

[root@localhost aweits]# sysctl net.ipv4.tcp_tsq_limit_output_interval=4
net.ipv4.tcp_tsq_limit_output_interval = 4

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 1.97 GBytes 282 Mbits/sec 84 sender
[ 4] 0.00-60.00 sec 1.97 GBytes 281 Mbits/sec receiver

[Klatsky, Carl]

In these latest results with net.ipv4.tcp_tsq_limit_output_interval=4 and all CCRs running with fq/pacing qdisc, why does BBR seem to get less throughput than RENO or CUBIC?

Regards,
Carl Klatsky

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Neal Cardwell]

On Tue, Feb 28, 2017 at 10:17 AM, Klatsky, Carl <Carl_K...@comcast.com> wrote:

In these latest results with net.ipv4.tcp_tsq_limit_output_interval=4 and all CCRs running with fq/pacing qdisc, why does BBR seem to get less throughput than RENO or CUBIC?

I would love to see tcpdump traces to confirm, but I suspect this is due to the set of known issues we have seen before on the bbr-dev list and elsewhere, in which:

(1) The current cwnd calculation (cwnd = cwnd_gain * bw * min_rtt + tso_budget) is not sufficient to reach full utilization on wifi LANs, because this does not allow enough data in flight to feed the aggregation mechanisms in play. This is related to some of the same issues that cause the TSQ limit itself to be tricky for wifi LANs.

(2) Then when the BBR cwnd is not sufficient to fully utilize the available bandwidth, the bw samples and hence bw estimate fall, and hence the pacing rate and cwnd can fall further. Eventually the rate settles at a level that the cwnd budget can sustain.

This set of issues is one I mentioned yesterday as an interesting area for further research, if there are interested researchers out there.

neal

 

Regards,
Carl Klatsky

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Andrew W Elble]

I did some additional testing with an Intel card:

[ 5.620205] iwlwifi 0000:0a:00.0: loaded firmware version 17.352738.0 op_mode iwlmvm
[ 6.040120] iwlwifi 0000:0a:00.0: Detected Intel(R) Dual Band Wireless AC 7260, REV=0x144

The results are somewhat similar, but only if I turn GSO off.

Does (2 * skb->truesize) give an "advantage" to wireless drivers that actually implement
TSO/GSO?

>> limit = max(2 * skb->truesize, sk->sk_pacing_rate >> 10);

GSO disabled:

sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=1
sudo sysctl net.ipv4.tcp_congestion_control=bbr

[ 4] 0.00-60.00 sec 409 MBytes 57.2 Mbits/sec 15 sender
[ 4] 0.00-60.00 sec 409 MBytes 57.2 Mbits/sec receiver

sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=4
sudo sysctl net.ipv4.tcp_congestion_control=bbr

[ 4] 0.00-60.00 sec 861 MBytes 120 Mbits/sec 34 sender
[ 4] 0.00-60.00 sec 860 MBytes 120 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=reno
net.ipv4.tcp_congestion_control = reno

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=1
net.ipv4.tcp_tsq_limit_output_interval = 1

[ 4] 0.00-60.00 sec 1.67 GBytes 238 Mbits/sec 10 sender
[ 4] 0.00-60.00 sec 1.66 GBytes 238 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=4
net.ipv4.tcp_tsq_limit_output_interval = 4

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=reno
net.ipv4.tcp_congestion_control = reno

[ 4] 0.00-60.00 sec 1.70 GBytes 244 Mbits/sec 42 sender
[ 4] 0.00-60.00 sec 1.70 GBytes 243 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=1
net.ipv4.tcp_tsq_limit_output_interval = 1

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_congestion_control = cubic

[ 4] 0.00-60.00 sec 1.51 GBytes 217 Mbits/sec 17 sender
[ 4] 0.00-60.00 sec 1.51 GBytes 217 Mbits/sec receiver

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=4
net.ipv4.tcp_tsq_limit_output_interval = 4

[root@localhost aweits]# sudo sysctl net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_congestion_control = cubic

[ 4] 0.00-60.00 sec 1.29 GBytes 185 Mbits/sec 190 sender
[ 4] 0.00-60.00 sec 1.29 GBytes 185 Mbits/sec receiver

GSO enabled:

RENO

localhost (records/2017):sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=1
net.ipv4.tcp_tsq_limit_output_interval = 1
localhost (records/2017):sudo sysctl net.ipv4.tcp_congestion_control=reno
net.ipv4.tcp_congestion_control = reno

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 2.09 GBytes 299 Mbits/sec 12 sender
[ 4] 0.00-60.00 sec 2.08 GBytes 298 Mbits/sec receiver

localhost (records/2017):sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=4
net.ipv4.tcp_tsq_limit_output_interval = 4
localhost (records/2017):sudo sysctl net.ipv4.tcp_congestion_control=reno
net.ipv4.tcp_congestion_control = reno

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 1.83 GBytes 262 Mbits/sec 48 sender
[ 4] 0.00-60.00 sec 1.83 GBytes 262 Mbits/sec receiver


CUBIC

localhost (records/2017):sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=1
net.ipv4.tcp_tsq_limit_output_interval = 1
localhost (records/2017):sudo sysctl net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_congestion_control = cubic

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 2.09 GBytes 300 Mbits/sec 127 sender
[ 4] 0.00-60.00 sec 2.09 GBytes 299 Mbits/sec receiver

localhost (records/2017):sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=4
net.ipv4.tcp_tsq_limit_output_interval = 4
localhost (records/2017):sudo sysctl net.ipv4.tcp_congestion_control=cubic
net.ipv4.tcp_congestion_control = cubic

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 1.96 GBytes 281 Mbits/sec 16 sender
[ 4] 0.00-60.00 sec 1.96 GBytes 280 Mbits/sec receiver

BBR

localhost (records/2017):sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=1
net.ipv4.tcp_tsq_limit_output_interval = 1
localhost (records/2017):sudo sysctl net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 1.74 GBytes 248 Mbits/sec 96 sender
[ 4] 0.00-60.00 sec 1.73 GBytes 248 Mbits/sec receiver

localhost (records/2017):sudo sysctl net.ipv4.tcp_tsq_limit_output_interval=4
net.ipv4.tcp_tsq_limit_output_interval = 4
localhost (records/2017):sudo sysctl net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-60.00 sec 1.81 GBytes 260 Mbits/sec 43 sender
[ 4] 0.00-60.00 sec 1.81 GBytes 259 Mbits/sec receiver

[Eric Dumazet]

On Tue, Feb 28, 2017 at 12:11 PM, Andrew W Elble <awe...@rit.edu> wrote:
>
> I did some additional testing with an Intel card:
>
> [ 5.620205] iwlwifi 0000:0a:00.0: loaded firmware version 17.352738.0 op_mode iwlmvm
> [ 6.040120] iwlwifi 0000:0a:00.0: Detected Intel(R) Dual Band Wireless AC 7260, REV=0x144
>
> The results are somewhat similar, but only if I turn GSO off.
>
> Does (2 * skb->truesize) give an "advantage" to wireless drivers that actually implement
> TSO/GSO?

This might give incentive, but really GSO is pure software.

Consider that TCP developers only test their stuff with GSO being enabled.

At Google we have a TCP test suites with 1679 tests, and none of them
is disabling GSO.

Any tests done with GSO off are really not interesting for us, we will
not spend time on this feature.

We might very soon kill support for non GSO, and remove a lot of code
complexity and bugs,
by removing one variable.

[Ben Greear]

On 05/10/2017 11:16 AM, Ben Greear wrote:
> Hello!
>
> I finally had a good reason to look closely at this patch. We were testing TCP upload
> performance on ath10k stations on an apu2 (weak 4-core processor) system. With up to
> about 6 stations doing upload, total throughput remained at around 300Mbps, but as we added
> more, performance fell drastically, down to around 50Mbps total throughput. UDP saw no
> drop-off.

I did some more tests today. We were seeing pretty unreliable upload TCP throughput using
the 9984 (wave-2) ath10k 4x4 NIC. I have 15 station vdevs on a single ath10k radio on the
4.9.29+ kernel. With a setting of 5, I see throughput for 3 of them at around 750Mbps.
When I run all 15, I see total throughput of around 450Mbps. When I set the
tcp_tsq_limit_output_interval to 200, I see about 680Mbps total throughput across 15 stations.

Each station had a tcp connection trying to run at 300Mbps. With setting of 200, the upload
is similar across all stations (ie, it is fair). At 5, a few might have 100Mbps, and the rest are
very slow.

This seems fairly reliable, and throughput goes up as I change from 5 to 200.

Thanks,
Ben

>
> With your patch backported to our 4.7 kernel, we see about 300Mbps scale all the way to 15
> stations (one stream per station). So, it is a big help on this system.
>
> In addition, I tested this on a high-end E5 processor system, where CPU usage
> is not an issue. In this case, we were using a wave-2 4x4 9984 ath10k NIC
> in an isolation chamber.
>
> Test case is TCP upload + download, 100 TCP streams, requesting 700Mbps in
> each direction. We have a 1Gbps Ethernet interconnect between these systems.
>
> In our 4.7 kernel, without your patch enabled, I see 700Mbps download, 315Mbps upload
> TCP throughput.
>
> With the value set to 5, I see 700Mbps download, and 320Mbps upload (not a big change).
>
> In the 4.9 kernel, I see 700Mbps download, 455Mbps upload without your patch (wow, big
> improvement somewhere, not sure it is stack or ath10k or what!!)
>
> With your patch to the 4.9 kernel and the value set to 5, I see 700Mbps download,
> 503Mbps upload, so a nice improvement.
>
> So, it looks like a win, at least when using ath10k and hoping for good bulk
> TCP throughput, and we will add this to our kernels for further testing.
>
> We are using 'reno' in all of these cases.
>
> Thanks,
> Ben

Ben Greear <gre...@candelatech.com>
Candela Technologies Inc http://www.candelatech.com

[Andrew W Elble]

Ben/Eric,

I've been quite distracted by other projects, but I've also had some good
results with making the ath10k code "do" GSO as well. When it's in a
consumable form I'll share it and results...

Thanks,

Andy