A question about tc fq pacing with BBR

[cf623...@gmail.com]

Hi ALL:

I  have been being running some tests with BBR on high loss rate path.  It seem that BBR have a overwhelming advantage than the loss-base congestion algorithm. 

What make me confused is that when i try to use tc fq qdisc with pacing enable together with BBR, i got a extremely lower throughput compared with BBR with default

pfifo_fast packet scheduler (or tc fq without pacing enable).  The details about my test is as follows:

 

I setup a bottleneck link with 50Mbps rate limited, 10ms delay and 20% loss rate using the Netropy network emulator and use Iperf tool to send/recv data. 

What i got is that when i use BBR with default pfifo_fast qdisc (or tc fq without pacing enable), I got an average throughput about 36Mbps. However when i use

 BBR with tc fq pacing enable, i got a throughput only 6.5Mpbs.    The kernel version is 4.9.0-rc2.

======================================= BBR with tc FQ NOPACING ====================

[root@localhost ~]# uname -r

4.9.0-rc2

[root@localhost ~]# 

[root@localhost ~]# uname -a

Linux localhost.localdomain 4.9.0-rc2 #1 SMP Thu Dec 1 19:13:24 CST 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# 

[root@localhost ~]# 

[root@localhost ~]# cat /proc/sys/net/ipv4/tcp_congestion_control 

bbr

[root@localhost ~]# 

[root@localhost ~]# tc -s -d qdisc show dev enp2s0 

qdisc mq 8001: root 

 Sent 141746513 bytes 95155 pkt (dropped 0, overlimits 0 requeues 0) 

 backlog 0b 0p requeues 0 

qdisc fq 8002: parent 8001:1 limit 10000p flow_limit 1000p buckets 1024 nopacing quantum 3028 initial_quantum 15140 

 Sent 141746513 bytes 95155 pkt (dropped 0, overlimits 0 requeues 0) 

 backlog 0b 0p requeues 0 

  12 flows (12 inactive, 0 throttled)

  0 gc, 0 highprio, 16131 throttled

[root@localhost ~]# 

[root@localhost ~]# 

[root@localhost ~]# iperf3 -c 172.23.4.15 -p 8080 -i 1 -b 50M -t 20 

Connecting to host 172.23.4.15, port 8080

[  4] local 172.23.4.88 port 45950 connected to 172.23.4.15 port 8080

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[  4]   0.00-1.00   sec  3.00 MBytes  25.1 Mbits/sec  383    107 KBytes       

[  4]   1.00-2.00   sec  4.38 MBytes  36.7 Mbits/sec  805    110 KBytes       

[  4]   2.00-3.00   sec  4.50 MBytes  37.7 Mbits/sec  855    110 KBytes       

[  4]   3.00-4.00   sec  4.62 MBytes  38.8 Mbits/sec  831    106 KBytes       

[  4]   4.00-5.00   sec  4.50 MBytes  37.7 Mbits/sec  823    122 KBytes       

[  4]   5.00-6.00   sec  4.50 MBytes  37.7 Mbits/sec  868    120 KBytes       

[  4]   6.00-7.00   sec  4.50 MBytes  37.7 Mbits/sec  819    116 KBytes       

[  4]   7.00-8.00   sec  4.62 MBytes  38.8 Mbits/sec  840    110 KBytes       

[  4]   8.00-9.00   sec  4.50 MBytes  37.7 Mbits/sec  820    106 KBytes       

[  4]   9.00-10.00  sec  4.62 MBytes  38.8 Mbits/sec  795   5.66 KBytes       

[  4]  10.00-11.00  sec  3.62 MBytes  30.4 Mbits/sec  677    113 KBytes       

[  4]  11.00-12.00  sec  4.25 MBytes  35.7 Mbits/sec  831    113 KBytes       

[  4]  12.00-13.00  sec  4.88 MBytes  40.9 Mbits/sec  817    116 KBytes       

[  4]  13.00-14.00  sec  4.38 MBytes  36.7 Mbits/sec  839    110 KBytes       

[  4]  14.00-15.00  sec  4.50 MBytes  37.7 Mbits/sec  826    117 KBytes       

[  4]  15.00-16.00  sec  4.50 MBytes  37.7 Mbits/sec  795    127 KBytes       

[  4]  16.00-17.00  sec  4.62 MBytes  38.8 Mbits/sec  840    116 KBytes       

[  4]  17.00-18.00  sec  4.62 MBytes  38.8 Mbits/sec  847    113 KBytes       

[  4]  18.00-19.00  sec  4.50 MBytes  37.7 Mbits/sec  802    113 KBytes       

[  4]  19.00-20.00  sec  4.62 MBytes  38.8 Mbits/sec  834    113 KBytes       

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[  4]   0.00-20.00  sec  88.2 MBytes  37.0 Mbits/sec  15947             sender

[  4]   0.00-20.00  sec  87.2 MBytes  36.6 Mbits/sec                  receiver

iperf Done.

========================================= BBR with tc FQ PACING =======================

[root@localhost ~]# 

[root@localhost ~]# tc qdisc replace dev enp2s0 parent 8001:1 fq flow_limit 1000 pacing   

[root@localhost ~]# 

[root@localhost ~]# tc -s -d qdisc show dev enp2s0                                                       

qdisc mq 8001: root 

 Sent 261875943 bytes 174747 pkt (dropped 0, overlimits 0 requeues 0) 

 backlog 0b 0p requeues 0 

qdisc fq 8002: parent 8001:1 limit 10000p flow_limit 1000p buckets 1024 quantum 3028 initial_quantum 15140 

 Sent 261875943 bytes 174747 pkt (dropped 0, overlimits 0 requeues 0) 

 backlog 0b 0p requeues 0 

  13 flows (13 inactive, 0 throttled)

  0 gc, 0 highprio, 16131 throttled

[root@localhost ~]# 

[root@localhost ~]# 

[root@localhost ~]# 

[root@localhost ~]# iperf3 -c 172.23.4.15 -p 8080 -i 1 -b 50M -t 20                                      

Connecting to host 172.23.4.15, port 8080

[  4] local 172.23.4.88 port 45954 connected to 172.23.4.15 port 8080

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[  4]   0.00-1.00   sec  4.22 MBytes  35.4 Mbits/sec  580   99.0 KBytes       

[  4]   1.00-2.00   sec  2.38 MBytes  19.9 Mbits/sec  429   59.4 KBytes       

[  4]   2.00-3.00   sec  1.75 MBytes  14.7 Mbits/sec  304   48.1 KBytes       

[  4]   3.00-4.00   sec  1.38 MBytes  11.5 Mbits/sec  234   39.6 KBytes       

[  4]   4.00-5.00   sec   896 KBytes  7.34 Mbits/sec  168   31.1 KBytes       

[  4]   5.00-6.00   sec   896 KBytes  7.34 Mbits/sec  159   31.1 KBytes       

[  4]   6.00-7.00   sec   512 KBytes  4.19 Mbits/sec  119   22.6 KBytes       

[  4]   7.00-8.00   sec   512 KBytes  4.19 Mbits/sec   91   19.8 KBytes       

[  4]   8.00-9.00   sec   512 KBytes  4.19 Mbits/sec   74   15.6 KBytes       

[  4]   9.00-10.00  sec   384 KBytes  3.15 Mbits/sec   54   5.66 KBytes       

[  4]  10.00-11.00  sec   256 KBytes  2.10 Mbits/sec   48   9.90 KBytes       

[  4]  11.00-12.00  sec   256 KBytes  2.10 Mbits/sec   54   17.0 KBytes       

[  4]  12.00-13.00  sec   256 KBytes  2.10 Mbits/sec   51   17.0 KBytes       

[  4]  13.00-14.00  sec   256 KBytes  2.10 Mbits/sec   36   17.0 KBytes       

[  4]  14.00-15.00  sec   384 KBytes  3.15 Mbits/sec   37   12.7 KBytes       

[  4]  15.00-16.00  sec   384 KBytes  3.15 Mbits/sec   70   19.8 KBytes       

[  4]  16.00-17.00  sec   384 KBytes  3.15 Mbits/sec   72   12.7 KBytes       

[  4]  17.00-18.00  sec   384 KBytes  3.15 Mbits/sec   64   9.90 KBytes       

[  4]  18.00-19.00  sec   384 KBytes  3.15 Mbits/sec   71   19.8 KBytes       

[  4]  19.00-20.00  sec   512 KBytes  4.19 Mbits/sec   90   33.9 KBytes       

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[  4]   0.00-20.00  sec  16.7 MBytes  7.01 Mbits/sec  2805             sender

[  4]   0.00-20.00  sec  15.8 MBytes  6.65 Mbits/sec                  receiver

iperf Done.

[root@localhost ~]# 

[root@localhost ~]# 

[root@localhost ~]# tc -s -d qdisc show dev enp2s0                  

qdisc mq 8001: root 

 Sent 283570543 bytes 189217 pkt (dropped 0, overlimits 0 requeues 0) 

 backlog 0b 0p requeues 0 

qdisc fq 8002: parent 8001:1 limit 10000p flow_limit 1000p buckets 1024 quantum 3028 initial_quantum 15140 

 Sent 283570543 bytes 189217 pkt (dropped 0, overlimits 0 requeues 0) 

 backlog 0b 0p requeues 0 

  13 flows (13 inactive, 0 throttled)

  0 gc, 0 highprio, 22766 throttled

[root@localhost ~]# 

There is no packets dropped in the tc qdisc queue.  

Several sample of the  BBR connection statistics from ss is as follow: 

/* BBR with tc FQ PACING */

bbr wscale:7,7 rto:212 rtt:11.701/0.449 mss:1448 cwnd:13 bytes_acked:1486 segs_out:39 segs_in:19 data_segs_out:37 send 12.9Mbps lastrcv:23522503 pacing_rate 33.5Mbps unacked:29 retrans:5/6 lost:5 sacked:16 rcv_space:29200 notsent:83984 minrtt:10.332

bbr wscale:7,7 rto:212 rtt:11.567/0.352 mss:1448 cwnd:29 bytes_acked:3302926 segs_out:2893 segs_in:2295 data_segs_out:2891 send 29.0Mbps lastsnd:1 lastrcv:23523509 pacing_rate 26.2Mbps unacked:36 retrans:8/572 lost:8 sacked:11 rcv_space:29200 notsent:966712 minrtt:10

bbr wscale:7,7 rto:213 rtt:12.917/0.114 mss:1448 cwnd:46 bytes_acked:5765974 segs_out:4988 segs_in:4001 data_segs_out:4986 send 41.3Mbps lastrcv:23524514 pacing_rate 22.4Mbps unacked:42 retrans:8/960 lost:8 sacked:16 rcv_space:29200 notsent:985344 minrtt:10

bbr wscale:7,7 rto:215 rtt:14.188/0.452 mss:1448 cwnd:36 bytes_acked:7571630 segs_out:6592 segs_in:5265 data_segs_out:6590 send 29.4Mbps lastsnd:2 lastrcv:23525520 lastack:1 pacing_rate 16.5Mbps unacked:55 retrans:3/1304 lost:3 sacked:37 rcv_space:29200 notsent:864800 minrtt:10

bbr wscale:7,7 rto:216 rtt:15.065/0.412 mss:1448 cwnd:28 bytes_acked:8900894 segs_out:7722 segs_in:6155 data_segs_out:7720 send 21.5Mbps lastsnd:1 lastrcv:23526525 pacing_rate 12.4Mbps unacked:23 retrans:5/1548 lost:5 sacked:9 rcv_space:29200 notsent:1023664 minrtt:10

bbr wscale:7,7 rto:215 rtt:14.295/1.196 mss:1448 cwnd:22 bytes_acked:9915942 segs_out:8600 segs_in:6866 data_segs_out:8598 send 17.8Mbps lastsnd:3 lastrcv:23527531 lastack:4 pacing_rate 10.0Mbps unacked:34 retrans:4/1714 lost:4 sacked:19 rcv_space:29200 notsent:910192 minrtt:10

bbr wscale:7,7 rto:217 rtt:16.891/1.048 mss:1448 cwnd:22 bytes_acked:10880310 segs_out:9415 segs_in:7519 data_segs_out:9413 send 15.1Mbps lastsnd:1 lastrcv:23528536 lastack:4 pacing_rate 9.2Mbps unacked:22 retrans:3/1875 lost:3 sacked:7 rcv_space:29200 notsent:1011776 minrtt:10

bbr wscale:7,7 rto:218 rtt:17.61/1.944 mss:1448 cwnd:20 bytes_acked:11665126 segs_out:10110 segs_in:8087 data_segs_out:10108 send 13.2Mbps lastrcv:23529541 lastack:2 pacing_rate 7.6Mbps unacked:47 retrans:4/2003 lost:4 sacked:33 rcv_space:29200 notsent:977192 minrtt:10

bbr wscale:7,7 rto:220 rtt:19.577/1.476 mss:1448 cwnd:9 bytes_acked:12386230 segs_out:10696 segs_in:8557 data_segs_out:10694 send 5.3Mbps lastsnd:1 lastrcv:23530547 lastack:1 pacing_rate 6.2Mbps unacked:13 retrans:1/2125 lost:1 sacked:5 rcv_space:29200 notsent:960680 minrtt:10

/* BBR with tc FQ NOPACING */

bbr wscale:7,7 rto:214 rtt:10.333/5.166 mss:1448 cwnd:10 bytes_acked:1 segs_out:3 segs_in:1 data_segs_out:1 send 11.2Mbps lastsnd:102 lastrcv:23624769 lastack:102 pacing_rate 33.5Mbps unacked:1 rcv_space:29200 minrtt:10.333

bbr wscale:7,7 rto:218 rtt:17.613/0.154 mss:1448 cwnd:74 bytes_acked:3621486 segs_out:3269 segs_in:2573 data_segs_out:3267 send 48.7Mbps lastsnd:1 lastrcv:23625775 lastack:1 pacing_rate 37.4Mbps unacked:142 retrans:9/621 lost:9 sacked:68 rcv_space:29200 notsent:925752 minrtt:10

bbr wscale:7,7 rto:219 rtt:18.416/0.07 mss:1448 cwnd:76 bytes_acked:8454910 segs_out:7414 segs_in:5890 data_segs_out:7412 send 47.8Mbps lastrcv:23626781 pacing_rate 38.8Mbps unacked:123 retrans:13/1447 lost:13 sacked:47 rcv_space:29200 notsent:931632 minrtt:10

bbr wscale:7,7 rto:219 rtt:18.625/0.191 mss:1448 cwnd:78 bytes_acked:13069686 segs_out:11557 segs_in:9179 data_segs_out:11555 send 48.5Mbps lastsnd:1 lastrcv:23627787 lastack:1 pacing_rate 39.4Mbps unacked:227 retrans:18/2299 lost:18 sacked:149 rcv_space:29200 notsent:753784 minrtt:10

bbr wscale:7,7 rto:219 rtt:18.855/0.127 mss:1448 cwnd:78 bytes_acked:17958134 segs_out:15697 segs_in:12476 data_segs_out:15695 send 47.9Mbps lastrcv:23628792 pacing_rate 39.1Mbps unacked:147 retrans:16/3142 lost:17 sacked:70 rcv_space:29200 notsent:961912 minrtt:10

bbr wscale:7,7 rto:219 rtt:18.735/0.156 mss:1448 cwnd:78 bytes_acked:22555534 segs_out:19840 segs_in:15738 data_segs_out:19838 send 48.2Mbps lastrcv:23629798 pacing_rate 39.8Mbps unacked:235 retrans:18/4023 lost:18 sacked:157 rcv_space:29200 notsent:824608 minrtt:10

bbr wscale:7,7 rto:220 rtt:19.187/0.025 mss:1448 cwnd:80 bytes_acked:27443982 segs_out:23984 segs_in:19056 data_segs_out:23982 send 48.3Mbps lastrcv:23630804 pacing_rate 40.3Mbps unacked:179 retrans:23/4847 lost:23 sacked:99 rcv_space:29200 notsent:866912 minrtt:10

bbr wscale:7,7 rto:219 rtt:18.745/0.224 mss:1448 cwnd:78 bytes_acked:32315054 segs_out:28126 segs_in:22431 data_segs_out:28124 send 48.2Mbps lastrcv:23631811 pacing_rate 39.0Mbps unacked:188 retrans:15/5616 lost:15 sacked:110 rcv_space:29200 notsent:832472 minrtt:10

bbr wscale:7,7 rto:219 rtt:18.318/0.148 mss:1448 cwnd:76 bytes_acked:37057254 segs_out:32265 segs_in:25733 data_segs_out:32263 send 48.1Mbps lastrcv:23632817 pacing_rate 38.6Mbps unacked:213 retrans:24/6455 lost:24 sacked:137 rcv_space:29200 notsent:903736 minrtt:10

Information of my NIC:

[root@localhost ~]# ethtool -i enp2s0 

driver: tg3

version: 3.137

firmware-version: sb

bus-info: 0000:02:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: yes

supports-priv-flags: no

Any suggestion? Very apprecaite!

Btw, any test result of BBR on  wireless path?  From my roughly test, it looks not very well compared with cubic/bic.  

Best Regards,

Devin Rao 

[Eric Dumazet]

Well, make sure to test the very typical case where your TCP sender has a 10Gbit interface, then later in the path, you have a link with a 100Mbit bottleneck.

If you are really willing to push 10Gbit, and therefore get 99% packet losses, then yes you can remove pacing.

Pacing is adding delays. Sometimes, delays are not needed, but often they are needed to avoid catastrophic consequences.

Future being hard to predict, at least from a TCP stack as implemented by computer, make sure you do not hurt the network by not using pacing with BBR.

[Yuchung Cheng]

could you provide tcpdump traces for your test (with and without pacing)? 

 

Btw, any test result of BBR on  wireless path?  From my roughly test, it looks not very well compared with cubic/bic.  

yes we've tested BBR on various mobile providers (that we've confirmed these connections aren't going through TCP-terminating proxies).

Persistent 20% TCP level loss rate is very rare in my experience. Both wifi and cellular link layer does a good job in reliable delivery.       

 

Best Regards,

Devin Rao 

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cf623...@gmail.com]

 

could you provide tcpdump traces for your test (with and without pacing)? 

Thanks for your reply. Please see the .RAR package.  It is easy to understand with the corresponding file name. 

 

Btw, any test result of BBR on  wireless path?  From my roughly test, it looks not very well compared with cubic/bic.  

yes we've tested BBR on various mobile providers (that we've confirmed these connections aren't going through TCP-terminating proxies).

Persistent 20% TCP level loss rate is very rare in my experience. Both wifi and cellular link layer does a good job in reliable delivery.       

Sorry about my unbiguous description.  My first tests(BBR with/without fq pacing) is primary concerned about the capibility of BBR to resist high loss rate on wire 

link, which means all paths including 50Mbps/10ms/20% loss rate bottleneck are wired.  

There isn't  any modification about the characteristics of the link(including bandrate, delay and loss rate), when i test BBR on wireless link.

Ok, I will check my environment first. 

 

Best Regards,

Devin Rao 

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

[Neal Cardwell]

Looking at the bbr_with_fq_pacing.pcap trace you provided with "tcptrace -l" shows:

     actual data pkts:      11555 

     rexmt data pkts:        3523

That suggests a retransmit rate of just over 30%. BBR was designed to be cautious with a loss rate that high.

The BBR paper in ACM Queue discusses BBR's response to packet loss, and Figure 8 shows how BBR behaves under a range of loss rates:

  http://queue.acm.org/detail.cfm?id=3022184

In that figure you can see how the design parameters chosen for BBR enable it to usually get close to the maximum throughput possible up to loss rates around 15%.

You might want to  try your experiments with lower loss rates, or a range of loss rates (0.1%, 1%, 5%, 10%, 15%). I suspect you will find results close to Figure 8. If not, please let us know.

cheers,

neal

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[cf623...@gmail.com]

Hi neal,

Thanks for your advise. It seems that it is the tc fq pacing delay the packet that cause the extra 10% retransmission (30% rexmit rate on my 20% loss rate 

bottleneck), i try to modify the time_next_packet variable in every fq_flow with:  f->time_next_packet = now + len/2.   I got a 20% rexmit rate with tc fq pacing 

enable .

The orignal fq get:

     actual data pkts:      13673           actual data pkts:          0      

     rexmt data pkts:        4268           rexmt data pkts:           0      

After modify fq with:  f->time_next_packet = now + len/2， what i get 

     actual data pkts:     117587           actual data pkts:          0            

     rexmt data pkts:       23980           rexmt data pkts:           0 

Best,

Devin

在 2016年12月22日星期四 UTC+8下午11:57:53，Neal Cardwell写道：