BBR + iperf3 latency increase

[Pablo Gil Pereira]

Hi guys,

I'm running performance experiments with BBR in the following architecture: client---router1---router2---server, with a 3 Mbps bottleneck and 80 ms latency. Both client and server have Linux kernel 4.9 and BBR has been enabled executing the following commands:

    > echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf

    > echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf

    > sysctl -p

When I run iperf3 on server and client, I get a cwnd doesn't coincides with BBR's operation point according to the ACM paper. In this case BDP = 3 Mbps * 80 ms = 30 kbytes. However, I get the following iperf3 client output

Connecting to host 10.10.2.1, port 5201

[  4] local 10.10.4.1 port 50886 connected to 10.10.2.1 port 5201

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[  4]   0.00-1.00   sec   875 KBytes  7.17 Mbits/sec    0   84.8 KBytes       

[  4]   1.00-2.00   sec   318 KBytes  2.61 Mbits/sec    0   65.0 KBytes       

[  4]   2.00-3.00   sec   318 KBytes  2.61 Mbits/sec    0   65.0 KBytes       

[  4]   3.00-4.00   sec   636 KBytes  5.22 Mbits/sec    0   65.0 KBytes       

[  4]   4.00-5.00   sec   318 KBytes  2.61 Mbits/sec    0   62.2 KBytes       

[  4]   5.00-6.00   sec   318 KBytes  2.61 Mbits/sec    0   62.2 KBytes       

[  4]   6.00-7.00   sec   318 KBytes  2.61 Mbits/sec    0   62.2 KBytes       

[  4]   7.00-8.00   sec   318 KBytes  2.61 Mbits/sec    0   62.2 KBytes   

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[  4]   0.00-10.00  sec  3.96 MBytes  3.32 Mbits/sec    0             sender

[  4]   0.00-10.00  sec  3.42 MBytes  2.87 Mbits/sec                  receiver

where the cwnd is twice the expected one. Besides, running a ping at the same time as iperf3 and the latency increases from 80 ms to 200 ms. Might I be missing something or BBR is not working correctly? I tried with netperf as well, getting the same results, which made me think of a problem in BBR configuration.

[Neal Cardwell]

On Fri, Jul 21, 2017 at 12:05 PM, Pablo Gil Pereira <pablogi...@gmail.com> wrote:

Hi guys,

I'm running performance experiments with BBR in the following architecture: client---router1---router2---server, with a 3 Mbps bottleneck and 80 ms latency. Both client and server have Linux kernel 4.9 and BBR has been enabled executing the following commands:

    > echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf

    > echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf

    > sysctl -p

When I run iperf3 on server and client, I get a cwnd doesn't coincides with BBR's operation point according to the ACM paper. In this case BDP = 3 Mbps * 80 ms = 30 kbytes. However, I get the following iperf3 client output

Did you reboot after running those commands? The qdisc sysctl doesn't take effect until the device is populated with a new qdisc. The easiest way to do that is to reboot.

That said, if you upgrade to the latest net-next or 4.13-rc1 Linux, then you should not need fq, and should not need a reboot in such cases. Though clearly the kernel upgrade would require a reboot. :-)

If you are still having problems after rebooting, please post a sender-side packet trace.

Thanks!
neal

[Pablo Gil Pereira]

I ran it again after reboot and I got the same results. Please, find attached a pcap file with the sender-side packet trace.

Thank you!

Pablo

[Neal Cardwell]

Thanks for the packet traces! Looking at the traces (graphs below), it seems like BBR is behaving as expected. The RTT for packets that I see in the trace is around 105ms or so, which is about what you'd expect for an 80ms two-way propagation delay and a link as slow as 3Mbps (the serialization delay for a 2-packet TSO chunk would be 8ms, so that's at least 80ms + 8ms for an RTT sample).

Keep in mind that the cwnd is intentionally slightly more than 2x the BDP, to allow for various delayed/aggregation ACK phenomena and quantization effects in offload mechanisms. For the details see the Internet Draft:

 https://tools.ietf.org/html/draft-cardwell-iccrg-bbr-congestion-control

Regarding the cwnd, keep in mind that for BBR the cwnd is usually not the bottleneck; usually (as here) it's the pacing rate. The pacing rate varies, to try to keep the amount of inflight data close to the BDP, to get full throughput and low delay. So to understand BBR behavior one has to look at the actual amount of inflight data, not the cwnd.

I am not sure why your ping tool shows a latency of 200ms. Perhaps you can double-check by tcpdumping the icmp packets as well as the TCP packets?

hope that helps,

neal

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pablo Gil Pereira]

The ping has values close to 100 ms after rebooting.

Thanks for your answer. I thought the cwnd was set to BDP, and that 2x was driving me crazy. Not only had I problems with this experiment because of that, but it was also affecting my analysis in other experiments. Now everything is clear. :-)

Regards,

Pablo

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.