Replicating Figure 6 from ACM queue paper

[Bruce Spang]

Hi!

In some of my recent research, I've been looking into non-loss-based congestion controllers and I'm very excited about BBR's design. I am also excited about the way BBR converges; Figure 6 in the ACM Queue article, which shows the throughput of five flows sharing a bottleneck, looks very promising.

I've been trying to replicate Figure 6 on mininet and am getting much less stable behavior (see the attached graph bbr-shared-link.png). I assume I've just set up the experiment wrong somehow, since TCP Vegas seems to be much more stable (see the attached vegas-shared-link.png). Do you happen to have a list of the environment parameters you used to create that figure (especially the fq qdisc settings), or even better a set of scripts I could use to reproduce it?

Thanks!

Bruce

[Soheil Hassas Yeganeh]

Hi Bruce,

Figure 6 is a two-machine experiment: one machine is used for the
sender flows and the other machine is used as the receiver.

The sender machine has a fq/pacing qdisc installed. The receiver
emulates a 100Mbps bottleneck link with 125pkt of buffer and 10ms of
delay (5ms ingress and 5ms egress) using netem.

The sender flows are started at time=0s, time=2s, time=4s, time=6s
and time=8s.

Hope this helps!

Thanks,
Soheil

> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bruce Spang]

Hi Soheil,

Thanks for the reply, that information definitely helps! I have two more questions though:

1. Do you know the tc commands that were used to emulate that bottleneck link? I’d rather not make up burst size settings and the like

2. Do you happen to know how the throughput data was collected for the graph? Was it just iperf, or something more complicated?

Thanks!

Bruce

[Samantha Steele]

Hi Soheil,

I wanted to follow up on Bruce's last post. I am trying to duplicate figure 6 and have been seeing similarly less consistent behavior. Are you using iperf or netperf flows? Are you measuring throughput using iperf or tcpdump or with a more sophisticated tool to measure throughput? Thanks for your help!

Best,
Samantha

[Soheil Hassas Yeganeh]

Hi Samantha,

The results are generated with netperf and the throughput time series
is calculated from packet captures.

Can you please send me your experiment setup (e.g., qdiscs on both
sides, netperf/netserver commands used, ...) so that I can take a
look?

Thank you!
Soheil

[Samantha Steele]

Hi Suheil,

Thanks for the info. We are now seeing better convergence on average, but there is a lot of variability between runs and the flows sometimes take much longer to converge. Is that expected behavior? Does this setup look correct? I've attached figures from a few different runs. The names describe whether iperf or netperf was used and whether bbr or reno was set on the server (we don't think that should affect results).  

client:

$ tc qdisc show

qdisc noqueue 0: dev lo root refcnt 2
qdisc fq 8001: dev ens4 root refcnt 5 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948 initial_quantum 14740 refill_delay 40.0ms

$ sysctl net.ipv4.tcp_congestion_control

net.ipv4.tcp_congestion_control = bbr

server:
$ tc qdisc show

qdisc noqueue 0: dev lo root refcnt 2
qdisc netem 8003: dev ens4 root refcnt 5 limit 125 delay 5.0ms rate 100Mbit
qdisc ingress ffff: dev ens4 parent ffff:fff1 ----------------
qdisc htb 1: dev ifb0 root refcnt 2 r2q 10 default 10 direct_packets_stat 157 direct_qlen 32
qdisc netem 20: dev ifb0 parent 1:10 limit 125 delay 5.0ms rate 100Mbit
$ tc class show dev ifb0
class htb 1:10 root leaf 20: prio 0 rate 100Mbit ceil 100Mbit burst 1600b cburst 1600b

$ sysctl net.ipv4.tcp_congestion_control

 net.ipv4.tcp_congestion_control = bbr or reno

** commands

server:

$netserver -p 5555

client (ports 1234 - 1238):

$ netperf -H [server ip] -p 5555 -l 50 -- -P [port]

Thanks,
Samantha

[Soheil Hassas Yeganeh]

Thank you Samantha for the information!

Looking at your qdisc configs, my first guess is that the queue size of netem is not large enough to accommodate packets of all the flows. Since netem does not have any fair queueing semantics, you may get unproportional losses from experiment to experiment for the 5 flows. Could you please try "limit 1024" or higher for your netem?

I mean:

tc qd replace dev ens4 root netem limit 1024 delay 5.0ms rate 100Mbit

tc qd replace dev eth0 root netem limit 1024 delay 5.0ms rate 100Mbit

Also, another common issue is that netperf's control channel for the late-coming flows can get starved. If that happens, the data channel may start sending data a long time later than the intended start time. Could you please try the "-s" instead?

Here is a script I usually use, to fire 5 netperf processes at 0s, 2s, ... to 8s:

for i in 0 2 4 6 8; do netperf -l $(expr 50 - $i) -H [server] -s $i -- -K bbr -P [port] & done

Please let me know whether the above resolve your issue.

Thanks again!

Soheil

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Eric Dumazet]

Also, if you really want to have small netem limit to emulate a small buffer limit at the bottleneck,

make sure to disable LRO and GRO on the server, otherwise GRO/LRO might add unexpected reorders and unfair selection of flows.

ethtool -K ens4 lro off gro off

[Shiyao Ma]

Hi Eric,

I am interested in the *reorder and unfair selection part*, can you briefly point me to the related kernel C file or the functions?

Thanks.

Best,

[Eric Dumazet]

Simply that if you drop packets in netem, if these logical packets are in fact up to 45 aggregated packets of a single flow,

the effect of dropping 45 packets of flow F1, and dropping 1 packet of flow F2, will likely have very different effect.

Your 'limit of 125' on netem is likely there to model a port limit in term of bytes, but netem does not have the capability to model a port memory limit.

You chose 125, assuming a packet would be MTU sized ? 125 * 1514 -> 189250 bytes

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Samantha Steele]

Hi Eric and Suheil,

Thanks very much for your help. With those settings we are now seeing very consistent convergence.

Best,
Samantha

[Soheil Hassas Yeganeh]

Great, thank you Samantha for the update!

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.