BBR vs CUBIC test on mobile wireless path

[James]

I am doing some research of BBR for my paper. I had porting BBR

to 4.4 kernel, tested the racing between BBR and CUBIC on cellular

phone. And I get something strange.

Test topology: 

PC  <----->  BRIDGE <-----> WIFI  <-----> PHONE (WITH BBR) 192.168.1.104

                                              ^

                                               |---------> PHONE (WITH CUBIC) 192.168.1.105

192.168.1.103            192.168.1.1

(The bridge is used to limit the network speed)

When I use 2.5Ghz with PHONE and WIFI, measured with iperf, start

iperf on phone at the same with command:

  ./iperf -c 192.168.1.103 -t 60 -Z bbr

  ./iperf -c 192.168.1.103 -t 60 -Z cubic

with 10Mbits bandwidth limit on bridge:

  BBR       5.64  4.27   5.31   6.82   5.73

  CUBIC     2.80  5.36   4.36   3.28   3.92

with 40Mbits bandwidth limit on bridge:

  BBR       21.5  19.8   18.5   20.1   22.6

  CUBIC     16.6  16.56  18.63  17.43  15.59

But when I test without bandwidth limit, BBR become worst:

  BBR       7.92  4.00  9.56  3.59

  CUBIC     37.3  41.7  35.5  42.2

I had saw BBR has issue when sharing deep buffers from:

  https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf

This should be the same issue. But the question is when limit the bandwidth

this issue is gone.

I am try to fix this issue by check both BW and RTT, when get half BW and 2*RTT

after CYCLE_LEN rounds elapsed, we need to probe more BW. The patch like the

following:(the rs->rtt_us may be replace with min_rtt measured like bbr_max_bw())

diff --git a/net/ipv4/tcp_bbr.c b/net/ipv4/tcp_bbr.c

index dbcc935..f9164b7 100644

--- a/net/ipv4/tcp_bbr.c

+++ b/net/ipv4/tcp_bbr.c

@@ -452,6 +452,15 @@ static void bbr_advance_cycle_phase(struct sock *sk)

        bbr->pacing_gain = bbr_pacing_gain[bbr->cycle_idx];

 }

+/* running in half full_bw and 2*RTT after CYCLE_LEN rounds elapsed */

+static bool bbr_need_more_available_bw(struct sock *sk, const struct rate_sample *rs)

+{

+       struct bbr *bbr = inet_csk_ca(sk);

+

+       return !rs->losses && bbr_max_bw(sk) < (bbr->full_bw >> 1) &&

+              bbr->min_rtt_us < (rs->rtt_us >> 1);

+}

+

 /* Gain cycling: cycle pacing gain to converge to fair share of available bw. */

 static void bbr_update_cycle_phase(struct sock *sk,

                                   const struct rate_sample *rs)

@@ -459,8 +468,16 @@ static void bbr_update_cycle_phase(struct sock *sk,

        struct bbr *bbr = inet_csk_ca(sk);

        if ((bbr->mode == BBR_PROBE_BW) && !bbr->lt_use_bw &&

-           bbr_is_next_cycle_phase(sk, rs))

+           bbr_is_next_cycle_phase(sk, rs)) {

+               if (bbr_need_more_available_bw(sk, rs)) {

+                       /* reset RTT since current min RTT doubled */

+                       bbr->min_rtt_us = rs->rtt_us;

+                       bbr->min_rtt_stamp = tcp_jiffies32;

+                       /* probe for more available bw */

+                       bbr->cycle_idx = CYCLE_LEN - 1;

+               }

                bbr_advance_cycle_phase(sk);

+       }

 }

 static void bbr_reset_startup_mode(struct sock *sk)

After patch,  BBR become better when racing with CUBIC.

Is this a good way to start? Any advice for the patch or this issue?

Are there some know issues of BBR on mobile or wireless path?

[Neal Cardwell]

Hi James,

Thanks for the report. We would have to look at the packet traces to be sure, but I suspect you are running into the known issues with the BBR cwnd computation being too conservative to reach full utilization for wifi LAN paths, due to the structure of the ACK stream in wifi bottlenecks. Here is an example bbr-dev thread with some discussion, traces, and analysis:

  https://groups.google.com/forum/#!msg/bbr-dev/zUrcENm9rZI/Ea28juVoFAAJ

We are actively working on this, and we'll post some patches when we have something ready for wider testing.

thanks,

neal

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James]

Hi Neal,

Thanks for your reply. I will post the pcap soon.

The performance for BBR is almost the same to CUBIC when no racing. only

when BBR and CUBIC used the same time, CUBIC get most of the bandwidth.

I saw the code, and it seem that after reach full bw, then keep in PROBE_BW state,

but the bw keep reduced to a small value, real rtt become bigger, min_rtt not change

until expried, BBR never have a chance to get more bw when racing with cubic.

After the min rtt is expried(bbr_min_rtt_win_sec), still return to PROBE_BW state.

But the PROBE_RTT state may cause bw reduce more worse.

Maybe something like the patch I posted is need, either enter STARTUP state or

use 5/4 pacing in PROBE_BW state to probe more BW I guess.

Any idea or something I can try to do?

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

[James]

pcap file can be download from https://drive.google.com/file/d/0Bx9se6bCgQ68aDlxeEJldUpnMVE/view?usp=sharing

在 2017年7月4日星期二 UTC+8下午10:26:57，Neal Cardwell写道：

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

[Neal Cardwell]

Hi James,

Thanks for the traces. Can you please add a little more detail about the traces?

For example:

o What sort of scenario is being traced here?

  o what are the sending and receiving machines?

  o what is the bottleneck link?

  o what is the "bridge", and what is it configured to do, exactly?

  o what congestion control algorithms are the flows running?

     o is the BBR in the trace vanilla BBR or a kernel with the patch you specified?

o This seems to be a receiver-side trace, based on the delays between the data and the ACKs; would you be able to take a trace from the sender, to show us what the sending congestion control is seeing? That's usually easier to interpret.

o Also interesting would be the output of "ss", to show the internal BBR model of BtlBw and RTprop; e.g.:

  (for i in `seq 1 60`; do ss -tin 'dst <destination_host>'; sleep 1; done) &

One interesting aspect is that (based on the attached screen shots of tcptrace output) the two flows you show seem to converge to a roughly fair share (About 19Mbps each) after a phase transition around 30 seconds into the trace. I could speculate as to the cause, but it would be nice to know more about the scenario before I speculate. :-)

Thanks!

neal

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[James]

Hi Neal,

Hi James,

Thanks for the traces. Can you please add a little more detail about the traces?

For example:

o What sort of scenario is being traced here?

  o what are the sending and receiving machines?

The send side are mobile devices and the receiving side is a notebook.

 

  o what is the bottleneck link?

The path between the mobile devices and the WIFI route is the bottleneck link.

It is only has ~46Mbits, WIFI to bridge and bridge to notebook have 100Mbits.

 

  o what is the "bridge", and what is it configured to do, exactly?

I used the bridge to emulate loss and bandwith control by using netem.

  o what congestion control algorithms are the flows running?

     o is the BBR in the trace vanilla BBR or a kernel with the patch you specified?

I use the vanilla BBR with bbr_min_rtt_win_sec set to 0;

o This seems to be a receiver-side trace, based on the delays between the data and the ACKs; would you be able to take a trace from the sender, to show us what the sending congestion control is seeing? That's usually easier to interpret.

o Also interesting would be the output of "ss", to show the internal BBR model of BtlBw and RTprop; e.g.:

  (for i in `seq 1 60`; do ss -tin 'dst <destination_host>'; sleep 1; done) &

I do not have a ss now, and will report after compile the ss command.

 

One interesting aspect is that (based on the attached screen shots of tcptrace output) the two flows you show seem to converge to a roughly fair share (About 19Mbps each) after a phase transition around 30 seconds into the trace. I could speculate as to the cause, but it would be nice to know more about the scenario before I speculate. :-)

The last file I posted has packet drop.

I had traced from the sender around 30 seconds in a less lost rate env.