BBR Report

[Lawrence Brakmo]

I ran some tests under a couple of different scenarios and have posted my results here: https://drive.google.com/file/d/0B4YZ_0yTgbJEa21CbUVLWFdrX2c/view?usp=sharing

My primary concern is that BBR can be very aggressive when it determines, incorrectly in many of my test instances, that losses are not correlated to congestion. When in this mode, any non-BBR traffic sharing the bottleneck will starve. Although in some instances, even some BBR flows are starved, probably because they did not get into the "losses are not correlated to congestion" mode.

[Yuchung Cheng]

Hi Larry

Thanks for the detailed testing of BBR. I have not go through the
report in detail (prepare to travel to netdev 1.2).

The major issue (high loss at various scenarios) you identified
matches what we have seen (before the upstream), and what
has been reported externally (i.e. thread from Brian Tierney). This
is the issue we are actively going after.

One of issue is the lag of BW windowed-max-filter will discard the
lower delivery rate, even when the reduction was driven by buffer
overflow. A common scenario we've seen on the Internet is, BBR is
slow-starting into a un-bloated buffer, but some cross-traffic shows
up and grabbing half bw, and drive buffer full. but until the bw
filter expires, BBR will continue to charge at previously full bw for
several rounds trips, inducing considerable amount losses.

How we improve bbr's processing on loss signals have been an active
research area. This part is further complicated by the drops caused by
token bucket policers. How'd NV handle these situations?

On Fri, Sep 30, 2016 at 7:49 AM, Lawrence Brakmo <bra...@gmail.com> wrote:
> I ran some tests under a couple of different scenarios and have posted my results here: https://drive.google.com/file/d/0B4YZ_0yTgbJEa21CbUVLWFdrX2c/view?usp=sharing
>
> My primary concern is that BBR can be very aggressive when it determines, incorrectly in many of my test instances, that losses are not correlated to congestion. When in this mode, any non-BBR traffic sharing the bottleneck will starve. Although in some instances, even some BBR flows are starved, probably because they did not get into the "losses are not correlated to congestion" mode.
>

> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Lawrence Brakmo]

Hi Yuchung,

As soon as I get some free time, I will look at the implementation of the bw filter in BBR. Vegas and NV also use windowed min(RTT) and max(bw) filters, but their windows are generally much smaller than BBR's. The max(bw) window is a minimum of 2 RTTs, but it also depends on the number of measurements it gets. The upstream NV patch waits until there are 60 measurements (RTT and bw) before making a cwnd reduction decision, and 20 for making a cwnd increase decision.

Regarding token bucket policers, the current NV implementation is meant for the DC and not really for the Internet (other than used as low priority). I am working on some new ideas for NV, but it will take a few more weeks to get done.

Have a great trip to netdev!

- Larry

> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Jay Vosburgh]

'Yuchung Cheng' via BBR Development <bbr...@googlegroups.com> wrote:

>Thanks for the detailed testing of BBR. I have not go through the
>report in detail (prepare to travel to netdev 1.2).

Yuchung,

Will you be having a session at netdev (perhaps as an ad-hoc at
some point) to discuss BBR?

I've done some testing of BBR between our data centers in the US
and UK (80ms RTT) to see how BBR responds to competing low-RTT flows
causing packet loss upstream from the sender. CUBIC handles this very
poorly; BBR does much better, but, as Larry described, I also see BBR
starving out CUBIC flows (e.g., a BBR host and a CUBIC host both sending
to the same 80ms distant peer).

-J

---
-Jay Vosburgh, jay.vo...@canonical.com

[Neal Cardwell]

Jay -

Yes, Yuchung and I will be talking about BBR, RACK, and how they fit with the rest of Linux TCP, in a netdev 1.2 talk this coming Wednesday, in Tokyo:

  http://netdevconf.org/1.2/schedule.html

We are actively working on reducing the buffer pressure exerted by BBR, which should help in the sorts of CUBIC vs BBR scenarios you describe (as well as BBR vs BBR scenarios).

thanks,

neal

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Yuchung Cheng]

Due to the time constraint (35m), the talk will only touch the very
high-level aspect of bbr. But let's meet and discuss the details! we
are there for the entire conf.

>> email to bbr-dev+u...@googlegroups.com.

[renaud sallantin]

Hi,

Is it possible to get the slides, or is there a recording of your speech?

BR,
Renaud

[Neal Cardwell]

On Mon, Oct 10, 2016 at 4:01 AM, renaud sallantin <renaud.s...@gmail.com> wrote:

Hi,

Is it possible to get the slides, or is there a recording of your speech?

BR,
Renaud

Here is the YouTube video of the talk Yuchung and I gave at the Linux netdev 1.2 conference in Tokyo last week. We discussed RACK loss recovery and BBR congestion control, and how they fit together with packet scheduling:

  https://www.youtube.com/watch?v=hIl_zXzU3DA

The slides and paper associated with this talk will be released later.

neal

[Brian Tierney]

Sorry if I missed it, but has there been a code update that addresses this issue?

[Neal Cardwell]

Hi Brian,

There have been no changes to the upstream Linux TCP BBR behavior since the initial release in v4.9.

However, internally the Google BBR team has been experimenting with a number of tweaks in the lab and on YouTube servers.

We wanted to share one experimental patch in particular (patch against net-next attached) that is aimed at reducing queue pressure, to reduce delay and packet drops. The idea is to try a bit harder to drain the queue on each pacing gain cycle by allowing ourselves to spend more time in the sub-unity pacing gain phase. We would be interested in feedback from anyone who has time to try it out.

In particular it should help behavior somewhat in cases like these, where there are multiple BBR flows sharing a WAN bottleneck with moderately sized buffers:

   http://downloads.es.net/pub/tierney/BBR/Berkeley-WashDC-4streams/

I should emphasize that this patch is still very experimental, and is only one tweak among many that we're testing. But we would be interested in feedback from anyone who has time to play with it.

Thanks!

neal

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

[Brian Tierney]

Unfortunately there seems to be no noticeable impact from the patch.

Here are a couple sample results:

before patch:

>bwctl -c bschool-ndt.net.unc.edu -t20 -P 2
[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-20.00  sec  2.09 GBytes   897 Mbits/sec  133147   

> bwctl -c perfsonar.nssl.noaa.gov -t20 -P 2
[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-20.00  sec  1.48 GBytes   635 Mbits/sec  108409  

after patch:

>bwctl -c bschool-ndt.net.unc.edu -t20 -P 2

[ ID] Interval           Transfer     Bandwidth       Retr

[SUM]   0.00-20.00  sec  1.94 GBytes   832 Mbits/sec  108628        

> bwctl -c perfsonar.nssl.noaa.gov -t20 -P 2

[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-20.00  sec  2.12 GBytes   911 Mbits/sec  138121

For this run tcptrace reports 1,421,927 packets sent, and 138,121 packets retransmitted, for a .097% retransmit rate, which pretty high, IMHO. 

And for comparison, cubic:

[SUM]   0.00-20.00  sec  1.10 GBytes   474 Mbits/sec  721           

[SUM]   0.00-20.00  sec  1.34 GBytes   574 Mbits/sec  620  

So BBR throughput is much better, but at the cost of a lot more retransmits. 

 

--

Brian Tierney, http://www.es.net/tierney
Energy Sciences Network (ESnet), Berkeley National Lab
http://fasterdata.es.net

[Neal Cardwell]

Thanks, Brian, for taking that for a spin. Would you be able to take a tcpdump trace of about 10 secs of the BBR transfers? Capturing just headers would be ideal (e.g. "tcpdump -s 100 -w trace.pcap").

thanks,

neal

[Dave Taht]

I am irked by sub 20 second tests. These plague the internet. I am
constantly seeing networks explode at T+22 seconds. Can you run your
tests for longer - at least a minute - please?

> Brian Tierney, http://www.es.net/tierney
> Energy Sciences Network (ESnet), Berkeley National Lab
> http://fasterdata.es.net
>

> --
> You received this message because you are subscribed to the Google Groups
> "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbr-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

[Brian Tierney]

tcpdump in: http://downloads.es.net/pub/tierney/BBR/Dec2016/

Dave: I ran a 60sec test too, and got the same retransmit rate.

[Neal Cardwell]

On Thu, Dec 15, 2016 at 6:03 PM, Brian Tierney <blti...@es.net> wrote:

tcpdump in: http://downloads.es.net/pub/tierney/BBR/Dec2016/

Thanks, Brian. That trace was very helpful. Looks like that path had a bottleneck of about 1 Gbps, with a shallow bottleneck buffer.

Unfortunately, the patch I posted on Dec 12th was not expected to help much for the shallow-buffered case. That patch was targeting an aspect of BBR behavior that shows up more in the kind of scenario reflected in your traces from Sep 22 of this year, where there were multiple flows and a decent amount of buffering (roughly a BDP of buffer):

   http://downloads.es.net/pub/tierney/BBR/Berkeley-WashDC-4streams/

If you have time to try that patch on that Berkeley->WashDC path, it would be interesting to see what results you get.

As for the shallow-buffered case, we will get back to you when we have a patch series ready for wider testing.

Thanks,

neal

[Brian Tierney]

The particular host in WashDC no longer seems to be up, but I found a host in Atlanta where I think all the routers in the path have decent buffering.

tcpdump available here:

http://downloads.es.net/pub/tierney/BBR/Dec2016/tcpdump.Berkeley-Atlanta.tar.gz

There are still a ton of retransmits.

I'll keep an eye out for the DC host coming back up.

[Neal Cardwell]

Thanks, Brian. I will take a look.

neal

[Brian Tierney]

Neal:

The host in WashDC is back, and I reran the test.

The patch seems to help a lot on that path.

Retransmits dropped from over 30000 to around 18000 packets for a 60 second 4 stream test.

Retransmits are still about 10x more than cubic tho.

tcpdump files (both with and without the patch, and cubic) are available here:

http://downloads.es.net/pub/tierney/BBR/Jan2017/

Let me know if you have other patches you'd like tested that might help on shallow-buffered paths.

[Neal Cardwell]

Thank you, Brian, for running all these tests!

We will analyze these and get back to you when we have more patches
for wider testing.

thanks,
neal

[Sean Hull]

NICE TESTING!!!

This is exactly what you get with BBR when the algorithm is so aggressive that it can not handle all loss scenerios. 

DCTCP and Vegas would be the stacks I respect after looking at your results.