BBR high RTT unfairness: Fifty Shades of Congestion Control: A Performance and Interactions Evaluation
374 views
Skip to first unread message
Dave Taht
May 29, 2019, 11:05:37 AM5/29/19
to BBR Development, bloat
I have been trying to work through this paper:
https://arxiv.org/pdf/1903.03852.pdf
which is enormous and well worth reading.
I have a theory, though, about TABLE XII, which contrasts four BBR
flows at different RTTs, in that BBRv1's probe phase makes a 200ms
assumption, thus
not seeing the real rtt at ong rtts, and thus the longest RTT flow
gets the most bandwidth on this test, and the second (testable) theory
is that were these rtts not exactly on the 100ms boundaries, we would
see more throughput fairness.
and thus we end up with the
tput
BBR(100ms) 0.94 95.90 1.02 1.02 0.62
BBR(200ms) 2.36 95.89 2.53 2.48
BBR(300ms) 1.09 95.89 1.20 1.14
BBR(400ms) 4.57 95.89 4.79 4.59
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
https://arxiv.org/pdf/1903.03852.pdf
which is enormous and well worth reading.
I have a theory, though, about TABLE XII, which contrasts four BBR
flows at different RTTs, in that BBRv1's probe phase makes a 200ms
assumption, thus
not seeing the real rtt at ong rtts, and thus the longest RTT flow
gets the most bandwidth on this test, and the second (testable) theory
is that were these rtts not exactly on the 100ms boundaries, we would
see more throughput fairness.
and thus we end up with the
tput
BBR(100ms) 0.94 95.90 1.02 1.02 0.62
BBR(200ms) 2.36 95.89 2.53 2.48
BBR(300ms) 1.09 95.89 1.20 1.14
BBR(400ms) 4.57 95.89 4.79 4.59
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
Anant Shah
May 29, 2019, 2:54:12 PM5/29/19
to Dave Taht, BBR Development
Great paper, thanks for sharing. A few months back when we tested BBRv1 at the CDN edge, we saw similar unfairness among BBR flows as well.
--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CAA93jw7HB4D2zjHx7okdFVPpkQerXb-qZ_8-5fC-4R7zA4VzDA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Neal Cardwell
May 29, 2019, 6:02:09 PM5/29/19
to Anant Shah, Dave Taht, BBR Development
Dave, thanks for raising this paper for discussion.
Please note that the 200ms constant in the BBRv1 PROBE_RTT mechanism does not limit the mechanism to working for RTTs of 200ms and lower. The mechanism can work just fine for higher RTTs. Attached is a screenshot illustrating an example of the RTT samples and min_rtt estimate for 3 flows, each with a 400ms RTT, and starting at different times. The diagram illustrates how the flows can coordinate to maintain good estimates of the two-way propagation delay, even with RTTs that are quite long.
In all of the cases I've seen with unfairness due to differing min_rtt values, the dominant factor is simply that with BBRv1 each flow has a cwnd that is basically 2*bw*min_rtt, which tends to try to maintain 1*bw*min_rtt in the bottleneck queue, which quite directly means that flows with higher min_rtt values maintain more packets in the bottleneck queue and therefore get a higher fraction of the bottleneck bandwidth. The most direct way I'm aware of to improve RTT fairness in the BBR framework is to get rid of that excess queue, or ensure that the amount of queue is independent of a flow's min_rtt estimate.
cheers,
neal
To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CA%2BXbCHVRCNb2G-kEgQdbDmHMBUTcUiazw-KGLVe%2BPH0ZM%2BXJiw%40mail.gmail.com.
Roland Bless
May 30, 2019, 7:31:09 AM5/30/19
to Dave Taht, BBR Development, bloat
Hi Dave,
On 29.05.19 at 17:05 Dave Taht wrote:
> I have been trying to work through this paper:
> https://arxiv.org/pdf/1903.03852.pdf
> which is enormous and well worth reading.
>
> I have a theory, though, about TABLE XII, which contrasts four BBR
> flows at different RTTs, in that BBRv1's probe phase makes a 200ms
> assumption, thus
> not seeing the real rtt at ong rtts, and thus the longest RTT flow
> gets the most bandwidth on this test, and the second (testable) theory
> is that were these rtts not exactly on the 100ms boundaries, we would
> see more throughput fairness.
Nope, the main reason for RTT unfairness in BBRv1 is its
CWnd cap at 2*(RTT_min*est_bw) (2*estimated bottleneck BDP share).
As we showed in http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
Section III: multiple BBR flows will always increase their CWnd up
to this point (except when the buffer capacity is smaller than a BDP).
Neal's explanation is in line with our findings.
Consequently, each flow will converge towards a share of RTT_min*est_bw
at the bottleneck queue, providing a larger bandwidth share for flows
with a larger RTT_min. See also Section V.F of our paper that also
evaluated RTT unfairness (moreover, the outcome depends also on the
bottleneck buffer size).
Unfortunately, they didn't test TCP-LoLa in this context, since it is
actually able to provide fairness among flows with different RTTs
(while still limiting the overall queuing delay). Moreover, Mario
and Felix improved the convergence speed by introducing FFBquick, see:
http://doc.tm.kit.edu/Poster/2019-FFBquick_Networking.pdf
for a quick glance on the challenges and the solution.
This was published as poster paper at Networking 2019:
M. Hock, R. Bless, F. Neumeister, M. Zitterbart: FFBquick: Fast
Convergence to Fairness for Delay-bounded Congestion Controls,
Networking 2019, Warsaw, Poland, May 20-22.
Regards
Roland
On 29.05.19 at 17:05 Dave Taht wrote:
> I have been trying to work through this paper:
> https://arxiv.org/pdf/1903.03852.pdf
> which is enormous and well worth reading.
>
> I have a theory, though, about TABLE XII, which contrasts four BBR
> flows at different RTTs, in that BBRv1's probe phase makes a 200ms
> assumption, thus
> not seeing the real rtt at ong rtts, and thus the longest RTT flow
> gets the most bandwidth on this test, and the second (testable) theory
> is that were these rtts not exactly on the 100ms boundaries, we would
> see more throughput fairness.
CWnd cap at 2*(RTT_min*est_bw) (2*estimated bottleneck BDP share).
As we showed in http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
Section III: multiple BBR flows will always increase their CWnd up
to this point (except when the buffer capacity is smaller than a BDP).
Neal's explanation is in line with our findings.
Consequently, each flow will converge towards a share of RTT_min*est_bw
at the bottleneck queue, providing a larger bandwidth share for flows
with a larger RTT_min. See also Section V.F of our paper that also
evaluated RTT unfairness (moreover, the outcome depends also on the
bottleneck buffer size).
Unfortunately, they didn't test TCP-LoLa in this context, since it is
actually able to provide fairness among flows with different RTTs
(while still limiting the overall queuing delay). Moreover, Mario
and Felix improved the convergence speed by introducing FFBquick, see:
http://doc.tm.kit.edu/Poster/2019-FFBquick_Networking.pdf
for a quick glance on the challenges and the solution.
This was published as poster paper at Networking 2019:
M. Hock, R. Bless, F. Neumeister, M. Zitterbart: FFBquick: Fast
Convergence to Fairness for Delay-bounded Congestion Controls,
Networking 2019, Warsaw, Poland, May 20-22.
Regards
Roland
Dave Taht
May 30, 2019, 9:38:42 AM5/30/19
to Roland Bless, BBR Development, bloat
On Thu, May 30, 2019 at 4:31 AM Roland Bless <roland...@kit.edu> wrote:
>
> Hi Dave,
>
> On 29.05.19 at 17:05 Dave Taht wrote:
> > I have been trying to work through this paper:
> > https://arxiv.org/pdf/1903.03852.pdf
> > which is enormous and well worth reading.
> >
> > I have a theory, though, about TABLE XII, which contrasts four BBR
> > flows at different RTTs, in that BBRv1's probe phase makes a 200ms
> > assumption, thus
> > not seeing the real rtt at ong rtts, and thus the longest RTT flow
> > gets the most bandwidth on this test, and the second (testable) theory
> > is that were these rtts not exactly on the 100ms boundaries, we would
> > see more throughput fairness.
>
> Nope, the main reason for RTT unfairness in BBRv1 is its
> CWnd cap at 2*(RTT_min*est_bw) (2*estimated bottleneck BDP share).
The striking thing about that table was that the 300ms result was the
>
> Hi Dave,
>
> On 29.05.19 at 17:05 Dave Taht wrote:
> > I have been trying to work through this paper:
> > https://arxiv.org/pdf/1903.03852.pdf
> > which is enormous and well worth reading.
> >
> > I have a theory, though, about TABLE XII, which contrasts four BBR
> > flows at different RTTs, in that BBRv1's probe phase makes a 200ms
> > assumption, thus
> > not seeing the real rtt at ong rtts, and thus the longest RTT flow
> > gets the most bandwidth on this test, and the second (testable) theory
> > is that were these rtts not exactly on the 100ms boundaries, we would
> > see more throughput fairness.
>
> Nope, the main reason for RTT unfairness in BBRv1 is its
> CWnd cap at 2*(RTT_min*est_bw) (2*estimated bottleneck BDP share).
~same as the 100ms result for throughput, while the ones on the 200
and 400ms ones were 2x and 4x respectively.
My thought was that at extraordinary RTTs (anything > planet girdling
e.g. > 200ms) that trying a probe of 250ms (or some degree of variance
periodically - 220ms, 260ms, 180ms) or changing the period of the
probe itself, would desync things and get closer to the real RTT,
particularly when BBR was duking it out with itself.
This would also make up for researchers (which includes myself until I
trained myself out of it) tending to always start a test with multiple
flows all at exactly the same time, which could be another flaw in
this dataset.
> As we showed in http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
> Section III: multiple BBR flows will always increase their CWnd up
> to this point (except when the buffer capacity is smaller than a BDP).
> Neal's explanation is in line with our findings.
> Consequently, each flow will converge towards a share of RTT_min*est_bw
> at the bottleneck queue, providing a larger bandwidth share for flows
> with a larger RTT_min. See also Section V.F of our paper that also
> evaluated RTT unfairness (moreover, the outcome depends also on the
> bottleneck buffer size).
synced flows...
> Unfortunately, they didn't test TCP-LoLa in this context, since it is
> actually able to provide fairness among flows with different RTTs
> (while still limiting the overall queuing delay).
shades of congestion control going forward (dctcp, bbrv2, lola, fu)
and a stable base of data to work from.
Me being me I'd also like to vary the fq and aqm algorithms using the
same test setups.
I'll ping the authors.
> Moreover, Mario
> and Felix improved the convergence speed by introducing FFBquick, see:
> http://doc.tm.kit.edu/Poster/2019-FFBquick_Networking.pdf
> for a quick glance on the challenges and the solution.
> This was published as poster paper at Networking 2019:
> M. Hock, R. Bless, F. Neumeister, M. Zitterbart: FFBquick: Fast
> Convergence to Fairness for Delay-bounded Congestion Controls,
> Networking 2019, Warsaw, Poland, May 20-22.
>
> Regards
> Roland
>
Roland Bless
May 31, 2019, 8:39:10 AM5/31/19
to Dave Taht, BBR Development, bloat
Hi Dave,
On 30.05.19 at 15:38 Dave Taht wrote:
> On Thu, May 30, 2019 at 4:31 AM Roland Bless <roland...@kit.edu> wrote:
>>
>> Hi Dave,
>>
>> On 29.05.19 at 17:05 Dave Taht wrote:
>>> I have been trying to work through this paper:
>>> https://arxiv.org/pdf/1903.03852.pdf
>>> which is enormous and well worth reading.
>>>
>>> I have a theory, though, about TABLE XII, which contrasts four BBR
>>> flows at different RTTs, in that BBRv1's probe phase makes a 200ms
>>> assumption, thus
>>> not seeing the real rtt at ong rtts, and thus the longest RTT flow
>>> gets the most bandwidth on this test, and the second (testable) theory
>>> is that were these rtts not exactly on the 100ms boundaries, we would
>>> see more throughput fairness.
>>
>> Nope, the main reason for RTT unfairness in BBRv1 is its
>> CWnd cap at 2*(RTT_min*est_bw) (2*estimated bottleneck BDP share).
>
> The striking thing about that table was that the 300ms result was the
> ~same as the 100ms result for throughput, while the ones on the 200
> and 400ms ones were 2x and 4x respectively.
Yes, correct.
> My thought was that at extraordinary RTTs (anything > planet girdling
> e.g. > 200ms) that trying a probe of 250ms (or some degree of variance
> periodically - 220ms, 260ms, 180ms) or changing the period of the
> probe itself, would desync things and get closer to the real RTT,
> particularly when BBR was duking it out with itself.
>
> This would also make up for researchers (which includes myself until I
> trained myself out of it) tending to always start a test with multiple
> flows all at exactly the same time, which could be another flaw in
> this dataset.
You are right and we ran into this trap also at first and
experienced similar effects in our BBR evaluations. Therefore,
we used starting times that are not multiple of 10s (0s, 23s, 31s, 38s,
42s, 47s), because BBR probes every 10s for the min RTT, cf. Section V.D
of our paper below.
>> As we showed in http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
>> Section III: multiple BBR flows will always increase their CWnd up
>> to this point (except when the buffer capacity is smaller than a BDP).
>> Neal's explanation is in line with our findings.
>> Consequently, each flow will converge towards a share of RTT_min*est_bw
>> at the bottleneck queue, providing a larger bandwidth share for flows
>> with a larger RTT_min. See also Section V.F of our paper that also
>> evaluated RTT unfairness (moreover, the outcome depends also on the
>> bottleneck buffer size).
>
> I get it, it's my point above about not seeing RTT_min properly with
> synced flows...
I'm not sure, my explanations hold even if all flows see the proper
RTT_min. It seems that the 300ms RTT flow has difficulties in getting
a larger share.
On 30.05.19 at 15:38 Dave Taht wrote:
> On Thu, May 30, 2019 at 4:31 AM Roland Bless <roland...@kit.edu> wrote:
>>
>> Hi Dave,
>>
>> On 29.05.19 at 17:05 Dave Taht wrote:
>>> I have been trying to work through this paper:
>>> https://arxiv.org/pdf/1903.03852.pdf
>>> which is enormous and well worth reading.
>>>
>>> I have a theory, though, about TABLE XII, which contrasts four BBR
>>> flows at different RTTs, in that BBRv1's probe phase makes a 200ms
>>> assumption, thus
>>> not seeing the real rtt at ong rtts, and thus the longest RTT flow
>>> gets the most bandwidth on this test, and the second (testable) theory
>>> is that were these rtts not exactly on the 100ms boundaries, we would
>>> see more throughput fairness.
>>
>> Nope, the main reason for RTT unfairness in BBRv1 is its
>> CWnd cap at 2*(RTT_min*est_bw) (2*estimated bottleneck BDP share).
>
> The striking thing about that table was that the 300ms result was the
> ~same as the 100ms result for throughput, while the ones on the 200
> and 400ms ones were 2x and 4x respectively.
> My thought was that at extraordinary RTTs (anything > planet girdling
> e.g. > 200ms) that trying a probe of 250ms (or some degree of variance
> periodically - 220ms, 260ms, 180ms) or changing the period of the
> probe itself, would desync things and get closer to the real RTT,
> particularly when BBR was duking it out with itself.
>
> This would also make up for researchers (which includes myself until I
> trained myself out of it) tending to always start a test with multiple
> flows all at exactly the same time, which could be another flaw in
> this dataset.
experienced similar effects in our BBR evaluations. Therefore,
we used starting times that are not multiple of 10s (0s, 23s, 31s, 38s,
42s, 47s), because BBR probes every 10s for the min RTT, cf. Section V.D
of our paper below.
>> As we showed in http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf
>> Section III: multiple BBR flows will always increase their CWnd up
>> to this point (except when the buffer capacity is smaller than a BDP).
>> Neal's explanation is in line with our findings.
>> Consequently, each flow will converge towards a share of RTT_min*est_bw
>> at the bottleneck queue, providing a larger bandwidth share for flows
>> with a larger RTT_min. See also Section V.F of our paper that also
>> evaluated RTT unfairness (moreover, the outcome depends also on the
>> bottleneck buffer size).
>
> I get it, it's my point above about not seeing RTT_min properly with
> synced flows...
RTT_min. It seems that the 300ms RTT flow has difficulties in getting
a larger share.
Reply all
Reply to author
Forward
0 new messages