v2 is weak against packet loss?
351 views
Skip to first unread message
Akira Nakajima
Feb 1, 2021, 6:57:15 PM2/1/21
to BBR Development
Result of my testing,
v1 is strong against packet loss (High throughput until 10% loss),
But v2 is weak against packet loss (1/8 throughput 1% loss)
Does this mean that we need choice v1 or v2 according to the packet loss?
I did not test in an ECN-enabled environment because of no having environment.
High throughput in an ECN-enabled environment?
v1 is strong against packet loss (High throughput until 10% loss),
But v2 is weak against packet loss (1/8 throughput 1% loss)
Does this mean that we need choice v1 or v2 according to the packet loss?
I did not test in an ECN-enabled environment because of no having environment.
High throughput in an ECN-enabled environment?
Neal Cardwell
Feb 1, 2021, 9:12:43 PM2/1/21
to Akira Nakajima, BBR Development
On Mon, Feb 1, 2021 at 6:57 PM Akira Nakajima <girn...@gmail.com> wrote:
Result of my testing,
v1 is strong against packet loss (High throughput until 10% loss),
But v2 is weak against packet loss (1/8 throughput 1% loss)
In our tests we are seeing BBRv2 able to maintain the majority of the available bandwidth up through 1% loss:
What is the test process you are using to produce the 1% loss?
Does this mean that we need choice v1 or v2 according to the packet loss?
BBRv2 is still not finalized, and it has parameters that can be tuned to allow it to tolerate the same level of loss as BBRv1. So let's start with a specific use case and discuss what kind of algorithm behavior makes sense. Is there a particular real-world scenario with a path that has a sustained exogenous/random loss rate above 1% that you are concerned about? Are you able to share some details about it?
I did not test in an ECN-enabled environment because of no having environment.
High throughput in an ECN-enabled environment?
Yes, BBRv2 should be able to fully utilize a bottleneck link that uses ECN.
best,
neal
--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/45daafb4-1602-46ac-995c-830d633cd555n%40googlegroups.com.
Akira Nakajima
Feb 5, 2021, 1:24:01 AM2/5/21
to BBR Development
I'm not thinking about specific use cases, I want to know the specifications.
I just want to know if BBRv2 replaces BBRv1 or uses it depending on the line quiality.
>> https://datatracker.ietf.org/meeting/105/materials/slides-105-iccrg-bbr-v2-a-model-based-congestion-control-00#page=6
This is measured in 2 minutes. When measured in 30 minutes, a throughput of 125 Mbps can be obtained.
In my environment, real-time rate display using iperf3 shows that the initial velocity is close to 1 Gbps, but it gradually decreases and converges to about 100 Mbps.
# tc qdisc add dev ens3 root netem rate 1Gbit limit 8657 delay 20m loss 1%
net.ipv4.tcp_congestion_control = bbr2
net.core.default_qdisc = fq
net.ipv4.tcp_ecn = 2
Linux cent82-sender2 5.10.0 #1 SMP Wed Jan 27 12:13:36 JST 2021 x86_64 x86_64 x86_64 GNU/Linux
# iperf3 -c 192.168.122.2 -t 60
Connecting to host 192.168.122.2, port 5201
[ 5] local 192.168.122.13 port 38876 connected to 192.168.122.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 133 MBytes 1.12 Gbits/sec 461 2.49 MBytes
[ 5] 1.00-2.00 sec 77.5 MBytes 650 Mbits/sec 945 5.61 MBytes
[ 5] 2.00-3.00 sec 95.0 MBytes 797 Mbits/sec 765 2.24 MBytes
[ 5] 3.00-4.00 sec 91.2 MBytes 766 Mbits/sec 675 2.42 MBytes
[ 5] 4.00-5.00 sec 76.2 MBytes 640 Mbits/sec 675 1.96 MBytes
[ 5] 5.00-6.00 sec 67.5 MBytes 566 Mbits/sec 505 1.86 MBytes
[ 5] 6.00-7.00 sec 75.0 MBytes 629 Mbits/sec 549 1.69 MBytes
[ 5] 7.00-8.00 sec 66.2 MBytes 556 Mbits/sec 270 1.62 MBytes
[ 5] 8.00-9.00 sec 65.0 MBytes 545 Mbits/sec 495 1.55 MBytes
[ 5] 9.00-10.00 sec 55.0 MBytes 461 Mbits/sec 488 1.67 MBytes
[ 5] 10.00-11.00 sec 65.0 MBytes 545 Mbits/sec 675 1.55 MBytes
[ 5] 11.00-12.00 sec 67.5 MBytes 566 Mbits/sec 642 1.55 MBytes
[ 5] 12.00-13.00 sec 62.5 MBytes 524 Mbits/sec 586 1.55 MBytes
[ 5] 13.00-14.00 sec 57.5 MBytes 482 Mbits/sec 543 1.24 MBytes
[ 5] 14.00-15.00 sec 58.8 MBytes 493 Mbits/sec 621 1.49 MBytes
[ 5] 15.00-16.00 sec 60.0 MBytes 503 Mbits/sec 402 1.80 MBytes
[ 5] 16.00-17.00 sec 73.8 MBytes 618 Mbits/sec 584 1.74 MBytes
[ 5] 17.00-18.00 sec 62.5 MBytes 525 Mbits/sec 459 1.17 MBytes
[ 5] 18.00-19.00 sec 46.2 MBytes 388 Mbits/sec 291 1.12 MBytes
[ 5] 19.00-20.00 sec 50.0 MBytes 419 Mbits/sec 331 1.07 MBytes
[ 5] 20.00-21.00 sec 41.2 MBytes 346 Mbits/sec 271 1.40 MBytes
[ 5] 21.00-22.00 sec 53.8 MBytes 451 Mbits/sec 410 1.19 MBytes
[ 5] 22.00-23.00 sec 48.8 MBytes 409 Mbits/sec 401 1.22 MBytes
[ 5] 23.00-24.00 sec 47.5 MBytes 399 Mbits/sec 395 1.24 MBytes
[ 5] 24.00-25.00 sec 40.0 MBytes 336 Mbits/sec 492 954 KBytes
[ 5] 25.00-26.00 sec 43.8 MBytes 367 Mbits/sec 202 1018 KBytes
[ 5] 26.00-27.00 sec 38.8 MBytes 325 Mbits/sec 404 771 KBytes
[ 5] 27.00-28.00 sec 30.0 MBytes 252 Mbits/sec 346 700 KBytes
[ 5] 28.00-29.00 sec 28.8 MBytes 241 Mbits/sec 262 665 KBytes
[ 5] 29.00-30.00 sec 27.5 MBytes 231 Mbits/sec 169 636 KBytes
[ 5] 30.00-31.00 sec 25.0 MBytes 210 Mbits/sec 209 585 KBytes
[ 5] 31.00-32.00 sec 26.2 MBytes 220 Mbits/sec 224 636 KBytes
[ 5] 32.00-33.00 sec 26.2 MBytes 220 Mbits/sec 170 686 KBytes
[ 5] 33.00-34.00 sec 26.2 MBytes 220 Mbits/sec 149 600 KBytes
[ 5] 34.00-35.00 sec 25.0 MBytes 210 Mbits/sec 259 560 KBytes
[ 5] 35.00-36.00 sec 21.2 MBytes 178 Mbits/sec 161 560 KBytes
[ 5] 36.00-37.00 sec 22.5 MBytes 189 Mbits/sec 255 510 KBytes
[ 5] 37.00-38.00 sec 30.0 MBytes 252 Mbits/sec 164 764 KBytes
[ 5] 38.00-39.00 sec 33.8 MBytes 283 Mbits/sec 168 757 KBytes
[ 5] 39.00-40.00 sec 30.0 MBytes 252 Mbits/sec 199 731 KBytes
[ 5] 40.00-41.00 sec 30.0 MBytes 252 Mbits/sec 232 734 KBytes
[ 5] 41.00-42.00 sec 26.2 MBytes 220 Mbits/sec 305 731 KBytes
[ 5] 42.00-43.00 sec 28.8 MBytes 241 Mbits/sec 208 636 KBytes
[ 5] 43.00-44.00 sec 22.5 MBytes 189 Mbits/sec 101 509 KBytes
[ 5] 44.00-45.00 sec 21.2 MBytes 178 Mbits/sec 138 498 KBytes
[ 5] 45.00-46.00 sec 18.8 MBytes 157 Mbits/sec 192 445 KBytes
[ 5] 46.00-47.00 sec 20.0 MBytes 168 Mbits/sec 215 467 KBytes
[ 5] 47.00-48.00 sec 18.8 MBytes 157 Mbits/sec 167 467 KBytes
[ 5] 48.00-49.00 sec 18.8 MBytes 157 Mbits/sec 103 390 KBytes
[ 5] 49.00-50.00 sec 13.8 MBytes 115 Mbits/sec 133 318 KBytes
[ 5] 50.00-51.00 sec 12.5 MBytes 105 Mbits/sec 112 318 KBytes
[ 5] 51.00-52.00 sec 12.5 MBytes 105 Mbits/sec 112 376 KBytes
[ 5] 52.00-53.00 sec 13.8 MBytes 115 Mbits/sec 165 318 KBytes
[ 5] 53.00-54.00 sec 12.5 MBytes 105 Mbits/sec 75 311 KBytes
[ 5] 54.00-55.00 sec 12.5 MBytes 105 Mbits/sec 108 296 KBytes
[ 5] 55.00-56.00 sec 11.2 MBytes 94.4 Mbits/sec 136 225 KBytes
[ 5] 56.00-57.00 sec 10.0 MBytes 83.9 Mbits/sec 78 280 KBytes
[ 5] 57.00-58.00 sec 10.0 MBytes 83.9 Mbits/sec 85 296 KBytes
[ 5] 58.00-59.00 sec 12.5 MBytes 105 Mbits/sec 79 267 KBytes
[ 5] 59.00-60.00 sec 12.5 MBytes 105 Mbits/sec 79 269 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 2.35 GBytes 337 Mbits/sec 19095 sender
[ 5] 0.00-60.02 sec 2.31 GBytes 331 Mbits/sec receiver
iperf Done.
regards.
Nakajima.
I just want to know if BBRv2 replaces BBRv1 or uses it depending on the line quiality.
>> https://datatracker.ietf.org/meeting/105/materials/slides-105-iccrg-bbr-v2-a-model-based-congestion-control-00#page=6
This is measured in 2 minutes. When measured in 30 minutes, a throughput of 125 Mbps can be obtained.
In my environment, real-time rate display using iperf3 shows that the initial velocity is close to 1 Gbps, but it gradually decreases and converges to about 100 Mbps.
# tc qdisc add dev ens3 root netem rate 1Gbit limit 8657 delay 20m loss 1%
net.ipv4.tcp_congestion_control = bbr2
net.core.default_qdisc = fq
net.ipv4.tcp_ecn = 2
Linux cent82-sender2 5.10.0 #1 SMP Wed Jan 27 12:13:36 JST 2021 x86_64 x86_64 x86_64 GNU/Linux
# iperf3 -c 192.168.122.2 -t 60
Connecting to host 192.168.122.2, port 5201
[ 5] local 192.168.122.13 port 38876 connected to 192.168.122.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 133 MBytes 1.12 Gbits/sec 461 2.49 MBytes
[ 5] 1.00-2.00 sec 77.5 MBytes 650 Mbits/sec 945 5.61 MBytes
[ 5] 2.00-3.00 sec 95.0 MBytes 797 Mbits/sec 765 2.24 MBytes
[ 5] 3.00-4.00 sec 91.2 MBytes 766 Mbits/sec 675 2.42 MBytes
[ 5] 4.00-5.00 sec 76.2 MBytes 640 Mbits/sec 675 1.96 MBytes
[ 5] 5.00-6.00 sec 67.5 MBytes 566 Mbits/sec 505 1.86 MBytes
[ 5] 6.00-7.00 sec 75.0 MBytes 629 Mbits/sec 549 1.69 MBytes
[ 5] 7.00-8.00 sec 66.2 MBytes 556 Mbits/sec 270 1.62 MBytes
[ 5] 8.00-9.00 sec 65.0 MBytes 545 Mbits/sec 495 1.55 MBytes
[ 5] 9.00-10.00 sec 55.0 MBytes 461 Mbits/sec 488 1.67 MBytes
[ 5] 10.00-11.00 sec 65.0 MBytes 545 Mbits/sec 675 1.55 MBytes
[ 5] 11.00-12.00 sec 67.5 MBytes 566 Mbits/sec 642 1.55 MBytes
[ 5] 12.00-13.00 sec 62.5 MBytes 524 Mbits/sec 586 1.55 MBytes
[ 5] 13.00-14.00 sec 57.5 MBytes 482 Mbits/sec 543 1.24 MBytes
[ 5] 14.00-15.00 sec 58.8 MBytes 493 Mbits/sec 621 1.49 MBytes
[ 5] 15.00-16.00 sec 60.0 MBytes 503 Mbits/sec 402 1.80 MBytes
[ 5] 16.00-17.00 sec 73.8 MBytes 618 Mbits/sec 584 1.74 MBytes
[ 5] 17.00-18.00 sec 62.5 MBytes 525 Mbits/sec 459 1.17 MBytes
[ 5] 18.00-19.00 sec 46.2 MBytes 388 Mbits/sec 291 1.12 MBytes
[ 5] 19.00-20.00 sec 50.0 MBytes 419 Mbits/sec 331 1.07 MBytes
[ 5] 20.00-21.00 sec 41.2 MBytes 346 Mbits/sec 271 1.40 MBytes
[ 5] 21.00-22.00 sec 53.8 MBytes 451 Mbits/sec 410 1.19 MBytes
[ 5] 22.00-23.00 sec 48.8 MBytes 409 Mbits/sec 401 1.22 MBytes
[ 5] 23.00-24.00 sec 47.5 MBytes 399 Mbits/sec 395 1.24 MBytes
[ 5] 24.00-25.00 sec 40.0 MBytes 336 Mbits/sec 492 954 KBytes
[ 5] 25.00-26.00 sec 43.8 MBytes 367 Mbits/sec 202 1018 KBytes
[ 5] 26.00-27.00 sec 38.8 MBytes 325 Mbits/sec 404 771 KBytes
[ 5] 27.00-28.00 sec 30.0 MBytes 252 Mbits/sec 346 700 KBytes
[ 5] 28.00-29.00 sec 28.8 MBytes 241 Mbits/sec 262 665 KBytes
[ 5] 29.00-30.00 sec 27.5 MBytes 231 Mbits/sec 169 636 KBytes
[ 5] 30.00-31.00 sec 25.0 MBytes 210 Mbits/sec 209 585 KBytes
[ 5] 31.00-32.00 sec 26.2 MBytes 220 Mbits/sec 224 636 KBytes
[ 5] 32.00-33.00 sec 26.2 MBytes 220 Mbits/sec 170 686 KBytes
[ 5] 33.00-34.00 sec 26.2 MBytes 220 Mbits/sec 149 600 KBytes
[ 5] 34.00-35.00 sec 25.0 MBytes 210 Mbits/sec 259 560 KBytes
[ 5] 35.00-36.00 sec 21.2 MBytes 178 Mbits/sec 161 560 KBytes
[ 5] 36.00-37.00 sec 22.5 MBytes 189 Mbits/sec 255 510 KBytes
[ 5] 37.00-38.00 sec 30.0 MBytes 252 Mbits/sec 164 764 KBytes
[ 5] 38.00-39.00 sec 33.8 MBytes 283 Mbits/sec 168 757 KBytes
[ 5] 39.00-40.00 sec 30.0 MBytes 252 Mbits/sec 199 731 KBytes
[ 5] 40.00-41.00 sec 30.0 MBytes 252 Mbits/sec 232 734 KBytes
[ 5] 41.00-42.00 sec 26.2 MBytes 220 Mbits/sec 305 731 KBytes
[ 5] 42.00-43.00 sec 28.8 MBytes 241 Mbits/sec 208 636 KBytes
[ 5] 43.00-44.00 sec 22.5 MBytes 189 Mbits/sec 101 509 KBytes
[ 5] 44.00-45.00 sec 21.2 MBytes 178 Mbits/sec 138 498 KBytes
[ 5] 45.00-46.00 sec 18.8 MBytes 157 Mbits/sec 192 445 KBytes
[ 5] 46.00-47.00 sec 20.0 MBytes 168 Mbits/sec 215 467 KBytes
[ 5] 47.00-48.00 sec 18.8 MBytes 157 Mbits/sec 167 467 KBytes
[ 5] 48.00-49.00 sec 18.8 MBytes 157 Mbits/sec 103 390 KBytes
[ 5] 49.00-50.00 sec 13.8 MBytes 115 Mbits/sec 133 318 KBytes
[ 5] 50.00-51.00 sec 12.5 MBytes 105 Mbits/sec 112 318 KBytes
[ 5] 51.00-52.00 sec 12.5 MBytes 105 Mbits/sec 112 376 KBytes
[ 5] 52.00-53.00 sec 13.8 MBytes 115 Mbits/sec 165 318 KBytes
[ 5] 53.00-54.00 sec 12.5 MBytes 105 Mbits/sec 75 311 KBytes
[ 5] 54.00-55.00 sec 12.5 MBytes 105 Mbits/sec 108 296 KBytes
[ 5] 55.00-56.00 sec 11.2 MBytes 94.4 Mbits/sec 136 225 KBytes
[ 5] 56.00-57.00 sec 10.0 MBytes 83.9 Mbits/sec 78 280 KBytes
[ 5] 57.00-58.00 sec 10.0 MBytes 83.9 Mbits/sec 85 296 KBytes
[ 5] 58.00-59.00 sec 12.5 MBytes 105 Mbits/sec 79 267 KBytes
[ 5] 59.00-60.00 sec 12.5 MBytes 105 Mbits/sec 79 269 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 2.35 GBytes 337 Mbits/sec 19095 sender
[ 5] 0.00-60.02 sec 2.31 GBytes 331 Mbits/sec receiver
iperf Done.
regards.
Nakajima.
2021年2月2日火曜日 11:12:43 UTC+9 Neal Cardwell:
Neal Cardwell
Feb 5, 2021, 10:19:52 AM2/5/21
to Akira Nakajima, BBR Development
On Fri, Feb 5, 2021 at 1:24 AM Akira Nakajima <girn...@gmail.com> wrote:
I'm not thinking about specific use cases, I want to know the specifications.
I just want to know if BBRv2 replaces BBRv1 or uses it depending on the line quiality.
>> https://datatracker.ietf.org/meeting/105/materials/slides-105-iccrg-bbr-v2-a-model-based-congestion-control-00#page=6
This is measured in 2 minutes. When measured in 30 minutes, a throughput of 125 Mbps can be obtained.
In my environment, real-time rate display using iperf3 shows that the initial velocity is close to 1 Gbps, but it gradually decreases and converges to about 100 Mbps.
# tc qdisc add dev ens3 root netem rate 1Gbit limit 8657 delay 20m loss 1%
Thanks for the details about your test scenario. It looks like you are installing netem on the sending machine. TCP performance results are not realistic when netem is installed on the sending machine, due to interactions between netem and mechanisms like TSQ (TCP small queues). To get realistic TCP performance results with netem, the netem qdisc has to be installed either on an intermediate "router" machine or on the ingress path of the receiving machine.
For examples on how to install netem on the ingress of a machine, see the ifb0 example in the "How can I use netem on incoming traffic?" section of this doc:
best,
neal
To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/cee2a609-492d-4dbb-909b-abef059d546en%40googlegroups.com.
Neal Cardwell
Feb 5, 2021, 10:33:38 AM2/5/21
to Akira Nakajima, BBR Development
On Fri, Feb 5, 2021 at 10:19 AM Neal Cardwell <ncar...@google.com> wrote:
On Fri, Feb 5, 2021 at 1:24 AM Akira Nakajima <girn...@gmail.com> wrote:I'm not thinking about specific use cases, I want to know the specifications.
I just want to know if BBRv2 replaces BBRv1 or uses it depending on the line quiality.
>> https://datatracker.ietf.org/meeting/105/materials/slides-105-iccrg-bbr-v2-a-model-based-congestion-control-00#page=6
This is measured in 2 minutes. When measured in 30 minutes, a throughput of 125 Mbps can be obtained.
In my environment, real-time rate display using iperf3 shows that the initial velocity is close to 1 Gbps, but it gradually decreases and converges to about 100 Mbps.
# tc qdisc add dev ens3 root netem rate 1Gbit limit 8657 delay 20m loss 1%Thanks for the details about your test scenario. It looks like you are installing netem on the sending machine. TCP performance results are not realistic when netem is installed on the sending machine, due to interactions between netem and mechanisms like TSQ (TCP small queues). To get realistic TCP performance results with netem, the netem qdisc has to be installed either on an intermediate "router" machine or on the ingress path of the receiving machine.For examples on how to install netem on the ingress of a machine, see the ifb0 example in the "How can I use netem on incoming traffic?" section of this doc:
I have added a section to the BBR FAQ for this:
"How can I test Linux TCP BBR with an emulated network?"BTW, another factor to consider is that when you emulate loss with netem, the netem qdisc makes drop decisions in terms of entire sk_buff TSO bursts (of up to 44 MTU-sized packets), rather than individual MTU-sized packets. This makes the loss process highly unrealistic relative to a drop process that drops 1% of MTU-size packets: the time in between drops can be up to 44x longer, and the drops are much burstier (e.g. dropping 44 MTU-sized packets in a single sk_buff).
neal
Akira Nakajima
Feb 12, 2021, 3:04:51 AM2/12/21
to BBR Development
Thanks for advise.
netem is setted on intermediater router.
Forget my measured speed of 1%. It has nothing to do with the intent of the question.
https://datatracker.ietf.org/meeting/105/materials/slides-105-iccrg-bbr-v2-a-model-based-congestion-control-00#page=6
As you have shown, v2 has dropped to the same speed as CUBIC with a packet loss rate of 3% or more.
I want to know why v2 abandoned this advantage of v1.
And does this mean that v1 and v2 should be used properly according to the package loss rate?
These two points.
Regards.
Nakajima.
netem is setted on intermediater router.
Forget my measured speed of 1%. It has nothing to do with the intent of the question.
https://datatracker.ietf.org/meeting/105/materials/slides-105-iccrg-bbr-v2-a-model-based-congestion-control-00#page=6
As you have shown, v2 has dropped to the same speed as CUBIC with a packet loss rate of 3% or more.
I want to know why v2 abandoned this advantage of v1.
And does this mean that v1 and v2 should be used properly according to the package loss rate?
These two points.
Regards.
Nakajima.
2021年2月6日土曜日 0:33:38 UTC+9 Neal Cardwell:
Matt Mathis
Feb 12, 2021, 5:48:46 PM2/12/21
to Akira Nakajima, BBR Development
If widely deployed, BBRv1's high loss tolerance would be a danger to the Internet, It is part of the reason that BBRv1 was so "unfair" to CUBIC, but the situation is actually much worse than that.
Since BBR can tolerate high loss, there are also guaranteed to be situations where it will cause high loss. The simplest example would be something like an under-buffered router (tiny drain time) with heavily aggregated traffic (100k flows) of widely varying RTTs (1-100mS). If you get the setup just right, this should be able to push the loss rate very high - perhaps even above 10%. Not only would this crush CUBIC, it also breaks DNS and just about anything else. You do not want this Internet.
There is an intrinsic tussle between the the worst case losses that any transit protocol can tolerate vs the worst case losses that it might inflict on other traffic. In highly aggregated traffic a transport stream can not distinguish between self induced losses caused by the aggregate behavior or losses that are the consequence of a noisy radio. One algorithm can't address both cases.
Thanks,
--MM--The best way to predict the future is to create it. - Alan Kay
We must not tolerate intolerance;
however our response must be carefully measured:
too strong would be hypocritical and risks spiraling out of control;
too weak risks being mistaken for tacit approval.
To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/626fbe2e-7827-460d-9d86-c807cd1848abn%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages