BBR and TCP rmem and wmem buffer values...
1,155 views
Skip to first unread message
Raghavendran N
Sep 27, 2016, 9:25:28 AM9/27/16
to BBR Development
Hell o BBR forum,
I wanted to know the TCP rmem and wmem values (buffers) will be automatically taken care of by BBR? I mean, I don't need to worry about the buffer values anymore? I was trying to get around the bufferbloat issues in 3G and 4G networks by reducing the rmem and wmem values coupled with fq_codel.
Appreciate your help/thoughts.
-raghav
Neal Cardwell
Sep 27, 2016, 10:10:46 AM9/27/16
to Raghavendran N, BBR Development
When it comes to rmem and wmem, there are two scenarios that come to mind:
(1) For users that have been dialing send or receive buffers down to
try to avoid bufferbloat in deep buffers: with BBR, this should no
longer be needed. BBR ought to be able to bound the amount of data in
flight to a sensible range, without any need for buffer tuning. That
sounds like what you were asking about, and the answer is: "with BBR,
this should just work". If it doesn't, please let us know.
(2) For users that have high-speed WAN paths with moderate-to-high
loss rates (e.g. due to shallow-buffered switches): BBR is often able
to fully utilize these pipes that previous congestion control
algorithms could not, so with BBR you may now be limited by your
kernel's send or receive buffer settings. Because TCP has an in-order
delivery model, and the TCP SACK specification considers SACKs as
"hints" rather than official and final acknowledgments of retaining
data, senders and receivers may have to keep several BDP of data in
the send and receive buffers when there is packet loss on repeated
rounds. This is because the TCP layer must wait until retransmissions
fill in all the holes in the sequence space, so the data can be passed
up to the receiving application. So to fully take advantage of
bandwidth on higher-loss paths you may need to increase the maximum
values of the kernel send and receive buffer sizes. For testing, you
might try something like:
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_wmem='4096 16384 16777216'
sysctl -w net.core.rmem_max=25165824
sysctl -w net.ipv4.tcp_rmem='4096 87380 25165824'
neal
> --
> You received this message because you are subscribed to the Google Groups
> "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbr-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
(1) For users that have been dialing send or receive buffers down to
try to avoid bufferbloat in deep buffers: with BBR, this should no
longer be needed. BBR ought to be able to bound the amount of data in
flight to a sensible range, without any need for buffer tuning. That
sounds like what you were asking about, and the answer is: "with BBR,
this should just work". If it doesn't, please let us know.
(2) For users that have high-speed WAN paths with moderate-to-high
loss rates (e.g. due to shallow-buffered switches): BBR is often able
to fully utilize these pipes that previous congestion control
algorithms could not, so with BBR you may now be limited by your
kernel's send or receive buffer settings. Because TCP has an in-order
delivery model, and the TCP SACK specification considers SACKs as
"hints" rather than official and final acknowledgments of retaining
data, senders and receivers may have to keep several BDP of data in
the send and receive buffers when there is packet loss on repeated
rounds. This is because the TCP layer must wait until retransmissions
fill in all the holes in the sequence space, so the data can be passed
up to the receiving application. So to fully take advantage of
bandwidth on higher-loss paths you may need to increase the maximum
values of the kernel send and receive buffer sizes. For testing, you
might try something like:
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_wmem='4096 16384 16777216'
sysctl -w net.core.rmem_max=25165824
sysctl -w net.ipv4.tcp_rmem='4096 87380 25165824'
neal
> You received this message because you are subscribed to the Google Groups
> "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbr-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Raghavendran N
Sep 27, 2016, 10:25:42 AM9/27/16
to Neal Cardwell, BBR Development
Hi Neal,
On Tue, Sep 27, 2016 at 7:40 PM, Neal Cardwell <ncar...@google.com> wrote:
> When it comes to rmem and wmem, there are two scenarios that come to mind:
>
> (1) For users that have been dialing send or receive buffers down to
> try to avoid bufferbloat in deep buffers: with BBR, this should no
> longer be needed. BBR ought to be able to bound the amount of data in
> flight to a sensible range, without any need for buffer tuning. That
> sounds like what you were asking about, and the answer is: "with BBR,
> this should just work". If it doesn't, please let us know.
>
This sounds awesome! Is it safe to assume that 4G falls in this
scenario (at least the download speeds are comparable with WiFi. I
have seen upwards of 25Mbps).
I am in the process of testing the BBR patch, but before that, I need
to upgrade to the latest version on my distribution and also need to
move from 4.1 to 4.9 :-)
I am really excited to give BBR a try, thank you!
> (2) For users that have high-speed WAN paths with moderate-to-high
> loss rates (e.g. due to shallow-buffered switches): BBR is often able
> to fully utilize these pipes that previous congestion control
> algorithms could not, so with BBR you may now be limited by your
> kernel's send or receive buffer settings. Because TCP has an in-order
> delivery model, and the TCP SACK specification considers SACKs as
> "hints" rather than official and final acknowledgments of retaining
> data, senders and receivers may have to keep several BDP of data in
> the send and receive buffers when there is packet loss on repeated
> rounds. This is because the TCP layer must wait until retransmissions
> fill in all the holes in the sequence space, so the data can be passed
> up to the receiving application. So to fully take advantage of
> bandwidth on higher-loss paths you may need to increase the maximum
> values of the kernel send and receive buffer sizes. For testing, you
> might try something like:
>
> sysctl -w net.core.wmem_max=16777216
> sysctl -w net.ipv4.tcp_wmem='4096 16384 16777216'
> sysctl -w net.core.rmem_max=25165824
> sysctl -w net.ipv4.tcp_rmem='4096 87380 25165824'
>
I might end up supporting both 3G/4G links with WAN links. In that
case, setting buffer sizes to the values you suggested should be ok
because they will be applied to the WAN links. Did I get that right?
On Tue, Sep 27, 2016 at 7:40 PM, Neal Cardwell <ncar...@google.com> wrote:
> When it comes to rmem and wmem, there are two scenarios that come to mind:
>
> (1) For users that have been dialing send or receive buffers down to
> try to avoid bufferbloat in deep buffers: with BBR, this should no
> longer be needed. BBR ought to be able to bound the amount of data in
> flight to a sensible range, without any need for buffer tuning. That
> sounds like what you were asking about, and the answer is: "with BBR,
> this should just work". If it doesn't, please let us know.
>
scenario (at least the download speeds are comparable with WiFi. I
have seen upwards of 25Mbps).
I am in the process of testing the BBR patch, but before that, I need
to upgrade to the latest version on my distribution and also need to
move from 4.1 to 4.9 :-)
I am really excited to give BBR a try, thank you!
> (2) For users that have high-speed WAN paths with moderate-to-high
> loss rates (e.g. due to shallow-buffered switches): BBR is often able
> to fully utilize these pipes that previous congestion control
> algorithms could not, so with BBR you may now be limited by your
> kernel's send or receive buffer settings. Because TCP has an in-order
> delivery model, and the TCP SACK specification considers SACKs as
> "hints" rather than official and final acknowledgments of retaining
> data, senders and receivers may have to keep several BDP of data in
> the send and receive buffers when there is packet loss on repeated
> rounds. This is because the TCP layer must wait until retransmissions
> fill in all the holes in the sequence space, so the data can be passed
> up to the receiving application. So to fully take advantage of
> bandwidth on higher-loss paths you may need to increase the maximum
> values of the kernel send and receive buffer sizes. For testing, you
> might try something like:
>
> sysctl -w net.core.wmem_max=16777216
> sysctl -w net.ipv4.tcp_wmem='4096 16384 16777216'
> sysctl -w net.core.rmem_max=25165824
> sysctl -w net.ipv4.tcp_rmem='4096 87380 25165824'
>
case, setting buffer sizes to the values you suggested should be ok
because they will be applied to the WAN links. Did I get that right?
Neal Cardwell
Sep 27, 2016, 10:32:20 AM9/27/16
to Raghavendran N, BBR Development
On Tue, Sep 27, 2016 at 10:25 AM, Raghavendran N <rag...@watchy.in> wrote:
> Hi Neal,
>
> On Tue, Sep 27, 2016 at 7:40 PM, Neal Cardwell <ncar...@google.com> wrote:
>> When it comes to rmem and wmem, there are two scenarios that come to mind:
>>
>> (1) For users that have been dialing send or receive buffers down to
>> try to avoid bufferbloat in deep buffers: with BBR, this should no
>> longer be needed. BBR ought to be able to bound the amount of data in
>> flight to a sensible range, without any need for buffer tuning. That
>> sounds like what you were asking about, and the answer is: "with BBR,
>> this should just work". If it doesn't, please let us know.
>>
> This sounds awesome! Is it safe to assume that 4G falls in this
> scenario (at least the download speeds are comparable with WiFi. I
> have seen upwards of 25Mbps).
Yes, 4G links would fall into this category, as would most last-mile
> Hi Neal,
>
> On Tue, Sep 27, 2016 at 7:40 PM, Neal Cardwell <ncar...@google.com> wrote:
>> When it comes to rmem and wmem, there are two scenarios that come to mind:
>>
>> (1) For users that have been dialing send or receive buffers down to
>> try to avoid bufferbloat in deep buffers: with BBR, this should no
>> longer be needed. BBR ought to be able to bound the amount of data in
>> flight to a sensible range, without any need for buffer tuning. That
>> sounds like what you were asking about, and the answer is: "with BBR,
>> this should just work". If it doesn't, please let us know.
>>
> This sounds awesome! Is it safe to assume that 4G falls in this
> scenario (at least the download speeds are comparable with WiFi. I
> have seen upwards of 25Mbps).
links (cellular, wifi, cable, DSL).
maximum buffer size settings even if some paths have lower BDPs. BBR
should only use a bigger buffer if it's needed.
Eric Dumazet
Sep 27, 2016, 10:43:30 AM9/27/16
to BBR Development, rag...@watchy.in
Although BBR will precisely control number of in-flight packets, making tcp_wmem[2] bigger can have an impact
on overall memory used by TCP sockets on a host.
TCP sendmsg() will happily accept MBytes of data pumped from application to kernel space.
So if a host has very tight memory requirements, you might consider using
sysctl net.ipv4.tcp_notsent_lowat=1000000
This will instruct TCP sendmsg() not filling the whole socket write queue.
Raghavendran N
Sep 27, 2016, 11:24:29 AM9/27/16
to Neal Cardwell, BBR Development
On Tue, Sep 27, 2016 at 8:01 PM, Neal Cardwell <ncar...@google.com> wrote:
> On Tue, Sep 27, 2016 at 10:25 AM, Raghavendran N <rag...@watchy.in> wrote:
>> Hi Neal,
>>
>> On Tue, Sep 27, 2016 at 7:40 PM, Neal Cardwell <ncar...@google.com> wrote:
>>> When it comes to rmem and wmem, there are two scenarios that come to mind:
>>>
>>> (1) For users that have been dialing send or receive buffers down to
>>> try to avoid bufferbloat in deep buffers: with BBR, this should no
>>> longer be needed. BBR ought to be able to bound the amount of data in
>>> flight to a sensible range, without any need for buffer tuning. That
>>> sounds like what you were asking about, and the answer is: "with BBR,
>>> this should just work". If it doesn't, please let us know.
>>>
>> This sounds awesome! Is it safe to assume that 4G falls in this
>> scenario (at least the download speeds are comparable with WiFi. I
>> have seen upwards of 25Mbps).
>
> Yes, 4G links would fall into this category, as would most last-mile
> links (cellular, wifi, cable, DSL).
Thank you!
> On Tue, Sep 27, 2016 at 10:25 AM, Raghavendran N <rag...@watchy.in> wrote:
>> Hi Neal,
>>
>> On Tue, Sep 27, 2016 at 7:40 PM, Neal Cardwell <ncar...@google.com> wrote:
>>> When it comes to rmem and wmem, there are two scenarios that come to mind:
>>>
>>> (1) For users that have been dialing send or receive buffers down to
>>> try to avoid bufferbloat in deep buffers: with BBR, this should no
>>> longer be needed. BBR ought to be able to bound the amount of data in
>>> flight to a sensible range, without any need for buffer tuning. That
>>> sounds like what you were asking about, and the answer is: "with BBR,
>>> this should just work". If it doesn't, please let us know.
>>>
>> This sounds awesome! Is it safe to assume that 4G falls in this
>> scenario (at least the download speeds are comparable with WiFi. I
>> have seen upwards of 25Mbps).
>
> Yes, 4G links would fall into this category, as would most last-mile
> links (cellular, wifi, cable, DSL).
forward to the ACM queue article and hoping it will be available
outside of ACM too.
Raghavendran N
Sep 27, 2016, 11:26:47 AM9/27/16
to Eric Dumazet, BBR Development
On Tue, Sep 27, 2016 at 8:13 PM, Eric Dumazet <edum...@google.com> wrote:
> Although BBR will precisely control number of in-flight packets, making
> tcp_wmem[2] bigger can have an impact
> on overall memory used by TCP sockets on a host.
Yup, thank you for that Eric. The server needs to be able to handle
> Although BBR will precisely control number of in-flight packets, making
> tcp_wmem[2] bigger can have an impact
> on overall memory used by TCP sockets on a host.
lots of incoming connections so, I guess we won't be able to set
really high values to the tcp_[rw]mem buffers.
>
> TCP sendmsg() will happily accept MBytes of data pumped from application to
> kernel space.
>
> So if a host has very tight memory requirements, you might consider using
>
> sysctl net.ipv4.tcp_notsent_lowat=1000000
>
> This will instruct TCP sendmsg() not filling the whole socket write queue.
>
> https://lwn.net/Articles/560082/
Thank you for the link. Appreciate it.
> This will instruct TCP sendmsg() not filling the whole socket write queue.
>
> https://lwn.net/Articles/560082/
Reply all
Reply to author
Forward
0 new messages