BBR Linux implementation to expose the estimated bw at bottleneck link?

[Morten Lindeberg]

Hi!

I am doing research on distributed stream processing systems, and would like for the application to "know" the available bandwidth on its TCP connections used for stream processing. 

Seems BBR tracks the available bandwidth as part of its operation. Is there by chance anyone that can help in understanding a way to have an TCP BBR implementation on Linux advertise this to e.g., an application somehow (e.g., periodically log it to file, advertise on some port, etc)?

Regards

Morten

[Neal Cardwell]

Hi,

If you are looking to use a command line tool to find the BBR estimated bandwidth, you can use the "ss" tool. For details, see this documentation:
  https://github.com/google/bbr/blob/master/Documentation/bbr-faq.md#how-can-i-monitor-linux-tcp-bbr-connections

For finding this information programmatically by modifying an application, you can use something like the following to extract the BBR bandwidth estimate and minimum RTT estimate for a given TCP socket file descriptor, fd:

#include <linux/inet_diag.h>
...
typedef unsigned long long u64;
...

  int fd;
  u64 bw;
 
  union tcp_cc_info info;
  socklen_t len = sizeof(info);

  if (getsockopt(fd, SOL_TCP, TCP_CC_INFO, &info, &len) < 0) {
    perror("getsockopt(TCP_CC_INFO)");
    exit(EXIT_FAILURE);
  }

  if (len >= sizeof(info.bbr)) {
    bw = ((u64)info.bbr.bbr_bw_hi << 32) | (u64)info.bbr.bbr_bw_lo;
    printf("bw: %lu bytes/sec\n", bw);
    printf("min_rtt: %u usec\n", info.bbr.bbr_min_rtt);
  }

I have added this example to the BBR FAQ.

best regards,

neal

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/9453a0b2-819d-4738-a21c-5035d23e3212n%40googlegroups.com.

[Bob McMahon]

Interesting. Is there value to adding BBR TCP_CC_INFO stats as iperf 2? It currently samples TCP_INFO for things like retries, RTT and CWND per the -e or --enhanced option on the client.

Bob

Bob

To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CADVnQy%3DiVEh6_9GiTDiUJpcxXqCTe1aHw_9TkLET%3DAGKhtuxYQ%40mail.gmail.com.

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

[Neal Cardwell]

Good question. That's up to you. I normally use "ss" to get this kind of detail for ongoing tests. And that works well enough for me.

thanks,

neal

[Bob McMahon]

Curious to what an app or a network forwarding plane/driver could/would do with this information? If there's really no actionable engineering by either set of engineers then I don't see a need to expose it. Plus ss is already there and per your comments that seems sufficient for stack/congestion control engineers.

Bob

[Bob McMahon]

Another question and idea. Is there some sort of signature from the aggregate of TCP_INFO class stats (including the CC ones) that fed and aggregated by a stats collector and engine that could detect network issues? And maybe then deploy countermeasures? 

Thanks,
Bob

[Bob McMahon]

Hi All,

A question for the transport experts. Let's say an intermediate forwarding plane device determined a packet requiring low latency was going to be late, would the best action be to set ECN bits and forward it so ECN could pace the source or rather immediately drop the packet?

Thanks,
Bob

[Dave Taht]

On Fri, Jun 10, 2022 at 11:41 AM 'Bob McMahon' via BBR Development
<bbr...@googlegroups.com> wrote:
>
> Hi All,
>
> A question for the transport experts. Let's say an intermediate forwarding plane device determined a packet requiring low latency was going to be late, would the best action be to set ECN bits and forward it so ECN could pace the source or rather immediately drop the packet?

A packet not marked as ECN capable (neither ECT1 and ECT0 set) MUST
NOT be marked with ECN, but dropped. In RFC3168 ECN a drop is
considered equivalent to a mark.

There isn't a distinct notion of "this packet needs low latency" baked
into RFC3168-style ECN. The concept was reserved for explicit diffserv
markings, or in my (FQ'd) world, "sparseness"). So I might argue with
a CS4 + ECN mark that it might still be better to drop rather than
mark.

L4S-style ECN is supposed to land in short queues by default, and the
folk over there never want to drop a packet ever.

> Thanks,
> Bob
>
>
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
>

> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvq%3DxZR_MY1PA_FTZUCO1PRrEqgAQc-dxW--irFo1nXnoA%40mail.gmail.com.



--
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

[Keith Winstein]

Hello Bob,

My 2c would be that there's probably no single right answer to this sort of question -- it depends on the application semantics, the network queueing and scheduling behavior, and the status quo of how endpoints will interpret and react to drops vs. marks.

In the abstract, dropping a packet can be preferable to marking if it means that other packets will arrive in a way that better satisfies application objectives.

If we're talking about "other packets" on the same flow, then this depends on the application semantics and the reason that the packet is going to be late. If the bottleneck link is fast but jittery (e.g. a Wi-Fi hop with tons of capacity but occasional 50-ms outages) and a packet is queued excessively because of that jitter, then there may be little benefit to dropping it. It won't make later packets arrive discernibly sooner. But if the reason is that the bottleneck link is capacity-limited (so dropping a packet will make later packets arrive sooner), and if the application would benefit from this on a net basis (e.g. it's transmitting a sequence of sensor readings where the receiver only cares about the most recent value, so every packet supersedes all previous), then yes, dropping a packet that contains old data is beneficial if it means later packets on the same flow will arrive sooner.

If we're talking about "other packets" on a different flow, then this depends on the scheduling policy at the bottleneck and the desires of contending flows. If there's a single FIFO queue, and dropping my late packet means that your packet will get delivered sooner, and if your application will benefit from that, then dropping is beneficial to the network as a whole. But if our flows are in different queues being serviced by some per-queue sharing policy, and if my queue will still have demand, then dropping one of my packets probably won't help you.

I'm sure there are more subtleties when you have to think about real-world issues -- networks and receivers (and senders) not behaving in an ideal manner, etc.

Sincerely,

Keith

--

You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvq%3DxZR_MY1PA_FTZUCO1PRrEqgAQc-dxW--irFo1nXnoA%40mail.gmail.com.

[Bob McMahon]

Thanks Keith for the detailed and considered response. I was mostly wondering from the transport layer (itself as a non-linear control system) if a drop or a mark had a "higher order."

From the predictive "modeling" per a set of live e2e paths, that seems unachievable to be order of jitter or latency (per previous discussions around queueing theory)

As we move towards latency-based performance (vs throughput) some holes seem to be that TTL is hop based vs time based and there is no way the forwarding plane can derive a time based set of rules for live e2e flows. Then, even if it could, what would a forwarding plane do to improve itself to provide a better set of e2e latency optimums? Also, wireless seems to need special considerations in much of this.

Bob 

[Morten Lindeberg]

Anyways, thanks a ton Neal! Great help! Just the kind of feature I was hoping existed!