CNI bandwidth plugin: configuratable latency(queue depth)?

[Xu Kangjie]

Hi,

I propose to have a latency parameter for the bandwidth plugin.

The bandwidth plugin rely on the linux tc token bucket filter to work, and 

currently, the latency parameter for the token bucket filter is fixed, which is

25 ms, eaily leads to head-of-line blocking and bufferbloat. Since there can

be different types of traffic existing in the same pod, some latency-sensitive

connection will suffer.

I have a simple test in local kind cluster, one TCP_RR conn for measuring latency and one TCP_STREAM for measuring throughput. two connections are in the same pod(rate limit: 100M), the results are listed below, 

- Data flow direction - Throughput(Mbps) for TCP_STREAM flow - Latency(us) for TCP_RR flow
- netperf-client -> netperf-server (egress rate limit) - 95.6 - 10760.05
- netperf-server -> netperf-client (ingress rate limit) - 95.6 - 226120.05

It's a local kind env, the ping latency for client and server is about 50us. The high is brought by the bandwidth plugin.

To reduce the high latency, we can make the latency param smaller in configurations. 

I also send a PR for reference: https://github.com/containernetworking/plugins/pulls

Thanks,

Kangjie