ESXi VMs cannot ingest more than 3.5Gbps unless Ring Buffer on DUT is set low

[Mark Wittling]

As we do more testing, we are noticing that not a single DUT VM, be it a Cent7 VM, or a Juniper vSRX VM, can receive more than 3.5Gbps. Most of the time, the VM "caps out" at receiving 3.2Gbps.  On the Cent7 DUT, we are running iftop to get our receive rate statistics.

The config we are using on Trex is an ip-based config (src_ip and default_gw). We are hitting underlay dPG ports. We are running both the Trex VM, and the DUT, on the same ESXi hypervisor host. The Trex VM has a CPU reservation of 12,200 Mhz.

When we started looking in VMWare, with the command:
vsish -e cat /net/portsets/DvsPortset-X/ports/yyyyyyy/vmxnet3/rxSummary

We see interesting things about Ring Buffers:
pkts rx ok:75409132
   bytes rx ok:26955684070
   unicast pkts rx ok:75409131
   unicast bytes rx ok:26955684010
   multicast pkts rx ok:0
   multicast bytes rx ok:0
   broadcast pkts rx ok:1
   broadcast bytes rx ok:60
   running out of buffers:25183929
   pkts receive error:0
   1st ring size:1024
   2nd ring size:256
   # of times the 1st ring is full:25183929
   # of times the 2nd ring is full:0
   fail to map a rx buffer:0
   request to page in a buffer:0
   # of times rx queue is stopped:0
   failed when copying into the guest buffer:0
   # of pkts dropped due to large hdrs:0
   # of pkts dropped due to max number of SG limits:0
   pkts rx via data ring ok:0
   bytes rx via data ring ok:0
   Whether rx burst queuing is enabled:0
   current backend burst queue length:0
   maximum backend burst queue length so far:0
   aggregate number of times packets are requeued:0
   aggregate number of times packets are dropped by PktAgingList:0
   # of pkts dropped due to large inner (encap) hdrs:0
   number of times packets are dropped by burst queue:0
   number of packets delivered by burst queue:0
   number of packets dropped by packet steering:0
   number of packets dropped due to pkt length exceeds vNic mtu:0

When we changed the Ring Buffer in the Cent7 DUT (ethtool -G eth0 512/1024/4096), we noticed something VERY strange. The lower ring buffer size increased the ability of the DUT to ingest a higher packet rate (3.5Gbps up to about 5Gbps).

And we also noticed that when we increased the Ring Buffer size on the DUT Cent7 VM, the Trex VM transmit packet rate dropped!