comparison of docker overlay network performance

[sgas]

I  have collected a  summary  of my findings so far  on docker overlay network  performance .  The table follows.   As is obvious there is significant performance penalty form Linux vxlan and bridge. 

End points	Mtu	NIC vxlan acceleration	Rate (Gbps)	Notes
Host to host	9000	N/A	9.63
Host-vxlan to host-vxlan	8950	no	7.01	Linux vxlan overhead (?)
Docker overlay	8950	no	3.96	Linux bridge overhead (?)
Docker overlay	1450	no	1.11	Small mtu penalty

Improving  performance  in both Linux vxlan and bridge, in addition to offload to NIC should also improve results. 

[Jana Radhakrishnan]

Good info. These performance number can be understood better if the details about test hardware and the test setup is provided. But yes if you are using iperf kind of tools then the single core performance of forwarding through a bridge maxes out at 700-750Kpps(The size of the packet doesn't matter much for bridge forwarding) because it needs to perform forwarding lookups which is always the long pole in any kind of packet performance tests. Again the bridge forwarding limit is determined by how fast the core is and that is the limit and a single core would never be a able to saturate a 10G link even in fastest cores available today when forwarding lookups are involved.

But single core performance is not the end of all in terms of the how this data needs to interpreted.. If you run a multi-core saturation tests with all cores being put to use to send Docker overlay traffic you can in fact saturate your 10G link. So as long as there are enough cores available and there are enough containers that can make use of these cores and as long as we can saturate that 10G link we are good. So the only time this would be a problem is on a truly single core(or less number of cores) hardware or when there is a single application which doesn't effectively utilize all the cores but requires all the bandwidth that the host network offers. 

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sgas]

Jana, good point.   The test setup info follows.

2 hosts  each with 1 cpu Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz with 32 cores,  10 Gbps ixgbe  NICs. 

I ran a comparison of Linux Bridge and  OVS in the following setup . 

iperf----veth1---veth2-- [bridge]--vxlan--phy----(LAN)----phy--vxlan--[bridge]---veth2---veth1--iperf

The performance is as follows:

OVS :               6.27 Gbits/sec

Linux Bridge    4.12 Gbits/sec

Linux bridge is only  about 2/3 rd as performant as  OVS !!

[sgas]

So , here is the updated  performance comparison table.  

End points

Mtu

NIC vxlan acceleration

Iperf Rate (Gbps)

Notes

Host to host	9000	N/A	9.63
Host-vxlan to host-vxlan	8950	no	7.01	Linux vxlan overhead (?)

Veth-ovs-vxlan to vxlan-ovs-veth	8950	no	6.27	Ovs overhead
Veth-lbr-vxlan to vxlan-lbr-veth	8950	no	4.12	Linux bridge overhead (?)

Docker overlay	8950	no	3.96	Linux bridge overhead (?)
Docker overlay	1450	no	1.11	Small mtu penalty