problem KVM + bond + bridge performance

[tschend]

Hi,

i have a problem with the performance of guest network speed when
using bond + bridge.

we are testing debian squeeze + kernel 3.2 form backports + kvm 1.2 +
ganeti 2.6.1. Works great so far.

The setup of the bridges looks like usual

eth4/eth5 <--> bond0 <---> gnt-br0

Bond settings is active/passiv. Switch is not configured in any way.

The tap of the test vm has the same mac as the bridge

gnt-br0 Link encap:Ethernet HWaddr 26:4b:49:53:a7:c8
tap0 Link encap:Ethernet HWaddr 26:4b:49:53:a7:c8

the active nic is marked as local in the bridge

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth5
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth4
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 68:05:ca:04:21:c6
Slave queue ID: 0

Slave Interface: eth5
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 68:05:ca:04:21:c7
Slave queue ID: 0

brctl showmacs gnt-br0

port no mac addr is local? ageing timer
1 00:50:56:9c:0c:10 no 0.00
2 26:4b:49:53:a7:c8 yes 0.00
1 3e:b6:00:6a:d3:da no 64.96
1 68:05:ca:04:21:c6 yes 0.00
1 68:05:ca:04:22:38 no 0.19
1 68:05:ca:04:22:86 no 0.50
1 aa:00:00:1f:5c:e0 no 58.93
1 aa:00:00:3d:8f:be no 65.28
2 aa:00:00:4b:f4:6c no 119.13
1 aa:00:00:4e:ff:92 no 65.09
1 aa:00:00:75:c8:fa no 288.39
1 aa:00:00:8d:6b:4b no 64.86
1 aa:00:00:a3:81:14 no 64.97
1 aa:00:00:b6:e1:8e no 64.54
1 aa:00:00:f5:da:bc no 57.38

When i run netio in the vm on the local node it looks pretty good

TCP connection established.
Packet size 1k bytes: 703.86 MByte/s Tx, 1.024 GByte/s Rx.
Packet size 2k bytes: 1.396 GByte/s Tx, 1.362 GByte/s Rx.
Packet size 4k bytes: 1.503 GByte/s Tx, 1.362 GByte/s Rx.
Packet size 8k bytes: 1.639 GByte/s Tx, 1.318 GByte/s Rx.
Packet size 16k bytes: 1.734 GByte/s Tx, 1.263 GByte/s Rx.
Packet size 32k bytes: 1.805 GByte/s Tx, 1.277 GByte/s Rx.
Done.

When i do netio to another ganeti node in the testcluster or vm on a
different node (the other node has no bonding!)

TCP connection established.
Packet size 1k bytes: 277.63 MByte/s Tx, 6656.17 KByte/s Rx.
Packet size 2k bytes: 309.53 MByte/s Tx, 1159.69 KByte/s Rx.
Packet size 4k bytes: 323.91 MByte/s Tx, 2002.59 KByte/s Rx.
Packet size 8k bytes: 321.66 MByte/s Tx, 2265.05 KByte/s Rx.
Packet size 16k bytes: 336.52 MByte/s Tx, 4963.96 KByte/s Rx.
Packet size 32k bytes: 305.09 MByte/s Tx, 2134.77 KByte/s Rx.
Done.

Looks like the traffic going in is being dropped or something.
I found something on the kvm mailinglist but this is pretty old and
already fixed.
Any ideas on this?

Regards
Thomas

[tschend]

Hi,

just for the records, i fixed the problem.

The intel x520-T2 10GBe dual port card was the problem.

The ixgbe driver version of kernel 3.2.0 (3.6.7-k) caused the error.

I installed 3.11.33 and turned off large receive offload (ethtool -K
ethX lro off) and it works like a chrarm now.

CHeers
Thomas

[Guido Trotter]

Great news! :)

Sorry we didn't help, but it was very hardware specific for us to know!

Guido

--
Guido Trotter
SRE - Corp Computing Services (aka Horsepower)
Google Germany