Bond interfaces flapping

[Mimmus]

Hi,

on some bare metal servers, we are observing flapping behaviour on bond interfaces:server starts correctly but it starts to flapping up/down indefinitely after a few seconds.

This is our network config:

/etc/modprobe.d/bonding.conf
# Prevent kernel from automatically creating bond0 when the module is loaded.
# This allows systemd-networkd to create and apply options to bond0.
options bonding max_bonds=0

/etc/systemd/network/10-eth.network
[Match]
Name=eno*
[Network]
Bond=bond0

/etc/systemd/network/15-eth.network
[Match]
Name=ens*
[Network]
Bond=bond1

/etc/systemd/network/20-bond.netdev
[NetDev]
Name=bond0
Kind=bond
[Bond]
Mode=active-backup
MIIMonitorSec=1s

/etc/systemd/network/25-bond.netdev
[NetDev]
Name=bond1
Kind=bond
[Bond]
Mode=active-backup
MIIMonitorSec=1s

/etc/systemd/network/30-bond0.network
[Match]
Name=bond0
[Network]
Address=10.11.0.12/23
Gateway=10.11.1.254
DNS=10.9.1.11
DNS=10.192.72.77
Domains=mydomain.com
LinkLocalAddressing=no
IPv6AcceptRA=no

/etc/systemd/network/35-bond1.network
[Match]
Name=bond1
[Network]
Address=192.168.99.190
LinkLocalAddressing=no
IPv6AcceptRA=no

The strange thing is that sometime bonding stabilizes and works; also other servers with same config works.

CoreOS is 1800.7.0

Any suggestion?

Thanks in advance

--

[Benjamin Gilbert]

On Wed, Oct 24, 2018 at 8:52 AM Mimmus <domenico...@gmail.com> wrote:

on some bare metal servers, we are observing flapping behaviour on bond interfaces:server starts correctly but it starts to flapping up/down indefinitely after a few seconds.

Could you post the relevant log entries from dmesg?

--Benjamin Gilbert

[Mimmus]

The issue is sneaky: I wasn't able to reproduce on the previous server after it stabilized but on another is actually fully reproducible.

dmesg output:

[ 1199.540919] bond0: link status definitely up for interface eno50, 10000 Mbps full duplex
[ 1199.540922] bond0: making interface eno50 the new active one
[ 1199.543363] bond0: first active interface up!
[ 1200.564864] bond0: link status definitely up for interface eno49, 10000 Mbps full duplex
[ 1202.603106] ixgbe 0000:03:00.1: removed PHC on eno50
[ 1202.718785] bond0: link status definitely down for interface eno50, disabling it
[ 1202.740897] bond0: making interface eno49 the new active one
[ 1202.770080] ixgbe 0000:03:00.0: removed PHC on eno49
[ 1203.422721] bond1: link status definitely down for interface ens3f1, disabling it
[ 1203.445599] bond1: link status definitely down for interface ens3f0, disabling it
[ 1203.467563] bond1: now running without any active interface!
[ 1203.756603] ixgbe 0000:03:00.1: registered PHC device on eno50
[ 1203.900797] bond0: link status definitely down for interface eno49, disabling it
[ 1203.922685] bond0: now running without any active interface!
[ 1203.947335] ixgbe 0000:03:00.1 eno50: detected SFP+: 5
[ 1204.231861] ixgbe 0000:03:00.0: registered PHC device on eno49
[ 1204.418823] ixgbe 0000:03:00.0 eno49: detected SFP+: 6
[ 1204.574185] netxen_nic: ens3f1 NIC Link is up
[ 1204.948977] netxen_nic: ens3f0 NIC Link is up
[ 1205.492671] bond1: link status definitely up for interface ens3f1, 10000 Mbps full duplex
[ 1205.516306] bond1: making interface ens3f1 the new active one
[ 1205.536738] bond1: first active interface up!
[ 1205.556318] bond1: link status definitely up for interface ens3f0, 10000 Mbps full duplex
[ 1207.550137] ixgbe 0000:03:00.1 eno50: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1208.014192] ixgbe 0000:03:00.0 eno49: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1208.052596] bond0: link status definitely up for interface eno50, 10000 Mbps full duplex
[ 1208.075343] bond0: making interface eno50 the new active one
[ 1208.096626] bond0: first active interface up!
[ 1208.115676] bond0: link status definitely up for interface eno49, 10000 Mbps full duplex

I can send the whole dmesg, if can be useful.

Thanks you very much

P.S. I was able to stop continous flapping, manually removing an interface from bond