Servers reboot unexpectedly on command execution

[Vladimir Palii]

Hello,

I'm experiencing reboots on 3 nodes when I issue the following command, "gnt-instance recreate-disk -n new_secondary_node instance_name”. The Server I was working on had 8 Instances and this command would successfully execute 7 times but fail on the 8th execution. This happened twice in 2 weeks

Here is how this fails:

The virtual machine was running on the primary server (Node02) and the secondary (Node04). During the re-creation of disks on another secondary server (Node03), three servers Node02, 03, and Node01 was rebooted. Strange thing that Node01 had nothing to do with rebuilding the disks. On Node01 and Node02 last logs are same and related to the drbd, and nothing on Node03.

Seems to be related to drbd. Any hints why it could happen?

Jul 26 19:30:02 node02 kernel: [619365.524477] block drbd8: asender terminated

Jul 26 19:30:02 node02 kernel: [619365.524484] block drbd8: Terminating drbd8_asender

Jul 26 19:30:02 node02 kernel: [619365.524596] block drbd8: Connection closed

Jul 26 19:30:02 node02 kernel: [619365.524608] block drbd8: conn( NetworkFailure -> Unconnected ) 

Jul 26 19:30:02 node02 kernel: [619365.524619] block drbd8: receiver terminated

Jul 26 19:30:02 node02 kernel: [619365.524622] block drbd8: Restarting drbd8_receiver

Jul 26 19:30:02 node02 kernel: [619365.524626] block drbd8: receiver (re)started

Jul 26 19:30:02 node02 kernel: [619365.524632] block drbd8: conn( Unconnected -> WFConnection ) 

Software:

Number of nodes in the cluster: 6

Distro: Debian  Wheezy   7.6

Hypervisor: xen-hypervisor-4.1-amd64:  4.1.6.lts1-8

ganeti:  2.10.7.1+grnet2+bpo2-1~wheezy all

drbd;   drbd8-util2:8.3.13-2

--

Vladimir.