ganeti QA: TestInstanceDeviceNames fails on DRBD/KVM cluster

[Rudolph Bott]

Hey Everyone,

I am trying to run the testsuite against a three-node DRBD/KVM Debian Buster cluster setup and currently the TestInstanceDeviceNames test fails with the following error:

Command: ssh gnt-test01 'gnt-instance modify --disk=test_device:remove kvm-test-instance01'
Stderr was:
Failure: prerequisites not met for this operation:
error type: wrong_state, error details:
Instance 'kvm-test-instance01' is marked to be up, can't remove volume from a running instance without using hotplug
---- FAILED <function TestInstanceDeviceNames at 0x7f789cb8b2a8>: Command 'gnt-instance modify --disk=test_device:remove kvm-test-instance01' on node gnt-test01 failed, exit code 1
>>>> 2019-07-17 08:52:13.428087 time=0:00:02.451124 <function TestInstanceDeviceNames at 0x7f789cb8b2a8>
---- BLOCK FAILED [RunInstanceTests] Create and exercise instances: Command 'gnt-instance modify --disk=test_device:remove kvm-test-instance01' on node gnt-test01 failed, exit code 1
^^^^ BLOCK 2019-07-17 08:52:13.428177 time=0:05:30.631610 [RunInstanceTests] Create and exercise instances

Adding of the same disk succeeds (that happens right before the step above):

<<<< 2019-07-17 08:52:10.976963 start <function TestInstanceDeviceNames at 0x7f789cb8b2a8>
Command: ssh gnt-test01 'gnt-instance modify --disk=-1:add,name=test_device,size=512M kvm-test-instance01'
Wed Jul 17 08:52:11 2019 * disk 2, size 512M
Wed Jul 17 08:52:12 2019  - INFO: Waiting for instance kvm-test-instance01 to sync disks
Wed Jul 17 08:52:12 2019  - INFO: Instance kvm-test-instance01's disks are in sync
Modified instance kvm-test-instance01
 - disk/2 -> add:size=512,mode=rw
Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.

Just for the fun of it I modified the code in qa_instance.py to use the --hotplug parameter. Now the command seems to fail but in a way where it does not affect the test evaluation:

Command: ssh gnt-test01 'gnt-instance modify --disk=test_device:remove --hotplug kvm-test-instance01'
Wed Jul 17 07:53:39 2019  - INFO: Trying to hotplug device...
Wed Jul 17 07:53:45 2019  - WARNING: Could not hotplug device: Error while executing backend function: Device disk-a47d61ab-6fb3-4178 should have been removed but is still there
Wed Jul 17 07:53:45 2019  - INFO: Continuing execution..
Modified instance kvm-test-instance01
 - disk/2 -> remove
 - disk/2 -> hotplug:failed
Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.

Attached to this mail you'll find the json config for the QA configuration and also the current cluster configuration. Packages/Versions used are 2.16.0 from Debian Buster. Only two things were changed:

* I manually added the file colors.py to /usr/share/ganeti/testsuite/qa from the sources as its missing in the ganeti-testsuite package (PR has been opened for that)

* I changed the sources to disable strict ssh host key checking (yeah, blame me later for that. im currently focused on the tests ;-) )

Maybe someone with some QA experience can help me out with this one :-)

On a side node - the cluster is itself virtualised using nested KVM.

Cheers,

Rudi