kvm_extra options not passed to qemu-kvm correctly?

[George Diamantopoulos]

Hello,

I've been trying out the new "kvm_extra" feature in ganeti-2.7.0-rc2 which allows passing arbitrary arguments to qemu-kvm, in order to use a virtio-scsi storage interface. Although the following options do not return an error when starting qemu-kvm from the command line, starting an instance where the kvm_extra option has been configured with the same values produces an error. Unfortunately, neither virtio-scsi nor kvm_extra are sufficiently documented online, so I might be doing something wrong here:

This is how I'm configuring kvm_extra. I've put this together by googling around for examples, I suppose it should do the following: 1) create a virtual SCSI initiator in the VM (-device virtio-scsi-pci), 2) Configure the two drives for use within the instance, and assign an ID to them so that we have a way of making references to them (-drive file=/dev/sdX,id=sdX...), 3) Attach these drives to the virtual SCSI initiator scsihw0 (-device scsi-hd,drive=sdX...). I've had to escape commas, which I discovered by trial and error:

gnt-instance modify -H kvm_extra="-device virtio-scsi-pci\,id=scsihw0 -drive file=/dev/sdb\,if=none\,id=sdb\,cache=writeback\,aio=native -device scsi-hd\,drive=sdb\,id=scsi1\,bus=scsihw0.0\,channel=0\,scsi-id=0\,lun=0\,bootindex=100 -drive file=/dev/sdc\,if=none\,id=sdc\,cache=writeback\,aio=native -device scsi-hd\,drive=sdc\,id=scsi2\,bus=scsihw0.0\,channel=0\,scsi-id=1\,lun=0\,bootindex=101" nas.mydomain.tld

gnt-instance succeeds in setting this:

Modified instance nas.mydomain.tld
 - hv/kvm_extra -> -device virtio-scsi-pci,id=scsihw0 -drive file=/dev/sdb,if=none,id=sdb,cache=writeback,aio=native -device scsi-hd,drive=sdb,id=scsi1,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,bootindex=100 -drive file=/dev/sdc,if=none,id=sdc,cache=writeback,aio=native -device scsi-hd,drive=sdc,id=scsi2,bus=scsihw0.0,channel=0,scsi-id=1,lun=0,bootindex=101
Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.

Starting the instance produces an error:

death ~ # gnt-instance start nas.mydomain.tld
Waiting for job 93062 for nas.mydomain.tld ...
Job 93062 for nas.mydomain.tld has failed: Failure: command execution error:
Could not start instance: Hypervisor error: Failed to start instance nas.mydomain.tld: exited with exit code 1 (qemu-system-x86_64: -device virtio-scsi-pci,id=scsihw0 -drive file=/dev/sdb,if=none,id=sdb,cache=writeback,aio=native -device scsi-hd,drive=sdb,id=scsi1,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,bootindex=100 -drive file=/dev/sdc,if=none,id=sdc,cache=writeback,aio=native -device scsi-hd,drive=sdc,id=scsi2,bus=scsihw0.0,channel=0,scsi-id=1,lun=0,bootindex=101: invalid option
)

Passing these options to qemu-kvm results in no complaining whatsoever, though. I can't really tell whether it's working as expected, but at least I can tell that kvm_extra is not passed properly to qemu-kvm.

Any clues? Thanks

[Guido Trotter]

I believe that's because we pass all as a single parameter, which of
course is wrong.
Can you try with this patch and let us know if it works better?

diff --git a/lib/hypervisor/hv_kvm.py b/lib/hypervisor/hv_kvm.py
index 37058e7..b2d158d 100644
--- a/lib/hypervisor/hv_kvm.py
+++ b/lib/hypervisor/hv_kvm.py
@@ -1369,7 +1369,7 @@ class KVMHypervisor(hv_base.BaseHypervisor):
kvm_cmd.extend(["-usbdevice", dev])

if hvp[constants.HV_KVM_EXTRA]:
- kvm_cmd.extend([hvp[constants.HV_KVM_EXTRA]])
+ kvm_cmd.extend([hvp[constants.HV_KVM_EXTRA].split(" ")])

# Save the current instance nics, but defer their expansion as parameters,
# as we'll need to generate executable temp files for them.

[George Diamantopoulos]

Hi,

I applied the patch, but I'm still getting similar errors. kvm_extra according to gnt-instance info is set to the following:

    - kvm_extra: -device virtio-scsi-pci,id=scsihw0 -drive file=/dev/sdb,if=none,id=sdb,cache=writeback,aio=native -device scsi-hd,drive=sdb,id=scsi1,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,bootindex=100 -drive file=/dev/sdc,if=none,id=sdc,cache=writeback,aio=native -device scsi-hd,drive=sdc,id=scsi2,bus=scsihw0.0,channel=0,scsi-id=1,lun=0,bootindex=101

I'm still getting the "invalid option" error. Perhaps I should be using a different convention when setting kvm_extra? I'm issuing the same modify command as before, I am not sure whether I should be using delimiters or anything like that.

[Guido Trotter]

Just to be sure: did you restart Ganeti after applying the patch?

Thanks,

Guido

[George Diamantopoulos]

As a matter of fact, no, I hadn't :-D

After restarting the daemons, I'm now getting:

Could not start instance: Hypervisor error: Failed to start instance nas.mydomain.tld: exited with exit code 1 (qemu-system-x86_64: -cpu host: could not open disk image ['-device virtio-scsi-pci,id=scsihw0 -drive file=/dev/sdb,if=none,id=sdb,cache=writeback,aio=native -device scsi-hd,drive=sdb,id=scsi1,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,bootindex=100 -drive file=/dev/sdc,if=none,id=sdc,cache=writeback,aio=native -device scsi-hd,drive=sdc,id=scsi2,bus=scsihw0.0,channel=0,scsi-id=1,lun=0,bootindex=101']: No such file or directory

[Guido Trotter]

Indeed, you're right. That's what I get from proposing patches without
trying them out myself. :)
The correct line is in ganeti 2.7.0 rc3 and is:
kvm_cmd.extend(hvp[constants.HV_KVM_EXTRA].split(" "))

rc3 is going to be out soon. :)

Thanks,

Guido

--
Guido Trotter
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370

[George Diamantopoulos]

Yeap, you nailed it this time :-).