R3.1 rc2 bug report: Failed to attach USB device to other VM than sys-usb.

[sudod...@gmail.com]

To use USB devices other than mice and storage devices it's necessary to attach the PCI USB card to another VM than sys-usb. The obvious way to do this is:
* shutdown sys-usb
* attach USB card to other VM using qubes-manager
* start the other VM

As of R3.1 rc2 this stopped working for me. When starting the VM I get this error:

--> Creating volatile image: /var/lib/qubes/appvms/usb-test/volatile.img...
--> Loading the VM (type = AppVM)...
Traceback (most recent call last):
File "/usr/bin/qvm-start", line 131, in <module>
main()
File "/usr/bin/qvm-start", line 115, in main
xid = vm.start(verbose=options.verbose, preparing_dvm=options.preparing_dvm, start_guid=not options.noguid, notify_function=tray_notify_generic if options.tray else None)
File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1876, in start
self.libvirt_domain.createWithFlags(libvirt.VIR_DOMAIN_START_PAUSED)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirt.libvirtError: internal error: libxenlight failed to create new domain 'usb-test'

This is libxl-driver.log

2016-01-15 14:04:14 CET libxl: error: libxl_pci.c:1000:do_pci_add: xc_assign_device failed: Operation not permitted
2016-01-15 14:04:14 CET libxl: error: libxl_create.c:1422:domcreate_attach_pci: libxl_device_pci_add failed: -3

When I try to start the VM anew:

--> Creating volatile image: /var/lib/qubes/appvms/usb-test/volatile.img...
--> Loading the VM (type = AppVM)...
Traceback (most recent call last):
File "/usr/bin/qvm-start", line 131, in <module>
main()
File "/usr/bin/qvm-start", line 115, in main
xid = vm.start(verbose=options.verbose, preparing_dvm=options.preparing_dvm, start_guid=not options.noguid, notify_function=tray_notify_generic if options.tray else None)
File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1868, in start
nd.dettach()
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 5249, in dettach
if ret == -1: raise libvirtError ('virNodeDeviceDettach() failed')
libvirt.libvirtError: Requested operation is not valid: PCI device 0000:00:1a.0 is in use by driver xenlight, domain usb-test

Note that the PCI device is thought to be attached to the VM I actually WANT to start.

I can't even restart sys-usb without getting the same error.
1) Does anyone have an idea what's going on here?
2) Is there a (xen) way to detach a PCI device from a non running VM? Such that Qubes does not have to be rebooted after such an error?

Thanks for your help!

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Jan 15, 2016 at 05:23:53AM -0800, sudod...@gmail.com wrote:
> To use USB devices other than mice and storage devices it's necessary to attach the PCI USB card to another VM than sys-usb. The obvious way to do this is:
> * shutdown sys-usb
> * attach USB card to other VM using qubes-manager
> * start the other VM
>
> As of R3.1 rc2 this stopped working for me. When starting the VM I get this error:
>
> --> Creating volatile image: /var/lib/qubes/appvms/usb-test/volatile.img...
> --> Loading the VM (type = AppVM)...
> Traceback (most recent call last):
> File "/usr/bin/qvm-start", line 131, in <module>
> main()
> File "/usr/bin/qvm-start", line 115, in main
> xid = vm.start(verbose=options.verbose, preparing_dvm=options.preparing_dvm, start_guid=not options.noguid, notify_function=tray_notify_generic if options.tray else None)
> File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1876, in start
> self.libvirt_domain.createWithFlags(libvirt.VIR_DOMAIN_START_PAUSED)
> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
> if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
> libvirt.libvirtError: internal error: libxenlight failed to create new domain 'usb-test'
>
> This is libxl-driver.log
>
> 2016-01-15 14:04:14 CET libxl: error: libxl_pci.c:1000:do_pci_add: xc_assign_device failed: Operation not permitted
> 2016-01-15 14:04:14 CET libxl: error: libxl_create.c:1422:domcreate_attach_pci: libxl_device_pci_add failed: -3

Take a look at /var/log/xen/console/hypervisor.log (or `xl dmesg`
output) - I guess you'll see a message about "shared RMRR", then look
here:
https://github.com/QubesOS/qubes-issues/issues/1544

In short: disable "pci_strictreset" property (using qvm-prefs tool) and
assign all your USB controllers there.

> When I try to start the VM anew:
>
> --> Creating volatile image: /var/lib/qubes/appvms/usb-test/volatile.img...
> --> Loading the VM (type = AppVM)...
> Traceback (most recent call last):
> File "/usr/bin/qvm-start", line 131, in <module>
> main()
> File "/usr/bin/qvm-start", line 115, in main
> xid = vm.start(verbose=options.verbose, preparing_dvm=options.preparing_dvm, start_guid=not options.noguid, notify_function=tray_notify_generic if options.tray else None)
> File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1868, in start
> nd.dettach()
> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 5249, in dettach
> if ret == -1: raise libvirtError ('virNodeDeviceDettach() failed')
> libvirt.libvirtError: Requested operation is not valid: PCI device 0000:00:1a.0 is in use by driver xenlight, domain usb-test
>
> Note that the PCI device is thought to be attached to the VM I actually WANT to start.

This is some bug in libvirt, which do not handle VM startup fail nicely.
Workaround: restart libvirtd service (terminate Qubes Manager before doing
that...).

> I can't even restart sys-usb without getting the same error.
> 1) Does anyone have an idea what's going on here?
> 2) Is there a (xen) way to detach a PCI device from a non running VM? Such that Qubes does not have to be rebooted after such an error?

You can try using qvm-pci utility - it does supports online
attaching/detaching PCI devices. But first make sure that target VM
(usb-test) is started with "iommu=soft swiotlb=8192" kernel parameters
(qvm-prefs tool, "kernelopts" property). Otherwise that VM may panic at
device assignment...

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWm6hSAAoJENuP0xzK19csNoUH/AjLXjnh7MzOg76P+jnqnUxD
Z0VrwtLWF7WdlqzASld0iPCO42HCxJXfTvlZ7Mgpbm9J9fiysrgmvnfUKlkxlK8B
MJ7E/ianIIEl/tpjfnEUBOt8oi9GM5NMUufBpFTwACM9xvBhh7jFU8ptQn0BUH1F
4LjMGNJccz2CFjxvUQnViZYqF3Hx6sMWvE6ZaC37cMcEFTGUf8+LA+Mqddw62wRl
8r67sVHlUljnHUiaPIce7Ws6W2oMGbUfjZ42KmGT8buJ9ep76BfRg5Y+muIV2J+m
um8g7w1aIqWbGB0OdEzHZdRqHujCC4EavK0ujpO1pWv3eU6kMG/hV9BlI5zHJvU=
=/d+P
-----END PGP SIGNATURE-----

[sudod...@gmail.com]

Am Sonntag, 17. Januar 2016 15:42:35 UTC+1 schrieb Marek Marczykowski-Górecki:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>

Thanks for the reply!

Indeed hypervisor.log says something about it being 'too risky to assign 0000:00:1a.0 with shared RMRR at ...'.
So I set pci_strictreset and pcidevs to the same values as for sys-usb, but I still get the same error when using a HVM (Windows). With an ordinary fedora23 appvm the procedure works. I guess using a HVM causes the problem and not Windows, because the error occurs before the VM actually gets started.

[morf...@gmail.com]

I believe I encountered the same issue:

- Qubes OS v3.1 installed on Thinkpad T420 using default options
- After install, sys-net has 4 mapped devices: Ethernet Controler, Wireless adapter, USB 00:1a, and USB 00:1d. sys-net does not require the USB devices for any reason. Unmapping the USB devices from sys-net was not a problem, network was still fine. Mapping the USB devices to any AppVM domain other than sys-net however caused this issue:

Failed to start domain <domain>
0000:00:1a.0 is in use by driver xenlight, domain <domain>

This occurred even after setting pci_strictreset=false via qvm-prefs.

The resolution was to update the BIOS from Lenovo. Now that I am using the most current BIOS build (148) I am able to map the camera USB device to any AppVM and it starts successfully. The webcam is fully operational in any AppVM domain.