Cannot assign USB controller to App VM anymore (Qubes 3.2)

[sbor...@gmail.com]

Dear Qubes community,

after using 3.1 and 3.2 in production on my primary laptop
(Lenovo X220), and having used that machine to test Qubes since R2,
I now have the need to make my built in camera available in an App VM (I choose untrusted, but may a dedicated one later on).

However, I am failing to pass through the
USB controller to the App VM. This
may never have worked with Qubes 3.x (didn't need it so far), but I definitely tested this in the 2.x days.
Since it was experimental(?) at the time, I chose not to install
a dedicated USB VM, so by default both USB controllers are
assigned to Dom0. This is what my system/hardware looks like
Please note that this is Qubes R3.2!!

lspci (in Dom0):
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)

lsusb (in Dom0):
Bus 002 Device 003: ID 0bdb:1911 Ericsson Business Mobile Networks BV
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP)
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Output of 'readlink /sys/bus/usb/devices/usb1'
../../../devices/pci0000:00/0000:00:1a.0/usb1

I assumed that the path of least resistance would be to attach
the USB controller with pci ID 00:1a.0 to my AppVM (untrusted).
So,

qvm-pci -a untrusted 00:1a.0
qvm-pci -l untrusted
['00:1a.0']

However, as apparently often seen (mailing list, FAQ), at that
point I fail to start the AppVM:

[user@dom0 ~]$ qvm-start untrusted
--> Creating volatile image: /var/lib/qubes/appvms/untrusted/volatile.img...
--> Loading the VM (type = AppVM)...
Traceback (most recent call last):
File "/usr/bin/qvm-start", line 136, in <module>
main()
File "/usr/bin/qvm-start", line 120, in main
xid = vm.start(verbose=options.verbose, preparing_dvm=options.preparing_dvm, start_guid=not options.noguid, notify_function=tray_notify_generic if options.tray else None)
File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1979, in start
self.libvirt_domain.createWithFlags(libvirt.VIR_DOMAIN_START_PAUSED)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirt.libvirtError: internal error: libxenlight failed to create new domain 'untrusted'

And xl dmesg shows:

XEN) [VT-D] It's disallowed to assign 0000:00:1a.0 with shared RMRR at da8d5000 for Dom5.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1a.0 to dom5 failed (-1)

Further, pci ID 00:1a.0 still shows up in dom0.

In the context of dedicated USB VMs there is a FAQ pertaining to this,
and clearly there are several github issues related to this. However,
e.g., after

qvm-prefs untrusted -s pci_strictreset false

I get exactly the same error (AppVM untrusted fails to start). I tried
the trick resetting USB to 2.0 (though given the age of the machine
I am not even sure that this is a 3.0 hub/device); again no effect --
as far as I can tell identical error.

Yesterday too late I found some discussions from 2015 in a Xen mailing list, where someone eventually succeeded using several options, but
I don't know how to set these in Qubes (via qvm-prefs??).

I should add that i tried again after rebooting as well, but no
change. So, I am puzzled as I know that this worked in Qubes 2.x.
Am I missing some small print in my attempts and/or in what order
should I try the tricks that might remedy this?

I guess I could try setting up a USB VM, but I assume I would run
into exactly the same issue. And aside from the need to assign the
camera, I don't exactly have a use scenario for a dedicated USB VM
on that machine.

Help appreciated, thanks in advance!

Stefan

[sbor...@gmail.com]

I seem to have it working; I'll outline the steps in case others
run into this. Nevertheless, I'd appreciate an 'authoritative answer'
since I was 'fishing blindly'. [More see inline]

Am Freitag, 2. März 2018 09:34:18 UTC+1 schrieb sbor...@gmail.com:
> Dear Qubes community,
>
> after using 3.1 and 3.2 in production on my primary laptop
> (Lenovo X220), and having used that machine to test Qubes since R2,
> I now have the need to make my built in camera available in an App VM (I choose untrusted, but may a dedicated one later on).
>
> However, I am failing to pass through the
> USB controller to the App VM.

[snip]

I reread

https://www.qubes-os.org/doc/assigning-devices/

and tried enabling 'permissive' mode as described for R3.2 in the above
documentation. However, this per se doesn't work, as the target file
(/sys/bus/pci/drivers/pciback/permissive)
is not writeable, even for root and even when triggered through systemd.

However, I then compared the 'kernelopts' of 'sys-net' to those of 'untrusted',
and noted that 'iommu=soft swiotlb=8192' where missing in the latter. So
I added those, together with forcing 'pci_strictreset False'.

After rebooting the whole machine, untrusted has grabbed the usb hub and sees
the camera. The expected loss of a USB port due to the strange 'wiring' of the Lenovo X220 is acceptable to me; furthermore, I do plan to attach the pci device only when I know that I'll need the camera.

[snip]

>
> And xl dmesg shows:
>
> XEN) [VT-D] It's disallowed to assign 0000:00:1a.0 with shared RMRR at da8d5000 for Dom5.
> (XEN) XEN_DOMCTL_assign_device: assign 0000:00:1a.0 to dom5 failed (-1)
>

For the record, xl dmesg is now telling me that
[VT-D] It's risky to assign .. with shared RMRR at .. for Dom4

what ever that means.

I don't know which of the options / changes did the trick, but one or more
of the above seems to enable the camera in 'untrusted'.

Best regards,

Stefan

[awokd]

On Fri, March 2, 2018 2:27 pm, sbor...@gmail.com wrote:

> https://www.qubes-os.org/doc/assigning-devices/
>
>
> and tried enabling 'permissive' mode as described for R3.2 in the above
> documentation. However, this per se doesn't work, as the target file
> (/sys/bus/pci/drivers/pciback/permissive)
> is not writeable, even for root and even when triggered through systemd.

Just tested again on R3.2 and the procedure in the doc works, but only for
re-assignable devices (ones using the pciback driver in lspci). Are you
sure you were trying to write the correct 0000:<BDF>? When I tried with an
un-assignable device, I got:

bash: echo: write error: No such device

>
>>
>> And xl dmesg shows:
>>
>>
>> XEN) [VT-D] It's disallowed to assign 0000:00:1a.0 with shared RMRR at
>> da8d5000 for Dom5. (XEN) XEN_DOMCTL_assign_device: assign 0000:00:1a.0
>> to dom5 failed (-1)
>>
>
> For the record, xl dmesg is now telling me that
> [VT-D] It's risky to assign .. with shared RMRR at .. for Dom4

Setting pci_strictreset to false is what changed this message from
"disallowed" to "risky".

> what ever that means.
>
> I don't know which of the options / changes did the trick, but one or
> more of the above seems to enable the camera in 'untrusted'.

Setting pci_strictreset to false.

[sbor...@gmail.com]

Thank you for looking into this!

Am Samstag, 3. März 2018 13:28:24 UTC+1 schrieb awokd:

> On Fri, March 2, 2018 2:27 pm, Stefan wrote:
>
> > https://www.qubes-os.org/doc/assigning-devices/
> >
> >
> > and tried enabling 'permissive' mode as described for R3.2 in the above
> > documentation. However, this per se doesn't work, as the target file
> > (/sys/bus/pci/drivers/pciback/permissive)
> > is not writeable, even for root and even when triggered through systemd.
>
> Just tested again on R3.2 and the procedure in the doc works, but only for
> re-assignable devices (ones using the pciback driver in lspci). Are you
> sure you were trying to write the correct 0000:<BDF>? When I tried with an
> un-assignable device, I got:
>
> bash: echo: write error: No such device
>

This sounds like the error I got; don't really know what the properties
of this USB hub are. I did not follow up as this isn't necessary

>
> >
> >>
> >> And xl dmesg shows:
> >>
> >>
> >> XEN) [VT-D] It's disallowed to assign 0000:00:1a.0 with shared RMRR at
> >> da8d5000 for Dom5. (XEN) XEN_DOMCTL_assign_device: assign 0000:00:1a.0
> >> to dom5 failed (-1)
> >>
> >
> > For the record, xl dmesg is now telling me that
> > [VT-D] It's risky to assign .. with shared RMRR at .. for Dom4
>
> Setting pci_strictreset to false is what changed this message from
> "disallowed" to "risky".
>
> > what ever that means.
> >
> > I don't know which of the options / changes did the trick, but one or
> > more of the above seems to enable the camera in 'untrusted'.
>
> Setting pci_strictreset to false.

It indeed looks like it. I reset everything (to the best of my recollection) and
played again with the various options to pass through the USB hub to 'untrusted'. And indeed, this time around, setting pci_strictreset to false
was enough. I was a 100% positive that I tried that before and it seemed not
to work. However, this time, after qvm-reset -s .. I rebooted before trying
whether this changed anything.

Anyways, thanks again for your help!

Stefan