Disposable sys-usb creation fails with "unable to recet PCI device"

[tetra...@danwin1210.me]

Following the directions here:
https://www.qubes-os.org/doc/disposablevm-customization/#create-the-sys-usb-disposablevm

I already had a sys-usb VM so did not need to hide USB controllers from
dom0.

After finishing with the given steps, I run `qvm-start disp-sys-usb` and
get the error:
```
$ qvm-start disp-sys-usb
Start failed: internal error: Unable to reset PCI device 0000:00:14.0: no FLR, PM reset or bus reset available, see /var/log/libvirt/libxl/libxl-driver.log for details
```

The corresponding log entry:
```
2020-01-21 01:57:18.598+0000: libxl: libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
```

`qvm-pci | grep USB` indicates that no-strict-reset is already
configured for all my USB devices.

Any suggestions?

[awokd]

tetrahedra via qubes-users:

> Following the directions here:
> https://www.qubes-os.org/doc/disposablevm-customization/#create-the-sys-usb-disposablevm

In step 5, did you include the option?

> I already had a sys-usb VM so did not need to hide USB controllers from
> dom0.
>
> After finishing with the given steps, I run `qvm-start disp-sys-usb` and
> get the error:
> ```
> $ qvm-start disp-sys-usb
> Start failed: internal error: Unable to reset PCI device 0000:00:14.0:
> no FLR, PM reset or bus reset available, see
> /var/log/libvirt/libxl/libxl-driver.log for details
> ```
>
> The corresponding log entry:
> ```
> 2020-01-21 01:57:18.598+0000: libxl:
> libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support
> reset from sysfs for PCI device 0000:00:14.0
> ```
>
> `qvm-pci | grep USB` indicates that no-strict-reset is already
> configured for all my USB devices.
>
> Any suggestions?
>

Did you detach the USB controller from your existing sys-usb (or at
least shut it down)?

--
- don't top post
Mailing list etiquette:
- trim quoted reply to only relevant portions
- when possible, copy and paste text instead of screenshots

[tetra...@danwin1210.me]

On Thu, Jan 23, 2020 at 02:22:20PM +0000, 'awokd' via qubes-users wrote:
>tetrahedra via qubes-users:
>> Following the directions here:
>> https://www.qubes-os.org/doc/disposablevm-customization/#create-the-sys-usb-disposablevm
>
>In step 5, did you include the option?

I used the Qube Manager GUI to attach but -- since the USB controllers
were still marked as attached to disp-sys-usb when I ran `qvm-pci` with
disp-sys-usb powered off, I assume the answer is "yes."

Just in case I removed all the USB controllers from disp-sys-usb, then
ran the step 5 command with all USB controllers (including the
`--persistent` option) and tried starting disp-sys-usb.

The original error ("unable to reset PCI device...") still occurs when
trying to start disp-sys-usb.

>Did you detach the USB controller from your existing sys-usb (or at
>least shut it down)?

I shut down sys-usb but did not detach the devices from it.

I tried removing the devices from sys-usb (so they were exclusively
attached to disp-sys-usb) but the error still appears after doing so.

[tetra...@danwin1210.me]

On Sat, Jan 25, 2020 at 05:35:20AM +0100, tetrahedra via qubes-users wrote:
>On Thu, Jan 23, 2020 at 02:22:20PM +0000, 'awokd' via qubes-users wrote:
>>tetrahedra via qubes-users:
>>>Following the directions here:
>>>https://www.qubes-os.org/doc/disposablevm-customization/#create-the-sys-usb-disposablevm
>>
>>In step 5, did you include the option?
>
>I used the Qube Manager GUI to attach but -- since the USB controllers
>were still marked as attached to disp-sys-usb when I ran `qvm-pci` with
>disp-sys-usb powered off, I assume the answer is "yes."
>
>Just in case I removed all the USB controllers from disp-sys-usb, then
>ran the step 5 command with all USB controllers (including the
>`--persistent` option) and tried starting disp-sys-usb.
>
>The original error ("unable to reset PCI device...") still occurs when
>trying to start disp-sys-usb.

The error is now also happening when I try to start sys-usb!

[tetra...@danwin1210.me]

On Sun, Jan 26, 2020 at 08:11:45AM +0100, tetrahedra via qubes-users wrote:
>>The original error ("unable to reset PCI device...") still occurs when
>>trying to start disp-sys-usb.
>
>The error is now also happening when I try to start sys-usb!

I was able to get disp-sys-usb start (without any attached USB
controllers!) and found another problem:

it looks like the underlying disp-sys-usb template started, rather
than an actual DispVM (the running VM is named `disp-sys-usb` instead of
`dispXXXX`) ...

[tetra...@danwin1210.me]

On Mon, Jan 27, 2020 at 02:18:42AM +0100, tetrahedra via qubes-users wrote:
>On Sun, Jan 26, 2020 at 08:11:45AM +0100, tetrahedra via qubes-users wrote:
>>>The original error ("unable to reset PCI device...") still occurs when
>>>trying to start disp-sys-usb.
>>
>>The error is now also happening when I try to start sys-usb!

It looks like no-strict-reset=True has to be passed *every time* you
attach a PCI device to a VM... that it was passed before when attaching
to a different VM is not enough!

Detaching all USB controllers from sys-usb and then manually reattaching
with
$ qvm-pci attach --option no-strict-reset=True --persistent sys-usb dom0:00_14.0

resulted in a slightly different error when trying to start sys-usb:
$ qvm-start sys-usb
Start failed: internal error: Unable to reset PCI device 0000:00:14.0: internal error: libxenlight failed to create new domain 'sys-usb', see /var/log/libvirt/libxl/libxl-driver.log for details
$ sudo tail /var/log/libvirt/libxl/libxl-driver.log

libxl: libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0

However attaching all controllers to disp-sys-usb (using the above
command) worked, and my USB devices are recognized by disp-sys-usb.

[tetra...@danwin1210.me]

On Mon, Jan 27, 2020 at 02:18:42AM +0100, tetrahedra via qubes-users wrote:
>it looks like the underlying disp-sys-usb template started, rather
>than an actual DispVM (the running VM is named `disp-sys-usb` instead of
>`dispXXXX`) ...

Testing this hypthothesis (by creating a file in the home directory on
disp-sys-usb, and rebooting it) indicates that I'm wrong and
disp-sys-usb actually is running as a disposable VM (the created file
vanishes after reboot).

[unman]

I run named disposable sys-usbs, from a usb template. In my experience
it is *not* necessary to pass the reset option on each boot. The option
is set once and applies on every boot.
(On one x230 I have a separate print usb, and just switch between the
standard named usb and print qube without any issues. Again the reset
option is set once.)

I cant account for what's happening in your set-up.
I'm assuming that your original sys-usb worked fine.
Take a step back: delete all the disposable sys-usb, and confirm that
your sys-usb works fine.
Then create a disposable template - confirm tat *that* works fine.
Then create disposable sys-usb.

If you hit a problem, tell us what hardware you have.

[tetra...@danwin1210.me]

On Mon, Jan 27, 2020 at 11:37:01AM +0000, unman wrote:
>I run named disposable sys-usbs, from a usb template. In my experience
>it is *not* necessary to pass the reset option on each boot. The option
>is set once and applies on every boot.
>(On one x230 I have a separate print usb, and just switch between the
>standard named usb and print qube without any issues. Again the reset
>option is set once.)
>
>I cant account for what's happening in your set-up.
>I'm assuming that your original sys-usb worked fine.
>Take a step back: delete all the disposable sys-usb, and confirm that
>your sys-usb works fine.
>Then create a disposable template - confirm tat *that* works fine.
>Then create disposable sys-usb.
>
>If you hit a problem, tell us what hardware you have.

You quoted a different message than the one you were replying to...

The confusion appears to have been that I thought no-strict-reset was a
setting applied to a PCI device. Instead it appears to be an option
applied to a specific connection between a PCI device and a VM.

Therefore, when *attaching* a PCI device to a VM, you must pass
`--option no-strict-rest=True` *each time you attach the device
manually.*

If you use `--persistent` with qvm-pci then naturally the option is
passed every time you start the VM.

This means that it is not sufficient to do:
```
qvm-pci attach --persistent --option no-strict-reset=True VMNAME DEVICE
qvm-pci attach --persistent OTHER_VMNAME DEVICE
```

Instead you must do:
```
qvm-pci attach --persistent --option no-strict-reset=True VMNAME DEVICE
qvm-pci attach --persistent --option no-strict-reset=True OTHER_VMNAME DEVICE
```

And under no circumstances may you do:
```
qvm-pci attach --persistent --option no-strict-reset=True VMNAME DEVICE
qvm-pci detach VMNAME DEVICE
qvm-pci attach --persistent VMNAME DEVICE
```

[unman]

On Tue, Jan 28, 2020 at 04:30:13AM +0100, tetra...@danwin1210.me wrote:
> On Mon, Jan 27, 2020 at 11:37:01AM +0000, unman wrote:
> > I run named disposable sys-usbs, from a usb template. In my experience
> > it is *not* necessary to pass the reset option on each boot. The option
> > is set once and applies on every boot.
> > (On one x230 I have a separate print usb, and just switch between the
> > standard named usb and print qube without any issues. Again the reset
> > option is set once.)
> >
> > I cant account for what's happening in your set-up.
> > I'm assuming that your original sys-usb worked fine.
> > Take a step back: delete all the disposable sys-usb, and confirm that
> > your sys-usb works fine.
> > Then create a disposable template - confirm tat *that* works fine.
> > Then create disposable sys-usb.
> >
> > If you hit a problem, tell us what hardware you have.
>
> You quoted a different message than the one you were replying to...
>

I quoted a message referring to disposable sys-usbs, and told you my own
experience of disposable sus-usbs.

> The confusion appears to have been that I thought no-strict-reset was a
> setting applied to a PCI device. Instead it appears to be an option
> applied to a specific connection between a PCI device and a VM.
>

Now *that* confusion is cleared up, I assume your problem has gone away?

[Andrey Arapov]

Hi tetrahedra,

> The original error ("unable to reset PCI device...") still occurs when trying to start disp-sys-usb.

Despite have the "no-strict-reset" set to True, you will continue to see the "Unable to reset PCI device: ... no FLR, PM reset or bus reset available" "error" message each time you are trying to attach a PCI device that does not support the FLR (Function Level Reset) [2].

The "no-strict-reset" enablement patch [1] allows you (libvirt) to assign a PCI device to domU, even when the device does not support any reset method .

The error message is kept there for the informational purposes so this way a person may become aware of that his PCI device may not be working as desired because it does not support any reset method, despite of which it still gets assigned to domU when "no-strict-reset" is set to True, thanks to the patch [1].

I think it'd be good to mention here the whole message from the "no-strict-reset" patch:

> This allows to assign PCI device to some VM, even when the device does not support any reset method. This may be dangerous in some cases (especially when the device is later assigned to some other VM). But in some cases it still makes sense - for example when the device is assigned to the same VM whole the time.

To check if your PCI device has a FLR function, run:

$ sudo lspci -vv -s 00:14.0

If you see output with "FLReset-" then your PCI device does not support FLR function. If output has "FLReset+" then it does.

There are very few PCI devices which support FLR function.

[1]: https://github.com/QubesOS/qubes-core-libvirt/blob/release4.0/patches.qubes/0010-Add-nostrictreset-attribute-to-PCI-host-devices.patch#L160

[2]: https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/util/virpci.c;hb=d7acab0bfeec5c9ae75db21b3519486e3586250c

Kind regards,
Andrey Arapov

[Andrey Arapov]

> > The original error ("unable to reset PCI device...") still occurs when trying to start disp-sys-usb.
> >
>
> Despite have the "no-strict-reset" set to True, you will continue to see the "Unable to reset PCI device: ... no FLR, PM reset or bus reset available" "error" message each time you are trying to attach a PCI device that does not support the FLR (Function Level Reset) [2].
>
> The "no-strict-reset" enablement patch [1] allows you (libvirt) to assign a PCI device to domU, even when the device does not support any reset method .

Hum, I have just realized that you have also noticed one more error:

>
> libxl_pci.c: libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
>
>

It looks like this error is related to this code https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libxl/libxl_pci.c;h=6f8f49c7c0a80478b244c52ae65e75f8a78c6481;hb=b03cee73197f4a37bf2941b9367105187355e638#l1150 [https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libxl/libxl_pci.c;h=6f8f49c7c0a80478b244c52ae65e75f8a78c6481;hb=b03cee73197f4a37bf2941b9367105187355e638#l1150]
where, it appears to me at the first sight, we are not patching it.

I raised that question here https://github.com/QubesOS/qubes-issues/issues/3518#issuecomment-579526805

Kind regards,
Andrey Arapov

[tetra...@danwin1210.me]

On Tue, Jan 28, 2020 at 12:22:00PM +0000, unman wrote:
>Now *that* confusion is cleared up, I assume your problem has gone away?

Yes (so far).

[tetra...@danwin1210.me]

On Tue, Jan 28, 2020 at 10:59:00PM +0000, 'Andrey Arapov' via qubes-users wrote:
>
>Hi tetrahedra,
>
>> The original error ("unable to reset PCI device...") still occurs when trying to start disp-sys-usb.
>
>Despite have the "no-strict-reset" set to True, you will continue to see the "Unable to reset PCI device: ... no FLR, PM reset or bus reset available" "error" message each time you are trying to attach a PCI device that does not support the FLR (Function Level Reset) [2].
>
>The "no-strict-reset" enablement patch [1] allows you (libvirt) to assign a PCI device to domU, even when the device does not support any reset method .
>
>The error message is kept there for the informational purposes so this way a person may become aware of that his PCI device may not be working as desired because it does not support any reset method, despite of which it still gets assigned to domU when "no-strict-reset" is set to True, thanks to the patch [1].

Hi Andrey,

All that may be true but it does not explain why the error message was
accompanying an *error condition* -- specifically that the VM refused to
start. If the error had simply been printed in the logs and the VM
started normally (with USB controllers) then it would not have been an
issue.

[tetra...@danwin1210.me]

On Tue, Jan 28, 2020 at 11:52:56PM +0000, 'Andrey Arapov' via qubes-users wrote:
>Hum, I have just realized that you have also noticed one more error:
>
>>
>> libxl_pci.c: libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
>>
>>
>
>
>It looks like this error is related to this code https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libxl/libxl_pci.c;h=6f8f49c7c0a80478b244c52ae65e75f8a78c6481;hb=b03cee73197f4a37bf2941b9367105187355e638#l1150 [https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libxl/libxl_pci.c;h=6f8f49c7c0a80478b244c52ae65e75f8a78c6481;hb=b03cee73197f4a37bf2941b9367105187355e638#l1150]
>where, it appears to me at the first sight, we are not patching it.
>
>I raised that question here https://github.com/QubesOS/qubes-issues/issues/3518#issuecomment-579526805

Thank you!