kernel-latest broke my system

[Fabrizio Romano Genovese]

In trying to make my wifi adapter working, I decided to try `kernel-latest` on Dom0, which installed kernel `5.10.11-1.fc25.qubes.x86_64`. The result is a system where I cannot start VMs (not even VMs with no devices connected to them) due to `libvirt` errors ( The kernel doesn't support reset from sysfs for PCI device ...).

I tried to go back to my old kernels by changing `xen.cfg` in `/boot/efi/EFI/qubes` (here I have options 5.4.90-1.qubes.x86_64 and 5.4.91-1.fc25.qubes.x86_64, besides the one I mentioned above). The real big problem is that these kernels do not seem to appear to work anymore. As soon as I change `default` in `xen.cgf` selecting one of these two kernels, I am not able to access the system (after I insert the LUKS passphrase I get black screen in the authorization manager. Moreover, from boot messages it seems that neither these kernel can start sys-net anymore).

Any suggestion is really appreciated, I spent the last week configuring my PC and I would literally break into tears if I had to re-do everything from scratch.

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Fabrizio Romano Genovese:

Did you also install kernel-latest-qubes-vm (in addition to
kernel-latest) in dom0? Then maybe that too happens to be somehow
broken on your system.

If you can log in on a console (Ctrl-Alt-F2) *after* all your
autostart VMs have failed to start, check 'qubes-prefs default_kernel'
and try setting the VM kernel to another version - i.e. to one of the
directory names in /var/lib/qubes/vm-kernels/ - like this:

qubes-prefs default_kernel 5.4.90-1
qubes-prefs default_kernel 5.4.91-1.fc25

If you can't log in at all, you could mount the root filesystem from a
Qubes installer console and edit the 'default_kernel' property inside
var/lib/qubes/qubes.xml on the root filesystem mountpoint.

Rusty
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEhLWbz8YrEp/hsG0ERp149HqvKt8FAmAcY8ZfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDg0
QjU5QkNGQzYyQjEyOUZFMUIwNkQwNDQ2OUQ3OEY0N0FBRjJBREYACgkQRp149Hqv
Kt9qzQ//VGZZvcU0kmZc/0rphRHJL8Ds5NHqQcFE/tijIxFGin6as/3r/l0hRP5L
xYA7IXoVlPj/Q2r+Ub5JnPJUl2/0WeYSU40PFJMALCBbjM0Vs1LqE55WbuNu1xBt
8uKlRKIxMfe+uJzFGjUBmNbV8elNCPpxMRHeAWG6iqvHfm/62RlEamKj+7xm1DTY
sOUJzbSFguHWd5P0e/orVpQfaYrfRGexwpLv7+LkM+aQ2i8pDgcnt+ueLtLYDS/F
yQVAd1PLEuxI5AxaKL4OFgDIT5X6EGSof3yayr7/CSCwPo9fMb2W/30ZOGa5+2J5
Y2mXfUfnbZsCBCVolOKNZhVC7we0JxcfDElsJ8nWNHhrOq7vtIKm/Z0NUL2fw7Yu
LFHKgpRRXwnICRETfTWpaCE1mfYx0XA1h+PDxVCNQsn6Qt49Zm993ZGrk6pAS6Dt
9zfDt2wB1qEIB0vd/6ao350B6eDkOB2JElldR6H/BXoFbv1m9h3YD7d8KazehpX5
f2NdT4Zg+ZmJYw4sZbRPIYaYjlNBumJnkF+XyM2ZN/dzDfMe4oY8/AuqxBGeWHln
BTDgX19GwfMviPeTv6K+FsmcaykA48j4sdw/ns6ZVexJ57KgC4cExsKiIT31jHBL
vgzUALinx/QwtCfwl7XxL1TakJ9tQqLt42E0UTA6rPBVDXS/074=
=hqBL
-----END PGP SIGNATURE-----

[Fabrizio Romano Genovese]

Yes, I installed kernel-latest-qubes-vm as well, while still using 5.4.90-1 on dom0, and things ran smoothly without much problems, even after several restarts. I did what you suggested but nothing really happens.

It turns out this is a bug of kernel 5.10.11-1.fc25.qubes.x86_64, which is being discussed here: https://github.com/QubesOS/qubes-issues/issues/6377

The bug will be probably fixed already in 5.10.12, the problem is that I don't see how I can install the upgrade if I cannot launch VMs.

What really puzzles me is that 5.10.11-1 somehow managed to break also 5.4.90-1.qubes.x86_64 and 5.4.91-1.fc25.qubes.x86_64.

In any case, elaborating on your hints it turns out that using kernel 5.4.91-1 I can get a shell by doing Ctrl+Alt+F2. I am greeted by a message saying that X won't start, but at least I can login.

Here I can launch VMs like vault, that have no device assigned (on kernel 5.10.11-1 I cannot launch anything).

Still no hope for sys-net: It says "Unable to reset PCI device <number>: no FLR,  PM reset or bus reset available, see /var/log/libvirt/libxl/libxl-driver.log for details".

Following the suggestion, the log says "unable to add device with path <path>", and "libxl_devices_destroy failed for 5".

...If I manage to get internet running, I may download the new, patched kernel and hopefully be back with a functioning computer. Any clue as what may be the next steps here?

[Fabrizio Romano Genovese]

...Following the suggestions on https://www.qubes-os.org/doc/pci-troubleshooting/ I was able to start sys-net and get internet running. I'm running qubes-dom0-update and it works, but the new kernel version is not yet available. I hope that installing the new kernel solves the problem, it should be a matter of hours before it can be installed. Fingers crossed! If things go well (let's hope!) I'll document what I did in https://github.com/QubesOS/qubes-issues/issues/6377 in case other people end up in the same nasty situation. :)