Qubes 3.2 no longer booting after dom0 update today

[vsanchez...@gmail.com]

Hi,

I have been using qubes 3.2 for more than a year now on a Lenovo x250 thinkpad.
Today after a qubes-dom0-update ( that appears in dnf history as date: 2018-08-16 12:56 , Action: I,U , Modified: 11 E<) I turned off the pc.

When I turned it on again a few hours later it simply wouldn’t finish qubes boot and restarted every time before getting to the window manager. So I got into grub from the splash screen to explore things and found an empty xen.cfg file in qubes subdirectory : I replaced it with one found in the qubes doc at the UEFI troubleshooting page (and tried out with the three kernel versions present in the boot partition). It didn’t improve things, boot still finished in restart. So I then chose the other items in grub’s menu and suppressed the quiet option from the xen.cfg. In this manner the subitem using kernel 4.14.57 could still not boot but the other two could (partially) with 4.9.56 and 4.4.14. Partially, because in both cases when the boot got through to the gui neither sys-net nor sys-firewall would start (the dom0 log indicates some crashing problems after starting dom0, see pictures).

Any suggestions to help me get back to my previous stable situation ? How difficult would it be to unroll completely the dom0 update that led to this ?
Thanks.

Vicente

[awokd]

Invalid opcode in dom1 makes me think a template got updated and is trying
to execute one of the new mitigations, but your microcode doesn't support
it. Check for a firmware update for your system.

If there isn't one, try setting your VMs to not auto-start by editing
/var/lib/qubes/qubes.xml, then seeing if you can get into Qubes and figure
out which template is causing it. Then switch your VMs to use a different
one.

[vsanchez...@gmail.com]

Thanks for your reply.
I tried upgrading the bios to the most recent version on Lenovo’s site that cites it mitigates variant 4 and 3a of spectre (1.32).

The problem remained the same in qubes though. And I noticed in dmesg for dom0 a message during systemd[1] bring up that says ‘Failed to start Load Kernel Modules.’ (It’s the only message in red).

I tried -as you suggested- to boot with no vm in autoboot and changed their templates from fedora 27 to fedora 26. Then I tried to launch manually sys-net or templates or other vms. They all try to run sys-net (its state icon becomes yellow then disappears), then fail and -some time later- they report ‘Error starting VM: Cannot execute qrexec-daemon’. I also tried using debian-8 template.

After each vm start failure dom0’s log (/var/log/xen/console/hypervisor.log attached) shows again a new iteration of the same ‘Unhandled invalid opcode fault/trap [#6, ec=0000]’ and a call to domain_crash_sync etc...

Any suggestions ?
Thanks again.

Vicente

[awokd]

On Fri, August 17, 2018 10:34 pm, vsanchez...@gmail.com wrote:

> Thanks for your reply.
> I tried upgrading the bios to the most recent version on Lenovo’s site
> that cites it mitigates variant 4 and 3a of spectre (1.32).

Was hoping this would fix it!

> The problem remained the same in qubes though. And I noticed in dmesg for
> dom0 a message during systemd[1] bring up that says ‘Failed to start Load
> Kernel Modules.’ (It’s the only message in red).

You can ignore that error message. You were testing those templates out
while running one of the older kernels, right? Only thing left is Xen
then:
https://www.qubes-os.org/doc/software-update-dom0/#how-to-downgrade-a-specific-package

[vsanchez...@gmail.com]

Thanks for the quick reply. Yes I am using 4.4.14 kernel.

I’ll try downgrading Xen next...after some sleep :)

[vsanchez...@gmail.com]

After you asked me to confirm the kernel version I was using, I rechecked and I actually was still using 4.14.57 for the default vm kernel not the 4.4.14 used in dom0... I changed that setting to 4.4.14 and all vms can run now. So the problem was probably in the more recent Linuxes and finally not with Xen. I’ll explore that.

Thanks again for your help.

[awokd]

On Sat, August 18, 2018 9:30 am, vsanchez...@gmail.com wrote:

> After you asked me to confirm the kernel version I was using, I rechecked
> and I actually was still using 4.14.57 for the default vm kernel not the
> 4.4.14 used in dom0... I changed that setting to 4.4.14 and all vms can
> run now. So the problem was probably in the more recent Linuxes and
> finally not with Xen. I’ll explore that.
>
> Thanks again for your help.

Glad you were able to get it back up and running!