Is it possible to boot qubes dom0 kernel without Xen?

[Claudia]

I've recently run into some hardware problems in Qubes which are not
present in the equivalent Fedora version (F25). I have a feeling Xen may
be a likely culprit, just simply because of how Xen controls the use of
certain hardware for security reasons (VT-d, and such).

If it were possible to boot Qubes without Xen, it would be a step
towards narrowing down the source of the problem. (The inverse --
installing Fedora *with* Xen -- is another option too, I suppose.)

Disabling VT-x and VT-d in BIOS is easy enough to do, and can probably
identify firmware bugs caused by virtualization, for example. However
even with VT-x/VT-d disabled, Qubes still boots under Xen.

So I was wondering, is it possible to run the Qubes dom0 kernel directly
on the hardware instead of under Xen? How might one go about this? And
how much work would it involve?

-------------------------------------------------
This free account was provided by VFEmail.net - report spam to ab...@vfemail.net

ONLY AT VFEmail! - Use our Metadata Mitigator to keep your email out of the NSA's hands!
$24.95 ONETIME Lifetime accounts with Privacy Features!
15GB disk! No bandwidth quotas!
Commercial and Bulk Mail Options!

[awokd]

Claudia:

> So I was wondering, is it possible to run the Qubes dom0 kernel directly
> on the hardware instead of under Xen? How might one go about this? And
> how much work would it involve?

I've had similar troubleshooting needs. Closest I found was to download
Fedora 25 and test under that. What's missing are any Qubes specific
patches to Fedora itself, which makes the testing results a bit iffy.

--
- don't top post
Mailing list etiquette:
- trim quoted reply to only relevant portions
- when possible, copy and paste text instead of screenshots

[Claudia]

'awokd' via qubes-users:

> Claudia:
>
>> So I was wondering, is it possible to run the Qubes dom0 kernel directly
>> on the hardware instead of under Xen? How might one go about this? And
>> how much work would it involve?
>
> I've had similar troubleshooting needs. Closest I found was to download
> Fedora 25 and test under that. What's missing are any Qubes specific
> patches to Fedora itself, which makes the testing results a bit iffy.
>

Yeah, I've been using Fedora 25 as a test reference, but like you said
there's still a big difference between Fedora and Qubes (Qubes kernel,
Qubes packages, Xen, VT-x, to name a few). It's still a needle in a
haystack.

But anyway... Success! It is in fact quite possible to run Qubes without
Xen, and surprisingly not all that difficult. (And on top of that, in my
case Qubes without Xen was able to detect the hardware it couldn't
detect under Xen!)

So, background info, any Xen dom0 kernel can run as a regular kernel
too. Basically the only difference is that CONFIG_XEN_DOM0 is turned on
in dom0 kernels, but it can still run without Xen. It's just that grub,
I assume, checks if it's running under Xen, and if so, doesn't create
non-Xen menu options (unless you do what I did below).

Here's basically what I had to do:

# tar -C / -cvzf ~/boot-backup.tar.gz boot/
# qubes-dom0-update grub2-efi
# chmod ugo+x /etc/grub.d/10_linux
# echo GRUB_CMDLINE_LINUX=\"$(cat /boot/efi/EFI/qubes/xen.cfg | grep
kernel | cut -d ' ' -f 4- | head -n 1)\" >> /etc/default/grub
# grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
# efibootmgr -c -d /dev/sda -p 1 -L "Fedora-Grub" -l
"\EFI\fedora\grubx64.efi"

Note: this will change your default EFI boot entry! You can change it
back with `efibootmgr -o`. Here's a nice tutorial if you're not familiar
with efibootmgr:
https://www.linuxbabe.com/command-line/how-to-use-linux-efibootmgr-examples

Reboot, and if necessary press f12 and select "Fedora-Grub" in UEFI boot
options. Log in and verify that /proc/xen is not present.

Note: Fedora documentation tells you to install the "shim" package as
well, and boot into the EFI shim to install a grub menu entry. I used
efibootmgr instead as it is much simpler and already installed. In any
case, don't try to run grub2-install. https://fedoraproject.org/wiki/GRUB_2

Note: This should work even with secure boot enabled, but I didn't try it.

Note: You should be able to do something similar on legacy boot systems.
In fact it should be even easier, since you're already using grub. The
important part is the chmod /etc/grub/10_linux.

Random thought: Maybe one day in the distant future, if/when Qubes
supports KVM, maybe we'll be able to switch back and forth between Xen
and KVM at boot time?

[Claudia]

Claudia:

> # tar -C / -cvzf ~/boot-backup.tar.gz boot/
> # qubes-dom0-update grub2-efi
> # chmod ugo+x /etc/grub.d/10_linux
> # echo GRUB_CMDLINE_LINUX=\"$(cat /boot/efi/EFI/qubes/xen.cfg | grep
> kernel | cut -d ' ' -f 4- | head -n 1)\" >> /etc/default/grub
> # grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
> # efibootmgr -c -d /dev/sda -p 1 -L "Fedora-Grub" -l
> "\EFI\fedora\grubx64.efi"

Amendment: Actually, you might have to install grub2-efi-modules as well
and run grub2-install. I had the prebuilt
/boot/efi/EFI/fedora/grubx64.efi working when grub2-efi-modules was
installed, but after I uninstalled it and deleted /boot/grub2/modules,
it apparently stopped working (complained about missing commands), even
though grub2-efi is supposed to be self-contained and is not supposed to
require grub2-efi-modules. I'm not sure -- I have to do more testing.
Just leaving this here as a note for now.

[Claudia]

Claudia:

> Claudia:
>> # tar -C / -cvzf ~/boot-backup.tar.gz boot/
>> # qubes-dom0-update grub2-efi
>> # chmod ugo+x /etc/grub.d/10_linux
>> # echo GRUB_CMDLINE_LINUX=\"$(cat /boot/efi/EFI/qubes/xen.cfg | grep
>> kernel | cut -d ' ' -f 4- | head -n 1)\" >> /etc/default/grub
>> # grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
>> # efibootmgr -c -d /dev/sda -p 1 -L "Fedora-Grub" -l
>> "\EFI\fedora\grubx64.efi"
>
> Amendment: Actually, you might have to install grub2-efi-modules as well
> and run grub2-install. I had the prebuilt
> /boot/efi/EFI/fedora/grubx64.efi working when grub2-efi-modules was
> installed, but after I uninstalled it and deleted /boot/grub2/modules,
> it apparently stopped working (complained about missing commands), even
> though grub2-efi is supposed to be self-contained and is not supposed to
> require grub2-efi-modules. I'm not sure -- I have to do more testing.
> Just leaving this here as a note for now.

From what I can tell, it looks like fedora's prebaked grubx64.efi only
contains a minimal set of modules, which is fine for booting regular
linux/Qubes without Xen, but is missing the modules necessary for Xen
and chainloading. So the original instructions should work as long as
you're okay with using two different loaders.

Your UEFI implementation should let you choose which loader to use
(xen.efi or grubx64.efi), so you can always boot from the original
xen.efi loader (I recommend keeping that as the default boot option). In
other words, you can just select the Fedora-Grub (grubx64.efi) boot
option in BIOS when you want to troubleshoot hardware without Xen, and
select Qubes (xen.efi) otherwise.

When you're finished troubleshooting, you can `dnf remove grub2-efi` and
(if necessary) `rm -r /boot/efi/EFI/fedora`, and use efibootmgr to
delete the Fedora-Grub boot entry from BIOS.

If you want to be able to boot both Xen and non-Xen versions from grub,
you'll have to use grub2-efi-modules and grub2-install instead of the
prebaked grubx64.efi. I think the Qubes installer does this if it
detects other OSes or other versions of Qubes. This also makes it
convenient to choose the Xen/kernel version and edit the command line at
boot. This requires Xen 4.9 and Grub 2.02, which I guess means you
probably need to be using the unstable repo? (I haven't quite worked out
the manual setup for this yet, but it must be possible because the
installer somehow does it automatically.)