[Advanced] Enabling nested virtualization in Qubes/HVM

[he...@ruggedinbox.com]

Input from the developers would be especially welcome.

I've been trying to enable nested virtualization in Qubes, which should be
possible without modifications, since Xen requires only the addition of
two lines to a conf file:

-------
Make sure you have the right support
Xen 4.4 or later
Intel CPU with EPT support

Add the following to your config file:

hap=1
nestedhvm=1

(Cite: http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen)
-------

1) What's the preferred way to accomplish this in Qubes?

2) It looks like the template packages qubes-gui-vm, xen-qubes-vm, and
xen-libs are set up to block installation of other in-guest virtualization
packages like qemu/libvirt (requiring the former to be removed for any
experimentation to proceed). It happens on both the fedora and debian.
Removing those packages would cause problems interacting with dom0. What's
going on here?

Thank you.

[Eric Shelton]

It is not that trivial - patches are required, due to some shortcomings in libvirt (Qubes R3+ uses libvirt to start and manage domains, not xl):

https://groups.google.com/forum/#!topic/qubes-devel/UzO0BsIfIow

Before you pursue nested virtualization, you may want to read the following, which covers most of the discussions on this:

https://groups.google.com/d/msg/qubes-users/zTHbWGpVpj8/2CVn7XGbBAAJ

https://groups.google.com/forum/#!msg/qubes-devel/Es14znX-62M/Z92h8h11AgAJ

In short, the Qubes team does not wish to have nested HVM as a standard feature, and that is a reasonable decision.  It is too poorly tested, and Xen 4.4 seems to be the high water mark for the feature, with it appearing to be slowly breaking down due to lack of use (and consequently user testing) and active maintenance.  It seems to have never really gotten beyond being a "toy" feature of Xen.  That may change down the road (I have seen mention of regression tests being implemented for nested HVM on xen-devel), but that is the current state of things.

If you know what you are doing, and can apply the patches noted above, proceed at your own peril.

Best,

Eric

[lastp...@gmail.com]

Cool but where is that config file located???

[unman]

On Thu, Jul 18, 2019 at 12:13:24PM -0700, lastp...@gmail.com wrote:
> On Sunday, 7 February 2016 23:12:25 UTC, he...@ruggedinbox.com wrote:
> > Input from the developers would be especially welcome.
> >
> > I've been trying to enable nested virtualization in Qubes, which should be
> > possible without modifications, since Xen requires only the addition of
> > two lines to a conf file:
> >
> > -------
> > Make sure you have the right support
> > Xen 4.4 or later
> > Intel CPU with EPT support
> >
> > Add the following to your config file:
> >
> > hap=1
> > nestedhvm=1
> >
> > (Cite: http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen)
> > -------
> >
> > 1) What's the preferred way to accomplish this in Qubes?
> >

There isnt one.

> > 2) It looks like the template packages qubes-gui-vm, xen-qubes-vm, and
> > xen-libs are set up to block installation of other in-guest virtualization
> > packages like qemu/libvirt (requiring the former to be removed for any
> > experimentation to proceed). It happens on both the fedora and debian.
> > Removing those packages would cause problems interacting with dom0. What's
> > going on here?
> >
> > Thank you.
>

As you correctly point out there are various features in Qubes which
make this difficult. In any case, nested virtualization in Xen is pretty
broken, and doesnt seem to work with more recent kernels. Look at the
tables on the page you cite: you have to go back before 4.7 to get
decent coverage, and anything after 4.9 looks dead.

So enabling this is a security hit for Qubes, and not a priority - (this
may change as there are periodic requests for the feature).

[awokd]

lastp...@gmail.com:

> On Sunday, 7 February 2016 23:12:25 UTC, he...@ruggedinbox.com wrote:

>> Add the following to your config file:
>>
>> hap=1
>> nestedhvm=1

> Cool but where is that config file located???
>

Note you are replying to a three year old post. I think there is more to
it than just those config values, but you'd add them as Xen options to
/boot/efi/EFI/qubes/xen.cfg (or /boot/grub2/grub.cfg). Try searching
this list and qubes-issues for "nested virtualization".

[Ilpo Järvinen]

IIRC, the xen package is built in Qubes with nested virtualization
disabled by default to reduce attack surface so you'd need to compile
it yourself.


--
i.