[Advanced] Enabling nested virtualization in Qubes/HVM

610 views
Skip to first unread message

he...@ruggedinbox.com

unread,
Feb 7, 2016, 6:12:25 PM2/7/16
to qubes...@googlegroups.com
Input from the developers would be especially welcome.

I've been trying to enable nested virtualization in Qubes, which should be
possible without modifications, since Xen requires only the addition of
two lines to a conf file:

-------
Make sure you have the right support
Xen 4.4 or later
Intel CPU with EPT support

Add the following to your config file:

hap=1
nestedhvm=1

(Cite: http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen)
-------

1) What's the preferred way to accomplish this in Qubes?

2) It looks like the template packages qubes-gui-vm, xen-qubes-vm, and
xen-libs are set up to block installation of other in-guest virtualization
packages like qemu/libvirt (requiring the former to be removed for any
experimentation to proceed). It happens on both the fedora and debian.
Removing those packages would cause problems interacting with dom0. What's
going on here?

Thank you.

Eric Shelton

unread,
Feb 8, 2016, 4:01:46 PM2/8/16
to qubes-users, he...@ruggedinbox.com
It is not that trivial - patches are required, due to some shortcomings in libvirt (Qubes R3+ uses libvirt to start and manage domains, not xl):


Before you pursue nested virtualization, you may want to read the following, which covers most of the discussions on this:


In short, the Qubes team does not wish to have nested HVM as a standard feature, and that is a reasonable decision.  It is too poorly tested, and Xen 4.4 seems to be the high water mark for the feature, with it appearing to be slowly breaking down due to lack of use (and consequently user testing) and active maintenance.  It seems to have never really gotten beyond being a "toy" feature of Xen.  That may change down the road (I have seen mention of regression tests being implemented for nested HVM on xen-devel), but that is the current state of things.

If you know what you are doing, and can apply the patches noted above, proceed at your own peril.

Best,
Eric

lastp...@gmail.com

unread,
Jul 18, 2019, 3:13:24 PM7/18/19
to qubes-users

Cool but where is that config file located???

unman

unread,
Jul 19, 2019, 8:25:58 AM7/19/19
to qubes-users
On Thu, Jul 18, 2019 at 12:13:24PM -0700, lastp...@gmail.com wrote:
> On Sunday, 7 February 2016 23:12:25 UTC, he...@ruggedinbox.com wrote:
> > Input from the developers would be especially welcome.
> >
> > I've been trying to enable nested virtualization in Qubes, which should be
> > possible without modifications, since Xen requires only the addition of
> > two lines to a conf file:
> >
> > -------
> > Make sure you have the right support
> > Xen 4.4 or later
> > Intel CPU with EPT support
> >
> > Add the following to your config file:
> >
> > hap=1
> > nestedhvm=1
> >
> > (Cite: http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen)
> > -------
> >
> > 1) What's the preferred way to accomplish this in Qubes?
> >

There isnt one.


> > 2) It looks like the template packages qubes-gui-vm, xen-qubes-vm, and
> > xen-libs are set up to block installation of other in-guest virtualization
> > packages like qemu/libvirt (requiring the former to be removed for any
> > experimentation to proceed). It happens on both the fedora and debian.
> > Removing those packages would cause problems interacting with dom0. What's
> > going on here?
> >
> > Thank you.
>

As you correctly point out there are various features in Qubes which
make this difficult. In any case, nested virtualization in Xen is pretty
broken, and doesnt seem to work with more recent kernels. Look at the
tables on the page you cite: you have to go back before 4.7 to get
decent coverage, and anything after 4.9 looks dead.

So enabling this is a security hit for Qubes, and not a priority - (this
may change as there are periodic requests for the feature).

awokd

unread,
Jul 19, 2019, 3:14:02 PM7/19/19
to qubes...@googlegroups.com
lastp...@gmail.com:
> On Sunday, 7 February 2016 23:12:25 UTC, he...@ruggedinbox.com wrote:

>> Add the following to your config file:
>>
>> hap=1
>> nestedhvm=1

> Cool but where is that config file located???
>
Note you are replying to a three year old post. I think there is more to
it than just those config values, but you'd add them as Xen options to
/boot/efi/EFI/qubes/xen.cfg (or /boot/grub2/grub.cfg). Try searching
this list and qubes-issues for "nested virtualization".

Ilpo Järvinen

unread,
Jul 20, 2019, 3:53:20 PM7/20/19
to qubes-users
IIRC, the xen package is built in Qubes with nested virtualization
disabled by default to reduce attack surface so you'd need to compile
it yourself.


--
i.
Reply all
Reply to author
Forward
0 new messages