How to nested virtualization with Xen?

[J. Eppler]

Hello,

Xen 4.4+ supports nested virtualization.
The Xen wiki page about nested virtualization, writes the following:

To use nested virtualization, you need to add the following two lines in your guest configuration file:

    hap=1
    nestedhvm=1

where can I find the the guest configuration for Qubes guests?
Does anybody has experience in using nested virtualization in qubes?
Xen - Xen
Xen - VirtualBox
Xen - KVM

Best regards
  J. Eppler

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Jan 24, 2016 at 09:20:14PM -0800, J. Eppler wrote:
> Hello,
>
> Xen 4.4+ supports nested virtualization.
> The Xen wiki page about nested virtualization

> <http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen>, writes the

> following:
>
> To use nested virtualization, you need to add the following two lines in
> your guest configuration file:
>
> hap=1
> nestedhvm=1
> where can I find the the guest configuration for Qubes guests?
> Does anybody has experience in using nested virtualization in qubes?
> Xen - Xen
> Xen - VirtualBox
> Xen - KVM

This isn't that easy because libvirt doesn't have support for it. Take a
look here:
https://groups.google.com/d/msgid/qubes-devel/e116b03e-16ad-42e1-9479-99047696dcc0%40googlegroups.com

Also we don't enable this feature on purpose, because we think it
greatly enlarge Xen attack surface.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWpfG+AAoJENuP0xzK19csX+IH/A0v1OAqRKOzrUF0zwIR2EsX
yUuo+iQo+bd1KzbbmAGaOXKw7wkmTwhsi8aVgoi16vUcaUg/kiN2nSWnOWa4jb1P
u/nvbaaQG7ZgubTnkXknI1pzhn9ZDo/hzUiucZIueZ1tRXxaY37IY5sCTn3/TM0R
R3OscvvFrMrp+1a6e7EVCP2bfZeZOggRWoZ8+1CgRkKU8jt5R5Y5QVe8z8w5CNZE
xK/JpRq9HYQIDB5qNaw2iip4U7SnSp4OdRqqTv9CkJF1Lu3Rk5xKmJzlhosTsvLY
JVQeJOxgc3r2qvumCZ7FXryNXQQoTK0YabXDm93YztszkcLAmcsGFoPN2L8jiCo=
=VEsb
-----END PGP SIGNATURE-----

[J. Eppler]

Hello,

is Erics patch implemented into libVirt?
The other question is can I enable it on purpose in Qubes?

I am pretty sure that it enlarges the Xen attack surface, but to which amount?
Furthermore I don't think that every hypervisor/virtual machine monitor enlarges
the Xen attack surface in the same way?

Best regards
  J. Eppler

[Eric Shelton]

On Monday, January 25, 2016 at 10:40:46 AM UTC-5, J. Eppler wrote:

is Erics patch implemented into libVirt?

Those patches probably were not upstreamed into libvirt, as I did get around to submitting it to the libvirt developers.  I should get around to it, since it does fix a couple of things other than nested HVM (for example, fully implementing the 'viridian' feature).

 

The other question is can I enable it on purpose in Qubes?

If you rebuild libvirt as discussed, then you just need to use the '--custom-config' option of qvm-start to use a modified .conf file for the VM that includes <nestedhvm> in the features section (and probably also <hap> - I forget).

 

I am pretty sure that it enlarges the Xen attack surface, but to which amount?
Furthermore I don't think that every hypervisor/virtual machine monitor enlarges
the Xen attack surface in the same way?

I'm not quite sure what you are asking.  The nested HVM code is Xen has never left the tech preview status, as far as I know.  I am not aware of the feature being widely adopted by users, and as a consequence, it is not anywhere as fully tested as other parts of Xen.  Further, I am not sure how well maintained it is.  It may be that as new features have been added since 4.4, and as various bug fixes have been made, that the nested HVM code has gotten broken in various ways.  For example, if you look at the two charts comparing the status in Jan 2014 and Dec 2014, in those 11 months it appears that VMware and HyperV were broken: http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen  Unless someone among the Xen developers is actively maintaining nested HVM support, I suspect this has only grown worse in Xen 4.6.

In short, if you equate poorly tested and infrequently used code with potentially buggy code, then use of the feature can significantly affect the attack surface (at least from the perspective of hostile code running within a VM for which nested HVM is enabled).

 

Does anybody has experience in using nested virtualization in qubes?
Xen - Xen
Xen - VirtualBox
Xen - KVM

Xen works, as that is what was needed for nested Qubes to work.  I also did some limited testing of KVM and VirtualBox, with positive results: https://groups.google.com/d/msg/qubes-devel/Es14znX-62M/Z92h8h11AgAJ  However, it never worked quite as far as being able to run OS X in KVM under Qubes - the VM would just crash instead.  So, it is not 100%.  Nested Qubes seemed to work OK, though.

My work with it last fall was experimental, but I gave up on it shortly after August.  Best of luck if you want to pursue it further.

Eric 

[J. Eppler]

Hello Eric,

thank you for your long answer. I thought it would be easier to enable and be more stable.
This means for now it is more an academic/research subject.

Best regards
  J. Eppler