How to export (H)VMs from Qubes/Xen to VMware vSphere

[[799]]

Hello,

Currently I still need to run a 2nd OS to use VMware Workstation to prepare/test VMs/Setup for customers.

I'd like to prepare VMs in Qubes and then migrate/export them to the customers environment which are mostly based on VMware vSphere/ESXi.

Questions:

a) How can I get a (H)VM out of Qubes into a VMware VM. If I know what to do, I can script this to get a good workflow.
Worst Case Szenario would be to backup the VM, then manually create a new VM in vSphere, boot with a live Linux and recover the VM - so mainly migrating the harddrive from Qubes/Xen to a VMware .vmdk/virtual harddrive

Other possible approach:

b) Is it possible to do "nested virtualization" and create something like a 'monster-VM" in Qubes in which I install VMware Workstation or ESXi to prepare/test VMs and then export them from there?

c) Do you have any other idea how to use Qubes as primary OS to provision VMs locally and migrate them to vSphere/ESXi at the end of the workflow?
Or is this task not solveable in a good way with Qubes?

Working with Qubes at the customer location would greatly improve security for both sides as I can use separate VMs for each customer or work with disposable VMs when connecting to the network.p

Kind regards

[799]

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Oct 13, 2017 at 01:20:14AM -0400, '[799]' via qubes-users wrote:
> Hello,
>
> Currently I still need to run a 2nd OS to use VMware Workstation to prepare/test VMs/Setup for customers.
>
> I'd like to prepare VMs in Qubes and then migrate/export them to the customers environment which are mostly based on VMware vSphere/ESXi.
>
> Questions:
>
> a) How can I get a (H)VM out of Qubes into a VMware VM. If I know what to do, I can script this to get a good workflow.
> Worst Case Szenario would be to backup the VM, then manually create a new VM in vSphere, boot with a live Linux and recover the VM - so mainly migrating the harddrive from Qubes/Xen to a VMware .vmdk/virtual harddrive

I assume you use Qubes 3.2.
You can get VM's disk image from
/var/lib/qubes/appvms/<name-of-vm>/root.img. This is raw disk image in
sparse file. You can convert it to vmdk using qemu-img tool, like this:

qemu-img convert -f raw -O vmdk /path/to/root.img /path/to/root.vmdk

This should work with HVM created on Qubes. But not for template-based
AppVMs - which heavily depends on Qubes infrastructure.

In case of Windows, you may need to prepare such operating system
for migration - look for migrating Windows installation to a different
hardware (AFAIR the tool is named sysprep).

> Other possible approach:
>
> b) Is it possible to do "nested virtualization" and create something like a 'monster-VM" in Qubes in which I install VMware Workstation or ESXi to prepare/test VMs and then export them from there?

No, nested virtualization is not supported (on purpose, because its
complexity -> large attack surface).

> c) Do you have any other idea how to use Qubes as primary OS to provision VMs locally and migrate them to vSphere/ESXi at the end of the workflow?
> Or is this task not solveable in a good way with Qubes?
>
> Working with Qubes at the customer location would greatly improve security for both sides as I can use separate VMs for each customer or work with disposable VMs when connecting to the network.p
>
> Kind regards
>
> [799]
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZ3zBrAAoJENuP0xzK19csygIH/2gNHJY3I1p8Jg0r735Ql/n/
2lnHus8UHQZrpr545epTf9uU1dRilwlpF4WLkXNAUdMjhjQ3L6HKNeOJKPW7Jiy0
Ss5BdIxQ0oTAiSKrS7TQ1xO3Jw07fV9ZCCmiSPwpWr8YecFBiKGRRi8P0VzKEg+h
Pdv949bq0pQf/V5G2Dw3ZPcocAvK9T6Y6B79OMXeS5YHqf3nh/dPIG5H53CWggSQ
RB0Hjga20XntFDvab4z7pOgk7qTrc3o+g/HEguOJ0yCP7HeMirkBsrXiluzsrbNl
hpTu/n1pXaljrnqmuAENLwNL6RKtu9Y1I3Wan1pB66pCrrAEuIPwcmToJ0xXdMI=
=S/aE
-----END PGP SIGNATURE-----

[[799]]

Hello,

as mentioned in the other thread, I was able to successfully create a cent os minimal HVM.

There are still same smaller problems, as for example the time from grub selection menue to the login prompt is ~10min, but at least I get a login prompt and can login.

*** Converting QEmu/Qubes disk to a VMware .vmdk image ***

>> Me: How can I get a (H)VM out of Qubes into a VMware VM.

> I assume you use Qubes 3.2. You can get VM"s disk image from

> /var/lib/qubes/appvms/<name-of-vm>/root.img. This is raw disk image in

> sparse file. You can convert it to vmdk using qemu-img tool, like this:

> qemu-img convert -f raw -O vmdk /path/to/root.img /path/to/root.vmdk

[USER@dom0 ~]$ ls -lah /var/lib/qubes/appvms/my-test/ | grep .img

-rw-rw-r-- 1 USER qubes 2.0G Oct 14 16:07 private.img

-rw-rw-r-- 1 USER qubes  20G Oct 14 17:26 root.img

-rw-rw-r-- 1 USER qubes  22G Oct 14 16:07 volatile.img

During the installation of Cent OS, I choosed to only use the 20GB root image, not the private 2GB image, as such I didn't understand the size ls is reporting.

[USER@dom0 home]$ du -sh /var/lib/qubes/appvms/my-test/* | grep .img

0 /var/lib/qubes/appvms/my-test/private.img

1.3G /var/lib/qubes/appvms/my-test/root.img

0 /var/lib/qubes/appvms/my-test/volatile.img

this filesizes make more sense to me.

As Marek suggested I tried to convert the image using qemu-img, but as this is not available in dom0 I used qemu-img-xen:

(my-test is my newly created Cent OS HVM, created from a CentOS minimal ISO)

cd /var/lib/qubes/appvms/my-test

qemu-img convert -f raw -O vmdk root.img root.vmdk

this throws an error direct after starting:

qemu-img: error while writing

I have also looked at

https://www.howtoforge.com/how-to-convert-a-xen-virtual-machine-to-vmware

but this uses another syntax (which I also tried but didn't work).

Strangely trying to get informatiom from my root.img via...

qemu-img-xen info root.img

.. results in an error:

qemu-img: Could not open 'root.img'

While I am in the folder and I can see the file with ls.

Any idea where to go from here?

Can someone try to run qemu-img-xen on their system to see if they can get any information out of their image files?

[799]

[Unman]

For info to work specify '-f raw'

I've never got qemu-img-xen in dom0 to work properly for conversion.
A reasonable workaround is to install qemu-img in a qube, and then
attach the root.img using qvm-block -A .

Then you can run qemu-img convert in the qube, and export it from there
as you wish. Works reasonably well.

unman

[Unman]

Also, for completeness I should say that while nested virtualisation
is not supported, it IS possible.
I've run both virtualbox and vmware from within qubes in the past - the
performance isn't great and the security considerations might put you
off, but it is possible.
You will have to do some hacking about to get the kernel modules
working, and I wouldn't generally recommend it. Vmware I couldn't get
working on Debian at all.

[pixel fairy]

On Thursday, October 12, 2017 at 10:20:30 PM UTC-7, [799] wrote:
> Hello,
>
> Currently I still need to run a 2nd OS to use VMware Workstation to prepare/test VMs/Setup for customers.
>
> I'd like to prepare VMs in Qubes and then migrate/export them to the customers environment which are mostly based on VMware vSphere/ESXi.

have you tried running vmware on a dedicated machine and using the vmware workstation binary as a remote interface?

I also need nested virtualization for developing hypervisor management software. this is how i get around it, only with virt-manager instead of vmware.