Qubes Migration: VMWare Virtual Machines

[Andrew Sorensen]

I'm currently in the process of migrating my existing workflow in Fedora 18 (KDE) to Qubes R2 Beta 2. Most of my programs and data are easy to transfer (though I'm setting up it from scratch). For example, I have a number of separate firefox profiles that correspond almost directly to the AppVMs I'm creating. There's a few areas where my workflow wasn't already what I should have in Qubes, but those are pretty easy to fix.

But there's a kicker: my virtual machine library. I have about 100 virtual machines on my machine and external drives that I need to be able to use. All of these machines are currently in VMWare Workstation, and range from Windows installations to test things on (with snapshots at different service packs and update sets), vulnerable Linux distributions to demonstrate security issues, and virtual machines to test things like Qubes itself (yes - VMWare provides nested VT-x), and other virtual appliances in VMDK format. I need to be able to use these virtual machines after migrating to Qubes.

It seems like I have 2 options to continue forward, none of which seems "easy":

1) Convert all my virtual machines to Xen, removing the need for VMWare.

This sounds like a great option, but I have lots of snapshots, and a few of my windows programs work poorly without accelerated graphics.

2) Install VMWare in Qubes.

I tried this by creating a HVM domain with Ubuntu, and installing VMWare Player. VMWare Player stated that I needed to uninstall Hyper-V to run VMWare products (huh?) I also tried this on Dom0, but I could not load the vmmon kernel module (I did quite a bit of research on this, and it's a conflict between Xen and VMWare).

I'd like to be able to do #1, but I know I'm sure I'd run into issues/inconveniences along the way - for example, pressing ctrl+alt+delete in Windows. I'm willing to contribute to the Qubes project (eg: improvements to the qubes-manager, integration of spice [1], etc), to make this work.

If I can do #2, I'd want to convert most of my virtual machines to Xen, and leave a few in VMWare Workstation.

I've also tried VirtualBox, in case I could convert all my virtual machines to it, but it just freezes on starting a virtual machine (at least in a HVM).

I understand running software on Dom0 is a security risk. I don't want to do it - but not being able to migrate because I can't move my virtual machines over will prevent me from being able to gain any benefits from Qubes.

[1] http://spice-space.org/

[Joanna Rutkowska]

The #1 is the way to go.

The #2 will not work. Even if we enabled nested VT-x support in future
Qubes (IIRC it's an optional feature in Xen 4.2), which would however be
quite stupid from a security perspective, as it increases the attack
surface on the hypervisor, chances are slim any real VMM would run with
any acceptable speed using nested root mode.

joanna.

[Marek Marczykowski]

The snapshots could be a problem...

I also have some VMWare (and Virtualbox) machines with a lot of snapshots, and
haven't solved this problem yet. I use them rather rarely (like once a year),
so currently have just separate harddisk with baremetal Linux and
VMware+Virtualbox on it.

>
> The #2 will not work. Even if we enabled nested VT-x support in future
> Qubes (IIRC it's an optional feature in Xen 4.2), which would however be
> quite stupid from a security perspective, as it increases the attack
> surface on the hypervisor, chances are slim any real VMM would run with
> any acceptable speed using nested root mode.
>
> joanna.
>

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[Joanna Rutkowska]

And why do you need RAM snapshots?

As far as disk snapshots are concerned, one can easily do them by hand,
e.g.:

cd /var/lib/qubes/appvms/myvm/
cp root.img root-snapshot1.img

and then link to the actual img file via ln -s. Never tried it, but
should work...

joanna.

[Andrew Sorensen]

I don't think it needs VT-x (AFAIK, VirtualBox only needs it for 64 bit
VMs), I think it's something else that's causing the issue (but I
couldn't say what without a lot more research).

I'm not sure what to say about the speed. I'm currently using Qubes
inside of VMWare Workstation, and I'm happy with it.
>
> joanna.
>

[Joanna Rutkowska]

If it doesn't use VT-x and only uses binary translaiton that it might
work in an HVM, but not in a PV domain (so not in AppVM or Dom0).

> I'm not sure what to say about the speed. I'm currently using Qubes
> inside of VMWare Workstation, and I'm happy with it.

Right, but that's because you use only PV domains, correct? And those do
not require neither VT-x nor binary translation. Or were you also able
to use an HVM domain on Qubes under VMWare?

joanna.

[Andrew Sorensen]

Looks like you are right. I tried to start Ubuntu 12.10 HVM domain
inside of VMWare, and the performance is awful (I didn't get it to a
desktop, just trying to get the system to boot).
>
> joanna.
>

[Andrew Sorensen]

It looks like I'm going to have to go with option #1. In this case, I'm going to see what's already implemented, and possibly suggest a few additional features to the qubes-manager:

eg:

* COW snapshot system, with a way to take and revert to different snapshots in the GUI.
* ability to save/resume a virtual machine's state on disk.
* ability to search the list of virtual machines, and hide VMs that aren't integrated ( qrexec_installed = false)
* ability to use spice instead of (vnc?) to control HVMs
* ability to place VMs on any drive I want (not just /var/lib/qubes)

I'll probably need to create a set of scripts similar to vmware tools that takes care of things like the duplicate mouse cursor, networking, etc).

[Joanna Rutkowska]

As wrote earlier -- you can use a poor man's solution right now (cp
root.img root-snapshot1.img).

If you want to write a patch to Qubes Manager to support it, then please
first write a command line tool that implements it, and preferably just
call this tool (or the function that implements the action) from Qubes
Manager.

Something like:

qvm-snapshot <vmnam> [--new | --revert] [name of snapshot]

> * ability to save/resume a virtual machine's state on disk.

That might be problematic because of the need to reconnect various
things to the restored VM, such as our GUI channel. I think this will be
quite difficult. Marek, what do you think?

> * ability to search the list of virtual machines, and hide VMs that
> aren't integrated ( qrexec_installed = false)

I guess you want to implement a new switch to qvm-ls here?

> * ability to use spice instead of (vnc?) to control HVMs

We don't use VNC nor any other similarly stinking protocol. We use
super-optimized-for-security-and-performance custom GUI protocol.

http://theinvisiblethings.blogspot.com/2012/03/windows-support-coming-to-qubes.html

Why would you need to use something else?

> * ability to place VMs on any drive I want (not just /var/lib/qubes)
>

I think qvm-prefs already has support for that -- check qvm-pref -s dir

> I'll probably need to create a set of scripts similar to vmware tools
> that takes care of things like the duplicate mouse cursor, networking, etc).
>

What do you mean?

joanna.

[Marek Marczykowski]

On 21.03.2013 11:36, Joanna Rutkowska wrote:
> On 03/20/13 19:51, Andrew Sorensen wrote:
>> It looks like I'm going to have to go with option #1. In this case, I'm
>> going to see what's already implemented, and possibly suggest a few
>> additional features to the qubes-manager:
>>
>> eg:
>>
>> * COW snapshot system, with a way to take and revert to different
>> snapshots in the GUI.
>
> As wrote earlier -- you can use a poor man's solution right now (cp
> root.img root-snapshot1.img).

This have few disadvantages:
- making and restoring snapshot takes some time
- need much more disk space than COW version (imaging Windows HVM with a lot
of software installed, say 40GB root.img; then 10 or more snapshots...)

COW snapshots isn't that hard as it sound, actually we already do it for
template-based VMs, right? Still this would need some coding, but not that much.
Another problem is converting existing VMWare snapshots to Linux dm-snapshot
one. Can be a problem, I know nothing about VMWare snapshot format, even not
sure if specification is available anywhere.

> If you want to write a patch to Qubes Manager to support it, then please
> first write a command line tool that implements it, and preferably just
> call this tool (or the function that implements the action) from Qubes
> Manager.
>
> Something like:
>
> qvm-snapshot <vmnam> [--new | --revert] [name of snapshot]
>
>> * ability to save/resume a virtual machine's state on disk.
>
> That might be problematic because of the need to reconnect various
> things to the restored VM, such as our GUI channel. I think this will be
> quite difficult. Marek, what do you think?

The state would include both disk image and RAM, right?
Connections needed to be restored:
- xenstore - already done by xen toolstack
- network/block backend - already done by xen toolstack
- qrexec - trivial, as it doesn't have any state (ongoing Qubes RPC
connection would be terminated anyway)
- gui - this can be problematic

I though many times about ability to restore gui connection. I think it is
complex, but doable. I've already wrote something about this AFAIR in some
thread regarding DispVM startup time.

>
>> * ability to search the list of virtual machines, and hide VMs that
>> aren't integrated ( qrexec_installed = false)
>
> I guess you want to implement a new switch to qvm-ls here?
>
>> * ability to use spice instead of (vnc?) to control HVMs
>
> We don't use VNC nor any other similarly stinking protocol. We use
> super-optimized-for-security-and-performance custom GUI protocol.
>
> http://theinvisiblethings.blogspot.com/2012/03/windows-support-coming-to-qubes.html
>
> Why would you need to use something else?
>
>> * ability to place VMs on any drive I want (not just /var/lib/qubes)
>>
>
> I think qvm-prefs already has support for that -- check qvm-pref -s dir
>
>> I'll probably need to create a set of scripts similar to vmware tools
>> that takes care of things like the duplicate mouse cursor, networking, etc).
>>
>
> What do you mean?
>
> joanna.
>

[Joanna Rutkowska]

On 03/21/13 17:21, Marek Marczykowski wrote:
>>> * COW snapshot system, with a way to take and revert to different
>>> >> snapshots in the GUI.
>> >
>> > As wrote earlier -- you can use a poor man's solution right now (cp
>> > root.img root-snapshot1.img).
> This have few disadvantages:
> - making and restoring snapshot takes some time
> - need much more disk space than COW version (imaging Windows HVM with a lot
> of software installed, say 40GB root.img; then 10 or more snapshots...)
>
> COW snapshots isn't that hard as it sound, actually we already do it for
> template-based VMs, right? Still this would need some coding, but not that much.
> Another problem is converting existing VMWare snapshots to Linux dm-snapshot
> one. Can be a problem, I know nothing about VMWare snapshot format, even not
> sure if specification is available anywhere.
>

Note that VMWare snapshots, AFAIU, contain both disk state, as well as
the RAM state.

joanna.

[Andrew Sorensen]

This depends on the state of the virtual machine when the snapshot is
created. The snapshot of RAM state and disk are always done separately.
> joanna.
>

[Zrubecz Laszlo]

On 21 March 2013 17:50, Joanna Rutkowska <joa...@invisiblethingslab.com> wrote:

> Note that VMWare snapshots, AFAIU, contain both disk state, as well as
> the RAM state.

It depends. It can contain the RAM state as well but then it is
written in a separate file.

It is also contains however all the actual VM settings wich are really
VMware related.



For the migration part the (somehow) standardized .ovf or .ova formats may help.

if some one really wanna poke with the files by hand (in command line)
then the - mostly undocumented - vmkfstools is the tool for that job.



--
Zrubi

[Joe]

I am trying to convert a bunch of VMWare VMs to something that Qubes 4 can run. But it is very hit or miss. I have been using qemu-img convert to combine multiple vmdk files into one raw image.

I usually would try "root-copy-from" during qvm-create, but usually the raw disk file is larger than the default root storage of 10GB. So I wind up creating the vm first, increasing the storage of root.. then dd copying to the root.
But most of the time, it fails to boot for a various set of reasons. I don't know if these VMWare files need more to be converted, they don't seem to have snapshots.
They always seem to have grub, initramfs, dracut issues.

Is there anything I am missing?
How do you normally convert from VMWare files?

Thanks.