Is it possible to create a fast clone/copy-on-write Qube?

[fshar...@gmail.com]

I want to do the following.

1. Create an HVM Qube. This Qube contains a clean install of an OS such as Windows 8 or Arch.

2. Clone the Qube from #1. The files that make up this Qube should just contain the delta from the parent Qube. Writes are made to the child Qube; reads go to the child Qube (if the data being read was written previously) or to the parent Qube if not (in other words, copy-on-write.) I want to install applications in this Qube, so changes should persist.

Hyper-V does this with differencing disks (see https://technet.microsoft.com/en-us/library/cc720381.aspx.)

It looks like Xen does this also, with fast cloning (see https://docs.citrix.com/en-us/xencenter/6-2/xs-xc-vms/xs-xc-vms-copy.html.)

My question is, can this be done in Qubes? From searching the Web and the docs, it looks like not, but I thought I'd see if I've overlooked anything.

One way would be to have a TemplateVM and a TemplateBasedVM, but have changes to the TemplateBasedVM (outside of /home) be persistent. I don't know if Qubes allows that.

One possible way (I think) would be to use Btrfs as the dom0 file system, but I don't know if that's allowed either.

Otherwise, I'll have to create the parent Qube and then clone it with qvm-clone, but it looks like that creates a full copy, which I'd rather avoid if possible due to the disk space consumption.

Thank you for any suggestions!

[Chris Laprise]

By default, the Qubes 4.0 volume manager will always create COW (not
full) clones. If you choose btrfs instead, the effect will also be the same.

One Qubes 3.2, you have to choose btrfs (or prepare a pool with a
similar filesystem having COW reflinks) for this type of cloning to work.

Keep in mind that (as with Hyper-V) these block level differencing
volumes will diverge increasingly over time as updates are applied. That
means the initial space savings will begin to shrink. I don't know if
Hyper-V has a solution to this, but on Linux one way to help prevent
divergence is deduplication (such as in btrfs).

Overall, these cloning options (using LVM or btrfs) should work more
smoothly than Hyper-V storage. Note the warnings re: volume spoilage on
the Microsoft page don't apply to Qubes; you still have to update each
cloned OS, but there is no need for you to keep track of volumes to
avoid spoilage.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[fshar...@gmail.com]

Hi Chris,

Thank you very much for your fast and helpful reply.

For the sake of my own learning, I've been trying to verify what you said by doing the following:

1. Clone a VM.
2. Compare the sizes of the parent and child VMs.
3. The child VM (which contains no delta from the parent VM) should have a much smaller size than the parent VM (which is about 10 GB).

I haven't been able to do this, I presume due to my ignorance. The above procedure would work with Hyper-V VMs, but it might be conceptually wrong for regard to Qubes/Xen/Linux. I'm new to logical volumes on Linux, thin provisioning, etc., though I've been trying to read up on them today. (If anyone knows of good conceptual material on this stuff, I'd certainly appreciate it.)

I ran qvm-clone to clone my Windows 8 HVM, named "windows". The clone is named "windows2". I have never touched "windows2", so it should have no delta from "windows".

In case it matters, the Windows 8 HVM started life as a Hyper-V .vhdx file, which I converted to a .vhd using Powershell, then to an .img file using StarWind Converter. I then imported it into Qubes using qvm-create with --root-move-from.

My Qubes install is plain vanilla and I haven't tweaked any settings, file systems, etc.

Here are some commands I've run to try to determine the actual size of "windows2", and the output (edited to remove irrelevant info):

$ sudo lvm lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
pool00 qubes_dom0 twi-aotz-- 43.98g 45.81 27.26
root qubes_dom0 Vwi-aotz-- 43.98g pool00 9.03
swap qubes_dom0 -wi-ao---- 5.98g
...
vm-windows-private qubes_dom0 Vwi-a-tz-- 2.00g pool00 0.00
vm-windows-private-1526795188-back qubes_dom0 Vwi-a-tz-- 2.00g pool00 0.00
vm-windows-root qubes_dom0 Vwi-a-tz-- 10.00g pool00 83.61
vm-windows-root-1526795191-back qubes_dom0 Vwi-a-tz-- 10.00g pool00 82.08
vm-windows2-private qubes_dom0 Vwi-a-tz-- 2.00g pool00 vm-windows-private 0.00
vm-windows2-root qubes_dom0 Vwi-a-tz-- 10.00g pool00 vm-windows-root 83.61
...

I'm guessing the LSize column means apparent size rather than actual size, so vm-windows2-root having size 10 G wouldn't be unexpected.

Now I want to find the block special files for these VMs.

$ ls -la /dev/qubes_dom0
...
lrwxrwxrwx 1 root root 8 May 23 13:24 vm-windows2-private -> ../dm-32
lrwxrwxrwx 1 root root 8 May 23 13:24 vm-windows2-root -> ../dm-31
lrwxrwxrwx 1 root root 8 May 23 13:24 vm-windows-private -> ../dm-18
lrwxrwxrwx 1 root root 8 May 23 13:13 vm-windows-private-1526795188-back -> ../dm-17
lrwxrwxrwx 1 root root 8 May 23 13:24 vm-windows-root -> ../dm-20
lrwxrwxrwx 1 root root 8 May 23 13:13 vm-windows-root-1526795191-back -> ../dm-19
lrwxrwxrwx 1 root root 8 May 23 13:13 vm-work-private -> ../dm-16

So the parent (vm-windows-root) is dm-20, and the child (vm-windows2-root) is dm-31.

$ sudo blockdev --getsize64 /dev/dm-20
10737418240
$ sudo blockdev --getsize64 /dev/dm-31
10737418240

Still getting 10 GB for both of them.

$ sudo fdisk -l /dev/dm-20
Disk /dev/dm-20: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0x18b40766

Device Boot Start End Sectors Size Id Type
/dev/mapper/qubes_dom0-vm--windows--root-part1 * 2048 718847 716800 350M 7 HPFS/NTFS/exFAT
/dev/mapper/qubes_dom0-vm--windows--root-part2 718848 25163775 24444928 11.7G 7 HPFS/NTFS/exFAT

$ sudo fdisk -l /dev/dm-31
Disk /dev/dm-31: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0x18b40766

Device Boot Start End Sectors Size Id Type
/dev/mapper/qubes_dom0-vm--windows2--root-part1 * 2048 718847 716800 350M 7 HPFS/NTFS/exFAT
/dev/mapper/qubes_dom0-vm--windows2--root-part2 718848 25163775 24444928 11.7G 7 HPFS/NTFS/exFAT

Still seeing the same 10 GB size for both VMs. The original .vhdx had a max size of 12 GB, with the first 350 MB allocated for the system partition.

$ sudo du -h /dev/dm-20
0 /dev/dm-20
$ sudo du -h /dev/dm-31
0 /dev/dm-31
$ du -h --apparent-size /dev/dm-20
0 /dev/dm-20
$ du -h --apparent-size /dev/dm-31
0 /dev/dm-31

$ df /dev/dm-20
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 1996484 0 1996484 0% /dev
$ df /dev/dm-31
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 1996484 0 1996484 0% /dev

Not sure what I'm seeing there. Could be those commands aren't appropriate for this type of file or storage?

Thank you!

[fshar...@gmail.com]

Ok, I discovered pvs to tell me the disk usage, ran it before and after cloning a ~10 GB Windows VM, and verified usage was the same before and after:

$ sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sdb2 qubes_dom0 lvm2 a-- 58.76g 8.71g

$ qvm-clone windows windows2
windows2: Cloning private volume
windows2: Cloning root volume

$ sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sdb2 qubes_dom0 lvm2 a-- 58.76g 8.71g

Qubes manager gives the disk of the clone VM as 0 before I run it:

https://imgur.com/NGAjTxg

Though afterward it gets the same disk size as the original:

https://imgur.com/hCGUVqs

So again I assume that's the virtual space allocated to the clone rather than the actual space.

[tas...@posteo.net]

On 26.05.2018 07:19, tas...@posteo.net wrote:

> On 26.05.2018 05:56, fshar...@gmail.com wrote:
>> Ok, I discovered pvs to tell me the disk usage, ran it before and
>> after cloning a ~10 GB Windows VM, and verified usage was the same
>> before and after:
>>
>> $ sudo pvs
>> PV VG Fmt Attr PSize PFree
>> /dev/sdb2 qubes_dom0 lvm2 a-- 58.76g 8.71g
>
>

> Actually, the answer is also there in the lvs output. The LSize shows
> the upper limit for the volume's size, and the Data% shows how much of
> that is used.
>
> When you clone a VM (lets say an appVM), you'll notice another private
> volume is created and shows the same figures as the volume it
> originates from. But looking at pool00 (the overall container for our
> VM volumes), the Data% barely changes if at all.

>
>
>>
>> $ qvm-clone windows windows2
>> windows2: Cloning private volume
>> windows2: Cloning root volume
>>
>> $ sudo pvs
>> PV VG Fmt Attr PSize PFree
>> /dev/sdb2 qubes_dom0 lvm2 a-- 58.76g 8.71g
>>
>> Qubes manager gives the disk of the clone VM as 0 before I run it:
>>
>> https://imgur.com/NGAjTxg
>>
>> Though afterward it gets the same disk size as the original:
>>
>> https://imgur.com/hCGUVqs
>>
>> So again I assume that's the virtual space allocated to the clone
>> rather than the actual space.
>

> Yes.
>
> The Linux storage options won't show a volume that grows from nothing
> and has a size equal to the delta from the parent volume. This is
> because the Thin LVM and Btrfs objects are not really hierarchical,
> and this can be a good thing because it usually means fewer demands on
> you attention... there is less 'care and feeding' necessary with the
> Linux options. But you do have to keep an eye on your overall free
> space, of course.
>
> If you really need to know the delta size between two related volumes,
> there may be a utility out there that scans the volume metadata and
> shows you. My guess is you would most likely find it for Btrfs, and it
> would be described as a tool related to deduplication (or 'dedup').
>
> And that brings me to another point: On Btrfs if you have two related
> files (.img volumes) but suspect they have diverged too much over
> time, you can tell Btrfs to perform deduplication to find new
> commonalities and reduce the overall space used.
>
> Chris

[fshar...@gmail.com]

> When you clone a VM (lets say an appVM), you'll notice another private
> volume is created and shows the same figures as the volume it
> originates from. But looking at pool00 (the overall container for our
> VM volumes), the Data% barely changes if at all.

I did see in Qubes Manager that some template-based VMs, sys-net, sys-firewall, and untrusted, have small sizes (< 100 MB) after I played around with them. I'm guessing that represents stuff that has accumulated in their private volumes.

> The Linux storage options won't show a volume that grows from nothing
> and has a size equal to the delta from the parent volume.

That's just what I was (mistakenly) looking for, based on my experience with Hyper-V.

> And that brings me to another point: On Btrfs if you have two related
> files (.img volumes) but suspect they have diverged too much over
> time, you can tell Btrfs to perform deduplication to find new
> commonalities and reduce the overall space used.

Btrfs looks interesting, but given I don't fully understand how Qubes 4 handles storage even in its vanilla configuration, I'd better get some more experience with it under my belt before I experiment with a non-default file system.

Thank you again for your help! This has cleared things up and taught me some things.

[awokd]

On Sun, May 27, 2018 1:43 am, fshar...@gmail.com wrote:

> Btrfs looks interesting, but given I don't fully understand how Qubes 4
> handles storage even in its vanilla configuration, I'd better get some
> more experience with it under my belt before I experiment with a
> non-default file system.

I find the diagram towards the beginning of
https://www.qubes-os.org/news/2017/10/03/core3/ helpful to see how the
pieces fit together.