Using btrfs for instant template/VM cloning (COW copy script)

[cprise]

I've found a good way to save drive space and time is to reflink-copy
(COW) the disk images when cloning a VM. This gives me a way to quickly
create new versions of existing VMs to test out new ideas or risky
procedures, and without using gigabytes of extra disk space.

This method requires that Qubes was installed with btrfs, or that
/var/lib/qubes was later moved to a btrfs volume; Doing this on an ext4
volume will fail.

Attached is a quick script I've created to handle the cloning of
templates. It could be adapted to Appvms, HVMs or other types.

Comments and suggestions are welcome.

[Jason M]

@cprise, this is mostly for you just in case you are not aware of these features, etc:

Here are a few other tips if running a BTRFS file system

1. Keep an eye on free disk space. df -h does not always report real usage.
btrfs fi sh /   # show file system info
btrfs fi df /   # disk free

If the disk fills up, you may end up not being able to even delete files to free space.  So be careful about any snapshots

2. Create a snapshot directory (/.snapshots) to generate backup of complete file system in a second or 2.  It does not take up extra space but anything new written to the file system would take additional space and anything deleted will not reduce space on file system unless snapshot is deleted.  Don't forget to track disk usage.
mkdir /.snapshots
btrfs su sn -r / /.snaphots/root-2015-04-18  # Create read-only snapshot
btrfs su de /.snapshots/root-2015-04-18  # Delete snapshot

the snapper program is real cool for generating snapshots.  Will backup system every hour, manual backups, restore, etc.  I have also use a script that allows you to rollback yum or apt-get updates.

3. If disk usage is high when showing a btrfs fi sh /, but not so much with btrfs fi df / or df -h, run a balance:
btrfs balance start / &  # Start balance; other options are available like -dusage 5
btrfs balance status /   # Check status

4. Run a scrub once a month to confirm bitrot or crc
btrfs scrub start /   # Start scrub
btrfs scrub status /  # Check status

Things to look into...

Normally I would keep each virtual machine within its own subvolume, which would require moving existing directory out of way and then recopying (--reflink) old file to new subvolume:
btrfs su cr vmname

Then may add the nodatacow option to the subvolume
chattr +C vmname

I say normally, but that is not in reference to Qubes or Xen since I am unsure if it would make a difference with the current way Qubes (Xen) creates COW versions or the root.img.  The nodatacow option will prevent your extents from growing quickly and also prevent having to run a balance more often since you will see higher disk usage when using btrfs fi sh /

Someone would have to test it out.



Ok, maybe this should have been number 1 tip...

When things go wrong with BTRFS, they go really wrong.  If system crashes for whatever reason and you can not boot; it will get stuck after entering password you will be so relieved you have a USB repair stick ready to go.

Make sure you have a USB or DVD that contains the most recent kernel (at least 3.19.5; not 4 since 4 has a kernel issue with BTRFS I just found out) with at least 3.18.1 btrfs-tools, cryptsetup.  You can build a Release 3 Qubes iso making sure to include the 3.19 kernel; then you can boot into troubleshoot mode, and when asked to unencrypt disk, select skip to do it manually; or maybe a recent enough version of Ubuntu or something like that.

Assuming you are using a a qubes USB stick and entered Troubleshoot mode and skipped mounting
cd /
cryptsetup luksOpen /dev/sda3 crypt     # mount btrfs drive; enter password when prompted
btrfs check --repair /dev/mapper/crypt  # Select 'Y' to delete log; let it complete

Now if you did not see any errors like
root 257 inode 123456 errors 400, nbytes wrong

You are done, disk repaired
shutdown -r now

Otherwise, you may want to attempt to fix those errors as well:
mount /dev/mapper/crypt /root   # mount the btrfs file system
btrfs inspect ino 12345 /root   # Will should you the corrupt filename
truncate -s 0 /path/to/123456   # Will make the length of the file 0.  You lose its contents :(
rm /path/to/123456              # unlink the file (delete)
#repeat for any other errors
umount /root                    # Unmount btrfs file system (/root)
shutdown -r now                 # You are done; reboot

[cprise]

> --

Thanks for those tips, Jason. So far I've found the easiest way to do
snapshots of individual VMs is simply to do a reflink copy of a VM's
folder into a subfolder of home (that way I do not forget about them)...
There's no requirement to create separate subvolumes so it suits me fine.

I am thinking about patching the qubes management library to enable a
use case where new Standalone VMs can be created from a template without
forcing a conventional copy of root.img. This could allow a greater
degree of freedom when testing new software or experimenting.

[2134619036784109437810943278]

Hallo,

very nice would be the integration of the snapshot-functions accessible for VM's direct in the Qubes VM Manger (if Qubes OS is running under btrfs):

- use btrfs Subvolume for the VM xyz
- save btrfs snapshot for the VM xyz
- timetravel btrfs to snapshot xyz for the VM xyz
- btrfs snapshot-copy for the VM1 to VM2
- save btrfs snapshot for the VM1, 2, 3 every hour
- one time snapshot for VM1, 2, 3...

The btrfs speed on my system is very very fast, so it is fun to work with this feature much more.

Kind Regards

[nrgaway]

On 20 May 2015 at 13:05, 2134619036784109437810943278 <kerste...@gmail.com> wrote:

Hallo,

very nice would be the integration of the snapshot-functions accessible for VM's direct in the Qubes VM Manger (if Qubes OS is running under btrfs):

- use btrfs Subvolume for the VM xyz
- save btrfs snapshot for the VM xyz
- timetravel btrfs to snapshot xyz for the VM xyz
- btrfs snapshot-copy for the VM1 to VM2
- save btrfs snapshot for the VM1, 2, 3 every hour
- one time snapshot for VM1, 2, 3...

I agree. Do you have any programming experience in order to implement these features?  Otherwise it's in my TODO list to complete one day.

Maybe we should open a qubes issue so this is not forgotten about?

Couple of other things that shoudl happen if using BTRFS are:

- disable COW for VM images?

- disable compression for VM images

- improved fstab entry

- Use BTRFS subvolume for ALL (each) VM image directory, so /var/lib/qubes/vm-templates/fedora-21 would be a subvolume

[Vít Šesták]

I wanted to do something similar, but with LVM snapshots instead of BTRFS. The main purpose was cloning for atomic online backup. (Consistency: assuming that such online snapshot of a block device is like a sudden power loss for the clone, which is something that journaling filesystems are ready for. Moreover, requesting a sync before backup would also improve the backup)

On BTRFS vs. LVM: Are there any significant advantages of one? I don't have experience with BTRFS, which is the main reason why I decided to use LVM. I also am not sure how would it work with BTRFS:

a. BTRFS subvolume containing some files related to the VM, mainly VM image. This is likely what Jason means, but I think there would be one extra indirection level compared to LVM.
b. Maybe BTRFS has some block subvolumes support, like ZFS has. In this case, it would be very similar to LVM. I think this is the best way on BTRFS, which is also a reason why chose LVM.
c. Use BTRFS ordinary subvolumes and attach them to virtual machines. I am, however, not sure how to make this securely working. I don't know how to attach the BTRFS subvolume device without adding some signifficant attack surface. As far as I know, the FS would be mounted in Dom0 and somehow (NFS? Oh, we don't have networking in Dom0…) shared with DomU. But in any case, I see many technical difficulties, larger attack surface and just few benefits.

Regards,
Vít Šesták 'v6ak'