Incremental / continuous backups?

[Loren Rogers]

What's a good approach for regular backups?

Does Qubes have a simple way of automatically saving VM snapshots? And,
is there a way to do this incrementally? I assume not, since the
encryption would block it?

Thanks,
Loren

[Jean-Philippe Ouellet]

This is a known problem area.

See discussions in:
- https://github.com/QubesOS/qubes-issues/issues/971
- https://github.com/QubesOS/qubes-issues/issues/858

[Chris Laprise]

I think the easiest, most efficient route currently available is to have
your VMs stored on a btrfs volume, then use snapshots and 'btrfs send'.
The latter creates a stream that contains the differences from the prior
snapshot, and this stream can be piped to a backup VM. If the backup VM
is untrusted, you can encrypt it in dom0 and have the backup VM store
them as incremental backup files.

A search on 'btrfs send backup' will bring up some guides.

Chris

[pixel fairy]

you can encrypt the volume and put snapshots on that. maybe the future of qubes-backup?

till then i use a script from dom0,
qvm-run backupkeeper "rm -rf QubesIncoming/*"
for i in `cat backuplist`;do qvm-run $i "qvm-copy-to-vm backupkeeper .";done

then rdiff-backup QubesIncoming /run/media/user/.../backups/rdiff

this way, the same drive can also keep appvm snapshots.
backupkeeper is just an appvm with no network access and a lot of space. the usb disks are setup with cryptsetup, ext4 and a backups folder owned by user. the first line is to get rid of old backups. qvm-copy-to-vm wont let you overwrite.

its no TimeMachine, and deleting and copying entire folders is inefficient. but does the job and easy to recover on any linux system.

would be nice to be able to initiate the file copy from dom0 and auto allow it. then it could run in the background.

> Thanks,
> Loren

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Actually you can "auto allow it". Simply click "Yes to all" during one
backup run - it will add appropriate rule to policy to allow the same
operation (same source and target VM) in the future without asking.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYLOWUAAoJENuP0xzK19cs3UAH/3CcPVZy9fTSCCI8r/8yR4i5
bx4Y6HDuNQeF3gYp18rsCKAc9U+iBZXMata+lHzN0htIRV3Uncv0orEdhlM7SBT/
0d1mB5DN93hZid+H1MtHD6UmwL3mFhkcFo14kYcEYkQO72whBwMzSXIVd5g5iLmg
m04+RGXt8fhEe1DoL0RcZj/Kcu3R4UxhJR0ODRl96K8n6bUEgfq8Y0SybmUqT9lz
lx2SL1TTUVmNC8ZiMCMni8ckm84vxrPe3u3gCW6KzzO4IlEZG9ON4MCvXnSin/BK
hL/IB2Jy/RkjU1MRa8uqKMCnxzTg3ueMKfqPCoJzTgGSTzZ/ZRzhNRJZOTiHuCw=
=MNuy
-----END PGP SIGNATURE-----

[Loren Rogers]

Very interesting - I'll look into these as options. Out of curiosity,
would it be possible to post your script to Gist? (To be sure I don't
miss something.) I'd like to explore this in more detail, and it would
be really helpful to see a working example.

In the past, I've had great success with this rsync backup script:
https://github.com/eaut/rsync-time-backup

What would be the costs/benefits of using VM snapshots instead of rsync?
Would it even be possible to run an rsync script like the one above in
dom0 that reached into the VMs? (I'm still learning the ins and outs of
Qubes.)

Loren

[pixel fairy]

On Friday, November 18, 2016 at 11:41:56 PM UTC-5, Loren Rogers wrote:

> Very interesting - I'll look into these as options. Out of curiosity,
> would it be possible to post your script to Gist? (To be sure I don't
> miss something.) I'd like to explore this in more detail, and it would
> be really helpful to see a working example.

im cleaning it up for general consumption. its a sync and inc style backup (thats just what i call it, im sure this style of back up has other names), meaning the appvms dont have direct access to the incremental backups. that way, malware can overwrite the past and you can laugh off ransomware.

heres the process in more detail.

1. install rdiff-backup (or use that script) in your chosen template vm, and make a backupvm. give it enough storage for all your other vms backups. to do this use the qubes vm manager, go into vm settings and set disk storage. since my online life is boring, 10 gigs is enough. most people here will want at least 100.

2. make your backups disks. $1 is the device. for this purpose, i usually dont bother with partitioning, and just use /dev/xvdi

#!/bin/bash
set -exu
cryptsetup -v --hash sha512 --cipher aes-xts-plain64 --key-size 512 --use-random --iter-time 5000 --verify-passphrase luksFormat $1
cryptsetup open $1 test
mkfs.ext4 /dev/mapper/test
mount /dev/mapper/test /mnt
df -h /mnt
umount /mnt
cryptsetup close test

then mount your new disk, and

sudo mkdir backups
chown -R user.user backups

you should keep at least two.

then, im dom0, keep a list of appvms you want to backup, and run this,

qvm-run backupvm "rm -rf QubesIncomming/*" # delete previous backup
for i in `cat backuplist.txt;do qvm-run $i "qvm-copy-to-vm backupvm .";done

youll have to accept a lot of copy permissions. when thats done,

mount a backup disk in backup vm and

backupvolume=`ls -d /media/run/user/*`
rdiff-backup QubesIncomming $backupvolume/backups

this last part could be replaced with any incremental backup, like rsync-time-backup. another possibility is duplicity for an online backup.

the above process is inefficient in that a full copy is made every time you make a backup. the only way i can think of to get around that would be to make a backup disk image for each targetvm and then attach all of them to the backupvm for the incremental backup. this would also be an easy way to exclude certain files like caches.

> What would be the costs/benefits of using VM snapshots instead of rsync?
> Would it even be possible to run an rsync script like the one above in
> dom0 that reached into the VMs? (I'm still learning the ins and outs of
> Qubes.)

you run commands in a vm from dom0, but you cant rsync across vms. you can also run commands in a vm from another vm, but you have to set up special permissions for that. see https://www.qubes-os.org/doc/qrexec3/

>
> Loren