Full proper backup of Dom0 possible?

[Otto Kratik]

For both Template and AppVM's it's easy enough to backup and restore the entire thing as needed using the built-in tools from Qubes Manager, Qmenu or command line. Anything goes wrong, just restore a backup.

But for dom0, all that seems to get backed up is the /home directory, meaning that any changes to other system areas such as /etc/qubes-rpc/policy/ for example would not get restored in the event of a reversion to a previously made backup after a fresh system install.

Is there any way to make a full proper backup of Dom0 that would include absolutely everything outside of the other VM's on the system, obviously? So that after performing a fresh system install and then restoring all backups (dom0, templates, appvms) you would end up with a complete byte-for-byte replica of the previous system that existed?

This could be done with external tools like DD of course, but that produces a gigantic image of the entire hard drive rather than just the needed parts, and also doesn't allow one to fully restore *just* dome0 while leaving the other templates/VMs alone, if desired.

Does such a versatile option exist?

[Chris Laprise]

I don't think there is any 100% proper way to backup dom0 root while
dom0 is running. However, its still possible to perform backups on the
root volume if you accept a little risk of filesystem 'corruption' which
should normally only manifest as a need to run fsck (the filesystem
journal will take care of the rest).

The way to start the process is with an LVM snapshot of the root volume:

sudo lvcreate -pr -kn -ay -s qubes_dom0/root -n root-snapshot

Then you can simply tar the resulting device:

sudo tar --sparse -czf - /dev/qubes_dom0/root-snapshot | qvm-run -p
backupvm "cat >root.tgz"

Since tar is using sparse mode, unused areas of the volume won't be
stored in the archive.

Another way is to use dd to compress the image directly, and most unused
areas should be read as zeros and compressed to almost nothing.:

sudo dd if=/dev/qubes_dom0/root-snapshot | gzip | qvm-run -p backupvm
"cat >root.gz"

Yet another option is to mount the root-snapshot volume and do a regular
tar of the files inside.

Since you're concerned with not backing up data unnecessarily, there is
also an incremental backup tool I'm working on:

https://github.com/tasket/sparsebak

Since it instantly knows where changes are in a disk volume – and the
unchanged data isn't scanned – the incremental backup size and speed is
proportional to the changes in the volume. This means that large volumes
with small changes are updated in just seconds. With this tool,
snapshots are automatically created and it can be used on any regular
Qubes lvm volume (but isn't limited to Qubes).

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Not yet. There is an open issue for this:

https://github.com/QubesOS/qubes-issues/issues/3550

I currently just make sure to create a backup of each such file in my
home directory so that they get included in my normal dom0 backups.

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEZQ7rCYX0j3henGH1203TvDlQMDAFAlz/BrgACgkQ203TvDlQ
MDDg2BAAxfBcm0aAv7DhJa8TOOGWGxv0CSqBH3CQZmh9rFW2N0exH9BHF2wI8LcC
AFZUNPHmWJ/dlSt7XW0fp80agQKNR0SuITgio30PZYK7eEMWV/wl975eOnlodHq1
idqDxaBmi2i4Of6lxaUzzU72hsAXjqYTCeUgepFrOGseLZHXJKSHb9IArl0kFJG7
a9AU3pDx56hqXqltAlutdJ8oUGT9qHjoSSLpsnjiG5jVC1PwYl5d3eytpTsVe/nT
cVqMauEI3VxsARnq+DtIVrUzAx9UyEHqcsKMVW6VStafeMSjxj5V9WRyiatQznBR
9D3cqww+MmjGNKFcf2q1c6xrmR8rCXrBMY9heaJo1FAfZruMViJcwpmoDm0aZSdZ
ic0qibr80TOeTho22/cRNDrrRQ42/LK/T910/xeBu8ZPDFGggOkNdDBR+XRwcmZq
YG1EltZ845sUfPxzW8jGiPMq8ixbWfsT+zG4izaj4Ad6wJleJhfpBkyXV5Y3Oq1C
rezLHe/2V0BhAYO0bYtAfFd5jJ1yHyaEjvIh0jAv3nS845gkYP3ILTk4kAJOZA2B
0gIRBwIfk6swhkgDAdZuzl5KBhrvfSQ/CUDpuPBo2sdsirsBIXPOH/0Xyzu8I4MA
6hNiMwNWciwrIKI0uSYV9kFnmYZKXACVQWFy4LAbSAuDvqCPq/0=
=BKpI
-----END PGP SIGNATURE-----

[Chris]

Hello Otto,


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Monday, June 10, 2019 8:04 PM, Otto Kratik <ottok...@gmail.com> wrote:

> But for dom0, all that seems to get backed up is the /home directory, meaning that any changes to other system areas such as /etc/qubes-rpc/policy/ for example would not get restored in the event of a reversion to a previously made backup after a fresh system install.

I use image for Linux (https://www.terabyteunlimited.com/image-for-linux.htm) to make an image of the entire system. I boot from USB/CD and save it directly to my NAS but you can save to external disk.

I have the BootIt collection which is great and have used it for years - highly recommended.

Regards,

Chris

-
Chris Willard
ch...@meliser.co.uk

Sent with ProtonMail Secure Email.

[Steve Coleman]

On 6/10/19 4:04 PM, Otto Kratik wrote:

I have a lightweight solution that may be good-enough for some people
out there, and a simple suggestion at the bottom.

I have been bitten by this dom0 backup omission more times that I can
remember. The problem is that the standard dom0 backup system does not
save any of my dom0 highly tweaked system configuration files. Thus
whenever restoring from backup is required, I am forced to manually
reconfigure everything manually from scratch.

In order to have 'the privilege' of running Qubes-OS here as my desktop
system I am forced to configure my machine according to "the standard"
configuration. I need to install specific software, 2FA, install cron
jobs, run compliance reports, just to maintain access to network resources.

Example: I just got back from short term disability, and found I was
locked out and I needed to breach my own systems numerous security
controls, rebuild, and reconfigure from the ground up. I'm still picking
up the pieces and am trying to get everything back together for the
inspections starting next week.

To save myself from having to go through this fiasco even one more time
I am now saving off the dom0 configuration information using a specific
list of those things that I have to hand modify. I then I push that copy
to a dedicated AppVM where it will be backed up just as any normal AppVM
would be. The hardest part is remembering to add any changed
configuration file to my configuration save list, though I am sure this
too can be automated.

The super simple command I am using to save this set of configuration
files is:

sudo tar cf - --derefrence --files-from=$FILELIST | \
qvm-run -a -p $DOCVM "cd /home/user/dom0-config ; tar xvf -"

Here I am deliberately expanding the directory tree on the other side,
but you might want instead to simply create an archive and label it with
a date time-stamp before moving the archive over. I use this tree to
diff and document my system within that dedicated AppVM. If any dom0
configuration files have changed it will be obvious. When recovering, by
simply moving this configuration tree back to dom0, it will put me back
to where I was before.

Apart from that there may be some rpm packages to install and scripts to
run, but that is Ok with me because I have all that documented and
scripted. I don't need to backup everything in dom0. Just the important
stuff in /etc, /usr/local/*, software archives, special rpm's, etc. If I
didn't have to edit it, run, or install it, then I really don't need to
back it up. Its simply a minimalist recovery capsule.

suggestion - If the standard dom0 command line backup tool could be
extended to allow a dom0 include-list argument, then it might mitigate
this whole problem. If the user could simply add and remove file
references to this include-list, then a full backup of dom0 might not be
necessary. The user then decides what is important to add to this list.
When restoring, the user would still have to move the individual
recovered files back into place, but at least the user would *have* all
the pieces needed to get up and running again.

steve

[Chris Laprise]

Hi Steve,

There is still a consistency problem with this kind of live-system
backup: the files could be in flux while they're being copied. Its also
fussy to configure, as you mentioned, and leaves out packages and
anything else the user forgot to plan.

I think the best solution for a safe and comprehensive dom0 backup is to
have Qubes simply snapshot the root lv at boot time, before its mounted
as read-write. It shouldn't take more than a few script lines in the
dom0 startup. Then dom0 can be backed up like any other vm.

[Chris Laprise]

On 6/11/19 6:50 PM, Chris Laprise wrote:
> I think the best solution for a safe and comprehensive dom0 backup is to
> have Qubes simply snapshot the root lv at boot time, before its mounted
> as read-write. It shouldn't take more than a few script lines in the
> dom0 startup. Then dom0 can be backed up like any other vm.

I created an issue with a 3-line (barely tested) example here:

https://github.com/QubesOS/qubes-issues/issues/5094

[Mike Keehan]

On Wed, 12 Jun 2019 10:29:54 -0400
Chris Laprise <tas...@posteo.net> wrote:

> On 6/11/19 6:50 PM, Chris Laprise wrote:
> > I think the best solution for a safe and comprehensive dom0 backup
> > is to have Qubes simply snapshot the root lv at boot time, before
> > its mounted as read-write. It shouldn't take more than a few script
> > lines in the dom0 startup. Then dom0 can be backed up like any
> > other vm.
>
> I created an issue with a 3-line (barely tested) example here:
>
> https://github.com/QubesOS/qubes-issues/issues/5094
>

Chris, have you some thoughts on how to restore such a backup?

Mike.

[Chris Laprise]

Good question. LVs can be renamed while they are active, so initiating a
reboot process right after lvrename should work. Alternately, lvrename
could be added to the shutdown process itself.

Another approach is to just mount the restored lv and retrieve the files
you need from it.

The other issue with backing up dom0 is what to do about the boot
partition. Its not active during normal use, so could be as simple as
remounting as read-only, then 'dd' or 'tar' to a file on the root fs,
then proceed with root snapshot and backup. Restoring it would be the
reverse.

[Chris Laprise]

Here is another snapshot example that occurs during shutdown; this
doesn't cause delays:

https://github.com/QubesOS/qubes-issues/issues/5094#issuecomment-502077678

As with templates, having automatic snapshots makes it easier to recover
from configuration mishaps. One can just lvrename the root-autosnap to
something else as a way to set aside the most recent good configuration
before making dom0 changes or before rebooting after making changes.