Missing data after Qubes restore from backup

[Crsi]

Hi,

I recently upgraded from R4.0 to R4.1 with a fresh installation.
I did make a backup of all my important Qubes to an external drive
beforehand and everything completed successfully. I also checked "verify
backup" -- no error reported.

On the new installation, I restored all my Qubes from the backup.
When starting to work with them again, I noticed that some data was
apparently missing from some (not all!) of the Qubes. It turned out that
in two of the restored VMs all data of the last six weeks (!) was
completely missing. It looked like the VM had a state like from my last
full VM backup using the backup utility (which was exactly that six
weeks ago).

I tried restoring the VM again -- but same result. Searching for some
issue like that only gave me [Issue #3498]. Apparently I was hit by the
"flaw" that the verification doesn't detect missing data.

I guess I have no chance to restore my data whatsoever, since the old
drive was wiped. Anyhow, just posting this: does someone has a clue
*why* that could have happened? I don't want anyone to loose their work
of six weeks, too.



[Issue #3498]: https://github.com/QubesOS/qubes-issues/issues/3498

[Qubes]

'Crsi' via qubes-users wrote:
> Hi,
>
> I recently upgraded from R4.0 to R4.1 with a fresh installation.
> I did make a backup of all my important Qubes to an external drive
> beforehand and everything completed successfully. I also checked "verify
> backup" -- no error reported.
>

Do you often reboot your VMs? When Qubes creates a backup of a running
VM it will do the backup using the VM state at startup. So if for
example you started your VM (the ones with the outdated/missing data) 6
weeks ago, but since starting you added/edited/deleted data and you
create a backup, without restarting the VM, the backup will not include
those changes.

[Crsi]

> Do you often reboot your VMs? When Qubes creates a backup of a
running VM it will do the backup using the VM state at startup. So if
for example you started your VM (the ones with the outdated/missing
data) 6 weeks ago, but since starting you added/edited/deleted data and
you create a backup, without restarting the VM, the backup will not
include those changes.

Apparently, no. However, I made sure no VM was running (my host system
crashed previously anyways). That means there's a difference between
powering down a VM (e.g. qvm-shutdown) and having a powered down VM due
to crash / power loss? Am I right that the backup tool uses the LV
`vm-<name>-private` for backup only, but my data was in
`vm-<name>-private-snap`?

Then, what about showing a warning that there exists a snap of a VM in
the qubes-backup utility (at the same place where "The VM is running,
backup will contain its state from before its start!" is shown as well),
even if a VM is not running currently? That would have definitively
saved me.

[Qubes]

'Crsi' via qubes-users wrote:
> > Do you often reboot your VMs? When Qubes creates a backup of a
> running VM it will do the backup using the VM state at startup. So if
> for example you started your VM (the ones with the outdated/missing
> data) 6 weeks ago, but since starting you added/edited/deleted data and
> you create a backup, without restarting the VM, the backup will not
> include those changes.
>
>
> Apparently, no. However, I made sure no VM was running (my host system
> crashed previously anyways). That means there's a difference between
> powering down a VM (e.g. qvm-shutdown) and having a powered down VM due
> to crash / power loss? Am I right that the backup tool uses the LV

Doing a gracefull shutdown or the VM crashing AFAIK is the same in terms
of the backup function as we have discussed. If your VM crashed the
state of the data on disk would still be the same compared to gracefully
shutting down the VM.

Do you still have access to the old system to re-do the backup?
Alternatively if you still have access to the old system you could
manually mount the private volume of the VM and copy the data across.

[Crsi]

On 8/14/22 18:09, Qubes wrote:
> Doing a gracefull shutdown or the VM crashing AFAIK is the same in
> terms of the backup function as we have discussed. If your VM crashed
> the state of the data on disk would still be the same compared to
> gracefully shutting down the VM.

My VM didn't crash -- the host system did. Therefore, I suspect the
volume `vm-<name>-private` was never updated with the content of
`vm-<name>-private-snap`, which probably occurs only on graceful VM
shutdown (or on VM crash, but then the host system cleans things up and
moves it correctly). If the host crashed, then this clean-up didn't take
place. What a bad situation, indeed.

>
> Do you still have access to the old system to re-do the backup?
> Alternatively if you still have access to the old system you could
> manually mount the private volume of the VM and copy the data across.

Sadly, I have no access to my old drive anymore as I pointed out earlier.

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Crsi:

> However, I made sure no VM was running (my host system
> crashed previously anyways). That means there's a difference between
> powering down a VM (e.g. qvm-shutdown) and having a powered down VM due to
> crash / power loss? Am I right that the backup tool uses the LV
> `vm-<name>-private` for backup only, but my data was in
> `vm-<name>-private-snap`?

Spot on. That was the situation in R4.0.

> Then, what about showing a warning that there exists a snap of a VM in the
> qubes-backup utility (at the same place where "The VM is running, backup
> will contain its state from before its start!" is shown as well), even if a
> VM is not running currently? That would have definitively saved me.

I know it doesn't help you anymore, but R4.1 avoids the whole problem
by cleanly stopping crashed volumes when the computer is restarted:

https://github.com/QubesOS/qubes-core-admin/pull/397

Rusty
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEhLWbz8YrEp/hsG0ERp149HqvKt8FAmL5IwNfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDg0
QjU5QkNGQzYyQjEyOUZFMUIwNkQwNDQ2OUQ3OEY0N0FBRjJBREYACgkQRp149Hqv
Kt+itA/+JGWiBR8lvKB3usGW96cI8ty4BePclBivLLxN6zeC0KpAxZK14QywIgOL
o0tguMM/H4Fc+F0ueRQymYCskzo/BWJlU06Fq9Bwp+re0p6Y6o0bzmx35e/mj8mb
CrMWqboqEjFihpxGxvk5BR8PceUTWvxFGNYGPZZQejNbanbkFeTINUXzMNR/dMjX
V0cIneqQtWcsJXBEBwS8KtMh+uHgumqgG+UcdymUuBU/FvwgrmOfOTSFrW/jgNEe
MImex5y9im/xrQ4RSDCJVc+FJNvq9HGGePFccrVo9tnJ3WY4khuYEckSe2wIL/5y
8TKHe5pQMTUSCbIqGNIFBWF5cFDw3zOPXPQGDx7HMruJNTtOJXa7RzO4qgdV6rcj
KMQeWVwKyZ2KyTYqJMY7M87c8IL9DFLmzZuGIQ/yTsw40uNoMdPpNTPgdGBTO0UB
hIqvf8DL9OBbUnXOLmgVtMLp4UjNxV+6Kf2k5Qk+NZxahMblUlezHXj3B0UeLmML
HcGW5N8ln+tUIcFHcrRNqZ6uf5CoWKZ47vNpedlLKHz2mXs2U6maFqg168A8tnAM
iL/bRAit1rctIyX9R/kSvcebetQRIrY7Sci4GjlNTwcCKpXehedm7D87jgA2gIS0
Sv4eXSUPPliDZiVGp0y1utywDZVdWzu9LyS6tUrG+hnaQCB4E3o=
=h1CV
-----END PGP SIGNATURE-----

[Andrew David Wong]

On 8/14/22 8:36 AM, 'Crsi' via qubes-users wrote:
> I also checked "verify backup" -- no error reported.

Just to clarify, the "verify only" option simulates a restore operation without actually writing any data from the backup to disk. This has two implications:

1. This option is not at all applicable when *creating* a backup (and should not even appear as an option there). [There's an open issue for a "create a backup, then verify it" feature, but it hasn't been implemented yet: https://github.com/QubesOS/qubes-issues/issues/1454]

2. Enabling this option when restoring will tell you that it was successful (if the simulation was successful), but no data on disk will change. This means, for example, that restoring from an older backup first, then restoring from a newer backup with "verify only" enabled, will result in the older VMs still being the ones on your disk.