Recover data from 'private-cow.img'

[Stickstoff]

Hello everyone,

I lost a somewhat important file from a software crash in an appvm.
Within the VM, I couldn't find a way to recover it. I copied the appvm
filesystem containers ('private.img', 'private-cow.img',
'private-cow.img.old').
As I understand it, 'private-cow.img' contains the old version of files
that were changed since last appvm startup.
However, '-cow.img' files contain no filesystem, but "binary patch"
data, thus can't be mounted or read directly or without their
corresponding'.img' files.
I found only little info online. Qubes documentation [1] says about
reverting template 'root-cow.img' changes:

> 1. Ensure that no other VMs uses this template.
> 2. Prepare snapshot device with root-cow.img.old instead of root-cow.img (/etc/xen/scripts/block-snapshot prepare).
> 3. Replace snapshot device-mapper target with snapshot-merge, other parameters (chunk size etc) remains untouched. Now kernel starts merging changes stored in root-cow.img.old into root.img. d-m device can be used normally (if needed).
> 4. Waits for merge completed: dmsetup status shows used snapshot blocks – it should be equal to metadata size when completed.
> 5. Replace snapshot-merge d-m target back to snapshot.
> 6. Cleanup snapshot device (if nobody uses it at the moment).
> 7. Move root-cow.img.old to root-cow.img (overriding existing file).

I tried to apply these steps to 'private.img' and 'private-cow.img', but
already failed with the second step, where I couldn't really figure out
how to use the '/etc/xen/scripts/block-snapshot' script.

Is there any hope to recover an older version of a changed/deleted file
from 'private-cow.img'?
Can anyone give me a pointer how to access its content, aka "apply its
content as a patch to 'private.ig'"?

Thank you in advance,

Stickstoff


[1] https://www.qubes-os.org/doc/template-implementation/

[David Hobach]

Use `qvm-volume revert` to go back to previous snapshots for a VM.

For the file pool you cited above there's just one previous revision available though. It should indeed be in the files you mentioned, but the other doc is somewhat unrelated.

If you start the VM, the old snaphot will be deleted, i.e. better create a backup before applying your changes.

[haaber]

> I lost a somewhat important file from a software crash in an appvm.

> However, '-cow.img' files contain no filesystem, but "binary patch"
> data, thus can't be mounted or read directly or without their
> corresponding'.img' files.

These are real disc-image files! There is a filesystem, but it is not in
sector 1 :) The trick is to mount it with an offset (see mount command).
To get the right offset, fdisk the file (it should have an old-style
MBR). If fdisk does not accept files (I forgot), try either cfdisk or
simple loop it in a device:

losetup /dev/loop42 imagefile.img
fdisk -p /dev/loop42
losetup -d /dev/loop42

This gives the starting sector of the partition, that is than handeld
over to mount as offset. And then you can grab data. Good luck!

[Stickstoff]

Thank you David and haaber for your help, it is much appreciated!

On 4/18/21 11:23 AM, David Hobach wrote:
> Use `qvm-volume revert` to go back to previous snapshots for a VM.

Oh, I saw that page [1] at some point, but didn't try it before. Thanks
for the reminder.

[dom0] qvm-volume info vmname:private
> Available revisions (for revert): old
looking good!

however:
[dom0] qvm-volume revert vmname:private old
> Got empty response from qubesd. See journalctl in dom0 for details.

Journal says:
> unhandled exception while calling src=b' dom0' meth=b' \
> admin.vm.volume.Revert' dest=b'vmname' arg=b' private' \
> len(untrusted_payload)=3
[..]
> NotImplementedError: Volume Filevolume has revert() not implemented

Both the revert command as well as the entry in journal are the same no
matter if I include or omit 'old' in the revert command.

On 4/18/21 11:25 AM, haaber wrote:
> These are real disc-image files! There is a filesystem, but it is not
> in sector 1 :)

Interesting and promising!
After way too long I figured out that the regular 'private.img' seems to
contain its filesystem beginning right at sector 1,
'private-cow.img.old' apparently as well, and 'private-cow.img' at
sector 560.
Testdisk could find them and correctly determine they are ext4
filesystems. But no matter if I loop-mount with offset, or deep-scan
with testdisk, or fsck, I can't find any data on the 'cow' images. The
regular 'private.img' is no problem here. The most I could get is
> /dev/xvdbc contains a DM_snapshot_cow file system
from fsck when letting it work on a rw copy of 'private-cow.img'.

Honestly, I don't understand why the last step fails so hard, after all
the files aren't corrupted, filesystems are detected, and the data
should just pop out by now?


I would be very grateful for hints :-)

Thank you,

Stickstoff


[1] https://www.qubes-os.org/doc/volume-backup-revert

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Stickstoff:

> [dom0] qvm-volume revert vmname:private old
> > Got empty response from qubesd. See journalctl in dom0 for details.
>
> Journal says:
> > unhandled exception while calling src=b' dom0' meth=b' \
> > admin.vm.volume.Revert' dest=b'vmname' arg=b' private' \
> > len(untrusted_payload)=3
> [..]
> > NotImplementedError: Volume Filevolume has revert() not implemented

The legacy 'file' storage driver just doesn't implement the required
functionality for 'qvm-volume revert' - one of the many reasons it
will be deprecated:

https://github.com/QubesOS/qubes-issues/issues/6399

> On 4/18/21 11:25 AM, haaber wrote:
> > These are real disc-image files! There is a filesystem, but it is not
> > in sector 1 :)

> After way too long I figured out that the regular 'private.img' seems to
> contain its filesystem beginning right at sector 1,
> 'private-cow.img.old' apparently as well, and 'private-cow.img' at
> sector 560.

private.img is a full disk image, but private-cow.img(.old) is more
like a patch as you said, and isn't mountable.

You could use this script (after backing up private.img and
private-cow.img.old):

https://github.com/rustybird/qubes-stuff/blob/master/dom0/bin/qvm-legacy-filevolume-revert

$ qvm-legacy-filevolume-revert vmname private

Rusty
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEhLWbz8YrEp/hsG0ERp149HqvKt8FAmB9t/hfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDg0
QjU5QkNGQzYyQjEyOUZFMUIwNkQwNDQ2OUQ3OEY0N0FBRjJBREYACgkQRp149Hqv
Kt/I5xAAmGRwSaNEzIt3ceEPbaJB6WVTUuqnwOUjRa5ds0/Fd0W7UWOc+kJVhdl+
QrshJn4kTRwf4N/DVuwhtzX146G+zJFf3PZegaTChXMoLCyxHg1zOsdSqaBUWILX
LUV7PFCKHTqhDX1XqoOYxhWPJpleYE3Gvzvv2IlqKXxFy8mP7guoMkNBjVoh5NVA
CBBKc7Z1MHk4+5OTJ8J3EFdrEXeODMzkUQw0GNs048MyI+Zig3z6r0t78h2JNFwY
Hb+BPPVZ3YdzVcjLu4FegEdy9EvTwdQ/z+K87DupKLMu5FNT9GRbvgHzdLNxq9je
6GOVrSLwkvPYo86LBaVE+9b73mX5oc8OKaLAK1dv3LB2f756IP4wU4dgFVmjkkjU
jlaHSqXFv3S6R44u31iVlDzDNclqcuZwvhyFkeAn//JG9E0Mh+er80N8qBLP3gMj
QMYFk8kWakpI56ceR8rQ9dignsWkBdd6+PBGAilHVX22Mx2C+2sfw0cu6QLRMstm
DzocEMwY0sEI2cZJZpcODToB4MhzWcPB1me5s2/etkAgCNWqodR/HHZ0T+j5ZXH5
20P0GMa2YpQFvIZ7uZxbj96aQfyBtPv7a+EkEBYK0f5zNPFst6KLVvVPiyTF6IT5
CRLyyXOAHmAxMCgGysyet9ZsshuMCeMbh8kJp5rBMLbPYcgW1Hw=
=N9oy
-----END PGP SIGNATURE-----

[haaber]

> The legacy 'file' storage driver just doesn't implement the required
> functionality for 'qvm-volume revert' - one of the many reasons it
> will be deprecated:
>
> https://github.com/QubesOS/qubes-issues/issues/6399
>

Awesome! Thank yu for that hint. When/how will it be changed ?? That
seems quite troublesome to change the internal storage type within a
running system ... will need re-install in some further time??