extract file from image backup

[higgin...@gmail.com]

Not sure if what I want to achieve is possible but I'll try and summarise below.

With other distros that I run (WIN10, Ubuntu, Debian) as well as QUBES - I make full disk images using CLONEZILLA and retain them on a spare PC, running as a local server.

All full disk restores work fine, including QUBES OS - which is now my primary system.

Ideally I'd like to be able to have access to individual files within the image copies(if needed).

For DEBIAN, Ubuntu and I assume WIN10(though not really bothered about that) I can go into terminal on my backup PC and key something along the lines of

sudo cat /dir-to-images/sdb1.ext4-ptcl-img.gz.* | sudo gzip -d -c | sudo partclone.ext4 -C -r -W -s - -O /dir-to-new-image/hda1.img

where all the key files take the form sdb1.ext4-ptcl-img.gz.*
with * being aa,ab,ac etc.

As long as I've created a restore file first (eg hda1.img in the above example) the above code works fine - and if I mount the hda1.img file, then all my folders and files are accessible.

I can then recover say an individual file without having to do full disk restore etc.

IS THIS POSSIBLE WITH QUBES?

The relevant files take the form sdc1.ext4-ptcl-img.gz.aa (a single file) and lots of files of the form sdc2.dd-img.aa, sdc2.dd-img.ab through to sdc2.dd-img.bd.

The above instruction works with the sdc1 file and opens up various system files - but I can't do anything to read the sdc2 series of files.

Am not sure what to expect if I could get it to work since each VM has its own DOCUMENTS, DOWNLOAD folders etc.

Is what I want to achieve possible and if so - grateful for any suggestions as to relevant code needed.

If not - any suggestions about a different approach to a simple "system" backup with access to individual files as needed?

[donoban]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I think you could restore /var/lib/qubes/appvms/vm-name/private.img
and mount it as a loop device. Then extract the file you want.

However I don't see too much benefit with your method. Are you getting
your whole hard disk backup on /dir-to-new-image/hda1.img? If yes it's
not very useful for getting then one single file.

I think it's simpler and more efficient just restoring an standard
Qubes backup (only the VM's that you need), start them, move the files
you want outside and remove it.

With Qubes backup you have possibility to restore individual VM's,
integrity check of the backup, possibility of encryption, and also
store related configuration of the vm (firewall, netVM, etc...).
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEznLCgPSfWTT+LPrmFBMQ2OPtCKUFAlqMKAkACgkQFBMQ2OPt
CKXgRQ/9Hi8UFzT+vgZ2Zs0J6rZ12GkQYGwbTtzFMbBXeuUmPhr1TFhKXCAkLEIV
GxBOOJdUvBeEcDwfIje7kQ1x3WdEQnArFFsmahP/N/iboS6nSWfChFJ0wt6MHRgJ
h6jXn/VoRzNpc6/ONMda7ErvHaDvNA1MGKXIBTO136cfJBv7CoOVGx/22KbDbXy5
j80NqW3nmOodHZ4IPmSgAV7B4r2ATGS0ki8plMNg/fmzM7ipzbFSl9Vgk0kIGkJq
hN5OUEz3WBWQE7K1UWDo8O0qW14tJ1lhfu4wDGSEhg2JPM2VkTVxxEciB8D0+tew
PEVlehCZ/1VJ5LK8irbo0ZcPzhEkI+mVFr+RmwLVvqwnn+0jkw0tUlWrTEhpEK2c
JkI8rr+ib4409sRZyqrNSpCI8zQZzF8cFFC/dIhr66Q3t2QHDzlayEf+qPDEls2I
T/qqNILdDtx27r0T86ORRsf2NTTaMKchz1I9YujK730JRDrsIO70rW3bQLfWxlfO
6BZuqMhslSnwQ/TWjsAtFS7l9nWxqEJEbDnUxGc6v/JJShBYg+0Rw/Cs1U+4KLI/
KU4KMqjC186b9QacmduuyTKlhTyWeZJB18sosMDWRgJbCbhR6mzL1oK2vHqG7q1U
wTIQZWI28s6XP9QbtnsmZdXfSKiYiluWlOEgSMuz7PzB1zrb6O8=
=97+k
-----END PGP SIGNATURE-----

[Yuraeitha]

I'm not sure about extracting from the backup (it's probably possible), but you can do something else though.

The concern is full restore, right?

How familiar are you with the qvm-backup/qvm-backup-restore exclusion or inclusion methods? I.e. only backup or restore a single or handful of AppVM's without the rest of the system, etc.

Also are you familiar with qvm-backup/qvm-backup-restore profiles?

I know it's still overkill to restore a full AppVM, having to delete it all again after extracting the specific file(s), but if you're not doing this already, atleast this can get you halfway there.

For example if you have a work AppVM, you can make a profile for it and do frequent backups of just that one AppVM and nothing else, then it'll go pretty fast too in encryption/decryption, compared to a full system procedure.

It's not perfect, but maybe it's something?

[Yuraeitha]

Apologies, missed your post donoban. But looping the backup seems interesting, I suppose it must be possible with the decryption too.

[Bernhard]

> > Apologies, missed your post donoban. But looping the backup seems
interesting, I suppose it must be possible with the decryption too. >

Yes, it is. I backup by data that way since Q4  - the qubes-backup may
be more "handy", but I prefer knowing every single detail on encryption,
etc myself. You may mount a luks-container in sys-usb (for example), and
then attach one-by-one your app-vm private.img  to sys-usb using the
qubes widget; after mounting them (ro of course) you can simply rsync
your data, most conveniently to the backup volume. Your app-vm will not
be exposed to usb that way.

If you have a full dd take of your qubes system (I understood you inital
mail like that), be aware that the some image files are rather like "dd
disk images" rather than "dd partition images", which means you cannot
use the most straightforward mount on the loop device (you never mount a
disc, but a partition!). Instead, have to read the offset of the
partition start using fdisk or similar, and provide this offset to the
mount command. A quick google reveals the details on this procedure :)

Bernhard

[Yuraeitha]

That's a pretty cool and nice trick, I'm definitely gonna play with it when I get the time for it. I feel the same about such programs too, complex backup mechanisms are a single point of failure... I might just adapt your habit there, even though this isn't my thread/issue, I'll still say thanks for sharing.

Hopefully that solves the OP problem though, it seems like it's exactly what he's looking for by the looks of it.

[Chris Laprise]

There is not much difference between the manual method OP describes and
the way qvm-backup does it (automatically).

If you follow the emergency backup recovery docs, it lays out manual
steps for recovering Qubes data as img files which you can then mount:

https://www.qubes-os.org/doc/backup-restore/#emergency-backup-recovery-without-qubes


--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[higgin...@gmail.com]

Thanks for comments.

For info - have used Debian for around 10 years till I swapped to Qubes around 1 year ago. Windows before that. I use CLONEZILLA as it gives easy backup/restore for my old (Occasional use Os's) + QUBES + my wife's UBUNTU.
In extreme case of needing an individual old file (NOT NEEDED YET) - I could restore an old IMAGE to a spare HDD on chosen PC and then copy it from there. Not elegant but it would work.

Am retired and just experimenting the other day for general education - to see if I could get an individual file from an old image copy. As per above - no problem on non-QUBES. (TO answer one point above - I did restore full disk image to hda1.img and could then access all files - this was no real effort - enter an instruction, wait a couple of minutes and its all there).
Ideally for me there would be something simple for me with QUBES.

Have no problem backing up VM's to separate HDD on main PC and copying back as needed - this may be the way forward for me - but I'd still do the CLONEZILLA (BACKUP EVERYTHING) approach as I get a complete system if say my SSD fails.
Backing up everything and then being able to select an odd file if ever needed just seems a simpler concept to me!

My preference is to have "BACKUP" even of VM's on separate PC (as per my CLONEZILLA images) but though one can apparently use SSH to send files through to separate SERVER PC, I haven't yet managed to get this work - I'm OK when using SSH to copy from EG my DEBIAN system to SERVER - but the "right" SSH instruction within the QUBES BACKUP PROCESS seems to be eluding me!

The restore /var/lib/qubes and loopback approach may be a bit beyond me. Can see the relevant files for my live system but not sure about the data within my clonezilla image.

If this is feasible from CLONEZILLA image stored on separate internal server, I might see if I can try and understand it - will have a look another day. Think I'll go hill-walking tomorrow!

Thanks again for all comments.

[Bernhard]

Backing up everything and then being able to select an odd file if ever
needed just seems a simpler concept to me!
> My preference is to have "BACKUP" even of VM's on separate PC (as per my CLONEZILLA images) but though one can apparently use SSH to send files through to separate SERVER PC, I haven't yet managed to get this work - I'm OK when using SSH to copy from EG my DEBIAN system to SERVER - but the "right" SSH instruction within the QUBES BACKUP PROCESS seems to be eluding me!
>
>
>

For data transfer  over network, consider   rsync --rsh=ssh 
source-IP:source-path/      destination-IP:destination-path/   

defined SSH aliases are OK instead of IPs as well as dns-names.  This
needs a running SSH server on the distant machine, of course.

Bernhard 

[higgin...@gmail.com]

Cheers.

Have used rsync before from Debian PC to server PC - but again not sure how to apply within constraint of a VM backup process.

I'll have a play - after my day on the hills tomorrow.

Cheers

[MirrorWay]

Consider using borgbackup.

Borgbackup is bandwidth-efficient (only sends deltas), versioned (later backups don't overwrite earlier ones), compressed, deduplicated, and encrypted.

Here is an outline of a Qubes 4.0 setup:

BACKUP:
In dom0, borg reads the source VM's private block device and computes the delta (of the raw data) against all previous backups. The delta is then compressed, encrypted, and then streamed to a net-connected VM, which in turn ssh's the data to your backup server.

RESTORE:
In dom0, borg again uses the net-connected VM to mount a FUSE fs (in dom0) containing the block device you backed up. You then pass that block device to the source (or other) VM (via loop mount + qvm-block). Inside the VM, you mount the device read-only, and can now restore single files or rsync all files (restores differences!).

For example, a 20GB backup may result in a 200MB daily delta that takes 5 minutes to backup remotly.

SECURITY:
* At no point does decrypted data leave dom0. The encryption key stays in dom0. Neither the remote server, nor the net-connected proxying VM, are trusted.
* You never mount the backup in dom0, only in the source or other VM
* Since the backup is orchestrated by dom0, the source VM cannot delete its own backups.

IMPLEMENTATION NOTE:
The key to using borg from dom0 is to use BORG_RSH= to tunnel out, something like
BORG_RSH='qvm-run -p <borg_proxy_vm> ssh'
(This doesn't quite work since dom0's qvm-run doesn't pass extra args to the command it runs, but you can workaround it using a script).

[mes...@sindominio.net]

On Wednesday, 21 February 2018 05:41:15 UTC+1, MirrorWay wrote:
> Consider using borgbackup.
>
> Borgbackup is bandwidth-efficient (only sends deltas), versioned (later backups don't overwrite earlier ones), compressed, deduplicated, and encrypted.
>
> Here is an outline of a Qubes 4.0 setup:
>
> BACKUP:
> In dom0, borg reads the source VM's private block device and computes the delta (of the raw data) against all previous backups. The delta is then compressed, encrypted, and then streamed to a net-connected VM, which in turn ssh's the data to your backup server.
>

> For example, a 20GB backup may result in a 200MB daily delta that takes 5 minutes to backup remotly.

Thanks for the instructions. I'm trying to get borg working in qubes 4.0, as you describe.

If I understand correctly the idea is to pass borg to qvm-backup as a command. Looking into qvm-backup I see there is no option anymore to do unencrypted backups. So the output sent into the borg command will be already encrypted. If the backups are encrypted before arriving to borg I'm wondering how do you get any deltas calculated (encrypted data should be different from one run to the next one even if the original data is the same).

Is there any hidden way to force qvm-backup into do it unencrypted? (so I can rely on borg to encrypt the backups) or how do you do it in qubes 4.0?

meskio

[MirrorWay]

> Thanks for the instructions. I'm trying to get borg working in qubes 4.0, as you describe.
>
> If I understand correctly the idea is to pass borg to qvm-backup as a command.

No, qvm-backup is not involved.

Run borg --read-special on the LVM thin volume containing the private data.

For example, for the personal VM, the thin volume is called
/dev/mapper/qubes_dom0-vm--personal--private
or if the personal VM is running, then
/dev/mapper/qubes_dom0-vm--personal--private--snap

(You don't need to shutdown the VM before running borg, though you may want to `xl pause` it, or backup a snapshot.)

[Leo Gaspard]

Just to point a security-related downside of doing this: this requires
borg, a program running in dom0, to interact with a remote backup server.

Hence, a flaw in the borg source code may lead to dom0 being compromised.

In my opinion, ideally borg would be run inside an AppVM dedicated to it
that would be passed other VMs' disk in read-only mode. That's still a
lot more attack surface than `qvm-backup` (and important attack
surface, as compromising borg gives read access to all VMs' hard disks),
but is way better than the borg-in-dom0 solution.

Even better would be to first encrypt low-in-storage high-in-importance
disks before passing them through to the borg VM, this way it couldn't
deduplicate but couldn't read them either were it compromised. This
could be especially useful for a vault VM. Maybe it could even be
extended to encrypting all the AppVMs' disks (maybe with file-level
encryption for eg. photo storage VMs), and no encryption for TemplateVMs
(thus giving deduplication).

Any opinions about this idea?

[MirrorWay]

> Just to point a security-related downside of doing this: this requires
>
> borg, a program running in dom0, to interact with a remote backup server.
>
> Hence, a flaw in the borg source code may lead to dom0 being compromised.

I agree.

> Maybe it could even be
>
> extended to encrypting all the AppVMs' disks (maybe with file-level
>
> encryption for eg. photo storage VMs), and no encryption for TemplateVMs
>
> (thus giving deduplication).
>
> Any opinions about this idea?

I think if the qubes StorageVM is ever implemented (that sees the private volumes in encrypted form only), then running an incremental backup utility there would be ideal.

[MirrorWay]

Sorry, I meant per-VM encryption, not Storage VM.

If per-VM encryption (where each private volume is encrypted in its own volume) is ever implemented, I think secure backup could become very simple and elegant: use qvm-block to attach each encrypted private volume you want to backup to "sys-backup", and then run the incremental backup utility in that VM. sys-backup is net-connected but only ever sees each private volume in encrypted form.