Feedback request: Incremental file-based backup PoC
247 views
Skip to first unread message
Vít Šesták
Feb 26, 2017, 5:07:50 PM2/26/17
to qubes-users
Hello,
I'd like to announce my PoC for incremental file-based backup for Qubes VMs. It is not a completely new backup system, it just wires together some existing software with Qubes principles to provide convenient and secure way to backup many VMs with minimum overhead and encrypt them by one password. I have tried hard not to expose dom0 to untrusted inputs from VMs. Design is described in README, though I will probably break README to multiple parts. Sources are here:
https://github.com/v6ak/qubes-incremental-backup-poc/
Now, I'd like to collect feedback on this. Whether you want to try it or just discuss or review the design without trying, you are welcome. Even if you think that the design is a complete crap, you are welcome: You might have a good point that I would like to know in this early stage.
Short QA:
Q: Why should I prefer this way to current Qubes backup mechanism?
A:
* Efficiency. Incremental backups can lower the backup size dramatically. File-based backups skip unused space, so it is even more efficient. And last but not least, file-based backup allows you to exclude ~/.cache and other stuff you don't want to backup.
* Comfort: The builtin backup does not handle storage. My approach allows you to automatically upload it to various storages.
Q: How much is it stable?
A: Well, I just got it working. Don't rely on it too much, please. The backup format also might change a bit, so using it in production might require reviewing commit messages.
Q: How much is it secure?
A: There was nontrivial effort to design it securely and implement it securely. In both areas are some improvable places (e.g., non-processing UTF-8 in dom0 or better authentication of files), but I hope it is reasonably secure. There are some potential minor leaks, mostly through error messages.
Q: I don't like Duplicity for some reason. Can I use something else instead?
A: Well, the code is modular enough, so that implementing some other backup backend should not be hard. On the other hand, no other current backend is implemented at the moment. If you have some suggestion, you are welcome. If you want to write some other backend, you are also welcome, but maybe you will want to wait a while until things are more stable.
Q: What backup backends are suitable for this scheme?
A: The most crucial part is splitting the backup process into two parts. One process runs in a DVM, but it does not have the access to the backup storage. Second process runs in a BackupStorageVM, but it does not have access to the filesystem you want to backup. Those two processes are connected through qrexec.
Q: Can I backup VM that is running?
A: If you have LVM-based private.img, you can. (See README for details.) If you have standard file-based private.img, you cannot. I don't plan to implement this feature for file-based private.img, because Qubes 4.0 will use LVM, so they don't seem to have a long future.
Q: What is the code quality?
A: Short answer: PoC. Long answer: Some parts look like a Bash script rewritten to Python (which is in few cases how it actually has happened), but I believe I can iteratively improve the suboptimal cases. Especially exception handling is something neglected – I am aware of Exceptions and the code expects them to some degree (using finally blocks etc.), but they are rarely reported to user in a friendly way. They usually bubble to the top level, so stacktrace is dumped.
Regards,
Vít Šesták
I'd like to announce my PoC for incremental file-based backup for Qubes VMs. It is not a completely new backup system, it just wires together some existing software with Qubes principles to provide convenient and secure way to backup many VMs with minimum overhead and encrypt them by one password. I have tried hard not to expose dom0 to untrusted inputs from VMs. Design is described in README, though I will probably break README to multiple parts. Sources are here:
https://github.com/v6ak/qubes-incremental-backup-poc/
Now, I'd like to collect feedback on this. Whether you want to try it or just discuss or review the design without trying, you are welcome. Even if you think that the design is a complete crap, you are welcome: You might have a good point that I would like to know in this early stage.
Short QA:
Q: Why should I prefer this way to current Qubes backup mechanism?
A:
* Efficiency. Incremental backups can lower the backup size dramatically. File-based backups skip unused space, so it is even more efficient. And last but not least, file-based backup allows you to exclude ~/.cache and other stuff you don't want to backup.
* Comfort: The builtin backup does not handle storage. My approach allows you to automatically upload it to various storages.
Q: How much is it stable?
A: Well, I just got it working. Don't rely on it too much, please. The backup format also might change a bit, so using it in production might require reviewing commit messages.
Q: How much is it secure?
A: There was nontrivial effort to design it securely and implement it securely. In both areas are some improvable places (e.g., non-processing UTF-8 in dom0 or better authentication of files), but I hope it is reasonably secure. There are some potential minor leaks, mostly through error messages.
Q: I don't like Duplicity for some reason. Can I use something else instead?
A: Well, the code is modular enough, so that implementing some other backup backend should not be hard. On the other hand, no other current backend is implemented at the moment. If you have some suggestion, you are welcome. If you want to write some other backend, you are also welcome, but maybe you will want to wait a while until things are more stable.
Q: What backup backends are suitable for this scheme?
A: The most crucial part is splitting the backup process into two parts. One process runs in a DVM, but it does not have the access to the backup storage. Second process runs in a BackupStorageVM, but it does not have access to the filesystem you want to backup. Those two processes are connected through qrexec.
Q: Can I backup VM that is running?
A: If you have LVM-based private.img, you can. (See README for details.) If you have standard file-based private.img, you cannot. I don't plan to implement this feature for file-based private.img, because Qubes 4.0 will use LVM, so they don't seem to have a long future.
Q: What is the code quality?
A: Short answer: PoC. Long answer: Some parts look like a Bash script rewritten to Python (which is in few cases how it actually has happened), but I believe I can iteratively improve the suboptimal cases. Especially exception handling is something neglected – I am aware of Exceptions and the code expects them to some degree (using finally blocks etc.), but they are rarely reported to user in a friendly way. They usually bubble to the top level, so stacktrace is dumped.
Regards,
Vít Šesták
Chris Laprise
Feb 26, 2017, 10:42:19 PM2/26/17
to Vít Šesták, qubes-users
On 02/26/2017 05:07 PM, Vít Šesták wrote:
> Hello,
> I'd like to announce my PoC for incremental file-based backup for Qubes VMs. It is not a completely new backup system, it just wires together some existing software with Qubes principles to provide convenient and secure way to backup many VMs with minimum overhead and encrypt them by one password. I have tried hard not to expose dom0 to untrusted inputs from VMs. Design is described in README, though I will probably break README to multiple parts. Sources are here:
>
> https://github.com/v6ak/qubes-incremental-backup-poc/
>
> Now, I'd like to collect feedback on this. Whether you want to try it or just discuss or review the design without trying, you are welcome. Even if you think that the design is a complete crap, you are welcome: You might have a good point that I would like to know in this early stage.
>
> Short QA:
Thanks, Vit. Qubes needs a way to do incremental backups. I have been
> Hello,
> I'd like to announce my PoC for incremental file-based backup for Qubes VMs. It is not a completely new backup system, it just wires together some existing software with Qubes principles to provide convenient and secure way to backup many VMs with minimum overhead and encrypt them by one password. I have tried hard not to expose dom0 to untrusted inputs from VMs. Design is described in README, though I will probably break README to multiple parts. Sources are here:
>
> https://github.com/v6ak/qubes-incremental-backup-poc/
>
> Now, I'd like to collect feedback on this. Whether you want to try it or just discuss or review the design without trying, you are welcome. Even if you think that the design is a complete crap, you are welcome: You might have a good point that I would like to know in this early stage.
>
> Short QA:
trying out btrfs-send. Marek expressed interest in a technique using LVM
thin-provisioning, possibly for R4.
BTW, the place in the Readme where it says "Attacker can obtain some
reasonably limited metadata about the backup"... this doesn't sound
correct to my ears. It almost sounds like you're trying to supply data
to the attacker, which I know is not the case. You may want to re-phrase.
In the list of limitations, you could mention that the backups cannot be
pruned (old backup sessions deleted independently of the others) since
that is a feature some users look for. Another downside you may want to
list is that duplicity (librsync) uses significant processing power to
compare old contents with new.
Chris
Vít Šesták
Feb 27, 2017, 2:58:21 AM2/27/17
to qubes-users
> It almost sounds like you're trying to supply data to the attacker
Good point. I'll rephrase it. Maybe the following is more reasonable:
* Attacker having access to your storage can learn how many VMs you have and limited metadata for each VM (see below). They however cannot learn VM names, lenghts of VM names (thanks to padding to constant length) or similarities in VM names like common prefix (thanks to synthetic IV).
* If attacker with access to your backup storage can store significant amount of data to one VM, she will be able to determine what files belong to the particular VM.
* Amount of metadata attacker can see for one VM depends on the backup backend. With Duplicity, they include backup timestamps and GPG-related metadata.
Ad backups cannot be pruned: You are right, but since even restore is not yet implemented, I don't see much point of mentioning prune in limitations. And prune seems to be much easier to implement.
Ad performance: This does not correspond to my experience, but you are not the first complaining about Duplicity performance, so I'll mention it.
Regards,
Vít Šesták 'v6ak'
Chris Laprise
Feb 27, 2017, 7:51:58 AM2/27/17
to Vít Šesták, qubes-users
On 02/27/2017 02:58 AM, Vít Šesták wrote:
>
> Ad backups cannot be pruned: You are right, but since even restore is not yet implemented, I don't see much point of mentioning prune in limitations. And prune seems to be much easier to implement.
I was referring to the ability to delete any backups (older than a given
>
> Ad backups cannot be pruned: You are right, but since even restore is not yet implemented, I don't see much point of mentioning prune in limitations. And prune seems to be much easier to implement.
date, for example) without ruining the set. IIRC, duplicity is incapable
of doing this since it produces increments that are dependent on the
prior "full backup" (the only way around this is frequent full backups).
In some backup systems, each backup appears complete to the system, even
though it was created with incremental deltas. A benefit of this is you
can delete any backup in the set to reclaim space, without affecting any
of the remaining backups---a single backup set can be updated in
rotation forever. Its akin to block-level deduplication. Another benefit
is that all backups except for the first will transmit only deltas; data
that never changes never gets re-transmitted.
Chris
Manuel Amador (Rudd-O)
Feb 27, 2017, 8:52:54 AM2/27/17
to Chris Laprise, Vít Šesták, qubes-users
On 02/27/2017 12:51 PM, Chris Laprise wrote:
>
> In some backup systems, each backup appears complete to the system,
> even though it was created with incremental deltas. A benefit of this
> is you can delete any backup in the set to reclaim space, without
> affecting any of the remaining backups---a single backup set can be
> updated in rotation forever. Its akin to block-level deduplication.
> Another benefit is that all backups except for the first will transmit
> only deltas; data that never changes never gets re-transmitted.
>
> Chris
>
Duplicati does this. It's really well-designed and does exactly what
>
> In some backup systems, each backup appears complete to the system,
> even though it was created with incremental deltas. A benefit of this
> is you can delete any backup in the set to reclaim space, without
> affecting any of the remaining backups---a single backup set can be
> updated in rotation forever. Its akin to block-level deduplication.
> Another benefit is that all backups except for the first will transmit
> only deltas; data that never changes never gets re-transmitted.
>
> Chris
>
you want. It also does encryption as well.
If this PoC is sound (it appears to be) then Duplicati can be repurposed
to substitute for Duplicity altogether.
File-based backup really does sound like the right thing to do, for so
many reasons. It's really dumb to back up big, big files, even if they
could be done incrementally (as I do with my own Duplicity-based
solution for dom0).
--
Rudd-O
http://rudd-o.com/
Manuel Amador (Rudd-O)
Feb 27, 2017, 8:54:09 AM2/27/17
to qubes...@googlegroups.com
On 02/26/2017 10:07 PM, Vít Šesták wrote:
> Q: Can I backup VM that is running?
> A: If you have LVM-based private.img, you can. (See README for details.) If you have standard file-based private.img, you cannot. I don't plan to implement this feature for file-based private.img, because Qubes 4.0 will use LVM, so they don't seem to have a long future.
I am terrified that Qubes 4.0 will force me to use LVM, because LVM is
> Q: Can I backup VM that is running?
> A: If you have LVM-based private.img, you can. (See README for details.) If you have standard file-based private.img, you cannot. I don't plan to implement this feature for file-based private.img, because Qubes 4.0 will use LVM, so they don't seem to have a long future.
frankly a disaster for data recovery and for data integrity as well. I
would be willing to write an adapter to use ZFS zvols instead. That
would be the right thing to do.
--
Rudd-O
http://rudd-o.com/
Manuel Amador (Rudd-O)
Feb 27, 2017, 8:59:46 AM2/27/17
to qubes...@googlegroups.com
On 02/27/2017 01:53 PM, Manuel Amador (Rudd-O) wrote:
>
> I am terrified that Qubes 4.0 will force me to use LVM, because LVM is
> frankly a disaster for data recovery and for data integrity as well. I
> would be willing to write an adapter to use ZFS zvols instead. That
> would be the right thing to do.
>
Honestly, ZFS on Linux should be integrated into the Qubes OS installer,
>
> I am terrified that Qubes 4.0 will force me to use LVM, because LVM is
> frankly a disaster for data recovery and for data integrity as well. I
> would be willing to write an adapter to use ZFS zvols instead. That
> would be the right thing to do.
>
and the system should ship and be installed with it. But if that's too
much work, then btrfs should be the default.
Inferior volume management systems with no data integrity protection
should not be used these days.
--
Rudd-O
http://rudd-o.com/
Chris Laprise
Feb 27, 2017, 6:39:02 PM2/27/17
to Manuel Amador (Rudd-O), Vít Šesták, qubes-users
Restic's CDF, and Time Machine's sparsebundles.
TM has one big advantage, though: File contents in the source do not
have to be scanned to detect deltas (with big files, that's a big deal).
Chris
Vít Šesták
Mar 1, 2017, 1:49:44 AM3/1/17
to qubes-users
Hello,
I've implemented some changes, most notably initial support for restores. I haven't updated the README yet.
The CLI is still subject of change, I believe it should be refactored.
I am tracking many ideas in Github issues. There are two notable groups:
* Help-wanted (currently two issues): https://github.com/v6ak/qubes-incremental-backup-poc/labels/help%20wanted
* [WIP] Goals – long-term meta-tasks that show visions. They will link related issues. https://github.com/v6ak/qubes-incremental-backup-poc/labels/goal
Now, I will respond to some comments:
Ad leaking data to the attacker: Hmm, it is not in limitations, it is in goals. So, I'll have to rephrase it differently. But in goals, I definitely agree that it sounds weird. It sounds like it was a goal to leak the data, not to limit the leaks to something reasonable.
Ad pruning old backups: Good point, I'll add it. AFAIK, this is always matter of some tradeoff.
Ad Duplicati: I've taken a brief look at it. The backup design looks good. What I am somewhat worried about is encryption:
By default, it uses AESCrypt format, which don't seem to be well-known. I am always skeptical when reading that something uses strong AES-256 encryption without further details – I tend to imaginate ECB or fixed-IV CTR without any authentication, maybe you know why… It is hard to find any details on AESCrypt, even the format description is not very verbose – it does not even mention mode of operation. I felt like reverse engineer when I finally found “CBC” in the source code. And I still don't know important details about authentication. I know there are two HMACs, but I don't know if it is mac-then-encrypt, mac-and-encrypt or encrypt-and-mac. Also, list of vulnerabilities (not very visible) does not suggest a good design: https://www.aescrypt.com/wishlist.html . And there is more to review, e.g. what implementation of AES is used and if it has any side channels. In general, I would like to avoid AESCrypt format at all.
Duplicati can also use GPG. Sure, even GPG can be used wrongly (most obvious: allowing to interchange various backup files within one VM – not much practical if they are large blocks), but I feel better with GPG that with AESCrypt. Unfortunately, I see it only with use of asymmetric encryption, which is not what I would want – not only for encryption strength against quantum computers, but also for implementation hassle – this would probably require generating and storing some extra keypair per VM (barring the fact it would be rather pair of pairs – for authentication and encryption). But maybe Duplicati can be somehow convinced to use GPG with symmetric encryption.
There is one more concern with Duplicati: How would we split it between BDVM (backup DVM) and BackupStorageVM? Is this possible without recompilation of the whole Duplicati? In Duplicity, I add a backup backend to BDVM that forwards everything to another backend in BackupStorageVM. So, BDVM loads dynamically my backend and qrexec handler in BackupStorageVM loads the desired Duplicity backend. I am almost sure this is somehow possible with Duplicati, but I am unsure about the effort required for this.
Do you have any ideas about mentioned implementation concerns related do Dupicati?
A side note on potential backup-block-level malleability: Not only this is a minor concern today, it should become non-concern when Merkle-tree-based whole-backup authentication is implemented.
On BTRFS/ZFS/LVM I don't know much about differences between them when used as pools for block devices. You might have a point, but discuss it in a separate thread, please. A note relevant for backup: It should be easy to add support for ZFS/BTRFS, I believe. You can see the current implementation for LVM:
https://github.com/v6ak/qubes-incremental-backup-poc/blob/f9291b3fbec459cbc2f44279f7a31a3613a49811/qubesvmtools.py#L29-L38L86-L109
https://github.com/v6ak/qubes-incremental-backup-poc/blob/f9291b3fbec459cbc2f44279f7a31a3613a49811/qubesvmtools.py#L29-L38L86-L109
Regards,
Vít Šesták 'v6ak'
I've implemented some changes, most notably initial support for restores. I haven't updated the README yet.
The CLI is still subject of change, I believe it should be refactored.
I am tracking many ideas in Github issues. There are two notable groups:
* Help-wanted (currently two issues): https://github.com/v6ak/qubes-incremental-backup-poc/labels/help%20wanted
* [WIP] Goals – long-term meta-tasks that show visions. They will link related issues. https://github.com/v6ak/qubes-incremental-backup-poc/labels/goal
Now, I will respond to some comments:
Ad leaking data to the attacker: Hmm, it is not in limitations, it is in goals. So, I'll have to rephrase it differently. But in goals, I definitely agree that it sounds weird. It sounds like it was a goal to leak the data, not to limit the leaks to something reasonable.
Ad pruning old backups: Good point, I'll add it. AFAIK, this is always matter of some tradeoff.
Ad Duplicati: I've taken a brief look at it. The backup design looks good. What I am somewhat worried about is encryption:
By default, it uses AESCrypt format, which don't seem to be well-known. I am always skeptical when reading that something uses strong AES-256 encryption without further details – I tend to imaginate ECB or fixed-IV CTR without any authentication, maybe you know why… It is hard to find any details on AESCrypt, even the format description is not very verbose – it does not even mention mode of operation. I felt like reverse engineer when I finally found “CBC” in the source code. And I still don't know important details about authentication. I know there are two HMACs, but I don't know if it is mac-then-encrypt, mac-and-encrypt or encrypt-and-mac. Also, list of vulnerabilities (not very visible) does not suggest a good design: https://www.aescrypt.com/wishlist.html . And there is more to review, e.g. what implementation of AES is used and if it has any side channels. In general, I would like to avoid AESCrypt format at all.
Duplicati can also use GPG. Sure, even GPG can be used wrongly (most obvious: allowing to interchange various backup files within one VM – not much practical if they are large blocks), but I feel better with GPG that with AESCrypt. Unfortunately, I see it only with use of asymmetric encryption, which is not what I would want – not only for encryption strength against quantum computers, but also for implementation hassle – this would probably require generating and storing some extra keypair per VM (barring the fact it would be rather pair of pairs – for authentication and encryption). But maybe Duplicati can be somehow convinced to use GPG with symmetric encryption.
There is one more concern with Duplicati: How would we split it between BDVM (backup DVM) and BackupStorageVM? Is this possible without recompilation of the whole Duplicati? In Duplicity, I add a backup backend to BDVM that forwards everything to another backend in BackupStorageVM. So, BDVM loads dynamically my backend and qrexec handler in BackupStorageVM loads the desired Duplicity backend. I am almost sure this is somehow possible with Duplicati, but I am unsure about the effort required for this.
Do you have any ideas about mentioned implementation concerns related do Dupicati?
A side note on potential backup-block-level malleability: Not only this is a minor concern today, it should become non-concern when Merkle-tree-based whole-backup authentication is implemented.
On BTRFS/ZFS/LVM I don't know much about differences between them when used as pools for block devices. You might have a point, but discuss it in a separate thread, please. A note relevant for backup: It should be easy to add support for ZFS/BTRFS, I believe. You can see the current implementation for LVM:
https://github.com/v6ak/qubes-incremental-backup-poc/blob/f9291b3fbec459cbc2f44279f7a31a3613a49811/qubesvmtools.py#L29-L38L86-L109
https://github.com/v6ak/qubes-incremental-backup-poc/blob/f9291b3fbec459cbc2f44279f7a31a3613a49811/qubesvmtools.py#L29-L38L86-L109
Regards,
Vít Šesták 'v6ak'
Marek Marczykowski-Górecki
Mar 2, 2017, 3:07:05 PM3/2/17
to Manuel Amador (Rudd-O), qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
See for example here[1] why btrfs is not a good idea as a default
filesystem. But we already have some support for it (qvm-clone use cp
- --reflink=auto). And same should apply to storage layer in Qubes 4.0
too. So, if you want to use it, go ahead. Just few issues needs to be
fixes, like this one:
https://github.com/QubesOS/qubes-issues/issues/1871
[1] https://github.com/QubesOS/qubes-issues/issues/2340#issuecomment-263562644
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJYuHtjAAoJENuP0xzK19csSnIH/2BDWXF+N2MiselU1qPK6OW5
YDpjbulfQFk5edqw9Tr9JOjgEQxVx0hgtoeGwZ4iE5MjUMY0ptcGa6w2OfA7FhZE
sixhqgWPBjLgmALwAjBqMR7SkCA8/xtKTF3I1ArkEfc5bxBiSKNxYs0xuqmG6gZz
ZYiaH7xHnuSaRYYlR+D0YMOniX/JmoDA5yuH6si4VhhvTNE/S3UKDbpmyF32Pp+z
VFESy1u6aLoTXk2W5A8+mi9rR/TpyWGXL9pWM2L1FCVNR+IhlsTqn1af2v9BbOnb
bhbAL/pr2c1yY5V804dqIrfl3LlR3ELyeaAMm7j5D6mBRGq3kX12NuvYH2IGzW8=
=JE3h
-----END PGP SIGNATURE-----
Hash: SHA256
filesystem. But we already have some support for it (qvm-clone use cp
- --reflink=auto). And same should apply to storage layer in Qubes 4.0
too. So, if you want to use it, go ahead. Just few issues needs to be
fixes, like this one:
https://github.com/QubesOS/qubes-issues/issues/1871
[1] https://github.com/QubesOS/qubes-issues/issues/2340#issuecomment-263562644
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJYuHtjAAoJENuP0xzK19csSnIH/2BDWXF+N2MiselU1qPK6OW5
YDpjbulfQFk5edqw9Tr9JOjgEQxVx0hgtoeGwZ4iE5MjUMY0ptcGa6w2OfA7FhZE
sixhqgWPBjLgmALwAjBqMR7SkCA8/xtKTF3I1ArkEfc5bxBiSKNxYs0xuqmG6gZz
ZYiaH7xHnuSaRYYlR+D0YMOniX/JmoDA5yuH6si4VhhvTNE/S3UKDbpmyF32Pp+z
VFESy1u6aLoTXk2W5A8+mi9rR/TpyWGXL9pWM2L1FCVNR+IhlsTqn1af2v9BbOnb
bhbAL/pr2c1yY5V804dqIrfl3LlR3ELyeaAMm7j5D6mBRGq3kX12NuvYH2IGzW8=
=JE3h
-----END PGP SIGNATURE-----
Marek Marczykowski-Górecki
Mar 2, 2017, 3:21:02 PM3/2/17
to Vít Šesták, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hash: SHA256
On Tue, Feb 28, 2017 at 10:49:43PM -0800, Vít Šesták wrote:
> Hello,
>
> I've implemented some changes, most notably initial support for restores. I haven't updated the README yet.
How do you handle restore? File-based backup was considered some time
> Hello,
>
> I've implemented some changes, most notably initial support for restores. I haven't updated the README yet.
ago, here:
https://groups.google.com/d/msg/qubes-devel/Gcrb7KQVcMk/CK-saQU_1HYJ
And restore process is a major problem there - you need some VM (with
appropriate trust) to extract the data into filesystem. While this can
be achieved for some template-based VMs (as long as you have those
templates), it gets really tricky for standalone VMs. And even more
tricky for non-Linux VMs. Especially when you want to have backup as
accurate as possible, including all file attributes etc (think of
restoring Windows system partition).
As for trusting particular DispVM - in Qubes 4.0 you can create DispVM
out of any AppVM, which means a backup for some VM may be handled by
DispVM based on this very VM - which should guarantee the same level of
trust. No need to select "most trusted DispVM" (until you think of
dom0...).
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
MgYKFmxh7HW0iGeA1ckvbSmc7qcA2nhhG9cOWNfp5qzNVyIb1KPO3w3RhUwyQvOd
Mj+3YlOQ9dHbLw1MHazXSndf3MNV5XXlyCCrjwRamNCVhqXwXVD6oZQyV3kAKXHc
p1jV0Gg7UfDEo809mn/O4+e1orim69fQCck4CbCqs7YkwvkIsx9uPRqHfRf1lpPh
85D1XmsTrM7Qe2CgREHC5xRqGk8rjh/V34sg9s5OAxoz2a8KGCcsTksYidTELfwv
+x6DiUnYeJYGrbDx6FfdzVc+o98040551LZjwvb9kP4qayYC03d4+5aBiCsao14=
=cbmM
-----END PGP SIGNATURE-----
Vít Šesták
Mar 3, 2017, 3:42:56 PM3/3/17
to qubes-users
Well, I don't see any problem with restore. This is roughly what dom0 does (or initiates):
1. create a new VM (with default or explicitly provided parameters and no netvm – as I have mentioned, I don't backup anything but files and VM name)
2. start a new DVM (we have chosen to trust the DVM…)
3. attach the private.img to that DVM and mount it
4. add permissions for the DVM (BackupStorageVM allows the DVM accessing only the specified subfolder of backup) – well, it currently gets RW access, which is suboptimal (R access should be enough), but I don't se a real threat here.
5. start the backup script in the DVM (The script deletes everything but lost+found on the attached FS and starts restore.)
Do you think I miss something important?
Ad DVMs in QubesOS 4: Even cooler than I thought. My idea was to use the same DVM (maybe I am influenced with the current Qubes) in order to be sure we have Duplicity (or another software) of the version we want. But you have a point on the trust.
This is even more interesting and it could be an important design shift. Maybe a template could know how to backup related AppVM's data. This is not a minor change, as it would allow using different backup backends for different VMs. And it is not so trivial, because we probably would want to use just one storage backend for all backups. Duplicity has a very simple API for backends (you have to implement just get/put/list/delete, it does not even have subfolders), but I am not sure about other backends.
Anyway, the storage backends are going to be a challenge, at least a bit. With Duplicity not supporting subfolders, I rely on some features of the storage backends (subfolder-compliant URI scheme and automatic subfolder creation). This works with file backend (which is fortunately somewhat universal thanks to various fuse-based filesystems). However, with rsync, I had to implement a hack that violates encapsulation. (See https://github.com/v6ak/qubes-incremental-backup-poc/commit/83a285307042eb9363b5ed77e9701dcfa7ab3514 .) And I currently have no idea about other backends. Maybe the storage backend in BackupStorageVM can be quite generic (i.e., agnostic to backup backend), but maybe we can be badly suprised…
TemplateVMs: I haven't published my idea, so I'll do it now: Backup list of explicitly installed packages and config files. I know, this is going to be package-manager-specific. There is not much a reason for storing backup of packages I can download.
Ad StandaloneVMs: I agree it is tricky here. Currently, I backup/restore just private.img of AppVM. We definitely can't perform fully OS-agnostic file-based backup without cooperation of the OS (that is not yet installed). There are several ways how to handle it:
a. Backup just private.img. I would be usually OK with this, because my StandaloneVMs don't have a large lists of packages, they are usually rather default installation with few additional packages installed.
b. Handle root.img like TemplateVM and private.img like template-based VM.
c. Handle root.img by block-based backup and private.img by file-based backup.
Ad attributes: Good point. I honestly don't know what Duplicity supports. I am sure it has some basic support, so I concluded it is good enough for me, but maybe I should review its limitations again.
Ad Windows: currently, it can theoretically mount private.img provided that there is ntfs module etc., so it should be possible to perform backup. I am not sure about attributes supported. The restore process for Windows will currently not work out-of-box without any further tuning.
Thank you for your notes. Maybe we should follow Pareto rule. Maybe it should not be a goal to use file-based backup on everything. We can use file-based backup in most cases and use the current block-based backup in other cases. Especially when we integrate both backup types together – we could consider the current block-based backup as just another backup backend.
My priorities are roughly:
1. Have good file-based backup and restore process for Linux template-based AppVMs.
2. Have similar process for Debian/Fedora TemplateVMs.
3. Have block-based backup backend.
4. Have good support for StandaloneVMs, Windows and other OSes.
I believe this corresponds not only to my needs, but also to needs of many others. I am generally not against implementing something that has lower priority for me, especially if there is a pull request. ☺
Regards,
Vít Šesták 'v6ak'
1. create a new VM (with default or explicitly provided parameters and no netvm – as I have mentioned, I don't backup anything but files and VM name)
2. start a new DVM (we have chosen to trust the DVM…)
3. attach the private.img to that DVM and mount it
4. add permissions for the DVM (BackupStorageVM allows the DVM accessing only the specified subfolder of backup) – well, it currently gets RW access, which is suboptimal (R access should be enough), but I don't se a real threat here.
5. start the backup script in the DVM (The script deletes everything but lost+found on the attached FS and starts restore.)
Do you think I miss something important?
Ad DVMs in QubesOS 4: Even cooler than I thought. My idea was to use the same DVM (maybe I am influenced with the current Qubes) in order to be sure we have Duplicity (or another software) of the version we want. But you have a point on the trust.
This is even more interesting and it could be an important design shift. Maybe a template could know how to backup related AppVM's data. This is not a minor change, as it would allow using different backup backends for different VMs. And it is not so trivial, because we probably would want to use just one storage backend for all backups. Duplicity has a very simple API for backends (you have to implement just get/put/list/delete, it does not even have subfolders), but I am not sure about other backends.
Anyway, the storage backends are going to be a challenge, at least a bit. With Duplicity not supporting subfolders, I rely on some features of the storage backends (subfolder-compliant URI scheme and automatic subfolder creation). This works with file backend (which is fortunately somewhat universal thanks to various fuse-based filesystems). However, with rsync, I had to implement a hack that violates encapsulation. (See https://github.com/v6ak/qubes-incremental-backup-poc/commit/83a285307042eb9363b5ed77e9701dcfa7ab3514 .) And I currently have no idea about other backends. Maybe the storage backend in BackupStorageVM can be quite generic (i.e., agnostic to backup backend), but maybe we can be badly suprised…
TemplateVMs: I haven't published my idea, so I'll do it now: Backup list of explicitly installed packages and config files. I know, this is going to be package-manager-specific. There is not much a reason for storing backup of packages I can download.
Ad StandaloneVMs: I agree it is tricky here. Currently, I backup/restore just private.img of AppVM. We definitely can't perform fully OS-agnostic file-based backup without cooperation of the OS (that is not yet installed). There are several ways how to handle it:
a. Backup just private.img. I would be usually OK with this, because my StandaloneVMs don't have a large lists of packages, they are usually rather default installation with few additional packages installed.
b. Handle root.img like TemplateVM and private.img like template-based VM.
c. Handle root.img by block-based backup and private.img by file-based backup.
Ad attributes: Good point. I honestly don't know what Duplicity supports. I am sure it has some basic support, so I concluded it is good enough for me, but maybe I should review its limitations again.
Ad Windows: currently, it can theoretically mount private.img provided that there is ntfs module etc., so it should be possible to perform backup. I am not sure about attributes supported. The restore process for Windows will currently not work out-of-box without any further tuning.
Thank you for your notes. Maybe we should follow Pareto rule. Maybe it should not be a goal to use file-based backup on everything. We can use file-based backup in most cases and use the current block-based backup in other cases. Especially when we integrate both backup types together – we could consider the current block-based backup as just another backup backend.
My priorities are roughly:
1. Have good file-based backup and restore process for Linux template-based AppVMs.
2. Have similar process for Debian/Fedora TemplateVMs.
3. Have block-based backup backend.
4. Have good support for StandaloneVMs, Windows and other OSes.
I believe this corresponds not only to my needs, but also to needs of many others. I am generally not against implementing something that has lower priority for me, especially if there is a pull request. ☺
Regards,
Vít Šesták 'v6ak'
Holger Levsen
Mar 3, 2017, 5:21:29 PM3/3/17
to Vít Šesták, qubes-users
On Fri, Mar 03, 2017 at 12:42:56PM -0800, Vít Šesták wrote:
> Well, I don't see any problem with restore. This is roughly what dom0 does (or initiates):
Vít, please design and *test* your restore. Nobody wants backup, everybody
> Well, I don't see any problem with restore. This is roughly what dom0 does (or initiates):
wants restore.
--
cheers,
Holger
Vít Šesták
Mar 4, 2017, 12:53:32 PM3/4/17
to qubes-users
Holger, restore is already implemented in some basic form. (I havre mentioned it here on Mar 1.) Both backup and restore are equally important: Without backup you have nothing to restore from.
Feel free to try it, but remember, this is early stage of development and you should read limitations. I have decided to publish it in such early stage in order to get feedback. (And I already got some valuable feedback.)
On testing: Some basic manual test passed, but I should create automated tests. (Maybe integration tests are more important than unit tests here – after ali, it mostly integrates existing products together.)
You can test restore scenario (at some level) now:
1. Backup.
2. Create new BackupStorageVM and new config directory for restore testing.
3. Run restore (./backup --action=restore other-args…) with specified config directory and VM name template. For example, you specify template “restore-testing-%” in order to add prefix “restore-testing-”.
4. Verify that restored VMs contain the desired data.
If you forget to specify VM name template, it will try to restore it under the original name. However, if the VM still exists, it will fail at creating the VM and will not continue. This is intentional: It will never overwrite existing VM by default. (In future, --force will probably overwrite the existing VM.)
Regards,
Vít Šesták 'v6ak'
Feel free to try it, but remember, this is early stage of development and you should read limitations. I have decided to publish it in such early stage in order to get feedback. (And I already got some valuable feedback.)
On testing: Some basic manual test passed, but I should create automated tests. (Maybe integration tests are more important than unit tests here – after ali, it mostly integrates existing products together.)
You can test restore scenario (at some level) now:
1. Backup.
2. Create new BackupStorageVM and new config directory for restore testing.
3. Run restore (./backup --action=restore other-args…) with specified config directory and VM name template. For example, you specify template “restore-testing-%” in order to add prefix “restore-testing-”.
4. Verify that restored VMs contain the desired data.
If you forget to specify VM name template, it will try to restore it under the original name. However, if the VM still exists, it will fail at creating the VM and will not continue. This is intentional: It will never overwrite existing VM by default. (In future, --force will probably overwrite the existing VM.)
Regards,
Vít Šesták 'v6ak'
Vít Šesták
Mar 5, 2017, 1:07:37 PM3/5/17
to qubes-users
On backup backends: I'd like to move the discussion to GitHub. I've summed up what we need and created some comparison table: https://github.com/v6ak/qubes-incremental-backup-poc/issues/35
Regards,
Vít Šesták 'v6ak'
Regards,
Vít Šesták 'v6ak'
Holger Levsen
Mar 6, 2017, 5:51:22 AM3/6/17
to qubes-users
Hi Vít,
On Sat, Mar 04, 2017 at 09:53:31AM -0800, Vít Šesták wrote:
> Holger, restore is already implemented in some basic form. (I havre mentioned it here on Mar 1.)
I'm sorry, I was wrong indeed and a bit too sarcastic too. I guess this was
due to your initial mail where there was no restore yet…
> Both backup and restore are equally important: Without backup you have nothing to restore from.
well, true. But if you don't regularily test what you backup and test that you
can actually restore, your backup is rather worthless. You don't want to find
out your restore is broken, once your regular install is gone…
> Feel free to try it
I'm sorry, I'm at my limits…
> On testing: Some basic manual test passed, but I should create automated tests. (Maybe integration tests are more important than unit tests here – after ali, it mostly integrates existing products together.)
I think so, yes.
> You can test restore scenario (at some level) now:
>
> 1. Backup.
> 2. Create new BackupStorageVM and new config directory for restore testing.
> 3. Run restore (./backup --action=restore other-args…) with specified config directory and VM name template. For example, you specify template “restore-testing-%” in order to add prefix “restore-testing-”.
> 4. Verify that restored VMs contain the desired data.
cool. Is that documented in your git repo too or just here? :)
and really: thank you very much for working on making the backup experience of
Qubes better. That's very very much appreciated!
--
cheers,
Holger
On Sat, Mar 04, 2017 at 09:53:31AM -0800, Vít Šesták wrote:
> Holger, restore is already implemented in some basic form. (I havre mentioned it here on Mar 1.)
due to your initial mail where there was no restore yet…
> Both backup and restore are equally important: Without backup you have nothing to restore from.
can actually restore, your backup is rather worthless. You don't want to find
out your restore is broken, once your regular install is gone…
> Feel free to try it
I'm sorry, I'm at my limits…
> On testing: Some basic manual test passed, but I should create automated tests. (Maybe integration tests are more important than unit tests here – after ali, it mostly integrates existing products together.)
> You can test restore scenario (at some level) now:
>
> 1. Backup.
> 2. Create new BackupStorageVM and new config directory for restore testing.
> 3. Run restore (./backup --action=restore other-args…) with specified config directory and VM name template. For example, you specify template “restore-testing-%” in order to add prefix “restore-testing-”.
> 4. Verify that restored VMs contain the desired data.
and really: thank you very much for working on making the backup experience of
Qubes better. That's very very much appreciated!
--
cheers,
Holger
Vít Šesták
Mar 6, 2017, 12:57:23 PM3/6/17
to qubes-users
Hi Holger,
I am generally OK with this level of sarcasm.
> But if you don't regularily test what you backup and test that you can actually restore, your backup is rather worthless. You don't want to find out your restore is broken, once your regular install is gone…
Sure. I remember having some hard time with decryption of the backup. Despite it is almost five years ago and I have finally succeeded, I remember it was not a good feeling.
Doing some checks is on my tasklist. I haven't decided how exactly it should look like, but there is some basic ticket on this: https://github.com/v6ak/qubes-incremental-backup-poc/issues/14 If you have some idea on this, you are welcome. (Maybe I should add help-wanted label here.)
> cool. Is that documented in your git repo too or just here? :)
Just here. Well, you can get some bare idea from running ./backup -h, but the testing process is not described elsewhere. 😔
Regards,
Vít Šesták 'v6ak'
I am generally OK with this level of sarcasm.
> But if you don't regularily test what you backup and test that you can actually restore, your backup is rather worthless. You don't want to find out your restore is broken, once your regular install is gone…
Doing some checks is on my tasklist. I haven't decided how exactly it should look like, but there is some basic ticket on this: https://github.com/v6ak/qubes-incremental-backup-poc/issues/14 If you have some idea on this, you are welcome. (Maybe I should add help-wanted label here.)
> cool. Is that documented in your git repo too or just here? :)
Regards,
Vít Šesták 'v6ak'
Vít Šesták
Mar 20, 2017, 12:49:20 PM3/20/17
to qubes-users
Hello again. I am close to 0.1 release, the first numbered release. Mostly documentation-related tasks are missing. Since my last update, I've made rather quality improvements (e.g., not using utf-8 where not needed, disconnecting BDVM from network and so on) than new features. I've also organized issues on Github in milestones.
But the reason why I am writing now are some changes I wish to make in upcoming releases and I might need your help.
First, the current backup storage has hit its limits. I want to create Merkle-tree-based backup storage that would be agnostic of backup backend you use. Aside of various security improvements and reliability improvement (atomicity of backup), this will allow one to use different backup backends for different VMs. I've planned this change for 0.2, but I am ready to postpone it if I feel there is not enough of agreement on its design. That's exactly the reason why I want your comments: https://github.com/v6ak/qubes-incremental-backup-poc/issues/37
Second, there is an ongoing discussion about backup backends. I consider Merkle-tree-based storage as a prerequisite of implementing another backup backends, but we can discuss now: https://github.com/v6ak/qubes-incremental-backup-poc/issues/35
And there are more issues tagged as “help wanted”. Even if you don't feel like contributing with your code, but you want to discuss the design, you are welcome: https://github.com/v6ak/qubes-incremental-backup-poc/labels/help%20wanted
Regards,
Vít Šesták 'v6ak'
But the reason why I am writing now are some changes I wish to make in upcoming releases and I might need your help.
First, the current backup storage has hit its limits. I want to create Merkle-tree-based backup storage that would be agnostic of backup backend you use. Aside of various security improvements and reliability improvement (atomicity of backup), this will allow one to use different backup backends for different VMs. I've planned this change for 0.2, but I am ready to postpone it if I feel there is not enough of agreement on its design. That's exactly the reason why I want your comments: https://github.com/v6ak/qubes-incremental-backup-poc/issues/37
Second, there is an ongoing discussion about backup backends. I consider Merkle-tree-based storage as a prerequisite of implementing another backup backends, but we can discuss now: https://github.com/v6ak/qubes-incremental-backup-poc/issues/35
And there are more issues tagged as “help wanted”. Even if you don't feel like contributing with your code, but you want to discuss the design, you are welcome: https://github.com/v6ak/qubes-incremental-backup-poc/labels/help%20wanted
Regards,
Vít Šesták 'v6ak'
Message has been deleted
Vít Šesták
Mar 20, 2017, 12:56:39 PM3/20/17
to qubes-users
Forgot to mention: The Merkle-tree-based storage will be an incompatible format change. On the positive side, I plan creating a migration script. I know this is going to be pain a bit, but the later I do it, the worse…
Regards,
Vít Šesták 'v6ak'
Regards,
Vít Šesták 'v6ak'
Reply all
Reply to author
Forward
0 new messages