Snapshots - Use of CoW
Drew White
Any chance that there will be added in the feature for snapshots?
even CoW snapshots would be good, then a consolidation option once done.
I have one issue where I want to do something, but I have to 7z the VM before I can do anything to it in-case it breaks.
I know that there are CoW options there, but how do I access them?
You already use the technology since 2.0, but I have not seen it go anything beyond that for the dispvm and the like.
the AppVMs, they have CoW for them, but it's not persistant on the CoW file, that file is destroyed after the VM is shut down.
I tried changing it to not be destroyed on shutdown when it used a CoW file, but Qubes just tell me to get stuffed and destroys it once the guest is shut down.
How can I fix this please?
johny...@sigaint.org
It's a good thing; if one gets compromised, it's a temporary compromise.
:)
If you want permanent changes, update your template.
But it sounds like you might want a "standalone VM":
https://www.qubes-os.org/doc/software-update-vm/
Also, using BTRFS as a root (or VM host) filesystem should allow you to
snapshot/rollback the standalone VM to your heart's content (and clone it
instantly). BTRFS is all about the copy-on-write.
JJ
Marek Marczykowski-Górecki
Hash: SHA256
will rollback changes done during last template run. Currently only one
such revision is stored.
In Qubes 4.0 there will be (actually this is already implemented) an
option to keep multiple of them, and not only for templates, but also
for normal AppVMs.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJX6Oj4AAoJENuP0xzK19csboIH/0Ia7XuTrCdSABbE13MmTCvX
m87DVH/N37DOIyh6eCIfp47ybtBiPlfm66OgSmmjca4ROk/WPPaMNDOeVSSnbJpx
rL4b3BDIFlfnWgdxEWGTlkpKq/VogvF+3A1WN1QpgQ0Ek2mRURqthYcQifiZ4ivI
NbHuwnn5USpfq2G5wLETbgxb/WvPDTbKACkEAvr+YpGgirKP2rJcMutkqwArgo1h
YhMzYPdGODh1TI16kLGrTg4xr4D4W9MLfSwxxf8rclFxRlSFC0ygemwF3H7dYrCa
0HQMYb1a2XEF4q4RXxiztBdKjwLUKW5suni3DvbaSWcpxcI8Yq+GI5DLllls0h8=
=bIXB
-----END PGP SIGNATURE-----
Drew White
> AppVM's are designed to toss changes, other than /home, /rw, /usr/local.
> It's a good thing; if one gets compromised, it's a temporary compromise.
> :)
>
> If you want permanent changes, update your template.
>
> But it sounds like you might want a "standalone VM":
Some of the ones I want to do this for are standalone. I have multiple standalones because I can't do what I want to do completely.
I have 1 for web development, 1 for pascal coding, 1 for pic programming, and several others. They don't have internet access, only LAN access. So they are secure because I limit what they can talk to on the network too.
They can talk to the respository that's on the LAN, so that I don't have to download updates from the internet 10 times just to update my PC. I download once, then update everything.
If it gets compromised, then I just restore from a backup, but since it can't talk to the internet, there is no real threat there.
> Also, using BTRFS as a root (or VM host) filesystem should allow you to
> snapshot/rollback the standalone VM to your heart's content (and clone it
> instantly). BTRFS is all about the copy-on-write.
On an SSD that isn't good, it decreases the life of the drive by over 50%.
This is why I use EXT2 Filesystem, even in the templates. (The templates were originally EXT3/4 if memory serves, as an LVM.
On Monday, 26 September 2016 19:23:11 UTC+10, Marek Marczykowski-Górecki wrote:
> Also in case of templates, you have qvm-revert-template-changes, which
> will rollback changes done during last template run. Currently only one
> such revision is stored.
>
> In Qubes 4.0 there will be (actually this is already implemented) an
> option to keep multiple of them, and not only for templates, but also
> for normal AppVMs.
So I'm stuck on 3.2
Having it in 4+ is more than useless to me.
If it was able to be instantiated into 3.2 somehow, that would be great. If not, then I would have to do it myself after learning every line of code in the final system.
I appreciate that it's going to be in there, but it would be beneficial to have it in a version that wasn't so restrictive.
johny...@sigaint.org
>> AppVM's are designed to toss changes, other than /home, /rw, /usr/local.
>> It's a good thing; if one gets compromised, it's a temporary compromise.
>> :)
>>
>> If you want permanent changes, update your template.
>>
>> But it sounds like you might want a "standalone VM":
>
> Hi JJ, I know it's meant to, but if I'm testing something, I want to
> install pre-requisites for things, then it may have to restart, then I
> want to take a snapshot, then perform the next step, then do another
> snapshot, and revert if I need to.
switching to btrfs root soon.
>> Also, using BTRFS as a root (or VM host) filesystem should allow you to
>> snapshot/rollback the standalone VM to your heart's content (and clone
>> it
>> instantly). BTRFS is all about the copy-on-write.
>
> I'm not going to use that on an SSD, that creates way too many writes. As
> far as I've read and know, correct me if I'm wrong, BTRFS is a journaling
> file system, meaning it writes the changes that it's going to make to the
> journal, and then writes the changes to the disk. This means it writes
> everything twice.
> On an SSD that isn't good, it decreases the life of the drive by over 50%.
"There are some optimizations for SSD drives, and you can enable them by
mounting with -o ssd. "
"Mount -o ssd_spread is more strict about finding a large unused region of
the disk for new allocations, which tends to fragment the free space more
over time. Mount -o ssd_spread is often faster on the less expensive SSD
devices. The default for autodetected SSD devices is mount -o ssd. "
Doesn't sound too bad to me. Are SSD's really that prone to wearing out
quickly?
Ignoring spare files, if you consider that a reflink copy of, say, a 2G
file (such as a VM .img file) takes almost no disk activity on BTRFS (just
a metadata cloning due to the COW nature), while ext4 churns away
reading/writing the full 2G of data, I'd say that BTRFS could *save* you a
whole bunch of wear on your SSD.
snapshotting and reverting under BTFS is near immediate, versus copying of
multiple large files; seems like a no-brainer in favor of BTRFS.
From another page:
"Btrfs is SSD-aware and exploits TRIM/Discard to allow the file system to
report unused blocks to the storage device for reuse. On SSDs, Btrfs
avoids unnecessary seek optimization and aggressively sends writes in
clusters, even if they are from unrelated files. This results in larger
write operations and faster write throughput, albeit at the expense of
more seeks later."
And:
https://oss.oracle.com/pipermail/btrfs-devel/2008-February/000513.html
And:
https://www.linux.com/learn/weekend-project-get-started-btrfs
which comments:
"the system uses a copy-on-write strategy that writes changed data to disk
first, then updates the references in the tree. This crash-proofs the
filesystem, but without the overhead of maintaining a journal."
(That article was six years ago, not sure if things have changed, but on a
quick search I couldn't find any reference to BTRFS using journaling.)
In fact, the wiki page on journaling filesystems:
https://en.wikipedia.org/wiki/Journaling_file_system#Alternatives
considers BTFS and COW-based filesystems as *alternatives* to journaling
filesystems:
"Full copy-on-write file systems (such as ZFS and Btrfs) avoid in-place
changes to file data by writing out the data in newly allocated blocks,
followed by updated metadata that would point to the new data and disown
the old, followed by metadata pointing to that, and so on up to the
superblock, or the root of the file system hierarchy. This has the same
correctness-preserving properties as a journal, without the write-twice
overhead."
So your fears may be unfounded, and you might be missing out on a tech
that provides exactly what you need. :)
> This is why I use EXT2 Filesystem, even in the templates. (The templates
> were originally EXT3/4 if memory serves, as an LVM.
day and age. :)
> Well, I can't use Qubes 4.0, because you are FORCING it to REQUIRE
> technology in the CPU that my Multi-CPU system doesn't have.
JJ
Drew White
Yes they do.
I had 2 SSDs, both doing the same thing. One was EXT4, the other was EXT2.
EXT4, after a check, had over 2 million errors on it. The EXT2 had 1 error.
> Ignoring spare files, if you consider that a reflink copy of, say, a 2G
> file (such as a VM .img file) takes almost no disk activity on BTRFS (just
> a metadata cloning due to the COW nature), while ext4 churns away
> reading/writing the full 2G of data, I'd say that BTRFS could *save* you a
> whole bunch of wear on your SSD.
>
> snapshotting and reverting under BTFS is near immediate, versus copying of
> multiple large files; seems like a no-brainer in favor of BTRFS.
>
> From another page:
>
> "Btrfs is SSD-aware and exploits TRIM/Discard to allow the file system to
> report unused blocks to the storage device for reuse. On SSDs, Btrfs
> avoids unnecessary seek optimization and aggressively sends writes in
> clusters, even if they are from unrelated files. This results in larger
> write operations and faster write throughput, albeit at the expense of
> more seeks later."
>
> And:
>
> https://oss.oracle.com/pipermail/btrfs-devel/2008-February/000513.html
>
> And:
>
> https://www.linux.com/learn/weekend-project-get-started-btrfs
>
> which comments:
>
> "the system uses a copy-on-write strategy that writes changed data to disk
> first, then updates the references in the tree. This crash-proofs the
> filesystem, but without the overhead of maintaining a journal."
>
> (That article was six years ago, not sure if things have changed, but on a
> quick search I couldn't find any reference to BTRFS using journaling.)
Then perhaps the things I read back then were mistaken, or worded what they said the wrong way.
> In fact, the wiki page on journaling filesystems:
>
> https://en.wikipedia.org/wiki/Journaling_file_system#Alternatives
>
> considers BTFS and COW-based filesystems as *alternatives* to journaling
> filesystems:
>
> "Full copy-on-write file systems (such as ZFS and Btrfs) avoid in-place
> changes to file data by writing out the data in newly allocated blocks,
> followed by updated metadata that would point to the new data and disown
> the old, followed by metadata pointing to that, and so on up to the
> superblock, or the root of the file system hierarchy. This has the same
> correctness-preserving properties as a journal, without the write-twice
> overhead."
>
> So your fears may be unfounded, and you might be missing out on a tech
> that provides exactly what you need. :)
>
> > This is why I use EXT2 Filesystem, even in the templates. (The templates
> > were originally EXT3/4 if memory serves, as an LVM.
>
> Flying without a journal (especially wrt metadata) is a bit bold in this
> day and age. :)
Not really.
> > Well, I can't use Qubes 4.0, because you are FORCING it to REQUIRE
> > technology in the CPU that my Multi-CPU system doesn't have.
>
> That's a thorn in my side as well, but possibly the price of progress.
It's not the price of progress, it should be an OPTION in Qubes 4.0. That way, if you don't have it, you can still use Qubes 4. But if you have it, then you are (supposedly) more secure and still using Qubes 4.