Adding New Disk Image to VM
505 views
Skip to first unread message
entr0py
Jun 20, 2016, 2:57:54 PM6/20/16
to qubes-users
Would appreciate if someone could check that I'm not doing something stupid... (Don't see any docs regarding this.)
Use case: I want to store my Documents on a separate .img (on same Qubes SSD) so that only config files remain on my appVM. This should ease the upgrade process when I want to start with fresh appVMs and reduce chances of user error / corruption of frequently moving large existing files.
Steps:
1. create sparse file in dom0:
`dd if=/dev/zero of=/var/lib/qubes/storage/myfiles.img bs=1024k seek=10000 count=0`
2. create filesystem on sparse file:
`mkfs -t ext4 myfiles.img`
3. add file as disk device to /var/lib/qubes/appvms/myappvm.conf:
<disk type='block' device='disk'>
<driver name='phy' />
<source dev='/var/lib/qubes/storage/myfiles.img' />
<target dev='xvde' bus='xen' />
</disk>
4. add /dev/xvde to /etc/fstab in appVM:
/dev/xvde /home/user/storage ext4 defaults,noatime 0 0
5. start appVM
Any issues? TIA!
-------------------------------------------------
ONLY AT VFEmail! - Use our Metadata Mitigator to keep your email out of the NSA's hands!
$24.95 ONETIME Lifetime accounts with Privacy Features!
15GB disk! No bandwidth quotas!
Commercial and Bulk Mail Options!
Use case: I want to store my Documents on a separate .img (on same Qubes SSD) so that only config files remain on my appVM. This should ease the upgrade process when I want to start with fresh appVMs and reduce chances of user error / corruption of frequently moving large existing files.
Steps:
1. create sparse file in dom0:
`dd if=/dev/zero of=/var/lib/qubes/storage/myfiles.img bs=1024k seek=10000 count=0`
2. create filesystem on sparse file:
`mkfs -t ext4 myfiles.img`
3. add file as disk device to /var/lib/qubes/appvms/myappvm.conf:
<disk type='block' device='disk'>
<driver name='phy' />
<source dev='/var/lib/qubes/storage/myfiles.img' />
<target dev='xvde' bus='xen' />
</disk>
4. add /dev/xvde to /etc/fstab in appVM:
/dev/xvde /home/user/storage ext4 defaults,noatime 0 0
5. start appVM
Any issues? TIA!
-------------------------------------------------
ONLY AT VFEmail! - Use our Metadata Mitigator to keep your email out of the NSA's hands!
$24.95 ONETIME Lifetime accounts with Privacy Features!
15GB disk! No bandwidth quotas!
Commercial and Bulk Mail Options!
Marek Marczykowski-Górecki
Jun 20, 2016, 3:11:11 PM6/20/16
to entr0py, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Mon, Jun 20, 2016 at 06:35:14PM +0000, entr0py wrote:
> Would appreciate if someone could check that I'm not doing something stupid... (Don't see any docs regarding this.)
>
> Use case: I want to store my Documents on a separate .img (on same Qubes SSD) so that only config files remain on my appVM. This should ease the upgrade process when I want to start with fresh appVMs and reduce chances of user error / corruption of frequently moving large existing files.
>
> Steps:
>
> 1. create sparse file in dom0:
> `dd if=/dev/zero of=/var/lib/qubes/storage/myfiles.img bs=1024k seek=10000 count=0`
You can use sparse files to save some space:
truncate -s 10G /var/lib/qubes/storage/myfiles.img
> 2. create filesystem on sparse file:
> `mkfs -t ext4 myfiles.img`
>
> 3. add file as disk device to /var/lib/qubes/appvms/myappvm.conf:
> <disk type='block' device='disk'>
> <driver name='phy' />
> <source dev='/var/lib/qubes/storage/myfiles.img' />
> <target dev='xvde' bus='xen' />
> </disk>
This config file is regenerated each time the VM is started, so your
changes will have no effect...
Recently I've posted an instruction about triggering qvm-block on VM
startup to achieve the same effect. Take a look here:
https://groups.google.com/d/topic/qubes-users/RogG5rXG_Pw/discussion
> 4. add /dev/xvde to /etc/fstab in appVM:
> /dev/xvde /home/user/storage ext4 defaults,noatime 0 0
>
> 5. start appVM
>
> Any issues? TIA!
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJXaD/GAAoJENuP0xzK19css1cIAJMLGDAF3OJMgoUAYfGy/8lG
LKkeMKn7tCJ8beuQ0WO+mDz4C4O3Na1p3x3M9ItEpLz079JmLh3Etm5YaPcjWz4w
qPst5QRwMVXxYt7MglDMJFriEO2/yJ662598QNk9RblxWxjZeBICNIjbqFf33k8Q
Tg/CUAIS2FB6qoYXMg2Z6CLFNaa1FUjtR2RkVJ7OJVlraEt9nAUSdbbe6HOmSaJp
9xB88hk1LwDT2wQI9OQKKfYVTuhCecp5aiA0DaPBrH0pD9tz/7RipAlRucikdz+L
iarLIF3J9ZClxFfhdkJaiTEKnVb5u/ViHyr29DfjTlwr7cU3gPF7WT0ZSUZRRBM=
=r80r
-----END PGP SIGNATURE-----
Hash: SHA256
On Mon, Jun 20, 2016 at 06:35:14PM +0000, entr0py wrote:
> Would appreciate if someone could check that I'm not doing something stupid... (Don't see any docs regarding this.)
>
> Use case: I want to store my Documents on a separate .img (on same Qubes SSD) so that only config files remain on my appVM. This should ease the upgrade process when I want to start with fresh appVMs and reduce chances of user error / corruption of frequently moving large existing files.
>
> Steps:
>
> 1. create sparse file in dom0:
> `dd if=/dev/zero of=/var/lib/qubes/storage/myfiles.img bs=1024k seek=10000 count=0`
truncate -s 10G /var/lib/qubes/storage/myfiles.img
> 2. create filesystem on sparse file:
> `mkfs -t ext4 myfiles.img`
>
> 3. add file as disk device to /var/lib/qubes/appvms/myappvm.conf:
> <disk type='block' device='disk'>
> <driver name='phy' />
> <source dev='/var/lib/qubes/storage/myfiles.img' />
> <target dev='xvde' bus='xen' />
> </disk>
changes will have no effect...
Recently I've posted an instruction about triggering qvm-block on VM
startup to achieve the same effect. Take a look here:
https://groups.google.com/d/topic/qubes-users/RogG5rXG_Pw/discussion
> 4. add /dev/xvde to /etc/fstab in appVM:
> /dev/xvde /home/user/storage ext4 defaults,noatime 0 0
>
> 5. start appVM
>
> Any issues? TIA!
- --
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJXaD/GAAoJENuP0xzK19css1cIAJMLGDAF3OJMgoUAYfGy/8lG
LKkeMKn7tCJ8beuQ0WO+mDz4C4O3Na1p3x3M9ItEpLz079JmLh3Etm5YaPcjWz4w
qPst5QRwMVXxYt7MglDMJFriEO2/yJ662598QNk9RblxWxjZeBICNIjbqFf33k8Q
Tg/CUAIS2FB6qoYXMg2Z6CLFNaa1FUjtR2RkVJ7OJVlraEt9nAUSdbbe6HOmSaJp
9xB88hk1LwDT2wQI9OQKKfYVTuhCecp5aiA0DaPBrH0pD9tz/7RipAlRucikdz+L
iarLIF3J9ZClxFfhdkJaiTEKnVb5u/ViHyr29DfjTlwr7cU3gPF7WT0ZSUZRRBM=
=r80r
-----END PGP SIGNATURE-----
Andrew David Wong
Jun 20, 2016, 3:12:54 PM6/20/16
to entr0py, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
On 2016-06-20 11:35, entr0py wrote:
> Would appreciate if someone could check that I'm not doing
> something stupid... (Don't see any docs regarding this.)
>
I think the reason there are no docs for this is that this seems like
going out of your way to use Qubes in a way that it was not intended
to be used. (Of course, you're perfectly within your rights to do that.)
> Use case: I want to store my Documents on a separate .img (on same
> Qubes SSD) so that only config files remain on my appVM. This
> should ease the upgrade process when I want to start with fresh
> appVMs and reduce chances of user error / corruption of frequently
> moving large existing files.
>
So, as you probably know, Qubes is designed with the expectation that
users will store their data (and config files) in AppVMs. Normally,
there's not much reason to need to "start with fresh AppVMs" due to
the way TemplateVMs work. (You already "start fresh" with respect to
the root filesystem each time you restart an AppVM anyway.)
Of course, there might be some reason to roll back to the previous
state of an AppVM, but that's what backups are for.
In the rare case where we actually need to migrate all the data from
one AppVM to a new one, and there's a large amount of that data, I
think it makes sense to use conventional data-integrity-verification
tools to do that (just as you would if migrating the data from one
conventional OS to another).
I'm not saying any of this to try to dissuade you from doing what you
want to do. I'm just offering another perspective and suggesting that,
depending your goals, this might not be necessary.
- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJXaEAwAAoJENtN07w5UDAw5lcP/RMrkmQFcTKvo6HU4BnG2vmp
V0hcScH3E6JkT7AmRns0YlS74bsngaLzRV3DNQVJGto/yuQ2YS2O7IUvWDd2IZLB
bPvCCKOy2dmIxFcEE9GYNbJ2UD+PqR5JhbffgB8SHi8oBk2cGu0mT5yfdqp2NBq2
vzG7phFYoLLoBMQBepGKaxZGS0dmwhOvzY4YpafoykTy7ihxQgEOTYO/AA3QOFrq
+qrpdehFleTBpvPsRb6Z+3SUO8eJgpmPGMEEIJmhDp2VIJU8bC7IjupK3LN7JaAQ
rslyAH9xL55JwXhsghuM4HFa14lcrBR6EYCCdIHRBYg/rnQMkxdPWlPnsLSSK+ri
muOOCWzRJz1oNFFSH1OwNBxLP534SofY6RgZZzC02HFHTuJzKsMkCcMb681pHQKk
szy15Qcfvf2JLFxB+UjG9kls0qtv8eCeHMXIzeAgou05K5rf5tqKaNQKELO8YiM+
BSj0hMvVHykc1xoQxo0rZfHNDynyHCoYPB7mi0oe4bJ6iS0vj/3IaAcwEEjof8E3
JOqstIqrhFRLMv6IocXmQ6TOhhS5mDbvyY6fPZv7iWH1/mgyO9eNOaW3RmuCTIcu
U/1ikefi6HoJ9C2fYSiEvcu6Y1D3yIsg+ysszu7eezYmBy4UOoNWASeb9qpelOQ1
wLdHy4bZcPqCVDXRcGEM
=5UNE
-----END PGP SIGNATURE-----
Hash: SHA512
On 2016-06-20 11:35, entr0py wrote:
> Would appreciate if someone could check that I'm not doing
> something stupid... (Don't see any docs regarding this.)
>
going out of your way to use Qubes in a way that it was not intended
to be used. (Of course, you're perfectly within your rights to do that.)
> Use case: I want to store my Documents on a separate .img (on same
> Qubes SSD) so that only config files remain on my appVM. This
> should ease the upgrade process when I want to start with fresh
> appVMs and reduce chances of user error / corruption of frequently
> moving large existing files.
>
users will store their data (and config files) in AppVMs. Normally,
there's not much reason to need to "start with fresh AppVMs" due to
the way TemplateVMs work. (You already "start fresh" with respect to
the root filesystem each time you restart an AppVM anyway.)
Of course, there might be some reason to roll back to the previous
state of an AppVM, but that's what backups are for.
In the rare case where we actually need to migrate all the data from
one AppVM to a new one, and there's a large amount of that data, I
think it makes sense to use conventional data-integrity-verification
tools to do that (just as you would if migrating the data from one
conventional OS to another).
I'm not saying any of this to try to dissuade you from doing what you
want to do. I'm just offering another perspective and suggesting that,
depending your goals, this might not be necessary.
- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJXaEAwAAoJENtN07w5UDAw5lcP/RMrkmQFcTKvo6HU4BnG2vmp
V0hcScH3E6JkT7AmRns0YlS74bsngaLzRV3DNQVJGto/yuQ2YS2O7IUvWDd2IZLB
bPvCCKOy2dmIxFcEE9GYNbJ2UD+PqR5JhbffgB8SHi8oBk2cGu0mT5yfdqp2NBq2
vzG7phFYoLLoBMQBepGKaxZGS0dmwhOvzY4YpafoykTy7ihxQgEOTYO/AA3QOFrq
+qrpdehFleTBpvPsRb6Z+3SUO8eJgpmPGMEEIJmhDp2VIJU8bC7IjupK3LN7JaAQ
rslyAH9xL55JwXhsghuM4HFa14lcrBR6EYCCdIHRBYg/rnQMkxdPWlPnsLSSK+ri
muOOCWzRJz1oNFFSH1OwNBxLP534SofY6RgZZzC02HFHTuJzKsMkCcMb681pHQKk
szy15Qcfvf2JLFxB+UjG9kls0qtv8eCeHMXIzeAgou05K5rf5tqKaNQKELO8YiM+
BSj0hMvVHykc1xoQxo0rZfHNDynyHCoYPB7mi0oe4bJ6iS0vj/3IaAcwEEjof8E3
JOqstIqrhFRLMv6IocXmQ6TOhhS5mDbvyY6fPZv7iWH1/mgyO9eNOaW3RmuCTIcu
U/1ikefi6HoJ9C2fYSiEvcu6Y1D3yIsg+ysszu7eezYmBy4UOoNWASeb9qpelOQ1
wLdHy4bZcPqCVDXRcGEM
=5UNE
-----END PGP SIGNATURE-----
entr0py
Jun 20, 2016, 4:43:38 PM6/20/16
to Marek Marczykowski-Górecki, Andrew David Wong, qubes-users
Marek Marczykowski-Górecki:
Andrew David Wong:
I don't think what I'm trying to do is un-Qubes-like at all. After all, we are trying to isolate and compartmentalize independent aspects of our systems. (This benefits not just security but scalability as well). (Isn't StorageVM on the horizon?) My Documents (perhaps Archives would be a more contrastful term) should **never** need to be moved because of any change in the underlying Operating System - regardless of how rarely such events might occur.
(Perhaps due to my ignorance, or unwillingness to research how config files function for each component of my Template,) I tend to perform fresh installs of my appVMs with any major change in the underlying Template. For example, moving from Debian Jessie to Stretch may involve any or all of the following: Whonix 13 -> 14, LibreOffice 4 -> 5, KDE 4 -> 5, etc. These are major upgrades and I don't want to assume that the devs have correctly anticipated all the prior configs that will work properly or optimally. In addition, appVMs can get polluted with scripts and bind-directories in /rw. I could just manually delete config directories but like I said, I can't anticipate what unintended consequences that might have unless I actually research it.
> In the rare case where we actually need to migrate all the data from
> one AppVM to a new one, and there's a large amount of that data, I
> think it makes sense to use conventional data-integrity-verification
> tools to do that (just as you would if migrating the data from one
> conventional OS to another).
>
Could you suggest some tools? I've been using tar --verify, qvm-copy, shasum, then hoping! that tar -xf doesn't corrupt anything :( Also, since I want to re-use some of the same VM names, and since I've had mixed results with renaming VMs in the past (in R2.x, not all .xml files got updated properly), I usually have to copy twice: first copy to an intermediary VM, delete original VM, create new VM with same name, copy from temp to new VM.
Again, this would be fine for several GBs but not something I should ever have to do for 100's of GBs. Even migrating to another OS should not require moving troves of data, since most data these days is OS-agnostic.
> On Mon, Jun 20, 2016 at 06:35:14PM +0000, entr0py wrote:
>> 3. add file as disk device to /var/lib/qubes/appvms/myappvm.conf:
>> <disk type='block' device='disk'>
>> <driver name='phy' />
>> <source dev='/var/lib/qubes/storage/myfiles.img' />
>> <target dev='xvde' bus='xen' />
>> </disk>
>
> This config file is regenerated each time the VM is started, so your
> changes will have no effect...
>
> Recently I've posted an instruction about triggering qvm-block on VM
> startup to achieve the same effect. Take a look here:
> https://groups.google.com/d/topic/qubes-users/RogG5rXG_Pw/discussion
>
Thanks! Saw that but didn't realize it was relevant (and this would have been easier :/ )
>> 3. add file as disk device to /var/lib/qubes/appvms/myappvm.conf:
>> <disk type='block' device='disk'>
>> <driver name='phy' />
>> <source dev='/var/lib/qubes/storage/myfiles.img' />
>> <target dev='xvde' bus='xen' />
>> </disk>
>
> This config file is regenerated each time the VM is started, so your
> changes will have no effect...
>
> Recently I've posted an instruction about triggering qvm-block on VM
> startup to achieve the same effect. Take a look here:
> https://groups.google.com/d/topic/qubes-users/RogG5rXG_Pw/discussion
>
Andrew David Wong:
> On 2016-06-20 11:35, entr0py wrote:
>> Would appreciate if someone could check that I'm not doing
>> something stupid... (Don't see any docs regarding this.)
>
>> Would appreciate if someone could check that I'm not doing
>> something stupid... (Don't see any docs regarding this.)
>
> I think the reason there are no docs for this is that this seems like
> going out of your way to use Qubes in a way that it was not intended
> to be used. (Of course, you're perfectly within your rights to do that.)
>
> going out of your way to use Qubes in a way that it was not intended
> to be used. (Of course, you're perfectly within your rights to do that.)
>
>> Use case: I want to store my Documents on a separate .img (on same
>> Qubes SSD) so that only config files remain on my appVM. This
>> should ease the upgrade process when I want to start with fresh
>> appVMs and reduce chances of user error / corruption of frequently
>> moving large existing files.
>
>> Qubes SSD) so that only config files remain on my appVM. This
>> should ease the upgrade process when I want to start with fresh
>> appVMs and reduce chances of user error / corruption of frequently
>> moving large existing files.
>
> So, as you probably know, Qubes is designed with the expectation that
> users will store their data (and config files) in AppVMs. Normally,
> there's not much reason to need to "start with fresh AppVMs" due to
> the way TemplateVMs work. (You already "start fresh" with respect to
> the root filesystem each time you restart an AppVM anyway.)
>
> users will store their data (and config files) in AppVMs. Normally,
> there's not much reason to need to "start with fresh AppVMs" due to
> the way TemplateVMs work. (You already "start fresh" with respect to
> the root filesystem each time you restart an AppVM anyway.)
>
> I'm not saying any of this to try to dissuade you from doing what you
> want to do. I'm just offering another perspective and suggesting that,
> depending your goals, this might not be necessary.
>
Your input is always appreciated.
> want to do. I'm just offering another perspective and suggesting that,
> depending your goals, this might not be necessary.
>
I don't think what I'm trying to do is un-Qubes-like at all. After all, we are trying to isolate and compartmentalize independent aspects of our systems. (This benefits not just security but scalability as well). (Isn't StorageVM on the horizon?) My Documents (perhaps Archives would be a more contrastful term) should **never** need to be moved because of any change in the underlying Operating System - regardless of how rarely such events might occur.
(Perhaps due to my ignorance, or unwillingness to research how config files function for each component of my Template,) I tend to perform fresh installs of my appVMs with any major change in the underlying Template. For example, moving from Debian Jessie to Stretch may involve any or all of the following: Whonix 13 -> 14, LibreOffice 4 -> 5, KDE 4 -> 5, etc. These are major upgrades and I don't want to assume that the devs have correctly anticipated all the prior configs that will work properly or optimally. In addition, appVMs can get polluted with scripts and bind-directories in /rw. I could just manually delete config directories but like I said, I can't anticipate what unintended consequences that might have unless I actually research it.
> In the rare case where we actually need to migrate all the data from
> one AppVM to a new one, and there's a large amount of that data, I
> think it makes sense to use conventional data-integrity-verification
> tools to do that (just as you would if migrating the data from one
> conventional OS to another).
>
Again, this would be fine for several GBs but not something I should ever have to do for 100's of GBs. Even migrating to another OS should not require moving troves of data, since most data these days is OS-agnostic.
Andrew David Wong
Jun 20, 2016, 5:40:00 PM6/20/16
to entr0py, Marek Marczykowski-Górecki, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
But that's the difference. Qubes is about *security* by
compartmentalization. Sure, maybe we can also achieve *scalability* by
compartmentalization as a happy byproduct, but that's not a primary
goal of Qubes. (If we ever have to choose between security and
scalability, security will win.)
In other words: Qubes is not just about isolation/compartmentalization
simpliciter, or isolation/compartmentalization for its own sake, or
anything like that. It's about using isolation/compartmentalization
*as a means to* creating a secure user endpoint.
> (Isn't StorageVM on the horizon?)
Yes, but it sounds like it will be different from what you're doing
here. From what I understand, the idea of the StorageVM is to isolate
the storage backend (disks and drivers) from what will be the AdminVM
(all of which is currently handled in dom0). This wouldn't change the
current standard practice of storing user data in AppVMs, so I don't
think it would directly solve your problem (isolating user data from
AppVMs).
> My Documents (perhaps Archives would be a more contrastful term)
> should **never** need to be moved because of any change in the
> underlying Operating System - regardless of how rarely such events
> might occur.
>
I agree with that sentiment, but that sounds like extra-Qubes (beyond
Qubes) functionality.
For example, if Fedora screws something up and makes it so that data
in a Fedora-based AppVM needs to be moved to a fresh AppVM, then
that's a problem caused by Fedora, not Qubes. Qubes is just a "user"
of Fedora (and Xen and all the other components). Qubes takes all
these components and puts them together into a reasonably secure,
integrated desktop operating system. It wouldn't make sense to blame
Qubes for Fedora's failure to safeguard user data in such a scenario.
Qubes doesn't claim to (and doesn't really try to) make up for any
shortcomings in Fedora (or any other OS that Qubes VMs might be based
upon). It just provides those as containers for us to use. What goes
on *inside* those containers is up to us.
Again, none of this means that what you're doing is wrong, and I
personally think protecting your data from the underlying OS is a
worthy goal.
> (Perhaps due to my ignorance, or unwillingness to research how
> config files function for each component of my Template,) I tend to
> perform fresh installs of my appVMs with any major change in the
> underlying Template. For example, moving from Debian Jessie to
> Stretch may involve any or all of the following: Whonix 13 -> 14,
> LibreOffice 4 -> 5, KDE 4 -> 5, etc. These are major upgrades and I
> don't want to assume that the devs have correctly anticipated all
> the prior configs that will work properly or optimally. In
> addition, appVMs can get polluted with scripts and bind-directories
> in /rw. I could just manually delete config directories but like I
> said, I can't anticipate what unintended consequences that might
> have unless I actually research it.
>
>
>> In the rare case where we actually need to migrate all the data
>> from one AppVM to a new one, and there's a large amount of that
>> data, I think it makes sense to use conventional
>> data-integrity-verification tools to do that (just as you would
>> if migrating the data from one conventional OS to another).
>>
>
> Could you suggest some tools? I've been using tar --verify,
> qvm-copy, shasum, then hoping! that tar -xf doesn't corrupt
> anything :(
I'm not an advanced sysadmin or anything, so I would just do something
very basic, like write a script that calculates the hash of each file
in the original VM (logging the output), make a backup, migrate the
data to the fresh VM, calculate all the hashes again (logging the
output again), and diff the two logs (before and after) to make sure
all the file hashes match. There are probably more efficient ways.
> Also, since I want to re-use some of the same VM names, and since
> I've had mixed results with renaming VMs in the past (in R2.x, not
> all .xml files got updated properly), I usually have to copy
> twice: first copy to an intermediary VM, delete original VM, create
> new VM with same name, copy from temp to new VM.
>
> Again, this would be fine for several GBs but not something I
> should ever have to do for 100's of GBs. Even migrating to another
> OS should not require moving troves of data, since most data these
> days is OS-agnostic.
>
True, you shouldn't have to, but note that silent data corruption can
occur due to degradation of the underlying storage medium even if you
never move the data. (This is sometimes colloquially referred to as
"bit rot.") You may have files that have been silently corrupted, and
you may not find out until you try to open the file N years from now
only to discover that it's unreadable. Certain new filesystems like
ZFS and Btrfs have built-in integrity-checking to detect this.
(Personally, I just make frequent backups and keep them all forever.
We live in an age of cheap hard disks and unlimited cloud storage.)
- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJXaGKfAAoJENtN07w5UDAwuO0P/jCwgvL5GJe6WZk2QNikxaL6
wUw+nuOFowOFn7ZwoL5cCB015LTLqk65js40Ba3kVj6Vj0qYq5eic+gFnCVeAIXO
tpJXG8n7yAveSn4kuuUqQLjWK4uo4J/mXQ4AoJLaoZvTal4nVKlmy1UBYJ/Nmu8+
ZlKvGILR3OPUig8LLTEBhBMNXK7y99ZxXWczOTl6Ea3qb7YN3PM27xzzYv19/qMO
ZJjqSgEslISmpXLZDMRWI06iii28KPnOTR6IyBfOOu5f3nlZ4XlDlj3czDKX1ucU
Kau+9VRBoW7oz22vYh/HZEHKblti/nui5L/P56w0CjSGAsbOFSUveSbgRzOVevEO
j5PjXy+wFb3KiSHeqjvNeFw3AqIrOnVtSBvKdsaQkZ1DfncNskGBlWUy1vD8Yc9A
sH931JBoZmv3bnilNf2QborqXxoXTbl+uWTneXWF13m9vv5sKWqcgCuMcG/BX4VB
zfsbOAuNaqSM2XTMes0kEwkJe4YgVh4gdiTYZVLEYBDTR9AOpxc/zILH33ByFZXu
wDFHBUhBWMrUFvy0A+46Bd0f0rMhhIbx4RrSGj+52/wzxgY/pL1lisvUMhZXIlXg
yYVEoNkSTdHkMRxDoTRyrIzEpPMDIO9N+qzqwxOB/H4vpHRDx49CPWZJxNPC25ZV
V07fRchDuZy8WibiF04e
=9rBg
-----END PGP SIGNATURE-----
Hash: SHA512
compartmentalization. Sure, maybe we can also achieve *scalability* by
compartmentalization as a happy byproduct, but that's not a primary
goal of Qubes. (If we ever have to choose between security and
scalability, security will win.)
In other words: Qubes is not just about isolation/compartmentalization
simpliciter, or isolation/compartmentalization for its own sake, or
anything like that. It's about using isolation/compartmentalization
*as a means to* creating a secure user endpoint.
> (Isn't StorageVM on the horizon?)
here. From what I understand, the idea of the StorageVM is to isolate
the storage backend (disks and drivers) from what will be the AdminVM
(all of which is currently handled in dom0). This wouldn't change the
current standard practice of storing user data in AppVMs, so I don't
think it would directly solve your problem (isolating user data from
AppVMs).
> My Documents (perhaps Archives would be a more contrastful term)
> should **never** need to be moved because of any change in the
> underlying Operating System - regardless of how rarely such events
> might occur.
>
Qubes) functionality.
For example, if Fedora screws something up and makes it so that data
in a Fedora-based AppVM needs to be moved to a fresh AppVM, then
that's a problem caused by Fedora, not Qubes. Qubes is just a "user"
of Fedora (and Xen and all the other components). Qubes takes all
these components and puts them together into a reasonably secure,
integrated desktop operating system. It wouldn't make sense to blame
Qubes for Fedora's failure to safeguard user data in such a scenario.
Qubes doesn't claim to (and doesn't really try to) make up for any
shortcomings in Fedora (or any other OS that Qubes VMs might be based
upon). It just provides those as containers for us to use. What goes
on *inside* those containers is up to us.
Again, none of this means that what you're doing is wrong, and I
personally think protecting your data from the underlying OS is a
worthy goal.
> (Perhaps due to my ignorance, or unwillingness to research how
> config files function for each component of my Template,) I tend to
> perform fresh installs of my appVMs with any major change in the
> underlying Template. For example, moving from Debian Jessie to
> Stretch may involve any or all of the following: Whonix 13 -> 14,
> LibreOffice 4 -> 5, KDE 4 -> 5, etc. These are major upgrades and I
> don't want to assume that the devs have correctly anticipated all
> the prior configs that will work properly or optimally. In
> addition, appVMs can get polluted with scripts and bind-directories
> in /rw. I could just manually delete config directories but like I
> said, I can't anticipate what unintended consequences that might
> have unless I actually research it.
>
>
>> In the rare case where we actually need to migrate all the data
>> from one AppVM to a new one, and there's a large amount of that
>> data, I think it makes sense to use conventional
>> data-integrity-verification tools to do that (just as you would
>> if migrating the data from one conventional OS to another).
>>
>
> Could you suggest some tools? I've been using tar --verify,
> qvm-copy, shasum, then hoping! that tar -xf doesn't corrupt
> anything :(
very basic, like write a script that calculates the hash of each file
in the original VM (logging the output), make a backup, migrate the
data to the fresh VM, calculate all the hashes again (logging the
output again), and diff the two logs (before and after) to make sure
all the file hashes match. There are probably more efficient ways.
> Also, since I want to re-use some of the same VM names, and since
> I've had mixed results with renaming VMs in the past (in R2.x, not
> all .xml files got updated properly), I usually have to copy
> twice: first copy to an intermediary VM, delete original VM, create
> new VM with same name, copy from temp to new VM.
>
> Again, this would be fine for several GBs but not something I
> should ever have to do for 100's of GBs. Even migrating to another
> OS should not require moving troves of data, since most data these
> days is OS-agnostic.
>
occur due to degradation of the underlying storage medium even if you
never move the data. (This is sometimes colloquially referred to as
"bit rot.") You may have files that have been silently corrupted, and
you may not find out until you try to open the file N years from now
only to discover that it's unreadable. Certain new filesystems like
ZFS and Btrfs have built-in integrity-checking to detect this.
(Personally, I just make frequent backups and keep them all forever.
We live in an age of cheap hard disks and unlimited cloud storage.)
- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----
wUw+nuOFowOFn7ZwoL5cCB015LTLqk65js40Ba3kVj6Vj0qYq5eic+gFnCVeAIXO
tpJXG8n7yAveSn4kuuUqQLjWK4uo4J/mXQ4AoJLaoZvTal4nVKlmy1UBYJ/Nmu8+
ZlKvGILR3OPUig8LLTEBhBMNXK7y99ZxXWczOTl6Ea3qb7YN3PM27xzzYv19/qMO
ZJjqSgEslISmpXLZDMRWI06iii28KPnOTR6IyBfOOu5f3nlZ4XlDlj3czDKX1ucU
Kau+9VRBoW7oz22vYh/HZEHKblti/nui5L/P56w0CjSGAsbOFSUveSbgRzOVevEO
j5PjXy+wFb3KiSHeqjvNeFw3AqIrOnVtSBvKdsaQkZ1DfncNskGBlWUy1vD8Yc9A
sH931JBoZmv3bnilNf2QborqXxoXTbl+uWTneXWF13m9vv5sKWqcgCuMcG/BX4VB
zfsbOAuNaqSM2XTMes0kEwkJe4YgVh4gdiTYZVLEYBDTR9AOpxc/zILH33ByFZXu
wDFHBUhBWMrUFvy0A+46Bd0f0rMhhIbx4RrSGj+52/wzxgY/pL1lisvUMhZXIlXg
yYVEoNkSTdHkMRxDoTRyrIzEpPMDIO9N+qzqwxOB/H4vpHRDx49CPWZJxNPC25ZV
V07fRchDuZy8WibiF04e
=9rBg
-----END PGP SIGNATURE-----
entr0py
Jun 21, 2016, 9:15:57 AM6/21/16
to Andrew David Wong, Marek Marczykowski-Górecki, qubes-users
Andrew, thanks for clarifications!
Andrew David Wong:
Andrew David Wong:
> True, you shouldn't have to, but note that silent data corruption can
> occur due to degradation of the underlying storage medium even if you
> never move the data.
>
I've got floppies that I've been meaning to archive (won't say how old they are - would reflect poorly on my age as well as my procrastination habits). Would be a miracle if they're still readable :(
> occur due to degradation of the underlying storage medium even if you
> never move the data.
>
entr0py
Jun 21, 2016, 9:16:30 AM6/21/16
to Andrew David Wong, Marek Marczykowski-Górecki, qubes-users
Working steps for anyone interested (based on Marek & Tomhet's posts: https://groups.google.com/forum/#!topic/qubes-users/RogG5rXG_Pw/discussion)
> Use case: I want to store my Documents on a separate .img (on same Qubes SSD) so that only config files remain on my appVM. This should ease the upgrade process when I want to start with fresh appVMs and reduce chances of user error / corruption of frequently moving large existing files.
3. set up Qubes-RPC in dom0:
add to /etc/qubes-rpc/attach-myfiles.img:
/usr/bin/qvm-block -A --force-root -f xvde myappVM dom0:/var/lib/qubes/storage/myfiles.img
add to /etc/qubes-rpc/policy/attach-myfiles.img:
myappVM dom0 allow
$anyvm $anyvm deny
4. set up fstab in appVM:
add to /rw/config/fstab:
/dev/xvde /home/user/storage ext4 defaults,noatime,discard 0 0
5. set up rc.local in appVM:
add to /rw/config/rc.local
cat /rw/config/fstab >> /etc/fstab
qrexec-client-vm dom0 attach-myfiles.img
mount -a
6. umount & detach will happen automatically when appVM shuts down.
Thanks everyone for help!
> Use case: I want to store my Documents on a separate .img (on same Qubes SSD) so that only config files remain on my appVM. This should ease the upgrade process when I want to start with fresh appVMs and reduce chances of user error / corruption of frequently moving large existing files.
1. create sparse file in dom0:
truncate -s 10G /var/lib/qubes/storage/myfiles.img
2. create filesystem on sparse file:
mkfs.ext4 myfiles.img
2. create filesystem on sparse file:
3. set up Qubes-RPC in dom0:
add to /etc/qubes-rpc/attach-myfiles.img:
/usr/bin/qvm-block -A --force-root -f xvde myappVM dom0:/var/lib/qubes/storage/myfiles.img
add to /etc/qubes-rpc/policy/attach-myfiles.img:
myappVM dom0 allow
$anyvm $anyvm deny
4. set up fstab in appVM:
add to /rw/config/fstab:
/dev/xvde /home/user/storage ext4 defaults,noatime,discard 0 0
5. set up rc.local in appVM:
add to /rw/config/rc.local
cat /rw/config/fstab >> /etc/fstab
qrexec-client-vm dom0 attach-myfiles.img
mount -a
6. umount & detach will happen automatically when appVM shuts down.
Thanks everyone for help!
Reply all
Reply to author
Forward
0 new messages