Shrinking a private volume

[799]

Hello,

I've migrated ~150gb of data into a Qubes Storage Qube.

After cleaning up older files I have reduced the data to 100gb.

Now I'd like to free the additional 50gb so that dom0 can use this capacity for other qubes.

Unfortunately I found out that shrinking volumes is not possible according to the Qubes Docs:

--- --- ---

From: Qubes Docs > Resize Disk Image

https://www.qubes-os.org/doc/resize-disk-image/

[...] ext4 and most other filesystems do not support online shrinking [...]

1. Create a new qube with smaller disk using Qube Manager or qvm-create
2. Move data to the new qube using qvm-copy, backup & restore, or OS utilities
3. Delete old qube using Qube Manager or qvm-remove

--- --- ---

As I have several files and subdirectories in the Storage Qubes, I can't run qvm-copy but as I don't have enough free space I am also unable to zip everything into one big file, qvm-copy this file to a new (smaller) storage qube and then delete the old qubes.

Is there a better to shrink the volume or migrate the data to another Qube?

Honestly I don't understand why it is not possible to shrink a volume as this is something which can done in linux.

Does someone has an idea how to "reclaim" the free capacity from an AppVM?

- O

[Stuart Perkins]

On Tue, 26 Feb 2019 22:30:35 +0100
799 <one7...@gmail.com> wrote:

>Hello,
>
>I've migrated ~150gb of data into a Qubes Storage Qube.
>After cleaning up older files I have reduced the data to 100gb.
>Now I'd like to free the additional 50gb so that dom0 can use this capacity
>for other qubes.
>
>Unfortunately I found out that shrinking volumes is not possible according
>to the Qubes Docs:
>--- --- ---
>From: Qubes Docs > Resize Disk Image
>https://www.qubes-os.org/doc/resize-disk-image/
>
>[...] ext4 and most other filesystems do not support online shrinking [...]
>

> 1. Create a new qube with smaller disk using Qube Manager or qvm-create
> 2. Move data to the new qube using qvm-copy, backup & restore, or OS
> utilities
> 3. Delete old qube using Qube Manager or qvm-remove

>
>--- --- ---
>
>As I have several files and subdirectories in the Storage Qubes, I can't
>run qvm-copy but as I don't have enough free space I am also unable to zip
>everything into one big file, qvm-copy this file to a new (smaller) storage
>qube and then delete the old qubes.
>Is there a better to shrink the volume or migrate the data to another Qube?
>
>Honestly I don't understand why it is not possible to shrink a volume as
>this is something which can done in linux.
>
>Does someone has an idea how to "reclaim" the free capacity from an AppVM?
>
>- O
>

Don't know Qubes Storage Qube yet...haven't upgraded to 4...but...
Can you mount an additional "drive" in the storage qube? Copy all the files with cp -r or rsync then rearrange things so the new disk image is the main one and delete the overgrown one? Or, conversely, mount the storage qube's main drive in a regular appVM and do the same thing there?

Stuart

[799]

Hello Stuart,

On Wed, 27 Feb 2019 at 01:57, Stuart Perkins <perkins...@gmail.com> wrote:

On Tue, 26 Feb 2019 22:30:35 +0100
799 <one7...@gmail.com> wrote:
>Honestly I don't understand why it is not possible to shrink a volume as
>this is something which can done in linux.
>Does someone has an idea how to "reclaim" the free capacity from an AppVM?

 

[...]

Can you mount an additional "drive" in the storage qube?  Copy all the files with cp -r or rsync then rearrange things so the new disk image is the main one and delete the overgrown one?  Or, conversely, mount the storage qube's main drive in a regular appVM and do the same thing there?

Of course I could copy data from the Old AppVM to an external HDD, create a smaller Qube and then copy the data back.

But this doesn't feel like how it should be done in 2019 ;-)

I thought that there might something like using resize2fs to make the mage smaller from within the AppVM and then use some magic to tell Qubes that it can free up space.

But, maybe another good idea which you have brought up is, to mount the old private.img and copy data to the new private.img of the new qube:

mkdir /tmp/old-private.img

sudo mount /var/lib/qubes/appvms/<OLDQUBE>/private.img

mkdir /tmp/new-private.img

sudo mount /var/lib/qubes/appvms/<NEWQUBE>/private.img

Then I could copy data from /tmp/old-private.img to /tmp/new-private.img

While this could be done from dom0, I don't want to do it from there, but I'd prefer to mount the old private.img into the new AppVM.

I think this something which can be done, from the new AppVM Qubes, to which I mount the old private.img additionally.

I just have to figure out how to do so, haven't found something yet, but afaik I've read it some time ago.

If someone has guidance, this would be helpfull and could work as a "poor man's" reduce disk alternative  ;-)

- O

[unman]

On Tue, Feb 26, 2019 at 10:30:35PM +0100, 799 wrote:
> Hello,
>
> I've migrated ~150gb of data into a Qubes Storage Qube.
> After cleaning up older files I have reduced the data to 100gb.
> Now I'd like to free the additional 50gb so that dom0 can use this capacity
> for other qubes.
>

You dont need to do anything. In 4.0 dom0 will simply use the
unallocated space. (Well, sometimes I restart a qube for this purpose.)
That 50GB is *already* available to other qubes.

[brenda...@gmail.com]

I’ve found it necessary sometimes to sudo fstrim -av from inside the VM in order for the dom0 lvm thin provisioned volumes to return the unallocated space back to the pool.

This doesn’t shrink the volume but it does tell the system that it doesn’t need to store anything there for that VM currently so other VMs can use the space.

Welcome to thin provisioning!

B

[Chris Laprise]

I agree. The default 'discard' config used by Qubes templates should
already have reclaimed most of the space. However, I found it doesn't
recover 100% of the deleted space, so fstrim is sometimes necessary to
get everything back.

OTOH, the upper limit for the volume's size remains the same. If that is
somehow an issue, the are options for shrinking it:

* If the remaining data is not large, you can qvm-copy it to a new VM
and use that instead.

* Otherwise, its possible to shrink the upper limit using a combination
of qvm-block attach, resize2fs (in a VM), and lvresize. This is one of
those "if you know really what you're doing" options.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[onw7two99]

Hello Brendan

I run sudo fstrim -av from within the AppVM (as root) but I haven't
notive any change.

The AppVM has currently a 145GB private disk attached, from which 91GB
is used, 44GB is free.

I would like to move this data to another storage qube, but I have only
58GB available (df -h) in dom0.

Therefore I thought that I can someone move the private disk of
"storage-qube-old" to "storage-qube-new".

I have looked into /var/lib/qubes/appvms/storage-qube-old and see 3
private images:

private-cow.img -> 147 GB

private-cow.img.old -> 147 GB

private-cow.img -> 147 GB

Is there a way to move this private disk to the new storage-qube?


Slightly offtopic:
 I haven't understand at which layer thin provisioning works in Qubes.

Neither in the AppVM nor in dom0 I see something like LVM, but as far as
I remember LVM was available in Qubes (I think have looked into it 1 or
2 years ago.

- O

[Chris Laprise]

On 3/23/19 5:34 AM, onw7two99 wrote:
> Hello Brendan
>
> On 2/28/19 3:14 AM, brenda...@gmail.com wrote:
>> I’ve found it necessary sometimes to sudo fstrim -av from inside the VM in order for the dom0 lvm thin provisioned volumes to return the unallocated space back to the pool.
>> This doesn’t shrink the volume but it does tell the system that it doesn’t need to store anything there for that VM currently so other VMs can use the space.
>> Welcome to thin provisioning!
>
> I run sudo fstrim -av from within the AppVM (as root) but I haven't
> notive any change.

The difference doesn't show until you shutdown the vm for the second
time (the current .old volume gets deleted). Also the Qubes disk widget
and Qubes Manager take some time to report the new values. But running
fstrim may not make any difference if you haven't been deleting many files.

>
> The AppVM has currently a 145GB private disk attached, from which 91GB
> is used, 44GB is free.
>
> I would like to move this data to another storage qube, but I have only
> 58GB available (df -h) in dom0.
>
> Therefore I thought that I can someone move the private disk of
> "storage-qube-old" to "storage-qube-new".
>
> I have looked into /var/lib/qubes/appvms/storage-qube-old and see 3
> private images:
>
> private-cow.img -> 147 GB
>
> private-cow.img.old -> 147 GB
>
> private-cow.img -> 147 GB
>
> Is there a way to move this private disk to the new storage-qube?

Yes, you can use the 'mv' command while the two vms are not running.
Keep in mind this can erase the existing private image of the
new/destination vm unless you rename that first (but it won't be
accessible after renaming it, unless you run 'qvm-block attach').

Also keep in mind that any problems the old vm had arising from anything
in /rw or /home will be carried over to the new vm.

>
>
> Slightly offtopic:
>  I haven't understand at which layer thin provisioning works in Qubes.
>
> Neither in the AppVM nor in dom0 I see something like LVM, but as far as
> I remember LVM was available in Qubes (I think have looked into it 1 or
> 2 years ago.

Since you have -cow.img files in /var/lib/qubes/appvms, your system
seems to have been installed with one of the non-LVM options. This could
be either Ext4 (rather limited) or Btrfs (flexible). Looking at the
output of the 'mount' command in dom0 will tell you which.

[onw7two99]

Hello Chris,

thank you once again for your feedback.

On 3/23/19 11:31 AM, Chris Laprise wrote:
> On 3/23/19 5:34 AM, onw7two99 wrote:
>> I run sudo fstrim -av from within the AppVM (as root) but I haven't
>> notive any change.
>
> The difference doesn't show until you shutdown the vm for the second
> time (the current .old volume gets deleted). Also the Qubes disk
> widget and Qubes Manager take some time to report the new values. But
> running fstrim may not make any difference if you haven't been
> deleting many files.

if I run ls -lah /var/lib/qubes/appvms/<APPVM> the size listed is the
"thick provisioned", is there a way to show the "real" file size in dom0
whch should be the used size  which is reported from within the AppVM
when using "df -h" ?

>> Is there a way to move this private disk to the new storage-qube?
>
> Yes, you can use the 'mv' command while the two vms are not running.
> Keep in mind this can erase the existing private image of the
> new/destination vm unless you rename that first (but it won't be
> accessible after renaming it, unless you run 'qvm-block attach').

I tried to mount the private.img from the old appvm to the new one, but
I think I don't have the correct syntax.

I tried:

qvm-block attach <AppVM-NEW> <AppVM-OLD>:private.img
qvm-block attach <AppVM-NEW> /var/lib/qubes/appvms/<AppVM-OLD>/private.img

How can I attach a private.img from one AppVM to another? Maybe even
once in read/write and in another scenario read only.

This could be usefull for doing forensics on an AppVM or clam av scans etc.

>> Neither in the AppVM nor in dom0 I see something like LVM, but as far as
>> I remember LVM was available in Qubes (I think have looked into it 1 or
>> 2 years ago.
>
> Since you have -cow.img files in /var/lib/qubes/appvms, your system
> seems to have been installed with one of the non-LVM options. This
> could be either Ext4 (rather limited) or Btrfs (flexible). Looking at
> the output of the 'mount' command in dom0 will tell you which.

Stupid me, I wasn't aware that I installed Qubes without LVM ... why
should I do so?

The only reason to fix this is likely to reinstall Qubes with LVM, but
what happens if I restore my old AppVMs from the non-LVM-setup?
Can you give some more details about the private.img vs private-cow.img
vs private-cow.img.old

From the Qubes docs
https://www.qubes-os.org/doc/template-implementation/ I learned that the
root.cow.img will hold the differences (from the appVM perspective) to
root.img (the root image of the template which is read only).

Still I haven't understand what the private.cow.img and
private.cow.img.old do.

Is there some more information regarding this topic?

- O

[unman]

We're all working on the assumption that you have Qubes 4.0 - can you
confirm that?

You can see what size is actually used on disk by using s flag to ls:
so 'ls -lsS' will show you the apparent size and the actual size.

In 3.2, to attach a private.img try:
qvm-block -A <qube> dom0:/var/.....

[799]

Hello Unman,

unman <un...@thirdeyesecurity.org> schrieb am So., 24. März 2019, 03:17:

[...]

We're all working on the assumption that you  have Qubes 4.0 - can you
confirm that?

Yes, of course - sorry for not mentioning this, but I don't see why someone would run Q3.2, I have switched to 4.0 shortly after the release and never looked back.

You can see what size is actually used on disk by using s flag to ls:
so 'ls -lsS' will show you the apparent size and the actual size.

Thanks, I wasn't aware of this command option and will review the man page for ls, to see it there are more interesting options 

In 3.2, to attach a private.img try:
qvm-block -A <qube> dom0:/var/.....

Will this also work under Qubes 4, as you are mentioning 3.2 specifically.

What are the command to do so in 4.0?

- O

[Chris Laprise]

On 3/24/19 4:15 AM, 799 wrote:
> Hello Unman,
>
> unman <un...@thirdeyesecurity.org <mailto:un...@thirdeyesecurity.org>>

> schrieb am So., 24. März 2019, 03:17:
>
> [...]
> We're all working on the assumption that you  have Qubes 4.0 - can you
> confirm that?
>
>
> Yes, of course - sorry for not mentioning this, but I don't see why
> someone would run Q3.2, I have switched to 4.0 shortly after the release
> and never looked back.
>
> You can see what size is actually used on disk by using s flag to ls:
> so 'ls -lsS' will show you the apparent size and the actual size.
>
>
> Thanks, I wasn't aware of this command option and will review the man
> page for ls, to see it there are more interesting options
>
> In 3.2, to attach a private.img try:
> qvm-block -A <qube> dom0:/var/.....

That's 3.x specific.

One way to do it in 4.0 (losetup assigns a device to the img file then
prints the device name):
$ sudo losetup -f /var/lib/qubes/...img
loop0

$ qvm-block attach newvm dom0:loop0

As for backup and restore, you can do this just fine between non-lvm and
lvm systems. If you do create a backup you may want to copy it to a
third device so there is an extra copy, then do a verify on the extra
copy just in case.