Qubes4: Cannot increase size of VM private storage

[RSS]

I do believe this is the first time in a couple of years of Qubes 3.2 &
Qubes 4 that this has happened:

I am trying to use the "Qube Settings" GUI to increase "private storage
max. size", and it is failing, silently at that.

Even qvm-extend on the dom0 command line is failing, also silently.

It looks like the space is actually being assigned to the volume, but
the file system size has not changed. So I keep increasing the assigned
space, but df output for /rw in the running qube is unchanged, even
after multiple qube restarts. (I know the space is actually being
assigned because I have to increase the size on each qvm-extend call.)

Further, inside the broken VM:

root@q4home:/home/user# resize2fs /dev/xvdb
resize2fs 1.44.2 (14-May-2018)
Filesystem at /dev/xvdb is mounted on /rw; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
resize2fs: Permission denied to resize filesystem

I do believe on bare-metal Linux, if the partition is bigger than the
file system, invoking resize2fs like this on a mounted file system would
just automatically increase the file system size to match the partition
size.

Instead, this strange "Permission denied".....

I am guessing this is a consequence of using the lvm_thin pool?

Is there some solution more elegant than creating a new VM and qvm-copy
all the files over?

[awokd]

On Sun, July 1, 2018 9:59 am, RSS wrote:
>
> root@q4home:/home/user# resize2fs /dev/xvdb
> resize2fs 1.44.2 (14-May-2018) Filesystem at /dev/xvdb is mounted on /rw;
> on-line resizing required old_desc_blocks = 1, new_desc_blocks = 1
> resize2fs: Permission denied to resize filesystem

Can you umount /rw (not sure that would work) or loop mount the private
partition and qvm-block it to a different VM, then resize it from in
there?

[RSS]

On Sun, 1 Jul 2018 11:27:57 -0000
"awokd" <aw...@danwin1210.me> wrote:

> On Sun, July 1, 2018 9:59 am, RSS wrote:
> >
> > root@q4home:/home/user# resize2fs /dev/xvdb
> > resize2fs 1.44.2 (14-May-2018) Filesystem at /dev/xvdb is mounted
> > on /rw; on-line resizing required old_desc_blocks = 1,
> > new_desc_blocks = 1 resize2fs: Permission denied to resize
> > filesystem
>
> Can you umount /rw (not sure that would work)

Strangely enough, I can umount /rw, but I cannot umount /home without
killing a bunch of "user" processes, and I have not found a subset of
user processes that releases /home for a umount *and* leaves X
standing. Nor am I able to get a root console on the VM using Xen from
dom0. (As I recall this was fairly trivial on Qubes 3.)

> or loop mount the private partition and qvm-block it to a different
> VM, then resize it from in there?

Something else that used to be trivial on Qubes 3. On 4, with a VM in a
thin pool, I can not find my way through the maze to even loop mount
the thing. Which is fine with me if things never break, but now I am
not so sure I want thin pools at all on my next install if they are are
such opaque tech. It sure seems to complicate picking up the pieces
after something breaks.

[awokd]

On Mon, July 2, 2018 8:51 am, RSS wrote:

>
> Strangely enough, I can umount /rw, but I cannot umount /home without
> killing a bunch of "user" processes, and I have not found a subset of user
> processes that releases /home for a umount *and* leaves X standing. Nor am
> I able to get a root console on the VM using Xen from
> dom0. (As I recall this was fairly trivial on Qubes 3.)

sudo xl console vmname

>> or loop mount the private partition and qvm-block it to a different VM,
>> then resize it from in there?
>
> Something else that used to be trivial on Qubes 3. On 4, with a VM in a
> thin pool, I can not find my way through the maze to even loop mount the
> thing. Which is fine with me if things never break, but now I am not so
> sure I want thin pools at all on my next install if they are are such
> opaque tech. It sure seems to complicate picking up the pieces after
> something breaks.

I know there's a way to do it, but I'm not finding it ATM either. An
alternative might be to run resize2fs in dom0 against the
/dev/mapper/[...]private partition you want to fix.

[RSS]

On Tue, 3 Jul 2018 12:01:00 -0000
"awokd" <aw...@danwin1210.me> wrote:

> On Mon, July 2, 2018 8:51 am, RSS wrote:
>
> > Strangely enough, I can umount /rw, but I cannot umount /home
> > without killing a bunch of "user" processes, and I have not found a
> > subset of user processes that releases /home for a umount *and*
> > leaves X standing. Nor am I able to get a root console on the VM
> > using Xen from dom0. (As I recall this was fairly trivial on Qubes
> > 3.)
>
> sudo xl console vmname

Thanks, that was the required medicine. Many moons ago I used this at
least once in Qubes 3.2, and spent a significant amount of time
searching for it this time, to no avail. I used a search engine and
grep'ed Qubes docs ("grep -ir xen * | grep -i console", among other
strings). Looking over those docs, I am not sure I can see a
straight-forward way to make it easier to find within the existing
structure.

(Well, except for an aesthetic nit-pick: IIRC the guidelines for Qubes
Docs mandate hard-wrapping one's markdown. It seems this is not
happening *a lot*, and that makes reading and using a text grep of the
doc's source much harder.)

For the archives, this is what I had to do once in possession of a root
console on the VM via xl console:

ps -ef | grep user

then killed all "user" processes in the VM

Some like systemd restarted themselves, but this was not an issue. Then

umount /rw
umount /home
umount /var/spool/cron

grep xvdb /etc/mtab

then produced null results, as /dev/xvdb was no longer mounted anywhere.

resize2fs /dev/xvdb

then prompted to first run fsck on /dev/xvdb. After this

resize2fs /dev/xvdb

just worked, the file system expanded to fill the volume, and
everything looks completely normal now after a reboot of the VM.

If someone believes there is a natural and desirable home for this
tidbit somewhere in the Qubes Docs, I would be happy to put in there.

[awokd]

On Fri, July 6, 2018 7:43 am, RSS wrote:

>
> resize2fs /dev/xvdb
>
> then prompted to first run fsck on /dev/xvdb. After this

Usually Qubes handles that resize2fs automatically; I suspect whatever
caused the need for fsck was preventing that.

> resize2fs /dev/xvdb
>
> just worked, the file system expanded to fill the volume, and everything
> looks completely normal now after a reboot of the VM.
>
> If someone believes there is a natural and desirable home for this
> tidbit somewhere in the Qubes Docs, I would be happy to put in there.

There is https://www.qubes-os.org/doc/resize-disk-image/, but I don't
think the process you described is intended. Does it happen every time you
resize a Linux AppVM's volumes (i.e. can you recreate it)? If so, might be
better to file an Issue instead. Anyways, glad it finally worked for you!

[RSS]

> > resize2fs /dev/xvdb
> > then prompted to first run fsck on /dev/xvdb. After this
>
> Usually Qubes handles that resize2fs automatically; I suspect whatever
> caused the need for fsck was preventing that.

I wish I could remember if anything odd happened at the time, that is
already several days ago and there has been a lot of water under the
bridge since then.

> > If someone believes there is a natural and desirable home for this
> > tidbit somewhere in the Qubes Docs, I would be happy to put in
> > there.
>
> There is https://www.qubes-os.org/doc/resize-disk-image/, but I don't
> think the process you described is intended. Does it happen every
> time you resize a Linux AppVM's volumes (i.e. can you recreate it)?
> If so, might be better to file an Issue instead. Anyways, glad it
> finally worked for you!

No, actually this is I believe the first time on either Qubes 3 or 4
that this has happened to me, and I tend to be resizing private storage
quite regularly. So it's basically a one-off that I have no way of
reproducing, and I don't recall many (any?) other people talking about
this on the list, so perhaps we can just let the mail archive take care
of it.

It might be worth noting here that after using Qubes 4 for several
months, I have had two significant glitches, this and another, and they
both were storage issues. Everything else has worked perfectly so far.

Thanks for the help!

[heinrich...@googlemail.com]

This exact problem happened to me recently in Qubes 4. The steps outlined here worked.

The commands for me were along the lines of: sudo xl console, pkill -u 1000, umount, e2fsck, resize2fs.

A HUGE THANK YOU FOR DOCUMENTING ALL OF THIS HERE! Excuse the yelling, this is my excitement about having this resolved :)

Additional note: Since I had snapd installed in the TemplateVM I needed to remove this too to get rid of its mounted file systems in the AppVM.