home directory is out of disk space

[Franz]

I am helping maintaining a Qubes system of a partner that is installed on an external SSD USB of 64GB. It is working flawlessy for more than a year.

Today, copying a large file to a userVM got the message home directory is out of disk space. I was not able to stop the copy and possibly it was too late to stop it. Qubes manager freezed and had to restart the system.

Booting worked fine, but after inserting the password to unlock the screen got this message:

The following installation problem was detected  while trying to start KDE:

$ home directory (/home/n) is out of disk space. KDE is unable to start.

Clicking on OK I get the following other message:

could not start ksmserver check your installation.

Trying to use failsafe KDE or xfce  gives the same results

So it seems that filling the userVM with the large file resulted in draining memory from Dom0. Should be so, because KDE is based on Dom0. Is that correct?

If this is correct it seems a bug because it makes no sense that problems of a single userVM result in a crash of the whole system beyond easy repair.

Regarding repair ... Is it possible to login to Dom0 terminal? I looked for this option, but could not find it.  My idea is to use Dom0 terminal to access userVM terminal (qvm-run -a userVM terminal) and try to delete what remains of the large file transfer. Hoping this solves my problem.

Best

Franz

[Marek Marczykowski-Górecki]

On 28.11.2013 23:00, Franz wrote:
> I am helping maintaining a Qubes system of a partner that is installed on
> an external SSD USB of 64GB. It is working flawlessy for more than a year.
>
> Today, copying a large file to a userVM got the message home directory is
> out of disk space. I was not able to stop the copy and possibly it was too
> late to stop it. Qubes manager freezed and had to restart the system.
>
> Booting worked fine, but after inserting the password to unlock the screen
> got this message:
>
> The following installation problem was detected while trying to start KDE:
> $ home directory (/home/n) is out of disk space. KDE is unable to start.
>
> Clicking on OK I get the following other message:
>
> could not start ksmserver check your installation.
>
> Trying to use failsafe KDE or xfce gives the same results
>
> So it seems that filling the userVM with the large file resulted in
> draining memory from Dom0. Should be so, because KDE is based on Dom0. Is
> that correct?

All the VM disks are in dom0 filesystem, so yes.

> If this is correct it seems a bug because it makes no sense that problems
> of a single userVM result in a crash of the whole system beyond easy repair.

You have limits for each VM (2GB by default). Normally VM uses only as much
disk space as files inside, but if you copy some big file - the VM will grow,
up to the limit. If you have lesser disk space than limits for all VMs, then
you can run out of space...

> Regarding repair ... Is it possible to login to Dom0 terminal? I looked for
> this option, but could not find it. My idea is to use Dom0 terminal to
> access userVM terminal (qvm-run -a userVM terminal) and try to delete what
> remains of the large file transfer. Hoping this solves my problem.

You should be able access text console (Alt-Ctrl-F2), then start the VM
(qvm-start) and access its console with "sudo xl console <vmname>".

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Franz]

On Thu, Nov 28, 2013 at 8:21 PM, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:

On 28.11.2013 23:00, Franz wrote:
> I am helping maintaining a Qubes system of a partner that is installed on
> an external SSD USB of 64GB. It is working flawlessy for more than a year.
>
> Today, copying a large file to a userVM got the message home directory is
> out of disk space. I was not able to stop the copy and possibly it was too
> late to stop it. Qubes manager freezed and had to restart the system.
>
> Booting worked fine, but after inserting the password to unlock the screen
> got this message:
>
> The following installation problem was detected  while trying to start KDE:
> $ home directory (/home/n) is out of disk space. KDE is unable to start.
>
> Clicking on OK I get the following other message:
>
> could not start ksmserver check your installation.
>
> Trying to use failsafe KDE or xfce  gives the same results
>
> So it seems that filling the userVM with the large file resulted in
> draining memory from Dom0. Should be so, because KDE is based on Dom0. Is
> that correct?

All the VM disks are in dom0 filesystem, so yes.

> If this is correct it seems a bug because it makes no sense that problems
> of a single userVM result in a crash of the whole system beyond easy repair.

You have limits for each VM (2GB by default). Normally VM uses only as much
disk space as files inside, but if you copy some big file - the VM will grow,
up to the limit. If you have lesser disk space than limits for all VMs, then
you can run out of space...

Ahhhh I increased a lot the userVM limit (do not remember how much) so this is why drained the system memory.  But this setting is too dangerous! One can very easily assign memory as needed to a userVM without being aware that the whole system is going to crash. Also how may one know how much memory it is possible to assign to a userVM, without risking a system crash?

I mean memory management is something basic that ordinary users need to use for normal work. This is not advanced stuff. It would be very helpful to have some automatic system that calculates how much total memory is available and prevents the user to assign memory to userVM beyond a safe limit. I encourage everybody able to do that to think a way to solve it.

But now, without an automated system, do we have a command or something to know how much memory is left free for the system?

> Regarding repair ... Is it possible to login to Dom0 terminal? I looked for
> this option, but could not find it.  My idea is to use Dom0 terminal to
> access userVM terminal (qvm-run -a userVM terminal) and try to delete what
> remains of the large file transfer. Hoping this solves my problem.

You should be able access text console (Alt-Ctrl-F2), then start the VM
(qvm-start) and access its console with "sudo xl console <vmname>".

Thanks Marek, but I am getting errors with qvm-start userVM:

Traceback (most recent call last):

File "/bin/qvm-start", line 126 in<module> main()

File "bin/qvm-start",line 79  in main qvm_collection.load()

etc 8 more lines

File  "parser.pxi", line 616 in lxml.etree._reiseParseError (scr/lxml/lxml.etree.c:84192

lxml:etree.XMLSyntaxError: Document is empty, line 1 column 1

Best

Franz

[Hakisho Nukama]

Is someone interested in writing a patch for qubes-manager to calculate
the amount of used disk space, when all AppVMs use all their
assigned space and warn the user about over-provisioning?

[Marek Marczykowski-Górecki]

Just to clarify: "memory" != "disk space".
In any case we'll love to accept a patch for qubes manager, which will add
some warning about such situation (as Hakisho Nakuma just described in
previous message).

>>> Regarding repair ... Is it possible to login to Dom0 terminal? I looked
>> for
>>> this option, but could not find it. My idea is to use Dom0 terminal to
>>> access userVM terminal (qvm-run -a userVM terminal) and try to delete
>> what
>>> remains of the large file transfer. Hoping this solves my problem.
>>
>> You should be able access text console (Alt-Ctrl-F2), then start the VM
>> (qvm-start) and access its console with "sudo xl console <vmname>".
>>
>> Thanks Marek, but I am getting errors with qvm-start userVM:
> Traceback (most recent call last):
> File "/bin/qvm-start", line 126 in<module> main()
> File "bin/qvm-start",line 79 in main qvm_collection.load()
> etc 8 more lines
> File "parser.pxi", line 616 in lxml.etree._reiseParseError
> (scr/lxml/lxml.etree.c:84192
> lxml:etree.XMLSyntaxError: Document is empty, line 1 column 1

Not good...
Check the size of /var/lib/qubes/qubes.xml. Its zero, you'll need to use one
of the file backup (stored in /var/lib/qubes/backup), hopefully you have the
current data there. Find the most recent one and place in
/var/lib/qubes/qubes.xml instead of empty file.
In any case you'll need some disk space to start the VM. Check "df" output if
you have some. If not, some hints how to free some disk space:
1. Clean yum cache:
sudo yum clean all
2. Decrease filesystem safety margin (5% by default):
sudo tune2fs -m 4 /dev/mapper/vg_dom0-lv_root
3. Remove some unneeded files in dom0 home (if you have one, most likely no)

[Axon]

Related issue: Why is the *minimum* 2 GB? Qubes VM Manager does let me
set the maximum AppVM "private storage max. size" lower than 2 GB (2048
MB). Is there a reason that the user should not be allowed to choose a
lower value?

Note that, with this restriction in place, the patch you're asking for
might constrain the user by the *number* of AppVMs instead of the
user-desired sum of all "private storage max. size" values.

I.e., if

(number of AppVMs * 2 GB) > (sum of all user-desired "private storage
max. size"),

then the user will not be able to use as much of their disk as they
wish. This could happen to users who (for whatever reason) have a large
number of small AppVMs.

Of course, if there's some danger associated with having <2 GB, then the
restriction may be appropriate.

[Axon]

Ah, I see now that this is just a consequence of the more general fact
that you cannot use Qubes VM Manager to decrease the value of "private
storage max. size" once it has been set.

This page[1] explains how to grow the disk image size, but is there any
way to shrink it? If a user discovers that they have allocated too much
space across all their AppVMs (i.e., more than the size of the physical
disk), then how can they reduce the sizes to more appropriate values? Is
it necessary to delete and re-make the AppVMs?

[1] http://qubes-os.org/trac/wiki/ResizeDiskImage

[Marek Marczykowski-Górecki]

Yes, this is why it should be only a warning, not a strict requirement.

>>
>> I.e., if
>>
>> (number of AppVMs * 2 GB) > (sum of all user-desired "private storage
>> max. size"),
>>
>> then the user will not be able to use as much of their disk as they
>> wish. This could happen to users who (for whatever reason) have a large
>> number of small AppVMs.
>>
>> Of course, if there's some danger associated with having <2 GB, then the
>> restriction may be appropriate.

This is just some reasonable default value.

> Ah, I see now that this is just a consequence of the more general fact
> that you cannot use Qubes VM Manager to decrease the value of "private
> storage max. size" once it has been set.
>
> This page[1] explains how to grow the disk image size, but is there any
> way to shrink it?

No, currently we don't allow to decrease private storage size, whatever
current value is. This is just to not damage the data during filesystem shrink
- there is a lot of things that can goes wrong, starting with the fact that
ext3/4 must be unmounted for such operation (and we don't want mount VM
filesystem in dom0 for security reasons).

> If a user discovers that they have allocated too much
> space across all their AppVMs (i.e., more than the size of the physical
> disk), then how can they reduce the sizes to more appropriate values? Is
> it necessary to delete and re-make the AppVMs?

Yes, currently this is the only option.

> [1] http://qubes-os.org/trac/wiki/ResizeDiskImage

[Axon]

Good point.

>>>
>>> I.e., if
>>>
>>> (number of AppVMs * 2 GB) > (sum of all user-desired "private storage
>>> max. size"),
>>>
>>> then the user will not be able to use as much of their disk as they
>>> wish. This could happen to users who (for whatever reason) have a large
>>> number of small AppVMs.
>>>
>>> Of course, if there's some danger associated with having <2 GB, then the
>>> restriction may be appropriate.
>
> This is just some reasonable default value.

Yes, it does seem to be a good balance between convenience (not having
to set or increase the value every time a new AppVM is created) and
conservation (2 GB is not terribly much given the average disk size
these days, though I suppose people with small SSDs or installing on USB
sticks might need to be a little careful).

>
>> Ah, I see now that this is just a consequence of the more general fact
>> that you cannot use Qubes VM Manager to decrease the value of "private
>> storage max. size" once it has been set.
>>
>> This page[1] explains how to grow the disk image size, but is there any
>> way to shrink it?
>
> No, currently we don't allow to decrease private storage size, whatever
> current value is. This is just to not damage the data during filesystem shrink
> - there is a lot of things that can goes wrong, starting with the fact that
> ext3/4 must be unmounted for such operation (and we don't want mount VM
> filesystem in dom0 for security reasons).
>

That makes sense. Thanks for the explanation.

>> If a user discovers that they have allocated too much
>> space across all their AppVMs (i.e., more than the size of the physical
>> disk), then how can they reduce the sizes to more appropriate values? Is
>> it necessary to delete and re-make the AppVMs?
>
> Yes, currently this is the only option.

OK. In that case, I think I'll just wait until I run out of space in an
AppVM before increasing the value (instead of increasing it at the time
of AppVM creation and guessing how much storage space I'll eventually
use), as this seems to be the safest way. If others agree, I (or someone
else) could add this as a "user tip" to the wiki.

>
>> [1] http://qubes-os.org/trac/wiki/ResizeDiskImage
>>
>
>

[Franz]

this gives:
tune2fs: file or directory non existent during opening of /dev/mapper/vg_dom0-lv_root

Would it be easier to fully delete some not much used VM?

best

Franz

[Marek Marczykowski-Górecki]

Yes, this is also an option. If qvm-remove doesn't work and you don't have
space to restore qubes.xml file, you can manually delete some VM private.img
file, then cleanup the rest with qvm-remove - when it will be working again.

[Olivier Médoc]

On 11/29/13 00:50, Hakisho Nukama wrote:
> On Thu, Nov 28, 2013 at 10:21 PM, Marek Marczykowski-G�recki

>> Marek Marczykowski-G�recki

>> Invisible Things Lab
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>>
> Is someone interested in writing a patch for qubes-manager to calculate
> the amount of used disk space, when all AppVMs use all their
> assigned space and warn the user about over-provisioning?
>

I've got exactly the same issue two days ago (qubes.xml deleted ...). In
AppVMs, some tools warn about low disk space which allow to stop current
actions that are consuming the disk space. I found that nothing like
this is present on dom0 by default.

Maybe the first point would be to warn the user when the dom0 disk space
is low (<4 or 8GB), independently from the over provisionning issue ? It
probably only requires to install some tools ?

[Olivier Médoc]

The "gnome-settings-daemon" has a plugin called "housekeeping". This is
probably causing the "Low disc space" warning in AppVMs.

Maybe somebody know a good tool independent from gnome/kde that uses the
standard notification feature ? (other than qubes-manager ?)

[Marek Marczykowski-Górecki]

On 29.11.2013 16:25, Olivier Médoc wrote:

> On 11/29/13 13:17, Olivier Médoc wrote:
>> I've got exactly the same issue two days ago (qubes.xml deleted ...). In
>> AppVMs, some tools warn about low disk space which allow to stop current
>> actions that are consuming the disk space. I found that nothing like this is
>> present on dom0 by default.

Actually qubes.xml problem can be easily fixed by not overriding existing
file, but save new one as temporary file and then move onto qubes.xml name.
But this doesn't solve general problem here. I though there is some KDE plugin
for disk space installed, perhaps just not enabled?

>> Maybe the first point would be to warn the user when the dom0 disk space is
>> low (<4 or 8GB), independently from the over provisionning issue ? It
>> probably only requires to install some tools ?
>
>
> The "gnome-settings-daemon" has a plugin called "housekeeping". This is
> probably causing the "Low disc space" warning in AppVMs.

Ah, so perhaps disabling g-s-d (which has been done to solve some keyboard
layout issues) isn't such a good idea?

> Maybe somebody know a good tool independent from gnome/kde that uses the
> standard notification feature ? (other than qubes-manager ?)

--
Best Regards,
Marek Marczykowski-Górecki

[Franz]

This last one worked Marek, Qubes is running again, and we can recover all data. So many many thanks.

I prepared a wiki page to keep trace of what worked. I had to add a new section that I called Troubleshooting to contain it:
https://qubes-os.org/trac/wiki/OutOfmemory

If there is something wrong or lacking pls let me know

best

Franz

[Olivier Médoc]

On 11/29/13 16:55, Marek Marczykowski-Górecki wrote:
> On 29.11.2013 16:25, Olivier Médoc wrote:
>> On 11/29/13 13:17, Olivier Médoc wrote:
>>> I've got exactly the same issue two days ago (qubes.xml deleted ...). In
>>> AppVMs, some tools warn about low disk space which allow to stop current
>>> actions that are consuming the disk space. I found that nothing like this is
>>> present on dom0 by default.
> Actually qubes.xml problem can be easily fixed by not overriding existing
> file, but save new one as temporary file and then move onto qubes.xml name.
> But this doesn't solve general problem here. I though there is some KDE plugin
> for disk space installed, perhaps just not enabled?
>
>>> Maybe the first point would be to warn the user when the dom0 disk space is
>>> low (<4 or 8GB), independently from the over provisionning issue ? It
>>> probably only requires to install some tools ?
>>
>> The "gnome-settings-daemon" has a plugin called "housekeeping". This is
>> probably causing the "Low disc space" warning in AppVMs.
> Ah, so perhaps disabling g-s-d (which has been done to solve some keyboard
> layout issues) isn't such a good idea?

If I remember it was possible to only disable to keyboard plugin, not
the whole gsd ?

[Axon]

On 11/29/13 07:55, Marek Marczykowski-Górecki wrote:
> On 29.11.2013 16:25, Olivier Médoc wrote:
>> On 11/29/13 13:17, Olivier Médoc wrote:
>>> I've got exactly the same issue two days ago (qubes.xml deleted ...). In
>>> AppVMs, some tools warn about low disk space which allow to stop current
>>> actions that are consuming the disk space. I found that nothing like this is
>>> present on dom0 by default.
>
> Actually qubes.xml problem can be easily fixed by not overriding existing
> file, but save new one as temporary file and then move onto qubes.xml name.
> But this doesn't solve general problem here. I though there is some KDE plugin
> for disk space installed, perhaps just not enabled?
>

There's a nice little KDE widget called "Hard Disk Space Usage" which is
already included in our dom0. I've been using it for a while. For me, it
shows the usage of both boot and root. It can be dragged to the desktop
or system tray just like any other widget.

[cprise]

On 11/29/13 10:55, Marek Marczykowski-Górecki wrote:
> On 29.11.2013 16:25, Olivier Médoc wrote:
>
>> The "gnome-settings-daemon" has a plugin called "housekeeping". This is
>> probably causing the "Low disc space" warning in AppVMs.
> Ah, so perhaps disabling g-s-d (which has been done to solve some keyboard
> layout issues) isn't such a good idea?
>

Disabling it seems to be causing another problem for me as well. An RSS
app called Liferea will load with no access to its settings... the
window will appear very small and in its misconfigured state will delete
all collected RSS data when an update is performed. If Liferea was
started from CLI, I see lots of warnings about gnome-settings-deamon not
responding. When I see the window size symptom, I know I have to reboot
the appvm before Liferea will work correctly again.

Gnome-terminal also has a problem when the appvm is in this state:
Opening additional tabs causes the window to shrink to a very small
size- but I can resize the window manually and continue using it.

I don't know why a reboot would fix the problem; AFAIK the GSD is still
disabled for the sake of preserving the global keyboard layout.

[yon...@gmail.com]

I'm had a very similar issue on 3.1

My harddrive was completely full. I tried to remove some files from inside the appVMs but that didn't clear any space on dom0 So I rebooted the system.
After reboot one AppVM wouldn't start. I got the
“Document is empty, line 1, column 1” Error.
If I tried to go into the vm settings a pop up with the same error shows and text below -
"This is most likely a bug in the Qubes Manager"

I looked into qubes.xml and it wasn't empty ( from size it looked similar to the other files in backup ) Even though I tried to replace it with some older versions from the backup but non worked.

In the end I noticed the appVM folder had an empty firewall.xml file
removing the file solved the start and setting issue
I had to recover my firewall rules but that wasn't a big issue