VM's fail to start after fixing chock-full LVM thinpool

[Patrick]

Hi everybody,

I have gotten myself in quite some mess, and I hope somebody can help me out! I am running 4.0RC2 and my system borked after I had copied too much data to my VM's which resulted in none of my VM's being able to write away data anymore, since all are part of an LVM thinpool (which is the standard configuration with this release). After this, my troubles started... After rebooting the system I found out none of the VM's would start again, and I figured it must have been because of having to write away some temporary data for which there was no room anymore (I did not do any more research at this time).

Instead of deleting some data, I opted to expand the LVM vgroup and tpool to a second SSD in my laptop. After a bit of playing around, I got this working properly. Now, after rebooting the system again, I acknowledged there was more than enough space for all LV's (VM logicall volumes) on my system. Yay!

However... I immediatelyy found out VM's still did not start on my system. After looking in journalctl and systemctl, I found out qubesd would not start: "failed to start Qubes OS daemon". I attached text files of the error outputs from both commands for all your scrutinizing eyes. Does anybody got any ideas on what is going wrong?

Best regards,
Patrick

[Tom Zander]

On Thursday, 7 December 2017 01:53:40 CET Patrick wrote:
> However... I immediatelyy found out VM's still did not start on my system.
> After looking in journalctl and systemctl, I found out qubesd would not
> start: "failed to start Qubes OS daemon". I attached text files of the
> error outputs from both commands for all your scrutinizing eyes. Does
> anybody got any ideas on what is going wrong?

I would hope qubesd logs somewhere else as well, as these files show nothing
of use. Just that it failed.

Try
qvm-run -p -u root ‘ls /usr/log’
and similar commands to check if there are more logs.

--
Tom Zander
Blog: https://zander.github.io
Vlog: https://vimeo.com/channels/tomscryptochannel

[Patrick]

Hi Tom,

> I would hope qubesd logs somewhere else as well, as these files show nothing
> of use. Just that it failed.
>
> Try
> qvm-run -p -u root ‘ls /usr/log’
> and similar commands to check if there are more logs.

These commands (almost?) all depend on qubesd it seems, therefore not very helpful. Here is the output of your suggestion; all other qvm commands have similar output

root@dom0 patrick]# qvm-run -p -u root Personal-Administration 'ls /usr/log'
Traceback (most recent call last):
File "/bin/qvm-run", line 9, in <module>
load_entry_point('qubesadmin==4.0.9', 'console_scripts', 'qvm-run')()
File "/usr/lib/python3.5/site-packages/qubesadmin/tools/qvm_run.py", line 129, in main
args = parser.parse_args(args, app=app)
File "/usr/lib/python3.5/site-packages/qubesadmin/tools/__init__.py", line 387, in parse_args
action.parse_qubes_app(self, namespace)
File "/usr/lib/python3.5/site-packages/qubesadmin/tools/__init__.py", line 170, in parse_qubes_app
namespace.domains += [app.domains[vm_name]]
File "/usr/lib/python3.5/site-packages/qubesadmin/app.py", line 85, in __getitem__
if not self.app.blind_mode and item not in self:
File "/usr/lib/python3.5/site-packages/qubesadmin/app.py", line 99, in __contains__
self.refresh_cache()
File "/usr/lib/python3.5/site-packages/qubesadmin/app.py", line 59, in refresh_cache
'admin.vm.List'
File "/usr/lib/python3.5/site-packages/qubesadmin/app.py", line 464, in qubesd_call
client_socket.connect(qubesadmin.config.QUBESD_SOCKET)
FileNotFoundError: [Errno 2] No such file or directory


Best regards,
Patrick

[Patrick]

Correction: the last line is not actually part of the error message! (my mistake...)

[Patrick]

I found the problem! :-D My /var/lib/qubes/qubes.xml file was corrupted, so it could not be parsed correctly by qubesd. I restored a previous version from /var/lib/qubes/backup and now I am back in business! Thanks anyway for checking out my problem. :-)

Best regards,
Patrick

[Tom Zander]

On Friday, 8 December 2017 01:05:32 CET Patrick wrote:
> I found the problem! My /var/lib/qubes/qubes.xml file was corrupted, so

> it could not be parsed correctly by qubesd. I restored a previous version
> from /var/lib/qubes/backup and now I am back in business! Thanks anyway
> for checking out my problem. :-)

Thanks for reporting this!

This looks like a show-stopper bug to me.
The system should never be able to corrupt a critical file like that due to
disk-full.

I reported it to the qubes devs;
https://github.com/QubesOS/qubes-issues/issues/3376

[Patrick]

Hi Tom,

> This looks like a show-stopper bug to me.
> The system should never be able to corrupt a critical file like that due to
> disk-full.

This problem could be averted by pulling dom0 out of tpool. Also, using "df" to check how full your partition is, will not necessarily say how much data you can still write on the partition if the disk itself is already quite full. So you should always use "lvs" (for example) on dom0 to check how much space is actually left available on tpool00 itself. The documentation about low disk space (deceptively called "Home directory is out of disk space error" and with an even more deceptive file name and page title "out-of-memory") on the Qubes website should be changed for 4.0 to reflect this.

> I reported it to the qubes devs

Thsnks a lot! :-)

Regards,
Patrick

[dimlev]

Hi Patrick,

> I found the problem! :-D My /var/lib/qubes/qubes.xml file was corrupted

Do you remember chaning any vm settings when the disk was full?