Qubes enters emergency mode on startup

[Richard]

Qubes (v 3.1.11-1) was running fine yesterday.  Today I upgraded the Templates and I had an issue where Firefox would not start in any of the AppVms.  So I decided to restart the machine and now Qubes enters emergency mode on startup.

I first receive a number of errors, including:

   NMI watchdog BUG: soft lockup - CPU#1 stuck for 23s! [migration/1:14] (seven similar messages)
   xhci_hcd 0000:03:00.0: can't setup: -110
   xhci_hcd 0000:03:00.0: init 0000:03:00.0 fail, -110

Then after I enter my disk password I receive the following after Reached Target Basic System:

   [59.235198] dracut-initqueue[318]: ln: failed to create symbolic link '/dev/resume' : File exists

   [246.024619] dracut-initqueue[318]: Warning: Could not boot

Then it proceeds until Reached target Basic System and shows:
   [59.235198] dracut-initqueue[318]: ln: failed to create symbolic link '/dev/resume' : File exists

   [246.024619] dracut-initqueue[318]: Warning: Could not boot

   [246.024619] dracut-initqueue[318]: Warning: /dev/mapper/qubes_dom0-root does not exist
   [246.024619] dracut-initqueue[318]: Warning: /dev/qubes_dom0/root does not exist

   Starting Dracut Emergency Shell

   Warning: /dev/mapper/qubes_dom0-root does not exist
   Warning: /dev/qubes_dom0/root does not exist

When I run journalctl I see some additional errors:

   dom0 systemd-modules-load[169]: Failed to find module
     'xen-blkback'
     'xen-netback'
     'evtchn'
     'gntdev'
     'netbk'
     'blkbk'
     'xen-scsibk'
     'usbbk'
     ' pckback'
     'xen-acpi-processor'
     'blktap2'

   dom0 systemd[1]: systemd-modules-load-service: main process exited, code=exited, status=1/FAILURE
   dom0 systemd[1]: Failed to start Load Kernal Modules
   dom0 systemd[1]: Unit systemd-modules-load.service entered failed state

Rebooting and trying different kernels (3.19.8-100.fc20.x86_64 and 4.1.13-8.pvops.qubes.x86_64) do not resolve the issue. 

Any ideas what is happening and how I can fix it?

Thanks,
Richard

[Tim W]

I might be wrong but almost all the time I have seen that error message on a machine reported to me ;NMI watchdog BUG: soft lockup - CPU#1 ....... It has been a hw failure issue.

Most of.the time it was powersupply related as its not givimg.the cpu enough voltage. Next same result but the capacitors next to the cpu on the mb went bad. On a latop it.could be a battery but usually I saw it on DTs but still psu related.

Maybe someone else knows of a software related cause. I. That area overclocki.g or messing with cpu bios settings if available is the only other area I could think of on that end.

Sorry I can not be more help orhave better news.

Do a search on that first part of.the erro up to the 23s and I thi k it should brimg up some stuff as I know I have seen other post getting this error. Maybe you will find sething Else could be the cause.

[Richard]

Thank you Tim for your suggestions.  I was seeing the NMI watchdog BUG error prior to yesterday, so I don't think it's the reason I now go to the Emergency Mode on Startup (but it could be that what ever was the issue, just decided to finally break).  I unfortunately have not been able to find anything in my searches of similar error messages to fix the issue.

Does anyone else have any suggestions on how I can fix the issue(s)?

Thanks,
Richard

[Tim W]

Is.this a laptop be uase if it is the battery could certai ly cause this and as you say just got bad enough to cause the fatal error.

[Richard H]

On Mon, Feb 1, 2016 at 1:23 PM, Tim W <timw...@gmail.com> wrote:

Is.this a laptop be uase if it is the battery could certai ly cause this and as you say just got bad enough to cause the fatal error.

No, it's a desktop that has no issues when I switch back to running Windows 8.1. 

[Richard H]

Anyone have any other ideas what the issue(s) might be, how I can fix them and get my desktop up and running again?
 

[Qubed One]

Richard:

I've had similiar issues multibooting on non-Qubes machines in the past.
The messages about "/dev/mapper/qubes_dom0-root does not exist"
(especially) and "Failed to find module" are telling you that your
filesystem isn't being decrypted or loaded for some reason. This might
be due to an incorrect disk decryption passphrase or an incorrect grub
configuration. That would also explain why choosing different kernels
doesn't resolve the issue. Unfortunately, that's all I can tell you
based on what you've posted.

PLEASE NOTE(!): This does not appear to have anything to do with Qubes
specifically, and a quick search for `/dev/mapper/* does not exist` (or
something similiar) would likely have yielded the same explanation. Read
about LVM (Logical Volume Management) for background info. This is why
having experience using linux in general is crucial, so that one is able
to determine where to start looking for answers (rather than having no
other choice but to ask and wait...).

Having said that, I fully understand that your computer worked, you did
some updating, restarted, then it didn't work. I fully understand how
this could appear to be a Qubes-related problem, and I'm absolutely not
trying to berate you. I'm only trying to emphasize the importance of
linux experience before learning Qubes. In fact, I'll go so far as to
say that, in my opinion, if someone was learning/using Qubes without
linux experience, they should always start with the assumption that any
issue is not Qubes-specific, instead of the other way around.

[Richard H]

Thank you for the additional guidance.  My research thus far has been pointing to a grub issue and as you pointed out, related to multibooting machines.  But my computer is NOT a multibooting machine.   When I installed Qubes, I unplugged my windows drive and then installed Qubes on a clean hard drive.  I only switch back to Windows when Qubes doesn't work.  I do this by unplugging the Qubes hard drive and plugging back the Windows drive.  So Qubes and Windows do not know that each other exists.  Furthermore, there are no other drives in the computer.

To further test the issue, I took the Qubes hard drive and plugged it into my other computer that is also running Qubes.  Again, it was the only hard drive connected.  I get the same issue of entering emergency mode. 

I'm convinced that the updates that I did through Qubes on January 31 somehow messed up a critical system file that locked me out of the system.  Having said this, I understand and don't disagree that this isn't specifically a Qubes issue, but I view (maybe incorrectly) that my operating system is Qubes and hope that I can turn to the Qubes community for assistance for issues that might not be strictly a Qubes issue but still affect Qubes.

All the best,

Richard

[Qubed One]

Richard H:

> Thank you for the additional guidance. My research thus far has been
> pointing to a grub issue and as you pointed out, related to multibooting
> machines. But my computer is NOT a multibooting machine. When I
> installed Qubes, I unplugged my windows drive and then installed Qubes on a
> clean hard drive. I only switch back to Windows when Qubes doesn't work.
> I do this by unplugging the Qubes hard drive and plugging back the Windows
> drive. So Qubes and Windows do not know that each other exists.
> Furthermore, there are no other drives in the computer.
>
> To further test the issue, I took the Qubes hard drive and plugged it into
> my other computer that is also running Qubes. Again, it was the only hard
> drive connected. I get the same issue of entering emergency mode.
>
> I'm convinced that the updates that I did through Qubes on January 31
> somehow messed up a critical system file that locked me out of the system.
> Having said this, I understand and don't disagree that this isn't
> specifically a Qubes issue, but I view (maybe incorrectly) that my
> operating system is Qubes and hope that I can turn to the Qubes community
> for assistance for issues that might not be strictly a Qubes issue but
> still affect Qubes.
>
> All the best,
> Richard
>

Absolutely. I didn't mean to imply that you shouldn't see turning to the
Qubes community for assistance as an option. I only meant that it might
not always be the fastest way to resolve a given issue. Give a man a
fish or teach a man to fish...

One other thought: assuming that it's only a grub issue and nothing
else, you could try booting from live media (doesn't have to be Qubes)
and mounting the Qubes hard disk, then chroot and regenerate that file:

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

(I've spent almost an hour unsuccessfully looking for exact step-by-step
instructions to point you to for the other steps, but most of what I've
found relates to Ubuntu, which isn't exactly the same. However, if you
can replace the `update-grub` command in Ubuntu instructions with the
command written above, that's really the main difference.)

[Richard H]

I very much appreciate you taking the time to try to find the step-by-step instructions.

I have tried to mounting the hard drive using Qubes I installed on my USB stick to help me with previous Qubes issues.  But this time as soon as I enter my disk encryption password I get a message that Qubes has started and then a Gdk-ERROR XDG_RUNTINE_DIR not set in the environment  Pane is dead error message.  ... Out of the frying pan into the fire, lol

It's getting late here, so I'll have to look at this further tomorrow.

Thanks again for your help.