qubes_dom0-root fsck failed with error code 4

[qber...@gmail.com]

Hello everyone. I'm running qubes 3.2 and I love it.

Today I messed up and it's got my stomach in knots. I switched to my windows ssd for a legacy task that i have yet to migrate to linux. After I finished, i told the laptop to shut down. Then, perhaps too quickly, i closed the lid, flipped it over, pulled off the cover, took out the windows ssd and put in the qubes ssd. When i flipped it back over, to my horror i realized the laptop was still powered on.

Immediately i fear i have either shorted the ssd or corrupted it.

When I boot, it reached the password screen. After I enter my password it enters emergency mode. As instructed, i ran journalctl. The only errors are near the end, where it says:
/dev/mapper/qubes_dom0-root contains a file system with errors, check forced.
Then fsck failed with error code 4, and then a file system check failed on qubes_dom0-root. (See attached picture)

I ran fsck /dev/mapper/qubes_dom0-root, but said no to every prompt, as I have no experience in this situation and I fear making things worse. But I did want to see what it had to say.

Next I boot to the qubes install USB stick and choose Rescue a Qubes system.
It reaches a point where Rescue asks me to choose:
1. find and mount the installed OS to /mnt/sysimage,
2. Mount it as read-only,
3. skip to shell, or
4. quit.

I chose to continue. I enter my password when prompted, then it tells me that it has been mounted under /mnt/sysimage, and that I am able to mount it with chroot /mnt/sysimage.

I ran chroot /mnt/sysimage, and it said:
bash: /tmp/.bashrc: Structure needs cleaning

I'm not sure what to do here but I assume I would be able to do something if I only knew what.

It appears I have two potential courses of action: run the fsck, say yes to everything and hope for the best, or perhaps attempt something else from the chroot. However, I am new to linux. Usually I'm fairly cautious and don't get myself into these sorts of situations, but alas, here I am.

I'm terrified of making things worse. I would rather wait to hear from the community on how to proceed. I would be grateful for any help in rescuing my data. I promise to do a full backup immediately after getting it running again.
I am at your mercy.

[qber...@gmail.com]

I forgot to attach the relevant photo. I apologize for the size of the photo

[qber...@gmail.com]

I've attached two screenshots, showing the prompts given to me by fsck, to which I said no, until I have guidance.

[qber...@gmail.com]

I went ahead and did fsck, saying yes to all prompts. It went a little crazy for a while, then finished. The shutdown command doesnt exist, so i lowered it off.
The next boot progressed till it sat at a umts something or other. Another poweroff, and the next boot finished.
I'm logged in now, but no VMs will start.

[qber...@gmail.com]

The Guest log for the VMs say the filesystems are read-only.
The fedora template also doesnt boot.

I've rebooted once again to the usb install stick and try to rescue. Again I chose option 1 to mount the partition, and when i run chmod /mnt/sysimage, i dont get any errors.
From here I woud like to run a checkdisk or something like in windows. Am I on the right track?

If I coud just get these VMs to boot I could get my data off of them and do a full reinstall.
Any help would be appreciated.

[qber...@gmail.com]

I'm unable to get the guest.log for the VMs off of the laptop, so I've attached a screenshot of the log of the most critical VM that i need to rescue.
It appears it is going into emergency mode, but I can't run a terminal or anything as qrexec never connects.
The qrexec.log is empty.

Is there any way get this to boot, or to extract a few files from within it to dom0?

[Chris Laprise]

Have you tried mounting an external drive and then running qvm-backup?

--

Chris Laprise, tas...@openmailbox.org
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Unman]

You dont *need* to get the VM to boot to get at the data.
All the private data in a template based VM is stored in the private.img.

You can get at the data just by mounting the private.img, and looking in
/home.

If I were you I would attach a backup drive, and copy off the files
from /var/lib/qubes/appvms. I would do this in preference to using
qvm-backup, (although once done you can try the backup route also.)

First secure the data.
Then try to get the systems running - or dont bother, and just reinstall
from scratch. If you haven't tried the backup just mount your backup
medium in a qube, mount the private.img files and copy the data cross to
your newly created qubes.

But get the data off the system first.

Good luck

unman

[qber...@gmail.com]

I was able to back up the VMs and restore them to another qubes install successfully!

I have since reinstalled qubes and restored everything, I'm back in business.

Thank you for the replies, and I will learn to use the private.img as well so I can be more prepared in the future.