Can’t boot after attempting to verify backup

[Ryan Tate]

On Friday I backed up my machine and attempted to verify the backup by launching qubes-backup-restore (GUI interface) and ticking the verify only box.

Is it possible this box does not work? I was a bit alarmed when qubes began unpacking the backup after verifying each qube, but thought perhaps it was checking the data.

This morning is my first time booting since then. It goes into emergency mode. Warnings include “dracut-initqueue timeout - starting timeout scripts” (many times) “Could not boot.” “/dev/mapper/qubes_dom0-root does not exist” “/dev/qubes_dom0/root does not exist”.

The only thing significant I can think I did to system Friday was the backup. It’s possible I also ran system update before or after backup and I vaguely recall a dom0 update but can’t recall if it was fRiday.

Thanks for any help. Worried it’s a wipe and restore situation :-(

[Ryan Tate]

PS not an April 1 joke sadly.

[Ryan Tate]

Looking over journalctl in emergency mode. It looks like it decrypts fine (“Reached target Encrypted Volumes. Reached Target System Initialization. Reached Target Basic System.”)

It seems like it sees many of my vms (many “inactive ‘/dev/qubes/dom0/vm-foo’ [10.0GB] inherit”)

But there is
“PARTIAL MODE. Incomplete logical volumes will be processed...
Check of pool qubes_dom0/pool100 failed (status: 1). Manual repair required!”

Then soon after come the warnings I posted in my first message.

[Ryan Tate]

Hmmm, looks an awful lot like this:

https://github.com/QubesOS/qubes-issues/issues/4479
https://groups.google.com/forum/#!msg/qubes-users/PR3-ZbZXo_0/wem9wbGTBgAJ
https://github.com/QubesOS/qubes-issues/issues/4857
:-\

Fwiw I am on stable, NOT testing (a question raised in that first thread).

So across these threads I am seeing ~4 other people publicly reporting same issue.

The solution is to reinstall (I guess?) but I see no reason to expect the issue would not resurface. This was just a routine update on a stable system that has been running Qubes for almost 2 years now. I could go to the pain of reinstalling from scratch, but from where I sit this seems to be some kind of bug.

[Chris Laprise]

I think its more related to this issue:

https://github.com/QubesOS/qubes-issues/issues/4791

Unfortunately qubes-backup-restore writes large amounts of data to disk
even when only verifying. If the dom0 root filesystem is growing past
the point the lvm pool can manage (i.e. the pool fills up, even if the
root filesystem thinks more space is available) then the system may
crash and be unable to boot normally.

Qubes is supposed to warn you when the lvm pool nears this full state,
but if you're not in front of your computer when it happens
(understandable when backing up or verifying) then it doesn't help.

As a remedy, you can run 'lvm' in the recovery shell then 'lvconvert
--repair' on qubes_dom0/pool00. But before doing the repair I would try
using vgscan, lvs and lvchange -ay to see which of the volumes can be
brought online (if any), and write down any errors you get in the process.

BTW, I had a similar issue with the lvm thinpool, but in my case the
metadata had been corrupted due to a system crash. Luckily lvconvert
--repair did work but IIRC I had to do something with the metadata
volume (maybe expand or duplicate it) that I can't remember exactly.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Ryan Tate]

> On Apr 1, 2019, at 11:48 AM, Chris Laprise <tas...@posteo.net> wrote:
>
> I think its more related to this issue:
>
> https://github.com/QubesOS/qubes-issues/issues/4791

I think you nailed it. The first time I ran the verification, I panicked when it said it was extracting data, as it seemed to have already “verified” various VMs, and I hit cancel. Then I ran again and kept my determination to finish, and it seemed to work and give a happy “Finished” message. (I did not see any lvm alerts, but as you note, I was probably away from the machine at various points.)

The issue above mentions that if a prior restore fails, and then you do another, you may run out of disk space. I think the equivalent happened to me, except instead of failing, it was cancelled (same result).

Right now I’m restoring my backup to a fresh Qubes install. At least I have my Qubes menu back, and fedora29, and my backup is about as fresh as one could hope.

Thanks for the help.