How to recover Qubes when keyboard / mice is dysfunctional due to USB qube setup issues?
Patrick Schleizer
> Caution: By assigning a USB controller to a USB qube, it will no
longer be available to dom0. This can make your system unusable if, for
example, you have only one USB controller, and you are running Qubes off
of a USB drive.
How can one recover from such a situation if there is no PS2
keyboard/mice available?
I guess... Unless there is a better way...? Boot the system using from
an external disk using a USB recovery operating system... Then modify
the local disk (with broken Qubes)... Then do what?
Cheers,
Patrick
cooloutac
ya that. exactly.
cooloutac
that would be the only way I would know of.
cooloutac
sorry i misunderstood. you could use the qubes keyboard proxy. or unhide it from dom0. think they are both explained in the docs there, but don't think either are recommended but if you have no choice.
Patrick Schleizer
when the disk is not booted (for recovery booted from USB), the gui
cannot be used since it refers to the USB booted and not internal disk
supposed to be recovered.
To undo it some file on the internal disk needs to be modified. Which
files needs what modification?
Yethal
Ron Hunter-Duvar
keyboard & mouse, and recovering from it is how I eventually made my
system unbootable. That plus intervening updates and other tweaks.
So not sure I have much to add on how to do it properly. But maybe
serves as a cautionary tale.
I used an old ps/2 keyboard to get control of the system again (no
mouse), and get the USB controller I had my main keyboard and my mouse
on assigned back to dom0 again (thankfully I have several controllers on
this motherboard, with keyboard, mouse and nothing else on one of them).
I got it working, but I think I messed up something on sys-usb. From
time to time after that I would get a kernel panic starting sys-usb
during boot. Then after a recent dom0 update, I got a "non-system disk
or disk error" BIOS error and that was it. Couldn't get it properly
recovered.
So I booted from a Ubuntu live USB, mounted the Qubes partitions, copied
everything off to a backup USB hard drive, then did a clean reinstall of
Qubes. After the reinstall, I've been more careful in setting up
sys-usb, and it's working fine so far.
Ron
Patrick Schleizer
Isn't it difficult to run grub from a chrooted disk without messing up
bootloader of the disk that was booted or messing up which devices grub
is referring to?
> Remove rd.qubeshideallusb parameter from grub [...]
If that's all... Then why not just do this during normal system boot at
grub?
Even if it's not hidden all the time from dom0... Won't the
keyboard/mice USB controller be quickly assigned to sys-usb, detached
from dom0 and still leave an unbootable system?
As I understand the documentation rd.qubeshideallusb is "only" for
improved security. One can render its system unusable even without using
rd.qubeshideallusb.
Unman
Interrupt the boot process and change the parameters to remove rd.qubeshideallusb, and add
rd.break=cleanup.
You'll be prompted to decrypt disks and then drop to shell.
The root filesystem will be mounted ro at /sysroot.
umount /sysroot.
Mount it rw somewhere else (e.g /mnt - you'll have to create that)
cd /mnt/var/lib/qubes/servicevms
mv sys-usb sys-usb.bak
umount /mnt
Remount /dev/mapper/blah on /sysroot
exit from the rescue shell
The system should then continue to boot - sys-usb wont be able to start
and so cant claim the usb devices - as you changed the boot parameters
they'll be available in dom0.
I don't currently have a USB keyboard with me so cant check this.
Let me know if it works. It should.
unman
Patrick Schleizer
> cd /mnt/var/lib/qubes/servicevms mv sys-usb sys-usb.bak
Please have a look at the following instructions, modified what you
wrote. I hope we could simplify/clarify for novice users and add this to
the Qubes documentation.
You should be able to fix this in grub: something like this -
Interrupt the boot process and change the parameters to remove
rd.qubeshideallusb, and add
rd.break=cleanup.
You'll be prompted to decrypt disks and then drop to shell.
The root filesystem will be mounted ro at /sysroot.
umount /sysroot
mount /dev/mapper/qubes_dom0-root /mnt/disk
chroot /mnt/disk
systemctl disable qubes-vm@sys-usb
exit
sudo umount /mnt/disk
reboot
Patrick Schleizer
> cd /mnt/var/lib/qubes/servicevms mv sys-usb sys-usb.bak
Please have a look at the following instructions, modified what you
wrote. I hope we could simplify/clarify for novice users and add this to
the Qubes documentation.
Interrupt the boot process and change the parameters to remove
rd.qubeshideallusb, and add
rd.break=cleanup.
You'll be prompted to decrypt disks and then drop to shell.
The root filesystem will be mounted ro at /sysroot.
mkdir /mnt/disk
mount /dev/mapper/qubes_dom0-root /mnt/disk
chroot /mnt/disk
yura...@gmail.com
A manual method, 3 steps to ensure a fix:
*) Modify the AppVM autostart in /var/lib/qubes/qubes.xml quick, easy and dirty.
*) Modify Grub or EFI settings (plenty of guides around), so Dom0 can see the USB controller. You either use EFI or Grub, not both at the same time. First identify which you use, before you can change the settings to unblock.
*) Undoing the sys-usb commands.
From my understanding, the reason the two commands
qubesctl top.enable qvm.sys-usb and qubesctl state.highstate
are so hard to undo, is because it's meant to be hack-proof, and it will cause the sys-usb to be automatically restoed again to its default state. Further reading if you're curious: https://github.com/QubesOS/qubes-issues/issues/2157
In order to easily undo it, I believe you may be able to edit /srv/salt/_tops/base/topd.top
Remove the sys-usb line in the file, don't change anything else.
Be warned, I haven't done this before, it's hypothetical whether it works.
To my knowledge, the first command not only enables the preconfigured sys-usb, it also push the sys-usb into the salt file (the top part of the command. The second command apparently enables the salt feature for whichever VM is in the list (default enabled in Qubes 4). If sys-usb is removed from the file, it should supposedly be outside salts protection area.
Now try restart, if everything went well (hopefully), it should be back to normal, and USB should be found by Dom0 again. You don't need to disable sys-usb, as long as it doesn't autostart (which we at this point fixed).
Keep in mind that the /var/lib/qubes/qubes.xml file may have changed in Qubes 4 compared to Qubes 3.2. Also SALT should be by default enabled in Qubes 4, so you might encounter this kind of memory protection thing in other areas in Qubes 4 in the future.
Everything here is edited through plain text files, you do not need chroot or anything of the sorts. As long as you can open up your encrypted drives, and gain read/Write access, have a text editor you can use, you're basically set.
Keep in mind, I'm not an expert. Be sure to make backups or at the very least backup/memorize what you did to the single file you changed, so you can reverse it, should it be needed.
Otto Kratik
In my case I had assigned a network card/controller to the Sys-Net VM, which Qubes hated so much compatibility-wise that it subsequently refused to boot altogether, simply because attempt to initialize that net card caused such catastrophic failure on startup that the entire OS became non-startable.
However, since devices assigned to VM's have specific serial identifiers that Qubes is looking for on bootup to assign to chosen VM, and since Qubes was in this case itself installed on a flash drive, it stood to reason that by simply booting it on a completely different laptop which did not have that relevant network card and serial number present, Qubes would simply ignore the requested assignment and boot normally, with no network card (not even the one in the second laptop) assigned to Sys-net upon boot completion. And this is exactly what happened.
From there, with system successfully booted on second laptop without problem card, it is/was still necessary to use Qubes Manager to graphically edit settings for that Sys-net VM, and look in its list of assigned devices, which *appears* empty but is not really (because a non-present device is still assigned to it "invisibly"). Because of this, it was needed to click the "unassign all" button (two arrows pointing left: << ) and then save settings, so that even non-visible/present devices (the problem card) became unassigned to that Sys-Net VM.
After that, I shut down and placed Qubes flash drive back in original laptop, booted, and everything worked fine. Problem network card was back to Dom0 and not assigned to Sys-Net, and all worked fine (with no networking, obviously, but still usable system).
This approach may or may not work for de-assigning USB controller from USB-VM, since when booted on a different laptop, that specific USB controller won't be found, and the assign request will be ignored, and you can unassign-all-devices from USB-VM on that second laptop, before shutting down and then booting again on original machine.
Like I said, possibly this tactic won't work for some unpredictable reason, but it worked in my case when network card was the "accidentally" assigned device causing non-startable system.