After last update system DEAD

[eva...@firemail.cc]

Hello,
Oh.... no... :(
After last update (kernel&xen) system almost dead!
It can boot and I have access to dom0 console, but mouse not work and
there is 100% cpu usage by (1.xvdb-1 / 7.xvdc-1) and other processes
eating to much cpu
How to remove last update? How to recover now?
Data is urgent. I have too old backups :( Please, help :(((

I guess now it's not possible to reinstall, because any full update wll
break the system again. And also my data. How to recover it? It's SSH
M.2 drive and I don't have other machine with M.2 slot to recover. Or
how to do this? :( :( :(
Thanks.

(new email because no access to my password and data :( )

[eva...@firemail.cc]

Okey... keep calm...
Time to time I can use dom0 console without troubles (sudo
qubes-dom0-update also still work by no fix still come :( )
I guess now need to somehow change new kernel to old one?
How to do this? I guess I'm at UEFI mode (new notebook)
1) boot from some qubes installed usb flash and I will see recovery
mode? Or not?
2) I tried to access /boot/efi/ to edit xen.cfg (only it must be
edited?), but no access. sudo cd /boot/efi/ (success) then-> sudo ls
show me my home folder. How to get access to /boot/efi and what I must
edit to return to my old kernel?
3) or any other recovery methods please :)
Thanks!

[awokd]

eva...@firemail.cc:

First thing to try is to rollback the dom0 kernel update. If you're
using UEFI boot, "sudo nano /boot/efi/EFI/qubes/xen.cfg" and change the
default kernel line at the top to point to the previous version number.
Save and reboot. If you're not sure how to do that, send me your xen.cfg
file.

[eva...@firemail.cc]

On 2019-05-26 15:57, awokd wrote:
> First thing to try is to rollback the dom0 kernel update. If you're
> using UEFI boot, "sudo nano /boot/efi/EFI/qubes/xen.cfg" and change
> the default kernel line at the top to point to the previous version
> number. Save and reboot. If you're not sure how to do that, send me
> your xen.cfg file.

Totally DEAD :( After reboot -> boot -> system freeze because high cpu
ussage -> reboot...

Now it boot only to console:

generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue
etc.

Also this message say that it's possible to mount boot.
Maybe it's good ideato mount it, change setting and then try to boot
again?

[awokd]

eva...@firemail.cc:

Yes, but be aware that it may mount the boot directory you're looking to
edit underneath a different location, so watch the messages when you do.

[eva...@firemail.cc]

On 2019-05-26 16:32, 'awokd' via qubes-users wrote:

> Yes, but be aware that it may mount the boot directory you're looking
> to edit underneath a different location, so watch the messages when
> you do.

What is correct "mount" command to do this?

And to fix filesystem then...

[eva...@firemail.cc]

On 2019-05-26 15:57, 'awokd' via qubes-users wrote:

Log(url): i DOT imgur DOT com SLASH Z8GbXLJ.jpg

it request to run fsch or e2fsck. What I must run first and how?

Then if filesystem will be fixed, how to mount boot to change it? I
guess this emergency console can help me, because dom0 console after
boot not usable now (system can freeze). Here is a good point to change
things. Hope that it's possible to recover filesystem here.

Thanks

[awokd]

eva...@firemail.cc:

Filesystem corruption is a different issue. In the emergency console,
"fsck /dev/mapper/qubes_dom0-root". Answer y to everything it asks, then
"reboot". If that lets you get back into Qubes, DO NOT DO ANYTHING ELSE
UNTIL RUNNING A FULL BACKUP. Then try editing xen.cfg as previously
described.

[eva...@firemail.cc]

> Filesystem corruption is a different issue. In the emergency console,
> "fsck /dev/mapper/qubes_dom0-root". Answer y to everything it asks,
> then "reboot". If that lets you get back into Qubes, DO NOT DO
> ANYTHING ELSE UNTIL RUNNING A FULL BACKUP. Then try editing xen.cfg as
> previously described.

Thanks! Booted.
I'm not sure that it's possible to make full backup now, because system
freeze time to time with this kernel. But I will try. It's a good idea.

[eva...@firemail.cc]

Maybe it's possible to return to this emergency mode? System freeze
until Qubes Manager start. It can freeze at 45% or at 91% or at any
point. I see high cpu usage. It freeze because of this. I shutdown it
before at this point and filesystem corrupt. Need the way to access
/boot/efi without actual system load or I will lose all data and nothing
will changed...

[eva...@firemail.cc]

By the way, I got the idea and switched to other tty before login to the
system where Qubes Manager and other tasks start. It give me some time
to use the system. And I found the bug at std error output:

!!!!!
wachdog: BUG: soft lockup - CPU#3 stuck for 23s! [1.xvda-1:4325]
!!!!



kernel changed successfully to previous one. What next? yes, now I will
do new backups ASAP. But what next? Never run qubes-dom0-update?

Thanks

[awokd]

eva...@firemail.cc:

Did you notice any other errors around the soft lockup? Try searching
for them; it would be good to figure out what's causing the lockup, but
it can be hard to do.

Most important thing is to backup your system before dom0 kernel
updates. If uptime is a priority for you, wait for a week or so when
dom0 kernel updates come out and monitor qubes-issues and the mailing
lists to see if others are reporting problems. If so, hold off until
they get addressed. If uptime is critical and you can afford it, get
another machine with the same hardware configuration, and test out
updates there first before applying them to your production system.

See also Chris's suggestion
https://www.mail-archive.com/qubes...@googlegroups.com/msg27411.html.

[Evastar]

> Did you notice any other errors around the soft lockup? Try searching

> for them; it would be good to figure out what's causing the lockup, but

> it can be hard to do.

Only one error: all system very unresponsive, :) And as I'm already note 100% cpu usage by some processes. Also I saw strange "battary" icon at taskbar. It was like zero with some little dot image at the bottom corner of zero(gray) battery.

System very quick take "dead lock" and freeze. Only Qubes Manager staring at this time. It freeze at 45%, at 91% and some time Qubes Manager load successfully, but after this system freeze. I guest problem not with some software, but how processor work with this kernel.

> Most important thing is to backup your system before dom0 kernel

updates.

I have this idea on my mind after I install this new update and before reboot, but other mind come "what could be wrong. I update it since Q4 exists and never had problems with update :)".

Again, what to do now? Never update or try to wait for month and then update?

Thank you very much for your help!

[awokd]

'Evastar' via qubes-users:

>
> Again, what to do now? Never update or try to wait for month and then update?

Short term, continue applying template updates as usual. If you run
"sudo qubes-dom0-update", you can check for updates, but make sure to
choose not to apply them with N. Wait for the next dom0 kernel release
before trying to upgrade dom0 again, then pick and choose from my
suggestions in the last email. :)

[g80vm...@riseup.net]

eva...@firemail.cc:

Don't touch your existing M.2 drive. Get a spare drive (spinning rust
is OK) at least as large as your M.2 drive and that you can attach to
your machine (e.g. with a USB SATA dock).

Download Tails to a USB stick. Boot your machine into Tails. Connect
your spare drive. `fdisk -l` to be sure you have the right device names
for all your disks. `dd` your M.2 to the spare drive. Now `sync` the
disk and detach it. You now have a backup of your current state.

If you _need_ to access some VM data now, you can (with major negative
security implications) decrypt your M.2 disk manually with `cryptsetup`,
navigate to <m.2>/var/lib/qubes/appvms/<yourvm> and use `losetup` to
help mount the VM's data disk in Tails so you can copy off whatever data
you urgently need.

[Chris Laprise]

Based on what others have posted about the update, you should try to
select the prior kernel version under "Advanced options" in the grub
boot menu.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Eva Star]

> Don't touch your existing M.2 drive. Get a spare drive (spinning rust
> is OK) at least as large as your M.2 drive and that you can attach to
> your machine (e.g. with a USB SATA dock).
>
> Download Tails to a USB stick. Boot your machine into Tails. Connect
> your spare drive. `fdisk -l` to be sure you have the right device names
> for all your disks. `dd` your M.2 to the spare drive. Now `sync` the
> disk and detach it. You now have a backup of your current state.
>
> If you _need_ to access some VM data now, you can (with major negative
> security implications) decrypt your M.2 disk manually with `cryptsetup`,
> navigate to <m.2>/var/lib/qubes/appvms/<yourvm> and use `losetup` to
> help mount the VM's data disk in Tails so you can copy off whatever data
> you urgently need.

Thanks! It must be at the qubes FAQ. Next time it will help. I got the idea with cloning drive (vs actual remove it and do something with it). And looks like it's not so hard to decrypt the whole drive and mount some qubes data disks. Hope I will run without troubles, but will know this opportunity for future use.

Chris, thanks! Now solved. I'm at EFI. It was not possible to select something on the boot time or I don't know how to do this.

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 26/05/2019 3.44 PM, Eva Star wrote:
>
>> Don't touch your existing M.2 drive. Get a spare drive (spinning rust
>> is OK) at least as large as your M.2 drive and that you can attach to
>> your machine (e.g. with a USB SATA dock).
>>
>> Download Tails to a USB stick. Boot your machine into Tails. Connect
>> your spare drive. `fdisk -l` to be sure you have the right device names
>> for all your disks. `dd` your M.2 to the spare drive. Now `sync` the
>> disk and detach it. You now have a backup of your current state.
>>
>> If you _need_ to access some VM data now, you can (with major negative
>> security implications) decrypt your M.2 disk manually with `cryptsetup`,
>> navigate to <m.2>/var/lib/qubes/appvms/<yourvm> and use `losetup` to
>> help mount the VM's data disk in Tails so you can copy off whatever data
>> you urgently need.
>
>
> Thanks! It must be at the qubes FAQ. Next time it will help.

Here are the emergency manual restore instructions (for future reference):

https://www.qubes-os.org/doc/backup-restore/#emergency-backup-recovery-without-qubes

> I got the idea with cloning drive (vs actual remove it and do something with it). And looks like it's not so hard to decrypt the whole drive and mount some qubes data disks. Hope I will run without troubles, but will know this opportunity for future use.
>
> Chris, thanks! Now solved. I'm at EFI. It was not possible to select something on the boot time or I don't know how to do this.
>

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEZQ7rCYX0j3henGH1203TvDlQMDAFAlzrDsgACgkQ203TvDlQ
MDDoQw//boohe5Dfja8JQE3qOYzFxpJVu2Ke0wpATl06aDdaLzcP7db73X8HgYr8
/MD6kOhQSboslN7SLoZXmrQffnANPwOB87P+KTkzDK80KUofNDwv2xMnzORqhLva
/7V7j9UGYRpmQRINPB1Ibon3wJ6930IeOd2jgnxFY7jubxk8aXFlIgzVl4tBMvna
kfVrTJbz9gRfKCfyGxsl8IlorEscdGLXCleJCTDZj7VBjwg4W6mo819h3FCebkDR
v5GZa0SgVAJQBsm7yUsa4+AWPr7YYni2oJAoK1A8fVXHGfBJFdPgdrv3RZvM43Qf
GMh6Ql9kyHG3b6MrYe57mXBg8/ijnTgyNCP+vMJF5V501tiEy5fyeyuAKwwDFIeA
iN2sCNMJf0j5HvXK+EcMkt8gg7p4/K7hYP/YyZuXMsgdKJEZaL46KQ76H6lF8Nlp
L0qcI4TpBE7gD2w1JjPHnlKbscq/ugPjFMmzghRbzqtFFS9q6pB7kC2UUZWyv7Ik
xCpjsp4cfwAkB1ZnFernMvtzPu9q3jBQU0bZ1zOc7KX0icEIVv9ntjpi78yrERCl
BVIXmx0GR4Wn7Dannk9K3WXvWxx7ExZZ4oK8HmUv4G5TtOV6w40LrqYzlDU/hagh
EXKqkKj9h6PcDQC87BbyZHf71RZgymU+mzATbw5RDJorSx3Pq18=
=necR
-----END PGP SIGNATURE-----