Need help after a failed in-place upgrade attempt

Viktor Ransmayr

unread,

Feb 16, 2024, 5:33:14 AMFeb 16

to qubes-users

Hello community,

I tried to upgrade my Qubes OS system from 4.1.2 to 4.2.0 using the "qubes-dist-upgrade" script.

The upgrade failed - and - now the system is in a 'weird' state.

None of the Fedora- or Debian-based VM have 'external / public' network access anymore.

The 'anon-whonix' VM however still does have 'external / public' network access - and - the update of templates through the Qubes Updater is also still working ...

I did perform a backup a backup of all VMs before starting the script - but - would like to skip a new install if possible ! - Any advice / ideas ?

With kind regards,

Viktor

haaber

unread,

Feb 17, 2024, 2:10:17 AMFeb 17

to qubes...@googlegroups.com

Hi

> I tried to upgrade my Qubes OS system from 4.1.2 to 4.2.0 using the
> "qubes-dist-upgrade" script.
>
> The upgrade failed - and - now the system is in a 'weird' state.
>
> None of the Fedora- or Debian-based VM have 'external / public'
> network access anymore.
>
> The 'anon-whonix' VM however still does have 'external / public'
> network access - and - the update of templates through the Qubes
> Updater is also still working ...
>

OK, I am not an expert on THIS question. Some general remarks: the
network card seems to work, right? So you need to check where the chain
breaks.

- go to sys-net (open terminal via widget) type ping 8.8.8.8 and see if
you come out

- go to sys-firewall (terminal via widget) and do the same.

if these two work, and an app-vm has no network, its config got lost.
Look at network settings of the corresponding appVM. It should be
sys-firewall in std setting, apart anon-whonix, of corse which uses
sys-whonix.

Viktor Ransmayr

unread,

Feb 17, 2024, 5:31:32 AMFeb 17

to haaber, qubes...@googlegroups.com

Hello 'Haaber',

Am Sa., 17. Feb. 2024 um 08:10 Uhr schrieb 'haaber' via qubes-users <qubes...@googlegroups.com>:

Hi
> I tried to upgrade my Qubes OS system from 4.1.2 to 4.2.0 using the
> "qubes-dist-upgrade" script.
>
> The upgrade failed - and - now the system is in a 'weird' state.
>
> None of the Fedora- or Debian-based VM have 'external / public'
> network access anymore.
>
> The 'anon-whonix' VM however still does have 'external / public'
> network access - and - the update of templates through the Qubes
> Updater is also still working ...
>
OK, I am not an expert on THIS question. Some general remarks: the
network card seems to work, right? So you need to check where the chain
breaks.

- go to sys-net (open terminal via widget) type ping 8.8.8.8 and see if
you come out

Working.

- go to sys-firewall (terminal via widget) and do the same.

Working as well.

if these two work, and an app-vm has no network, its config got lost.
Look at network settings of the corresponding appVM. It should be
sys-firewall in std setting, apart anon-whonix, of corse which uses
sys-whonix.

Not working. - I changed the settings from "default (sys-firewall) (current)" to "sys-firewall" in one App-VM ...

An additional / new info is, that an update check for 'dom0' does no longer work !

I did not explicitely try that before ...

Does it make sense to try to just restore 'dom0' as a start ?

With kind regards,

Viktor

Viktor Ransmayr

unread,

Feb 19, 2024, 3:10:29 PMFeb 19

to haaber, qubes-users

Hello community,

Am Sa., 17. Feb. 2024 um 08:10 Uhr schrieb 'haaber' via qubes-users <qubes...@googlegroups.com>:

Hi
> I tried to upgrade my Qubes OS system from 4.1.2 to 4.2.0 using the
> "qubes-dist-upgrade" script.

...
>
OK, I am not an expert on THIS question. Some general remarks: the
network card seems to work, right? So you need to check where the chain
breaks.

- go to sys-net (open terminal via widget) type ping 8.8.8.8 and see if
you come out

Working.

- go to sys-firewall (terminal via widget) and do the same.

Working as well.

if these two work, and an app-vm has no network, its config got lost.
Look at network settings of the corresponding appVM. It should be
sys-firewall in std setting, apart anon-whonix, of corse which uses
sys-whonix.

Not working. - I changed the settings from "default (sys-firewall) (current)" to "sys-firewall" in one App-VM ...

An additional / new info is, that an update check for 'dom0' does no longer work !

I did not explicitely try that before ...

Does it make sense to try to just restore 'dom0' as a start ?

Any feedback - or - any other ideas what I could try ?

With kind regards,

Viktor

haaber

unread,

Feb 20, 2024, 5:10:44 AMFeb 20

to qubes...@googlegroups.com

Hi

Am Sa., 17. Feb. 2024 um 08:10 Uhr schrieb 'haaber' via qubes-users <qubes...@googlegroups.com>:

Hi
> I tried to upgrade my Qubes OS system from 4.1.2 to 4.2.0 using the
> "qubes-dist-upgrade" script.
...
>
OK, I am not an expert on THIS question. Some general remarks: the
network card seems to work, right? So you need to check where the chain
breaks.

- go to sys-net (open terminal via widget) type ping 8.8.8.8 and see if
you come out

Working.

- go to sys-firewall (terminal via widget) and do the same.

Working as well.

if these two work, and an app-vm has no network, its config got lost.
Look at network settings of the corresponding appVM. It should be
sys-firewall in std setting, apart anon-whonix, of corse which uses
sys-whonix.

Not working. - I changed the settings from "default (sys-firewall) (current)" to "sys-firewall" in one App-VM ...

An additional / new info is, that an update check for 'dom0' does no longer work !

all updates go via tor network (sys-whonix) by default. You could click on the blue qube widget -> sys-wonix -> run terminal and see if sys-whonix has network. But I guess not. Here is why:

https://www.qubes-os.org/doc/firewall/

I wild-guess that you are in a "half-state" where one part of the system expects iptables, another one nftables ...

Did you download / start to download new (debian/fedora) Templates or are they the "old" ones?

I did not see any other user jump to your help, and I am not good enough to fix that alone for you. So honestly, at your place I would

(1) backup data (again)

(2) extract the list of manually installed packages in each of your templates and stock them on your backup drive

("apt-mark showmanual > manual.packages.list" in a terminal is your friend, no root priv needed)

(3) re-install a clean 4.2

(4) replay your manual installs of packages in your templates:

"cat manual.packages.list | apt-get install " or something of this type should work (run as root)

(5) restore your data.

It's a pain and takes half a day, but I fear that it is, at the end of the day, faster than any other solution...

good luck!

Viktor Ransmayr

unread,

Feb 20, 2024, 2:13:20 PMFeb 20

to haaber, qubes...@googlegroups.com

Hello 'Haaber',

Am Di., 20. Feb. 2024 um 11:10 Uhr schrieb 'haaber' via qubes-users <qubes...@googlegroups.com>:

...

all updates go via tor network (sys-whonix) by default. You could click on the blue qube widget -> sys-wonix -> run terminal and see if sys-whonix has network. But I guess not. Here is why:

https://www.qubes-os.org/doc/firewall/

I wild-guess that you are in a "half-state" where one part of the system expects iptables, another one nftables ...

Did you download / start to download new (debian/fedora) Templates or are they the "old" ones?

I did not see any other user jump to your help, and I am not good enough to fix that alone for you. So honestly, at your place I would

(1) backup data (again)

(2) extract the list of manually installed packages in each of your templates and stock them on your backup drive

("apt-mark showmanual > manual.packages.list" in a terminal is your friend, no root priv needed)

(3) re-install a clean 4.2

(4) replay your manual installs of packages in your templates:

"cat manual.packages.list | apt-get install " or something of this type should work (run as root)

(5) restore your data.

It's a pain and takes half a day, but I fear that it is, at the end of the day, faster than any other solution...

good luck!

Thanks a lot !

This is exactly the feedback I was hoping for.

I'll investigate further on my side & will provide an update from my side before the end of the week ...

With kind regards,

Viktor

Viktor Ransmayr

unread,

Apr 4, 2024, 1:45:37 PMApr 4

to haaber, qubes...@googlegroups.com

Hello 'Haaber' & Qubes OS community,

Am Di., 20. Feb. 2024 um 20:12 Uhr schrieb Viktor Ransmayr <viktor....@gmail.com>:

...

It took much longer due to private reasons - but - I can report that I was able to fully recover from the backups !

What I did different than suggested was that I started with a clean re-install of Qubes OS 4.1 ...

Now I've started a second attempt of an in-place upgrade - and - are already running into issues again at STAGE 1:

Here is the dom0 - log:

###

[vr@dom0 ~]$
[vr@dom0 ~]$ sudo qubes-dist-upgrade --update
WARNING: /!\ MAKE SURE YOU HAVE MADE A BACKUP OF ALL YOUR VMs AND dom0 DATA /!\
-> Launch upgrade process? [y/N] y
---> Allow shutdown of unnecessary VM (use --keep-running to exclude some): fedora-feedly-vm fedora-qubes-study-vm? [y/N] y
---> (STAGE 1) Do you want to make a dom0 snapshot? [y/N] y
WARNING: Sum of all thin volume sizes (<2.83 TiB) exceeds the size of thin pools and the size of whole volume group (<475.34 GiB).
Logical volume "Qubes41UpgradeBackup" created.
--> If upgrade to 4.2 fails, you can restore your dom0 snapshot with sudo lvconvert --merge qubes_dom0/Qubes41UpgradeBackup. Reboot after restoration.
---> (STAGE 1) Updating dom0...
Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...
Qubes OS Repository for Dom0 2.9 MB/s | 3.0 kB 00:00
Qubes OS Repository for Dom0 6.7 MB/s | 192 kB 00:00

kernel-latest.x86_64 1000:6.7.7-1.qubes.fc32 qubes-dom0-cached
kernel-latest-devel.x86_64 1000:6.7.7-1.qubes.fc32 qubes-dom0-cached
kernel-latest-modules.x86_64 1000:6.7.7-1.qubes.fc32 qubes-dom0-cached
kernel-latest-qubes-vm.x86_64 1000:6.7.7-1.qubes.fc32 qubes-dom0-cached
qubes-usb-proxy-dom0.noarch 1.2.0-1.fc32 qubes-dom0-cached
Qubes OS Repository for Dom0 2.9 MB/s | 3.0 kB 00:00
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
kernel-latest x86_64 1000:6.7.7-1.qubes.fc32 qubes-dom0-cached 12 M
kernel-latest-devel x86_64 1000:6.7.7-1.qubes.fc32 qubes-dom0-cached 15 M
kernel-latest-modules x86_64 1000:6.7.7-1.qubes.fc32 qubes-dom0-cached 76 M
kernel-latest-qubes-vm x86_64 1000:6.7.7-1.qubes.fc32 qubes-dom0-cached 18 M
Upgrading:
qubes-usb-proxy-dom0 noarch 1.2.0-1.fc32 qubes-dom0-cached 25 k

Transaction Summary
================================================================================
Install 4 Packages
Upgrade 1 Package

Total size: 121 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : kernel-latest-modules-1000:6.7.7-1.qubes.fc32.x86_64 1/6
Running scriptlet: kernel-latest-modules-1000:6.7.7-1.qubes.fc32.x86_64 1/6
Installing : kernel-latest-devel-1000:6.7.7-1.qubes.fc32.x86_64 2/6
Running scriptlet: kernel-latest-devel-1000:6.7.7-1.qubes.fc32.x86_64 2/6
Installing : kernel-latest-qubes-vm-1000:6.7.7-1.qubes.fc32.x86_6 3/6
Running scriptlet: kernel-latest-qubes-vm-1000:6.7.7-1.qubes.fc32.x86_6 3/6
--> Building files for 6.7.7-1.qubes.fc32.x86_64 in /var/lib/qubes/vm-kernels/6.7.7-1.fc32
---> Generating modules.img
resize2fs 1.45.5 (07-Jan-2020)
--> Done.

Installing : kernel-latest-1000:6.7.7-1.qubes.fc32.x86_64 4/6
Upgrading : qubes-usb-proxy-dom0-1.2.0-1.fc32.noarch 5/6
Cleanup : qubes-usb-proxy-dom0-1.1.5-1.fc32.noarch 6/6
Running scriptlet: kernel-latest-1000:6.7.7-1.qubes.fc32.x86_64 6/6
/usr/lib/kernel/install.d/20-grub.install: line 81: grub2-get-kernel-settings: command not found
cat: /sys/power/resume: No such file or directory
Generating grub configuration file ...
Found theme: /boot/grub2/themes/qubes/theme.txt
Found linux image: /boot/vmlinuz-6.7.7-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.7.7-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.6.2-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.6.2-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.1.75-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.1.75-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.1.12-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.1.12-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-5.15.94-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-5.15.94-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.7.7-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.7.7-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.6.2-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.6.2-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.1.75-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.1.75-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.1.12-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.1.12-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-5.15.94-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-5.15.94-1.qubes.fc32.x86_64.img
done
Generating grub configuration file ...
Found theme: /boot/grub2/themes/qubes/theme.txt
Found linux image: /boot/vmlinuz-6.7.7-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.7.7-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.6.2-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.6.2-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.1.75-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.1.75-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.1.12-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.1.12-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-5.15.94-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-5.15.94-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.7.7-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.7.7-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.6.2-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.6.2-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.1.75-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.1.75-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-6.1.12-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-6.1.12-1.qubes.fc32.x86_64.img
Found linux image: /boot/vmlinuz-5.15.94-1.qubes.fc32.x86_64
Found initrd image: /boot/initramfs-5.15.94-1.qubes.fc32.x86_64.img
done

Running scriptlet: qubes-usb-proxy-dom0-1.1.5-1.fc32.noarch 6/6
Verifying : kernel-latest-1000:6.7.7-1.qubes.fc32.x86_64 1/6
Verifying : kernel-latest-devel-1000:6.7.7-1.qubes.fc32.x86_64 2/6
Verifying : kernel-latest-modules-1000:6.7.7-1.qubes.fc32.x86_64 3/6
Verifying : kernel-latest-qubes-vm-1000:6.7.7-1.qubes.fc32.x86_6 4/6
Verifying : qubes-usb-proxy-dom0-1.2.0-1.fc32.noarch 5/6
Verifying : qubes-usb-proxy-dom0-1.1.5-1.fc32.noarch 6/6

Upgraded:
qubes-usb-proxy-dom0-1.2.0-1.fc32.noarch
Installed:
kernel-latest-1000:6.7.7-1.qubes.fc32.x86_64
kernel-latest-devel-1000:6.7.7-1.qubes.fc32.x86_64
kernel-latest-modules-1000:6.7.7-1.qubes.fc32.x86_64
kernel-latest-qubes-vm-1000:6.7.7-1.qubes.fc32.x86_64

Complete!
---> (STAGE 1) Updating Templates VMs and StandaloneVMs...
debian-11-podman: OK
debian-11-docker-desktop: OK
debian-12-vrsq: OK
debian-12-minimal: OK
debian-11: OK
fedora-38-minimal-vr: OK
debian-12: OK
fedora-38-minimal-vr-vscodium: ERROR (exception No such domain: 'disp-mgmt-fedora-38-minimal-vr-')
fedora-38-minimal-vr-podman: OK
fedora-39-minimal: ERROR (exit code 20, details in /var/log/qubes/mgmt-fedora-39-minimal.log)
fedora-39: OK
fedora-38: OK
fedora-38-vrsq: OK
whonix-gw-16: ERROR (exit code 20, details in /var/log/qubes/mgmt-whonix-gw-16.log)
vrqt-fl39-01: OK
vrqt-fl39-02: OK
whonix-ws-16: OK
[vr@dom0 ~]$

###

Is it save to proceed ?

With kind regards,

Viktor

Viktor Ransmayr

unread,

Apr 5, 2024, 3:08:19 PMApr 5

to haaber, qubes-users

On Thu, 4 Apr 2024, 19:45 Viktor Ransmayr, <viktor....@gmail.com> wrote:

Hello 'Haaber' & Qubes OS community,

Am Di., 20. Feb. 2024 um 20:12 Uhr schrieb Viktor Ransmayr <viktor....@gmail.com>:
...
Am Di., 20. Feb. 2024 um 11:10 Uhr schrieb 'haaber' via qubes-users <qubes...@googlegroups.com>:

...
all updates go via tor network (sys-whonix) by default. You could click on the blue qube widget -> sys-wonix -> run terminal and see if sys-whonix has network. But I

It took much longer due to private reasons - but - I can report that I was able to fully recover from the backups !

What I did different than suggested was that I started with a clean re-install of Qubes OS 4.1 ...

Let's close this thread !

Reply all

Reply to author

Forward