updating Raspbian to latest takes CBRIDGE 64bit image off network

[Tony Brack]

This is a recurring theme with my cluster. The distributed image seems to flash OK and comes online. Cluster is a v2.4 board with a 2G Pi4b, 3 Pi Zeros and 1 Pi Zero W (although we never get to booting these).

2022-04-04-2-bullseye-ClusterCTRL-arm64-lite-CBRIDGE.zip

1. burn image to my microSD
2. boot into network
3. sudo apt update -y
4. sudo apt upgrade
5. sudo apt install -y cockpit
6. sudo reboot

After this, the network does not connect and further, attempting to run raspi-config to configure WiFi also results in an S1 error, whatever that is. The image itself, prior to update/upgrade seems to connect just fine. I don't know what additional info I can get you, and it is not easy to capture since I am blown off the network.

Tony

[Tony Brack]

Further investigation: something in the general "upgrade" portion of this is what sends clusterctrl off the rails. When I tried to check things out from the console before reloading the image, even hostnamectl was hanging. "apt update -y" indicates that we bumped the repository version from 1.3 to 1.6.

bracka@centauri:~ $ sudo apt update -y
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:3 http://security.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:4 http://archive.raspberrypi.org/debian bullseye InRelease [23.6 kB]
Get:5 http://deb.debian.org/debian bullseye/main armhf Packages [7,945 kB]
Get:6 http://deb.debian.org/debian bullseye/main arm64 Packages [8,072 kB]
Get:7 http://deb.debian.org/debian bullseye/main Translation-en [6,240 kB]
Get:8 http://deb.debian.org/debian bullseye/contrib armhf Packages [40.2 kB]
Get:9 http://deb.debian.org/debian bullseye/contrib arm64 Packages [41.0 kB]
Get:10 http://deb.debian.org/debian bullseye/contrib Translation-en [46.9 kB]
Get:11 http://deb.debian.org/debian bullseye/non-free armhf Packages [56.3 kB]
Get:12 http://deb.debian.org/debian bullseye/non-free arm64 Packages [73.0 kB]
Get:13 http://deb.debian.org/debian bullseye/non-free Translation-en [92.4 kB]
Get:14 http://deb.debian.org/debian bullseye-updates/main armhf Packages [12.0 kB]
Get:15 http://deb.debian.org/debian bullseye-updates/main arm64 Packages [12.0 kB]
Get:16 http://deb.debian.org/debian bullseye-updates/main Translation-en [7,929 B]
Get:17 http://security.debian.org/debian-security bullseye-security/main armhf Packages [217 kB]
Get:18 http://security.debian.org/debian-security bullseye-security/main arm64 Packages [218 kB]
Get:19 http://security.debian.org/debian-security bullseye-security/main Translation-en [145 kB]
Get:20 http://security.debian.org/debian-security bullseye-security/non-free Translation-en [344 B]
Get:21 http://archive.raspberrypi.org/debian bullseye/main armhf Packages [311 kB]
Get:22 http://archive.raspberrypi.org/debian bullseye/main arm64 Packages [302 kB]
Fetched 24.1 MB in 10s (2,452 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
117 packages can be upgraded. Run 'apt list --upgradable' to see them.
N: Repository 'http://deb.debian.org/debian bullseye InRelease' changed its 'Version' value from '11.3' to '11.6'

If I skip the 'apt upgrade -y' and simply install and enable cockpit it looks like I am running, so there seems to be something in the latest general upgrade packages that conflicts with what is in the 2022-04-04 image, or more to the point, some dependency gets broken. I may try loading this image on a non-clustered 64 bit Pi and see what happens. Again, any insights as to what to collect (and possibly how) are appreciated.

I suppose I could list what it WANTS to upgrade from/to?

I'd rather be part of the solution than part of the problem. 

[Tony Brack]

bracka@centauri:~ $ sudo apt list --upgradable
Listing... Done
avahi-daemon/stable 0.8-5+deb11u1 arm64 [upgradable from: 0.8-5]
base-files/stable 11.1+deb11u6 arm64 [upgradable from: 11.1+deb11u3]
bash/stable 5.1-2+deb11u1 arm64 [upgradable from: 5.1-2+b3]
bind9-host/stable-security 1:9.16.37-1~deb11u1 arm64 [upgradable from: 1:9.16.27-1~deb11u1]
bind9-libs/stable-security 1:9.16.37-1~deb11u1 arm64 [upgradable from: 1:9.16.27-1~deb11u1]
bluez-firmware/stable,stable 1.2-4+rpt10 all [upgradable from: 1.2-4+rpt8]
bluez/stable 5.55-3.1+rpt2 arm64 [upgradable from: 5.55-3.1+rpt1]
cifs-utils/stable,stable-security 2:6.11-3.1+deb11u1 arm64 [upgradable from: 2:6.11-3.1]
console-setup-linux/stable,stable 1.205+rpt1 all [upgradable from: 1.205]
console-setup/stable,stable 1.205+rpt1 all [upgradable from: 1.205]
curl/stable-security 7.74.0-1.3+deb11u5 arm64 [upgradable from: 7.74.0-1.3+deb11u1]
dhcpcd5/stable 1:8.1.2-1+rpt9 arm64 [upgradable from: 1:8.1.2-1+rpt5]
dirmngr/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
distro-info-data/stable,stable 0.51+deb11u3 all [upgradable from: 0.51+deb11u1]
dpkg-dev/stable,stable 1.20.12 all [upgradable from: 1.20.9]
dpkg/stable 1.20.12 arm64 [upgradable from: 1.20.9]
firmware-atheros/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]
firmware-brcm80211/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]
firmware-libertas/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]
firmware-misc-nonfree/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]
firmware-realtek/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]
git-man/stable-security,stable-security 1:2.30.2-1+deb11u1 all [upgradable from: 1:2.30.2-1]
git/stable-security 1:2.30.2-1+deb11u1 arm64 [upgradable from: 1:2.30.2-1]
gnupg-l10n/stable,stable,stable-security,stable-security 2.2.27-2+deb11u2 all [upgradable from: 2.2.27-2+deb11u1]
gnupg-utils/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
gnupg/stable,stable,stable-security,stable-security 2.2.27-2+deb11u2 all [upgradable from: 2.2.27-2+deb11u1]
gpg-agent/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
gpg-wks-client/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
gpg-wks-server/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
gpg/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
gpgconf/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
gpgsm/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
gpgv/stable,stable-security 2.2.27-2+deb11u2 arm64 [upgradable from: 2.2.27-2+deb11u1]
isc-dhcp-client/stable,stable-security 4.4.1-2.3+deb11u1 arm64 [upgradable from: 4.4.1-2.3]
isc-dhcp-common/stable,stable-security 4.4.1-2.3+deb11u1 arm64 [upgradable from: 4.4.1-2.3]
keyboard-configuration/stable,stable 1.205+rpt1 all [upgradable from: 1.205]
libavahi-common-data/stable 0.8-5+deb11u1 arm64 [upgradable from: 0.8-5]
libavahi-common3/stable 0.8-5+deb11u1 arm64 [upgradable from: 0.8-5]
libavahi-core7/stable 0.8-5+deb11u1 arm64 [upgradable from: 0.8-5]
libc-bin/stable 2.31-13+rpt2+rpi1+deb11u5 arm64 [upgradable from: 2.31-13+rpt2+rpi1+deb11u2]
libc-dev-bin/stable 2.31-13+rpt2+rpi1+deb11u5 arm64 [upgradable from: 2.31-13+rpt2+rpi1+deb11u2]
libc-devtools/stable 2.31-13+rpt2+rpi1+deb11u5 arm64 [upgradable from: 2.31-13+rpt2+rpi1+deb11u2]
libc-l10n/stable,stable 2.31-13+rpt2+rpi1+deb11u5 all [upgradable from: 2.31-13+rpt2+rpi1+deb11u2]
libc6-dbg/stable 2.31-13+rpt2+rpi1+deb11u5 arm64 [upgradable from: 2.31-13+rpt2+rpi1+deb11u2]
libc6-dev/stable 2.31-13+rpt2+rpi1+deb11u5 arm64 [upgradable from: 2.31-13+rpt2+rpi1+deb11u2]
libc6/stable 2.31-13+rpt2+rpi1+deb11u5 arm64 [upgradable from: 2.31-13+rpt2+rpi1+deb11u2]
libcamera-apps-lite/stable 0~git20230125+9f08463-1 arm64 [upgradable from: 0~git20220228+0dc5ea0-1]
libcamera0/stable 0~git20230124+9b860a66-1 arm64 [upgradable from: 0~git20220510+22656360-1]
libcurl3-gnutls/stable-security 7.74.0-1.3+deb11u5 arm64 [upgradable from: 7.74.0-1.3+deb11u1]
libcurl4/stable-security 7.74.0-1.3+deb11u5 arm64 [upgradable from: 7.74.0-1.3+deb11u1]
libdpkg-perl/stable,stable 1.20.12 all [upgradable from: 1.20.9]
libexpat1/stable,stable-security 2.2.10-2+deb11u5 arm64 [upgradable from: 2.2.10-2+deb11u3]
libfreetype-dev/stable 2.10.4+dfsg-1+deb11u1 arm64 [upgradable from: 2.10.4+dfsg-1]
libfreetype6-dev/stable 2.10.4+dfsg-1+deb11u1 arm64 [upgradable from: 2.10.4+dfsg-1]
libfreetype6/stable 2.10.4+dfsg-1+deb11u1 arm64 [upgradable from: 2.10.4+dfsg-1]
libgssapi-krb5-2/stable,stable-security 1.18.3-6+deb11u3 arm64 [upgradable from: 1.18.3-6+deb11u1]
libk5crypto3/stable,stable-security 1.18.3-6+deb11u3 arm64 [upgradable from: 1.18.3-6+deb11u1]
libkrb5-3/stable,stable-security 1.18.3-6+deb11u3 arm64 [upgradable from: 1.18.3-6+deb11u1]
libkrb5support0/stable,stable-security 1.18.3-6+deb11u3 arm64 [upgradable from: 1.18.3-6+deb11u1]
libksba8/stable-security 1.5.0-3+deb11u2 arm64 [upgradable from: 1.5.0-3]
libldap-2.4-2/stable,stable-security 2.4.57+dfsg-3+deb11u1 arm64 [upgradable from: 2.4.57+dfsg-3]
libldap-common/stable,stable,stable-security,stable-security 2.4.57+dfsg-3+deb11u1 all [upgradable from: 2.4.57+dfsg-3]
libnftables1/stable 0.9.8-3.1+deb11u1 arm64 [upgradable from: 0.9.8-3.1]
libntfs-3g883/stable,stable-security 1:2017.3.23AR.3-4+deb11u3 arm64 [upgradable from: 1:2017.3.23AR.3-4+deb11u1]
libpam-chksshpwd/stable 1.4.0-9+deb11u1+rpt2 arm64 [upgradable from: 1.4.0-9+deb11u1+rpt1]
libpam-modules-bin/stable 1.4.0-9+deb11u1+rpt2 arm64 [upgradable from: 1.4.0-9+deb11u1+rpt1]
libpam-modules/stable 1.4.0-9+deb11u1+rpt2 arm64 [upgradable from: 1.4.0-9+deb11u1+rpt1]
libpam-runtime/stable,stable 1.4.0-9+deb11u1+rpt2 all [upgradable from: 1.4.0-9+deb11u1+rpt1]
libpam-systemd/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]
libpam0g/stable 1.4.0-9+deb11u1+rpt2 arm64 [upgradable from: 1.4.0-9+deb11u1+rpt1]
libpcre2-8-0/stable 10.36-2+deb11u1 arm64 [upgradable from: 10.36-2]
libpcre2-posix2/stable 10.36-2+deb11u1 arm64 [upgradable from: 10.36-2]
libssl1.1/stable 1.1.1n-0+deb11u3+rpt1 arm64 [upgradable from: 1.1.1n-0+deb11u1]
libsystemd0/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]
libtasn1-6/stable 4.16.0-2+deb11u1 arm64 [upgradable from: 4.16.0-2]
libtiff5/stable-security 4.2.0-1+deb11u3 arm64 [upgradable from: 4.2.0-1+deb11u1]
libtirpc-common/stable,stable,stable-security,stable-security 1.3.1-1+deb11u1 all [upgradable from: 1.3.1-1]
libtirpc-dev/stable,stable-security 1.3.1-1+deb11u1 arm64 [upgradable from: 1.3.1-1]
libtirpc3/stable,stable-security 1.3.1-1+deb11u1 arm64 [upgradable from: 1.3.1-1]
libudev1/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]
libwbclient0/stable,stable-security 2:4.13.13+dfsg-1~deb11u5 arm64 [upgradable from: 2:4.13.13+dfsg-1~deb11u3]
libxml2/stable,stable-security 2.9.10+dfsg-6.7+deb11u3 arm64 [upgradable from: 2.9.10+dfsg-6.7+deb11u1]
linux-libc-dev/stable 1:1.20230106-1 arm64 [upgradable from: 1:1.20220331-1]
locales/stable,stable 2.31-13+rpt2+rpi1+deb11u5 all [upgradable from: 2.31-13+rpt2+rpi1+deb11u2]
logrotate/stable 3.18.0-2+deb11u1 arm64 [upgradable from: 3.18.0-2]
nano/stable 5.4-2+deb11u2 arm64 [upgradable from: 5.4-2]
nftables/stable 0.9.8-3.1+deb11u1 arm64 [upgradable from: 0.9.8-3.1]
ntfs-3g/stable,stable-security 1:2017.3.23AR.3-4+deb11u3 arm64 [upgradable from: 1:2017.3.23AR.3-4+deb11u1]
openssh-client/stable 1:8.4p1-5+deb11u1 arm64 [upgradable from: 1:8.4p1-5]
openssh-server/stable 1:8.4p1-5+deb11u1 arm64 [upgradable from: 1:8.4p1-5]
openssh-sftp-server/stable 1:8.4p1-5+deb11u1 arm64 [upgradable from: 1:8.4p1-5]
openssl/stable 1.1.1n-0+deb11u3+rpt1 arm64 [upgradable from: 1.1.1n-0+deb11u1]
publicsuffix/stable,stable 20220811.1734-0+deb11u1 all [upgradable from: 20211207.1025-0+deb11u1]
raspberrypi-bootloader/stable 1:1.20230106-1 arm64 [upgradable from: 1:1.20220331-1]
raspberrypi-kernel/stable 1:1.20230106-1 arm64 [upgradable from: 1:1.20220331-1]
raspberrypi-net-mods/stable,stable 1.3.4 all [upgradable from: 1.3.3]
raspberrypi-sys-mods/stable 20221019 arm64 [upgradable from: 20220224]
raspi-config/stable,stable 20221214 all [upgradable from: 20220506]
raspinfo/stable,stable 20221220-1 all [upgradable from: 20190624-1]
rpi-eeprom/stable 16.0-1 arm64 [upgradable from: 13.14-1]
rpiboot/stable 20221215~105525 arm64 [upgradable from: 20220504~214218]
rsyslog/stable,stable-security 8.2102.0-2+deb11u1 arm64 [upgradable from: 8.2102.0-2]
ssh/stable,stable 1:8.4p1-5+deb11u1 all [upgradable from: 1:8.4p1-5]
sudo/stable-security 1.9.5p2-3+deb11u1 arm64 [upgradable from: 1.9.5p2-3]
systemd-sysv/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]
systemd-timesyncd/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]
systemd/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]
tzdata/stable,stable,stable-updates,stable-updates 2021a-1+deb11u8 all [upgradable from: 2021a-1+deb11u3]
udev/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]
unzip/stable,stable-security 6.0-26+deb11u1 arm64 [upgradable from: 6.0-26]
userconf-pi/stable,stable 0.5 all [upgradable from: 0.2]
wireless-regdb/stable,stable 2022.04.08-2~deb11u1 all [upgradable from: 2020.04.29-2]
zlib1g-dev/stable,stable-security 1:1.2.11.dfsg-2+deb11u2 arm64 [upgradable from: 1:1.2.11.dfsg-2+deb11u1]
zlib1g/stable,stable-security 1:1.2.11.dfsg-2+deb11u2 arm64 [upgradable from: 1:1.2.11.dfsg-2+deb11u1]

[Tony Brack]

I started going through the packages manually, going for what I thought were innocuous packages. The system came off the rails right after I installed the new 'udev' and rebooted. I am going to approach this from the other side:

• Install the latest raspbian 64 bit minimal image on an SD card and apply the packages I want
• manually install clusterctrl per:
  8086 Support - How do I manually install clusterctrl?

If anybody is following: Is there anything I need to be aware of?

[Chris Burton]

Hi, 

2022-04-04-2-bullseye-ClusterCTRL-arm64-lite-CBRIDGE.zip

1. burn image to my microSD
2. boot into network
3. sudo apt update -y
4. sudo apt upgrade
5. sudo apt install -y cockpit
6. sudo reboot

After this, the network does not connect and further,

Installing cockpit is activating/installing NetworkManager but the image is setup to use dhcpcd.

If you're starting from a fresh image you should be able to prevent it from being installed and then upgrade/install as normal

sudo apt-mark hold network-manager
sudo apt update
sudo apt upgrade -y

sudo apt install -y cockpit
sudo reboot

If you want to fix an existing image you can switch it from using NetworkManager to dhcpcd and it should be back to normal after a reboot.

sudo raspi-config nonint do_netconf 1
sudo reboot

 

attempting to run raspi-config to configure WiFi also results in an S1 error, whatever that is. The image itself, prior to update/upgrade seems to connect just fine. I don't know what additional info I can get you, and it is not easy to capture since I am blown off the network.

If you're using WiFi then you'd probably want the CNAT image. I tested seting up WiFi on an updated CBRIDGE image without issue but if you're still seeing problems please provide the steps to replicate.

 Chris.

[Tony Brack]

Hi again, and thanks for getting back!

I have been working on this a bit more and have an update:

I used the image builder to burn a brand new minimal image without clusterctrl software and it would not even boot, so the problem may be with Raspbian. I still think the Pi is OK.

I then built the microSD for the many-eth time and avoided all packages that I figured may be contributing to my grief. I now have a working system but have not yet checked out the following packages. Some combination of these is the problem. Of course, I am not going to allow automatic updates! In your opinion are any of these "safer" to try?

bracka@centauri:~ $ sudo apt list --upgradable

Listing... Done

dhcpcd5/stable 1:8.1.2-1+rpt9 arm64 [upgradable from: 1:8.1.2-1+rpt5]

firmware-atheros/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]

firmware-brcm80211/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]

firmware-libertas/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]

firmware-misc-nonfree/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]

firmware-realtek/stable,stable 1:20221012-1~bpo11+1+rpt1 all [upgradable from: 1:20210315-3+rpt5]

isc-dhcp-client/stable,stable-security 4.4.1-2.3+deb11u1 arm64 [upgradable from: 4.4.1-2.3]

isc-dhcp-common/stable,stable-security 4.4.1-2.3+deb11u1 arm64 [upgradable from: 4.4.1-2.3]

libudev1/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]

raspberrypi-bootloader/stable 1:1.20230106-1 arm64 [upgradable from: 1:1.20220331-1]

raspberrypi-kernel/stable 1:1.20230106-1 arm64 [upgradable from: 1:1.20220331-1]

raspberrypi-net-mods/stable,stable 1.3.4 all [upgradable from: 1.3.3]

raspberrypi-sys-mods/stable 20221019 arm64 [upgradable from: 20220224]

raspinfo/stable,stable 20221220-1 all [upgradable from: 20190624-1]

rpi-eeprom/stable 16.0-1 arm64 [upgradable from: 13.14-1]

rpiboot/stable 20221215~105525 arm64 [upgradable from: 20220504~214218]

udev/stable 247.3-7+deb11u1 arm64 [upgradable from: 247.3-7]

wireless-regdb/stable,stable 2022.04.08-2~deb11u1 all [upgradable from: 2020.04.29-2]

bracka@centauri:~ $ sudo apt list --upgradable 2>/dev/null | grep -i upgrad | wc -l

18

I am using a hardwired connection (static DHCP from my router). WiFi was just a test since the hard-wired connection was not getting a DHCP address. I was unable to manually assign an address (using ifconfig) once I was in this state. Note that I have not tested isc-dhcp, dhcpd, or the kernel individual updates yet. By the way, my experience is repeatable and the recent Raspbian boot images boot fine on my other (non-ClusterHAT) Pis. I have 2 more clusters that I can try to debug with as well.

Regarding network:

I have managed to cobble together a working configuration starting with your image, and I am going to burn another different microSD and plug away with the packages that are left. I can try some of your suggestions and will keep you posted on any progress.

[Chris Burton]

Hi, 

I am using a hardwired connection (static DHCP from my router). WiFi was just a test since the hard-wired connection was not getting a DHCP address. I was unable to manually assign an address (using ifconfig) once I was in this state. Note that I have not tested isc-dhcp, dhcpd, or the kernel individual updates yet. By the way, my experience is repeatable and the recent Raspbian boot images boot fine on my other (non-ClusterHAT) Pis. I have 2 more clusters that I can try to debug with as well.

With the supplied ClusterCTRL images I install all updates and haven't seen any problems but I don't use cockpit which as per my previous reply will switch you to NetworkManager and break things.

Chris. 

[Tony Brack]

Hi Chris,

I can confirm the behavior and you were right. It is specifically related to cockpit bringing in and activating NetworkManager. I have confirmed this by successively activating and deactivating both it and dhcpd. The other issue(s) may have been a microSD corruption of some sort and went away with reimaging. One thing still remains outstanding though, and that is that only the pre-built clusterctrl images seem to be affected. A vanilla, standalone Pi built straight off the Pi imager that uses dhcpd to get a DHCP provided IP address (hard wired) as well as running cockpit does not have the same problem as your images. (cbridge, cnat & client all behave the same)

Once I figure out what is going on there, I'll post it in a different thread. For now, simply disabling NetworkManager seems to work for getting my cluster up and running with a DHCP provided IP address.