Building a new Raspbian (bullseye) microSD with clusterctrl

[Tony Brack]

I decided to start a new thread so that there wouldn't be crosstalk with the upgrade issues I have been experiencing. I have gotten over the hump with a new image built on a different microSD after my first attempt failed with some odd sort of boot sector write protect error.

Configuration:

• 2GB Pi4b
• ClusterHAT v2.4
• 3 x Pi Zero (P1-P3)
• 1 x Pi Zero W (P4)
• hardwired ethernet - static assigned via DHCP

I built a new image:

1. use the Raspberry Pi imager and build a Raspbian 11 Lite (64 bit) image
2. boot as per normal
3. sudo apt update -y
4. sudo apt upgrade -y
5. sudo install -y cockpit ... (and a bunch of other stuff)
6. install clusterctrl using the web page instructions, as per Ubuntu
   8086 Support - How do I manually install clusterctrl?
   
7. use raspi-config to enable the i2c interface
8. reboot

I now have a running configuration that sees the 2 USB serial gadgets for each Pi, although I still haven't quite figured out their behavior. I intend to configure their management using conserver. Very happy so far! 

bracka@centauri:/dev $ hostnamectl
   Static hostname: centauri
         Icon name: computer
        Machine ID: 7c529d9b459e49f7a92b5b11268d5360
           Boot ID: 7e9fedcdc7d44d39b473e4caae7b8ab5
  Operating System: Debian GNU/Linux 11 (bullseye)
            Kernel: Linux 5.15.84-v8+
      Architecture: arm64

bracka@centauri:/dev $ sudo apt upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Thank You! So far, So good!

Now, the next part of this is to make it use the Pi Zeros through either a bridged or nat configuration. Is this a simple matter of following the procedure to change to one or the other using the documented clusterctrl reconfiguration procedure?

cbridge is preferred

[Tony Brack]

I forgot to post: resulting status ... network to a while to appear! It looks like I just need to configure routing and a DHCP server? Are there any pointers on how to deal with the network gadgets?

bracka@centauri:/dev $ sudo clusterctrl status
clusterhat:1
clusterctrl:False
maxpi:4
throttled:0x0
hat_version:2.4
hat_version_major:2
hat_version_minor:4
hat_size:4
hat_uuid:de91a4ce-ac7f-11e9-a2a3-2a2ae2dbcce4
hat_vendor:8086 Consultancy
hat_product_id:0x0004
hat_alert:0
hat_hub:1
hat_wp:1
hat_led:1
hat_wplink:0
hat_xra1200p:True
p1:1
p2:1
p3:1
p4:1

bracka@centauri:/dev $ ls -al ttypi*
lrwxrwxrwx 1 root root 7 Feb  3 22:53 ttypi1 -> ttyACM2
lrwxrwxrwx 1 root root 7 Feb  3 22:53 ttypi1a -> ttyACM3
lrwxrwxrwx 1 root root 7 Feb  3 22:53 ttypi2 -> ttyACM0
lrwxrwxrwx 1 root root 7 Feb  3 22:53 ttypi2a -> ttyACM1
lrwxrwxrwx 1 root root 7 Feb  3 22:53 ttypi3 -> ttyACM6
lrwxrwxrwx 1 root root 7 Feb  3 22:53 ttypi3a -> ttyACM7
lrwxrwxrwx 1 root root 7 Feb  3 22:53 ttypi4 -> ttyACM4
lrwxrwxrwx 1 root root 7 Feb  3 22:53 ttypi4a -> ttyACM5

bracka@centauri:/dev $ ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.40  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::69eb:75f2:e7ed:524c  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:ad:7d:78  txqueuelen 1000  (Ethernet)
        RX packets 8988  bytes 2403214 (2.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1559  bytes 212408 (207.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ethpi1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.205.182  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::2898:c9db:7443:2e5a  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:01  txqueuelen 1000  (Ethernet)
        RX packets 2215  bytes 66169 (64.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 1613 (1.5 KiB)
        TX errors 58  dropped 0 overruns 0  carrier 0  collisions 0

ethpi2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.60.62  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::5971:411e:e6a8:915  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:02  txqueuelen 1000  (Ethernet)
        RX packets 1396  bytes 43379 (42.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4  bytes 824 (824.0 B)
        TX errors 232  dropped 0 overruns 0  carrier 0  collisions 0

ethpi3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.193.150  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::2a70:4d53:4435:4bdb  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:03  txqueuelen 1000  (Ethernet)
        RX packets 2214  bytes 66141 (64.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25  bytes 5767 (5.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ethpi4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.111.226  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::ff4d:da57:2885:32fb  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:04  txqueuelen 1000  (Ethernet)
        RX packets 500  bytes 17995 (17.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15  bytes 3372 (3.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 32  bytes 3997 (3.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 32  bytes 3997 (3.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.3.158  netmask 255.255.254.0  broadcast 192.168.3.255
        inet6 fe80::c471:7109:b0f:e553  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:ad:7d:79  txqueuelen 1000  (Ethernet)
        RX packets 2206  bytes 899875 (878.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 88  bytes 12649 (12.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[Chris Burton]

HI,

I forgot to post: resulting status ... network to a while to appear! It looks like I just need to configure routing and a DHCP server? Are there any pointers on how to deal with the network gadgets?

If you want to bridge or use NAT for the network then I'm afraid it's not something I've looked at recently when using NetworkManger - I would advise looking at their documentation on bridging with a goal of bridging eth0 with the ethpiX interfaces.

Chris.

 

[Tony Brack]

For a globally accessible cluster (traditional cluster) one would generally want bridging. I am not sure you provided any passthrough rules to the NAT configuration anyway. I cracked open the clusterctrl control file and see what you are doing, so this helps ... easier for me to read than the python code I seem to remember.

Anyhow, this all appears to be working fine now, except the serial (USB) consoles. I noted that the moment I installed cockpit, NetworkManager came up and left dhcpd enabled. This has been working since the 2022-04-04 image came out, but what the heck. I did some research and these ARE conflicts, so this is a Raspbian/Debian/cockpit bug. As a workaround I disabled NetworkManager in favor of dhcpd ... and in the process lost network management in cockpit. Given that I know squat about NetworkManager, I may have to go through the DOCs and reconsider this.

sudo systemctl disable NetworkManager.service

Do you know what happens if I disable dhcpd in favor of NetworkManager? Supposedly NetworkManager also has a built in dhcp daemon of some sort, hence the conflict. They ARE supposed to play nice together, but maybe not so much. If I find out more, once I finalize things, I'll forward something. There are some rules, like NetworkManager is not supposed to meddle if something else is managing the IF.

Thanks for bringing this to my attention!

[Chris Burton]

Hi,

For a globally accessible cluster (traditional cluster) one would generally want bridging. I am not sure you provided any passthrough rules to the NAT configuration anyway. I cracked open the clusterctrl control file and see what you are doing, so this helps ... easier for me to read than the python code I seem to remember.

You can see the NAT rules in the script - https://github.com/burtyb/clusterhat-image/blob/master/build/create.sh#L236 they only apply when traffic is from the internal "fallback" IP range and isn't going out of the br0 interface, so basically it's used when traffic goes out over wlan0 or eth0 (when not in the bridge). When it's bridged the rules do nothing - bridging it setup in /etc/network/interfaces.d/clusterctrl (https://github.com/burtyb/clusterhat-image/tree/master/files/usr/share/clusterctrl clusterctrl.cbridge ).

 

Anyhow, this all appears to be working fine now, except the serial (USB) consoles. I noted that the moment I installed cockpit, NetworkManager came up and left dhcpd enabled. This has been working since the 2022-04-04 image came out, but what the heck. I did some research and these ARE conflicts, so this is a Raspbian/Debian/cockpit bug. As a workaround I disabled NetworkManager in favor of dhcpd ... and in the process lost network management in cockpit. Given that I know squat about NetworkManager, I may have to go through the DOCs and reconsider this.

sudo systemctl disable NetworkManager.service

Do you know what happens if I disable dhcpd in favor of NetworkManager? Supposedly NetworkManager also has a built in dhcp daemon of some sort, hence the conflict. They ARE supposed to play nice together, but maybe not so much. If I find out more, once I finalize things, I'll forward something. There are some rules, like NetworkManager is not supposed to meddle if something else is managing the IF.

The supplied images use dhcpcd for fallback IP on the Pi Zeros if DHCP fails and on the controller when using CBRIDGE if it doesn't get an IP on eth0 it will fallback so the controller can talk to the pi zeros - if using CNAT then it will get the fallback IP on br0 (as it's not bridged externally) which again allows it to talk to the Pi Zeros which will also get fallback IPs. On the controller it's also used for the IP on the VLAN interfaces used for USBBOOT (booting Pi Zeros without SD cards) setup in (/etc/network/interfaces.d/clusterctrl). 

Chris.