Updated test images (2022-04-04-1)

180 views
Skip to first unread message

Chris Burton

unread,
Apr 24, 2022, 4:43:14 PM4/24/22
to ClusterHAT
I've uploaded new test images based on 2022-04-04 Bullseye (32/64-bit) and Buster to https://dist.8086.net/clusterctrl/testing/

When using the Bullseye images please remember you need to set a user/password before booting if you're using it headless otherwise you won't be able to login (the Px images need this too as they can't setup over the serial console).

This can either be done using Raspberry Pi Imager or manually with a userconf.txt file in the boot partition (see https://www.raspberrypi.com/news/raspberry-pi-bullseye-update-april-2022/ ).

The Bullseye usbboot archives also need a userconf.txt. Once the usbboot archive has been extracted you can use something like this to create a password for p1.

cd /var/lib/clusterctrl/nfs/p1
set +o history # Turn off bash history temporarily
# Replace myusername and mypassword with your details
echo "myusername:"$(echo 'mypassword' | openssl passwd -6 -stdin) > boot/userconf.txt
set -o history # Turn history back on again

You can then "touch boot/ssh" to enable SSH if needed and run "usbboot-init 1" as normal and it's ready to power on p1.

It can take 5 minutes or more for a first boot of all images so please be patient!

If you have problems with the images or suggestions, please open a ticket at https://secure.8086.net/billing/submitticket.php?step=2&deptid=1 or create a new thread with details of the image filename, model of Pi you're using, any additional commands you've ran etc. so I can try to replicate the problem.

Chris.

Ben Heininger

unread,
Apr 26, 2022, 10:32:25 AM4/26/22
to ClusterHAT
Hi,
Do I need to have them all on the Buster Images?
I just realized that I downloaded the buster P# images but the Bullseye cBridge x64 image(I thought I needed the x64 version for the Pi4 - Is that true?).

Images Downloaded:
2022-04-04-1-bullseye-ClusterCTRL-arm64-CBRIDGE.zip
2022-04-04-1-buster-ClusterCTRL-armhf-lite-p1.zip
2022-04-04-1-buster-ClusterCTRL-armhf-lite-p2.zip
2022-04-04-1-buster-ClusterCTRL-armhf-lite-p3.zip
2022-04-04-1-buster-ClusterCTRL-armhf-lite-p4.zip

I cant get the P# to get an IP address from my router. 
I see IPs marked with * as their name but I am unable to SSH to them.
I created /boot/ssh before installing the cards.
I used the Pi Imager and set the username and password the same for all 5 devices.
I can SSH into the Pi4 but not the zeros (I tried from my PC and from the Pi4 controller).
The Pi4 Controller again has the cmdline.old:
cmdline.old:
console=serial0,115200 console=tty1 root=PARTUUID=50a67c3e-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles init=/usr/sbin/reconfig-clusterctrl cbridge systemd.run=/boot/firstrun.sh systemd.run_success_action=reboot systemd.unit=kernel-command-line.target

cmdline.txt:
console=serial0,115200 console=tty1 root=PARTUUID=1a5b1414-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles

I haven't checked the P# cmdline files yet as it is nearly midnight and I'm going to bed.
Is there a writeup about how to setup the cBridge? (All I can find is cNAT Tutorials)
And I definitely waited 5+ minutes before doing anything

Thanks
Ben

Ben Heininger

unread,
Apr 26, 2022, 11:00:18 AM4/26/22
to ClusterHAT
A quick update.
I managed to find the P1-4 mac address and assigned them all static ips and rebooted the router and the cluster (Complete) and now they all appear with ip addresses in my router but i can only ssh into the cBridge but not the P1-4.

ben@cbridge:~ $ ssh b...@192.168.1.11
ssh: connect to host 192.168.1.11 port 22: Connection timed out

Ben

Chris Burton

unread,
Apr 27, 2022, 8:02:03 AM4/27/22
to ClusterHAT
Hi, 
Do I need to have them all on the Buster Images?
I just realized that I downloaded the buster P# images but the Bullseye cBridge x64 image(I thought I needed the x64 version for the Pi4 - Is that true?).

The 64-bit images can only be used on 64-bit capable hardware (3B,3B+,3A+,4,400,CM3,CM3+,CM4,Zero 2 W), the others can be used on all models of Pi.
 
Images Downloaded:
2022-04-04-1-bullseye-ClusterCTRL-arm64-CBRIDGE.zip
2022-04-04-1-buster-ClusterCTRL-armhf-lite-p1.zip
2022-04-04-1-buster-ClusterCTRL-armhf-lite-p2.zip
2022-04-04-1-buster-ClusterCTRL-armhf-lite-p3.zip
2022-04-04-1-buster-ClusterCTRL-armhf-lite-p4.zip

I cant get the P# to get an IP address from my router. 
I see IPs marked with * as their name but I am unable to SSH to them.
I created /boot/ssh before installing the cards.
I used the Pi Imager and set the username and password the same for all 5 devices.
I can SSH into the Pi4 but not the zeros (I tried from my PC and from the Pi4 controller).

After powering on with "clusterctrl on" waiting say 10 minutes to make sure they've properly booted up in case you have slow SD cards and then what does the output these commands look like?

cat /proc/device-tree/hat/* ; echo ; dmesg|grep "cdc\|usb" ; lsusb -t; sudo ifconfig -a ; sudo brctl show  ;clusterctrl status
 
You can either send to sup...@8086.net.uk or reply here with the info.

Is there a writeup about how to setup the cBridge? (All I can find is cNAT Tutorials)

From a Cluster HAT networking side of things there shouldn't be anything to setup - once the Pi Zeros are booted they should get an IP from the same DHCP server as the controller Pi so you can login directly from the rest of the network - from there you can setup as you need.

Chris.

Ben Heininger

unread,
Apr 27, 2022, 9:27:47 AM4/27/22
to ClusterHAT
Ok,
So despite creating the SSH file after creating the images to the SD cards the initial setup deleted the SSH file.
So I recreated the SSH file just now turned on the Pi4 and SSHed to that successfully and then turned on the clusterhat.
(I also noticed that while recreating the SSH files that each P1-4's cmdline.txt file was also .OLDed and replaced with a default cmdline.txt.)
Once all the lights were on the P1-4 I tried to ping them with no success.
I checked the router and they have been assigned their static IPs.
I cannot ping them and I cannot SSH to them.

Am I missing something?

Thanks
Ben

Ben Heininger

unread,
Apr 27, 2022, 9:32:05 AM4/27/22
to ClusterHAT
Sorry, I missed your update Chris.
cat /proc/device-tree/hat/* ; echo ; dmesg|grep "cdc\|usb" ; lsusb -t; sudo ifconfig -a ; sudo brctl show  ;clusterctrl status
hatZC4:ClusterHAT0x00040x002516aeb902-9d28-11ea-bb37-0242ac1300028086 Consultancy
[    0.176754] usbcore: registered new interface driver usbfs
[    0.176817] usbcore: registered new interface driver hub
[    0.176882] usbcore: registered new device driver usb
[    0.177245] usb_phy_generic phy: supply vcc not found, using dummy regulator
[    1.396473] usbcore: registered new interface driver r8152
[    1.396537] usbcore: registered new interface driver lan78xx
[    1.396591] usbcore: registered new interface driver smsc95xx
[    1.444886] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.15
[    1.444908] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.444924] usb usb1: Product: xHCI Host Controller
[    1.444938] usb usb1: Manufacturer: Linux 5.15.32-v8+ xhci-hcd
[    1.444951] usb usb1: SerialNumber: 0000:01:00.0
[    1.446926] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.15
[    1.446946] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.446960] usb usb2: Product: xHCI Host Controller
[    1.446974] usb usb2: Manufacturer: Linux 5.15.32-v8+ xhci-hcd
[    1.446987] usb usb2: SerialNumber: 0000:01:00.0
[    1.475705] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.15
[    1.475726] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.475741] usb usb3: Product: xHCI Host Controller
[    1.475755] usb usb3: Manufacturer: Linux 5.15.32-v8+ xhci-hcd
[    1.475768] usb usb3: SerialNumber: fe9c0000.xhci
[    1.477113] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.477378] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.15
[    1.477397] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.477413] usb usb4: Product: xHCI Host Controller
[    1.477426] usb usb4: Manufacturer: Linux 5.15.32-v8+ xhci-hcd
[    1.477439] usb usb4: SerialNumber: fe9c0000.xhci
[    1.479687] usbcore: registered new interface driver uas
[    1.479816] usbcore: registered new interface driver usb-storage
[    1.490658] usbcore: registered new interface driver usbhid
[    1.490671] usbhid: USB HID core driver
[    1.699235] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[    1.849952] usb 1-1: New USB device found, idVendor=2109, idProduct=3431, bcdDevice= 4.21
[    1.849988] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[    1.850004] usb 1-1: Product: USB2.0 Hub
[    2.151259] usb 1-1.2: new high-speed USB device number 3 using xhci_hcd
[    2.251776] usb 1-1.2: New USB device found, idVendor=0424, idProduct=2504, bcdDevice= 0.01
[    2.251816] usb 1-1.2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    2.551236] usb 1-1.2.1: new low-speed USB device number 4 using xhci_hcd
[    2.663657] usb 1-1.2.1: New USB device found, idVendor=413c, idProduct=2106, bcdDevice= 1.01
[    2.663693] usb 1-1.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.663709] usb 1-1.2.1: Product: Dell QuietKey Keyboard
[    2.663724] usb 1-1.2.1: Manufacturer: DELL
[    2.673680] input: DELL Dell QuietKey Keyboard as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2.1/1-1.2.1:1.0/0003:413C:2106.0001/input/input0
[    2.735994] hid-generic 0003:413C:2106.0001: input,hidraw0: USB HID v1.10 Keyboard [DELL Dell QuietKey Keyboard] on usb-0000:01:00.0-1.2.1/input0
[    2.815245] usb 1-1.2.2: new low-speed USB device number 5 using xhci_hcd
[    2.931439] usb 1-1.2.2: New USB device found, idVendor=046d, idProduct=c077, bcdDevice=72.00
[    2.931471] usb 1-1.2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.931487] usb 1-1.2.2: Product: USB Optical Mouse
[    2.931501] usb 1-1.2.2: Manufacturer: Logitech
[    2.940243] input: Logitech USB Optical Mouse as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2.2/1-1.2.2:1.0/0003:046D:C077.0002/input/input1
[    2.940776] hid-generic 0003:046D:C077.0002: input,hidraw1: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:01:00.0-1.2.2/input0
[    6.154722] usbcore: registered new interface driver brcmfmac
[    9.211248] usb 1-1.4: new high-speed USB device number 6 using xhci_hcd
[    9.314394] usb 1-1.4: New USB device found, idVendor=05e3, idProduct=0608, bcdDevice=85.36
[    9.314432] usb 1-1.4: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[    9.314448] usb 1-1.4: Product: USB2.0 Hub
[   10.262068] usb 1-1.4: USB disconnect, device number 6
[   11.767265] usb 1-1.4: new high-speed USB device number 7 using xhci_hcd
[   11.868815] usb 1-1.4: New USB device found, idVendor=05e3, idProduct=0608, bcdDevice=85.36
[   11.868844] usb 1-1.4: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[   11.868860] usb 1-1.4: Product: USB2.0 Hub
[  407.603114] usb 1-1.4.4: new full-speed USB device number 8 using xhci_hcd
[  418.759109] usb 1-1.4.4: new high-speed USB device number 9 using xhci_hcd
[  418.860303] usb 1-1.4.4: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[  418.860335] usb 1-1.4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  418.860352] usb 1-1.4.4: Product: ClusterCTRL
[  418.860366] usb 1-1.4.4: Manufacturer: 8086 Consultancy
[  418.860380] usb 1-1.4.4: SerialNumber: 1
[  418.934425] usbcore: registered new interface driver cdc_ether
[  418.944339] rndis_host 1-1.4.4:1.0 eth1: register 'rndis_host' at usb-0000:01:00.0-1.4.4, RNDIS device, 00:22:82:ff:fe:01
[  418.944505] usbcore: registered new interface driver rndis_host
[  418.951167] cdc_acm 1-1.4.4:1.2: ttyACM0: USB ACM device
[  418.953538] cdc_acm 1-1.4.4:1.4: ttyACM1: USB ACM device
[  418.953998] usbcore: registered new interface driver cdc_acm
[  418.954017] usbcore: registered new interface driver rndis_wlan
[  418.954214] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[  419.103048] usb 1-1.4.1: new full-speed USB device number 10 using xhci_hcd
[  422.399077] usb 1-1.4.1: new high-speed USB device number 11 using xhci_hcd
[  422.500376] usb 1-1.4.1: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[  422.500392] usb 1-1.4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  422.500398] usb 1-1.4.1: Product: ClusterCTRL
[  422.500422] usb 1-1.4.1: Manufacturer: 8086 Consultancy
[  422.500435] usb 1-1.4.1: SerialNumber: 4
[  422.508257] rndis_host 1-1.4.1:1.0 eth1: register 'rndis_host' at usb-0000:01:00.0-1.4.1, RNDIS device, 00:22:82:ff:fe:04
[  422.510409] cdc_acm 1-1.4.1:1.2: ttyACM2: USB ACM device
[  422.513941] cdc_acm 1-1.4.1:1.4: ttyACM3: USB ACM device
[  422.743126] usb 1-1.4.2: new high-speed USB device number 12 using xhci_hcd
[  422.844282] usb 1-1.4.2: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[  422.844302] usb 1-1.4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  422.844308] usb 1-1.4.2: Product: ClusterCTRL
[  422.844313] usb 1-1.4.2: Manufacturer: 8086 Consultancy
[  422.844318] usb 1-1.4.2: SerialNumber: 3
[  422.851960] rndis_host 1-1.4.2:1.0 eth1: register 'rndis_host' at usb-0000:01:00.0-1.4.2, RNDIS device, 00:22:82:ff:fe:03
[  422.854059] cdc_acm 1-1.4.2:1.2: ttyACM4: USB ACM device
[  422.856160] cdc_acm 1-1.4.2:1.4: ttyACM5: USB ACM device
[  423.083110] usb 1-1.4.3: new high-speed USB device number 13 using xhci_hcd
[  423.184282] usb 1-1.4.3: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[  423.184304] usb 1-1.4.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  423.184310] usb 1-1.4.3: Product: ClusterCTRL
[  423.184315] usb 1-1.4.3: Manufacturer: 8086 Consultancy
[  423.184320] usb 1-1.4.3: SerialNumber: 2
[  423.192043] rndis_host 1-1.4.3:1.0 eth1: register 'rndis_host' at usb-0000:01:00.0-1.4.3, RNDIS device, 00:22:82:ff:fe:02
[  423.194386] cdc_acm 1-1.4.3:1.2: ttyACM6: USB ACM device
[  423.196616] cdc_acm 1-1.4.3:1.4: ttyACM7: USB ACM device
[  434.911229] Modules linked in: rndis_wlan cdc_acm rndis_host cdc_ether rfcomm cmac algif_hash aes_arm64 algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc bridge 8021q garp stp llc nft_chain_nat xt_MASQUERADE nf_nat nft_counter xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 snd_soc_hdmi_codec nft_compat nf_tables nfnetlink brcmfmac brcmutil cfg80211 rfkill v3d gpu_sched raspberrypi_hwmon vc4 cec drm_kms_helper bcm2835_v4l2(C) bcm2835_isp(C) bcm2835_codec(C) snd_soc_core i2c_brcmstb snd_compress snd_bcm2835(C) bcm2835_mmal_vchiq(C) snd_pcm_dmaengine v4l2_mem2mem videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common vc_sm_cma(C) snd_pcm i2c_bcm2835 videodev mc snd_timer rpivid_mem snd syscopyarea sysfillrect sysimgblt fb_sys_fops nvmem_rmem uio_pdrv_genirq uio i2c_dev drm drm_panel_orientation_quirks backlight fuse nfsd ip_tables x_tables ipv6
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/0p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
            |__ Port 2: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 4: Dev 7, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 11, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 1: Dev 11, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 1: Dev 11, If 1, Class=CDC Data, Driver=rndis_host, 480M
            |__ Port 1: Dev 11, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 11, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 11, If 0, Class=Communications, Driver=rndis_host, 480M
            |__ Port 2: Dev 12, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 2: Dev 12, If 0, Class=Communications, Driver=rndis_host, 480M
            |__ Port 2: Dev 12, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 12, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 12, If 1, Class=CDC Data, Driver=rndis_host, 480M
            |__ Port 2: Dev 12, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 3: Dev 13, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 3: Dev 13, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 3: Dev 13, If 0, Class=Communications, Driver=rndis_host, 480M
            |__ Port 3: Dev 13, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 3: Dev 13, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 3: Dev 13, If 1, Class=CDC Data, Driver=rndis_host, 480M
            |__ Port 4: Dev 9, If 0, Class=Communications, Driver=rndis_host, 480M
            |__ Port 4: Dev 9, If 1, Class=CDC Data, Driver=rndis_host, 480M
            |__ Port 4: Dev 9, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 4: Dev 9, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 4: Dev 9, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 4: Dev 9, If 5, Class=CDC Data, Driver=cdc_acm, 480M
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.10  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::1bf5:272d:ec9:107  prefixlen 64  scopeid 0x20<link>
        ether e4:5f:01:94:fa:d1  txqueuelen 1000  (Ethernet)
        RX packets 7672  bytes 945268 (923.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 371  bytes 50394 (49.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

brint: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.19.180.254  netmask 255.255.255.0  broadcast 172.19.180.255
        inet6 fe80::f064:bfff:fe74:70d5  prefixlen 64  scopeid 0x20<link>
        ether f2:64:bf:74:70:d5  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 63  bytes 7134 (6.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether e4:5f:01:94:fa:d1  txqueuelen 1000  (Ethernet)
        RX packets 7710  bytes 1104834 (1.0 MiB)
        RX errors 0  dropped 125  overruns 0  frame 0
        TX packets 2411  bytes 144508 (141.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ethpi1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe01  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:01  txqueuelen 1000  (Ethernet)
        RX packets 516  bytes 16799 (16.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 1905 (1.8 KiB)
        TX errors 5510  dropped 0 overruns 0  carrier 0  collisions 0

ethpi2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe02  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:02  txqueuelen 1000  (Ethernet)
        RX packets 516  bytes 16799 (16.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 112  bytes 19493 (19.0 KiB)
        TX errors 5800  dropped 0 overruns 0  carrier 0  collisions 0

ethpi3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe03  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:03  txqueuelen 1000  (Ethernet)
        RX packets 516  bytes 16976 (16.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 91  bytes 16242 (15.8 KiB)
        TX errors 5800  dropped 0 overruns 0  carrier 0  collisions 0

ethpi4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe04  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:04  txqueuelen 1000  (Ethernet)
        RX packets 515  bytes 16771 (16.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17  bytes 2724 (2.6 KiB)
        TX errors 5916  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 21  bytes 2379 (2.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21  bytes 2379 (2.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether e4:5f:01:94:fa:d3  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bridge name     bridge id               STP enabled     interfaces
br0             8000.e45f0194fad1       no              eth0
                                                        ethpi1
                                                        ethpi2
                                                        ethpi3
                                                        ethpi4
brint           8000.000000000000       no
clusterhat:1
clusterctrl:False
maxpi:4
throttled:0x0
hat_version:2.5
hat_version_major:2
hat_version_minor:5
hat_size:4
hat_uuid:16aeb902-9d28-11ea-bb37-0242ac130002
hat_vendor:8086 Consultancy
hat_product_id:0x0004
hat_alert:0
hat_hub:1
hat_wp:1
hat_led:1
hat_wplink:0
hat_xra1200p:True
p1:1
p2:1
p3:1
p4:1

Chris Burton

unread,
Apr 27, 2022, 5:25:47 PM4/27/22
to ClusterHAT
Hi, 
[  434.911229] Modules linked in: rndis_wlan cdc_acm rndis_host cdc_ether rfcomm cmac algif_hash aes_arm64 algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc bridge 8021q garp stp llc nft_chain_nat xt_MASQUERADE nf_nat nft_counter xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 snd_soc_hdmi_codec nft_compat nf_tables nfnetlink brcmfmac brcmutil cfg80211 rfkill v3d gpu_sched raspberrypi_hwmon vc4 cec drm_kms_helper bcm2835_v4l2(C) bcm2835_isp(C) bcm2835_codec(C) snd_soc_core i2c_brcmstb snd_compress snd_bcm2835(C) bcm2835_mmal_vchiq(C) snd_pcm_dmaengine v4l2_mem2mem videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common vc_sm_cma(C) snd_pcm i2c_bcm2835 videodev mc snd_timer rpivid_mem snd syscopyarea sysfillrect sysimgblt fb_sys_fops nvmem_rmem uio_pdrv_genirq uio i2c_dev drm drm_panel_orientation_quirks backlight fuse nfsd ip_tables x_tables ipv6

This line points to a kernel problem - does "dmesg|grep WATCHDOG" show a line like "NETDEV WATCHDOG: ethpi1 (rndis_host): transmit queue 0 timed out" ?

If you run "clusterctrl hub off;sudo rmmod cdc_ether rndis_wlan rndis_host;sleep 3;clusterctrl  hub on" and wait a few seconds can "ping p1.local" (or the other Pi Zeros you've powered on)?

Chris.

Ben Heininger

unread,
Apr 28, 2022, 6:17:30 AM4/28/22
to ClusterHAT
Hi Chris,
I ran your script above and received the following:

ben@cbridge:~ $ dmesg|grep WATCHDOG
[  450.019377] NETDEV WATCHDOG: ethpi3 (rndis_host): transmit queue 0 timed out


ben@cbridge:~ $ clusterctrl hub off;sudo rmmod cdc_ether rndis_wlan rndis_host;sleep 3;clusterctrl  hub on
rmmod: ERROR: Module cdc_ether is in use by: rndis_host

ben@cbridge:~ $ ping p1.local -c3
PING p1.local (192.168.1.11) 56(84) bytes of data.
64 bytes from 192.168.1.11 (192.168.1.11): icmp_seq=3 ttl=64 time=0.591 ms

--- p1.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.505/0.599/0.701/0.080 ms

ben@cbridge:~ $ ping p2.local -c3
PING p2.local (192.168.1.12) 56(84) bytes of data.
64 bytes from 192.168.1.12 (192.168.1.12): icmp_seq=3 ttl=64 time=0.602 ms

--- p2.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.550/0.821/1.313/0.348 ms

ben@cbridge:~ $ ping p3.local -c3
ping: p3.local: Name or service not known


ben@cbridge:~ $ ping p4.local -c3
PING p4.local (192.168.1.14) 56(84) bytes of data.
64 bytes from 192.168.1.14 (192.168.1.14): icmp_seq=3 ttl=64 time=0.500 ms

--- p4.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.500/0.805/1.373/0.401 ms

It looks like P3 isnt booting or something.
I can now ssh into all of them except P3.
After connecting to all of them I ran the dmesg|grep WATCHDOG and received the same message that is at the top.

What does that mean about P3?
Should I try the Bullseye P1-4 Images instead?

Ben Heininger

unread,
Apr 28, 2022, 6:31:35 AM4/28/22
to ClusterHAT
At boot up before i see the huge RPi logo I get this message at the top of the screen:
[    1.478079] hub 4-0:1.0: config failed, hub doesn't have any ports! (err -19)

I ran dmesg|grep hub and found all of these:

ben@cbridge:~ $ dmesg|grep hub
[    0.172819] usbcore: registered new interface driver hub
[    1.445648] hub 1-0:1.0: USB hub found
[    1.445745] hub 1-0:1.0: 1 port detected
[    1.447643] hub 2-0:1.0: USB hub found
[    1.447709] hub 2-0:1.0: 4 ports detected
[    1.476346] hub 3-0:1.0: USB hub found
[    1.476410] hub 3-0:1.0: 1 port detected
[    1.478016] hub 4-0:1.0: USB hub found
[    1.478079] hub 4-0:1.0: config failed, hub doesn't have any ports! (err -19)
[    1.856046] hub 1-1:1.0: USB hub found
[    1.856263] hub 1-1:1.0: 4 ports detected
[    2.261770] hub 1-1.2:1.0: USB hub found
[    2.261930] hub 1-1.2:1.0: 4 ports detected
[    2.446868] hub 1-1.4:1.0: USB hub found
[    2.447228] hub 1-1.4:1.0: 4 ports detected
[   55.659414] hub 1-1.4:1.0: USB hub found
[   55.665053] hub 1-1.4:1.0: 4 ports detected

Not sure if this helps but here it is.

I rebooted the RPi4 controller and all the ssh connections to the P1,2,4 remained up on my PC.
After the reboot i am still unable to ping or ssh into the P3.

Chris Burton

unread,
Apr 28, 2022, 7:49:00 AM4/28/22
to ClusterHAT
Hi, 
After the reboot i am still unable to ping or ssh into the P3. 
 
 For the P3 problem I'd advise rewriting the SD card if you haven't made any changes to it yet.
 
Should I try the Bullseye P1-4 Images instead?
 
Unless you have a specific reason to use Buster I would advise Bullseye as it will receive support for longer.

Chris. 

Ben Heininger

unread,
Apr 28, 2022, 7:56:57 AM4/28/22
to ClusterHAT
Hi Chris,
Thanks for your patience.

I figured since I couldn't get P3 to work or the other 3 PZeros to update I reimaged the P1-4 with the bulseye images.
I had to run your cmd:
 clusterctrl hub off;sudo rmmod cdc_ether rndis_wlan rndis_host;sleep 3;clusterctrl  hub on
to turn them on.
I can now ping all 4 Pi Zeros and SSH to them.

I took the cluster hat off and all the Zeros and swapped 3 & 4 around to test incase P3 was dead or if the usb port for P3 was dead.
I also used the imager to set-up SSH.

I now get the following from now:
ben@cbridge:~ $ dmesg|grep WATCHDOG
[  503.012438] NETDEV WATCHDOG: ethpi1 (rndis_host): transmit queue 0 timed out
[  503.012455] NETDEV WATCHDOG: ethpi2 (rndis_host): transmit queue 0 timed out
Thanks again,
Ben

Ben Heininger

unread,
Apr 28, 2022, 8:54:30 AM4/28/22
to ClusterHAT
Following the email I sent you just before,
After I managed to ping and SSH into all the Zeros, I "sudo apt update && sudo apt full-upgrade -y" and then rebooted them, I still had to run  "clusterctrl hub off;sudo rmmod cdc_ether rndis_wlan rndis_host;sleep 3;clusterctrl  hub on" and wait a minute to be able to ssh back to them.
They dont seem to come back online without that code.

Chris Burton

unread,
Apr 28, 2022, 4:08:44 PM4/28/22
to ClusterHAT
Hi,
Following the email I sent you just before,
After I managed to ping and SSH into all the Zeros, I "sudo apt update && sudo apt full-upgrade -y" and then rebooted them, I still had to run  "clusterctrl hub off;sudo rmmod cdc_ether rndis_wlan rndis_host;sleep 3;clusterctrl  hub on" and wait a minute to be able to ssh back to them.
They dont seem to come back online without that code.
 
Thanks for the update,  I'm going through the kernels to see if I can see when the problem started.

Chris.

Ben Heininger

unread,
Apr 29, 2022, 12:01:41 AM4/29/22
to ClusterHAT
Are these 2 commands the same thing?
clusterctrl hub on
and
clusterhat on

Chris Burton

unread,
Apr 29, 2022, 7:35:52 AM4/29/22
to ClusterHAT
Hi,
Are these 2 commands the same thing?
clusterctrl hub on
and
clusterhat on

clusterctrl/clusterhat are the same command they're just symlinks.

"clutserctrl hub on" / "clusterctrl hub off" turns the USB hub on/off the Cluster HAT on (it's used by "clusterctrl init" which turns the USB hub off/on to work out which USB port the ClusterHAT is plugged into to get the USB path for p1-p4 usbboot).

"clusterctrl on"  / "clusterctrl off" turns power on/off for P1-P4

Chris.

Ben Heininger

unread,
May 9, 2022, 10:45:39 PM5/9/22
to ClusterHAT
Hi Chris,
Now when I run "clusterctrl hub off;sudo rmmod cdc_ether rndis_wlan rndis_host;sleep 3;clusterctrl  hub on" I receive an error:

rmmod: ERROR: Module cdc_ether is in use by: rndis_host
and I can no longer ssh into the Pi Zeros.
Thanks
Ben

Ben Heininger

unread,
May 10, 2022, 8:42:33 PM5/10/22
to ClusterHAT
Hi,
I managed to ssh into the Zeros, despite having fast SD cards they took about 15min to load.
I still receive this error when booting the controller Pi4
dmesg|grep hub
[    1.461928] hub 4-0:1.0: config failed, hub doesn't have any ports! (err -19)

Chris Burton

unread,
May 14, 2022, 12:50:55 PM5/14/22
to ClusterHAT
Hi, 
Now when I run "clusterctrl hub off;sudo rmmod cdc_ether rndis_wlan rndis_host;sleep 3;clusterctrl  hub on" I receive an error:
rmmod: ERROR: Module cdc_ether is in use by: rndis_host
and I can no longer ssh into the Pi Zeros.

It looks like ModemManager is causing the "NETDEV WATCHDOG: ethpi1 (rndis_host): transmit queue 0 timed out" problem again.

If you run the following commands on the controller and reboot are you able to communicate with p1-4 as normal?

sudo systemctl disable ModemManager.service
sudo systemctl stop ModemManager.service
sudo apt -y purge modemmanager

Chris. 

Ben Heininger

unread,
May 15, 2022, 1:12:10 AM5/15/22
to ClusterHAT
That seems to have fixed it Chris.
I still receive "[    1.461475] hub 4-0:1.0: config failed, hub doesn't have any ports! (err -19)" at boot up but so far so good.
I'll let you know if i come across any other issues.

Thanks again
Regards 
Ben

Reply all
Reply to author
Forward
0 new messages