Cannot access nodes after fresh 32-bit Bullseye (ClusterHAT) images

[baq...@gmail.com]

Hello!

It has been a while for me to get back to the forum for assistance. Things were running smoothly with Buster but the upgrade bug bit me. I’ve forgotten how to enable the nodes. Minicom is installed at /usr/bin but since ttyp1 etc. are not accessible, I cannot logon to the nodes.

IP addresses are from pi-hole (DHCP) server and there is no Wi-Fi.

Please direct me to some documentation. Thanks.

Kind regards.

Hardware

RPI3 + 4 RPi0 v1.3

ClusterHAT 1.x (red board)

Images

32-bit Bullseye CBRIDGE

32-bit Bullseye p1, p2, p3, p4

SSH access to RPi3 enabled during imaging with rpi_imager

see session below

$ clusterctrl on p1

$ clusterctrl on p2

$ clusterctrl on p3

$ clusterctrl on p4

$ ls /dev/ttypi*

ls: cannot access '/dev/ttypi*': No such file or directory

$ lsusb

Bus 001 Device 015: ID 3171:0020 8086 Consultancy ClusterCTRL

Bus 001 Device 020: ID 3171:0020 8086 Consultancy ClusterCTRL

Bus 001 Device 019: ID 3171:0020 8086 Consultancy ClusterCTRL

Bus 001 Device 017: ID 3171:0020 8086 Consultancy ClusterCTRL

Bus 001 Device 004: ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 001 Device 003: ID 0424:ec00 Microchip Technology, Inc. (formerly SMSC) SMSC9512/9514 Fast Ethernet Adapter

Bus 001 Device 002: ID 0424:9514 Microchip Technology, Inc. (formerly SMSC) SMC9514 Hub

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

$ hostnamectl

Static hostname: raspbari20

Icon name: computer

Machine ID: 9238d63688754aca8380e6ed362b7e2b

Boot ID: 68ae26a318084c2988ea9311c2c4d6a5

Operating System: Raspbian GNU/Linux 11 (bullseye)

Kernel: Linux 5.15.61-v7+

Architecture: arm

$ lsusb -t

/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M

|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M

|__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=smsc95xx, 480M

|__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M

|__ Port 4: Dev 15, If 4, Class=Communications, Driver=, 480M

|__ Port 4: Dev 15, If 2, Class=Communications, Driver=, 480M

|__ Port 4: Dev 15, If 0, Class=Communications, Driver=, 480M

|__ Port 4: Dev 15, If 5, Class=CDC Data, Driver=, 480M

|__ Port 4: Dev 15, If 3, Class=CDC Data, Driver=, 480M

|__ Port 4: Dev 15, If 1, Class=CDC Data, Driver=, 480M

|__ Port 2: Dev 19, If 5, Class=CDC Data, Driver=, 480M

|__ Port 2: Dev 19, If 3, Class=CDC Data, Driver=, 480M

|__ Port 2: Dev 19, If 1, Class=CDC Data, Driver=, 480M

|__ Port 2: Dev 19, If 4, Class=Communications, Driver=, 480M

|__ Port 2: Dev 19, If 2, Class=Communications, Driver=, 480M

|__ Port 2: Dev 19, If 0, Class=Communications, Driver=, 480M

|__ Port 3: Dev 20, If 0, Class=Communications, Driver=, 480M

|__ Port 3: Dev 20, If 5, Class=CDC Data, Driver=, 480M

|__ Port 3: Dev 20, If 3, Class=CDC Data, Driver=, 480M

|__ Port 3: Dev 20, If 1, Class=CDC Data, Driver=, 480M

|__ Port 3: Dev 20, If 4, Class=Communications, Driver=, 480M

|__ Port 3: Dev 20, If 2, Class=Communications, Driver=, 480M

|__ Port 1: Dev 17, If 4, Class=Communications, Driver=, 480M

|__ Port 1: Dev 17, If 2, Class=Communications, Driver=, 480M

|__ Port 1: Dev 17, If 0, Class=Communications, Driver=, 480M

|__ Port 1: Dev 17, If 5, Class=CDC Data, Driver=, 480M

|__ Port 1: Dev 17, If 3, Class=CDC Data, Driver=, 480M

|__ Port 1: Dev 17, If 1, Class=CDC Data, Driver=, 480M

$ ifconfig

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.16.0.14 netmask 255.255.254.0 broadcast 172.16.1.255

inet6 fe80::9f7b:9fe5:845b:d8be prefixlen 64 scopeid 0x20<link>

ether b8:27:eb:97:16:13 txqueuelen 1000 (Ethernet)

RX packets 511693 bytes 514195544 (490.3 MiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 159485 bytes 11136204 (10.6 MiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

brint: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.19.180.254 netmask 255.255.255.0 broadcast 172.19.180.255

inet6 fe80::1c75:10ff:fe2f:b86 prefixlen 64 scopeid 0x20<link>

ether 1e:75:10:2f:0b:86 txqueuelen 1000 (Ethernet)

RX packets 0 bytes 0 (0.0 B)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 144 bytes 16345 (15.9 KiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

ether b8:27:eb:97:16:13 txqueuelen 1000 (Ethernet)

RX packets 536843 bytes 515631863 (491.7 MiB)

RX errors 0 dropped 1688 overruns 0 frame 0

TX packets 159485 bytes 13040508 (12.4 MiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

inet 127.0.0.1 netmask 255.0.0.0

inet6 ::1 prefixlen 128 scopeid 0x10<host>

loop txqueuelen 1000 (Local Loopback)

RX packets 59 bytes 6137 (5.9 KiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 59 bytes 6137 (5.9 KiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

$ which minicom

/usr/bin/minicom

[Chris Burton]

Hi,

It has been a while for me to get back to the forum for assistance. Things were running smoothly with Buster but the upgrade bug bit me. I’ve forgotten how to enable the nodes. Minicom is installed at /usr/bin but since ttyp1 etc. are not accessible, I cannot logon to the nodes.

The first thing I would suggest trying is after powering on the Pi Zeros to wait at least 10-15 minutes as it could take that long for their initial reboots depending on the speed of the SD cards.

Do you know the filenames/version numbers of the ClusterCTRL images you're using?

Chris. 

[baq...@gmail.com]

Hello Chris,

The images are 32-bit for all Pi models:

CBRIDGE - Desktop Controller (FULL)

P1 - Lite Bullseye image P1

P2 - Lite Bullseye image P2

P3 - Lite Bullseye image P3

P4 - Lite Bullseye image P4

In other words:

2022-09-06-2-bullseye-ClusterCTRL-armhf-full-CBRIDGE.img
2022-09-06-2-bullseye-ClusterCTRL-armhf-lite-p1.img
2022-09-06-2-bullseye-ClusterCTRL-armhf-lite-p2.img
2022-09-06-2-bullseye-ClusterCTRL-armhf-lite-p3.img
2022-09-06-2-bullseye-ClusterCTRL-armhf-lite-p4.img

I did change the default admin credentials. It worked for the host (i.e. Controller) but I'd have to locate a mini-HDMI plug that works (I have two but they're not working!) to test the Zeros. Also, the Zeros (& the 2s) have been unavailable since July 2021 (at the only retail store in my area). (Eben promised that a fresh supply would cross the pond soon but we're still wating.) So I'm unable to swap any of these right now for a sanity check.

Back to your other suggestion about waiting 1-15 minutes; I did that twice but it didn't seem to change anything.

Couldn't run the following since wiringpi is no longer in the official repository:

gpio write 22 1

Using the troubleshooting guide, here are the responses for my situation:

1. The ALERT LED goes from dim to off
2. The output from the command clusterctrl status follows:

$ clusterctrl status
clusterhat:1
clusterctrl:False
maxpi:4
throttled:0x50000
hat_version:1.3
hat_version_major:1
hat_version_minor:3
hat_size:4
hat_uuid:622aef87-17d6-4022-a370-781e1c3fe2b9
hat_vendor:8086 Consultancy
hat_product_id:0x0004
hat_alert:0
p1:1
p2:1
p3:1
p4:1
$

3. Hub control not supported on Cluster HAT v1.x

4. Done

5. The P1-P4 LEDs come on

6. Output provided in initial post above

7. Can interchange SD cards to re-image if that is a required route towards issue resolution

8. No /dev/ttypi* devices detected

Thanks again for your guidance.

Regards.

[baq...@gmail.com]

Hello Chris,

I now get to see the Pn nodes with:

$ ls /dev/ttypi*
/dev/ttypi1   /dev/ttypi2   /dev/ttypi3   /dev/ttypi4
/dev/ttypi1a  /dev/ttypi2a  /dev/ttypi3a  /dev/ttypi4a
$ ping p1
ping: p1: Name or service not known
$

but I cannot minicom p1 like I used to in the past. minicom simply freezes and never display the login prompt.

Your guidance will be sincerely appreciated.

Regards.

[baq...@gmail.com]

Also,

$ uname -a ;cat /proc/device-tree/hat/* ; echo ; dmesg|grep "cdc\|usb" ; lsusb -t; sudo ifconfig -a ; sudo brctl show  ;clusterctrl status
Linux raspbari20 5.15.84-v7+ #1613 SMP Thu Jan 5 11:59:48 GMT 2023 armv7l GNU/Linux
hatZC4:ClusterHAT0x00040x0013622aef87-17d6-4022-a370-781e1c3fe2b98086 Consultancy
[    0.158447] usbcore: registered new interface driver usbfs
[    0.158514] usbcore: registered new interface driver hub
[    0.158585] usbcore: registered new device driver usb
[    0.158958] usb_phy_generic phy: supply vcc not found, using dummy regulator
[    0.159168] usb_phy_generic phy: dummy supplies not allowed for exclusive requests
[    1.969816] usbcore: registered new interface driver lan78xx
[    1.969885] usbcore: registered new interface driver smsc95xx
[    2.899448] dwc_otg 3f980000.usb: DWC OTG Controller
[    2.899489] dwc_otg 3f980000.usb: new USB bus registered, assigned bus number 1
[    2.899539] dwc_otg 3f980000.usb: irq 89, io mem 0x00000000
[    2.899984] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.15
[    2.900009] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.900028] usb usb1: Product: DWC OTG Controller
[    2.900045] usb usb1: Manufacturer: Linux 5.15.84-v7+ dwc_otg_hcd
[    2.900061] usb usb1: SerialNumber: 3f980000.usb
[    2.902217] usbcore: registered new interface driver usb-storage
[    2.909597] usbcore: registered new interface driver usbhid
[    2.909612] usbhid: USB HID core driver
[    3.242129] usb 1-1: new high-speed USB device number 2 using dwc_otg
[    3.482414] usb 1-1: New USB device found, idVendor=0424, idProduct=9514, bcdDevice= 2.00
[    3.482453] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.802115] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[    3.932531] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00, bcdDevice= 2.00
[    3.932569] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.075135] SMSC LAN8700 usb-001:003:01: attached PHY driver (mii_bus:phy_addr=usb-001:003:01, irq=POLL)
[    4.076067] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-3f980000.usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:97:16:13
[    4.172120] usb 1-1.4: new high-speed USB device number 4 using dwc_otg
[    4.303527] usb 1-1.4: New USB device found, idVendor=05e3, idProduct=0608, bcdDevice=85.36
[    4.303556] usb 1-1.4: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[    4.303576] usb 1-1.4: Product: USB2.0 Hub
[   11.032857] usbcore: registered new interface driver brcmfmac
[  334.729804] usb 1-1.4.4: new full-speed USB device number 5 using dwc_otg
[  347.061111] usb 1-1.4.4: new high-speed USB device number 6 using dwc_otg
[  347.192409] usb 1-1.4.4: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[  347.192445] usb 1-1.4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  347.192466] usb 1-1.4.4: Product: ClusterCTRL
[  347.192483] usb 1-1.4.4: Manufacturer: 8086 Consultancy
[  347.192500] usb 1-1.4.4: SerialNumber: 1
[  347.283654] usbcore: registered new interface driver cdc_ether
[  347.296212] rndis_host 1-1.4.4:1.0 eth1: register 'rndis_host' at usb-3f980000.usb-1.4.4, RNDIS device, 00:22:82:ff:fe:01
[  347.297467] usbcore: registered new interface driver rndis_host
[  347.306106] cdc_acm 1-1.4.4:1.2: ttyACM0: USB ACM device
[  347.310344] cdc_acm 1-1.4.4:1.4: ttyACM1: USB ACM device
[  347.310465] usbcore: registered new interface driver cdc_acm
[  347.310474] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[  347.316295] usbcore: registered new interface driver rndis_wlan
[  347.491178] usb 1-1.4.1: new full-speed USB device number 7 using dwc_otg
[  354.311748] usb 1-1.4.1: new high-speed USB device number 8 using dwc_otg
[  354.443199] usb 1-1.4.1: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[  354.443234] usb 1-1.4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  354.443254] usb 1-1.4.1: Product: ClusterCTRL
[  354.443271] usb 1-1.4.1: Manufacturer: 8086 Consultancy
[  354.443288] usb 1-1.4.1: SerialNumber: 4
[  354.447847] rndis_host 1-1.4.1:1.0 eth1: register 'rndis_host' at usb-3f980000.usb-1.4.1, RNDIS device, 00:22:82:ff:fe:04
[  354.470131] cdc_acm 1-1.4.1:1.2: ttyACM2: USB ACM device
[  354.478283] cdc_acm 1-1.4.1:1.4: ttyACM3: USB ACM device
[  354.801812] usb 1-1.4.2: new high-speed USB device number 9 using dwc_otg
[  354.932955] usb 1-1.4.2: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[  354.932978] usb 1-1.4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  354.932989] usb 1-1.4.2: Product: ClusterCTRL
[  354.932997] usb 1-1.4.2: Manufacturer: 8086 Consultancy
[  354.933006] usb 1-1.4.2: SerialNumber: 3
[  354.940751] rndis_host 1-1.4.2:1.0 eth1: register 'rndis_host' at usb-3f980000.usb-1.4.2, RNDIS device, 00:22:82:ff:fe:03
[  354.953420] cdc_acm 1-1.4.2:1.2: ttyACM4: USB ACM device
[  354.957872] cdc_acm 1-1.4.2:1.4: ttyACM5: USB ACM device
[  355.261841] usb 1-1.4.3: new high-speed USB device number 10 using dwc_otg
[  355.413373] usb 1-1.4.3: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[  355.413404] usb 1-1.4.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  355.413419] usb 1-1.4.3: Product: ClusterCTRL
[  355.413432] usb 1-1.4.3: Manufacturer: 8086 Consultancy
[  355.413445] usb 1-1.4.3: SerialNumber: 2
[  355.423362] rndis_host 1-1.4.3:1.0 eth1: register 'rndis_host' at usb-3f980000.usb-1.4.3, RNDIS device, 00:22:82:ff:fe:02
[  355.434936] cdc_acm 1-1.4.3:1.2: ttyACM6: USB ACM device
[  355.445428] cdc_acm 1-1.4.3:1.4: ttyACM7: USB ACM device
[  882.638705] usbcore: deregistering interface driver rndis_wlan
[  882.703484] usbcore: deregistering interface driver rndis_host
[  882.705106] rndis_host 1-1.4.3:1.0 ethpi2: unregister 'rndis_host' usb-3f980000.usb-1.4.3, RNDIS device
[  882.767478] rndis_host 1-1.4.2:1.0 ethpi3: unregister 'rndis_host' usb-3f980000.usb-1.4.2, RNDIS device
[  882.847720] rndis_host 1-1.4.1:1.0 ethpi4: unregister 'rndis_host' usb-3f980000.usb-1.4.1, RNDIS device
[  882.937353] rndis_host 1-1.4.4:1.0 ethpi1: unregister 'rndis_host' usb-3f980000.usb-1.4.4, RNDIS device

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=smsc95xx, 480M
        |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M

            |__ Port 4: Dev 6, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 4: Dev 6, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 4: Dev 6, If 0, Class=Communications, Driver=, 480M
            |__ Port 4: Dev 6, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 4: Dev 6, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 4: Dev 6, If 1, Class=CDC Data, Driver=, 480M
            |__ Port 2: Dev 9, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 9, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 9, If 1, Class=CDC Data, Driver=, 480M
            |__ Port 2: Dev 9, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 2: Dev 9, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 2: Dev 9, If 0, Class=Communications, Driver=, 480M
            |__ Port 3: Dev 10, If 0, Class=Communications, Driver=, 480M
            |__ Port 3: Dev 10, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 3: Dev 10, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 3: Dev 10, If 1, Class=CDC Data, Driver=, 480M
            |__ Port 3: Dev 10, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 3: Dev 10, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 8, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 8, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 8, If 0, Class=Communications, Driver=, 480M
            |__ Port 1: Dev 8, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 1: Dev 8, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 1: Dev 8, If 1, Class=CDC Data, Driver=, 480M

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.14  netmask 255.255.254.0  broadcast 172.16.1.255
        inet6 fe80::9f7b:9fe5:845b:d8be  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:97:16:13  txqueuelen 1000  (Ethernet)

        RX packets 17220  bytes 2324670 (2.2 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 1079  bytes 203039 (198.2 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

brint: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.19.180.254  netmask 255.255.255.0  broadcast 172.19.180.255

        inet6 fe80::784c:84ff:fe23:f886  prefixlen 64  scopeid 0x20<link>
        ether 7a:4c:84:23:f8:86  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 69  bytes 7722 (7.5 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether b8:27:eb:97:16:13  txqueuelen 1000  (Ethernet)

        RX packets 19447  bytes 2447319 (2.3 MiB)
        RX errors 0  dropped 141  overruns 0  frame 0
        TX packets 1237  bytes 236138 (230.6 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)

        RX packets 23  bytes 2543 (2.4 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 23  bytes 2543 (2.4 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bridge name    bridge id        STP enabled    interfaces
br0        8000.b827eb971613    no        eth0
brint        8000.000000000000    no        

clusterhat:1
clusterctrl:False
maxpi:4
throttled:0x50000
hat_version:1.3
hat_version_major:1
hat_version_minor:3
hat_size:4
hat_uuid:622aef87-17d6-4022-a370-781e1c3fe2b9
hat_vendor:8086 Consultancy
hat_product_id:0x0004
hat_alert:0
p1:1
p2:1
p3:1
p4:1

[baq...@gmail.com]

Hello Chris,

Please consider the issue closed. Once again a self-inflicted user/operator, I presume.

I have been dutifully switching off and on the main board (since I have headless access) and using to clusterctrl to turn on the nodes for once every few days since I started this thread. The nodes were never detectable. But earlier this morning, zenmap/nmap reported all four nodes up an running for the first time. I've signed on to p1 in headless mode right now and will subsequently step through the other nodes in due course. The nodes are the back-end of my infrastructure support and administration.

Bottom line, your images are perfect. Perhaps as you have indicated previously, some cable issue at my end. Again thanks for your work and support. Long live ClusterHAT!!!

Regards.