Pi4b Clusterhat 2.4 (bridge) image does not communicate via USB

86 views
Skip to first unread message

Tony Brack

unread,
May 9, 2021, 11:57:11 PMMay 9
to ClusterHAT
Greetings All,

I have used numerous up to date cnat or cbridge images to try to set up my Pi4b with a ClusterHAT 2.4. I am seeing MAC addresses on the Pi4b as well as serial devices, but neither is able to communicate with the Pi ZEROs. I have also tried to attach a Pi ZERO (P20) image with a USB connection, also to no avail. 32 and 64 bit images show the same behavior. I have seen extremely intermittent success. One of the Pis is a Pi ZERO-W, so I should be able to post its view of the connection.

Currently the Pi4b and Pi ZEROs have:
- 2020-08-20-8-ClusterCTRL-arm64-lite-CBRIDGE
- 2020-12-02-1-ClusterCTRL-armhf-lite-p1 to p5

I have applied the latest apt update to all of these images.
I have created a script that reports most of the current state:

   Static hostname: Pi4b-64-cluster
         Icon name: computer
        Machine ID: b659d7240f29459aab330cb98fbf6d05
           Boot ID: 6c0d87d483324813ac0b916d2e3235c6
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 5.10.17-v8+
      Architecture: arm64
-- testing clusterhat --
clusterhat:1
clusterctrl:False
maxpi:4
throttled:0x0
hat_version:2.4
hat_version_major:2
hat_version_minor:4
hat_size:4
hat_uuid:de91a4ce-ac7f-11e9-a2a3-2a2ae2dbcce4
hat_vendor:8086 Consultancy
hat_product_id:0x0004
hat_alert:0
hat_hub:1
hat_wp:1
hat_led:1
hat_wplink:0
hat_xra1200p:True
p1:1
p2:1
p3:1
p4:1
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.120  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::79dc:886f:660d:fa2d  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:ad:52:49  txqueuelen 1000  (Ethernet)
        RX packets 486550  bytes 61897779 (59.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 189856  bytes 43163844 (41.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

brint: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.19.180.254  netmask 255.255.255.0  broadcast 172.19.180.255
        inet6 fe80::6057:baff:fe79:157c  prefixlen 64  scopeid 0x20<link>
        ether 62:57:ba:79:15:7c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 52  bytes 5909 (5.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether dc:a6:32:ad:52:49  txqueuelen 1000  (Ethernet)
        RX packets 486432  bytes 70446842 (67.1 MiB)
        RX errors 0  dropped 89  overruns 0  frame 0
        TX packets 291836  bytes 47540932 (45.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ethpi1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe01  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:01  txqueuelen 1000  (Ethernet)
        RX packets 180  bytes 38348 (37.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1  bytes 134 (134.0 B)
        TX errors 310880  dropped 0 overruns 0  carrier 0  collisions 0

ethpi2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe02  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:02  txqueuelen 1000  (Ethernet)
        RX packets 25232  bytes 740753 (723.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1  bytes 134 (134.0 B)
        TX errors 311112  dropped 0 overruns 0  carrier 0  collisions 0

ethpi3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe03  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:03  txqueuelen 1000  (Ethernet)
        RX packets 25045  bytes 704174 (687.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 1420 (1.3 KiB)
        TX errors 310996  dropped 0 overruns 0  carrier 0  collisions 0

ethpi4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe04  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:04  txqueuelen 1000  (Ethernet)
        RX packets 25043  bytes 704118 (687.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 65  bytes 15411 (15.0 KiB)
        TX errors 277704  dropped 0 overruns 0  carrier 0  collisions 0

ethpi20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe14  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:14  txqueuelen 1000  (Ethernet)
        RX packets 25066  bytes 712878 (696.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34  bytes 8347 (8.1 KiB)
        TX errors 310822  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1562  bytes 2806072 (2.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1562  bytes 2806072 (2.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.15  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 2001:1970:55e2:6500:6b0d:9a41:6f11:7e14  prefixlen 64  scopeid 0x0<global>
        inet6 2001:1970:55e2:6500::72ae  prefixlen 128  scopeid 0x0<global>
        inet6 fe80::ec5c:6172:550a:44b0  prefixlen 64  scopeid 0x20<link>
        ether 6e:aa:e8:7f:6d:57  txqueuelen 1000  (Ethernet)
        RX packets 155447  bytes 160697508 (153.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9447  bytes 1372562 (1.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gw.home.brack.c 0.0.0.0         UG    213    0        0 br0
default         192.168.0.1     0.0.0.0         UG    307    0        0 wlan0
172.19.180.0    0.0.0.0         255.255.255.0   U     0      0        0 brint
192.168.0.0     0.0.0.0         255.255.255.0   U     307    0        0 wlan0
192.168.1.0     0.0.0.0         255.255.255.0   U     213    0        0 br0
PING 192.168.1.120 (192.168.1.120) 56(84) bytes of data.
64 bytes from 192.168.1.120: icmp_seq=1 ttl=64 time=0.104 ms

--- 192.168.1.120 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.104/0.104/0.104/0.000 ms
PING 192.168.1.121 (192.168.1.121) 56(84) bytes of data.
From 192.168.1.120 icmp_seq=1 Destination Host Unreachable

--- 192.168.1.121 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING 192.168.1.122 (192.168.1.122) 56(84) bytes of data.
From 192.168.1.120 icmp_seq=1 Destination Host Unreachable

--- 192.168.1.122 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING 192.168.1.123 (192.168.1.123) 56(84) bytes of data.
From 192.168.1.120 icmp_seq=1 Destination Host Unreachable

--- 192.168.1.123 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING 192.168.1.124 (192.168.1.124) 56(84) bytes of data.
From 192.168.1.120 icmp_seq=1 Destination Host Unreachable

--- 192.168.1.124 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING 192.168.1.125 (192.168.1.125) 56(84) bytes of data.
From 192.168.1.120 icmp_seq=1 Destination Host Unreachable

--- 192.168.1.125 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

? (192.168.0.1) at 02:00:00:00:00:04 [ether] on wlan0
gw.home.brack.ca (192.168.1.1) at 00:0c:29:80:d2:68 [ether] on br0
p1.home.brack.ca (192.168.1.121) at <incomplete> on br0
p2.home.brack.ca (192.168.1.122) at <incomplete> on br0
p3.home.brack.ca (192.168.1.123) at <incomplete> on br0
p4.home.brack.ca (192.168.1.124) at <incomplete> on br0
p20.home.brack.ca (192.168.1.125) at <incomplete> on br0
CNCSBRACKAD3C.home.brack.ca (192.168.1.13) at 18:66:da:40:aa:f8 [ether] on br0
thunderbolt.home.brack.ca (192.168.1.167) at 54:bf:64:2f:5d:16 [ether] on br0
-- listing serial ports --
crw-rw---- 1 root gpio  254, 0 May  9 12:05 /dev/gpiochip0
crw-rw---- 1 root gpio  254, 1 May  9 12:05 /dev/gpiochip1
crw-rw---- 1 root gpio  246, 0 May  9 12:05 /dev/gpiomem
crw-rw---- 1 root video 238, 0 May  9 12:05 /dev/rpivid-h264mem
crw-rw---- 1 root video 240, 0 May  9 12:05 /dev/rpivid-hevcmem
crw-rw---- 1 root video 239, 0 May  9 12:05 /dev/rpivid-intcmem
crw-rw---- 1 root video 237, 0 May  9 12:05 /dev/rpivid-vp9mem
lrwxrwxrwx 1 root root       7 May  9 12:05 /dev/ttypi1 -> ttyACM0
lrwxrwxrwx 1 root root       7 May  9 12:05 /dev/ttypi1a -> ttyACM1
lrwxrwxrwx 1 root root       7 May  9 12:05 /dev/ttypi2 -> ttyACM2
lrwxrwxrwx 1 root root       7 May  9 12:06 /dev/ttypi20 -> ttyACM8
lrwxrwxrwx 1 root root       7 May  9 12:06 /dev/ttypi20a -> ttyACM9
lrwxrwxrwx 1 root root       7 May  9 12:05 /dev/ttypi2a -> ttyACM3
lrwxrwxrwx 1 root root       7 May  9 12:05 /dev/ttypi3 -> ttyACM4
lrwxrwxrwx 1 root root       7 May  9 12:05 /dev/ttypi3a -> ttyACM5
lrwxrwxrwx 1 root root       7 May  9 12:05 /dev/ttypi4 -> ttyACM6
lrwxrwxrwx 1 root root       7 May  9 12:05 /dev/ttypi4a -> ttyACM7

I am about to test with a standard DW2 ethernet and serial 'gadget' Pis and will post the results of that testing as well.

Tony

Tony Brack

unread,
May 10, 2021, 2:04:08 AMMay 10
to ClusterHAT
I took the same Pi ZERO that I used as P20 (updated P5 image) and plugged it in to another machine running another cbridge image without a ClusterHAT and was able to communicate with both the serial interface and the network interface. The working host is a Pi4b running 32 bit:

pi@flirc-4b:~ $ hostnamectl

   Static hostname: flirc-4b
         Icon name: computer
        Machine ID: 062cd7dd815f4306b6b44abe2752e446
           Boot ID: 48819624fc074c1696968d498e9bdd77
  Operating System: Raspbian GNU/Linux 10 (buster)
            Kernel: Linux 5.10.17-v7l+
      Architecture: arm

pi@flirc-4b:~ $ ping -c 1 p20
PING p20.home.brack.ca (192.168.1.125) 56(84) bytes of data.
64 bytes from p20.home.brack.ca (192.168.1.125): icmp_seq=1 ttl=64 time=0.540 ms

--- p20.home.brack.ca ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.540/0.540/0.540/0.000 ms

pi@flirc-4b:~ $ arp -a | sort -k2
? (192.168.0.1) at 02:00:00:00:00:04 [ether] on wlan0
p20.home.brack.ca (192.168.1.125) at 00:22:82:ff:ff:14 [ether] on br0
laptop (192.168.1.167) at 54:bf:64:2f:5d:16 [ether] on br0
gw.home.brack.ca (192.168.1.1) at 00:0c:29:80:d2:68 [ether] on br0

pi@flirc-4b:~ $ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gw.home.brack.c 0.0.0.0         UG    205    0        0 br0
default         192.168.0.1     0.0.0.0         UG    303    0        0 wlan0
172.19.180.0    0.0.0.0         255.255.255.0   U     0      0        0 brint
192.168.0.0     0.0.0.0         255.255.255.0   U     303    0        0 wlan0
192.168.1.0     0.0.0.0         255.255.255.0   U     205    0        0 br0

Thanks

Tony Brack

unread,
May 10, 2021, 2:20:39 AMMay 10
to ClusterHAT
From the failing Pi4b (with a ClusterHAT) - reconnecting p20
pi@Pi4b-64-cluster:~ $ ifconfig ethpi20
ethpi20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe14  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:14  txqueuelen 1000  (Ethernet)
        RX packets 538  bytes 26910 (26.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17  bytes 2516 (2.4 KiB)
        TX errors 4988  dropped 0 overruns 0  carrier 0  collisions 0

pi@Pi4b-64-cluster:~ $ arp p20
Address                  HWtype  HWaddress           Flags Mask            Iface
p20.home.brack.ca                (incomplete)                              br0
pi@Pi4b-64-cluster:~ $ ping -c 1 p20
PING p20.home.brack.ca (192.168.1.125) 56(84) bytes of data.
From Pi4b-64-cluster.home.brack.ca (192.168.1.120) icmp_seq=1 Destination Host Unreachable

--- p20.home.brack.ca ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

-- dmesg output --
[38862.733439] EXT4-fs error (device mmcblk0p2): htree_dirblock_to_tree:1030: inode #41948: block 11294: comm apt-get: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=4294967295, rec_len=65535, name_len=255, size=4096
[46646.658111] usb 1-1.2: USB disconnect, device number 14
[46646.661046] rndis_host 1-1.2:1.0 ethpi20: unregister 'rndis_host' usb-0000:01:00.0-1.2, RNDIS device
[46646.691297] br0: port 6(ethpi20) entered disabled state
[46646.716088] device ethpi20 left promiscuous mode
[46646.716106] br0: port 6(ethpi20) entered disabled state
[50420.624449] usb 1-1.2: new full-speed USB device number 15 using xhci_hcd
[50433.188669] usb 1-1.2: new high-speed USB device number 16 using xhci_hcd
[50433.290118] usb 1-1.2: New USB device found, idVendor=3171, idProduct=0020, bcdDevice= 1.00
[50433.290137] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[50433.290150] usb 1-1.2: Product: ClusterCTRL
[50433.290162] usb 1-1.2: Manufacturer: 8086 Consultancy
[50433.290174] usb 1-1.2: SerialNumber: 20
[50433.308170] rndis_host 1-1.2:1.0 eth1: register 'rndis_host' at usb-0000:01:00.0-1.2, RNDIS device, 00:22:82:ff:fe:14
[50433.315441] cdc_acm 1-1.2:1.2: ttyACM8: USB ACM device
[50433.318496] cdc_acm 1-1.2:1.4: ttyACM9: USB ACM device
[50433.397261] rndis_host 1-1.2:1.0 ethpi20: renamed from eth1
[50433.595102] br0: port 6(ethpi20) entered blocking state
[50433.595115] br0: port 6(ethpi20) entered disabled state
[50433.595444] device ethpi20 entered promiscuous mode
[50433.617777] br0: port 6(ethpi20) entered blocking state
[50433.617790] br0: port 6(ethpi20) entered forwarding state
[50452.053278] cdc_acm 1-1.2:1.2: failed to set dtr/rts
[50457.173346] cdc_acm 1-1.2:1.2: failed to set dtr/rts
[50470.997640] cdc_acm 1-1.2:1.4: failed to set dtr/rts
[50479.445795] cdc_acm 1-1.2:1.2: failed to set dtr/rts
[50490.965950] cdc_acm 1-1.2:1.4: failed to set dtr/rts

Tony Brack

unread,
May 10, 2021, 5:53:09 AMMay 10
to ClusterHAT
I think I have found something: The interface on the Pi ZERO (P1) shows up as:

usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.19.181.1  netmask 255.255.255.0  broadcast 172.19.181.255
        inet6 fe80::33f0:28a0:23fb:ac27  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:ff:01  txqueuelen 1000  (Ethernet)
        RX packets 116  bytes 15114 (14.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 535  bytes 101198 (98.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The same interface on the other side of the USB on the Pi4b shows up as:

ethpi1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe01  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:01  txqueuelen 1000  (Ethernet)
        RX packets 401  bytes 56352 (55.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1  bytes 134 (134.0 B)
        TX errors 429490  dropped 0 overruns 0  carrier 0  collisions 0

p1.home.brack.ca (192.168.1.121) at <incomplete> on br0

pi@Pi4b-64-cluster:/boot $ ping -c 1 p1
PING p1.home.brack.ca (192.168.1.121) 56(84) bytes of data.
From Pi4b-64-cluster.home.brack.ca (192.168.1.120) icmp_seq=1 Destination Host Unreachable

--- p1.home.brack.ca ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

Tony Brack

unread,
May 10, 2021, 10:39:54 PMMay 10
to ClusterHAT
Attached is a shell script that reports (hopefully) all the debug data in the troubleshooting guide as well as a copy of the most recent report from my configuration. I am going to retry with ClusterCTRL-2020-02-13-lite-1-CBRIDGE as the Pi4b image.
report.sh
report.txt

Tony Brack

unread,
May 11, 2021, 12:10:45 AMMay 11
to ClusterHAT
... setting up the old version on second uSD card results in a working bridged server (albeit 32 bit)
- see attached report-old.txt 

report-old.txt

Chris Burton

unread,
May 11, 2021, 3:00:16 AMMay 11
to ClusterHAT
Hi,

It looks like you have an IP on wlan0 and a non-standard fallback/DHCP IP on br0 so are you connecting both or have you changed the fallback IP in /etc/dhcpcd.conf on the controller?

If you've connected both then I'd try using one or the other not both and if you use WiFi then you'll need to use the CNAT image.

If you've set the br0 fallback IP on the controller you'll need to alter it on the Pi Zeros too.

If br0 is getting the IP via DHCP on Ethernet with CBRIDGE but the Pi Zero usb0 isn't getting an IP then you can alter the fallback IP in /etc/dhcpcd.conf to the correct IP over serial.

You should be able to log into the Pi Zero P1 over USB Gadget serial from the controller with "screen /dev/ttypi1", press enter and wait a few seconds for the login prompt (press CTRL-a, then press k and y to exit screen).

The MAC addresses should be different on both sides of the usb0/ethpiX USB Gadget Ethernet interface.

Chris.

Tony Brack

unread,
May 11, 2021, 10:51:41 PMMay 11
to ClusterHAT
Hi Chris,

Thanks for getting back.

I am connecting to 2 networks and have not altered the configuration in any way, so I will try to answer you as best I can. I have tried multiple "vanilla" cbridge images with the same result, so I have no doubt that I am missing something stupid and obvious. I provided the script so that it can be updated to capture all the important stuff for the next guy! This is not my only cluster, and the rest work. (I think they're running CNAT but not sure)

I will try to follow you:

> It looks like you have an IP on wlan0 and a non-standard fallback/DHCP IP on br0 so are you connecting both or
> have you changed the fallback IP in /etc/dhcpcd.conf on the controller?

I have 2 routers. wlan0 is connected to my ISP router (192.168.0.1/24) and gets its address from a DHCP static mapping, and br0 is getting a similar IP address from my external gateway that bypasses my ISP router (192.169.1.1/24). I have not touched anything either here or on my 2 working clusters.

> If you've connected both then I'd try using one or the other not both and if you use WiFi then you'll need to use the CNAT image.

I have done this with and without WiFi enabled, ... no difference, but I will try again later and advise. CNAT has the same (similar) problem with all kernels tested. Yes: I have tried multiple.

> If you've set the br0 fallback IP on the controller you'll need to alter it on the Pi Zeros too.

No, I have not explicitly set anything on BR0. It is running DHCP (defaulted) and communicates with the outside world properly. I am running headless with ssh and cockpit as well. That is how I am generating these reports.

> If br0 is getting the IP address via DHCP on Ethernet with CBRIDGE but the Pi Zero usb0 isn't getting an IP then you
> can alter the fallback IP in /etc/dhcpcd.conf to the correct IP over serial.

No, the serial ports are not working either. The 5th Pi (mapped to P20) works properly when I connect to my other cluster, or a system that is just running the CBRIDGE image without a hat (used for testing), or even my laptop for that matter, but not when connected on this machine. I am going to try a fresh uSD on a different spare Pi I have kicking around. Again, all of these are getting statically mapped addresses via DHCP/MAC. I took painstaking caution to ensure we also do not have a collision. Again, P20 which works properly everywhere else is the proof. It is a P5 image and is NOT going through the HAT.

*** interesting ***
I checked P1 (4th Pi is a Pi ZERO W, ... Why'd you do that? LOL) and logged in over the WiFi. 
It is NOT *successfully* getting its address from the DHCP server and falling back ... ?
(incidentally: it still did not work properly from a CNAT either)

>>> I am going to try this MicroSD on another Pi ZERO W off my laptop.
 
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.19.181.1  netmask 255.255.255.0  broadcast 172.19.181.255
        inet6 fe80::33f0:28a0:23fb:ac27  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:ff:01  txqueuelen 1000  (Ethernet)
        RX packets 116  bytes 15114 (14.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 535  bytes 101198 (98.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The same interface on the other side of the USB on the Pi4b shows up as:

ethpi1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:82ff:feff:fe01  prefixlen 64  scopeid 0x20<link>
        ether 00:22:82:ff:fe:01  txqueuelen 1000  (Ethernet)
        RX packets 401  bytes 56352 (55.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1  bytes 134 (134.0 B)
        TX errors 429490  dropped 0 overruns 0  carrier 0  collisions 0

p1.home.brack.ca (192.168.1.121) at <incomplete> on br0

pi@Pi4b-64-cluster:/boot $ ping -c 1 p1
PING p1.home.brack.ca (192.168.1.121) 56(84) bytes of data.
From Pi4b-64-cluster.home.brack.ca (192.168.1.120) icmp_seq=1 Destination Host Unreachable

--- p1.home.brack.ca ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
*** interesting - This tells me the bridge isn't bridging ... ***

I am attaching dhcpd.conf from the CBRIDGE (Pi4b-64-cluster). It seems to be mostly vanilla and has not been edited.

> You should be able to log into the Pi Zero P1 over USB Gadget serial from the controller
> with "screen /dev/ttypi1", press enter and wait a few seconds for the login prompt (press CTRL-a,
> then press k and y to exit screen).

screen (anything) just sits unresponsive, but the devices ARE created

No, none of these work, ... not even P20 which should be the easiest one. If I unplug P20 and plug it in anywhere else I can communicate via the serial gadget, but not in this CBRIDGE with either of these: 

lrwxrwxrwx 1 root root 7 Feb 13  2020 /dev/ttypi20 -> ttyACM0
lrwxrwxrwx 1 root root 7 Feb 13  2020 /dev/ttypi20a -> ttyACM1

You will note that the correct IP addresses were registered in the DNS server on the router (pfSense) and that arp shows the correct MAC addresses mapping to the expected IP addresses:

pi@Pi4b-64-cluster:~ $ arp -a | sort -k2
gw.home.brack.ca (192.168.1.1) at 00:0c:29:80:d2:68 [ether] on br0
p1.home.brack.ca (192.168.1.121) at 00:22:82:ff:ff:01 [ether] on br0
p2.home.brack.ca (192.168.1.122) at 00:22:82:ff:ff:02 [ether] on br0
p3.home.brack.ca (192.168.1.123) at 00:22:82:ff:ff:03 [ether] on br0
p4.home.brack.ca (192.168.1.124) at 00:22:82:ff:ff:04 [ether] on br0
p20.home.brack.ca (192.168.1.125) at 00:22:82:ff:ff:14 [ether] on br0

In most of my report script outputs I get the correct IP addresses and no mention of the MAC address in arp:

gw.home.brack.ca (192.168.1.1) at 00:0c:29:80:d2:68 [ether] on br0
p1.home.brack.ca (192.168.1.121) at <incomplete> on br0
p2.home.brack.ca (192.168.1.122) at <incomplete> on br0
p3.home.brack.ca (192.168.1.123) at <incomplete> on br0
p4.home.brack.ca (192.168.1.124) at 00:22:82:ff:ff:04 [ether] on br0
p20.home.brack.ca (192.168.1.125) at <incomplete> on br0

pi@Pi4b-64-cluster:~ $ ping p1
PING p1.home.brack.ca (192.168.1.121) 56(84) bytes of data.
^C
--- p1.home.brack.ca ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 106ms

pi@Pi4b-64-cluster:~ $ ping p2
PING p2.home.brack.ca (192.168.1.122) 56(84) bytes of data.

--- p2.home.brack.ca ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 84ms

^C
pi@Pi4b-64-cluster:~ $ ping p3
PING p3.home.brack.ca (192.168.1.123) 56(84) bytes of data.
^C
--- p3.home.brack.ca ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 198ms

pi@Pi4b-64-cluster:~ $ ping p4
PING p4.home.brack.ca (192.168.1.124) 56(84) bytes of data.
^C
--- p4.home.brack.ca ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 129ms

pi@Pi4b-64-cluster:~ $ ping p20
PING p20.home.brack.ca (192.168.1.125) 56(84) bytes of data.
^C
--- p20.home.brack.ca ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 92ms

The network is working: (silly, but to be thorough)

pi@Pi4b-64-cluster:~ $ ping flirc-4b
PING flirc-4b.home.brack.ca (192.168.1.85) 56(84) bytes of data.
64 bytes from flirc-4b.home.brack.ca (192.168.1.85): icmp_seq=1 ttl=64 time=0.586 ms
64 bytes from flirc-4b.home.brack.ca (192.168.1.85): icmp_seq=2 ttl=64 time=0.216 ms
64 bytes from flirc-4b.home.brack.ca (192.168.1.85): icmp_seq=3 ttl=64 time=0.209 ms
64 bytes from flirc-4b.home.brack.ca (192.168.1.85): icmp_seq=4 ttl=64 time=0.217 ms
64 bytes from flirc-4b.home.brack.ca (192.168.1.85): icmp_seq=5 ttl=64 time=0.232 ms
^C
--- flirc-4b.home.brack.ca ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 100ms
rtt min/avg/max/mdev = 0.209/0.292/0.586/0.147 ms

> The MAC addresses should be different on both sides of the usb0/ethpiX
> USB Gadget Ethernet interface.

They are.  00:22:82:ff:fe:01 versus 00:22:82:ff:ff:01

Thanks again,
Tony
dhcpcd.conf

Tony Brack

unread,
May 14, 2021, 8:57:45 AMMay 14
to ClusterHAT
Hi All,

I finally gave up and did 2 things:

1. swapped out the physical Pi
2. switched to CNAT

The cluster is now up and running, but I have to go through either the Pi ZERO W or the CNAT host to access any of the other Pis. This was not the intent. This configuration still does not work properly in a bridged configuration, including the serial ports.

Tony

Reply all
Reply to author
Forward
0 new messages