bridged network installation

[Stu Midgley]

Afternoon

After plaing around with the cluster hat, I went with a bridged network installation.  The relavent config files...

sdm900@clusterpi0:~ $ cat /etc/systemd/network/bridge-br0.netdev

[NetDev]

Name=br0

Kind=bridge

sdm900@clusterpi0:~ $ cat /etc/systemd/network/br0-members.network

[Match]

Name=eth0 usb*

[Network]

Bridge=br0

sdm900@clusterpi0:~ $ cat /etc/dhcpcd.conf

# A sample configuration for dhcpcd.

# See dhcpcd.conf(5) for details.

# Allow users of this group to interact with dhcpcd via the control socket.

#controlgroup wheel

# Inform the DHCP server of our hostname for DDNS.

hostname

# Use the hardware address of the interface for the Client ID.

clientid

# or

# Use the same DUID + IAID as set in DHCPv6 for DHCPv4 ClientID as per RFC4361.

# Some non-RFC compliant DHCP servers do not reply with this set.

# In this case, comment out duid and enable clientid above.

#duid

# Persist interface configuration when dhcpcd exits.

persistent

# Rapid commit support.

# Safe to enable by default because it requires the equivalent option set

# on the server to actually work.

option rapid_commit

# A list of options to request from the DHCP server.

option domain_name_servers, domain_name, domain_search, host_name

option classless_static_routes

# Respect the network MTU. This is applied to DHCP routes.

option interface_mtu

# Most distributions have NTP support.

#option ntp_servers

# A ServerID is required by RFC2131.

require dhcp_server_identifier

# Generate SLAAC address using the Hardware Address of the interface

#slaac hwaddr

# OR generate Stable Private IPv6 Addresses based from the DUID

slaac private

# Example static IP configuration:

#interface eth0

#static ip_address=192.168.0.10/24

#static ip6_address=fd51:42f8:caae:d92e::ff/64

#static routers=192.168.0.1

#static domain_name_servers=192.168.0.1 8.8.8.8 fd51:42f8:caae:d92e::1

# It is possible to fall back to a static IP if DHCP fails:

# define static profile

#profile static_eth0

#static ip_address=192.168.1.23/24

#static routers=192.168.1.1

#static domain_name_servers=192.168.1.1

# fallback to static profile on eth0

#interface eth0

#fallback static_eth0

denyinterfaces usb* eth* wlan*

interface br0

With all this in place, now on boot, I get a bridged interface and when I turn on the cluster hat and start the pi's all their usb network devices go into the bridge as well... and boot as normal!

5 pi's on my network now.

[thejame...@gmail.com]

Nice! Was you using the CBRIDGE or CNAT image for your controller??

[Stu Midgley]

I was using the default lite images from raspberrypi.org .  I wanted to stay within the "normal" installation as much as possible.

To get ssh active on boot, touch /boot/ssh on the sd card.

Then to get the pi0's to boot using ethernet mode you follow this tutorial (or any other like it on the web)

https://www.circuitbasics.com/raspberry-pi-zero-ethernet-gadget/

this puts the pi into device mode and loads modules to use usb for ethernet on boot.

This ONLY needs to be done on the pi0 images.

The host needs no modification cept for those posted above.

[James Porter]

I'm looking to use a Pi4 as the controlller and four Pi Zeros (non wifi version). I wanted to connect the Pi4 to my network via wifi. So, I'm guessing in order to bridge the Pi zeroes and the pi4, I'll need an additional wifi dongle?

I stumbled across this page yesterday, which suggests that's the case.... 

https://code.metcarob.com/node/174 

--
You received this message because you are subscribed to a topic in the Google Groups "ClusterHAT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clusterhat/Q01z9t2arKM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/40853162-e0d3-41af-9e41-2e6ab5bebdc8n%40googlegroups.com.

[Peter Cross]

No, you'll need to use the CNAT image for the controller. The onboard wifi works fine. You do not need an additional dongle. 

Cheers!

Peter J. Cross
San Antonio, TX

"Experience has taught mankind the necessity of auxiliary precautions"
-James Madison, Federalist Paper No. 51

Please consider the environment before printing this email

You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/CACsi_506gCQMjEnQHtepct6cbBaT-KTdyw%2BiL%2BvC_m5EsPjO0w%40mail.gmail.com.

[James Porter]

So cnat image and create the bridge bridging eth0 and wlan0 and it should work??

To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/CAACnC-zU3n0HLsJnii1GO6bm-2Xp%2BBsTXMzW2Z4pYQCJV-ztFQ%40mail.gmail.com.

[Chris Burton]

Hi,

So cnat image and create the bridge bridging eth0 and wlan0 and it should work??

The cnat image is for NAT - it can be used with either WiFi or Ethernet connection but I'd imagine mostly used for WiFi which doesn't support bridging in the same way you can normally bridge Ethernet.

If you really must bridge WiFi to Ethernet+the Pi Zeros then you should be able to use the CBRIDGE image and mangle the MAC addresses with ebtables (unless you have a weird router in which case it might just work) - see the "Bridging with a wireless NIC" section on https://wiki.debian.org/BridgeNetworkConnections for examples how to use ebtables.

p.s. Please start a new thread for a new issue/question.

[James Porter]

Thanks. (And sorry about the thread mangling!!)

--


You received this message because you are subscribed to a topic in the Google Groups "ClusterHAT" group.


To unsubscribe from this topic, visit https://groups.google.com/d/topic/clusterhat/Q01z9t2arKM/unsubscribe.


To unsubscribe from this group and all its topics, send an email to clusterhat+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/d827f28b-2f9d-43ad-8317-7904ffaa7d0fn%40googlegroups.com.

[Stu Midgley]

Extra, you need to

apt install bridge-utils

 systemctl enable systemd-networkd

[James Porter]

Thanks!!

--
You received this message because you are subscribed to a topic in the Google Groups "ClusterHAT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clusterhat/Q01z9t2arKM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clusterhat+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/e9b42b4e-4a9c-4d40-b6e8-09dd94ee044dn%40googlegroups.com.

[Samuel Montgomery-Blinn]

This is also the approach I am trying, using default raspbian images, manual install of clusterctrl and usbboot-init, and I really appreciate this thread for the simple bridge network setup.

However, I still struggled with the pi0 side of things. I was trying to get them to netboot from my pi4 if possible, rather than install a SD card at all. And the tutorial for pi0 gadget mode's first steps are literally to change files on the SD card.

Anyway, when I clusterctrl on p1, my pi4 /var/log/kern.log shows:

Jan 20 03:34:54 pi4 kernel: [  623.304742] usb 1-1.3.4: USB disconnect, device number 5

Jan 20 03:35:06 pi4 kernel: [  635.156887] usb 1-1.3.4: new full-speed USB device number 6 using xhci_hcd

Jan 20 03:35:06 pi4 kernel: [  635.292769] usb 1-1.3.4: New USB device found, idVendor=0a5c, idProduct=2763, bcdDevice= 0.00

Jan 20 03:35:06 pi4 kernel: [  635.292779] usb 1-1.3.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0

Jan 20 03:35:06 pi4 kernel: [  635.292788] usb 1-1.3.4: Product: BCM2708 Boot

Jan 20 03:35:06 pi4 kernel: [  635.292796] usb 1-1.3.4: Manufacturer: Broadcom

I tried semi-following this tutorial on rpiboot:

https://www.raspberrypi.org/forums/viewtopic.php?f=49&t=199994#p1254966

(Using a BTRFS filesystem on a USB3 external drive.)

But in the end I just said... screw it and put an SD card in each pi0. If you do (or someone else) wants to take this on (rpiboot over USB) in a similarly "vanilla raspbian" with your simple bridge settings... that would be amazing.

-Sam

[Samuel Montgomery-Blinn]

And actually, I ran into trouble getting this to work even with SD cards and enabling gadget mode, I think something isn't quite configured correctly. I made the changes to dhcpc.conf and /etc/systemd/network and when I power on p1, any ideas?

==> /var/log/kern.log <==

Jan 20 05:17:37 pi4 kernel: [ 6786.682528] usb 1-1.3.4: new full-speed USB device number 8 using xhci_hcd

Jan 20 05:17:38 pi4 kernel: [ 6787.642548] usb 1-1.3.4: new high-speed USB device number 9 using xhci_hcd

Jan 20 05:17:38 pi4 kernel: [ 6787.773629] usb 1-1.3.4: New USB device found, idVendor=0525, idProduct=a4a2, bcdDevice= 5.04

Jan 20 05:17:38 pi4 kernel: [ 6787.773639] usb 1-1.3.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0

Jan 20 05:17:38 pi4 kernel: [ 6787.773647] usb 1-1.3.4: Product: RNDIS/Ethernet Gadget

Jan 20 05:17:38 pi4 kernel: [ 6787.773655] usb 1-1.3.4: Manufacturer: Linux 5.4.83+ with 20980000.usb

Jan 20 05:17:38 pi4 kernel: [ 6787.815547] cdc_ether 1-1.3.4:1.0 usb0: register 'cdc_ether' at usb-0000:01:00.0-1.3.4, CDC Ethernet Device, 4a:ea:6f:9d:e3:b7

Jan 20 05:17:38 pi4 kernel: [ 6787.815707] usbcore: registered new interface driver cdc_ether

Jan 20 05:17:38 pi4 kernel: [ 6787.815849] usbcore: registered new interface driver cdc_subset

==> /var/log/daemon.log <==

Jan 20 05:17:38 pi4 systemd-udevd[2475]: Using default interface naming scheme 'v240'.

Jan 20 05:17:38 pi4 dhcpcd[539]: br0: fe80::968f:cfff:fe7c:b550 is unreachable

==> /var/log/kern.log <==

Jan 20 05:17:38 pi4 kernel: [ 6787.904582] br0: port 2(usb0) entered blocking state

Jan 20 05:17:38 pi4 kernel: [ 6787.904590] br0: port 2(usb0) entered disabled state

Jan 20 05:17:38 pi4 kernel: [ 6787.904750] device usb0 entered promiscuous mode

Jan 20 05:17:38 pi4 kernel: [ 6787.906380] br0: port 2(usb0) entered blocking state

Jan 20 05:17:38 pi4 kernel: [ 6787.906389] br0: port 2(usb0) entered forwarding state

Jan 20 05:17:38 pi4 kernel: [ 6787.939340] br0: port 2(usb0) entered disabled state

==> /var/log/daemon.log <==

Jan 20 05:17:38 pi4 systemd-networkd[392]: usb0: Gained carrier

Jan 20 05:17:38 pi4 systemd-networkd[392]: usb0: Configured

Jan 20 05:17:38 pi4 systemd-networkd[392]: usb0: Lost carrier

Jan 20 05:17:39 pi4 dhcpcd[539]: br0: soliciting an IPv6 router

Jan 20 05:17:39 pi4 dhcpcd[539]: br0: Router Advertisement from fe80::968f:cfff:fe7c:b550

Jan 20 05:17:39 pi4 dhcpcd[539]: br0: fe80::968f:cfff:fe7c:b550 is reachable again

Jan 20 05:17:40 pi4 avahi-daemon[307]: Joining mDNS multicast group on interface usb0.IPv6 with address fe80::48ea:6fff:fe9d:e3b7.

Jan 20 05:17:40 pi4 avahi-daemon[307]: New relevant interface usb0.IPv6 for mDNS.

Jan 20 05:17:40 pi4 systemd-networkd[392]: usb0: Gained IPv6LL

Jan 20 05:17:40 pi4 avahi-daemon[307]: Registering new address record for fe80::48ea:6fff:fe9d:e3b7 on usb0.*.

Jan 20 05:17:49 pi4 dhcpcd[539]: br0: fe80::968f:cfff:fe7c:b550 is reachable again

Jan 20 05:17:49 pi4 dhcpcd[539]: br0: fe80::968f:cfff:fe7c:b550 is reachable again

[Samuel Montgomery-Blinn]

Disregard this request for help, as rebooting the host pi4 seems to have resolved the issue. Up and running! Thank you again for this simple bridge setup.

[Stu Midgley]

great, glad it has helped :)

--

You received this message because you are subscribed to the Google Groups "ClusterHAT" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/f5376109-4971-4271-b53d-d57e2f40d9den%40googlegroups.com.

--

Dr Stuart Midgley
sdm...@gmail.com

[Samuel Montgomery-Blinn]

In an effort to be helpful in return, on the pi0 nodes, adding:

ip link set usb0 mtu 7418

To the end of /etc/rc.local (well, above exit 0, of course) significantly increases the transfer speed, or at least, it did for me. (from 500 kb/s to 3 MB/s).

-Sam

[Stu Midgley]

ah, that's a good point.

Where did you get that size from?

You received this message because you are subscribed to a topic in the Google Groups "ClusterHAT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clusterhat/Q01z9t2arKM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/8180d714-553c-4967-a53a-9e88d8b07d6cn%40googlegroups.com.

[Samuel Montgomery-Blinn]

I got the size from this rpi forum thread:

https://www.raspberrypi.org/forums/viewtopic.php?f=49&t=199994#p1254966

I do have another issue that I'm finally starting to be able to at least understand. Something very weird (or at least unexpected, undesired?) is happening when I clusterhat on p1 or clusterhat off p1. My MAC addresses change for the bridge, which causes my routers/switches to get quite unhappy, among other issues. Since I *only* have done your bridge setup, and manual install of clusterhat script, I'm keeping this in this thread for now as possibly others could be encountering this issue?

ifconfig output BEFORE clusterhat on:

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.1.244  netmask 255.255.255.0  broadcast 192.168.1.255

        inet6 fe80::6cf4:cf16:b8bc:8ce3  prefixlen 64  scopeid 0x20<link>

        inet6 2600:1700:6b1:5070:14d6:301f:682:32cd  prefixlen 64  scopeid 0x0<global>

        inet6 2600:1700:6b1:5070::28  prefixlen 128  scopeid 0x0<global>

        ether dc:a6:32:39:02:16  txqueuelen 1000  (Ethernet)

        RX packets 15137  bytes 1135497 (1.0 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 187  bytes 17982 (17.5 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::dea6:32ff:fe39:216  prefixlen 64  scopeid 0x20<link>

        ether dc:a6:32:39:02:16  txqueuelen 1000  (Ethernet)

        RX packets 15141  bytes 1347777 (1.2 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 212  bytes 20726 (20.2 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ifconfig output AFTER clusterhat on:

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.1.244  netmask 255.255.255.0  broadcast 192.168.1.255

        inet6 fe80::6cf4:cf16:b8bc:8ce3  prefixlen 64  scopeid 0x20<link>

        inet6 2600:1700:6b1:5070:14d6:301f:682:32cd  prefixlen 64  scopeid 0x0<global>

        inet6 2600:1700:6b1:5070::28  prefixlen 128  scopeid 0x0<global>

        ether 2e:47:5d:a0:85:d4  txqueuelen 1000  (Ethernet)

        RX packets 15232  bytes 1143413 (1.0 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 192  bytes 18604 (18.1 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::dea6:32ff:fe39:216  prefixlen 64  scopeid 0x20<link>

        ether dc:a6:32:39:02:16  txqueuelen 1000  (Ethernet)

        RX packets 15236  bytes 1357023 (1.2 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 217  bytes 21348 (20.8 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

usb0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

        inet6 fe80::2c47:5dff:fea0:85d4  prefixlen 64  scopeid 0x20<link>

        ether 2e:47:5d:a0:85:d4  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The differences being:

1. after I have a usb0 interface, of course

2. before, both br0 and eth0 have the same ether value (ether dc:a6:32:39:02:16) whereas after, the br0 ether has changed (ether 2e:47:5d:a0:85:d4)

Unfortunately I just don't know enough (yet) about this bridge setup to understand if this is something I configured wrong, or what's going on here. Here are my network config files:

pi@pi4:/etc/systemd/network $ cat br0-members.network

[Match]

Name=eth0 usb*

[Network]

Bridge=br0

pi@pi4:/etc/systemd/network $ cat bridge-br0.netdev

[NetDev]

Name=br0

Kind=bridge

And my /etc/dhcpcd.conf matches yours exactly. Any ideas?

-Sam

[Chris Burton]

Hi, 

1. after I have a usb0 interface, of course

2. before, both br0 and eth0 have the same ether value (ether dc:a6:32:39:02:16) whereas after, the br0 ether has changed (ether 2e:47:5d:a0:85:d4)

Unfortunately I just don't know enough (yet) about this bridge setup to understand if this is something I configured wrong, or what's going on here. Here are my network config files:

You might be able to modify how I do it for the ClusterHAT CBRIDGE image but as I don't use systemd to configure the network interfaces it won't work as is but the config for the br0 interface runs a small copyMAC script to copy the MAC address from eth0 to br0 so you don't have a problem with the MAC/IP changing when the bridge is brought up.

It looks like you want to take a DIY approach but you might want to look at more of the scripts and files I use to build the ClusterHAT images for inspiration to set fixed MAC addresses on the Pi Zeros, etc.

Chris.

[Samuel Montgomery-Blinn]

OK, I finally got things to work. The keys to me were:

1. disabling networking.service and dhcpcd.service

2. specifying the eth0 MAC address for the br0.network file

systemctl mask networking.service

systemctl mask dhcpcd.service

mv /etc/network/interfaces /etc/network/interfaces~

sed -i '1i resolvconf=NO' /etc/resolvconf.conf

ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

systemctl enable systemd-networkd

systemctl enable systemd-resolved

root@pi4:/etc/systemd/network# ls -al

total 24

drwxr-xr-x 2 root root 4096 Jan 26 16:14 .

drwxr-xr-x 5 root root 4096 Jan 11 12:50 ..

-rw-r--r-- 1 root root   31 Jan 26 16:12 01-br0.netdev

-rw-r--r-- 1 root root   40 Jan 26 16:13 02-eth0.network

-rw-r--r-- 1 root root  107 Jan 26 16:14 02-usbX.network

-rw-r--r-- 1 root root   75 Jan 26 16:13 03-br0.network

lrwxrwxrwx 1 root root    9 Jan 11 12:52 99-default.link -> /dev/null

root@pi4:/etc/systemd/network# cat 01-br0.netdev

[NetDev]

Name=br0

Kind=bridge

root@pi4:/etc/systemd/network# cat 02-eth0.network

[Match]

Name=eth0

[Network]

Bridge=br0

root@pi4:/etc/systemd/network# cat 02-usbX.network

[Match]

Name=usb0 usb1 usb2 usb3 usb4 usb5 usb6 usb7 usb8 usb9

[Network]

Bridge=br0

[Link]

MTUBytes=7418

root@pi4:/etc/systemd/network# cat 03-br0.network

[Match]

Name=br0

[Network]

DHCP=yes

[Link]

MACAddress=dc:a6:32:39:02:16

ET VOILA! Now:

root@pi4:/etc/systemd/network# clusterhat on p1

(wait...)

root@pi4:/etc/systemd/network# networkctl

IDX LINK             TYPE               OPERATIONAL SETUP

  1 lo               loopback           carrier     unmanaged

  2 eth0             ether              degraded    configured

  3 wlan0            wlan               off         unmanaged

  4 br0              bridge             routable    configured

  5 usb0             ether              carrier     configured

Everything works now! Woo hoo!

Now, about booting those pi0s over the network... another day. Thanks all.

-Sam