New combined SD / usbboot(rpiboot) images ready for testing

[Chris Burton]

Hi,

The test images are currently available at https://dist.8086.net/clusterctrl/testing/ (zip files will show up as they're created).

The images work with the ClusterHAT and the ClusterCTRL devices I've been working on.

They support booting Pi Zeros/A+/3A+/CM1/CM3/CM3+ from either SD (and eMMC for compute modules) or without an SD card using rpiboot, as before the controller can be setup with either a bridged (CBRIDGE) or NAT (CNAT) network.

If you have some spare time please give them a try and let me know of success or failure either here or via https://secure.8086.net/portal/submitticket.php?step=2&deptid=1

*For this test release only*

- SSH is enabled on both controller and Pi Zero images.

- Auto login is enabled via serial console (controller) and ttyGS0 (Pi Zeros) accessible using /dev/ttypiX (you can use "minicom p1" or "screen /dev/ttypi2" etc. as before).

Login: pi/clusterctrl

Controller images can be used on SD or USB Flash (3B+ or 3B with OTP set).

Pi Zero "pX" images can be used on Pi Zeros/A+/3A+/CM1/CM3/CM3+.

Not all 3A+ are able to boot via USB, sadly there's no further information on which can boot other than the first few thousand can't :(.

Image Changes

To reconfigure the SD image type you need to append " init=/sbin/reconfig-clusterctrl X" to /boot/cmdline.txt where X is "cnat" (NAT) or "cbridge" (BRIDGE) for the controller and "pX" where X is 1-252 for the Pi Zeros.

As above the hostname of the bridged controller image is now "cbridge" and not "controller".

Merges in the functionality of the usbboot test image.

Image filenames are now prefixed with ClusterCTRL.

Added restrictions to limit NFS access (/etc/hosts.allow & /etc/hosts.deny) to the internal network IP range 172.19.180.0/24.

rpiboot runs as a service "clustercrtl-rpiboot", progress can be seen in "/var/log/daemon.log".

"clusterhat init" no longer runs from rc.local, there is now a "clusterctrl-init" service.

Gadgets are setup as Ethernet and two serial ports (the first port /dev/ttypiX is used for a serial console. I plan to use the second port /dev/ttypiXa to allow comms with the Pi Zero from the controller).

Switched from g_cdc to RNDIS for USB Gadget Ethernet.

The above change allows the SD images to be used on a Pi Zero/A+/3A+/CM1/CM3/CM3+ connected to a Windows 10 or Linux machine allowing access via Ethernet/Serial.

USB Gadget interface is setup using libcomposite. With the SD image this is setup in the "clusterctrl-composite" service, for usbboot it's setup within initramfs allong with internal networks VLAN 10 interface.

initramfs are automatically rebuilt when the kernel is upgraded (usbboot only)

Filesystem automatically expanded to size of SD/USB.

Probably lots of other things I can't remember right now.

Network Setup

The network is configured the same way as the previous ClusterHAT images. The usbboot root filesystem uses initramfs allowing usb0 on the Pi Zero to be used for the external network (the same as the SD images) and usb0.10 for the internal network for the NFS filesystem.

Controller types

cnat.local (CNAT)

- Network address translation is setup to work with a controller connected via either WiFi or Ethernet. Due to the fallback IPs the controller will have IP 172.19.181.254 on br0 and 172.19.181.X (X=pX number) for the Pi Zeros.

cbridge.local (CBRIDGE)

- Pi Zeros network is bridged with controllers Ethernet port, controller+Pi Zeros get an IP from DHCP server on local network. Will fallback to 172.19.181.254 on the controller and 172.19.181.X (X=pX number) for the Pi Zeros if no DHCP server is available on the network.

Network Bridges

br0 - External network

- CNAT - Controller has the fallback IP 172.19.181.254 on br0, the Pi Zeros will also fallback to 172.19.181.X where X is the pX number.

- CBRIDGE - The controller and Pi Zeros will obtain an IP address from the DHCP server on the local Ethernet connection.

[ When using WiFi on the controller you must use the CNAT image to enable the Pi Zeros to access the internet ]

brint - Internal network only used for usbboot on Pi Zeros/etc.

- 172.19.180.254 on the controller and Pi Zeros (usbboot only) have 172.19.180.X where X is pX.

How to boot Pi Zeros without SD cards.

Overview

The root filesystems are stored in /var/lib/clusterctrl/nfs/

In previous releases of the test image the USB cable to the ClusterHAT needed to be in a specific USB port on the controller. The USB path is now detected on boot for Cluster HAT v2.x . For those using Cluster HAT v1.x you will need to configure the USB port by uncommenting the appropriate line in "/etc/default/clusterctrl" to allow the USB path links in "/var/lib/clusterctrl/boot/" to be update on boot (you can also run "sudo clusterctrl init" to update the links without rebooting).

Filesystem Setup

To start with the root filesystems in /var/lib/clusterctrl/nfs/pX/ are empty.

First download a usbboot archive file from https://dist.8086.net/clusterctrl/testing/ with wget

Then reconfigure the root filesystem

P=1 # The pX number

DIR=/var/lib/clusterctrl/nfs/p$P

tar -C $DIR -axf <path to>/ClusterCTRL-2019-04-08-lite-1-usbboot.tar.xz

# Replace "lite" with "std" or "full" depending on which image you use.

/var/lib/clusterctrl/nfs/p$P/usr/share/clusterctrl/reconfig-usbboot $P

Booting

Ensure the Pi Zeros either have no SD card in them or the SD card doesn't contain a bootcode.bin file in the first FAT partition and then power on as normal "clusterctrl on pX".

You can watch the boot process with "tail -f /var/log/kern.log /var/log/daemon.log". daemon.log shows rpiboot transferring the files to the Pi Zero and kern.log shows the network interfaces being configured. It will then take a few more seconds for the Pi Zero to boot over the USB network before you can access them using the normal "pX.local" from the controller.

If you use this with a 3rd generation Pi (3A+, CM3 or CM3+) be aware a halted Pi is still running at some level so you'll see it trying to boot over and over in the logs for rpiboot, so I'd advise powering off after halting if using these models as nodes.

Thanks,

 Chris.

[Chris Burton]

Hi,

I didn't expect to be updating them so quickly but I've uploaded test images based on buster (2019-06-20) to https://dist.8086.net/clusterctrl/testing/

I'm seeing (what looks to be minor) issues with some of the service and interface files (I imagine there will be other things too) so I need to fix those but Pi Zeros are booting from SD and over USB and communicating OK so hopefully it shouldn't take too long to get things working on buster.

As before if you see anything please let me know.

Chris.

[Chaim Krause]

Making SDs based on these images now. Pick a few specific things you would like me to test and I will try to do so.

[Chris Burton]

 Hi,

Making SDs based on these images now. Pick a few specific things you would like me to test and I will try to do so.

Thanks for taking a look - just using it as normally would be a good start and let me know if anything looks off.

Things I've noticed so far are mostly generic Buster issues.

- ClusterHAT v1.x use RPi.GPIO which doesn't work on the Pi 4 - It looks like support will be added soon (https://sourceforge.net/p/raspberry-gpio-python/tickets/168/) :).

- Shutting down a Pi Zero (usbboot) causes an ifdown "/bin/sh: 1: test: -eq: unexpected operator" error but looks to work OK.

- wiringpi - needs to be updated.

- "screen" directory missing ( https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929418 ?).

- "w" and "uptime" don't show any users logged in ( /var/run/utmp missing ).

Chris.

[Ciprian Manea]

Hi,

I'm (beta) testing the newer buster images, and even if I reconfigure the controller with cbridge, when it boots it still has the cnat profile/type

Also, I don't seem to be able to connect/ping any of the Zero's

Any hints are welcome 🤔

Thanks!

--
You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/932015df-0ed9-4e16-88eb-ade224c1d787%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ciprian Manea]

Okay, I downloaded initially the (lite) BRIDGE version of the Controller, but after booting it the hostname was still cnat

Then I reconfigured as cbridge, rebooted, and hostname was still cnat

Now I reconfigured it as cnat, rebooted, hostname as cnat and I can ssh into the Zero's 😎

Thanks,

[Chris Burton]

Hi, 

Okay, I downloaded initially the (lite) BRIDGE version of the Controller, but after booting it the hostname was still cnat

I'm not sure what's going on there as the bridge image doesn't start out being cnat so I'd advise reimaging and/or making sure you're on the latest version "cat /etc/debian_version" should say 10.0.

Chris.

[Bruce Rindahl]

Running CBRIDGE TEST02 now.  Everything seems to be working fine on first try.

[Chris Burton]

Hi, 

I didn't expect to be updating them so quickly but I've uploaded test images based on buster (2019-06-20) to https://dist.8086.net/clusterctrl/testing/

I've uploaded a buster TEST3 (it's doing the zip on the .img files now).

This only has a couple of fixes so if you have TEST2 you should be able to merge in these changes with a text editor.

https://github.com/burtyb/clusterhat-image/blob/c85bfa273081aaf10ae6b495f942f13f14f7bc14/files/sbin/copyMAC - added the if statement to prevent errors when using a Pi without eth0.

https://github.com/burtyb/clusterhat-image/blob/c85bfa273081aaf10ae6b495f942f13f14f7bc14/files/lib/systemd/system/clusterctrl-rpiboot.service - I'd missed ".service" off the end.

I've also updated the build script and files I've been using ( https://github.com/burtyb/clusterhat-image ).

Chris.

[Bruce Rindahl]

On Sunday, June 23, 2019 at 1:51:00 PM UTC-7, Chris Burton wrote:

 

The images work with the ClusterHAT and the ClusterCTRL devices I've been working on.

They support booting Pi Zeros/A+/3A+/CM1/CM3/CM3+ from either SD (and eMMC for compute modules) or without an SD card using rpiboot, as before the controller can be setup with either a bridged (CBRIDGE) or NAT (CNAT) network.

Thanks,

 Chris.

Am I reading this right?  You have a HAT that will use one or more Compute Modules instead of Zeros?  Really interested in that .

Thanks

Bruce 

[Chris Burton]

Hi, 

Am I reading this right?  You have a HAT that will use one or more Compute Modules instead of Zeros?  Really interested in that .

 

It's not a HAT but I've been working on https://twitter.com/burtyb/status/1137081578360885248 for 3 compute modules. As they're all "nodes" it needs a Raspberry Pi (etc.) to control them using the std clusterhat/clusterctrl tool to power on/off/etc. and NAT/bridge the network traffic.

Chris.

[Jon Beltran de Heredia]

ClusterCTRL triple... any chance that will be available to the public any time soon? So appealing!

[Chris Burton]

Hi, 

ClusterCTRL triple... any chance that will be available to the public any time soon? So appealing!

I hope to have a few beta ClusterCTRL units on Tindie soon - where beta means the firmware and "clusterctrl" command will need to be updated for fixes.

All that'll be needed to update the firmware (ATmega328 or ATtiny4313) is a Raspberry Pi with 40 pin GPIO header and the supplied wires (where needed) to connect it up and run a few commands.

Chris.

[Jon Beltran de Heredia]

That is very interesting Chris, I will watch this one with interest. Would you by chance know the internal eMMC read speed? That is my main interest in using those instead of Pi Zeros with the ClusterHAT. RPi 3B+ SD card got me around 22Mb/s read speed and RPi 4 gets me double that (~44Mb/s), but eMMC has been known to deliver a lot more on other boards (just not sure if Raspberry bandwidth for eMMC will be as limited as it usually is). Thanks!

[Ciprian Manea]

Noob Q:

What’s the (best) use case for:

a) ClusterCTRL

b) ClusterHAT

Thanks!

--

You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/0f8f8f9e-bd55-410f-bad0-24c824a71275%40googlegroups.com.

[Chris Burton]

Hi, 

That is very interesting Chris, I will watch this one with interest. Would you by chance know the internal eMMC read speed? That is my main interest in using those instead of Pi Zeros with the ClusterHAT. RPi 3B+ SD card got me around 22Mb/s read speed and RPi 4 gets me double that (~44Mb/s), but eMMC has been known to deliver a lot more on other boards (just not sure if Raspberry bandwidth for eMMC will be as limited as it usually is). Thanks!

It's not that much better on the CM3+ as all of the Pi before the 4 had the same SD controller - 23.9 MB/s sequential read from the eMMC (32GB).

Connecting the CM3+ to a Pi4 controller (USB3 root filesystem) and doing reading from NFS isn't much worse though at 19.4 MB/s (9.6MB/s for a Pi Zero).

Chris.

[Jon Beltran de Heredia]

Very useful to know, thanks Chris. I was more excited about the potential bandwidth gains than about the extra computation (my aims are more data-intensive than compute-intensive), so this saves me from exploring that direction. RPi 4 with some SSD via USB3 should then probably be the best bandwidth you can get on Raspberry world.

 -- Jon

[Ciprian Manea]

?

[Chris Burton]

Hi,

On Thursday, 4 July 2019 08:20:27 UTC+1, Ciprian Manea wrote:

?

On Tue 2. Jul 2019 at 11.20, Ciprian Manea wrote:

Noob Q:

What’s the (best) use case for:

a) ClusterCTRL

b) ClusterHAT

I'm didn't really want to get into this as IMHO it's a personal thing.

One sits on a Pi, the other will hopefully be a collection of boards offering different options where some may be more/less appropriate for different people/uses.

Chris.

[Cypress Twist]

ClusterCTRL looks very interesting. Would be very interested in one with 8 or more slots.

[Matha Goram]

Hello Chris,

I know you will but nevertheless have to ask (since I see almost daily image updates - thank you!). Will you post a message indicating a "stable" release? Given all the juggling acts I've had to with Buster on the other machines (mainly my ignorance about iptables-persistent and netfilter-persistent), right now my ClusterHAT is chugging along fine but, of course, kicking tires is what I like to do with the RPi farm.

Kind regards.

[Chris Burton]

Hi, 

I know you will but nevertheless have to ask (since I see almost daily image updates - thank you!). Will you post a message indicating a "stable" release? Given all the juggling acts I've had to with Buster on the other machines (mainly my ignorance about iptables-persistent and netfilter-persistent), right now my ClusterHAT is chugging along fine but, of course, kicking tires is what I like to do with the RPi farm.

Yes, there will be an announcement here once it's ready for general use.

Chris. 

[enno]

Hi Chris,

first let me say thank you for all the input you provided to these awesome projekt and also thanks to all the folks who share their experience and knowledge with the community!

I just use TEST7 lite bridged on RP4 with Cluster HAT 2.3 and four 1.3 zeros on it. By using five SD cards everything went well right from the start and I can conect to each node.

By testing usbboot, I just followed the instruction/Overview above:

How to boot Pi Zeros without SD cards.

Overview

The root filesystems are stored in /var/lib/clusterctrl/nfs/

In ....

These does not work in my case. By the way, I'm a newbie in cluster computing and I'm eager to learn.

Additional I try:

$ git clone --depth=1 https://github.com/burtyb/usbboot
$ cd usbboot
$ sudo apt-get install libusb-1.0-0-dev
$ make
$ sudo ./rpibootCode

indeed all power LED of the zeros light up, but doesn't flickering and nor node is reachable. It seems no one of the zeros realy boot the system. cbridge respont:

pi@cbridge:/var/lib/clusterctrl/usbboot $ sudo make
cc -Wall -Wextra -g -o bin2c bin2c.c
./bin2c msd/bootcode.bin msd/bootcode.h
./bin2c msd/start.elf msd/start.h
cc -Wall -Wextra -g -o rpiboot main.c -lusb-1.0
main.c: In function ‘main’:
main.c:601:2: warning: ‘libusb_set_debug’ is deprecated: Use libusb_set_option instead [-Wdeprecated-declarations]
  libusb_set_debug(ctx, verbose ? LIBUSB_LOG_LEVEL_WARNING : 0);
  ^~~~~~~~~~~~~~~~
In file included from main.c:1:
/usr/include/libusb-1.0/libusb.h:1300:18: note: declared here
 void LIBUSB_CALL libusb_set_debug(libusb_context *ctx, int level);
                  ^~~~~~~~~~~~~~~~
pi@cbridge:/var/lib/clusterctrl/usbboot $ sudo ./rpiboot
Waiting for BCM2835/6/7
Failed to claim interface
Failed to claim interface
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7
Second stage boot server
File read: start.elf
Second stage boot server done
pi@cbridge:/var/lib/clusterctrl/usbboot $

I can't figure out which steps of your "Guide to using the RPIBOOT [test]..." still necessary. Should I perform all these steps manually?

Thanks,
enno

[Chris Burton]

Hi, 

These does not work in my case. By the way, I'm a newbie in cluster computing and I'm eager to learn.

Most of the time it no longer works for me either, occasionally it will boot but then won't reboot - rpiboot sends the files to the Pi Zero but it's as through it never starts the kernel on the Pi Zero as I see nothing on the serial console.

 

Additional I try:

$ git clone --depth=1 https://github.com/burtyb/usbboot
$ cd usbboot
$ sudo apt-get install libusb-1.0-0-dev
$ make
$ sudo ./rpibootCode

To use another rpiboot you'd first need to stop the existing service "sudo service clusterctrl-rpiboot stop". My usbboot repo is really old now so if you're going to try with a different one I'd go with the official one ( https://github.com/raspberrypi/usbboot ). Also you'd need to point it to the correct directly and enable overlays by running it with something like "sudo ./rpiboot -m 2000 -d /var/lib/clusterctrl/boot/ -o -l -v" 

  

I can't figure out which steps of your "Guide to using the RPIBOOT [test]..." still necessary. Should I perform all these steps manually?

No, the idea of the new images is none of that is needed. You'd just need to follow the steps in the announcement message to download/expand one of the "ClusterCTRL*usbboot*xz" images for the file system and configure it for each pX and then power it on.

I'll be able to spend more time looking at it next week so hopefully things will be working again soon :).

Chris.