Docker swarm on ClusterHat using rpiboot

614 views
Skip to first unread message

Gábor Babusa

unread,
Feb 27, 2018, 8:54:12 AM2/27/18
to ClusterHAT
Hi all,

I've purchased a Cluster Hat, and before I start playing with it, I'm  curious about can I use Docker swarm on it if I use rpiboot mode (Pi zero's without SD card), and if yes how can I start?

Will Sargent

unread,
Feb 27, 2018, 12:22:27 PM2/27/18
to Gábor Babusa, ClusterHAT

On Tue, Feb 27, 2018 at 5:54 AM, Gábor Babusa <gba...@gmail.com> wrote:
Hi all,

I've purchased a Cluster Hat, and before I start playing with it, I'm  curious about can I use Docker swarm on it if I use rpiboot mode (Pi zero's without SD card), and if yes how can I start?

--
You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/0ebd1185-d404-482e-9b0e-8633adaa385c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Will Sargent

unread,
Feb 27, 2018, 12:25:34 PM2/27/18
to Gábor Babusa, ClusterHAT
Ah, you want rpiboot mode specifically.  The above blog post doesn't cover that, but it looks like it should work fine as everything works the same post boot:



Gábor Babusa

unread,
Mar 1, 2018, 1:03:13 PM3/1/18
to ClusterHAT
root@clustercontroller:/home/pi# ls -l /var/lib/clusterhat/boot/1-*
lrwxrwxrwx 1 root root 15 Nov 27 17:02 /var/lib/clusterhat/boot/1-1.2.1 -> ../nfs/p4/boot/
lrwxrwxrwx 1 root root 15 Nov 27 17:02 /var/lib/clusterhat/boot/1-1.2.2 -> ../nfs/p3/boot/
lrwxrwxrwx 1 root root 15 Nov 27 17:02 /var/lib/clusterhat/boot/1-1.2.3 -> ../nfs/p2/boot/
lrwxrwxrwx 1 root root 15 Nov 27 17:02 /var/lib/clusterhat/boot/1-1.2.4 -> ../nfs/p1/boot/

root@clustercontroller:/home/pi# lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=smsc95xx, 480M
        |__ Port 4: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M

The second part looks a little bit different than in the description (although 3 of 4 zero are plugged)


2018. február 27., kedd 18:25:34 UTC+1 időpontban Will Sargent a következőt írta:
Ah, you want rpiboot mode specifically.  The above blog post doesn't cover that, but it looks like it should work fine as everything works the same post boot:



On Tue, Feb 27, 2018 at 9:21 AM, Will Sargent <will.s...@gmail.com> wrote:
On Tue, Feb 27, 2018 at 5:54 AM, Gábor Babusa <gba...@gmail.com> wrote:
Hi all,

I've purchased a Cluster Hat, and before I start playing with it, I'm  curious about can I use Docker swarm on it if I use rpiboot mode (Pi zero's without SD card), and if yes how can I start?

--
You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.

Gábor Babusa

unread,
Mar 1, 2018, 1:50:45 PM3/1/18
to ClusterHAT

Hardware: Rpi2 with 16GB SD, SSH enabled, WLAN, USB mouse  + keyboard attached
               1 Zero no camport,
               1 Zero with camport
               1 Zero with wifi and camport

Rpi2 has both IP address 172.x.x.x / 192.x.x.x

when I log in and give 'clusterhat on' command, all the 4 leds turn on each by each.



Gábor Babusa

unread,
Mar 2, 2018, 4:44:00 AM3/2/18
to ClusterHAT
Browsing the topics I've found this:

"It was the USB cable... the short one that shipped with the HAT didn't work..."

Ouch, I had to check it, when I will go home from work..


Gábor Babusa

unread,
Mar 2, 2018, 1:41:58 PM3/2/18
to ClusterHAT
Although I'm rather connected to Rpi2 with HDMI then serial, I can't see 'Turning on p1-4' messages, only orange leds turn on, and when I give 'screen -rx rpiboot' command, only 'waiting for BMC2835/6/7' meggage appeal.

Gábor Babusa

unread,
Mar 2, 2018, 3:10:32 PM3/2/18
to ClusterHAT
Hmm, is it looks good? :)

root@controller:~# lsusb -t

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=smsc95xx, 480M
        |__ Port 2: Dev 9, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 3: Dev 24, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 3: Dev 24, If 0, Class=Communications, Driver=cdc_ether, 480M
            |__ Port 3: Dev 24, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 3: Dev 24, If 1, Class=CDC Data, Driver=cdc_ether, 480M
            |__ Port 1: Dev 22, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 1: Dev 22, If 1, Class=CDC Data, Driver=cdc_ether, 480M
            |__ Port 1: Dev 22, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 22, If 0, Class=Communications, Driver=cdc_ether, 480M
            |__ Port 2: Dev 20, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 2: Dev 20, If 0, Class=Communications, Driver=cdc_ether, 480M
            |__ Port 2: Dev 20, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 20, If 1, Class=CDC Data, Driver=cdc_ether, 480M

        |__ Port 4: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
root@controller:~#

(3 of 4 Zero attached)

"clusterhat off;clusterhat hub off;sleep 2;clusterhat hub on, clusterhat on" helped
then screen -rx rpiboot worked :)

Chris Burton

unread,
Mar 3, 2018, 5:19:57 AM3/3/18
to ClusterHAT
Hi,


On Friday, 2 March 2018 20:10:32 UTC, Gábor Babusa wrote:
Hmm, is it looks good? :)

It looks as it should if you have 3 Pi Zeros (P2/P3/P4).

Chris.

Gábor Babusa

unread,
Mar 6, 2018, 1:48:01 PM3/6/18
to ClusterHAT
Hi,

Correct.

But is it ok, that next reboot I also have to give 'clusterhat hub off/on' commands?

Can I connect to the Zeros by SSH?

Chris Burton

unread,
Mar 6, 2018, 3:25:57 PM3/6/18
to ClusterHAT
Hi, 
But is it ok, that next reboot I also have to give 'clusterhat hub off/on' commands?

If you have a Cluster HAT 1.x then yes you would need to turn shutdown the Pi Zeros before rebooting the controller and then turn the ports on again but if you have a v2.x then the Pi Zeros should stay powered up and recover from the NFS timeouts.
 
Can I connect to the Zeros by SSH?

Yes, you should be able to either SSH to the "internal" IP address of the Pi Zero
ssh p...@172.19.180.1 # for P1
ssh p...@172.19.180.2 # for P2
etc.

You should also be able to SSH by name
ssh p...@p1.local
ssh p...@p2.local
etc.

But please be aware if you've installed Docker then it has possibly broke the network config the Cluster HAT expects - see https://groups.google.com/d/msg/clusterhat/_vLhJfPO_eo/yJVdfOZ1DwAJ

Chris.

Gábor Babusa

unread,
Mar 6, 2018, 6:34:32 PM3/6/18
to ClusterHAT

Hi

root@controller:/home/pi# lsusb -t

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=smsc95xx, 480M
        |__ Port 2: Dev 14, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 3: Dev 32, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 3: Dev 32, If 0, Class=Communications, Driver=cdc_ether, 480M
            |__ Port 3: Dev 32, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 3: Dev 32, If 1, Class=CDC Data, Driver=cdc_ether, 480M
            |__ Port 1: Dev 30, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 1: Dev 30, If 1, Class=CDC Data, Driver=cdc_ether, 480M
            |__ Port 1: Dev 30, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 30, If 0, Class=Communications, Driver=cdc_ether, 480M
            |__ Port 4: Dev 34, If 1, Class=CDC Data, Driver=cdc_ether, 480M
            |__ Port 4: Dev 34, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 4: Dev 34, If 0, Class=Communications, Driver=cdc_ether, 480M
            |__ Port 4: Dev 34, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 28, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 2: Dev 28, If 0, Class=Communications, Driver=cdc_ether, 480M
            |__ Port 2: Dev 28, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 28, If 1, Class=CDC Data, Driver=cdc_ether, 480M
        |__ Port 4: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 4: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M

But I have to reset clusterhat hub after every reboot, otherwise rpiboot hangs with 'waiting for BMC2835/6/7'

other problem is that now I can ping only p1, and therefore I can SSH to it only

Chris Burton

unread,
Mar 7, 2018, 2:45:44 PM3/7/18
to ClusterHAT
Hi, 
But I have to reset clusterhat hub after every reboot, otherwise rpiboot hangs with 'waiting for BMC2835/6/7'

If the Pi Zeros have already booted then that's to be expected. 
 
other problem is that now I can ping only p1, and therefore I can SSH to it only

If you're still having problems then please provide more information.

Which version of the Cluster HAT software are you using?

If you run "clusterhat on" and watch the screen session for the Pi Zeros to download the files and be accessible via SSH.

Then run "brctl show;echo;gpio readall ; cat /proc/device-tree/hat/* ; echo ; dmesg|grep "cdc\|usb" ; lsusb -t; sudo ifconfig -a;sudo iptables -S".

And after rebooting run the above commands again, which should give a clue as to what's going on.

Either reply with the output here or send to sup...@8086.net.uk and I'll take a look.

Chris.

Gábor Babusa

unread,
Mar 7, 2018, 5:12:41 PM3/7/18
to ClusterHAT
Hello,

Thx for your support.

I thought that I use the latest, but today I realise, it was 'ClusterHAT-2017-09-07-lite-1-usbboot', so I reimaged my SD card with 'ClusterHAT-2017-11-29-lite-2-usbboot'

Now, I don't need to install anything, and a 'clusterhat on', 'screen -rx rpiboot' working. But I have some hardware issue (connection error, wrong usb cable and/or one/some bad Pi Zeros). It needs further investigations. One thing is certain, the position p1 not reachable, so I remove the Pi Zero from it.
I could ssh to the other 3 Pi (with password) and when I wanted to do the passwordless setup, RPi2 suddenly stopped accepting commands from SSH, but luckily I could shutted it down with the attached keyboard.

I'm going to get a new USB data cable, and a wifiless Pi Zero is on its way for the P1 position.
So something is not too stable now, but I hope I can fix it within a few days.

Gábor Babusa

unread,
Mar 8, 2018, 2:59:39 PM3/8/18
to ClusterHAT
root@controller:/home/pi# for I in 1 2 3 4; do echo -n "p$I:";ssh pi@p$I.local uptime;done
p1: 19:57:54 up 3 min,  0 users,  load average: 0.61, 0.61, 0.29
p2: 19:57:57 up 4 min,  0 users,  load average: 0.46, 0.52, 0.26
p3: 19:57:59 up 6 min,  0 users,  load average: 0.42, 0.51, 0.29
p4: 19:58:01 up 5 min,  0 users,  load average: 0.31, 0.46, 0.26

:)

Gábor Babusa

unread,
May 25, 2020, 7:40:30 PM5/25/20
to ClusterHAT
update: after an hour something strange happened:I can login to all te PiZeros:

pi@cnat:~ $ for I in 1 2 3 4; do echo -n "p$I:";ssh p$I uptime;done
p1: 01:37:38 up  1:30,  0 users,  load average: 0,23, 0,31, 0,54
p2: 01:37:41 up  1:28,  0 users,  load average: 0,42, 0,31, 0,23
p3: 01:37:44 up  1:30,  0 users,  load average: 0,70, 0,38, 0,56
p4: 01:37:47 up  1:29,  0 users,  load average: 0,35, 0,35, 0,59

Befote it only p2, p4 worked


???????????????? :D


After two years, I've restarted mx clusterpi project.
I've rebuilt it with a Rpi 2 v1.1, an official 7" touchscreen (with an Y powercord), and 4 Pi Zero.
I've used the ClusterCTRL-2020-02-13-full-1-CNAT image for the RPi 2, and the ClusterCTRL-2020-02-13-lite-1-usbboot for the Zeros

I generated an ssh-key for passwordless login.
I can login into the Zeros....

BUT:
After a short time (10-15mins) I can only login 2 of 4 Zeros.

But if I stop/start controller and wait 2-3 mins to reload, I can reach all Zeros, for a short period of time again

I have a clusterhat v2.2

pi@cnat:~ $ lsusb -t

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=smsc95xx, 480M
        |__ Port 3: Dev 42, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 56, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 56, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 1: Dev 56, If 0, Class=Communications, Driver=rndis_host, 480M
            |__ Port 1: Dev 56, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 1: Dev 56, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 1: Dev 56, If 1, Class=CDC Data, Driver=rndis_host, 480M
            |__ Port 4: Dev 50, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 4: Dev 50, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 4: Dev 50, If 1, Class=CDC Data, Driver=rndis_host, 480M
            |__ Port 4: Dev 50, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 4: Dev 50, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 4: Dev 50, If 0, Class=Communications, Driver=rndis_host, 480M
            |__ Port 2: Dev 52, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 2: Dev 52, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 2: Dev 52, If 0, Class=Communications, Driver=rndis_host, 480M
            |__ Port 2: Dev 52, If 5, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 52, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 2: Dev 52, If 1, Class=CDC Data, Driver=rndis_host, 480M
            |__ Port 3: Dev 58, If 3, Class=CDC Data, Driver=cdc_acm, 480M
            |__ Port 3: Dev 58, If 1, Class=CDC Data, Driver=rndis_host, 480M
            |__ Port 3: Dev 58, If 4, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 3: Dev 58, If 2, Class=Communications, Driver=cdc_acm, 480M
            |__ Port 3: Dev 58, If 0, Class=Communications, Driver=rndis_host, 480M
            |__ Port 3: Dev 58, If 5, Class=CDC Data, Driver=cdc_acm, 480M
        |__ Port 5: Dev 6, If 0, Class=Vendor Specific Class, Driver=rtl8192cu, 480M

Gábor Babusa

unread,
May 26, 2020, 1:15:15 AM5/26/20
to ClusterHAT

After a night, still working: :)

pi@cnat:~ $ for I in 1 2 3 4; do echo -n "p$I:";ssh p$I uptime;done
p1: 07:14:39 up  7:07,  0 users,  load average: 0,06, 0,03, 0,00
p2: 07:14:42 up  7:05,  0 users,  load average: 0,05, 0,04, 0,00
p3: 07:14:45 up  7:07,  0 users,  load average: 0,08, 0,03, 0,01
p4: 07:14:48 up  7:06,  0 users,  load average: 0,21, 0,12, 0,10

Gábor Babusa

unread,
May 27, 2020, 6:02:48 AM5/27/20
to ClusterHAT

docker.JPG

Status report :)

Gábor Babusa

unread,
Jun 2, 2020, 3:13:14 AM6/2/20
to ClusterHAT
Status report2 :)

docker2.JPG


Gábor Babusa

unread,
Jun 2, 2020, 3:14:27 AM6/2/20
to ClusterHAT

docker service create --name ping hypriot/rpi-alpine-scratch ping 8.8.8.8

Gábor Babusa

unread,
Jun 2, 2020, 4:18:27 AM6/2/20
to ClusterHAT
Issues:

There is a 'blessing moment' when all the nodes are working. (Ok, maybe a few minutes) Then it starts collapse. The Zero's starts the process, them the 'outsider' nodes goes 'down'. ' Heartbeat failure' they say. The Zeros come back usually, after a few ours, but the 'outsider' nodes won't, until I leave/re-join then to the swarm.



Gábor Babusa

unread,
Jun 2, 2020, 4:48:50 AM6/2/20
to ClusterHAT

docker4.JPG


 

docker5.JPG


Is there any idea why it it happen? I use an 2,5A official RPi power supply for powering a RPi 2b,  4 Pi Zero, an official 7" Touchscreen and a wifi dongle. It could be power issue?


Other thing: as it seen on the picture, the only working node which not run container is the x68 based. Is it an error or a feature? (all the other nodes are arm based. The x86 based rus with Raspbian PC OS.



Chris Burton

unread,
Jun 2, 2020, 8:13:36 AM6/2/20
to ClusterHAT
Hi, 

Is there any idea why it it happen? I use an 2,5A official RPi power supply for powering a RPi 2b,  4 Pi Zero, an official 7" Touchscreen and a wifi dongle. It could be power issue?


I've used a Pi+official display+Cluster HAT in the past and it was very tight on power so it could cause issues. On the controller you should be able to check for current/previous power issues with "vcgencmd get_throttled" or "clusterctrl status" if it says anything other than "0x0" it has throttled due to either temp or power.

Chris.

Gábor Babusa

unread,
Jun 2, 2020, 8:31:28 AM6/2/20
to ClusterHAT
Hi,

Thanks for the ansver, I use a micro USB Y power cable, one goes to the display, the other to the RPi 2. Before I use this, I saw a yellow ϟ icon on the righ-top corner of the desktop.

 vcgencmd get_throttled
throttled=0x50000


clusterctrl status
clusterhat:1
clusterctrl:False
maxpi:4
hat_version:2.2
hat_version_major:2
hat_version_minor:2
hat_size:4
hat_uuid:8274a91e-5ac2-4292-8878-db8692a5f28c
hat_vendor:8086 Consultancy
hat_product_id:0x0004
throttled:0x50000
hat_alert:0
hat_hub:1
hat_wp:1
hat_led:1
hat_wplink:-1
hat_xra1200p:False
p1:1
p2:1
p3:1
p4:1

Chris Burton

unread,
Jun 2, 2020, 1:46:40 PM6/2/20
to ClusterHAT
Hi, 
Thanks for the ansver, I use a micro USB Y power cable, one goes to the display, the other to the RPi 2. Before I use this, I saw a yellow ϟ icon on the righ-top corner of the desktop.

 vcgencmd get_throttled
throttled=0x50000

The icon means you had power issues which agrees with the decoded get_throttled output so it might be worth trying to power display and Pi+Cluster HAT through separate power supplies if you can to see if it helps.

Chris.

Gábor Babusa

unread,
Jun 2, 2020, 4:15:38 PM6/2/20
to ClusterHAT
Hi,

I've supplied the display with a different source.

After the USB came up, starting the docker and re-join the other nodes , it looks like:

docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
w7zbjh3wyh14c28kk1zianyc6     adblock             Ready               Active                                  19.03.9
qfmd5edjx3mfcsj48qy9ss1zu *   cnat                Ready               Active              Leader              19.03.9
7l66xa3mn80heh2zc4y87dyp7     cubietruck          Ready               Active                                  19.03.9
xd08l1tnpzkjt8jgerbr5fsl8     media               Ready               Active                                  19.03.9
ok42sinrvn9vufnfqb3dhoeeg     p1                  Ready               Active                                  19.03.9
qgwwnjpfd6ircflwfhycuj5gf     p2                  Down                Active                                  19.03.9
pnnj92xk7klwujv5sda9ww4p3     p3                  Ready               Active                                  19.03.9
fnxd1zj823zvos4i5ldrihuw8     p4                  Ready               Active                                  19.03.9

one-or two minutes after, than I've started docker, p2 went down and constantly is in this state since half an hour. And can't be reachable with ssh.
But just this, and not altering each other, progress?

vcgencmd get_throttled
throttled=0x0

docker6.JPG

(media is x86 based, and not run container, maybe I will remove it later, or test with an x86 based one)

Gábor Babusa

unread,
Jun 2, 2020, 5:21:40 PM6/2/20
to ClusterHAT
:( 2 of 4 Zero works only

Chris Burton

unread,
Jun 3, 2020, 3:30:08 AM6/3/20
to ClusterHAT
Hi, 
one-or two minutes after, than I've started docker, p2 went down and constantly is in this state since half an hour. And can't be reachable with ssh.
But just this, and not altering each other, progress?

If they're going awol after a few minutes (please remember I know nothing of docker) it might be worth trying my suggest in this post https://groups.google.com/d/msg/clusterhat/Uhha-CdaP4E/jWLclXc3AQAJ

Chris. 
Reply all
Reply to author
Forward
0 new messages