I have a cluster hat... what to do with it?

[Gordon]

Hey there.

I got a cluster hat and 4 zero 2. Working well. I installed docker swarm and was playing around a bit. Cool so far... but... what to do with it?

2 things would be interesting for me, but I have no idea if it is possible and how.

1) I would like to speed up my database (MariaDB). I collect "big data" and the database has many write - but also read - actions. And also a lot of records already. I can "feel" that the engine is getting slower.
The idea in my head... to use the zeros to speed it up... somehow. Like 2 are used for data collecting and writing into the database and the other 2 are used for reading and process the data. I guess the bottleneck here is would be the database itself, which is on the host pi (pi4) on an SSD. But maybe it is possible to put it partly into the RAM?

2) I am working as well with Python. It is also used to collect the data and to process the data. Partly it uses a lot of CPU power (deep learning). So it would be helpfull if the work load could be split to the whole cluster.

Sorry for my english, I hope you understand what I want to say :-)
Is there any way to achieve it? The best would be without the need to study rocket science. :-)

[Ian Goldsmith]

I think you’re asking a rocket science question, there are inly rocket science answers.

Sql Database operations need to happen in order, the previous operation needs to have updated the state before a new one can be done… you can sync to another though for the read operations.. to mostly happen from a mirror.

keeping lots of it in ram can improve speed but hard to do that distributed  - perhaps with sharding.  this isn’t the right hardware though if you really want speed.  One more powerful x86 cpu with a heap of ram will wipe the floor on speed & even power efficiency.  These systems are great for just small scale testing of stuff

--
You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/a6611a49-0b17-4d81-8b34-612af9965c63n%40googlegroups.com.

[hanur13]

Would you mind sharing your sources/github/etc to install docker swarm?
I've give up to install k3s in my clusterhat.

[Ted Serreyn]

I was hoping to do clustering/docker swarm only to find out that this is not really possible any more with the raspberry pi zero W.  GRRR.

can’t run with qty 4 pi zero 2 apparently either. 

Ted Serreyn

Serreyn Network Services, LLC

On Sep 17, 2022, at 8:08 AM, hanur13 <nurul...@gmail.com> wrote:

Would you mind sharing your sources/github/etc to install docker swarm?

--

You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/b58b1333-d499-443a-a948-5be560fdbbc6n%40googlegroups.com.

[Ben Heininger]

I too would like to know how you got Docker Swarm running on the clusterhat.
All of the current tutorials are out of date and no longer work properly.
It would be nice if people could update their tutorials so that they work with new images/setups/software versions

I have a Pi4 4gb with 4x pi zero 1.3 and have had many issues getting things to work.

The only thing i have managed to do was write a bash script to call mpiexec to run a python script on all the Pis to fetch the names and temperatures of each pi and display that in terminal.

(I did this months ago and now it doesnt work since my last apt update/upgrade)

I tried running docker sandboxed web browser and KASM Workspaces but that also failed miserably.

After nearly 12 months of trying to use this cluster for something all I have learnt is that most things dont work for the beginner/learner.

Everything else to me seems like Rocket Science (╯°□°）╯︵ ┻━┻

In saying that Chris has been very helpful when I have asked for help.

The problem seems to be that everyone who writes a tutorial is a Rocket Scientist and they write their Tutorial as everyone else understands what is going on and they put not help in for "If this doesn't work do this...".

[tsw...@gmail.com]

I have 2 clusterhats and have found them to be a nice compact cluster for experimenting and investigating parallel programming and docker swarm. I have configured these so that I can have both of them on my LAN (all the Pis have static addresses on the LAN).

I use one clusterhat for MPI programming in C and Python
- the controller is a 3 Model B
- the RPi Zeros are a combination of plain and W models (WiFi not enabled)
- they are all running Raspbian 10 (buster)
  - STD on the controller and Lite on the Zeros
- the controller provides NFS services
- the Zeros mount a user directory from the controller using NFS
  - this allows any work I do in the user directory on the controller to be seen by the Zeros
- I have installed both MPICH and MPI4PY, individually on all systems
  - I installed these from source
- I ssh into the shared user directory on the controller to do my programming
- parallel programs using all 5 systems are just a bit slower than using just the Zeros

  - my favorite parallel program is a Sieve of Eratosthenes where I can ask for the count of primes to 1,000,000,000

The other clusterhat I use for Docker Swarm
- the controller is a 3 Model B
- the RPi Zeros are a combination of plain and W models (Wifi not enabled)
- they are all running Raspbian 10 (buster)
  - STD on the controller and Lite on the Zeros
- I have installed docker and docker-compose and configured a Swarm
  - the Swarm is 1 manager and 4 workers
- I have run both the voting app and the dockercoin app on this cluster

Note that the controller is an arm32v7 and the Zeros are arm32v6. This is important.

I built all the app images on the controller AND on one of the Zeros. I had to figure out the use of multiple architectures in the Docker images -- this involves creating a manifest for the multi-architecture images.

Something like this with a suitable docker repo (locally or docker hub):

docker push tswoyer/dc-worker:arm32v7

docker push tswoyer/dc-worker:arm32v6
docker manifest create tswoyer/dc-worker:latest tswoyer/dc-worker:arm32v6 tswoyer/dc-worker:arm32v7
docker manifest push --purge tswoyer/dc-worker:latest

Then I can use

image: tswoyer/dc-worker:latest

in my compose file and get the right executable for the right architecture.

My biggest frustration is trying to reproduce work from the Internet related to the differences between x86 examples and ARM examples -- sometimes there is a lot of work to translate an x86 implementation to one that works on ARM. Sometimes I can't find an ARM image that works for me and I can't figure out how to build one.

Now, outside of my learning and experimenting, I haven't found a real-world use for the clusters.

[Tom Szolyga]

I have 2 clusterhats also.  I am interested in using them to play games like chess that require searching a large tree for the best move.  Has anyone seen such a use case?

[Peter Cross]

Someone wanted to get Docker running. I made this tutorial a few years ago and was able to get it running then. Please let me know if it works for you and if it doesn't please tell me at what step you broke down. 

Cheers!

Peter J. Cross
San Antonio, TX

"Experience has taught mankind the necessity of auxiliary precautions"
-James Madison, Federalist Paper No. 51

Please consider the environment before printing this email

To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/2f00d1cf-a8af-4c80-85e2-73abab0cd099n%40googlegroups.com.

[Ben Heininger]

@pjc
I followed you tutorial doc and the crontab (13 > b > ii ) ;
a. @reboot sudo iptables -A FORWARD -i br0 -o br0 -j ACCEPT
b. @reboot /sbin/clusterhat on

my clusterhat turned on but I still couldnt ping the Zeros.

[Douglas Goodall]

If you really want to get the power of the cluster focused, don't use Python. Write in C++ instead. Python is interpreted and will always be orders of magnitude

slower than compiled code.