ketchup (k3sup) kubernetes easy in

1,163 views
Skip to first unread message

Ciprian Manea

unread,
Mar 24, 2020, 5:54:39 PM3/24/20
to clust...@googlegroups.com
Hi there,

Just a heads-up for k3s lovers, there’s a new post (https://medium.com/@alexellisuk/walk-through-install-kubernetes-to-your-raspberry-pi-in-15-minutes-84a8492dc95a) on how to install quickly Kubernetes even on ClusterHATs

I just tested it on a ClusterHAT v2.2 just updated to Buster Lite, CNAT with RPi 3B+ and everything is nice and up (base Kubernetes, no extra apps yet). The “whole” setup is listed below:

 curl -sLS https://get.k3sup.dev | sudo sh

 k3sup install --ip 172.19.181.254 --user pi

 k3sup join --ip 172.19.181.1 --server-ip 172.19.181.254 --user pi

 k3sup join --ip 172.19.181.2 --server-ip 172.19.181.254 --user pi

 k3sup join --ip 172.19.181.3 --server-ip 172.19.181.254 --user pi

 k3sup join --ip 172.19.181.4 --server-ip 172.19.181.254 --user pi


 sudo systemctl status k3s

 curl -sLS https://dl.get-arkade.dev | sudo sh

 ark

 export KUBECONFIG=`pwd`/kubeconfig

 kubectl get node -o wide

 kubectl top pod --all-namespaces  



Best,
Ciprian

Eythymios Iosifidis

unread,
Mar 25, 2020, 5:52:54 AM3/25/20
to ClusterHAT
Hello Ciprian!

Thank you very much for this info. I was personally looking for something like this.
One question, can this also be applied in a CBRIDGE configuration? I presume yes, right?

Best Regards,
Efthimis

Ciprian Manea

unread,
Mar 25, 2020, 6:08:12 AM3/25/20
to clust...@googlegroups.com
it should work in CBRIDGE as well, but you should use different IPs (than cnap/mine) and make sure your DHCPd gives static IPs to the ClusterHAT and Zero's

--
You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/97526a61-90c2-4aae-9483-015637116c25%40googlegroups.com.

Ciprian Manea

unread,
Mar 25, 2020, 6:14:37 AM3/25/20
to clust...@googlegroups.com
Oh, and the most important thing — which I forgot 😊, all your ClusterHAT Pi’s must have these cg* added to cmdline.txt then reboot them to activate it

PS. I prefer ipv6.disable=1 but you won’t have to set it 😎

$ cat /boot/cmdline.txt 

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=dafeb7ce-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait ipv6.disable=1 quiet cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Message has been deleted

Dario Molinari

unread,
Mar 25, 2020, 4:08:12 PM3/25/20
to ClusterHAT
Well, this method works a treat also on a Pi3B with ClusterHat 1.2, 1x Zero v1.3 and 3x v1.2 on the buster lite cnat image. Thanks a million Ciprian and Rancher! :-D

pi@cnat:~ $ kubectl get node -o wide          
NAME   STATUS   ROLES    AGE     VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
p4     Ready    <none>   20m     v1.17.2+k3s1   172.19.181.4    <none>        Raspbian GNU/Linux 10 (buster)   4.19.97+         containerd://1.3.3-k3s1
p2     Ready    <none>   5m46s   v1.17.2+k3s1   172.19.181.2    <none>        Raspbian GNU/Linux 10 (buster)   4.19.97+         containerd://1.3.3-k3s1
p1     Ready    <none>   12m     v1.17.2+k3s1   172.19.181.1    <none>        Raspbian GNU/Linux 10 (buster)   4.19.97+         containerd://1.3.3-k3s1
cnat   Ready    master   31m     v1.17.2+k3s1   192.168.5.234   <none>        Raspbian GNU/Linux 10 (buster)   4.19.97-v7+      containerd://1.3.3-k3s1
p3     Ready    <none>   114s    v1.17.2+k3s1   172.19.181.3    <none>        Raspbian GNU/Linux 10 (buster)   4.19.97+         containerd://1.3.3-k3s1
pi@cnat:~ $ kubectl top pod --all-namespaces
NAMESPACE     NAME                                      CPU(cores)   MEMORY(bytes)    
kube-system   coredns-d798c9dd-pbb7j                    9m           8Mi              
kube-system   local-path-provisioner-58fb86bdfd-xm6xg   8m           5Mi              
kube-system   metrics-server-6d684c7b5-7ttp2            17m          9Mi              
kube-system   svclb-traefik-whq8f                       0m           1Mi              
kube-system   traefik-6787cddb4b-vgc9k                  12m          10Mi

Dario Molinari

unread,
Mar 25, 2020, 7:33:43 PM3/25/20
to ClusterHAT
Admittedly the cluster is creaking even just running dashboard... see screenshot
Screenshot_20200325_233142.png

Ciprian Manea

unread,
Mar 25, 2020, 11:58:47 PM3/25/20
to clust...@googlegroups.com
But look 👀 at those beautiful colors 

--
You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.

Eythymios Iosifidis

unread,
Mar 26, 2020, 4:30:27 AM3/26/20
to ClusterHAT
Hello!

wow!

How can i have this dashboard? I have only reached up to the point of having the below status in my cluster:

i@cbridge:~ $ sudo  kubectl get node
NAME      STATUS   ROLES    AGE   VERSION
p1        Ready    <none>   13h   v1.17.2+k3s1
p4        Ready    <none>   13h   v1.17.2+k3s1
p3        Ready    <none>   13h   v1.17.2+k3s1
p2        Ready    <none>   13h   v1.17.2+k3s1
cbridge   Ready    master   16h   v1.17.2+k3s1
pi@cbridge:~ $
Thanks!


Τη Πέμπτη, 26 Μαρτίου 2020 - 5:58:47 π.μ. UTC+2, ο χρήστης Ciprian Manea έγραψε:
But look 👀 at those beautiful colors 
To unsubscribe from this group and stop receiving emails from it, send an email to clust...@googlegroups.com.

Dario Molinari

unread,
Mar 26, 2020, 4:39:41 AM3/26/20
to ClusterHAT
Great stuff Eythymios,

Pick up from Ciprian's instructions ("kubectl top pod --all-namespaces") half way down the page here: https://medium.com/@alexellisuk/walk-through-install-kubernetes-to-your-raspberry-pi-in-15-minutes-84a8492dc95a
I was having some problems accessing the dashboard until I installed "kubectl" on my laptop and ran "kubectl proxy" from it

Good luck! ;-)
Dario

P.S. bear in mind that the default container tech for k3s is containerD, not docker

Dario Molinari

unread,
Mar 26, 2020, 4:40:27 AM3/26/20
to ClusterHAT
True! I could watch them for hours! :-D

Thanks for the suggestions btw Ciprian!
Dario


On Thursday, 26 March 2020 03:58:47 UTC, Ciprian Manea wrote:
But look 👀 at those beautiful colors 
To unsubscribe from this group and stop receiving emails from it, send an email to clust...@googlegroups.com.

Ciprian Manea

unread,
Mar 26, 2020, 4:47:44 AM3/26/20
to clust...@googlegroups.com
Hi,

The problem with the dashboard is that it seems to eat the whole ClusterHAT up as RPi Zero’s have quite slow CPUs

metrics-server (running by default in k3s) seems to be a (close guarded) secret, so if you’re happy with text only, just use

 kubectl get node -o wide

 kubectl top pod --all-namespaces  


And maybe there’s some juice left in your ClusterHAT to try/run something else there 😊

P.S. all merits go to https://github.com/alexellis for being the one coding / building them


Best,
Ciprian

To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/8958b5a4-28ed-4b02-a4f9-34ecfabef26f%40googlegroups.com.

Ciprian Manea

unread,
Mar 26, 2020, 6:04:20 AM3/26/20
to clust...@googlegroups.com
Oh, well. my Zero’s are still warming up air with no dashboard installed at all 🤔

$ kubectl top node

NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   

cnat   818m         20%    612Mi           62%       

p1     1001m        100%   240Mi           50%       

p2     1223m        122%   217Mi           45%       

p3     1028m        102%   217Mi           45%       

p4     730m         73%    250Mi           52%       


Eythymios Iosifidis

unread,
Mar 27, 2020, 7:23:42 AM3/27/20
to ClusterHAT
Hello!

I am now having real dilema, I would like to choose between k3s and docker swarm. It seems that the k3s is more resource intensive and has limited support
to the docker swarm, am I correct?

What is your opinion on this?

Thank you!

Dario Molinari

unread,
Mar 27, 2020, 8:01:14 AM3/27/20
to ClusterHAT
Same conundrum here, my thoughts are to have to look at a different approach: 
1. tasks orchestration (Puppet Bolt/Ansible/etc.)
2. distributed computing (Dispy, Slurm, etc.) 
3. or parallel programming in occam or similars

I am no developer so it's either going to be 1. or 2.
Either way, more hacking ahead! :-D

Cheers
Dario

Ciprian Manea

unread,
Mar 27, 2020, 9:03:15 AM3/27/20
to clust...@googlegroups.com
4. parallel programming in Golang 😎

To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/5f10939d-8ea6-442c-9ab2-1b8cf9adb8b3%40googlegroups.com.

Eythymios Iosifidis

unread,
Mar 27, 2020, 9:35:03 AM3/27/20
to ClusterHAT
Hey!
These arguments are really great but what about the performance?

Well sorry for asking all these questions but i am really having a trouble deciding here :)

Thanks again!

Τη Παρασκευή, 27 Μαρτίου 2020 - 3:03:15 μ.μ. UTC+2, ο χρήστης Ciprian Manea έγραψε:
4. parallel programming in Golang 😎

Dario Molinari

unread,
Mar 27, 2020, 9:58:19 AM3/27/20
to ClusterHAT
Eythymios, Rudi got Swarm running on it, see his post https://groups.google.com/d/msg/clusterhat/0Xj-b-95qnQ/ucPQhUWWBgAJ maybe it's better?

Cheers
Dario

Ciprian Manea

unread,
Mar 27, 2020, 12:58:07 PM3/27/20
to clust...@googlegroups.com
IMO, Zero’s are too small even for k3s

there are lite-er setups with containerd.io, but honestly — who has time to find all the quirks in that too?

I’d prefer to learn & practice some barebone Golang, on top of ClusterHAT Pi/cnat images and I know they [should] work with minimal fiddling 👌


To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/50c8ccac-3ae6-44a2-b38c-01fc01bd5fdc%40googlegroups.com.

Ciprian Manea

unread,
Mar 28, 2020, 1:37:36 AM3/28/20
to clust...@googlegroups.com
(also from the "Kubernetes in 15 minutes" link I posted here)
  • SD card — 32GB recommended, larger is up to you, but Kubernetes writes to disk a lot and could kill a card, so I tend to prefer buying more smaller cards.

Eythymios Iosifidis

unread,
Mar 28, 2020, 12:11:38 PM3/28/20
to ClusterHAT
Hey there!

Thank you all for providing your opinions ideas and references for setting a k3s. Especially Ciprian.

Below you can find my fresh and current configuration. However as you can observe the cluster is under load just sitting there idle :)

pi@cbridge:~ $ sudo  kubectl top node
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%  
cbridge   484m         12%    572Mi           14%      
p1        1414m        141%   194Mi           44%      
p2        976m         97%    189Mi           43%      
p3        1023m        102%   193Mi           44%      
p4        1046m        104%   191Mi           44%   



Τη Σάββατο, 28 Μαρτίου 2020 - 7:37:36 π.μ. UTC+2, ο χρήστης Ciprian Manea έγραψε:
(also from the "Kubernetes in 15 minutes" link I posted here)
  • SD card — 32GB recommended, larger is up to you, but Kubernetes writes to disk a lot and could kill a card, so I tend to prefer buying more smaller cards.

On Thu, Mar 26, 2020 at 5:58 AM Ciprian Manea <cipria...@gmail.com> wrote:
But look 👀 at those beautiful colors 
To unsubscribe from this group and stop receiving emails from it, send an email to clust...@googlegroups.com.

Stephen Greszczyszyn

unread,
Jun 29, 2020, 4:43:08 PM6/29/20
to ClusterHAT
I followed all the steps and got k3s working on my Pi3A+ with clusterhat, but for some reason kubectl was not responsive when even running "kubectl get nodes". Now after restarting I can't even really log in, even from USB/console with the ClusterHat disconnected.  I guess I'm going to try it again and start from scratch with the 32-bit CNAT images.  Does anyone have any ideas or should I try the 64-bit images?

Thanks,

Stephen

Stephen Greszczyszyn

unread,
Jul 7, 2020, 4:12:39 PM7/7/20
to ClusterHAT
I'm still struggling with this - trying to get k3s to load on the master/controller node, but seems like there are memory allocation errors.  I'm running a Pi3A+ that I think only comes with 512MB RAM, which likely isn't enough :(

Jul 07 21:09:21 cnat k3s[900]: time="2020-07-07T21:09:21.684697491+01:00" level=info msg="Waiting for master node  startup: resource name may not be empty"

Jul 07 21:09:21 cnat k3s[900]: time="2020-07-07T21:09:21.797304053+01:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"

Jul 07 21:09:21 cnat k3s[900]: time="2020-07-07T21:09:21.821111918+01:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"

Jul 07 21:09:21 cnat k3s[900]: containerd: fork/exec /var/lib/rancher/k3s/data/39a08c3192d96b9e0e5f20e12690a943b118bfe386f1d3df65a573a7a317787d/bin/containerd: cannot allocate memory

Jul 07 21:09:21 cnat systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

Jul 07 21:09:21 cnat systemd[1]: k3s.service: Failed with result 'exit-code'.

Ciprian Manea

unread,
Jul 7, 2020, 4:30:20 PM7/7/20
to clust...@googlegroups.com
yes, that must be it.

try a 3B/3B+ (it worked for me w/o issues)

Raspberry Pi 3A+ specs
  • SoC: Broadcom BCM2837B0 quad-core A54 (ARMv8) 64-bit @ 1.4GHz.
  • GPU: Broadcom VideoCore IV.
  • Networking: 2.4GHz and 5GHz 802.11b/g/n/ac wireless LAN.
  • RAM: 512MB LPDDR2 SDRAM.
  • Bluetooth: Bluetooth 4.2, Bluetooth Low Energy (BLE)

To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/69c11616-a4de-4757-ba7c-9313a3b5ae4bo%40googlegroups.com.

Stephen Greszczyszyn

unread,
Jul 7, 2020, 4:41:38 PM7/7/20
to ClusterHAT
Yes, that was it - just switched to a Pi3B with 1GB RAM and it worked.  So I guess I need a Pi3 minimum as a base - there is not much I can do even with k3s on the clusterhat, but still cool that it can even run!  I'd love someday to have a Pi4 base with 4 x Pi4 compute modules in a hat using USB3 for networking.  That would be pretty cool.

Ciprian Manea

unread,
Jul 7, 2020, 4:55:50 PM7/7/20
to clust...@googlegroups.com
have a look at https://turingpi.com/ (and try to get CM3+ while they're hot 🤓)

To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/7998675c-8bc8-48fb-aa0b-26883b9880ceo%40googlegroups.com.

Stephen Greszczyszyn

unread,
Jul 7, 2020, 5:21:24 PM7/7/20
to clust...@googlegroups.com
I was following Jeff Geerling on Youtube and his turingpi and other pi clusters.  I did have one of the turingpis on order but cancelled it as I already have a stack of 4 x Pi3 and 4 x Pi4 where I'm messing around with K3s and kubeadm clusters.  By the way, thanks for the tip on k3sup and getting things working on the clusterhat!

Bart Woloszyn

unread,
Jul 12, 2020, 12:56:50 PM7/12/20
to ClusterHAT
You are the man Ciprian! Just got it workng on cbridge and learned a lot on the way:) I am really greatfull to you, Alex Ellis and Chris Burton for this lesson, thank you!
Cheers:)
B

nimrod....@gmail.com

unread,
Aug 27, 2020, 10:53:40 AM8/27/20
to ClusterHAT
Hi,

I am having issue with accessing the dashboard on my laptop. Any advice on this?

Ciprian Manea

unread,
Aug 31, 2020, 3:00:06 AM8/31/20
to ClusterHAT
Oh, lookie: replacing RPi Zero's with Banana Pi M2 Zero gives lower standby CPU usage

$ kubectl top node

NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   

cnat   399m         9%     443Mi           51%       

p1     158m         3%     199Mi           40%       

p2     152m         3%     185Mi           37%       

p3     132m         3%     197Mi           40%       

p4     159m         3%     192Mi           38%       



sgre...@gmail.com

unread,
Aug 31, 2020, 4:34:54 PM8/31/20
to ClusterHAT
That's cool, I wonder if we could run something useful like OpenFaaS on this cluster: https://docs.openfaas.com/community/

Then again, one bananapi costs the same as 4 PiZeros!

Mina Ramses

unread,
Aug 31, 2020, 7:20:21 PM8/31/20
to clust...@googlegroups.com
Getting ssh handshake failed when tried to k3sup install --ip 
At the time i was able to ssh using the same key ssh -i /home/pi/.ssh/id_rsa pi@cnat
Any idea what's wrong?

To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/c29eb550-5d20-461d-92ab-13d0f7a102a0n%40googlegroups.com.

Ciprian Manea

unread,
Aug 31, 2020, 11:45:36 PM8/31/20
to clust...@googlegroups.com
Maybe it used to be expensive, not anymore 

Indeed, some shops have the Zero v1.3 even at about €5.50, but at pimoroni.com you’re limited at 1 per order.

Yes, you may deploy about anything with ark 

🙃

Ciprian Manea

unread,
Sep 1, 2020, 12:29:20 AM9/1/20
to clust...@googlegroups.com
openfaas works like a charm on the M2 Zero's (which I have them on top of an RPi 4B 1G 😎)

$ kubectl top pod -A

NAMESPACE     NAME                                     CPU(cores)   MEMORY(bytes)   

kube-system   coredns-8655855d6-b8b6k                  14m          7Mi             

kube-system   local-path-provisioner-6d59f47c7-sgzwk   13m          10Mi            

kube-system   metrics-server-7566d596c8-qd86j          9m           15Mi            

kube-system   svclb-traefik-54krl                      0m           1Mi             

kube-system   svclb-traefik-kssmv                      0m           1Mi             

kube-system   svclb-traefik-lck9l                      0m           2Mi             

kube-system   svclb-traefik-pk929                      0m           1Mi             

kube-system   svclb-traefik-ss27z                      0m           1Mi             

kube-system   traefik-758cd5fc85-nmtj5                 16m          11Mi            

openfaas      alertmanager-5c76b75f44-qck24            1m           6Mi             

openfaas      basic-auth-plugin-d4c78979-hsm5z         1m           3Mi             

openfaas      faas-idler-6446b647cc-8bj9m              1m           2Mi             

openfaas      gateway-d646c99cd-v77kn                  13m          14Mi            

openfaas      nats-7df64f5d8-r6dtk                     6m           4Mi             

openfaas      prometheus-759f55bcbd-6xmml              25m          23Mi            

openfaas      queue-worker-85dc5d8dc8-xbwwd            1m           2Mi             

openfaas-fn   figlet-54647f7fc6-jhmlx                  5m           2Mi             



you may choose to support Alex on Github https://github.com/sponsors/alexellis [and ask him questions]

🖖

sgre...@gmail.com

unread,
Sep 1, 2020, 5:38:00 AM9/1/20
to ClusterHAT
Thanks for the tips!  I did get my zeros from pimoroni at around £5, so the banana Pis are 4x the price.  I guess it makes sense as they have 4x the CPU.  I'll use them now for other projects.  I just ordered 4x banana pi from aliexpress for $20.

Really neat to see how small one can get a usable k8s cluster.

Mina Ramses

unread,
Sep 1, 2020, 6:13:08 PM9/1/20
to clust...@googlegroups.com
Please if anyone can help .. 
Did anyone get the same issue before ?
"SSH Handshake failed" while trying to: "k3sup install --ip " ?, any guidance will be really appreciated. 

On Thu, Aug 27, 2020, 9:53 AM nimrod....@gmail.com <nimrod....@gmail.com> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/c29eb550-5d20-461d-92ab-13d0f7a102a0n%40googlegroups.com.

Jimmy MG Lim

unread,
Dec 28, 2020, 9:14:38 AM12/28/20
to ClusterHAT
Hi wtorek,

wonder if you/anyone can share how you got your cbridge version working? i my cnat version was fine getting responses from 4xzeros but when i tried with another sd loaded with cbridge, I cant seem to see them on peer devices on local network, id assume they would get new set of ip addresses via dhcp from my router when using bridge mode?

note that i just swapped my controller but the zeros (images) are the same ones I used for cnat. wonder if thats an issue?

cheers
Jimmy

Jimmy MG Lim

unread,
Jan 3, 2021, 7:39:01 AM1/3/21
to ClusterHAT
Hi, happy new year all.

i think the issue i had was bridge was on eth0. and the ip i had was designated on wlan. in any case, I was testing k3sup and when joining from agent node to server (@cnat 172.19.181.254).. the agent did not trigger any issues when joining. however when i am listing the nodes, there are two issues. agent node(s) not showing up and the internal IP is wrong.

```
NAME   STATUS   ROLES    AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
cnat   Ready    master   51m   v1.19.5+k3s2   192.168.0.186   <none>        Raspbian GNU/Linux 10 (buster)   5.4.51-v7l+      containerd://1.4.3-k3s1
```

I checked the systemd service on the controller (which is my host) and k3s service was triggering with the right ip (--tls-san 172.19.181.254).

anyone can shed any light on this? (1) why internalip is stuck on wifiip (2) agent nodes not appearing.

cheers for help!
Jimmy

Jimmy MG Lim

unread,
Jan 3, 2021, 11:00:58 AM1/3/21
to ClusterHAT
hi Mina..

did you try to ssh into each zero before you tried installing? it looks like ssh might be disabled in your zeros. drop an empty ssh file into /boot folder and restart your zeros.

this might help. if you have gone past that stage. did you manage to get all zero agents added to your controller/server?

cheers
Jimmy

Jimmy MG Lim

unread,
Jan 4, 2021, 4:10:49 PM1/4/21
to ClusterHAT
Just an update. managed to get cbridge working fine. caveat is that zero(s) will not work with k3s as arm6 is dropped. Running k3s check-config on the zeros yields:

- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: missing (fail)
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)

config cpusets failing thus none of the agents will be able to spin up k3s (on zeros) .. the base controller however works fine with rbpi4.

bit of a shame as k3s should technically run on weaker zeros. so far concluded that zeros wont work for k3s despite above posts. cluster can perhaps be built for swarm or nomad instead.

cheers

hanur13

unread,
Sep 25, 2021, 5:00:37 AM9/25/21
to ClusterHAT
Hello,
I have similar issue with "SSH Handshake failed" while trying to: "k3sup install --ip " when installing k3sup.
I am installing Armbian image on Banana Pi M2 Zero.

> did you try to ssh into each zero before you tried installing? it looks like ssh might be disabled in your zeros. drop an empty ssh file into /boot folder and restart your zeros.
I succed ssh into p1-p4 on Banana Pi


> if you have gone past that stage. did you manage to get all zero agents added to your controller/server?
do you mean to put the public key of controller to the ~/.ssh/authorized_keys to each of p1-p4?

partially I am following this but didn't work

Nurul

hanur13

unread,
Sep 26, 2021, 12:47:22 AM9/26/21
to ClusterHAT
Do you add that line  (cg*..) to /boot/armbian_first_run.txt for Armbian running on BananaPi?
similar to /boot/cmdline.txt in Pi7s

Nurul
On Wednesday, March 25, 2020 at 7:14:37 PM UTC+9 cipria...@gmail.com wrote:
Oh, and the most important thing — which I forgot 😊, all your ClusterHAT Pi’s must have these cg* added to cmdline.txt then reboot them to activate it

PS. I prefer ipv6.disable=1 but you won’t have to set it 😎

$ cat /boot/cmdline.txt 

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=dafeb7ce-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait ipv6.disable=1 quiet cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Ciprian Manea

unread,
Sep 26, 2021, 2:45:38 AM9/26/21
to clust...@googlegroups.com
armbian/bananaPi is a bit different animal / beast

the cgroup* in that old email are meant only for Pi Zero's

--
You received this message because you are subscribed to the Google Groups "ClusterHAT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clusterhat+...@googlegroups.com.

hanur13

unread,
Sep 27, 2021, 6:58:55 AM9/27/21
to ClusterHAT
Ok so I guest I need not to add the line to the armbian.

I am following your begining post to install k3sup

> k3sup install --ip 172.19.181.254 --user pi

And I have a following question which might be very basic:
I have an issue with using above command "Error: unable to connect to 172.19.181.254:22 over ssh", so I use below command to install k3sup:
$ k3sup install --local

However, there are an issue on in cgroup*  although I already manually added "cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory" at /boot/cmdline.txt:
...
[INFO]  Failed to find memory cgroup, you may need to add "cgroup_memory=1 cgroup_enable=memory" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)
...

Do you know how I could thoubleshoot it?


Nurul

Ciprian Manea

unread,
Sep 27, 2021, 7:39:45 AM9/27/21
to clust...@googlegroups.com
Have you restarted after the cmdline change? 🙃

You received this message because you are subscribed to a topic in the Google Groups "ClusterHAT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clusterhat/iUcfVqJ1aL0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clusterhat+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/clusterhat/fbbd6932-9e43-4ba7-a6ab-b274c34f96cdn%40googlegroups.com.

hanur13

unread,
Sep 29, 2021, 11:54:02 PM9/29/21
to ClusterHAT
Thanks for the insight.

After the flash sd card, I directly changed the cmdline by plug and modify it using another  pc.
When bring the RPi on, seems that it created a new cmdline.txt while modified one were rename into cmdline.old
After modify cmdline.txt again directly and reboot, there is no issue of memory cgroup appear. 

Chris Burton

unread,
Sep 30, 2021, 1:51:14 AM9/30/21
to ClusterHAT
Hi, 
After the flash sd card, I directly changed the cmdline by plug and modify it using another  pc.
When bring the RPi on, seems that it created a new cmdline.txt while modified one were rename into cmdline.old
After modify cmdline.txt again directly and reboot, there is no issue of memory cgroup appear.

On first boot the ClusterCTRL images reconfigures to the right image (cnat/cbridge/p1/etc) using cmdline.txt.

If you want to add your own options to cmdline.txt before first boot then you can add them before the " init=..." option as reconfig-clusterctrl removes "init=" and everything after it 

Chris. 

hanur13

unread,
Oct 11, 2021, 2:17:06 AM10/11/21
to ClusterHAT
If you want to add your own options to cmdline.txt before first boot then you can add them before the " init=..." option as reconfig-clusterctrl removes "init=" and everything after it 
Thanks for your explanation, Chris.

Initially I put that "init=.." option in the middle that make an issue with loop of booting.

Nurul
Reply all
Reply to author
Forward
0 new messages