K3D Kuberntes Cluster - Running RabbitMQ

Brian Greco

unread,

Mar 14, 2022, 8:46:30 PM3/14/22

to rabbitmq-users

Hello,

I use to easily run RabbitMQ within a local K3D Kubernetes cluster. Something has change I believe in the last several versions of K3D that now prevents the pod from starting. I completed the below also on a Minikube cluster and it worked fine and the RabbitMQ logs appears to be the same. The issue is easy to replicate:

Create a local K3D cluster

Save the following to rabbitmq.yaml

apiVersion: rabbitmq.com/v1beta1

kind: RabbitmqCluster

metadata: name: hello-world

kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
kubectl apply -f rabbitmq.yaml -n vehicles

kubectl get pods --all-namespaces

The rabbitmq pod will not start and if I describe the pod, I don't see any reasons as to why.

Was wonder if anyone else is running RabbitMQ within K3D and had any suggestions.

Thanks in advance,

brian

Message has been deleted

Aitor Alberto Perez Cedres

unread,

Mar 15, 2022, 12:45:28 PM3/15/22

to rabbitm...@googlegroups.com

Hello,

I usually run RabbitMQ within a local K3S Kubernetes without issues. I don't think there's a huge difference of running k3s inside a VM v.s. running k3s in Docker.

Do you get "stuck" the Pod in "Pending" state?

Describing the StatefulSet may also provide more hints. A common gotcha as well is not having a default storage class, or not enough resources assigned to Docker/k3s to run RabbitMQ with its default requests (2 cpu 2 GB memory).

Best regards,

Aitor

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Brian Greco <blgr...@gmail.com>
Sent: 15 March 2022 00:46
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: [Suspected Spam] [rabbitmq-users] K3D Kuberntes Cluster - Running RabbitMQ

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/877d6d04-8155-4680-b532-c99cf2055603n%40googlegroups.com.

Brian Greco

unread,

Mar 15, 2022, 2:40:25 PM3/15/22

to rabbitm...@googlegroups.com

Hi Aitor,

Thanks for responding. This is the high level status of all the resources and following this is the describe for the stateful set.

Thanks,

brian

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 4m12s
kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 4m9s
kube-system service/metrics-server ClusterIP 10.43.225.127 <none> 443/TCP 4m7s
kube-system service/traefik LoadBalancer 10.43.173.135 172.27.0.3,172.27.0.4 80:30702/TCP,443:32064/TCP 3m42s
vehicles service/test-rabbit-nodes ClusterIP None <none> 4369/TCP,25672/TCP 3m28s
vehicles service/test-rabbit ClusterIP 10.43.84.112 <none> 5672/TCP,15672/TCP,15692/TCP 3m28s

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/svclb-traefik 2 2 2 2 2 <none> 3m42s

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 1/1 1 1 4m9s
kube-system deployment.apps/local-path-provisioner 1/1 1 1 4m8s
rabbitmq-system deployment.apps/rabbitmq-cluster-operator 1/1 1 1 3m48s
kube-system deployment.apps/metrics-server 1/1 1 1 4m8s
kube-system deployment.apps/traefik 1/1 1 1 3m42s

NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-96cc4f57d 1 1 1 3m58s
kube-system replicaset.apps/local-path-provisioner-84bb864455 1 1 1 3m58s
rabbitmq-system replicaset.apps/rabbitmq-cluster-operator-755c76fc56 1 1 1 3m48s
kube-system replicaset.apps/metrics-server-ff9dbcb6c 1 1 1 3m58s
kube-system replicaset.apps/traefik-55fdc6d984 1 1 1 3m42s

NAMESPACE NAME READY AGE
vehicles statefulset.apps/test-rabbit-server 0/1 3m28s

NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system job.batch/helm-install-traefik-crd 1/1 16s 4m6s
kube-system job.batch/helm-install-traefik 1/1 17s 4m6s

NAMESPACE NAME ALLREPLICASREADY RECONCILESUCCESS AGE
vehicles rabbitmqcluster.rabbitmq.com/test-rabbit False Unknown 3m29s

----------------------------------------------------------------------------------------------------------------

Name: test-rabbit-server
Namespace: vehicles
CreationTimestamp: Tue, 15 Mar 2022 14:33:23 -0400
Selector: app.kubernetes.io/name=test-rabbit
Labels: app.kubernetes.io/component=rabbitmq
app.kubernetes.io/name=test-rabbit
app.kubernetes.io/part-of=rabbitmq
Annotations: rabbitmq.com/createdAt: 2022-03-15T18:33:23Z
Replicas: 1 desired | 1 total
Update Strategy: RollingUpdate
Partition: 0
Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/component=rabbitmq
app.kubernetes.io/name=test-rabbit
app.kubernetes.io/part-of=rabbitmq
Annotations: prometheus.io/port: 15692
prometheus.io/scrape: true
Service Account: test-rabbit-server
Init Containers:
setup-container:
Image: rabbitmq:3.9.13-management
Port: <none>
Host Port: <none>
Command:
sh
-c
cp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins /operator/enabled_plugins ; echo '[default]' > /var/lib/rabbitmq/.rabbitmqadmin.conf && sed -e 's/default_user/username/' -e 's/default_pass/password/' /tmp/default_user.conf >> /var/lib/rabbitmq/.rabbitmqadmin.conf && chmod 600 /var/lib/rabbitmq/.rabbitmqadmin.conf
Limits:
cpu: 100m
memory: 500Mi
Requests:
cpu: 100m
memory: 500Mi
Environment: <none>
Mounts:
/operator from rabbitmq-plugins (rw)
/tmp/default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
/tmp/erlang-cookie-secret/ from erlang-cookie-secret (rw)
/tmp/rabbitmq-plugins/ from plugins-conf (rw)
/var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
/var/lib/rabbitmq/mnesia/ from persistence (rw)
Containers:
rabbitmq:
Image: rabbitmq:3.9.13-management
Ports: 4369/TCP, 5672/TCP, 15672/TCP, 15692/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Limits:
cpu: 2
memory: 2Gi
Requests:
cpu: 1
memory: 2Gi
Readiness: tcp-socket :amqp delay=10s timeout=5s period=10s #success=1 #failure=3
Environment:
MY_POD_NAME: (v1:metadata.name)
MY_POD_NAMESPACE: (v1:metadata.namespace)
K8S_SERVICE_NAME: test-rabbit-nodes
RABBITMQ_ENABLED_PLUGINS_FILE: /operator/enabled_plugins
RABBITMQ_USE_LONGNAME: true
RABBITMQ_NODENAME: rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
K8S_HOSTNAME_SUFFIX: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
Mounts:
/etc/pod-info/ from pod-info (rw)
/etc/rabbitmq/conf.d/10-operatorDefaults.conf from rabbitmq-confd (rw,path="operatorDefaults.conf")
/etc/rabbitmq/conf.d/11-default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
/etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf from rabbitmq-confd (rw,path="userDefinedConfiguration.conf")
/operator from rabbitmq-plugins (rw)
/var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
/var/lib/rabbitmq/mnesia/ from persistence (rw)
Volumes:
plugins-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: test-rabbit-plugins-conf
Optional: false
rabbitmq-confd:
Type: Projected (a volume that contains injected data from multiple sources)
ConfigMapName: test-rabbit-server-conf
ConfigMapOptional: <nil>
SecretName: test-rabbit-default-user
SecretOptionalName: <nil>
rabbitmq-erlang-cookie:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
erlang-cookie-secret:
Type: Secret (a volume populated by a Secret)
SecretName: test-rabbit-erlang-cookie
Optional: false
rabbitmq-plugins:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
pod-info:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels['skipPreStopChecks'] -> skipPreStopChecks
Volume Claims:
Name: persistence
StorageClass:
Labels: app.kubernetes.io/component=rabbitmq
app.kubernetes.io/name=test-rabbit
app.kubernetes.io/part-of=rabbitmq
Annotations: <none>
Capacity: 10Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 5m28s statefulset-controller create Claim persistence-test-rabbit-server-0 Pod test-rabbit-server-0 in StatefulSet test-rabbit-server success
Normal SuccessfulCreate 5m28s statefulset-controller create Pod test-rabbit-server-0 in StatefulSet test-rabbit-server successful

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/Ku3Crx2y1Rw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/SN6PR05MB52772D0F1AC6102796EDF4EEC8109%40SN6PR05MB5277.namprd05.prod.outlook.com.

Aitor Alberto Perez Cedres

unread,

Mar 17, 2022, 8:39:22 AM3/17/22

to rabbitm...@googlegroups.com

It looks like the Pod is created correctly, from the StatefulSet perspective. I tried to reproduce the problem locally with K3D and I found that, by default, the cluster created doesn't have enough resources to accommodate the default resource request (1 CPU 2 GB RAM). If that's your case as well, you should be able to confirm by describing the Pod, you should observe an event about insufficient CPU and/or memory. In that case, you can remove the default resource requests using the following manifest:

apiVersion: rabbitmq.com/v1beta1

kind: RabbitmqCluster

metadata:

name: hello-world

spec:

resources:

requests: {}

limits: {}

Hope this helps,

Aitor

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Brian Greco <blgr...@gmail.com>

Sent: 15 March 2022 18:40
To: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com>
Subject: Re: [rabbitmq-users] K3D Kuberntes Cluster - Running RabbitMQ

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAHaDBwkuugZnrmLk5fSkofmmHAOZzZsoaSDuyyJ01CU_2OQfYg%40mail.gmail.com.

Brian Greco

unread,

Mar 18, 2022, 7:05:18 AM3/18/22

to rabbitm...@googlegroups.com

Aitor,

That corrected my issue. I greatly appreciate your help.

Best Regards,

Brian

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/SN6PR05MB52773E94FB78B1BEC7734132C8129%40SN6PR05MB5277.namprd05.prod.outlook.com.

Reply all

Reply to author

Forward