Rabbitmq pods are stuck in pending state

639 views

Skip to first unread message

Nataraj S

unread,

Nov 11, 2022, 6:51:44 AM11/11/22

to rabbitmq-users

After deploying below ulrs pods are created successfully but stuck in pending state, kindly provide the solution for this.

kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"

kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

kubectl describe pod/hello-world-server-0

output for the above command

Name: hello-world-server-0
Namespace: default
Priority: 0
Node: <none>
Labels: app.kubernetes.io/component=rabbitmq
app.kubernetes.io/name=hello-world
app.kubernetes.io/part-of=rabbitmq
controller-revision-hash=hello-world-server-65d4dfc66d
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=hello-world
service.istio.io/canonical-revision=latest
statefulset.kubernetes.io/pod-name=hello-world-server-0
Annotations: kubectl.kubernetes.io/default-container: rabbitmq
kubectl.kubernetes.io/default-logs-container: rabbitmq
prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true
sidecar.istio.io/status:
{"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["workload-socket","workload-certs","istio-envoy","istio-data","is...
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/hello-world-server
Init Containers:
setup-container:
Image: rabbitmq:3.10.2-management
Port: <none>
Host Port: <none>
Command:
sh
-c
cp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins /operator/enabled_plugins ; echo '[default]' > /var/lib/rabbitmq/.rabbitmqadmin.conf && sed -e 's/default_user/username/' -e 's/default_pass/password/' /tmp/default_user.conf >> /var/lib/rabbitmq/.rabbitmqadmin.conf && chmod 600 /var/lib/rabbitmq/.rabbitmqadmin.conf
Limits:
cpu: 100m
memory: 500Mi
Requests:
cpu: 100m
memory: 500Mi
Environment: <none>
Mounts:
/operator from rabbitmq-plugins (rw)
/tmp/default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
/tmp/erlang-cookie-secret/ from erlang-cookie-secret (rw)
/tmp/rabbitmq-plugins/ from plugins-conf (rw)
/var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
/var/lib/rabbitmq/mnesia/ from persistence (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t5h4c (ro)
istio-init:
Image: docker.io/istio/proxyv2:1.14.2
Port: <none>
Host Port: <none>
Args:
istio-iptables
-p
15001
-z
15006
-u
1337
-m
REDIRECT
-i
*
-x

-b
*
-d
15090,15021,15020
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 10m
memory: 40Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t5h4c (ro)
Containers:
rabbitmq:
Image: rabbitmq:3.10.2-management
Ports: 4369/TCP, 5672/TCP, 15672/TCP, 15692/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Limits:
cpu: 2
memory: 2Gi
Requests:
cpu: 1
memory: 2Gi
Readiness: http-get http://:15020/app-health/rabbitmq/readyz delay=10s timeout=5s period=10s #success=1 #failure=3
Environment:
MY_POD_NAME: hello-world-server-0 (v1:metadata.name)
MY_POD_NAMESPACE: default (v1:metadata.namespace)
K8S_SERVICE_NAME: hello-world-nodes
RABBITMQ_ENABLED_PLUGINS_FILE: /operator/enabled_plugins
RABBITMQ_USE_LONGNAME: true
RABBITMQ_NODENAME: rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
K8S_HOSTNAME_SUFFIX: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
Mounts:
/etc/pod-info/ from pod-info (rw)
/etc/rabbitmq/conf.d/10-operatorDefaults.conf from rabbitmq-confd (rw,path="operatorDefaults.conf")
/etc/rabbitmq/conf.d/11-default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
/etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf from rabbitmq-confd (rw,path="userDefinedConfiguration.conf")
/operator from rabbitmq-plugins (rw)
/var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
/var/lib/rabbitmq/mnesia/ from persistence (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t5h4c (ro)
istio-proxy:
Image: docker.io/istio/proxyv2:1.14.2
Port: 15090/TCP
Host Port: 0/TCP
Args:
proxy
sidecar
--domain
$(POD_NAMESPACE).svc.cluster.local
--proxyLogLevel=warning
--proxyComponentLogLevel=misc:error
--log_output_level=default:info
--concurrency
2
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 10m
memory: 40Mi
Readiness: http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
Environment:
JWT_POLICY: third-party-jwt
PILOT_CERT_PROVIDER: istiod
CA_ADDR: istiod.istio-system.svc:15012
POD_NAME: hello-world-server-0 (v1:metadata.name)
POD_NAMESPACE: default (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
HOST_IP: (v1:status.hostIP)
PROXY_CONFIG: {}

ISTIO_META_POD_PORTS: [
{"name":"epmd","containerPort":4369,"protocol":"TCP"}
,{"name":"amqp","containerPort":5672,"protocol":"TCP"}
,{"name":"management","containerPort":15672,"protocol":"TCP"}
,{"name":"prometheus","containerPort":15692,"protocol":"TCP"}
]
ISTIO_META_APP_CONTAINERS: rabbitmq
ISTIO_META_CLUSTER_ID: Kubernetes
ISTIO_META_INTERCEPTION_MODE: REDIRECT
ISTIO_META_WORKLOAD_NAME: hello-world-server
ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/default/statefulsets/hello-world-server
ISTIO_META_MESH_ID: cluster.local
TRUST_DOMAIN: cluster.local
ISTIO_KUBE_APP_PROBERS: {"/app-health/rabbitmq/readyz":{"tcpSocket":{"port":5672},"timeoutSeconds":5}}
Mounts:
/etc/istio/pod from istio-podinfo (rw)
/etc/istio/proxy from istio-envoy (rw)
/var/lib/istio/data from istio-data (rw)
/var/run/secrets/istio from istiod-ca-cert (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t5h4c (ro)
/var/run/secrets/tokens from istio-token (rw)
/var/run/secrets/workload-spiffe-credentials from workload-certs (rw)
/var/run/secrets/workload-spiffe-uds from workload-socket (rw)
Conditions:
Type Status
PodScheduled False
Volumes:
workload-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
workload-certs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
istio-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-podinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
istio-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 43200
istiod-ca-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-ca-root-cert
Optional: false
persistence:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: persistence-hello-world-server-0
ReadOnly: false
plugins-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: hello-world-plugins-conf
Optional: false
rabbitmq-confd:
Type: Projected (a volume that contains injected data from multiple sources)
ConfigMapName: hello-world-server-conf
ConfigMapOptional: <nil>
SecretName: hello-world-default-user
SecretOptionalName: <nil>
rabbitmq-erlang-cookie:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
erlang-cookie-secret:
Type: Secret (a volume populated by a Secret)
SecretName: hello-world-erlang-cookie
Optional: false
rabbitmq-plugins:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
pod-info:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels['skipPreStopChecks'] -> skipPreStopChecks
kube-api-access-t5h4c:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m28s default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

kubectl describe pvc persistence-hello-world-server-0

output:

Name: persistence-hello-world-server-0
Namespace: default
StorageClass: robin
Status: Pending
Volume:
Labels: app.kubernetes.io/component=rabbitmq
app.kubernetes.io/name=hello-world
app.kubernetes.io/part-of=rabbitmq
Annotations: volume.beta.kubernetes.io/storage-provisioner: robin
volume.kubernetes.io/selected-node: robincnp.tcl.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: hello-world-server-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 87s persistentvolume-controller waiting for first consumer to be created before binding
Normal Provisioning 26s (x8 over 87s) robin_csi-provisioner-robin-5476477774-zgtj9_1c138711-ded5-423f-9405-465ac2e4c764 External provisioner is provisioning volume for claim "default/persistence-hello-world-server-0"
Warning ProvisioningFailed 26s (x8 over 87s) robin_csi-provisioner-robin-5476477774-zgtj9_1c138711-ded5-423f-9405-465ac2e4c764 failed to provision volume with StorageClass "robin": rpc error: code = Aborted desc = Error: {"message": "Your Robin License has EXPIRED and cluster is locked.\nPlease contact your Robin Account Manager to get new license or visit https://get.robin.io to renew your license. Robin Version: 5.3.11-217.", "operation_allowed": false}
Normal ExternalProvisioning 12s (x6 over 87s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "robin" or manually created by system administrato

Michal Kuratczyk

unread,

Nov 11, 2022, 6:58:22 AM11/11/22

to rabbitm...@googlegroups.com

Hi,

Nothing to do with RabbitMQ - a persistent volume can't be provisioned, so the pod won't start. The reason seems to be an expired Robin license:

"Your Robin License has EXPIRED and cluster is locked."

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/0b7909c1-b1c8-48a5-853f-d90b9de35eb4n%40googlegroups.com.

Michał

RabbitMQ team

Reply all

Reply to author

Forward

0 new messages