Still need to manually stop, join_cluster and start every node?

108 views
Skip to first unread message

Kok How Teh

unread,
Sep 19, 2019, 3:46:18 AM9/19/19
to rabbitmq-users
Deployed the sample yaml files but doesn't work as expected. I still need to run a batch job on the cluster to manually stop, join_cluster and start for every pod. In addition, is there a way to configure vhosts, users and credentials using rabbitmq.conf file? Otherwise I have to do it in kerbernetes batch job after all the pods have booted up. My cluster DNS works fine as it is able to resolve `kubernetes` to `kubernetes.default.svc.cluster.local`

Michael Klishin

unread,
Sep 19, 2019, 3:49:34 AM9/19/19
to rabbitmq-users
I'm afraid we need a tad more detail on what exactly you have tried and what the observations are.
We strongly prefer to not guess on this list as guessing is too expensive.

See [1] to learn  when peer discovery will NOT be performed, [2] for troubleshooting advice and [3] for log file locations.
All peer discovery events are logged, in quite a bit of detail at debug level specifically.

[4] explains what features are available for virtual host, user, permission and topology preconfiguration.


On Thu, Sep 19, 2019 at 10:46 AM Kok How Teh <funco...@gmail.com> wrote:
Deployed the sample yaml files but doesn't work as expected. I still need to run a batch job on the cluster to manually stop, join_cluster and start for every pod. In addition, is there a way to configure vhosts, users and credentials using rabbitmq.conf file? Otherwise I have to do it in kerbernetes batch job after all the pods have booted up. My cluster DNS works fine as it is able to resolve `kubernetes` to `kubernetes.default.svc.cluster.local`

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/900a4450-ac71-49f6-8a6e-8379bfe8f5f2%40googlegroups.com.


--
MK

Staff Software Engineer, Pivotal/RabbitMQ
Message has been deleted
Message has been deleted

Kok How Teh

unread,
Sep 19, 2019, 9:50:00 PM9/19/19
to rabbitmq-users
Observed this log:

2019-09-20 01:24:41.775 [info] <0.238.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2019-09-20 01:24:41.775 [debug] <0.238.0> Peer discovery backend supports initialisation.
2019-09-20 01:24:41.775 [debug] <0.238.0> Peer discovery Kubernetes: initialising...
2019-09-20 01:24:41.775 [debug] <0.238.0> HTTP client proxy is not configured
2019-09-20 01:24:41.775 [debug] <0.238.0> Peer discovery backend initialisation succeeded.
2019-09-20 01:24:41.775 [info] <0.238.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2019-09-20 01:24:41.775 [info] <0.238.0> Peer discovery backend does not support locking, falling back to randomized delay
2019-09-20 01:24:41.775 [info] <0.238.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.
2019-09-20 01:24:41.808 [debug] <0.238.0> Response: {ok,{{"HTTP/1.1",200,"OK"},[{"date","Fri, 20 Sep 2019 01:24:41 GMT"},{"content-length","1209"},{"content-type","application/json"},{"audit-id","cf1cb95a-d478-4099-98d8-2af7ad3ec847"}],"{\"kind\":\"Endpoints\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"svc-rabbitmq\",\"namespace\":\"default\",\"selfLink\":\"/api/v1/namespaces/default/endpoints/svc-rabbitmq\",\"uid\":\"11d24c1d-d9b9-11e9-99d0-02c5c8b619a6\",\"resourceVersion\":\"14768605\",\"creationTimestamp\":\"2019-09-18T02:07:30Z\",\"labels\":{\"app\":\"svc-rabbitmq\",\"component\":\"rabbitmq\"}},\"subsets\":[{\"notReadyAddresses\":[{\"ip\":\"10.0.14.251\",\"hostname\":\"rabbitmq-0\",\"nodeName\":\"ip-10-0-13-45.ap-southeast-1.compute.internal\",\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"default\",\"name\":\"rabbitmq-0\",\"uid\":\"53a4357f-db45-11e9-99d0-02c5c8b619a6\",\"resourceVersion\":\"14768599\"}},{\"ip\":\"10.0.56.109\",\"hostname\":\"rabbitmq-1\",\"nodeName\":\"ip-10-0-59-58.ap-southeast-1.compute.internal\",\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"default\",\"name\":\"rabbitmq-1\",\"uid\":\"53a69cfd-db45-11e9-99d0-02c5c8b619a6\",\"resourceVersion\":\"14768604\"}},{\"ip\":\"10.0.72.133\",\"hostname\":\"rabbitmq-2\",\"nodeName\":\"ip-10-0-94-74.ap-southeast-1.compute.internal\",\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"default\",\"name\":\"rabbitmq-2\",\"uid\":\"53a98dda-db45-11e9-99d0-02c5c8b619a6\",\"resourceVersion\":\"14768509\"}}],\"ports\":[{\"name\":\"http\",\"port\":15672,\"protocol\":\"TCP\"},{\"name\":\"amqp\",\"port\":5672,\"protocol\":\"TCP\"}]}]}\n"}}
2019-09-20 01:24:41.808 [info] <0.238.0> k8s endpoint listing returned nodes not yet ready: rabbitmq-0, rabbitmq-1, rabbitmq-2
2019-09-20 01:24:41.808 [info] <0.238.0> All discovered existing cluster peers: 
2019-09-20 01:24:41.808 [info] <0.238.0> Discovered no peer nodes to cluster with
2019-09-20 01:24:41.811 [info] <0.43.0> Application mnesia exited with reason: stopped
2019-09-20 01:24:41.881 [info] <0.238.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

Michael Klishin

unread,
Sep 19, 2019, 9:50:46 PM9/19/19
to rabbitmq-users
It does not really explain what you have tried, so we have to guess.

The logs demonstrate a successful response from


but then the plugin logs

> 2019-09-20 01:24:41.808 [info] <0.238.0> k8s endpoint listing returned nodes not yet ready: rabbitmq-0, rabbitmq-1, rabbitmq-2

The plugin did discover some nodes but according to the Kubernetes API response, their pods are not yet ready.
Such nodes are not considered to be eligible peers.

Are you sure you are using a stateful set? What readiness probe looks like?


On Fri, Sep 20, 2019 at 4:45 AM Kok How Teh <funco...@gmail.com> wrote:
Observed this log:
2019-09-20 01:24:41.775 [info] <0.238.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2019-09-20 01:24:41.775 [debug] <0.238.0> Peer discovery backend supports initialisation.
2019-09-20 01:24:41.775 [debug] <0.238.0> Peer discovery Kubernetes: initialising...
2019-09-20 01:24:41.775 [debug] <0.238.0> HTTP client proxy is not configured
2019-09-20 01:24:41.775 [debug] <0.238.0> Peer discovery backend initialisation succeeded.
2019-09-20 01:24:41.775 [info] <0.238.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2019-09-20 01:24:41.775 [info] <0.238.0> Peer discovery backend does not support locking, falling back to randomized delay
2019-09-20 01:24:41.775 [info] <0.238.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.
2019-09-20 01:24:41.808 [debug] <0.238.0> Response: {ok,{{"HTTP/1.1",200,"OK"},[{"date","Fri, 20 Sep 2019 01:24:41 GMT"},{"content-length","1209"},{"content-type","application/json"},{"audit-id","cf1cb95a-d478-4099-98d8-2af7ad3ec847"}],"{\"kind\":\"Endpoints\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"svc-rabbitmq\",\"namespace\":\"default\",\"selfLink\":\"/api/v1/namespaces/default/endpoints/svc-rabbitmq\",\"uid\":\"11d24c1d-d9b9-11e9-99d0-02c5c8b619a6\",\"resourceVersion\":\"14768605\",\"creationTimestamp\":\"2019-09-18T02:07:30Z\",\"labels\":{\"app\":\"svc-rabbitmq\",\"component\":\"rabbitmq\"}},\"subsets\":[{\"notReadyAddresses\":[{\"ip\":\"10.0.14.251\",\"hostname\":\"rabbitmq-0\",\"nodeName\":\"ip-10-0-13-45.ap-southeast-1.compute.internal\",\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"default\",\"name\":\"rabbitmq-0\",\"uid\":\"53a4357f-db45-11e9-99d0-02c5c8b619a6\",\"resourceVersion\":\"14768599\"}},{\"ip\":\"10.0.56.109\",\"hostname\":\"rabbitmq-1\",\"nodeName\":\"ip-10-0-59-58.ap-southeast-1.compute.internal\",\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"default\",\"name\":\"rabbitmq-1\",\"uid\":\"53a69cfd-db45-11e9-99d0-02c5c8b619a6\",\"resourceVersion\":\"14768604\"}},{\"ip\":\"10.0.72.133\",\"hostname\":\"rabbitmq-2\",\"nodeName\":\"ip-10-0-94-74.ap-southeast-1.compute.internal\",\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"default\",\"name\":\"rabbitmq-2\",\"uid\":\"53a98dda-db45-11e9-99d0-02c5c8b619a6\",\"resourceVersion\":\"14768509\"}}],\"ports\":[{\"name\":\"http\",\"port\":15672,\"protocol\":\"TCP\"},{\"name\":\"amqp\",\"port\":5672,\"protocol\":\"TCP\"}]}]}\n"}}
2019-09-20 01:24:41.808 [info] <0.238.0> k8s endpoint listing returned nodes not yet ready: rabbitmq-0, rabbitmq-1, rabbitmq-2

On Thursday, September 19, 2019 at 3:46:18 PM UTC+8, Kok How Teh wrote:
Deployed the sample yaml files but doesn't work as expected. I still need to run a batch job on the cluster to manually stop, join_cluster and start for every pod. In addition, is there a way to configure vhosts, users and credentials using rabbitmq.conf file? Otherwise I have to do it in kerbernetes batch job after all the pods have booted up. My cluster DNS works fine as it is able to resolve `kubernetes` to `kubernetes.default.svc.cluster.local`

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e25f0e90-c6aa-43c0-a4a6-5efde7bf1027%40googlegroups.com.

Kok How Teh

unread,
Sep 19, 2019, 10:10:20 PM9/19/19
to rabbitmq-users
Fixed. That was due to me using "Parallel" podManagementPolicy. ;)
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

Michael Klishin

unread,
Sep 19, 2019, 10:14:10 PM9/19/19
to rabbitmq-users
Would you mind sharing the example that exhibited the problem and a diff of the fix with this list?
It would help other RabbitMQ and Kubernetes users ;)

Thank you for reporting back to the list.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e45340f2-fbf0-489b-b76d-2b19bc9ad3c9%40googlegroups.com.

Kok How Teh

unread,
Sep 19, 2019, 10:15:16 PM9/19/19
to rabbitmq-users
apiVersion: apps/v1 
kind: StatefulSet
metadata:
  name: rabbitmq
  labels: 
    app: rabbitmq
    component: rabbitmq
spec:
  serviceName: svc-rabbitmq
  podManagementPolicy: "OrderedReady" # Default is OrderedReady. The pods need to start ONE AT A TIME! Do NOT use Parallel!

Michael Klishin

unread,
Sep 19, 2019, 10:16:25 PM9/19/19
to rabbitmq-users
Thank you, this confirms my hypothesis. Have a good day :)

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e8def398-efb4-44af-917b-8ada5bdd5081%40googlegroups.com.

Michael Klishin

unread,
Sep 19, 2019, 10:38:28 PM9/19/19
to rabbitmq-users
The docs were updated to mention pod management policy [1]. Thanks again!

Kok How Teh

unread,
Sep 19, 2019, 10:44:06 PM9/19/19
to rabbitmq-users
Does it make sense to highlight that OrderedReady is the default Pod Management Policy so that not explicitly specifyinf this in the Statefulset manifest would also work?


--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Michael Klishin

unread,
Sep 20, 2019, 12:23:24 AM9/20/19
to rabbitmq-users
Done.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/4bcbb9be-9524-41a5-a337-c881ce4be446%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages