RMQ Cluster Crashed
126 views
Skip to first unread message
Satheesh Kumar
Dec 15, 2023, 1:12:49 AM12/15/23
to rabbitmq-users
Hi Team,
I am using RabbitMQ version 3.10 and it is running on AWS EKS Cluster. There are about 8 AWS Nodes running and in each node only RMQ pod is running.
Yesterday AWS terminated one of the node due to Hardware Issue. But Entire stack went down. We lost all the connections until we restore the RMQ
When we analyzed the logs only thing i able to found below log
Node 'rab...@test-rabbitmq2-6.test-rabbitmq2.test.svc.cluster.local' not responding **".
Just want to know. if only one node is going down then why entire cluster is broken
I am using RabbitMQ version 3.10 and it is running on AWS EKS Cluster. There are about 8 AWS Nodes running and in each node only RMQ pod is running.
Yesterday AWS terminated one of the node due to Hardware Issue. But Entire stack went down. We lost all the connections until we restore the RMQ
When we analyzed the logs only thing i able to found below log
Node 'rab...@test-rabbitmq2-6.test-rabbitmq2.test.svc.cluster.local' not responding **".
Just want to know. if only one node is going down then why entire cluster is broken
Satheesh Kumar
Dec 20, 2023, 4:05:19 AM12/20/23
to rabbitmq-users
Any update on this
kjnilsson
Dec 20, 2023, 4:43:56 AM12/20/23
to rabbitmq-users
Hi,
We'd need a lot more information and detail including logs, cluster configuration and details on applications to look at this. We don't know what you mean by "entire stack went down" - Did your apps crash? did they not reconnect?
sometimes when a node is force terminated and leaves a dangling TCP connection it could take a minute for it to be detected as down - after that what happens may depend on your partition handling configuration.
Reply all
Reply to author
Forward
0 new messages