RabbitMQ Connection Issue

[burak bingöl]

Hello all,

We use rabbitmq 3.8.9 version on k8s cluster with 5 nodes cluster.

We have over 1300 quorum queues on the cluster.

Our application is getting timeout exceptions while sending messages to rabbitmq.

We are getting hundreds of these repeated log messages from rabbitmq;

2022-01-12 01:57:31.735 [warning] <0.274.0> rabbit_sysmon_handler busy_dist_port <0.830.0> [{name,delegate_management_1},{initial_call,{delegate,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.166>,unknown}

2022-01-12 01:31:39.699 [info] <0.4835.1> queue 'Q1***' in vhost '/': detected a new leader {'%2F_Q1***','rabbit@Node****'} in term 3  

2022-01-12 01:31:28.763 [info] <0.19708.0> queue 'Q1***' in vhost '/': declining vote to {'%2F_Q1***','rabbit@Node***'} for term 3, current term 4

2022-01-12 01:31:27.766 [info] <0.19083.0> queue 'Q2***' in vhost '/': granting vote for {'%2F_ Q2***  ','rabbit@Node***'} with last indexterm {14190,1} for term 3 previous term was 2

We checked these for troubleshooting:

The application didn't restart at that time, rabbitmq nodes also seem available.

There isn't a network issue between rabbitmq and application, we checked the amqp and TCP packages with Wireshark.

What could be the root cause of that issue?

Best regards.

[Wes Peng]

You have provided too little info.

Such as system error logs, disk IO, FD numbers, network connection counting, queues size etc,  all these conditions can influence the running status of RMQ cluster.

Regards

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/efbcfb76-907e-4a99-9356-86db1354db65n%40googlegroups.com.

[burak bingöl]

Hello the current state of the cluster is below, I have checked the logs but there isn't error log on that period of time.

I took ss of observer tabs, you can find them below.

12 Ocak 2022 Çarşamba tarihinde saat 14:20:50 UTC+3 itibarıyla Wes Peng şunları yazdı:

[Wes Peng]

I am not sure, but you have only 1GB memory on the node?

That's not enough since you have a busy cluster.

regards.

[burak bingöl]

I realized it now but why is it saying there's 1 gb memory.

I am using helm to deploy rabbitmq so where is it getting this value? From resource request attribute?

12 Ocak 2022 Çarşamba tarihinde saat 15:15:01 UTC+3 itibarıyla Wes Peng şunları yazdı:

[Wes Peng]

One of your screencut just shows that.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/03a4090b-3742-410b-894f-7c90929b6ee4n%40googlegroups.com.

[burak bingöl]

Yes you are right, it seems on the observer 1GB but on the overview screen, it seems 
4.5 GB, why are they different?

I also tried to find where the 1 gb value comes from but I didn't find anything. How can I change the 1 gb value, which parameter should I change, do you have any idea about it?

13 Ocak 2022 Perşembe tarihinde saat 10:58:12 UTC+3 itibarıyla Wes Peng şunları yazdı:

[Nick West]

Is this not just a display of the total memory used? In my observer, this "Total" value fluctuates.