Rabbitmq server sometimes freezes and has a high cpu usage under ets.

张子炫

unread,

Mar 21, 2022, 8:24:59 AM3/21/22

to rabbitmq-users

What does it mean ? How can I fix it? It has been acting like this for several months. We really hope to find the answer.

张子炫

unread,

Mar 21, 2022, 8:27:29 AM3/21/22

to rabbitmq-users

Everyone gets really tired about this issue.

Wes Peng

unread,

Mar 21, 2022, 8:27:49 AM3/21/22

to rabbitm...@googlegroups.com

You should provide us more details including logs, versions, architecture and deployment description.

Thanks

On Mon, Mar 21, 2022 at 20:25 张子炫 <zhangzi...@gmail.com> wrote:

What does it mean ? How can I fix it? It has been acting like this for several months. We really hope to find the answer.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f9eddbd1-7013-49a5-8700-4c0361a64f0bn%40googlegroups.com.

张子炫

unread,

Mar 21, 2022, 8:41:59 AM3/21/22

to rabbitmq-users

RabbitMQ 3.9.13 With Erlang 24.3 running in docker.

There are no error logs in console output.

Thank you for your quick reply.

Message has been deleted

Michal Kuratczyk

unread,

Mar 21, 2022, 11:23:11 AM3/21/22

to rabbitm...@googlegroups.com

Hi,

Can you share anything about your workload? Number of connections, msg/s, queue types, etc. Ideally some screenshots from the Management UI / monitoring dashboards when this happens. Also, even if there are no error logs - if there are logs at all that are printed during that time, this would help us understand what could be the trigger for this.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/41c60612-4a03-4a1c-9feb-1bf482a4dba8n%40googlegroups.com.

--

Michał

RabbitMQ team

Michal Kuratczyk

unread,

Mar 22, 2022, 4:48:35 AM3/22/22

to rabbitm...@googlegroups.com

I have received some logs from this thread and believe the reason must be high connection churn - there are dozens of connections established every second.

RabbitMQ is not meant to be used like that: https://www.rabbitmq.com/connections.html#high-connection-churn

Best,

Michał
RabbitMQ team

--
Michał
RabbitMQ team
RabbitMQ team

Message has been deleted

Wes Peng

unread,

Mar 22, 2022, 4:49:18 PM3/22/22

to rabbitm...@googlegroups.com

Maybe the OP should consider using a proxy like HAProxy in front of the MQ cluster.

Thanks

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0sqhvF57t0L42CNoL5GnuS8p8iviR_czOFnfBoKAsv4Nw%40mail.gmail.com.

Michal Kuratczyk

unread,

Mar 22, 2022, 5:07:47 PM3/22/22

to rabbitm...@googlegroups.com

Hi,

It's not normal. It is most likely a bug in your application that overloads RabbitMQ.

The logs you shared with me showed hundreds of connections being opened in a very short time and many of them unexpectedly closed soon after.

This is called connection churn and I shared a link that discusses this phenomenon. If that's what your app does, there is not much RabbitMQ can do about this.

If you don't think that's the case then please share full logs and additional monitoring data. Explain what you app does when this happens and try to reproduce the

problem in a different environment (we have a tool for generating all kinds of load that can be helpful: https://github.com/rabbitmq/rabbitmq-perf-test).

If we have a way to reproduce this or sufficient monitoring details to understand what could be going on, we will fix it. But most likely it's your application opening connections over and over.

Best,

On Tue, Mar 22, 2022 at 9:49 PM 张子炫 <zhangzi...@gmail.com> wrote:

We have 2200 consumers on a single rabbitmq node currently . Is this phenomenon normal under the workload?

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/fd853897-7125-4f42-96d7-9af3bdcaa010n%40googlegroups.com.

--

Michał

RabbitMQ team

张子炫

unread,

Mar 22, 2022, 9:55:26 PM3/22/22

to rabbitmq-users

Thank you. Consumers have a healthcheck that ping mq every 60 seconds. Maybe that's the problem. We'll modify the healthcheck behavior next time we run into this issue.

张子炫

unread,

Mar 31, 2022, 9:52:20 PM3/31/22

to rabbitmq-users

张子炫

unread,

Mar 31, 2022, 9:52:41 PM3/31/22

to rabbitmq-users

So it happens again

Wes Peng

unread,

Mar 31, 2022, 10:11:24 PM3/31/22

to rabbitm...@googlegroups.com

It is just impolite to throw out some pics to let users guess what you are trying to say.

You should be able to describe your questions more clearly with a few words and paste the relative logs here.

Thanks

张子炫

unread,

Mar 31, 2022, 10:20:50 PM3/31/22

to rabbitmq-users

Sorry about that, pictures are above.I will paste logs later.

张子炫

unread,

Mar 31, 2022, 10:28:17 PM3/31/22

to rabbitmq-users

Log is kind of huge. I upload it as an attachment.

rabbit20220401.log

Michal Kuratczyk

unread,

Apr 1, 2022, 3:49:01 AM4/1/22

to rabbitm...@googlegroups.com

It seems like nothing has changed in your environment since you initially reported this problem - just take a look at the beginning of your log file - within the first second

(2022-04-01 02:05:50), you opened 162 AMQP/TLS connections. That means that RabbitMQ had to perform 162 TLS handshakes and get ready to serve these connections.

AMQP is not a stateless protocol - RabbitMQ needs to remember publishers and consumers, to later send them acknowledgements/confirms, push messages to them and so on.

Your application should use long-running connections.

Best,

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/a416627d-e93b-4dcd-b2d9-2657ca0fced6n%40googlegroups.com.