High CPU spikes in 3.7.17 version

[Kushagra Bindal]

Hi Luke/Michael,

We are working on RabbitMQ 3.7.17 version right now. A few months back we have upgraded it from 3.6.10 to 3.7.17 version. Along with that we have updated the Erlang version from 19.3.4 to 22.0.7.

Since last few weeks we are observing very high CPU utilization on the newly deployed machines. 

PS: Both 3.6.10 and 3.7.17 are working on set of machine deployment configurations.

CPU utilization on 3.6.10 deployment is: 4-5 % on regular load condition.

CPU utilization on 3.7.17 deployment is: 50-60 % on regular load condition.

Can you guys please help in identifying the actual reason of these spikes. This is impacting badly to us.

Regards

Kushagra

[Kushagra Bindal]

Hi All,

Please let me know if anybody is having any proposed solution for this issue. 

We are struggling in managing the health of the system.

Regards

Kushagra

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/86bfb781-26f0-4fa6-a605-bd708e85359f%40googlegroups.com.

--

Regards,
Kushagra Bindal
+91-9013792807

[Luke Bakken]

Hi Kushagra,

Please be patient and do not reply to your own questions. The people who help out on this mailing list generally try to answer questions within a week. If you require urgent support, please see this page: https://www.rabbitmq.com/#support

Some questions for you:

* Do you have any monitoring in place? https://www.rabbitmq.com/monitoring.html

* What operating system and version are you using?

You say you upgraded RabbitMQ a few months back, yet these CPU issues started a few weeks ago. What changed?

Thanks,

Luke

On Monday, November 25, 2019 at 5:23:56 AM UTC-8, Kushagra Bindal wrote:

Hi All,

Please let me know if anybody is having any proposed solution for this issue. 

We are struggling in managing the health of the system.

Regards

Kushagra

On Mon, Nov 25, 2019 at 4:44 PM Kushagra Bindal <bindal....@gmail.com> wrote:

Hi Luke/Michael,

We are working on RabbitMQ 3.7.17 version right now. A few months back we have upgraded it from 3.6.10 to 3.7.17 version. Along with that we have updated the Erlang version from 19.3.4 to 22.0.7.

Since last few weeks we are observing very high CPU utilization on the newly deployed machines. 

PS: Both 3.6.10 and 3.7.17 are working on set of machine deployment configurations.

CPU utilization on 3.6.10 deployment is: 4-5 % on regular load condition.

CPU utilization on 3.7.17 deployment is: 50-60 % on regular load condition.

Can you guys please help in identifying the actual reason of these spikes. This is impacting badly to us.

Regards

Kushagra

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/86bfb781-26f0-4fa6-a605-bd708e85359f%40googlegroups.com.

[Kushagra Bindal]

Hi Luke,

No, we do not having such monitoring in place as of now.

We are using Centos7.7 version.

We have promoted our changes in October 1st week (this includes rabitmqadmin plugin upgrade as well), at that time there were around 5-10 vhost available. But then gradually when the vhost count of new deployment come closer to 3.6.10 version's vhost then now continuously it is operating on 50-60% CPU usage. On the otherside older version of RabbitMQ deployment is working properly with having same configuration (same Core, memory & OS).

Please let me know if you need any other input on this from my side.

Regards

Kushagra

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/86bfb781-26f0-4fa6-a605-bd708e85359f%40googlegroups.com.

[Luke Bakken]

Hi Kushagra,

Having monitoring in place is crucial. You could have application changes that are resulting in more messages being published, high connection churn, etc - https://www.rabbitmq.com/monitoring.html

Without monitoring we can't rule out application issues that have caused this. You can claim that there haven't been application changes but without monitoring to prove it we are both just hoping that's the case.

Please run this script on all nodes in your 3.7 cluster and send me the archives - https://github.com/rabbitmq/support-tools/blob/master/scripts/rabbitmq-collect-env

Could you please enable the rabbitmq_top plugin in both environments and report what is shown in the management "admin" section? Take a screenshot from both environments.

Please run this command on all nodes in your 3.7 cluster, capture the output, and attach the output to your response:

rabbitmq-diagnostics runtime_thread_stats

Finally, I can spend a little bit of time looking into this but you should be running either the latest 3.7.X version or 3.8.X version.

Thanks,

Luke

[Kushagra Bindal]

Hi Luke,

Based on our conversation, we had a detailed discussion with our senior leadership and we have decided to move to the latest stable version provided in the another thread i.e. 3.8.1.

Please provide details of below binaries that are compatible with 3.8.1 version of rabbitmq.

1. Erlang

2. RabbitMQAdmin

Based on your further suggestions, this is also have been decided to enable monitoring tool to capture the health of the system. Based on your last email, you asked to enable below two monitoring tools, I guess.

1. rabbitmq_diagnostics

2. rabbitmq_top

Can you please provide me more details around this, how to enable these tools in our environment? and what are different commands to monitor them.

Along with that please do let us know other monitoring tools that we can introduce to better track the health of the system.

Regards

Kushagra

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7364cdd0-afb3-4d18-a149-9d5c40e88e5f%40googlegroups.com.

[Luke Bakken]

Hi Kushagra,

Please use the official RabbitMQ documentation to answer your own questions, if possible. If you can't find the information you need, let us know how you tried to find it and we can assist.

Please provide details of below binaries that are compatible with 3.8.1 version of rabbitmq.

1. Erlang

2. RabbitMQAdmin

https://www.rabbitmq.com/which-erlang.html

https://www.rabbitmq.com/management-cli.html

Based on your further suggestions, this is also have been decided to enable monitoring tool to capture the health of the system. Based on your last email, you asked to enable below two monitoring tools, I guess.

1. rabbitmq_diagnostics

2. rabbitmq_top

Can you please provide me more details around this, how to enable these tools in our environment? and what are different commands to monitor them.

https://www.rabbitmq.com/monitoring.html

https://www.rabbitmq.com/community-plugins.html

https://www.rabbitmq.com/rabbitmq-diagnostics.8.html

Thanks,

Luke

[Kushagra Bindal]

Thanks Luke for providing the details.

I can see 3.8.2 is released, so can we use this version for our production use case?

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7b5bc5ca-d71d-4846-9ffe-da94678a9372%40googlegroups.com.

[Luke Bakken]

Yes, use 3.8.2.

On Monday, December 2, 2019 at 11:50:35 PM UTC-8, Kushagra Bindal wrote:

Thanks Luke for providing the details.

I can see 3.8.2 is released, so can we use this version for our production use case?

On Mon, Dec 2, 2019 at 10:13 PM Luke Bakken <lba...@pivotal.io> wrote:

Hi Kushagra,

Please use the official RabbitMQ documentation to answer your own questions, if possible. If you can't find the information you need, let us know how you tried to find it and we can assist.

Please provide details of below binaries that are compatible with 3.8.1 version of rabbitmq.

1. Erlang

2. RabbitMQAdmin

https://www.rabbitmq.com/which-erlang.html

https://www.rabbitmq.com/management-cli.html

Based on your further suggestions, this is also have been decided to enable monitoring tool to capture the health of the system. Based on your last email, you asked to enable below two monitoring tools, I guess.

1. rabbitmq_diagnostics

2. rabbitmq_top

Can you please provide me more details around this, how to enable these tools in our environment? and what are different commands to monitor them.

https://www.rabbitmq.com/monitoring.html

https://www.rabbitmq.com/community-plugins.html

https://www.rabbitmq.com/rabbitmq-diagnostics.8.html

Thanks,

Luke

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7b5bc5ca-d71d-4846-9ffe-da94678a9372%40googlegroups.com.

[Kushagra Bindal]

Thanks Luke.. I am using erlang version 22.1.8 with this. 

Will let you know as soon as it will get completed.

On Tue, Dec 3, 2019 at 7:34 PM Luke Bakken <lba...@pivotal.io> wrote:

Yes, use 3.8.2.

On Monday, December 2, 2019 at 11:50:35 PM UTC-8, Kushagra Bindal wrote:

Thanks Luke for providing the details.

I can see 3.8.2 is released, so can we use this version for our production use case?

On Mon, Dec 2, 2019 at 10:13 PM Luke Bakken <lba...@pivotal.io> wrote:

Hi Kushagra,

Please use the official RabbitMQ documentation to answer your own questions, if possible. If you can't find the information you need, let us know how you tried to find it and we can assist.

Please provide details of below binaries that are compatible with 3.8.1 version of rabbitmq.

1. Erlang

2. RabbitMQAdmin

https://www.rabbitmq.com/which-erlang.html

https://www.rabbitmq.com/management-cli.html

Based on your further suggestions, this is also have been decided to enable monitoring tool to capture the health of the system. Based on your last email, you asked to enable below two monitoring tools, I guess.

1. rabbitmq_diagnostics

2. rabbitmq_top

Can you please provide me more details around this, how to enable these tools in our environment? and what are different commands to monitor them.

https://www.rabbitmq.com/monitoring.html

https://www.rabbitmq.com/community-plugins.html

https://www.rabbitmq.com/rabbitmq-diagnostics.8.html

Thanks,

Luke

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7b5bc5ca-d71d-4846-9ffe-da94678a9372%40googlegroups.com.

--

Regards,
Kushagra Bindal
+91-9013792807

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/8f16523e-4624-44cc-af9f-8186e2411c35%40googlegroups.com.