RabbitMQ - High CPU utilization on CentOS

[jeevan kamkar]

Hi,

I am trying to do a performance test - CPU, Memory, iOPS & round trip delay on an application which is hosted on multiple servers and each servers are having multiple cores. Here are the details

Configurations:

• RabbitMQ server - 2.4GHz, 8 cores, 12GB RAM
• Other servers - 2.4GHz 8 cores, 8GB RAM (there are 6 such servers)
• Number of concurrent users (loaded using Jmeter) - 100
• Number of concurrent connections @ RabbitMQ - 500
• RabbitMQ version - 3.61
• Erlang version - 18.2
  
• CentOS kernel version - 3.10
  

Observations:

• The CPU usage observed to be almost 40% during overnight test.
• Only 1 core seem to have been used as the TOP command gives almost 98% CPU for one particular thread. However, overall CPU utilization in TOP command is 40%
• When see memory on TOP command, approximately 4GB is untraceable, neither appears in TOP as used memory or free memory!

Attached configuration file which I am using. Please help me to arrive at solutions for both untraced memory, high CPU utilization & fine tuning it to run on multiple cores.

Warm Regards

Jeevan

[Jean-Sébastien Pédron]

On 31/05/2016 06:54, jeevan kamkar wrote:
> Hi,

Hi!

> I am trying to do a performance test - CPU, Memory, iOPS & round trip
> delay on an application which is hosted on multiple servers and each
> servers are having multiple cores. Here are the details
>
> Configurations:
>

> * Other servers - 2.4GHz 8 cores, 8GB RAM (there are 6 such servers)
> * Number of concurrent users (loaded using Jmeter) - 100
> * Number of concurrent connections @ RabbitMQ - 500

Could you please describe the workload in more details?

o How many producers/consumers?
o How many messages published and at which rate?
o What is the size of those messages?
o How many queues do they use?
o What is the topology of the exchanges/bindings/queues?

--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[jeevan kamkar]

Hello Jean-Sébastien Pédron,

Thanks for the reply. Please find the details below.

    o  How many producers/consumers?  - 10 consumers
    o  How many messages published and at which rate? - Message rate defined per queue as in below table
    o  What is the size of those messages? - Size is defined in the table beow
    o  How many queues do they use? - 10 queues, each queue is connected with the consumer
    o  What is the topology of the exchanges/bindings/queues?  - Direct exchange

Queue_Name	Message Rate (per sec)	Message Size (KB)
Queue 1	34 to 200	2.1 to 9.4
Queue 2	140 to 1500	2.1 to 9.4
Queue 3	6 to 200	2.1 to 9.4
Queue 4	80 to 500	2.1 to 9.4
Queue 5	72 to 1000	2.1 to 9.4
Queue 6	11 to 500	2.1 to 9.4
Queue 7	40 to 500	2.1 to 9.4
Queue 8	20 to 400	2.1 to 9.4
Queue 9	8 to 200	2.1 to 9.4
Queue 10	8 to 200	2.1 to 9.4

Kindly help us to resolve this issue.

Thanks in advance

Best Regards
Jeevan

[Michael Klishin]

For10 queues, 500 concurrent connections, likely as many channels, and between 400 and over 5K messages published
a second 40% of total CPU or roughly one core constantly used doesn't sound unreasonably high.

For just one core to be used you'd have to limit your Erlang VM to just 1 scheduler, which isn't the case by default.
RabbitMQ startup banner (can be found in the logs) should mention how many schedulers are used.

You can enable HiPE compilation (a JIT compiler in the runtime) which will trade off server startup time
for lower CPU use:
https://github.com/rabbitmq/rabbitmq-server/blob/stable/docs/rabbitmq.config.example#L284

and tweak scheduler flags (the effect of which is highly workload-dependent), see

https://github.com/rabbitmq/rabbitmq-server/issues/612
+sbt on http://erlang.org/doc/man/erl.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ