Performance bottlenecks in RabbitMQ 3.6.1

452 views

Skip to first unread message

Yohan Wadia

unread,

Apr 26, 2016, 12:03:08 AM4/26/16

to rabbitmq-users

Hi,

We are currently using RabbitMQ v3.6.1 for one of our applications and we are facing poor performance from RabbitMQ. Currently the RabbitMQ is installed on 3 AWS m3.large instances in a cluster. We have configured few parameters of RabbitMQ for obtaining high levels of utilization such as vm_memory_high_watermark, 0.8, however still we dont see any improvements in performance. We have tested the Queue with a prefetch of 10, 100, 200, 250 and 500 values, with optimal consumer utilization of 2-3%.

RabbitMQ is currently installed on Ubuntu 14.04.

Can anyone help us out with fine tunning/ performance optimization of this setup? We are looking to handle approx 5000 concurrent users/ sec.

Here is the output of the RabbitMQ config file:

# cat /etc/rabbitmq/rabbitmq.config

[

{rabbit,

[

{handshake_timeout, 60000},

{heartbeat, 600},

{channel_max, 128},

{vm_memory_high_watermark, 0.8},

{disk_free_limit, 5000000000},

{cluster_nodes, {['rabbit@mq-node1', 'rabbit@mq-node2', 'rabbit@mq-node3'], disc}}]},

{kernel,

[

]},

{rabbitmq_management,

[

]},

{rabbitmq_shovel,

[{shovels,

[

]}

]},

{rabbitmq_stomp,

[

]},

{rabbitmq_mqtt,

[

]},

{rabbitmq_amqp1_0,

[

]},

{rabbitmq_auth_backend_ldap,

[

]}].

Output of rabbitmqctl as well:

# rabbitmqctl status

Status of node 'rabbit@mq-rabbit' ...

[{pid,1559},

{running_applications,

[{rabbitmq_management,"RabbitMQ Management Console","3.6.1"},

{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.1"},

{webmachine,"webmachine","1.10.3"},

{mochiweb,"MochiMedia Web Server","2.13.0"},

{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.1"},

{rabbit,"RabbitMQ","3.6.1"},

{os_mon,"CPO CXC 138 46","2.2.14"},

{ssl,"Erlang/OTP SSL application","5.3.2"},

{public_key,"Public key infrastructure","0.21"},

{crypto,"CRYPTO version 2","3.2"},

{amqp_client,"RabbitMQ AMQP Client","3.6.1"},

{rabbit_common,[],"3.6.1"},

{inets,"INETS CXC 138 49","5.9.7"},

{mnesia,"MNESIA CXC 138 12","4.11"},

{compiler,"ERTS CXC 138 10","4.9.4"},

{xmerl,"XML parser","1.3.5"},

{syntax_tools,"Syntax tools","1.6.12"},

{asn1,"The Erlang ASN1 compiler version 2.0.4","2.0.4"},

{ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},

{sasl,"SASL CXC 138 11","2.3.4"},

{stdlib,"ERTS CXC 138 10","1.19.4"},

{kernel,"ERTS CXC 138 10","2.16.4"}]},

{os,{unix,linux}},

{erlang_version,

"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:64] [kernel-poll:true]\n"},

{memory,

[{total,60350672},

{connection_readers,171560},

{connection_writers,155896},

{connection_channels,357208},

{connection_other,150416},

{queue_procs,2240928},

{queue_slave_procs,0},

{plugins,1054320},

{other_proc,18294808},

{mnesia,72160},

{mgmt_db,1122600},

{msg_index,63600},

{other_ets,1363576},

{binary,2484384},

{code,26676279},

{atom,951441},

{other_system,5191496}]},

{alarms,[]},

{listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},

{vm_memory_high_watermark,0.8},

{vm_memory_limit,6274695168},

{disk_free_limit,5000000000},

{disk_free,6818217984},

{file_descriptors,

[{total_limit,924},{total_used,6},{sockets_limit,829},{sockets_used,3}]},

{processes,[{limit,1048576},{used,270}]},

{run_queue,0},

{uptime,2957},

{kernel,{net_ticktime,60}}]

Michael Klishin

unread,

Apr 26, 2016, 1:33:39 AM4/26/16

to rabbitm...@googlegroups.com

It's not possible to suggest much without knowing your workload (routing topology, total number of queues, data locality), whether

mirroring is used, to how many nodes, and so on. Consumer utilisation of 2-3% is very low

and hardly can be considered optimal as far as RabbitMQ goes.

I'm not sure what metric you mean. 5K concurrent connections, 5K messages/second, 5K inbound connections/second?

In general this has been discussed dozens of times before on this list.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.