Optimum way to use RabbitMQ to have better throughput

[Amit Khosla]

Hi Team,

We are currently using (RabbitMQ 3.6.10,  Erlang 19.3.4). 3 node cluster of 16 Core CPU and 64 GB RAM.

We have policy set as HA-all.

We are using pause-minority to handle Network partitioning.

The issue we are facing is RabbitMQ is not able to take much load.

We have total 1600 queues in around 10 vhosts.

We have around 25 queues routed by single routing queue attached to Topic Exchange in a vhost for 25 nodes (one queue.for one node).

We are able to handle load of around 5K (sum of mesg/sec in each queue, Actual publish rate is way too low).

Can we think of increasing horizontal scaling by moving to say HA-3?

Will Pause minority be a valid option if we think of scale RabbitMQ horizontally?

Thanks & Regards
Amit Khosla

[Michal Kuratczyk]

Hi,

Use perf-test (https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/) to see what throughput you can achieve. If it's significantly higher than you currently see, the problem is on the application side.

However:

1. RabbitMQ 3.6 has been out of support for 3 years.

2. Mirrored queues are deprecated. Once you upgrade to a contemporary version, switch to using quorum queues

3. A single quorum queue should easily support ~25000 msg/s

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADGMEYWxY23OzRhe%2Bn%2B5wBMi%2BYF_QDxOzHwoHwqyk%3DfYXrNJEg%40mail.gmail.com.

--

Michał

RabbitMQ team

[Amit Khosla]

Thanks for the reply Michal !!

Just wanted to check with HA-all on 3 nodes, using quorum queues for 2kb messages, will we be able to get 25K msg/sec from same machine size?

I will definitely look forward towards moving to new version. After doing so, will I be able to scale horizontally as well for the same set of queues?

Thanks & Regards

Amit

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0vjPVEtKEZeQTCpT1QdvZMQ0HemY9sC54o40x1PRQg6WA%40mail.gmail.com.

--

[Michal Kuratczyk]

1. Just use perf-test to check. It shouldn't take more than a few minutes to test it against your current cluster and then you can repeat it on the new one, using quorum queues 

2. You are unlikely to need more than 3 nodes for the kind of numbers that you mentioned. Quorum queues don't require HA policies - they just replicate across 3 nodes by default. You can have a 5-node cluster if needed - each quorum queue will still use 3 out of those 5 nodes.

3. Bonus point - make sure you actually need all those queues being replicated. That obviously has a performance impact and we often see users just replicating all queues "just in case" when there are absolutely no business requirements to do so (but of course there can be in your case).

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADGMEYUyDVzpT_BrHgcVR6%2BpqntG2NtVy7nWZhXfjFodLexVSA%40mail.gmail.com.

--

Michał

RabbitMQ team

[Amit Khosla]

Thanks Michal !

I will run the test and share the results.

Thanks & Regards

Amit

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0vnENzq_s9cieu7REML9XxupTVQhHEaBCv9%2BMQOWosKKQ%40mail.gmail.com.

[Amit Khosla]

Hi,

I am trying to install three node cluster of RabbitMQ 3.8.19, but nodes unable to join cluster.

Installed Erlang Vm (erlang-24.0.4-1.el8.x86_64.rpm) and Rabbitmq server(rabbitmq-server-3.8.19-1.el7.noarch.rpm) on 3 different machines of centos 7.9  and were successful in starting the RabbitMQ server but with three different clusters on 3 machines, 

When we tried to cluster these rabbitmq nodes we are facing Erlang distribution failed error, it might be due to Erlang cookie mismatch can anyone help us

Error message :

Error: unable to perform an operation on node 'rabbit@keng03-dev01-ins01-dmq67-app-1627533565-1'. Please see diagnostics information and suggestions below.

The most common reasons for this are:

• Target node is unreachable (e.g. due to hostname resolution, TCP connection, or firewall issues)
• CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
• Target node is not running

In addition to the diagnostics info below:

• See the CLI, clustering, and networking guides on https://rabbitmq.com/documentation.html to learn more
• Consult server logs on node rabbit@keng03-dev01-ins01-dmq67-app-1627533565-1
• If a target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS

attempted to contact: ['rabbit@keng03-dev01-ins01-dmq67-app-1627533565-1']

rabbit@keng03-dev01-ins01-dmq67-app-1627533565-1:

• connected to epmd (port 4369) on keng03-dev01-ins01-dmq67-app-1627533565-1
• epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
• TCP connection succeeded but Erlang distribution failed
• suggestion: check if the Erlang cookie is identical for all server nodes and CLI tools
• suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
• suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
• suggestion: see the CLI, clustering, and networking guides on https://rabbitmq.com/documentation.html to learn more

Current node details:

• node name: 'rabbitmqcli-616-rabbit@keng03-dev01-ins01-dmq67-app-1627533565-2'
• effective user's home directory: /var/lib/rabbitmq
• Erlang cookie hash: AFJEXwyuc44Sp8oYi00SOw== '''

Can someone please help in resolving this issue.

Thanks & Regards

Amit

[Amit Khosla]

Sorry, a correction here....version of erlang is erlang-23.2.1-1.el7.x86_64.rpm. for 24, we were facing challenge in installing it on CentOS.

[Michal Kuratczyk]

The error message contains the common reasons for this failure. You can also find them in https://www.rabbitmq.com/clustering.html

Check the Erlang cookie, DNS resolution, network connectivity, etc.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADGMEYVP_Q4pAOs%3DxKOBmnmq1q1UhOR2bc1%2Bv-TR5s7RBhhuzQ%40mail.gmail.com.

--

Michał

RabbitMQ team