RabbitMQ Cluster and HA Mirrored Queue Throughput

[John Lao]

RabbitMQ Devs - 

I am examining the RabbitMQ Cluster and HA (using mirrored queues) setup.I am playing with 3 nodes (VMs) and trying to see if the throughput will increase when I add nodes to my cluster (see attached results). 

My results are not conclusive. It seems there is no increase in throughput when I add nodes? Is my expectation correct with mirrored queues? I follow the standard setup suggested by below links. 

Anyway I can improve the throughput when adding nodes to the cluster?

http://www.rabbitmq.com/clustering.html 

http://www.rabbitmq.com/ha.html  

  

Thanks,

John

[Michael Klishin]

You can if you understand how the system works:

https://content.pivotal.io/blog/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine

Mirroring means replicating data. It will not increase your throughput. It is not a feature

for scaling out.

We are aware of the fact that when data locality is poor, the current mirroring algorithm

actually can display lower throughput with more nodes. We plan to replace it in a 4.x release.

Data locality between publishers, queue masters and consumers is another important topic

that has been discussed numerous times on the list.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Your charts do not cover to how many nodes you mirror in the 3 node case. Mirroring to all

nodes when there are just 3 is not unreasonable but in general, mirroring to every node is unnecessary

and is the least efficient option possible.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Some earlier discussions on this list:

 * https://groups.google.com/d/msg/rabbitmq-users/51HpGotu_Mo/kxL2d94hAQAJ

 * https://groups.google.com/d/msg/rabbitmq-users/p6CnpiBXf6E/-dvYlG7PBAAJ

[John Lao]

Thanks Michael - I will go thru the links you provided.

Basically we wanted to achieve high availability, that messages are not interrupted for any node failures. But we also wanted to know how we can achieve increase in throughput thru horizontal scaling at the same time.  

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

MK

Staff Software Engineer, Pivotal/RabbitMQ