Quorum queues performance scalibility

[Shivanshu Kumar]

Hi Team,
I am using 3 node cluster version of rmq (3.11.10) with quorum queues .
I notices that it achieved 6k rps with 16 vcpu and 16GB ram and all the 3 nodes on the same zone.
When i increased the number of producers (80 -> 100) the publishers started to go in flow control state post 6.43k rps
Any suggestions as to what can cause this ?
PS: The consumers are pretty fast and scale well under high load.

[Michal Kuratczyk]

Hi,

You haven't provided enough information.

Have a look at the resources I shared before.

Try to reproduce using https://perftest.rabbitmq.com/

If you are using perf-test - share the command

At the very least, describe what you are doing - how many queues? What's the message size?

What are your Grafana dashboards showing? What does the observer show?

It's impossible to advise on performance / resolve performance issues without this kind of information.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/13a2ac7f-e39f-48b9-81d2-620445c7f7ffn%40googlegroups.com.

--

Michał

RabbitMQ team

[Michal Kuratczyk]

I'd still highly recommend using perf-test to reproduce the workload, so that you can investigate it in another environment and play with different scenarios.

When you say ~6k "rps", do you mean 6k messages published (coming into RabbitMQ)? If so, given what you described, you actually have 6k*4~=24k msg/s

or more based on the topics published/bindings. If you expect to be able to publish 40k/s that will be multiplied by 4 or more, then most likely quorum queues can't handle this currently unless you have a cluster large enough that this would be spread better. Quorum queues on a given node share a write-ahead log which generally can handle tens of thousands of messages per second. Using streams could be an option though. With streams you could just deliver the messages to 1 stream and 4 consumers could consume them (no need for the fanout).

I'm not sure if you are adding publishers because you have to or because you think it will speed things up. If the latter, it's most likely counter productive - more publishers mean:

* more Erlang processes running on your CPUs (each connection requires some processes and resources in general)

* more work for the queue to send confirmations

Run `rabbitmq-diagnostics observer` and take some screenshots - this will give us an idea of what's going on.

Best,

On Fri, Mar 31, 2023 at 8:09 AM Shivanshu Kumar <shivan...@gmail.com> wrote:

Not using perf test this is production setup wherin a fanout exchange is routing messages to 4 quorum queues and a topic exchange which is further routing to 4 other queues based on pattern matching . The avg message size is of 2Kb
The producers in this case is other rmq via shovels of prefetch 1000 

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/838074d2-797c-4920-904d-f506a7df2541n%40googlegroups.com.

--

Michał

RabbitMQ team