How to increase Queue's throughput?

[Giri Chintala]

Hi, I am running a performance test on RabbitMQ 3.6.10 and trying to find how much throughput i can get from a single Queue with default With default settings

Test procedure

1. one publisher publishing persistent messages and confirms required, and one consumer consuming without any rate limit. $ bin/runjava com.rabbitmq.perf.PerfTest -f persistent -c 10000 -s 5000 -u test
• Throughput  ~7000/sec
• Publisher connection is in flow and Channel is flow, But Queue is running.
  
• Disk Queue length is close to 0/sec
  
2. two publishers publishing persistent messages and confirms required, and two consumers consuming without any rate limit. $ bin/runjava com.rabbitmq.perf.PerfTest -f persistent -c 10000 -s 5000 -u test  -x 2 -y 2
• Unfortunately throughput is bit below ~7000/sec, So i think Queue is the bottleneck here.
  
• Publisher connection is in flow and Channel is flow, But Queue is running.
  
• Disk Queue length is close to 0/sec.
3. Then i tried to run test for  different queues.
   
• $ bin/runjava com.rabbitmq.perf.PerfTest -f persistent -c 10000 -s 5000
• $ bin/runjava com.rabbitmq.perf.PerfTest -f persistent -c 10000 -s 5000
• Throughput is almost doubled and linearly increased up to 25K before hitting the IOPS limits.
  

I tried to increase credit_flow_default_credit to  400,100, msg_store_credit_disc_bound 2000, 500, But didn't get throughput improvement for a single queue.

is there any thing else that i need to modify to improved throughput on a single Queue?


any help on this is greatly appreciated!

Thanks
Giri

[Michael Klishin]

Step 3 is where you've answered your own question. RabbitMQ was designed with the assumption

that multiple queues are used at the same time and a queue is a reasonable unit of concurrency to choose.

See https://groups.google.com/d/msg/rabbitmq-users/p6CnpiBXf6E/-dvYlG7PBAAJ.

There is a parallelism and CPU utilisation section in http://www.rabbitmq.com/queues.html

as well.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Note that during step 2 you actually use 2 queues (one per consumer) but each

of them gets messages from both exchanges IIRC, which is a bit counterintuitive in PerfTest.

But it doesn't really change the conclusion.

Also, we recommend increasing credit_flow values only if you really know

what the setting represents and what are the risks.

In all likelihood it won't have much of an impact in the longer term (e.g. not for the first

few seconds of a PerfTest run).

A much much better idea is to have continuous monitoring of

 * Network bandwidth (ingress in particular but also egress). We often see that developers

   assume that network bandwidth is infinite no matter what the message size is. Then they discover

   they need a 10 GBit/s link to push through message bodies alone.

 * I/O throughput

 * CPU utilisation

Those 3 is the bare minimum you need to monitor if you want to make informed decisions

about throughput in your distributed system (this is applicable beyond RabbitMQ).

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Giri Chintala]

Thanks Michael for detailed information!! 

I forgot that Queue is limited to a single CPU core!!  I will try  rabbitmq-sharding to improve Queue throughput!

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

I'd recommend changing your system's architecture to use more than one queue

but if rabbitmq-sharding is good enough, so be it.

[Giri Chintala]

We have multiple Queues, one for each different type of messages.

Some of the message types are high rate compare to others. 

[Michael Klishin]

You can use N queue per message type (that would take away "total ordering"

but so will rabbitmq-sharding).

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.