rabbitmq 3.5.3 more queues or more consumers ?
39 views
Skip to first unread message
Rajneesh Mitharwal
Sep 4, 2015, 9:43:44 AM9/4/15
to rabbitmq-users
Hi,
I recently started using rabbitmq as queue broker to enqueue users generated messages in queues and simple process them in asynchronous way. Since I want to keep maintaining order of messages coming from a user and also deliver them in lowest end to end latency, I had 500 queues and 500 consumer. (20 consumer processes and each having 25 threads). All consumers are single threaded just to maintain order. So total 500 consumer threads. Each Message size is ~1KB on avg. Below ~3K msg/s producing rate mean_90 latency is roughly 3 milli seconds but this latency increases dramatically like on rate of 4K msg/s mean_90 latency ~200 milli seconds. and this keeps increasing as producing rate increases. for stats: on rate of 8K msg/s mean_90 latency is ~ 2 seconds. My prefetch count for each consumer thread is 25 and acking each individual messages.
During my series of test I changed this 1:1 mapping between # of queues and # of consumers. Now I added 1000 consumers ( for each queue there are 2 consumers) for 500 queues. and found that latency decreased to half of earlier.
I found following patter on end to end latency
Producer rate no of queues no of consumers latency
r x x l
r x 2x l/2
r x 4x l/4
So my questions is keeping a lot of queues with single threaded consumer on them is good or having few queues and a lot of consumers is good ?
I recently started using rabbitmq as queue broker to enqueue users generated messages in queues and simple process them in asynchronous way. Since I want to keep maintaining order of messages coming from a user and also deliver them in lowest end to end latency, I had 500 queues and 500 consumer. (20 consumer processes and each having 25 threads). All consumers are single threaded just to maintain order. So total 500 consumer threads. Each Message size is ~1KB on avg. Below ~3K msg/s producing rate mean_90 latency is roughly 3 milli seconds but this latency increases dramatically like on rate of 4K msg/s mean_90 latency ~200 milli seconds. and this keeps increasing as producing rate increases. for stats: on rate of 8K msg/s mean_90 latency is ~ 2 seconds. My prefetch count for each consumer thread is 25 and acking each individual messages.
During my series of test I changed this 1:1 mapping between # of queues and # of consumers. Now I added 1000 consumers ( for each queue there are 2 consumers) for 500 queues. and found that latency decreased to half of earlier.
I found following patter on end to end latency
Producer rate no of queues no of consumers latency
r x x l
r x 2x l/2
r x 4x l/4
So my questions is keeping a lot of queues with single threaded consumer on them is good or having few queues and a lot of consumers is good ?
Michael Klishin
Sep 4, 2015, 9:46:09 AM9/4/15
to rabbitm...@googlegroups.com, Rajneesh Mitharwal
On 4 September 2015 at 16:43:47, Rajneesh Mitharwal (rajneesh...@gmail.com) wrote:
> So my questions is keeping a lot of queues with single threaded
> consumer on them is good or having few queues and a lot of consumers
> is good ?
There seems to be two identical threads started at about the same time.
> So my questions is keeping a lot of queues with single threaded
> consumer on them is good or having few queues and a lot of consumers
> is good ?
Answered in https://groups.google.com/forum/#!topic/rabbitmq-users/_BqqivEZsVQ.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
Reply all
Reply to author
Forward
0 new messages