How to improve the queue speed?

[Feng Ye]

I noticed the lazy queue and test it, which is  a great function, the speed is stable now.
But I found a problem, when I just only publish message to lazy queue, the publish speed can up to 30k/s, the disk write is 30k/s too. After a while I started the consumer, but the  publish and consume speeds down to 15k/s, and the disk read is 15k/s, disk write is 15k/s too. It looks like the performance is limited by disk.
I use iostat to watch the rabbitmq server process, the io wait is %0, which shows that there is no any io pressure on my machine.

How to improve the disk read and disk write speed of the rabbitmq server? And the batch read 16384 can be changed? 

thx!

[Michael Klishin]

On 25 December 2015 at 12:13:36, Feng Ye (jim.y...@gmail.com) wrote:
> How to improve the disk read and disk write speed of the rabbitmq
> server?

Use HiPE. Use more than 1 queue or rabbitmq-sharding (or similar), which trades total ordering
for parallelism.

And, well, use more than 1 node: when data locality is accounted for, you can go to
throughput rates that are close to 2 orders of magnitude higher [1].

1. http://googlecloudplatform.blogspot.ru/2014/06/rabbitmq-on-google-compute-engine.html 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Feng Ye]

Are there some special setting for the cluster on GCE?I can not setup cluster in goolge.

[Michael Klishin]

On 26 December 2015 at 13:20:56, Feng Ye (jim.y...@gmail.com) wrote:
> Are there some special setting for the cluster on GCE?I can not
> setup cluster in goolge.

We are not aware of anything. Everything on http://www.rabbitmq.com/ec2.html,
in particular sections on the hostname resolution requirements (discussed numerous times
on this list), should be applicable to GCE.

Google offers RabbitMQ on GCE as far as I know: 

[shawn li]

On Friday, December 25, 2015 at 10:55:54 PM UTC+8, Michael Klishin wrote:

Use HiPE. Use more than 1 queue or rabbitmq-sharding (or similar), which trades total ordering
for parallelism.

maybe 'batch publish' is another way to improve throughput, which has been discussed before: http://rabbitmq.1065348.n5.nabble.com/Batching-messages-td22852.html

[Feng Ye]

In currently, I've batch the message in my application with 200 message one time. And the speed from 50000/s down to 250/s, the cpu usage and others resource usage down to normal range.

It looks like the broker can't accept too more single request, other wise it will be very slow.

I also expect that the batch can be done by mq broker, but not in our application ourself.

Thx!