Can you provide hardware recommendation for a dedicated RabbitMQ server?

[Marek T]

Our current throughput averages at 100 messages per second with ~100 queues (using MSMQ). I believe this should be no problem for even a basic RabbitMQ deployment. The queues only rarely grow large, but when they do, they might grow to 100K messages (we have small messages <1KiB each).

This is very hard to find on google so I am asking here: What is a reasonable hardware configuration for a dedicated RabbitMQ broker? What should we focus on? RAM? (how much?). Is having SSD storage important? How big? Any general advice for hw configuration?

[Michael Klishin]

 On 17 February 2015 at 19:16:53, Marek T (marek...@gmail.com) wrote:
> Our current throughput averages at 100 messages per second
> with ~100 queues (using MSMQ). I believe this should be no problem
> for even a basic RabbitMQ deployment. The queues only rarely
> grow large, but when they do, they might grow to 100K messages
> (we have small messages <1KiB each).

That's nothing.

> This is very hard to find on google so I am asking here: What is a
> reasonable hardware configuration for a dedicated RabbitMQ
> broker? What should we focus on? RAM? (how much?). Is having SSD
> storage important? How big? Any general advice for hw configuration?

For that configuration you can go with 1 core, 1 GB of RAM on a shitty HDD.
Note that RabbitMQ configuration by default allows it to use 40% of total system RAM.

Growing that to 100% is a very bad idea (the OS and other services need RAM to operate)
but you can go to 60-70% with 2 GB of RAM, and perhaps 50-60% with 1 GB.

As for SSD's, RabbitMQ message store does appends and garbage collection, so it doesn't do
a lot of random I/O but would benefit from an SSD.

See this thread on the old mailing list:
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-November/thread.html#32028

We provide a decent benchmarking tool, feel free to run it against VMs with different
configurations. For your workload, I'd recommend specifying 50-100 consumers. Note that
the tool uses a fanout exchange by default (your workload may vary), but this too is tunable.

https://www.rabbitmq.com/java-tools.html 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Marek T]

Michael,

thank you for your reply.

How would your recommendation change if the hardware needed to handle an additional queue with 10K messages/second which occasionally grows to 10M messages due to consumers (let's assume 10 consumers) not keeping up?

If we want to ensure maximum performance, is it enough to optimize for RAM and just multiply the approximate volume of messges (10Mx500bytes=5GB), taking the 40% watermark into account => 12.5GB of RAM? 

My assumptions is that RabbitMQ benefits from multi-core processors, is there a practical limit when adding any more processors will be unnecessary? (depending on number of queues/consumers?)

Would it be beneficial to separate this huge queue away from the others in a separate broker?

[Michael Klishin]

 On 19 February 2015 at 14:39:25, Marek T (marek...@gmail.com) wrote:
> How would your recommendation change if the hardware needed
> to handle an additional queue with 10K messages/second which
> occasionally grows to 10M messages due to consumers (let's assume
> 10 consumers) not keeping up?

It depends on message size distribution. 10K per second can be handled by a development
laptop fairly easily if message size is < 4K (so, 4 cores, 4-8 GB of RAM, fast HDD should be OK).

There is currently a fixed cost per message out of the box because our default queue index implementation
is RAM-only. I believe it is about 20 bytes per message.

CloudAMQP folks have a LevelDB-based index:
https://github.com/cloudamqp/msg_store_eleveldb_index

Note that it includes native code and may be OS-specific in some ways.

> If we want to ensure maximum performance, is it enough to optimize
> for RAM and just multiply the approximate volume of messges (10Mx500bytes=5GB),
> taking the 40% watermark into account => 12.5GB of RAM?

Yes, plus the queue index entry cost mentioned above. At this total RAM size you can go
up to 70-80% with the limit, 40% sounds unreasonably low to me.

> My assumptions is that RabbitMQ benefits from multi-core processors,
> is there a practical limit when adding any more processors will
> be unnecessary? (depending on number of queues/consumers?)
> Would it be beneficial to separate this huge queue away from the
> others in a separate broker?

The benefit from extra cores will be greatest when you have more queues and connections/channels
going. For 1K queues and 1K connections I'd expect 4 and 8 cores to work about as well.
16 should make a difference but shouldn't be necessary for the described workload.

Again, the typical "please test it with PerfTest with a similar workload and see for yourself" suggestion
applies but your thinking is largely correct.

[Simon MacMullen]

On 19/02/15 12:14, Michael Klishin wrote:
> There is currently a fixed cost per message out of the box because our default queue index implementation
> is RAM-only. I believe it is about 20 bytes per message.

It's about 200 bytes.

Note that in RabbitMQ 3.5.0 you will only need to pay this cost for
large messages - see http://next.rabbitmq.com/persistence-conf.html

> CloudAMQP folks have a LevelDB-based index:
> https://github.com/cloudamqp/msg_store_eleveldb_index
>
> Note that it includes native code and may be OS-specific in some ways.

It's also not very battle-tested as I understand it.

Cheers, Simon

[Simon MacMullen]

On 19/02/15 11:39, Marek T wrote:
> If we want to ensure maximum performance, is it enough to optimize for
> RAM and just multiply the approximate volume of messges
> (10Mx500bytes=5GB), taking the 40% watermark into account => 12.5GB of RAM?

Assuming you want all messages to be in RAM all the time, you can't just
take 500 bytes as the message body and multiply it up. There's also
message metadata from AMQP (e.g message properties), and the queue
maintains message metadata not from AMQP (e.g. internal message ID,
place in queue and so on). You should do your own measurements but it is
typical to use about 1kB per message in addition to the message body,
when the message is fully RAM-resident.

Also, the queue will start to page messages out half way to the
watermark (by default) - see
http://www.rabbitmq.com/memory.html#memsup-paging

Cheers, Simon

[Marek T]

Thank you for the pointers so far.

Can you provide a proven CPU recommendation for running a high performing cluster? From what I could find in older discussions, hyperthreading does not help much (http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-March/018940.html)

What is the ideal CPU to run RabbitMQ?