about increasing the vm_memory_high_watermark limit

[jayashree gn]

Hi, 

We have been recently hitting max memory limit with rabbitmq cluster, i have vm_memory_high_watermark set to default 0.4. which on a 64 bit ubuntu server gets me 1.5 GB of memory for rabbitmq. 

When the limit hits above 700 MB the cluster slows down and starts blocking the publishers.

at which point new connections to the server from the client fails. and everything starts to really slow down. 

Two Questions:

1. What can i do to get things back to moving? Right now, i have tried to restart the cluster and cleanup the vhosts that are no longer used. But mostly wait, and eventually rabbitmq server stabilizes

2. What is the safe limit to increases the vm_memory_high_watermark setting? I am weighing options between, increasing the instance size vs increasing the memory limit. 

Is there something i must be cautious about when i increase the vm_memory_high_watermark limit? 

Thank you!

[Michael Klishin]

On 24 March 2015 at 19:13:09, jayashree gn (jaish...@gmail.com) wrote:
> 1. What can i do to get things back to moving? Right now, i have
> tried to restart the cluster and cleanup the vhosts that are no
> longer used. But mostly wait, and eventually rabbitmq server
> stabilizes

Free up RAM. Vhosts do not contribute to RAM use in any substantial way. Messages and message index entries typically use the most memory.

> 2. What is the safe limit to increases the vm_memory_high_watermark
> setting? I am weighing options between, increasing the instance
> size vs increasing the memory limit.
> Is there something i must be cautious about when i increase the
> vm_memory_high_watermark limit?

The OS and filesystem need RAM to operate. On a machine with 3.75 GB of RAM it's reasonable to use
a value around 0.75 (0.7  — 0.8), given that the machine primarily runs a RabbitMQ node.

You can also configure how soon RabbitMQ queues will being moving their contents to disk
by tweaking vm_memory_high_watermark_paging_ratio:

https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.config.example#L188

I'd not set it higher than 0.9 but the exact value is difficult to recommend. if you have more or less constant
ingress message flow, I'd say 0.75 is reasonable.

Finally, OS swappiness (vm.swappiness on Linux) configuration also matters because it may be the OS that
begins moving RAM contents to disk.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ