RabbitMQ and Swap for Paging

[alif...@workxpress.com]

I'm running a RabbitMQ HA cluster in GKE and have a couple of questions that I haven't been able to get answered elsewhere. We've recently switched over to use RabbitMQ as queue for our entire worker communication methodology in Kubernetes-run microservices. I had been managing the cluster for over a year without there being much load but now that we're using it so heavily we've hit the high-watermark for memory a few times and I need to know how to ensure the best performance.

Since we're running in GKE we don't have the ability to setup any lasting swap space on the servers, and I wasn't sure how this would affect the queue paging once the high-watermark is hit. While I saw it's recommended in the "Production Checklist" on Rabbit's website, I haven't been able to find whether it's necessary and what the implications are for not having any setup. A ServerFault post pointed me here by saying "The lore from the RabbitMQ mailing list is to not set the vm_memory_high_watermark to more than 50% as garbage collection can double the usage. And swap is considered evil as it can induce notable delays." I haven't found any post here that indicates the statement about swap being evil is true, so my first question is whether this is the case.

My second question is whether brokers can reconnect once the work in memory is paged out to disk and how we would notify the worker that they need to retry? We use all NodeJS to connect and publish/read to/from the queues.

Thanks!

[Martin Schröder]

2018-04-24 16:34 GMT+02:00 <alif...@workxpress.com>:
> evil as it can induce notable delays." I haven't found any post here that
> indicates the statement about swap being evil is true, so my first question
> is whether this is the case.

The comment is mine and it seems I was wrong:
https://www.rabbitmq.com/memory.html
states "It is strongly recommended that OS swap or page files are enabled."

So yes, swap is allowed but of course it's not a permanent replacement
for more RAM. :-)

Best
Martin

[alif...@workxpress.com]

Thanks, I definitely agree that RAM > swap and that's sorta the approach we're going about now since I don't have much of a choice. I'm hoping in my use case that we can page to disk without having swap available. I don't know the mechanics behind how RabbitMQ handles that, or whether it's reliant upon the Erlang environment and/or linux to handle swapping. 

Any additional info would be greatly appreciated.

-alex

[alif...@workxpress.com]

Does anyone else have any input that may help answer my question about paging without swap?

[Michael Klishin]

RabbitMQ does not use swap directly. The OS does, and therefore all the knobs you have

available are outside of RabbitMQ.

If you cap max queue lengths [1] and have a reasonably low (even default) paging watermark ratio,

running without swap doesn't sound like a crazy idea to me. However, only real world workload monitoring

can tell how often your nodes are actually swapped out.

Generally speaking it is a lot more common to see OOM events on RabbitMQ nodes than swapping, on this

list and what I see elsewhere.

HTH.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ