How to config/run RabbitMQ in low-memory environment?

[Jens Troeger]

Context

Over the past two days or so, the Redis store on a AWS EC2 t2.micro instance (1G memory) kept failing to persist its db to disk:

1137:M 28 Nov 2019 15:28:22.628 * 1 changes in 900 seconds. Saving...

1137:M 28 Nov 2019 15:28:23.397 # Can't save in background: fork: Cannot allocate memory

This, according to a SO thread, is/can be caused if there is not enough memory available to duplicate the existing store. So I looked into the memory consumers on that machine.

RabbitMQ

The server runs fairly recent versions of RabbitMQ and Erlang:

RabbitMQ version: 3.8.0

Node name: rabbit@ip-10-0-1-98

Erlang configuration: Erlang/OTP 21 [erts-10.3.5.5] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64] [hipe]

Erlang processes: 330 used, 1048576 limit

Looking at the memory diagnostics for RabbitMQ:

$ sudo rabbitmq-diagnostics memory_breakdown

Reporting memory breakdown on node rabbit@ip-10-0-1-98...

other_proc: 0.0227 gb (26.64%)

code: 0.0224 gb (26.25%)

binary: 0.0148 gb (17.29%)

other_system: 0.0103 gb (12.07%)

allocated_unused: 0.0092 gb (10.82%)

other_ets: 0.0027 gb (3.13%)

atom: 0.0012 gb (1.38%)

queue_procs: 0.0009 gb (1.06%)

connection_other: 0.0003 gb (0.39%)

connection_channels: 0.0002 gb (0.26%)

metrics: 0.0002 gb (0.25%)

connection_readers: 0.0002 gb (0.23%)

mnesia: 0.0001 gb (0.11%)

quorum_ets: 0.0 gb (0.05%)

msg_index: 0.0 gb (0.03%)

connection_writers: 0.0 gb (0.03%)

plugins: 0.0 gb (0.01%)

queue_slave_procs: 0.0 gb (0.0%)

quorum_queue_procs: 0.0 gb (0.0%)

mgmt_db: 0.0 gb (0.0%)

reserved_unallocated: 0.0 gb (0.0%)

which adds up to 0.0848 gb. From status:

Calculation strategy: rss

Memory high watermark setting: 0.4 of available memory, computed to: 0.4155 gb

Next, looking at htop I see the following:

with 81 beam.smp processes, each of which has 1686M vmem footprint. The complete command line is:

/usr/lib/erlang/erts-10.3.5.5/bin/beam.smp -W w -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.0/ebin  -noshell -noinput -s rabbit boot -sname rabbit@ip-10-0-1-98 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rab...@ip-10-0-1-98.log" -rabbit lager_upgrade_file "/var/log/rabbitmq/rabbit@ip-10-0-1-98_upgrade.log" -rabbit feature_flags_file "/var/lib/rabbitmq/mnesia/rabbit@ip-10-0-1-98-feature_flags" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.0/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@ip-10-0-1-98-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@ip-10-0-1-98" -ra data_dir "/var/lib/rabbitmq/mnesia/rabbit@ip-10-0-1-98/quorum" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

Looking at the htop output here, I am puzzled about the large number of processes running and heir memory consumption. That leads me to think that that’s the cause of Redis failing to fork.

Questions

Is the above analysis and conclusion agreeable?

How can I configure RabbitMQ/Erlang to tone down its memory footprint?

Much thanks!
Jens

[Luke Bakken]

Hi Jens,

My guess is that htop is showing threads within the beam.smp process as individual entries in the output, because multiple processes would be attempting to use the same listening ports and RabbitMQ would not be able to start. However, I started RabbitMQ on my OS X and Linux workstations and beam.smp was shown only once in the htop / top output so I can't explain what you are seeing.

You should not run Redis on the same VM as RabbitMQ as they will compete for resources (as you can see).

I think you should start by stopping the RabbitMQ service and ensuring that no beam.smp processes are running. If they remain, then multiple instances of RabbitMQ are running (and I don't know how).

Thanks -
Luke