How to handle high allocated_unused memory in Rabbitmq 3.8.3 erlang 22.3

[Xiaoyan Song]

Hello all,

When I observed high allocated_unused memory, how should I to further diagnostic and how to fix it?

rmq1:~$ sudo rabbitmq-diagnostics memory_breakdown
Reporting memory breakdown on node rabbit@rmq1...
allocated_unused: 8.9385 gb (79.3%)
plugins: 0.5488 gb (4.87%)
mgmt_db: 0.541 gb (4.8%)
queue_slave_procs: 0.5244 gb (4.65%)
binary: 0.3715 gb (3.3%)
queue_procs: 0.1815 gb (1.61%)
connection_other: 0.0702 gb (0.62%)
connection_readers: 0.0401 gb (0.36%)
other_system: 0.0322 gb (0.29%)
code: 0.031 gb (0.28%)
connection_channels: 0.0303 gb (0.27%)
mnesia: 0.0156 gb (0.14%)
connection_writers: 0.0145 gb (0.13%)
metrics: 0.0083 gb (0.07%)
other_ets: 0.0067 gb (0.06%)
atom: 0.0016 gb (0.01%)
msg_index: 0.0012 gb (0.01%)
quorum_ets: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
other_proc: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)

rmq1:~$ top
top - 05:10:04 up 295 days, 19:15,  1 user,  load average: 11.85, 10.02, 9.74
Tasks: 206 total,   1 running, 101 sleeping,   0 stopped,   0 zombie
%Cpu(s): 22.7 us,  6.9 sy,  0.0 ni, 67.8 id,  1.5 wa,  0.0 hi,  1.1 si,  0.0 st
KiB Mem : 28759132 total,  5752364 free, 11476472 used, 11530296 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 25488260 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8548 rabbitmq  20   0 15.077g 9.805g   4668 S 618.8 35.7 315618:24 beam.smp
  870 root      20   0       0      0      0 D   6.2  0.0  10943:36 jbd2/sdb1-8
12773 prismad+  20   0   32196   3436   2884 R   6.2  0.0   0:00.01 top
27625 root      20   0       0      0      0 I   6.2  0.0   0:00.36 kworker/5:2-eve

Thanks

xiaoyan

[Xiaoyan Song]

Add rabbitmqctl status for more information

rmq1:~$ sudo rabbitmqctl status
Status of node rabbit@rmq1 ...
Runtime

OS PID: 8548
OS: Linux
Uptime (seconds): 9136324
RabbitMQ version: 3.8.3
Node name: rabbit@rmq1
Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:128] [hipe]
Erlang processes: 16809 used, 1048576 limit
Scheduler run queue: 2
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_top
 * rabbitmq_management
 * rabbitmq_prometheus
 * rabbitmq_web_dispatch
 * rabbitmq_consistent_hash_exchange
 * rabbitmq_management_agent
 * rabbitmq_event_exchange
 * amqp_client
 * cowboy
 * prometheus
 * cowlib

Data directory

Node data directory: /var/rabbitmq/mnesia/rabbit@rmq1

Config files

 * /etc/rabbitmq/rabbitmq.config

Log file(s)

 * /var/log/rabbitmq/rab...@rmq1.log
 * /var/log/rabbitmq/rabbit@rmq1_upgrade.log

Alarms

(none)

Memory

Calculation strategy: rss
Memory high watermark setting: 0.6 of available memory, computed to: 17.6696 gb
allocated_unused: 8.8931 gb (84.06 %)
queue_slave_procs: 0.4594 gb (4.34 %)
binary: 0.3566 gb (3.37 %)
mgmt_db: 0.3195 gb (3.02 %)
plugins: 0.2939 gb (2.78 %)
queue_procs: 0.1542 gb (1.46 %)
connection_other: 0.0394 gb (0.37 %)
other_system: 0.0324 gb (0.31 %)
code: 0.031 gb (0.29 %)
connection_readers: 0.0305 gb (0.29 %)
connection_channels: 0.0284 gb (0.27 %)
mnesia: 0.0157 gb (0.15 %)
metrics: 0.0083 gb (0.08 %)
connection_writers: 0.0081 gb (0.08 %)
other_ets: 0.0064 gb (0.06 %)
atom: 0.0016 gb (0.01 %)
msg_index: 0.0013 gb (0.01 %)
quorum_ets: 0.0 gb (0.0 %)
other_proc: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)

File Descriptors

Total: 1954, limit: 65438
Sockets: 905, limit: 58892

Free Disk Space

Low free disk space watermark: 44.174 gb
Free disk space: 1022.0011 gb

Totals

Connection count: 905
Queue count: 1339
Virtual host count: 2

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
Interface: [::], port: 15672, protocol: https, purpose: HTTP API over TLS (HTTPS)
Interface: [::], port: 15691, protocol: https/prometheus, purpose: Prometheus exporter API over TLS (HTTPS)
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
Interface: [::], port: 15691, protocol: https/prometheus, purpose: Prometheus exporter API over TLS (HTTPS)
Interface: [::], port: 15672, protocol: https, purpose: HTTP API over TLS (HTTPS)