Memory fragmentation after RabbitMQ Server run several days

[zhenbo zeng]

Hi 

There are memory fragmentation at some of my RabbitMQ use case, the memory from `top` showed is much bigger than that from erlang:memory(total), and the memory detail at 3.7.x show that preallocated memory is several GB.

I run `recon_alloc:fragmentaion(current)` and `erlang:system_info({allocator, binary_alloc})` and find that the `mbcs_usage` from all instances are lower than 5%.

Is there anyone has experience to tune the VM arguments, or any solution to solve the VM memory fragmentation.

I can reproduce the problem in test environment

3 machines, 1 MQ server machine, 1 consumer machine, 1 producer machine

producers send different size of messages, from 100 Bytes to more then 10 KBytes

Ror example:

Run consumer script:

bin/runjava com.rabbitmq.perf.PerfTest -h amqp://admin:ad...@172.19.36.20  --auto-delete false --producers 0 --consumers 10 -e e1 --routing-key rk1 --queue q1 --use-millis -a

bin/runjava com.rabbitmq.perf.PerfTest -h amqp://admin:ad...@172.19.36.20  --auto-delete false --producers 0 --consumers 10 -e e1 --routing-key rk1 --queue q2 --use-millis -a

Run each produce scripts 5 minutes, 

bin/runjava com.rabbitmq.perf.PerfTest -h amqp://admin:ad...@172.19.36.20  --producers 2000 -consumers 0 --predeclared -e e1 --routing-key rk1 --queue q1 --use-millis --size 137 --rate 500 --auto-delete false 

bin/runjava com.rabbitmq.perf.PerfTest -h amqp://admin:ad...@172.19.36.20  --producers 2000 -consumers 0 --predeclared -e e1 --routing-key rk1 --queue q1 --use-millis --size 7237 --rate 500 --auto-delete false

bin/runjava com.rabbitmq.perf.PerfTest -h amqp://admin:ad...@172.19.36.20  --producers 2000 -consumers 0 --predeclared -e e1 --routing-key rk1 --queue q1 --use-millis --size 11137 --rate 500 --auto-delete false 

Maybe need to do garbage_collect manually: rabbitmqctl eval "[erlang:garbage_collect(P)||P<-erlang:processes()]."

After running several produce scripts, there are 2.2G preallocated memory.

Run `recon_alloc:fragmentaion(current)` and `erlang:system_info({allocator, binary_alloc})` to get memory allocation information, the attachments are the output of these commands.

[Michael Klishin]

There is no shortage of allocator flags to try. Some of them have been discussed on this list before, others are covered in an Erlang runtime engineer talk available on YouTube (also linked to multiple times on this list).

If you have a way to reproduce, you are 50% there. Tweak the flags and see their effect. Rinse and repeat.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<binary_alloc.txt>

<fragmentation.txt>