Hello,
We are not pushing much data through RabbitMQ, so have set a relatively small memory limit of 384 MB. After upgrading to 3.8.3 our containers are always killed on restart.
The following docker-compose file illustrates the issue:
version: '2'
networks:
rabbitnet-1:
rabbitnet-2:
volumes:
rabbitvol-1:
rabbitvol-2:
services:
rabbitmq-3.8.2:
image: rabbitmq:3.8.2-alpine
environment:
RABBITMQ_VM_MEMORY_HIGH_WATERMARK: 200MiB
RABBITMQ_NODENAME: "rabbitmq01@localhost"
RABBITMQ_USE_LONGNAME: "true"
mem_limit: 402653184 # 384M
memswap_limit: 402653184
networks:
- rabbitnet-1
volumes:
- rabbitvol-1:/var/lib/rabbitmq
rabbitmq-3.8.3:
image: rabbitmq:3.8.3-alpine
environment:
RABBITMQ_VM_MEMORY_HIGH_WATERMARK: 200MiB
RABBITMQ_NODENAME: "rabbitmq01@localhost"
RABBITMQ_USE_LONGNAME: "true"
mem_limit: 402653184 # 384M
memswap_limit: 402653184
networks:
- rabbitnet-2
volumes:
- rabbitvol-2:/var/lib/rabbitmq
Starting this works fine.
Stopping and then restarting (maybe one or two times) leads to:
rabbitmq-3.8.3_1 | 2020-03-24 08:35:06.452 [info] <0.320.0> WAL: recovering ["/var/lib/rabbitmq/mnesia/rabbitmq01@localhost/quorum/rabbitmq01@localhost/00000002.wal"]
rabbitmq-alloc_rabbitmq-3.8.3_1 exited with code 0
Here RabbitMQ was killed by the OoM killer.
Looking at the files mentioned:
$ docker-compose exec rabbitmq-3.8.2 ls -lh /var/lib/rabbitmq/mnesia/rabbitmq01@localhost/quorum/rabbitmq01@localhost/
total 20K
-rw-r--r-- 1 rabbitmq rabbitmq 5 Mar 24 08:38 00000001.wal
-rw-r--r-- 1 rabbitmq rabbitmq 5.3K Mar 24 08:38 meta.dets
-rw-r--r-- 1 rabbitmq rabbitmq 5.3K Mar 24 08:38 names.dets
$ docker-compose exec rabbitmq-3.8.3 ls -lh /var/lib/rabbitmq/mnesia/rabbitmq01@localhost/quorum/rabbitmq01@localhost/
total 451M
-rw-r--r-- 1 rabbitmq rabbitmq 451.2M Mar 24 08:38 00000001.wal
-rw-r--r-- 1 rabbitmq rabbitmq 5.3K Mar 24 08:38 meta.dets
-rw-r--r-- 1 rabbitmq rabbitmq 5.3K Mar 24 08:38 names.dets
It's clear that that file cannot be read entirely within the memory allocation.
I did not see anything that could cause this in the release notes and I would not expect such a big change in a minor release. Is there a way to tweak this behavior?
Greetings,
Jeroen