Quorum Queues are consuming disk space indefinitely

1,096 views
Skip to first unread message

Tobias Sette

unread,
Aug 26, 2020, 7:52:09 PM8/26/20
to rabbitm...@googlegroups.com
Hi. The directory mnesia/*/quorum is growing indefinitely in size on my
cluster nodes, even without any stale messages (ready, unacked) in that
queues. Is that the expected behavior? What can I do to stop it?

root@rabbitmq1:/var/lib/rabbitmq/mnesia/rabbit@rabbitmq1# du -hs *
4.0K    DECISION_TAB.LOG
4.0K    LATEST.LOG
4.0K    cluster_nodes.config
36K     msg_stores
4.0K    nodes_running_at_shutdown
45G     quorum
4.0K    rabbit_durable_exchange.DCD
4.0K    rabbit_durable_exchange.DCL
4.0K    rabbit_durable_queue.DCD
4.0K    rabbit_durable_queue.DCL
4.0K    rabbit_durable_route.DCD
4.0K    rabbit_durable_route.DCL
4.0K    rabbit_runtime_parameters.DCD
4.0K    rabbit_runtime_parameters.DCL
4.0K    rabbit_serial
4.0K    rabbit_topic_permission.DCD
4.0K    rabbit_user.DCD
4.0K    rabbit_user.DCL
4.0K    rabbit_user_permission.DCD
4.0K    rabbit_user_permission.DCL
4.0K    rabbit_vhost.DCD
40K     schema.DAT
4.0K    schema_version
root@rabbitmq1:/var/lib/rabbitmq/mnesia/rabbit@rabbitmq1# du -hs
quorum/rabbit\@rabbitmq1/*
513M    quorum/rabbit@rabbitmq1/00000263.wal
28K     quorum/rabbit@rabbitmq1/2F_CA-524XNFOC6DG2
636K    quorum/rabbit@rabbitmq1/2F_CA-BBQW94TV44R1
56K     quorum/rabbit@rabbitmq1/2F_EVE5PJ2SY960EMK
12G     quorum/rabbit@rabbitmq1/2F_EVEDD724WUPEMGT
24K     quorum/rabbit@rabbitmq1/2F_PROJIDE4YKZPGRZ
33G     quorum/rabbit@rabbitmq1/2F_PROLCFHRY2Y90SY
8.0K    quorum/rabbit@rabbitmq1/meta.dets
8.0K    quorum/rabbit@rabbitmq1/names.dets


Currently, I have a cluster with 6 nodes and 8 queues. 6 are quorum
queues, each one with a leader and 3 followers.

My default setup consists of only 3 nodes (and 2 followers for each
quorum queue). I ended up with 6 nodes because of disk alarms in the
first 3 nodes.

--
Att,

Tobias

"If technology does not liberate all people for the pursuit of higher aspirations in human achievement, then all it's technical potential will be meaningless." - Jacque Fresco

Michael Klishin

unread,
Aug 27, 2020, 11:46:02 AM8/27/20
to rabbitmq-users
What version of RabbitMQ is used? Are there any MQTT clients that may be short lived?

Karl Nilsson

unread,
Aug 27, 2020, 12:54:10 PM8/27/20
to rabbitm...@googlegroups.com

Which RabbitMQ version are you using? Is the average message size very large? Is there a large message backlog?

It is not normal for a queue that is regularly empties to grow indefinitely. 

Cheers
Karl

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/660d6d8c-c0c7-03fe-930e-e6669b1714fd%40tobias.ws.

--
Karl Nilsson

Tobias Sette

unread,
Aug 27, 2020, 8:18:15 PM8/27/20
to rabbitm...@googlegroups.com

I'm using RabbitMQ 3.8.6 and Erlang 23.0.3. There is no usage of MQTT and no backlog in any queue.

Right now I don't have a single message in "ready" state but I still have low disk space. It seems like quorum queues are allocating disk space but not freeing them up.

Karl Nilsson

unread,
Aug 28, 2020, 4:00:48 AM8/28/20
to rabbitm...@googlegroups.com
Quorum queues will use a bit of disk during normal operation which means stale messages will be stored on disk for some time potentially. How large are the messages approximately?


--
Karl Nilsson

Karl Nilsson

unread,
Sep 1, 2020, 4:53:35 AM9/1/20
to rabbitm...@googlegroups.com
I'm now back in the office so would like to look into this a bit further. If possible could you provide me with the approximate message size of the messages in the problematic queues? Perhaps a few screenshots of the management ui queue screen as well?

Cheers
Karl
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Karl Nilsson <kjni...@gmail.com>
Sent: 28 August 2020 9:00 AM
To: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com>
Subject: Re: [rabbitmq-users] Quorum Queues are consuming disk space indefinitely
 

Tobias Sette

unread,
Sep 1, 2020, 1:09:43 PM9/1/20
to rabbitm...@googlegroups.com

Hi. I don't have a precise way to see the average message size, yet. I gathered the information I have here https://drive.google.com/drive/folders/1v-F9xLk6IYC4eioeDZXqOir8FUXrJtig?usp=sharing

Karl Nilsson

unread,
Sep 2, 2020, 3:39:38 AM9/2/20
to rabbitm...@googlegroups.com
It doesn't need to be precise just a ball-park figure.

Anyhow I have a theory - I suspect this is something that stems from queues that don't have many messages at any one time but are never completely emptied. When queues are never emptied quorum queues will truncate the log too infrequently and combined with largish messages could cause excessive disk use.

I will look at improving this - it may just be that we need to tweak an internal default and/or make this configurable.

Cheers
Karl
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Tobias Sette <m...@tobias.ws>
Sent: 01 September 2020 6:09 PM

Tobias Sette

unread,
Sep 3, 2020, 12:04:54 AM9/3/20
to rabbitm...@googlegroups.com

The average message size is something like 100 KiB.


On 08/31 I moved to a new cluster and, so far, the disk usage is not growing indefinitely (see image attached). But the previous cluster was running for more time.

Screenshot from 2020-09-03 00-59-14.png

Tobias Sette

unread,
Dec 23, 2020, 12:18:35 AM12/23/20
to rabbitm...@googlegroups.com

Hello. Just want to inform that the issue still happens in RabbitMQ 3.8.9.

The last time it happened it seemed to be related with the amount of unacked messages caused by dead connections. However, the disk usage was more than 80 GiB, and I'm confident the size of all messages was inferior to 1 GiB.

Karl Nilsson

unread,
Dec 23, 2020, 2:48:53 AM12/23/20
to rabbitm...@googlegroups.com
What do you mean by dead connections causing unacked messages? A connection that is closed will cause unacked messages to be returned to the queue. 

It is right however that a faulty consumer that holds on to unacked messages indefinitely will halt log truncation. 

--
Karl Nilsson
Reply all
Reply to author
Forward
Message has been deleted
0 new messages