Increasing Preallocated memory

[Marco Lettieri]

Hello,
i'm figuring out why on one of my rabbitmq instance the preallocated memory by runtime increase everyday of 50-60MB.
Attached the screenshot of memory detail on just rebooted machine and 6 days uptime.

[Wesley Peng]

what rabbitmq and erlang version you were using?

regards.

[Marco Lettieri]

RabbitMQ 3.8.9 Erlang 23.1.2 on a Debian 4.9.240-2 virtual machine with 4gb RAM and 4 cores

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/taMFCY0Q9Js/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/019D2EB7-B954-4582-880B-37662B896AC1%40web.de.

[Wesley Peng]

Is there any error log in the unnormal node?

regards

[Marco Lettieri]

if you mean  /var/log/rabbitmq/rab...@rabbit01.log i've checked and there are no errors at all

[Wesley Peng]

any special setup such as DLX, Delayed Message Plugin, backup exchange,  lazy queues?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/131ce4e5-4268-45bf-9d6e-7b4635b21bf5n%40googlegroups.com.

[Marco Lettieri]

none, i have 3 extra topic exchange from default and 386 queue, 9 durable and all other autodelete.
On the 9 durable queues there are about 20/s incoming/get

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/taMFCY0Q9Js/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/20F186EC-5545-4627-9517-5F9088443162%40web.de.

[Marco Lettieri]

about 3 days uptime

[Marco Lettieri]

seems many durable queues make this issue happen.
deleting unused durable queues the quorum table decrease and the memory increase less, but still increase day by day of about 10-20MB
Attached is 7days uptime

[Karl Nilsson]

I see you are using quorum queues, possibly with very low thoughput. Quorum queues will cache the tail of the Raft log in memory until the WAL fills up and and the in-memory entries are flushed to disk. This means quorum queues will consume memory in a saw-tooth pattern (continuous increase until WAL limit is reached after it will drop down). By default the WAL is quite large to deal with high-throughput scenarios but this is configured. Please see: https://www.rabbitmq.com/quorum-queues.html#resource-use for more details.

Cheers

Karl

[Marco Lettieri]

I guess the durable queues are quorum queues, right?

Because they are the only "special" feature I use. Consider that the durable queues have very low traffic on them, 1-2 messages per hour of 20-30 bytes

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/cfd67bb9-fa3c-4f5f-8c44-a6522e7a688fn%40googlegroups.com.

[Karl Nilsson]

ok yeah at that rate it will take a long time to fill the default wal, if you want it to recover memory earlier set a smaller wal size

[Marco Lettieri]

so this doesn't explain the image above... if quorum (durable) queues have low traffic why does the memory increase?
the wal now is at default size

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/fb4585d0-42be-416a-a033-d36bc2cc7e86n%40googlegroups.com.

[Karl Nilsson]

because every message is written to the WAL _and_ kept in memory, messages are only deleted from the log of operations when the WAL fills up even if they have been removed from the queue. Setting a smaller WAL size will reclaim resources more frequently.

[Marco Lettieri]

But the documentation said # if quorum queues are not used, configure a lower max WAL segment

https://www.rabbitmq.com/mqtt.html#consensus

So is it suggested in my case configure a lower WAL?

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/8a12d547-13aa-4b6c-ac57-56919840ad9an%40googlegroups.com.

[Karl Nilsson]

From what you've told me you are asking about why the memory increases over time - I explained that it is due to how quorum queues work internally (memory is only reclaimed when the wal file fills up). As your throughput is very low you can safely set a lower WAL size to have the quorum queue related memory reclaimed more often (as the WAL will fill up earlier). Else you can just want for it to happen automatically when it reaches the default WAL size which is around 500MiB of disk use.

[Marco Lettieri]

sorry, probably something is not clear to me, but if the wal will clear the memory at 512MB doesn't explain why the memory exceeds the memory limit and after 1-2 week the server exceeds high watermark limit.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/98c8851b-aed5-4c97-9122-c16ce573b084n%40googlegroups.com.

[Karl Nilsson]

you never mentioned reaching the memory limit. when you how much does the quorum queue tables memory breakdown use?

[Marco Lettieri]

You're right, i've never mentioned this, sorry.
The quorum table was about 70MB, when just booted is about 20MB, but the big difference was preallocated by runtime, unused of about 800MB

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/5f9de884-1899-46af-aa90-f276a0e19f9cn%40googlegroups.com.

[Karl Nilsson]

the runtime will adjust the pre-allocation based on previous memory use - I'd still try lowering the WAL limit as it may bring peak memory usage down and with it the pre-allocation amount. Possibly.

[Marco Lettieri]

Hello,
i've done as you suggested but the memory quorum queue tables increase anyway and also preallocated by runtime, unused



Something is still not clear to me: the durable queue are automatically quorum? I've created for sample a quorum queue and the feature section make it evident, but in normal durable queue is not quorum.


So if the quorum tables is connected to quorum queues, but i've only some durable, how this table increase memory?

This are part of my queues, and memory used/persistent is really low

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/960d6cce-daea-4a6d-a5af-74619564c935n%40googlegroups.com.

[Karl Nilsson]

Quorum Queues are always durable but you can also create durable classic queues, in fact, this is the default. As I've explained before the quorum tables memory use will increase until the WAL is full then it will drop down. What max WAL size did you configure? Do you use MQTT?

There are a lot of information in the quorum queue manual page: https://www.rabbitmq.com/quorum-queues.html

[Marco Lettieri]

I've setted 256M wal max size, yes i use mqtt plugin

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3c030fe9-ff27-4c64-a860-d708cedcb2a6n%40googlegroups.com.

[Karl Nilsson]

ok I see - the MQTT plugin also uses Raft which currently means some memory used by the plugin shows up as "quorum tables" - see https://www.rabbitmq.com/mqtt.html#consensus

[Marco Lettieri]

yes, i've read that and i've setted in config
raft.wal_max_size_bytes = 67108864

still not explain why quorum and unused increase without limit

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/9d43e4a8-46aa-429f-8b92-793061ec5d26n%40googlegroups.com.

[Marco Lettieri]

Now is 4 days uptime and is 45MB quorum table, what will happen on 67MB (wal max size)?

[Karl Nilsson]

When the WAL file size reaches 67MB it will flush all in memory log entries ("quorum tables") to disk and remove them from memory. 

One thing to note is that the memory use and the WAL file use are only loosely correlated. If you have many small log entries it may use more memory than on disk due to varying per-disk overhead and how the erlang runtime manages memory.

[Marco Lettieri]

so what is the best way to have many mqtt connections and don't face up this problem?
When the wal is reached the quorum table are removed from memory but connections and queues still alive, right?
I don't understand why technically the mqtt queues or the mqtt plugin generate this quorum increasing memory usage

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/2b38509f-7f25-440d-aad1-da565ef2e5ecn%40googlegroups.com.

[Karl Nilsson]

It is explained in the documentation link I sent you a few messages ago: https://www.rabbitmq.com/mqtt.html#consensus

[Marco Lettieri]

Hello Karl,
we're about at 10 days uptime and i see quorum table increasing anyway, also with raft.wal_max_size_bytes = 67108864

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ed0a615d-09ef-4daf-adb2-006de968f78en%40googlegroups.com.

[Marco Lettieri]

After changed the parameter wal i still see memory increasing (slower of course).
Uptime 89 days, 1GB memory used. I know if i reboot it will return to around 400MB