Sudden & Permanent increase in Memory consumption.

[yagyans...@gmail.com]

Hi. I am using Prometheus version 2.12.0. I am running Alertmanager 0.21.0 in cluster mode. Since, last 9 days, I have observed twice that the memory consumption by Prometheus increased by 10-12% and it remained to the increased value there after. Interesting thing to note here is that both the times the increase came around 11:15/11:30am ish. I noticed that the number of Read IOps on my Prometheus instance also increased at that time to almost 4000. How to go about in finding the cause?
I do not see any increase in number of time series for my Prometheus.

Thanks in advance!

[Ben Kochie]

Prometheus performs compactions at regular intervals. This is likely what generated some IO.

Note, if you're just looking at RSS, this is not going to tell the whole story. Depending on which version of Go you built with, it may not be fast at reclaiming RSS memory.

Look at the go_memstats_alloc_bytes value to see what Go is really using.

I would also recommend upgrading to the latest release. There have been a number of memory use related improvements in the year and a half since 2.12.0

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/27b3be20-cd23-4bde-824c-566d5369517en%40googlegroups.com.

[Yagyansh S. Kumar]

Thanks, Ben. I'll upgrade to a newer Prometheus version and check if the issue still persists.

But I still have one doubt here, I am running this Prometheus instance for almost an year now, but I have noticed this memory increase recently only. First time on 31st Jan and 2nd time on Feb 8. If it really is because of compaction, why didn't he happen before?

[yagyans...@gmail.com]

Also, I had a look at Go memory utilization. I see that Go memory utilization(go_memstats_alloc_bytes) is around 50% of the total memory used by Prometheus(process_resident_memory_bytes)

[Ben Kochie]

RSS is not a true number. It's more of an estimate than anything. The Linux kernel tracking of process memory is fairly lazy. Especially when you get into more complex things like MADV_DONTNEED vs MADV_FREE.

If you really want to get a more realistic number, you can look at PSS, but it's an expensive number for the kernel to calculate. Hence why people look at RSS.

The difference between Go alloc and RSS is mostly memory that Go doesn't need anymore, but the kernel hasn't come around to reclaim it.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b9bb9202-a89d-4961-9d26-e2259f416b25n%40googlegroups.com.