Ratio of WAL size and required memory for Prometheus instance

[Yaron Idan]

I have a prometheus pod running on my cluster requesting 24gb of memory. It runs great most of the time but when it needs to read a large WAL it OOMs and the WAL has to be deleted manually. 

I've saw multiple messages suggesting that the instance just needs more memory in order to replay the WAL, but I wanted to get a better grasp of how the size of WAL relates to the memory the pod needs - if I can find out said ratio, I can probably alert well in advance if the WAL grows too big and needs to be handled with before we get in a crashloop.

Any idea how can I find out this information?

Thanks.

[Ben Kochie]

There is no single correct answer. It depends on your ingestion rate and labels.

Just give the pod more memory so that it does not crash. You do not want to be too tight on memory anyway, as Prometheus needs some non-process memory for page cache in the pod in order to perform queries.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d1b4df82-6be0-4b50-8731-9fe9890e41f0n%40googlegroups.com.

[Yaron Idan]

Thanks for the information, my pod is already requesting 24gb and the host it's running on has 32gb available, so I'm running out of space to maneuver on these machines. 

I can create a dedicated node group for prometheus alone but I'm trying to avoid that if possible since it will make it more of a special creature inside our cluster, and I'm usually try to avoid that whenever possible.

[Ben Kochie]

Without knowing the rest of your configuration, there's not much anyone can do to help.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cf7cc795-217d-4855-b369-6e802283d6e8n%40googlegroups.com.