Prometheus pod OOM killed

[hamisa]

Hi there, 
I'm running prometheus 2.17 on my openshift 3.9.43 cluster. 

when the prometheus pod collapse (deleted, evicted ...) the newly created pod start reading the wal segments in addiotion to run prometheus again. 

in that process the prometheus' pod container get into OOM state, and recreated just afterwards which causes CrashLoopBack to the pod. 

is there any solution to this problem? 

thanks in advance.

[Ben Kochie]

You will need to give the instance more memory in order to complete the recovery.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b967175e-9658-4d6e-ac27-bb62a5a2f70bn%40googlegroups.com.

[hamisa]

thanks for the quick reply. 

is there any way for me to disable it, or at least reduce the amount of memory it consumes? 

or for a change reduce the data capacity that being loaded from the wal? 

thanks in advance.

[Ben Kochie]

There have been some small improvements to the WAL, reducing the amount needed for startup. But you will have to upgrade to the latest release. This will not help the current situation you have, as the improvements come after upgrade.

Either way, there is almost never any choice about allocating more memory when you're hitting an OOM. Prometheus always uses the minimum amount of memory it can for operating. If it needs more memory, you need to decrease your metric cardinality or increase the allocated memory. There's no magic fix.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/95931435-aab7-4859-94b1-aee7743841bfn%40googlegroups.com.

[hamisa]

Thanks for the quick & detailed response !