Prometheus keeps crashing with OOM

[Udo Offermann]

Hello,

We are running Prometheus 2.37.1 on a VMWare instance with 32GB memory. 

Prometheus keeps crashing with OOM when running some simple queries in the web frontend.

Here some key values from the TSDB status page:

Number of Series   49245

Number of Chunks 143382

Number of Label Pairs 3585

What could be the reason for Prometheus crashing all the time?

For example: I just ran java_lang_Memory_HeapMemoryUsage_used a few times and the memory usage went up to 22GB! After executing node_load1, prometheus allocates nearly 29GB. Finally Prometheus crashed after I increased the range two 1 week.

Any help is welcome

Udo

[Udo Offermann]

It is interesting that Prometheus does its work completely unobtrusively during data collection - memory consumption remains constant at 4 GB throughout the day.

Only when I start executing queries, the memory consumption goes through the roof.

[Brian Candler]

Maybe worth trying an update to v2.37.6? I saw one memory-related thing in release notes for v2.37.3; it doesn't sound like what you're seeing, but would be easy enough to rule it out.