Prometheus k8s container eating up all memory and alerting

[Josep Manel Andrés i Moscardó]

Hi All,

I've been trying to understand how prometheus internal memory handling works and how to configure it, but I cannot find anything related to adjusting it in the latest documentation. In previous versions I can see I could specify heap size so it doesn't go beyond. What should I do ? I have the container constantly alarming when it hits 85% of memory consumed.

Thanks.

[Julius Volz]

Hi Josep,

https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion is a good starting point for more recent versions of Prometheus, although I know Brian and others have added more memory optimizations since then.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bae65c4b-6b7c-4893-be6d-4971c6d44b35%40googlegroups.com.

[Julius Volz]

(as for tuning flags in Prometheus 2.x, most of them have gone away, except for data retention flags and whether to compress the write-ahead-log (WAL) or not)

[Josep Manel Andrés i Moscardó]

Ok, thanks a lot! I will take a look this week.

Missatge de Julius Volz <juliu...@gmail.com> del dia dg., 12 d’abr. 2020 a les 19:35:

--

------------------------------------------------
http://www.arruixaqueplou.org
http://www.terradelfoc.blogspot.com
-------------------------------------------------

[Ervin B]

Also make sure that there are no leftover .tmp directories in your Prometheus data directory. When the compaction process fails, these aren't properly cleaned up and remain in memory. This can build up to a considerable amount over time.

# assuming Prometheus data is in /data
$ find /data -type d -name "*.tmp" -exec rm -rf {} \;

On Monday, 13 April 2020 12:18:08 UTC+2, Josep Manel Andrés i Moscardó wrote:

Ok, thanks a lot! I will take a look this week.

Missatge de Julius Volz <juliu...@gmail.com> del dia dg., 12 d’abr. 2020 a les 19:35:

(as for tuning flags in Prometheus 2.x, most of them have gone away, except for data retention flags and whether to compress the write-ahead-log (WAL) or not)

On Sun, Apr 12, 2020 at 7:31 PM Julius Volz <juliu...@gmail.com> wrote:

Hi Josep,

https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion is a good starting point for more recent versions of Prometheus, although I know Brian and others have added more memory optimizations since then.

On Sun, Apr 12, 2020 at 6:58 PM Josep Manel Andrés i Moscardó <tita...@gmail.com> wrote:

Hi All,

I've been trying to understand how prometheus internal memory handling works and how to configure it, but I cannot find anything related to adjusting it in the latest documentation. In previous versions I can see I could specify heap size so it doesn't go beyond. What should I do ? I have the container constantly alarming when it hits 85% of memory consumed.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bae65c4b-6b7c-4893-be6d-4971c6d44b35%40googlegroups.com.

[Ervin B]

PS. Slight clarification: they don't really "stay in memory" as they're not used all the time. If the compaction process continuously or periodically fails (due to OOM for example), those directories can't be processed and they technically do stay in memory.