Prometheus eats up all memory with constant load

[Tim Schwenke]

Is it expected that Prometheus will take all the memory it gets over time? I'm running Prometheus in a very "stable" environment with about 200 containers / targets in addition with Cadvisor and Node Exporter over 3 to 4 nodes. And now after a few weeks of uptime Prometheus reached the memory limits and I get memory limit hit events all the time.

 Here are screenshots of relevant dashboards:

https://github.com/trallnag/random-data/issues/1#issuecomment-716463651

Could be something wrong with my setup or is it all ok? The performance of queries in Grafana etc is not impacted

[l.mi...@gmail.com]

I was recently dealing with some heavily loaded instances that would occasionally get OOM killed.

The first tip is always to look at the metrics that being exported to get an idea of what's the expected memory usage, I found https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion to be pretty accurate.

The maths there mentions "double for GC" so you get a low and high watermark thresholds and you usage should be somewhere between "1x" (the base memory usage of data in Prometheus) and "2x" (if you double that base to account for Go GC). On top of that there will be some memory from queries, recording rules and so on, but what I found was that my real usage was close to 2x (so accounting for GC) so most of that extra memory needed will be hidden in the "double for GC" memory usage.
I did drop a lot of high cardinality metrics to get the memory usage down, mostly from exporters like cadvisor and node_exporter that expose a lot of detailed metrics. metric_relabel_configs in your scrape target configuration can be used to drop individual expensive metrics.

Next thing to know is that metrics churn can cost you a lot - if a target exposes some unique metric for just a few minutes, then it disappear, you will still pay the memory cost of it until the next head compaction. This I found is a bit harder to quantify, but you plot Prometheus memory usage over time you will see memory usage dropping every time after head compaction, how often does that happen can be tweaked with storage.tsdb.max-block-duration flag.
If you do have a lot of churn the extra memory pressure from some operations, like head compaction, can push your instances over the memory limit, so having a low value of storage.tsdb.max-block-duration might help, chances are you already set a custom value for that flag if you're using Thanos (see https://thanos.io/v0.16/components/sidecar.md/). This was mostly problematic on a big Prometheus instances (256GB of RAM) where default Go garbage collection thresholds would make it act a bit too lazy, and leave too much to be reclaimed, so when there was a spike in memory pressure GC would react too slow compared to how fast new allocations are happening, tweaking GOGC environment variables improved that for me.

 After trimming all unnecessary metrics the final thing that help was to:

1. enforce tsdb block duration: --storage.tsdb.max-block-duration=2h --storage.tsdb.min-block-duration=2h is a good start, but we've set it to only 30m on biggest boxes where we see a lot of churn

2. set GOGC=40 - this means you're trading more CPU usage for lower RSS memory usage, a different value might work better for you, I didn't find any benefit of setting it lower (30 would eat more CPU without giving me any memory savings).

[Ben Kochie]

It looks like the head chunks has been growing without compacting. Is the disk full? What's in the logs? What's in the data directory?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f89538c2-c07b-424d-a377-614f0098a337n%40googlegroups.com.

[Tim Schwenke]

Thanks for your answers, @mierzwa  and Ben

@mierzwa, I have checked the robust perception blog post you have linked and it really seems like my Prometheus is using too much memory. I have only around 20k series, but with quite high cardinality over a long period of time. If we just look at a few hours or days the churn should be very low / zero and only increase when ever an API endpoint gets called the first time and so on.

@Ben, the Prometheus is using AWS EFS for storage and so the disk size should be unlimited. It sits currently at just 13 GB while resident size is at around 2 GB and virtual memory around 7 GB. The node Prometheus is running on has 8 GB of RAM. Now that you have mentioned storage, I have read quite some time ago that Prometheus does not officially support NFS and I think the AWS EFS volume is mounted via NFS into the node.

---------------------------------------------------------

Here is the info I get from the Prometheus web interface:

https://trallnag.htmlsave.net/ 

This links to a html document that I saved 

[Tim Schwenke]

I think I will just kick the bucket and move my Promstack to a dedicated single node AWS ECS cluster, this way I can use EBS for persistent storage

[David Karlsen]

We see something similar for recent versions of Prometheus — but something else might cause it. What version are you at?

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/06d6ba09-4558-4cbd-9efd-f6f8bf31d90fn%40googlegroups.com.

--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

[Tim Schwenke]

The currently latest version v2.22.0

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/06d6ba09-4558-4cbd-9efd-f6f8bf31d90fn%40googlegroups.com.

[Bartłomiej Płotka]

Hi,

That's expected with the applications built with the newer Go versions. On Linux, they will "reserve" whatever memory is available in the host, but it's actually might not be allocations. This might be a memory that the kernel can take whenever it needs, so those cases will not trigger OOMs. To see what happens look for working sets metric instead of RSS.

See https://www.bwplotka.dev/2019/golang-memory-monitoring/ for details 🤗

[Tim Schwenke]

@All, I have switched from EFS to EBS (so no more NFS) and the head chunks have stopped growing. So it really seems to have something to do with that even though AWS's EFS NFS is supposed to be completely POSIX compliant. Weird.

@bwpl...@gmail.com, I see, thanks. Your post is also helpful for getting alert rules right

[Julien Pivotto]

On 22 Nov 12:15, Tim Schwenke wrote:
> @All, I have switched from EFS to EBS (so no more NFS) and the head chunks
> have stopped growing. So it really seems to have something to do with that
> even though AWS's EFS NFS is supposed to be completely POSIX compliant.
> Weird.

There might have a large range of issues, like client caching, coming
into play with network storage.

> >>>>>>> <https://groups.google.com/d/msgid/prometheus-users/f89538c2-c07b-424d-a377-614f0098a337n%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>>>>>> .

> >>>>>>>
> >>>>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "Prometheus Users" group.
> >>>>
> >>> To unsubscribe from this group and stop receiving emails from it, send
> >>>> an email to prometheus-use...@googlegroups.com.
> >>>>
> >>> To view this discussion on the web visit

> >>>> https://groups.google.com/d/msgid/prometheus-users/06d6ba09-4558-4cbd-9efd-f6f8bf31d90fn%40googlegroups.com
> >>>> <https://groups.google.com/d/msgid/prometheus-users/06d6ba09-4558-4cbd-9efd-f6f8bf31d90fn%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>>> .

> >>>>
> >>>
> >>>
> >>> --
> >>> --
> >>> David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen
> >>>
> >>>
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bf3431d6-1b0b-47e6-821c-4df379cca8a5n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie