Memory leak?

[Jacek S.]

I'm using a Prometheus server from an official docker. Currently v1.3.1

Since few weeks I observe that every week it's crashing - so I need to restart it. From a grafana dashboards looks like machine memory is slowly growing, and process memory reported by prometheus itself is also growing

Server memory:

Process memory:

How can I proceed to gather more details? What could be a possible issue?

Jacek

[Julius Volz]

Hi,

Please take usage questions to the prometheus-users mailing list (http://groups.google.com/forum/#!forum/prometheus-users) in the future.

The thing you're seeing here is that Prometheus by default will keep ~1 million chunks in memory, and its memory usage will grow as long as there are fewer than that many chunks in memory (it's not a leak - it will flatten out eventually, at the configured chunk limit). That usually is equivalent to a couple of gigabytes of RAM, with all the overhead.

Have a look at this for tuning your Prometheus instance to use less RAM:

https://prometheus.io/docs/operating/storage/#memory-usage

Specifically, you'll want to set -storage.local.memory-chunks much lower than the default.

Cheers,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/4f5f0772-8022-4b21-b538-d4acd9c72226%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jacek Suchenia]

Thanks Julius

This list is linked in a README file, I think that you should add a users mailing list too.

Thanks for pointing us to proper documentation

Jacek

2016-11-10 7:35 GMT+01:00 Julius Volz <juliu...@gmail.com>:

Hi,

Please take usage questions to the prometheus-users mailing list (http://groups.google.com/forum/#!forum/prometheus-users) in the future.

The thing you're seeing here is that Prometheus by default will keep ~1 million chunks in memory, and its memory usage will grow as long as there are fewer than that many chunks in memory (it's not a leak - it will flatten out eventually, at the configured chunk limit). That usually is equivalent to a couple of gigabytes of RAM, with all the overhead.

Have a look at this for tuning your Prometheus instance to use less RAM:

https://prometheus.io/docs/operating/storage/#memory-usage

Specifically, you'll want to set -storage.local.memory-chunks much lower than the default.

Cheers,

Julius

On Wed, Nov 9, 2016 at 8:56 PM, Jacek S. <jacek.s...@gmail.com> wrote:

I'm using a Prometheus server from an official docker. Currently v1.3.1

Since few weeks I observe that every week it's crashing - so I need to restart it. From a grafana dashboards looks like machine memory is slowly growing, and process memory reported by prometheus itself is also growing

Server memory:

Process memory:

How can I proceed to gather more details? What could be a possible issue?

Jacek

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/4f5f0772-8022-4b21-b538-d4acd9c72226%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jacek Suchenia
jacek.s...@gmail.com

[Julius Volz]

Thanks for letting me know, I'm suggesting to replace that outdated reference with a link to our community page:

https://github.com/prometheus/prometheus/pull/2179