Prometheus correlation between memory usage and timeseries

[Julio Leal]

Hi everyone
I'm trying understand and create an end of life of my prometheus instance.
I think that my promethes will die as my number of timeseries increases and I need more memory ram.
How can I create a correlation between my timeseries growth and my memory ram growth?

I already try to use:

1. container_memory_working_set_bytes with prometheus_tsdb_head_series
   
2. go_memstats_alloc_bytes with prometheus_tsdb_head_series
   
3. go_memstats_heap_inuse_bytes with prometheus_tsdb_head_series
   
4. process_resident_memory_bytes with prometheus_tsdb_head_series
   

The combination closest that I can was number 3 that I can a correlation of 0.56

Is there another way to create this correlation between Memory Ram and timeseries to mensure end of life or the growth of the prometheus instance?

[Ben Kochie]

I usually recommend looking at `process_resident_memory_bytes / prometheus_tsdb_head_series`.

The current typical use is around 8KiB per series, mainly due to the indexing of series.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f84d34ab-9c82-45b1-855d-9081e9d46cedn%40googlegroups.com.

[Julio Leal]

Ben, thank you so much for your answer.

My problem is that the prometheus_tsdb_head_series changes a lot over time.

Is there any way to get how much my timeseries is increasing over time (for example, in the last 3 months)?

[Ben Kochie]

That is a different question than you asked.

Time series depends on the jobs you are scraping. This varies greatly from one user to the next.

You can graph avg_over_time(prometheus_tsdb_head_series[$__interval]) in Grafana to see the trend over time.