Correlation between memory used and timeseries

[Julio Leal]

Hi everyone

I'm trying understand and create an end of life of my prometheus instance.

I think that my promethes will die as my number of timeseries increases and I need more memory ram.

How can I create a correlation between my timeseries growth and my memory ram growth?

I already try to use:

1. container_memory_working_set_bytes with prometheus_tsdb_head_series
2. go_memstats_alloc_bytes with prometheus_tsdb_head_series
3. go_memstats_heap_inuse_bytes with prometheus_tsdb_head_series
4. process_resident_memory_bytes with prometheus_tsdb_head_series

The combination closest that I can was number 3 that I can a correlation of 0.56

Is there another way to create this correlation between Memory Ram and timeseries to mensure end of life or the growth of the prometheus instance?

[l.mi...@gmail.com]

This:
go_memstats_alloc_bytes / prometheus_tsdb_head_series

gives you bytes per time series. This number is the base of my capacity planning. It depends on how many labels end up on your metrics but usually it's fairly stable. We usually see around 3-4KB per time series.

If you have read_read or lots of expensive queries this number will include that, which can make it less stable.

process_resident_memory_bytes depends on a few factors.

First GOGC settings, by default it's 100 and so process_resident_memory_bytes might get around 2x the size of Go heap as the result.

Second data on disk and some recent but not yet written to disk time series will be mmapped and that memory will end up in process_resident_memory_bytes, but the OS will manage this so process_resident_memory_bytes might change in ways that are a bit unpredictable.

In general there's a lot of different components that eat memory, usually the biggest one is all the scraped time series, that's why I only focus on

go_memstats_alloc_bytes / prometheus_tsdb_head_series ratio for capacity planning.

If you take your average "bytes per time series", then multiply that by the number of time series you must store, then by 2 (to account for GC and other things eating memory) then you usually get a rough idea how much memory you need.

[Bryan Boreham]

You may be interested in this talk I gave at PromCon recently, showing some detail around where the memory is used: https://youtu.be/vc5LgoiP_CA

Bryan