What is the Prometheus's memory elimination strategy?

[白洋洋]

The picture below shows the memory that Prometheus used.As we can see,the average memory is about 33G,and that's a lot of memory.Then I use the command "free -h" to check the memory usage.The result  is shown that the  most  memory the prometheus used is cache.Would you tell me What is the Prometheus's memory elimination strategy?Under what circumstances will Prometheus clean up the cache ?Thank you.

[Ben Kochie]

Prometheus only uses what memory it needs. You can see this with process_resident_memory_bytes and go_memstats_alloc_bytes. Cache memory is not part of Prometheus, it is controlled by the kernel.

On Wed, Jul 8, 2020 at 8:47 AM 白洋洋 <alexb...@gmail.com> wrote:

The picture below shows the memory that Prometheus used.As we can see,the average memory is about 33G,and that's a lot of memory.Then I use the command "free -h" to check the memory usage.The result  is shown that the  most  memory the prometheus used is cache.Would you tell me What is the Prometheus's memory elimination strategy?Under what circumstances will Prometheus clean up the cache ?Thank you.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5739771b-3d2a-4d8a-9577-c04d40d7c926o%40googlegroups.com.

[白洋洋]

Ok,thank you .But the expression "container_memory_usage_bytes{container_name="prometheus"}", which is a  metrics of kubernetes-cadvisor, shows the memory that Prometheus used is about 33G,Isn't it a little too high？Is it that most of this memory is cache, and we don't need to worry about it?（The memory of the computer is 48G）

在 2020年7月8日星期三 UTC+8下午2:56:08，Ben Kochie写道：

Prometheus only uses what memory it needs. You can see this with process_resident_memory_bytes and go_memstats_alloc_bytes. Cache memory is not part of Prometheus, it is controlled by the kernel.

On Wed, Jul 8, 2020 at 8:47 AM 白洋洋 <alexb...@gmail.com> wrote:

The picture below shows the memory that Prometheus used.As we can see,the average memory is about 33G,and that's a lot of memory.Then I use the command "free -h" to check the memory usage.The result  is shown that the  most  memory the prometheus used is cache.Would you tell me What is the Prometheus's memory elimination strategy?Under what circumstances will Prometheus clean up the cache ?Thank you.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Stuart Clark]

On 2020-07-08 08:44, 白洋洋 wrote:
> Ok,thank you .But the expression
> "container_memory_usage_bytes{container_name="prometheus"}", which is
> a metrics of kubernetes-cadvisor, shows the memory that Prometheus
> used is about 33G,Isn't it a little too high？Is it that most of this
> memory is cache, and we don't need to worry about it?（The memory of
> the computer is 48G）

From your other checks it suggests most is indeed cache.

As with all databases lots of memory is used to hold values stored on
disk, to speed up queries. As mentioned this cache memory is managed by
the kernel which will use as much as is available (totally free memory
is a waste, so it is better to use it as cache) but throw it away if the
memory is needed for applications.

Prometheus itself does need a reasonable amount of memory to hold
recently scraped data, handle compaction and process queries. This
memory is managed by the application, with the amount needed dependent
on the amount of data scraped (time series & scrape interval) and the
number & complexity of queries.

Prometheus will continue to operate if there isn't much cache memory
available, but performance will be affected at some point - disk access
(even SSD) is generally significantly slower than memory access. Disk IO
usage would increase as well as CPU IO time.

--
Stuart Clark

[白洋洋]

Ok, I see，Thank you very much!

在 2020年7月8日星期三 UTC+8下午3:51:18，Stuart Clark写道：

[Ben Kochie]

Prometheus memory use depends on how much data you're feeding it.

Look at prometheus_tsdb_head_series and rate(prometheus_tsdb_head_samples_appended_total[5m]).

I usually recommend planning to shard your Prometheus by namespace when the head series is over 2 million.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4361317e-718e-4be9-bb06-4b5a21775dd1o%40googlegroups.com.

[白洋洋]

Hi,Ben,many thanks, I used "prometheus_tsdb_head_series" and "prometheus_tsdb_head_chunks" to check my prometheus,and the results are as follows.But I don't think I really understand what it means.For example,how much memory does one serie or chunk take?Would you tell me how to understand the result ?Thank you very much!

在 2020年7月8日星期三 UTC+8下午4:07:00，Ben Kochie写道：

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4361317e-718e-4be9-bb06-4b5a21775dd1o%40googlegroups.com.