Prometheus v2.24 Memory Leak

[Shubham Shrivastav]

I've deployed Prometheus using helm chart https://github.com/prometheus-community/helm-charts

The RAM utilization keeps on increasing slowly over a couple of days.

TSDB Status

Head Stats

Number of Series 111147

Number of Chunks 123102

2021-07-28T02:00:00.080Z (1627437600080)
2021-07-28T03:00:29.028Z (1627441229028)

NameCount
apiserver_request_duration_seconds_bucket
15808
etcd_request_duration_seconds_bucket
4344
container_tasks_state
2330
apiserver_response_sizes_bucket
2168
container_memory_failures_total
1864
promtail_log_entries_bytes_bucket
1800
kubelet_runtime_operations_duration_seconds_bucket
1272
kube_pod_container_status_waiting_reason
1239
storage_operation_duration_seconds_bucket
1218
kube_pod_container_status_last_terminated_reason
1062

I'm not sure what the problem is. 
Help appreciated

TIA

Shubham 

[Ben Kochie]

What is the graph for go_memstats_alloc_bytes{job="prometheus"}?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bd288ed2-b3bc-4c33-b41e-50a6142878a5n%40googlegroups.com.

[Shubham Shrivastav]

Here it is for a period of a week
Dont see much of a jump here

[Ben Kochie]

Sounds like this is page cache use, not actual use. That is not a memory leak, just the kernel doing normal memory management.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a245a9cc-9ba2-402c-9ee2-9dad65868e3fn%40googlegroups.com.

[Shubham Shrivastav]

Then why does RAM consumption keep on increasing?

[Shubham Shrivastav]

Update: 

Kubectl top gives me 
CPU : 141m

Memory: 292Mi

Is the graph wrong?

[Ben Kochie]

It's not wrong, it just doesn't matter. Cached data sits in memory until the memory is needed for other work.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c2ea8e36-6676-44b7-8640-102dcf018eaen%40googlegroups.com.

[Shubham Shrivastav]

Not sure I understand. 

“Cached data sits in memory” > you mean Prometheus will keep on consuming more RAM until it runs out of it?

[Ben Kochie]

No, the memory is in the Prometheus process, it's part of the kernel page cache. It's "assigned" to the container, which is why it looks like the container is using more memory.

But it's page cache, so it doesn't matter. Please read some Linux documentation on how memory management works.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4aa5d40f-a21d-43ec-be30-6d77fc06ad4fn%40googlegroups.com.