Prometheus, Kubernetes and memory leak.

[Olga Chukanova]

Hello!

Using prometheus as monitoring-system for Kubernetes and have a problem with memory usage. Looks like memory leak - it starts from 4GB of memory and increases by time.

Number of collected metrics (max_over_time(prometheus_tsdb_head_series[1d])): 758779

May be someone had same problem, or can suggest what could be the problem?

Thank you in advance!

Graph:

[Ben Kochie]

What is the query for that graph?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAH81ULG3ADVN9MGZ%2BeUif%2B0i1n2i%2B4KEk_PYmt0-qyQwT4fRcw%40mail.gmail.com.

[Olga Chukanova]

Hello!

I'm using this one:

topk(5, sum(container_memory_working_set_bytes{namespace="monitoring", pod=~"prometheus-core.*$", container=~"prometheus.*"}) by (namespace, pod, container))

and the same I can see at Kubernetes-dashboard, so it looks true.

I tried to increase the scrape interval to 20s, but it didn't work. 

My prom instance has remote write settings, here there are:

        queue_config:
          capacity: 30000
          max_samples_per_send: 15000
          max_shards: 10
          min_shards: 1
          batch_send_deadline: 5s
          min_backoff: 30ms
          max_backoff: 100ms

Have no idea where to start looking for the problem.

чт, 2 мар. 2023 г. в 19:34, Ben Kochie <sup...@gmail.com>:

--

Best regards,

O. Chukanova, monitoring engineer.

+7 (968) 760-89-95

[Ben Kochie]

With over 750k series and a remote write configuration, 8-9GiB of memory doesn't seem unusual.