Understanding prometheus memory usage
266 views
Skip to first unread message
Björn Johansson
Feb 9, 2023, 4:56:48 AM2/9/23
to Prometheus Users
I'm struggling to understand how prometheus uses memory. I've done some reading and I find pieces of information here and there but I don't get the full picture.
Some context first.
We are running prometheus in kubernetes. Not the operator, just vanilla prometheus.
We are monitoring memory usage with the "container_memory_limit_utilization" and "container_memory_used_bytes" metrics.
I've seen in some previous conversation that these metrics are not ideal for monitoring usage since they include the page cache as well. This is where I start getting confused.
Kubernetes doesn't make any distinction if memory is the page cache or the actual memory used by prometheus. The pod will OOM and be evicted once it hits the limit.
Will page cache be released when needed and before the limit is hit?
Can you please help me understand how this works.
How should I monitor prometheus so that I can be sure it has the memory it needs?
Thanks
Björn
Some context first.
We are running prometheus in kubernetes. Not the operator, just vanilla prometheus.
We are monitoring memory usage with the "container_memory_limit_utilization" and "container_memory_used_bytes" metrics.
I've seen in some previous conversation that these metrics are not ideal for monitoring usage since they include the page cache as well. This is where I start getting confused.
Kubernetes doesn't make any distinction if memory is the page cache or the actual memory used by prometheus. The pod will OOM and be evicted once it hits the limit.
Will page cache be released when needed and before the limit is hit?
Can you please help me understand how this works.
How should I monitor prometheus so that I can be sure it has the memory it needs?
Thanks
Björn
Ben Kochie
Feb 9, 2023, 5:03:27 AM2/9/23
to Björn Johansson, Prometheus Users
Kubernetes will not OOM a pod for page cache use. The container_memory_used_bytes includes page cache and is not what you want to compare. What you want to do is look at is container_memory_rss or container_memory_working_set_bytes.
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6428d3f7-2fff-4783-9e4d-13f1a5e442c8n%40googlegroups.com.
Björn Johansson
Feb 9, 2023, 10:40:28 AM2/9/23
to Ben Kochie, Prometheus Users
Ok, gotcha thanks.
Part of the reason I was confused is because Monitoring in gcp has a metric called "container_memory_used_bytes" where you can filter on evictable and non-evictable.
If filtering on non-evictable the value seems to correlate with "container_memory_working_set_bytes".
Just adding that here in case someone else stumbles upon this conversation.
Part of the reason I was confused is because Monitoring in gcp has a metric called "container_memory_used_bytes" where you can filter on evictable and non-evictable.
If filtering on non-evictable the value seems to correlate with "container_memory_working_set_bytes".
Just adding that here in case someone else stumbles upon this conversation.
Björn Johansson
Software Developer
Cell: +46 72 052 58 49
Storytel Sweden AB
Box 24167
104 51 Stockholm
104 51 Stockholm
Visit address: Västra Torggatan 19, 65224 Karlstad
Reply all
Reply to author
Forward
0 new messages