Memory usage discrepancy

[Anoop]

Hi,

I am using Prometheus metric "container_memory_working_set_bytes" to display the memory usage graph in %. However, it is showing a higher value than what I can see when I use the linux command "free -h".

The actual usage using linux command:

Based on this, it is actually taking less than 19%.

Now I am using below query to get the % value for memory usage:

sum by (node) (container_memory_working_set_bytes{id="/",node="test-instance-e1"}) / sum by (node) (machine_memory_bytes{node=" test-instance-e1"}) * 100

But, this is showing 41%, which is not correct. 

Can someone please guide me if there is any mistake in the Prometheus query or if I can usage some other metrics/query to get the current memory usage?

Thank You,

Anoop

[Harkishen Singh]

You don't require to sum by a node when you are already querying with exact labels since I expect the output to be one. 

BTW, try with container_memory_usage_bytes instead of container_memory_working_set_bytes

[Anoop Mohan]

Thanks  Harkishen Singh for your suggestion.

But, when we use container_memory_usage_bytes instead of container_memory_working_set_bytes it is showing above 55%. 

Also referred some of the blogs which were also recommending to use container_memory_working_set_bytes  instead of  container_memory_usage_bytes  to get the accurate result. 

Ref: https://medium.com/faun/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d

Thanks & Regards,

Anoop Mohan

Mob# +91-7293009486

E-mail : anoopmo...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bb710ee9-6cec-432d-9ec0-5703720ea5b1n%40googlegroups.com.

[Anoop Mohan]

HI,

Anyone have any suggestions on this?

Thanks & Regards,

Anoop Mohan

Mob# +91-7293009486

E-mail : anoopmo...@gmail.com

[Harkishen@Timescale]

how about using node:node_memory_utilisation:ratio * 100 ?

[mspr...@us.ibm.com]

Welcome to the club.

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/golang-nuts/LsOYrYc_Occ/LbjLAsL6BwAJ

[Bartłomiej Płotka]

Hey, I wrote about this here https://www.bwplotka.dev/2019/golang-memory-monitoring/ at some point as well.

Unless something changes by the exporter that actually gives your this metric (it's not Prometheus question, Prometheus is merely collecting those from other exporters) container_memory_working_set_bytes used to be the best bet indeed. AND what you see is somehow expected. 

Even "container_memory_working_set_bytes" is not exactly 1:1 to `Total - Available` for node `free -h` as there are so many caches that kernel uses memory for, that there will be some differences. One is that free is just some utility on the container, vs working set are (if we trust cadvisor doing it well) is what cgroup is showing from outside, on the host.  (see: https://github.com/google/cadvisor/issues/638)

What you see it's quite a big difference though. What is an exact value for just container_memory_working_set_bytes{id="/",node="test-instance-e1"}  ?  Don't use `sum`, you might accidentally sum something irrelevant, etc. Since you expect one series, why you sum?

As per Anoop, your findings are quite interesting indeed. I seen similar inconsistencies, but well, we don't have anything else more accurate. The memory strategy is changing every Go and kernel version (and for the good!). The memory bookkeeping of the kernel is so dynamic that those things can happen. Ideally, you always have some room before your memory limit to not worry about those inconsistencies. 

Kind Regards,

Bartek Płotka (@bwplotka)

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3546a8f3-f03c-4a82-8e2e-bd3586398279n%40googlegroups.com.