Expressions about Monitoring Kube Pods Memory Usage

[Lijing Zhang]

Hi my freinds,

I'm trying to monitor pods memory usage. I use dashboard from Grafana.

Former panel use this expression,

sum (container_memory_working_set_bytes{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}) by (pod_name)

and I think there could be another two,

sum (container_memory_rss{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}) by (pod_name)                // rss: Resident Set Size

sum (container_memory_usage_bytes{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}) by (pod_name)

So, what's the different between these 3 metrics? Which one you prefer about monitoring

Thanks in advance.

[Julius Volz]

The cAdvisor internal structure gives some explanations in comments about the different reported memory types:

https://github.com/google/cadvisor/blob/08f0c2397cbca790a4db0f1212cb592cc88f6e26/info/v1/container.go#L338:6

Not sure, but I think mostly I've seen people just use "container_memory_usage_bytes" (which includes all somehow used memory)?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c3d2d16c-1317-4c9a-ae8a-946faf7f83ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Chodúr]

Hi, I actually end up using the "container_memory_rss" in kubernets because this ammount of memory I found out is the one which causes pod to OOM when it hits the requests limit.

It should not contain swap(mostly turned off in kubernetes anyway) and cache memory.

so there is:

"container_memory_cache" containing cached memory

"container_memory_swap" containing swap memory

"container_memory_rss" this should be the resident set sizethat should mean "really allocated memory in RAM"

and the "container_memory_usage_bytes" which imho shoul be summary of those as

container_memory_usage_bytes = container_memory_cache + container_memory_swap + container_memory_rss

I'm not expert on this so I might be terribly wrong in that case I'm sorry :)

As Julius stated mostly I've seen "container_memory_usage_bytes" and the "container_memory_rss"
to be used.  

[Lijing Zhang]

@Martin,

I observed from result,

sum ((container_memory_rss{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}) + (container_memory_cache{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}) - container_memory_usage_bytes{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}) by (pod_name)

equals 0, except when pod_name="kube-proxy". FYI

在 2018年5月18日星期五 UTC+8下午1:07:28，Martin Chodúr写道：

[Martin Chodúr]

That's weird regarding the "kube-proxy".

You are running kube-proxy in pod?
Our kube proxy is running on the host directly so I cannot compare it to our case.

can you post results for the kube-proxy?
container_memory_rss
container_memory_cache
container_memory_cache
container_memory_swap

As I sayed I'm not 100% sure myself so I'd be glad to get this clear.

Dne pátek 18. května 2018 8:38:08 UTC+2 Lijing Zhang napsal(a):

[Lijing Zhang]

@Martin, on my lab there are kube-proxy pods, every node has one.

And this is the result of kube-proxy memory usage, FYI

container_memory_rss    13.8 MiB
container_memory_cache  20 KiB
container_memory_swap   4.38MiB

container_memory_usage_bytes  19.64MiB


rss + cache - usage_bytes = -5.82MiB

在 2018年5月18日星期五 UTC+8下午4:15:00，Martin Chodúr写道：

[Martin Chodúr]

This is suspicious.. the swap should be equal to 0 if you are running it as pod

do you have swap turned off on the hosts?

container_memory_rss + container_memory_cache + container_memory_swap = 18.2

which is.. let's say pretty close to the 19.64 of "usage_bytes" that could be theoretically caused by the setup which I don't know what it looks like.


Dne pátek 18. května 2018 10:40:09 UTC+2 Lijing Zhang napsal(a):

[naveen...@gmail.com]

How to get this detailed memory usage? What command you have used?