Calculating Kubernetes resource usage vs limits

[julian....@liveramp.com]

Hi,

I'm looking to calculate Kubernetes resource usage vs limits by pod. I'm having the most trouble with CPU resource usage vs limits. Limits is simple, but I'm not sure how to calculate CPU usage on the same scale.

Below are my queries:

CPU limits:
max(kube_pod_container_resource_limits_cpu_cores{namespace="$deployment_namespace",pod=~"^$deployment_name.*"}) by (container)


CPU usage:
sum by (container_name)(rate(container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",container_name!="POD",pod_name=~"^$deployment_name.*"}[1m]))
/
count by (container_name)(container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",container_name!="POD",pod_name=~"^$deployment_name.*"})

The CPU usage query appears to show as a fraction of 1? I'd like something comparable to kube_pod_container_resource_limits_cpu_cores, though.

Here are my memory queries, as well.

Memory limits:
max(kube_pod_container_resource_limits_memory_bytes{namespace="$deployment_namespace",pod=~"^$deployment_name.*"}) by (container)

Memory usage:
avg(container_memory_usage_bytes{namespace="$deployment_namespace",pod_name=~"^$deployment_name.*",container_name!="POD"}) by (container_name)

Julian

[Ben Kochie]

For CPU use, you could simplify it by doing

avg by (container_name)(rate(container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",container_name!="POD",pod_name=~"^$deployment_name.*"}[1m])) 

This should return a float of the number of cores used for a set of pods.  You can then compare that to kube_pod_container_resource_limits_cpu_cores.  I think the only real difficulty here is that the container metrics are labeled with "container_name" and the kube metrics it's just "container".  This is going to make it hard to compare.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6910ebe3-1d41-4431-878c-74c971ff3d4b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[julian....@liveramp.com]

Oh oh I figured it out.

max by (container_name)(sum by (container_name, pod_name)(rate(container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",container_name!="POD",pod_name=~"^$deployment_name.*"}[1m])))

I was googling around and came upon this: https://books.google.com/books?id=aEvZDQAAQBAJ&pg=PA457&lpg=PA457&dq=%22container_cpu_usage_seconds_total%22+%22top%22&source=bl&ots=mrzzkCZ8NQ&sig=bSwB4QGqyNpjbSi51_XVANNr_M0&hl=en&sa=X&ved=0ahUKEwj5joaPiYLVAhXG6IMKHeleACg4ChDoAQghMAA#v=onepage&q=%22container_cpu_usage_seconds_total%22%20%22top%22&f=false

cADvisor records the CPU usage of each of the CPU cores that the container has access to, which is why we have two values returned, one for "cpu00" and one for "cpu01"

On Tuesday, July 11, 2017 at 3:40:55 AM UTC-4, Ben Kochie wrote:

For CPU use, you could simplify it by doing

avg by (container_name)(rate(container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",container_name!="POD",pod_name=~"^$deployment_name.*"}[1m])) 

This should return a float of the number of cores used for a set of pods.  You can then compare that to kube_pod_container_resource_limits_cpu_cores.  I think the only real difficulty here is that the container metrics are labeled with "container_name" and the kube metrics it's just "container".  This is going to make it hard to compare.

On Mon, Jul 10, 2017 at 6:11 PM, <julian....@liveramp.com> wrote:

Hi,

I'm looking to calculate Kubernetes resource usage vs limits by pod. I'm having the most trouble with CPU resource usage vs limits. Limits is simple, but I'm not sure how to calculate CPU usage on the same scale.

Below are my queries:

CPU limits:
max(kube_pod_container_resource_limits_cpu_cores{namespace="$deployment_namespace",pod=~"^$deployment_name.*"}) by (container)


CPU usage:
sum by (container_name)(rate(container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",container_name!="POD",pod_name=~"^$deployment_name.*"}[1m]))
/
count by (container_name)(container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",container_name!="POD",pod_name=~"^$deployment_name.*"})

The CPU usage query appears to show as a fraction of 1? I'd like something comparable to kube_pod_container_resource_limits_cpu_cores, though.

Here are my memory queries, as well.

Memory limits:
max(kube_pod_container_resource_limits_memory_bytes{namespace="$deployment_namespace",pod=~"^$deployment_name.*"}) by (container)

Memory usage:
avg(container_memory_usage_bytes{namespace="$deployment_namespace",pod_name=~"^$deployment_name.*",container_name!="POD"}) by (container_name)

Julian

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.