Alerting on "cadvisor" Container Memory Limits

[Dave Cadwallader]

Goal:

Alert when a container is close to using its memory limit.

Background:

I'm using the cadvisor agent, which gives me two useful metrics:

• container_memory_usage_bytes : how much memory a container is using
  
• container_spec_memory_limit_bytes : what is the memory limit I've defined for that container

Using these, I can answer the question "what percent of its memory limit has this container used?" using this expression:

(container_memory_usage_bytes / container_spec_memory_limit_bytes) * 100

...and I can alert on this with something like:

(container_memory_usage_bytes / container_spec_memory_limit_bytes) * 100 > 90

...to tell me when the container is using more than 90% of its allocated memory.

Problem:

Everything above works fine, except when a container does not have a memory limit defined.  Then the divisor is 0, and the result is +Inf, meaning that the alert triggers incorrectly since +Inf matches > 90.

Is there a way I can either exclude metrics where container_spec_memory_limit_bytes = 0, or provide a range for the alert such that +Inf values don't trigger it?  Or perhaps I should rewrite this query altogether?

Thanks!

[Julius Volz]

If the limit series exists, but is 0, you can select only those that have a non-zero limit:

  (container_memory_usage_bytes / (container_spec_memory_limit_bytes != 0)) * 100 > 90

That way you'll only get alerts for the ones that have a non-zero limit.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/26954812-4bc9-47a7-bcad-54908dab4874%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.