alert for container absence from cadvisor metrics

[tom.to...@gmail.com]

Hi,
I would like to write an alert checking if a Docker container is running. I am thinking to check for the cont by name now. (Would be better to check for image label IMO but cadvisor doesn't expose those. Maybe in future; or container-exporter).

Would anybody know how to put it from the top of her/his head? What metric to check and how to put it in the expression?

Thanks!

[Julius Volz]

I'm not using Docker/cAdvisor myself, but something like the following maybe?

ALERT ContainerMissing

  IF absent(container_last_seen{name="<container-name>"})

  WITH {}

  SUMMARY "..."

  DESCRIPTION "..."

That would alert if the container_last_seen metric is not present at all for a given container name. If instead you want to alert if an existing last-seen timestamp is growing too old (1 hour, in this example), you could do something like:

ALERT ContainerLastSeenOld

  IF time() - container_last_seen{name="<container-name>"} > 3600

  WITH {}

  SUMMARY "..."

  DESCRIPTION "..."

...or if you want to combine the two in one alert (I guess that makes most sense, as both conditions would indicate that a container is gone / was never seen?):

ALERT ContainerMissing

  IF absent(container_last_seen{name="<container-name>"}) or on (name) (time() - container_last_seen{name="<container-name>"} > 3600)

  WITH {}

  SUMMARY "..."

  DESCRIPTION "..."

Just keep in mind that I haven't tested these at all and just scribbled these down :)

There's also an issue open for including the docker container name and other metadata in cAdvisor: https://github.com/google/cadvisor/issues/688

And there's the container-exporter which already exports more info: https://github.com/docker-infra/container_exporter/blob/master/exporter.go

Cheers,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tom.to...@gmail.com]

Hi Julius, thanks for the answer it was really helpful!