Monitoring Unhealthy Nomad Jobs

[hch...@instartlogic.com]

Hi there!

Are there recommended best practices for monitoring unhealthy jobs in Nomad? We're currently deploying jobs across regions and datacenters. After deployment, is there a good way to check that all the services are running correctly and to alert us if not? We currently use Prometheus as our monitoring system.

We've looked at the option of using Consul health checks as many of you do. However, if a task never comes up at all there is no Consul health check. It would also be great is there is a way to monitor for accidental stopping of jobs. For example, if someone accidentally stops a job that's actually needed.

Thanks in advance.

[Matt Veitas]

I don't think there is anything out of the box, but we spent a day writing a number of simple python scripts that we run every 30 seconds to query the Consul and Nomad APIs to get information about jobs and their health and then report these metrics to our monitoring system. So far it's working well.

-Matt

[Shantanu Gadgil]

Hi,
Are these scripts available opensource somewhere???☺️😊

[Matt Veitas]

Not yet, but this is something we (my company) might consider in the future