alerting on time drift

[joh...@digitalis.io]

I am trying to figure our a way of alerting on time drift using Prometheus and AlertManager - just wondering if anyone has come up with an approach for this and would like to share it? I need to check for time being in-sync on my servers and would like to see if there is any good way of doing this via Prom.

Thanks,

Johnny

[Ben Kochie]

The node_exporter has an NTP ping check that does a basic offset check between the node and a remote server.  Of course these checks are far more frequent than a typical NTP client so please don't use a public NTP server.

Then it's as easy as writing an offset alert.

ALERT NodeClockDrift

  IF abs(node_ntp_drift_seconds) * 1000 > 100

  FOR 1m

    LABELS {

    severity = "info",

  }

  ANNOTATIONS {

    summary = "Node clock is drifting",

    description = "The time on the node has drifted {{$value}}ms from the NTP server",

    runbook = "http://some/doc/path",

  }

I also have a script that collects metrics from NTPd, and I'm working on one for chrony (an NTPd replacement) to collect NTP client/sever metrics.  Sometime soon I'll add that to the node exporter's repo for use with the textfile collector.

--

Any views or opinions presented are solely those of the author and do not necessarily represent those of the company. digitalis.io is a trading name of Digitalis.io Ltd. Company Number: 98499457 Registered in England and Wales. Registered Office: Kemp House, 152 City Road, London, EC1V 2NX, United Kingddom

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johnny Miller]

Thanks Ben - thats very helpful. We usually host an internal ntp server and each node prefers that one, so we can use that one to avoid hammering a public NTP server.

Much appreciated!

--

Johnny Miller

Co-Founder & CTO

+44 (0) 7490 960801 | joh...@digitalis.io | https://digitalis.io