How. to deal with outdated series in alerts.

[Mounya A]

Hello all, 

       I have a question - how do we deal with  labels that are no longer there (intentionally stopped) in alerts. Will there be any threshold time to consider it as stale or unwanted. 
       I have configured an alert when rate(requests[1m]) == 0. It is firing alerts for labels, that didn't show up in the past 7 days. I have intentionally stopped and don't want to alert in such conditions. How to deal with this. 
        

      Thanks in advance.

[Matthias Rampke]

Restart the application that produces metrics.

Generally, the client libraries will remember metrics even if they are not being incremented anymore (it cannot know that they won't be again). Restarting clears the "seen" label set in each process.

Side note: this can also bite you the other way – if an endpoint is unexpectedly never being requested after a restart (say, it wasn't hooked up correctly in the code), this alert will not detect it, because generic instrumentation of HTTP calls cannot know which valid paths there *should* be.

In some cases, we alert like this because we want to check if a specific business action has occurred. Instead of relying on the automatic HTTP metrics, we can separately instrument the code with another metric that we control, and where we can "increment by zero" for all possible label combinations on startup.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d7c7ed77-0f69-4a5a-b36a-40c95c669546n%40googlegroups.com.

[Mounya A]

Thank you so much. It helped me.