Alert on NaN value

891 views
Skip to first unread message

eem...@gmail.com

unread,
May 25, 2017, 5:37:30 AM5/25/17
to Prometheus Users
HI,

Is there any way to configure an alert firing when some metric has NaN as value?

thanks!

Brian Brazil

unread,
May 25, 2017, 5:44:17 AM5/25/17
to eem...@gmail.com, Prometheus Users
On 25 May 2017 at 10:37, <eem...@gmail.com> wrote:
HI,

Is there any way to configure an alert firing when some metric has NaN as value?

The expression x != x should do the trick, as NaN doesn't equal itself.

This seems like an unusual alert though, how are these NaNs occurring?

--

eem...@gmail.com

unread,
May 25, 2017, 6:26:44 AM5/25/17
to Prometheus Users, eem...@gmail.com
Hi Brian,

Works fine your trick, thanks!.

Im using a custom exporter.
It extracts data from a file created by a testing system, that is running like a cron, every 10 min.
The testing system could produce some specific N/A metrics, or not be able to generate the file for some time.
So, Im exporting to NaN in both cases.

N/A --> NaN
no file < 15 min --> NaN

I also have another metric with the timestamp of the last OK metrics. its useful, but it doesnt cover the first case of N/A metrics

Brian Brazil

unread,
May 25, 2017, 6:29:48 AM5/25/17
to eem...@gmail.com, Prometheus Users
On 25 May 2017 at 11:26, <eem...@gmail.com> wrote:
Hi Brian,

Works fine your trick, thanks!.

Im using a custom exporter.
It extracts data from a file created by a testing system, that is running like a cron, every 10 min.
The testing system could produce some specific N/A metrics, or not be able to generate the file for some time.
So, Im exporting to NaN in both cases.

N/A --> NaN
no file < 15 min --> NaN

I also have another metric with the timestamp of the last OK metrics. its useful, but it doesnt cover the first case of N/A metrics

You should only use NaN when it makes mathematical sense to do so. If you're not able to generate the file, then don't update the file.

Brian
 



El jueves, 25 de mayo de 2017, 11:44:17 (UTC+2), Brian Brazil escribió:
On 25 May 2017 at 10:37, <eem...@gmail.com> wrote:
HI,

Is there any way to configure an alert firing when some metric has NaN as value?

The expression x != x should do the trick, as NaN doesn't equal itself.

This seems like an unusual alert though, how are these NaNs occurring?

--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fe29df42-0eb7-4e37-994a-75e8eaa0ed23%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages