How to set up alerting when I have no data points.

[Hayk Davtyan]

Hello!
How can I set up an alert expression in case I don't have any data points.

I want to write the correct alert expression about processes for multiple instances and dimension, afterwards, send an alert if the state of the process is 'not running'.

e.g.  netdata_apps_processes_processes_average{instance=~"myserver1|my-server2|my-serverN", dimension=~'nginx|mongod|redis-server'} this expression returns no data when the process is stopped.

Using these <<absent()>>  or <<OR on() vector(0)>>  functions/operators also aren't useful for multiple instances and dimensions, because they don't return labels, hence I can not define which instance and dimension has a problem.

[Brian Brazil]

You have to tell Prometheus the labels you'd like to return one way or the other, as we can't pull them from thin air. It sounds like you want https://www.robustperception.io/absent-alerting-for-scraped-metrics

--

Brian Brazil

www.robustperception.io

[Hayk Davtyan]

But I have several hundred instances/servers and processes. How can I get (instance, dimension) labels of stopped processes, using RegEx. Absent works only one instance/dimension.

[Julien Pivotto]

On 22 Jan 05:00, Hayk Davtyan wrote:
> But I have several hundred instances/servers and processes. How can I get
> (instance, dimension) labels of stopped processes, using RegEx. Absent
> works only one instance/dimension.

You can try:

up == 0


or, it up stays 1:

netdata_apps_processes_processes_average offset 1h unless
netdata_apps_processes_processes_average


In which case you will get for 1h a very precise alert.

>
> On Wednesday, January 22, 2020 at 4:35:51 PM UTC+4, Brian Brazil wrote:
> >
> > On Wed, 22 Jan 2020 at 12:28, Hayk Davtyan <hayk...@gmail.com
> > <javascript:>> wrote:
> >
> >> Hello!
> >> How can I set up an alert expression in case I don't have any data points.
> >> I want to write the correct alert expression about processes for multiple
> >> instances and dimension, afterwards, send an alert if the state of the
> >> process is 'not running'.

> >> e.g. *netdata_apps_processes_processes_average{instance=~"myserver1|my-server2|my-serverN",
> >> dimension=~'nginx|mongod|redis-server'}* this expression returns no data

> >> when the process is stopped.

> >> Using these *<<absent()>> * or *<<OR on() vector(0)>>*

> >> functions/operators also aren't useful for multiple instances and
> >> dimensions, because they don't return labels, hence I can not define which
> >> instance and dimension has a problem.
> >>
> >
> > You have to tell Prometheus the labels you'd like to return one way or the
> > other, as we can't pull them from thin air. It sounds like you want
> > https://www.robustperception.io/absent-alerting-for-scraped-metrics
> >
> > --
> > Brian Brazil
> > www.robustperception.io
> >
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f041f9e-d801-413d-9eff-8e91a65520a5%40googlegroups.com.


--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu