Time of day alert

[Aleksandar Ilic]

Hello everyone,

I was wondering if there is any way to set an alert only to be triggered at a specific time of day.

As I saw for alertmanager there is PR open on GitHub but wondering if there is any workaround for this or any other way.

Best Regards

[David Leadbeater]

A simple way is to use the hour() PromQL function, e.g. something like:

up == 1 unless on() hour() < 9 or hour() > 17

To alert about up jobs during “working hours” (for the sake of an example, you should be able to graph this and see there’s no results outside those hours).

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9bb0e0d1-488b-49b0-b005-878d736c5bden%40googlegroups.com.

[Christian Hoffmann]

Hi,

On 11/21/20 2:41 PM, Aleksandar Ilic wrote:
> I was wondering if there is any way to set an alert only to be triggered
> at a specific time of day.
>
> As I saw for alertmanager there is PR open on GitHub but wondering if
> there is any workaround for this or any other way.

What David suggested would be the simplest method, I think. It has a
drawback though: The alert will resolve once the working hours pass.

We use the following pattern instead:

* Create a pseudo-alert InhibitOutOfWorkingHours which fires except
during the relevant working hours. We want this to be localtime-aware,
which is why we don't use hour() but a textfile collector-provided
metric called localtime_hour, etc.

* We use alert relabeling to automatically add the label
inhibited_by=<alertname> to all Inhibit.* alerts (i.e.
{alertname="InhibitOutOfWorkingHours",inhibited_by="InhibitOutOfWorkingHours"})

* We set up a generic inhibition rule which takes source alerts with
alertname="Inhibit.+" and inhibits target alerts with equal inhibited_by
labels.

* The alert(s) which should be inhibited during certain timeframes are
modified to have the appropriate label (e.g.
inhibited_by="InhibitOutOfWorkingHours")


Might sound complicated, but it works fine so far and is supported by
our configuration management logic.

Kind regards,
Christian

[zhengwei y]

It looks like a good idea.But I'm using alertManager to receive all the alerts, including calling the alertManager API directly to create the alerts, and I can't forbid the client not to call for a certain period of time, so I want to set it on alertManager to be silent  

[zhengwei y]

It's looks good, but I want to be able to manage Settings directly on alertManager instead of changing my alert rules