Re: [prometheus-developers] Alerting rule for gauge metric with new label value

[Bjoern Rabenstein]

[Redirecting this to prometheus-users@ and bcc'ing
prometheus-developers@ because it is about using Prometheus, not
developing it.]

On 05.06.21 10:00, karthik reddy wrote:
>
>
> Hello Developers,
>
> Please let me know how to create alerting rule such that, whenever
> Prometheus scrapes a gauge metric with a new label value from Pushgateway,
> I need to check that value range and raise an alert if it is out of range.
>
> For example:
> I want to alert if file_size>100 for newly added files, id is different and
> random for each file
> file_size{job=”pushgateway”, id=F234} 80 (in GB)
> file_size{job=”pushgateway”, id=F129} 40 (in GB)
>
> whenever new file_size(job=”pushgateway”, id=F787} 23 is added to
> Prometheus, I should be check 23>100? and send an alert mail such that,
> “file with id F787 size exceeded”.

I think you could craft something with `absent` and `offset` so that
the alert only fires if the corresponding time series wasn't there a
certain amount af time ago.

However, this all smells quite event-driven: Pushing something like an
event to the Pushgateway, then creating a one-shot alert based on that
"event"... Perhaps you are shoehorning Prometheus into something it's
not good at? A Prometheus alert is usually something that keeps firing
for as long as the alerting condition persists. Are files larger than
100GiB suddenly fine once they have been around for a while? (And how
long is that "a while"?)

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in