Alerting rule for gauge metric with new label value

[karthik reddy]

Hello Developers,

Please let me know how to create alerting rule such that, whenever Prometheus scrapes a gauge metric with a new label value from Pushgateway, I need to check that value range and raise an alert if it is out of range.

For example:
I want to alert if file_size>100 for newly added files, id is different and random for each file
file_size{job=”pushgateway”, id=F234} 80 (in GB)
file_size{job=”pushgateway”, id=F129} 40 (in GB)

whenever new file_size(job=”pushgateway”, id=F787} 23 is added to Prometheus, I should be check 23>100? and send an alert mail such that, “file with id F787 size exceeded”.

[Bjoern Rabenstein]

[Redirecting this to prometheus-users@ and bcc'ing
prometheus-developers@ because it is about using Prometheus, not
developing it.]

I think you could craft something with `absent` and `offset` so that
the alert only fires if the corresponding time series wasn't there a
certain amount af time ago.

However, this all smells quite event-driven: Pushing something like an
event to the Pushgateway, then creating a one-shot alert based on that
"event"... Perhaps you are shoehorning Prometheus into something it's
not good at? A Prometheus alert is usually something that keeps firing
for as long as the alerting condition persists. Are files larger than
100GiB suddenly fine once they have been around for a while? (And how
long is that "a while"?)

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in