absent function with regex not working

[Parvez Mulani]

Hi team,

We are trying to setup an alert using absent and regex on label values where

If No data received from any of the 4 sources ( SOURCE-10, SOURCE-11, SOURCE-13, SOURCE-14 ) , alert! Does anyone know the best way to handle this using absent and regex ?

These are the queries we intent to try but not sure if it works as we currently do not have a best way to test this out.

absent(test_metric{deployment='production', persisted='true', test_id=~'(SOURCE-10|SOURCE-11|SOURCE-13|SOURCE-14)'}) == 1

absent(test_metric{deployment='production', persisted='true', test_id=~'(SOURCE-1.*)'}) == 1

If anyone can share an example or any inputs would be great, thanks.

[Brian Candler]

There are several ways to test the expression:

1. The easiest is simply to type the alerting expression as-is into the PromQL browser built into prometheus.  If it returns any non-empty result set, then it's generating an alert.

Note that in this case you don't need the == 1 on the end, because absent returns an empty set when not absent (not a value of zero).  For example, try these:

absent(up{instance=~".*wombat.*"})

absent(up{instance=~".*server.*"})    # where you have a hostname containing 'server'

By switching to graph mode you can see historical alert conditions too, i.e. whether alerts were firing at previous points in time.

2. There is a facility for doing off-line unit tests, where you provide static timeseries data and then test your alerts against it:
https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/

This is good for more formal testing in a CI/CD pipeline, and for regression testing, or testing a rule in multiple scenarios.

Note that in general, it's better *not* to test on the absence/presence of a metric, but to have another metric which gives a success/fail value (like the "up" metric, which shows 1 for a successful scrape and 0 for a failed scrape).  If you control your own metric generation, this is what you should aim for. However, there are some circumstances where you can't control the metrics being generated, so absent() might be necessary.