Unit testing absent data

[Alex Finkel]

Hey,

I was just wondering if there's a way to write a unit test for an alert that includes missing data?

The context is that I'm trying to figure out if an alert we have is correct if data is missing at certain points in time. The alert is for a sync job and it should fire the job hasn't succeeded in > 24 hours. However, it's been written using a 'for: 24h' clause instead of our usual pattern, which is to include the threshold in the expression, e.g. 'time() - max_over_time(last_success_time[24h])'.

I know that the max_over_time pattern is resilient to missing data at any point during the interval, but I'm wondering in the other case, if the data goes missing for 20 minutes, will that reset the 'for' counter and make the alert useless?

Rather than specifically getting an answer to this question, I'm more interested in knowing how I might be able to write a unit test that demonstrates this problem.

Cheers and thanks in advance for your advice,

Alex

[Ian Billett]

Hey Alex,

If you haven't seen it already, take a look at promtool. It's a swiss army knife for prometheus, which includes functionality to unittest alerts.

I'm not certain if it handles missing data in the way you desire, but would be a good starting point.

Best,

Ian

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4de05e25-6f37-4972-97f8-e77cf1288cc5n%40googlegroups.com.

[Julius Volz]

"pint" from Cloudflare could be interesting to look at: https://groups.google.com/g/prometheus-users/c/WKMYPnDZRG8/m/Pvo9RGd8BgAJ

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4de05e25-6f37-4972-97f8-e77cf1288cc5n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com