Temporary recording rules

[Cedric Den Haese]

Hi everybody,

Inspired by the following article, I want to compare a metrics value with the value from the previous days at the same time using the offset modifier.

https://medium.com/qubit-engineering/using-seasonality-in-prometheus-alerting-d90e68337a4c

(The syntax used there seems to be outdated, but the concept remains the same.)

To do this, I have set up a recording rule to get the value at the days before and put them in a new metric with the "offset" label, as instructed by the article.

Since I am only interested in the average (and standard deviation) of those values at this time, I can calculate those by using "avg without (offset)" (again from the article).

Since I don't actually need the metric with the offset label, this made me wonder if prometheus supports temporary recording rules.

I am thus looking for a way to set up recording rules that are evaluated each time and can be used for further expressions, but aren't saved for the complete retention period.

Is there a way to do this in prometheus?

[Tristan Colgate]

You can probably get away with using subqueries now.
https://prometheus.io/blog/2019/01/28/subquery-support/

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0e348303-9070-4890-93b3-ae51eab64e70%40googlegroups.com.



--
Tristan Colgate-McFarlane
----
"You can get all your daily vitamins from 52 pints of guiness, and a
glass of milk"

[Weston Greene]

Cedric, would you be willing to paste what you came up with?

Tristan, would you be willing to elaborate?

I have been inspired by your article as well, Tristan, but like Cedric didn't want to clutter my timeline with all the offsets.

This is a working progress. It isn't dynamic and fails if I don't have at least 3 weeks of data:

    (

        abs(

            (

                (

                    (

                        my:metric

                        -

                        avg_over_time(my:metric  offset 7d [3h:])

                    )

                ) - (

                    (

                        avg_over_time (

                            (

                                my:metric

                                -

                                my:metric  offset 7d

                            ) [3h:]

                        ) + avg_over_time (

                            (

                                my:metric  offset 7d

                                -

                                my:metric  offset 14d

                            ) [3h:]

                        )

                    ) / 2

                )

            ) / (

                (

                stddev_over_time (

                    (

                        my:metric

                        -

                        my:metric  offset 7d

                    ) [3h:]

                ) + stddev_over_time (

                    (

                        my:metric  offset 7d

                        -

                        my:metric  offset 14d

                    ) [3h:]

                )

                ) / 2

            )

        ) > 80 < +Inf unless (

        avg_over_time (my:metric  offset 21d [3h:])

        )

    )

On Tuesday, July 23, 2019 at 10:56:09 AM UTC-4, Tristan Colgate wrote:

You can probably get away with using subqueries now.
https://prometheus.io/blog/2019/01/28/subquery-support/

On Tue, 23 Jul 2019 at 14:03, Cedric Den Haese
<cedric....@gmail.com> wrote:
>
> Hi everybody,
>
> Inspired by the following article, I want to compare a metrics value with the value from the previous days at the same time using the offset modifier.
> https://medium.com/qubit-engineering/using-seasonality-in-prometheus-alerting-d90e68337a4c
> (The syntax used there seems to be outdated, but the concept remains the same.)
>
> To do this, I have set up a recording rule to get the value at the days before and put them in a new metric with the "offset" label, as instructed by the article.
> Since I am only interested in the average (and standard deviation) of those values at this time, I can calculate those by using "avg without (offset)" (again from the article).
>
> Since I don't actually need the metric with the offset label, this made me wonder if prometheus supports temporary recording rules.
> I am thus looking for a way to set up recording rules that are evaluated each time and can be used for further expressions, but aren't saved for the complete retention period.
>
> Is there a way to do this in prometheus?
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Cedric Den Haese]

Weston, I ended up keeping the recording rules since they don't seem to take that much extra space anyway.

It also makes long expressions like yours easier to understand, and you can also look at the values separately that are used to calculate the final expression.

In my opinion temporary recording rules would be the ideal solution for this, but apparently there's no support for that yet.

Op maandag 29 juli 2019 21:39:01 UTC+2 schreef Weston Greene:

[Weston Greene]

Thanks for the input! 

How did you determine that they don't take up that much additional space? Isn't that the equivalent of making your storage space `X + (X * N)` (where X is the count of timeseries and N is the number of weeks you include for seasonality) (assuming you account for seasonality for every timeseries you ingest)? 

Like, if I have 10 timeseries, I'd have 80 recording rule results, bringing my total to 90 metrics in storage? That seems astronomical! 

What am I missing?

[Simon Pasquier]

On Wed, Aug 7, 2019 at 1:32 PM Weston Greene <weston...@gmail.com> wrote:
>
> Thanks for the input!
>
> How did you determine that they don't take up that much additional space? Isn't that the equivalent of making your storage space `X + (X * N)` (where X is the count of timeseries and N is the number of weeks you include for seasonality) (assuming you account for seasonality for every timeseries you ingest)?
>
> Like, if I have 10 timeseries, I'd have 80 recording rule results, bringing my total to 90 metrics in storage? That seems astronomical!
>
> What am I missing?

Prometheus compresses very well timeseries that don't change over time.
See https://www.youtube.com/watch?v=b_pEevMAC3I for more details

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/66369ee2-3c65-4587-9ebd-78000c8bbbd1%40googlegroups.com.

[Weston Greene]

Finally got around to watching this. Thank you. I feel more at ease increasing these recording rules.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/66369ee2-3c65-4587-9ebd-78000c8bbbd1%40googlegroups.com.