Calculate Time Until Gauge Reaches Target

[Dave Cadwallader]

Let's say we have two gauges, progress_current and progress_goal.

progress_current is gradually increasing until it eventually meets progress_goal.

How can we write an expression that will tell us the time in seconds until progress_current meets progress_goal?

I have seen a bunch of examples for predicting when a disk will fill, but these all seem to rely on the assumption that a gauge is trending downwards, and we are predicting when it meets zero.  

Such as:

time() - (node_filesystem_free_bytes{mountpoint="/"} /
deriv(node_filesystem_free_bytes{mountpoint="/"}[4h]) < 0)

In this case we want to predict it trending upwards until it reaches some other gauge target.

Thanks!

[Brian Candler]

> I have seen a bunch of examples for predicting when a disk will fill, but these all seem to rely on the assumption that a gauge is trending downwards, and we are predicting when it meets zero. 

Here is a better recipe as a starting point, using predict_linear():

https://groups.google.com/g/prometheus-users/c/PCT4MJjFFgI/m/kVfOW069BQAJ

> How can we write an expression that will tell us the time in seconds until progress_current meets progress_goal?

In principle, I think you should just be able to replace the metric in that expression with (progress_goal - progress_current) and see when that hits zero.  Something like this:

    expr: |
        ((progress_goal - progress_current) > 0) / ((progress_goal - progress_current) -
        ((progress_goal - predict_linear(progress_current{fstype!~"fuse.*|nfs.*"}[12h], 604800)) < 0)) * 604800

(That's syntactically valid, but otherwise untested. Obviously you should remove or change the label selectors on progress_current to suit the metric in question). 

I see +progress_goal and -progress_goal in the denominator so it should be possible to simplify it further, I think to this:

    expr: |
        ((progress_goal - progress_current) > 0) / 
        (( predict_linear(progress_current{fstype!~"fuse.*|nfs.*"}[12h], 604800) - progress_current) > progress_goal) * 604800

The value of this expression is intended to return *the time until the goal is reached*, using linear interpolation over the past 12 hours, to predict where it will be in 7 days. If the goal isn't expected to be reached in 7 days then it will return no value.

I can attempt to justify that expression graphically:

    |           C2 ^
    |          /   |
    |         /    |
  G +.^......x.....|....
    | |     /      D
    | N    /       |
    | |   /        |
    | v C1         v
    |
    +---0d------7d----> time

G is the progress_goal, which I'm assuming is constant

C1 is the value of progress_current now (time = 0d), which is less than G

C2 is the future predicted value of progress_current (at time = 7d), which is greater than G

N (numerator) is G - C1

D (denominator) is  C2 - C1

And geometrically, I think the ratio of the crossover time (0d...x) to the full time (0d...7d) is the same as the ratio of N to D.  Hence (N/D)*7d gives the time to crossover.

Give it a go. Note that if progress_goal is a metric, rather than a constant, then it will need to have exactly the same label set as progress_current; or you will need to add some on(...) or ignoring(...) clauses as appropriate.

[Brian Candler]

Grr, I think I messed up the threshold detection ("> progress_goal") on the denominator.

Take 2:

    expr: |
        ((progress_goal - progress_current) > 0) / 

        (( predict_linear(progress_current{fstype!~"fuse.*|nfs.*"}[12h], 604800) > progress_goal) - progress_current) * 604800

[Brian Candler]

Also remember a separate alert for "progress_current >= progress_goal" if you need to know if the goal has already been met. The expression I gave only gives an expected time to reach the goal, if the goal *hasn't* already been met.