how to predict a date in a future on a threshold from the predict_linear function

[Jérôme Loyet]

Hello all,

I have the percentage of disk usage on a metric. I can use predict_linear(disk_usage_percentage[30d], 30*24*60*60) to give me a prediction in 1 month from the past month of metrics. fine

but how could I retrieve the date on which the predict_linear function will reach 80% for instance ? if that's possible :-)

Thank you

regards

++ Jerome

[Brian Candler]

I had the a very similar requirement.  It was a tricky query to build from scratch, but simple when you've worked it out, so I'm happy to share :-)

- name: DiskRate12h
  interval: 1h
  rules:
  # Warn if rate of growth over last 12 hours means filesystem will fill in 7 days
  - alert: DiskFilling7d
    expr: |
        node_filesystem_avail_bytes / (node_filesystem_avail_bytes -
        (predict_linear(node_filesystem_avail_bytes,fstype!~"fuse.*|nfs.*"}[12h], 604800) < 0)) * 604800
    for: 24h
    labels:
      severity: warning
    annotations:
      summary: 'Filesystem will be full in {{ $value | humanizeDuration }} at current 12h growth rate'

I'm using "node_filesystem_avail_bytes" rather than "disk_usage_percentage", but as they both trend down to zero, you should be able to replace it.  Replace the time periods as appropriate.

The logic goes something like this: say V is the variable you're interested in (node_filesystem_avail_bytes in this case)

* we take the current value of V; call it V1

* predict_linear(V[12h], 604800) is the expected value in 7 days time based on the trend over the last 12 hours; call it V2

* filter that with < 0, so we get no value unless it's predicted to be below 0 in 7 days

     ^  V1
     |    \
     |     \
     +--0---x--7----> time
             \
              V2

To find where the cut is on the time axis, you note that V1 is to (V1 + (-V2)) as x is to 7 days.  That is, V1/(V1-V2) is the ratio of the lines V1...x and V1...V2.  And therefore that's also the fraction of 604800 seconds to the zero crossing point x.

Your problem is slightly different: you want to know when the free space percentage will fall below 20, not when it falls below zero.  I'll leave that as an exercise :-)  I think just substituting (disk_space_percentage-20) everywhere in place of the variable is a good starting point, but you have to be careful what happens if the current value is already below 20.

HTH,

Brian.

[Brian Candler]

Spotted typo:

    expr: |
        node_filesystem_avail_bytes / (node_filesystem_avail_bytes -

        (predict_linear(node_filesystem_avail_bytes{fstype!~"fuse.*|nfs.*"}[12h], 604800) < 0)) * 604800

And of course, I was wrong to say that disk_usage_percentage trends down to zero - it will presumably trend up to 100 as the disk fills.