Add parameters in rules files expression

[inmar tzemach]

Is it possible to add parameters in rules files expression?

Something like:

    expr: (100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > {{ $treshhold}}

Thanks

[Brian Candler]

No, but you can expand the rules files using a preprocessor of your choice.

If you want the threshold to be different for different time series, then see https://www.robustperception.io/using-time-series-as-alert-thresholds

[inmar tzemach]

Thanks!

How can I change the threshold from outside for giving expression while Prometheus running? 

[Brian Candler]

After changing the rules file, you'll need to trigger a prometheus reload, either by hitting a special API endpoint or sending it a SIGHUP. See https://prometheus.io/docs/prometheus/latest/management_api/#reload

(Note: *don't* restart the prometheus process! That's a much slower operation as it has to dump out and reload its WAL and can take several minutes on a busy server)

[sayf.eddi...@gmail.com]

FWIW, we use saltstack based jinja2 templating with a configured systemd reload (ExecReload=/bin/kill -HUP $MAINPID) to update the thresholds, and it works fine