can I dynamically set alert expression and not have predefined or hardcoded rules

[Adarsh Kumar Pandey]

Hie there .. I hope you are good and safe.

From what I have read, I have an understanding that in prometheus we have to mention all the alert generating rules in a rules file before we can use it to generate alerts.

Is there any other possibility to have the alerting rules comming from some other place into the alerting rule file..?

thanks

[Brian Candler]

Yes: write a script to write the new alerting rules to a file. Prometheus can be configured to pick up all the files in a directory, e.g.

rule_files:

  - "rules/*.yml"

Note: you will need to signal (e.g. "killall -HUP prometheus") to pick up rules when they are changed.

[Adarsh Kumar Pandey]

Thanks so much Brian I really needed an insight on this ... so I guess you have implemented this. If its not too much to ask then can I ask for a sample code please.

[Brian Candler]

No I haven't - my alerting rules are implemented by hand.  I don't have to add any new rules when adding new hosts, because the rules apply to all relevant hosts automatically.

These links might be useful:

https://www.robustperception.io/using-time-series-as-alert-thresholds

https://www.robustperception.io/how-to-have-labels-for-machine-roles/

https://www.robustperception.io/exposing-the-software-version-to-prometheus/

[Adarsh Kumar Pandey]

Much appreciate your response..I will look into these links..thanks !

[Adarsh Kumar Pandey]

I went through the links and I could not quite find what I wanted... I want the alerting rule to be created by any application that I'm monitoring or something like that...

I have one more question, what type of expressions I can put in alerting rules I mean only using metrics or a simple boolean expression also works or I need to make a custom metric for that...

[Brian Candler]

Like I said: if you want to create rules dynamically, then write out the rules and drop them into a file which prometheus will pick up.

But I suspect you ought to gain some familiarity with writing alerting rules by hand first.

I have one more question, what type of expressions I can put in alerting rules I mean only using metrics or a simple boolean expression also works or I need to make a custom metric for that...

You put in PromQL expressions - the same expressions which you'd use in the web interface to query prometheus.  The alert will fire when the rule returns a value.

So for example, say you have 100 metrics called "node_filesystem_avail_bytes".  You can write an expression which filters it down, e.g.

node_filesystem_avail_bytes{fstype!~"fuse.*|nfs.*"} < 100000000

If any of those returns a result, then the alert is triggered.  If this query returns no metrics at the current time, then there's no alert.

Documentation:

https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/