AlertManager - alerts based on logged messages to the alert log

[Dimuthu de Silva]

Hi,

Is there a way to send alerts on alertmanager based on a non-prometheus based metrics such as reading from a log file? (Sending alerts when a error is logged)

Thanks in advance :)

[Brian Candler]

"mtail" and "grok_exporter" can be used to turn logs into metrics, by counting lines that match a pattern.  You can scrape these with Prometheus and alert on them as normal.

"loki" is a log storage engine which has a LogQL language similar to PromQL, and will soon be gaining the ability to alert on them:

https://www.youtube.com/watch?v=TcmvmqbrDKU (see 29:35 for alerting)

Think carefully about what actual alerts you want to send out and when:

https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit

https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped