converting custom sciprt to prometheus

[deln...@gmail.com]

There's a custom script that checks specific tables on different databases. It's quite complex, but as a result it outputs the exit status in case there's a difference in some combinations of table definitions. Besides this it also outputs a description message with details which databases and tables have discrepancies.  I understand that NE text file collector can be used for triggering alert, but I have no idea how to insert description. Point is that exit code is always the same in case of an error, while the output is always unique, and correlated to specific table.

Any ideas are welcome

[Brian Candler]

Descriptions don't really belong in prometheus metrics - they belong in a logging system or a traditional database.  You said "the output is always unique", which makes Prometheus a particularly bad place to store this.

The only place you could put a description is in a label, but in general this only makes sense if you have a small, fixed set of labels:

table_differences{type="missing_rows"} 3

table_differences{type="mismatched_rows"} 5

What you definitely *don't* want is:

table_differences{info="row 17 in table foo does not match row 34 in table bar"} 1  # DON'T DO THIS!

This is because every time you make a new value for a label - or indeed, any metric with a new *combination* of labels - this creates a new timeseries.  You risk cardinality explosion, i.e. huge number of active timeseries, which is very bad for Prometheus.

Have a look at whether Loki meets your needs.  It's designed for storing logs but with prometheus-like labels, so you can quickly cross-reference from logs to metrics and vice versa.