Requesting Feedback: promTEd - A scheduling daemon to execute monitoring scripts
Jan Bergner
Hello everyone,
this is the first time I share some of my work with a community, but in this case, I have been encouraged to do so by quite some people, so here it is.
I always felt that the text-file-way to get custom metrics from own scripts into Prometheus was not as scalable as I wanted it. (I am mostly on Debian Linux.)
Of course, I can create a cronjob or systemd timer for each script that generates useful information and have the output redirected into a text file that is digestible by node-exporter, but if an environment requires a lot of scripts, it gets messy to keep track.
That is why I wrote promTEd (prometheus textfile exporter daemon) - a scheduling script that can execute everything in a particular directory according to a timer prefix. The script can be found here:
Essentially, I have a directory with executables like "5_scriptname.sh" and the scheduling script will interpret the prefix as number of minutes between executions.
It will continuously iterate over all the files in the directory,
check if a re-execution is due based on the age of the
corresponding text file and if that is the case, it will execute
the script and write the output to the proper file.
Notice, that this is not an exact scheduler like cron, but
rather a "good enough"-scheduler. (Like, if more than five
minutes have passed, re-execute.)
Also, it will provide some "meta metrics" like runtime of each script and its exit code.
I can picture some additional yet more complex improvements like privilege dropping or handing parameters to scripts via config file, but at this point, it "works for me".
However, I wonder if the Prometheus community finds this approach useful and/or has suggestions. That is why I share it.
Looking forward to everyone's thoughts.
Regards,
Jan