Prometheus CustomCollector

[Siddhant Gupta]

Hi,

I am new to Prometheus and I am trying to write a prometheus exporter to collect some custom metrics from a linux app. The exporter needs to execute a binary to get the required metrics. I saw some simple examples and I am able to get the exporter working. Now I want to make the exporter ready for shipping and want to design it according to industry standards.

1. I don't understand what is a CustomCollector and if I need to implement one for my use case. I couldn't even find a good document explaining when and where it should be used.

2. I have added a timer loop, which collects the metrics at the defined interval, I am wondering if there is a way to manage this interval from the prometheus server.

https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling

This link says that the exporter should not collect metrics at arbitrary intervals, however it doesn't say how to sync it with the prometheus server.

3. I am writing the exporter in python, is there a standard directory layout/coding practice that I should follow?

I went through a couple of prometheus documentation, but they are at a very high level and does not include implementation details and the blogs that have examples are very basic and does not show how to scale and make it ready to ship.

Any help and directions would be appreciated.

This is the blog I followed to implement my simple exporter: https://trstringer.com/quick-and-easy-prometheus-exporter/

Siddhant

[Brian Candler]

On Thursday, 23 June 2022 at 05:04:42 UTC+1 siddhant...@gmail.com wrote:

2. I have added a timer loop, which collects the metrics at the defined interval, I am wondering if there is a way to manage this interval from the prometheus server.

https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling

This link says that the exporter should not collect metrics at arbitrary intervals, however it doesn't say how to sync it with the prometheus server.

If you need to "do work" to collect metrics, you would do it whenever you receive a scrape.  That's what a Custom Collector is for: it's code which is executed at scrape time, which has whatever special logic you need to generate metrics on demand.

However in the normal case, you are simply setting gauges or incrementing counters in response to external events as part of the application's normal flow.  You would do those gauge or counter updates whenever the events occur.  Then the metrics are being maintained continuously, and so the scrape only needs to return the current values of those metrics.  In that case, no custom collector is required.

The example code you posted has a dummy polling loop to update metrics intermittently.  That's not what you'd do in real life. Rather, you'd replace "run_metrics_loop" with "run_my_application", and insert code to update metrics within the application, at strategic points where it does work (say when it processes a REST API request, or whatever else it does that needs measuring)

 

3. I am writing the exporter in python, is there a standard directory layout/coding practice that I should follow?

Often you're integrating the exporter into an existing application, in which case you just insert it into the layout of that application.

You said you're instrumenting a "linux app": is that linux app also written in Python?  If so, add your exporter to the application.  If not, how are you collecting data from it?

If the app is not in Python, and you can modify the app's source code, but you don't want to integrate an exporter in whatever language the app is written in, then here are some other approaches which don't involve writing a separate exporter:

- the app can send update messages to statsd_exporter, which maintains gauges and counter state

- the app can write metrics to text files, to be picked up by node_exporter textfile collector

If you can't modify the app, then at very worst it will probably emit log messages.  In that case, you can use mtail or grok_exporter to read the log files and generate metrics.  But that's a last resort.

[Siddhant Gupta]

Thank you for the detailed answer, specially on point 2. This clarifies the use case of custom collectors. Based on your answer, I am convinced that a custom collector is what I should be using.

The linux app is immutable but maintains and provides an interface to poll metrics. Its like using top to get the CPU and memory usage. The data exporter will be sitting on the host next to the application and will make calls to the app when the server wants to scrape to collect the metrics from the app.