scheduling data collection in a custom exporter

[Cory Robinson]

Hi Prometheus Users,

I am working on writing a (my first) custom exporter that collects metrics from an application's REST API endpoint that I have no control over.

Now that I am half way through it, I am questioning myself whether I am taking the right approach to get the collected metrics published to my /metrics endpoint for Prometheus to scrape, and I could use some guidance.

Let me attempt to explain what I'm doing.

My approach:

Metrics are published on a /metrics endpoint that Prometheus will scrape every 15 minutes. But, how do metrics get to the /metrics endpoint?

My idea was to have my exporter collect data from an external REST API on it's own schedule and publish the metrics to the /metrics endpoint so they would be ready and waiting for Prometheus to scrape them.

This way, Prometheus would be scraping /metrics on it's own scrape_interval = 15m, and my exporter would be collecting data and publishing to /metrics on it's own export_interval = 15m. I might even lag ahead the export_interval by a minute or so, because each of the ~10 API calls it makes could take up ~20 seconds to complete.

Is it necessary to have the 2 different schedules? Or would the Prometheus scrape_interval schedule be expected to trigger the API calls in the exporter?

Basically I don't want to bog down that external applications performance or look like a DOS attack on the external application by continuously barraging it with requests when I only need the data once every 15 minutes or so.

Hopefully I explained this alright, but let me know if you need any clarification.

Thanks,

Cory

[Brian Brazil]

On Fri, 22 May 2020 at 13:52, Cory Robinson <crr...@gmail.com> wrote:

Hi Prometheus Users,

I am working on writing a (my first) custom exporter that collects metrics from an application's REST API endpoint that I have no control over.

Now that I am half way through it, I am questioning myself whether I am taking the right approach to get the collected metrics published to my /metrics endpoint for Prometheus to scrape, and I could use some guidance.

Let me attempt to explain what I'm doing.

My approach:

Metrics are published on a /metrics endpoint that Prometheus will scrape every 15 minutes. But, how do metrics get to the /metrics endpoint?

My idea was to have my exporter collect data from an external REST API on it's own schedule and publish the metrics to the /metrics endpoint so they would be ready and waiting for Prometheus to scrape them.

This way, Prometheus would be scraping /metrics on it's own scrape_interval = 15m, and my exporter would be collecting data and publishing to /metrics on it's own export_interval = 15m. I might even lag ahead the export_interval by a minute or so, because each of the ~10 API calls it makes could take up ~20 seconds to complete.

Is it necessary to have the 2 different schedules? Or would the Prometheus scrape_interval schedule be expected to trigger the API calls in the exporter?

It's expected that a Prometheus scrape would trigger the API calls. However a very slow API that takes 200s to query is going to be challenging to work with, among other things the maximum sane scrape interval is 120s, and in such a case caching results can be okay. You'll want Prometheus scraping no less often than every 120s one way or the other.

Beian

 

Basically I don't want to bog down that external applications performance or look like a DOS attack on the external application by continuously barraging it with requests when I only need the data once every 15 minutes or so.

Hopefully I explained this alright, but let me know if you need any clarification.

Thanks,

Cory

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a119b024-37a0-4f5a-a239-990fc35c0cbc%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[Ben Kochie]

A good reference is the official Prometheus docs on writing exporters

https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling

--

[Brian Candler]

For slow exporters, you may prefer to run the export on a 15-minute schedule (e.g. from cron) and write the results into a file which is picked up by node_exporter's textfile-collector.

The client(s) then scrape the metrics from node_exporter as normal, and get an immediate reply with the most recent data.  node_exporter also gives additional metrics including the timestamp on the file, so you can alert if the process which writes the file has not updated them for too long.