Best export option for rate-limited API

[Dan Rice]

Hi all!

I am writing an exporter that pulls stats from a remote API to which I need to limit my requests -- ideally to no more than like 100-200 per day. Which of these is my best option for keeping the number of requests low?

1. Exporter queries the remote API at each scrape but I increase scrape interval for this target. Can it practically be longer than 2 minutes these days?
2. Exporter caches the remote API data and reports the values back to Prometheus as current.
3. Exporter caches the remote API data and reports the values back to Prometheus with timestamps.

Or is there something I’m not thinking of?

Thanks!

[Brian Brazil]

On Tue, 22 Sep 2020 at 16:21, Dan Rice <dan....@sparkmeter.io> wrote:

Hi all!

I am writing an exporter that pulls stats from a remote API to which I need to limit my requests -- ideally to no more than like 100-200 per day. Which of these is my best option for keeping the number of requests low?

1. Exporter queries the remote API at each scrape but I increase scrape interval for this target. Can it practically be longer than 2 minutes these days?

 

1. Exporter caches the remote API data and reports the values back to Prometheus as current.

This is the only really practical option when you can only hit the API every 15m. The others will cause you issues with staleness.

Brian

 

1. Exporter caches the remote API data and reports the values back to Prometheus with timestamps.

Or is there something I’m not thinking of?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/60c04a76-c574-4c39-8511-24ce1cfa9c37n%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[Brian Candler]

On Tuesday, 22 September 2020 16:21:41 UTC+1, Dan Rice wrote:

> Exporter caches the remote API data and reports the values back to Prometheus as current.

A simple way to implement this is to write the metrics to a file, to be picked up by node_exporter's textfile collector.  It exposes a metric which is the modification timestamp of the file, so you can determine when the data was last updated, and alert if it becomes too stale.

[Dan Rice]

Thanks, Brian and Brian!

I'm running this as a container so I'm avoiding using the node_exporter, but it's only a couple hundred lines of code for me to do it from scratch using the Python client. I was going back and forth, though, on making the API calls asynchronously and updating a set of Gauges, versus using a custom collector and hitting the API synchronously. Sounds like I should go with something more like the former.