Scrape Notification

[Stephen Perkins]

Hey all,

I'm trying to write an exporter.  I want my graph to display info only when the test is run and NOT when it is not running.

Something like:

I have not found a mechanism to be notified when a scrape has been completed.    My current workaround is that I:

1. publish test values
   
2. wait for the scrape interval plus a few seconds.  This should result in test values being scraped
3. publish zeros

Is there a way to be notified when a scrape occurs so that i can clear my values until the next test run?

Thanks in advance for any info,

- Steve

P.S.  I do not really want to "push" results into the pushgateway.  However, this may be the recommended solution...

[Brian Candler]

Prometheus is not an event database.  If you want to show the results of individual tests (or the duration of individual tests) then this isn't the tool for you.  Use a normal SQL database, or a noSQL like ElasticSearch, InfluxDB etc.

Prometheus is a metric database, so if you want to show the *number* or *rate* of tests (or test failures/successes) then it will do a good job.  To do that, you should expose a counter of the number of test runs/passes/failures, or an accumulated total of the test run times, or a histogram of test run times.

Then it doesn't matter how often it's scraped: the data is always correct.  The graph can show how many tests ran in a certain amount of time by looking at how the counter has changed over that time.  If you have a histogram you can also answer questions like "what percentage of tests completed in under 5 seconds".  But you are still thinking about tests in the aggregate, not as individual events.

Resetting the data on a scrape is not a good idea, because it breaks as soon as two prometheus servers scrape the same endpoint - e.g. a high-availability pair, or a laptop scraping the data to test a new collector.

You mention pushgateway.  That's rarely the right solution for any application.  You could use it here, if the only thing you care about is the result and/or timestamp of the *last* test that was run.  To accumulate counters or totals between test runs, have a look at statsd_exporter instead.

[Stephen Perkins]

Hey Brian,

Thanks for the reply.  I do agree that I'm not using Prometheus as intended.  The rationale for my question is that I'm interested in working "around" what I would consider "limitations" of Grafana.  I want to coordinate the visualization of events alongside the time series data I am collecting.  By allowing Prometheus to scrape one set of numbers at specific instances and then posting zeros for others, I am creating a fake "time series" that really hosts events.  Grafana will just ingest that data as any other and then present a beautiful synchronized visualization.

I got it working by getting a callback from the webserver.  Once Prometheus has scraped the page, the webserver initiates the callback, I zero out the result.

Best regards,

- Steve

 

P.S.  I already send event data to Grafana using annotations.  This gives the nice visualization, but doesn't allow the end user to easily download the event data for offline processing.

[Brian Candler]

On Sunday, 28 June 2020 19:24:16 UTC+1, Stephen Perkins wrote:

 The rationale for my question is that I'm interested in working "around" what I would consider "limitations" of Grafana.  I want to coordinate the visualization of events alongside the time series data I am collecting.  By allowing Prometheus to scrape one set of numbers at specific instances and then posting zeros for others, I am creating a fake "time series" that really hosts events.  Grafana will just ingest that data as any other and then present a beautiful synchronized visualization.

I got it working by getting a callback from the webserver.  Once Prometheus has scraped the page, the webserver initiates the callback, I zero out the result.

...

P.S.  I already send event data to Grafana using annotations.  This gives the nice visualization, but doesn't allow the end user to easily download the event data for offline processing.

Sounds like Loki might be what you're looking for.  It's written by the Grafana team, and gives you a way to store and visualise event logs which can be directly linked to associated metrics, as they share the same label sets.

https://www.youtube.com/watch?v=TcmvmqbrDKU