prometheus scrape cycle sync with node exporter

[Puneet Singh]

Hi,
I have a custom node exporter from which prometheus scrapes every 30 seconds.
Issue is that sometimes -
a) if the data volume is large ,node exporter takes longer , On top of that , 

b) i need to clear the metrics to remove the old stale values before i set the metrics to the respective values again. 
These scenario a) causes mixup of data from 2 iterations   and in scenario b) scrape cycle may get 0 values .



Expected behaviour -> Iteration 1 starts ->  Node exporter collects raw data -> clear metrics -> process data and set metrics -->Iteration 1 complete -> Prometheus Scrapes Data.

Problem is :  a)  Iteration 1 starts ->  Node exporter collects raw data -> clear metrics --> (  Prometheus Scrapes Data ) ->  process data and set metric #1  --> . process and set metric #N     -->Iteration 1 complete

b) Iteration1 starts ->   Node exporter collects raw data -> clear metrics -->  process data and set metric#1 --> (  Prometheus Scrapes Data )  -->.. process and set metric#n-->  Iteration 1 complete


So, Is there a way to enforce some syncronization between node exporter and prometheus scrape cycle ? example- 
Scenario 1) The node exporter prepared data in 20 seconds and Prometheus should start scraping from ( say) 30th second of every minute i.e.
02:34:30
Scenario 2) To expose all the metrics in one shot , example - 
Iteration2 starts ->   Node exporter collects raw data -> clear metrics -->  process data and set metric#1 -->  -->.. process and set metric#n-->  Iteration 2 complete -> expose metric.

something like - metric1.labels(STATE=x).set (y)
                              metric1.expose()

I am okay to scrape old data , but dont want to -

 mix Iteration 1 and Iteration 2's data or ,
 scrape the data when node exporter has cleared metric.

Scenario 3: make scrape wait till data collection iteration is complete.


Please do let me know if more information is required from my end here.


Regards
Puneet

[Brian Candler]

"i need to clear the metrics to remove the old stale values before i set the metrics to the respective values again."

I don't understand what you mean by that. Surely you just change the metric from the old value, to the new value?  What sort of metric is it?  Perhaps you are trying to use a gauge when a counter would be the right thing?

That is, don't return a metric for "the number of things that happened in the last 30 seconds": return a metric for "the total number of things that happened since the exporter started".

> Is there a way to enforce some syncronization between node exporter and prometheus scrape cycle ?

No, and that's by design. You should be able to point two, three or more prometheus instances at the same exporter, and it should continue to work. You should be able to make test scrapes using curl, and it shouldn't break scraping by other prometheus servers. If your exporter changes state when a scrape takes place, then it's not following the intended model.

Gauges should return *the current* or *most recently known* value of a metric, and counters should return *the current* or *most recently known* value of the counter.

You say that sometimes your additional data collection takes longer than 30 seconds.  In that case, I'd suggest you write a periodic task which triggers itself every 30 seconds, and writes out the results to a file for the node_exporter textfile collector to pick up.  If it takes longer than 30 seconds, then in principle it's fine to miss a cycle: you keep exporting the old values for longer.  Any values which are gauges will update to the most recent value on the next scrape, and any values which are counters should count all the events which occurred between the two scrapes.

With that model, you can have as many prometheus servers scraping the same exporter as you like, without creating extra load; and metrics will be delayed by no more than 30 seconds.