Prometheus exporter - reading CSV file having data from the past day

[Alexandre Allexandre]

Hello,

I am writing a Prometheus exporter which has to rad different CSV files. Each of them contains one entire day of data from the past (the goal is to have the exporter read a new CSV file each day. One CSV file is uploaded to the server each day, containing the data of the previous day. 

IN the CSV file, I have the same metrics every 5 mns. e.g : 

Date;Time;data
23.03.20;23:55:00;1
23.03.20;23:50:00;50
23.03.20;23:45:00;3

I struggle to add theses data in Prometheus properly. 

class CSVCollector(object):
  def collect(self):
    # We list all the min files in the current directory
    list_min = glob.glob("min*.csv")
    metric = GaugeMetricFamily(
                'day_tests_seconds',
                'kw', labels=["jobname"])
    for min in list_min :
      with open(min) as csv_file:
        csv_reader = csv.reader(csv_file, delimiter=';')
        line_count = 0
        for row in csv_reader:
            if line_count == 1:
                correct_date_format = row[0][:6] + "20" + row[0][6:]
                datetime_row = correct_date_format + ';' + row[1]
                timestamp = int(time.mktime(datetime.datetime.strptime(datetime_row, "%d.%m.%Y;%H:%M:%S").timetuple()))
                metric.add_metric(str(line_count), int(row[4]), timestamp)
            line_count += 1
    yield metric  
     




if __name__ == '__main__':
  # Usage: json_exporter.py port endpoint
  start_http_server(int(sys.argv[1]))
  REGISTRY.register(CSVCollector())
  while True: time.sleep(1)

Prometheus just read the first line, add it as a metric and read exactly the same again each time he scraps the exporter. What am I doing wrong ? I feel like this data should be a Jauge, as it goes up and down, but Prometheus seems that he doesn't want different data from the same Jauge in one scrap?

[Brian Candler]

Sorry, but I'm afraid you cannot backfill historical data into prometheus.  Prometheus will only scrape the current/latest value.  Backfill is a feature being considered for the future.

For now, you will need to look at a different storage engine.  I suggest VictoriaMetrics, which is prometheus-compatible (i.e. it supports prometheus API and promQL and can be configured as a remote-write endpoint for prometheus) and it *does* support the back-filling that you want to do.  However it's not a remote-read endpoint, so for example prometheus' alerting engine can't read data from it.  If you're only ingesting daily then this is unlikely to be a limitation.

Also check out other storage integrations which support both remote write and remote read, as you could populate the backend directly but prometheus can still read from them.

[Aliaksandr Valialkin]

On Tue, Apr 14, 2020 at 1:38 PM Brian Candler <b.ca...@pobox.com> wrote:

Sorry, but I'm afraid you cannot backfill historical data into prometheus.  Prometheus will only scrape the current/latest value.  Backfill is a feature being considered for the future.

For now, you will need to look at a different storage engine.  I suggest VictoriaMetrics, which is prometheus-compatible (i.e. it supports prometheus API and promQL and can be configured as a remote-write endpoint for prometheus) and it *does* support the back-filling that you want to do.  However it's not a remote-read endpoint, so for example prometheus' alerting engine can't read data from it.  If you're only ingesting daily then this is unlikely to be a limitation.

An additional important detail is that VictoriaMetrics supports CSV data ingestion additionally to backfilling.

 

Also check out other storage integrations which support both remote write and remote read, as you could populate the backend directly but prometheus can still read from them.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fc3f5c0b-c187-4698-9028-14144a7f38c4%40googlegroups.com.

--

Best Regards,

Aliaksandr