Help needed for using Node Exporter for parsing CSV file

[Navjyot Ramteke]

Hi All,

           We have a linux process which writes variables/statistics periodically into .csv file. We need to feed/save these variables/statistics in prometheus so that it can be viewed on Grafana.

Our process periodically writes a new file in a folder(linux virtual machine).

Folder:

stats_13_12_12_00.csv  >>>> written on 13th dec 12 am

stats_13_12_12_05.csv

stats_13_12_12_10.csv

stats_13_12_12_15.csv

stats_13_12_12_20.csv

stats_13_12_12_25.csv

in this example, the interval is 5 min. so next file will be written at 12:30 am.

the contents of the file is given below:

.csv file:

600,200,400,10,"abc",,20

explanation:

total_bytes=600

incoming_bytes=200

outgoing_bytes=400,

error_bytes=10,

interface_name="abc"

invalid_bytes=  <no value, so its not applicable>

rejected_bytes = 20

our csv file will just contains the values. the ordering of variables are fixed. if some variable/statistics is not applicable, then its value will be undefined (as shown below).

our file will contain a single row. we will hav 3000-4000 such variables in each file.

do we have any existing node exporter which can read csv file?

if not, any reference on how to write new node exporter?

Regards

Navjyot.

 

[Brian Candler]

> do we have any existing node exporter which can read csv file?

No. But if you write a simple script to read the csv file and write it back out in openmetrics format, then you can use node_exporter's textfile collector. For example:

total_bytes{interface_name="abc"} 600

incoming_bytes{interface_name="abc"} 200

outgoing_bytes{interface_name="abc"} 400

error_bytes{interface_name="abc"} 10

invalid_bytes{interface_name="abc"} 0

rejected_bytes{interface_name="abc"} 20

But you should beware that:

1. Having different filenames for different times of day won't work. You should create a single metrics file which has the *current* values, and let prometheus scrape it periodically. It will timestamp the values with the scrape time.  In general, you cannot import historical data into prometheus except as part of a one-time backfill operation.

2. You should not export the value which represents the counts from the last 5 minute period. Rather, you should record monotonically incrementing counters, i.e. the counter value which represents to *total* bytes since the system started up or was last reset.

See: https://prometheus.io/docs/concepts/metric_types/

If your data source can only generate individual counts per 5-minute period, then change your integration: use something like statsd_exporter to maintain the counts, and push data to statsd_exporter every accounting period. statsd_exporter will accumulate the values, and then get prometheus to scrape that.

If you're not prepared to do that, then prometheus is probably not the right system for you. Use a SQL database or similar instead.

> if not, any reference on how to write new node exporter?

Exporters simply respond to HTTP requests and return values in openmetrics format. There are client libraries available in many languages.

https://prometheus.io/docs/instrumenting/clientlibs/