Loading a large csv file to Prometheus

[Abdelouahab Khelifati]

Hello all, 

How can I load a huge csv file into Prometheus? I tried https://github.com/stohrendorf/csv-prometheus-exporter but it doesn’t seem to be working.

Thanks!

Best,

Abdel

[Ben Kochie]

You will want to convert the CSV to OpenMetrics format.

See: https://prometheus.io/docs/prometheus/latest/storage/#backfilling-from-openmetrics-format

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/4b580edc-8815-4ffd-9349-a8de58de5422n%40googlegroups.com.

[Abdelouahab Khelifati]

Hello Ben and thanks for the prompt answer!

Is there a tool to do such a conversion or should I implement it myself? 

Also, would you have a few dataset examples of time series data in the appropriate format for Prometheus? 

My data currently looks something like this: 

time, id_sensor, value1, value2, ... , value n

2022--01-03, 'd02', 0.54, 0.24, ... , 0.34

2022--01-03, 'd04', 0.35, 0.94, ... , 0.53

2022--01-04, 'd02', 0.59, 0.22, ... , 0.95

...

Best,

Abdel 

[Ben Kochie]

You will have to do this yourself.

Also, that doesn't really look like time-series data.

Prometheus is a monitoring system, not a generic database.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/71430c18-3524-403d-9a22-b4e4161e32d3n%40googlegroups.com.

[Abdelouahab Khelifati]

Thanks Ben for the answer! 

Would you have an example of how such data should look? 

> Also, that doesn't really look like time-series data. Prometheus is a monitoring system, not a generic database.

Would that mean that I cannot store such data in standalone Prometheus? My understanding is that Prometheus includes a full TSDB system. 

The data sample I showed is a simplistic multivariate time series belonging to 2 sensors (d02 and d04). Isn't this manageable by Prometheus's TSDB? 

Thanks! 

Best,

Abdel 

[Stuart Clark]

On 27/03/2023 13:01, Abdelouahab Khelifati wrote:
> Thanks Ben for the answer!
>
> Would you have an example of how such data should look?
>
> > Also, that doesn't really look like time-series data. Prometheus is
> a monitoring system, not a generic database.
> Would that mean that I cannot store such data in standalone
> Prometheus? My understanding is that Prometheus includes a full TSDB
> system.

Prometheus does contain a TSDB, but it isn't designed as a general
purpose database. Instead it is designed to be used for metrics that are
scraped at a constant rate, only storing numeric values per metric (with
optional string labels). If you are wanting something more generic, you
are better off looking at other databases (both timeseries based and
otherwise).

--
Stuart Clark

[Abdelouahab Khelifati]

Hello Stuart, thanks for reaching out!

I see your point! 

My data does contain only numerical data, has a constant rate, and only has one string label. Would you say that it is possible but not optimal or simply not possible to use Prometheus as a general-purpose database? 

Also, would it be possible to point out the main technical factors rendering Prometheus unsuitable as a general-purpose database? 

Thanks again for your tremendous help! 

Best,

Abdel

[Stuart Clark]

On 2023-03-27 13:22, Abdelouahab Khelifati wrote:
> Hello Stuart, thanks for reaching out!
>
> I see your point!
>
> My data does contain only numerical data, has a constant rate, and
> only has one string label. Would you say that it is possible but not
> optimal or simply not possible to use Prometheus as a general-purpose
> database?
>
> Also, would it be possible to point out the main technical factors
> rendering Prometheus unsuitable as a general-purpose database?
>

Well the TDSB in Promethues could be used elsewhere if you created code
(as I think the TDSB code is fairly abstracted from other parts of
Prometheus) but otherwise all interactions are via the rest of
Prometheus. So for example the main way of populating the database is
via scraping targets. There is no way to change existing data (other
than deleting things). There is only a single schema (each timeseries
contains a single float64 per data point with a list of string based
labels [the metric name is actually a label]).

So I'd suggest that the TDSB is as it stands very much not a
general-purpose database - it is just a integral part of Promtheus,
designed for the use case of regular live numeric metric values, with
the ability to extract values via PromQL for graphing/alerts/etc. If
your use case aligns with that, then it could be a good fit, but if you
are wanting other things (being able to change values, doing a lot of
bulk importing, wanting to run/scale the database separately, wanting
database reliability/HA similar to other general purpose databases) it
may be a poor fit.

--
Stuart Clark

[Abdelouahab Khelifati]

Got it, thanks a lot for your detailed answer! 

In our use case, we don't need to change existing values. I think I would love to still try Prometheus on our data and queries. 

Back to my initial question about loading data to Prometheus from a CSV file. I realize this cannot be done and the data has to be converted. I am just wonder what format it should be converted to and if I can get a sample of the required format. Also, are there any resources/tutorials about loading historical data to Prometheus? 

Thanks a lot! 

Best,

Abdel