TSDB Import Command

[Al]

I would like to know if there is a general estimate as to when the tsdb import functionality will be merged into master and made available in a release?  I have a use case where I need to import large batches of metrics at the end of every day.  I was initially planning on using Postgres with TimescaleDB along with the Promscale connector as this would be suitable due to the fact we can simply do bulk inserts of the metrics and specify value of the time field.  I would much rather stick with simply using Prometheus as we can avoid creating yet another database to manage.  

On another note, I'm trying to get an understanding of how the TSDB import feature would work with setup running with Thanos?  I imagine that once the blocks are imported into the local Prometheus instance, the thanos sidecar would simply read the block(s) and upload them to the configured storage bucket?  I realize this touches on a different application not directly part of Prometheus so if I'm best asking this in the Thanos group, please let me know.

Thanks for your help.

[Brian Candler]

I don't know if that CSV import supports your use case: there's a difference between back-filling an entire timeseries with historical data, and pumping new blocks of data in every 24 hours.  Doing the latter raises various questions - how would alerting rules work, for instance?

Your idea of using TimescaleDB seems reasonable for this use case.  VictoriaMetrics is another option to look at, as it supports ingestion in a whole range of formats:

https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-import-time-series-data

and it can be queried directly using a superset of PromQL with a Prometheus-compatible API (that is, you can point a Grafana dashboard at it and pretend it's a Prometheus server)

[Al]

For this specific use-case, it's understood that we wouldn't have alerting rules as the metrics would ingested only after they've been created.  It would be used purely for inspecting historical trends and identifying outliers over weeks, months. etc.   I'm leaning more towards TimescaleDB as we already have many instances of Postgres internally so it would end up being less work to get this up vs implementing something new such as VictoriaMetrics.

I was thinking we could simply spin up a Prometheus instance, run the tsdb import and then have thanos sidecar upload the block(s) to object storage.  Once that's done, the prometheus instance could be destroyed as we would always query the data from thanos query.   Given that we wouldn't be alerting on this data, would the TSDB import still be suitable?

[Aliaksandr Valialkin]

On Wed, Oct 14, 2020 at 6:12 PM Al <alain.l...@gmail.com> wrote:

For this specific use-case, it's understood that we wouldn't have alerting rules as the metrics would ingested only after they've been created.  It would be used purely for inspecting historical trends and identifying outliers over weeks, months. etc.   I'm leaning more towards TimescaleDB as we already have many instances of Postgres internally so it would end up being less work to get this up vs implementing something new such as VictoriaMetrics.

I'd recommend trying both solutions in parallel - TimescaleDB and VictoriaMetrics - and then choosing the most suitable solution for your case.

---

Best Regards,

Aliaksandr Valialkin, CTO VictoriaMetrics