Importing data from MRTG/RRD

[Carlos Mendioroz]

Hi,

I guess the expression "I like what I see so far" applies, I'm really happy with the power of the Prometheus framework.

Now, I do have some "history" in some MRTG RRD files that I would like to write into the TSDB, so I can see it in one place.

I've been looking and I've not seen any such tool or idea talked about. Am I missing something that makes this a bad idea ?

Not hard to do it, just curious about nobody having mentioned it before.

[Carlos Mendioroz]

I tend to write too fast, I guess I already know why...

Prometheus DB is not intended for durability and by default expires data at 15 days.

Importing MRTG's year makes litle or no sense. So If I want to get rid of MRTG, integrating a long filtered history metrics repo (Influxdb ?) that can also be leveraged by Grafana seems the way to go.

[Simon Pasquier]

Depending on how many metrics you collect, Prometheus might be able to
store 1 year of data.
As for importing MRTG data in Prometheus, nothing exists to my knowledge.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/90c49c1f-6649-462b-a697-0b383fa9511a%40googlegroups.com.

[Ben Kochie]

On Thu, Oct 17, 2019 at 4:37 PM Carlos Mendioroz <cmend...@gmail.com> wrote:

I tend to write too fast, I guess I already know why...

Prometheus DB is not intended for durability and by default expires data at 15 days.

This is a misunderstanding. The Prometheus TSDB is intended for durability. But it is only as durable as the server it's on.

This misunderstanding comes from the fact that the Prometheus creators came from Google-level durability needs. We would say the same thing about MRTG's RRA files. They're not any more durable than Prometheus.

There are Prometheus users doing many-year storage in Prometheus. Just like with any SQL database, it takes careful planning of storage durability, backups, etc.

 

Importing MRTG's year makes litle or no sense. So If I want to get rid of MRTG, integrating a long filtered history metrics repo (Influxdb ?) that can also be leveraged by Grafana seems the way to go.

--

[Carlos Mendioroz]

On Thursday, October 17, 2019 at 12:44:29 PM UTC-3, Ben Kochie wrote:

On Thu, Oct 17, 2019 at 4:37 PM Carlos Mendioroz <cmend...@gmail.com> wrote:

I tend to write too fast, I guess I already know why...

Prometheus DB is not intended for durability and by default expires data at 15 days.

This is a misunderstanding. The Prometheus TSDB is intended for durability. But it is only as durable as the server it's on.

This misunderstanding comes from the fact that the Prometheus creators came from Google-level durability needs. We would say the same thing about MRTG's RRA files. They're not any more durable than Prometheus.

That may be (being a misunderstanding) but there are a couple of sentences in the storage area of Prometheus site that drove me in that direction:

• Thus, ,,, and should thus be treated as more of an ephemeral sliding window of recent data.
• If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory.

Also, given that by default exporters generate a lot of metrics, persisting all those seems like an overkill.

 

There are Prometheus users doing many-year storage in Prometheus. Just like with any SQL database, it takes careful planning of storage durability, backups, etc.

 

 

 That might be, but my current understanding leads me to think that persisting a subset of the metrics in a dedicated long term reference DB is a better way to go. Would you disagree ?

[Ben Kochie]

On Thu, Oct 17, 2019 at 6:29 PM Carlos Mendioroz <cmend...@gmail.com> wrote:

On Thursday, October 17, 2019 at 12:44:29 PM UTC-3, Ben Kochie wrote:

On Thu, Oct 17, 2019 at 4:37 PM Carlos Mendioroz <cmend...@gmail.com> wrote:

I tend to write too fast, I guess I already know why...

Prometheus DB is not intended for durability and by default expires data at 15 days.

This is a misunderstanding. The Prometheus TSDB is intended for durability. But it is only as durable as the server it's on.

This misunderstanding comes from the fact that the Prometheus creators came from Google-level durability needs. We would say the same thing about MRTG's RRA files. They're not any more durable than Prometheus.

That may be (being a misunderstanding) but there are a couple of sentences in the storage area of Prometheus site that drove me in that direction:

• Thus, ,,, and should thus be treated as more of an ephemeral sliding window of recent data.

Yeaaaaaaaaa, I just recently replaced that text with our current recommendation. :-D It will be updated on the site in the next release.

https://github.com/prometheus/prometheus/pull/6134

• If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory.

This should probably be updated, there are better tools and practices now.

 

Also, given that by default exporters generate a lot of metrics, persisting all those seems like an overkill.

The whole idea behind Prometheus is to collect as much data as possible, so you have it in case you need it, rather than have to be picky about what you collect and store. One of the drivers of what inspired Prometheus was that the exist monitoring systems could not keep all the data we needed. Signals that might seem overkill at one point, might be extremely useful in the event of a failure or outage.

 

There are Prometheus users doing many-year storage in Prometheus. Just like with any SQL database, it takes careful planning of storage durability, backups, etc.

 

 

 That might be, but my current understanding leads me to think that persisting a subset of the metrics in a dedicated long term reference DB is a better way to go. Would you disagree?

It all depends on your scale and needs. This is more of a "how long is a piece of string" question. Without knowing the details of your needs, I can't answer it. With the details you've provided so far, I would say it's a premature optimization.

The main point of this whole story is that sadly the Prometheus developers have been intentionally, and greatly, under-selling our TSDB for a very long time. It's actually pretty damn awesome. But we're a bunch of pessimistic, oncall-SRE-hardened, engineers. We don't have a marketing team to fluff it up.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d74c210b-8d30-43bf-8633-abca06ab3609%40googlegroups.com.

[Carlos Mendioroz]

Thanks again.

I'm fine with having all the metrics in the near time (e.g. node exporter) but I need/want some metrics for a longer term and for different reasons sometimes.

Key metrics for sizing like space used, IOPs, load average and free/used memory plus bytes/packets TX/RX are great on the long term, some health like temps and fan revs too plus some non node related like home power, PV generation, temperature, humidity, wind and rain, playing with fridge temp and heating temp.

Those I want to persist more in a secure manner. May be 50 metrics, not 2000 like I'm scraping now.

That's 40x on quantity, times 12x in frequency (5" to 1' e.g.). Would change 24 GB into 50 MB for a year of history.

(compression aside)