Prometheus Approach for Long Term Storage and Historical Data Mining

[Johnny Private]

Hi All,

I have the following use case for Prometheus:

1) Acquiring data for ~1 - 2 months (~1 TB) while using a storage.local.retention period of about 6 months so no data is lost during the acquisition period.

2) Using Grafana to review and data mine the data during the acquisition period

3) Once the acquisition period is over, I want to migrate the stored data to another location and have it available to continue data mining in the future (for at least 2 years) using Grafana without losing any of the acquired data.

4) While datamining in the future, I'd like to have the prometheus data set grow as little as possible since it will only be used to data mine the historical data already acquired.

From what I've read so far, I should be able to do the following:

A) Once done the initial 1 or 2 month data acquisition, I should be able to shut Prometheus down and take a backup of the prometheus directory containing all the files, perhaps with tar, and move the tar to it's new location on a different server.

B) Untar in the new location and modify the storage.local.retention for the configuration in the new location to something extremely large (ie: maybe the equivalent of ten years time -- is there a max I should be aware of?)

C) Start Prometheus up in the new location and happily data mine away without fear of losing anything from the dataset for years.

D) Keep the tar archive file around in case I need to untar again because something goes wrong while data mining in the future.

Is this considered the best way to handle my use case right now?  Have I got anything wrong or missed anything here?

Thanks,

Johnny

[Ben Kochie]

I would consider using the remote write option and cortex.

https://github.com/weaveworks/cortex

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2fb3d7eb-ad48-4635-899c-50636021e776%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Wilkie]

Thanks for the recommendation Ben!

Hi Johnny - I'm one of the authors of Cortex, let me know if you have any questions about it.

Thanks

Tom

On Mon, Oct 2, 2017 at 5:17 AM Ben Kochie <sup...@gmail.com> wrote:

I would consider using the remote write option and cortex.

https://github.com/weaveworks/cortex

On Mon, Oct 2, 2017 at 12:39 AM, Johnny Private <johnnyp...@gmail.com> wrote:

Hi All,

I have the following use case for Prometheus:

1) Acquiring data for ~1 - 2 months (~1 TB) while using a storage.local.retention period of about 6 months so no data is lost during the acquisition period.

2) Using Grafana to review and data mine the data during the acquisition period

3) Once the acquisition period is over, I want to migrate the stored data to another location and have it available to continue data mining in the future (for at least 2 years) using Grafana without losing any of the acquired data.

4) While datamining in the future, I'd like to have the prometheus data set grow as little as possible since it will only be used to data mine the historical data already acquired.

From what I've read so far, I should be able to do the following:

A) Once done the initial 1 or 2 month data acquisition, I should be able to shut Prometheus down and take a backup of the prometheus directory containing all the files, perhaps with tar, and move the tar to it's new location on a different server.

B) Untar in the new location and modify the storage.local.retention for the configuration in the new location to something extremely large (ie: maybe the equivalent of ten years time -- is there a max I should be aware of?)

C) Start Prometheus up in the new location and happily data mine away without fear of losing anything from the dataset for years.

D) Keep the tar archive file around in case I need to untar again because something goes wrong while data mining in the future.

Is this considered the best way to handle my use case right now?  Have I got anything wrong or missed anything here?

Thanks,

Johnny

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2fb3d7eb-ad48-4635-899c-50636021e776%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmr6Th7x-7Wvn-bh6dtaw%2BqTvk1vbzqRFHj5SU-%2BvTWfcw%40mail.gmail.com.

[Stuart Clark]

> Thanks for the recommendation Ben!
>
> Hi Johnny - I'm one of the authors of Cortex, let me know if you have any
> questions about it.
>

Another option is to use another datastore for long term storage via the
generic read/write method, such as InfluxDB or OpenTDSB.

--
Stuart Clark

[Julius Volz]

Brian also published a blog post today about their new remote read/write adapter for CrateDB: https://www.robustperception.io/using-the-cratedb-prometheus-adapter/

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1ee9d863f1b38bac85586cc6da811a24.squirrel%40secure.tradeline.org.uk.

[Johnny Private]

Hi All!  Thanks for the responses.  I'll check these ideas out.  Much appreciated.

Johnny

On Monday, October 2, 2017 at 3:10:45 PM UTC-4, Julius Volz wrote:

Brian also published a blog post today about their new remote read/write adapter for CrateDB: https://www.robustperception.io/using-the-cratedb-prometheus-adapter/

On Mon, Oct 2, 2017 at 8:17 PM, Stuart Clark <stuart...@jahingo.com> wrote:

> Thanks for the recommendation Ben!
>
> Hi Johnny - I'm one of the authors of Cortex, let me know if you have any
> questions about it.
>

Another option is to use another datastore for long term storage via the
generic read/write method, such as InfluxDB or OpenTDSB.

--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.