Philosophy on Long-Term Storage of Business Metrics

[dcadwa...@gmail.com]

Hey all!

I'm looking to achieve two things:

1. Set up monitoring for the health of our infrastructure
2. Start tracking business metrics for everyone in the company to rally around

For #1, Prometheus is an obvious choice.  For #2, we're looking to answer questions that involve long-term analysis like:

How is our customer-facing performance doing compared to last week?  Last month?  Last year? 

I've been reading some things on this mailing list that seem to indicate that Prometheus was not designed for keeping data longer than a few weeks.  However, I just read Classes of Container Monitoring (by Brian Brazil) which states:

A well-used monitoring system collects all sorts of information, from raw hardware utilization and counts of API requests to high-level business metrics.

This seems to indicate that I should consider Prometheus for case #2.   So now I'm confused!

I know that the upcoming 2.0 release will improve performance, but will it also change the philosophy of Prometheus' data retention, allowing it to be used for long-term storage of business metrics?

If not, I'm thinking about using Prometheus for monitoring and InfluxDB for business metrics.  But that seems like a shame, since InfluxDB doesn't allow for math across measurements, which limits its usefulness in distilling down multiple metrics into percentages, etc.

Any insights / recommendations would be appreciated!

Thanks!

Dave

[Stuart Clark]

Use both :-)

Monitor everything using Prometheus and then plug in InfluxDB via the generic read and write hooks, which then means you can have long term storage queried via PromQL
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[dcadwa...@dnanexus.com]

Thanks Stuart!  I assume you're referring to this pull request?  If so I have a few follow-ups:

1. Is this considered a best practice in the Prometheus community?  I'm looking to follow Brian's advice of collecting both "raw hardware utilization" and "high-level business metrics." but I'd rather stay within the realm of doing what's "normal".  When I start to get into bleeding-edge-out-in-the-weeds territory, it's a smell to me that I'm missing something.  If not a lot of folks are doing this, does that mean that they maintain two totally different systems for short-term and long-term metrics?  Or that most people aren't tracking long-term metrics at all?

2. Are there any docs or other write-ups for this feature?

3. If I used this would I be replacing even Prometheus' short-term storage with an InfluxDB backend, or would Prometheus somehow know to use one backend for short-term data and one for long-term?

Thanks again!

Dave

[dcadwa...@dnanexus.com]

Found the answer to question 2: https://github.com/prometheus/docs/pull/703

[Brian Brazil]

On 15 June 2017 at 18:48, <dcadwa...@gmail.com> wrote:

Hey all!

I'm looking to achieve two things:

1. Set up monitoring for the health of our infrastructure
2. Start tracking business metrics for everyone in the company to rally around

For #1, Prometheus is an obvious choice.  For #2, we're looking to answer questions that involve long-term analysis like:

How is our customer-facing performance doing compared to last week?  Last month?  Last year? 

I've been reading some things on this mailing list that seem to indicate that Prometheus was not designed for keeping data longer than a few weeks.  However, I just read Classes of Container Monitoring (by Brian Brazil) which states:

A well-used monitoring system collects all sorts of information, from raw hardware utilization and counts of API requests to high-level business metrics.

This seems to indicate that I should consider Prometheus for case #2.   So now I'm confused!

I know that the upcoming 2.0 release will improve performance, but will it also change the philosophy of Prometheus' data retention, allowing it to be used for long-term storage of business metrics?

The sort of business metrics you'd typically store in Prometheus are mainly those related to operational monitoring and engineer, and thus not often the full set of metrics that the business side of a company wants.

If your requirement is long-term storage of business metrics for the company generally, then unless you data set happens to be particularly small, a business intelligence/data warehouse systerm is probably more along the lines of what to look for.

Brian

 

If not, I'm thinking about using Prometheus for monitoring and InfluxDB for business metrics.  But that seems like a shame, since InfluxDB doesn't allow for math across measurements, which limits its usefulness in distilling down multiple metrics into percentages, etc.

Any insights / recommendations would be appreciated!

Thanks!

Dave

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ec1364c1-a224-4e51-8671-c6aadd5ee1e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Dave Cadwallader]

Thanks Brian. So would that mean something like Splunk?  Or are there alternatives, OSS possibilities, etc?

On Thu, Jun 15, 2017 at 4:07 PM Brian Brazil <brian....@robustperception.io> wrote:

On 15 June 2017 at 18:48, <dcadwa...@gmail.com> wrote:

Hey all!

I'm looking to achieve two things:

1. Set up monitoring for the health of our infrastructure
2. Start tracking business metrics for everyone in the company to rally around

For #1, Prometheus is an obvious choice.  For #2, we're looking to answer questions that involve long-term analysis like:

How is our customer-facing performance doing compared to last week?  Last month?  Last year? 

I've been reading some things on this mailing list that seem to indicate that Prometheus was not designed for keeping data longer than a few weeks.  However, I just read Classes of Container Monitoring (by Brian Brazil) which states:

A well-used monitoring system collects all sorts of information, from raw hardware utilization and counts of API requests to high-level business metrics.

This seems to indicate that I should consider Prometheus for case #2.   So now I'm confused!

I know that the upcoming 2.0 release will improve performance, but will it also change the philosophy of Prometheus' data retention, allowing it to be used for long-term storage of business metrics?

The sort of business metrics you'd typically store in Prometheus are mainly those related to operational monitoring and engineer, and thus not often the full set of metrics that the business side of a company wants.

If your requirement is long-term storage of business metrics for the company generally, then unless you data set happens to be particularly small, a business intelligence/data warehouse systerm is probably more along the lines of what to look for.

Brian

 

If not, I'm thinking about using Prometheus for monitoring and InfluxDB for business metrics.  But that seems like a shame, since InfluxDB doesn't allow for math across measurements, which limits its usefulness in distilling down multiple metrics into percentages, etc.

Any insights / recommendations would be appreciated!

Thanks!

Dave

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ec1364c1-a224-4e51-8671-c6aadd5ee1e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 15 June 2017 at 23:15, Dave Cadwallader <dcadwa...@gmail.com> wrote:

Thanks Brian. So would that mean something like Splunk?  Or are there alternatives, OSS possibilities, etc?

That's getting a bit outside my area. It really depends on what sort of business metrics you're interested in and what you want to use them for.

Brian

 

On Thu, Jun 15, 2017 at 4:07 PM Brian Brazil <brian.brazil@robustperception.io> wrote:

On 15 June 2017 at 18:48, <dcadwa...@gmail.com> wrote:

Hey all!

I'm looking to achieve two things:

1. Set up monitoring for the health of our infrastructure
2. Start tracking business metrics for everyone in the company to rally around

For #1, Prometheus is an obvious choice.  For #2, we're looking to answer questions that involve long-term analysis like:

How is our customer-facing performance doing compared to last week?  Last month?  Last year? 

I've been reading some things on this mailing list that seem to indicate that Prometheus was not designed for keeping data longer than a few weeks.  However, I just read Classes of Container Monitoring (by Brian Brazil) which states:

A well-used monitoring system collects all sorts of information, from raw hardware utilization and counts of API requests to high-level business metrics.

This seems to indicate that I should consider Prometheus for case #2.   So now I'm confused!

I know that the upcoming 2.0 release will improve performance, but will it also change the philosophy of Prometheus' data retention, allowing it to be used for long-term storage of business metrics?

The sort of business metrics you'd typically store in Prometheus are mainly those related to operational monitoring and engineer, and thus not often the full set of metrics that the business side of a company wants.

If your requirement is long-term storage of business metrics for the company generally, then unless you data set happens to be particularly small, a business intelligence/data warehouse systerm is probably more along the lines of what to look for.

Brian

 

If not, I'm thinking about using Prometheus for monitoring and InfluxDB for business metrics.  But that seems like a shame, since InfluxDB doesn't allow for math across measurements, which limits its usefulness in distilling down multiple metrics into percentages, etc.

Any insights / recommendations would be appreciated!

Thanks!

Dave

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.

To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ec1364c1-a224-4e51-8671-c6aadd5ee1e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io