Per-metric data stores in ruby_client

[Николай Марков]

Problem

I would like to use different stores for different metrics.

E.g. for metrics that I don't scare to lose I would like to use faster storage, and for metrics I scare to lose - more reliable storage like DirectFileStore.

But seems like it is possible for now just to use a single store for all metrics, because there is a one "global" config setting `Prometheus::Client.config.data_store=`

I'm ready to make a PR with such functionality. What do you think about such feature?

[Daniel Magliola]

Hello,

Do you have an example of a use case for this?

In general, you wouldn't "care to lose" any metrics. On a server restart, you'd lose all of them, which Prometheus would see as a reset, and deal with it normally.

That is actually the best practice, and we recommend emptying the metrics temp directory on startup, if you have a persistent file system, to ensure metrics get reset.

Sorry if I misunderstood your use case, however. An example would probably help.

Thank you!

Daniel

[Николай Марков]

I will try to describe the problem.

We send our app-metrics to statsd and we have a statsd_exporter for prometheus.
One of the metric we have is a counter which indicates a count of some "cron job" executions.
And we have an alertmanager which checks the following query:

sum(increase(extremist_material_notifier_executions_count {environment="production"}[1d2h])) < 1

This worker may run on different servers and statsd automatically adds "instance" label for each metric so this metric may have a different "instance" label on each run - this is why we use "sum" in alert-query.
Example of metrics:

extremist_material_notifier_executions_count{environment="production", instance="tasks1"}

extremist_material_notifier_executions_count{environment="production", instance="tasks2"}

extremist_material_notifier_executions_count{environment="production", instance="tasks3"}

extremist_material_notifier_executions_count{environment="production", instance="tasks4"}

As you know statsd forgets its metrics on restarts so when it happens we have a "counter reset" and "increase"-function does not handle it well - it does not show increase when counter was e.g. 100 before the restart and disappears after statsd restart.

So I would like to use "textfile collector" module to persist such metrics and expose all other metrics through "/metrics" endpoint as usual - this is why I would like to use different stores for different metrics.

среда, 12 мая 2021 г. в 23:14:29 UTC+3, dmag...@gmail.com:

[Brian Brazil]

On Tue, 18 May 2021 at 21:49, Николай Марков <main...@gmail.com> wrote:

I will try to describe the problem.

We send our app-metrics to statsd and we have a statsd_exporter for prometheus.
One of the metric we have is a counter which indicates a count of some "cron job" executions.
And we have an alertmanager which checks the following query:

sum(increase(extremist_material_notifier_executions_count {environment="production"}[1d2h])) < 1

This worker may run on different servers and statsd automatically adds "instance" label for each metric so this metric may have a different "instance" label on each run - this is why we use "sum" in alert-query.
Example of metrics:

extremist_material_notifier_executions_count{environment="production", instance="tasks1"}

extremist_material_notifier_executions_count{environment="production", instance="tasks2"}

extremist_material_notifier_executions_count{environment="production", instance="tasks3"}

extremist_material_notifier_executions_count{environment="production", instance="tasks4"}

As you know statsd forgets its metrics on restarts so when it happens we have a "counter reset" and "increase"-function does not handle it well - it does not show increase when counter was e.g. 100 before the restart and disappears after statsd restart.

So I would like to use "textfile collector" module to persist such metrics and expose all other metrics through "/metrics" endpoint as usual - this is why I would like to use different stores for different metrics.

Have you considered the pushgateway for this? Cluster-level cron jobs are what it's designed for.

Brian

 

среда, 12 мая 2021 г. в 23:14:29 UTC+3, dmag...@gmail.com:

Hello,

Do you have an example of a use case for this?

In general, you wouldn't "care to lose" any metrics. On a server restart, you'd lose all of them, which Prometheus would see as a reset, and deal with it normally.

That is actually the best practice, and we recommend emptying the metrics temp directory on startup, if you have a persistent file system, to ensure metrics get reset.

Sorry if I misunderstood your use case, however. An example would probably help.

Thank you!

Daniel

On Saturday, May 8, 2021 at 6:21:50 PM UTC+1 main...@gmail.com wrote:

Problem

I would like to use different stores for different metrics.

E.g. for metrics that I don't scare to lose I would like to use faster storage, and for metrics I scare to lose - more reliable storage like DirectFileStore.

But seems like it is possible for now just to use a single store for all metrics, because there is a one "global" config setting `Prometheus::Client.config.data_store=`

I'm ready to make a PR with such functionality. What do you think about such feature?

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/12a310b2-e69e-4865-a936-21bfab6111a6n%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[Николай Марков]

> Have you considered the pushgateway for this? Cluster-level cron jobs are what it's designed for.

Yes, we do.
But I hope that Prometheus ruby client will support text-file store first of all as it is a much simpler solution for us, as we already have a text-file collector module.

I can open a PR with such a feature (per metric store support + text-file store).
WDYT?

--

С уважением,

Николай Марков.

[Daniel Magliola]

My personal opinion is that this use case is a bit too niche for adding to the client. I don't think it'll be very common for people to need different data stores for different metrics.

In terms of a text file store, you can make your own store and have it in your codebase, and set it in the Prometheus config, so we wouldn't need to add it to the client at all.

And my suggestion is that you set all your metrics to use this same store. 

However, if you *must* have different stores, you *could* hack your way around this by setting the config to the store you need before declaring the metrics that need a different one, and then resetting it back. 

You can only do this if your code is not multi-threaded at the time of declaring those metrics, so you want to make sure to define them before worker threads start up if you have any.

Again, that's not what I'd recommend as a general use case, it's definitely a hack that is not explicitly supported. 

I'd suggest using the same store for all of your metrics, but if you absolutely need to do this, that could work.

Hope that helps,

Daniel