Latest metric value from multipe Unicorn workers

[Mindaugas Niaura]

I work as SRE at a company which runs Ruby on Rails application deployed with Unicorn. Not so long time ago, we started to migrate from StatsD (https://github.com/statsd/statsd) based metrics to Prometheus. We wrote wrapper library which uses https://gitlab.com/gitlab-org/prometheus-client-mmap (which also is fork of https://github.com/prometheus/client_ruby) and so far so good, everything went as planned during PoC.

But recently we found some limitation which we are banging head on how to solve. Many people are used to StatD and in application code they used just to increase gauges values (from multiple locations) and aggregating such values gives you "current" value.

Now with Prometheus, we have multiple muti-thread workers which updates their own gauge values (workers are labeled e.g. with id). So let's say one process stores 10 in gauge and after some minutes another worker stores 5. How we shall know what value is the latest?

Prometheus client_ruby added most_recent aggregations https://github.com/prometheus/client_ruby/pull/172, which helps on single Unicorn, but let's say we want to know what is the latest value between two multiworker Unicorns?

Few proposed solutions:

1. Use Pushgateway https://prometheus.io/docs/practices/pushing/ - we really want to avoid running it, if we can, as it will become single point of failure, and probably we also will have some performance issues due to high rate of pushes.
2. Use https://github.com/discourse/prometheus_exporter - that would run as separate instance and seems to be option. But sill how to find what is the latest value between Unicorns?

Maybe someone had similar issues and come up with some different solutions? Any feedback is very welcome.

[Brian Candler]

On Thursday, 17 March 2022 at 10:28:18 UTC nia...@gmail.com wrote:

I work as SRE at a company which runs Ruby on Rails application deployed with Unicorn. Not so long time ago, we started to migrate from StatsD (https://github.com/statsd/statsd) based metrics to Prometheus. We wrote wrapper library which uses https://gitlab.com/gitlab-org/prometheus-client-mmap (which also is fork of https://github.com/prometheus/client_ruby) and so far so good, everything went as planned during PoC.

But recently we found some limitation which we are banging head on how to solve. Many people are used to StatD and in application code they used just to increase gauges values (from multiple locations) and aggregating such values gives you "current" value.

Now with Prometheus, we have multiple muti-thread workers which updates their own gauge values (workers are labeled e.g. with id). So let's say one process stores 10 in gauge and after some minutes another worker stores 5. How we shall know what value is the latest?

What do you want to happen? Do you want the scraped result to be 15? In that case it's a counter not a gauge, and you need to add to it.  Or do you want two separate timeseries showing values 5 and 10? Then you need separate gauges in each thread with their own labels.

 

Maybe someone had similar issues and come up with some different solutions?

There is statsd_exporter, which takes the same protocol messages as statsd, but exposes gauges and counters for scraping.  This is an easy way to aggregate counters across multiple processes and/or threads, since update messages from multiple sources can update the same counter, and it doesn't matter whether they are separate processes or threads.

I suspect it means you'll have to revert your code back to using statsd client though :-) 

[Mindaugas Niaura]

It also would mean what we would need to support both systems - Prometheus and Statsd (we use Prometheus for our infra monitoring so it will no go away any soon), which is not ideal.

Maybe some solution can be implemented exposing metrics with timestamps https://prometheus.io/docs/instrumenting/exposition_formats/#text-format-example, which Prometheus support. Each thread will expose metric with timestamp and we would rewrite timestamp on scrape. When we probably can use last_over_time() to find what is the latest value.

[Brian Candler]

On Friday, 18 March 2022 at 09:45:59 UTC nia...@gmail.com wrote:

It also would mean what we would need to support both systems - Prometheus and Statsd (we use Prometheus for our infra monitoring so it will no go away any soon), which is not ideal.

I think you misunderstand me.  statsd_exporter does *not* use statsd.  It's standalone.  But you send messages to it in statsd format.  It's kind-of like pushgateway, except it knows how to increment counters.

With statsd_exporter, you would be using only prometheus.