Create histogram with recording rules

[Suki Sahota]

I have a metric I'm collecting which is a gauge vector of timestamps. I'd like to take each one of those timestamps and subtract it from the time now and turn it into a histogram. Can I do this with recording rules?

Right now I have a gauge vector of:

<timestamp_1, timestamp_2, ..., timestamp_n>.

I'd like a histogram of [timeNow - timestamp_i]. I'm specifically looking to use recording rules.

Thanks, Suki

[Julius Volz]

For the histogram buckets, is your intention to have a few preconfigured "age" buckets like 0s-15s, 0s-1m, 0s-5m, and so on (cumulative buckets, all starting at 0, as in normal Prometheus histograms), and then a count for how many timestamps you have that fall into each age bucket?

You could make one recording rule for *each bucket*, with all the recording rules sharing the same output metric name. For example, for a bucket with a range of 0s - 1m, you could have this kind of rule:

record: my_timestamp_age_bucket

expr: count(time() - my_timestamp_seconds <= 60)

labels:

  le: "60"

...and equivalently for the other buckets.

If you wanted to have a *non-cumulative* histogram (and don't need to use histogram_quantile() on it), the rule would look like this (e.g. for a bucket from 15s - 1m):

record: my_timestamp_age_bucket

expr: count(time() - my_timestamp_seconds > 15 <= 60)

labels:

  # some other label(s) here to designate your bucket range

Remember to include a catch-all bucket for very long durations with an upper bound of +Inf, so you don't miss recording them.

Note that with the "count()" approach I use above, it will only create a bucket time series for buckets with at least one timestamp falling into the bucket, as otherwise "count()" will just return an empty result. If you want to create a bucket with the value of "0" instead, you can append "or vector(0)" to the rule expression:

record: my_timestamp_age_bucket

expr: count(time() - my_timestamp_seconds <= 60) or vector(0)

labels:

  le: "60"

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d542aed9-56bf-4737-9720-775b4f27d03dn%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Suki Sahota]

I like your approach Julius. But I have a question about all of the buckets that I create. If I create all of the buckets with the same output metric name, will each bucket overwrite the previous buckets? If not, will the output metric name now be a histogram that contains all of the buckets? Thanks for your reply!

[Julius Volz]

The buckets are differentiated via an "le" label (at least for the cumulative histogram variant, for a non-cumulative one you can make up your own bucket upper/lower bound label names), and time series are identified by their metric name and label set. So the recording rules for the different buckets wouldn't overwrite each other's time series.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/517249f0-e960-4f68-9e1f-15895b8f218an%40googlegroups.com.

[Suki Sahota]

Thank you so much Julius! I think this solution works great. Just what I was looking for with recording rules.