Is Prometheus and PromQL suitable for working on a metric that doesn't change much?

[Florian Luce]

Hi everyone,

I have a use case where I'm trying to track the use of a feature that isn't often used, and I've decided to use a counter.

To give you some stats, for the moment this counter will be incremented 50 times over 24h on average.

This functionality is implemented within a service that is deployed and replicated on 10 to 20 pods (infra k8s), with metrics scrapped at regular frequency (30sec). We have a label on the metrics to identify the pods and avoid collisions, so this metric evolves very little and is spread over a number of time series.

Here's a small example of the “flat” side of this metric

 

The first problem we had to solve was losing the 0 to 1 transition (we tested the feature beta created timestamps zero injection, but it generated a significant CPU overload, so we didn't activate it). 

So we went with a request like this : 

clamp_min(

sum (max_over_time(import_processed_total{}[1m]) or vector(0))

- sum (max_over_time(import_processed_total{}[1m] offset 1m) or vector(0)),

0)

And i fix the "Min interval" of query options in grafana to 1m.

It's still imperfect at the end of time series, but arrives at a result close to reality if I analyze it over time windows of 24 / 48 hours.

However, it becomes unusable if I use this approach over 30 days.

The questions I have are the following:

• Is there a different approach (promql query) to exploit this metric without losing precision?
• Is Prometheus suitable for this kind of use case?
• Couldn't an “adaptive metrics” approach be a solution for cleaning up this metric and generating a synthetic version for one day, which can then be analyzed over 30 days?

Thx for the read and futur answers

[Brian Candler]

I see at least two distinct issues there.

1. "Is Prometheus and PromQL suitable for working on a metric that doesn't change much?" - quite simply, "yes". Prometheus uses delta compression, so adjacent identical values compress extremely well.  Indeed, Prometheus is often used for metrics which *never* change, so long as the labels are static, for example:

node_os_info{id="ubuntu", id_like="debian", name="Ubuntu", pretty_name="Ubuntu 22.04.5 LTS", version="22.04.5 LTS (Jammy Jellyfish)", version_codename="jammy", version_id="22.04"} 1

The overhead of scraping this repeatedly is tiny.

2. You have a specific issue with distributed counters. Ideally you'd use sum(import_processed_total) to get the total amount of work done over all pods, but that's not reliable because parts of the counter will *vanish* when the pod terminates, and you don't want the total counter to go down.

I think the best solution is to accumulate your counters in some other external process, such as statsd_exporter. Send a '+1' whenever you do some work. The value scraped from statsd_exporter will be the total amount of work done, independently of which pod has performed the work. That is fine for both total work done and for calculating the overall rate of processing work.

[Florian Luce]

Hey Brian,

Thx for the answer, if i understand correctly prom. can be used for this use case, but I need to integrate a new component (aka statsd_exporter) to clean metrics and remove "non static" label value. 

Your approach allows you to go from, several time series, e.g:

• import_process_total{pod=“ pod-sdsdsf”}
• import_process_total{pod=“ pod-zzze”}
• import_process_total{pod=“ pod-fdssfdf”}
• ...

to a single 

• import_process_total{}

And then I can use the classic “sum” operator to get the total of the (processed) import, increase...

Is that the idea?

[Brian Candler]

You wouldn't even need the "sum" operator, since that sums across multiple timeseries. There would be a single import_process_total timeseries.

Queries might be:

import_process_total - import_process_total offset 1h

rate(import_process_total[1h])

... etc