Time series with change interval much less than scrape interval

[Mark Selby]

I am struggling with PromQL over an issue dealing with a metric that
changes less frequently than the scrape interval. I am trying to use
Prometheus as a pseudo event tracker and hoping to get some advice on
how to best try and accomplish my goal.

I have a random job that runs at different intervals depending on the
situation. Some instances of the job run every five minutes and some run
only once an hour or once a day. The job creats a node_exporter
textfile snippet that gets scraped on 30 second interval.

Below is an example of a metric that changes only every five minutes with
the lesser scrape interval. In this scenario all the points with same
value are from the same job run. I really only care about one of those.

I have no way to know what the interval is between sets for all my
different jobs. All I know is that when the value changes, a new set is
in play.

What I want to do in "reduce" my dataset to deal with only distinct
values. I want to collapse these 27 entries below into 3 by taking either
the first or last value of each "set".

I can not find a PromQL function/operator that does what I want. Maybe I
need to use recording rules?

All and any help is greatly appreciated.

metric_name{instance="hostname.example.net", job="external/generic", mode="pull", name="snafu"}

9973997301 @1673997343.774
9973997301 @1673997373.764
9973997301 @1673997403.764
9973997301 @1673997433.764
9973997301 @1673997463.764
9973997301 @1673997493.764
9973997301 @1673997523.764
9973997301 @1673997553.764
9973997301 @1673997583.764

9973997601 @1673997613.764
9973997601 @1673997643.764
9973997601 @1673997673.764
9973997601 @1673997703.774
9973997601 @1673997733.764
9973997601 @1673997763.764
9973997601 @1673997793.764
9973997601 @1673997823.764
9973997601 @1673997853.863

9973997901 @1673997913.764
9973997901 @1673997943.767
9973997901 @1673997973.764
9973997901 @1673998003.764
9973997901 @1673998033.764
9973997901 @1673998063.764
9973997901 @1673998093.764
9973997901 @1673998123.764
9973997901 @1673998153.764

[Stuart Clark]

On 18/01/2023 00:15, Mark Selby wrote:
> I am struggling with PromQL over an issue dealing with a metric that
> changes less frequently than the scrape interval. I am trying to use
> Prometheus as a pseudo event tracker and hoping to get some advice on
> how to best try and accomplish my goal.

I think this is the fundamental issue you are facing. Prometheus isn't
an event system. It is designed for metrics, which are pretty different
to events. It sounds like you should look at a system like Loki,
Elasticsearch or a general purpose SQL or key/value database, as they
are likely to be a much better fit for you than a timeseries database
and ecosystem that is designed for handling metrics.

--
Stuart Clark