keep stalled metric until prometheus scrapes new value for the metric

[nesa...@gmail.com]

Hi,

We have a program which exports metric whenever an event happens in the system, so we are generating a metric like this:

custom_metric{id="39a"} 1

and once another event happens in the system, a new metric generates like this:

custom_metric{id="15c"} 1

The problem is, since this metric is generated once in a while, the metrics get stalled after 5 minutes and e.g. there will be a gap between id=39a and id=15c when I want to see the history of "custom_metric", like the picture:

Is it possible to tell prometheus to keep the last metric until new one scraped by prometheus so the graph would be connected

[Stuart Clark]

Is that ID something to do with the event or related to something else?

Label should be very carefully used to ensure cardinality is controlled. Things like user IDs, IP addresses or other unique references should not be used.

If that is a unique ID that label should be removed. If you are needing information at the per event level you would want an events sys3(such as Elasticsearch or Loki) rather than Prometheus (which is for metrics).

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[nesa...@gmail.com]

I'm not sure if it is a bad idea to use Prometheus, as this is some sort of software version and if you take a look at this article, it explains on how to expose software version to the Prometheus:

https://www.robustperception.io/exposing-the-software-version-to-prometheus

[Brian Candler]

On Sunday, 18 October 2020 16:27:56 UTC+1, nesa...@gmail.com wrote:

Is it possible to tell prometheus to keep the last metric until new one scraped by prometheus so the graph would be connected

No, because these are two different timeseries.

You say "this metric is generated once in a while", but don't say how it's generated or scraped.

To avoid the staleness problem, you need to scrape every 2 minutes or faster.  You don't necessarily need to *generate* it every 2 minutes - you could run a cronjob every 30 minutes (say) which writes the metric into a file, which is exported by node_exporter textfile collector.  But you need to *scrape* it every 2 minutes.