How to configure staleness

[Dave Cadwallader]

From: https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness

If no sample is found (by default) 5 minutes before a sampling timestamp, no value is returned for that time series at this point in time. 

Is it possible to change this default on a per-metric basis?  I have one metric that I only need to scrape every hour.  If I query in the few minutes after the scrape, the data shows up.  But otherwise the metrics are missing.  

If I'm thinking about this all wrong, please let me know. :)  The scrape is relatively costly on the exporter side, so I would rather not increase the scrape interval.

[Ben Kochie]

I think this documentation is obsolete.  Prometheus 2.0 has a very different staleness handling method.

See: https://promcon.io/2017-munich/talks/staleness-in-prometheus-2-0/

Either way, Prometheus is not going to handle hourly scraping well, the practical upper limit of scrape interval is 2 minutes.  I would recommend changing the way your exporter works, I would probably do something like run it as a cron job and use the pushgateway or node_exporter textfile collector.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/185c4fba-583e-45ea-8532-aa2ff72b414e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Kochie]

Another option is to use a query like this:

max_over_time(metric[1h])

This will make Prometheus search backwards 1 hour for data.

[Stevie]

Is there any way that I can know the metrics on PushGateWay are stale ?

On Friday, May 11, 2018 at 2:45:14 PM UTC+8, Ben Kochie wrote:

I think this documentation is obsolete.  Prometheus 2.0 has a very different staleness handling method.

See: https://promcon.io/2017-munich/talks/staleness-in-prometheus-2-0/

Either way, Prometheus is not going to handle hourly scraping well, the practical upper limit of scrape interval is 2 minutes.  I would recommend changing the way your exporter works, I would probably do something like run it as a cron job and use the pushgateway or node_exporter textfile collector.

On Fri, May 11, 2018 at 8:29 AM, Dave Cadwallader <dcadwa...@gmail.com> wrote:

From: https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness

If no sample is found (by default) 5 minutes before a sampling timestamp, no value is returned for that time series at this point in time. 

Is it possible to change this default on a per-metric basis?  I have one metric that I only need to scrape every hour.  If I query in the few minutes after the scrape, the data shows up.  But otherwise the metrics are missing.  

If I'm thinking about this all wrong, please let me know. :)  The scrape is relatively costly on the exporter side, so I would rather not increase the scrape interval.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Cody Boggs]

I'm not as well-versed with the push gateway, but as far as I know staleness is only relevant from prometheus' perspective, not the gateway's. Prometheus marks a series stale if it hasn't been present for the last couple fo scrapes of the target whence it came.

Also, unless my memory fails, the push gateway has no concept of staleness or expiration, and keeps all things until its restarted.

Cody Boggs 

Freshtracks.io

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7e84ac14-8e79-48f2-93d7-ecc05e712d08%40googlegroups.com.