Graph Issues grafana and peometheus

[Tareerat Pansuntia]

hi, everyone

I have configured Prometheus to scrape data every 30 minutes. This means Prometheus will scrape data every thirty minutes and then display it in Grafana. However, the graph in Grafana shows the data for only a brief moment and then disappears. After thirty minutes, the graph reappears again. What could be causing this issue, and how can I fix it?

 In the graph , the green line appears every thirty minutes and then disappears until the next thirty-minute interval. I have set the scrape_interval for that website to every 30 minutes. What could be causing the data to disappear in between, and how can I fix this issue?  

Thanks in advance,

[Ben Kochie]

This is Prometheus staleness handling. In order to allow queries to function, Prometheus graph query evaluation (range query) is actually a series of query evaluations over time. Each one is independent of the next.

In order to support millisecond accurate timestamps, Prometheus will look back up to 5 minutes for samples from each evaluation timestamp.

There are a couple ways around this.

* Scrape faster. It's perfectly common and normal to scrape every 15s-60s in Prometheus. It basically doesn't take any more storage space due to the way Prometheus compresses samples. In fact, scraping slower wastes data by creating incomplete blocks.

* Use a function like `avg_over_time(metric[$__rate_interval])` to effectively increase the lookback. This needs to be combined with setting the "min step" in the Grafana query options to 30min to match your scrape interval.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/13d491cd-f8c0-42a3-bb3e-9f5a0fb94454n%40googlegroups.com.

[Ben Kochie]

The easiest thing to do is change your scrape interval to 1m. This would eliminate the problem. Don't worry, this is a perfectly normal interval for Prometheus.

On Wed, Jul 24, 2024 at 4:29 AM Tareerat Pansuntia <tareera...@gmail.com> wrote:

  And I would like to know if Prometheus can store metrics for only five minutes or up to five minutes before they disappear. Why does the graph show gaps at certain intervals?"

  Thanks in advance,   

ในวันที่ พ. 24 ก.ค. 2024 เวลา 09:20 Tareerat Pansuntia <tareera...@gmail.com> เขียนว่า:

Thank you for help me

but

 I am still a beginner in this field. Could you please help me understand how to use the avg_over_time(metric[$__rate_interval]) function? I will look for more information on my own afterward. For the metric in question, if I use probe_success, should I write it as avg_over_time(probe_duration_seconds{instance="xx", job="blackbox_exporter"}[30m])?  

Cloud you recommend me about function prometheus ?

Thanks in advance,   

ในวันที่ พฤ. 18 ก.ค. 2024 เวลา 15:10 Ben Kochie <sup...@gmail.com> เขียนว่า: