Prometheus exporter collection behaviour

[Arun Raj]

hi All,

   We are running prometheus server in our testing environment which scraps 3 different exporters.( node exporter, kafka exporter and custom exporter). 

   1. All the three exporters are pulling its metrics in different timing even though the scrap interval are same. Is it expected behaviour ?

   2. Our application's very time sensitive, we would like to scrap for every 60 seconds. But we could see the collection time difference are like 61 seconds or 62 seconds at sometimes. Is there any way to make it consistent ? Or Is it expected behaviour ?

Regards,

Arun Raj. R

[Stuart Clark]

Both are expected.

The scrape interval is an average, so gaps between scrapes might be a bit longer or shorter each time. Different scrapes with the same scrape interval will happen at different times to prevent everything happening at once, overwhelming the server and network.

Ideally most metrics should be counters rather than gauges which are a lot more resistant to slight differences in timings between scrapes and across jobs & targets.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Arun Raj]

hi Clark,

    Our applications are telco based applications. So far, they have been using push model for every 5 minutes. The every 5 minute is a telecom standard guideline. We thought of consume those files every 5 minutes from our custom exporter. We can't miss the second, Is there any way in prometheus to accomplish this ?

Regards,

Arun Raj. R

[Stuart Clark]

On 2020-02-16 15:06, Arun Raj wrote:
> hi Clark,
>
> Our applications are telco based applications. So far, they have
> been using push model for every 5 minutes. The every 5 minute is a
> telecom standard guideline. We thought of consume those files every 5
> minutes from our custom exporter. We can't miss the second, Is there
> any way in prometheus to accomplish this ?
>

The maximum scrape rate for Prometheus is about 2 minutes due to
staleness.

Can you switch to say scraping every minute?

> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/db38d816-c77d-46ff-be54-0cde43e27ac1%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/prometheus-users/db38d816-c77d-46ff-be54-0cde43e27ac1%40googlegroups.com?utm_medium=email&utm_source=footer

--
Stuart Clark

[Julien Pivotto]

To be fair at my current place we have a Prometheus instance for our Network team with a scrape interval of 5 minutes; and we have just adapted 2 things:

1. Loopback delta is set to 15 minute
2. We use irate, not rate.

However this Prometheus instance is isolated and is only doing this.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/be85e01278a5f6a8ec15861bf0d833d3%40Jahingo.com.

[Ben Kochie]

Scrape interval is not an average.

It should be very consistent. The timing is controlled from the start of the scrape, not the end. A typical healthy Prometheus setup will be on the exact same millisecond each time.

For example, you can see the scrape interval is on the exact same millisecond by looking at the `up` metric for a task. Prometheus assigns the timestamp of the metrics based on the start of the scrape, not the completion.

https://prometheus.demo.do.prometheus.io/graph?g0.range_input=1h&g0.expr=up%7Bjob%3D%22prometheus%22%7D%5B5m%5D&g0.tab=1

What you're probably talking about is the splay between tasks. The Prometheus server attempts to spread out scrapes over the scrape interval by applying a consistent hash to the list of targets.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9387C3B3-BF29-45E3-81A7-0CD5C9F5296C%40Jahingo.com.