Staggering scrapes to different targets?

[John Dexter]

Hello all,

I was thinking that if you use the same scrape-interval for many targets, doesn't this mean every 15/30/60s your Prometheus server sends out a huge barrage of requests all at once, potentially to many modules on the same server?

I wondered if this can lead to any issues with network spikes and so on, and if Prometheus allows/encourages all the various scrapes to be staggered slightly so there is a constant stream of polls not so 'lumpy'? Maybe it naturally works this way anyway, that polls are run sequentially rather than in parallel?

Thanks for any insight,

John.

[Stuart Clark]

Prometheus will ensure that a specific target is scraped every 15/30/60s
as configured, but other than the period between scrapes being fixed the
exact "second past the minute" will differ between targets. So for
example if you had two targets which had a scrape interval of 60s, you
might find that the scraping happened at 10s past the minute for target
1 and 40s past the minute for target 2.

--
Stuart Clark

[Ben Kochie]

Prometheus already does this. Each target within a job is scheduled in a separate goroutine (lightweight thread), with a millisecond resolution offset generated by the hash of the target labels.

If you query for `time() - timestamp(up{job="some-job"})`, you can see exactly how the offset works.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGJRanhhPZ26Zit71-BZWwmn4VZYrd6fp7%3DM6ipnqSvefpx4kA%40mail.gmail.com.

[Jiacai Liu]

AFAIK, prometheus depend on scrape interval to determine timestamp
of datapoints, if it schedules scrape at different interval, this
assume may not be held.

On Wed, 03 Mar 2021 at 00:43, John Dexter <jo...@jdxsolutions.com>
wrote:

[Stuart Clark]

On 03/03/2021 08:39, Jiacai Liu wrote:
> AFAIK, prometheus depend on scrape interval to determine timestamp of
> datapoints, if it schedules scrape at different interval, this assume
> may not be held.

Scrapes will always be made at the interval configured. It is just
"when" a scrape happens that varies between targets - you won't have all
targets being scraped at the same time, but they all will be being
scraped exactly the configured scrape interval after the previous one.

--
Stuart Clark