Why there are spikes when the counter increase at the same rate?

stav alfi

unread,

Oct 24, 2020, 3:50:25 PM10/24/20

to Prometheus Users

Intro

I have a counter that is increasing by 1 every 1 second.

the scrape interval is 1 second as well

as you can see - pretty normal.

I would expect that the `rate(test_total[1m])` function will have the same result over time, but it doesn't:

Question:

Why the rate isn't equal to 1 exectly on each timestamp? why there are spikes?

Ben Kochie

unread,

Oct 24, 2020, 4:34:02 PM10/24/20

to stav alfi, Prometheus Users

Can you share the raw data?

curl -s 'http://localhost:9090/api/v1/query?query=test_total[15m]'

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/701610c2-8c97-4b38-a386-db9ba0040fbdn%40googlegroups.com.

stav alfi

unread,

Oct 24, 2020, 6:45:26 PM10/24/20

to Prometheus Users

here is an update graph with the raw data:

~ curl -s 'http://localhost:9090/api/v1/query?query=test_total[15m]'

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"test_total","instance":"host.docker.internal:3000","job":"prometheus"},"values":[[1603579215.318,"2503"],[1603579216.315,"2504"],[1603579226.318,"2514"],[1603579231.316,"2519"],[1603579240.509,"2528"],[1603579241.508,"2529"],[1603579243.508,"2531"],[1603579249.510,"2537"],[1603579254.511,"2542"],[1603579256.508,"2544"],[1603579269.475,"2557"],[1603579270.476,"2558"],[1603579283.474,"2571"],[1603579288.474,"2576"],[1603579289.474,"2577"],[1603579291.474,"2579"],[1603579292.475,"2580"],[1603579295.474,"2583"],[1603579315.442,"2603"],[1603579319.443,"2607"],[1603579321.440,"2609"],[1603579322.441,"2610"],[1603579323.439,"2611"],[1603579324.439,"2612"],[1603579341.405,"2629"],[1603579343.405,"2631"],[1603579349.407,"2637"],[1603579350.405,"2638"],[1603579353.407,"2641"],[1603579354.407,"2642"],[1603579364.371,"2652"],[1603579373.371,"2661"],[1603579380.370,"2668"],[1603579397.337,"2685"],[1603579399.336,"2687"],[1603579406.337,"2694"]]}]}}

-----------

by looking at the raw data, it seems that even though I set promethues to scrape my app every 1 sec, it doesn't do that (?) :

global:

scrape_interval: 1s

evaluation_interval: 1s

Brian Candler

unread,

Oct 25, 2020, 3:32:17 AM10/25/20

to Prometheus Users

It could be that some of your scrapes are failing. With a 1 second scrape interval, the scrape time limit will be 0.5 seconds by default, I believe.

What does a graph of

up{job="prometheus",instance="host.docker.internal:3000"}

scrape_duration_seconds{job="prometheus",instance="host.docker.internal:3000"}

show? What happens if you change your scrape interval to 10 seconds?

How are you generating this "test_total" metric, i.e. what exporter is running on port 3000? And can you show the definition of the "prometheus" scrape job in your prometheus.yml ? (Aside: it would be normal to give the scrape job a more descriptive name, since you're not scraping prometheus itself)

Brian Brazil

unread,

Oct 25, 2020, 4:05:45 AM10/25/20

to Brian Candler, Prometheus Users

On Sun, 25 Oct 2020 at 07:32, Brian Candler <b.ca...@pobox.com> wrote:

It could be that some of your scrapes are failing. With a 1 second scrape interval, the scrape time limit will be 0.5 seconds by default, I believe.

It'll be a second. It's only the blackbox exporter that subtracts out half a second (by default), as if the probe times out there's still useful metrics about how it failed that we want to try to get back to Prometheus.

Brian

What does a graph of

up{job="prometheus",instance="host.docker.internal:3000"}
scrape_duration_seconds{job="prometheus",instance="host.docker.internal:3000"}

show? What happens if you change your scrape interval to 10 seconds?

How are you generating this "test_total" metric, i.e. what exporter is running on port 3000? And can you show the definition of the "prometheus" scrape job in your prometheus.yml ? (Aside: it would be normal to give the scrape job a more descriptive name, since you're not scraping prometheus itself)

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fb295a71-7cfa-4101-bd32-53a3add4999ao%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

stav alfi

unread,

Oct 25, 2020, 4:55:20 PM10/25/20

to Prometheus Users

I can't reproduce it anymore. the graph is valid (I think).

I will update this thread if there will be any change. Thank you all for the help.

--------------

curl -s 'http://localhost:9090/api/v1/query?query=counter_test[15m]'

Reply all

Reply to author

Forward