Now if I wait some time (up to a minute) and hit refresh, I'll see this:
I.e. counter is growing the same way, but rate returns zeroes for all but a major increases.
Here are my queries:
Green: 60 * sum(rate(api_calls_duration_seconds_count[1m]))
Yellow: 60 * sum(avg_over_time(api_calls_duration_seconds_count[1m]))
Moreover, if I try to calculate latency (red) by dividing _sum by _count, it only comes up for the second major spike, even when rate() gets it right:
Here is the query for latency:
sum(increase(api_calls_duration_seconds_sum[1m]))
/
sum(increase(api_calls_duration_seconds_count[1m]))
Another way to demonstrate it:
$ curl -s 'http://localhost:9090/api/v1/query_range?query=60%20*%20sum(rate(api_calls_duration_seconds_count%5B1m%5D))&start=1515031319&end=1515034907&step=60' | grep -o '"0"' | wc -l
46
$ curl -s 'http://localhost:9090/api/v1/query_range?query=60%20*%20sum(rate(api_calls_duration_seconds_count%5B1m%5D))&start=1515031320&end=1515034907&step=60' | grep -o '"0"' | wc -l
58
I.e. I moved start time 1 second forward and there are 12 more zero points in the result - i.e. all those small pics from the first screenshot are gone.
Any ideas what's happening here?
Thanks,
Zaar
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fedb1ff1-c9f9-43ef-a24a-c98a6969f32b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
$ curl -s 'http://localhost:9090/api/v1/query_range?query=60%20*%20sum(rate(api_calls_duration_seconds_count%5B1m%5D))&start=1515031319&end=1515034907&step=60' | grep -o '"0"' | wc -l
With start values from 1515031310 to 1515031390, and rate windows of 60, 61 and 67 seconds (step remained on 60 seconds for all 3). Here are the results: ("." signifies OK data, "!" - missing data, i.e. mostly zeroes):
60: ..........!!!!!!!!!!..................................................!!!!!!!!!!.
61: ...........!!!!!!!!!...................................................!!!!!!!!!.
67: .................!!!.........................................................!!!.
It also quite puzzling why my latency query returns mostly nothing.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
Hey Zaar,In case you're still looking for an answer, the one I found was to useincrease(foo[70s]) * 60 / 70wherever I would have wanted to use increase(foo(60s]) instead (all this assuming a 10s collection interval). The reasoning behind it is to compute an increase over 6 successive collection intervals (i.e. 7 successive data points, or 70s). But Prometheus will see that it's actually (on average 60s worth of data) and extrapolate that to 70s. Hence the * 60 / 70 part at the end to undo that extrapolation. It is not obvious by any stretch of the imagination, but it will produce much better numbers than increase().Or you can dofoo - foo offset 1mbut that will not take into account counter resets (if you care about them at all) and will take twice as much CPU.Cheers,Alin.
--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/GLKaSYQN_gI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0a0771ab-6f0b-49ab-8e88-82a75de85256%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b1fe227c-9696-45de-9f2d-3ee4b69c8eb7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Whoever runs into this issues and lands here - here is the article (not mine) that outlines the issue very well (scroll to the last part): https://www.stroppykitten.com/technical/prometheus-grafana-statistics