Rate and scrape interval

[van...@gmail.com]

Hi everyone.

I have some misunderstanding with scrape interval and rate. I thought that rate doesn't depend on scrape interval, but my test case shows me different result.

Test:

I am producing constant 1rps load

Query: sum(rate(reco_http_requests_count  { response_code = '500'}[1m] )) by (response_code)

1 part: use scrape interval 5 seconds

2 part: use scrape interval 15 seconds

So I see from graph that my results depend on scrape interval. Is it truth?

[van...@gmail.com]

I expect to see line around 1 rps

[Ben Kochie]

I assume this sum is necessary because you have a number of backends that need to be aggregated.

Because it's only 1qps, depending on how many instances you have, what you're seeing is interpolation noise.  For slow moving counters Prometheus has to use interpolation in order to estimate the real rate per second over 1 minute in your case.

This is why you're seeing funny results.

If you increase the simulation to get about 1 request per second per instance, things should smooth out.

On Thu, Apr 20, 2017 at 10:26 AM, <van...@gmail.com> wrote:

I expect to see line around 1 rps

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bd8ff708-626c-44b1-8fff-2225be34713f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[van...@gmail.com]

Thanks for fast reply.

I don't think that it is related with multiple nodes.

But I think that I found problem:

We are using kamon and kamon-prometheus in our application.

There should be invariant kamon.metric.tick-interval <= kamon.prometheus.refresh-interval >= prometheus scrape interval

It was violated.

On Thursday, April 20, 2017 at 12:30:32 PM UTC+4, Ben Kochie wrote:

I assume this sum is necessary because you have a number of backends that need to be aggregated.

Because it's only 1qps, depending on how many instances you have, what you're seeing is interpolation noise.  For slow moving counters Prometheus has to use interpolation in order to estimate the real rate per second over 1 minute in your case.

This is why you're seeing funny results.

If you increase the simulation to get about 1 request per second per instance, things should smooth out.

On Thu, Apr 20, 2017 at 10:26 AM, <van...@gmail.com> wrote:

I expect to see line around 1 rps

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Brian Brazil]

On 20 April 2017 at 10:16, <van...@gmail.com> wrote:

Thanks for fast reply.

I don't think that it is related with multiple nodes.

But I think that I found problem:

We are using kamon and kamon-prometheus in our application.

There should be invariant kamon.metric.tick-interval <= kamon.prometheus.refresh-interval >= prometheus scrape interval

It was violated.

kamon-prometheus is collecting data independent of when a scrape happens, this leads to weirdness like this. Switching to something that only collects data at scrape time would remove this problem.

Brian

 

On Thursday, April 20, 2017 at 12:30:32 PM UTC+4, Ben Kochie wrote:

I assume this sum is necessary because you have a number of backends that need to be aggregated.

Because it's only 1qps, depending on how many instances you have, what you're seeing is interpolation noise.  For slow moving counters Prometheus has to use interpolation in order to estimate the real rate per second over 1 minute in your case.

This is why you're seeing funny results.

If you increase the simulation to get about 1 request per second per instance, things should smooth out.

On Thu, Apr 20, 2017 at 10:26 AM, <van...@gmail.com> wrote:

I expect to see line around 1 rps

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bd8ff708-626c-44b1-8fff-2225be34713f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1b04a798-57bc-45b8-badf-689f7f232795%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 20 April 2017 at 11:43, Evgeny Vanslov <van...@gmail.com> wrote:

Brian, thanks for response.

I agree and will try to find better library instead of kamon, but now I have to fix this issue.

There isn't one publicly available that I'm aware of if it's the Akka stats you care about. Switching that bit of the code to be instrumented with client_java would be the best approach. I'm aware of someone working on this, but there's no signs yet that it'll become open source.

if it's not the Akka stats you're interested in, using client_java directly would be the recommended approach.

Unfortunately, fix of intervals haven't helped.

Current question:

reco_http_requests_count{response_code="500",request_type="GET",kamon_category="histogram",kamon_name="http_requests"} 70.0 1492684842737

if prometheus got as response (as text format) such string, does prometheus think that it is current value of counter or difference from last scrape?

Current value, that's always the case for Prometheus counters. See https://www.youtube.com/watch?v=67Ulrq6DxwA

Brian

--

Brian Brazil

www.robustperception.io

[van...@gmail.com]

Thanks for very interesting video.

As I understood: i can't use kamon library with prometheus properly, because kamon resets counters on each snapshot (Prometheus can handle restarts, but it isn't able to distinguish restart and normal case if current metric is greater than previous)

Is it true?

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1b04a798-57bc-45b8-badf-689f7f232795%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 20 April 2017 at 15:01, <van...@gmail.com> wrote:

Thanks for very interesting video.

As I understood: i can't use kamon library with prometheus properly, because kamon resets counters on each snapshot (Prometheus can handle restarts, but it isn't able to distinguish restart and normal case if current metric is greater than previous)

Is it true?

There's ways of making it kinda-work, but it's never going to be as good as using a Prometheus client library.

Brian

 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d683804e-ef1c-4add-8405-d52dcb24d718%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[van...@gmail.com]

Can you share available workarounds? I would like to understand all our possible solutions (prometheus client, kamon with workaround, cinnamon) 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d683804e-ef1c-4add-8405-d52dcb24d718%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 20 April 2017 at 18:53, <van...@gmail.com> wrote:

Can you share available workarounds? I would like to understand all our possible solutions (prometheus client, kamon with workaround, cinnamon) 

How the existing prometheus-kamon integration works is the workaround.

Brian

 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2bdd39fc-bee1-4445-a45d-585a92d216c7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[fann...@gmail.com]

I found using kamon-akka and kamon-prometheus frustrating and went and forked kamon-akka to natively record the metrics as prometheus metrics. I'm talking to my employers about open sourcing this.