Why Prometheus recording rule's result can differ from manual query?

[Alexandre Figura]

Hello all,

I have the following recording rule:

record: istio_slo:error_rate:ratio_rate2w
expr: |
  sum(rate(istio_requests_total{destination_app!="unknown", response_code=~"5.*"}[2w]))
  by (destination_service_namespace, destination_app, destination_service_name)
  /
  sum(rate(istio_requests_total{destination_app!="unknown"}[2w]))

  by (destination_service_namespace, destination_app, destination_service_name)

If I run manually the query inside the recording rule, I get a result different from the one returned by the recording rule itself:

Here is how I use the recording rule (in Grafana):

istio_slo:error_rate:ratio_rate2w{destination_service_namespace="$namespace", destination_app="$app", destination_service_name=~"$service"}

And the query:

sum(rate(istio_requests_total{destination_service_namespace="$namespace", destination_app="$app", destination_service_name=~"$service", response_code=~"5.*"}[2w]))  
/
sum(rate(istio_requests_total{destination_service_namespace="$namespace", destination_app="$app", destination_service_name=~"$service"}[2w]))

How would you explain such difference between results?

Thanks,

Have a nice day.

[Bjoern Rabenstein]

On 04.03.20 03:07, Alexandre Figura wrote:
>
> How would you explain such difference between results?

What's your retention time? If it is 15d (the default), then the
recording rule would actually see all the 2w of data you are querying
over in the `rate` range (because it is executed regularly for the
current state of your data). However, in your manual query, you are
executing the query for data in the past that now doesn't have enough
history anymore to really fill the 2w range selection.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in