Grafana query and Prometheus recorded rules

[Davide D'Amico]

Hi all,

In Grafana I have something like:

avg(

    label_replace(n_http_request_sec {n_instance="$instance"} offset 1d, "offset", "1d", "__name__", ".*") or

    label_replace(n_http_request_sec {n_instance="$instance"} offset 2d, "offset", "2d", "__name__", ".*") or

    label_replace(n_http_request_sec {n_instance="$instance"} offset 3d, "offset", "3d", "__name__", ".*") or

    label_replace(n_http_request_sec {n_instance="$instance"} offset 4d, "offset", "4d", "__name__", ".*") or

    label_replace(n_http_request_sec {n_instance="$instance"} offset 5d, "offset", "5d", "__name__", ".*") or

    label_replace(n_http_request_sec {n_instance="$instance"} offset 6d, "offset", "6d", "__name__", ".*") or

    label_replace(n_http_request_sec {n_instance="$instance"} offset 7d, "offset", "7d", "__name__", ".*")

) without (offset, instance, job)

to compute some anomaly detection (more info on https://groups.google.com/d/msgid/prometheus-users/fe876c1e-ded7-45e4-974e-ddb0b916a675%40googlegroups.com) and it works for my workload, so I am moving that to a recorded prometheus metrics with:

      # Avg over last week's samples this time

      - record: job:nginx_http_req_sec:avg_1w

        expr: avg(

                label_replace(n_http_request_sec offset 1d, "offset", "1d", "__name__", ".*") or

                label_replace(n_http_request_sec offset 2d, "offset", "2d", "__name__", ".*") or

                label_replace(n_http_request_sec offset 3d, "offset", "3d", "__name__", ".*") or

                label_replace(n_http_request_sec offset 4d, "offset", "4d", "__name__", ".*") or

                label_replace(n_http_request_sec offset 5d, "offset", "5d", "__name__", ".*") or

                label_replace(n_http_request_sec offset 6d, "offset", "6d", "__name__", ".*") or

                label_replace(n_http_request_sec offset 7d, "offset", "7d", "__name__", ".*")

            ) without (offset, instance, job)

 

What I see is that there is a "delay" in the data between the grafana panel and the datapoints generated from job:nginx_http_req_sec:avg_1w,

where the datapoints of job:nginx_http_req_sec:avg_1w are the same as the grafana ones shifted forward of 30s.

 

Is this something related to the scrape_interval or to some other interval variable?

 

Relevant prometheus config:

global:

  scrape_interval:     30s

  scrape_timeout:      15s

  evaluation_interval: 30s

Edit: C&P and format.

Thanks in advance,

d.

[Davide D'Amico]

As short term solution I'm recording something like:

label_replace(n_http_request_sec offset 259170s, "offset", "259170s", "__name__", ".*")

instead of:

label_replace(n_http_request_sec offset 3d, "offset", "3d", "__name__", ".*")

where 259170s = 3d - 30s (evaluation_interval)

and it seems working.

d.