delta in promql vs Grafana delta on dashboard

[Helena Lundell]

I am trying to identify an alert for an increasing Kafka Lag. The metrics are in prometheus and I use Grafana 9.4 for visualisation.

In Grafana I have a dasboard where I can see that the lag is increasing or decreasing. I have choosen to display the delta in the legend and it shows a number that does not really make sense, but it shows something.

This is the base promql:
avg by(consumergroup, topic, cluster) (kafka_consumergroup_lag{namespace="ns-kafka-int", consumergroup=~".*$container", cluster="$cluster"})
 
When I add delta to this query I get no data:
avg by(consumergroup, topic, cluster) (delta(kafka_consumergroup_lag{namespace="ns-kafka-int", consumergroup=~".*$container", cluster="$cluster"}[$__interval]))

So adding delta gives me no result even though when I exclude delta I can see that the metric has been changing.

The metric is a gaugue.

Question:
Am I doing something wrong?
Can I trust the delta from the Grafana dashboard(someone might know)?
Can I expect a positive or negative value to identify and increase?

[Ben Kochie]

Grafana defaults to a "minimum step" of 15 seconds. I bet your scrape interval is longer than 15 seconds, so "$__interval" is matching no data.

You can either adjust the datasource minimum step to your shortest scrape interval, or set the "min step" in the query options to match the scrape interval of your metric.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0f37aa72-dd90-4329-8e19-5587460b80c4n%40googlegroups.com.

[Helena Lundell]

Thanks for the fast reply and sorry for my late response.

We have a scraping on 15 seconds. My query interval is 30 min. So I should have data. And the Grafana dashboard below where I add difference(changed from delta) on legend values I see what I want.  

This difference from the Grafana panel is exactely what I am looking for and thought delta would give me. 

Is there an alternative to delta? Like first-last?  

[Ben Kochie]

Use `$__rate_interval` to make sure you get enough samples for `delta()` to work. (delta(), rate(), increase(), function similarly).

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/63cf733e-87bd-4530-9531-5a3a29556d8an%40googlegroups.com.

[Helena Lundell]

I was using $_interval since delta is for gauge. I now changed to $_rate_interval and it makes no difference whatsoever. I also really made sure to set an interval where I know I have a lag I want to catch. 

Am I really understanding delta correctly. 

I expect to get the difference between first value in the series and the last value of the series. 

If I start with 0, lots happens in between, lag goes down to 0 by the end of my interval. I expect to get 0. 

If I start on 0, lag increases to 10, I expect to get 10

If I start on 10 and goes down to 2, I expect to -8