Prometheus and Grafana - same query but different result?

[Rafał Radecki]

Hi All.

I am currently configuring graphs in grafana based on metrics from prometheus. I have a problem that fot the same query I get different results:

Grafana:

sum(increase(hadoop_nodemanager_containerskilled{env="uat",unit="si"}[24h])) -> 19

Prometheus:

sum(increase(hadoop_nodemanager_containerskilled{env="uat",unit="si"}[24h])) -> 35

Additionally in Prometheus I see:

query:

increase(hadoop_nodemanager_containerskilled{env="uat",unit="si"}[24h])

result:

{component="yarn",env="uat",instance="node2:40001",job="reactvar",name="NodeManagerMetrics",unit="si"} 3.0006688819090885

{component="yarn",env="uat",instance="node1:40001",job="reactvar",name="NodeManagerMetrics",unit="si"} 5.001834806853806

{component="yarn",env="uat",instance="node3:40001",job="reactvar",name="NodeManagerMetrics",unit="si"} 19.012688825216884

{component="yarn",env="uat",instance="node4:40001",job="reactvar",name="NodeManagerMetrics",unit="si"} 8.002809822812276

I tried to set different "step" and "resolution" for Grafana but this has not helped. In Prometheus scrape_interval=30s.

Can someone help?

BR,

Rafal.

[Julius Volz]

You haven't mentioned what kind of widget (singlestat?) you're using to display the query in Grafana, but my guess is that it's requesting the query for a different time (or time range) than Prometheus is. If you're using singlestat, I think it actually does a range query instead of an instant query in the background, and then averages over it or something (it's configurable in the Grafana UI). Also, you can open your browser's network inspector and see what query Grafana and Prometheus really send to the backend.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a48043e5-9d83-420a-85b4-bc1f6f80ec4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julius Volz]

Ah, you did mention graphs, but then you mention a single output number. Are the time ranges aligned though?