Thanos Query APIs are returning different values, depending upon the selection of start time and end time.

[Kishore Kumar]

Hello Prometheus Community,

          I hope the person reading this has an awesome day, and thanks for helping out.

I have a thanos query API endpoint, which if I query for a smaller start and end timestamp, it is giving a different value compared to the same query which has a bigger difference between start and end timestamps. I request to know what might be causing this unexpected behaviour?

1) Query with 6 second difference between start and end timestamp:

<endpoint>/thanosquery/api/v1/query_range?query=sum(increase(envoy_cluster_upstream_rq

{

kubernetes_namespace="sample_namespacce",

kubernetes_pod_name="sample_pod_name",

envoy_cluster_name="sample_cluster_name",

}[3m])/3) 

by (kubernetes_namespace,envoy_cluster_name)

&start=1731304828&end=1731304833&step=1s

2) Query with 2 minutes difference between start and end timestamp:

<endpoint>/thanosquery/api/v1/query_range?query=sum(increase(envoy_cluster_upstream_rq

{

kubernetes_namespace="sample_namespacce",

kubernetes_pod_name="sample_pod_name",

envoy_cluster_name="sample_cluster_name",

}[3m])/3) 

by (kubernetes_namespace,envoy_cluster_name)

&start=1731304800&end=1731304860&step=1s

Thank you for your time,

Kishore.

[Brian Candler]

Can you show what exactly is different in the output between the two cases, that you don't expect to be different?

[Kishore Kumar]

Hi Brian,

         I'm attaching responses for the previously mentioned queries. The only differences between query1 and query2 are start and end timestamps.

Lets say for a particular timestamp 1731304830 (present in both the json files)

Query1 (5 seconds window) returns a value 764.0958754210372

Query2 (60 seconds window) returns a value 42.8

As per my use case, regardless of the start and end timestamp, the value returned should be the same, as my step frequency is 1 second in both cases

Setting max_source_resolution=0s also does not seem to help.

Thank you,

Kishore.