Different results from same query on executing twice

[Ashmita]

Hi,

I am executing same query twice producing 2 different results. I have cross checked the timeframes and filters, all inputs seem correct but the results deviate too much between the executions.

What can be the possible cause of this?

Query:

process_cpu_usage{container='ec-xx-xx-v1'}

http://xx.xx.xx.xx/api/v1/query_range?start=1632127500&end=1632127860&step=10s&query=process_cpu_usage%7Bcontainer%3D%27ec-xx-xx-v1%27%7D

Result 1: 

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"process_cpu_usage","container":"ec-xx-xx-v1","endpoint":"web","instance":"xx.xx.xx.xx:8080","job":"ec-xx-xx","namespace":"ec","pod":"ec-xx-xx-v1-7f67465644-sl5nz","service":"ec-xx-xx"},"values":[[1632127500,"0.00026984992218995554"],[1632127510,"0.00022961338012735005"],[1632127520,"0.00023518311282263695"],[1632127530,"0.00023518311282263695"],[1632127540,"0.00023694847682949357"],[1632127550,"0.0002235921484748602"],[1632127560,"0.0002235921484748602"],[1632127570,"0.00022786081552209455"],[1632127580,"0.00023294510124391955"],[1632127590,"0.00023294510124391955"],[1632127600,"0.00022904436055841393"],[1632127610,"0.0002454010621767065"],[1632127620,"0.0002454010621767065"],[1632127630,"0.00022646683320757025"],[1632127640,"0.00024127932343283401"],[1632127650,"0.00024127932343283401"],[1632127660,"0.0002744682928706199"],[1632127670,"0.000271482408315234"],[1632127680,"0.000271482408315234"],[1632127690,"0.00024818152333497956"],[1632127700,"0.0004945948675971917"],[1632127710,"0.0004945948675971917"],[1632127720,"0.000250044588666984"],[1632127730,"0.00025306061460246424"],[1632127740,"0.00025306061460246424"],[1632127750,"0.0002324170178636118"],[1632127760,"0.00023710522919737198"],[1632127770,"0.00023710522919737198"],[1632127780,"0.00025918896512630877"],[1632127790,"0.00025716835976627466"],[1632127800,"0.00025716835976627466"],[1632127810,"0.00023743275328602447"],[1632127820,"0.0004958346818344523"],[1632127830,"0.0004958346818344523"],[1632127840,"0.0002374846750827883"],[1632127850,"0.0002943738539537926"],[1632127860,"0.0002943738539537926"]]}]}}

Result 2: 

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"process_cpu_usage","container":"ec-xx-xx-v1","endpoint":"web","instance":"xx.xx.xx.xx:8080","job":"ec-xx-xx","namespace":"ec","pod":"ec-xx-xx-v1-7f67465644-sl5nz","service":"ec-xx-xx"},"values":[[1632127500,"0.0000911854823590461"],[1632127510,"0.00008031800718621397"],[1632127520,"0.00009711552383552008"],[1632127530,"0.00009711552383552008"],[1632127540,"0.00007739860289849959"],[1632127550,"0.00009063076981304917"],[1632127560,"0.00009063076981304917"],[1632127570,"0.00007699932224869722"],[1632127580,"0.00012044574767573395"],[1632127590,"0.00012044574767573395"],[1632127600,"0.00008063313901524975"],[1632127610,"0.00009366396274970667"],[1632127620,"0.00009366396274970667"],[1632127630,"0.00007783129262198461"],[1632127640,"0.00008724516058957322"],[1632127650,"0.00008724516058957322"],[1632127660,"0.00008264823524951832"],[1632127670,"0.00009466906082607281"],[1632127680,"0.00009466906082607281"],[1632127690,"0.00008071745763340187"],[1632127700,"0.00008996126176758427"],[1632127710,"0.00008996126176758427"],[1632127720,"0.00007900892075375929"],[1632127730,"0.00008954506478524276"],[1632127740,"0.00008954506478524276"],[1632127750,"0.00008100053071147762"],[1632127760,"0.0000883203670154625"],[1632127770,"0.0000883203670154625"],[1632127780,"0.00007852370060172425"],[1632127790,"0.0000949312524493128"],[1632127800,"0.0000949312524493128"],[1632127810,"0.00007614114114890825"],[1632127820,"0.0000903200674807327"],[1632127830,"0.0000903200674807327"],[1632127840,"0.00007734656080272437"],[1632127850,"0.00009295713215565754"],[1632127860,"0.00009295713215565754"]]}]}}

[Bjoern Rabenstein]

On 27.09.21 07:16, Ashmita wrote:
>
> The above data is being requested from 2 prometheus instances which are
> load balanced over OpenShift. Any idea on how can we modify the API request
> to fetch data from either one of them or receive complete set of values
> from all instances to calculate aggregated value?

You could try to find out if your loadbalancer has any option for
sticky sessions or consistent routing so that you will hit the same
Prometheus server as long as it is still up.

Prometheus itself is deliberately not a distributed system and has no
built-in mechanims to stitch together replies from different
Prometheus servers. One idea behind https://thanos.io/ was to provide
such a layer. So you could look into that. More generally, you might
want to check out the many remote storage solutions:
https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in