kubernetes deployed prometheus has no "gaps" after upgrades

[Mario Cornaccini]

hi,

i just updated our kube-prometheus custom ressource, which manages a deployment of two prometheus pods, which flawlessly upgraded the pods one after each other..(deployment strategy)

but with grafana i see no gaps in my graphs.
grafana has the datasource, which points to the k8s service, which is a "loadbalancer", which has two endpoints; one for each prometheus pod. (maybe thats a question for the grafana mailinglist then)

i take it that my two prometheus do not sync timeseries; 

(as opposed to alertmanager which does gossip)

das anybody know why there seem to be no gaps in prometheus scraping timeseries ?
although i do see a gap when i look at each pods' cpu-usage. of course, it's because they got restarted.

just curious..
cheers,

fil

[Brian Candler]

> grafana has the datasource, which points to the k8s service, which is a "loadbalancer", which has two endpoints; one for each prometheus pod.

Then each query from grafana goes to *either* prometheus A *or* prometheus B.  If both prometheus servers are scraping the same endpoints then they'll have roughly the same data in them, so you'll see roughly the same graphs.

However, using a loadbalancer in this way is not recommended, as it's not deterministic which prometheus server your query will hit.

I suggest you look at promxy, which is designed for this use case and can do useful things like filling any gaps from data in prometheus A with data from prometheus B, or vice versa.

>  take it that my two prometheus do not sync timeseries; 

That is correct.  They'll both independently scrape and store.  I say they'll have "roughly" the same data because they won't scrape at the same points in time.

> but with grafana i see no gaps in my graphs.

Firstly, restarting prometheus is pretty quick.

Secondly, PromQL queries will look back in time up to 5 minutes (by default) to find a previous data point.  Up to that interval, your grafana graphs will look flat but without gaps.

[Mario Cornaccini]

 i guess this 5 min looking back is what did the trick for me; thanks! 

the datasource configuration comes from kube-prometheus; 
i did not yet ever notice a slightly different output after a reload; i guess thats not a problem/use-case for me then.

i'll have a look at promxy though anyway.

cheers,

fil