Missing timeseries on remote write

NandhaKumar V

unread,

Mar 8, 2022, 7:07:27 AM3/8/22

to Prometheus Users

Hi,

We have a remote_write configured to a local http server. Every two hours (precisely even hours), two to three minutes of data missing on remote endpoint. When I closely looked Prometheus ui and api, it seems that has old data for that period.

Any hint on what to look at?

Remote endpoint:

data from api:

Brian Candler

unread,

Mar 8, 2022, 9:08:34 AM3/8/22

to Prometheus Users

Prometheus versions, at sending and receiving side?

NandhaKumar V

unread,

Mar 8, 2022, 10:58:15 AM3/8/22

to Prometheus Users

On the sending side: 2.33.1, receiving side we have our webserver (locally in the pod)

l.mi...@gmail.com

unread,

Mar 8, 2022, 11:12:59 AM3/8/22

to Prometheus Users

Is it possible this is related to https://github.com/prometheus/prometheus/issues/10377? That's one reason for gaps in metrics on v2.33.x

On Tuesday, 8 March 2022 at 12:07:27 UTC v.na...@gmail.com wrote:

NandhaKumar V

unread,

Mar 10, 2022, 1:20:20 PM3/10/22

to Prometheus Users

I tried downgrading to v2.32.1, the problem remains the same. Prometheus API gives prev result (same can be seen in Prometheus UI), but on the remote endpoint it is missing). Surprisingly it is not only at the 2hour mark, but at different intervals as well.

Please let me know if there are pointers to check?

Screen Shot 2022-03-10 at 10.17.40 AM.png

Reply all

Reply to author

Forward