too old sample - Error appending remote write

149 views

Skip to first unread message

rksakt...@gmail.com

unread,

Dec 8, 2023, 2:37:19 AM12/8/23

to Prometheus Users

Hi All,

We are using prometheus version 2.48.0 and enabled out_of_order_time_window to 1d.

Promethues A and Prometheus B are in same version

Prometheus A normally remote writes to Prometheus B. Prometheus A receives data from a fleet of Prometheus Agents .

Prometheus A was done for 45 minutes and once recovered it received all the data from Prometheus Agent but those metrics didn't reach our Prometheus B which is our long term storage.

We generated the backfill metrics in Openmetrics format from Prometheus A for the missing time tried to push it to Prometheus B using the remote write endpoint using promtool.

It failed with below error.

HTTP 500 - Too old sample on Prometheus A

On Prometheus B we can see the below error.

{"caller":"write_handler.go:76","component":"web","err":"too oldsample","level":"error","msg":"Error appending remote write"}