Reg: Error on ingesting out-of-order samples

[RAJESH DASARI]

Hi ,

when our system is booted up , system time was in sync with ntpd
service, and prometheus also started running at that time(08:51 AM),
after some 5 minutes, system time got in sync with GPS receiver and
time was advanced by 10 mins(09:16 AM).

we have noticed the below error logs from prometheus.


14:24:41.837951 info CONTROLLER prometheus[1343]: level=warn
ts=2021-03-31T08:54:41.837Z caller=scrape.go:1372 component="scrape
manager" scrape_pool=prometheus target=http://pm:9090/metrics
msg="Error on ingesting out-of-order samples" num_dropped=398

we are noticing the Error on ingesting out-of-order samples continuously.

is there any way to recover from this issue, without any manual action
(like restarting prometheus). Could some please help me with this ?

prometheus version: 2.23.0
we don't have any rules configured and also we don't have multiple targets.

Regards,
Rajesh.

[RAJESH DASARI]

Could someone please help me with the above query ? whenever time goes
forward by 10 mins or so prometheus should be able to scrape the next
samples right ? or in my case samples were dropped continuously? How
to recover from that error. my scraping interval is 15 seconds.

Thanks,
Rajesh Dasari.

[Harald Koch]

On Thu, Apr 8, 2021, at 22:36, RAJESH DASARI wrote:
> is there any way to recover from this issue, without any manual action
> (like restarting prometheus). Could some please help me with this ?

Upgrade to 2.25.2 or later - most of these out-of-order issues were fixed.

(There's still an issue with recording rules and out-of-order samples, though)

--
Harald

[Julien Pivotto]

Can you update the github issue with more details? I think could be room
for improvement here indeed.

https://github.com/prometheus/prometheus/issues/8710



>
> --
> Harald
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a476bd4e-3c74-482b-8a38-e38a6064edf7%40www.fastmail.com.

--
Julien Pivotto
@roidelapluie