Suddenly - Error on ingesting samples that are too old or too far into the future

[Samuel Horowitz]

Something has happened -- I am not sure what, exactly, and suddenly my prometheus setup has become completely hosed. To my knowledge no configuration changes were made to Prometheus last week. 

I have several different exporters -- node_exporter running on some CentOS and Ubuntu servers, and then various other things like page_speed_exporter and something for UniFi, etc. They were all working fine. I made some changes to Grafana dashboards and reporting and everything looked great.

As I checked on my dashboards this morning they all report no data. And here is what's in my prometheus logs:

Oct 19 18:34:04 monitoring prometheus[782]: level=warn ts=2020-10-19T18:34:04.758Z caller=scrape.go:1170 component="scrape manager" scrape_pool=node target=http://monitoring_server:9100/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1492

Oct 19 18:34:04 monitoring prometheus[782]: level=warn ts=2020-10-19T18:34:04.758Z caller=scrape.go:945 component="scrape manager" scrape_pool=node target=http://monitoring_server:9100/metrics msg="appending scrape report failed" err="out of bounds"

Oct 19 18:34:10 monitoring prometheus[782]: level=warn ts=2020-10-19T18:34:10.120Z caller=scrape.go:1170 component="scrape manager" scrape_pool=redis_exporter_targets target="http://remote_server1:9100/metrics" msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=178

Oct 19 18:34:10 monitoring prometheus[782]: level=warn ts=2020-10-19T18:34:10.120Z caller=scrape.go:945 component="scrape manager" scrape_pool=redis_exporter_targets target="http://remote_server2:9101/scrape?target=redis%3A%2F%2Fcanvas-redis%3A63791" msg="appending scrape report failed" err="out of bounds"

Time does not seem to be out of sync, though?

$ ssh remote_server1 date -u && ssh monitoring_server date -u

Mon Oct 19 18:32:28 UTC 2020

Mon Oct 19 18:32:28 UTC 2020

I don't understand what's going on. I think the local time zone is set on some of the servers to UTC and some to Eastern, but if the actual date/time is correct then all should be well? When I scrape the node_exporter URL from the command line, the time stamp seems to be the expected UTC time, but I have no metrics, for any kind of exporter or server, all dropping last Friday.

The monitoring server is a Ubuntu 18 Virtual Machine running in Parallels Desktop for Mac v15. It's possible that Parallels itself was updated last Friday, but I can't fathom what effect that would have or how to troubleshoot further even if that is the cause.

[Samuel Horowitz]

And how could the monitoring be out of time sync when scraping itself? Doesn’t make sense to me??

---

From: promethe...@googlegroups.com <promethe...@googlegroups.com> on behalf of Samuel Horowitz <s...@horowitz.es>
Sent: Monday, October 19, 2020 2:37:30 PM
To: Prometheus Users <promethe...@googlegroups.com>
Subject: [prometheus-users] Suddenly - Error on ingesting samples that are too old or too far into the future

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c7878525-68f7-4b07-8f36-6f6a8dfde3d1n%40googlegroups.com.

[Samuel Horowitz]

Three quick updates:

1. Prometheus is running on Ubuntu 20.04, all packages are up to date.

2. I have tried setting the timezone on the Prometheus VM to New York (matching at least some servers), no difference (set it back to UTC).

3. I have tried downgrading the host VM software, no difference.