suspending a VM that is running prometheus?

[Harald Koch]

I'm running prometheus at home, to monitor a few VMs on my homelab, as well as my Linodes. My homelab hypervisor runs Arch Linux and libvirtd.

Sometimes I need to patch the OS on the hypervisor and reboot it. The process up until now has been to suspend the guests, reboot the server, and restore the guests. (Yeah, I know, this is not recommended for prometheus - but it's a homelab, not a production environment . And I've seen weird clock-skew errors on production VMware before... :).

This has been working for a couple of years at this point. But recently (like, the last month or so) I've been getting prometheus "Error on ingesting out-of-order samples" errors spewing in the logs after this operation, and no data stored in prometheus. Restarting the prometheus process fixes it immediately; the errors stop and data is stored in the database again.

The problem has something to do with the combination of suspending/resuming the VM, and clock synchronization (hypervisor -> guest, or NTP is unknown). Looking for advice on how to debug the problem (or someone who has already encountered this...)

Thanks!

--

Harald

[Laurent Dumont]

When you resume the VM, does NTP resync? I guess that since you just pause the VM, once it unpause, the time will be incorrect and that would explain the out-of-order message. The scaped host will be in the future from the perspective of the Prometheus server.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5a03f52b-c718-4b05-99e6-4330fb67608d%40www.fastmail.com.

[Ben Kochie]

For working hypervisor combinations, running NTP on the guest VM is unnecessary. For example, with KVM/QEMU, the guest can sync directly with the hypervisor.

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock

https://www.linux-kvm.org/page/KVMClock

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOAKi8yqTXQthunTsh5pGr2Yqu8XTtYFzjhWNk6xGtaCe19Ndw%40mail.gmail.com.

[Harald Koch]

On Thu, Nov 5, 2020, at 03:54, Ben Kochie wrote:

For working hypervisor combinations, running NTP on the guest VM is unnecessary. For example, with KVM/QEMU, the guest can sync directly with the hypervisor.

Thanks for your reply!

I agree, but the the state of the art continues to be that this ... doesn't work very well. Even RedHat's KVM+QEMU documentation recommends that the guests run NTP.

I'm still curious what the source of the issue is. Laurent Dumont mentioned that the clocks on the scraped target and the prometheus host will be different, but based on what I've seen, there are no timestamps in the exporter outputs, so why would this matter?

Even with NTPD running, the libvirtd resume operation resets the clock in the guest (I'm running the qemu-agent in the guest, and have the SYNC_TIME=1 option set on the host). The reboot takes about 2 minutes, so even if some samples get scraped into the TSDB with strange future timestamps, shouldn't this should correct after a few minutes, once the prometheus host catches back up to the present?

And the strange part for me - restarting prometheus immediately fixes the issue. If there were future-dated samples collected, wouldn't the problem persist after restarting the daemon?

Still confused,

--

Harald

[Laurent Dumont]

How long do you usually wait before restarting the Prometheus process?

https://www.robustperception.io/debugging-out-of-order-samples

There are a couple of different possible root cause so I'm 100% sure which one you are hitting. Prometheus itself might not pick up the time change to the correct one once the VM has resumed. It might take multiple scraping cycles for all targets. You can check the "prometheus_target_scrapes_sample_out_of_order_total" counter or "timestamp(up)" do see what issue you are hitting.

Either way, a server shutdown + start is not too bad ;)