time series missing data

[Stephen Eaton]

Hi All,

I'm a fairly new prometheus user and have some questions regarding missing data in my time-series.  I'm sure its related to the wmi exporter however just want to make sure that I'm not doing anything out of the ordinary.

I'm scraping about 2 dozen hosts using both the node exporter for my linux hosts and the wmi_exporter for windows hosts (mix of 2008, 2012r2 and 2016).

Node exporter all metrics are fine, nothing missing from any of the node time series.

However with wmi_exporter scraped data I'm seeing missing data on a number of hosts, not all though, nothing out of the ordinary in terms of system load at any point during this time.  Mainly occurring on disk and memory metrics, however also seeing on system_threads and in a few cases CPU.

All jobs are being scraped at 20sec intervals and have two seperate jobs configured, one for my nodes and one for my wmi-nodes.  I'm not seeing anything reported regarding missed scrapes etc from prometheus dashboards, and in most cases (not all) it appears to be deterministic.

All these are running on a mix of hypervisors, majority being on Nutanix AHV hypervisor and then XenServer, appears to be happening on both.

What scrape periods should I be looking at for jobs?  is 20 sec too much?

you can see an example here of what I'm seeing

 

Stephen....

[Ben Kochie]

20s scrape interval is usually fine.  I don't know about the reliability of the wmi exporter.

A couple of things you can look into:

Look at the `up` metric.  This will tell you if Prometheus is able to scrape consistently.

Same goes with the `scrape_samples_scraped` metric.  If you see `up` is OK, but the number of samples scraped isn't, you might be having issues with the exporter.

These are both metrics automatically created by Prometheus[0].

[0]: https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8af5b253-a694-493d-9a0b-cff8cb1da370%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.