How much time node_exporter holds data locally?

[Arunlal A]

I am looking into the internals of node exporter and it's working. I could't get a documentation for the total time that the node_exporter holds data locally on the machine. Can somebody help me to understand this ?

I went through this issue

I have done a test. I disabled the node from Prometheus scrape configuration for 30 minutes and could see that the data lost.

So in case of any connectivity issue (n/w) from Prometheus server to any remote servers where we're running node_expoter, what will happen? Is there any option to increase the hold time at node_exporter (something similar to spool option in metric beat) ?

Host operating system: output of uname -a

CentOS 6

node_exporter version: output of node_exporter --version

node_exporter, version 0.18.0 (branch: HEAD, revision: f97f01c46cfde2ff97b5539b7964f3044c04947b)
  build user:       root@77cb1854c0b0
  build date:       20190509-23:12:18
  go version:       go1.12.5

Are you running node_exporter in Docker?

No

Screenshot of loosing data: Link

[Benjamin Ritcey]

node_exporter does not hold data locally - it collects the information at scrape time.  As Brian wrote in that issue:

"In the case of the node exporter, virtually all the data is collected at each scrape and returned to the server. If the server can't talk to the client you'll lose information, but as we're exporting raw counters rather than rates this leads to a reduction in data granularity rather than losing all information about that time period."

In practice, you mostly shouldn't care about brief interruptions - if you're scraping every 10 seconds, losing 30 seconds of samples will still give you useful CPU rates over time - e.g., the 5 minute average will still be reasonably accurate.

Incidentally, this is why it's recommended to do your scraping from a Prometheus server that's "close" to the instance being scraped - e.g., on the same network segment or at least same data center (or pod or what have you).  You don't want to deal with potentially lossy links by scraping over the WAN.

Lots of Prom instances, with some mechanism to give you a global overview (Thanos/Cortex/VictoriaMetrics/federation/xxx).

[Arunlal A]

Thanks Benjamin!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/62325382-e017-49ba-b948-e9bdee181ba5%40googlegroups.com.

--

Best wishes,

Arunlal A

Linux Systems Architecture 

https://www.crybit.com/about

[Arunlal A]

I am expecting this behavior for all the other collectors, Isn't it ?

[Ben Kochie]

Yes, this is a core behavior of Prometheus-style metrics endpoints. Monitoring endpoints are "stateless", and only return "now" data. This allows for simultaneous access from multiple monitoring services, avoiding the need to keep complex tracking configuration and state inside the client.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CACUTJXB1PZoo0h6tYBuSCxMEOYe3y3P23Hq7wViO4wD9mNWfrg%40mail.gmail.com.

[Arunlal A]

Cool, thanks Ben!