Strange network behavior detected via node exporter

36 views
Skip to first unread message

iono sphere

unread,
Jul 23, 2021, 11:55:50 PM7/23/21
to Prometheus Users
I got the Prometheus metrics as follow:

CPU:
100 - (avg by (instance) (irate(node_cpu_seconds_total{job="node-exporter",mode="idle"}[1m])) * 100)

Received Bytes:
rate(node_network_receive_bytes_total[20s])

Transmitted Bytes:
rate(node_network_transmit_bytes_total[20s])


On July 22, I got this strange behavior on my server. 

See the attached incident_2.png. The data for CPU, receiving bytes and transmit bytes are missing for 3 minutes. All other data panels such as RAM, HDD and UP do not have any missing data and have valid data.

Next, please see the attached incident_1.png. After the data missing period, CPU, Received Bytes and Transmitted Bytes have these spikes nonstop every 5 minutes for a period of time. On Received Bytes and Transmitted Bytes, the spikes are from {device="lo", instance="x.x.x.x:xxxxx", job="node-exporter"}. 

So does anyone know what could be the possible cause of this?

Regards,
Iono
incident_1.png
incident_2.png
Reply all
Reply to author
Forward
0 new messages