Node Exporter Not Showing /dev/sdb1

[Emmanuel Yong]

Dear All,

Currently running the following:

Operating System: CentOS 7.4

Kernel Version: 4.13.5-1.el7.elrepo.x86_64.

Node Exporter Version: 0.15.2

Issue is my system is configured with 2 hard disk /dev/sda and /dev/sdb but node exporter is not picking up statistics for /dev/sdb1.

Running Node Exporter with debug, i am seeing the following

time="2017-12-19T08:34:48+08:00" level=debug msg="Ignoring device: sdb1" source="diskstats_linux.go:175"

time="2017-12-19T08:34:48+08:00" level=debug msg="Node /sys/bus/node/devices/node0 CPU 0 is missing package_throttle" source="cpu_linux.go:173"

time="2017-12-19T08:34:48+08:00" level=debug msg="OK: cpu collector succeeded after 0.000676s." source="collector.go:126"

time="2017-12-19T08:34:48+08:00" level=debug msg="Ignoring device: sda1" source="diskstats_linux.go:175"

time="2017-12-19T08:34:48+08:00" level=debug msg="Ignoring device: sda2" source="diskstats_linux.go:175"

time="2017-12-19T08:34:48+08:00" level=debug msg="OK: diskstats collector succeeded after 0.000232s." source="collector.go:126"

Checking /proc/diskstats shows all my disk

[root@srv] # cat /proc/diskstats | grep "$DISK "

   8       0 sda 45964 804 14826044 1072708 15387760 49702654 1661887264 4947207 0 2501554 6045458

   8       1 sda1 183 0 12754 2105 3 0 24 6 0 1292 2111

   8       2 sda2 45750 804 14809146 1070023 15387757 49702654 1661887240 4947201 0 2500572 6006610

   8      16 sdb 289 0 13960 3075 0 0 0 0 0 2368 3075

   8      17 sdb1 258 0 9808 2592 0 0 0 0 0 1935 2592

  11       0 sr0 18 0 2056 423 0 0 0 0 0 397 423

 253       0 dm-0 47539 0 14805410 1211494 64924929 0 1661887240 15747577 0 2533200 16966232

 253       1 dm-1 88 0 4328 742 0 0 0 0 0 534 742

How can i get node exporter to get the required disk stats for /dev/sdb1

[Ben Kochie]

The default regexp for `--collector.diskstats.ignored-devices` ignores the partitions because the stats for the block device include them.  As a result, if you were to sum() the stats, you would get invalid results.

It would be nice if these kernel metrics were better formatted so we knew exactly which stats were cumulative / related or not.

If you want to include partition stats, you can adjust the ignored-devices regexp, but beware you will need to adjust your queries to compensate for the cumulative data coming from the main block device.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/234d8c26-9a0e-443f-8e08-df319e567b86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Kochie]

You might want to review https://www.kernel.org/doc/Documentation/iostats.txt

[Emmanuel Yong]

Hi Ben,

My question is both /dev/sda and /dev/sdb are disk but i am able to see statistic for /dev/sda but not /dev/sdb

How should i configure?

Thank You

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

You said sdb1, which is a partition, not a block device.  Please test the node_exporter output with this command:

curl "http://localhost:9100/metrics" | grep "node_disk_bytes_read"

Also make sure if you are running the node_exporter in Docker, you have followed the documentation[0].  Containerizing the node_exporter can lead to missing/invalid data due to namespaces.

[0]: https://github.com/prometheus/node_exporter#using-docker

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7dcfe88a-5c18-4973-a498-c001de864810%40googlegroups.com.

[Emmanuel Yong]

Dear Ben,

Sorry for confusion, this is a virtual machine and having 2 virtual disks.

1st virtual disk is mounted under /dev/sda

2nd virtual disk is mounted under /dev/sdb

[root@srv system]# curl "http://localhost:9100/metrics" | grep "node_disk_bytes_read"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 94384  100 94384    0     0  8907k   # HELP node_disk_bytes_read The total number of bytes read successfully.

   0 # TYPE node_disk_bytes_read counter

--:-node_disk_bytes_read{device="dm-0"} 7.597888512e+09

-:-node_disk_bytes_read{device="dm-1"} 2.215936e+06

-node_disk_bytes_read{device="sda"} 7.60845312e+09

 --node_disk_bytes_read{device="sdb"} 7.14752e+06

:--:node_disk_bytes_read{device="sr0"} 1.052672e+06

-- --:--:-- 10.0M

The above shows sdb, but if i want to gather free space for /dev/sdb using node_filesystem_avail is not showing

[root@srv system]# curl "http://localhost:9100/metrics" | grep "node_filesystem_avail" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 94390 100 94390 0 0 6700k 0 --:--:-- --:# HELP node_filesystem_avail Filesystem space available to non-root users in bytes. --:-- # TYPE node_filesystem_avail gauge --:--:-node_filesystem_avail{device="/dev/mapper/centos_ph71v--esd01-root",fstype="ext4",mountpoint="/"} 6.4188174336e+10 node_filesystem_avail{device="/dev/sda1",fstype="ext4",mountpoint="/boot"} 8.23144448e+08 node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 4.159393792e+09 - 76node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 8.37009408e+08 81k

Thank You

[Ben Kochie]

The node exporter collects filesystem free space based on the filesystems listed in /proc/mounts.  It uses a syscall to get the actual stats.

If /dev/sdb1 is not mounted, you will not get metrics.  Where is /dev/sdb1 mounted?  What kind of filesystem is it?

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2ec7bad0-f160-46ea-86b2-2077b5242bf4%40googlegroups.com.

[Emmanuel Yong]

Dear Ben,

Thank you so much for that pointer, i realized that i did not placed the partition into /etc/fstab and i did a system maintenance some time back and the partition was not auto mounted.

After i remount the partition, all is good.

This i need to record as part of SOP for troubleshooting