Node exporter||Need help reading non-root FS mount point metrics

[NAGARJUNA D.N]

Hi,

I have an FS mounted for PostgreSQL WAL logs which have limited permissions.

Where the disk usage not reported to Prometheus and am getting the following error.

Nov 14 18:22:30 timescale-masterdb node_exporter[7368]: level=debug ts=2020-11-14T12:52:30.707Z caller=filesystem_linux.go:94 collector=filesystem msg="Error on statfs() system call" rootfs=/pg-data/postgresql/12/main/pg_wal err="permission denied"

Kindly suggest what permission I have to give for node_exporter users without affecting postgres. 

Thanks

[Matthias Rampke]

Looking at the statfs man page, it suggests that this error would happen if it is lacking "search" permission (the execute bit) on any of the enclosing directories.

I would recommend giving +rx on all directories in that path (/pg-data, /pg-data/postgresql, /pg-data/postgresql/12 and so on) – either to all (a+rx) or the group that owns them and add the node exporter to that group. I think this is not a security issue because it allows anyone / that group to `ls -l` the directories but not to access the files within, unless those files' permissions are way too wide.

/MR

This communication is for informational purposes only.  Any comments or statements made herein do not necessarily reflect those of Blue Pi Consulting India Private Limited, its subsidiaries and affiliates. This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Blue Pi Consulting India Private Limited, its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9293ba11-a9f0-4a5f-ba78-ce15ce10d1e6n%40googlegroups.com.

[NAGARJUNA D.N]

Hi,

You are right, the main folder under /pg-data/postgresql/12 has no permission to other users, I can't change this permission as postgres won't come up if I do so.

drwx------ 20 postgres postgres 4096 Nov 16 21:24 main/

I have added to a group like below but no luck. Kindly help

 - postgres:x:116:node_exporter

[NAGARJUNA D.N]

Hi,

The below metrics of the mount point is not exported.

node_filesystem_size_bytes 

node_filesystem_avail_bytes 

node_filesystem_free_bytes

I can still see other metrics exported for the same FS.

node_disk_discarded_sectors_total{device="nvme1n1"} 0

node_disk_discards_completed_total{device="nvme1n1"} 0

node_disk_discards_merged_total{device="nvme1n1"} 0

node_disk_io_now{device="nvme1n1"} 0

node_disk_io_time_seconds_total{device="nvme1n1"} 24109.884000000002

node_disk_io_time_weighted_seconds_total{device="nvme1n1"} 62920.544

node_disk_read_bytes_total{device="nvme1n1"} 1.009441195008e+12

node_disk_read_time_seconds_total{device="nvme1n1"} 5962.584

node_disk_reads_completed_total{device="nvme1n1"} 3.858302e+06

node_disk_reads_merged_total{device="nvme1n1"} 2240

node_disk_write_time_seconds_total{device="nvme1n1"} 94167.781

node_disk_writes_completed_total{device="nvme1n1"} 1.732477e+07

node_disk_writes_merged_total{device="nvme1n1"} 3.421928e+06

node_disk_written_bytes_total{device="nvme1n1"} 3.616515354624e+12

node_filesystem_device_error{device="/dev/nvme1n1",fstype="ext4",mountpoint="/pg-data/postgresql/12/main/pg_wal"} 

Have described my problem here - https://stackoverflow.com/questions/64786158/node-exporter-unable-to-export-specific-volume-mount-point-metrics-err-permis/64831390#64831390

[Matthias Rampke]

The directory permissions only grant access to the user, not the group, so adding the exporter to the group won't help. Given PostgreSQL is this strict about permissions, you will have to run the exporter either as root or as the postgres user :/

The other metrics are collected in different ways (from /proc mostly) so they are not subject to this permission check.

/MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b40328ef-6599-48de-ba09-123589c4d0bcn%40googlegroups.com.

[NAGARJUNA D.N]

Would you please suggest how can I run node exporter under root user?

Regards,

Nagarjuna

[NAGARJUNA D.N]

Have changed permission of exporter executable to postgres:postgres, but no luck.

[NAGARJUNA D.N]

Have mounted my directory like this.

/dev/nvme0n1    2.0T  1.2T  702G  63% /pg-data -> This looks Ok.

/dev/nvme1n1    344G   11G  318G   4% /pg-data/postgresql/12/main/pg_wal -> This has issue now.

Do you think the same path to the different FS is the reason? 

fstab entries:

LABEL=cloudimg-rootfs / ext4 defaults,discard 0 0

LABEL=pg-data /pg-data ext4 defaults  0 0

LABEL=pg-wal /pg-data/postgresql/12/main/pg_wal ext4 defaults  0 0

[Matthias Rampke]

How to change the user that the exporter runs as depends on how you run it (maybe a systemd unit?) – that will specify the user. The ownership of the executable itself is not so important.

That /pg-data is okay is what I would expect, since the permission problem is further down.

/MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d55da416-b5eb-4142-b953-090e8b44fd00n%40googlegroups.com.

[NAGARJUNA D.N]

Thanks, much. I fixed it by updating systemd and running as a root user.