Node Exporter & Systemd

[Jonathan Sloan]

Hello,q

I seem to be having some troubling getting NE to ignore a few devices 'sr0/sda[1-3]. I've tried multiple things to try and get this work, exact for manually telling it to ignore sr0|sda1|sda2|sda3 etc. When using the below command.

 

node_exporter --web.listen-address=0.0.0.0:9100 --web.max-requests=16 --collector.processes

It does not ignore sda[1-3] I know sr* is not part of the standard regex they use. But this standard regex doesn't appear to work with systemd because the \\d chars are removed from systemd.

/usr/sbin/node_exporter --web.listen-address=0.0.0.0:9100 --web.max-requests=16 --collector.processes --collector.diskstats.ignored-devices=^(sr|ram|loop|fd|(h|s|v|xv)d[a-z]|nvmed+nd+p)d+$ --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) --collector.filesystem.ignored-fs-types=^(rootfs|tmpfs|autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

The issue appears to be around the char '\'. I believe systemd removes these chars and thus you need to encode them from my googling around with systemd-escape which I did shown below, but it still doesn't appear to work..

[root@linda058 20:16 compose_files]$ systemd-escape '^(sr|ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$'

\x5e\x28sr\x7cram\x7cloop\x7cfd\x7c\x28h\x7cs\x7cv\x7cxv\x29d\x5ba\x2dz\x5d\x7cnvme\x5c\x5cd\x2bn\x5c\x5cd\x2bp\x29\x5c\x5cd\x2b\x24

[root@linda058 20:18 compose_files]$ systemd-escape -u '\x5e\x28sr\x7cram\x7cloop\x7cfd\x7c\x28h\x7cs\x7cv\x7cxv\x29d\x5ba\x2dz\x5d\x7cnvme\x5c\x5cd\x2bn\x5c\x5cd\x2bp\x29\x5c\x5cd\x2b\x24'

^(sr|ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$

[Christian Hoffmann]

Hi Jonathan,

On 3/28/20 2:20 AM, Jonathan Sloan wrote:
> I seem to be having some troubling getting NE to ignore a few devices
> 'sr0/sda[1-3]. I've tried multiple things to try and get this work,
> exact for manually telling it to ignore sr0|sda1|sda2|sda3 etc. When
> using the below command.
>  
> |
> node_exporter
> --web.listen-address=0.0.0.0:9100--web.max-requests=16--collector.processes
> |
>
> It does not ignore sda[1-3] I know sr* is not part of the standard regex
> they use. But this standard regex doesn't appear to work with systemd
> because the \\d chars are removed from systemd.
>
> |
> /usr/sbin/node_exporter
> --web.listen-address=0.0.0.0:9100--web.max-requests=16--collector.processes
> --collector.diskstats.ignored-devices=^(sr|ram|loop|fd|(h|s|v|xv)d[a-z]|nvmed+nd+p)d+$

> --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)--collector.filesystem.ignored-fs-types=^(rootfs|tmpfs|autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

>
> |
>
> The issue appears to be around the char '\'. I believe systemd removes
> these chars and thus you need to encode them from my googling around
> with systemd-escape which I did shown below, but it still doesn't appear
> to work..

One simple solution would be moving the options to a seperate file. This
is what the suggested default config does:
https://github.com/prometheus/node_exporter/tree/master/examples/systemd

If you don't want to to this, it should still be possible to solve this
within the unit file.
I have posted an example below which seems to work for me. Neither the
ExecStart value nor the parameter values are quoted in my case.

Kind regards
Christian


# systemctl --version
systemd 245 (245.2-2-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP
+LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS
+KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

# cat /etc/systemd/system/prometheus-node_exporter.service
[Unit]
Description=Node Exporter

[Service]
User=node_exporter
ExecStart=/usr/bin/prometheus-node-exporter
--collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$

[Install]
WantedBy=multi-user.target

# ps aux | grep node-exp
node_ex+ 3113 0.0 0.2 1161372 15848 ? Ssl 09:50 0:00
/usr/bin/prometheus-node-exporter
--collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$

[Brian Candler]

The trouble with backslashes is that they have special meanings in all sorts of places - not only to regexs but also to the shell and to systemd - and working out how many times you have to double them up is very hard.

I suggest you simply replace \d with [0-9] and then life suddenly becomes much easier.  I raised this before at https://github.com/prometheus/node_exporter/issues/1562 (see point 3).

The actual regular expression contains \d.  But this is shown in the Go source code as \\d because in a Go string, "\\" is a single backslash:

ignoredDevices = kingpin.Flag("collector.diskstats.ignored-devices", "Regexp of devices to ignore for diskstats.").Default("^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$").String()

./collector/systemd_linux.go: unitBlacklist          = kingpin.Flag("collector.systemd.unit-blacklist", "Regexp of systemd units to blacklist. Units must both match whitelist and not match blacklist to be included.").Default(".+\\.(automount|device|mount|scope|slice)").String()

And similarly, when kingpin (CLI library) displays the default value in its help output, the single backslash is doubled up:

# /opt/node_exporter/node_exporter --help
...

      --collector.diskstats.ignored-devices="^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$"

      --collector.systemd.unit-blacklist=".+\\.(automount|device|mount|scope|slice)"

But depending on the context, e.g. if this is inside a systemd unit file or a shell script setting an environment variable, the doubling-up may have to be done multiple times.

[Jonathan Sloan]

Thanks for the info. When I have more time again I'll try some more combos if needed, but was able to get this to work with the below, I've also never put the NE args directly in a unit file they've always been sourced in exactly like the default config which is where I got the unit file from.

$ systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

 $ tail -n1 /etc/sysconfig/node_exporter
OPTIONS="--web.listen-address=0.0.0.0:9100 --web.max-requests=16 --collector.processes --collector.diskstats.ignored-devices=^(sr0|sda1|sda2|sda3|dm-0|dm-1|dm-2|dm-3)$ --collector.filesystem.ignored-mount-points=^/(boot|dev|proc|sys|var/lib/docker/.+)($|/) --collector.filesystem.ignored-fs-types=^(rootfs|tmpfs|autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"