node_exporter stopping when run as systemd service on Linux

Rudy Gardelein

unread,

Nov 9, 2021, 7:38:46 AM11/9/21

to Prometheus Users

Host operating system: output of uname -a

Linux xxx 3.10.0-1160.45.1.el7.x86_64 #1 SMP Fri Sep 24 10:17:16 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.2.2 (branch: HEAD, revision: 2664536)
build user: root@b9cb4aa2eb17
build date: 20210806-13:44:18
go version: go1.16.7
platform: linux/amd64

node_exporter command line flags

N.A.

Are you running node_exporter in Docker?

NO

What did you do that produced an error?

I am running node_exporter as a systemd service with the following service file in /etc/systemd/system:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
RemainAfterExit=true
ExecStart=/var/opt/EAI/prometheus/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

The service is running only for a short time and is getting stopped regularly. The systemd status is:

● node_exporter.service - Node Exporter
Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2021-11-08 19:42:15 UTC; 14h ago
Main PID: 14098 (code=killed, signal=TERM)

Nov 08 19:04:28 xxx node_exporter[14098]: level=info ts=2021-11-08T19:04:28.554Z caller=node_exporter.go:115 collector=timex
Nov 08 19:04:28 xxx node_exporter[14098]: level=info ts=2021-11-08T19:04:28.554Z caller=node_exporter.go:115 collector=udp_queues
Nov 08 19:04:28 xxx node_exporter[14098]: level=info ts=2021-11-08T19:04:28.554Z caller=node_exporter.go:115 collector=uname
Nov 08 19:04:28 xxx node_exporter[14098]: level=info ts=2021-11-08T19:04:28.554Z caller=node_exporter.go:115 collector=vmstat
Nov 08 19:04:28 xxx node_exporter[14098]: level=info ts=2021-11-08T19:04:28.554Z caller=node_exporter.go:115 collector=xfs
Nov 08 19:04:28 xxx node_exporter[14098]: level=info ts=2021-11-08T19:04:28.554Z caller=node_exporter.go:115 collector=zfs
Nov 08 19:04:28 xxx node_exporter[14098]: level=info ts=2021-11-08T19:04:28.554Z caller=node_exporter.go:199 msg="Listening on" address=:9100
Nov 08 19:04:28 xxx node_exporter[14098]: level=info ts=2021-11-08T19:04:28.563Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
Nov 08 19:42:15 xxx systemd[1]: Stopping Node Exporter...
Nov 08 19:42:15 xxx systemd[1]: Stopped Node Exporter.

What did you expect to see?

I expect the node_exporter service to keep on running, and to automatically restart upon failure or system reboot.

What did you see instead?

The node_exporter is stopping in systemd for no apparent reason. I would like to know how it can be run as a service in a consistent way without interruption.

Brian Candler

unread,

Nov 9, 2021, 2:09:01 PM11/9/21

to Prometheus Users

Here is the node_exporter.service that I use:

--------

[Unit]

Description=Prometheus Node Exporter

Documentation=https://github.com/prometheus/node_exporter

After=network-online.target

[Service]

User=xxxxx

EnvironmentFile=/etc/default/node_exporter

ExecStart=/opt/node_exporter/node_exporter $OPTIONS

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-user.target

--------

and /etc/default/node_exporter:

--------

OPTIONS='--collector.textfile.directory=/var/lib/node_exporter --collector.systemd --collector.ntp'

--------

It doesn't have any problem with premature shutdown.

Venkata Bhagavatula

unread,

Nov 10, 2021, 4:17:44 AM11/10/21

to Brian Candler, Prometheus Users

Hi Brian,

Are you getting any error when you start it manually as the user?

/opt/node_exporter/node_exporter --collector.textfile.directory=/var/lib/node_exporter --collector.systemd --collector.ntp

Thanks n Regards,

Chalapathi

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/88690113-44bd-4337-a56d-b2a771e78434n%40googlegroups.com.

Brian Candler

unread,

Nov 10, 2021, 4:52:15 AM11/10/21

to Prometheus Users

No errors, it all works perfectly. I only posted my systemd config as an example of a working one you can copy.

I think your problem is that you have put "RemainAfterExit=true" - you should remove this. node_exporter does not daemonize, it stays running in the foreground, and this is normal for systemd to manage.

Rudy Gardelein

unread,

Nov 22, 2021, 8:07:58 AM11/22/21

to Prometheus Users

Brian and all,

I'm affraid the issue is still not resolved, despite many attempts with multiple configurations, including your suggestion to remove the RemainAfterExit. The current systemd configuration looks like this:

-----

[Unit]

Description=Node Exporter

Documentation=https://github.com/prometheus/node_exporter

After=network-online.target

[Service]

User=prometheus

Group=prometheus

EnvironmentFile=/var/opt/EAI/prometheus/node_exporter/node_exporter.cfg

ExecStart=/var/opt/EAI/prometheus/node_exporter/node_exporter $OPTIONS

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-user.target

-----

The node_exporter.cfg contains the following:

-----

OPTIONS='--collector.systemd --collector.ntp'

-----

Any help to get this issue resolved would be highly appreciated.

Best regards,

Rudy

Op woensdag 10 november 2021 om 10:52:15 UTC+1 schreef Brian Candler:

Rudy Gardelein

unread,

Nov 23, 2021, 5:56:47 AM11/23/21

to Prometheus Users

All,

The issue is explained and resolved. Found out that the node_exporter service was getting stopped by a puppet agent with a policy for removing node_exporter. So the issue is not caused by systemd or node_exporter but is caused by puppet. I have disabled puppet and the node_exporter service now keeps running.