how can I monitor custom systemd services?

[Joey Jojo]

Hello,

I have a node exporter setup and installed in a Linux Ubuntu server and everything works fine. I've had to setup a few different custom SystemD services located in /etc/systemd/system/ and I'd like to know how I can whitelist them into the node_exporter.service which is also located in /etc/systemd/system

This is the configuration in there right now:

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
     --collector.textfile.directory /var/lib/node_exporter/textfile_collector \
     --collector.filesystem.ignored-fs-types="^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$" \
     --collector.filesystem.ignored-mount-points "^/mnt/folder_dump" \
     --collector.systemd \
     --collector.systemd.unit-whitelist="(apache2|ssh|rsyslog|nginx|nuxt).service"

Question is, how can I add a custom SystemD service called let's say, "nuxt_sitename-a.service" as well as "nuxt_sitename-b.service" etc...

When I try to add it in the --collector.systemd.unit-whitelist= I don't see anything with that name in Prometheus query node_systemd_unit_state related to the nuxt_sitename-a/b but I do see all the other whitelist services like SSH, Apache etc..

Does anybody know how to get node exporter to see

[Christian Hoffmann]

Hi,

On 3/31/20 10:56 PM, Joey Jojo wrote:
> I have a node exporter setup and installed in a Linux Ubuntu server and
> everything works fine. I've had to setup a few different custom SystemD
> services located in /etc/systemd/system/ and I'd like to know how I can
> whitelist them into the node_exporter.service which is also located in
> /etc/systemd/system
>
> This is the configuration in there right now:
>

[...]

> Question is, how can I add a custom SystemD service called let's say,
> "nuxt_sitename-a.service" as well as "nuxt_sitename-b.service" etc...

> When I try to add it in the /*--collector.systemd.unit-whitelist= */I

> don't see anything with that name in Prometheus query

> /*node_systemd_unit_state */related to the nuxt_sitename-a/b but I do

> see all the other whitelist services like SSH, Apache etc..

> **/**/

>
> Does anybody know how to get node exporter to see

Your approach sounds correct -- this is what we are using as well.
Can you confirm that the unit is loaded? systemctl list-units must
return it; it being returned from systemctl list-unit-files is not
enough. Maybe you still need to start and/or reference/enable it?

Can you confirm that node_exporter is running using the updated
whitelist, e.g. did you perform a full node_exporter restart?

Kind regards,
Christian

[Joey Jojo]

Hey Christian,

Thanks for getting back to me. Yes I just ran this and looks active and running:

root@aws-mtl-dev3:~# systemctl list-units |grep nuxt
nuxt_site-a.service                                          loaded active running   Nuxt Vue JS Service
nuxt_site-b.service                                          loaded active running   Nuxt Vue JS Service
nuxt_site-c.service                                          loaded active running   Nuxt Vue JS Service
nuxt_usa-a.service                                                loaded active running   Nuxt Vue JS Service
nuxt_usa-b.service                                                loaded active running   Nuxt Vue JS Service
nuxt_usa-c.service                                                loaded active running   Nuxt Vue JS Service

And this is my Node Exporter file:

And this is what I have in the collector

     --collector.systemd \
     --collector.systemd.unit-whitelist="(apache2|ssh|rsyslog|nginx|nuxt_*).service"

I am trying to use nuxt_* (wildcard) to see if it an pickup all of those custom services.

Any idea why it doesn't show up in Prometheus when I query it?

[Christian Hoffmann]

Hi,

you need to change your pattern: _* matches zero or more underscores. What you want instead is _.+ which matches an underscore followed by one or more arbitrary characters.

The pattern is a regular expression, not a glob.

Kind regards,
Christian