Create custom metrics and add it to prometheus

[BHARATH KUMAR]

Hello All,

I want to calculate the pod age running on every server. I wrote a shell script. I added this script under the folder /var/lib/node_exporter.

I created a cronjob for this script to run every minute 

* * * * * root bash /var/lib/node_exporter/custom_metrics.sh > /var/lib/node_exporter/apt1-prom

I store the above cronjob in the /etc/cron.d folder with filename: prom-apt1.

But I am not able to see the metrics I created in Prometheus UI.

But similarly, I created another shell script file to fetch some metrics. I followed the same procedure as above.

* * * * * root bash /var/lib/node_exporter/custom1.sh > /var/lib/node_exporter/apt-prom

I store the above cronjob in the /etc/cron.d folder with filename: prom-apt.

The metrics which I mentioned in custom1.sh, I am able to see those metrics in Prometheus UI.

Could anyone help me?

Thanks & regards,

Bharath Kumar.

[Stuart Clark]

What is the contents of those files in /var/lib/node_exporter?

--
Stuart Clark

[Brian Candler]

The problem is that the filename must end with ".prom", i.e. you need to create "apt1.prom" not "apt1-prom"

Also, you need to enable the functionality with a flag to node_exporter:

    node_exporter --collector.textfile.directory=/var/lib/node_exporter

See the documentation here: https://github.com/prometheus/node_exporter#textfile-collector

[Brian Candler]

Aside: you might also want to look at kube-state-metrics, which can give you a range of pod metrics including the pod start time. Then an expression like time()-kube_pod_start_time should give you the pod age.

[BHARATH KUMAR]

Sorry for the mistake in the above message. I created the filename with ".prom" only.

I added the --collector.textfile.directory=/var/lib/node_exporter as one of the scripts was working fine. but another file was issue.

[BHARATH KUMAR]

In /var/lib/node_exporter --> In this folder, the scripts(custom metrics script) will be available.

[BHARATH KUMAR]

The main issue is that I am not able to see the metrics in Prometheus UI. I am unable to understand the root cause.

Am I missing anything?

[Brian Candler]

> The main issue is that I am not able to see the metrics in Prometheus UI. I am unable to understand the root cause.

Almost certainly because they are not being collected.  So you need to break this down further to find out why.

(1) Show the output of this command:

ls -l /var/lib/node_exporter

* If the filenames don't end with ".prom" then that is your root cause.  node_exporter only picks up files whose names match the pattern *.prom

* If the files aren't readable by the user that node_exporter is running as, then that's the root cause.

* Are you running under RedHat/CentOS with SELinux?  If so, try putting it into permissive mode.  Maybe there's some SELinux policy which prevents access to files in /var/lib/node_exporter.

(2) Then show the contents of one of those files:

head /var/lib/node_exporter/foobar.prom

If the files are not in valid openmetrics format, then they won't be picked up either.  You can test them with "promtool check metrics":

/path/to/promtool check metrics </var/lib/node_exporter/foobar.prom

If that shows an error, then modify your script to generate files in the correct format. There are examples you can copy from here: https://github.com/prometheus-community/node-exporter-textfile-collector-scripts

(3) Show the command line that you're running node_exporter with:

ps auxwww | grep node_exporter

If the flag "--collector.textfile.directory=/var/lib/node_exporter" is not present, then that's your root cause.

(4) Show the output of this command:

curl localhost:9100/metrics | grep foobar

where "foobar" is one of the custom metrics you are generating.  Does it appear here?  If not, then you can rule out any issues with prometheus itself, it's definitely node_exporter not picking up your custom metrics.  If it does, then you'll need to dig further onto prometheus side (e.g. your scrape job config)

> In /var/lib/node_exporter --> In this folder, the scripts(custom metrics script) will be available.

You don't want the scripts in that directory - only the *output* from those scripts.  However, as long as the scripts themselves don't end with ".prom" then they will be ignored, so that won't be a problem.

[Brian Candler]

Also: you should clarify where exactly you're running these scripts, i.e. which host has the /var/lib/node_exporter/*.prom files

* If it's on the prometheus server itself, then you'll need to run node_exporter on the prometheus server, *and* you'll need to get prometheus to scrape itself - i.e. a scrape job that collects from localhost:9100/metrics

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

* If it's on some remote host, then you'll need to get prometheus to scrape *that* host on x.x.x.x:9100/metrics

If you're already scraping node_exporter on all your hosts - for example, you can see metrics like "node_filesystem_avail_bytes", on the instance where your custom metrics are being created - then you should be fine.