--collector.textfile.directory is fetching empty results in promotheus

[Somesh Rao]

hi, i am trying to monitor cpu and ran utlization of each process of an odoo instance, i am using prometheus and node expoter for this task. i found that we wrote a bash script that fetches details of odoo instance and then store the output in a .prom the --collector.text.directory=/path/to/that/.prom file will pick the details and i will be able to query from prometheus. 

curl -G --data-urlencode 'query=odoo_instance_filestore_size_bytes{instance="name"}' http://localhost:9090/prometheus/api/v1/query

{"status":"success","data":{"resultType":"vector","result":[]}}

but i am getting an empty results even after making all the error to configure everything right. can anyone please help me with this, let me know what am i doing wrong. 

additional details :

@localhost:~# ls -ltr /var/lib/node_exporter/textfile_collector/odoo_metrics.prom
-rwxr-xr-x 1 prometheus prometheus 1092 Nov 21 04:29 /var/lib/node_exporter/textfile_collector/odoo_metrics.prom

@localhost:~# cat /var/lib/node_exporter/textfile_collector/odoo_metrics.prom
# Odoo Instances Resource Usage Metrics
# HELP odoo_instance_process_count Number of processes for the Odoo instance
# TYPE odoo_instance_process_count gauge
odoo_instance_process_count{instance="2hp39r1l"} 6
# HELP odoo_instance_cpu_usage_percent Total CPU usage percentage for the Odoo instance
# TYPE odoo_instance_cpu_usage_percent gauge
odoo_instance_cpu_usage_percent{instance="2hp39r1l"} 0
# HELP odoo_instance_memory_usage_bytes Total memory usage in bytes for the Odoo instance
# TYPE odoo_instance_memory_usage_bytes gauge
odoo_instance_memory_usage_bytes{instance="2hp39r1l"} 633397248
# HELP odoo_instance_filestore_size_bytes Filestore size in bytes for the Odoo instance
# TYPE odoo_instance_filestore_size_bytes gauge
odoo_instance_filestore_size_bytes{instance="2hp39r1l"} 2237538
# HELP odoo_instance_database_size_bytes Database size in bytes
# TYPE odoo_instance_database_size_bytes gauge
odoo_instance_database_size_bytes{instance="2hp39r1l.nip.io"} 20550447

@localhost:~# cat /lib/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=127.0.0.1:9100 \
  --collector.systemd \
  --collector.cpu.info \
  --collector.meminfo \
  --collector.diskstats \
  --collector.processes \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

[Brian Candler]

1. Scrape node_exporter using curl, to see if the metrics are returned.

curl -fsS x.x.x.x:9100/metrics 

curl -fsS x.x.x.x:9100/metrics | grep odoo_

If the metrics are present, then the issue is with your prometheus scraping or with your usage of the PromQL API.

2. If they are not, then look at the node x.x.x.x where node_exporter is running. Validate the contents of the file using promtool:

/path/to/promtool check metrics </var/lib/node_exporter/textfile_collector/odoo_metrics.prom

If not, the error will tell you what's wrong with your metrics file.

3. If that's OK, then check node_exporter is running with the correct flags

ps auxwww | grep node_exporter

systemctl show node_exporter.service | grep  ^ExecStart

If not:

- did you remember "systemctl daemon-reload"? 

- check you don't also have a /etc/systemd/system/node_exporter.service which overrides /lib/systemd/system/node_exporter.service

4. If that's OK, then check logs from node_exporter

journalctl -eu node_exporter

5. Check that the intermediate directories have world r-x permissions (or r-x for the prometheus user)

ls -ld /var/lib/node_exporter

ls -ld /var/lib/node_exporter/textfile_collector

6. Check for errors indicating apparmor or selinux access denials

dmesg | tail -100