How is node_exporter populating process_open_fds?

[Kristopher Kahn]

I am seeing a discrepancy on hosts between what is returned by

# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 13

I am not able to find any equivalent output of lsof on the machine that returns that value (13 - or near it). I am thinking I am probably not using the proper flags/commands. How is node_exporter populating that value? I'm wondering if I'm inadvertently checking mounted filesystems or something.

[Julius Volz]

Hi Kristopher,

Assuming this is on Linux / Unix, this value is determined by looking at how many entries there are in the /proc/<PID>/fd directory, where PID is the process ID of the Node Exporter. Note that the "process_xxx" metrics are default metrics included by the Prometheus Go client library about the exporting process itself (so in your case, about the Node Exporter), and not about the machine. When I compare the number of entries in /proc/<PID>/fd, I do see three additional descriptors in lsof that are not listed in that directory, which are the binary itself, and the directory the binary is contained in, and the root directory. So maybe that difference in accounting is where your discrepancy comes from?

$ lsof -c node_exporter

COMMAND     PID   USER   FD      TYPE DEVICE SIZE/OFF    NODE NAME
node_expo 13943 julius  cwd       DIR   8,18     4096 7999933 /XXXXXXXX/prometheus/node_exporter
node_expo 13943 julius  rtd       DIR   8,18     4096       2 /
node_expo 13943 julius  txt       REG   8,18 19027879 4989900 /XXXXXXXX/prometheus/node_exporter/node_exporter
node_expo 13943 julius    0u      CHR  136,1      0t0       4 /dev/pts/1
node_expo 13943 julius    1u      CHR  136,1      0t0       4 /dev/pts/1
node_expo 13943 julius    2u      CHR  136,1      0t0       4 /dev/pts/1
node_expo 13943 julius    3u     IPv6  98870      0t0     TCP *:9100 (LISTEN)
node_expo 13943 julius    4u  a_inode   0,13        0   12663 [eventpoll]
node_expo 13943 julius    5r     FIFO   0,12      0t0   98866 pipe
node_expo 13943 julius    6w     FIFO   0,12      0t0   98866 pipe
node_expo 13943 julius    7u     IPv6  99715      0t0     TCP ip6-localhost:9100->ip6-localhost:44322 (ESTABLISHED)

$ l /proc/13943/fd     

total 0
dr-x------ 2 julius julius  0 Feb 10 12:37 ./
dr-xr-xr-x 9 julius julius  0 Feb 10 12:37 ../
lrwx------ 1 julius julius 64 Feb 10 12:37 0 -> /dev/pts/1
lrwx------ 1 julius julius 64 Feb 10 12:37 1 -> /dev/pts/1
lrwx------ 1 julius julius 64 Feb 10 12:37 2 -> /dev/pts/1
lrwx------ 1 julius julius 64 Feb 10 12:37 3 -> 'socket:[98870]'
lrwx------ 1 julius julius 64 Feb 10 12:37 4 -> 'anon_inode:[eventpoll]'
lr-x------ 1 julius julius 64 Feb 10 12:37 5 -> 'pipe:[98866]'
l-wx------ 1 julius julius 64 Feb 10 12:37 6 -> 'pipe:[98866]'
lrwx------ 1 julius julius 64 Feb 10 12:37 7 -> 'socket:[99715]'

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/87bc4a80-2507-45a7-8c64-4ae2c95aee52n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Kristopher Kahn]

Thank you Julius, it's more clear to me now that it's a stat on node_exporter itself, good to know.