fluentd_tail_file_position and fluentd_tail_file_inode is not sync with pos_file

[Sarthak Sahu]

Hi All,

I am using tail plugin with "pos_file_compaction_interval" option. With this option pos_file is getting updated by removing all deleted tailed file information.

But it shows all deleted tailed file information  while scrapping fluentd_tail_file_position  and fluentd_tail_file_inode metrics.

Expectation: fluentd_tail_file_position and fluentd_tail_file_inode should show the information as per pos_file. It should not show the deleted tailed file information.

Please let me know why these metrics still showing all deleted tailed files information.

Regards

Sarthak Sahu

 

[Suvelee Sarpotdar]

Hi Sarthak,

We are consuming in tail prometheus metrics as well and I found this bug sometime back. Here is the thread where I first reported it - https://groups.google.com/g/fluentd/c/5GUoAPn2faQ/m/F9z_AMQMDgAJ. 

So to explain the bug in short, in_tail_prometheus plugin doesn't check if Fluentd has removed a file from its watch list. I found  a way to check if the file is unwatched or not. But a bigger issue is the prometheus library doesn't provide you a way to remove one entry from its map/dictionary. The only way the prometheus library allows to removal of the values is de-register the entire map. Once we de-register, metrics will become completely blank for a moment. Then we re-register the keys if you will and add the values for metrics for the files that are still being watched.

I had potential code fix ready however the prometheus plugin has undergone some changes. I will need to work my changes on top of those changes to submit a fix. 

Thanks,

Suvelee