Real-Time FIM: Practical Limit on the Nubmer of Monitored Files/Dirs
Peter Larin
I'm new to Wazuh and I'm tasked to set up realtime FIM on an Ubuntu web server with numerous vhosts. The main purpose is to catch in realtime cryptominers, encryptors and other malware made possible through imperfect php code.
It works fine as a proof of concept when I monitor the defailt dirs like /bin, and one vhost in /var/www. The integration with VirusTotal is also very responsive.
However, things change when I try to monitor all of vhosts, and we are talking here about a few million dirs and files in total. I have set up all possible restrictions (monitor only .php and a few more extensions), exclusions (do not monitor cache dirs and so on), however, FIM is behaving in a strange way. I remove fim.db and fim.db-journal, restart the agent, then fim.db and fim.db-journal reappear instantly (from some backup location??) and they never grow, they stay the same size all along. After my last restart (14 hours ago!) they haven't grown by a single byte. When I try to query fim.db it shows 0 records, however it is several MB in size. Before restarting the agent 14hrs ago, I added a couple new files to /bin but the Inventory still hasn't picked them up, so I was wondering what is actually happening. In htop I see a single process wazuh-syscheckd occupying a single CPU core at almost 100%.
So my question is: is is hopeless to trying to monitor millions of files?
And another one: will it help if I switch off the realtime mode - maybe in the absence of it, the agent will index the hashes faster, then switch the realtime mode back on? Any other suggestions? Thank you! Peter
rodrigo....@wazuh.com
What you are experiencing is not uncommon as many users experience issues when monitoring that magnitude of files.
Tuning does help a little but the limiting factor will continue to be the server's memory and processing resources.
Recommended Actions for mitigation:
Review the Diff Check Settings
The <diff> option can consume a large amount of memory when enabled on large files.
Consider disabling it or setting a lower <limit> to control resource usage.
Refer to the Wazuh diff check documentation: https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/syscheck.html#diff
Exclude Unnecessary Directories
Avoid scanning entire drives or large system paths.
Use the <ignore> tag to exclude non-critical directories and reduce scanning load.
See the documentation on ignored files: https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/syscheck.html#ignore.
Check the size and count of monitored files
If the total size or number of files is large, the scan will take significant time and memory, especially if diff checks are enabled.
Analyze System Resources
Use tools like Process Explorer (on Windows) or top / htop (on Linux) to monitor:
Total system memory usage
Memory consumption by wazuh-agent.exe
Number of open handles for the agent process
Review Agent Configuration
Certain configurations can cause long-term memory issues, such as specifying files instead of directories in <directories> blocks, if realtime or whodata options are enabled for them.
Peter Larin
Do you think it will help to switch off realtime mode to let FIM create the full db, and then re-enable realtime?
Thank you
rodrigo....@wazuh.com
You could increase the capacity with something like this:
sysctl -w fs.inotify.max_user_watches=1048576
sysctl -w fs.inotify.max_queued_events=32768
But this does not guarantee that other parts of the monitoring won't also reach full capacity.
The reality is that the FIM module is not designed to track such a large number of files.
To answer your last question I do believe that if you temporarily disable realtime FIM to wait for a full sync will help, yes.
But keep in mind that other bottlenecks might occur as I said above.