fim.db management

[Veera]

Hi,

While configuring FIM to scan a few NFS volumes, the FIM database gets created as expected. However, I noticed that whenever there is any change in ossec.conf or in the list of volumes configured under Syscheck, the file /var/ossec/queue/fim/db/fim.db starts rebuilding again from around 12 KB.

Is this expected behavior?

Also, if we add one or two additional volumes to Syscheck (FIM), will the existing data in /var/ossec/queue/fim/db/fim.db be lost and rebuilt from scratch, including the newly added volumes for event generation during the next scheduled scan?

In the long run, is it recommended to scan only one Filesystem  per agent?
sometimes scanning 2 or more FS by this method only generate events for only one Filesystem.


Case 2:
If I have three volumes(File systems) configured for Syscheck scanning, is it possible to schedule them separately (for example: Volume 1 on Monday, Volume 2 on Wednesday, and Volume 3 on Friday)?
Would this approach conflict with the way /var/ossec/queue/fim/db/fim.db is maintained and used?

[Pablo Ariel Gonzalez]

Hi Veera,

In general, seeing the FIM database (fim.db) being re-initialized after Syscheck/FIM configuration changes can be expected, as the baseline may need to be rebuilt when the monitored scope or settings change. However, the exact behavior and the best approach depend heavily on the environment and configuration.

To provide more accurate guidance, could you please share a bit more context?

• Which Wazuh version (manager and agent) are you using?

• Are all the NFS filesystems being scanned from a single agent, or is it possible to distribute them across multiple agents/hosts?

• Approximately how large are the filesystems (number of files / total size)?

• What scan frequency do you need for each volume (e.g. daily, weekly, near-real-time)?

• Are you using (or considering) realtime or whodata monitoring for any of these paths?

• What operating system and mount options are used for the NFS volumes?

With this information, we can better determine whether rebuilding the FIM database is expected in your case and suggest a more suitable design (scope, scheduling, or agent distribution).

[Veera]

Hello,
Thanks you  response.
Here are the answers to your questions:

Which Wazuh version (manager and agent) are you using?

Wazuh Manager&agent = 4.14.0-1

 

Are all the NFS filesystems being scanned from a single agent, or is it possible to distribute them across multiple agents/hosts?

Yes, NFS filesystems are being scanned one volume per  agent.
I am planning to distribute them across multiple agents in the future. But for now , I am experiencing this issue of whenever multiple NFS filesystems are scanned from a single agent , events for only one NFS filesystem are generated.(not sure ,why)

Approximately how large are the filesystems (number of files / total size)?

approximately 25TB-70TB data with approx 10-150 million files.

What scan frequency do you need for each volume (e.g. daily, weekly, near-real-time)?

weekly for now... 

Are you using (or considering) realtime or whodata monitoring for any of these paths?

no , I have set realtime=no,  whodata(unsure) not mentioned.

What operating system and mount options are used for the NFS volumes?

OS: RHEL 9
Mount options: nfsvers=3, soft,retry=1,rw,rsize=32768,wsize=32768,sec=sys

Please let me know if you need any more information.
Any specific documentation to refer for scanning multiple NFS filesystems would be helpful.

[Pablo Ariel Gonzalez]

Hi Veera,

There is no specific Wazuh documentation focused on File Integrity Monitoring configurations for large NFS filesystems, mainly because this is not the most common FIM use case. This does not mean it is not possible or that it is “unsupported”; rather, it requires carefully defining the most appropriate configuration based on the size of the environment, the number of files, and the actual monitoring objective.

For volumes of the scale you mentioned (tens of terabytes and millions of files), it is critical to clearly understand what needs to be monitored in order to achieve a stable and manageable implementation.

To better assist you, could you please clarify a few points?

* What types of changes do you actually need to detect?

• Other types of changes
• Permission or ownership changes

• File creation or deletion

• File content modifications

• Other types of changes

* Do these controls need to apply to the entire NFS filesystem, or can they be limited to specific directories or file types?

* Is the primary goal security/auditing or inventory/visibility?

If possible, it would also be very helpful if you could share your current Syscheck/FIM configuration.
Please review it before sharing and obfuscate any sensitive or personal data if necessary.

With this information, we will be able to evaluate how to properly tune FIM for this type of implementation.

[Veera]

Hi ,

Here are details,

 What types of changes do you actually need to detect?

basic file operations such as:

WRITE: Data being written to a file.
CREATE: New files or directories being created.
DELETE/REMOVE: Files or directories being deleted.
SETATTR: File attributes being changed (e.g., permissions, ownership).
GETATTR: File attributes being read.
RENAME: Files or directories being renamed or moved.
LINK: Hard links being created or removed.
READDIR: Directory contents being listed
. 

Do these controls need to apply to the entire NFS filesystem, or can they be limited to specific directories or file types?

Yes , at the moment we cannot limit to specific directories or file types,
 we need to monitor the entire NFS filesystem for all types of changes listed above.

 Here , can you provide examples of how to configure Wazuh to monitor NFS mounts for full mount and limited to specific directories or file types?
 That will help us to understand better the configuration options available.

 Is the primary goal security/auditing or inventory/visibility?

 I would say both are equally important.

Looking for Data visibility / Inventory,  Compliance / Audit, Threat detection / Response ,  Operational / Performance insights are expected.


Attached here is the syscheck configuration we are using currently for NFS mounts monitoring.
Review and suggest any changes if required to meet the above requirements.

Refer  https://groups.google.com/g/wazuh/c/Wfyk5uU-apY/m/9T3OcJXNBAAJ  also ... 

Thanks 

[Pablo Ariel Gonzalez]

Hi Veera,

   I am analyzing the information you have provided so that I can offer you the best alternative for your use case. I will write to you again when I have more information to share.

Thanks,

[Pablo Ariel Gonzalez]

Hi Veera,

   Based on the requirements you described, what you are looking for goes beyond a File Integrity Monitoring (FIM) use case. Most of the events you expect (such as READDIR, GETATTR, and detailed access auditing) fall under file activity and access monitoring, not FIM. In addition, on NFS mounts the agent cannot reliably observe or attribute these types of access events in the same way it can on local filesystems, so features such as realtime monitoring or who-data attribution do not provide the audit semantics you are expecting.

   From a scalability perspective, asking a single agent to scan tens of terabytes and millions of files is not realistic. This not only impacts local resources, but also forces massive metadata and content reads over the network. Even with scans every 12 hours, it is very unlikely that a full scan could complete consistently, and FIM also has practical design limits that make this scenario unsuitable.

   For these reasons, the recommended approach is to split responsibilities, while still keeping Wazuh as the central platform for management and correlation:

• Access auditing: use mechanisms such as auditd (or equivalent solutions) on the NFS server or on the hosts mounting the NFS to capture file activity and access events.
  This information can be sent to Wazuh using an agent or, where appropriate, by ingesting logs via localfile, allowing it to be correlated and managed centrally.

• Inventory / data visibility: rely on capabilities provided by the storage system itself or on external batch processes.
  For example, in CephFS, this is supported by the Metadata Server (MDS), which maintains global filesystem metadata and allows inventory and statistics to be generated without traversing the entire filesystem from a client.
  The resulting inventory or summary data can also be forwarded to Wazuh for analysis and tracking.

• File integrity: use Wazuh FIM strictly for file integrity monitoring, applied to a reduced scope that is relevant from a security perspective, rather than as a full NFS scanner.

   This approach aligns better with both the functional requirements and the scale of the environment, while allowing Wazuh to remain the central point for ingestion, correlation, and alerting without forcing FIM to cover use cases it was not designed for.

   If you have any questions, need further clarification, or would like us to review how this set of components could work in your environment, please let us know and we can go into more detail.

[Veera]

Hi  Pablo ,

Thanks for the Brief analysis.

Based on  your inputs, I have to re-align the scanning requirements.

What types of changes do you actually need to detect?

The changes that can be detected using check_all="yes" with realtime="no" are the only ones required to be captured in FIM.

Do these controls need to apply to the entire NFS filesystem, or can they be limited to specific directories or file types?

Yes, we cannot be restricted to specific directories or file types.

Is the primary goal security/auditing or inventory/visibility?

To use Wazuh FIM strictly for file integrity monitoring, focusing only on a reduced, security-relevant scope rather than performing a full NFS scan.”

Now ,
Is it feasible to complete the FIM scan for NFS volumes up to 50 TB in size?
What scan interval would you recommend for the syscheck schedule (instead of every 12 hours)?
Should we limit each scanner/Wazuh agent to monitoring 2–3 volumes, with a maximum total capacity of 50 TB per agent?
What would be the best recommended configuration of syscheck settings for the above?

[Pablo Ariel Gonzalez]

Veera,

Regarding your questions, there are a few key points to clarify first. The main limiting factor for FIM is not the filesystem size (in TB), but the total number of monitored files. Wazuh includes the file_limit parameter as a safeguard, with a default value of 100,000 files, specifically to prevent severe performance impact.

Although this limit can be increased, FIM maintains a local database and performs full filesystem traversal, which introduces practical scalability limits by design, especially in large NFS environments.

Scan feasibility and scheduling: 

For large NFS volumes, the most appropriate approach is to distribute the load across as many agents as necessary, so that each agent handles a number of files it can scan consistently.

A good practice is to define the scan interval based on the actual time required to complete a full scan. As a general rule, the interval should be at least twice the duration of a full scan, to avoid overlapping executions.
In this scenario, we recommend starting with a long test scan (for example, around 48 hours) and then adjusting the schedule based on how long the process actually takes in your specific environment.

Syscheck configuration considerations:

For this use case:

• Using check_all="yes" is not recommended unless all checks are strictly required, as it significantly increases the per-file cost.

• It is preferable to enable only the required attributes (mtime, size, permissions, ownership, etc.).

• If the system runs other resource-intensive processes, it may be advisable to disable scan_on_start, to avoid resource contention when the agent restarts.

If you have any questions or need further clarification, please let us know and we can go into more detail.

[Veera]

Thanks for the  recommendations.

I will be setting up the same ..

• If the system runs other resource-intensive processes, it may be advisable to disable scan_on_start, to avoid resource contention when the agent restarts.  - Since this host is a dedicated scanner only (no other resource-intensive workloads), scan_on_start can remain enabled? . However, please suggest any recommended tuning options to better utilize the system resources and optimize scan performance.  

• we recommend starting with a long test scan (for example, around 48 hours) and then adjusting the schedule based on how long the process actually takes in your specific environment.    -  Please clarify how we can determine the real scan time for a specific endpoint (for example, a scanner host with 3 NFS volumes), so that the schedule can be set appropriately.

• From the syscheck snippet shared earlier,   For an NFS-dedicated scanner, is it acceptable / recommended to exclude local paths like /etc from FIM, so the scanner focuses only on the required NFS mount paths as they are reported in fim events.

• with all your recommendations , is that ok to schedule 2 or more volumes in a single scanner/agent?

[Pablo Ariel Gonzalez]

Veera:

Below are our answers point by point.

1) scan_on_start on a dedicated scanner:

On a host dedicated exclusively to FIM, it is acceptable to keep scan_on_start enabled. This ensures the baseline is rebuilt after a restart, as long as restarts are infrequent and controlled.

2) How to determine the real scan time:

The actual scan duration should be determined by reviewing the Wazuh agent logs:

• identify when the syscheck/FIM scan starts,

• identify when the scan finishes,

• the difference between those timestamps is the real scan time.

As a general rule, the scan interval should be at least twice the duration of a full scan, to avoid overlapping executions. A good starting point is to run a long test scan (for example, around 48 hours) and then adjust the schedule based on the observed behavior.

3) Excluding local paths on an NFS-dedicated scanner:

The file_limit parameter applies at the agent level, not per filesystem or per path. All files from all monitored paths count toward the same limit.

On a scanner dedicated to NFS:

• it is valid and recommended to exclude most local operating system paths, as they do not add value to the use case and only consume part of the file_limit,

• however, keeping /etc monitored is often a good practice, since it contains critical system configuration and usually has a low file count, resulting in minimal performance impact.

In short:

• exclude /bin, /usr, /lib, etc.,

• keep /etc if you want to preserve basic OS integrity without significantly affecting NFS scanning.

4) Monitoring multiple volumes per scanner/agent:

It is possible to monitor more than one NFS volume per agent, but the key factor is not the number of volumes, it is the total number of files being monitored.

If you don't have a reliable estimate of the file count per volume in advance, the safest approach is to start with one volume per agent, measure scan duration and stability, and only then evaluate adding more volumes if the behavior remains consistent.

5) Final configuration considerations:

For this scenario, we recommend:

• avoiding check_all="yes" unless it is strictly required,

• keeping realtime="no",

• adjusting the scan frequency based on real measurements from your environment.

[Veera]

Hi Pablo,

I have configured the NFS volumes using the recommended FIM settings:
check_sum="yes", check_owner="yes", check_group="yes", check_perm="yes", check_size="yes", realtime="no", and recursion_level="100".

The Wazuh agents are configured to monitor single or multiple volumes, each maintained within the defined file limit, with a scan schedule of 48 hours. However, even after 144 hours, FIM events are being reported from only one server, while no events are received from the other servers.

Additionally, the server that is reporting events has three configured volumes, but FIM activity is observed from only one of those volumes.

Could you please advise which logs should be reviewed for troubleshooting and suggest the next steps to investigate this behavior?

Thanks ...

[Pablo Ariel Gonzalez]

Veera,

First, with check_sum="yes" enabled, FIM must read the full contents of every file in order to calculate checksums. On large NFS volumes, this significantly increases scan time and I/O cost, and it is one of the main reasons scans can take much longer than expected.

Given that, it is very likely that:

• the scan has only completed the first volume so far, and

• the other volumes have not been reached yet, so no events are generated from them at this stage.

Additionally, if the agent reaches the configured file_limit before all volumes are scanned, FIM will stop registering new files. When this happens, you should see corresponding warnings in the agent log (/var/ossec/logs/ossec.log) indicating that the file limit has been reached or that files are being skipped.

Recommended next step

To validate this and get a clear baseline, we recommend a partial test:

1. Configure the agent to monitor only a single NFS volume.

2. Let the scan run until it fully completes (confirm start and end messages in ossec.log).

3. Measure how long that scan actually takes.

4. Once confirmed, gradually add additional volumes and observe how scan time and behavior change.

This approach helps ensure that scans can complete reliably and provides concrete data to size the number of volumes per agent and the scan interval appropriately.

If you’d like, you can share the relevant ossec.log excerpts (scan start/end and any file limit warnings), and we can help review them.

[Veera]

Thanks for the update.

Is the Recommended next steps,  to be followed with check_sum="yes" enabled,  or to be disabled?

Shall i reduce the time to 24 hours  instead of 48 hours , so that I can have the results in 48n hours?

[Veera]

  With the 3-volume setup, the FIM process starts and stops over a period of 3 days.
However, events are reported only on 27th Jan. How can we correlate the "File integrity monitoring scan ended." message with the events shown in the FIM console?
If FIM events are generated at the end of the second scheduled scan, why are they not appearing in the logs?  

[root@my-vm1 Jan]# zgrep "File integrity monitoring scan started" ossec-20.log.gz
2026/01/20 09:19:02 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
2026/01/20 14:23:49 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
2026/01/20 16:20:02 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
2026/01/20 16:20:09 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan started" ossec-21.log.gz
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan started" ossec-22.log.gz
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan started" ossec-23.log.gz
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan started" ossec-24.log.gz
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan started" ossec-25.log.gz
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan started" ossec-26.log.gz
2026/01/26 06:03:40 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan" ossec-26.log.gz
2026/01/26 06:03:40 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan" ossec-25.log.gz
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan" ossec-24.log.gz
2026/01/24 06:02:06 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan" ossec-23.log.gz
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan" ossec-21.log.gz
[root@my-vm1 Jan]# zgrep "File integrity monitoring scan" ossec-20.log.gz
2026/01/20 09:19:02 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
2026/01/20 14:23:49 wazuh-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds
2026/01/20 14:23:49 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
2026/01/20 14:23:50 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.
2026/01/20 16:20:02 wazuh-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 172800 seconds
2026/01/20 16:20:02 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
2026/01/20 16:20:09 wazuh-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 172800 seconds
2026/01/20 16:20:09 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
[root@my-vm1 Jan]#

[root@my-vm1 Jan]# zgrep "File integrity monitoring scan ended" ossec-2*.log.gz
ossec-20.log.gz:2026/01/20 14:23:50 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.
ossec-24.log.gz:2026/01/24 06:02:06 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.
[root@my-vm1 Jan]#grep "File integrity monitoring scan ended" ../../../ossec.log
[root@my-vm1 Jan]#

[Pablo Ariel Gonzalez]

Veera:

The behavior you are observing (multiple “scan started” messages and very few “scan ended” messages) is expected in this scenario and is directly related to the cost of the scan. With check_sum="yes" enabled, FIM must read the full contents of every file in order to calculate checksums. On large NFS volumes, this makes the scan extremely slow and it can take several days to complete.

FIM performs scans sequentially and does not store the traversal progress. This means that:

• if a scan starts but does not complete (for example due to an agent restart, configuration reload, or frequency change),

• the next scan will start again from the beginning of the traversal,

• and volumes that have not yet been reached will not generate events because they are not yet part of the baseline.

This explains why you may see:

• activity only from a single volume,

• events concentrated on a specific date,

• and repeated scan restarts without consistent progress.

This behavior does not indicate a malfunction; it means the scan is operating at the edge of what is feasible with the current configuration.

As a practical way to move forward and demonstrate progress, we recommend a temporary adjustment:

• Disable check_sum temporarily to significantly reduce scan cost.

• Keep the remaining checks enabled (owner, group, permissions, size).

• Configure a long scan interval (for example, 4 days).

• Avoid agent restarts or configuration changes while the scan is running.

This allows you to confirm whether the scan can complete successfully and whether all volumes are reached. Once that is validated, you can continue working toward a more refined final solution based on measured behavior.

[Veera]

Hi Pablo,

Apologies if my previous message was confusing — sharing the full grep logs may have been misleading. I am able to clearly track the exact 48-hour FIM scan start and end timestamps, so that part is confirmed.

However, my concern is different. I realize I may be asking a lot of questions, but I’ve had to test multiple scenarios and want to ensure I understand the behavior correctly.

At the moment, I have not yet disabled check_sum="yes".

Case 1:

One Wazuh agent is scanning a single volume larger than 50 TB. The file count is roughly half of the configured file_limit. The FIM scan took 146 hours and 10 minutes to complete. A second scan has just started today and is still in progress. The configured scan frequency is 48 hours, but so far, no events have been generated.

Case 2:

Another Wazuh agent is scanning four volumes with a combined size of about 4 TB and a file count of up to 50 million. The FIM scan completed in approximately three days, but events were generated for only one of the volumes.

My question is: If a FIM scan for an individual volume completes successfully, when should the related events be expected?

Additionally, I reconfigured the multiple volumes to be scanned one by one and was able to successfully receive alerts for the second volume on the same agent. However, after that, events from the first volume stopped appearing.

So in a multi-volume setup on a single agent, are events generated and reported independently per volume, rather than all at the same time?

Thanks in advance for your guidance.

[Pablo Ariel Gonzalez]

Veera:

Thanks for the detailed explanation of both cases.

When should FIM events be expected after a scan completes? A key point to clarify is that FIM does not generate events simply because a scan finishes.

The message “File integrity monitoring scan ended” only indicates that the filesystem traversal has completed and the baseline is consistent. FIM events are generated only when detectable changes occur, such as file creation, deletion, metadata changes, or content changes based on the enabled checks.

These events can be generated:

• during a scan, as directories are traversed, or

• after a scan, when a subsequent scan compares the current state against the baseline.

It is therefore normal for a scan to complete successfully and still produce no events, if no relevant changes occurred. To confirm coverage for a specific volume, the correct approach is to perform a controlled change (for example, create or modify a file) and verify that the event is detected.

Multi-volume behavior and large file counts: In a multi-volume setup on a single agent, events are not generated per volume at the same time. FIM scans paths sequentially at the agent level, so events may appear first for the volume traversed earlier, while others appear later or not at all if no changes are detected.

Regarding the case with ~50 million files: while increasing file_limit makes this technically possible, it is well beyond the default and recommended operating range of FIM. In such scenarios, long scan times, uneven event visibility across volumes, and sensitivity to scan order are expected behaviors rather than errors.

If needed, the next step to validate behavior is to perform controlled changes on each volume and observe whether those changes generate events, or to reduce scan cost (for example by disabling check_sum) to make behavior more predictable