Wazuh 3.11 - whodata does not work in File Server with close to 500000 files.

[Ranjith Kesavan]

Hi Team, 

we have a Windows file server with close to 500000 files in 80000 folders. Wazuh agent is configured to monitor two root directories with recursion set to 30 and who-data is enabled. we are getting syscheck logs for some files but not all. Also there is no audit data in it. We also have another server with just few files and whodata is working perfect with the same setup. Is there a limitation with OSSEC FIM regarding how many files we can monitor with who-data or in general ? Any help is appreciated. 

Thank you,

Ranjith. 

[Carlos Ridao]

Hello Ranjith,

I will try to help you here. There is no limit of files FIM can monitor with Whodata. Lets see if there is something wrong in your configuration. 

Could you please paste here the syscheck configuration you are using for the Windows file server with a lot of files and for the other server too?

Also, can you attach an extract of your `/var/ossec/log/ossec.log` file (assuming default installation folder) from the Windows file server? You can use this command to do so:

tail -n100/var/ossec/log/ossec.log

With this information I may be able to help you.

Regards,

Carlos.

[Carlos Ridao]

Hi Ranjith,

First of all, thank you for responding with more details and attaching those logs. Examining them I saw you have a lot of `filesize is larger than the maximum allowed (1024 MB). File skipped` messages. Can you ensure the syscheck events you are NOT receiving are related to those files only?

Just in case you want to get rid of those messages, you can change the maximum size allowed. This way you will also monitor those large files BUT take into account that the larger the monitored file the longer will take Wazuh to perform the scans. Also it is NOT recommended to increase that value if you want to use `report_changes` functionality. In case you want to increase the maximum size, you can do it by adding `syscheck.file_max_size=1024` to `local_internal_options.conf` and changing that value to anything between 0 and 4095. A value of 0 means to disable the size restriction at all. 

Regarding the missing syscheck events and audit data, I will need more information to understand what's happening on your environment. To obtain that information I need you to enable Syscheck debug mode in your windows file server by adding the following line to `local_internal_options.conf` file, located in the same path Wazuh is installed, and then restarting Wazuh:

syscheck.debug=2

Once debug mode is enabled and you have restarted wazuh, I need you to trigger one alert by modiying a file inside one of those monitored dir. See if an alert is raised for that file and also attach here the `ossec.log`.

If possible, leave debug mode enabled during some time and send me the `ossec.log` just like you did last time. Then you can disable again debug mode by removing that line from `local_internal_options.conf` and restarting Wazuh.

Finally, keep in mind that having 500000 files in 80000 folders, a queue_size of 60000 and events_per_second set to 999 in the `ossec.conf` means that if your network is not fast enough you may lose some events.

Kind regards,

Carlos.

[Ranjith Kesavan]

The only message I see regarding Whodata in the debug logs is 

2020/04/09 17:38:21 ossec-agent[7800] syscheck.c:187 at Start_win32_Syscheck(): INFO: (6003): Monitoring directory: 'd:\safanad', with options 'perm | size | owner | group | md5sum | sha1sum | sha256sum | mtime | inode | whodata | attributes'.

2020/04/09 17:38:21 ossec-agent[7800] syscheck.c:187 at Start_win32_Syscheck(): INFO: (6003): Monitoring directory: 'd:\dfssoftware', with options 'perm | size | owner | group | md5sum | sha1sum | sha256sum | mtime | inode | whodata | attributes'.

There is no logs regarding Auditd at all. 

Any help is appreciated. 

Thank you,

Ranjith. 

On Thursday, April 9, 2020 at 6:29:09 PM UTC+4, Ranjith Kesavan wrote:

HI Carlos,

Thank you for your quick response. Please find attached the ossec.log after debugging is enabled and file access was performed. 

The file size is set intentionally. Most of the files in the folder are small. who-data does not work for any of these. Sometimes, we dont recieve syscheck alerts on even smaller files. Regarding the queue size, both the file server and Wazuh manager are in the same VLAN. So we are not concerned about the network latency. 

[Suat Toksöz]

Hi Ranjith Kesavan,

Are you getting logs for the 500,000 folders ? What is the log size you need? Also, what is the wazuh manager structure (master, worker ) you are handling this operations?

Thanks

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f368bbb8-27b5-44f3-9515-a8ddd2dbb703%40googlegroups.com.

--

Best regards,

Suat Toksoz

[Ranjith Kesavan]

Hello Suat, 

No. We have close to 82000 folders and 500000 files. we dont get logs from all the folders. Not sure how many folders I am getting logs from as the file access is dynamic. Regarding the architecture, we have Wazuh manager configured on a different server. This issue happens on Windows fileserver where we have a Windows agent Installed. Yes. I am managing the operations. 

Thank you,

Ranjith Kesavan.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f368bbb8-27b5-44f3-9515-a8ddd2dbb703%40googlegroups.com.

[Jose Luis Carreras Marin]

Good morning Ranjith Kesavan, first of all, sorry for the late response.

I've been analyzing this problem in depth. I'll tell you the critical points I've concluded:

• In wazuh version 3.11, when syscheck launches the scan_on_start, it takes a long time to send all the necessary alerts for new files. This delay is established by these syscheck variables:

# Syscheck checking/usage speed. To avoid large cpu/memory
# usage, you can specify how much to sleep after generating
# the checksum of X files. The default is to sleep one second
# per 100 files read.
syscheck.sleep=1
syscheck.sleep_after=100

• While these alerts are being sent, whodata is waiting for the scan to be completed. 

• Seeing the large amount of files you try to monitor, the scan could take even hours. Also, your configuration shows a "frequency" of 2 hours (7200 seconds), which makes me think that most of the time, whodata is waiting for this reason.

For all this, my recommendation is an update to 3.12, as syscheck has received many important changes, and now whodata works with much greater reliability.

If you need any help, do not hesitate to ask us, here you have relevant information in our documentation:

https://documentation.wazuh.com/3.12/upgrade-guide/upgrading/latest_wazuh3_minor.html#upgrading-latest-minor

I also leave you here the 3.12 update notes, where you can see, among others, that File Integrity Monitoring (syscheck) now works with a SQL database, you can switch between disk and memory storage:

https://documentation.wazuh.com/3.12/release-notes/release_3_12_0.html

[Ranjith Kesavan]

Thank you Jose, 

I will test the upgrade and update you.