Wazuh who-data not working

Nate

unread,

Jan 16, 2020, 7:06:28 PM1/16/20

to Wazuh mailing list

Hello!

We are running Wazuh 3.9 and have enabled who-data for both Windows and Linux instances. Unfortunately, it does not appear to be working (i.e., the syscheck log data does not record any "audit" data).

We have enabled in ossec.conf and restarted the wazuh agent.

I do see this in the logs for a Linux instance I am testing with:

/var/ossec/logs/ossec.log

2020/01/16 23:47:14 ossec-syscheckd: ERROR: (6642): Audit health check couldn't be completed correctly.

2020/01/16 23:47:14 ossec-syscheckd: WARNING: Audit events reader thread not started.

And running auditctl -l only produces:

-a never,task

Our results in the log look like this:

"syscheck":{

"path":"/etc/hosts.allow",

"size_before":"421",

"size_after":"433",

"perm_after":"100644",

"uid_after":"0",

"gid_after":"0",

"md5_before":"4b8ee210c257bc59f2b1d4fa0cbbc3da",

"md5_after":"acb2289fba96e77cee0a2c3889b49643",

"sha1_before":"d3452e66d5cfd3bcb5fc79fbcf583e8dec736cfd",

"sha1_after":"b87a0e558ca67073573861b26e3265fa0ab35d20",

"sha256_before":"6504e867b41a6d1b87e225cfafaef3779a3ee9558b2aeae6baa610ec884e2a81",

"sha256_after":"bfa1c0ec3ebfaac71378cb62101135577521eb200c64d6ee8650efe75160978c",

"uname_after":"root",

"gname_after":"root",

"mtime_before":"2018-07-10T14:04:25",

"mtime_after":"2018-07-10T14:05:28",

"inode_after":268234,

"diff":"10a11,12\n> 10.0.12.34\n",

"event":"modified",

},

And not like this:

"syscheck":{

"path":"/etc/hosts.allow",

"size_before":"421",

"size_after":"433",

"perm_after":"100644",

"uid_after":"0",

"gid_after":"0",

"md5_before":"4b8ee210c257bc59f2b1d4fa0cbbc3da",

"md5_after":"acb2289fba96e77cee0a2c3889b49643",

"sha1_before":"d3452e66d5cfd3bcb5fc79fbcf583e8dec736cfd",

"sha1_after":"b87a0e558ca67073573861b26e3265fa0ab35d20",

"sha256_before":"6504e867b41a6d1b87e225cfafaef3779a3ee9558b2aeae6baa610ec884e2a81",

"sha256_after":"bfa1c0ec3ebfaac71378cb62101135577521eb200c64d6ee8650efe75160978c",

"uname_after":"root",

"gname_after":"root",

"mtime_before":"2018-07-10T14:04:25",

"mtime_after":"2018-07-10T14:05:28",

"inode_after":268234,

"diff":"10a11,12\n> 10.0.12.34\n",

"event":"modified",

"audit":{

"user":{

"id":"0",

"name":"root"

},

"group":{

"id":"0",

"name":"root"

},

"process":{

"id":"82845",

"name":"/bin/nano",

"ppid":"3195"

},

"login_user":{

"id":"1000",

"name":"smith"

},

"effective_user":{

"id":"0",

"name":"root"

}

},

Any ideas? Thank you!

Juan Pablo Saez

unread,

Jan 17, 2020, 11:07:23 AM1/17/20

to Wazuh mailing list

hello Nate,

I do see this in the logs for a Linux instance I am testing with:
/var/ossec/logs/ossec.log

2020/01/16 23:47:14 ossec-syscheckd: ERROR: (6642): Audit health check couldn't be completed correctly.
2020/01/16 23:47:14 ossec-syscheckd: WARNING: Audit events reader thread not started.

Let's see what could be causing this error:

Before running the Who-data mode, the manager first verifies whether the auditd service is working. Seems like a bug in this verification is causing the manager to wrongly identify the auditd thread as stopped.

Could you start the Wazuh agent and check whether the auditd is active or not ? # systemctl status auditd
If the agent logs keep showing the Audit events reader thread not started. warning but, at the same time, the auditd service is running, the error is in the manager side.

On the other hand, I think you can bypass this bug disabling the whodata auditd startup healtcheck. You should include the further block inside the <syscheck> configuration section.

<whodata>
    <startup_healthcheck>no</startup_healthcheck>
</whodata>

Let me know how it goes. Greetings,
JP Sáez

Nate

unread,

Jan 17, 2020, 12:12:08 PM1/17/20

to Wazuh mailing list

Thank you for your help, JP!

The auditd service is actively running:

auditd (pid 6461) is running...

When I add the startup_healthcheck to the config, this is the error that results in the logs:

2020/01/17 16:54:30 ossec-syscheckd: INFO: (6026): Audit health check is disabled. Real-time Whodata could not work correctly.

2020/01/17 16:54:30 ossec-syscheckd: WARNING: Audit events reader thread not started.

Based on what you have said, it sounds like the error then is on the manager side? Any ideas on what steps we need to take on the manager side?

Juan Pablo Saez

unread,

Jan 21, 2020, 5:57:17 AM1/21/20

to Wazuh mailing list

Hello again Nate,

Based on what you have said, it sounds like the error then is on the manager side? Any ideas on what steps we need to take on the manager side?

Thanks for reporting the asked data and trying the suggested workaround. To ease this issue resolution, let me briefly explain how Whodata mode works:

On windows systems, the who-data information is extracted through setting audit policies and subscribing to event channels (This is done automatically for most supported Windows system).
On linux systems, the who-data information is gathered from a combination of inotify watchers and the auditd log: The basic info about the file event comes from inotify and the Who-data info comes from the audit log. This is the case for linux agents and managers.

Why is who-data mode failing in your linux agents and/or manager?

As I mentioned above, It could be a communication issue between Wazuh and the auditd process but we need to research yet. I'm afraid I need some more information to recreate your environment:

I'd like you to check if this error occurs in both linux agent and manager.
Are both agents and manager on Wazuh v3.9?
Could you check the auditd versions in the system with this problem? auditctl -v

Let me know how it goes. Greetings, JP Sáez

Nate

unread,

Jan 21, 2020, 11:54:56 AM1/21/20

to Wazuh mailing list

Thank you for your response!

It appears to be impacting both Windows and Linux endpoints.
Both agents and managers are running Wazuh v 3.9.2
Sample endpoint auditd version: auditctl version 2.6.5
Manager auditd version is the same at 2.6.5
I do not see the error in the manager logs /var/ossec/logs/ossec.conf (And the managers are not running the wazuh agent, only wazuh-manager. I am not sure if that matters?)

Thank you for your help!

Nate

unread,

Jan 23, 2020, 11:42:13 AM1/23/20

to Wazuh mailing list

I also enabled syscheck.debug=2 on /var/ossec/etc/internal_options.conf

But it does not provide much info:

2020/01/23 16:39:28 ossec-syscheckd[16316] syscheck_audit.c:1200 at audit_health_check(): DEBUG: (6279): Whodata health-check: Starting...

2020/01/23 16:39:28 ossec-syscheckd[16316] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB

2020/01/23 16:39:28 ossec-syscheckd[16316] syscheck_audit.c:934 at audit_healthcheck_thread(): DEBUG: (6255): Whodata health-check: Reading thread active.

2020/01/23 16:39:28 ossec-syscheckd[16316] syscheck_audit.c:1221 at audit_health_check(): DEBUG: (6257): Whodata health-check: Waiting creation event...

2020/01/23 16:39:29 ossec-syscheckd[16316] syscheck_audit.c:1170 at filterkey_audit_events(): DEBUG: (6251): Match audit_key: 'key="wazuh_hc"'

2020/01/23 16:39:38 ossec-syscheckd[16316] syscheck_audit.c:376 at audit_init(): ERROR: (6642): Audit health check couldn't be completed correctly.

2020/01/23 16:39:38 ossec-syscheckd[16316] syscheck.c:479 at main(): WARNING: Audit events reader thread not started.

2020/01/23 16:39:38 ossec-syscheckd[16316] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB

2020/01/23 16:39:38 ossec-syscheckd[16316] run_check.c:171 at start_daemon(): DEBUG: (6201): Setting SCHED_BATCH returned: '0'

2020/01/23 16:39:38 ossec-syscheckd[16316] syscheck_audit.c:938 at audit_healthcheck_thread(): DEBUG: (6256): Whodata health-check: Reading thread finished.

Juan Pablo Saez

unread,

Jan 24, 2020, 6:10:41 AM1/24/20

to Wazuh mailing list

Hello again Nate,

I’ve been reviewing our messages trying to figure out where is the error. I think the issue is in the already existing auditd rule you mentioned in your first message ( I should have noticed it back on that moment ):

-a never,task

Whodata won’t work whenever the rule above is loaded. Removing it from the auditd rules and restarting the Wazuh manager should be enough for Whodata to run normally. This rule is included by default on EC2 instances, maybe this is your case. The audit rule are usually located on the following paths:

/etc/audit/audit.rules
/etc/audit/rules.d/audit.rules

You could also delete the rule using auditctl -D.

Let me know how it goes. Also, I don't forget about your windows endpoints; let's have the Linux side working first and then we continue with the windows ones.

Best regards, JP Sáez

And running auditctl -l only produces:
-a never,task

Nate

unread,

Jan 24, 2020, 11:45:05 AM1/24/20

to Wazuh mailing list

Thank you, JP Sáez. I think we are getting closer.

I removed the rules from both an instance and the manager, restarted the auditd service on both, and restarted wazuh-agent and wazuh-manager.

Results on an instance:

sudo auditctl -l

No rules

Results on the manager:

sudo auditctl -l

-w /etc -p wa -k wazuh_fim

-w /usr/bin -p wa -k wazuh_fim

-w /usr/sbin -p wa -k wazuh_fim

-w /bin -p wa -k wazuh_fim

-w /sbin -p wa -k wazuh_fim

Log results on the instance still has:

2020/01/24 16:25:47 ossec-syscheckd: WARNING: Audit events reader thread not started.

But it no longer has the "ERROR: (6642): Audit health check couldn't be completed correctly."

Log results on the master look promising:

2020/01/24 16:44:06 ossec-syscheckd: INFO: (6018): Starting file integrity monitoring real-time Whodata engine.

Nate

unread,

Jan 24, 2020, 4:29:35 PM1/24/20

to Wazuh mailing list

I noticed when looking at the instance's configuration page, it said it was synchronized and Enable auditing (who-data) = no was set on the monitored directories.

So I set:

agent.remote_conf=0

in /var/ossec/etc/local_internal_options.conf

And then it worked on the instance after restarting the agent!

2020/01/24 21:24:22 ossec-syscheckd: INFO: (6019): File integrity monitoring real-time Whodata engine started.

# sudo auditctl -l

-w /etc -p wa -k wazuh_fim

-w /usr/bin -p wa -k wazuh_fim

-w /usr/sbin -p wa -k wazuh_fim

-w /bin -p wa -k wazuh_fim

-w /sbin -p wa -k wazuh_fim

-w /boot -p wa -k wazuh_fim

So, my next question is, why is this happening? I looked at both ossec.conf and agent.conf in the /var/ossec/etc/shared/ and /var/ossec/etc/shared/default (for both the manager and instance), and they all have whodata=yes.

Juan Pablo Saez

unread,

Jan 27, 2020, 3:48:03 AM1/27/20

to Wazuh mailing list

Hello again Nate,

I'm glad it is correctly working now, at least in the manager.

Could you paste here the agent `ossec.conf` and `agent.conf`? I want to recreate the environment. In the meantime, if you keep having issues with the Whodata FIM mode on the instance, I would like you to check again what happens after disabling the `startup_healthcheck` option.

Let me know how it goes. Best regards,

JP Sáez

Nate

unread,

Jan 27, 2020, 12:43:10 PM1/27/20

to Wazuh mailing list

Thank you for your reply.

Here is the ossec.conf:

<ossec_config>

<address>sanitized</address>

</server>

<config-profile>amzn, amzn2017, amzn2017.09</config-profile>

<notify_time>10</notify_time>

<time-reconnect>60</time-reconnect>

<auto_restart>yes</auto_restart>

<crypto_method>aes</crypto_method>

</client>

<client_buffer>

<queue_size>5000</queue_size>

<events_per_second>500</events_per_second>

</client_buffer>

<check_unixaudit>yes</check_unixaudit>

<check_files>yes</check_files>

<check_trojans>yes</check_trojans>

<check_dev>yes</check_dev>

<check_sys>yes</check_sys>

<check_pids>yes</check_pids>

<check_ports>yes</check_ports>

<check_if>yes</check_if>

<rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>

<rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>

<system_audit>/var/ossec/etc/shared/system_audit_rcl.txt</system_audit>

<system_audit>/var/ossec/etc/shared/system_audit_ssh.txt</system_audit>

<skip_nfs>yes</skip_nfs>

</rootcheck>

<scan-on-start>yes</scan-on-start>

</wodle>

<scan-on-start>yes</scan-on-start>

<java_path>wodles/java</java_path>

<ciscat_path>wodles/ciscat</ciscat_path>

</wodle>

<run_daemon>yes</run_daemon>

<log_path>/var/log/osquery/osqueryd.results.log</log_path>

<config_path>/etc/osquery/osquery.conf</config_path>

<add_labels>yes</add_labels>

</wodle>

<scan_on_start>yes</scan_on_start>

</wodle>

<scan_on_start>yes</scan_on_start>

<ignore>/etc/hosts.deny</ignore>

<ignore>/etc/mail/statistics</ignore>

<ignore>/etc/random-seed</ignore>

<ignore>/etc/random.seed</ignore>

<ignore>/etc/adjtime</ignore>

<ignore>/etc/httpd/logs</ignore>

<ignore>/etc/utmpx</ignore>

<ignore>/etc/wtmpx</ignore>

<ignore>/etc/cups/certs</ignore>

<ignore>/etc/dumpdates</ignore>

<ignore>/etc/svc/volatile</ignore>

<ignore>/sys/kernel/security</ignore>

<ignore>/sys/kernel/debug</ignore>

<nodiff>/etc/ssl/private.key</nodiff>

<skip_nfs>yes</skip_nfs>

<remove_old_diff>yes</remove_old_diff>

<restart_audit>yes</restart_audit>

</syscheck>

<log_format>command</log_format>

</localfile>

<log_format>full_command</log_format>

<command>netstat -tulpn | sed 's/$[[:alnum:]]\+$\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+$.*$:$[[:digit:]]*$\ \+$[0-9\.\:\*]\+$.\+\ $[[:digit:]]*\/[[:alnum:]\-]*$.*/\1 \2 ==

\3 == \4 \5/' | sort -k 4 -g | sed 's/ == $.*$ ==/:\1/' | sed 1,2d</command>

<alias>netstat listening ports</alias>

</localfile>

<log_format>full_command</log_format>

</localfile>

<active-response>

<ca_store>/var/ossec/etc/wpk_root.pem</ca_store>

<ca_verification>yes</ca_verification>

</active-response>

<log_format>plain</log_format>

</logging>

</ossec_config>

<ossec_config>

<log_format>audit</log_format>

<location>/var/log/audit/audit.log</location>

</localfile>

<log_format>syslog</log_format>

<location>/var/ossec/logs/active-responses.log</location>

</localfile>

<log_format>syslog</log_format>

<location>/var/log/messages</location>

</localfile>

<log_format>syslog</log_format>

<location>/var/log/secure</location>

</localfile>

<log_format>syslog</log_format>

<location>/var/log/maillog</location>

</localfile></ossec_config>

And here is the agent.conf:

<agent_config os="Linux">

<ignore>/etc/mnttab</ignore>

<ignore>/etc/hosts.deny</ignore>

<ignore>/etc/mail/statistics</ignore>

<ignore>/etc/random-seed</ignore>

<ignore>/etc/adjtime</ignore>

<ignore>/etc/httpd/logs</ignore>

<ignore>/etc/utmpx</ignore>

<ignore>/etc/wtmpx</ignore>

<ignore>/etc/cups/certs</ignore>

<ignore>/etc/dumpdates</ignore>

<ignore>/etc/svc/volatile</ignore>

<ignore>/etc/puppetlabs/mcollective/facts.yaml</ignore>

<ignore>/etc/mcollective/facts.yaml</ignore>

</syscheck>

<log_format>syslog</log_format>

<location>/var/ossec/logs/active-responses.log</location>

</localfile>

</agent_config>

<agent_config os="Windows">

<directories check_all="yes" whodata="yes">%WINDIR%/win.ini</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/system.ini</directories>

<directories check_all="yes" whodata="yes">C:\autoexec.bat</directories>

<directories check_all="yes" whodata="yes">C:\config.sys</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/CONFIG.NT</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/AUTOEXEC.NT</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/at.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/attrib.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/cacls.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/debug.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/drwatson.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/drwtsn32.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/edlin.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/eventcreate.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/eventtriggers.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/ftp.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/net.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/net1.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/netsh.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/rcp.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/reg.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/regedit.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/regedt32.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/regsvr32.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/rexec.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/rsh.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/runas.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/sc.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/subst.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/telnet.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/tftp.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/tlntsvr.exe</directories>

<directories check_all="yes" whodata="yes">%WINDIR%/System32/drivers/etc</directories>

<directories check_all="yes" whodata="yes">C:\Documents and Settings/All Users/Start Menu/Programs/Startup</directories>

<directories check_all="yes" whodata="yes">C:\Users/Public/All Users/Microsoft/Windows/Start Menu/Startup</directories>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\batfile</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\cmdfile</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\comfile</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\exefile</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\piffile</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\AllFilesystemObjects</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\Directory</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\Folder</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\Protocols</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Policies</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Security</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Internet Explorer</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\KnownDLLs</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurePipeServers\winreg</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\URL</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon</windows_registry>

<windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components</windows_registry>

<ignore>%WINDIR%/System32/LogFiles</ignore>

<ignore>%WINDIR%/Debug</ignore>

<ignore>%WINDIR%/WindowsUpdate.log</ignore>

<ignore>%WINDIR%/iis6.log</ignore>

<ignore>%WINDIR%/system32/wbem/Logs</ignore>

<ignore>%WINDIR%/system32/wbem/Repository</ignore>

<ignore>%WINDIR%/Prefetch</ignore>

<ignore>%WINDIR%/PCHEALTH/HELPCTR/DataColl</ignore>

<ignore>%WINDIR%/SoftwareDistribution</ignore>

<ignore>%WINDIR%/Temp</ignore>

<ignore>%WINDIR%/system32/config</ignore>

<ignore>%WINDIR%/system32/spool</ignore>

<ignore>%WINDIR%/system32/CatRoot</ignore>

<registry_ignore>HKEY_LOCAL_MACHINE\Security\Policy\Secrets</registry_ignore>

<registry_ignore>HKEY_LOCAL_MACHINE\Security\SAM\Domains\Account\Users</registry_ignore>

<registry_ignore type="sregex">\Enum$</registry_ignore>

</syscheck>

<log_format>eventlog</log_format>

<location>Application</location>

</localfile>

<log_format>eventlog</log_format>

<location>Security</location>

</localfile>

<log_format>eventlog</log_format>

<location>System</location>

</localfile>

</agent_config>

Reply all

Reply to author

Forward