About wazuh-syscheckd error that occurs in wazuh server

[Keisuke Moriai]

Hi.

I would like to ask about the following error that is currently occurring frequently on the wazuh server.
"FIM real-time queue is full. Some real-time events may be lost."

At the moment, this occurs after a while after starting the wazuh service.
After it occurs, this error is output once every 10 seconds.

The version of wazuh is 4.3.10.
I have tried it and it does not affect the operation.
However, I would like to do something about it because the log is flooded with this error.

As a last resort, it would be good if we could prevent it from appearing in the alert log.

I would like to know the conditions under which this occurs and how to handle it.

The configuration file is at the end of this email.

Sorry for my poor English, but thank you in advance.

------------------------------------------------------------------------------------

[root@sample-01 ~]# cat /var/ossec/etc/ossec.conf
<!--
  Wazuh - Manager - Default configuration for rhel 8.6
  More info at: https://documentation.wazuh.com
  Mailing list: https://groups.google.com/forum/#!forum/wazuh
-->

<ossec_config>
  <global>
    <jsonout_output>yes</jsonout_output>
    <alerts_log>yes</alerts_log>
    <logall>no</logall>
    <logall_json>no</logall_json>
    <email_notification>no</email_notification>
    <agents_disconnection_time>10m</agents_disconnection_time>
    <agents_disconnection_alert_time>0</agents_disconnection_alert_time>
  </global>

  <alerts>
    <log_alert_level>3</log_alert_level>
  </alerts>

  <!-- Choose between "plain", "json", or "plain,json" for the format of internal logs -->
  <logging>
    <log_format>plain</log_format>
  </logging>

  <remote>
    <connection>secure</connection>
    <port>1514</port>
    <protocol>tcp</protocol>
    <queue_size>131072</queue_size>
  </remote>

  <!-- Policy monitoring -->
  <rootcheck>
    <disabled>yes</disabled>
  </rootcheck>

  <wodle name="cis-cat">
    <disabled>yes</disabled>
  </wodle>

  <!-- Osquery integration -->
  <wodle name="osquery">
    <disabled>yes</disabled>
  </wodle>

  <!-- System inventory -->
  <wodle name="syscollector">
    <disabled>yes</disabled>
  </wodle>

  <sca>
    <enabled>no</enabled>
  </sca>

  <vulnerability-detector>
    <enabled>no</enabled>
  </vulnerability-detector>

  <!-- File integrity monitoring -->
  <syscheck>
    <disabled>no</disabled>
    <scan_on_start>yes</scan_on_start>
    <frequency>43200</frequency>
    <alert_new_files>yes</alert_new_files>

    <directories realtime="yes">/etc,/usr/bin,/usr/sbin,/bin,/sbin,/boot,/lib,/lib64</directories>
    <directories realtime="yes">/var/ossec/</directories>


    <!-- Files/directories to ignore -->
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
    <ignore>/etc/random-seed</ignore>
    <ignore>/etc/random.seed</ignore>
    <ignore>/etc/adjtime</ignore>
    <ignore>/etc/httpd/logs</ignore>
    <ignore>/etc/utmpx</ignore>
    <ignore>/etc/wtmpx</ignore>
    <ignore>/etc/cups/certs</ignore>
    <ignore>/etc/dumpdates</ignore>
    <ignore>/etc/svc/volatile</ignore>
    <ignore>/var/ossec/logs/</ignore>
    <ignore>/var/ossec/queue/</ignore>
    <ignore>/var/ossec/var/run/</ignore>
    <ignore>/var/ossec/stats/</ignore>

    <!-- File types to ignore -->
    <ignore type="sregex">.log$|.swp$</ignore>

    <!-- Check the file, but never compute the diff -->
    <nodiff>/etc/ssl/private.key</nodiff>

    <skip_nfs>yes</skip_nfs>
    <skip_dev>yes</skip_dev>
    <skip_proc>yes</skip_proc>
    <skip_sys>yes</skip_sys>

    <!-- Nice value for Syscheck process -->
    <process_priority>10</process_priority>

    <!-- Maximum output throughput -->
    <max_eps>100</max_eps>

    <!-- Database synchronization settings -->
    <synchronization>
      <enabled>yes</enabled>
      <interval>5m</interval>
      <max_interval>1h</max_interval>
      <max_eps>10</max_eps>
      <queue_size>30000</queue_size>
    </synchronization>
  </syscheck>

  <!-- Active response -->
  <global>
    <white_list>127.0.0.1</white_list>
    <white_list>^localhost.localdomain$</white_list>
    <white_list>10.0.18.2</white_list>
  </global>

  <command>
    <name>disable-account</name>
    <executable>disable-account</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>restart-wazuh</name>
    <executable>restart-wazuh</executable>
  </command>

  <command>
    <name>firewall-drop</name>
    <executable>firewall-drop</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>host-deny</name>
    <executable>host-deny</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>route-null</name>
    <executable>route-null</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>win_route-null</name>
    <executable>route-null.exe</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>netsh</name>
    <executable>netsh.exe</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <!--
  <active-response>
   <disabled>yes</disabled>
  </active-response>
  -->

  <!-- Log analysis -->
  <localfile>
    <log_format>command</log_format>
    <command>df -P</command>
    <frequency>360</frequency>
  </localfile>

  <localfile>
    <log_format>full_command</log_format>
    <command>netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d</command>
    <alias>netstat listening ports</alias>
    <frequency>360</frequency>
  </localfile>

  <localfile>
    <log_format>full_command</log_format>
    <command>last -n 20</command>
    <frequency>360</frequency>
  </localfile>

  <ruleset>
    <!-- Default ruleset -->
    <decoder_dir>ruleset/decoders</decoder_dir>
    <rule_dir>ruleset/rules</rule_dir>
    <rule_exclude>0215-policy_rules.xml</rule_exclude>
    <list>etc/lists/audit-keys</list>
    <list>etc/lists/amazon/aws-eventnames</list>
    <list>etc/lists/security-eventchannel</list>

    <!-- User-defined ruleset -->
    <decoder_dir>etc/decoders</decoder_dir>
    <rule_dir>etc/rules</rule_dir>
  </ruleset>

  <rule_test>
    <enabled>yes</enabled>
    <threads>1</threads>
    <max_sessions>64</max_sessions>
    <session_timeout>15m</session_timeout>
  </rule_test>

  <!-- Configuration for wazuh-authd -->
  <auth>
    <disabled>no</disabled>
    <port>1515</port>
    <use_source_ip>no</use_source_ip>
    <purge>yes</purge>
    <use_password>no</use_password>
    <ciphers>HIGH:!ADH:!EXP:!MD5:!RC4:!3DES:!CAMELLIA:@STRENGTH</ciphers>
    <ssl_manager_cert>etc/sslmanager.cert</ssl_manager_cert>
    <ssl_manager_key>etc/sslmanager.key</ssl_manager_key>
    <ssl_auto_negotiate>no</ssl_auto_negotiate>
  </auth>

  <cluster>
    <name>sample-cluster</name>
    <node_name>sample-master</node_name>
    <node_type>master</node_type>
    <key>4dc329d2a35483a505a028f0b824d73c</key>
    <port>1516</port>
    <bind_addr>0.0.0.0</bind_addr>
    <nodes>
        <node>sample.local</node>
    </nodes>
    <hidden>no</hidden>
    <disabled>no</disabled>
  </cluster>

</ossec_config>

<ossec_config>
  <localfile>
    <log_format>audit</log_format>
    <location>/var/log/audit/audit.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/ossec/logs/active-responses.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/messages</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/secure</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/maillog</location>
  </localfile>

</ossec_config>

------------------------------------------------------------------------------------

[Awwal Ishiaku]

Hi Keisuke
It looks like you are monitoring some rapidly changing directories in realtime. Refer to the following lines in your configuration:

    <directories realtime="yes">/etc,/usr/bin,/usr/sbin,/bin,/sbin,/boot,/lib,/lib64</directories>
    <directories realtime="yes">/var/ossec/</directories>

The contents of some of these directories change frequently and they are flooding the FIM real-time queue. Some of them may be json logs that are constantly changing.

We recommend that you investigate further to determine which directory is less appropriate to monitor in real-time and disable realtime monitoring for them.

Use real-time monitoring for files and directories that change less frequently to avoid event flooding.

Regards.

[Awwal Ishiaku]

We recommend that you disable realtime monitoring for the Wazuh directory because of the log and alert files in the directoty.
Or at least ignore the directories that contain logs and alerts.
Thanks.

[Keisuke Moriai]

Thank you for your response.

The same error occurred even if I set "ignore" for files in the wazuh directory that were frequently changed.
So I explicitly set the scan to scan only files that are not frequently changed.

I will wait for a while to get the results and will report back on the results.

2023年3月13日月曜日 15:29:09 UTC+9 Awwal Ishiaku:

[Keisuke Moriai]

Hi , Awwal.

Thanks to your advice I was able to solve the problem.
Thank you.

I will open another chat with you again because I have another problem.

I am not sure if you will be assigned to me, but I look forward to seeing you again.

2023年3月13日月曜日 17:34:30 UTC+9 Keisuke Moriai: