Wazuh-agent 3.7.2 not monitoring with multiple apache log file

[Kazim Koybasi]

Hi All,

We upgraded wazuh-agent to 3.7.2 in our linux web servers. Our web server have multiple apache log files. Before upgrade it was analyzing log files and producing alerts. Now it show that it monitored apache log files but does not produce any alerts. I enabled debug mode for wazuh agent but cant identify reason for that. How can solve this situation? How can I analyze problem?

Best Regards.

[Juan Carlos]

Hello Kazim,

A very helpful tool in analyzing this type of issue is to see what you are receiving on the manager.

In order to do so you may temporarily configure the <logall> option in /var/ossec/etc/ossec.conf to yes.

This will save every log message received in /var/ossec/logs/archive/archives.log

Note that this will add a timestamp, source and destination information before the message, so the actual message will come after this.

If the original message is:

[Tue Sep 30 11:30:13.262255 2014] [core:error] [pid 20101] [client 99.47.227.95:34567] AH00037: Symbolic link not allowed or link target not accessible: /usr/share/awstats/icon/mime/document.png

Then in /var/ossec/logs/archive/archives.log you should see:

2019 Mar 29 08:13:37 (agent) 192.168.1.20->/var/log/apache.log [Tue Sep 30 11:30:13.262255 2014] [core:error] [pid 20101] [client 99.47.227.95:34567] AH00037: Symbolic link not allowed or link target not accessible: /usr/share/awstats/icon/mime/document.png

If you're not seeing the message, verify that other messages from the agent are reaching the manager, and that the agent is indeed reading the log. If the agent is connected and has the log open, on the agent side you may change the variable logcollector.debug to 2 in /var/ossec/etc/ossec.conf and each time a log message is collected the file /var/ossec/logs/ossec.log will get a line similar to:

2019/03/29 08:31:54 ossec-logcollector[28208] read_syslog.c:119 at read_syslog(): DEBUG: Read 1 lines from /var/log/apache.log

Otherwise, if you are seeing the log message on the manager then it means the means the agent is correctly relaying the information to the manager.

You can then verify if this log message would trigger an alert by using the /var/ossec/bin/ossec-logtest utility, I personally prefer to pipe the message to it:

echo '[Tue Sep 30 11:30:13.262255 2014] [core:error] [pid 20101] [client 99.47.227.95:34567] AH00037: Symbolic link not allowed or link target not accessible: /usr/share/awstats/icon/mime/document.png' | /var/ossec/bin/ossec-logtest




In order for the message to generate an alert the output of this must end with:
**Alert to be generated.

If the message is correctly decoded and triggers a rule but it doesn't state that then it means that rule does not generate an alert.

Certain errors like the one I have used as an example will not generate an alert in order to reduce noise in the system. The criteria for this decision may vary, so you may use a custom child rule or overwrite that one with a custom rule if you wish, for that we recommend you read this section of our documentation:
https://documentation.wazuh.com/current/user-manual/ruleset/custom.html

Finally, if you consider that your custom rule is of general interest we encourage the community to contribute to Wazuh (which is an open source project) by either opening an issue or submitting a Pull Request to the ruleset repository:

https://github.com/wazuh/wazuh-ruleset/

Don't forget to change back the logall and logcollector.debug variables in order to avoid using unnecessary disk space.

Best Regards,
Juan Carlos tello

[Kazim Koybasi]

Hello Juan,

Thanks you for informations. I applied instruction that you mentioned. When I opened logall there was no logs came from our webserver . I opened /var/ossec/bin/ossec-logtest and pasted text which is taken from our web server. 
  After that some alerts triggered but when I open debug parameter for agent.debug and logcollector.debug after a while nothing is written ossec.log in wazuh-agent. There is much apache logs which is produced in this web servers. May there any other limits that hangs agent because of lots of logs ? Are there anything that I can try to solve issue?

Best Regards.

[Juan Carlos]

Ok, it would seem the agent is not relaying the logs to the manager.

In order to see the status of the different agents you may run in the manager:

/var/ossec/bin/agent_control -l

Or in the agent:

grep status /var/ossec/var/run/osse-agentd.state

Further information of whether the agent is having issues connecting to the manager can be found in the agent's /var/ossec/logs/ossec.log file.

There is an anti-flooding mechanism in place for the agent to avoid overloading the manager or the network which by default is 500 Events Per Second (EPS). More information is available here:

https://documentation.wazuh.com/current/user-manual/capabilities/antiflooding.html

And by default only a maximum of 10000 lines are read each time an agent interacts with a logfile, which by default should happen every 2 seconds.

More information at the bottom of that same page:

https://documentation.wazuh.com/current/user-manual/capabilities/antiflooding.html#anti-flooding-in-agent-modules

However you should be seeing at least 500 EPS for this agent if any of these anti-flooding mechanisms were taken place.

Best Regards,

Juan Carlos Tello

[Kazim Koybasi]

Hello Juan,

I investigate the issue and also checked you instructions but no success. After I upgrade Wazuh to 3.8.2 problems is resolved. Thanks for your help.

Best Regards,

On Thursday, 28 March 2019 19:32:18 UTC+3, Kazim Koybasi wrote: