Monitoring Wazuh activity

[Flek Kontrec]

Hello,

I'm using Wazuh Manager v4.2.5 and I'd like to detect situations where Wazuh practically stops working - events and alerts are not created anymore, Kibana graphs are empty but Wazuh services are still running. This happened when Elasticsearch shards reached their limit. Reaching shards limit is just one of the issues which can happen so I'm searching for a way how to detect them.

I've checked alerts for Wazuh server agent (agent.id 000) and noticed that alerts stopped being created when shards limit was reached and started being created when the shard issue was resolved. So my idea is to somehow detect situations where alerts for Wazuh server agent are not created for 1 hour, for example. Is there a way to set this up or is there some better way to get a notice that something's wrong with Wazuh?

Any advice is appreciated!

Regards

[Andres Micalizzi]

Hi Flekkontrek,

It's an interesting use case. It would be interesting to consider what would be considered wrong behaviour.

A first option is to regularly monitor the manager's and agent's ossec.log and other log files for error and/or warning messages (Not all error messages cause alerts, and not all errors.
cat /var/ossec/logs/ossec.log | grep 'ERROR \| WARNING' >>  /path-for-file/log-report.log - Can be used to copy only error messasges and warnings to a new file so it is easier to read. Since the logs are by default rotated every 24hs, you could do this process once a day, so you read only the logs daily.

Another possible option is determine which wazuh daemons you have usually enabled an monitor if one is stopped:

/var/ossec/bin/wazuh-control status - Will print a list of the status of all daemons (you can add the name of daemon to check the status of only that daemon).

This command could be fired from the manager (using command monitoring) periodically and with a custom decoder and rules, you could generate alerts if a daemon is shut down unexpectedly.

To launch the command every 2 minutes:

<localfile>

    <log_format>full_command</log_format>

    <command>/var/ossec/bin/wazuh-control status</command>

    <frequency>120</frequency>

</localfile>

Then you could use a custom decoder and rules to decode the response.

I hope this examples help you clear up your doubts. In case of further questions, do not hesitate to ask.
Cheers.

[Flek Kontrec]

Hi,

the shard limit error was written in the log file but this is just one case where something went wrong and data was missed. I believe there are lots of error/warning messages written in the log files so if I create alerts/notifications for them, it will be a storm.

About wazuh-control status, each Wazuh service was up and running, so this won't help.

If my idea isn't good and can't be achieved, maybe I need some other monitoring tool to watch over Wazuh.