Wazuh integrations

[Рокси[АрмаСрбија]]

Hello everyone,

Firstly I would like to explain our system. We work with Wazuh in production environment with around 1600 agents. All agents have Sysmon installed for log enrichment, and also Vulnerability and FIM modules are activated. We have a wazuh server cluster(master and three workers) but only one worker is for collecting Wazuh agent logs. Whole system is around 100 million events per day, while agents are responsible for 25 millions. Alert EPS is above 300 in working hours.

Our main issue currently is Wazuh integrator not being able to keep up, meaning that integrations like Telegram notifications, Shuffle, MISP trigger very late like there is some limit but we couldnt find anywhere in documentation where that line is. We think that its probably to much events for integrator to work like intended, maybe it was designed for much less agents. We are sure that integrations themselves are not the issue, network is not a problem. Integrations work, it is just that they are delayed. If we take notifications for example, mail notifications work perfectly(because they are not part of integrations) and are almost realtime. While telegram integration has to check every event, which is probably limited and then call webhook to send message.

If you have any ideas or questions to help us we would 

appreciate it.

[Parash Mani Kafle]

The high event volume (100M daily, 25M from agents) and the delays with your integrations, it's likely that Wazuh's integrations (Telegram, MISP, etc.) are struggling to keep up with the processing load.
Few things that needs to be considered are as follows:
Integration Throughput: The integrations may have rate limits that are too slow to handle such a high volume. Implementing a multithreaded process could help by sending high volume logs to the API more efficiently.
API Limits: When the integrations relies on external APIs (e.g., Telegram), ensure that no rate limits are being hit on those services.

You can configure a loadbalancer in  such a way that all the logs that are being received on the environment can be loadbalanced among all the worker nodes and configure the integration script on all the nodes so that a single manager doesn't need to forward those events to the external endpoints.
For more information, You can visit the URL given below:
https://documentation.wazuh.com/current/user-manual/wazuh-server-cluster/load-balancers.html
https://documentation.wazuh.com/current/user-manual/manager/integration-with-external-apis.html

Additionally, you can reduce the data sent to external APIs by creating custom rules to only forward the necessary logs. For example, if similar alerts are expected within the same time frame, You can create a rule based on the time frame and only send a alerts that is really necessary to the external platforms:
For an example:

<rule id="30202" level="10" frequency="10" timeframe="120">
  <if_matched_sid>30201</if_matched_sid>
  <description>ModSecurity: Multiple attempts blocked.</description>
  <mitre>
    <id>T1110</id>
  </mitre>
  <group>access_denied,modsecurity,pci_dss_10.2.4,pci_dss_11.4</group>
</rule>
In this case, if rule 30201 is triggered 10 times within 2 minutes, only a single alert with rule ID 30202 will be sent.

For more reference, Please visit the URL given below:
https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/rules.html#if-matched-sid