Migrating from AIO to distributed

[Unai]

Hello everyone,
I've been working with a AIO setup for 3 years. Now that I have 100 agents and +200 syslog senders, I think the finest option would be to migrate to a distributed setup, because the performance of the AIO isn't great...
Wich would be the best way to migrate to a distributed setup? I would like to mantain all the data. I have arround 6 milions logs per day.
Wich architecture do you recommend?

Thanks in advance,

[Bony V John]

Hi,

Please allow me some time, I'm working on this and will get back to you with an update as soon as possible.

[Bony V John]

Hi,

Based on your input, it would be better to move from an all-in-one deployment to a distributed deployment. Since you are ingesting around 6 million logs per day, the current server is facing performance issues, and you need to retain the existing data, I would suggest moving to a cluster deployment instead of a normal distributed deployment.

In this case, I recommend adding extra nodes to your current Wazuh environment instead of migrating everything to a new setup.

From your current all-in-one Wazuh environment, you can add two extra indexer nodes and one extra manager worker node. This will convert the current setup into a cluster environment, improve performance, and keep the existing data available on your current server.

You can follow the below guidance.

Set up three new servers in the same network as the current Wazuh server:

• Two servers for Wazuh indexer nodes

• One server for a Wazuh manager worker node

For adding the two new indexer nodes, you can refer to the Wazuh documentation below for step-by-step guidance on certificate configuration and adding new nodes:

https://documentation.wazuh.com/current/user-manual/wazuh-indexer-cluster/add-wazuh-indexer-nodes.html#all-in-one-deployment

In the certificate creation step, make sure you include the current server details, the two new indexer server IP addresses, and the new Wazuh manager worker node IP address.

After generating the certificates, continue with the remaining indexer configuration steps from the documentation to add the new indexer nodes to the existing Wazuh indexer cluster.

After adding the indexer nodes, you can add the new Wazuh manager worker node.

For that, refer to the Wazuh manager node addition documentation. In that documentation, you can skip the certificate generation part because the certificates were already generated while adding the indexer nodes. Use those generated certificates on the new Wazuh manager node.

After copying the newly generated certificates from the Wazuh indexer server to the new manager node, follow the Configuring existing components section in the documentation.

After that, add a load balancer in front of both Wazuh manager nodes so that agents and syslog sources can connect through the load balancer.

You can use Nginx as the load balancer and configure it to distribute traffic between the Wazuh manager nodes based on load. You can refer to the Wazuh load balancer documentation for the configuration guidance.

Please let me know if you have any further questions or doubts.

[Unai]

Thank you for your quick response!
I have a few questions on this setup.
1. If i use a load balancer, will the agents and syslogs sources connect to nginx IP? If that was the case, should i change the config on all the hosts? Could i exchange the IP from the original AIO server with the Nginx server to keep the config on all the hosts? How does this work?
2. Right now, my server has 6vCPU, 16GB of RAM and 800GB for storage. What requirements do you recommend for the new servers?

Thanks!!

[Bony V John]

Hi,

Yes, if you introduce a load balancer in front of the Wazuh managers, you need to configure the agents and syslog sources to connect to the Nginx IP address. On the Nginx server, you must configure the load balancer to forward the traffic to the Wazuh manager nodes.

If you do not want to update the configuration on all endpoints, you can change the current AIO server IP address to a new IP and assign the AIO server’s existing IP address to the Nginx server. This will allow the agents and syslog sources to continue connecting to the same IP address and avoid configuration changes across all endpoints.

Note that when changing the AIO server IP address, I recommend doing this before adding the new manager and indexer nodes. The updated AIO server IP address must be used when generating the certificates and configuring the dashboard and indexer components. Therefore, the recommended order is:

1. Change the AIO server IP address.

2. Update the existing Wazuh configuration with the new IP address.

3. Generate the certificates using the new node IP addresses.

4. Add the new manager and indexer nodes.

5. Assign the old AIO IP address to the Nginx server.

6. Configure Nginx to forward the agent and syslog traffic to the manager nodes.

You should also plan for a maintenance window because moving the existing IP address may temporarily interrupt agent, syslog, dashboard, API, and indexer connectivity. Make sure the old IP address is completely removed from the AIO server before assigning it to Nginx to avoid an IP conflict.

Regarding server resources, the required capacity cannot be calculated accurately based only on the number of endpoints and syslog sources. Resource requirements vary depending on the environment. For example, the event ingestion rate differs based on the operating system, agent configuration, enabled modules, server activity, and workload. The ingestion rate can also vary significantly from one day to another.

Similarly, the number of syslog sources alone is not enough to predict the event rate. One device may generate only a few events per second, while another may generate thousands.

The ruleset also affects resource usage. The number and complexity of rules, decoders, correlations, and generated alerts can change the CPU, memory, storage, and indexing requirements. Retention periods, archive logging, shard configuration, and replication settings must also be considered.

I recommend referring to the Wazuh hardware requirements documentation to estimate the approximate resources required for your environment. However, do not size the environment only for the average ingestion rate. Include sufficient additional capacity to handle ingestion spikes. This will help the Wazuh environment run smoothly and provide better performance.

[Unai]

Understood, thank you for your help!
We will follow your recommendations.