Wazuh Single Host to distributed

[PATAN MAHAMMAD KHAN]

Hi Team,

We are running Wazuh 4.3.10 on single host, wherein Manager, Dashboard, Indexer are installed on same host with the help of Assistance. 

From last couple of weeks we are encountering log delay, Log loose because of resource incompatibility and high EPS, Hence we have been asked to go distributed deployment for smooth functionality. 

Now, I am planning to add extra server for the existing setup, wherein the plan was to separate the Indexer into a newly adding server and Manager, Dashboard will remain same on current server.

Please help me with complete step-wise document to achieve this upgradation without disturbing the current setup.

[Marcos Darío Buslaiman]

Hi Patan Mahammad Khan,

Thanks for using Wazuh!!

I am going to prepare a guide with the steps so that you can do this.
As soon as I have it, I'll send it to you.

Regards

[Marcos Darío Buslaiman]

Hi Patan Mahammad Khan,
I will describe the steps that you will need to execute to get the wazuh-indexer on a different server than your all-in-one installation.
Basically, we will need to install the same version of indexer on your new server and redeploy the certificates due to the IP address should change, so for filebeat and dashboard that connect with Wazuh-indexer, needs to configure the IP of this new indexer and create new certificates to replace the current ones.
On the other hand, it will also be necessary to move the data from the current path of indexer to the new one.

So you will need to follow these steps:
1) Stop wazuh Indexer on your current host.
systemctl stop wazuh-indexer

2) Install the dependencies and the wazuh Indexer on the new server, but without starting the service yet, by running this.

yum -y install coreutils
yum -y install wazuh-indexer-4.3.10-1

Move from the current wazuh-indexer to the new one, the path /var/lib/wazuh-indexer

scp -pr /var/lib/wazuh-indexer root@<NEW_INDEXER_IP>:/var/lib/

Verify that the files have been moved and that their ownership and group
ls -ltr /var/lib/wazuh-indexer

Ensure that the ownership and group be wazuh-indexer user and group.
chown -R wazuh-indexer:wazuh-indexer

3)On the new indexer server, download the script and template to create the new certificates.
curl -sO https://packages.wazuh.com/4.4/wazuh-install.sh
curl -sO https://packages.wazuh.com/4.4/config.yml

Edit ./config.yml and change the IP address of each service, like this example,
nodes:
  # Wazuh indexer nodes
  indexer:
    - name: node-1
      ip: <NEW_INDEXER_IP>
    #- name: node-2
    #  ip: <indexer-node-ip>
    #- name: node-3
    #  ip: <indexer-node-ip>

  # Wazuh server nodes
  # If there is more than one Wazuh server
  # node, each one must have a node_type
  server:
    - name: wazuh-1
      ip: <CURRENT_WAZUH_MANAGER_IP>
    #  node_type: master
    #- name: wazuh-2
    #  ip: <wazuh-manager-ip>
    #  node_type: worker
    #- name: wazuh-3
    #  ip: <wazuh-manager-ip>
    #  node_type: worker

  # Wazuh dashboard nodes
  dashboard:
    - name: dashboard
      ip: <CURRENT_WAZUH_DASHBOARD_IP>

Execute the script to create the certificates.

bash ./wazuh-certs-tool.sh -A

Create the tar files with the certificates and remove the creation folder
tar -cvf ./wazuh-certificates.tar -C ./wazuh-certificates/ .
rm -rf ./wazuh-certificates

mkdir /etc/wazuh-indexer/certs
tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
chmod 500 /etc/wazuh-indexer/certs
chmod 400 /etc/wazuh-indexer/certs/*
chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

4) Configure the Wazuh-Indexer editing the config file /etc/wazuh-indexer/opensearch.yml
These steps are the same as this document.
https://documentation.wazuh.com/4.3/installation-guide/wazuh-indexer/step-by-step.html#configuring-the-wazuh-indexer
Edit the /etc/wazuh-indexer/opensearch.yml configuration file and replace the following values:

 a)  network.host: Sets the address of this node for both HTTP and transport traffic. The node will bind to this address and use it as its publish address. Accepts an IP address or a hostname.

Use the same node address set in config.yml to create the SSL certificates.

 b) node.name: Name of the Wazuh indexer node as defined in the config.yml file. For example, node-1.


5) Sent the tar file with the certificates from your new indexer server to the Wazuh-Manager/Dashboard server.

scp wazuh-certificates.tar root@<CURRENT_WAZUH_MANAGER_IP>:/root

6) Go to your Wazuh-manager host and take a backup of your current certificates

mkdir /root/filebeat_backup_certs
mkdir /root/dashboard_backup_certs

mv /etc/filebeat/cert/* /root/filebeat_backup_certs
mv /etc/wazuh-dashboard/certs/* /root/dashboard_backup_certs

Stop the filebeat and dashboard service

systemctl stop filebeat
systemctl stop wazuh-dashboard

Deploy the certificates for Filebeat and Wazuh-dashboard
These steps are the same of this document:
https://documentation.wazuh.com/current/installation-guide/wazuh-server/step-by-step.html#deploying-certificates
https://documentation.wazuh.com/current/installation-guide/wazuh-dashboard/step-by-step.html#deploying-certificates

Filebeat:
Note Make sure that a copy of the wazuh-certificates.tar file, created during the initial configuration step, is placed in your working directory.

Replace <server-node-name> with your Wazuh server node certificate name, the same one used in config.yml when creating the certificates. Then, move the certificates to their corresponding location.

NODE_NAME=<server-node-name>

tar -xf ./wazuh-certificates.tar -C /etc/filebeat/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem
mv -n /etc/filebeat/certs/$NODE_NAME.pem /etc/filebeat/certs/filebeat.pem
mv -n /etc/filebeat/certs/$NODE_NAME-key.pem /etc/filebeat/certs/filebeat-key.pem
chmod 500 /etc/filebeat/certs
chmod 400 /etc/filebeat/certs/*
chown -R root:root /etc/filebeat/certs


Add the new Indexer IP on /etc/filebeat/filebeat.yml
output.elasticsearch.hosts:
        - <NEW_INDEXER_IP>:9200

Verify that the filename of the files /etc/filebeat/certs/  matches with the name on your /etc/filebeat/filebeat.yml config file
 ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
 ssl.key: "/etc/filebeat/certs/filebeat-key.pem"

if they don't match modify the configuration file to match with your certificates.

Start the service:
systemctl daemon-reload
systemctl enable filebeat
systemctl start filebeat

Validate if filebeat is correctly running and accessing the new indexer.

filbeat test output.


Dashboard:

Replace <dashboard-node-name> with your Wazuh dashboard node name, the same one used in config.yml to create the certificates, and move the certificates to their corresponding location.

NODE_NAME=<dashboard-node-name>

tar -xf ./wazuh-certificates.tar -C /etc/wazuh-dashboard/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem
mv -n /etc/wazuh-dashboard/certs/$NODE_NAME.pem /etc/wazuh-dashboard/certs/dashboard.pem
mv -n /etc/wazuh-dashboard/certs/$NODE_NAME-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem
chmod 500 /etc/wazuh-dashboard/certs
chmod 400 /etc/wazuh-dashboard/certs/*
chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs

Configuring the Wazuh dashboard
Edit the /etc/wazuh-dashboard/opensearch_dashboards.yml file and replace the IP address of your new Indexer:

opensearch.hosts: https://<NEW_INDEXER_IP>:9200

As well that filebeat, verify that the filename of the files /etc/wazuh-dashboard/certs/ matches with the name on your /etc/wazuh-dashboard/opensearch_dashboards.yml config file
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"

Start the service
systemctl daemon-reload
systemctl enable wazuh-dashboard
systemctl start wazuh-dashboard

Now you will need to check if you are receiving new alerts and if they are correctly inserted on Wazuh-Indexer.

Please, just let me know if you have any doubts or questions.

Regards

Marcos

[PATAN MAHAMMAD KHAN]

Hi Marcos,

Thanks for the step wise guide, I will work on it and let you know any challenges.

[PATAN MAHAMMAD KHAN]

Hi Marcos,

Requesting you to please clear these details 

1. Instead of moving indexer and huge historical logs to the new server,  I could like to  move the Manager, Dashboard and other required services to the new server!  Will that be possible and feasible? If yes, Can you please share the step wise document for the same
2. Is there any provision to read the logs to after saving it to local database with the help SCP.

[PATAN MAHAMMAD KHAN]

Hi Marcos, 

Can you please look into this issue.

[Marcos Darío Buslaiman]

Hi,
Sorry for the delay,
Regarding your question, yes, is possible to move the wazuh manager and wazuh dashboard to other server, here is important to know if you are using a Load Balancer to connect your agents with the Wazuh manager, if not, keep in mind that if you change the IP of the Wazuh Manager you will need to update the IP on all your agents, in case that you keep the same IP address, as well that the first option (move Indexer) you will need to move a folder, in this case,  "/var/ossec/*" to your new server in order to keep the client keys and all the configurations, custom decoders, rules, etc.

In any case will be necessary to create new certificates, due to you will be changing the IP address of your indexer or Wazuh-Manager and Dashboard.
Also, it is very important to mention, that in any case, we recommend taking snapshots of your current servers in case of any error you will have a backup of your current environment.

Please let me know which option you prefer and then I will send you the steps in case you need to move (Manager and Dashboard to a new server):

Regards
Marcos

[PATAN MAHAMMAD KHAN]

Hi Marcos,

Thanks for the response, I would like to move the manager and dashboard to the new server and wazuh indexer will remain in same existing server. (I hope that is the best thing, rather moving huge data of 3TB from old to new server).

Please let me know the step wise procedure for the same or suggest if any other simple way.

[Marcos Darío Buslaiman]

Hi Patan Mahammad Khan,
In order to analyze the difficulty and the steps to follow, it is important to know if you will change the IP of the manager or not, and also, if you connect all your agents to the manager, or if are you using a Load Balancer between the agents and your manager?

Regards

[PATAN MAHAMMAD KHAN]

Hi Marcos,

I will be changing the IP address of the manager.

[Marcos Darío Buslaiman]

Hi 

Sorry for the late response, 

It's very important to mention here that with this change of the IP of manager and dashboard you will need to re-create the certificates and deploy them on the new servers.

And another important point to take into account is that all of your agents should be configured with this new IP of the Wazuh Manager.

I will recommend you take a snapshot of the servers before changing anything in order to revert to the current state.

1) As the new servers have different IP address you will need to create the certificates by following this guide:

https://documentation.wazuh.com/4.3/user-manual/certificates.html

2) Once that the certificate is generated you will need to replace them on your indexer server. Ref Doc https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#deploying-certificates
Take a backup of the current ones then replace them.

NODE_NAME=<indexer-node-name>

tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
chmod 500 /etc/wazuh-indexer/certs
chmod 400 /etc/wazuh-indexer/certs/*
chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

Then restart the service 
systemctl restart wazuh-indexer

3) install Wazuh Manager and Filebeat on the new server you can follow this guide. and deploy the certificates created on step 2

https://documentation.wazuh.com/4.3/installation-guide/wazuh-server/step-by-step.html#wazuh-server-node-installation

4) install Wazuh Dashboard and deploy the certificates like the step before.

https://documentation.wazuh.com/current/installation-guide/wazuh-dashboard/step-by-step.html

Please let me know if you have any doubts or questions.

Regards