Urgent please, My wazuh-dashboard service status has stopped

[Juan Ferdinan]

Hi Wazuh Teams,

My wazuh-dashboard service doesn't work, after I updated to wazuh version 4.5

I have tried start and restarting the service and it is running successfully, but within seconds it returns to stop

Thanks & Regards

Juan

[Olusegun Adenrele Oyebo]

Hello Juan,

Thank you for using Wazuh.

What steps did you take to proceed with the upgrade to 4.5?

What is your current architecture? All-in-one or distributed architecture

Kindly run the below commands on the various components and revert with the output:

• For the wazuh-indexer: journalctl -u wazuh-indexer
• For the wazuh manager: journalctl -u wazuh-manager
• For filebeat: journalctl -u filebeat
• For wazuh dashboard: journalctl -u wazuh-dashboard

Also revert with the /var/ossec/logs/ossec.log file from the wazuh server.

What is the current resource utilization on the node where the wazuh-dashboard is installed. Run the below commands to verify:

• df -H (free storage)
• top (CPU/Memory utilization)

Will be expecting your feedback so as to assist you further.

Best Regards.

[Juan Ferdinan]

Hi,

At first I was patching the Ubuntu OS by running the apt update command, then apt upgrade. And I didn't realize, it turned out that the wazuh was also updated up to version 4.5 and my dashboard wazuh had changed as previously attached.

What is your current architecture? All-in-one

For the wazuh-indexer: journalctl -u wazuh-indexer

For the wazuh manager: journalctl -u wazuh-manager

For filebeat: journalctl -u filebeat

For wazuh dashboard: journalctl -u wazuh-dashboard

/var/ossec/logs/ossec.log

df -H (free storage)

top (CPU/Memory utilization)

Best Regrads

Juan

[Olusegun Adenrele Oyebo]

Hello Juan,

Thanks for you feedback.

From the wazuh dashboard logs I can see that you have a fatal error "FATAL {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] to......" though it's not complete from your screenshot but kindly confirm the error looks like this "FATAL {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [****]/[****]". If that is the case, it means that you have reached your maximum number of shards per node which seems to be the reason why the wazuh-dashboard service is stopping intermittently.

You might need to increase the current number shard limits allocated. Kindly run the below command on the wazuh server to see the current maximum shard limit and revert with a screenshot of the output: curl -k -u admin:<admin_password> -XGET "https://<ip_address>:9200/_cluster/settings". Replace <ip_address> with the ip address of the wazuh server and <admin_password> with the admin password.

Will be expecting your feedback so as to assist you further.

Best Regards.

[Juan Ferdinan]

Hi,

I don't know why the sentence is incomplete, do you know the command to complete the sentence

by the way, here are the results

[Juan Ferdinan]

Hi,

I found a quick solution, i.e. I deleted some indexes, but I don't think it's the right solution. Do you have a better solution?

[Olusegun Adenrele Oyebo]

Dear Juan,

Sorry for the late response.

One way to view the the complete output of the logs is to redirect it to a text file like below and view the file:

• journalctl -u wazuh-dashboard > logs.txt

Deleting old indexes that are not needed is actually what we recommend. For you to check the indices stored, you can run the below command:

• curl -k -u admin:<admin_password> -XGET "https://<ip_address>:9200/_cat/indices?v&h=index,pri,rep" replace <admin_password> with the password of the user admin and also <ip_address> with the IP address of the wazuh indexer.

You can now use the below command to delete the old indexes:

• curl -k -u admin:<admin_password> -XDELETE "https://<ip_address>:9200/<index_to_delete>". Replace <admin_password> with the password of the user admin and also <ip_address> with the IP address of the wazuh indexer and <index_to_delete> with the index to delete e.g.  curl -k -u admin:Test123 -XDELETE "https://192.168.227.139:9200/wazuh-alerts-4.x-2023.08.20"

Another way which I stated in my previous mail is increasing the current shard limit. Though we don't recommend this as it can lead to issues later on the future. I'll just run you through how you can go about it below:

• Say you want to increase the limit from 1000 to 3000, kindly use the command: curl -k -u admin:<admin_password> -X PUT https://<ip_address>:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'. Replace <admin_password> with the password of the user admin and also <ip_address> with the IP address of the wazuh indexer.

I hope this was helpful and resolves the wazuh-dashboard issue. Do not hesitate to get back to us in case you have any other query

Best Regards.

[Juan Ferdinan]

Hi,

Sorry for the late response

what happens when I increase the limit from 1000 to 3000?Is this the only way or are there others?

Best Regards,

[Olusegun Adenrele Oyebo]

Hello Juan,

Thanks for reaching out again.

Increasing the shard limits from 1000 to 3000 will allow for more processing of search queries and also will help accommodate a larger historical data retention period but as I stated earlier in the previous mail, it's not what we recommend as it can lead to performance issues in the future.

What we recommend is deleting the old indices not needed.

Another way which we can recommend is adding more nodes to your Wazuh indexer cluster. You can check the link on more information on how to configure wazuh indexer cluster.

I hope this was able to provide more clarity. Do not hesitate to reach out to us again in case you have another other query.

Best Regards.

[Juan Ferdinan]

Hi,

It seems like I prefer to try creating a multi-node cluster system, but there are several things I want to ask, because I don't understand the tutorial at this link https://documentation.wazuh.com/current/installation-guide/wazuh- indexer/step-by-step.html#installing-the-wazuh-indexer-step-by-step

1. Should I run this method first on my current wazuh server? And will there be any impact when I try to do it?

2. What I understand, means I have to create 2 more servers to serve as a multi-node cluster?

3. Does the hardware specification of the 2 servers have to be exactly the same as the master node (wazuh my current server)?

4. Are the 2 servers in the same condition as the master node, which must have a wazuh server, wazuh indexer, wazuh dashboard and etc or is it just the wazuh indexer?

Sorry for my many questions, I hope to get the answers

Best Regards

Juan

[Olusegun Adenrele Oyebo]

Hello Juan,

Good to hear from you again and sorry for the late response.

1. Should I run this method first on my current wazuh server? And will there be any impact when I try to do it? Yes you will have to initiate the process on your current Wazuh server by first editing the config.yml file which should include the information of your additional nodes and regenerating the certificate. Use the summary below as guide to do this:

•  Edit the config.yml file on your current server and add the IP address and node information of the additional nodes you want to add.
• Generate the certificates for the multi-node cluster. You can check the link on how to download the config.yml file and certificate generation tool if you don't have them on the server.
• Next you will need to edit the /etc/wazuh-indexer/opensearch.yml file and include the ip address and node name of the additional wazuh indexer nodes as specified in your config.yml file and also configure the plugins.security.nodes_dn section by uncommenting and adding if applicable. You can use the link as guide for the configuration.
• After performing the above steps and compressing the certificate files, you will then need to rename the existing certificates in the directory /etc/wazuh-indexer/certs/ to .old, go back to the directory where you have your compressed certificate file and deploy the newly generated certificates using the steps outlined in the link and then restart the wazuh indexer service with command: systemctl restart wazuh-indexer
• Copy the compressed certificate to all the additional nodes (both Wazuh indexer and Wazuh manager) and perform the steps of installing wazuh indexer, configuring and certificate deployment on them using the link as guide.
• Next you will need to edit the /etc/filebeat/filebeat.yml on your existing node and include the ip address of your additional nodes as described in the link.
• After editing the file, rename the certificates in the directory /etc/filebeat/certs/ to .old, go back to the directory where you have your compressed certificate file and deploy the new certificate using the link as guide. Restart the filebeat service: systemctl restart filebeat  and run filebeat test: filebeat test output
• Install the Wazuh manager server on your additional nodes, configure and deploy the newly generated on them using the link as guide.
• Edit the wazuh dashboard file /etc/wazuh-dashboard/opensearch_dashboards.yml and include the ip address of your wazuh indexer additional nodes using the format as described in the link.
• After editing the file, rename the certificates that are present in the /etc/wazuh-dashboard/certs/ directory to .old, go back to the directory where you have the compressed certificate file and deploy the new certificates using the link as guide. Restart the dashboard service: systemctl restart wazuh-dashboard.

There could be little impact/downtime as a result of the certificate deployment and restarting of the services.

2. What I understand, means I have to create 2 more servers to serve as a multi-node cluster? Yes you can create 2 more servers for Wazuh manager and 2 more for Wazuh indexer.

3. Does the hardware specification of the 2 servers have to be exactly the same as the master node (wazuh my current server)? You can leave the additional nodes to be same specification as the master. You can also check the link below for the requirement specification details:

• Wazuh Indexer requirements
• Wazuh server requirements

4. Are the 2 servers in the same condition as the master node, which must have a wazuh server, wazuh indexer, wazuh dashboard and etc or is it just the wazuh indexer? If I can get this question correctly, you're asking if the additional nodes must have other components installed (wazuh server, indexer and dashboard) just like the existing architecture? If that is your question, No, you only need to install the individual component only example like installing only wazuh indexer on one of the additional node, etc.

I hope this was helpful and provided clarity. Do not hesitate to get back to us for further clarification or any other query.

Best regards.