everytime i upgrade Wazuh it breaks.

[Brad Nelson]

Hi,

I hope I am doing something wrong but every time I update wazuh to a newer version I can no longer access the wazuh dashboard. I get the following error:  Wazuh dashboard server is not ready yet

I have done this on 3 different ubuntu servers, and this happens every time. I end up having to uninstall wazuh and do a fresh install.  The steps i take are simply "apt upgrade" and after its completed i can't access the dashboard, ive rebooted the ubuntu server and get the same issue.

Is there something i am not doing right when updating these?

[Francisco Tuduri]

Hi Brad!

It sounds like you might be missing some steps or verifications during the upgrade process. Wazuh provides a comprehensive guide on how to upgrade the central components, which includes the Wazuh Manager, Wazuh Indexer, and Wazuh Dashboard. This guide outlines all the necessary steps and verifications needed for a successful upgrade.

You can find the upgrade guide here: https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html

Please review it to ensure you haven't missed any steps.
(Also note that there is a troubleshooting guide that may help you encounter any of the problems listed there: https://documentation.wazuh.com/current/upgrade-guide/troubleshooting.html)

Another thing to consider is the version that you are trying to upgrade to. Those links I shared all point to the current version (4.8). If you are trying to upgrade to a different version make sure to select the right one on the drop list box at the top.

Regarding the "Wazuh dashboard server is not ready yet" message, it usually indicates that the Dashboard is not being able to connect to the Wazuh Indexer.

To diagnose this issue, you can check the status of the indexer with:
systemctl status wazuh-indexer

and review the logs for errors or warnings:
cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

As for the Dashboard you can check:
journalctl -u wazuh-dashboard

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

If you continue to experience problems, please share the output of these commands and provide details about your deployment and the version you are currently using.

Regards

[Brad Nelson]

Hi Francisco thanks for your reply.

I checked out the link you sent me, i got to the step that says Upgrading the Wazuh indexer

when i ran this command: curl -X DELETE "https://10.1.2.3:9200/_index_template/ss4o_*_template" -u admin:PASSHIDDEN -k
I get this error: curl: (28) Failed to connect to 10.1.2.3 port 9200 after 131092 ms: Connection timed out

when I run this command: cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

I get: cat: /var/log/wazuh-indexer/wazuh-cluster.log: No such file or directory

When i run this command: journalctl -u wazuh-dashboard

I get this which i cant read the entire line:

Apr 26 17:13:29 wazuh opensearch-dashboards[60207]: {"type":"response","@timest>
Apr 26 17:13:29 wazuh opensearch-dashboards[60207]: {"type":"response","@timest>

when i run cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

I do get errors and i attached them to a log file for your review called "Wazuh errors.txt"

I want to remind you that this a fairly new install of 8.0 and i ran apt-update and apt-upgrade to get it to 8.1 and thats when it broke and its where we are right now. I really dont want to uninstall and reinstall to fix this as it will be my 4th or 5th install to fix a broke wazuh in the past 8 months.

[Brad Nelson]

Here is some more information to help you assist me.

# filebeat test output

attached to this message

# cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

No such file or directory

# cat /var/log/filebeat/filebeat | grep -i -E "error|warn"

No results

# cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"

attached to this message

[Francisco Tuduri]

Hi Brad,

I checked out the link you sent me, i got to the step that says Upgrading the Wazuh indexer

when i ran this command: curl -X DELETE "https://10.1.2.3:9200/_index_template/ss4o_*_template" -u admin:PASSHIDDEN -k
I get this error: curl: (28) Failed to connect to 10.1.2.3 port 9200 after 131092 ms: Connection timed out

This step needs the indexer to be running, and it looks like it is not. Either because it was stopped or because something broke during the upgrade.

Try to start the indexer with:

systemctl start wazuh-indexer

And check the status with:

systemctl status wazuh-indexer

If there are no errors continue with the upgrade guide. If there are errors, please share them here.

It is very strange that you get an error when attempting to read wazuh-cluster.log
Is there any content in /var/log/wazuh-indexer/?

One more question. You mentioned that this was a new install of 4.8, and then you attempted the upgrade to 4.8.1. Previous to the attempted upgrade was everything working correctly?

[Brad Nelson]

When i run the status wazuh-indexer i see this:

me@wazuh:/home/brad# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: activating (start) since Sun 2024-08-11 15:58:36 UTC; 55s ago
       Docs: https://documentation.wazuh.com
   Main PID: 18117 (java)
      Tasks: 39 (limit: 9317)
     Memory: 4.3G
        CPU: 1min 22.587s
     CGroup: /system.slice/wazuh-indexer.service
             └─18117 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopen>

Aug 11 15:58:36 wazuh systemd[1]: Starting Wazuh-indexer...
Aug 11 15:58:46 wazuh systemd-entrypoint[18117]: WARNING: A terminally deprecated method in java.lang.System has been c>
Aug 11 15:58:46 wazuh systemd-entrypoint[18117]: WARNING: System::setSecurityManager has been called by org.opensearch.>
Aug 11 15:58:46 wazuh systemd-entrypoint[18117]: WARNING: Please consider reporting this to the maintainers of org.open>
Aug 11 15:58:46 wazuh systemd-entrypoint[18117]: WARNING: System::setSecurityManager will be removed in a future release
Aug 11 15:58:52 wazuh systemd-entrypoint[18117]: WARNING: A terminally deprecated method in java.lang.System has been c>
Aug 11 15:58:52 wazuh systemd-entrypoint[18117]: WARNING: System::setSecurityManager has been called by org.opensearch.>
Aug 11 15:58:52 wazuh systemd-entrypoint[18117]: WARNING: Please consider reporting this to the maintainers of org.open>
Aug 11 15:58:52 wazuh systemd-entrypoint[18117]: WARNING: System::setSecurityManager will be removed in a future release

It is very strange that you get an error when attempting to read wazuh-cluster.log
Is there any content in /var/log/wazuh-indexer/?

Yes, there are gc.log files and wazuh-indexer-cluster.log

One more question. You mentioned that this was a new install of 4.8, and then you attempted the upgrade to 4.8.1. Previous to the attempted upgrade was everything working correctly?

Yes 4.8 was a fresh install, i had about 20 agents connected and working correctly. I then ran apt-update and apt-upgrade and thats when it broke. And this is the 2nd time i have run apt-update/apt-upgrade and broke wazuh with the same error.

[Brad Nelson]

In case you want to see whats in that file, here it is attached

[Francisco Tuduri]

I see that systemctl status wazuh-indexer shows Active: activating (start), when the normal state is Active: active (running). 

Does it get stuck at 'activating' or does it eventually change its state? could you please run systemctl status wazuh-indexer again to check that?

Also, could you share the indexer configuration with sudo cat /etc/wazuh-indexer/opensearch.yml?

[Brad Nelson]

Here is the command you requested:

 wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)

     Active: failed (Result: exit-code) since Sun 2024-08-11 15:59:43 UTC; 2 days ago
       Docs: https://documentation.wazuh.com
    Process: 18117 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (c>
   Main PID: 18117 (code=exited, status=78)
        CPU: 1min 42.417s

Aug 11 15:58:52 wazuh systemd-entrypoint[18117]: WARNING: System::setSecurityManager has been called by org.opensearch.>
Aug 11 15:58:52 wazuh systemd-entrypoint[18117]: WARNING: Please consider reporting this to the maintainers of org.open>
Aug 11 15:58:52 wazuh systemd-entrypoint[18117]: WARNING: System::setSecurityManager will be removed in a future release

Aug 11 15:59:42 wazuh systemd-entrypoint[18117]: ERROR: [1] bootstrap checks failed
Aug 11 15:59:42 wazuh systemd-entrypoint[18117]: [1]: memory locking requested for opensearch process but memory is not>
Aug 11 15:59:42 wazuh systemd-entrypoint[18117]: ERROR: OpenSearch did not exit normally - check the logs at /var/log/w>
Aug 11 15:59:43 wazuh systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=78/CONFIG
Aug 11 15:59:43 wazuh systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Aug 11 15:59:43 wazuh systemd[1]: Failed to start Wazuh-indexer.
Aug 11 15:59:43 wazuh systemd[1]: wazuh-indexer.service: Consumed 1min 42.417s CPU time.

and attached is the log file opensearch.yml

[Francisco Tuduri]

It looks like the indexer is having problems locking the memory.
Please, follow this guide to set appropriate values: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#memory-locking
Note that the recommended values for the -Xms and -Xmx parameters is half of the system RAM.
Let me know if you have any problem with this.
Regards!

[Brad Nelson]

HI Francisco,

I followed the article and then when i restart wazuh-indexer i get this:

root@wazuh:/home/brad# systemctl restart wazuh-indexer
Job for wazuh-indexer.service failed because the control process exited with error code.
See "systemctl status wazuh-indexer.service" and "journalctl -xeu wazuh-indexer.service" for details.

when running status i show:

     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)

    Drop-In: /etc/systemd/system/wazuh-indexer.service.d
             └─wazuh-indexer.conf
     Active: failed (Result: exit-code) since Wed 2024-08-14 18:41:40 UTC; 1min 44s ago
       Docs: https://documentation.wazuh.com
   Main PID: 3265 (code=exited, status=1/FAILURE)
        CPU: 5.632s

Aug 14 18:41:40 wazuh systemd-entrypoint[3329]:         at org.opensearch.common.settings.Settings.fromXContent(Settings.java:629)
Aug 14 18:41:40 wazuh systemd-entrypoint[3329]:         at org.opensearch.common.settings.Settings$Builder.loadFromStream(Settings.>
Aug 14 18:41:40 wazuh systemd-entrypoint[3329]:         ... 10 more
Aug 14 18:41:40 wazuh systemd-entrypoint[3388]: encountered [2] errors parsing [/etc/wazuh-indexer/jvm.options]
Aug 14 18:41:40 wazuh systemd-entrypoint[3388]: [1]: encountered improperly formatted JVM option in [/etc/wazuh-indexer/jvm.options>
Aug 14 18:41:40 wazuh systemd-entrypoint[3388]: [2]: encountered improperly formatted JVM option in [/etc/wazuh-indexer/jvm.options>
Aug 14 18:41:40 wazuh systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Aug 14 18:41:40 wazuh systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Aug 14 18:41:40 wazuh systemd[1]: Failed to start Wazuh-indexer.
Aug 14 18:41:40 wazuh systemd[1]: wazuh-indexer.service: Consumed 5.632s CPU time.
~

[Brad Nelson]

Update: I rebooted that server and now its working after those changes.

[Francisco Tuduri]

It's great to hear that! Thanks for letting us know

Don't hesitate to reach out again if you find any problems.

Regards!