Indexer failed to start in 4.8

[Veera]

Hi ,

I have build a new manager,  indexer and dashboard in version 4.8 by following the documentation "step by step" .  

However i have the problem in dashboard with  "Wazuh dashboard server is not ready yet"

while debugging it seems the indexer is failing to start .  its a 2 node cluster and I have  attached the logs here for analysis. 

[Lamya Imam]

Hello Veera!

Can you please ensure that the certificate names from -> /etc/filebeat/certs, matches with the:
filebeat config file at -> /etc/filebeat/filebeat.yml?
ossec configuration file at -> /var/ossec/etc/ossec.conf under the indexer block configuration?
And, dashboard config file at -> /etc/wazuh-dashboard/opensearch_dashboards.yml?

Ensure that the indexer name and IP in the config file(opensearch.yml) is similar to the  config.yml file.

Let me know! I will be waiting for your response!

[Veera]

Thanks .  I am able to correct the certificates , path and other minor misses  and have the new setup  have wazuh-indexer , wazuh-manager and filebeat are fine.

filebeat tests the output successfully  from both indexers , when tested from  both servers. 

However the dashboard 4.8.0.1  is showing blank .

[root@new-wazuh ~]# systemctl status  wazuh-dashboard
× wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2024-06-25 09:21:11 EEST; 20s ago
   Duration: 6.303s
    Process: 49380 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (code=exited, status=1/FAILURE)
   Main PID: 49380 (code=exited, status=1/FAILURE)
        CPU: 7.251s

Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["info","s>
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["info","s>
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["error",">
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["warning">
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["fatal",">
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["info","p>
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]:  FATAL  {"error":{"root_cause":[{"type":"index_not_found_exception">
Jun 25 09:21:11 new-wazuh.novalocal systemd[1]: wazuh-dashboard.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 09:21:11 new-wazuh.novalocal systemd[1]: wazuh-dashboard.service: Failed with result 'exit-code'.
Jun 25 09:21:11 new-wazuh.novalocal systemd[1]: wazuh-dashboard.service: Consumed 7.251s CPU time.

Files attached for analysis .

1. Internal or external IP to be used in the opensearch.hosts  of /etc/wazuh-dashboard/opensearch_dashboards.yml?

2. Also in the version 4.8.0-1.x86_64 , the mentioned file (in DOC)  /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml is missing .

Still the Index file too ?? 

[root@ new-wazuh   ~]# rpm -qa wazuh-dashboard
wazuh-dashboard-4.8.0-1.x86_64
[root@ new-wazuh   ~]# rpm -ql wazuh-dashboard |grep wazuh.yml
[root@ new-wazuh   ~]#

The file /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml manually copied into , but not  working

[Lamya Imam]

Hello Veera!

This could happen if the Dashboard cannot communicate with the Indexer. The settings for that communication are done in the /etc/wazuh-dashboard/opensearch_dashboards.yml file, with the opensearch.hosts setting. You should configure the address of your Wazuh Indexer server/servers in that file and restart the Dashboard.  For multiple Wazuh indexer nodes in the same cluster the instructions are stated on the Installing the Wazuh dashboard step by step documentation. 

Yo do not need to manually copy the wazuh.yml  file. The settings on the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml are used by the Wazuh plugin to communicate with the Server's API, but that step is done after the Dashboards communication with the Indexer and the login occurs. 

Let me know if it worked!

[Veera]

Hi Lamya,

Ignoring the  usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml ,  I have followed the  Installing the Wazuh dashboard step by step documentation (Before this post)  and I repeated it  multiple times  without luck . However I tried it again to find the same result . 

I followed the steps  installation-guide , step-by-step method  in the Order without any Error or failure ... Indexer, dashboard, Filebeat and then Dashboard in the order.

However accessing the URL of dashboard with https://<IP>:443  , with public or private IP are not working . 

Refer to the logs  attached. 

[Lamya Imam]

Hi Veera,

I will need more information about the configuration of both Wazuh servers to understand how the servers are set up. Please provide the following configuration files:
- ossec.conf
- config.yml
- opensearch.yml
- filebeat.yml

Please do mention the node names when sharing the config files, like: ossec.conf (node1) and ossec.conf (node2)
Also, could you please elaborate what you meant by "with public or private IP"?

Will be waiting to for your response!

[Veera]

Hi Lamya,

Attached   the configurations files  mentioned by you .

The nodes used here are running on a cloud environment , where they have both public and private IP attached to it .

An interface  eth0  have the below configuration  can be accessed over the  external network or internet using the public IP 10.X.X. 21

 inet 192.168.0.28/24 brd 192.168.0.255 scope global dynamic noprefixroute eth0

So I am trying to access the dashboard with the https://<public_IP_of_192.168.0.28>:443

 

For example , In an another case , I have a single master wazuh server installed whose internal IP is  192.168.0.111  and  the dashboard can be accessed successfully on the external IP on 10.150.160.15 (dashboard) .

Thanks 

[Lamya Imam]

Hello Veera,

The dashboard is not ready can occur because it cannot query the indexer. For that I would need you to share the wazuh-indexer-cluster.log by using the following command:
# cat /var/log/wazuh-indexer/wazuh-indexer-cluster.log 

Please ensure that you configured the Indexer properly following the documentation: https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#configuring-the-wazuh-indexer

In the node2_opensearch.yml file, it says: cat: /etc/wazuh-dashboard/opensearch_dashboards.yml: No such file or directory. Does that mean you have installed the dashboard in node1? Also, I could not find the indexer configuration /etc/wazuh-indexer/opensearch.yml of node2. Please do share the opensearch.yml of node2 as well. 

Also, ensure that the indexer certificate has proper permissions:
ll /etc/wazuh-indexer/certs/

After configuring the certificate properly as mentioned in the document,  restart the indexer service and check the status.

Reference: https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#configuring-the-wazuh-indexer

Please share your findings and let me know if you face any issue!