Indexer failed to start in 4.8

[Veera]

Hi ,

I have build a new manager,  indexer and dashboard in version 4.8 by following the documentation "step by step" .  

However i have the problem in dashboard with  "Wazuh dashboard server is not ready yet"

while debugging it seems the indexer is failing to start .  its a 2 node cluster and I have  attached the logs here for analysis. 

[Lamya Imam]

Hello Veera!

Can you please ensure that the certificate names from -> /etc/filebeat/certs, matches with the:
filebeat config file at -> /etc/filebeat/filebeat.yml?
ossec configuration file at -> /var/ossec/etc/ossec.conf under the indexer block configuration?
And, dashboard config file at -> /etc/wazuh-dashboard/opensearch_dashboards.yml?

Ensure that the indexer name and IP in the config file(opensearch.yml) is similar to the  config.yml file.

Let me know! I will be waiting for your response!

[Veera]

Thanks .  I am able to correct the certificates , path and other minor misses  and have the new setup  have wazuh-indexer , wazuh-manager and filebeat are fine.

filebeat tests the output successfully  from both indexers , when tested from  both servers. 

However the dashboard 4.8.0.1  is showing blank .

[root@new-wazuh ~]# systemctl status  wazuh-dashboard
× wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2024-06-25 09:21:11 EEST; 20s ago
   Duration: 6.303s
    Process: 49380 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (code=exited, status=1/FAILURE)
   Main PID: 49380 (code=exited, status=1/FAILURE)
        CPU: 7.251s

Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["info","s>
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["info","s>
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["error",">
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["warning">
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["fatal",">
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]: {"type":"log","@timestamp":"2024-06-25T06:21:11Z","tags":["info","p>
Jun 25 09:21:11 new-wazuh.novalocal opensearch-dashboards[49380]:  FATAL  {"error":{"root_cause":[{"type":"index_not_found_exception">
Jun 25 09:21:11 new-wazuh.novalocal systemd[1]: wazuh-dashboard.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 09:21:11 new-wazuh.novalocal systemd[1]: wazuh-dashboard.service: Failed with result 'exit-code'.
Jun 25 09:21:11 new-wazuh.novalocal systemd[1]: wazuh-dashboard.service: Consumed 7.251s CPU time.

Files attached for analysis .

1. Internal or external IP to be used in the opensearch.hosts  of /etc/wazuh-dashboard/opensearch_dashboards.yml?

2. Also in the version 4.8.0-1.x86_64 , the mentioned file (in DOC)  /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml is missing .

Still the Index file too ?? 

[root@ new-wazuh   ~]# rpm -qa wazuh-dashboard
wazuh-dashboard-4.8.0-1.x86_64
[root@ new-wazuh   ~]# rpm -ql wazuh-dashboard |grep wazuh.yml
[root@ new-wazuh   ~]#

The file /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml manually copied into , but not  working

[Lamya Imam]

Hello Veera!

This could happen if the Dashboard cannot communicate with the Indexer. The settings for that communication are done in the /etc/wazuh-dashboard/opensearch_dashboards.yml file, with the opensearch.hosts setting. You should configure the address of your Wazuh Indexer server/servers in that file and restart the Dashboard.  For multiple Wazuh indexer nodes in the same cluster the instructions are stated on the Installing the Wazuh dashboard step by step documentation. 

Yo do not need to manually copy the wazuh.yml  file. The settings on the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml are used by the Wazuh plugin to communicate with the Server's API, but that step is done after the Dashboards communication with the Indexer and the login occurs. 

Let me know if it worked!

[Veera]

Hi Lamya,

Ignoring the  usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml ,  I have followed the  Installing the Wazuh dashboard step by step documentation (Before this post)  and I repeated it  multiple times  without luck . However I tried it again to find the same result . 

I followed the steps  installation-guide , step-by-step method  in the Order without any Error or failure ... Indexer, dashboard, Filebeat and then Dashboard in the order.

However accessing the URL of dashboard with https://<IP>:443  , with public or private IP are not working . 

Refer to the logs  attached. 

[Lamya Imam]

Hi Veera,

I will need more information about the configuration of both Wazuh servers to understand how the servers are set up. Please provide the following configuration files:
- ossec.conf
- config.yml
- opensearch.yml
- filebeat.yml

Please do mention the node names when sharing the config files, like: ossec.conf (node1) and ossec.conf (node2)
Also, could you please elaborate what you meant by "with public or private IP"?

Will be waiting to for your response!

[Veera]

Hi Lamya,

Attached   the configurations files  mentioned by you .

The nodes used here are running on a cloud environment , where they have both public and private IP attached to it .

An interface  eth0  have the below configuration  can be accessed over the  external network or internet using the public IP 10.X.X. 21

 inet 192.168.0.28/24 brd 192.168.0.255 scope global dynamic noprefixroute eth0

So I am trying to access the dashboard with the https://<public_IP_of_192.168.0.28>:443

 

For example , In an another case , I have a single master wazuh server installed whose internal IP is  192.168.0.111  and  the dashboard can be accessed successfully on the external IP on 10.150.160.15 (dashboard) .

Thanks 

[Lamya Imam]

Hello Veera,

The dashboard is not ready can occur because it cannot query the indexer. For that I would need you to share the wazuh-indexer-cluster.log by using the following command:
# cat /var/log/wazuh-indexer/wazuh-indexer-cluster.log 

Please ensure that you configured the Indexer properly following the documentation: https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#configuring-the-wazuh-indexer

In the node2_opensearch.yml file, it says: cat: /etc/wazuh-dashboard/opensearch_dashboards.yml: No such file or directory. Does that mean you have installed the dashboard in node1? Also, I could not find the indexer configuration /etc/wazuh-indexer/opensearch.yml of node2. Please do share the opensearch.yml of node2 as well. 

Also, ensure that the indexer certificate has proper permissions:
ll /etc/wazuh-indexer/certs/

After configuring the certificate properly as mentioned in the document,  restart the indexer service and check the status.

Reference: https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#configuring-the-wazuh-indexer

Please share your findings and let me know if you face any issue!

[Lamya Imam]

Hello Veera!

Firstly, I wanted to mention something regarding our email communications to help streamline our conversations. When making or replying to queries, I would request you to please use "Reply All". This approach helps ensure that other users in the community can also benefit from the shared information and any responses that follow.

Could you please check the status of your dashboard:
systemctl status wazuh-dashboard

Without the dashboard service running, you will not be able to see the dashboard.
If the dashboard service is up and running, please share the screenshot of the browser. 

After that, 
Check if the certificates are configured properly by running the command:
filebeat output test

Also, share the output of the command.

Kindly share your findings here so that we can analyze it. 

[Veera]

Hi Lamya,

Welcome back !!!

I usually use "Reply ALL"  and still unware of where I have missed .. Will ensure  the same in Future.

Attached is my  Dashboard screen and  the output of the commands mentioned .

The same IP used in the config.yml is the initial setup which is the equivalent of Internal IP of  the Dashboard server. 

As mention on my post on Jun25th , The Dashboard service use to run for a few minutes after to show the below error.

Please have a check and comment on my IP's as I use to access the Dashboard with Public IP of the Dashboard server. 

[Lamya Imam]

Hello Veera!

There could be an issue with the Dashboard configuration.
Could you please provide the following using the command below:
cat /etc/wazuh-dashboard/opensearch_dashboards.yml
ll /etc/wazuh-dashboard/certs/

Will be waiting for your response!

[Veera]

Hi  Lamya,

Here are the details ..

[root@new-wazuh ~]# cat /etc/wazuh-dashboard/opensearch_dashboards.yml
server.host: 0.0.0.0
server.port: 443
opensearch.hosts: ["https://192.168.0.28:9200", "https://192.168.0.20:9200"]
opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home

[root@new-wazuh ~]# ll /etc/wazuh-dashboard/certs/
total 12
-r-------- 1 wazuh-dashboard wazuh-dashboard 1704 Jun 24 20:48 dashboard-key.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1294 Jun 24 20:48 dashboard.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1204 Jun 24 20:48 root-ca.pem
[root@new-wazuh ~]#

[Lamya Imam]

Hello Veera,

Sorry about the inconveniences that you are facing. After thorough review, the configuration seems okay to me. But there was an error in the Dashboard status:
FATAL  {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.kibana]",>

I would ask you to share the Wazuh dashboard log for further investigation:
# journalctl -u wazuh-dashboard
# cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

Looking forward to hearing from you soon!

[Veera]

Hi Lamya,

Attached is the  journalctl   logs of the dashboard.

But unable to locate the file  wazuhapp.log and under  /usr/share/wazuh-dashboard/data/wazuh/logs/ .

The dashboard is still unreachable using the  private IP   192.168.0.28 or PublicIP(10.x.x.x)  of the same .

[Lamya Imam]

Hello Veera,

The issue could be due to a lack of indentation  in Wazuh Dashboard configuration files: 

• The vertical space between default and url  at /usr/share/wazuh-dashboard//data/wazuh/config/wazuh.yml [It will look as shown on the screenshot]
• The space between opensearch.hosts: after the "," at  /etc/wazuh-dashboard/opensearch_dashboards.yml  [There will be no space  after the comma ","  ] 

Can you please check these for the fix as shown on the screenshot.

Reference:
https://documentation.wazuh.com/current/installation-guide/wazuh-dashboard/step-by-step.html#starting-the-wazuh-dashboard-service

Let me know if it solves the issue!

[Veera]

Hi Lamya,

Thanks . I have corrected  the  

• The space between opensearch.hosts: after the "," at  /etc/wazuh-dashboard/opensearch_dashboards.yml  [There will be no space  after the comma ","  ] 

but the below file indentation is set as expected. no changes made.

• The vertical space between default and url  at /usr/share/wazuh-dashboard//data/wazuh/config/wazuh.yml [It will look as shown on the screenshot]

After the dashboard service restarted , still the same issue.

Below are the ports allowed for the server's network.

ALLOW IPv6 to ::/0
ALLOW IPv4 514/tcp from 0.0.0.0/0
ALLOW IPv4 1514/tcp from 0.0.0.0/0
ALLOW IPv4 1516/tcp from 0.0.0.0/0
ALLOW IPv4 443/tcp from 0.0.0.0/0
ALLOW IPv4 9200/tcp from 0.0.0.0/0
ALLOW IPv4 1515/tcp from 0.0.0.0/0
ALLOW IPv4 514/udp from 0.0.0.0/0
ALLOW IPv4 55000/tcp from 0.0.0.0/0
ALLOW IPv4 to 0.0.0.0/0
ALLOW IPv4 22/tcp from 0.0.0.0/0
ALLOW IPv4 1514/udp from 0.0.0.0/0
ALLOW IPv4 6300-6400/tcp from 0.0.0.0/0

Logs and screenshots attached.

[Lamya Imam]

Hello Veera,

I believe the IP address you are using for the dashboard is not correct. Instead of  10.x.x.x which is a private IP, use the public IP of the dashboard  (192.168.0.28 -> if this is the IP where you have installed the dashboard). Edit both the configuration files at /etc/wazuh-dashboard/opensearch_dashboards.yml and  /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml and put the public IP instead.

Then restart the Wazuh dashboard and check if it is accessible now:
systemctl restart wazuh-dashboard

If the issue still persists, please share the full wazuh indexer log:
cat /var/log/wazuh-indexer/wazuh-indexer-cluster.log 

It is a bit odd that you do not have the wazuhapp.log file as it should be there after the installation [Screenshot for reference].
Could you please share the output of this command:
ls -lrt /usr/share/wazuh-dashboard/data/wazuh/logs/


From the journalctl logs, we can see this error:
"@timestamp":"2024-06-24T21:12:34Z","tags":["error","opensearch","data"],"pid":36140,"message":"[index_not_found_exception]: no such index [.kibana_1]"} 

Could you please run this command and check if the kibana indices were created [See the screenshot for reference]:
curl https://<indexer_ip>:9200/_cat/indices/ -u <user>:<pass> -k

If it is not there then you might have to re-create the kibana indices, but I would suggest you to remove the wazuh dashboard and re-install it. 

Also, ensure that you have created the certificates for the public IP of the dashboard, otherwise you have to re-create them as well. 

Let me know if you need further assistance on this!

[Veera]

Hi Lamya,

Let me answer your suggestions one by one..

I believe the IP address you are using for the dashboard is not correct. Instead of  10.x.x.x which is a private IP, use the public IP of the dashboard  (192.168.0.28 -> if this is the IP where you have installed the dashboard). Edit both the configuration files at /etc/wazuh-dashboard/opensearch_dashboards.yml and  /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml and put the public IP instead.

Then restart the Wazuh dashboard and check if it is accessible now:
systemctl restart wazuh-dashboard

NO Luck ... Even trying  with Private or public IP's  on both files with all probabilities.   Also the IP's 10.x.x.x are public and 192.x.x.x are private.   192.168.0.28   is where the dashboard is installed and  it has the public IP as 10.x.x.23 (Refer to the attachment)

If the issue still persists, please share the full wazuh indexer log:
cat /var/log/wazuh-indexer/wazuh-indexer-cluster.log 

No such file exists .. also no file with the name /var/log/*-indexer-*  available  (Refer to the attachment)

It is a bit odd that you do not have the wazuhapp.log file as it should be there after the installation [Screenshot for reference].
Could you please share the output of this command:
ls -lrt /usr/share/wazuh-dashboard/data/wazuh/logs/

NO such file exists . Actually when the dashboard is installed or re-installed ,  there are no directories inside /usr/share/wazuh-dashboard/data/   .. I have manually created the directories and  set dashboard as owner/group for the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml

From the journalctl logs, we can see this error:
"@timestamp":"2024-06-24T21:12:34Z","tags":["error","opensearch","data"],"pid":36140,"message":"[index_not_found_exception]: no such index [.kibana_1]"} 

Could you please run this command and check if the kibana indices were created [See the screenshot for reference]:
curl https://<indexer_ip>:9200/_cat/indices/ -u <user>:<pass> -k

Yes , this  "[index_not_found_exception]: no such index [.kibana_1]"}   is causing the dashboard service to fail  and this problem exists from Jun 25). If dashboard service is running then only we can access it on either Ptrivate or public IP .   (Refer to the attachment)
I too tried many options , still no luck 

If it is not there then you might have to re-create the kibana indices, but I would suggest you to remove the wazuh dashboard and re-install it. 

Dashboard re-installed and hit the same   Hurdle ...  !!! . Review the logs for kibana indices

Also, ensure that you have created the certificates for the public IP of the dashboard, otherwise you have to re-create them as well

Yes , Certificates are created for Public IP   but as the dashboard service is never is running state , we have to wait ..

Thanks for your help and time ...  

[Veera]

Please Hold-on .. I am upgrading to 4.8.1

[Veera]

Hi Lamya ,

After following the document pages to upgrade to 4.8,1  , I faced the same issue . 

@timestamp":"2024-06-24T21:12:34Z","tags":["error","opensearch","data"],"pid":36140,"message":"[index_not_found_exception]: no such index [.kibana_1]"

I reset the password with the tool   wazuh-passwords-tool.sh for all  in the "master wazuh  server" and  and ensure the same is applied to the filebeat and Indexer .

Then it started working . 

I would summarize that the Upgrade might help , but somewhere there was a mismatch is the passwords caused the issue .

Thanks for your help  and inputs  .. 

[Lamya Imam]

Hello Veera,

I am glad your issue has been resolved. Thank you for letting me know.

Regards,

Lamya Imam Security Engineer, Operations

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/UaOCMLs1LM0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/cd28d60e-56cf-492e-9213-35cb0b2b8e59n%40googlegroups.com.