Wazuh dashboard server is not ready yet after upgrade to 4.6.0-1

[Jerry Bike]

Hello,

after upgrade my ubuntu 22.04 LTS server monthly upgrade via apt upgrade the wazuh komponents were upgraded from 4.5 to 4.6 and after reboot the dashboard stopped working.

It is all in one instalation. I found weird behavior that service listening on tcp6 port 9200 but ipv4 address. 

~# netstat -a -n |grep 9200
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN

I do not use ipv6 at all.

All advices are appreciated

[Ingeniux Network Operations]

I have same issue after upgrading from 4.5 to 4.6. Cluster is green, filebeat test is green. dashboard fails to load.

Logs originally showed:

Nov 02 16:21:39 wazuh01 opensearch-dashboards[24809]: {"type":"log","@timestamp":"2023-11-02T23:21:39Z","tags":["error","opensearch","data"],"pid":24809,"message":"[security_exception]: OpenSearch Security not initialized for indices:admin/get"}
Nov 02 16:21:42 wazuh01 opensearch-dashboards[24809]: {"type":"log","@timestamp":"2023-11-02T23:21:42Z","tags":["error","opensearch","data"],"pid":24809,"message":"[security_exception]: OpenSearch Security not initialized for indices:admin/get"}
Nov 02 16:21:44 wazuh01 opensearch-dashboards[24809]: {"type":"log","@timestamp":"2023-11-02T23:21:44Z","tags":["error","opensearch","data"],"pid":24809,"message":"[search_phase_execution_exception]: all shards failed"}

I re-ran the indexer-security-init.sh then the password tool since it reverted all my users/passwords to fix users. Now the all shards failed and admin indicdes error is gone but the dashboar still fails with the resource already exists error.

logs are below... I've tried deleting the kibana_4 index but it just repeats. I've also deleted _1-3 and now it just says _1 already exists.

Nov 03 10:00:28 wazuh01 opensearch-dashboards[61442]: {"type":"log","@timestamp":"2023-11-03T17:00:28Z","tags":["info","savedobjects-service"],"pid":61442,"message":"Starting saved objects migrations"}

Nov 03 10:00:28 wazuh01 opensearch-dashboards[61442]: {"type":"log","@timestamp":"2023-11-03T17:00:28Z","tags":["info","savedobjects-service"],"pid":61442,"message":"Detected mapping change in \"properties.visualization-visbuilder\""}

Nov 03 10:00:28 wazuh01 opensearch-dashboards[61442]: {"type":"log","@timestamp":"2023-11-03T17:00:28Z","tags":["info","savedobjects-service"],"pid":61442,"message":"Creating index .kibana_4."}

Nov 03 10:00:28 wazuh01 opensearch-dashboards[61442]: {"type":"log","@timestamp":"2023-11-03T17:00:28Z","tags":["error","opensearch","data"],"pid":61442,"message":"[resource_already_exists_exception]: index [.kibana_4/X_f76AjgRDueL1v0LVwmCQ] already exists"} Nov 03 10:00:28 wazuh01 opensearch-dashboards[61442]: {"type":"log","@timestamp":"2023-11-03T17:00:28Z","tags":["warning","savedobjects-service"],"pid":61442,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_4/X_f76AjgRDueL1v0LVwmCQ] already exists"}

Nov 03 10:00:28 wazuh01 opensearch-dashboards[61442]: {"type":"log","@timestamp":"2023-11-03T17:00:28Z","tags":["warning","savedobjects-service"],"pid":61442,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_4 and restarting OpenSearchDashboards."}

Nov 03 10:00:28 wazuh01 opensearch-dashboards[61442]: {"type":"log","@timestamp":"2023-11-03T17:00:28Z","tags":["info","savedobjects-service"],"pid":61442,"message":"Detected mapping change in \"properties.visualization-visbuilder\""} 

[Jerry Bike]

Today, I have checked in lab, that new instalation of 4.6 is in netstat visible only in line TCP6 but it works in lab. SO there is not maybe issue with TCP6 binding but something else.

Dne pátek 3. listopadu 2023 v 18:38:28 UTC+1 uživatel Ingeniux Network Operations napsal:

[suricata]

Hí Jerry,

The same happens to me. After many years, it is the first time that the dashboard stopped working in an update and there is no way to fix it.

[Jeremias Ignacio Posse]

Hi Jerry Bike I need more detail about the errors and your environment to check that everything works correctly.  

type these commands and send me the answer be careful not to send any confidential/private data or public IPs thanks!

Could you share the result of the following commands to check the situation?

systemctl status wazuh-manager

curl -k -X GET "https://<api_url>:55000/" -H "Authorization: Bearer $(curl -u <api_user>:<api_password> -k -X POST 'https://<api_url>:55000/security/user/authenticate?raw=true')"

curl https://<WAZUH_INDEXER_IP>:9200/_cat/indices/wazuh-alerts-* -u <wazuh_indexer_user>:<wazuh_indexer_password> -k


filebeat test output

[Jerry Bike]

Hello,

I thought that I have sent the ansfers but it looks like not ...

So once more :-) :

~# systemctl status wazuh-manager
● wazuh-manager.service - Wazuh manager
     Loaded: loaded (/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-12-04 16:35:28 UTC; 13min ago
    Process: 908 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/SUCCESS)
      Tasks: 142 (limit: 9389)
     Memory: 2.2G
        CPU: 13min 44.573s
     CGroup: /system.slice/wazuh-manager.service
             ├─1395 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
             ├─1397 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
             ├─1400 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
             ├─1403 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
             ├─1446 /var/ossec/bin/wazuh-authd
             ├─1464 /var/ossec/bin/wazuh-db
             ├─1486 /var/ossec/bin/wazuh-execd
             ├─1500 /var/ossec/bin/wazuh-analysisd
             ├─1513 /var/ossec/bin/wazuh-syscheckd
             ├─1579 /var/ossec/bin/wazuh-remoted
             ├─1580 /var/ossec/bin/wazuh-remoted
             ├─1612 /var/ossec/bin/wazuh-logcollector
             ├─1647 /var/ossec/bin/wazuh-monitord
             └─1661 /var/ossec/bin/wazuh-modulesd

Dec 04 16:35:19 siem-zcm env[908]: Started wazuh-execd...
Dec 04 16:35:20 siem-zcm env[908]: Started wazuh-analysisd...
Dec 04 16:35:21 siem-zcm env[908]: Started wazuh-syscheckd...
Dec 04 16:35:22 siem-zcm env[908]: Started wazuh-remoted...
Dec 04 16:35:24 siem-zcm env[908]: Started wazuh-logcollector...
Dec 04 16:35:25 siem-zcm env[908]: Started wazuh-monitord...
Dec 04 16:35:25 siem-zcm env[1659]: 2023/12/04 16:35:25 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
Dec 04 16:35:26 siem-zcm env[908]: Started wazuh-modulesd...
Dec 04 16:35:28 siem-zcm env[908]: Completed.
Dec 04 16:35:28 siem-zcm systemd[1]: Started Wazuh manager.


***************************************************************************

curl -k -X GET "https://<api_url>:55000/" -H "Authorization: Bearer $(curl -u <api_user>:<api_password> -k -X POST 'https://<api_url>:55000/security/user/authenticate?raw=true')"

when I change my server and credentials the output is:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    59  100    59    0     0   3958      0 --:--:-- --:--:-- --:--:--  4214
{"title": "Unauthorized", "detail": "Invalid token"}

*************************************************************************************

curl https://<WAZUH_INDEXER_IP>:9200/_cat/indices/wazuh-alerts-* -u <wazuh_indexer_user>:<wazuh_indexer_password> -k

when I change my server and credentials the output is:

curl: (7) Failed to connect to siem-zcm port 9200 after 0 ms: Connection refused

but when I change to localhost, the result is OK:

.......

green open wazuh-alerts-4.x-2023.11.30 079b3Q9qQnmM1zbp2XVh6w 3 0  35843 0  37.5mb  37.5mb
green open wazuh-alerts-4.x-2023.12.04 WIRQ13axSf-M6NjQEfHCaA 3 0  23653 0  41.1mb  41.1mb
green open wazuh-alerts-4.x-2023.12.03 -p4Hw-qMSxG9_AMtd1KPLQ 3 0  27548 0  26.4mb  26.4mb
green open wazuh-alerts-4.x-2023.12.02 JWZk6WR_QaW1qRZNgJNO5g 3 0  28459 0    24mb    24mb
green open wazuh-alerts-4.x-2023.12.01 aYVHj2MQQzKbBkUVv9Yg8w 3 0  33962 0  34.1mb  34.1mb

.....

**********************************
filebeat test output

elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 7.10.2

Today, I have tried to do apt upgrade on Ubuntu with automatical upgrade to 4.7 but result is the same

Thanks for next advice

Dne pondělí 6. listopadu 2023 v 19:52:52 UTC+1 uživatel Jeremias Ignacio Posse napsal:

[Jerry Bike]

This is the  dashboard status

systemctl status wazuh-dashboard

wazuh-dashboard.service - wazuh-dashboard

     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)

     Active: active (running) since Mon 2023-12-04 17:29:47 UTC; 2min 21s ago

   Main PID: 4037 (node)

      Tasks: 11 (limit: 9389)

     Memory: 176.7M

        CPU: 6.208s

     CGroup: /system.slice/wazuh-dashboard.service

             └─4037 /usr/share/wazuh-dashboard/node/bin/node --no-warnings --max-http-header-size=65536 --unhandled-rejections=warn /usr/share/wazuh-dashboard/src/cli/dist -c /etc/wazuh-dashboard/opensearch_dashb>

Dec 04 17:31:54 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:31:54Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:31:56 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:31:56Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:31:57 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:31:57Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:31:59 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:31:59Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:32:00 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:32:00Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:32:02 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:32:02Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:32:03 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:32:03Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:32:05 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:32:05Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:32:06 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:32:06Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

Dec 04 17:32:08 wazuh-server opensearch-dashboards[4037]: {"type":"log","@timestamp":"2023-12-04T17:32:08Z","tags":["info","savedobjects-service"],"pid":4037,"message":"Detected mapping change in \"properties.visuali>

~

po 4. 12. 2023 v 18:09 odesílatel Jerry Bike <jerry...@gmail.com> napsal:

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/Ckv0UEzEwbA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/afbed036-0934-4731-80cd-d5db14e2c26an%40googlegroups.com.

[Jerry Bike]

Hello,

after long time without answers I have found the article https://www.reddit.com/r/Wazuh/comments/17nlhed/wazuh_dashboard_server_is_not_ready_yet_resolved/ and use similar way to solve my issue.

systemctl stop wazuh-dashboard

curl -k -u admin:<password> https://localhost:9200/_cat/indices/.kib*

There were 2 files

green open .kibana_1 b8jXntI8Ru--q_X9449YYA 1 0 4 1 30.5kb 30.5kb

green open .kibana_3 8JMUbD03Sb2SzvMrvFsYrw 1 0 0 0   208b   208b

curl -k -X DELETE -u admin:<password> https://127.0.0.1:9200/.kibana_1

curl -k -X DELETE -u admin:<password> https://127.0.0.1:9200/.kibana_3

systemctl start wazhuh-dashboard

And dashboard works again :-)

po 4. 12. 2023 v 18:33 odesílatel Jerry Bike <jerry...@gmail.com> napsal:

[Faber Andres Cubides]

hello Jerry

Thanks for the information.
The same problem occurred to me, after updating the operating system wazuhdashboard did not load, I followed the steps you mentioned and they were useful, it now works for me
thank you

[Jerrybikecz]

Hello,

I’m glad to read it helped you. 

Jerry

7. 3. 2024 v 19:04, Faber Andres Cubides <pard...@gmail.com>:

hello Jerry

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/4395e075-72e1-4720-b6b0-779b1512bac8n%40googlegroups.com.