Unable to start wazuh-indexer

Chris Curtis

unread,

Apr 10, 2023, 6:23:27 PM4/10/23

to Wazuh mailing list

Hi,

I have an all-in-one instance of wazuh which is a couple of weeks old. It has only 2 agents at this point.

After performing an os update and rebooting the sever, the wazuh-indexer server gets an error starting, which stops the UI from loading as well.

Version is
{"error":0,"data":[{"WAZUH_VERSION":"v4.4.0"},{"WAZUH_REVISION":"40405"},{"WAZUH_TYPE":"server"}]}

Output from service status
● wazuh-indexer.service - Wazuh-indexer
Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2023-04-10 22:07:16 UTC; 7s ago
Docs: https://documentation.wazuh.com
Process: 9472 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 9472 (code=exited, status=1/FAILURE)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.Command.main(Command.java:101)
Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Apr 10 22:07:16 wazuh-server systemd[1]: wazuh-indexer.service: main process exited, code=exited, status=1/FAILURE
Apr 10 22:07:16 wazuh-server systemd[1]: Failed to start Wazuh-indexer.
Apr 10 22:07:16 wazuh-server systemd[1]: Unit wazuh-indexer.service entered failed state.
Apr 10 22:07:16 wazuh-server systemd[1]: wazuh-indexer.service failed.

The other thing that had changed was an hour or so earlier I had created a index management policy as per attached. I'm not sure if this is related but thought it worth mentioning.

Hopefully someone can provide some guidance so I can get it back online.

index-policy.txt

Leandro David Sayanes

unread,

Apr 11, 2023, 8:10:44 AM4/11/23

to Wazuh mailing list

Hi Chris Curtis I will try to help you,

First let's check some logs

You can run the following commands and send me the output please:

systemctl status wazuh-dashboard

Dasboard error logs:

journalctl -u wazuh-dashboard cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

Also share the status:

systemctl status wazuh-indexer

If Wazuh-Indexer shows active share the log file by running:

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

wazuh-cluster is the default name. if a custom name is set change it there.

You can also try restarting the services to fix any connection problems that occurred at runtime by running the following command:

systemctl restart wazuh-indexer
systemctl restart wazuh-dashboard

Finally, you can get more help in this document:

https://documentation.wazuh.com/current/user-manual/wazuh-dashboard/troubleshooting.html

I will be waiting for any news!
Best regards!

Chris Curtis

unread,

Apr 12, 2023, 10:03:43 PM4/12/23

to Wazuh mailing list

Hi,

Thank you for replying. I really appreciate it.

systemctl status wazuh-dashboard

● wazuh-dashboard.service - wazuh-dashboard

Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: disabled)

Active: active (running) since Mon 2023-04-10 08:24:03 UTC; 1 day 13h ago

Main PID: 1741 (node)

CGroup: /system.slice/wazuh-dashboard.service

└─1741 /usr/share/wazuh-dashboard/bin/../node/bin/node --no-warnings --max-http-header-size=65536 --unhandled-rejections=warn /usr/share/wazuh-dashboard/bin/../src/cli/dist -c /etc/wazuh-dashboard/opensearch_dashboards.yml

Apr 11 22:21:45 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:21:45Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:21:48 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:21:48Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:21:50 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:21:50Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:21:53 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:21:53Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:21:55 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:21:55Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:21:58 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:21:58Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:22:00 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:22:00Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:22:03 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:22:03Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:22:05 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:22:05Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Apr 11 22:22:08 wazuh-server opensearch-dashboards[1741]: {"type":"log","@timestamp":"2023-04-11T22:22:08Z","tags":["error","opensearch","data"],"pid":1741,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

No output from the wazuhapp.log grep. The last entries in that log are:

{"date":"2023-03-29T08:38:41.810Z","level":"info","location":"initialize","message":"Wazuh dashboard index: .kibana"}

{"date":"2023-03-29T08:38:41.811Z","level":"info","location":"initialize","message":"App revision: 06"}

{"date":"2023-03-29T08:38:41.811Z","level":"info","location":"initialize","message":"Total RAM: 3872MB"}

systemctl status wazuh-indexer

● wazuh-indexer.service - Wazuh-indexer

Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)

Active: failed (Result: exit-code) since Mon 2023-04-10 22:07:16 UTC; 24h ago

Docs: https://documentation.wazuh.com

Process: 9472 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=1/FAILURE)

Main PID: 9472 (code=exited, status=1/FAILURE)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.Command.main(Command.java:101)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

Apr 10 22:07:16 wazuh-server systemd[1]: wazuh-indexer.service: main process exited, code=exited, status=1/FAILURE

Apr 10 22:07:16 wazuh-server systemd[1]: Failed to start Wazuh-indexer.

Apr 10 22:07:16 wazuh-server systemd[1]: Unit wazuh-indexer.service entered failed state.

Apr 10 22:07:16 wazuh-server systemd[1]: wazuh-indexer.service failed.

Restarting the wazuh-indexer service didn’t work – same errors.

Restarting the wazuh-dashboard service worked fine.

The wazuh-cluster log has nothing since shutdown messages from the system reboot.

I can’t find any opensearch logs, or at least they are not obvious.

Here is a find of *.log

/var/log/wazuh-install.log

/var/log/wazuh-indexer/wazuh-cluster_deprecation.log

/var/log/wazuh-indexer/wazuh-cluster_index_indexing_slowlog.log

/var/log/wazuh-indexer/wazuh-cluster_index_search_slowlog.log

/var/log/wazuh-indexer/wazuh-cluster_task_detailslog.log

/var/log/wazuh-indexer/wazuh-cluster.log

/var/log/wazuh-indexer/gc.log

/var/log/wazuh-passwords-tool.log

/var/ossec/logs/alerts/2023/Mar/ossec-alerts-28.log

/var/ossec/logs/alerts/2023/Apr/ossec-alerts-13.log

/var/ossec/logs/alerts/alerts.log

/var/ossec/logs/archives/2023/Mar/ossec-archive-28.log

/var/ossec/logs/archives/2023/Apr/ossec-archive-13.log

/var/ossec/logs/archives/archives.log

/var/ossec/logs/firewall/2023/Mar/ossec-firewall-28.log

/var/ossec/logs/firewall/2023/Apr/ossec-firewall-13.log

/var/ossec/logs/firewall/firewall.log

/var/ossec/logs/active-responses.log

/var/ossec/logs/integrations.log

/var/ossec/logs/cluster.log

/var/ossec/logs/api.log

/var/ossec/logs/ossec.log

/var/ossec/stats/totals/2023/Mar/ossec-totals-29.log

/var/ossec/stats/totals/2023/Mar/ossec-totals-30.log

/var/ossec/stats/totals/2023/Mar/ossec-totals-31.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-01.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-02.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-03.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-04.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-05.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-06.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-07.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-08.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-09.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-10.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-11.log

/var/ossec/stats/totals/2023/Apr/ossec-totals-12.log

/usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log

/usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp-plain.log

/usr/share/wazuh-dashboard/node_modules/d3-collection/yarn-error.log

/usr/share/wazuh-dashboard/node_modules/is-arrayish/yarn-error.log

/usr/share/wazuh-dashboard/plugins/alertingDashboards/node_modules/d3-collection/yarn-error.log

/usr/share/wazuh-dashboard/plugins/reportsDashboards/node_modules/nwsapi/dist/lint.log

/usr/share/wazuh-dashboard/plugins/wazuh/node_modules/is-arrayish/yarn-error.log

Best regards,

Chris

Leandro David Sayanes

unread,

Apr 14, 2023, 1:15:58 PM4/14/23

to Wazuh mailing list

Hi Chris Curtis!

Regarding to this log:

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)

Apr 10 22:07:16 wazuh-server systemd-entrypoint[9472]: at org.opensearch.cli.Command.main(Command.java:101)

We will need more information and to see the OpenSearch logs, could you check them ? they are usually in

/var/log/opensearch but wazuh-indexer might put them here /usr/share/wazuh-dashboard or here /var/log/wazuh-dashboard

Just in case, also check that issue:
https://github.com/opensearch-project/opensearch-build/issues/1477

If we cannot get more information from the logs and as a last option you can reinstall the Indexer, check this issue:

https://github.com/wazuh/wazuh/issues/15523

Here is a guide to retrieve the info:

https://wazuh.com/blog/recover-your-data-using-wazuh-alert-backups/

I hope this help you!

Message has been deleted

bhuvanesh ammisetty

unread,

Feb 15, 2024, 1:57:05 AM2/15/24

to Wazuh | Mailing List

@7ee74ae0] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-02-15T12:21:49,507][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [index1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@7ee74ae0] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)

i am getting this error in my wazuh -indexer.please help me troubleshoot

Reply all

Reply to author

Forward