wazuh-indexer failed to start - OpenSearchException[Unable to load plugin class [org.opensearch.securityanalytics.SecurityAnalyticsPlugin]]

[ryanrudolf]

Hello wazuh team,

My wazuh server has been up and running for several months but today I'm not able to access it. I've restarted the services and wazuh-indexer fails to start.

This is the error I'm getting -

-- Logs begin at Mon 2025-02-24 09:21:38 EST, end at Mon 2025-02-24 09:37:52 EST. --
Feb 24 09:21:43  systemd[1]: Starting Wazuh-indexer...
-- Subject: Unit wazuh-indexer.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit wazuh-indexer.service has begun starting up.
Feb 24 09:21:46  systemd-entrypoint[1400]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 24 09:21:46  systemd-entrypoint[1400]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.8.0.jar)
Feb 24 09:21:46  systemd-entrypoint[1400]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 24 09:21:46  systemd-entrypoint[1400]: WARNING: System::setSecurityManager will be removed in a future release
Feb 24 09:21:47  systemd-entrypoint[1400]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 24 09:21:47  systemd-entrypoint[1400]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.8.0.jar)
Feb 24 09:21:47  systemd-entrypoint[1400]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 24 09:21:47  systemd-entrypoint[1400]: WARNING: System::setSecurityManager will be removed in a future release
Feb 24 09:21:51  systemd-entrypoint[1400]: uncaught exception in thread [main]
Feb 24 09:21:51  systemd-entrypoint[1400]: OpenSearchException[Unable to load plugin class [org.opensearch.securityanalytics.SecurityAnalyticsPlugin]]; nested: ClassNotFoundException[org.opensearch.securityanalytics.SecurityAnalyticsPlugin];
Feb 24 09:21:51  systemd-entrypoint[1400]: Likely root cause: java.lang.ClassNotFoundException: org.opensearch.securityanalytics.SecurityAnalyticsPlugin
Feb 24 09:21:51  systemd-entrypoint[1400]:         at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at java.base/java.lang.Class.forName0(Native Method)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at java.base/java.lang.Class.forName(Class.java:467)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.plugins.PluginsService.loadPluginClass(PluginsService.java:758)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:719)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.node.Node.<init>(Node.java:454)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.node.Node.<init>(Node.java:381)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.cli.Command.main(Command.java:101)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Feb 24 09:21:51  systemd-entrypoint[1400]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Feb 24 09:21:51  systemd-entrypoint[1400]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Feb 24 09:24:43  systemd[1]: wazuh-indexer.service: start operation timed out. Terminating.
Feb 24 09:24:43  systemd[1]: wazuh-indexer.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit wazuh-indexer.service has entered the 'failed' state with result 'timeout'.
Feb 24 09:24:43  systemd[1]: Failed to start Wazuh-indexer.
-- Subject: Unit wazuh-indexer.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit wazuh-indexer.service has failed.

I've already restarted the service multiple times and rebooted the VM multiple times but still getting the error. Its strange that all of a sudden the service will fail to start. Please help. Thank you.

[Delfina Lizarralde Bressan]

Hi Ryan!

The error indicates that the indexer is trying to load the Security Analytics plugin, but it can’t find the expected class.
This “ClassNotFoundException” usually means that the SecurityAnalyticsPlugin jar might be missing or was inadvertently removed or corrupted.
Inspect the /usr/share/wazuh-indexer/lib/ or /usr/share/wazuh-indexer/plugins/ directory for the opensearch-securityanalytics jar file. If it’s missing, you might need to reinstall using
/usr/share/wazuh-indexer/bin/opensearch-plugin install <PLUGIN_NAME>

Let me know how this goes.
Regards.

[ryanrudolf]

Additional troubleshooting I've performed -

list installed plugins -

/usr/share/wazuh-indexer/bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security

opensearch-security-analytics
opensearch-sql

Since it is complaining about opensearch-security, I tried to remove it -

/usr/share/wazuh-indexer/bin/opensearch-plugin remove opensearch-security-analytics

Restart wazuh-indexer and it now works!

But why did it fail on that plugin? When i try to re-install the plugin it gives error -

/usr/share/wazuh-indexer/bin/opensearch-plugin install opensearch-security-analytics

-> Installing opensearch-security-analytics
-> Failed installing opensearch-security-analytics
-> Rolling back opensearch-security-analytics
-> Rolled back opensearch-security-analytics
A tool for managing installed opensearch plugins

Non-option arguments:
[String] -- command

Option             Description
------             -----------
-E <KeyValuePair>  Configure a setting
-h, --help         Show help
-s, --silent       Show minimal output
-v, --verbose      Show verbose output
ERROR: Unknown plugin opensearch-security-analytics

[ryanrudolf]

Is there any way I can re-add the opensearch-security-analytics plugin?

[Delfina Lizarralde Bressan]

Hi ryan!

Could you share what error are you encountering? You can check Indexer logs by running:
cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"