Opensearch - Not initizialed

[christineIT]

Good morning.
I hope all is well team. An urgent matter. We are seeing that the opensearch is not initialized.

• Wazuh dashboard - Errors

● wazuh-dashboard.service - wazuh-dashboard
   Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-10-09 00:30:28 CEST; 9h ago
 Main PID: 3901 (node)
   CGroup: /system.slice/wazuh-dashboard.service
           └─3901 /usr/share/wazuh-dashboard/node/fallback/bin/node /usr/share/wazuh-dashboard/src/cli/dist

Oct 09 09:37:57 ip-.eu-west-1.compute.internal opensearch-dashboards[3901]: {"type":"log","@timestamp":"2024-10-09T07:37:57Z","tags":["error","opensearch","data"],"pid":3901,"message":"[ResponseError]: Response Error"}
Oct 09 09:38:00 ip-.eu-west-1.compute.internal opensearch-dashboards[3901]: {"type":"log","@timestamp":"2024-10-09T07:38:00Z","tags":["error","opensearch","data"],"pid":3901,"message":"[ResponseError]: Response Error"}

• /usr/share/wazuh-indexer/bin/indexer-security-init.sh

OpenSearch Version: 2.13.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]. This is not an error, will keep on trying ...

filebeat test output
elasticsearch: https://xxx.xxx.xxx.xxx:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.10.192
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 503 Service Unavailable: OpenSearch Security not initialized.

• Wazuh version --> 4.9
• Wazuh manager and indexer OK
  
  Any ideas on how to solve it?
  
  

[Stuti Gupta]

Hi  christineIT

The "ERROR 503 Service Unavailable: OpenSearch Security not initialized" message indicates that the OpenSearch security plugin has not been fully initialized or configured. This can happen when OpenSearch is not able to initialize its security settings, causing it to become unavailable.

Can you please run the following command: /usr/share/wazuh-indexer/bin/indexer-security-init.sh

Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Also share the output of the following command:
Check the cluster health with:
curl -XGET -k -u user:pass "https://localhost:9200/_cluster/health"
Please share the indexer logs:
cat /var/log/wazuh-indexer/wazuh-cluster.log

In case cluster health is read then please perform the following solutions:
Solution1:  Delete Unassigned Shards: You can use the command: curl -k -XGET -u user:pass "https://<elasticsearxch>:9200/_cat/shards" | grep UNASSIGNED | awk '{print $1}' | xargs -i curl -k -XDELETE -u user:pass "https://<indexer_ip>:9200/{}"  
 https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html

Solution 2: Delete the indices manually
It is necessary to delete old indices to if they are no use. It is necessary to check what the indices stored in the environment, the following API call can help:
GET _cat/indices
The API call to delete indices is:
DELETE <index_name>
Or CLI command
 # curl -k -u admin:admin -XDELETE https://<WAZUH_INDEXER_IP>:9200/wazuh-alerts-4.x-YYYY.MM.DD
You can use wildcards (*) to delete more indices in one query.
Note: this cannot be retrieved unless there are backups of the data either using snapshots or Wazuh alerts backups.

Solution 3 : Index management policies:
Since storage space has a cost and a limit, you may have to delete old data to ensure you can maintain the retention period that you need.
Alerts generated by Wazuh are sent to an indexer daily index named wazuh-alerts-4.x-YYYY.MM.DD by using the default configuration. You can create policies that govern the lifecycle of the indices based on different phases.
Four phases can be defined in a Lifecycle Policy:
Hot phase. For recent data that is actively accessed.
Warm phase. Data that you may wish to access, but less often.
Cold phase. Similar to the warm phase you may also freeze indices to reduce overhead.
Delete phase. Data that reaches this phase is deleted.
You can follow the steps mentioned in this document https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html. You can also take snapshots of the indices that automatically back up your Wazuh indices in local or Cloud-based storage and restore them at any given time. To do so please refer to https://wazuh.com/blog/index-backup-managementhttps://wazuh.com/blog/wazuh-index-management/

Solution4 : You can add a indexer node.
Adding a new node to the Wazuh indexer cluster can enhance the capacity and resilience of the security monitoring infrastructure. https://documentation.wazuh.com/current/user-manual/upscaling/adding-indexer-node.html

Hope this helps

[christineIT]

Thank you very much Stuti Gupta for the information. I am providing the requested data.

With all this, what operative can be applied to solve the problem with Opensearch and wazuh?

/indexer-security-init.sh --options --accept-red-cluster
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to xxx.xxx.xxx.xxx:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.13.0
Contacting opensearch cluster 'opensearch' ...
Clustername: wazuh-cluster
Clusterstate: RED
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
ERR: .opendistro_security index state is RED.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml
   FAIL: Configuration for 'config' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml
   FAIL: Configuration for 'roles' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-7 [ACTIVE]
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml
   FAIL: Configuration for 'rolesmapping' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-8 [ACTIVE]
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml
   FAIL: Configuration for 'internalusers' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-9 [ACTIVE]
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml
   FAIL: Configuration for 'actiongroups' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-10 [ACTIVE]
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml
   FAIL: Configuration for 'tenants' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-11 [ACTIVE]
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml
   FAIL: Configuration for 'nodesdn' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-12 [ACTIVE]
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml
   FAIL: Configuration for 'whitelist' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-13 [ACTIVE]
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml
   FAIL: Configuration for 'audit' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-14 [ACTIVE]
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
...

cluster health
[root@ip- bin]# curl -XGET -k -u admin:---- "https://localhost:9200/_cluster/health"
curl: (7) Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
[root@ip- bin]# curl -XGET -k -u admin:---- "https://xxx.xxx.xxx.xxx:9200/_cluster/health"
OpenSearch Security not initialized.

cat /var/log/wazuh-indexer/wazuh-cluster.log
[2024-10-09T13:52:06,959][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-10-09T13:52:08,794][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:08,795][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:08,796][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:08,797][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:08,800][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@13b78ed4] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-10-09T13:52:11,297][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:11,298][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:11,300][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:11,301][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:11,960][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-10-09T13:52:13,797][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:13,798][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:13,799][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-10-09T13:52:13,801][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)

kind regards!!

[christineIT]

Hi team, 

could you please indicate if you can provide any update on the above post?

Thanks in advance 

[Stuti Gupta]

Hi 

As you can see  Clusterstate: RED To solve this please perform the following solutions:

Solution1:  Delete Unassigned Shards: You can use the command: curl -k -XGET -u user:pass "https://<elasticsearxch>:9200/_cat/shards" | grep UNASSIGNED | awk '{print $1}' | xargs -i curl -k -XDELETE -u user:pass "https://<indexer_ip>:9200/{}"  
 https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html

Solution 2: Delete the indices manually
It is necessary to delete old indices to if they are no use. It is necessary to check what the indices stored in the environment, the following API call can help:
GET _cat/indices
The API call to delete indices is:
DELETE <index_name>
Or CLI command
 # curl -k -u admin:admin -XDELETE https://<WAZUH_INDEXER_IP>:9200/wazuh-alerts-4.x-YYYY.MM.DD
You can use wildcards (*) to delete more indices in one query.
Note: this cannot be retrieved unless there are backups of the data either using snapshots or Wazuh alerts backups.

Solution 3 : Index management policies:
Since storage space has a cost and a limit, you may have to delete old data to ensure you can maintain the retention period that you need.
Alerts generated by Wazuh are sent to an indexer daily index named wazuh-alerts-4.x-YYYY.MM.DD by using the default configuration. You can create policies that govern the lifecycle of the indices based on different phases.
Four phases can be defined in a Lifecycle Policy:
Hot phase. For recent data that is actively accessed.
Warm phase. Data that you may wish to access, but less often.
Cold phase. Similar to the warm phase you may also freeze indices to reduce overhead.
Delete phase. Data that reaches this phase is deleted.
You can follow the steps mentioned in this document https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html. You can also take snapshots of the indices that automatically back up your Wazuh indices in local or Cloud-based storage and restore them at any given time. To do so please refer to https://wazuh.com/blog/index-backup-managementhttps://wazuh.com/blog/wazuh-index-management/

Solution4 : You can add a indexer node.
Adding a new node to the Wazuh indexer cluster can enhance the capacity and resilience of the security monitoring infrastructure. https://documentation.wazuh.com/current/user-manual/upscaling/adding-indexer-node.html

Hope this helps