Error in the dashboard, the data is not displayed

49 views
Skip to first unread message

Henry Valero

unread,
Sep 2, 2025, 10:58:55 AM (5 days ago) Sep 2
to Wazuh | Mailing List
Hi,

I have the wazuh with 22 agents, and since last week it has been generating these errors. I have doubled the RAM memory of the servers and the error persists. The SIEM installation is distributed, one server for each component (03 servers). The hardware resources are sufficient, they do not reach the limit

How can I resolve this error?

Atte,
Henry
02.png
01.png
03.png

Henry Valero

unread,
Sep 2, 2025, 1:17:35 PM (5 days ago) Sep 2
to Wazuh | Mailing List
HI,
i run this command:
cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -iE "err|warn1crit"
and the result is:

[2025-09-02T01:15:44,167][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T01:15:46,658][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T01:15:46,664][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T01:15:46,668][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T01:15:46,672][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:41:43,709][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/lib/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2025-09-02T14:41:52,400][WARN ][stderr                   ] [node-1] WARNING: A restricted method in java.lang.foreign.Linker has been called
[2025-09-02T14:41:52,400][WARN ][stderr                   ] [node-1] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[2025-09-02T14:41:52,401][WARN ][stderr                   ] [node-1] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[2025-09-02T14:41:58,970][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2025-09-02T14:42:05,666][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
[2025-09-02T14:42:06,143][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/DHUkpErrQSy6ZPdIxQIagw] already exists
[2025-09-02T14:42:06,627][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:06,707][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:06,710][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:06,713][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:06,716][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,717][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,717][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,717][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,717][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,718][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,718][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:06,718][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,718][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,718][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:06,718][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@41d8cb32] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-09-02T14:42:07,292][INFO ][o.o.p.PluginsService     ] [node-1] PluginService:onIndexModule index:[.opensearch-observability/DHUkpErrQSy6ZPdIxQIagw]
[2025-09-02T14:42:08,436][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:08,447][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:08,449][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:08,450][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:08,467][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:08,485][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:08,494][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:08,516][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-09-02T14:42:08,961][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)

and when I run the  GET _cluster/health  command, I get this result in the picture attached

Can I help me.
Atte,
Henry
cluster-state.png

Md. Nazmur Sakib

unread,
Sep 3, 2025, 12:28:00 AM (5 days ago) Sep 3
to Wazuh | Mailing List

Hello Henry,


Check if the JVM configuration is tuned.

Start by checking your memory with:

free -h


Then, update the heap size in the /etc/wazuh-indexer/jvm.options file. For example, if your server has 12 GB of RAM, set the heap size to 6 GB as shown below:

-Xms6g

-Xmx6g


Here are some key points to keep in mind to configure heap memory:

Use no more than 50% of your available RAM.

Don’t set the heap size over 32 GB.
Make sure not to add any additional space before -Xm


After making these changes, restart the Wazuh indexer for them to take effect:

systemctl restart wazuh-indexer


You can refer to this link for more details:

https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#memory-locking


I can see a number of unassigned shares. The issue can be due to some unassigned system shards.


Check the unassigned shards and their unassignment reason.

curl -k -XGET -u admin:<admin_user’s_PASSWORD> "https://127.0.0.1:9200/_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state" | grep UNASSIGNED



curl -k -XGET -u admin:<admin_user’s_PASSWORD> "https://127.0.0.1:9200//_cluster/allocation/explain?pretty"


The full trace is not present on the cluster.log you've shared. Could you please run the following commands and share the output

- systemctl restart wazuh-indexer

- systemctl status wazuh-indexer

- journalctl -xeu wazuh-indexer.service


(It is a plus if you can get all the last logs from /var/log/wazuh-indexer/wazuh-cluster.log, with the corresponding logs to this last restart are enough. You can share the full log in a text file in the attachment.)


Looking forward to your update on the issue.

Henry Valero

unread,
Sep 3, 2025, 10:53:40 AM (4 days ago) Sep 3
to Wazuh | Mailing List
Hello Nazmur,
I made the suggested changes and ran the indicated commands, these are the results of the commands:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15487  100 15487    0     0   237k      0 --:--:-- --:--:.opendistro-alerting-config                          0     r      UNASSIGNED        CLUSTER_RECOVERED
--.opendistro-alerting-alert-history-2025.08.25-000002 0     r      UNASSIGNED        CLUSTER_RECOVERED
 -.opendistro-alerting-alert-history-2025.07.25-1      0     r      UNASSIGNED        CLUSTER_RECOVERED
-.opensearch-alerting-config-lock                     0     r      UNASSIGNED        CLUSTER_RECOVERED
:-.opendistro-alerting-alerts                          0     r      UNASSIGNED        CLUSTER_RECOVERED
-:--  240k

{
  "index" : ".opendistro-alerting-config",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2025-09-03T14:06:57.912Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "M1q3YHEWSMKmn6kqibCWmg",
      "node_name" : "node-1",
      "transport_address" : "mi_ip_server:9300",
      "node_attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "a copy of this shard is already allocated to this node [[.opendistro-alerting-config][0], node[M1q3YHEWSMKmn6kqibCWmg], [P], s[STARTED], a[id=xRZ48ryiS8Sg18BKysKuBA]]"
        }
      ]
    }
  ]
}

  • I also attach the requested log
I await your instructions
Atte,
Henry
log
Reply all
Reply to author
Forward
0 new messages