Wazuh - Facing some issues [IndexerConnector, All shards failed, ConnectionError]

[DogthMaul]

Hello.

Im facing the following issues since the v4.8.2 Wazuh upgrade:

• indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-******', retrying until the connection is successful.
• opensearch-dashboards[800]: {"type":"log","@timestamp":"2024-10-03T13:30:05Z","tags":["error","opensearch","data"],"pid":800,"message":"[search_phase_execution_exception]: all shards failed"}
• opensearch-dashboards[800]: {"type":"log","@timestamp":"2024-10-03T13:29:21Z","tags":["error","opensearch","data"],"pid":800,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

I have changed the block. In the block I have put the correct certificates from /etc/filebeat/certs and the IP from the /etc/filebeat/filebeat.yml file.
I have also saved the admin keys in the wazuh-keystore.
I have also created a retention policy for the indexes but it seems to be stuck as well.

The errors I am experiencing at the user level are as follows:

• Vulnerability Detection events are not updating. Detections with status “solved” keep appearing in the Dashboard.
• There are thousands of alerts that are false positives that I would like to filter out.
• The index retention policy seems to have deleted quite a few, but does not finish executing. It gets stuck.
• With a “GET _cluster/health” I see that the status is “yellow” and the number of unassigned shards is 66.

I have checked hundreds of forums without being able to fix it.

Thank you very much in advance.

[Othniel Ebolum]

Hello Dogmaul,

The error relates to the indexer configuration and connection.

The first troubleshooting step is to check the status of your Wazuh indexer(s)

sudo systemctl status wazuh-indexer

Ensure that is it running and has no errors.

A quick fix to issue is to restart the indexer service

sudo systemctl restart wazuh-indexer

You stated you have checked the indexer connection configuration block on the manager's ossec.conf file, but i recommend cross-checking again and ensuring the block is also enabled and not disabled. if any changes are made on your investigation, always restart the Wazuh-manager service after changes to implement them.

I would also recommend changing the passwords of the service users for your Wazuh infrastructure as probably the kibana user credential is no longer the same after the upgrade.

Kindly follow the documentation on password management to help with this https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

After all these, you can decide to restart all modules just to check their status. You can use the wazuh-control tool to do so located in this directory on your wazuh server /var/ossec/bin.

Kindly browse through other troubleshooting steps you can follow after an upgrade has been done here https://documentation.wazuh.com/current/upgrade-guide/troubleshooting.html

Best regards, 

[DogthMaul]

Hello, Othniel.

Thanks for the answer. I will summarize the details gathered after following the steps you indicate.

sudo systemctl restart wazuh-indexer

● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-10-04 09:18:33 CEST; 2 days ago
       Docs: https://documentation.wazuh.com
   Main PID: 29073 (java)
      Tasks: 161 (limit: 18791)
     Memory: 9.0G
        CPU: 4h 29min 50.434s
     CGroup: /system.slice/wazuh-indexer.service
             └─29073 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms7893m -Xmx7893m -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-6989345212597649386 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/wazuh-indexer -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m" -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy -XX:MaxDirectMemorySize=4139778048 -Dopensearch.path.home=/usr/share/wazuh-indexer -Dopensearch.path.conf=/etc/wazuh-indexer -Dopensearch.distribution.type=rpm -Dopensearch.bundled_jdk=true -cp "/usr/share/wazuh-indexer/lib/*" org.opensearch.bootstrap.OpenSearch -p /run/wazuh-indexer/wazuh-indexer.pid --quiet

Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Oct 07 00:00:01 sicmad8809 systemd-entrypoint[29073]:         at java.base/java.lang.Thread.run(Thread.java:833)

If I restart the service, the same status is displayed.

This is my vulnerability-detection and indexer blocks:

 <vulnerability-detection>
    <enabled>yes</enabled>
    <index-status>yes</index-status
    <feed-update-interval>60m</feed-update-interval>
  </vulnerability-detection>

  <indexer>
    <enabled>yes</enabled>
    <hosts>
      <host>https://127.0.0.1:9200</host>
    </hosts>
    <ssl>
      <certificate_authorities>
        <ca>/etc/filebeat/certs/root-ca.pem</ca>
      </certificate_authorities>
      <certificate>/etc/filebeat/certs/wazuh-server.pem</certificate>
      <key>/etc/filebeat/certs/wazuh-server-key.pem</key>
    </ssl>
  </indexer>

You can check the certs paths are right:

root@ #########  :/# ll /etc/filebeat/certs/
total 20
dr-x------ 2 root root 4096 Oct  7 07:58 ./
drwxr-xr-x 5 root root 4096 Oct  7 07:58 ../
-r-------- 1 root root 1204 May 12  2023 root-ca.pem
-r-------- 1 root root 1704 May 12  2023 wazuh-server-key.pem
-r-------- 1 root root 1285 May 12  2023 wazuh-server.pem

And the host IP is the one that its displayed at the filebeat.yml:

root@#########:/# cat /etc/filebeat/filebeat.yml
# Wazuh - Filebeat configuration file
output.elasticsearch.hosts:
        - 127.0.0.1:9200
#        - <elasticsearch_ip_node_2>:9200
#        - <elasticsearch_ip_node_3>:9200

output.elasticsearch:
  protocol: https
  username: ${username}
  password: ${password}
  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/wazuh-server.pem"
  ssl.key: "/etc/filebeat/certs/wazuh-server-key.pem"

If I reset the password for kibana, in which configuration files should I set it or how should I save it?

I have to clarify that the main reason for opening the case is that the Vulnerability Detection events are not updated. Neither new ones appear nor the patched ones disappear.

Thanks for all !!

[Othniel Ebolum]

Hello Dogthmul,

Your configuration and status of your indexer looks ok to me,

Concerning the change of password for your users i gave reference to follow here in the password management section of our documentation, please go over it and it will help you change your passwords successfully.

Also now to focus on the reason for your ticket which you say is the "Vulnerability Detection events are not updated."  then a clear indication from the troubleshooting upgrade guide i sent points to this section Vulnerability detection seems to be disabled or has a problem
"

1. Verify that the vulnerability index wazuh-states-vulnerabilities-* has been correctly created. You can check this under Indexer Management > Index Management > Indices configuration.

2. If the index wasn't created, check the Wazuh manager logs for any errors or warnings, as the issue might be related to errors mentioned in previous sections:"
   

Kindly go through and follow the troubleshooting steps.

Best Regards, 

[DogthMaul]

I think it has to do with the state of the cluster:

curl -k -u admin:############## -X GET "https://localhost:9200/_cluster/health?pretty"
{
  "cluster_name" : "wazuh",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 341,
  "active_shards" : 341,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 6,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 98.27089337175792
}

Thanks.