Wazuh does not populate wazuh-alerts-*

[Commercial League]

Hi,

I updated wazuh from 4.13 to 4.14 and everything was working fine. After about 10 days wazuh-alerts stopped being populated with data. I checked the indexes and I do not have any new index files since 9.11.2025.

In the log file there were many errors:

Nov 24, 2025 @ 09:55:38.000 indexer-connector ERROR  HTTP response code said error, status code: 400.
Nov 24, 2025 @ 09:55:40.000 indexer-connector ERROR  HTTP response code said error, status code: 400.
Nov 24, 2025 @ 09:55:42.000 indexer-connector ERROR  HTTP response code said error, status code: 400.
Nov 24, 2025 @ 09:55:43.000 indexer-connector ERROR  HTTP response code said error, status code: 400.
Nov 24, 2025 @ 09:55:45.000 indexer-connector ERROR  HTTP response code said error, status code: 400.
Nov 24, 2025 @ 09:55:47.000 indexer-connector ERROR  HTTP response code said error, status code: 400.
Nov 24, 2025 @ 09:55:49.000 indexer-connector ERROR  HTTP response code said error, status code: 400.
Nov 24, 2025 @ 09:55:50.000 indexer-connector ERROR  HTTP response code said error, status code: 400.

I suggested (wrongly) that I hit a bug so I updated to the minor version 4.14.1 which did not fix the issue but introduces a new error:

2025/11/24 12:50:26 indexer-connector: ERROR: Client error, status code: 400.
2025/11/24 12:50:27 indexer-connector: WARNING: Document operation failed for index 'wazuh-states-inventory-groups-wazuh_cluster' - type: 'json_parse_exception', reason: 'Unrecognized character escape 'a' (code 97)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 86]'
2025/11/24 12:50:27 indexer-connector: ERROR: Client error, status code: 400.
2025/11/24 12:50:29 indexer-connector: WARNING: Document operation failed for index 'wazuh-states-inventory-groups-wazuh_cluster' - type: 'json_parse_exception', reason: 'Unrecognized character escape 'a' (code 97)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 86]'
2025/11/24 12:50:29 indexer-connector: ERROR: Client error, status code: 400.
2025/11/24 12:50:30 indexer-connector: WARNING: Document operation failed for index 'wazuh-states-inventory-groups-wazuh_cluster' - type: 'json_parse_exception', reason: 'Unrecognized character escape 'a' (code 97)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 86]'
2025/11/24 12:50:30 indexer-connector: ERROR: Client error, status code: 400.
2025/11/24 12:50:32 indexer-connector: WARNING: Document operation failed for index 'wazuh-states-inventory-groups-wazuh_cluster' - type: 'json_parse_exception', reason: 'Unrecognized character escape 'a' (code 97)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 86]'
2025/11/24 12:50:32 indexer-connector: ERROR: Client error, status code: 400.
2025/11/24 12:50:34 indexer-connector: WARNING: Document operation failed for index 'wazuh-states-inventory-groups-wazuh_cluster' - type: 'json_parse_exception', reason: 'Unrecognized character escape 'a' (code 97)

I have enabled wazuh_modules.debug=2 in order to collect some additional information:

2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Event Log Readers.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Guests.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Hyper-V Administrators.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_IIS_IUSRS.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Network Configuration Operators.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_OpenSSH Users.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Performance Log Users.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Performance Monitor Users.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Power Users.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Remote Desktop Users.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Remote Management Users.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Replicator.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_System Managed Accounts Group.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_User Mode Hardware Operators.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:996 at operator()(): DEBUG: Added document for insertion with id: 983_Users.
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:1079 at operator()(): WARNING: Document operation failed for index 'wazuh-states-inventory-groups-wazuh_cluster' - type: 'json_parse_exception', reason: 'Unrecognized character escape 'a' (code 97)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 86]'
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:1129 at operator()(): ERROR: Client error, status code: 400.

Any ideas how to fix it or probably what I messed?

Kind regards,

Nikolay

[Fabian Ruiz]

Hi  Nikolay,

From the error it looks like some inventory group are being sent with invalid JSON escaping, Could you share what docummentation do you use to upgrade your environment?

Did you make sure to follow our upgrade guide? https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html

Review all the steps to verify that everything has been updated as indicated in the documentation, You could also share the indexer/manager logs to see if we can find anything else.

Check the indexer status with the following command:

curl -k -u <USER>:<PASSWORD> -X GET "https://<WAZUH_INDEXER_IP>:9200/_cluster/health?pretty"

Pleare share your results to check them.

[Commercial League]

Hi Fabian,

here is the indexer status:

{
  "cluster_name": "wazuh-indexer-cluster",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 4,
  "number_of_data_nodes": 4,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 2094,
  "active_shards": 4002,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

I have prepared document with the credentials for each node of my cluster (2 servers, 4 indexers, 1 dashboard, 1 load balancer) and I compare it with the upgrade manual. At this point I should be done the upgrade through the instructions. However it is possible for example the wazuh-manager to be offline during the filebeat upgrade but I took all the steps in the manual and I found no errors. 

I am not sure if I missed something in the previous release upgrades because the notes change so the upgrade manual.

I attach the last 5000 lines of ossec.log because it is 500MB. The cluster log is way smaller.

Thank you in advance.

[Fabian Ruiz]

Hi  Nikolay,

According to what I see in the indexer logs, there is an indexer node that does not appear to be connected to the cluster (10.20.15.12:9300). It is possible that the configuration of this node is not correct, but it may be causing the query to the /wazuh-states-inventory-groups-wazuh_cluster index to affect its performance because not all shards are available, which generates an exception in this call to the manager.

[2025-11-24T09:24:25,704][WARN ][r.suppressed             ] [wind1] path: /wazuh-states-inventory-groups-wazuh_cluster/_search, params: {scroll=1m, index=wazuh-states-inventory-groups-wazuh_cluster}
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed

Caused by: org.opensearch.transport.NodeNotConnectedException: [wind2][10.20.15.12:9300] Node not connected

[Commercial League]

Hi Fabian,

This is probably because of the update. The status is:

/_cat/nodes?v

10.20.15.13           59          98   2    0.45    0.20     0.18 dimr      data,ingest,master,remote_cluster_client -               wind3
10.20.15.12           70          98   2    0.09    0.11     0.09 dimr      data,ingest,master,remote_cluster_client -               wind2
10.20.15.14           72          96   3    0.22    0.17     0.17 dimr      data,ingest,master,remote_cluster_client -               wind4
10.20.15.11           74          98   4    0.26    0.33     0.29 dimr      data,ingest,master,remote_cluster_client *               wind1

[Commercial League]

Is it possible just to drop this wazuh-states-inventory-groups-wazuh_cluster index or empty it in order to continue working properly?

[Commercial League]

Hi,

We identified two separate issues - the first one is that the indexer open shards were above the limit but because of the 500mb log file we somehow missed it.

The second one is probably bug where the windows groups are probably not correctly escaped which causes the 

2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:1079 at operator()(): WARNING: Document operation failed for index 'wazuh-states-inventory-groups-wazuh_cluster' - type: 'json_parse_exception', reason: 'Unrecognized character escape 'a' (code 97)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 86]'
2025/11/24 12:57:14 indexer-connector[14449] indexerConnector.cpp:1129 at operator()(): ERROR: Client error, status code: 400.

 

Kind regards,

Nikolay

[Fabian Ruiz]

Hi Nikolay,

We would need to perform further tests to see what might be happening.

You can run this query to see if the return is correct:  path: /wazuh-states-inventory-groups-wazuh_cluster/_search, params: {scroll=1m, index=wazuh-states-inventory-groups-wazuh_cluster}

On the other hand, we must check connectivity with the indexer:

filebeat test output

[Commercial League]

Hi Fabian,

I attach the both results. The result from the indexer is the output of this command run in the Indexer Dev Tools

GET /wazuh-states-inventory-groups-wazuh_cluster/_search?scroll=1m

Kind regards,

Nikolay

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/x6edxTvnNDo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/wazuh/c97fbdae-cd3b-4420-bee6-9983eb96844fn%40googlegroups.com.

[Fabian Ruiz]

Hi Nikolay

You can check the status of the processes of wazuh, Based on what we see, you should have no problems indexing events.

/var/ossec/bin/wazuh-control status