wazuh-csyslogd ERROR date or location not NULL or p is NULL

[Alex]

Hi,

We encountered the error in the image below. Yesterday, we discovered that the max shard limit was reached and no more shards were being produced. We increased the shard count using the code below, but the "wazuh-csyslogd ERROR date or location not NULL or p is NULL" error still persists. What is the solution to this?


curl -u admin:SecretPassword -X PUT https://localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "5000" } }' -k

[Olusegun Adenrele Oyebo]

Hello Alex,

The error "date or location, not NULL or p is NULL" seems to be different from the issue of the shard limit exceeded. It error seems to be related to a program (csyslogd) that forwards alerts via syslog. Do you have any syslog configuration in your Wazuh manager's ossec.conf file? If yes,  can you check if there are any network restrictions or firewalls preventing logs from reaching the destination? Also, ensure that the necessary ports (514 in this example) are open for communication.

If your shard limit has been exceeded, kindly note that increasing the shard limit is not what we recommend as it could provoke inoperability and performance issues in your Elasticsearch/Wazuh indexer cluster. I can also see that you increased it to as much as 5000 which seems to be too much.

One thing we can recommend is deleting old indices. To delete old indices that are not needed, you can first check the indices stored by running the below command:

• curl -k -u admin:<admin_password> -XGET "https://<ip_address>:9200/_cat/indices?v" replace <admin_password> with the password of the user admin and also <ip_address> with the IP address of the wazuh indexer.

You can now use the below command to delete the old indexes:

• curl -k -u admin:<admin_password> -XDELETE "https://<ip_address>:9200/<index_to_delete>". Replace <admin_password> with the password of the user admin and also <ip_address> with the IP address of the wazuh indexer and <index_to_delete> with the index to delete e.g. curl -k -u admin:Test123 -XDELETE "https://192.168.227.139:9200/wazuh-alerts-4.x-2023.08.20"

If you want to delete indices for a whole month, you can use the wildcard (*). For example, you want to delete for the whole of September:

• curl -k -u admin:Test123 -XDELETE "https://192.168.227.139:9200/wazuh-alerts-4.x-2023.09*"

You can also check the below link on indexer lifecycle management which will help you to automate the process going forward:

• https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.html

Another thing we can recommend in the long run is adding more nodes to your cluster:

• https://documentation.wazuh.com/current/user-manual/wazuh-indexer-cluster/add-wazuh-indexer-nodes.html

If the issue persists, kindly share the full output of the below command (reference):

• cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

Let me know if you still need further assistance with this.

Best regards.