ERROR index_not_found_exception

[Oleksandr Kotliarov]

Hey group.

I have a problem with my Wazuh with displaying of logs.

When I checked logs, I noticed the next lines with errors:

Sep 22, 2021 @ 10:15:00 ERROR Could not create wazuh-monitoring-2021.09.22 index on elasticsearch due to validation_exception Sep 22, 2021 @ 10:15:00 ERROR index_not_found_exception Sep 22, 2021 @ 10:20:00 ERROR Error searching or creating 'wazuh-statistics-2021.38w' due to 'validation_exception' Sep 22, 2021 @ 10:20:00 ERROR Error searching or creating 'wazuh-statistics-2021.38w' due to 'validation_exception'

I couldn't find here no hints about how to fix it, so I restored Wazuh from backup. First it worked fine, but then the same problem started again.

[Alejandro Ruiz Becerra]

Hello there. Thank yo for using Wazuh.

In order to be able to help you, could you please provide information about your environment? This includes ElasticSearch and Wazuh versions, as well as if ODFE or Xpack is being used as a security plugin.

I await for your reply.

[Oleksandr Kotliarov]

Wazuh version is 4.1.5

ElasticSearch version is 7.10.2

Speaking about OpenDistro, I have the next information:

(name - version)

opendistro-sql - 1.13.2.0

opendistro-knn - 1.13.0.0

opendistro-performance-analyzer - 1.13.0.0

opendistro_security - 1.13.1.0

opendistro-job-scheduler - 1.13.0.0

opendistro-anomaly-detection - 1.13.0.0

opendistro-reports-scheduler - 1.13.0.0

opendistro-alerting - 1.13.1.0

opendistro-index-management - 1.13.2.0

opendistro-asynchronous-search - 1.13.0.1

середа, 22 вересня 2021 р. о 12:24:14 UTC+3 alejandro.r...@wazuh.com пише:

[Alejandro Ruiz Becerra]

Hello again

After some research I think this could be a problem related with having too many shards in Kibana. To check this, take a look a the Kibana logs with this command 

journalctl -u kibana | grep -i -E "error|warn"   

and look for the following logs:

[validation_exception]: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}

This means you reached the shards limit count (1000 by default in the node). To fix this issue, there are multiple options:

• Delete indices. This frees shards. You could do it with old indices you don't want/need. Or even, you could automate it with ILM/ISM policies to delete old indices after a period of time as explained in this post: https://wazuh.com/blog/wazuh-index-management.

• Add more nodes to your Elasticserach cluster.

You can read more about Shard and how to manage them here.

However, if this is not your case, please share the Kibana logs so we can see what's happening.

[Oleksandr Kotliarov]

I tried to apply policy to indices so they changed the state from hot to cold, seems like it works some time, but then shards happened to be full again.

How can I delete some indices? Can't find this option.

пт, 24 вер. 2021 о 14:43 Alejandro Ruiz Becerra <alejandro.r...@wazuh.com> пише:

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/vygLVxjl3nE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6ac3fd68-8a60-458e-9d7a-65534e47b7fbn%40googlegroups.com.

[Oleksandr Kotliarov]

Also, is there any way to continue adequate working of Wazuh without expansion on more nodes? I have some thoughts that if I add one more node, it will be full again after some time. So that's the way, just add more nodes when it stops?

вівторок, 19 жовтня 2021 р. о 10:24:54 UTC+3 Oleksandr Kotliarov пише:

[Matias Ezequiel Moreno]

Hi, thanks you for using Wazuh,

I was reviewing the problem, to be able to delete the indexes manually you can have this guide as a reference

https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-delete-index.html

Exactly if new nodes are added to the elasticsearch cluster, the problem is likely to be fixed momentarily (we would have to check if there is some other source that is adding indexes)

Perhaps according to what this guide mentions, the policy can be modified so that the indexes are eliminated every day.

currently what is the time that was configured for the elimination of the indexes?

[Oleksandr Kotliarov]

At the moment it's configured for every 30 days, but I'll try to configure it to eliminate on daily basis.

пʼятниця, 22 жовтня 2021 р. о 20:45:20 UTC+3 matias...@wazuh.com пише:

[Oleksandr Kotliarov]

We have expanded the number of shards from 1000 to 3000 and now everything works fine.

Thank you for your help!

понеділок, 25 жовтня 2021 р. о 14:16:27 UTC+3 Oleksandr Kotliarov пише: