Index policy "Still initializing, please wait a moment"

[meganie]

Hello, I've created a state management policy for my indices:

{ "policy": { "policy_id": "wazuh-index-state-policy", "description": "Wazuh index state management for OpenDistro to move indices into a cold state after 30 days and delete them after 70 days.", "last_updated_time": 1695565951406, "schema_version": 17, "error_notification": null, "default_state": "hot", "states": [ { "name": "hot", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "replica_count": { "number_of_replicas": 1 } } ], "transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "30d" } } ] }, { "name": "warm", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "replica_count": { "number_of_replicas": 0 } } ], "transitions": [ { "state_name": "cold", "conditions": { "min_index_age": "50d" } } ] }, { "name": "cold", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "read_only": {} } ], "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "70d" } } ] }, { "name": "delete", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "delete": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "wazuh-alerts*" ], "priority": 100, "last_updated_time": 1679387560421 } ] } }

It gets applied automatically.

But the state of the indices never changes and it just shows "Still initializing, please wait a moment"

[Md. Nazmur Sakib]

Hi Meganie,

Hope you are doing well. Thank you for using Wazuh.

Can you answer the following questions so that I can have a better understanding of your problem?

This is happening for every managed index? 

None get passed Initializing?

How many managed indices are you running? 

Cluster is in green health?

If you’re using ODFE(Open Distro for Elasticsearch) you can at least change the job interval time for quicker testing to every 1 minute in the cluster settings. Once that’s done try applying a test policy to an index so it creates a managed index job and confirm the following:

Get the document ID of the internal managed index, you can _search the .opendistro-ism-config and find it.

Then confirm you see a log on one of the nodes that says it’s been scheduled that looks like:

“Scheduling job id for index .”

Then wait 1 minute for it to run. If it does actually run, you should see a newly created document in the lock index: .opendistro-job-scheduler-lock which should have the index name and job id.

If you can find that then it confirms it's at least running and if it still is always stuck then it means it’s failing on something in between somehow.

Also note that, if an index with health in red exists, this prevents the indexing from initializing. It doesn’t matter if you try to apply policy to the “red” index or not.

Check this document to learn more about Index State Management:

https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index-lifecycle-error-handling.html

I hope this helps. Please let me know if you need any further information.

Regards

Md. Nazmur Sakib

[meganie]

This is happening for every managed index? yes

None get passed Initializing? no

How many managed indices are you running? 71 at the moment

Cluster is in green health? yes

I don't even recall installing OpenDistro. I've just followed the step-by-step guides to install the cluster:

https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html

https://documentation.wazuh.com/current/installation-guide/wazuh-server/step-by-step.html

https://documentation.wazuh.com/current/installation-guide/wazuh-dashboard/step-by-step.html

[Md. Nazmur Sakib]

Hi Meganie,

I hope you are doing well. Sorry for the late reply.

I was looking at your problem. As per my findings, this issue happens if there are unassigned shards. I was unable to recreate the issue with all assigned shades. If there are unassigned shards you could re-assign or simply delete that index to resolve this issue. But I can see from your screenshot there are no unassigned shades.

You can try restarting the service to initialize the saved objects again. This will initialize the index with the required mappings. Check after restarting the dashboard.

systemctl restart wazuh-dashboard

I hope this helps.

Regards

Md. Nazmur Sakib

[meganie]

Hello, thanks for your reply.

I've already restarted the dashboard, other server components or the complete server. But it didn't change.

It actually never worked since I've installed the cluster a couple of months ago. But now the drives are filling and I've actually noticed it. I've also already updated the server a couple of times after the initial install.

Also new indices were created after my first post without a change.

[Md. Nazmur Sakib]

Hi Meganie,

Is the index policy working for your newly created indices?  or it is also stuck in the initialing stage for new indices?

Regards

Md. Nazmur Sakib

[meganie]

No, it's also not working for newly created indices as you can see in the screenshots of my last message. Also stuck in the initial stage.

[meganie]

I've noticed another problem regarding indexing: I've added the "azure-logs" wodle yesterday and the new fields just show: "Unindexed fields can not be searched"