Wazuh stopped displaying new logs

[Андрей Рыжков]

Hello, Wazuh Team!

Since yesterday, the logs in the Dashboard have ceased to be displayed. And on the server, according to `tail -1 /var/ossec/logs/alerts/alerts.json`, the logs are still coming.



The server is running Ubuntu Server 20.04. At the moment, his memory is more than 85% full.



At the same time, the Wazuh Index Manager is also configured:
`{ "id": "wazuh_index", "seqNo": 954149, "primaryTerm": 16, "policy": { "policy_id": "wazuh_index", "description": "Настройка индексов для Wazuh. «Горячие» логи будут храниться 14 дней и перейдут в состояние «теплых». «Теплые» логи будут храниться 14 дней и перейдут в состояние «холодные». «Холодные» логи будут храниться 30 дней и будут удалены", "last_updated_time": 1656480604197, "schema_version": 12, "error_notification": null, "default_state": "hot", "states": [ { "name": "hot", "actions": [ { "replica_count": { "number_of_replicas": 5 } } ], "transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "14d" } } ] }, { "name": "warm", "actions": [ { "replica_count": { "number_of_replicas": 5 } } ], "transitions": [ { "state_name": "cold", "conditions": { "min_index_age": "14d" } } ] }, { "name": "cold", "actions": [], "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "30d" } } ] }, { "name": "delete", "actions": [ { "delete": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "wazuh-statistics-*" ], "priority": 100, "last_updated_time": 1652251599303 }, { "index_patterns": [ "wazuh-monitoring-*" ], "priority": 100, "last_updated_time": 1652251599304 }, { "index_patterns": [ "wazuh-alerts-*" ], "priority": 100, "last_updated_time": 1652251599304 }, { "index_patterns": [ "security-auditlog-*" ], "priority": 100, "last_updated_time": 1652251599304 } ] } }`

Errors that show up when using grep -i error /var/log/wazuh-indexer/wazuh-cluster.log:

[2022-06-29T15:30:56,892][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.head

less=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -D

io.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2970m, -Xmx2970m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30,

-Djava.io.tmpdir=/tmp/opensearch-9378270008976337784, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:u

tctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemo

rySize=1557135360, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]

[2022-06-29T15:31:11,772][WARN ][stderr ] [node-1] java.util.ServiceConfigurationError: com.sun.tools.attach.spi.AttachProvider: Provider sun.tools.attach.AttachProviderImpl could not be instantiated

[2022-06-29T15:31:21,386][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly.

[2022-06-29T15:31:30,595][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendis

tro_security)

[2022-06-29T15:31:33,384][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] get managed-index failed: NoShardAvailableActionException[No shard available for [org.opensearch.action.get.MultiGetShardRequest@e744904]]

[2022-06-29T15:31:33,665][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to get ISM policies with templates: Failed to execute phase [query], all shards failed

[2022-06-29T15:31:34,160][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:34,403][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:34,426][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:34,430][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:36,453][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:36,459][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:36,474][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:36,478][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:38,946][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:38,956][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:38,960][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T15:31:38,963][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)

[2022-06-29T16:36:04,703][ERROR][o.o.a.t.ADTaskManager ] [node-1] Failed to update realtime task for detector m-PzuoABHZQttFqXiOHL

[2022-06-29T16:36:04,893][ERROR][o.o.a.AnomalyDetectorJobRunner] [node-1] Failed to update latest realtime task for detector m-PzuoABHZQttFqXiOHL

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [yzfXL52dQO-jUfhar0Y2DA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [hSgKt3QpRjubA2fqB2EcFA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [g-qIdiuPRdCKoix6kCZxPA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [DndQWXrNSuWUB0NvxWVKYQ#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [TcefhaYoRT-3EaVhbzhmvg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [NFxBO-8QTaCgii_rJuZ4zQ#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [KeUn6lloTSuYLRWq4ltMVA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [vdLnyHgGTnuRapMaQLjO_g#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,446][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [mC1NwILISnK5wvxbMMuUbg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [cXOFA8r8Ri2HhtY1A_1Kkg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [IX2Rvap3TpKq0nsHTWDzaw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [UL4Z1xL6SPiJZAQjOIyWdg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [OvvfnoO8RoGnaBbsW9lprg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [HTGx1L3_Qj2amyvH8PVWiw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [jjhmG2XBQoSDzXIu59QXFg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [dltHKDE8Qs6I0Abjxgh2EQ#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [gvjGKARZRJOn9FOf6owNZg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [tpGj6M_9RYeBSzlxK1Y5vA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,447][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [tUnxL9qyQfiNFuRmpaJ5Ng#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [ChvMHtdbTQmgNiUyWXeW_w#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [JCxW2k4cRRuJOhY8oj_lcA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [NmqNkrvqT7ijurxLeLv7FA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [YRR5kuIKRHKby2Xjh60liw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [zNzL3RFnR_eVIKA-5nU6bw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [RBzpIJRFS3eWdq2bqVQZEQ#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [HuHUQlCLS8CDWRYIJdl3Bw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [O3q7JfwpTJC4A_g1IMtnLw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,448][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [7TbPWtorQI2alXaZa0H_dQ#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [2k9kbJsETrGZDpMzH8IFlg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [88tH0mIWSJGw_9vSjh5c7g#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [NUKaNvH3RvSMSPhhwDZgUg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [w9AjKOPVS_SE8nBMMCFemw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [ZN-JmopaTYi_qG09O2UT7A#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [VRiuAKL1Rl6FMVvMp5FIfA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [fJGJvGMsTCCG5yibWWExvw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [FfbM61lgQFW3aBT27uLqqA#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [Eif46K7iSPOASwSBNSwvyQ#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [MSSQTfhMRcOqfDabii7q_w#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,449][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [nfZ20yoRTpGCd2wHWHo1vg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,450][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [iCo0uLzWQyWQOSFkJMaNfQ#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,450][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [pUroHLZkSuqfMIlHLhQqaw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,450][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [pOxR3fXyRGa4lBzo0yT3jg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,450][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [ppjdvOzaRjCfzL2PmEsHlw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:41:30,450][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [d5cNa_8dS9CPk1juuobuDw#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blo

cked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

[2022-06-29T16:46:04,647][ERROR][o.o.a.t.ADTaskManager ] [node-1] Failed to update realtime task for detector m-PzuoABHZQttFqXiOHL

[2022-06-29T16:46:04,670][ERROR][o.o.a.AnomalyDetectorJobRunner] [node-1] Failed to update latest realtime task for detector m-PzuoABHZQttFqXiOHL      

[Adrián Jesús Peña Rodríguez]

Hello,

After reviewing the errors you have shared, it seems that this problem is related to the machine's memory. By default, when wazuh-indexer thinks the disk is full, it goes into read-only mode.
This happens because the disk occupancy exceeds X percentage. To solve this problem the first thing to do is to either free up some space or increase the disk size.

Once this is done we will have to unlock the indexes, for this we will execute the following in the "dev-tools" of Kibana:

This will unlock the indexes for the Opensearch cluster. After freeing the space and doing the above the problem should be solved.
If you have any questions feel free to ask.

Regards.

[Renzo Geelhoed]

Hi, same problem here, dashboard stayed empty after my server ran out of diskspace. I extended my disk and now I have 200GB free. But still my dashboard displays no recent information. I ran PUT _cluster settings and the output was ok. But still no events. I run a single vm.

Error output from grep -i error /var/log/wazuh-indexer/wazuh-cluster.log:

2022-06-29T12:24:17,757][ERROR][o.o.s.a.s.InternalOpenSearchSink] [node-1] Unable to index audit log {"audit_cluster_name":"wazuh-cluster","audit_transport_headers":{"_system_index_access_allowed":"false"},"audit_node_name":"node-1","audit_trace_task_id":"JlXV7FPSRsW0L9vVT1hG8A:36651","audit_transport_request_type":"CreateIndexRequest","audit_category":"INDEX_EVENT","audit_request_origin":"REST","audit_request_body":"{}","audit_node_id":"JlXV7FPSRsW0L9vVT1hG8A","audit_request_layer":"TRANSPORT","@timestamp":"2022-06-29T10:24:17.756+00:00","audit_format_version":4,"audit_request_remote_address":"127.0.0.1","audit_request_privilege":"indices:admin/auto_create","audit_node_host_address":"10.8.10.124","audit_request_effective_user":"wazuh","audit_trace_indices":["<wazuh-alerts-4.x-{2022.06.29||/d{yyyy.MM.dd|UTC}}>"],"audit_node_host_name":"10.8.10.124"} due to

[2022-06-29T12:24:18,783][ERROR][o.o.s.a.s.InternalOpenSearchSink] [node-1] Unable to index audit log {"audit_cluster_name":"wazuh-cluster","audit_transport_headers":{"_system_index_access_allowed":"false"},"audit_node_name":"node-1","audit_trace_task_id":"JlXV7FPSRsW0L9vVT1hG8A:36703","audit_transport_request_type":"CreateIndexRequest","audit_category":"INDEX_EVENT","audit_request_origin":"REST","audit_request_body":"{}","audit_node_id":"JlXV7FPSRsW0L9vVT1hG8A","audit_request_layer":"TRANSPORT","@timestamp":"2022-06-29T10:24:18.782+00:00","audit_format_version":4,"audit_request_remote_address":"127.0.0.1","audit_request_privilege":"indices:admin/auto_create","audit_node_host_address":"10.8.10.124","audit_request_effective_user":"wazuh","audit_trace_indices":["<wazuh-alerts-4.x-{2022.06.29||/d{yyyy.MM.dd|UTC}}>"],"audit_node_host_name":"10.8.10.124"} due to

[2022-06-29T12:24:19,765][ERROR][o.o.s.a.s.InternalOpenSearchSink] [node-1] Unable to index audit log {"audit_cluster_name":"wazuh-cluster","audit_transport_headers":{"_system_index_access_allowed":"false"},"audit_node_name":"node-1","audit_trace_task_id":"JlXV7FPSRsW0L9vVT1hG8A:36751","audit_transport_request_type":"CreateIndexRequest","audit_category":"INDEX_EVENT","audit_request_origin":"REST","audit_request_body":"{}","audit_node_id":"JlXV7FPSRsW0L9vVT1hG8A","audit_request_layer":"TRANSPORT","@timestamp":"2022-06-29T10:24:19.764+00:00","audit_format_version":4,"audit_request_remote_address":"127.0.0.1","audit_request_privilege":"indices:admin/auto_create","audit_node_host_address":"10.8.10.124","audit_request_effective_user":"wazuh","audit_trace_indices":["<wazuh-alerts-4.x-{2022.06.29||/d{yyyy.MM.dd|UTC}}>"],"audit_node_host_name":"10.8.10.124"} due to

I've searched and search but I am unable to found a solution.

Can you help me out here please?

Kind regards,

Renzo

[Adrián Jesús Peña Rodríguez]

Hi Renzo,

Normally the error you are reporting is related to Opensearch having reached the default shard limit (1000). Possible solutions can be:
- Add more nodes to the Opensearch cluster.
- Delete old indexes that are no longer needed.
- Increase the shard limit.

The problem may not be the same, I suggest you to open a new thread so we can help you, this way we keep each one of the threads with only one topic and these are more focused.

In any case you can find more information here:
- https://groups.google.com/g/wazuh/c/U0MdHBfpiR8
- https://groups.google.com/g/wazuh/c/lc-NvBVAQcI
- https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster (link to Elasticsearch instead of Opensearch)

[Renzo Geelhoed]

Hi Adrián,

Thanks for your reply, you are right that I hade to open a new thread, sorry for this.

I read through de 3rd link and decided to create a quick and dirty fix so I ran curl -XPUT https://localhost:9200/_cluster/settings -H 'Content-type: application/json' --data-binary $'{"transient":{"cluster.max_shards_per_node":2000}}' -k -u admin:admin

And this solved my issue for now.

I will look into the other links as soon as I can, but for now the system is up and running.

Thanks and regards,

Renzo

[Adrián Jesús Peña Rodríguez]

No problem Renzo, don't worry about the thread, I'm glad you found it useful!!.

For any other questions feel free to open a new thread and we'll get back to you as soon as possible.

Regards,
Adrián Peña

[Андрей Рыжков]

Hi Adrián!

Thank you for your reply.

Can you please suggest the best way for me to remove the old indexes? I do not quite understand the mechanism of work, since the Wazuh Index Manager was configured for me. Or were there problems?

среда, 29 июня 2022 г. в 23:13:29 UTC+10, Adrián Jesús Peña Rodríguez:

[Андрей Рыжков]

Okay, I used the Dev Tool and applied the following

This is how I was able to remove the old indexes and free up some hard disk space:

Thank for  help

четверг, 30 июня 2022 г. в 08:36:30 UTC+10, Андрей Рыжков:

[Adrián Jesús Peña Rodríguez]

Hi!

Yes that's the way to remove indexes, I'm glad the problem is solved, if you have any other questions feel free to open a new thread.

Regards,
Adrián Peña