Security Events just suddenly stopped showing up?

858 views
Skip to first unread message

Raphael Pepi

unread,
Aug 22, 2023, 7:34:06 AM8/22/23
to Wazuh mailing list
Ive been working on integrating Suricata IDS into my wazuh server/agents. despite not making any changes to the Wazuh Server config (running server 4.5.0) this morning when absolutely no one was even using the systems suddenly all security events for all agents have stopped reporting? . Does ANyone have any ideas on how to get them to start reporting again? ive looked over the ossec.logs on ther server and agents and dont see anything amiss.. Its very Bizarre

Im not even sure where to start looking

Raphael Pepi

unread,
Aug 22, 2023, 7:43:58 AM8/22/23
to Wazuh mailing list
literally all my servers look like this:
Screen Shot 2023-08-22 at 8.34.58 PM.png
No one was working on the servers or making changes to the wazuh server or agnets at 8am this morning, the've just all stopped.


Javier Medeot

unread,
Aug 22, 2023, 8:51:58 AM8/22/23
to Wazuh mailing list
Hello Raphael.

Make sure indexer and server are running

sudo systemctl status wazuh-indexer.service
sudo systemctl status wazuh-manager.service
sudo systemctl status filebeat.service
filebeat test output


Check if alerts were still generated over that period of time by looking at the /var/ossec/logs/alerts/alerts.json file. You can also check in the indexer API to see if there's an alert index for today

curl https://<WAZUH_INDEXER_IP>:9200/_cat/indices/wazuh-alerts-* -u <wazuh_indexer_user>:<wazuh_indexer_password> -k

You would see something like this

green open wazuh-alerts-4.x-2023.08.22 TSj3p71XR5qTvwNR3JicwQ 3 0 3 0  53.5kb  53.5kb
green open wazuh-alerts-4.x-2023.08.21 VjfArkNxSOK8-DBht8YnSg 3 0  480 0   1.1mb   1.1mb
green open wazuh-alerts-4.x-2023.08.20 svIJ4pKvRk2fILuMNFCxtQ 1 0 0 0 208b 208b


You can look for error logs in /var/ossec/logs/ossec.log, /var/log/filebeat/filebeat, /var/log/wazuh-indexer/

Please tell me about your findings. Also review any changes you might have made when working on integrating Suricata since you mentioned it. Thank you.
Message has been deleted

Raphael Pepi

unread,
Aug 23, 2023, 11:13:29 AM8/23/23
to Wazuh mailing list
strangely the long message i wrote with all the info didn"t post. heres the abbreviated version.

indexer appears to be functioning incorrectly. Reason wehy? unknown nothing was done to its config at all.

 heres the errors and warnings i get during  indexer startup.

[2023-08-23T15:00:32,761][WARN ][stderr                   ] [node-1] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

[2023-08-23T15:00:32,761][WARN ][stderr                   ] [node-1] SLF4J: Defaulting to no-operation (NOP) logger implementation

[2023-08-23T15:00:32,762][WARN ][stderr                   ] [node-1] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

[2023-08-23T15:00:32,787][INFO ][o.o.s.s.t.SSLConfig      ] [node-1] SSL dual mode is disabled

[2023-08-23T15:00:43,297][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.

[2023-08-23T15:00:43,298][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.

[2023-08-23T15:00:43,299][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Message routing enabled: false

[2023-08-23T15:00:45,780][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection (35 Times)

[2023-08-23T15:00:47,035][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually

[2023-08-23T15:00:50,428][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)

org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:205) ~[opensearch-2.6.0.jar:2.6.0]

at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:191) ~[opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.6.0.0.jar:2.6.0.0]

at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.6.0.0.jar:2.6.0.0]

at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.6.0.0.jar:2.6.0.0]

at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.6.0.0.jar:2.6.0.0]

at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:465) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:581) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:208) [opensearch-security-2.6.0.0.jar:2.6.0.0]

at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:99) [opensearch-security-2.6.0.0.jar:2.6.0.0]

at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.6.0.0.jar:2.6.0.0]

at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.6.0.0.jar:2.6.0.0]

at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.6.0.0.jar:2.6.0.0]

at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.6.0.0.jar:2.6.0.0]

at java.lang.Thread.run(Thread.java:833) [?:?]

[2023-08-23T15:00:51,644][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/-A-wQ7KyTsq2a488bLKUyA] already exists

[2023-08-23T15:00:52,326][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)

[2023-08-23T15:00:52,358][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)

[2023-08-23T15:00:52,364][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)

[2023-08-23T15:00:52,369][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)


The above set of 4 erros repeats about 6 times in between alot of INFO lines then seems to go away.

eventually it gets to this:

[2023-08-23T15:01:54,483][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[wazuh-alerts-4.x-2022.10.14][2], [wazuh-monitoring-2022.41w][0], [wazuh-alerts-4.x-2022.10.14][0]]]).

[2023-08-23T15:01:54,532][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration

[2023-08-23T15:02:50,375][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Cancel background move metadata process.

[2023-08-23T15:02:50,376][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Performing move cluster state metadata.

[2023-08-23T15:02:50,376][INFO ][o.o.i.i.MetadataService  ] [node-1] Move metadata has finished.

[2023-08-23T15:05:49,596][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep

[2023-08-23T15:05:50,379][INFO ][o.o.i.i.PluginVersionSweepCoordinator] [node-1] Canceling sweep ism plugin version job

^[[B^[[A[2023-08-23T15:10:49,598][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep

and thats where its sitting


Whats missing from making indexer startup error free, and why did this change all of a sudden? 

Raphael Pepi

unread,
Aug 23, 2023, 11:49:32 AM8/23/23
to Wazuh mailing list
Additional information, i decided to do a reboot to see if that resolved any issues with libraries etc.

What i found after the reboot is that wazuh-indexer had failed to startup after reboot.
When i then started it manually, it succeed after some time, and then checking my system state i found the elasticsearch.service seems to have failed.
it appears these 2 cannot co-exist peacefully. 

Additionally, i see in the sysdlog this error in wazuh-indexer startrup:

Aug 23 15:17:24 monitoring-prd systemd-entrypoint[5856]: WARNING: A terminally deprecated method in java.lang.System has been called

Aug 23 15:17:24 monitoring-prd systemd-entrypoint[5856]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/>

Aug 23 15:17:24 monitoring-prd systemd-entrypoint[5856]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch

Aug 23 15:17:24 monitoring-prd systemd-entrypoint[5856]: WARNING: System::setSecurityManager will be removed in a future release

Aug 23 15:17:26 monitoring-prd systemd-entrypoint[5856]: WARNING: A terminally deprecated method in java.lang.System has been called

Aug 23 15:17:26 monitoring-prd systemd-entrypoint[5856]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/op>

Aug 23 15:17:26 monitoring-prd systemd-entrypoint[5856]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security

Aug 23 15:17:26 monitoring-prd systemd-entrypoint[5856]: WARNING: System::setSecurityManager will be removed in a future release


and the follwoing error/warn in the indexer log:


[2023-08-23T15:18:04,026][WARN ][r.suppressed             ] [node-1] path: /.kibana/_count, params: {index=.kibana}

org.opensearch.action.search.SearchPhaseExecutionException: all shards failed

at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:663) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:698) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:471) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.6.0.jar:2.6.0]

at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.6.0.jar:2.6.0]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]

Javier Medeot

unread,
Aug 24, 2023, 10:27:11 AM8/24/23
to Wazuh | Mailing List

Hello Raphael.

Since this happened all of a sudden without any modifications from your side, maybe some limit was reached. And as there is mention of shards failing in your logs, I believe you've reached a limit for the number of indices you can handle. I can see you have indices from about a year ago. Maybe your system can't handle this any longer (unless you free some resources when Elasticsearch is down?).

I think you should implement an index retention policy to free up some space by removing indices that you no longer need. You should also review your indexer cluster size, maybe you are short of nodes for the volume of information you need to handle and need to add more nodes to the cluster.

You can try by closing or removing indices.

- https://opensearch.org/docs/2.6/api-reference/index-apis/close-index/
- https://opensearch.org/docs/2.6/api-reference/index-apis/delete-index/
- https://opensearch.org/docs/2.6/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/

And as a reference, to create an index retention policy to automatically delete old indices, you can check this old Wazuh blog post. Implementing a policy in Wazuh indexer is very similar.

- https://wazuh.com/blog/wazuh-index-management/

Please tell me if you too think this could be the issue here and any progress you make. Thank you.

Raphael Pepi

unread,
Aug 24, 2023, 10:22:43 PM8/24/23
to Wazuh | Mailing List
My indecies were indeed full and the culprit.  This gives me 2 really big questions

Why doesn't Wazuh ship with a default retention policy of 3 months (more than fair for most auditors), and/or documentation during the setup guide that explains both how to do this and the importance of it? This seems like a critically important step of setup that is severely missing. The Blogs that you sent were not so helpful because they do not show anything close to current menu structure or how to setup with a current version I kind of had to just stumble my way through it which is less than ideal. .
For current systems hers the menu path>  Left Hamburger> OpenSearch Plugins > Index Management > Indicies or: https://<your.wazuh.site>/app/opensearch_index_management_dashboards# 

Is there Something Similar for Vulnerability reporting? I ask because i have a similar issue regarding vulnerability reporting just suddenly stopping, but only for Ubuntu 18 Machines (majority of our reporting systems). 
and there didn't seem to be indicies related to vulnerability, at least not in this indicies cache. 

Thanks for a push in the right direction.

Javier Medeot

unread,
Aug 25, 2023, 9:40:38 AM8/25/23
to Wazuh | Mailing List
You are right Raphael, we need to make it easy for users. For now, I added a comment to a related docs improvement issue here: https://github.com/wazuh/wazuh-documentation/issues/6156#issuecomment-1693357495

And about vulnerabilities reporting, what do you mean by reporting? Do you mean you stopped seeing Wazuh alerts about vulnerabilities found in your Ubuntu 18 systems? Can you give an example? There a different types of vulnerability scans which lead to creating vulnerability alerts which in turn, as you know, end up being indexed. Please share some details about your vulnerability detection configuration and what you were expecting to be informed about to see what could be the problem here.

Thank you.

Raphael Pepi

unread,
Aug 26, 2023, 3:10:10 AM8/26/23
to Wazuh | Mailing List
The vulnerability scanner keeps running at the specified times, but the events arent visible in the events screen or dashboard.

This looks eerily similar to the out of indexes issue i just solved, but having removed 80% of the index (everything older than 3 months) the vulnerability scanner hasn't started displaying valuers again, which makes me think the indexes are stored elsewhere, or that there is some other resource that is locked related to ubuntu 18. I would like to clean whatever resource that is locked/full now for these so that I get vuln status for my servers agin.


Quite Simply, All of my ubuntu 18 servers are showing this type of result on their Vulnerabilities screen:
Screen Shot 2023-08-26 at 1.35.17 PM.png


Please See my thread "Wazuh Vulnerability Scanner suddenly stopped reporting results" for additional details

Javier Medeot

unread,
Aug 28, 2023, 8:36:25 AM8/28/23
to Wazuh | Mailing List
Hello Raphael.

Since June 2023, Ubuntu 18.04 reached the end of standard support. When this happens, the vulnerabilities feed inform about this issue as we've found and you can read in this comment. We're working on improvements for the vulnerability detector module for a future release. As soon as this release is out, false negatives in operating systems with no maintenance such as recently with Ubuntu 18 will be addressed.

I hope this helps toclarify the issue. Tell me if there's anything else you need to know about this. Thank you.
Reply all
Reply to author
Forward
0 new messages