Missing events (filebeat service crash)
Franck Ehret
Hi Wazuh,
I'm contacting you again because I noticed that I'm missing data for the last 30 days:
As you can see, there are gaps in the data. The cause is the filebeat service crashing and restarting after updates. I found that about it yesterday.
So, I have several questions:
- How can I find the root cause?
- How can I resolve this issue?
- How can I re-import my missing data (I know it's possible, but I can't remember how)?
I tried an AI analysis in parallel, and apparently these logs mention the crash:
12:18
00007ff5a21ea7c0: 0000000000000000 0000000000000000
filebeat
12:18
00007ff5a21ea7b0: 0000000000000000 0000000000000000
filebeat
12:18
00007ff5a21ea7a0: 0000000000000000 0000000000000000
filebeat
12:18
stack: frame={sp:0x7ff5a21ea8a0, fp:0x0} stack=[0x7ff5a19eb1e8,0x7ff5a21eade8)
filebeat
12:18
runtime: unknown pc 0x7ff5ca68d02c
filebeat
12:18
goroutine 0 [idle]:
filebeat
12:18
PC=0x7ff5ca68d02c m=5 sigcode=18446744073709551610
filebeat
12:18
SIGABRT: abort
filebeat
12:18
runtime/cgo: pthread_create failed: Operation not permitted
filebeat
12:18
According AI, Filebeat 7.10.2 seems to be ‘outdated’ for RHEL 9, but I can't comment on that.
In any case, thank you in advance for your help! And happy holidays! 😉
Franck
PS: I tried to post yesterday, but I didn't see my post and I guess something didn't work. So if this is a duplicate, you can delete yesterday's!
Translated with DeepL.com (free version)
Stuti Gupta
Hi Franck,
Yes, you can re-ingest the missing data. First, check whether the compressed alert files are available under /var/ossec/logs/alerts/2025/. If they are present, you can restore them by following the steps and using the script provided in the Wazuh documentation here: https://documentation.wazuh.com/current/migration-guide/restoring/wazuh-central-components.html#restoring-old-logs.
If the alerts are still not showing on the dashboard after this, please share the output of the
GET _cluster/health command from Indexer Management → Dev Tools,
and also share any warnings or errors from the indexer logs using: cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn".