recovery.json not being digested

[Chad Linderman]

Greetings,

My Wazuh 4.4 server recently decided to act a little odd. I used the instructions from here to recover the missing data: https://wazuh.com/blog/recover-your-data-using-wazuh-alert-backups/

My manifest.yml files looks like the newly updated one after the addition of the recovery.json line:

____________________________________________________
module_version: 0.1

var:
  - name: paths
    default:
      - /var/ossec/logs/alerts/alerts.json
      - /tmp/recovery.json
  - name: index_prefix
    default: wazuh-alerts-4.x-

input: config/alerts.yml

ingest_pipeline: ingest/pipeline.json
_____________________________________________________

My recovery.json file has a healthy amount of data in in. So I know the recovery.py script works fine. However, the data is not populating in Wazuh and the size of the recovery.json file is not decreasing even after restarting the Filebeat service or even rebooting the entire server.

I checked the filebeat.log file and nothing stands out from what I can see. The only error I see is that there's a "ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure" which doesn't seem to be related.

Can someone please point me in the right direction as to where I should troubleshoot next?

Thank you.

[Marcos Darío Buslaiman]

Hi Chad,
Thanks for using Wazuh!, 
I'm glad to help you with this issue, I will try to replicate your scenario so I will need some information from your side.
Could you share the output of the following command?
filebeat test output
the filebeat config file /etc/filebeat/filebeat.yml
ls -l/usr/share/filebeat/module/wazuh/alerts/manifest.yml

On the other hand, I would like to know the status of your wazuh-indexer,
Are you receiving new events right now?
What is the status of your Indexer?

 curl -k -u  <username>:<password>  "https://localhost:9200/_cluster/health?pretty"

Please, prior share your data, remember to remove all your sensitive data.

Regards
Marcos

[Chad Linderman]

Hi Marcos,

The filebeat test output showed all OKs.

The Filebeat.yml file shows

The Manifest.yml files shows the following:

module_version: 0.1

var:
  - name: paths
    default:
      - /var/ossec/logs/alerts/alerts.json
      - /tmp/recovery.json
  - name: index_prefix
    default: wazuh-alerts-4.x-

input: config/alerts.yml

ingest_pipeline: ingest/pipeline.json

We are still receiving new events. So that part is working. It just seems like the recovery.json file is not being digested.

The Health shows the following:

{
"cluster_name" : "wazuh-cluster",
"status" : "yellow",
"timed_out" : false,
"number _of_nodes" : 1,
"number_of_data_nodes" : 1,
"discovered_master" : true,
"discovered_cluster_manager" true,
"active_primary_shards" : 174,
"active_shards" : 174,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 4,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 97.75280898876404
}

Thank you,

Chad 

[Marcos Darío Buslaiman]

Hi Chad,
I think that we need to verify the filebeat log by changing it to debug mode, to do that you need follow this steps.
Create a directory to add the configuration for the deamon.
mkdir /etc/systemd/system/filebeat.service.d/
Add a file (debug.conf) with the following content:
[Service]
Environment="BEAT_LOG_OPTS=-d \"*\""
Then reload the deamon config and restart filebeat.

 systemctl daemon-reload
 systemctl restart filebeat

Then you can check if your recovery.json file is being read by the filebeat process,  checking like this.
 journalctl -u filebeat --no-pager | grep recovery

For instance, this is the log when the recovery.json file has been modified by the recover script and filebeat detect that and performs the ingest to Wazuh-Indexer.
....
May 03 21:37:06 wazuh-CL1 filebeat[7022]: 2023-05-03T21:37:06.604Z        DEBUG        [input]        log/input.go:530        Update existing file for harvesting: /tmp/recovery.json, offset: 31634
May 03 21:37:06 wazuh-CL1 filebeat[7022]: 2023-05-03T21:37:06.604Z        DEBUG        [input]        log/input.go:584        File didn't change: /tmp/recovery.json
May 03 21:37:07 wazuh-CL1 filebeat[7161]: 2023-05-03T21:37:07.652Z        DEBUG        [input]        file/states.go:68        New state added for /tmp/recovery.json
May 03 21:37:07 wazuh-CL1 filebeat[7161]: 2023-05-03T21:37:07.652Z        DEBUG        [publisher]        pipeline/client.go:231        Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::67212194-2049 PrevId: Finished:true Fileinfo:<nil> Source:/tmp/recovery.json Offset:31634 Timestamp:2023-05-03 21:20:26.514675004 +0000 UTC TTL:-1ns Type:log Meta:map[] FileStateOS:67212194-2049 IdentifierName:native} TimeSeries:false}
May 03 21:37:07 wazuh-CL1 filebeat[7161]: 2023-05-03T21:37:07.652Z        INFO        log/input.go:157        Configured paths: [/var/ossec/logs/alerts/alerts.json /tmp/recovery.json]

To disable the debug mode,  you will need to remove the lines on the /etc/systemd/system/filebeat.service.d/debug.conf file, then reload and restart the filebeat service.

Please, let me know about your tests.

Regards!

[Marcos Darío Buslaiman]

Hi Chad,
I forgot to mention that another easy way to check if the file is being read by filebeat is with the lsof command.
lsof /tmp/recovery.json

For instance, executing to alert.json file
[root@localhost ~]# lsof /var/ossec/logs/alerts/alerts.json
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
filebeat  1187  root    8r   REG  253,0   203563 13568111 /var/ossec/logs/alerts/alerts.json
wazuh-int 2221 wazuh    4r   REG  253,0   203563 13568111 /var/ossec/logs/alerts/alerts.json
wazuh-ana 2331 wazuh   15w   REG  253,0   203563 13568111 /var/ossec/logs/alerts/alerts.json

[Chad Linderman]

Hi Marcos,

I am not sure if I scared the Filebeat application or what. However, everything is working properly now. Perhaps I was just impatient?

Thank you for all of your help.

[Marcos Darío Buslaiman]

Hi Chad,
That's great to hear, I'm glad you were able to fix the problem.
If you have any other questions, please don't hesitate to contact us.

I hope you have a nice weekend.

Marcos