Busted Elasticsearch fixed but missing logs

[Chad Linderman]

Greetings everyone,

I did a quick search, but I didn't find an answer. I recently took over a Wazuh environment from a user who left. 

The environment was broken due some Shard & Index issues. I am missing about 20 days of logs. I just resolved the issue with the Shard/Index issues. However, how can I get Wazuh to grab all the missing logs & events for those days?

Thank you.

[Dario Menten]

Hello Chad,
Thank you for sharing here in the community.

Yes, you can restore the Elasticsearch indices from the Wazuh archived alerts, you need to follow the steps described in this blog post: Recover your data using Wazuh alerts backups
Please follow the guide and you will be able to recover the data of the gap in alerts you have.

I hope this could be helpful.

​

[Chad Linderman]

Hello Dario,

You are amazing! I was able to get my recovery.json file. It looks like it has all of my missing data, since it is suck a large file.

However, I am having an issue with Elastic importing my recovery.json file after I restart Filebeat. I checked the logs and it says "file is inactive /tmp/recovery.json" and the alerts are still missing. 

We are using Elastic 7.10.2. When I get to the part where we modify the .yml files. The instructions for the manifest.yml file have instructions which look a little different than what we have. However, it looks pretty straightforward on adding a new json entry. I added "- /tmp/recovery.json" in the "var" section underneath "- /var/ossec/logs/alerts/alerts.json" which is under default:

var:
  - name: paths
        default:
          - /var/ossec/logs/alerts/alerts.json
          - /tmp/recovery.json

Since we're only using Elastic 7.X & a Single Architecture, I assume I only need to change the manifest.yml & restart filebeat.

I do not have a 02-recovery.conf file, which is what I would expect to only be there for Elastic 6.x. However, I do have a filebeat.yml file. 

Do I also need to change my /etc/filebeat/filebeat.yml file too? It doesn't even have any paths in it. That section only reads

filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

Do I need to set archives to enabled:true in here, or should I make it like this, which I read in another thread?

filebeat.modules:

  - module: wazuh

    alerts:

      enabled: true

      input:

        paths:

          - /var/ossec/logs/alerts/alerts.jsn

         - /tmp/recovery.json 

I'm not sure how to get around the "file is inactive" error.

Thank you for your help.

Chad

[Dario Menten]

Hello Chad,
I think you are doing the configurations correctly. And I think Filebeat is complaining that the file is inactive because you first filled it and then told Filebeat to read it, and that is not the procedure, this needs to be dynamic.
You need to set up the recovery.py and run the command, after that, you configure Filebeat to read it and restart Filebeat to then start to collect the lines from recovery.json and inject them into Elasticsearch.
Now that you have your Filebeat configured, please remove your recovery.json file, and execute again the recovery.py command, and it should start injecting the missed alerts as expected.
I hope to be helpful.

Kind Regards.

​

[Chad Linderman]

Hi Dario,

You are the man! Everything worked great!

Thanks for helping a total novice like me out.

Chad

[Dario Menten]

Hello Chad,

Glad to know it worked!

Thank you for contributing to the community.

Kind regards.