Import archives folder logs

[George B]

Hello cummunity, 

I had only backup the archives folder from the logs from my old infrastructure. 

Is it impossible to make these alerts and appear on the wazuh dashboard?.

[Mariano Koremblum]

Hi!

Do you want to get to load to Kibana your archives logs only once or do you want to continue logging to such files and at the same time index it on Kibana? Any further information that you can provide to us will be helpful as well.

I will be waiting for your reply,

Mariano Koremblum

[George B]

Hello,

I had installed the wazuh in a VPS cloud for 6 months and a week ago I change VPS and wanted to transfer it all to the new VPS. So I got backed up all the configuration files. But I only got the archives folder from the logs and not the alerts. 

So I couldn't display the logs on the dashboard. 

I want to load only these logs and not continue to load such logs. 

[Mariano Koremblum]

Hi George,

Do you have access to your previous installation or such files are definitively lost?

[George B]

Hi 

all files from the old vps are lost 

[George B]

I have all the configuration files. 

Of all the old infrastructure the only thing I don't have is the all folder logs. 

I only have the archives folder from the log folder 

[Mariano Koremblum]

Can we know exactly which files did you backup? Were the archives ("logall" options) activated from the very beginning of your former Wazuh installation?

[George B]

I got the whole archives folder containing inside the following files:
2021 ( folder) 

2022 ( folder) 

archives.json 

archives.log

And I want these log files to load on kibana 

[Mariano Koremblum]

You didn't either backup the elastic files?

[George B]

What elastic files do you mean? 

[Mariano Koremblum]

Hi again George,

I am sorry for the late reply, I had to ask the team about this issue. Unfortunately, recovering the alerts from the archives is not a trivial task.

First of all, I would recommend you to backup every log file again, including the new installation ones. To do so, be sure to save the whole /var/ossec/logs/ directory.

Then, in your (old) backup archives directory, you should have such files and some subdirectories containing compressed old archives files, we will only care about the json files. To fully restore the previous alerts you should uncompress all the *.json.gz compressed files. This can be done by using the gzip command:

# gzip -dvk FileName.json.gz

(you can replace “FileName” with “*” on every directory to uncompress all the json files)

After having all the uncompressed old archives files, you should stop the manager and then dump the contents of the json files, in order, to the alerts.json file. But you should only dump the logs with an alert level equal to or higher than the alert level you have set on your manager’s ossec.conf file. To get such logs, you can use the egrep tool as follows:

# egrep "(\"level\":[3-9])|(\"level\":[1][0-6])" FileName.json

(To dump the output please use the >> operator pointing to the new alerts.json file)

After completing the alerts dumping you can start your manager again.

If you have the time and the patience, I would suggest doing this process once for every json file (stop the manager, dump alerts from a single file, start the manager, check the alerts on Kibana and repeat).

Please, let us know if you have any doubt related to this process

​

[George B]

Thank you so much for the reply 

The egrep command you wrote, it will bring me all the logs from level 3-9 or all ? 

[Mariano Koremblum]

Hi George, you are welcome!

It will bring every log from level 3 to 16 (as 3 is the default minimum alerting level, you can change it if you want higher level logs)

Let us know if you still need guidance!

Regards,

Mariano Koremblum

[George B]

Hello again! 

Thank you so much helped me

I have a problem. I have put several logs so far and it works fine but it shows an error in the dashboard  as alerts appear on wazuh dashboard  
I get the following error:

Search Error 

⚠️ Too Many Requests 

circuit_breaking_exception 

[parent] Data too large, data for [<reduce_aggs>] would be [1034333776/986.4mb, which is larger than the limit of [1020054732/972.7mb], real usage: [1034333776/986.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=9064/8.8kb, in_flight_requests=2858/2.7kb, accounting=102750816/97.9mb] 

I haven't been put all yet. This shows as the logs appear on the wazuh dashboard 

[Mariano Koremblum]

Hi George,

Where exactly are you getting that error?

I will be waiting for your reply

[Saiful Alam Shihab]

hi Mariano 

Is it possible to load archive log from json format to kibana. 

The scenary is like i need to analyse my log with kibana which is avaiable on /var/ossec/log/archive/ *.json.gz. i will unzip the folder and want to view in kibana or CSV format or in excel or any other way when i need. 

Shihab

[Mariano Koremblum]

Hi Shihab!

Sure, you can add your custom files to be indexed by Kibana. To do so, please check the following link: https://documentation.wazuh.com/current/user-manual/elasticsearch/configure-indices.html

Best regards,

Mariano Koremblum

[Saiful Alam Shihab]

Hi Mariano

from the link(https://benheater.com/tag/wazuh/) i have configured accordingly. I have another question to ask , When i create wazuh-archive-* index all of the alert and archive log found in this index. so i have to follow newly created index wazuh-archive-* for alert and archive logs? 
is there any way to find the alert/archive with specific field?

is there any way to import json format log to kibana by extratinf the ***.log.gz file.

Thanks 

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/c1782676-4a4a-48e4-a2ba-26872c543c6cn%40googlegroups.com.

--

Saiful Alam Shihab Associate Manager Security Operations Centre (SOC) Enterprise InfoSec Consultants +8801676767940 | www.eic.com.bd House-15,Road-7,Niketan,Gulshan,Dhaka-1212

[Benjamin Heater]

Hello, Saiful. Thanks for checking out my blog and I'm glad you got the wazuh-archives-* pattern created. You may have missed the part just below about de-duplicating the alerts data in both Archives and Alerts.

https://benheater.com/hunting-with-wazuh-adding-context/#de-duplicating-logs-in-elasticsearch

[Md. Sohan Prodhan]

Hello, Benjamin Heater. 

I have followed all the steps from the website but unfortunately, I am getting no logs from the Index that I have created (as shown in the screenshot). I have attached the image here. Please do give it a look.

Thanks

Md.Sohan Prodhan 

[Benjamin Heater]

Go back through the steps again and make sure you check all the spelling on index names. Your index pattern match looks wrong in Wazuh Dashboards.

Should be wazuh-archives-* , so in your case, you've got a typo and a bad pattern match.