Disk Space Full-Wazuh3.13
siddha...@gmail.com
Julia Magan Rodriguez
Hello Siddharth,
I would recommend doing a few checks to find out what is taking up so much disk space.
First of all, please run du -sh /var/ossec to see how much disk space Wazuh is taking up.
Then, if the problem of disk space is caused by Wazuh, you should check which file is taking up disk space. Usually, those files are logs files, which are located at /var/ossec/logs. You can run the same command as before, but with the specified directory you want to check, for example, du -sh /var/ossec/logs. Old files are rotated into folders sorted by date:
/var/ossec/logs/alerts/year/month/day
/var/ossec/logs/archives/year/month/day
You can delete or move files that no longer interest you. Furthermore, when your alerts are sent to Elastic, it is not necessary to keep your logs in your manager. You can also, apply a data retention policy to remove old logs and use Opendistro IML for the elasticsearch indices, it is up to you. Here you can learn more about it: https://wazuh.com/blog/wazuh-index-management/
If the alerts file is taking up much disk space, you could run cat /var/ossec/logs/alerts/alerts.log | grep Alert | sort | cut -d '.' -f 1 | uniq -c , so you can check which alerts are repeated.
Finally, I would recommend you check Wazuh configuration in /var/ossec/etc/ossec.conf and disable options like logall and logall_json because, by default, alerts will be generated on important events or of security relevance, so with logall option enabled, you are storing all events even if they do not match a rule.
<logall>no</logall>
<logall_json>no</logall_json>
I hope I have helped you and I look forward to your response.
Best regards.
siddharth jha
--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0d5941e9-6e96-44d1-8ab1-72d59cd21308n%40googlegroups.com.
Julia Magan Rodriguez
Hello Siddharth,
When you run cat /var/ossec/logs/alerts/alerts.log | grep Alert | sort | cut -d '.' -f 1 | uniq -c you will be able to see something like this:
root@focal:/var/ossec/logs/alerts# cat alerts.log| grep Alert | sort | cut -d '.' -f 1 | uniq -c
1 ** Alert 1618558437
19 ** Alert 1618558440
57 ** Alert 1618558441
27 ** Alert 1618558442
95 ** Alert 1618558443
70 ** Alert 1618558444
1 ** Alert 1618558451
1 ** Alert 1618558462
2 ** Alert 1618559333
3 ** Alert 1618559341
Here, you are seeing how many times an alert is repeated. For example, Alert 1618558443 is repeated 95 times.
I suggest you run cat /var/ossec/logs/alerts.log | grep Rule | sort | uniq -c , so you will be able to see which rule is firing and how many times. For example:
1 Rule: 502 (level 3) -> 'Ossec server started
76 Rule: 510 (level 7) -> 'Host-based anomaly detection event (rootcheck)
1 Rule: 5403 (level 4) -> 'First time user executed sudo
3 Rule: 5501 (level 3) -> 'PAM: Login session opened
1 Rule: 5715 (level 3) -> 'sshd: authentication success
Here, Rule 510 has been fired 76 times.
All these alerts are being saved in /var/ossec/logs/alerts/alerts.log, which is the file you are searching in. If you run the same command but with /var/ossec/logs/alerts/year/month/day/ossec-alerts-day.log, you will be able to see the alerts that have been rotated.
Finally, I would recommend you to run du -ah /var/ossec/ | sort -n -r | head -n 20 so you will be able to see at once, how much the Wazuh directories occupy.
siddharth jha
--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/rzf59mnX3KU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/af9dc1ab-45f7-4231-80d6-61099385519en%40googlegroups.com.
Julia Magan Rodriguez
Hello,
I am sorry, I forgot to add the alerts folder to the path, the running command should be:
cat /var/ossec/logs/alerts/alerts.log |grep Rule | sort |uniq -c
In the information of how much space Wazuh is occupying, Wazuh is not occupying 200GB. It seems that you have Elasticsearch installed in the same server and it hasn’t enough shards. Could you check how much memory is elasticsearch taking up?du -sh /usr/share/elasticsearch
You are reaching the maximum number of shards as you can see in the next picture:
.png?part=0.1&view=1)
This problem is generating level 2 events, and those are generating alerts.
You could add nodes to the cluster or increase the maximum number of shards possible:
curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'
Also, in the output you have attached, I can see that rule 1002, the one that is generated because of the problem above, has been fired 6394457 times. This rule has level 2, so maybe, you have lowered the alert level, by default is 3. You can check it at /var/ossec/etc/ossec.conf:
<alerts>
<log_alert_level>3</log_alert_level>
</alerts>
Regards.
siddharth jha
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/42ef456a-b3aa-4be3-bf41-b0c365855342n%40googlegroups.com.
Julia Magan Rodriguez
Hello,
So that’s the folder that is taking up so much space. I recommend that you delete the logs that are stored there unless you want to save them for some reason, then I suggest you move them to another site. Before deleting the file, I recommend that you do:du -ah /var/ossec/logs/alerts/2021/Apr/
You will be able to see how much space the logs take up and the date of the logs. Then run:cat /var/ossec/logs/alerts/2021/Apr/ossec-alerts-day.log |grep Rule | sort |uniq -c
And you can check which alerts are being fired all the time.
Finally, if you want to see an alert when an event happens, for example, the firewall logs you want to check, I would recommend you increase the level of this event, instead of lowering the log_alert_level. To do this, you will have to change the rules, you can see how it’s done here: https://documentation.wazuh.com/current/learning-wazuh/replace-stock-rule.html
Regards.
siddharth jha
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/36a4ca4f-1e45-489a-9f12-d754b4bb2f9cn%40googlegroups.com.
Julia Magan Rodriguez
Hello,
Sorry if I didn’t specify it enough, but in the command that I gave you, day is in bold and blue because you have to change it to the number of the day of the log. You can see the day in the name of the log. For example, if you want to see what is in /var/ossec/logs/alerts/2021/Apr/ossec-alerts-29.log, you should run cat /var/ossec/logs/alerts/2021/Apr/ossec-alerts-29.log |grep Rule | sort |uniq -c
Regards.
siddharth jha
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b0274c86-8028-43d8-b77c-c224b7728155n%40googlegroups.com.
Julia Magan Rodriguez
Hello,
Sorry for the late response. I’ve been checking the thread and I saw that you have some log files that aren’t rotated.
First of all, I recommend you set an interval for rotating the logs, you can see how it’s done here. This will rotate and compress the logs to save disk space.
If the problem persists, you can apply:
0 6,14,22 * * * /bin/rm /var/ossec/logs/alerts/2020/* -Rf && /bin/systemctl restart wazuh-manager
More info at: https://github.com/wazuh/wazuh/issues/4425#issuecomment-577585313
Anyway, we will take a look at this problem and will come back as soon as possible.
Regards.
siddharth jha
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/dec668bf-468c-4038-a01e-78583f0b407dn%40googlegroups.com.
Julia Magan Rodriguez
Hello,
To specify the Interval you should add <rotate_interval>10h</rotate_interval>, or change the number if this line already exists in /var/ossec/etc/ossec.conf. https://documentation.wazuh.com/3.13/user-manual/reference/ossec-conf/global.html#rotate-interval
When you change analysisd.min_rotate_interval=600 in /var/ossec/etc/internal_options.conf, you are telling the minimum value of rotate_interval option, but you aren’t specifying the interval. This is done using the <rotate_interval> option. Keep in mind that /var/ossec/etc/internal_options.conf changes every time you upgrade your Wazuh version. So if you want to change this value (or any other value), and you want it to persists even if you upgrade Wazuh, you should add it to /var/ossec/etc/local_internal_options.conf.
Also, you could take a look here. This sets the size limit of alert files with a maximum allowed value of 1TiB and a minimum allowed value of 1MiB.
Answering the question of if you should write 2021 instead of 2020, yes you should, since you won’t have any 2020 log.
Finally, it is not necessary to upgrade your Wazuh version, since the problems do not depend on the version you have.
Regards.
siddharth jha
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/7a634f3a-94e0-409c-a8db-65f13dcbd9a8n%40googlegroups.com.
Julia Magan Rodriguez
Hello,
Before running 0 6,14,22 * * * /bin/rm /var/ossec/logs/alerts/2020/* -Rf && /bin/systemctl restart wazuh-manager, please run crontab -e.
For the problem with Kibana, could you please start a new thread? It seems to be a different problem and, this way, we can give it more visibility to other users.
Regards.