Best way to purge all data with a few exceptions

[Justin Pederson]

Hey all,

I have been using SO for a little while now and I have noticed a problem. It seems like there is a point in which elastic stack consumes all the hard drive space and will not delete old data out. All the indexes go in to a read only state and curator isn't deleting them. What is the best way to take care of that problem?

Also, we have decided to purge the database at this point and start over. A lot of this is due to the fact that we have been playing with this tool for a while and we want to have a clean slate to go from here with what we have implemented.

I want to keep my dashboards that I made in elastic. Also I have some configuration in the threshold document in snort config. It would also be nice to keep my firewall rules. What is the best way to go about purging all the old data, keeping what I outlined, and make sure I don't run into the space problem?

[Wes Lambert]

You'll want to ensure that you aren't exceeding the threshold defined via LOG_SIZE_LIMIT in /etc/nsm/securityonion.conf.  This settings governs at what point indices will be deleted by Curator and our closed index deletion script.

If LOG_SIZE_LIMIT is set to be greater than the watermark threshold for ES, you will end up with indices that are marked read-only by ES to prevent data corruption.

Otherwise, you may need to look at a greater amount of storage if you are exceeding the threshold value before data can be purged on the next minutely run -- either that or reducing the overall throughput.

Are you referring to purging the ES indices, or securityonion_db, or both?  For Elasticsearch, I believe you should be able to run so-elastic-reset.  For securityonion_db, use sguil-db-purge with the appropriate flags (try with -h).

You can export dashboards from the Kibana API, or from inside the GUI if needed, then re-import them through either.  Also, check out nsm_server_backup-config and nsm_sensor_backup-config for config backups.

For ufw, check out the rules in /etc/ufw/after.rules and user.rules.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/a94f6659-2376-431e-9af9-05bc15626845%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Justin Pederson]

Thanks for the reply Wes,

Would the so-elastic-reset delete the dashboards that I have made in Kibana?

Currently I can't get the so services start on this server. So I don't think I will be able to backup the dashboards that I made API or GUI.

I have edited the LOG_SIZE_LIMIT a few times. I still have space on the storage, but I keep running into curator not keeping up with with the removing the logs fast enough I think... I am not sure what the cause of the problem is, but ES keeps making the indexes read only and the whole box locks up the storage.

[Wes Lambert]

so-elastic-reset should only remove the logstash-* indices, IIRC.

Have you tried running the docker logs command to see why the container/services won't start (or have you checked the various logs)?

What percentage of disk space relative to the drive on which /nsm is mounted have you specified?

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/8def81a7-e9ac-4e0f-b62a-21505562a254%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Justin Pederson]

Hey Wes,

It has been a while. I have gotten pulled off on some different projects lately. The SO services won't start due to disk space. I had the indexes lock up once before when I was on default configuration. I re-configured the threshold on where the indexes would lock to try and fix the problem and it locked up on me again. Here is where I am ad with drive space: 2019-07justin@OnionBasket:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 173M 1.4G 11% /run
/dev/mapper/securityonion--vg-root 2.5T 2.3T 0 100% /
tmpfs 7.9G 132K 7.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda2 473M 143M 306M 32% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/1001
tmpfs 1.6G 8.0K 1.6G 1% /run/user/1000
tmpfs 1.6G 0 1.6G 0% /run/user/1002
justin@OnionBasket:~$ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 8183032 0 8183032 0% /dev
tmpfs 1642596 176812 1465784 11% /run
/dev/mapper/securityonion--vg-root 2578698504 2447982516 0 100% /
tmpfs 8212960 132 8212828 1% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 8212960 0 8212960 0% /sys/fs/cgroup
/dev/sda2 483946 145998 312963 32% /boot
tmpfs 1642596 0 1642596 0% /run/user/1001
tmpfs 1642596 8 1642588 1% /run/user/1000
tmpfs 1642596 0 1642596 0% /run/user/1002
justin@OnionBasket:~$

I am really tempted to just reinstall and start over. But I have some time into the dashboards that I built in Kibana.

Thanks in advance,
JP

[Wes Lambert]

Hi Justin,

I would also take a look in /var/ and see what kind of space is getting used there.  It may be due to the size of Wazuh logs or securityonion_db.  securityonion_db is managed by days, but not by size, and Wazuh logs can grow quite a bit if you have more than a few active/chatty agents.

Otherwise, unless you are ingesting faster than you can purge, the NSM scripts should take care of /nsm and LOG_SIZE_LIMIT should take care of the Elastic indices.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/877df5e2-b1cc-4c8f-bcf7-7440e266860d%40googlegroups.com.