Disk full (help on clean up)

[Richard Giesige]

Hello,

I some updates and then noticed that my sensor stopped working around the 20th of the month. I went and did some investigation on what was using up all the disk space.

I found that the bro exe extraction is using 495Gig's of space. I have files dating back a couple of months that are in that folder.

I was wondering if there was an automatic clean up for this path?

If not is there a way to move some of the older files to the master server? I have 7TB of storage on the master and only 1 TB on the sensor.

Also I changed the keep logs on securityonion.conf from 60 to 30 days but when I ran the update from the master I didn't see it clean out the path for bro extractions.

Output of du:
449G ./bro/extracted
364K ./bro/logs/stats/www
45M ./bro/logs/stats
45M ./bro/logs
2.6G ./bro/spool/tmp/post-terminate-2015-09-12-00-00-13-4751-crash
6.0M ./bro/spool/tmp/post-terminate-2015-09-12-00-00-15-4987-crash
0 ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5156-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5156-crash
0 ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5159-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5159-crash
0 ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5157-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5157-crash
0 ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5161-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5161-crash
0 ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5166-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5166-crash
0 ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5173-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5173-crash
0 ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5170-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-12-00-00-17-5170-crash
352M ./bro/spool/tmp/post-terminate-2015-09-17-18-25-02-26341-crash
0 ./bro/spool/tmp/post-terminate-2015-09-17-18-25-04-28020-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-17-18-25-04-28020-crash
0 ./bro/spool/tmp/post-terminate-2015-09-17-18-25-04-28018-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-17-18-25-04-28018-crash
261M ./bro/spool/tmp/post-terminate-2015-09-17-18-55-02-9920-crash
0 ./bro/spool/tmp/post-terminate-2015-09-17-18-55-05-11631-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-17-18-55-05-11631-crash
0 ./bro/spool/tmp/post-terminate-2015-09-17-18-55-05-11632-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-17-18-55-05-11632-crash
95M ./bro/spool/tmp/post-terminate-2015-09-18-00-00-03-14783-crash
0 ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16457-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16457-crash
0 ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16458-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16458-crash
0 ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16461-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16461-crash
0 ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16466-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16466-crash
0 ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16468-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16468-crash
0 ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16476-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16476-crash
0 ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16482-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-18-00-00-05-16482-crash
312M ./bro/spool/tmp/post-terminate-2015-09-18-10-35-03-22798-crash
184M ./bro/spool/tmp/post-terminate-2015-09-18-12-10-02-4952-crash
325M ./bro/spool/tmp/post-terminate-2015-09-18-14-15-03-32675-crash
8.9M ./bro/spool/tmp/post-terminate-2015-09-19-00-00-03-32535-crash
0 ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1818-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1818-crash
0 ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1819-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1819-crash
0 ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1823-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1823-crash
0 ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1821-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1821-crash
0 ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1828-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1828-crash
0 ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1830-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1830-crash
0 ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1834-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-19-00-00-04-1834-crash
299M ./bro/spool/tmp/post-terminate-2015-09-19-01-30-03-15286-crash
274M ./bro/spool/tmp/post-terminate-2015-09-19-02-35-02-15103-crash
256M ./bro/spool/tmp/post-terminate-2015-09-19-03-45-03-16078-crash
266M ./bro/spool/tmp/post-terminate-2015-09-20-00-00-02-20004-crash
0 ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21595-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21595-crash
0 ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21596-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21596-crash
0 ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21600-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21600-crash
0 ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21603-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21603-crash
0 ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21610-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21610-crash
0 ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21613-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21613-crash
0 ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21617-crash/extract_files
6.0M ./bro/spool/tmp/post-terminate-2015-09-20-00-00-04-21617-crash
1.3G ./bro/spool/tmp/post-terminate-2015-09-20-00-40-02-13021-crash
6.6G ./bro/spool/tmp
40K ./bro/spool/installed-scripts-do-not-touch/site
12K ./bro/spool/installed-scripts-do-not-touch/auto
52K ./bro/spool/installed-scripts-do-not-touch
du: cannot read directory `./bro/spool/proxy/.state': Permission denied
0 ./bro/spool/proxy/.state
28K ./bro/spool/proxy
du: cannot read directory `./bro/spool/surrapp-sen-eth7-7/.state': Permission denied
0 ./bro/spool/surrapp-sen-eth7-7/.state
0 ./bro/spool/surrapp-sen-eth7-7/extract_files
28K ./bro/spool/surrapp-sen-eth7-7
du: cannot read directory `./bro/spool/manager/.state': Permission denied
0 ./bro/spool/manager/.state
0 ./bro/spool/manager
du: cannot read directory `./bro/spool/surrapp-sen-eth7-2/.state': Permission denied
0 ./bro/spool/surrapp-sen-eth7-2/.state
0 ./bro/spool/surrapp-sen-eth7-2/extract_files
28K ./bro/spool/surrapp-sen-eth7-2
du: cannot read directory `./bro/spool/surrapp-sen-eth7-1/.state': Permission denied
0 ./bro/spool/surrapp-sen-eth7-1/.state
0 ./bro/spool/surrapp-sen-eth7-1/extract_files
28K ./bro/spool/surrapp-sen-eth7-1
du: cannot read directory `./bro/spool/surrapp-sen-eth7-3/.state': Permission denied
0 ./bro/spool/surrapp-sen-eth7-3/.state
0 ./bro/spool/surrapp-sen-eth7-3/extract_files
28K ./bro/spool/surrapp-sen-eth7-3
du: cannot read directory `./bro/spool/surrapp-sen-eth7-4/.state': Permission denied
0 ./bro/spool/surrapp-sen-eth7-4/.state
0 ./bro/spool/surrapp-sen-eth7-4/extract_files
28K ./bro/spool/surrapp-sen-eth7-4
du: cannot read directory `./bro/spool/surrapp-sen-eth7-5/.state': Permission denied
0 ./bro/spool/surrapp-sen-eth7-5/.state
0 ./bro/spool/surrapp-sen-eth7-5/extract_files
28K ./bro/spool/surrapp-sen-eth7-5
du: cannot read directory `./bro/spool/surrapp-sen-eth7-6/.state': Permission denied
0 ./bro/spool/surrapp-sen-eth7-6/.state
0 ./bro/spool/surrapp-sen-eth7-6/extract_files
28K ./bro/spool/surrapp-sen-eth7-6
6.6G ./bro/spool
456G ./bro
0 ./sensor_data/surrapp-sen-eth0/dailylogs
0 ./sensor_data/surrapp-sen-eth0/portscans
0 ./sensor_data/surrapp-sen-eth0/sancp
0 ./sensor_data/surrapp-sen-eth0
0 ./sensor_data/surrapp-sen-eth1/dailylogs
0 ./sensor_data/surrapp-sen-eth1/portscans
0 ./sensor_data/surrapp-sen-eth1/sancp
0 ./sensor_data/surrapp-sen-eth1
0 ./sensor_data/surrapp-sen-eth2/dailylogs
0 ./sensor_data/surrapp-sen-eth2/portscans
0 ./sensor_data/surrapp-sen-eth2/sancp
0 ./sensor_data/surrapp-sen-eth2
0 ./sensor_data/surrapp-sen-eth3/dailylogs
0 ./sensor_data/surrapp-sen-eth3/portscans
0 ./sensor_data/surrapp-sen-eth3/sancp
0 ./sensor_data/surrapp-sen-eth3
0 ./sensor_data/surrapp-sen-eth4/dailylogs
0 ./sensor_data/surrapp-sen-eth4/portscans
0 ./sensor_data/surrapp-sen-eth4/sancp
0 ./sensor_data/surrapp-sen-eth4
0 ./sensor_data/surrapp-sen-eth5/dailylogs
0 ./sensor_data/surrapp-sen-eth5/portscans
0 ./sensor_data/surrapp-sen-eth5/sancp
0 ./sensor_data/surrapp-sen-eth5
0 ./sensor_data/surrapp-sen-eth6/dailylogs
0 ./sensor_data/surrapp-sen-eth6/portscans
0 ./sensor_data/surrapp-sen-eth6/sancp
0 ./sensor_data/surrapp-sen-eth6
0 ./sensor_data/surrapp-sen-eth7/dailylogs
0 ./sensor_data/surrapp-sen-eth7/portscans
0 ./sensor_data/surrapp-sen-eth7/sancp
0 ./sensor_data/surrapp-sen-eth7/argus
12G ./sensor_data/surrapp-sen-eth7
12G ./sensor_data
du: cannot read directory `./elsa/data/elsa/log/tmp': Permission denied
0 ./elsa/data/elsa/log/tmp
du: cannot read directory `./elsa/data/elsa/log/new': Permission denied
0 ./elsa/data/elsa/log/new
du: cannot read directory `./elsa/data/elsa/log/cur': Permission denied
0 ./elsa/data/elsa/log/cur
122M ./elsa/data/elsa/log
72M ./elsa/data/elsa/tmp/buffers
72M ./elsa/data/elsa/tmp
239G ./elsa/data/elsa/mysql
239G ./elsa/data/elsa
0 ./elsa/data/sphinx/log
226G ./elsa/data/sphinx
464G ./elsa/data
464G ./elsa
931G .

Thanks,

Rich

[Doug Burks]

Hi Rich,

Replies inline.

On Wed, Sep 23, 2015 at 7:28 PM, Richard Giesige
<gies...@oregonstate.edu> wrote:
> Hello,
>
> I some updates and then noticed that my sensor stopped working around the 20th of the month. I went and did some investigation on what was using up all the disk space.
>
> I found that the bro exe extraction is using 495Gig's of space. I have files dating back a couple of months that are in that folder.
>
> I was wondering if there was an automatic clean up for this path?

Yes, /nsm/bro/extracted/ *should* be cleaned up once your disk hits
the disk usage threshold.

What's the output of the following?
grep CRIT_DISK_USAGE /etc/nsm/securityonion.conf

> If not is there a way to move some of the older files to the master server?

Yes, you can move or remove the files in /nsm/bro/extracted/.

> I have 7TB of storage on the master and only 1 TB on the sensor.

Any particular reason why? We keep data as close to its point of
origin as possible which means that sensors need much more disk space
than a master server.

--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Richard Giesige]

Thanks Doug for replying, well I have a large distributed environment so I was hoping that we could have central storage for files and information on the master server so I didn't have to log into five different machines to get the bro-cut info. Sort of like how bro master and sensors work, where all the info is passed to the master server allowing for centralization. But if security onion is configured the opposite I will rework the environment.

Output of 'grep CRIT_DISK_USAGE /etc/nsm/securityonion.conf'
CRIT_DISK_USAGE=90

[Doug Burks]

On Thu, Sep 24, 2015 at 10:57 AM, Richard Giesige
<gies...@oregonstate.edu> wrote:
> Thanks Doug for replying, well I have a large distributed environment so I was hoping that we could have central storage for files and information on the master server so I didn't have to log into five different machines to get the bro-cut info. Sort of like how bro master and sensors work, where all the info is passed to the master server allowing for centralization. But if security onion is configured the opposite I will rework the environment.

Bro logs (as well as most other data types) stay on the sensors
themselves. If you're running ELSA, you log into the central ELSA web
interface on the master server and you can query all the Bro logs
across all sensors without having to drop to the command line and/or
use bro-cut.

[Richard Giesige]

Doug,

What's the easiest way to verify that the removal of files is happening? Is it stored in the logs?

[Doug Burks]

Take a look at /var/log/nsm/sensor-clean.log.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

[Matt .]

I had a standalone box that with 1TB of space which wasn't really enough space. For that system to function I had to set the threshold at something like 80% instead of 90% to keep the drives from filling.