Disk space issue (non /nsm partition)

[Doug Lamb]

Hey all,

First of all, great class in Augusta, Doug. Thanks!

Now for the fun part. When I set up our SO box, I dedicated a huge partition to the /nsm directory, since that's where the good stuff lives. Everything's kept pruned to 90% wonderfully on that partition and we have no probs.

But after neglecting the alerts for a few days, it seems our root partition has filled up and both mysql and snort crashed due to no disk space. I have all the mysql tables repaired with myisamchk, and it starts okay, but booms right back too full disk usage:

/dev/sda5 222G 211G 0 100% /
udev 16G 4.0K 16G 1% /dev
tmpfs 6.3G 876K 6.3G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 16G 0 16G 0% /run/shm
/dev/mapper/vg0-lv0 9.1T 7.4T 1.3T 86% /nsm
overflow 1.0M 60K 964K 6% /tmp

When I search for large files on the root partition, I get these:

root@infha012:/# find . -type f -size +1000000k -exec ls -lh {} \; | awk '{ print $8 ": " $9 ": " $5 }'

17:43: ./proc/kcore: 128T
12:32: ./var/lib/mysql/securityonion_db/data_infha012@002deth1_20130604.MYD: 1009M
12:33: ./var/lib/mysql/securityonion_db/data_infha012@002deth1_20130605.MYD: 997M
12:27: ./var/lib/mysql/securityonion_db/data_infha012@002deth1_20130525.MYD: 1.5G
13:34: ./var/lib/mysql/securityonion_db/data_infha012@002deth1_20130510.MYD: 985M
12:41: ./var/lib/mysql/securityonion_db/data_infha012@002deth1_20130621.MYD: 1011M
12:30: ./var/lib/mysql/securityonion_db/data_infha012@002deth1_20130531.MYD: 1009M
05:28: ./var/lib/mysql/syslog_data/syslogs_index_1135869758.MYD: 2.4G
12:07: ./var/lib/mysql/syslog_data/syslogs_index_1296778477.MYD: 2.9G
13:17: ./var/lib/mysql/syslog_data/syslogs_index_1306782350.MYD: 4.9G
14:39: ./var/lib/mysql/syslog_data/syslogs_archive_1265091027.ARN: 11G
05:35: ./var/lib/mysql/syslog_data/syslogs_index_964386516.MYD: 2.4G
17:43: ./var/lib/mysql/ibdata1: 19G

Two questions:

1.) Is there a way to manually clear space on the root partition without negatively affecting the DB, or will I need to go in through mysql and remove data older than X days? Other ways to config/fix?

2.) Was it a mistake to put the /nsm directory on it's own partition? I'm thinking the disk space check would've caught this and taken the root partition into consideration when it runs and trims back the space.

Thanks in advance for any insight.

Doug Lamb

[Doug Lamb]

Was able to clear enough space by moving older DB files off of the root partition based on info from the sguil wiki (http://nsmwiki.org/MySQL#Archving_the_Sguil_Database):

1) Stop sguild (nsm, in our case)
2) Stop mysqld
3) cd /path/to/db/sguildb
4) rm *YYYYMMDD* OR mv *YYYYMMDD* /path/to/archive
5) rm event.* tcphdr.* udphdr.* icmphdr.* data.* sancp.*
6) start mysqld
7) start sguild (again, nsm)

This freed up enough space to be able to start mysql and nsm server/sensor processes effectively. Still pondering my second question though, although it isn't as critical at this point. Sorry for my ignorance, but not a db guru by any means.

>
> 2.) Was it a mistake to put the /nsm directory on it's own partition? I'm thinking the disk space check would've caught this and taken the root partition into consideration when it runs and trims back the space.
>

Thanks,
Doug L.

[Chris White]

Hello Doug L,

If you're still interested in number 2). It is my understanding that the focus of the pruning script is the pcaps which represent the largest set of data. The check likely checks df on the daily logs directory(ies) for the sensors in /nsm/sensor_data.

This solution works just fine if you have a distributed deployment where the vast majority of data hits /nsm, this becomes problematic when you're running a standalone instance due to the DB location on /var which is the second fastest growing data set.

I agree that a separate partition makes sense, but essentially the pruning script doesn't take into account the standalone scenario. If you wanted to salvage the current install you could try reducing your days_to_keep variable for the DB to keep its size in check, or you might symlink /var to /nsm/var.

Hope this helps,

Chris White

[Doug Lamb]

On Tuesday, August 6, 2013 12:51:46 AM UTC-4, Chris White wrote:

>
> I agree that a separate partition makes sense, but essentially the pruning script doesn't take into account the standalone scenario. If you wanted to salvage the current install you could try reducing your days_to_keep variable for the DB to keep its size in check, or you might symlink /var to /nsm/var.
>

Thanks for the insight, Chris. Y'know, the simplest answers usually go overlooked, and such is the case here. Hadn't even considered symlinking /var to the big array. Appreciate your help!

Doug L.

[Chris White]

Awesome,

I'm glad I was able to help, happy hunting.

Chris White