Hard disk full for daily logs

[Jose Campos]

Hello

I was seeing that in the daily logs directory there are snor.log.date files, which are filling my hard drive.

I ran this command, du --max-depth = 1 -h / nsm / sensor_data / *, and it showed me this.

4.0K / nsm / sensor_data / cyberseg-virtual-machine-ens32 / dailylogs
4.0K / nsm / sensor_data / cyberseg-virtual-machine-ens32 / sancp
4.0K / nsm / sensor_data / cyberseg-virtual-machine-ens32 / portscans
16K / nsm / sensor_data / cyberseg-virtual-machine-ens32
4.0K / nsm / sensor_data / cyberseg-virtual-machine-ens33 / argus
8.8M / nsm / sensor_data / cyberseg-virtual-machine-ens33 / snort-1
24G / nsm / sensor_data / cyberseg-virtual-machine-ens33 / dailylogs
4.0K / nsm / sensor_data / cyberseg-virtual-machine-ens33 / sancp
4.0K / nsm / sensor_data / cyberseg-virtual-machine-ens33 / portscans

Can someone help me with this?

Regards,

[Steven J]

Hi Jose, 

I'm sure one of the seniors here will have a more beneficial answer, though I struggled with disk space for a long time and much of those were chewed up in log files.

In my case, it took some time for our client to follow their process to allocate more drive space for the instance. 
While I waited I simply compressed each log file and deleted the original, noting they are text files. Once they were older than 30 days, I move them into a separate folder for each month.  From there, as disk space was still at a premium, I moved the compressed files into an offline storage medium. 

Measuring how long you are required/desired to retain log files as compared to the space each file consumes, will help in determining how much disk space should be allocated for log file retention.  

Also, this thread may be useful.  

https://groups.google.com/forum/#!msg/security-onion/c5iE_Ni3XIQ/ncbczsWSHwcJ
Steven.

Steven Malm

Roc-Analyst I

Lyrical Security

174 Spadina Ave, Suite 400, Toronto, ON, Canada - M5T 2C2

1-855-561-4604 ext. 55
mobile: (705) 440-3339

e-mail: sjm@lyricalsecurity.com

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Steven J]

Hi Jose, 

I'm sure one of the seniors here will have a more beneficial answer, though I struggled with disk space for a long time and much of those were chewed up in log files.

In my case, it took some time for our client to follow their process to allocate more drive space for the instance. 
While I waited I simply compressed each log file and deleted the original, noting they are text files. Once they were older than 30 days, I move them into a separate folder for each month.  From there, as disk space was still at a premium, I moved the compressed files into an offline storage medium. 

Measuring how long you are required/desired to retain log files as compared to the space each file consumes, will help in determining how much disk space should be allocated for log file retention.  

Also, this thread may be useful.  

https://groups.google.com/forum/#!msg/security-onion/c5iE_Ni3XIQ/ncbczsWSHwcJ
Steven.

Steven Malm

Roc-Analyst I

Lyrical Security

174 Spadina Ave, Suite 400, Toronto, ON, Canada - M5T 2C2

1-855-561-4604 ext. 55
mobile: (705) 440-3339

e-mail: sjm@lyricalsecurity.com

On Wed, Jun 27, 2018 at 1:23 PM, Jose Campos <jca...@cyberseg.com> wrote:

[Wes Lambert]

Hi Jose,

Bro logs files and PCAP are stored in /nsm/  The snort.log.* files are your PCAP files.  These files are purged every minute by a script controlled by a cron job.  The default percentage of disk utilization at which the script purges data (from the oldest to the newest) is 90%.

PCAP data can consume large amounts of disk space.  There are ways to limit the amount recorded, such as BPF:

https://github.com/Security-Onion-Solutions/security-onion/wiki/BPF#granular-bpfconf

I would even consider adding a separate drive/array for /nsm, as described here:

https://github.com/Security-Onion-Solutions/security-onion/wiki/NewDisk

Thanks,

Wes

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Jose Campos]

Hi Wes

Thank you very much for the support to all, if that manages to see, that 90% of disk use did not happen, it eliminates them. I create a cronjob to eliminate it every minute, because it stores up to 28 or 30 logs, and eliminates the first, I think it did not help much.

By the way version 16.04 is excellent.

Regards,

[Jose Campos]

Hi Steven

Thank you very much for the support, I was considering the idea of creating a separate disk to store those logs (even if they are in binary), in the same way I do not think it affects the performance of the sensor.

Regards,

[ledin...@gmail.com]

Also, don't forget about trimpcap - this tool has been invaluable to me in being able to stretch my HDD space far beyond my needs.

Once installed, I then wrote a simple shell script to fire off once daily via cron - keeps my PCAPs from getting out of control.

[ledin...@gmail.com]

Jose - I also setup a dedicated HDD for my NSM and mySQL directories and it made a huge difference in terms of performance. My situation is somewhat different that yours I expect - I am running SO on an ESXi host and I can tell you for certain that the NSM and mySQL data definitely hammers the disk. The performance stats I gathered during testing were pretty surprising. I think this will be similar in a standalone machine scenario but may not have the same overall impact as it does when on a virtualized platform.

[Jose Campos]

Hello Ledin

If my scenario is the same as yours, I thought about creating another virtual machine to store the logs there and on the side of the NSM leave everything without low performance.

Thank you for your comments.

[ledin...@gmail.com]

Well, if I understand correctly, I would be concerned about the inter-VM communications performance. If using a high-performance Ethernet based NAS infrastructure, much less of a concern. But if just using directly attached storage from the virtualized pool, then you will probably want to scrutinize that to ensure you're not creating more bottlenecks.

That said, I'm not really sure what another dedicated VM provides. If concerned about CPU, RAM, etc., then I think you can get more bang for your buck by adding the resources you would use for the 2nd VM to your original SO VM...again, just my opinion. I'm sure others will have comments as well...

[Jose Campos]

Hello

Well, I had two scenarios, but I think it would be better to add a partition to the SO virtual machine, what do you recommend?

[ledin...@gmail.com]

Add a partition for which - All logs and PCAPs? Also, do you really mean you're adding new disk space here or a new dedicated partition on an already existing disk within the system?

If the former, that will definitely help on the storage space issue and may help with performance.

But if the latter, I'm not sure what a dedicated partition on the same disk that's already being used will provide except possibly more complexity...

[Steven J]

Hi Jose, if you can provision disk space for another VM instance, you may be better served just adding that space to what your now have.  In my case, we monitor a few hundred work stations and more than a handful of servers, 100GB was a little low, at 200GB I have one instance at 85+%, the other about 75%.

There is an upside to running in a VM, it adds an extra layer of segregation to keep your sensor inside. :-)

In the link I included in my first comment, Doug Burks shows a couple more commands you can run to show if anything else is consuming disk space, I don't know how much disk space you allocated for the instance but 24gb may not be terrible.  

At 200GB, df -h shows me running at 85% consumed, with about 42GB consumed in log files.

In securityonion.conf I have:
DAYSTOKEEP=30 

DAYSTOREPAIR=7

WARN_DISK_USAGE=80
CRIT_DISK_USAGE=90
.

In case this helps a bit.

Steven Malm

Roc-Analyst I

Lyrical Security

174 Spadina Ave, Suite 400, Toronto, ON, Canada - M5T 2C2

1-855-561-4604 ext. 55
mobile: (705) 440-3339

e-mail: sjm@lyricalsecurity.com

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.