Retention policy

[coriumintl]

I wanted to make sure I understood what was stored where as my company is working on tightening the retention policy.

If I set it so that pcaps only hang around for 30 days, the Sguil/ELSA/Bro data doesn't hold the network traffic just information about the alert?

[Doug Burks]

Full packet capture logs in
/nsm/sensor_data/HOSTNAME-INTERFACE/dailylogs/ are controlled by
CRIT_DISK_USAGE in /etc/nsm/securityonion.conf.

Alerts in Sguil are controlled by DAYSTOKEEP in /etc/nsm/secuityonion.conf.

Data in ELSA is controlled by log_size_limit in /etc/elsa_node.conf.

Raw Bro logs in /nsm/bro/logs/ (not Bro logs in ELSA) are controlled
by CRIT_DISK_USAGE in /etc/nsm/securityonion.conf.

On Thu, Jul 9, 2015 at 10:09 AM, coriumintl <be...@coriumintl.com> wrote:
> I wanted to make sure I understood what was stored where as my company is working on tightening the retention policy.
>
> If I set it so that pcaps only hang around for 30 days, the Sguil/ELSA/Bro data doesn't hold the network traffic just information about the alert?
>

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[coriumintl]

Given my sensor where the PCAPs are.

I'm good with just setting CRIT_DISK_USAGE since Bro, ELSA, and Sguil data are stored on my server instance then?

[Doug Burks]

Bro logs stay on the sensor.

ELSA logs stay in the ELSA database on the sensor.

Sguil data is stored in the Sguil database on the master server.

[coriumintl]

So over time I could hit a wall if I set CRIT_DISK_USAGE to less than the size of the ELSA log_size_limit then?

I guess I'm asking if I should be writing a cron job to dump folders in dailylogs that are older than 30 days, or am i going to have difficulty with massaging CRIT_DISK_USAGE since i'm more concerned with the PCAPs?

[Doug Burks]

I'm not sure I understand what you're saying.

CRIT_DISK_USAGE defaults to 90% and log_size_limit defaults to about
half your disk space. This means that, with those defaults, you're
giving (roughly) half your disk to full packet capture and (roughly)
the other half to ELSA.

[coriumintl]

I guess the log_size_limit isn't a percentage like CRIT_DISK_USAGE is.

So does that give the opportunity that setting CRIT_DISK_USAGE to a percentage such that log_size_limit is larger than which would prevent any pcaps lasting for more than a day?

[Doug Burks]

Yes, they are independent values and you need to ensure that you set
them properly according to your retention requirements.

On Fri, Jul 10, 2015 at 10:01 AM, coriumintl <be...@coriumintl.com> wrote:
> I guess the log_size_limit isn't a percentage like CRIT_DISK_USAGE is.
>
> So does that give the opportunity that setting CRIT_DISK_USAGE to a percentage such that log_size_limit is larger than which would prevent any pcaps lasting for more than a day?
>

[thegree...@gmail.com]

Reviving this old threat, I think it still applies.
What if I want to keep bro/zeek logs longer than raw PCAPs, can there be different parameters so that retention policy is different for both?
Thanks

[Steven J]

Thining outside the box, can you archive the ones you want to keep into an external storage medium?  I would think Cyberchef

 tools would still be useful.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.

[thegree...@gmail.com]

El lunes, 8 de abril de 2019, 20:55:56 (UTC-5), Steven J escribió:
> Thining outside the box, can you archive the ones you want to keep into an external storage medium?  I would think Cyberchef
>  tools would still be useful.
>
>
>
> On Mon, Apr 8, 2019 at 8:34 PM <thegree...@gmail.com> wrote:
> El viernes, 10 de julio de 2015, 9:08:56 (UTC-5), Doug Burks escribió:
>
> > Yes, they are independent values and you need to ensure that you set
>
> > them properly according to your retention requirements.
>
> >
>
> > On Fri, Jul 10, 2015 at 10:01 AM, coriumintl <be...@coriumintl.com> wrote:
>
> > > I guess the log_size_limit isn't a percentage like CRIT_DISK_USAGE is.
>
> > >
>
> > > So does that give the opportunity that setting CRIT_DISK_USAGE to a percentage such that log_size_limit is larger than which would prevent any pcaps lasting for more than a day?
>
> > >
>
> > > --
>
> > > You received this message because you are subscribed to the Google Groups "security-onion" group.
>

> > > To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

>
> > > To post to this group, send email to securit...@googlegroups.com.
>
> > > Visit this group at http://groups.google.com/group/security-onion.
>
> > > For more options, visit https://groups.google.com/d/optout.
>
> >
>
> >
>
> >
>
> > --
>
> > Doug Burks
>
> > Need Security Onion Training or Commercial Support?
>
> > http://securityonionsolutions.com
>
>
>
> Reviving this old threat, I think it still applies.
>
> What if I want to keep bro/zeek logs longer than raw PCAPs, can there be different parameters so that retention policy is different for both?
>
> Thanks
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to the Google Groups "security-onion" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.

Yes, external storge could be useful if it is available, But if no external storage is available, is there a way to apply a different retention policy to Zeek logs (which are small) and pcap (which are huge)?

[Steven J]

For Pcaps you could leverage this to get a little more mileage out of your disk space.
https://github.com/Security-Onion-Solutions/security-onion/wiki/Trimming-PCAPs

As for the Bro/Zeek logs, would the compression from archiving them after a specific number of days get you more wiggle room?

Ideally though, if your organization has a defined Data Retention policy, they will have to provision enough disk space to satisfy their requirements.

https://github.com/security-onion-solutions/security-onion/wiki/Hardware

Scroll down to Storage for a primer on space needed.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

[Doug Burks]

Hi thegreenmantiz,

Assuming that you are running Setup and choosing Best Practices, this enables the Elastic Stack and sets LOG_SIZE_LIMIT to about half of your disk space.  Roughly the other half will be used by pcap.  Since pcap requires more disk space than Bro logs, you will naturally be able to retain more days worth of Bro logs (in Elasticsearch) than pcap.  If you want to adjust that ratio after running Setup, you can adjust LOG_SIZE_LIMIT in /etc/nsm/securityonion.conf.  For more information, please see:

https://securityonion.readthedocs.io/en/latest/curator.html?highlight=log_size_limit

On Mon, Apr 8, 2019 at 8:34 PM <thegree...@gmail.com> wrote:

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.

For more options, visit https://groups.google.com/d/optout.

--

Doug Burks
CEO
Security Onion Solutions, LLC