For those of you running SO at home... hard drive space question.

[allp...@gmail.com]

For those of you that run this at home, what is your hard drive setup?

How much space do you have?
How long does that last?
How heavy is your network use?
Is it in RAID?
What do you have on a separate partition/hard drive(s)

Network specs:

50 Mbit FIOS
Sophos UTM
Two NAS units
Two printers, rarely used.

I am a heavy user of the Internet. Both my wife and I stream music, videos and I do a lot of research. We also shop online.

I am thinking 30 days of logs minimum with 90 days maximum.
I was thinking 3 2 TB hard drives in RAID 5 for the data (/nsm?)
And two 500 GB drives for the OS (RAID 1)

Thoughts?

Thank you!

[Heine Lysemose]

Hi

Have a look here to see the calculation...https://github.com/Security-Onion-Solutions/security-onion/wiki/Hardware#storage

If you fully utilize the 50Mbit all day (24 hours). you will need to come up with about 540GB per day...

Regards,

Lysemose

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[allp...@gmail.com]

I saw that calculation, but that is not really realistic - just a maximum possible. I am trying to get sense of what other home users are actually seeing and what they have in terms of storage and network usage.

I have had SO running for about 4-5 days 24/7 and I have used 10 GB of space on the HD. This is for the entire SO install as well. 3.8 GB is in the /etc/nsm folder. (I am guessing that is where most the data is stored, but not sure.) A gig a day for 60 days would only be 60 GB. If I double that for the other PC, that still only 120 GB. If I go high, and say 5 gig a day for 60 days that is 300 GB.

I am guessing that I can easily get by with three 2 TB RAID 5 drives.

Thank you.

[Jeff]

I've been running SO at home for a few months (looks like my oldest logs go back to 23 April). My internet usage sounds similar to yours, lots of streaming music and video, work from home, etc.

I don't think I have changed any retention settings from their defaults.
According to sostat I have 78 days of full packet capture.

I am using 1.1TB total for the whole system

1.0TB /nsm/sensor_data
37.0GB /nsm/elsa
1.9GB /nsm/bro
121.5MB /nsm/server_data

1.4GB /var/lib/mysql (Sguil database)

You can probably get historical traffic data from your Sophos UTM to confirm your usage, but I'd expect that the 3x 2TB Raid 5 for /nsm (and maybe /var?) and the 500GB Raid 1 for the rest of the system should be fine for at least 30 days, and probably 90+ days.

I should add that I am only looking at perimeter traffic (to/from the internet) if you're going to be logging any internal traffic you'll have to take that into consideration.

Jeff

[allp...@gmail.com]

Thank you Jeff, that is what I am looking for. I will be pulling all traffic on my internal trusted network. There is not much beyond the normal Windows chatter for that with the exception of some light NAS use. I might exempt that IP though. I just have to figure out how.

[Doug Burks]

On Thu, Jul 9, 2015 at 9:35 PM, <allp...@gmail.com> wrote:
> Thank you Jeff, that is what I am looking for. I will be pulling all traffic on my internal trusted network. There is not much beyond the normal Windows chatter for that with the exception of some light NAS use. I might exempt that IP though. I just have to figure out how.

Have you considered using BPF to filter the traffic for that IP?
https://github.com/Security-Onion-Solutions/security-onion/wiki/BPF


--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[allp...@gmail.com]

I knew there had to be a way, but I did not know what it was yet. I will look into it. :)

Thanks again!

[wedgeshot]

I'm using a Dell Optiplex 755 w/ 8GIG RAM with a single 1TB drive. I have an Intel card installed for eth1(monitoring).

Here is some history. We are a family of four who use devices frequently. Kids like Netflix and youtube of course and I watch my friends Plex content.

/nsm/sensor_data/SO-server-eth1/dailylogs/ - 58 days
607G .
15G ./2015-05-15
15G ./2015-05-16
13G ./2015-05-17
8.7G ./2015-05-18
4.9G ./2015-05-19
5.0G ./2015-05-20
4.3G ./2015-05-21
7.7G ./2015-05-22
12G ./2015-05-23
4.6G ./2015-05-24
7.7G ./2015-05-25
986M ./2015-05-26
3.3G ./2015-05-27
2.3G ./2015-05-28
5.5G ./2015-05-29
6.1G ./2015-05-30
5.5G ./2015-05-31
7.1G ./2015-06-01
21G ./2015-06-02
7.5G ./2015-06-03
13G ./2015-06-04
6.6G ./2015-06-05
8.6G ./2015-06-06
8.7G ./2015-06-07
2.7G ./2015-06-08
9.3G ./2015-06-09
8.4G ./2015-06-10
4.5G ./2015-06-11
15G ./2015-06-12
29G ./2015-06-13
33G ./2015-06-14
5.8G ./2015-06-15
4.3G ./2015-06-16
809M ./2015-06-17
9.7G ./2015-06-18
6.1G ./2015-06-19
31G ./2015-06-20
20G ./2015-06-21
8.7G ./2015-06-22
6.0G ./2015-06-23
7.6G ./2015-06-24
8.8G ./2015-06-25
11G ./2015-06-26
8.3G ./2015-06-27
20G ./2015-06-28
22G ./2015-06-29
22G ./2015-06-30
8.0G ./2015-07-01
7.0G ./2015-07-02
11G ./2015-07-03
12G ./2015-07-04
2.6G ./2015-07-05
41G ./2015-07-06
15G ./2015-07-07
11G ./2015-07-08
8.6G ./2015-07-09
12G ./2015-07-10
2.4G ./2015-07-11

/nsm/bro/logs/ - 58 days
582M .
6.8M ./2015-05-15
7.5M ./2015-05-16
8.2M ./2015-05-17
9.0M ./2015-05-18
9.9M ./2015-05-19
9.9M ./2015-05-20
8.2M ./2015-05-21
9.3M ./2015-05-22
7.4M ./2015-05-23
5.7M ./2015-05-24
6.4M ./2015-05-25
12M ./2015-05-26
13M ./2015-05-27
13M ./2015-05-28
14M ./2015-05-29
14M ./2015-05-30
15M ./2015-05-31
14M ./2015-06-01
10M ./2015-06-02
14M ./2015-06-03
12M ./2015-06-04
9.6M ./2015-06-05
8.4M ./2015-06-06
7.9M ./2015-06-07
8.3M ./2015-06-08
9.0M ./2015-06-09
8.5M ./2015-06-10
10M ./2015-06-11
11M ./2015-06-12
13M ./2015-06-13
16M ./2015-06-14
12M ./2015-06-15
8.5M ./2015-06-16
7.0M ./2015-06-17
15M ./2015-06-18
12M ./2015-06-19
15M ./2015-06-20
12M ./2015-06-21
7.1M ./2015-06-22
8.0M ./2015-06-23
11M ./2015-06-24
11M ./2015-06-25
8.8M ./2015-06-26
7.7M ./2015-06-27
8.4M ./2015-06-28
11M ./2015-06-29
13M ./2015-06-30
7.5M ./2015-07-01
6.5M ./2015-07-02
11M ./2015-07-03
10M ./2015-07-04
8.8M ./2015-07-05
14M ./2015-07-06
13M ./2015-07-07
6.7M ./2015-07-08
7.1M ./2015-07-09
12M ./2015-07-10
1.1M ./2015-07-11
13M ./stats

[allp...@gmail.com]

Thank you Wedgeshot.

I took a look at my firewall to see how much data was actually going through the Internet interface. This is the one I plan to mirror.

My calculations (Based on 1024 and not a 1000) show that with a single 3 TB drive and an average of 150 GB a month, I could get 1.71 years of storage for NSM sensor data (Packet capture). But that includes my streaming video, and that is on a different NIC. :-) My current average for the internal NIC is about 16 GB/month, but July is not over yet.

I am going to use 25 GB for my sizing. (See attached image)

I am now thinking:

WD RED drives.
OS: Mirrored 1 TB drives (2 TB drives are $30 more ea.)
NSM: Mirrored 3 TB drives (3 TB is $26 more ea.)
13 in 3.0 Ghz range
8 GB RAM

Second option:

Same as above, no mirrored drives. (It's a home network...)

Question:

Based on everyone's experience, how do you feel about my calculations and my size choices?

* In June we got Netflix, and it shows. :-)

Wedgeshot, I show that you should be able to get about 3 months on 1 TB for just the NSM data. You seem to be right on track for that.

Thank you.

[Doug Burks]

RAID is probably overkill for a sensor on a home network (my home
sensor has a single SSD).

However, if you really need the redundancy, I'd just create a RAID1
for the entire volume.

On Fri, Jul 17, 2015 at 6:03 PM, Tri0x <gmccl...@allpcwork.com> wrote:
> In addition to my above question, how many of you run RAID for your SO install? Do you just run the whole thing on RAID or do you run the logs in RAID?
>
> Thanks!

>
> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

[Brant Hale]

I run a 1TB disk for /nsm on my 50/5 link with internet accessible servers and lots of home use 3 people.  I have about 60 days of pcaps.

single quad core intel /12 gig of ram/(2) 1TB sata disks/ 2sniffing interfaces - internet and internal traffic.

 

[Tri0x]

Thank you Brant.

Based on usage I am seeing with you and one other, I am looking at getting either a pair of 2 TB or 3 TP WD Purple drives. These allow for good read/write access at the same time.

Thank you Doug.

I will just Mirror my Purple drives.

[Michał Purzyński]

michal@nsm1:/nsm/bro/logs$ du -sh *

3.1M 2015-07-14
8.9M 2015-07-15
9.3M 2015-07-16
11M 2015-07-17
4.7M 2015-07-18
4.6M 2015-07-19
12M 2015-07-20

I have the NSM VM in VMware, with a single physical disk, 500GB. Too
lazy to enable full packet capture, but when I did I remember saving
around 100-120GB / month (I do lots of video conferencing).