Hardware Suggestions

[Ronald Swanson]

I'm on a 10Gbps link and was wondering what hardware would the community recommend that I purchase if I were planning a deployment that would save packet captures up to 30 days. I would also like to keep Alert data for up to 180 days.

Any suggestions are much appreciated.

[Michal Purzynski]

On 8/1/13 11:25 PM, Ronald Swanson wrote:
> I'm on a 10Gbps link and was wondering what hardware would the community recommend that I purchase if I were planning a deployment that would save packet captures up to 30 days. I would also like to keep Alert data for up to 180 days.
>
> Any suggestions are much appreciated.
>

1. You need to measure your real transfer. Netflow data is _very_ useful
for it, and many other things too, so if you are not collecting it
already, start doing it. Google://nfsen

2. Think about your network design. What do you want to see? What do you
want to store. Note - it might be a bit different (it's here).

3. I don't see any value of storing encrypted traffic - this includes
HTTPS, ESP, SSH. I might see it useful to feed Snort and Bro with it,
just not pcap it :) 10Gbit is 1250MB/sec. Over 105TB / day. I bet your
real traffic varies depending on a time, day, and is much less. Unless
it's not ;)

4. Do you use load balancers? If you terminate HTTPS on them, you can
feed the "backend" traffic to the NSM and have it in clear. That's what
you want to sent to Bro+Snort+Netsniff-ng.

5. Get a real capture cards. I've seen Myricom doing over 630Mbit/sec
per core here with no packet loss at all, for Snort. It all depends on
the traffic and enabled rules - I've seen it doing 80Mbit/sec with loss,
too :) Snort can use Myricom via a vendor provided libpcap. Bro has
support, too (might need to use the version from git). Plus, Myricom +
Sniffer 10G is actually cheap. And if you don't want to - than get Intel
x520 for pf_ring.

6. If you get Myricom (or other capture cards, or pf_ring with DNA
enabled) you need to split sensors to multiple physical host. Hardware
packet capture can service a single application only. So one sensor will
do Bro, another Snort, yet another packet capture. That's what I'm doing
now.

7. Forget about Prads and Argus. They are single thread applications,
dying not much above 100Mbit/sec with no way to split the load across
multiple cores. You don't need them - Bro has all you need, and even
more. Including session data. It's a Security Onion specialty, that you
can pivot from Bro logs to full capture transcript :-)

8. Can't tell you how much of CPU power you need. It's too traffic and
rules dependent. Googling, you will find lots of people throwing pretty
much random numbers, like "Snort can do 100Mbit/sec per core". Or 80. Or
250. It's all true and in the same time - for them, in the specific
environment, with some ruleset that's different than yours.

9. Disable http_agent, use Bro for it. It stores HTTP(s) (meta)data by
default. And much more. Isn't Bro just great? Oh, and I forgot - it
scales very well.

10. While you separate packet capture, think about the box performance
and RAID groups. And the time to rebuild a very large RAID. And MTBF.
And the RAID type. Controller throughput and overbooking. 8 x 1TB SATA
7.2k disks in RAID5 can do over 650MB/sec of constant write here. Of
course your number will be different, but I would not exceed more than
11 disks per RAID5 group + 1 spare. Or 12 in RAID10 + 1 spare.

11. Separate sensor on yet another host. How much disk space do you need
there? I've no idea, trying to measure it myself, on a multiple
10Gbit/sec networks, in a few data centers.

12. Ask. Think. Design. Ask again. There are people with a lot of
traffic here. Like me ;)

[Ronald Swanson]

On Thursday, August 1, 2013 9:25:30 PM UTC, Ronald Swanson wrote:
> I'm on a 10Gbps link and was wondering what hardware would the community recommend that I purchase if I were planning a deployment that would save packet captures up to 30 days. I would also like to keep Alert data for up to 180 days.
>
> Any suggestions are much appreciated.

Wow thanks for the detailed response. Really great information.

Your point on capturing encrypted data is spot on, I disabled captures on that traffic long ago.

The NIC info is great, I'll look into Myricom.

7. When you say forget about PRADs and Argus, I assume you mean disable them?
Currently I've disabled Bro because it was using a lot of resources on my sensor and I was not using it. I'll definitely take your advice on this and check this out.

8. Haha yes that is exactly what I came across!! I agree, processing power will depend on rule sets. I am currently trimming and editing where I can to drop the MATCH% shown in Sguil. Is there a recommended % I should be aiming for?

9. OK.

I really appreciate the advice!

[Michal Purzynski]

On 8/2/13 3:42 PM, Ronald Swanson wrote:
> On Thursday, August 1, 2013 9:25:30 PM UTC, Ronald Swanson wrote:
>> I'm on a 10Gbps link and was wondering what hardware would the community recommend that I purchase if I were planning a deployment that would save packet captures up to 30 days. I would also like to keep Alert data for up to 180 days.
>>
>> Any suggestions are much appreciated.
> Wow thanks for the detailed response. Really great information.
>
> Your point on capturing encrypted data is spot on, I disabled captures on that traffic long ago.
>
> The NIC info is great, I'll look into Myricom.
>
> 7. When you say forget about PRADs and Argus, I assume you mean disable them?
> Currently I've disabled Bro because it was using a lot of resources on my sensor and I was not using it. I'll definitely take your advice on this and check this out.

Than better invest into having Bro running, it can replace Argus + Prads
in no time and that's just the beginning it can do. And it scales, but
agreed - needs a lot of power. Worth every Ghz, though.

>
> 8. Haha yes that is exactly what I came across!! I agree, processing power will depend on rule sets. I am currently trimming and editing where I can to drop the MATCH% shown in Sguil. Is there a recommended % I should be aiming for?

I used to have almost 200% and that was bad... I'd say less than 100%?
There are some Snort people here, they might know better.

That being said I just disabled all the rules and I'm enabling one after
another, sometimes using categories. I wish Pulledport had a three pass
rule set management, so I could disable all, enable categories and
disable single offending rules. Like the one telling me about all HTTP
404 as ... an indicator of compromise. Really... guys...

> 9. OK.
>
> I really appreciate the advice!
>

You're welcome, the more people with a huge traffic in here, the more
interesting feedback and ideas around.