Heavy Nodes vs the preferred distributed deployment

[Jake Sallee]

I'm coming back to SO after an extended hiatus in that magical place we all like to call ... no funding land.

But now I finally come home. The skies are blue, the birds are chirping, and I have been given some monies to deploy SO in my network!

So here is the deal, I am looking at the deployment options for SO and I am torn between the preferred (according to the SO website) distributed deployment and running a master + n(heavy nodes) deployment.

I have funding sufficient to deploy five beefy servers as heavy nodes and a master server running in a VM.

"beefy" im my case is:
~100TB raw storage (~50TB in RAID 10)
128GB RAM (will bump to 256 if the budget allows)
32 CPU cores (64 if I enable HT, although I'm not sure what the current conventional wisdom is about that.)

I'm hoping that will allow me to handle 1Gb/sec per server (at absolute peak, normal usage will be significantly lower)

I haven't tried to rework the design for a distributed deployment yet, but I am trying to figure out what the benefit would be over the heavy node route.

AFAICT in the distributed model the work load is split out over more servers in a slightly more economical fashion, but given my budget it seems like I would need more boxes to really get the benefits of that design.

Any guidance anyone can offer is most welcome!

PS: Please forgive me if this has already been answered, I searched the group posts but came up empty.

[Philip Robson]

I had a similar discussion with Wes today (probably a few posts down). I see that it boils down to scalability. Going with the distributed deployment you can add more storage nodes in the future if you want more log storage and to spread the load out.

Phil

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Jake Sallee]

Phil:

I did see that conversation and I think the main point is as you say, the ability to expand your log storage by just adding storage nodes.  This would allow you to keep NIDS, HIDS, OSSEC, etc longer without also increasing your PCAPs space too.

I think, after further reflection, I will use heavy nodes to begin with and expand with storage nodes and / or forward nodes as necessary.

What does everyone here think of that?

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/lLUbQMoHP-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Wes Lambert]

Jake,

Keep in mind, not only will you need to think about storage, but you'll need to think about having to manage Elasticsearch and Logstash instances for each and every heavy node, which may turn out to be more problematic for you.

Thanks,

Wes   

On Tue, Apr 17, 2018 at 4:25 PM, Jake Sallee <elcra...@gmail.com> wrote:

Phil:

I did see that conversation and I think the main point is as you say, the ability to expand your log storage by just adding storage nodes.  This would allow you to keep NIDS, HIDS, OSSEC, etc longer without also increasing your PCAPs space too.

I think, after further reflection, I will use heavy nodes to begin with and expand with storage nodes and / or forward nodes as necessary.

What does everyone here think of that?

On Tue, Apr 17, 2018 at 2:43 PM, 'Philip Robson' via security-onion <security-onion@googlegroups.com> wrote:

I had a similar discussion with Wes today (probably a few posts down). I see that it boils down to scalability. Going with the distributed deployment you can add more storage nodes in the future if you want more log storage and to spread the load out.

Phil

On Tue, 17 Apr 2018, 20:37 Jake Sallee, <elcra...@gmail.com> wrote:

I'm coming back to SO after an extended hiatus in that magical place we all like to call ... no funding land.

But now I finally come home.  The skies are blue, the birds are chirping, and I have been given some monies to deploy SO in my network!

So here is the deal, I am looking at the deployment options for SO and I am torn between the preferred (according to the SO website) distributed deployment and running a master + n(heavy nodes) deployment.

I have funding sufficient to deploy five beefy servers as heavy nodes and a master server running in a VM.

"beefy" im my case is:
~100TB raw storage (~50TB in RAID 10)
128GB RAM (will bump to 256 if the budget allows)
32 CPU cores (64 if I enable HT, although I'm not sure what the current conventional wisdom is about that.)

I'm hoping that will allow me to handle 1Gb/sec per server (at absolute peak, normal usage will be significantly lower)

I haven't tried to rework the design for a distributed deployment yet, but I am trying to figure out what the benefit would be over the heavy node route.

AFAICT in the distributed model the work load is split out over more servers in a slightly more economical fashion, but given my budget it seems like I would need more boxes to really get the benefits of that design.

Any guidance anyone can offer is most welcome!

PS:  Please forgive me if this has already been answered, I searched the group posts but came up empty.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/lLUbQMoHP-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Jake Sallee]

Wes:

I'm still learning ELK so when you say "more problematic" how problematic are we talking here?

I'm not adverse to doing the work, but I also don't want my inexperience to cause me to make a foolish decision.

If we are talking about managing shards, indices, etc then that is standard fare for managing an ELK instance. 

 If I understand correctly, if I have multiple boxes in the distributed deployment then I could save some of that management overhead by not having to manage the ELK specific items on the forward nodes (on account of them not running the ELK stack), but I would still need to manage them on the master and the storage nodes correct?

Am I missing something?  Thank you for your input BTW.  I really appreciate it.

[Wes Lambert]

Jake,

I'm more so referring to having to ensure the mappings for each indices are in line with all of the others, that each and every instance is up and running okay, resource consumption, and just anything that can go wrong.  I'm not saying you couldn't manage it, it just might be easier with a minimal amount of storage nodes vs X number of instances on sensors.

You would still need to manage them on the master and storage nodes, however it would require far less work, especially as you expand the number of sensor boxes.

Thanks,

Wes

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Jake Sallee]

Wes:

Thank you for your clarification!

I think you have convinced me. I did some soul searching (and some inventory searching) and I came up with some more hardware that I think I can use as storage nodes.

Is it permissible to post my hypothetical deployment here (in a new thread perhaps) to get some feedback on the design and hardware?

[Wes Lambert]

Jake,

Please feel free to post for feedback.  While we may not be able to provide an absolute prescription, we should be able to get you pointed in the right direction.

Thanks

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Jake Sallee]

Since we have deviated from my original question I will post my setup in a new thread so hopefully it will be more readily available to people searching in the future.

[Jake Sallee]

Here is the post:

https://groups.google.com/forum/#!topic/security-onion/3pIrhHRVjfE