Scaling ES with Data nodes on SO Master site

[Audrius J]

Hi,

I saw some topics, where some advise was to send logs to SO Master server for parsing and storing.
It may be a good idea, but if we take in to account that data can grow, we need to start increase ES data nodes on Master site.
Do you have an idea how you will deal with that, I mean join more ES data nodes to SO Master ES Cluster?

Regards,
Audrius

[Wes Lambert]

Hi Audrius,

We are still working on the best way to implement this.  As always, we welcome any feedback.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Doug Burks]

Hi Audrius,

One option would be to scale out using redis and additional cross
cluster nodes like this:

- Configure logstash on the master server to output to redis.

- Add one or more "sensors" that are really just cross cluster nodes
(no sniffing processes, just logstash and elasticsearch). Run Setup
and choose Experimental, Production, Sensor Only, Custom, Disable
everything except Salt and Elastic (store logs locally).

- Configure logstash on those cross cluster nodes with a redis input
to consume from the master server.

I did a quick proof of concept using the latest 14.04.5.7 ISO image
and everything seems to work fine.

On Wed, Jan 24, 2018 at 3:16 PM, Audrius J <aud...@gmail.com> wrote:

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

[Audrius J]

Hi Doug,

Thanks for some ideas!
If I uderstood you correctly, in that case, these additional nodes will be single cluster instantces too (like a sensor is).
I was also thinking about some additinal options.
One of them is to make a master node as ES master node and additional ES nodes as an ES Data nodes by exposing required ports to form a cluster.
In that case all nodes on master site will be just bigger ES cluster, and yuo will be able to search them (and all sensor nodes) by using kibana on SO Master.
But need to test this out...

Regards,
Audrius

[Audrius J]

So, had a weekend to work on this, but faced some issues.
And these are mainly docker related. So will ask here.
There is a way to make a cluster (on Master site) by joining additional ES Data nodes (let's imagine 2) and making of them the small central cluster with 3 nodes. The idea behind this is to make a master server just master node and not store data on it at all. In that case it will be lightweight and more resistant to a failures.
In meanwhile I can't find a way how to announce discovery.zen.ping.unicast.hosts to other ES DATA nodes...
Can you think about the way it can work? Because I think, I am missing some internals of SO.

Regards,
Audrius

[Doug Burks]

Hi Audrius,

I think you would have to change ELASTICSEARCH_PUBLISH_IP in
/etc/nsm/securityonion.conf and then add firewall rule(s) to allow the
traffic. However, please keep in mind that this is not an officially
supported solution at this time.

The redis-cross-cluster option I mentioned earlier in this thread has
been added to RC2, which is in testing now:
https://groups.google.com/d/topic/security-onion-testing/_NuzYTnN38c/discussion

Also, thanks to the detailed performance testing that you did
previously, RC2 now supports Bro logs in JSON by default.

If you'd like to help us test RC2, that'd be great. Thanks!

[Audrius J]

Hi Doug,

waw... so many improvements and you saved me the day :)
I will start downloading new image and start with deployment.
Actually, you fixed some things I noticed before, so that's great.
Addition of redis should be good idea, but also we should limit the memory usage of it. I mean , if something goes wrong with current ES storage nodes, it will start to save logs in the queue (in memory). So it can go up, until it will kill the master node.
So in my configuration I add an threshold, where redis starts to discard logs.
In current production deployment, my logs comes to logstash, and logstash outputs them to redis.
In redis output configuration I add a threshold:
It looks like this, but you need to change accordingly:

output {
redis {
host => [ "host1", "host2", "host3" ]
shuffle_hosts => true
data_type => "list"
key => "logstash"
congestion_interval => 1
congestion_threshold => 50000000
}
}

Regards,
Audrius

[Doug Burks]

Hi Audrius,

Good idea, I've added this to our RC3 list:
https://github.com/Security-Onion-Solutions/security-onion/issues/1208

Thanks!

[steve baker]

Is there anywhere else where the topic of larger scale enterprise cluster design and configuration is discussed in depth? I'm looking to learn more about this topic and only seem to be able to find bits and pieces like this in the group list.

TIA,
Steve

[Wes Lambert]

Hi Steve,

We haven't formally disseminated a  "Best Practices" architecture as we have not moved to GA yet, but I believe our recommended architecture would be using forward nodes (no Elastic components) communicating with a master server running Redis and one or more storage nodes running Elasticsearch, Logstash, and Curator, and load balancing the logs coming from the Master/Redis.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.