ELK replacing ELSA?

[Kris Springer]

I've been watching the ELK blog and reading some of the posts and watching Doug's video demos, and it looks like a great step forward. It seems to be assumed that everyone already knows what ELK is, but since it's new to me I'll just ask the question that I haven't seen any mention of.

Is ELK replacing ELSA, or just adding more tools to the existing SO system? Also, will this be affecting the Squert interface? Does ELK have anything to do with reading the Sguil database, or only the ELSA database?

[Kevin Branch]

In short, Elastic Stack (formerly called ELK) is eventually replacing ELSA.  Presently syslog-ng feeds log data to ELSA and you use the ELSA web interface to mine that log data.  In the Tech Preview, syslog-ng sends the log data instead to Logstash which parses out relevant fields, does some enrichment, and then feeds it to Elasticsearch which indexes and stores the log data.  Then you use the Kibana web interface to mine the data with the help of very cool dashboards.  Sguil does not change in all of this.  It still feeds off of the sguil mysql database which is not touched by any of the Elastic Stack stuff.  

I found this diagram to be helpful in getting my head wrapped around the Tech Preview stuff:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Elastic-Architecture

Kevin

On Thu, Aug 10, 2017 at 11:05 AM, Kris Springer <kspr...@innovateteam.com> wrote:

I've been watching the ELK blog and reading some of the posts and watching Doug's video demos, and it looks like a great step forward.  It seems to be assumed that everyone already knows what ELK is, but since it's new to me I'll just ask the question that I haven't seen any mention of.

Is ELK replacing ELSA, or just adding more tools to the existing SO system?  Also, will this be affecting the Squert interface?  Does ELK have anything to do with reading the Sguil database, or only the ELSA database?

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Kris Springer]

Excellent explanation. I thought that was the case. Would it be safe to assume the attached diagram is the future architecture once there's a stable release?

[Wes]

On Thursday, August 10, 2017 at 1:52:28 PM UTC-4, Kris Springer wrote:
> Excellent explanation. I thought that was the case. Would it be safe to assume the attached diagram is the future architecture once there's a stable release?

Kris,

The attached diagram would be conceptually similar to what is planned in regard to replacement of ELSA.

Thanks,
Wes

[Kevin Branch]

The one adjustment I would make to that diagram is to show syslog-ng's role:

• Barnyard sends events directly to syslog-ng which then passes them on as json input to Logstash.
  
• Bro spools its logs to local files that syslog-ng tails and then passes on as json input to Logstash.
  
• OSSEC spools raw log events to the archives.log file that syslog-ng tails and then passes on as json input to Logstash.
  
• OSSEC sends alert events directly to syslog-ng which then passes them on as json input to Logstash.
• Syslog-ng also passes as json input to Logstash, pretty much all locally generated syslog messages as well as all syslog messages sent to the server by remote devices.

Regards,

Kevin

On Thu, Aug 10, 2017 at 1:52 PM, Kris Springer <kspr...@innovateteam.com> wrote:

Excellent explanation.  I thought that was the case.  Would it be safe to assume the attached diagram is the future architecture once there's a stable release?

[Kris Springer]

Kevin, are you saying that's how it works now because of the Docker containers? Or are you saying that's the future stable release functionality?

In short: Will syslog-ng eventually be replaced by Logstash?

[Kevin Branch]

That is how Tech Preview 3 works.  Whether or not that is projected to stay that way, Wes or Doug would be more qualified to address.  I'm actually curious if we are considering sticking with Docker as a part of SO for the long term.  I'm starting to get to like it.  

It does strike me that for certain log streams, syslog-ng would not need to remain part of the pipeline.  Maybe that will be approached in future Tech Preview releases.  For example the OSSEC alerts.log and archives.log data could all be passed along as a single json stream for direct Logstash consumption with lots of field decoding already in place, if SO were to upgrade from OSSEC to Wazuh HIDS which supports that.  Also, I believe Logstash could read the Bro log files directly.  At least I've experimented with that before and don't recall any trouble.   Banyard (Snort/Suricata alerts) could also be directly read from the sguild log by Logstash with a one-liner tweak in one of the sguild scripts.  

I think right now everything goes through syslog-ng because that is how it needs to be in the ELSA/OSSEC environment of production SO.  The data pipelines may shift more later.

Kevin

On Thu, Aug 10, 2017 at 2:49 PM, Kris Springer <kspr...@innovateteam.com> wrote:

Kevin, are you saying that's how it works now because of the Docker containers? Or are you saying that's the future stable release functionality?

In short: Will syslog-ng eventually be replaced by Logstash?

[Doug Burks]

Hi Kevin,

Replies inline.

On Thu, Aug 10, 2017 at 3:24 PM, Kevin Branch
<ke...@branchnetconsulting.com> wrote:
> That is how Tech Preview 3 works. Whether or not that is projected to stay
> that way, Wes or Doug would be more qualified to address. I'm actually
> curious if we are considering sticking with Docker as a part of SO for the
> long term. I'm starting to get to like it.

Yes, Docker will continue to be the distribution method for our
Elastic stack integration even after we reach full release. At that
point, we may decide to start moving other components into their own
Docker images...

> It does strike me that for certain log streams, syslog-ng would not need to
> remain part of the pipeline. Maybe that will be approached in future Tech
> Preview releases. For example the OSSEC alerts.log and archives.log data
> could all be passed along as a single json stream for direct Logstash
> consumption with lots of field decoding already in place, if SO were to
> upgrade from OSSEC to Wazuh HIDS which supports that. Also, I believe
> Logstash could read the Bro log files directly. At least I've experimented
> with that before and don't recall any trouble. Banyard (Snort/Suricata
> alerts) could also be directly read from the sguild log by Logstash with a
> one-liner tweak in one of the sguild scripts.
>
> I think right now everything goes through syslog-ng because that is how it
> needs to be in the ELSA/OSSEC environment of production SO. The data
> pipelines may shift more later.

Yes, the current pipeline in TP3 was really the path of least
resistance of taking our existing architecture and integrating the
Elastic stack. It may change in the future. However, one thing that
I'm keeping in mind is that some folks may want to run stripped down
sensors without logstash and so it may make sense to have an option
where a sensor can just run our existing syslog-ng to transport the
logs over to a separate host running Logstash for ingestion into
Elasticsearch.

Interested in any feedback!

> Kevin
>
> On Thu, Aug 10, 2017 at 2:49 PM, Kris Springer <kspr...@innovateteam.com>
> wrote:
>>
>> Kevin, are you saying that's how it works now because of the Docker
>> containers? Or are you saying that's the future stable release
>> functionality?
>>
>> In short: Will syslog-ng eventually be replaced by Logstash?
>>
>> --
>> Follow Security Onion on Twitter!
>> https://twitter.com/securityonion
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "security-onion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to security-onio...@googlegroups.com.
>> To post to this group, send email to securit...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

[Brant Hale]

I like the use of docker for the components.   I think this will give some flexibility and options for the future as well as giving us some control over resource utilization.  

I am interested in how the architecture with master servers and sensors will translate to the new platform.   Is a 3+ node elastic stack cluster for redundancy and HA  in the future? I am guessing the mysql for sguil would play into this and complicate it.

Good work Doug!  It is exciting to see Security Onion evolve. 

Brant

 

>> email to security-onion+unsubscribe@googlegroups.com.
>> To post to this group, send email to security-onion@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Kevin Branch]

Hi Doug,

Good point about not all sensors being in the best position to handle the CPU load of logstash.   Syslog-ng could certainly provide transport of logs from an SO sensor over to an SO server's logstash instance.  You might also keep your eye on Elastic's Filebeat.  It is lightweight and could monitor multiple log files on an SO sensor and stream the log lines to logstash on an SO server.  It can even do filtering and some light processing before handing the log data over to logstash for the fancy stuff.  I use it on all my clients' standalone SO systems to stream snort events extracted from sguild.log, to logstash on my cloud server where I have a Kibana master NIDS dashboard for high level review of snort events across my entire client base.  It has been hassle free for me, even with it streaming across OpenVPN tunnels.

Kevin

On Fri, Aug 11, 2017 at 6:48 AM, Doug Burks <doug....@gmail.com> wrote:

>> email to security-onion+unsubscribe@googlegroups.com.
>> To post to this group, send email to security-onion@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Doug Burks]

Hi Brant,

Replies inline.

On Fri, Aug 11, 2017 at 11:38 AM, Brant Hale <bran...@gmail.com> wrote:
> I like the use of docker for the components. I think this will give some
> flexibility and options for the future as well as giving us some control
> over resource utilization.
>
> I am interested in how the architecture with master servers and sensors will
> translate to the new platform. Is a 3+ node elastic stack cluster for
> redundancy and HA in the future?

I can't make any promises right now, but I agree it would be nice to
have that kind of flexibility.

> I am guessing the mysql for sguil would
> play into this and complicate it.

In theory, I could see a master server running sguild and mysql (as
we've been doing) and Kibana, with that Kibana instance querying
Elasticsearch instance(s) which may live on other hosts.

> Good work Doug! It is exciting to see Security Onion evolve.

Thanks, Brant!

>> >> email to security-onio...@googlegroups.com.
>> >> To post to this group, send email to securit...@googlegroups.com.

>> >> Visit this group at https://groups.google.com/group/security-onion.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > Follow Security Onion on Twitter!
>> > https://twitter.com/securityonion
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "security-onion" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to security-onio...@googlegroups.com.
>> > To post to this group, send email to securit...@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/security-onion.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Doug Burks
>>
>> --
>> Follow Security Onion on Twitter!
>> https://twitter.com/securityonion
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "security-onion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to security-onio...@googlegroups.com.
>> To post to this group, send email to securit...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.

[Doug Burks]

Hi Kevin,

Yes, filebeat is an option as well. To avoid scope creep, we may need
to stick with syslog-ng for now and then later consider filebeat once
the initial Elastic integration is complete.

On Fri, Aug 11, 2017 at 12:53 PM, Kevin Branch

>> >> email to security-onio...@googlegroups.com.
>> >> To post to this group, send email to securit...@googlegroups.com.

>> >> Visit this group at https://groups.google.com/group/security-onion.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > Follow Security Onion on Twitter!
>> > https://twitter.com/securityonion
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "security-onion" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to security-onio...@googlegroups.com.
>> > To post to this group, send email to securit...@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/security-onion.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Doug Burks
>>
>> --
>> Follow Security Onion on Twitter!
>> https://twitter.com/securityonion
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "security-onion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to security-onio...@googlegroups.com.
>> To post to this group, send email to securit...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.