Splunk as a Security Onion sensor manager

[Ludwig Goon]

So I had a short epiphany of using spunk and spunk forwarders to centralize data from the sensors and it would also allow me to send data to two (spunk) managers for high availability installations with multiple sensors.

So far I have thought of the following advantages and disadvantages and wanted to know if anyone else is using or can contribute to the idea of using spunk as a Security Onion manager?

Advantages:
1. Snort and Bro logs can be forwarded to spunk from multiple sensors, even encrypted!
2. Could use some Splunk apps to visualize and alert on SO data, including SO app itself.
3. Can provide performance and uptime, NAGIOS like metrics.
4. Can use DBconnect spunk app to send database data to splunk.

DisAdvantages:
1. Elsa and OSSEC log data can't send at this time, also no known visual interface.
2. Can't use pcap/session data extractions on the sensors from splunk?
3. Same holds true for binary.
4. Difficult to configure snorby, squert, and squil?

Help me DB1 KHAN-OBI! (DB1=Doug Burks 1)

[Doug Burks]

Hi Ludwig,

Have you looked at Splunk pricing? Many folks find it to be
cost-prohibitive. In fact, ELSA was written by Martin Holste because
his organization couldn't afford Splunk. Most of the things that
you'll want to do with Splunk can be done with ELSA (for free).
Please let us know what questions you have about ELSA and we'll be
glad to help.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Ludwig Goon]

So this is for my company since we are considering Splunk. I think if you spend hundred-thousand in budget for a SIEM you can spend about $100K for splunk and even lower. Thus the barrier to entry for major IT companies is affordable.

However for small companies and developers I agree the cost would be prohibitive. I did ask splunk about offering a personal license for enthusiasts and small companies and they do have discounts for 501c or non-profits. So it's something that's still in the works. 

So for Elsa, the major issue I have is dashboards, and they having to use Kibana and a bunch of platforms to do what Splunk does in one.

other concerns with Elsa are:

1. Search interface and language is not intuitive (in need of an update)

2. Dashboards or the dependence on other platforms.

3. Limited data formats.

4. No meta language API or alerting capability.

[Doug Burks]

Replies inline.

On Sat, Oct 11, 2014 at 12:59 PM, Ludwig Goon <lag...@gmail.com> wrote:
> So this is for my company since we are considering Splunk. I think if you
> spend hundred-thousand in budget for a SIEM you can spend about $100K for
> splunk and even lower. Thus the barrier to entry for major IT companies is
> affordable.
>
> However for small companies and developers I agree the cost would be
> prohibitive. I did ask splunk about offering a personal license for
> enthusiasts and small companies and they do have discounts for 501c or
> non-profits. So it's something that's still in the works.
>
> So for Elsa, the major issue I have is dashboards, and they having to use
> Kibana and a bunch of platforms to do what Splunk does in one.

ELSA doesn't use Kibana. Are you thinking of Elasticsearch perhaps?

> other concerns with Elsa are:
>
> 1. Search interface and language is not intuitive (in need of an update)

Personally, I think the language is fairly intuitive. Quoting from
https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation#Syntax:
"Query syntax is loosely based on Google search syntax. "

Additionally, we've made most common queries a single click:
http://blog.securityonion.net/2014/01/new-securityonion-web-page-package.html

> 2. Dashboards or the dependence on other platforms.

Not sure that I understand what you're saying here.

> 3. Limited data formats.

Not sure that I understand. ELSA is based on syslog-ng so it can read
any standard log file and accepts standard syslog, which is a fairly
universal standard. In addition, it can query databases:
http://ossectools.blogspot.com/2012/09/integrating-org-data-in-elsa.html

> 4. No meta language API or alerting capability.

Yes, ELSA does have an API and alerting:
https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation#Command-line_Interface_and_API
https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation#Alerts

[Ludwig Goon]

Oh yeah your right I was thinking of Elastisearch.

Limited Data formats also include unstructured data formats or multilined data such as wifi accesss point data or SNMP data.

I will have to check the "google search" like capability on my personal instance. 

So is all the data from the database searchable in ELSA?

[Doug Burks]

Replies inline.

On Sat, Oct 11, 2014 at 1:22 PM, Ludwig Goon <lag...@gmail.com> wrote:
> Oh yeah your right I was thinking of Elastisearch.
>
> Limited Data formats also include unstructured data formats or multilined
> data such as wifi accesss point data or SNMP data.

Looks like syslog-ng supports multi-line logs:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.5-guides/en/syslog-ng-ose-v3.5-guide-admin/html/reference-source-file.html

> I will have to check the "google search" like capability on my personal
> instance.

The link I sent previously details query syntax:
https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation#Syntax

If you have further questions about syntax, please let us know.

> So is all the data from the database searchable in ELSA?

ELSA allows you to search more data than any of our other interfaces.
From ELSA, you have access to:
- NIDS alerts (Snort/Suricata)
- HIDS alerts from OSSEC agents
- host logs from OSSEC agents
- host logs from standard syslog senders
- session data (Bro conn.log)
- asset data (Bro software.log)
- transaction data (Bro protocol logs - dns, ftp, http, irc, ssl, smtp, etc.)
- other Bro logs (weird.log, intel.log, tunnel.log, etc.)

Additionally, you can pivot from ELSA to full packet capture and have
CapME render an ASCII transcript of the TCP stream or download the
pcap.

[Ludwig Goon]

Good information. But I still need to have sensors send data to two managers simultaneously. So if I can get splunk to forward bro and snort logs then I would say I am 60% there correct?

Re-watched your video, Also I see some performance issues with ELSA that may be a hinderance in high volume corporate environments. Plus the fact that I will have 10 to 20 users logging into the manager performing analytics.

Don't get me wrong, I think ELSA is cool. However I need to develop some of the same capabilities and features using Splunk. I will get those issues and screenshots to you next week. Enjoy the Holiday!

Also can you provide information/costs for on-site training should we decide to implement Security Onion fully? I did discuss this with my manager and he is very open.

thanks!

[Doug Burks]

Replies inline.

On Sat, Oct 11, 2014 at 1:56 PM, Ludwig Goon <lag...@gmail.com> wrote:
> Good information. But I still need to have sensors send data to two managers
> simultaneously. So if I can get splunk to forward bro and snort logs then I
> would say I am 60% there correct?

You don't have to use Splunk just to send data to two managers. You
can do this with ELSA:
https://groups.google.com/d/topic/security-onion/6iqe-bRpbEQ/discussion

I believe ELSA also has some native store-and-forward functionality.

> Re-watched your video, Also I see some performance issues with ELSA that may
> be a hinderance in high volume corporate environments. Plus the fact that I
> will have 10 to 20 users logging into the manager performing analytics.

Please provide more information about these "performance issues" and
we'll help you troubleshoot them.

> Don't get me wrong, I think ELSA is cool. However I need to develop some of
> the same capabilities and features using Splunk. I will get those issues and
> screenshots to you next week. Enjoy the Holiday!
>
> Also can you provide information/costs for on-site training should we decide
> to implement Security Onion fully? I did discuss this with my manager and he
> is very open.

Will respond via private email.

[Brad Shoop]

I wrote a Splunk app, including forwarder add ons, that is available on Splunkbase (see http://eyeis.net for details). It's a bit dated but is still quite effective. The app was written for use with the free version of Splunk to help introduce users to the data SO generates (mainly home users or those new to SO/NSM). Many of the dashboards may load slowly in high eps environments.

But to Doug's point Splunk is expensive and Bro generates a massive ton of data which means high cost and slower search performance with Splunk. You may find you prefer the performance of 20 people running simultaneous sub second queries with ELSA to the much longer wait times with Splunk.

Splunk is prettier to show SO off and the Splunk app is very useful for getting "the lay of the land" but ELSA is a better hunting tool once you're on the ground in my opinion and you cannot underestimate the value of the capME integration. Spend the money on better ways to leverage the data you'll be collecting, like training, learning how to care and feed a distributed SO deployment, intel feeds (CIF for free starters) and an Emerging Threats subscription.

Brad

[Ludwig Goon]

thanks brad. We are planning on using and updating the Bro App and the Security Onion app, However since we are a company that is implementing Splunk we definitely want to integrate Security Onion.