Need to know more insight on logstash config file and bro inputs

[Blason R]

Hi Wes/Doug,

I am bit struggling with Bro input to my other ELK and using config files kept /etc/logstash/conf.d as a reference point.

I would really appreciate if someone can shed some light on those config files? Or is there any wiki written on those? At least I did not find it.

Well my query is, which are the logstash config files being used to input the bro data from //nsm/bro/logs/current into elasticsearch?

As I discussed in my earlier post since I am using SO as sensor which will talk to my ELK host which is laready in network and processing the data/logs from my other security devices/servers. I am still struggling with injesting data from SO sensor to elasticsearch.

Once data in injected and indexed I can build the Kibana visualzations and I know current SO template might not work.

Hence I would really appreciate if someone can help me from SO perspective. in understanding data injestion.

[Wes Lambert]

The config files to process Bro data can be found in /etc/logstash/conf.d/11*preprocess*bro*.conf.

Keep in mind, if you are sending this over via syslog, you may need to alter the config files, or have another config file to parse the syslog and get the data into the expected format for the Bro config files.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Blason R]

On Monday, October 23, 2017 at 5:06:35 PM UTC+5:30, Wes wrote:
> The config files to process Bro data can be found in /etc/logstash/conf.d/11*preprocess*bro*.conf.
>
>
> Keep in mind, if you are sending this over via syslog, you may need to alter the config files, or have another config file to parse the syslog and get the data into the expected format for the Bro config files.
>
>
> Thanks,
> Wes
>
>
> On Mon, Oct 23, 2017 at 12:26 AM, Blason R <blas...@gmail.com> wrote:
> Hi Wes/Doug,
>
>
>
> I am bit struggling with Bro input to my other ELK and using config files kept /etc/logstash/conf.d as a reference point.
>
>
>
> I would really appreciate if someone can shed some light on those config files? Or is there any wiki written on those? At least I did not find it.
>
>
>
> Well my query is, which are the logstash config files being used to input the bro data from //nsm/bro/logs/current into elasticsearch?
>
>
>
> As I discussed in my earlier post since I am using SO as sensor which will talk to my ELK host which is laready in network and processing the data/logs from my other security devices/servers. I am still struggling with injesting data from SO sensor to elasticsearch.
>
>
>
> Once data in injected and indexed I can build the Kibana visualzations and I know current SO template might not work.
>
>
>
> Hence I would really appreciate if someone can help me from SO perspective. in understanding data injestion.
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to the Google Groups "security-onion" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
>
> To post to this group, send email to securit...@googlegroups.com.

>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.

How is SO sending those logs to logstash? And sending logs over syslog is the only option available? I guess? Can you please elabore little more on config changes if I am sending those over syslog?

[Wes Lambert]

Currently, in a standalone configuration syslog-ng forwards logs to Logstash for ingestion.

See the following more details:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Elastic-Architecture#detailed-data-flow-diagram

I wouldn't say sending via syslog is the only option.  You could install a Logtsash forwarder, Beat, etc. to push the logs to Logstash, that is your choice..  The reason I mentioned syslog was because you would simply add a new destination.

In regard to the config files, you could use the same config files on your install, but you will need to make sure prerequisites are met (defined within the config files) so that the Bro config files are receiving messages in the expected format (this means, if sending by syslog, stripping out info the Bro config files do not care about).

Thanks,

Wes 

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Blason R]

So do we have already have logstash forwarder installed on sensor only? I guess not. And if I am using the same config file on my ELK I need to use port 1514 on ELK right? and my sensor should be forwarding to 1514? And these changes I need to do on syslog-ng right or bro config?

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/309rowVwdAk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.

[Blason R]

By the way!! Just curious to know; do I need to forward the snort data as well along with Bro to detect attacks or signatures?

[Wes]

> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/309rowVwdAk/unsubscribe.
>

> To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.

So do we have already have logstash forwarder installed on sensor only? I guess not.

I do not believe Filebeat is installed by default.

And if I am using the same config file on my ELK I need to use port 1514 on ELK right? and my sensor should be forwarding to 1514? And these changes I need to do on syslog-ng right or bro config?

Another destination should be configured (to go to your Logstash on separate Elastic instance) in syslog-ng.

This depends on the port on which your Logstash instance is listening.

So do we have already have logstash forwarder installed on sensor only? I guess not. And if I am using the same config file on my ELK I need to use port 1514 on ELK right? and my sensor should be forwarding to 1514? And these changes I need to do on syslog-ng right or bro config?

If you want alert data (besides Bro Notices), then yes, you will need to forward the Snort data as well.

See: https://github.com/Security-Onion-Solutions/security-onion/wiki/ThirdPartyIntegration#how-do-i-send-ids-alerts-to-an-external-system

Thanks,
Wes

[Roy]

I've got Filebeat on all sensors sending bro logs to a cluster of Logstash boxes which is filtering and sending to a dedicated Elastic cluster. See attached filebeat.yml and logstash conf files...

[Blason R]

Hi Wes,

I started receiving a logs in my ELK after using thsoe config file and really thanks for that. I am using syslog-ng to send the messages.

Ony thing is certain fileds are not mapping out and I loaded the logstash templated. Am I missing anything here?

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Wes Lambert]

What do you mean by not mapping out?  Fields not getting parsed or indexed?

Thanks,

Wes

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

[Blason R]

Yep that's right. Certain fields are not getting indexed. Not sure why I mean I followed templates and all those log stash config files. Especially snort data fields with ET rules.

To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Wes Lambert]

Blason,

If you are referring to the Logstash template from here:

https://github.com/Security-Onion-Solutions/elastic-test/blob/master/etc/logstash/logstash-template.json

... it does not necessarily contain all of the mappings at them moment.  This is actually something we are currently working on.  You will need to load the logstash index-pattern file, found here:

https://github.com/Security-Onion-Solutions/elastic-test/blob/master/kibana/dashboards/dashboards/index-pattern/logstash-*.json

Thanks,

Wes

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/309rowVwdAk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Blason R]

Hey Wes,

How do I load that /logstash-*.json in Elasticsearch? Can you please give me the command pls?

You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/309rowVwdAk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/309rowVwdAk/unsubscribe.

[Doug Burks]

Hi Blason,

I mentioned to you previously that we cannot provide any support
whatsoever for running our dashboards elsewhere:
https://groups.google.com/d/topic/security-onion/cQCwlsnvl9E/discussion

The same goes for our logstash config. It is designed for our
pipeline config and our specific software versions. We cannot
guarantee that it will work elsewhere and we cannot provide any
support for running it elsewhere.

Doug Burks

[Blason R]

Hi Doug,

Yes I agree on the part and really appreciate for the assistance provided, infact I just wanted to know how the same logstash*.json file is uploaded on SO master server.

>>>>>> security-onion+unsubscribe@googlegroups.com.
>>>>>> To post to this group, send email to security-onion@googlegroups.com.

>>>>>> Visit this group at https://groups.google.com/group/security-onion.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>>
>>>>> --
>>>>> Follow Security Onion on Twitter!
>>>>> https://twitter.com/securityonion
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "security-onion" group.
>>>>>
>>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>> an email to security-onion+unsubscribe@googlegroups.com.
>>>>>
>>>>>
>>>>> To post to this group, send email to security-onion@googlegroups.com.

>>>>> Visit this group at https://groups.google.com/group/security-onion.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>> --
>>>> Follow Security Onion on Twitter!
>>>> https://twitter.com/securityonion
>>>> ---
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "security-onion" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/security-onion/309rowVwdAk/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to

>>>> security-onion+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to security-onion@googlegroups.com.

>>>> Visit this group at https://groups.google.com/group/security-onion.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> Follow Security Onion on Twitter!
>>> https://twitter.com/securityonion
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "security-onion" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to security-onion+unsubscribe@googlegroups.com.
>>> To post to this group, send email to security-onion@googlegroups.com.

>>> Visit this group at https://groups.google.com/group/security-onion.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> Follow Security Onion on Twitter!
>> https://twitter.com/securityonion
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "security-onion" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/security-onion/309rowVwdAk/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> security-onion+unsubscribe@googlegroups.com.
>> To post to this group, send email to security-onion@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--

Doug Burks

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/309rowVwdAk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Blason R]

Hi Roy,

I guess this does not contain the Snort ET logs. Any idea where the snort logs are being written so that those can be included with filebeat.