Bro logs to json without breaking elsa

[Audrius J]

Hi,

First thanks Doug for your effort of this distro!
Next, we like elsa very much, but despite this we also send logs to our ELK cluster, there logs are parsed and enriched.
But to save processing power we also would like to save bro logs in json format and send them to elk cluster. This is possible in bro, but this should brake elsa. There is a good way to avoid such issue?

Regards,
Audrius

[Doug Burks]

Hi Audrius,

One option would be to add a new destination to
/etc/syslog-ng/syslog-ng.conf that would output Bro logs in JSON
format to your desired location.

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks

[Kevin Branch]

I have used Logstash on the Security Onion sensor to parse Bro files using the csv filter that is pretty high performance compared to grok, like this for intel.log records:

    csv {

      columns => ["ts","uid","srcip","srcport","dstip","dstport","fuid","file_mime_type","file_desc","indicator","indicator_type","where","node","sources"]

      # a literal <tab> must be between the two quotes below

      separator => "    "

    }

but like Doug said, using syslog-ng to pump out json directly would probably be even higher performance, leaving most or all of the parsing load to your ELK cluster.  See:

https://dustinoprea.com/2014/09/16/convert-syslog-events-to-a-json-stream/

My trouble with feeding Bro records into ELK has been that the amount of hardware needed for ELSA to keep up with those high-volume Bro records like bro_conn and bro_http, is not enough for ELK to handle those same records.  You get more with ELK, but it costs more to get it :)

Kevin

On Tue, Mar 14, 2017 at 5:34 AM, Doug Burks <doug....@gmail.com> wrote:

Hi Audrius,

One option would be to add a new destination to
/etc/syslog-ng/syslog-ng.conf that would output Bro logs in JSON
format to your desired location.

On Tue, Mar 14, 2017 at 2:47 AM, Audrius J <aud...@gmail.com> wrote:
> Hi,
>
> First thanks Doug for your effort of this distro!
> Next, we like elsa very much, but despite this we also send logs to our ELK cluster, there logs are parsed and enriched.
> But to save processing power we also would like to save bro logs in json format and send them to elk cluster. This is possible in bro, but this should brake elsa. There is a good way to avoid such issue?
>
> Regards,
> Audrius
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Audrius J]

Thanks for advise!
I just tried to do it via syslog-ng and it seems that I need to create a template for it and maintain it. But I would like to avoid it, because depending on the bro version, they change the fields in bro logs and I always need to check if nothing is changed after upgrade.

With syslog-ng, with this config:
destination d_broconn_json {file("/var/log/nsm/sensor_logs/bro_conn.json" template("$(format-json --scope selected_macros --scope nv_pairs)\n"));};
log {source(s_bro_conn); destination(d_broconn_json);};

I get:
{
"TAGS": ".source.s_bro_conn",
"SOURCEIP": "127.0.0.1",
"PROGRAM": "bro_conn",
"PRIORITY": "notice",
"MESSAGE": "1489597229.165013\tCAiskl1FtF2zl3Veve\t10.0.60.29\t49218\t239.255.255.250\t1900\tudp\t-\t6.016471\t744\t0\tS0\tT\tF\t0\tD\t6\t912\t0\t0\t(empty)\t-\t-\tids-lab002-eth1",
"HOST_FROM": "ids-lab002",
"HOST": "ids-lab002",
"FILE_NAME": "/nsm/bro/logs/current/conn.log",
"FACILITY": "user",
"DATE": "Mar 15 17:01:36"
}

But I would like to get something like this:
{
"ts": 1489604626.970392,
"uid": "Cr43uD3ZMHyXnN0Iti",
"id.orig_h": "10.0.60.21",
"id.orig_p": 5779,
"id.resp_h": "64.233.161.128",
"id.resp_p": 80,
"trans_depth": 1,
"method": "GET",
"host": "storage.googleapis.com",
"uri": "/update-delta/gkmgaooipdjhmangpemjhigmamcehddo/17.96.0/17.95.0/d0166ee64ec93189b34813cd2def8e77219d8fea86e0956ca782741ad45bf2d9.crxd",
"user_agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36",
"request_body_len": 0,
"response_body_len": 204746,
"status_code": 200,
"status_msg": "OK",
"tags": [],
"resp_fuids": ["FG82hfZM0RtSMn3E5"]
}

For now we are using Logstash with CSV and Grok filters and it works almost great, but because of amount of logs and performance optimization it would be great to have an option to save logs in ES native format and in that case we will also avoid problems with bro updates/upgrades.
We could log everything to json, but because it breaks elsa, I want to avoid it...

In our case, ELK is running on its own dedicated cluster. So no ES on sensor, just filebeat for shipping logs.
Do you have any different ideas?

[Audrius J]

Ok, it seems, that I found that I want here https://gist.github.com/J-Gras/f9f86828f9e9d9c0b8f0908bc3573bb0.
Short testing showed required results with some caveats.

Saving logs in json format has its own price ad it is - readability. Some symbols are encoded, to avoid problems with some special characters. For example "&" is encoded to "\u0026" and etc...
It seems, that bro creators have chosen TAB separator for a good reason...
So, thanks for ideas and help!

Regards,
Audrius