Syslog Flow Help (Sophos XG Logstash)

190 views

Skip to first unread message

boyhi...@yahoo.com

unread,

Nov 9, 2018, 11:21:08 AM11/9/18

to security-onion

I'm trying to get Sophos XG logs parsed in logstash. For some reason (presumable human error) I'm having a harder time than I did wit the UTM/SG model.

I have logse being sent from Sophos over SYSLOG (514) to SO.

The logs are naturally parsed as bro_syslog and also device="SFW"

No matter what I modify I cannot seem to process the logs with a new conf file.

Again I know it's a me issue, but any direction on modifying the flow of these to properly parse would be great. The firewall logs are the only item I have coming via Syslog if that helps.

Wes Lambert

unread,

Nov 10, 2018, 7:23:58 AM11/10/18

to securit...@googlegroups.com

Hi Josh,

Would you happen to have a sample log you can share?

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

https://twitter.com/therealwlambert

https://securityonion.net/

Josh Silvestro

unread,

Nov 10, 2018, 8:58:06 AM11/10/18

to securit...@googlegroups.com

Sophos XG Sample:

date=2018-11-10 time=08:54:23 timezone="EST" device_name="SFVH" device_id=C01001P8R29TF3F log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=4 policy_type=1 user_name="" user_gp="" iap=12 ips_policy_id=0 appfilter_policy_id=0 application="QUIC" application_risk=1 application_technology="Network Protocol" application_category="Infrastructure" in_interface="Port1" out_interface="Port2" src_mac=00: 0:00: 0:00: 0 src_ip=Z.Z.Z.Z src_country_code=R1 dst_ip=X.X.X.X dst_country_code=USA protocol="UDP" src_port=38870 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=Y.Y.Y.Y tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="2579803808" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0

Wes Lambert

unread,

Nov 10, 2018, 9:52:51 AM11/10/18

to securit...@googlegroups.com

Josh,

This is a basic, non-optimized example, but you could try something like the following:

filter {

if [message] =~ "SFVH" {

mutate {

add_field => {

"message1" => "%{type}"

"message2" => "%{message}"

"entire_message" => "%{message1}%{message2}"

}

replace => { "type", "sophos" }

add_tag => [ "sophos_xg", "firewall" ]

}

kv {

source => "entire_message"

}

mutate {

remove_field => ["entire_message"]

}

Thanks,

Wes

On Sat, Nov 10, 2018 at 8:58 AM Josh Silvestro <josh.si...@gmail.com> wrote:

Sophos XG Sample:

date=2018-11-10 time=08:54:23 timezone="EST" device_name="SFVH" device_id=C01001P8R29TF3F log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=4 policy_type=1 user_name="" user_gp="" iap=12 ips_policy_id=0 appfilter_policy_id=0 application="QUIC" application_risk=1 application_technology="Network Protocol" application_category="Infrastructure" in_interface="Port1" out_interface="Port2" src_mac=00: 0:00: 0:00: 0 src_ip=Z.Z.Z.Z src_country_code=R1 dst_ip=X.X.X.X dst_country_code=USA protocol="UDP" src_port=38870 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=Y.Y.Y.Y tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="2579803808" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

Wes Lambert

unread,

Nov 10, 2018, 9:54:51 AM11/10/18

to securit...@googlegroups.com

Actually, that should be:

filter {

if [message] =~ "SFVH" {

mutate {

add_field => {

"message1" => "%{type}"

"message2" => "%{message}"

"entire_message" => "%{message1}%{message2}"

}

replace => { "type" => "sophos" }

Josh Silvestro

unread,

Nov 11, 2018, 6:59:50 AM11/11/18

to securit...@googlegroups.com

Wes,

Thanks! That did what I was trying to do. Some of the logs look like they're having issues with double quotes and outputting the following:

    "application_category": "\\u0022\\u0022",
    "dstzonetype": "\\u0022\\u0022",
    "out_interface": "\\u0022\\u0022",
    "ips": "192.168.10.133",
    "vconnid": "\\u0022\\u0022",
    "action": "\\u0022Deny\\u0022",

I tried adding

gsub => ["entire_message","\"","'"]

But it's still happening, I've not ran in to this before, any suggestions?

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/yv5YnoYIarw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

Thank You,

Joshua Silvestro

Wes

unread,

Nov 12, 2018, 7:47:31 AM11/12/18

to security-onion

Hi Josh,

Could you provide another example log for which this issue occurs?

Thanks,
Wes

Josh Silvestro

unread,

Nov 12, 2018, 8:49:35 AM11/12/18

to securit...@googlegroups.com

It causes the Kibana view to be spaced out and vertical prints each letter.

"_source": {
    "vconnid": "\\u0022\\u0022",
    "sent_pkts": "0",
    "ips_policy_id": "0",
    "appresolvedby": "\\u0022Signature\\u0022",
    "device_id": "C01001P8R29TF3F",
    "port": 56574,
    "firewall{\"ts\":\"2018-11-12T13:46:01.699136Z\",\"uid\":\"Cw6AAA1FslAYuiZUO\",\"id.orig_h\":\"Y.Y.Y.Y\",\"id.orig_p\":41409,\"id.resp_h\":\"X.X.X.X\",\"id.resp_p\":514,\"proto\":\"udp\",\"facility\":\"DAEMON\",\"severity\":\"INFO\",\"message\":\"device": "\\u0022SFW\\u0022",
    "syslog-host": "sly-secon",
    "@timestamp": "2018-11-12T13:46:03.887Z",
    "syslog-priority": "notice",
    "dstzonetype": "\\u0022\\u0022",
    "syslog-file_name": "/nsm/bro/logs/current/syslog.log",
    "host": "gateway",
    "user_name": "\\u0022\\u0022",
    "tags": [
      "syslogng",
      "bro",
      "firewall",
      "sophos",
      "internal_source",
      "conf_file_9000"
    ],
    "source_ips": "Z.Z.Z.Z",
    "in_interface": "\\u0022Port1.10\\u0022",
    "hb_health": "\\u0022No",
    "srczonetype": "\\u0022\\u0022",
    "application": "\\u0022\\u0022",
    "syslog-host_from": "sly-secon",
    "iap": "0",
    "application_risk": "0",
    "event_type": "firewall",
    "syslog-sourceip": "127.0.0.1",
    "fw_rule_id": "0",
    "@version": "1",
    "dir_disp": "\\u0022\\u0022",
    "message": "\\u0022\\u0022",
    "sent_bytes": "0",
    "source_ip": "192.168.10.102",
    "duration": "0",
    "device_name": "\\u0022SFVH\\u0022",
    "priority": "Information",
    "application_technology": "\\u0022\\u0022",
    "user_gp": "\\u0022\\u0022",
    "icmp_code": "0",
    "date": "2018-11-12",
    "syslog-tags": ".source.s_bro_syslog",
    "policy_type": "0",
    "app_is_cloud": "0\"}",
    "action": "\\u0022Deny\\u0022",
    "dstzone": "\\u0022\\u0022",
    "destination_geo.country_code2": "protocol=\\u0022ICMP\\u0022",
    "timestamp": "2018-11-12T13:46:03.887Z",
    "icmp_type": "8",
    "out_interface": "\\u0022\\u0022",
    "recv_bytes": "0",
    "src_country_code": "dst_ip=192.168.10.1",
    "ips": "W.W.W.W",
    "source_mac": "XX:XX:XX:XX:XX",
    "log_type": "\\u0022Firewall\\u0022",
    "application_category": "\\u0022\\u0022",
    "syslog-facility": "user",
    "srczone": "\\u0022\\u0022",
    "logstash_time": 0.006455898284912109,
    "log_subtype": "\\u0022Denied\\u0022",
    "recv_pkts": "0",
    "log_component": "\\u0022Appliance",
    "appfilter_policy_id": "0"

Wes Lambert

unread,

Nov 12, 2018, 9:02:38 AM11/12/18

to securit...@googlegroups.com

Hi Josh,

Could you please re-attach a raw log (via syslog or the original message body) that causes this behavior?

Thanks,

Wes

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

Josh Silvestro

unread,

Nov 14, 2018, 8:26:42 AM11/14/18

to securit...@googlegroups.com

Grabbed a log from syslog, so it looks like it's actually making its way over that way?

syslog.log:{"ts":"2018-11-14T13:20:10.183584Z","uid":"C4y92i2f5Yf4gFxBhd","id.orig_h":"X.X.X.X","id.orig_p":39627,"id.resp_h":"Y.Y.Y.Y","id.resp_p":514,"proto":"udp","facility":"DAEMON","severity":"WARNING","message":"device=\u0022SFW\u0022 date=2018-11-14 time=08:19:15 timezone=\u0022EST\u0022 device_name=\u0022SFVH\u0022 device_id=C01001P8R29TF3F log_id=020804407002 log_type=\u0022IDP\u0022 log_component=\u0022Signatures\u0022 log_subtype=\u0022Drop\u0022 priority=Warning idp_policy_id=5 fw_rule_id=8 user_name=\u0022username\u0022 signature_id=15 signature_msg=\u0022Reset outside window\u0022 classification=\u0022Potentially Bad Traffic\u0022 rule_priority=2 src_ip=W.W.W.W src_country_code=USA dst_ip=Z.Z.Z.Z dst_country_code=R1 protocol=\u0022TCP\u0022 src_port=443 dst_port=6319 platform=\u0022All\u0022 category=\u0022Misc\u0022 target=\u0022All\u0022"}

Wes Lambert

unread,

Nov 14, 2018, 6:14:48 PM11/14/18

to securit...@googlegroups.com

It may be easier to parse if you can fix/modify that upfront, before it gets to Logstash.

Thanks,

Wes

On Wed, Nov 14, 2018 at 8:26 AM Josh Silvestro <josh.si...@gmail.com> wrote:

Grabbed a log from syslog, so it looks like it's actually making its way over that way?

syslog.log:{"ts":"2018-11-14T13:20:10.183584Z","uid":"C4y92i2f5Yf4gFxBhd","id.orig_h":"X.X.X.X","id.orig_p":39627,"id.resp_h":"Y.Y.Y.Y","id.resp_p":514,"proto":"udp","facility":"DAEMON","severity":"WARNING","message":"device=\u0022SFW\u0022 date=2018-11-14 time=08:19:15 timezone=\u0022EST\u0022 device_name=\u0022SFVH\u0022 device_id=C01001P8R29TF3F log_id=020804407002 log_type=\u0022IDP\u0022 log_component=\u0022Signatures\u0022 log_subtype=\u0022Drop\u0022 priority=Warning idp_policy_id=5 fw_rule_id=8 user_name=\u0022username\u0022 signature_id=15 signature_msg=\u0022Reset outside window\u0022 classification=\u0022Potentially Bad Traffic\u0022 rule_priority=2 src_ip=W.W.W.W src_country_code=USA dst_ip=Z.Z.Z.Z dst_country_code=R1 protocol=\u0022TCP\u0022 src_port=443 dst_port=6319 platform=\u0022All\u0022 category=\u0022Misc\u0022 target=\u0022All\u0022"}

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

Josh Silvestro

unread,

Nov 14, 2018, 6:19:03 PM11/14/18

to securit...@googlegroups.com

I would agree however the sophos xg logging has limited to no options for changing output.

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/yv5YnoYIarw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

Wes Lambert

unread,

Nov 15, 2018, 9:04:36 PM11/15/18

to securit...@googlegroups.com

Hi Josh,

It looks like you'll want to consider somehow escaping/converting/dropping the double quote conversion on the Logstash side.

I'm not sure what that would look like off the top of my head, but I can look at that as time allows.

Thanks,

Wes

Josh Silvestro

unread,

Dec 1, 2018, 8:42:40 AM12/1/18

to securit...@googlegroups.com

I was able to get that figured out. I just created a filter to drop the \\ and u0022 and works fine. I'm still getting tables that are really spaced out on the field side. I figured out the issue is for some reason it also adds the "message" value as a field.

I'm trying to drop the field, but since it's variable it seems to not be working.

Field starts:

{"ts":"2018-12-01T13:09:47

Any suggestions on parsing this to drop?

Thank You,

Joshua Silvestro

Philip Robson

unread,

Dec 1, 2018, 1:06:30 PM12/1/18

to securit...@googlegroups.com

You can do ?something:{data:something}? For optional fields.

Philip Robson

unread,

Dec 1, 2018, 1:07:00 PM12/1/18

to securit...@googlegroups.com

Sorry should be ?()? Missed the brackets I thinm

Edwin

unread,

May 6, 2019, 7:13:06 PM5/6/19

to security-onion

Hi Josh,

Do you mind sharing the Logstash conf file you ended up using? I have a similar issue with my Sophos XG logs coming through SYSLOG (514).

Regards,
Edwin