Security Onion - Elasticsearch - Sophos Logs
Josh Silvestro
Now that working with Doug & Wes I was able to get some issues worked out, I'd like to start parsing Sophos UTM logs as most of my SO sensors do receive Sophos UTM logs. I started working with Grok and using the debugger, still learning that. But also wondered if anyone else uses Sophos firewalls? It may make sense for a default conf file in the SO installations?
Some sample events below.
Sample Event:
10:07-09:32:32 firewall_hostname ulogd[25637]: id="2105" severity="info" sys="SecureNet" sub="ips" name="UDP flood detected" action="UDP flood" fwrule="60013" initf="eth1" srcmac="11:11:11:11:11:11" dstmac="00:00:00:00:00:00" srcip="100.100.100.100" dstip="24.24.24.24" proto="17" length="1521" tos="0x00" prec="0x00" ttl="58" srcport="8080" dstport="8080"
Sample Event:
10:07-12:18:09 firewall_hostname httpproxy[18485]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="X.X.X.X" dstip="X.X.X.X" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="6903" request="0x8f74000" url="https://outlook.office365.com/" referer="" error="" authtime="0" dnstime="140397" cattime="290" avscantime="0" fullreqtime="75294216" device="0" auth="0" ua="" exceptions="" category="156" reputation="trusted" categoryname="Web Mail" country="United States"
Wes Lambert
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
Josh Silvestro
For anyone who may be looking for Sophos logs:
Place in /etc/logstash/conf.d/sophos_utm.conf
# Sophos UTM
filter {
if [type] == "2017" {
kv {
source => "message"
}
mutate {
convert => [ "destination_port", "integer" ]
convert => [ "source_port", "integer" ]
convert => [ "ip_version", "integer" ]
replace => { "type" => "firewall" }
rename => { "dstip" => "destination_ip" }
rename => { "dstport" => "destination_port" }
rename => { "dstmac" => "destination_mac" }
rename => { "name" => "rule_name" }
rename => { "srcip" => "source_ip" }
rename => { "srcport" => "source_port" }
rename => { "srcmac" => "source_mac" }
rename => { "filteraction" => "filter_action" }
add_tag=> [ "sophos","firewall" ]
}
}
}
#END
*Check your type, for some reason mine came in as 2017.
Doug Burks
<securit...@googlegroups.com> wrote:
> Fair enough, thanks. I actually found a tool called KV that made it super simple, modified some fields to work with existing dashboards thanks! Instead of starting a new forum. I'd like to install Xpack for alerting/monitoring on custom searches. Is there a recommended way to do this? I located and ran "bin/elasticsearch-plugin install x-pack" but it's erroring telling me I don't have Java in my path. You're smarter than I, but I feel like x-pack would be a good thing to have by default in SO.
We cannot include x-pack in Security Onion because it is not open source.
If you're looking for alerting/monitoring on custom searches, have you
considered ElastAlert (already included in Security Onion)?
https://github.com/Security-Onion-Solutions/security-onion/wiki/ElastAlert
--
Doug Burks
Josh Silvestro
I've not checked out ElastAlert, but will give it a go. My wanting to try x-pack was due to the default security searches and alerting. For clarification, I understand the you cannot "include" but does that mean I cannot add either?
Wes Lambert
Doug,
I've not checked out ElastAlert, but will give it a go. My wanting to try x-pack was due to the default security searches and alerting. For clarification, I understand the you cannot "include" but does that mean I cannot add either?
Blason R
> Josh,
>
>
> You could certainly add X-Pack yourself, but we wouldn't be able to support it.
>
>
> Thank,
> Wes
>
>
>
> On Oct 8, 2017 4:19 PM, "'Josh Silvestro' via security-onion" <securit...@googlegroups.com> wrote:
> Doug,
>
>
>
> I've not checked out ElastAlert, but will give it a go. My wanting to try x-pack was due to the default security searches and alerting. For clarification, I understand the you cannot "include" but does that mean I cannot add either?
>
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to the Google Groups "security-onion" group.
>
>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.
By the way guys , where are are the elastalert files are kept? I need to see if those can be modified aaccording to my need?
And Josh would you mind please sharing the entire logstash config file Sophos UTM?
Doug Burks
> By the way guys , where are are the elastalert files are kept? I need to see if those can be modified aaccording to my need?
Per my previous email in this thread, you can read more about
ElastAlert on the ElastAlert page on our Wiki:
https://github.com/Security-Onion-Solutions/security-onion/wiki/ElastAlert
From that page:
"ElastAlert rules are stored in /etc/elastalert/rules/."
--
Doug Burks
Josh Silvestro
Understood, and that makes perfect sense. I don't see anything in the wiki, may be overlooking it. But any steps on installing additional elasticsearch plugins? Currently following steps via their site there's a java message. I can get that figured out, just wasn't sure if there was a "right" way with SO.
Blason, a few posts back I did post the config. If it looks sparse that's because it's KV doing all the work. It auto pareses the message field looking for "=" and other info. Other then the renames I also attached everyone seems 100% in my use/review so far.
Doug Burks
<securit...@googlegroups.com> wrote:
> Doug,
>
> Understood, and that makes perfect sense. I don't see anything in the wiki, may be overlooking it. But any steps on installing additional elasticsearch plugins? Currently following steps via their site there's a java message. I can get that figured out, just wasn't sure if there was a "right" way with SO.
Keep in mind that your Elastic components are running inside Docker
containers. So any configuration would have to be done inside the
Docker container itself.
--
Doug Burks
Josh Silvestro
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/sOD7kkFK2V0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
Wes
> Good point. And this should be accomplished via "docker ps" & then docker attach "container id" , right?
>
>
> On Mon, Oct 9, 2017 at 7:33 AM, Doug Burks <doug....@gmail.com> wrote:
> On Mon, Oct 9, 2017 at 6:51 AM, 'Josh Silvestro' via security-onion
>
>
> > Doug,
>
> >
>
> > Understood, and that makes perfect sense. I don't see anything in the wiki, may be overlooking it. But any steps on installing additional elasticsearch plugins? Currently following steps via their site there's a java message. I can get that figured out, just wasn't sure if there was a "right" way with SO.
>
>
>
> Hi Josh,
>
>
>
> Keep in mind that your Elastic components are running inside Docker
>
> containers. So any configuration would have to be done inside the
>
> Docker container itself.
>
>
>
>
>
> --
>
> Doug Burks
>
>
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/sOD7kkFK2V0/unsubscribe.
>
>
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
> --
>
>
>
>
>
> Thank You,
> Joshua Silvestro
I normally do something like:
sudo docker exec -it [container] /bin/bash
But you should be able to use docker attach.
Thanks,
Wes
Josh Silvestro
{"type":"log","@timestamp":"2017-10-12T15:38:36Z","tags":["warning","config"],"pid":1,"message":"Settings for \"xpack\" were not applied, check for spelling errors and ensure the plugin is loaded."}
I know SO is not supporting this, but figured I'd throw it up and maybe someone else has worked through this issue already.
Josh Silvestro
# Default Kibana configuration from kibana-docker.
server.name: kibana
server.host: "0"
elasticsearch.url: http://elasticsearch:9200
#elasticsearch.username: elasticsearch
#elasticsearch.password: changeme
xpack.monitoring.ui.container.elasticsearch.enabled: true
logging.dest: /var/log/kibana/kibana.log
Getting this error in the Kibana.logs
{"type":"log","@timestamp":"2017-10-12T16:26:15Z","tags":["warning","config"],"pid":1,"message":"Settings for \"xpack\" were not applied, check for spelling errors and ensure the plugin is loaded."}
Wes Lambert
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Josh Silvestro
Hi Josh,I don't think so. You could try having a look here:Thanks,Wes
On Thu, Oct 12, 2017 at 12:45 PM, 'Josh Silvestro' via security-onion <security-onion@googlegroups.com> wrote:
I know you can't provide support on this. But is there anything that'd be weird about the SO install/setup that would make the default install steps not work? I wonder if the issue lies in the Kibana.yml? Xpack was commented out, I uncommented that and still getting issues.
# Default Kibana configuration from kibana-docker.
server.name: kibana
server.host: "0"
elasticsearch.url: http://elasticsearch:9200
#elasticsearch.username: elasticsearch
#elasticsearch.password: changeme
xpack.monitoring.ui.container.elasticsearch.enabled: true
logging.dest: /var/log/kibana/kibana.log
Getting this error in the Kibana.logs
{"type":"log","@timestamp":"2017-10-12T16:26:15Z","tags":["warning","config"],"pid":1,"message":"Settings for \"xpack\" were not applied, check for spelling errors and ensure the plugin is loaded."}
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/sOD7kkFK2V0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.