Security Onion and Hive Alert Integration

[Chris V]

Has anyone had any luck integrating the two? I tried following the guide and documentation https://securityonion.readthedocs.io/en/latest/hive.html

*My goal is simple, just to send the alerts from SO to Hive as intended by the documentation.*

*Security Onion Master host:*
First step was to create a hive.yaml file and modify it with url , port , and api key.

Second, place hive.yaml file in the elastalert/rules/ directory.

After that I see nothing coming into Hive. Was there an initial elastalert base setup I should have performed prior to creating the hive.yaml file?

*Hive host:*
I also noticed I did not have thehive4py setup. So I set that up using the "pip install thehive4py" still see nothing. So I downloaded the repo using git clone [http://git.com/thehive4py path] then tried to install thehive4py using the "./setup.py" script. I still get nothing.

I am leaning towards my hive and thehive4py integration not properly functioning. Unless there is a gotcha with the elastalert setup I missed as I only created the yaml file, modified it, and placed it in the elastalert rules folder.

Any tips or suggestions would be appreciated.

Thanks everyone!

[Nick S]

It isn't working for me either. I noticed while reviewing the logs it has something to do with the "alert" command in the elastalert container.
I know my thehive works correctly because I can import and export from MISP and cortex with no problem. It only the elastalert integration that is a problem.
---------------------ELASTALERT STDERR.log-----------------------------------------------------------
INFO:elastalert:Queried rule TheHive - New IDS Alert! from 2019-07-23 20:55 UTC to 2019-07-23 21:05 UTC: 9 / 9 hits
ERROR:root:Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1332, in alert
return self.send_alert(matches, rule, alert_time=alert_time, retried=retried)
File "/opt/elastalert/elastalert/elastalert.py", line 1421, in send_alert
alert.alert(matches)
File "elastalert/alerts.py", line 2151, in alert
alert_config[alert_config_field] = alert_config_value.format(**context)
KeyError: 'alert'

ERROR:root:Uncaught exception running rule TheHive - New IDS Alert!: 'alert'
--------------------------------------------------------------------------------------


----------elastalert rule-----------
# hive.yaml
# Elastalert rule to forward IDS alerts from Security Onion to a specified TheHive instance.
#
es_host: elasticsearch
es_port: 9200
name: TheHive - New IDS Alert!
type: frequency
index: "*:logstash-*"
num_events: 1
timeframe:
minutes: 10
buffer_time:
minutes: 10
allow_buffer_time_overlap: true

filter:
- term:
event_type: "bro_intel"

alert:
- "hivealerter"

hive_connection:
hive_host: http://1.1.1.1
hive_port: 9000
hive_apikey: 1111111111111111111111

hive_proxies:
http: ''
https: ''

hive_alert_config:
title: '{rule[name]} -- {match[alert]}'
type: 'external'
source: 'SecurityOnion'
description: '{match[message]}'
severity: 2
tags: ['elastalert, SecurityOnion']
tlp: 3
status: 'New'
follow: True

hive_observable_data_mapping:
- ip: '{match[source_ip]}'
- ip: '{match[destination_ip]}'

[Philip Robson]

Alert has to match a field from bro. The example rule is from an ids event.

I have thehive alerter working fine with custom alerts from different sources.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/19ce94f0-2f8a-4da9-be47-095884e7b873%40googlegroups.com.

[Nick S]

> To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/19ce94f0-2f8a-4da9-be47-095884e7b873%40googlegroups.com.

Thanks for replying Philip. you got it. I removed {match[alert]} since that is not a field in the bro_intel and it worked as expected. Thank you for catching that.

[Philip Robson]

No problem, I had the same issue :)

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/eb7be37e-cbe3-4344-83b3-eeb3c3645215%40googlegroups.com.

[Chris V]

so all you had to do was remove {match[alert]} and it worked? or did you replace it with something else?

[Philip Robson]

You dont need it but I changed it to match a field from the data the alert was based on.

On Wed, 24 Jul 2019, 22:29 'Chris V' via security-onion, <securit...@googlegroups.com> wrote:

so all you had to do was remove {match[alert]} and it worked? or did you replace it with something else?

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/538b12b3-e8f1-447d-a9e2-a8187521ade0%40googlegroups.com.

[Justin Hussey]

I still can't get security onion to send alerts to the Hive.  I would like some help if anyone could, please.  

So far, I've installed the hive (training VM) using virtual box on top of SO (running as my host OS on a dedicated workstation.)

I've created a yaml file ( /etc/elastalert/rules/hive.yaml) as instructed "https://securityonion.readthedocs.io/en/latest/hive.html?highlight=hive"

I am unsure if it's necessary or not to run an elasticsearch instance on my Hive-VM.  From my understanding, the hive.yaml file takes Snort alert events, and sends them to the hive VM --> port 9000.  I don't see anything pointing to the Hive's Elasticsearch instance except for what's written in /etc/thehive/application.conf.

I can successfully "curl www.testmyids.com" and receive GPL ATTACK_RESPONSE id check returned root in Squert, (plus i verified the event_type for the alert = "snort")

I can successfully log in and view my Hive instance, Though, I am still uncertain of what configuration needs to be tweaked to get alerts sent to the Hive.

Lastly, I ran rule-update, and so-restart to restart everything, but still no dice...

I've looked through and tweaked (to the best of my knowledge) elasticsearch.yml on both SO & Hive.  Also I've edited what I could in application.conf.  Not sure for the next troubleshooting steps.  Below is my hive.yaml.

# hive.yaml

# Elastalert rule to forward IDS alerts from Security Onion to a specified TheHive instance.

#

es_host: elasticsearch

es_port: 9200

name: TheHive - New IDS Alert!

type: frequency

index: "*:logstash-ids*"

num_events: 1

timeframe:

    minutes: 10

buffer_time:

    minutes: 10

allow_buffer_time_overlap: true

filter:

- term:

    event_type: "snort" *Didn't change this, because am thinking my suricata & Emerging Threats alerts are tied to this*

alert: hivealerter

hive_connection:

  hive_host: http://*Hive_IP_Address*

  hive_port: 9000

  hive_apikey: nAF752L1AswoI2r7z85nvmG9sAmryBJD *Got this key from the cortex part of /etc/thehive/application.conf*

hive_proxies:

  http: ''

  https: ''

hive_alert_config:

  title: '{rule[name]}' *Followed advice and removed {match[alert]}

[Wes Lambert]

You'll want to make sure that the user you are attempting to auth as using the apikey has alert creation privileges, and you may also want to try testing the rule with so-elastalert-test to ensure you are actually getting hits.

If you are getting hits with so-elastalert-test, try tailing /var/log/elastalert/elastalert_stderr.log when alerts are being generate to see if any error messages are generated when trying to connect to TheHive.

Thanks,

Wes 

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/81e67d6d-70ec-4b37-991e-1e0887c0f273%40googlegroups.com.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Justin Hussey]

Wes, thank you sir.  (I am a fan of your work w/ SO, and am very appreciative for you to reaching out directly).  I realized my API key was wrong, so I fixed that part.  Then I ran a script to continuously curl www.testmyids.com, and am not sure how to interpret the results...

I have 2x screenshots because I ran the curl script and the hive.yaml rule twice.

thank you

- Justin 

 

[Philip Robson]

You mentioned that you used the cortex api key.

As wez said. You need to create a hive user with alert rights, generate an api key and use that.

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/91eb5eb8-36fe-4fd7-8f18-8069a58b6c82%40googlegroups.com.

[Philip Robson]

Next stage will be to try and generate ids events. What does it look like under NIDS in the dashboard?

[Wes Lambert]

As Philip mentioned, you'll want to make sure you are actually generating IDS events, first and foremost.  You can also try using tcpreplay with the sample PCAPs in the /opt/samples/ directory to replay them to your sniffing interface.

After you have confirmed events are being generated and fed into Elasticsearch, try using so-elastalert-test to search back the necessary number of minutes/hours, etc, to confirm you are getting hits.  After that is successful, try letting the rule run and watch the log for any clues.

Thanks,

Wes

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAHXW2w-ySNx%2BPqp8zQ%3D8r5KWaU2iu4RT39yXBpaAcPq0rc9XVQ%40mail.gmail.com.

[Justin Hussey]

I am pretty sure I am generating successful IDS events (I see them in Squert & Kibana) though I don't quite understand the mechanism for how Elastalert sends anything to TheHive.  I am guessing when SO starts, it runs Elastalert (amongst many other things/services) which triggers the /etc/elastalert/hive.yaml which instructs ElasticSearch/Logstash to send IDS-alert events to TheHive (which is a bridged VM running on my SO).  My hive.yaml file is attached, with ***edited*** portions.  I still don't understand why TheHive has it's own Elasticsearch instance?  Should I be sending the alerts to TheHive on port 9300 (TheHive's Elasticsearch service port)? 

Attached is the so-elastalert-test .txt file.  This was taken <10 minutes after running a bash script (IDS event maker using the Zeus test pcap and curl testmyids.com).

Is there any instructional video that explains how alerts get sent to The Hive using Elastalert?  

Thank you for all the help... Sorry for so many questions.
- Jusitn

[Justin Hussey]

On Monday, December 30, 2019 at 1:40:14 AM UTC-6, Philip Robson wrote:

Next stage will be to try and generate ids events. What does it look like under NIDS in the dashboard?

On Mon, 30 Dec 2019, 07:37 Philip Robson, <mrf...@googlemail.com> wrote:

You mentioned that you used the cortex api key.

As wez said. You need to create a hive user with alert rights, generate an api key and use that.

On Mon, 30 Dec 2019, 02:52 Justin Hussey, <justin...@gmail.com> wrote:

Wes, thank you sir.  (I am a fan of your work w/ SO, and am very appreciative for you to reaching out directly).  I realized my API key was wrong, so I fixed that part.  Then I ran a script to continuously curl www.testmyids.com, and am not sure how to interpret the results...

I have 2x screenshots because I ran the curl script and the hive.yaml rule twice.

thank you

- Justin 

 

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

[Wes Lambert]

Keep in mind, so-elastalert-test will not generate alerts unless you tell it to (--alert):

Ex.

sudo so-elastalert-test -o '--alert' -r /etc/elastalert/rules/rule.yaml

Otherwise, you could tail the elastalert log for more clues.  It sounds like it is triggering on NIDS alerts from the rule test, but so-elatalert-test, as I mentioned will not generate alerts from Elastalert unless configured to do so.

Again, I would recommend tailing /var/log/elastalert/elastalert_stderr.log for more clues as you generate alerts (keeping in mind the delay from Elasticsearch ingestion/refresh, as well as the minutely run of Elastalert).

Thanks,

Wes

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/67ab21b7-966c-42eb-98af-384bd345e818%40googlegroups.com.

[Justin Hussey]

My error-log troubleshooting skills are rusty.  I seem to be getting a lot of *ConnectionRefusedError: [Errno 111] Connection refused* --though I'm unsure if it's a broken .py script or what?

I have a feeling I'll get this figured out... next year ;)

Happy new year!

- Justin

[Jan Hartmann]

HI justin,

thy to switch your hive_connections settings 

hive_connection:

  hive_host: http://192.168.1.7

  hive_port: 9000

to 

hive_connection:

  hive_host: http://192.168.1.7:9000

  hive_port: 9000

not sure why the alerter ignoring the hive_port setting.

[Wes Lambert]

Yes, Elastalert has been updated, so we will need to update the docs to reflect that.  I'll post back when updated.

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/1d62592b-c990-41f7-a3a7-291f9c65fc0f%40googlegroups.com.

[Justin Hussey]

Hey Wes,

Sorry to harp on this again, but I finally got Hive working correctly, and I really see a LOT of usefulness with it. (Response actions and/or pre-configured analytics...)

The issue was in /usr/sbin/so-elastalert-start... (needed to add "--network=thehive_default" line in order for elastalert-docker-net to talk to my hive-docker-network)

However, whenever sudo soup is ran; the so-elastalert-start will be reverted to its *correct* form, so, not sure how to keep that permanent.  Maybe a cronjob will fix it or something.

I attached the file for convenience.

so, all in all; there are 3 main files the user should care about when implementing Hive on Security Onion:
-  /etc/elastalert/rules

- thehives application.conf file

- /usr/sbin/so-elastaler-start (elastalerts docker instance)

On Tuesday, January 14, 2020 at 6:48:36 AM UTC-6, Wes wrote:

Yes, Elastalert has been updated, so we will need to update the docs to reflect that.  I'll post back when updated.

On Tue, Jan 14, 2020 at 6:38 AM Jan Hartmann <har...@gmail.com> wrote:

HI justin,

thy to switch your hive_connections settings 

hive_connection:

  hive_host: http://192.168.1.7

  hive_port: 9000

to 

hive_connection:

  hive_host: http://192.168.1.7:9000

  hive_port: 9000

not sure why the alerter ignoring the hive_port setting.

Am Dienstag, 31. Dezember 2019 23:19:08 UTC+1 schrieb Justin Hussey:

My error-log troubleshooting skills are rusty.  I seem to be getting a lot of *ConnectionRefusedError: [Errno 111] Connection refused* --though I'm unsure if it's a broken .py script or what?

I have a feeling I'll get this figured out... next year ;)

Happy new year!

- Justin

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/1d62592b-c990-41f7-a3a7-291f9c65fc0f%40googlegroups.com.