Bro enhancement - Alienvault OTXv2

[Ron Rosson]

Has anyone looked into what is needed to use Alienvaults OTXv2 Bro Connector (https://github.com/AlienVault-Labs/OTX-Apps-Bro-IDS). I think this would be an awesome addition to SO.

TIA
-Ron

[Doug Burks]

Hi Ron,

I'm not sure why we would need this in Security Onion. We already
have Bro and its logs already go to ELSA.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Ron Rosson]

Doug,

This connector brings in the IOC's in bro format to be used with bro's intel plugin. Keep in mind OTX stands for "Open ThreatIntel Exchange"

-Ron

[riema...@gmail.com]

Ron,

I'm kind of a newb at this so PLEASE feel free to correct anything I get wrong...

I think I've got everything working, so here's what I did:

cd /opt/bro/share/bro/site
sudo git clone https://github.com/AlienVault-Labs/OTX-Apps-Bro-IDS.git
cd /OTX-Apps-Bro-IDS
sudo vim config.cfg

[change the following configurations:
base_path=/opt/bro/share/bro/site
api_key=<your_api_key_here>]

sudo python otxv2-bro.py first_run

That should get everything pulled in and configured, assuming you already have a an API key and have selected whatever pulses you subscribe to. Check your local.bro file to see that it is loading the connector. At the bottom of /opt/bro/share/bro/site/local.bro, there should be a line like this:

@load OTX-Apps-Bro-IDS

From there, restart bro (don't use "sudo broctl restart," per Doug's comment here: https://groups.google.com/forum/#!searchin/security-onion/critical$20stack/security-onion/ZBArSyuzbpM/86jQPEKd3OMJ):

sudo nsm_sensor_ps-restart --only-bro

From there, I think it's just a matter of creating a cron job to run the following at your desired interval: python otxv2-bro.py check_new. Here's how I set that up (again, suggestions welcome!):

I wrote the following to a shell script that I put in the \OTX-Apps-Bro-IDS directory (and did sudo chmod +x OTX_update.sh):

####################################################
#!/bin/sh
#script to monitor Security Onion activity for the past hour to alert on inactivity

SCRIPT_PATH="otxv2-bro.py"
PYTHON="/usr/bin/python"

DATE=`date`
REPORT=/root/so-OTX-update.txt

echo "Last run: $DATE" > $REPORT

cd /opt/bro/share/bro/site/OTX-Apps-Bro-IDS/ &&
$PYTHON $SCRIPT_PATH check_new >> $REPORT

if sed -n '2p' $REPORT | grep -qP [1-9]+; then
nsm_sensor_ps-restart --only-bro
echo "Bro restarted..." >> $REPORT
else
echo "No new pulses found..." >> $REPORT
####################################################

I also made the otxv2-bro.py file executable with chmod (same as above), but I don't think it was necessary.

Then I created a cron job to run the shell script every 15 minutes and saved it as /etc/cron.d/OTX_update. Here's the shell script:

####################################################
# /etc/cron.d/OTX_update
#
# fetch new OTX pulses for /opt/bro/share/bro/site/OTX-Apps-Bro-IDS

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

00,15,30,45 * * * * root /opt/bro/share/bro/site/OTX-Apps-Bro-IDS/OTX_update.sh
####################################################

I just now got it configured, so no hits yet to verify this is working as intended. I can see in my Bro logs that the script is loaded though, and the Python script updated the __load__.bro file with the addresses of the .intel files that were written to \pulses. So short of trying to look through Bro debug logs, I think this should work.

Good luck.

-de

[Seth Hall]

> On Sep 1, 2015, at 5:06 PM, riema...@gmail.com wrote:
>
> I just now got it configured, so no hits yet to verify this is working as intended. I can see in my Bro logs that the script is loaded though, and the Python script updated the __load__.bro file with the addresses of the .intel files that were written to \pulses. So short of trying to look through Bro debug logs, I think this should work.

The only side issue with the way they’ve created this is that Bro needs to be restarted to incorporate new pulses. Existing pulses that are updated should have their data automatically pulled into Bro.

It looks to me like you did everything correct. :)

.Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

[Brian Haugli]

Just got this running and it's awesome! Great work!

b

[Seth Hall]

There is another option for integrating OTX data into Bro now. Stephen Hosom just posted a script he wrote to manage the pulses he subscribes to. It’s nice in that it doesn’t attempt to modify your Bro scripts and you can load new pulses without needing to restart Bro (all data is aggregated into a single file).

https://github.com/hosom/bro-otx

.Seth

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.

> For more options, visit https://groups.google.com/d/optout.

--

[Brian Haugli]

I'll say this is a much cleaner way to do it. Tested and works with new SO release as well.

Brian

[Stafford Waltho]

Have tried to install this on my instance of security onion. Everything went ok and the pulses i subscribe to have been downloaded. However, nothing is showing in my intel.log, even when I send traffic to an IP in one of the pulses (8.8.8.8).

I have Critical Stack Intel on Security Onion up and running and is working ok, the intel.log is writing ok but not for OTX ?

Any ideas ?

[Wes]

Have you checked to see if the AV script is loaded in loaded_scripts.log?

Thanks,
Wes

[Curtis]

Has anyone else had success with this? I'm trying to set it up as well--I've gotten close, but not entirely successful. It may be that I just don't fully understand the route the IOC's take from my feed to Bro, and ultimately viewable in ELSA. (If you could provide this, that may be all I need).

First, I followed the steps laid out by riema, and everything seemed to work well; the sensor was pulling in my indicators and populating "/opt/bro/share/bro/site/OTX-Apps-Bro-IDS/pulses," and even placing them in __load__.bro. But when I checked for my indicators in the ELSA Intel dashboard, there were indicators from a different OTX feed than my own. For some reason, it seemed the IOCs weren't transferring to the intel.dat file.. is this the file that ELSA references when searching for these indicators?

Anyway, I started over and tried Stephen Hosom's solution--the setup went well, but I couldn't see my IOCs in __load__.bro, nor intel.dat. I ran "sudo broctl check" and it gives me an error:

<bro scripts failed.
fatal error in /opt/bro/share/bro/site/local.bro, line 104: can't find site/otx>

Hosom's installation instructions say to add "@load site/otx" into the local.bro file... so I'm stuck here.

I seemed to have more success with riema's approach, but couldn't make my IOC's/pulses searchable. I'll probably revert back to that method for now and see if I can get it working, but if anyone has any input, I'm all ears.

Thanks,
Curt

[Brian Haugli]

For the local.bro edit, use the full path after @load.

Also, this writes an otx.dat for the iocs, not to the intel.dat file. It will reside in the bro-otx folder.