Adding Custom Rules (AlienVault OTX)

[arnydo]

Good afternoon,

Recently deployed SELKS and really happy with it.

I am going through the setup of integrating AlienVault OTX pulses with Suricada.

I have updated the configs as described but I am not seeing them actually being loaded.

Is there something special that needs to be done?

https://github.com/AlienVault-OTX/OTX-Suricata

[Peter Manev]

On Tue, Jun 27, 2017 at 9:51 PM, arnydo <arn...@gmail.com> wrote:
> Good afternoon,
>
> Recently deployed SELKS and really happy with it.
>

Thanks for trying SELKS out!

> I am going through the setup of integrating AlienVault OTX pulses with
> Suricada.
>
> I have updated the configs as described but I am not seeing them actually
> being loaded.

Which configs did you update?

For example:
You can use Scirius to do that -
https://github.com/StamusNetworks/scirius#creating-source by using
"Individual signature files"

>
> Is there something special that needs to be done?
>
> https://github.com/AlienVault-OTX/OTX-Suricata
>
>

> --
> IRC: Let's talk about SELKS on Freenode IRC on the #SELKS channel
> Wiki: https://github.com/StamusNetworks/SELKS/wiki
> GitHub: https://github.com/StamusNetworks/SELKS
> Blog: https://www.stamus-networks.com/theblog/
> Twitter: @StamusN
> g+: Stamus Networks
> ---
> You received this message because you are subscribed to the Google Groups
> "SELKS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to selks+un...@googlegroups.com.
> To post to this group, send email to se...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Regards,
Peter Manev

[arnydo]

Pevma,

I appended suricata.yaml with (sorry for the bad crop):

There were no apparent errors with the config but nothing showed up in the GUI or logs showing that the rules were enabled.

[Peter Manev]

On Tue, Jun 27, 2017 at 11:13 PM, arnydo <arn...@gmail.com> wrote:
> Pevma,
>
> I appended suricata.yaml with (sorry for the bad crop):
>

I dont see any file appended.
However in SELKS you add rules via Scirius (as i was mentioning in my
previous post) not via the suricata.yaml directly since Scirius is the
ruleset manager for the distro.

[arnydo]

Good morning,

Since these files are updated regularly from the API scripts from OTX, how can I set it to always look in the "/etc/suricata/rules/otx_iprep.rules" file?

On Wed, Jun 28, 2017 at 12:16 AM, Kyle Parrish <arn...@gmail.com> wrote:

> I see OTX is referenced in the article posted on SANS but it doesn't go into
> detail...
>
> https://www.sans.org/reading-room/whitepapers/critical/continuous-monitoring-build-world-class-monitoring-system-enterprise-small-office-home-37477
>
> Get Outlook for iO
>
>
>
> On Tue, Jun 27, 2017 at 5:37 PM -0400, "Kyle Parrish" <arn...@gmail.com>
> wrote:
>
>> Okay. How would I go about adding the source if I already have the .rules
>> file in the /etc/suricata/rules folder? I also have some custome categories
>> in there as well. These are generated by Calling the Alienvault OTX API.
>>
>> From what I read it looks like you have to provide a uri for the file when
>> adding via Scirius.

You can do either way - url or file.
First you need to create a new/additional source - Click on the
"Source" tab -> click "Add"

For the upload you can choose "url" or "upload" if you have the file
on your machine you can choose "upload" -> browse to the file , select
it then choose if you would like to add it to the current ruleset and
click "Submit".  Then you can go to the "Suricata" tab select "Ruleset
actions" and update the ruleset on  SELKS.

[Peter Manev]

On Wed, Jun 28, 2017 at 2:11 PM, arnydo <arn...@gmail.com> wrote:
>
> Good morning,
>
> Since these files are updated regularly from the API scripts from OTX, how can I set it to always look in the "/etc/suricata/rules/otx_iprep.rules" file?

Is there an URL that they are available at? or is it only local run scripts?

>
> On Wed, Jun 28, 2017 at 12:16 AM, Kyle Parrish <arn...@gmail.com> wrote:
>
> > I see OTX is referenced in the article posted on SANS but it doesn't go into
> > detail...
> >
> > https://www.sans.org/reading-room/whitepapers/critical/continuous-monitoring-build-world-class-monitoring-system-enterprise-small-office-home-37477
> >
> > Get Outlook for iO
> >
> >
> >
> > On Tue, Jun 27, 2017 at 5:37 PM -0400, "Kyle Parrish" <arn...@gmail.com>
> > wrote:
> >
> >> Okay. How would I go about adding the source if I already have the .rules
> >> file in the /etc/suricata/rules folder? I also have some custome categories
> >> in there as well. These are generated by Calling the Alienvault OTX API.
> >>
> >> From what I read it looks like you have to provide a uri for the file when
> >> adding via Scirius.
>
> You can do either way - url or file.
> First you need to create a new/additional source - Click on the
> "Source" tab -> click "Add"
>
> For the upload you can choose "url" or "upload" if you have the file
> on your machine you can choose "upload" -> browse to the file , select
> it then choose if you would like to add it to the current ruleset and
> click "Submit". Then you can go to the "Suricata" tab select "Ruleset
> actions" and update the ruleset on SELKS.
>
>

[arnydo]

There is not a url available. These script are locally run and pull from an API to Alienvault based on my personal OTX account.

[Peter Manev]

On Wed, Jun 28, 2017 at 3:26 PM, arnydo <arn...@gmail.com> wrote:
> There is not a url available. These script are locally run and pull from an
> API to Alienvault based on my personal OTX account.

So then you should use the described process and make sure the rules
are always written in the same file. Then the update cronjob on SELKS
should pick that up daily.

[arnydo]

Which process was that? I don't think you verified which method should be used to add a new source pointing to a file already residing in /etc/suricata/rules...

[Peter Manev]

On Wed, Jun 28, 2017 at 4:01 PM, arnydo <arn...@gmail.com> wrote:
> Which process was that? I don't think you verified which method should be
> used to add a new source pointing to a file already residing in
> /etc/suricata/rules...

[arnydo]

If I upload it via the web GUI, will any updates to the .rules file in the /etc/suricata/rules (from the OTX API) be reflected? Or will I have to re-upload each time there is an update (daily/hourly/etc)?

[arnydo]

I apologize if I am missing something...but I am getting errors following your steps.

If possible, can you review the steps outlined to setup integration with Alienvaults OTX API and let me know if we are going about this correctly.

https://github.com/AlienVault-OTX/OTX-Suricata

During setup, this is what it requests to be added to the Suricata configs:

root@SELKS01:~/OTX-Suricata/otx-suricata# python suricata.py --key my-key-here -dd /etc/suricata/rules/
['suricata.py', '--key', 'my-key-here', '-dd', '/etc/suricata/rules/']
Wrote related iprep rules to otx_iprep.rules
Wrote 2304 IPv4 & IPv6 to /etc/suricata/rules/reputation.list
========================================
To leverage generated files, enable the suricata iprep feature in suricata.yaml
A default configuration for iprep with these rules can be enabled by appending the following to suricata.yaml
========================================
NOTE: Please read the docs to adapt for your environment
========== Start YAML Snippet ==========
reputation-categories-file: /etc/suricata/rules/categories.txt
default-reputation-path: /etc/suricata/rules/
reputation-files:
 - reputation.list
rule-files:
 - /etc/suricata/rules/otx_iprep.rules
==========  End YAML Snippet  ==========
Wrote 392 md5 hash files to /etc/suricata/rules/
Wrote 392 rules to /etc/suricata/rules/otx_file_rules.rules
========================================
To leverage generated files, enable the suricata file feature in suricata.yaml
A default configuration for the file feature with these rules can be enabled by append the following to suricata.yaml
The following was a snippet from 'http://jasonish-suricata.readthedocs.org/en/latest/file-extraction/file-extraction.html'
========================================
NOTE: Please read the docs to adapt for your environment
========== Start YAML Snippet ==========
- file-log:
    enabled: yes
    filename: files-json.log
    append: yes
    force-magic: no
    force-md5: no
    waldo: file.waldo
==========  End YAML Snippet  ==========

[Peter Manev]

On Thu, Jun 29, 2017 at 3:01 PM, arnydo <arn...@gmail.com> wrote:
> I apologize if I am missing something...but I am getting errors following
> your steps.
>
> If possible, can you review the steps outlined to setup integration with
> Alienvaults OTX API and let me know if we are going about this correctly.
>
> https://github.com/AlienVault-OTX/OTX-Suricata
>

Actually i was just discussing that with Eric.
It would be best if we can do that (add local files from SELKS from
the command line - which is not possible now) like we can add using
url.
For example:
https://github.com/StamusNetworks/SELKS/blob/SELKS4-dev/staging/config/hooks/chroot-inside-Debian-Live.chroot#L167

Would you please open a feature request for that on Scirius please -
https://github.com/StamusNetworks/scirius/issues

Thank you