Elastalert Missing Alerts (Again for me).

[Josh Silvestro]

Hello,

I had started a post many months ago about elastalert missing alerts. At the time I chalked it up to resource issues (even though I couldn't find evidence of that).

Since then I've re-deployed in a heavy setup. Alerts are still being missed and the boxes do not appear to be remotely using all of their resources.

Example: I have an alert for when an account is disabled. I see the data in Kibana within a minute or less of it happening. Elastalert never fires, doesn't throw an error. I even have the time period set to 60 minutes to catch it and never catches. The query I use in elastalert, again returns the data in Kibana.

Anything else I can check or need to change?

[Bryant Treacle]

Josh,

I have been experimenting with elastalert over the last few months and may be able to help. Can you provide an example of the rule? Also if you test the rule does it return hits and matches?

I wrote a script that will test elastalert rules and another one to walk you through the creation of new rules. Always looking for someone to test them!

Here is my GitHub page if you want to check them out.
https://github.com/bryant-treacle

Bryant

[Josh Silvestro]

Bryant,

Thanks for the response and the cool scripts! I did run your test script and it did properly generate the event I was expecting. 

Bryant

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

[Bryant Treacle]

Josh

If you look in the *:elastalert-* index in the discover tab do you see the results there?  If so are there warning triangles next to the fields.

Bryant

On Jul 26, 2018 5:00 PM, "Josh Silvestro" <josh.si...@gmail.com> wrote:

Bryant,

Thanks for the response and the cool scripts! I did run your test script and it did properly generate the event I was expecting. 

On Thu, Jul 26, 2018 at 4:26 PM, Bryant Treacle <treacle...@gmail.com> wrote:

Josh,

I have been experimenting with elastalert over the last few months and may be able to help. Can you provide an example of the rule?  Also if you test the rule does it return hits and matches? 

I wrote a script that will test elastalert rules and another one to walk you through the creation of new rules.  Always looking for someone to test them!

Here is my GitHub page if you want to check them out.
https://github.com/bryant-treacle

Bryant

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

[Josh Silvestro]

Bryant, 
Thanks for the continued follow up. No I checked yesterday before posting, and checked again now to be sure. Looking back over 24 hours, for that rule only '0' hits appears. For fun I excluded  0 hits, and no results were found. 

On Thu, Jul 26, 2018 at 5:47 PM, Bryant Treacle <treacle...@gmail.com> wrote:

Josh

If you look in the *:elastalert-* index in the discover tab do you see the results there?  If so are there warning triangles next to the fields.

Bryant

On Jul 26, 2018 5:00 PM, "Josh Silvestro" <josh.si...@gmail.com> wrote:

Bryant,

Thanks for the response and the cool scripts! I did run your test script and it did properly generate the event I was expecting. 

On Thu, Jul 26, 2018 at 4:26 PM, Bryant Treacle <treacle...@gmail.com> wrote:

Josh,

I have been experimenting with elastalert over the last few months and may be able to help. Can you provide an example of the rule?  Also if you test the rule does it return hits and matches? 

I wrote a script that will test elastalert rules and another one to walk you through the creation of new rules.  Always looking for someone to test them!

Here is my GitHub page if you want to check them out.
https://github.com/bryant-treacle

Bryant

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Bryant Treacle]

What filter are you using.  I believe that hits means that the filter matched a log and that a match happens when the the log matches the rule.

On Fri, Jul 27, 2018, 9:36 AM Josh Silvestro <josh.si...@gmail.com> wrote:

Bryant, 
Thanks for the continued follow up. No I checked yesterday before posting, and checked again now to be sure. Looking back over 24 hours, for that rule only '0' hits appears. For fun I excluded  0 hits, and no results were found. 

On Thu, Jul 26, 2018 at 5:47 PM, Bryant Treacle <treacle...@gmail.com> wrote:

Josh

If you look in the *:elastalert-* index in the discover tab do you see the results there?  If so are there warning triangles next to the fields.

Bryant

On Jul 26, 2018 5:00 PM, "Josh Silvestro" <josh.si...@gmail.com> wrote:

Bryant,

Thanks for the response and the cool scripts! I did run your test script and it did properly generate the event I was expecting. 

On Thu, Jul 26, 2018 at 4:26 PM, Bryant Treacle <treacle...@gmail.com> wrote:

Josh,

I have been experimenting with elastalert over the last few months and may be able to help. Can you provide an example of the rule?  Also if you test the rule does it return hits and matches? 

I wrote a script that will test elastalert rules and another one to walk you through the creation of new rules.  Always looking for someone to test them!

Here is my GitHub page if you want to check them out.
https://github.com/bryant-treacle

Bryant

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

[Josh Silvestro]

Here's a sample of how most of my alerts are laid out (removed alert section details) :

////////////////////////

es_host: elasticsearch
es_port: 9200
type: frequency
generate_kibana_link: false

name: "An account has been locked out"
alert_subject: "Account Lockout: {0}"
alert_subject_args:
- "event_data.TargetUserName"

index: "*:logstash-beats*"
num_events: 1
timeframe:
minutes: 60
filter:
- query:
query_string:
query: "event_id: \"4740\""

alert:
- "email"

////////////////////////

FYI - Originally I had this for example set to "terms" but after having issues thought I'd try query, same results. Some searches are a bit more lengthy as well. This was just the first I grabbed.

[Josh Silvestro]

Circling back on this. I cut out half of my alerts, which sucks, and I'm still having some misses on alerts.

We started piping AV logs in to SO, as our solution is one i love, but apparently does not alert on even HIGH level threats if they're handled -_- so I have data in SO, and created an elastalert to alert on any HIGH events so we can review.

But this I received one alert and thought great :) went in to investigate and found out there were 3 other alerts that never triggered. FYI - I did in alert yaml set a query_key: "computer_name" so in theory it shouldn't have silenced the other 3.

Is there anything I can do to increase elastalert resources if that's what we think the issue is? Coming from a heavy Splunk background we never really had any resource issues or limitations on number of alerts that could be in place, so this feels odd to me.

[Josh Silvestro]

And to add on to this, I did just go back and run that so-elastalert-test-rule script and ALL 4 events did hit! So I'm not sure what the back end cause is?

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Philip Robson]

Hi Josh,

I have noticed some oddity's with missed alerts through my months of testing, I am creating more rules so back to testing. A rule I have been playing with stopped working partially, I modified the rule to set the buffer_time to 2 minutes as the default in SO seems to be 1 minute (I did find the config file for this at one point). Once I changed it the issue I had cleared, this is just 1 test and will look to test further.

Also just noticed the mention of so-elastalert-test-rule. I will have to test this as I was looking for the standard test command.

Thanks
Phil

[Philip Robson]

Hi Josh,

I think it depends on the timescale of the events. I played with realert: 0 to try and get every event but even if the buffer is 1 minute and I get 3 events in that time only 1 will come through.

Now I believe there is something that can be added to show all data in the alert. I cannot remember what that is at the moment.

I will try and feed back any findings.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

[Josh Silvestro]

Yea - to my understanding it shouldn't work that way and should alert on each unique event.

On Mon, Dec 10, 2018 at 1:43 PM 'Philip Robson' via security-onion <securit...@googlegroups.com> wrote:

Hi Josh,

I think it depends on the timescale of the events. I played with realert: 0 to try and get every event but even if the buffer is 1 minute and I get 3 events in that time only 1 will come through.

Now I believe there is something that can be added to show all data in the alert. I cannot remember what that is at the moment.

I will try and feed back any findings.

On Mon, 10 Dec 2018, 18:17 'Philip Robson' via security-onion <securit...@googlegroups.com wrote:

Hi Josh,

I have noticed some oddity's with missed alerts through my months of testing, I am creating more rules so back to testing.  A rule I have been playing with stopped working partially, I modified the rule to set the buffer_time to 2 minutes as the default in SO seems to be 1 minute (I did find the config file for this at one point).  Once I changed it the issue I had cleared, this is just 1 test and will look to test further.

Also just noticed the mention of so-elastalert-test-rule.  I will have to test this as I was looking for the standard test command.

Thanks
Phil

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Dustin Lee]

Hi,

I've recently experienced this as well and it required me to read a bit more about how Quentin implemented the run_every, timeframe, and buffer_time logic. More here: https://github.com/Yelp/elastalert/issues/805

Also, Security Onion has run_every set to 1 minute by default but can be altered if desired.

- Dustin

[Bryant Treacle]

Josh, 

      This is really interesting especially since  so-elastalert-test-rule returned all the alerts.  One thing I was thinking about for a while was the impact of having a large number of elastalert rules querying a large data set every minute.  I wrote a script to find kerberos golden tickets on a large data  set and it take over a minute to parse through all the data (700 million records).  I am curious if elastarert is running the rule again before all the results from the previous queries are returned and somehow elastalert has incomplete data sitting in memory.  In the link that Dustin provided, the fix was to change elastatert to only run every 15 minutes which would give elastatert enough time to process all the logs and properly alert.  Have you tried extending the run_every setting in the config.yaml out 15 minutes? 

Bryant

[Wes Lambert]

To add, we have also suggested setting buffer_time here to the desired value:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Elastalert#timeframe

Thanks,

Wes

You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Bryant Treacle]

Josh,

    I hope all is going well.  In regard to your issue of not seeing all the alerts populate in Kibana.  Have you tried to set your realert option to zero. I found this today reading of the Elastalert  docs and wondering if it could help.

realert:  It defaults to one minute, which means that if ElastAlert is run over a large time period which triggers many matches, only the first alert will be sent by default. If you want every alert, set realert to 0 minutes. (Optional, time, default 1 minute)

I applied this to a whitelist and every event that matched was sent to the elastalert index.  The only caution is that every record that was a match was sent to the index. I wrote a bad whitelist for testing and had 10K+ records send to the elastalert index.  On the flip side, I now have the record for each event that matched.

Bryant  

[Josh Silvestro]

Interesting, I'll have to give that a go. That being said. I have issue where I get no alert at all. So realert shouldn't be an issue in this case.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

[Suat Toksoz]

Hi,

Is this email group is still active?

Thanks

On Wednesday, December 12, 2018 at 11:46:34 PM UTC+3, Bryant Treacle wrote:

Josh, 

      This is really interesting especially since  so-elastalert-test-rule returned all the alerts.  One thing I was thinking about for a while was the impact of having a large number of elastalert rules querying a large data set every minute.  I wrote a script to find kerberos golden tickets on a large data  set and it take over a minute to parse through all the data (700 million records).  I am curious if elastarert is running the rule again before all the results from the previous queries are returned and somehow elastalert has incomplete data sitting in memory.  In the link that Dustin provided, the fix was to change elastatert to only run every 15 minutes which would give elastatert enough time to process all the logs and properly alert.  Have you tried extending the run_every setting in the config.yaml out 15 minutes? 

Bryant

On Wed, Dec 12, 2018 at 1:44 PM Dustin Lee <dle...@gmail.com> wrote:

Hi,

I've recently experienced this as well and it required me to read a bit more about how Quentin implemented the run_every, timeframe, and buffer_time logic. More here: https://github.com/Yelp/elastalert/issues/805

Also, Security Onion has run_every set to 1 minute by default but can be altered if desired.

- Dustin

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/MKMUU6TvzqU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to securit...@googlegroups.com.

[Chris Morgret]

Hi  Suat,

It is. If you have a question about something please start a new email please.

Thanks,

Chris

You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/154695a9-8cfa-48b3-bbd6-3e0307b277f9o%40googlegroups.com.