ossec-logtest returns Level 0 but still getting email alerts Level 2

[Daniel Bray]

I'm running ossec-hids-server-2.8.2-49.el6.art.x86_64 (Atomic repo)

I've updated /var/ossec/rules/local_rules.xml with the following rule:

  <rule id="100005" level="0">

    <if_sid>1002</if_sid>

    <hostname>testserver1|testserver2</hostname>

    <program_name>mip</program_name>

    <regex>HAEngine\.*INFO|HAEngine\.*WARNING|Failed to send pseudo-TCP segment frame</regex>

    <description>Ignore MIP Alerts</description>

  </rule>

I've tested the rule with:

ossec-testrule: Type one log per line.

Nov 12 13:48:50 testserver1 mip:  : HAEngine : WARNING   :     2 : Replay protection check failed

**Phase 1: Completed pre-decoding.

       full event: 'Nov 12 13:48:50 testserver1 mip:  : HAEngine : WARNING   :     2 : Replay protection check failed '

       hostname: 'testserver1'

       program_name: 'mip'

       log: ' : HAEngine : WARNING   :     2 : Replay protection check failed '

**Phase 2: Completed decoding.

       No decoder matched.

**Phase 3: Completed filtering (rules).

       Rule id: '100007'

       Level: '0'

       Description: 'Ignore MIP Alerts'

       

I've restarted everything, but the servers are still generating alerts:

OSSEC HIDS Notification.

2015 Nov 12 14:58:37

Received From: (testserver1)

Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."

Portion of the log(s):

Nov 12 14:58:36 testserver1 mip:  : HAEngine : WARNING   :     2 : Replay protection check failed

 --END OF NOTIFICATION

Can anybody shed some light on what's going on, or what I should try next?

[Santiago Bassett]

Hi Daniel,

not sure if that matters but is your local rule in the same <group name="syslog,errors,">, as rule 1002 is? You sure you restarted the manger right?

Best

--

---
You received this message because you are subscribed to the Google Groups "ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dan (ddp)]

On Thu, Nov 12, 2015 at 8:37 PM, Santiago Bassett
<santiago...@gmail.com> wrote:
> Hi Daniel,
>
> not sure if that matters but is your local rule in the same <group
> name="syslog,errors,">, as rule 1002 is? You sure you restarted the manger
> right?
>

Or are you sure the manager restarted? Most of the time when I've seen
this behavior on the list analysisd did not actually stop, so it
didn't pickup the new rules. Running `/var/ossec/bin/ossec-control
stop`, then verifying all of the processes are stopped is a prudent
course of action.

[Daniel Bray]

Yes, all my local rules are under the <group name="local,syslog,"> and yes, I made sure to stop and restart everything.

[Daniel Bray]

On Friday, November 13, 2015 at 8:51:45 AM UTC-5, dan (ddpbsd) wrote:

Or are you sure the manager restarted? Most of the time when I've seen
this behavior on the list analysisd did not actually stop, so it
didn't pickup the new rules. Running `/var/ossec/bin/ossec-control
stop`, then verifying all of the processes are stopped is a prudent
course of action.

Hmmm, not sure this would cause it, but this is what I saw:

sudo /var/ossec/bin/ossec-control stop

Killing ossec-monitord ..

Killing ossec-logcollector ..

Killing ossec-remoted ..

Killing ossec-syscheckd ..

Killing ossec-analysisd ..

Killing ossec-maild ..

ossec-execd not running ..

OSSEC HIDS v2.8 Stopped

sudo ps aux| grep ossec

ossecm    4828  0.0  0.0  10508   260 ?        S    Nov10   0:00 /var/ossec/bin/ossec-maild

So, it stopped everything, except ossec-maild. I missed this the first time, because I specifically checked for analysisd instead of just "ossec".  So, I manually killed the ossec-maild process and started everything back. I'm waiting to see if it generates an alert.

[Daniel Bray]

On Friday, November 13, 2015 at 10:33:04 AM UTC-5, Daniel Bray wrote:

 I'm waiting to see if it generates an alert.

Nope, issue remains. Very confusing.  

[dan (ddp)]

I think if you stat ossec-analysisd in debug mode it outputs the rule
IDs it loads. Is 100005 in there?

I've put the rule in /var/ossec/rules/local_rules.xml and changing the
hostnames to match my systems. Then running `echo ' : HAEngine :
WARNING : 2 : Replay protection check failed' | logger -t mip`
gives me the log in question in /var/log/messages.
And here are the results:
** Alert 1447429935.7071: - local,syslog,
2015 Nov 13 10:52:15 arrakis->/var/log/messages
Rule: 100005 (level 4) -> 'Ignore MIP Alerts'
Nov 13 10:52:14 arrakis mip: : HAEngine : WARNING : 2 : Replay
protection check failed

So it works (I changed the level so it shows up) with more than just
ossec-logtest.

[Pedro S.]

Hi Daniel,

The alerts you changed to level 0 it isn't the same that you write some lines before, isn't it?

You turn to 0 rule SID 100005 but the alert you show us has SID 1002.

For testing purposes try to deactivate (change to level 0) rule 1002 and check if it is still generating these alerts.

[dan (ddp)]

On Fri, Nov 13, 2015 at 10:59 AM, Pedro S. <sna...@gmail.com> wrote:
> Hi Daniel,
>
> The alerts you changed to level 0 it isn't the same that you write some
> lines before, isn't it?
> You turn to 0 rule SID 100005 but the alert you show us has SID 1002.
>

The log message used in the ossec-logtest example matches the log
message that is in the alert. The problem is that ossec-logtest shows
that the log message should match rule 100005, but ossec-analysisd is
matching the log message to 1002.

> For testing purposes try to deactivate (change to level 0) rule 1002 and
> check if it is still generating these alerts.
>

Don't do this. There's no reason to change that to 0. Even for
testing. I've been using OSSEC for a little while now, and I don't
think that would have ever helped with anything.

>
>
>
>
> El viernes, 13 de noviembre de 2015, 7:44:37 (UTC-8), Daniel Bray escribió:
>>
>> On Friday, November 13, 2015 at 10:33:04 AM UTC-5, Daniel Bray wrote:
>>>>
>>>> I'm waiting to see if it generates an alert.
>>>
>>>
>>
>>
>> Nope, issue remains. Very confusing.
>

[Pedro S.]

My confusion was the rule he wrote here has SID 100005 and the logtest result has SID 100007, sorry about that.

Still i'll try to create a generic rule to make sure OSSEC is loading new rules.

Anyways if Dan already has tested it, the rule is working, it should be your OSSEC is not loading the rule properly.

[dan (ddp)]

On Fri, Nov 13, 2015 at 11:16 AM, Pedro S. <sna...@gmail.com> wrote:
> My confusion was the rule he wrote here has SID 100005 and the logtest
> result has SID 100007, sorry about that.
>

You're right, I totally missed that. Now I'm wondering what 100007 is.

[Daniel Bray]

Sorry about that, it is just a simple typo. I didn't want to copy&paste the actual rule, as it had some semi-private information in it.  I copied and pasted my actual rule 100005 to a test rule 100007, so please just ignore that.  Here is the actual updated test rule I'm trying:

  <rule id="100007" level="0">

    <if_sid>1002</if_sid>

    <hostname>testserver</hostname>

    <program_name>mip</program_name>

    <regex>HAEngine\.*INFO|HAEngine\.*WARNING|Failed to send pseudo-TCP segment frame</regex>

    <description>Ignore MIP Alerts</description>

  </rule>

Here is the current log entry I'm testing:

Nov 13 16:07:17 testserver mip:  : HAEngine : WARNING   :     2 : Replay protection check failed

And here is the current results:

**Phase 1: Completed pre-decoding.

       full event: 'Nov 13 16:07:17 testserver mip:  : HAEngine : WARNING   :     2 : Replay protection check failed'

       hostname: 'testserver'

       program_name: 'mip'

       log: ' : HAEngine : WARNING   :     2 : Replay protection check failed'

**Phase 2: Completed decoding.

       No decoder matched.

**Phase 3: Completed filtering (rules).

       Rule id: '100007'

       Level: '0'

       Description: 'Ignore MIP Alerts'

However, the email alerts are still coming in. I'm trying to start some of this up in debug mode, so I can gather further information.

You received this message because you are subscribed to a topic in the Google Groups "ossec-list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ossec-list/uXdwCE64oRU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ossec-list+...@googlegroups.com.

[Pedro Sánchez de Castro]

I'm wondering.. maybe you can activate archives log (logall option) and check if the alert is working, i mean, if the alert shows on archives we will know that the issue is mail related and no about rules decoding.

[dan (ddp)]

Ok, this information is working for me as well. I have tested it on a
local install and an agent/server install (changing the hostname as
appropriate).

Is the agent name testserver? Do the hostname of the system and the
agent name match?

[Daniel Bray]

On Friday, November 13, 2015 at 12:17:09 PM UTC-5, dan (ddpbsd) wrote:

Ok, this information is working for me as well. I have tested it on a
local install and an agent/server install (changing the hostname as
appropriate).

Is the agent name testserver? Do the hostname of the system and the
agent name match?

 Yes, that all matches up. In fact, I've tried with multiple hostnames or just one hostname, and each time the logtest catches it as "Level: '0' - Description: 'Ignore MIP Alerts'"no matter what I throw at it, but the emails/alerts keep coming in as "Rule: 1002 fired (level 2)". 

I'm even waiting for the email to come in, grabbing the "Portion of the log(s):" from the email and pasting it into the logtest, and each time it comes up as "Level: '0' - Description: 'Ignore MIP Alerts'".

[dan (ddp)]

Try setting the rule to level 2

[Daniel Bray]

On Friday, November 13, 2015 at 2:03:45 PM UTC-5, dan (ddpbsd) wrote:

Try setting the rule to level 2

Doing that results in:

**Phase 3: Completed filtering (rules).

       Rule id: '100007'

       Level: '2'

       Description: 'Ignore MIP Alerts'

**Alert to be generated.

 

[dan (ddp)]

I was hoping it would help with the production use, but since it was
working for me I guess that doesn't matter. I'm pretty much stumped at
the moment.

[Daniel Bray]

On Fri, Nov 13, 2015 at 2:16 PM, dan (ddp) <ddp...@gmail.com> wrote:

I was hoping it would help with the production use, but since it was
working for me I guess that doesn't matter. I'm pretty much stumped at
the moment.

I'm running this on CentOS 6 with ossec-hids-server-2.8.2-49.el6.art.x86_64 (Atomic)

I'm curious if it's an issue with the version I'm using. 

[dan (ddp)]

I've never used the RPMs, and I don't have a centos box handy to try
them out at the moment.

[Pedro S.]

Okay try this:

Temporaly remove "<options>alert_by_email</options>" from rule 1002 on syslog_rules.xml.

Now add "<options>alert_by_email</options>" in your custom rule.

Restart OSSEC and generate the alert.

What im trying here is to stop OSSEC from sending 1002 rule email, i think that "alert_by_email" option force OSSEC to send an email alert and stop him to keep looking to reach 100007 rule. Just guessing.

Btw, sorry for my english, as you would imagine, it is not my mother language.

[Daniel Bray]

On Friday, November 13, 2015 at 2:30:24 PM UTC-5, Pedro S. wrote:

Okay try this:

Temporaly remove "<options>alert_by_email</options>" from rule 1002 on syslog_rules.xml.

Now add "<options>alert_by_email</options>" in your custom rule.

Restart OSSEC and generate the alert.

What im trying here is to stop OSSEC from sending 1002 rule email, i think that "alert_by_email" option force OSSEC to send an email alert and stop him to keep looking to reach 100007 rule. Just guessing.

Btw, sorry for my english, as you would imagine, it is not my mother language.

OK, I'm a little lost as to what this is trying to prove, but the updated settings are in place. I'm waiting for an alert to come through. 

Interesting discovery that I just noticed, within the WebUI. For every alert that comes in, I'm seeing two entries now. The new one that Dan had me update, which changes it to Alert level 1 (not sending an email). The other that I'm used to seeing is Alert level 2 (sending en email). So, it would appear that my local_rule is working, but it is not overwriting the default rules. This is very confusing. I've changed the local_rule back to level 2 along with the requested "<options>alert_by_email</options>" update.

[Daniel Bray]

On Monday, November 16, 2015 at 7:47:24 AM UTC-5, Daniel Bray wrote:

OK, I'm a little lost as to what this is trying to prove, but the updated settings are in place. I'm waiting for an alert to come through. 

With the updated alert_by_email settings, this has stopped the email alerts. I see it hitting the WebUI as alert level 2, but no emails are coming in. 

[Daniel Bray]

On Monday, November 16, 2015 at 8:28:27 AM UTC-5, Daniel Bray wrote:

With the updated alert_by_email settings, this has stopped the email alerts. I see it hitting the WebUI as alert level 2, but no emails are coming in. 

Unfortunately, with everything put back to the default settings, this issue remains. I'm seeing other issues with some filters as well. Not sure what else to do. It must be a bad install or version I'm running. 

[Pedro S.]

Hi Daniel, sorry for late response.

I don't know for real what is happening with your alerts but i'll keep giving you some advices, we'll see if we can make this work.

Maild read directly from alerts.log, search for "mail" flag and if it is present send the email, that means if your alerts is printing out into alerts.log file it should be sent by email.

So, first try to locate the alert 10005 (or 100007) in your alerts.log file.

Second, in your ossec.conf file between <email_alerts> tags include the following for better testing: <do_not_delay /> and do_not_group

It is very important that the alert your looking to be send via email actually be present on alerts.log file.

Good luck! Keep us up to date.

[Daniel Bray]

Thank you Pedro. I've actually taken a step back from this, and I'm trying to figure out why the emails are getting sent in the first place. If the default level is 7, and I haven't changed that:

  <global>

    <email_notification>yes</email_notification>

    <email_to>mye...@mydomain.com</email_to>

    <smtp_server>my.smtp.server</smtp_server>

    <email_from>os...@mydomain.com</email_from>

    <white_list>127.0.0.1</white_list>

    <logall>yes</logall>

  </global>

  <alerts>

    <log_alert_level>1</log_alert_level>

    <email_alert_level>7</email_alert_level>

  </alerts>

....then I do not understand why level 2 emails are coming in:

Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."

--

---
You received this message because you are subscribed to a topic in the Google Groups "ossec-list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ossec-list/uXdwCE64oRU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ossec-list+...@googlegroups.com.

[lostinthetubez]

On the OSSEC server, open /var/ossec/rules/syslog_rules.xml. Search for rule 1002, right there towards the top. Note the options element, which contains alert_by_email. That option tells OSSEC to ignore your email_alert_level and just send an email every time this rule matches.  As you have seen, rule 1002 is a catch-all heuristics rule that attempts to identify problems in logs based on certain keywords.

 

If you are still attempting to troubleshoot your 100007 rule and you can’t seem to figure it out using ossec-logtest, an alternative approach would be selectively eliminating rule filters to see which element is causing 100007 to not match. For example, remove the hostname element, restart ossec, then trigger the error condition and see if the filtering works. If that doesn’t help, restore the hostname element and repeat those steps with the program_name element removed, then with the regex element removed and so on. I’ve got a few 1002 filter rules in local_rules.xml and there’s no magic to it. Just have to make sure all your rule filters are setup correctly.

--

---
You received this message because you are subscribed to the Google Groups "ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+...@googlegroups.com.

[Daniel Bray]

On Wednesday, November 25, 2015 at 1:46:15 PM UTC-5, LostInThe Tubez wrote:

On the OSSEC server, open /var/ossec/rules/syslog_rules.xml. Search for rule 1002, right there towards the top. Note the options element, which contains alert_by_email. That option tells OSSEC to ignore your email_alert_level and just send an email every time this rule matches.  As you have seen, rule 1002 is a catch-all heuristics rule that attempts to identify problems in logs based on certain keywords.

Thank you, that explains why level 2 alerts are generating the emails for the "BAD_WORDS". I was under the impression that the default level of 7 was for all types of rules, but that is clear now.

I'm now left with the feeling of that is the main cause of these alerts coming in, even though I have the filters in local_rules.xml, level 2 alerts are still coming in. Even when logtest shows that it should stop. Here is another simple example of a local_rule working for logtest, but still generating email alerts .

/var/ossec/rules/local_rules.xml

  <rule id="100010" level="0">

    <program_name>accelerator</program_name>

    <regex>Update peer failed with code 22</regex>

    <description>Ignore Expand Warnings</description>

  </rule>

/var/ossec/bin/ossec-logtest

2015/11/25 19:15:23 ossec-testrule: INFO: Reading local decoder file.

2015/11/25 19:15:24 ossec-testrule: INFO: Started (pid: 6713).

ossec-testrule: Type one log per line.

Nov 25 19:11:45 x.x.x.x accelerator[4124]:     Update peer failed with code 22.

**Phase 1: Completed pre-decoding.

       full event: 'Nov 25 19:11:45 x.x.x.x accelerator[4124]:     Update peer failed with code 22.'

       hostname: 'x.x.x.x'

       program_name: 'accelerator'

       log: '    Update peer failed with code 22.'

**Phase 2: Completed decoding.

       No decoder matched.

**Phase 3: Completed filtering (rules).

       Rule id: '100010'

       Level: '0'

       Description: 'Ignore Expand Warnings'

So, even though logtest shows it will be a Level: '0', I still get an email alert as:

[lostinthetubez]

Let's keep things simple for the purposes of troubleshooting. Verify a basic rule works, then you can get as complex as you like. Try using this:

<rule id="100010" level="0">

<if_sid>1002</if_sid>
<match>Update peer failed with code 22</match>
<description>testing </description>
</rule>

Also, copy/paste the exact alert message when/if you get one. Be very careful not to replace white space if you are sanitizing the data. It will allow us to corroborate what you are seeing.

From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Daniel Bray
Sent: Wednesday, November 25, 2015 12:20 PM
To: ossec-list <ossec...@googlegroups.com>
Subject: Re: [ossec-list] ossec-logtest returns Level 0 but still getting email alerts Level 2

[Daniel Bray]

I've pretty much already done that, and have had the same results. I've tried it with <if_sid>1002</if_sid> and without, I've tried it with <match> or with <regex>. Each time, logtest catches it as a "will not alert", but the emails still continue to come in.

[dan (ddp)]

And strangely enough, this works just fine for me (ignored when fed
through logger).

Can you update to the latest OSSEC source from github and try that?

[Daniel Bray]

On Friday, November 27, 2015 at 8:16:39 AM UTC-5, dan (ddpbsd) wrote:

And strangely enough, this works just fine for me (ignored when fed
through logger).

Can you update to the latest OSSEC source from github and try that?

Updated to latest github update, and issue remains. Logtest shows Level 0, alerts come to email as level 2.

Side note: Kudos to the developers, the upgrade was VERY easy over top the existing RPM install:

git clone https://github.com/ossec/ossec-hids.git
cd ossec-hids

./install

 - You already have OSSEC installed. Do you want to update it? (y/n): y

 - Do you want to update the rules? (y/n): y

....done! Nice and quick.

[dan (ddp)]

On Mon, Nov 30, 2015 at 9:59 AM, Daniel Bray <dbra...@gmail.com> wrote:
> On Friday, November 27, 2015 at 8:16:39 AM UTC-5, dan (ddpbsd) wrote:
>>
>> And strangely enough, this works just fine for me (ignored when fed
>> through logger).
>>
>> Can you update to the latest OSSEC source from github and try that?
>
>
> Updated to latest github update, and issue remains. Logtest shows Level 0,
> alerts come to email as level 2.
>

Last idea at the moment:
Copy archives.log. Open the copy in a text editor. Find an entry you
want to test against and delete everything else.
Delete the archives.log header from your chosen entry.
Run that through ossec-logtest:
`cat copy-of-archives.log | /var/ossec/bin/ossec-logtest`

See if it still gets reported as a 0. Maybe there's some odd spacing
issue that isn't maintained when copy/pasting it.

>
> Side note: Kudos to the developers, the upgrade was VERY easy over top the
> existing RPM install:
> git clone https://github.com/ossec/ossec-hids.git
> cd ossec-hids
> ./install
> - You already have OSSEC installed. Do you want to update it? (y/n): y
> - Do you want to update the rules? (y/n): y
> ....done! Nice and quick.
>

[Daniel Bray]

On Mon, Nov 30, 2015 at 11:26 AM, dan (ddp) <ddp...@gmail.com> wrote:

Last idea at the moment:
Copy archives.log. Open the copy in a text editor. Find an entry you
want to test against and delete everything else.
Delete the archives.log header from your chosen entry.
Run that through ossec-logtest:
`cat copy-of-archives.log | /var/ossec/bin/ossec-logtest`

See if it still gets reported as a 0. Maybe there's some odd spacing
issue that isn't maintained when copy/pasting it.

Still gets reported as 0, but email is Level 2.

[Ryan Schulze]

--

Is this the only rule in your local_rules.xml that isn't working, or are all rules in your local_rules.xml not working?

[Daniel Bray]

On Mon, Nov 30, 2015 at 5:28 PM, Ryan Schulze <ry...@dopefish.de> wrote:

Is this the only rule in your local_rules.xml that isn't working, or are all rules in your local_rules.xml not working?

So far, this is the only rule that I just can't seem to stop emailing. I have other rules, and when I check them against ossec-logtest, they come back as "Level: 0", which I've correctly configured. I wait, and no email alerts come in, which is the expected behavior. In fact, I have about 12 rules filtering out various known issues. It's just this one that will not stop emailing, and wouldn't you know it, it is rather a common "alert" that comes in a few hundred times a day. 

[Daniel Bray]

On Tuesday, December 1, 2015 at 9:38:30 AM UTC-5, Daniel Bray wrote:

On Mon, Nov 30, 2015 at 5:28 PM, Ryan Schulze  wrote:

Is this the only rule in your local_rules.xml that isn't working, or are all rules in your local_rules.xml not working?

So far, this is the only rule that I just can't seem to stop emailing. I have other rules, and when I check them against ossec-logtest, they come back as "Level: 0", which I've correctly configured. I wait, and no email alerts come in, which is the expected behavior. In fact, I have about 12 rules filtering out various known issues. It's just this one that will not stop emailing, and wouldn't you know it, it is rather a common "alert" that comes in a few hundred times a day. 

Spoke too soon. I've found another rule that is not working (new rule, just enabled today):

  <rule id="100013" level="0">

    <hostname>10.10.10.10</hostname>

    <regex>decrypt: mac verify failed for connection|decrypt: replay check failed</regex>

    <description>Ignore mac verify and replay check alerts</description>

  </rule>

Here is the test log entry:

Dec  1 17:09:28 10.10.10.10 46508: *Dec  1 17:12:31.321: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=4705 spi=F1D214CD seqno=F1D214CD

ossec-logtest shows:

**Phase 1: Completed pre-decoding.

       full event: 'Dec  1 17:09:28 10.10.10.10 46508: *Dec  1 17:12:31.321: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=4705 spi=F1D214CD seqno=F1D214CD'

       hostname: '10.10.10.10'

       program_name: '46508'

       log: '*Dec  1 17:12:31.321: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=4705 spi=F1D214CD seqno=F1D214CD'

**Phase 2: Completed decoding.

       No decoder matched.

**Phase 3: Completed filtering (rules).

       Rule id: '100013'

       Level: '0'

       Description: 'Ignore mac verify and replay check alerts'

However, the emails are still getting generated with:

Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."

Portion of the log(s):

 

Dec  1 17:09:28 10.10.10.10 46508: *Dec  1 17:12:31.321: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=4705 spi=F1D214CD seqno=F1D214CD