Create a rule to match nothing / or is triggered by no output

[Robert Micallef]

Hi,

I have been researching this for a few days and cannot find anything. I would like a rule to alert me if a program is not running.

On the agent I added a process monitoring where a command: ps -ef | grep program-name is run.

On the server side I have tried creating a rule to send an e-mail out if the output is nothing. Basically if the program is running, that command will output the PID and paths etc, and if not running it will output nothing.

I tried using <check_diff/> but the output changes frequently (even though the process keeps running), so we cannot use that as an option.

Could someone please point me in the right direction?

Thanks,
Robert

[dan (ddp)]

Try using something like:
ps auxww | grep smtpd: | sed 's/ */ /g' | cut -d ' ' -f 2,12

That should give you a list of PIDs and (in the case of smtpd, maybe
try 11 for non privsep daemons) processes.

Or use one of the many daemons designed to make sure a process is running.

> Thanks,
> Robert
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ossec-list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ossec-list+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Robert Micallef]

Thanks a lot Dan. That worked like a charm. It didn't cross my mind to grep only the PID.

I used the <check_diff /> option and:
ps -ef | grep process-name | awk '{ print $2 }'

It is working well now. Can you also please tell me what I did wrong with this rule?

I created a script to output the Memory Usage. The output will be the percentage used. Ex: 67.5%. I want an alert when it is over 80%.

I have OSSEC running the script with the following:

  <localfile>
    <log_format>full_command</log_format>
    <command>sh /var/ossec/scripts/memusage.sh</command>
    <alias>mem-usage</alias>
  </localfile>

On the server I created the following rule:

<rule id="100074" level="7" ignore="7200">
   <if_sid>530</if_sid>
   <match>ossec: output: 'mem-usage':</match>
   <regex>^8|^9|^10</regex>
   <description>High Memory Usage</description>
</rule>

To test that this is working I then created this rule:

<rule id="100075" level="7" ignore="7200">
   <if_sid>530</if_sid>
   <match>ossec: output: 'mem-usage':</match>
   <regex>^1|^2|^3|^4|^5|^6|^7</regex>
   <description>Test Memory Usage</description>
</rule>

I left it running for a few days and I see no alerts. Any idea how to fix this please?

Thanks.

[dan (ddp)]

Turn on the log all option on the server and provide us with a sample log message.

> Thanks.

[Robert Micallef]

Hi Dan,

From archives.log:

2013 Dec 27 11:31:01 (m-s-comm1) 10.152.1.227->mem-usage ossec: output: 'mem-usage':
70.85%

From alerts.log I see nothing at those timestamps.

Am I looking at the correct logs?

Thanks.

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "ossec-list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ossec-list/QeNptAfzGQQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ossec-list+...@googlegroups.com.

[dan (ddp)]

On Fri, Dec 27, 2013 at 8:41 AM, Robert Micallef <rober...@gmail.com> wrote:
> Hi Dan,
>
> From archives.log:
>
> 2013 Dec 27 11:31:01 (m-s-comm1) 10.152.1.227->mem-usage ossec: output:
> 'mem-usage':
> 70.85%
>
> From alerts.log I see nothing at those timestamps.
>
> Am I looking at the correct logs?
>

Yes, archives.log gives you a sample of the log message you are trying
to match against.
From reading the documentation or looking at the mailing list
archives, you can see that there is a header on this log message. So
the log we want to test against is:
ossec: output: 'mem-usage':70.85%

I don't have ossec available at the moment to copy/paste the whole
ossec-logtest output for you, but it's easy enough for you to recreate
on your own. The important part I want to look at first is what is
predecoded as the "log" field. This is what <match> and <regex>
entries will be looking at:

log: 'ossec: output: 'mem-usage': 79,whatever%'

From that one line we can tell that your regex is not correct, the
first character is not a number.

You can either adjust your rule to account for this, or create a
decoder to put the % in a field and check against it in your rule. I
personally think the decoder option would be easier, but I've written
a few in the past.

[Robert Micallef]

Hi Dan,

Thanks for the feedback. I cannot figure out how to get the decoder to work.

However are you sure that the actual log is being decoded as: 'ossec: output: 'mem-usage': 79,whatever%'

I tried modifying the rule as follows:

<rule id="100074" level="7" ignore="7200">
   <if_sid>530</if_sid>

   <match>ossec: output: 'mem-usage':7</match>

   <description>High Memory Usage</description>
</rule>

According to ossec-logtest the rule should be triggered, and yet it isn't.

[dan (ddp)]

On Fri, Dec 27, 2013 at 10:00 AM, Robert Micallef <rober...@gmail.com> wrote:
> Hi Dan,
>

> Thanks for the feedback. I cannot figure out how to get the decoder to work.
>

<decoder name="ossec-mem">
<parent>ossec</parent>
<prematch offset="after_parent">'mem-usage': </prematch>
<regex offset="after_prematch>^(\d+.\d+)%</regex>
<order>extra_data</order>
</decoder>

With that you should be able to include somethinglike:
<extra_data>^7</extra_data>
in your rule (untested though, so test first).

> However are you sure that the actual log is being decoded as: 'ossec:
> output: 'mem-usage': 79,whatever%'
>

Yes, I'm sure. You can verify for yourself.

> I tried modifying the rule as follows:
>
>
> <rule id="100074" level="7" ignore="7200">
> <if_sid>530</if_sid>
> <match>ossec: output: 'mem-usage':7</match>
>

Double check your spacing.

> <description>High Memory Usage</description>
> </rule>
>
> According to ossec-logtest the rule should be triggered, and yet it isn't.
>

Did you restart the ossec processes on the server after changing your rule?

[Robert Micallef]

Hi Dan,

Thanks for your help so far. I have tried searching before asking again and as far as I can see this should work.

The decoder works. I used ossec-logtest and up to phase 2, the percentage is taken in extra_data

<decoder name="ossec-mem">

<parent>ossec</parent>

  <prematch offset="after_parent">'mem-usage': </prematch>

  <regex offset="after_prematch>^(\d+.\d+)%</regex>
  <order>extra_data</order>
</decoder>

However I cannot get the rule to trigger. Below is the rule I defined. I used /d to test.

<group name="memory-usage">

<rule id="100080" level="0">

<decoded_as>ossec-mem</decoded_as>

<description>Custom Mem Usage Alerts</description>

</rule>

<rule id="100081" level="7">

<if_group>memory-usage</if_group>

<extra_data>\d</extra_data>

<description>Test_Mem_Usage</description>

</rule>

I also tried this instead of the one above:

<rule id="100081" level="7">
<if_sid>100080</if_sid>

<extra_data>\d</extra_data>

<description>Test_Mem_Usage</description>

</rule>

I can't figure out why it's not working.

Thanks again.

[dan (ddp)]

On Mon, Dec 30, 2013 at 8:13 AM, Robert Micallef <rober...@gmail.com> wrote:
> Hi Dan,
>

> Thanks for your help so far. I have tried searching before asking again and
> as far as I can see this should work.
>
> The decoder works. I used ossec-logtest and up to phase 2, the percentage is
> taken in extra_data
>
> <decoder name="ossec-mem">
> <parent>ossec</parent>
> <prematch offset="after_parent">'mem-usage': </prematch>
> <regex offset="after_prematch>^(\d+.\d+)%</regex>
> <order>extra_data</order>
> </decoder>
>
> However I cannot get the rule to trigger. Below is the rule I defined. I
> used /d to test.
>
> <group name="memory-usage">
> <rule id="100080" level="0">
> <decoded_as>ossec-mem</decoded_as>
> <description>Custom Mem Usage Alerts</description>
> </rule>
>
> <rule id="100081" level="7">
> <if_group>memory-usage</if_group>
> <extra_data>\d</extra_data>

I believe extra_data should be a number (and I don't think the field
is regex capable).

[Robert Micallef]

Hi Dan,

Ok fixed finally.

I modified the rule to have ossec as decoder not ossec-mem.

<group name="memory-usage">
<rule id="100080" level="0">

<decoded_as>ossec</decoded_as>

<description>Custom Mem Usage Alerts</description>
</rule>

<rule id="100081" level="7">
<if_group>memory-usage</if_group>

<extra_data>^7|^8|^9|^100</extra_data>

<description>Test_Mem_Usage</description>

</rule>

This way the alerts are triggered if over 80%. They are being triggered in ossec-logtest but I can't see them in alerts.log or the WebUI.

I modified the scripts on the agents to return only a single number and then modified the local_decoder which now looks like this:

<decoder name="ossec-mem">

<parent>ossec</parent>

  <prematch offset="after_parent">'mem-usage': </prematch>

  <regex offset="after_prematch>^(\d+)%</regex>
  <order>extra_data</order>
</decoder>

In the archives.log I see the following output:

2013 Dec 30 15:26:28 (m-s-comm1) 10.152.1.227->mem-usage ossec: output: 'mem-usage':
71%

In ossec-logtest I see the following output:

**Phase 1: Completed pre-decoding.

full event: 'ossec: output: 'mem-usage': 71%'

hostname: 'm-p-log1'

program_name: '(null)'

log: 'ossec: output: 'mem-usage': 71%'

**Phase 2: Completed decoding.

decoder: 'ossec'
extra_data: '71'

**Phase 3: Completed filtering (rules).

Rule id: '100081'

Level: '7'

Description: 'Test_Mem_Usage'

**Alert to be generated.

I don't get it. The alert should be triggered. Any ideas?

Thanks.

[dan (ddp)]

On Mon, Dec 30, 2013 at 9:34 AM, Robert Micallef <rober...@gmail.com> wrote:
> Hi Dan,
>

Are you receiving logs that should trigger this alert? Did you restart
the OSSEC processes on the server after putting this rule/decoder in
place?

[Robert Micallef]

Yes I restarted the process on both the server and the agent. The agent is set to send the output every 10 seconds (to test). The server is receiving the output of the command as I could see when tailing the archive.log. Could the server not generate alerts in real time?

[dan (ddp)]

On Mon, Dec 30, 2013 at 1:35 PM, Robert Micallef <rober...@gmail.com> wrote:
> Yes I restarted the process on both the server and the agent. The agent is
> set to send the output every 10 seconds (to test). The server is receiving
> the output of the command as I could see when tailing the archive.log. Could
> the server not generate alerts in real time?
>

Anything is possible. Are there other alerts in alerts.log while this
activity is going on?

[Robert Micallef]

Yes but all from log monitoring. But I checked just now, and it has been running for some time now and I still can't see an alert.  One thing I noticed is that without the custom decoder and having the rule set to match the output and alert when it sees any number by using the regex "/d" alerts are generated (although not for over 80% as I need it) which could mean that the regex is not matching the actual log. As I posted earlier, in archives.log I find the following log:

2013 Dec 30 15:26:28 (m-s-comm1) 10.152.1.227->mem-usage ossec: output: 'mem-usage':
71%

The percentage is always in a line beneath the log. I don't know if that makes a difference.

In ossec-logtest I input the following as a single line to test:
ossec: output: 'mem-usage': 71%

[dan (ddp)]

On Mon, Dec 30, 2013 at 3:29 PM, Robert Micallef <rober...@gmail.com> wrote:
> Yes but all from log monitoring. But I checked just now, and it has been
> running for some time now and I still can't see an alert. One thing I
> noticed is that without the custom decoder and having the rule set to match
> the output and alert when it sees any number by using the regex "/d" alerts
> are generated (although not for over 80% as I need it) which could mean that
> the regex is not matching the actual log. As I posted earlier, in
> archives.log I find the following log:
>
> 2013 Dec 30 15:26:28 (m-s-comm1) 10.152.1.227->mem-usage ossec: output:
> 'mem-usage':
> 71%
>
> The percentage is always in a line beneath the log. I don't know if that
> makes a difference.
>
> In ossec-logtest I input the following as a single line to test:
> ossec: output: 'mem-usage': 71%
>

Yeah, that might make a difference. ossec-logtest doesn't really work
with multi-line logs, so it probably isn't matching.
I'm not entirely sure how to account for that log silliness off hand,
I'd have to play around with it.

[Robert Micallef]

Hi Dan,

Thank you so much for your help. It was that. I changed the config on the agent to <command> from <full_command> and the output was sent in one line instead of two.

Thanks again for your help. Much appreciated.