Problem adding custom rules using full commands

[Tim Eberhard]

[My apologies for posting this to ossec-dev. I typed in the wrong
google group. This was intended for ossec-list]

All,

I've been trying to write some rules for my lab OSSEC box and test
them before we roll OSSEC out to production. I'm having some problems
writing rules when using the full command. I've tried to follow the
examples written here:
http://www.ossec.net/doc/manual/monitoring/process-monitoring.html

But it seems all my added checks/rules don't work properly.

Basic info:
-Linux - 2.6.18-128.1.6.el5
-OSSEC 2.5.1
-Stand alone server

Here is the example rule I've been trying to get to work...

Check for changes to the system start up services
ossec.conf:
<localfile>
<log_format>full_command</log_format>
<command> /sbin/chkconfig --list | grep '3:on'</command>
</localfile>

In local_rules.xml:
<rule id="510004" level="7">
<if_sid>530</if_sid>
<match>ossec: output: ‘/sbin/chkconfig </match>
<check_diff />
<description>The system start up services have changed</description>
</rule>

Upon changing the start up and removing an item I get an alert when
OSSEC notices the start up script file change..It just doesn't seem to
fire off my alert that I have configured.

OSSEC HIDS Notification.
2010 Nov 02 07:16:40
Received From: ossec->syscheck
Rule: 553 fired (level 7) -> "File deleted. Unable to retrieve checksum."
Portion of the log(s):

File '/etc/rc.d/rc0.d/K03yum-updatesd' was deleted. Unable to retrieve checksum.
--END OF NOTIFICATION

Anyone care to tell me what obvious item I'm missing? This holds true
for half a dozen items that I am using full_command for and trying to
check. Another example is below:

Check for changes to the SUID binaries
ossec.conf:
<localfile>
<log_format>full_command</log_format>
<command> find / -user root -perm -4000 -print</command>
</localfile>

In local_rules.xml:
<rule id="510005" level="7">
<if_sid>530</if_sid>
<match>ossec: output: ‘find / -user root </match>
<check_diff />
<description>SUID root binaries have been changed</description>
</rule>

Thanks for any assistance/input you can provide.
-Tim Eberhard

[dan (ddp)]

On Tue, Nov 2, 2010 at 4:13 PM, Tim Eberhard <xmi...@gmail.com> wrote:
> [My apologies for posting this to ossec-dev. I typed in the wrong
> google group. This was intended for ossec-list]
>
> All,
>
> I've been trying to write some rules for my lab OSSEC box and test
> them before we roll OSSEC out to production. I'm having some problems
> writing rules when using the full command. I've tried to follow the
> examples written here:
> http://www.ossec.net/doc/manual/monitoring/process-monitoring.html
>
> But it seems all my added checks/rules don't work properly.
>
> Basic info:
> -Linux - 2.6.18-128.1.6.el5
> -OSSEC 2.5.1
> -Stand alone server
>
> Here is the example rule I've been trying to get to work...
>
> Check for changes to the system start up services
> ossec.conf:
>  <localfile>
>  <log_format>full_command</log_format>
>  <command> /sbin/chkconfig  --list | grep '3:on'</command>

Is the space in the <command> above intentional or a paste-o? I don't
know if it will affect the output or not...

[Tim Eberhard]

That's how it sits today. I'll remove them and see if that helps things at all.

[Tim Eberhard]

I removed that extra white space and it doesn't appear to have helped
anything. Checking the log file..

2010/11/03 07:54:23 ossec-logcollector: INFO: Monitoring full output
of command(360): netstat -tan |grep LISTEN | grep -v$
2010/11/03 07:54:23 ossec-logcollector: INFO: Monitoring full output
of command(360): awk -F: '($3 == "0") {print}' /etc/$
2010/11/03 07:54:23 ossec-logcollector: INFO: Monitoring full output
of command(360): rpm -qa
2010/11/03 07:54:23 ossec-logcollector: INFO: Monitoring full output
of command(360): awk -F: '($2 == "") {print}' /etc/s$
2010/11/03 07:54:23 ossec-logcollector: INFO: Monitoring full output
of command(360): find / -user root -perm -4000 -print
2010/11/03 07:54:23 ossec-logcollector: INFO: Started (pid: 21501).

What is the next step in troubleshooting custom rules like this? I
apologize if this is a standard question.. if someone would show me
how to go about this I'll do my best to spoon feed myself :)

Thanks again for your help,
-Tim Eberhard

[dan (ddp)]

Try running logcollector in debug mode.
Try it with 1 full_command to see if you can get that working. I
recommend the ones that aren't quite so system intensive.

Here's my setup:
<localfile>
<log_format>full_command</log_format>
<command>netstat -an |grep LISTEN | grep -v '127.0.0.1'</command>
</localfile>

<rule id="510000" level="7">
<if_sid>530</if_sid>
<match>ossec: output: 'netstat -an |grep LISTEN</match>
<check_diff />
<description>Listened ports have changed.</description>
</rule>

This works for me.

[cnk]

Hey Tim,

Here are a couple more suggestions based on previously discussed
issues around command about monitoring:

-You could try enabling "logall" in ossec.conf to make sure the
command output is getting into ossec

<global>
<logall>yes</logall>
</global>

-Check if the last entry file is being created

Commnd output last entries are stored under
/var/ossec/queue/diff/<hostname>/<rule number>/

I tested your chkconfig example and it's working fine here.

Cheers,

Dale

On Wed, Nov 3, 2010 at 10:59 AM, Tim Eberhard <xmi...@gmail.com> wrote:

[Tim Eberhard]

What am I missing here?

root@ossec:/home/teberhard# /var/ossec/bin/ossec-logcollector -d
2010/11/03 09:29:47 ossec-logcollector: DEBUG: Starting ...
root@ossec:/home/teberhard# /var/ossec/bin/ossec-logcollector -dt
2010/11/03 09:29:57 ossec-logcollector: DEBUG: Starting ...
root@ossec:/home/teberhard# /var/ossec/bin/ossec-logcollector -d -t
2010/11/03 09:30:00 ossec-logcollector: DEBUG: Starting ...
root@ossec:/home/teberhard# /var/ossec/bin/ossec-logcollector -d -t -c
/var/ossec/etc/ossec.conf
2010/11/03 09:30:10 ossec-logcollector: DEBUG: Starting ...

[dan (ddp)]

Nothing that I can see. Even with -d it goes into the background.
DEBUG messages are logged to ossec.log

[Tim Eberhard]

So changing it to logall and then changing a start up item via
chkconfig..I don't see anything. Nothing in my alerts about it being
changed either.

My log file is below. Does the syscheckd error matter in this case?

root@ossec:/var/ossec/logs# tail -f ossec.log
2010/11/03 09:56:56 ossec-logcollector(1950): INFO: Analyzing file:
'/var/log/nmap-out-bird.log'.
2010/11/03 09:56:56 ossec-logcollector: INFO: Monitoring full output
of command(360): netstat -tan |grep LISTEN | grep -v '127.0.0.1'
2010/11/03 09:56:56 ossec-logcollector: INFO: Monitoring full output
of command(360): awk -F: '($3 == "0") {print}' /etc/passwd
2010/11/03 09:56:56 ossec-logcollector: INFO: Monitoring full output
of command(360): rpm -qa
2010/11/03 09:56:56 ossec-logcollector: INFO: Monitoring full output
of command(360): awk -F: '($2 == "") {print}' /etc/shadow
2010/11/03 09:56:56 ossec-logcollector: INFO: Monitoring full output

of command(360): find / -user root -perm -4000 -print

2010/11/03 09:56:56 ossec-logcollector: INFO: Started (pid: 23635).
2010/11/03 09:56:56 ossec-analysisd(1210): ERROR: Queue
'/queue/alerts/ar' not accessible: 'Connection refused'.
2010/11/03 09:56:56 ossec-analysisd(1301): ERROR: Unable to connect to
active response queue.
2010/11/03 09:56:56 ossec-analysisd: INFO: Connected to
'/queue/alerts/execq' (exec queue)
2010/11/03 09:57:57 ossec-syscheckd: INFO: Starting syscheck scan
(forwarding database).
2010/11/03 09:57:57 ossec-syscheckd: INFO: Starting syscheck database
(pre-scan).
2010/11/03 09:57:57 ossec-syscheckd: INFO: Initializing real time file
monitoring (not started).
2010/11/03 09:59:34 ossec-syscheckd: ERROR: Invalid internal state
(missing '/etc/alternatives/jaxp_parser_impl').
2010/11/03 10:01:27 ossec-syscheckd: INFO: Finished creating syscheck
database (pre-scan completed).
2010/11/03 10:01:39 ossec-syscheckd: INFO: Ending syscheck scan
(forwarding database).
2010/11/03 10:01:59 ossec-syscheckd: INFO: Starting real time file monitoring.
2010/11/03 10:01:59 ossec-rootcheck: INFO: Starting rootcheck scan.

[cnk]

Sorry I forgot to mention that you will need to look in archive.log for the logall output.

> Nothing that I can see. Even...

[dan (ddp)]

On Wed, Nov 3, 2010 at 1:04 PM, Tim Eberhard <xmi...@gmail.com> wrote:
> So changing it to logall and then changing a start up item via
> chkconfig..I don't see anything. Nothing in my alerts about it being
> changed either.
>

I don't see the chkconfig check being run in your logs.

Make sure the command runs, then change it, and make sure it runs again.

> My log file is below. Does the syscheckd error matter in this case?
>

I'd see what kind of file that is, but it shouldn't matter in this
instance. I get the same error for a "broken" symlink.

[Tim Eberhard]

Good point. I removed them all in an attempt to focus on one. I've
readded it but I still don't see any change.

So I let OSSEC syscheck run, then changed /bin/mount's permissions to
be SUID. I see it show up in archives.log during the second check. But
it never fires off an alert.

2010 Nov 03 11:57:43 ossec->find / -user root -perm -4000 -print
ossec: output: 'find / -user root -perm -4000 -print':
/usr/sbin/suexec
/usr/libexec/openssh/ssh-keysign
/usr/bin/at
/usr/bin/gpasswd
/usr/bin/passwd
/usr/bin/sudoedit
/usr/bin/sudo
/usr/bin/chsh
/usr/bin/chage
/usr/bin/crontab
/usr/bin/newgrp
/usr/lib/vmware-tools/sbin64/vmware-hgfsmounter
/bin/ping
/bin/su
/lib/dbus-1/dbus-daemon-launch-helper
/sbin/pam_timestamp_check
/sbin/unix_chkpwd
/lib64/dbus-1/dbus-daemon-launch-helper
2010 Nov 03 11:58:05 hostname->/var/log/messages Nov 3 11:58:05
hostname ntpd[2372]: sendto(XX.XX.XX.XX) (fd=20): Invalid argument
2010 Nov 03 12:03:45 ossec->netstat -tan |grep LISTEN | grep -v
'127.0.0.1' ossec: output: 'netstat -tan |grep LISTEN | grep -v
'127.0.0.1'':
tcp 0 0 0.0.0.0:3306 0.0.0.0:*
LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:*
LISTEN
tcp 0 0 :::80 :::*
LISTEN
tcp 0 0 :::22 :::*
LISTEN
tcp 0 0 :::443 :::*
LISTEN
tcp 0 0 :::8443 :::*
LISTEN
2010 Nov 03 12:03:45 ossec->awk -F\ '($3 == "0") {print}' /etc/passwd
ossec: output: 'awk -F\ '($3 == "0") {print}' /etc/passwd':
root:x:0:0:root:/root:/bin/bash
bobtest2:x:0:0::/home/bobtest2:/bin/bash
2010 Nov 03 12:03:46 ossec->rpm -qa ossec: output: 'rpm -qa':
libSM-1.0.1-3.1
<SNIP, removed RPM output>
2010 Nov 03 12:03:46 ossec->find / -user root -perm -4000 -print
ossec: output: 'find / -user root -perm -4000 -print':
/usr/sbin/suexec
/usr/libexec/openssh/ssh-keysign
/usr/bin/at
/usr/bin/gpasswd
/usr/bin/passwd
/usr/bin/sudoedit
/usr/bin/sudo
/usr/bin/chsh
/usr/bin/chage
/usr/bin/crontab
/usr/bin/newgrp
/usr/lib/vmware-tools/sbin64/vmware-hgfsmounter
/bin/ping
/bin/su
/bin/mount
/lib/dbus-1/dbus-daemon-launch-helper
/sbin/pam_timestamp_check
/sbin/unix_chkpwd
/lib64/dbus-1/dbus-daemon-launch-helper
2010 Nov 03 12:04:38 hostname ->/var/log/messages Nov 3 12:04:37
hostname ntpd[2372]: sendto(XX.XX.XX.XX) (fd=20): Invalid argument

[dan (ddp)]

I'll have to play with this tomorrow when I have access to my OSSEC setup.

Could you perhaps post your ossec.conf and associated rules? REMEMBER
to remove passwords and IPs for anything you don't want to be public
knowledge. ;)

I'll be able to plug your rules and configs into my setup to see if it
works for me.

[Tim Eberhard]

Sure thing.

Attached is the ossec.conf & local_rules.xml. I've scrubbed it a bit :)

Thanks again for all your help

[dan (ddp)]

I just wanted to apologize for not getting to this yet. I haven't
forgotten about you, just had a "string of mondays." It's at the top
of my list though, so tonight or tomorrow.

[Tim Eberhard]

That's alright Dan, you're helping me. I just greatly appreciate your
assistance. I am in no rush as I am still using ossec in the lab
trying to get all of these issues ironed out before we go to deploy
it.

Thanks again for your help,
-Tim Eberhard

[dan (ddp)]

I focused on just one (netstat) command to start with. It's working on
my centos 5.5 system.
I liked your other commands, so I'll see if I can get them to work also.

Here's the ossec.conf entry:
<localfile>
<log_format>full_command</log_format>
<command>netstat -tan |grep LISTEN | grep -v '127.0.0.1'</command>
<frequency>120</frequency>
<alias>netstat</alias>
</localfile>

The <frequency> is the minimum amount of time between runs of the
command. I added that just to help with testing. The <alias> gives it
prettier output than 'ossec: output: 'netstat -tan'.

Here's the rule that matches the entry above:
<rule id="520000" level="7">
<if_sid>530</if_sid>
<match>ossec: output: 'netstat'</match>

<check_diff />
<description>Listened ports have changed.</description>
</rule>

When this is running you can look in /var/ossec/queue/diff/AGENT_NAME/
for what's being output. There will be directories corresponding to
Rule IDs. Inside of those directories will be (hopefully) multiple
files. last-entry will be the output from the last time the command
was run, and state.EPOCH (EPOCH being the unix time) will be the
previous output.

Copy/paste can sometimes be unreliable (don't know why, different
encodings I'd guess). So make sure all of the single quotes really are
single quotes in your ossec config and rules file.

[Tim Eberhard]

So on a side note. Now that the output works.. how do I get it to show
the previous output? Here is the alert I get when I turn off/on snmpd

OSSEC HIDS Notification.
2010 Nov 08 09:20:14

Received From: ossec->/sbin/chkconfig --list | grep '3\on'
Rule: 510005 fired (level 7) -> "The system start up services have changed"
Portion of the log(s):

ossec: output: '/sbin/chkconfig --list | grep '3\on'':
acpid 0:off 1:off 2:off 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
cfenvd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cfexecd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:on 3:on 4:on 5:on 6:off
lm_sensors 0:off 1:off 2:on 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcstrans 0:off 1:off 2:on 3:on 4:on 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:off 3:on 4:on 5:on 6:off
microcode_ctl 0:off 1:off 2:on 3:on 4:on 5:on 6:off
mysqld 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off

--END OF NOTIFICATION

[Jefferson, Shawn]

Same problem I am running into... there seems to be a limit on the length of the email alert.

[Tim Eberhard]

Dan,

Using your alias method (very cool, thanks for the trick) I am now
able to get the other full commands to work.

I simply created an alias for the command as I suspected the quotes in
the command itself might be screwing up local_rules.xml's parsing.
Thanks again for your help. I greatly appreciate it.

<localfile>
<log_format>full_command</log_format>
<command>awk -F: '($3 == "0") {print}' /etc/passwd</command>
<alias>passcheck</alias>
</localfile>

<rule id="510002" level="7">
<if_sid>530</if_sid>
<match>ossec: output: 'passcheck'</match>
<check_diff />
<description>A user with the UID of 0 has been found.</description>
</rule>

OSSEC HIDS Notification.
2010 Nov 08 09:53:36

Received From: ossec->passcheck
Rule: 510002 fired (level 7) -> "A user with the UID of 0 has been found."
Portion of the log(s):

ossec: output: 'passcheck':

root:x:0:0:root:/root:/bin/bash
bobtest2:x:0:0::/home/bobtest2:/bin/bash

Previous output:
ossec: output: 'passcheck':

root:x:0:0:root:/root:/bin/bash

--END OF NOTIFICATION