ossec and system updates: forcing immediate syscheck

[Eric Hankins]

All,

Wanted to ping the group for thoughts/opinions on interactions between
file integrity checks and administrative operating system updates.

For example, in the case of a large-scale ossec implementation where
multiple groups are tasked with updating various pieces of the system,
i.e. one group is responsible for the OS installs themselves, and
another group handles the apps/services running on them, and they
might not always know what each other are up to. The result is a
stream of alerts that are effectively false positives, because the
file checksum changes are due to purposeful maintenance taking place.

The task to overcome this is to make ossec a functional component of
the update process, by making it play nice with scheduled system
maintenance. There are two components to this:

1) Be able to force an immediate syscheck to 're-baseline' the file
integrity checksum database immediately following whatever
admin-triggered action resulted in changes to things on the
filesystem. Ideally this 're-baseline' mode would ignore syscheck file
scanning throttles like syscheck.sleep and syscheck.sleep_after
because an administratively-triggered syscheck operation during a
scheduled maintenance window should probably run as fast as possible.

2) Be able to squelch the alerts that result from the 're-baseline'
syscheck, as everything found by this operation will likely be
purposeful and not worthy of an alert.

So, with these objectives in mind, some questions spring to mind:

Is there currently a way to force a syscheck? Will a simple agent
restart result in it beginning one? A potentially useful feature here
would be to send the agent a signal, say, SIGUSR1 to trigger this
special syscheck, ignoring any throttling options in the process.

As for alerting, it gets a little complicated. The obvious,
oversimplified method would be for the agent to simply not alert when
it executes the special 're-baseline' syscheck. But this is (equally
obviously) a horrible idea, as any intruder with a clue will simply
send SIGUSR1 or whatever should trigger the immediate syscheck and
happily have his rootkit rolled into the file integrity checksum list
without being noticed.

So, the alert squelching clearly needs to happen at the ossec server.
Extending the concept of maintenance windows, time slices in which
alerts may safely be ignored and not emailed out, to the server could
be one way to accomplish this. Preferably, this would be implemented
such that maintenance windows could be updated dynamically without
restarting the ossec server. One could do this in a custom fashion
today by writing alerts to a database, and layer some custom scripts
atop it that simply purge alerts for a host during a time period as
dictated by the maintenance window.

Anyway, just curious what the community thinks about this. Happy to
submit feature requests based on what we come up with.

best,
-e

[Mike Freemon]

I am interested in this topic as well. In my case, I have a number of servers that I have yum automatically installing the latest updates. Of course, this triggers a flood of false positives. One idea I was thinking about was to develop a yum plugin that would calculate new checksums as yum executes and forward those new checksums to ossec without triggering an alert. I will quickly confess that I have not yet looked at either the design or the code of either yum or ossec to assess whether or not this is a feasible idea or not. I look forward to this discussion.

- Mike

[Peter M. Abraham]

Greetings:

Ditto... we do a lot of security patching for our customers who are
tied to our ossec server, and it is not uncommon to get several
hundred ossec alerts within minutes to as much as within 24 hours of
the update which makes it harder to follow real ossec reports that
matter.

Thank you.

[Daniel Cid]

Hi Eric,

I understand your pain in there :) What I have done in the past (which
worked for me) was to do the
following:

1-Configured syscheck to run at a determined interval instead of a
frequency (in my case to run
every day after 9pm):

<syscheck>
<scan_time>21:00</scan_time>
<scan_on_start>no</scan_on_start>
</syscheck>

2-After that, I created a sample local rule to ignore syscheck alerts
from 9am to 7pm:

<rule id="102246" level="3">
<if_group>syscheck</if_group>
<time>9 am - 7 pm</time>
<description>Administrator initiated syscheck during business
hours.</description>
</rule>

The reason I did that was because since the scans were only initiated
after 9pm, if we ever got
one during "normal" business hours is because the administrator
initiated them and we should ignore.

3-Whenever anything got updated, we would run agent control to perform
a integrity checking
immediately (in this case on all the agents):

# /var/ossec/bin/agent_control -r -a

*If you wanted only in one agent:
# /var/ossec/bin/agent_control -r -u <agent_id>

Since we would run that during normal business hours, the alert would
be ignored and reduce tremendously
the noise ...

So, that's what we did to deal with this issue. Not the perfect
solution, but worked... Note that we didn't
completely ignore the changes, but set them to level 3, in case we
needed to review it later...

Hope it helps and gives some ideas.

--
Daniel B. Cid
dcid ( at ) ossec.net

[Reggie Griffin]

I have a few questions related to this thread.

1. I noticed that <syscheck><frequency></frequency><syscheck> exists on
the client side. Is it possible to set
the <scan_time></scantime> there?

2. Is it possible to set multiple <scan_time>s?

This is my 4th post to the list in the last few months. Hopefully this
one gets a response.

-Reggie

[Daniel Cid]

Hi Reggie,

Yes, you can set the scan time on the client side, but you can only have one per
agent. I don't think it is very useful to scan more than once per day,
but we can
add support for this in the future.

thanks,

--
Daniel B. Cid
dcid ( at ) ossec.net

[Reggie Griffin]

Daniel,

Thank you for the response. Mostly I want to be able to set a specific
time and/or time/day that will
coincide with system updates and also have a regular scan schedule.

The syscheck default is every two hours.

<!-- Frequency that syscheck is executed - default every 2 hours -->
<frequency>21600</frequency>

OSSEC is a pretty amazing tool. We have been reviewing commercial event
correlation tools that are
very expensive. As of yet we haven't found anything that I would say
outclasses OSSEC other than
some have a nicer web interface or have specific support for a few
proprietary log formats.

Keep up the good work!

[Joël Stienlet]

Hello Mike,
Sorry for digging this up, but a search on google lead me to this discussion. Is there any news on this plugin idea?
My 2 cent: there is a md5 whitelist database by Xavier Mertens. So it should be possible to use a local repository cache, compute the md5 sums of all the files in every package, and send them to the whitelist database. This method should work with most distros. Not tested yet, just an idea.
Best regards,
Joël