Force/burst output on apt-get and software updates

[Gerard Petersen]

Hi all,

I've been looking for an easier way to relate output to my own updates. Assume I do a system and/or software update in de morning and the regular 22 hour syscheck cycle invokes in the afternoon.I get a lot of changes I need to interpret, but shouldn't because they are known. Although false positives, it does consume my time.

I've been thinking about stopping the daemons, running the updates, and while restart them, forcing the checks. This would result in controlled output bursts, but also in increased risk due to the daemons temporarly not watching the system ... So, not really a solution.

Setting most parts of the FS to 'realtime' monitoring. This way Ossec knows straight away, but you get a performance hit. Also not ideal.

Anybody any suggestions on how to handle this?

Thanx a lot!

Kind regards,

Gerard.

[Jeremy Rossi]

>I've been looking for an easier way to relate output to my own updates.
>Assume I do a system and/or software update in de morning and the regular
>22 hour syscheck cycle invokes in the afternoon.I get a lot of changes I
>need to interpret, but shouldn't because they are known. Although false
>positives, it does consume my time.
>
>I've been thinking about stopping the daemons, running the updates, and
>while restart them, forcing the checks. This would result in controlled
>output bursts, but also in increased risk due to the daemons temporarly not
>watching the system ... So, not really a solution.

Agreed, stopping is not the correct answer. But have you thought about
forcing syscheck just after you have completed the updates?

http://ossec-docs.readthedocs.org/en/latest/manual/syscheck/#id1

This should bring the events closer together and *Maybe* easier to
dealwith.

>Setting most parts of the FS to 'realtime' monitoring. This way Ossec knows
>straight away, but you get a performance hit. Also not ideal.

The performance of realtime is normally less if you avg it out over the
full day. This is due to OSSEC making use of inotify (in kernel method
of asked when have files changed) to tell ossec when to do work.

>
>Anybody any suggestions on how to handle this?

Hard problem for ossec and anything file intergrity monitoring tool.

-Jeremy Rossi

[Gerard Petersen]

Dear Jeremy,

Sorry for the late reply (totally forgot about this post), and thanx for the reply. I Do run a "/var/ossec/bin/agent_control -r -a" after an apt-get update. I still seem to get hits on other moments in time. Looking at the binary files that change, they are related to the last upgraded packages, cant be coincidence. 

Maybe I need to look at my agent configurations and check whether or not they are correct.

Thanx again.

Kind regards,

Gerard.

[Michael Starks]

On 06/16/2014 05:49 AM, Gerard Petersen wrote:
> I've been thinking about stopping the daemons, running the updates, and
> while restart them, forcing the checks. This would result in controlled
> output bursts, but also in increased risk due to the daemons temporarly
> not watching the system ... So, not really a solution.

One possibility is to clear the syscheck database just before the
updates so the next round creates a baseline.

> Setting most parts of the FS to 'realtime' monitoring. This way Ossec
> knows straight away, but you get a performance hit. Also not ideal.

I have not noticed any performance impact and I have used OSSEC on
hundreds of systems. Are you seeing otherwise?

[Gerard Petersen]

Dear Michael,

Thanx for the reply. Cleaning out the database might be an idea, I'll give that a try. Is it acceptable operating procedure for ossec to clean out the database without stopping the agents or is there temporary agent shutdown involved?

One more thing I noticed. All hits that are not around the time of (re)starting the syscheck are coming from the ossec server. The ossec server undergoes the same updates as the systems running the agents. The only conclusion I can draw from this is that "agent_control -r -a" does not initiate the agent on the servers, which would seem odd. Can you confirm this?

On the performance part. I do see quite some CPU spikes when I update parts of the web framework that is monitored realtime. I understand the inode concept. It's just I find them rather high. I'll have a look at load indicator which is more of a averaged realtime value as opposed to the realtime cpu spikes. What indicators do you use to confirm for yourself that there is no (or little) load increase?

Thanx a lot.

Kind regards,

Gerard.

[Gerard Petersen]

In "The only conclusion I can draw from this is that "agent_control -r -a" does not initiate the agent on the servers, which ..."

The word "servers" .. should be singular instead of plural :)

[Michael Starks]

On 06/28/2014 05:18 AM, Gerard Petersen wrote:
> Is it acceptable operating procedure for ossec to clean
> out the database without stopping the agents or is there temporary agent
> shutdown involved?

Yes. Just run ./bin/syscheck_control -u <id>

> One more thing I noticed. All hits that are not around the time of
> (re)starting the syscheck are coming from the ossec server. The ossec
> server undergoes the same updates as the systems running the agents. The
> only conclusion I can draw from this is that "agent_control -r -a" does
> not initiate the agent on the servers, which would seem odd. Can you
> confirm this?

I just looked at a couple of Linux agents and it worked for me. Keep in
mind that syscheck is both scheduled and real-time, depending on how it
is set up on your systems. So you could have alerts that are
instantaneous and every x hours.

> On the performance part. I do see quite some CPU spikes when I update
> parts of the web framework that is monitored realtime. I understand the
> inode concept. It's just I find them rather high. I'll have a look at
> load indicator which is more of a averaged realtime value as opposed to
> the realtime cpu spikes. What indicators do you use to confirm for
> yourself that there is no (or little) load increase?

No one yells at me. :) Seriously, while there are spikes I have noticed
that they are not sustained. I guess my applications are not that
sensitive to this kind of thing.

[Gerard Petersen]

Hi Michael,

Thanx again for the response. As long as I don't answer any phone calls there's no yelling here either ;) ... My servers are running several Wordpress stacks. Allthough temporarily, load can influence website response times.

In regards to the update question. When an update is always accompanied by a reset, alerts should not occur other then during my updates (or with malicious behaviour ofcourse).

Been thinking about the order of actions to take for a proper update cycle.

- clearout database

- update files

- init syscheck

Curious what happens. I'll give this a try.

Kind regards,

Gerard.

[Julien T]

Hi the list,

My understanding is clearing/init syscheck is on the server or could it be initiated on client side?

it seems first as there is no syscheck_control on agent...

Because in the second case, on debian/ubuntu, you can use apt Pre/Post-invoke [1].

I don't know if there is an equivalent for rpm based system (%post seems to imply editing specfile) or non-linux but that makes thing a lot easier. I use this system with Aide HIDS and it works very well.

Would be nice if the client could notify server, an update is pending/done, do the previous procedure.

Cheers,

Julien

[1] https://www.debian.org/doc/manuals/securing-debian-howto/ch4.fr.html#s4.10

http://wiki.itadmins.net/doku.php?id=debian:apt-xtra