syscheckd high cpu usage

[Jefferson, Shawn]

Hi,

 

I have OSSEC installed on Ubuntu 10.04.2 LTS 64-bit, and the syscheckd process is taking a lot of CPU time, and has for the past couple of days.  I haven’t seen this behaviour on other installations, but on three of these systems that are configured similiarly.  Any suggestions on where to look?  Rootkitcheck?

 

You can see this one has been running syscheck for days…

 

2011/05/05 20:05:21 ossec-syscheckd: INFO: Starting syscheck scan (forwarding database).

2011/05/05 20:05:21 ossec-syscheckd: INFO: Starting syscheck database (pre-scan).

2011/05/06 22:21:01 ossec-agentd: INFO: Event count after '20000': 4664877->3811296 (81%)

2011/05/08 06:35:39 ossec-agentd: INFO: Event count after '20000': 4195430->3534200 (84%)

2011/05/09 15:46:25 ossec-agentd: INFO: Event count after '20000': 4407799->3661232 (83%)

2011/05/11 01:30:02 ossec-agentd: INFO: Event count after '20000': 4909642->3973976 (80%)

 

 

 

Shawn

 

 

[Jeremy Lee]

I think I recall seeing issues with syscheck when files grow too large and too fast. Are you monitoring a particular file that would fall into both those categories?

[Tanishk Lakhaani]

Else u can do one more thing... Configure the agent to perform the syscheck scan off-production hours at a particular time instant.. At times it happens that due to heavy usage of the system or heavy load on the system, syscheck takes a lot of time, resulkting in high CPU utilisation.

Rgds
Tanishk

Sent from BlackBerry® on Airtel

---

From: Jeremy Lee <jpl...@gmail.com>

Sender: ossec...@googlegroups.com

Date: Wed, 11 May 2011 10:29:37 -0700

To: <ossec...@googlegroups.com>

ReplyTo: ossec...@googlegroups.com

Subject: Re: [ossec-list] syscheckd high cpu usage

[dan (ddp)]

Please provide some information about how you have these systems
configured (especially syscheck settings), and what they do.

> Shawn
>
>

[Jefferson, Shawn]

This has been done.  Syscheck starts dutifully at the time I specified and then runs for 4 days straight.

[Jefferson, Shawn]

Hi,

Two of these systems run Snort, the third runs some packet capturing software.

2011/05/11 10:07:19 ossec-syscheckd: INFO: Started (pid: 11418).
2011/05/11 10:07:19 ossec-rootcheck: INFO: Started (pid: 11418).
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/bin'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/boot'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/dev'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/etc'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/lib'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/lib64'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/sbin'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/usr/bin'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/usr/lib'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/usr/lib64'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/usr/sbin/'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/usr/share'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/usr/local'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/var/ossec/bin'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/var/ossec/etc'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/var/ossec/active-response'.
2011/05/11 10:07:19 ossec-syscheckd: INFO: Monitoring directory: '/etc/snort'.

Rootcheck is setup as per the default ossec.conf that is installed with 2.5.1.

-----Original Message-----
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of dan (ddp)
Sent: Wednesday, May 11, 2011 11:40 AM
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] syscheckd high cpu usage

Please provide some information about how you have these systems
configured (especially syscheck settings), and what they do.

On Wed, May 11, 2011 at 1:05 PM, Jefferson, Shawn
<Shawn.J...@bcferries.com> wrote:
>
> Hi,
>
> I have OSSEC installed on Ubuntu 10.04.2 LTS 64-bit, and the syscheckd process is taking a lot of CPU time, and has for the past couple of days.  I haven't seen this behaviour on other installations, but on three of these systems that are configured similiarly.  Any suggestions on where to look?  Rootkitcheck?
>

> You can see this one has been running syscheck for days.

[Michael Starks]

On 05/11/2011 12:05 PM, Jefferson, Shawn wrote:
> Hi,
> I have OSSEC installed on Ubuntu 10.04.2 LTS 64-bit, and the syscheckd
> process is taking a lot of CPU time, and has for the past couple of
> days. I haven’t seen this behaviour on other installations, but on three
> of these systems that are configured similiarly. Any suggestions on
> where to look? Rootkitcheck?

Try running syscheck in debug mode with the -d argument. We might be
able to get more information about what it is trying to scan.

[Daniel Cid]

Can you also try the latest snapshot? I fixed a bug on syscheck a
little while ago related to it (it was going 100% on my
ubuntu server as well):

https://bitbucket.org/dcid/ossec-hids/

*just go on download source to get it.

Thanks,

[Jefferson, Shawn]

I have installed with the latest source from this location. The version number is still 2.5.1? I want to make sure I'm using the correct source.

It hasn't seemed to help so far, but I'll let it run for awhile longer and see if it eventually stops.

-----Original Message-----
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Daniel Cid
Sent: Wednesday, May 11, 2011 6:21 PM
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] syscheckd high cpu usage

[Jefferson, Shawn]

Hi,

I ran syscheck in debug mode, but not very much was written to the ossec.log (or should I look somewhere else?)

2011/05/13 11:33:51 ossec-rootcheck: DEBUG: Starting ...
2011/05/13 11:33:51 ossec-rootcheck: Starting queue ...
2011/05/13 11:33:51 ossec-syscheckd: INFO: (unix_domain) Maximum send buffer set to: '124928'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: (unix_domain) Maximum send buffer set to: '124928'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Started (pid: 2353).
2011/05/13 11:33:55 ossec-rootcheck: INFO: Started (pid: 2353).
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/bin'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/boot'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/dev'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/etc'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/lib'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/lib64'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/sbin'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/usr/bin'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/usr/lib'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/usr/lib64'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/usr/sbin/'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/usr/share'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/usr/local'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/var/ossec/bin'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/var/ossec/etc'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/var/ossec/active-response'.
2011/05/13 11:33:55 ossec-syscheckd: INFO: Monitoring directory: '/etc/snort'.
2011/05/13 11:34:07 ossec-syscheckd: Setting SCHED_BATCH returned: 0
2011/05/13 11:34:57 ossec-syscheckd: INFO: Starting syscheck scan (forwarding database).
2011/05/13 11:34:57 ossec-syscheckd: INFO: Starting syscheck database (pre-scan).

Then it basically locked up one CPU at 100% for a couple of hours.

-----Original Message-----
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Michael Starks
Sent: Wednesday, May 11, 2011 5:15 PM
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] syscheckd high cpu usage

[Jefferson, Shawn]

Hi Daniel,

Unfortunately the latest version that I downloaded, did not fix this problem. Syscheckd still pegs the CPU at 100% for days, and doesn't stop. Syscheckd -d doesn't give much detail unfortunately. Anything else I can try to narrow the problem down?

[Jefferson, Shawn]

Hi,

Responding to myself here...

The high CPU utilization issue seems to be caused by checking either /lib or /lib64 on my Ubuntu 64-bit machine. This will cause the machine to lock one CPU at 100% for days and days. Should I not be checking those directories (maybe something about Linux I'm not aware of precludes this.)?

[Daniel Cid]

Can you try this snapshot:

https://bitbucket.org/dcid/ossec-hids/get/78e0ab251a6c.tar.gz

Should have a fix for that...

Thanks,

[Jefferson, Shawn]

I get this error when trying to install this version:

/usr/bin/ld: cannot find -lssl
/usr/bin/ld: cannot find -lcrypto
collect2: ld returned 1 exit status
make[1]: *** [auth1] Error 1
make[1]: Leaving directory `/root/installs/dcid-ossec-hids-78e0ab251a6c/src/os_auth'

[dan (ddp)]

Try installing the openssl development package for your distro.

On Tue, May 24, 2011 at 6:54 PM, Jefferson, Shawn

[Daniel Cid]

Or try the latest snapshot again:

https://bitbucket.org/dcid/ossec-hids/get/64b1ba1a779c.tar.gz

It should compile fine even without openssl support...

Thanks,

[Jefferson, Shawn]

Yes, that worked, thanks. Testing it now.

[Jefferson, Shawn]

Hi,

Unfortunately this new version did not fix the problem for me. What I have done in the meantime though, is exclude these directories from checking: /lib, /lib64 and /dev. Any of those and the CPU is spiked for days and syscheck never seems to stop (well, I haven't let it go longer than 4 or 5 days).

Thanks,
Shawn