Memory Leak in ossec-csyslogd and ossec-dbd

[Steve]

I have been experiencing a memory leak in ossec-csyslogd and ossec-
dbd. I've reviewed a message on the list last Nov (thanks for the
help Joe!) and tried to apply the patch by Daniel (https://
bitbucket.org/ddpbsd/ossec-hids/changeset/4b86abf62d5b) to address
this. Didn't seem to work. I'm looking for suggestions on how to
proceed. Right now I have a crontab entry to restart OSSEC every 4
hours. Occasionally I get a level 12 alert from OSSEC about rule 5108
system running out of memory and that it killed a process.

System Info:
CentOS 64bit, 4 GB RAM
OSSEC 2.6
About 100 agents running 2.5.1 and 2.6
MySQL database
About 1M events per hour

Try #1 - Made manual changes to src/shared/read-alert.c in the
original 2.6 source to free dstip. Copied newly compiled binaries
into /var/ossec/bin/ directory. Different size binary, but no change
in memory consumption. These progs all seem statically linked where a
linked library with the memory leak isn't the cause of the problem,
right?

Try #2 - Downloaded latest snapshot dcid-ossec-hids-2f056a266d49 and
compiled new binaries. Copied ossec-dbd and ossec-csyslogd binaries
only into /var/ossec/bin directory. Still no change in memory
consumption.

Try #3 - Tried upgrade of OSSEC with the entire dcid-ossec-
hids-2f056a266d49 source tree. The install worked, but no agents
could communicate with the server. Even tried deleting an agent key
and regenerating the key for the agent, clearing the queue/rids files,
etc... but still no communication. So, I reloaded the original 2.6
binaries and config files. This might be a completely different
issue. No weird messages on the server, but the client was giving a
couple of errors. I'll leave this situation alone unless someone
pings me.

At this point I'm back on #2. For example, first thing in the
morning, I'll take a look and those two programs are eating up all my
RAM. I restart and it slowly builds back up.

[steve.lodin@HqOssec01 ~]$ date
Fri Mar 2 09:23:11 EST 2012
[steve.lodin@HqOssec01 ~]$ ps axuww|grep ossec
ossecm 16615 21.7 26.3 1109516 1064544 ? S Mar01 197:43 /var/
ossec/bin/ossec-dbd
ossecm 16620 0.0 24.6 1002632 997144 ? S Mar01 0:18 /var/
ossec/bin/ossec-csyslogd

[steve.lodin@HqOssec01 ~]$ sudo service ossec restart
Stopping OSSEC: [ OK ]
Starting OSSEC: [ OK ]
[steve.lodin@HqOssec01 ~]$ date
Fri Mar 2 09:36:52 EST 2012
[steve.lodin@HqOssec01 ~]$ ps axuww|grep ossec
ossecm 22467 0.9 0.0 48000 3040 ? S 09:37 0:00 /var/
ossec/bin/ossec-dbd
ossecm 22472 0.0 0.0 6460 944 ? S 09:37 0:00 /var/
ossec/bin/ossec-csyslogd

[steve.lodin@HqOssec01 ~]$ date
Fri Mar 2 11:13:28 EST 2012
[steve.lodin@HqOssec01 ~]$ ps axuww|grep ossec
ossecm 22467 2.0 3.3 178908 133972 ? S 09:37 1:57 /var/
ossec/bin/ossec-dbd
ossecm 22472 0.0 3.0 129268 123800 ? S 09:37 0:02 /var/
ossec/bin/ossec-csyslogd

Thanks for any suggestions or help!

Steve

[Andreas Piesk]

On 02.03.2012 22:17, Steve wrote:
>
> Thanks for any suggestions or help!
>

you could use valgrind (http://valgrind.org) to report memleaks, for instance

valgrind <binary> <args>

or more detailed

valgrind --leak-check=yes <binary> <args>

regards,
-ap

[Steve Lodin]

Thanks Andreas.

I've got valgrind running on both binaries and it looks like there might be some interesting leak results.  Any suggestions on how to get this fixed?  Sorry, relative newbie to OSSEC and I'm not sure how to get this into the bug fix process.

Cheers,
Steve

--
Cell: +1-317-840-9088
LinkedIn: http://www.linkedin.com/in/stevelodin
Twitter: http://twitter.com/stevelodin

[Andreas Piesk]

On 05.03.2012 19:28, Steve Lodin wrote:
>
> I've got valgrind running on both binaries and it looks like there might be some interesting leak
> results. Any suggestions on how to get this fixed? Sorry, relative newbie to OSSEC and I'm not
> sure how to get this into the bug fix process.
>

you could open an issue at bitbucket (https://bitbucket.org/dcid/ossec-hids/) and upload your
findings there, so others can take a look, verify and hopefully fix some leaks.

or you post your findings with an explanation how you got them on the mailing list. but if the data
is multi-megabytes in size, bitbucket is the better choice.

regards,
-ap

[Andreas Piesk]

On 05.03.2012 19:28, Steve Lodin wrote:
>

> I've got valgrind running on both binaries and it looks like there might be some interesting leak
> results. Any suggestions on how to get this fixed? Sorry, relative newbie to OSSEC and I'm not
> sure how to get this into the bug fix process.
>

while fixing memleaks in other ossec parts i took a look at your issue and uploaded a patch
(ossec-memleaks.patch) to bitbucket. please let me know if the patch fixes your problem.

regards,
-ap

[Steve Lodin]

I've had this patch running for the past two days and all indications are this eliminated the memory leak in read-alert.c that affects ossec-csyslogd.

Thanks!!

Steve

[Andreas Piesk]

On 20.03.2012 20:52, Steve Lodin wrote:
> I've had this patch running for the past two days and all indications are this eliminated the memory
> leak in read-alert.c that affects ossec-csyslogd.

good to hear.

commited the patch in my fork https://bitbucket.org/pieska/ossec-hids along with another memleak
fix. you're welcome to try my fork (it contains only bugfixes).

regards,
-ap