Performance testing of OSSEC

[Christopher Moraes]

Hello everyone,

I have just joined the ossec-list.  I am evaluating (performance testing) OSSEC for my organization to see whether OSSEC can handle a volume of 2500-3000 events per second.

I have created the following test setup - 

-  Dual Core Linux server with 4 GB RAM 

-  OSSEC is installed on this server (local mode) and is configured to monitor 4 log files, which I am filling using a script.

I ran the test yesterday and OSSEC updated the stats folder only at eod.    Is there anyway to get OSSEC to report on the stats either 

i.  in real time?  

ii. at least every hour?

Thanks and regards,

Chris

[Aamir Niazi]

You can check this link
http://www.ossec.net/main/manual/manual-syscheck/realtime-file-integrity-monitoring/

You can also set the frequency in ossec.conf to whatever you like. Hope this helps.

Typos, courtesy of the wireless device.

---

From: Christopher Moraes <cmora...@gmail.com>

Sender: ossec...@googlegroups.com

Date: Tue, 21 Sep 2010 12:17:52 -0400

To: <ossec...@ossec.net>

ReplyTo: ossec...@googlegroups.com

Subject: [ossec-list] Performance testing of OSSEC

[Christopher Moraes]

Hi Aamir,

Thanks for your reply.  I went through the link you sent.  Currently I am only testing the performance of the log analysis components.  (We intend to use only log-analysis and leave out the file integrity checking and rootkit detection.)

You mentioned that this can be setup in the ossec.conf file.  Do you have a link that explains the various configuration options in ossec.conf or would you happen to know what the syntax of the configuration should be?   My existing file is very sparse - it just lists the rules xml and the list of log files to be monitored.

Thanks.

[dan (ddp)]

The following link outlines the various options:
http://www.ossec.net/main/manual/configuration-options/

If the stats you're looking to have updated at the ones in
/var/ossec/stats, there aren't really any configuration options for
those.

[Christopher Moraes]

Thank you.  That was exactly the link I was looking for.

Since /var/ossec/stats is not configurable, is there any other way I can measure the volume of eps that ossec (log-collector and analysisd) handle for a test run?

For e.g. if I have OSSEC monitor a log file for an hour, is there any way I can generate a report showing how many EPS were processed?

[dan (ddp)]

Not that I'm aware of, beyond grepping the logfile for events during
the time you want and counting that. If you turn on the logall option,
you could compare that number to the number of events in the archive
logfile.

[dan (ddp)]

It looks like the stats kept in '/var/ossec/stats/totals/' are updated hourly.
Not sure how to decode them off-hand though.
I'm guessing something like:
hour-events-alerts-something?

[dan (ddp)]

I think this is the code that writes out these stats:
/* Print total for the hour */
fprintf(flog, "%d--%d--%d--%d--%d\n\n",
thishour,
hourly_alerts, hourly_events, hourly_syscheck,hourly_firewall);

This would go with the following line from
/var/ossec/stats/totals/Sep/ossec-totals-01.log on my system:
14--80516--101757--1--0

The following line:
15-370010-1-83
is hour-SID-level-firedtimes (maybe the number of times it fired
during that hour...)

If this doesn't answer your other thread as well, you can look in the
ossec source (src/analysisd/analysisd.c) for more answers.

[Michael Starks]

On 09/21/2010 02:10 PM, Christopher Moraes wrote:
> Hi Aamir,
>
> Thanks for your reply. I went through the link you sent. Currently I
> am only testing the performance of the log analysis components. (We
> intend to use only log-analysis and leave out the file integrity
> checking and rootkit detection.)

The short answer is that ossec-analysisd is not the bottleneck. Disk I/O
and kernel parameters will slow things down first. Specifically, I
recall UDP buffers having an affect. Seems like a good topic for Week of
OSSEC. I know Daniel in particular has done extensive testing in this
area. From what I recall, I think you can reasonably expect around 1,000
eps on a well-tuned system.

--
Michael Starks
[I] Immutable Security
http://www.immutablesecurity.com

[Christopher Moraes]

I ran a couple of tests yesterday, where I generated about 2500 EPS in the log files being monitored.  

Today, the OSSEC stats have been updated and looking at the hourly-average figures, I see that OSSEC is currently processing 2500 EPS.  (The hourly-average reports a figure of 89,830,xxx for 4 consecutive hours).  Resource usage is barely noticeable.   CPU% for analysisd hit a high of 5% for a bit of time, but largely remained in the 1-2% range.  

I'm not sure if the log events I'm generating are stressing analysisd enough.   I would like to add logs that make analysisd test against more rules.  If anyone has sample logs that I could use to test, I would greatly appreciate if you can share them with me.

I'm planning to do another test with around 4000 EPS being generated in the log files.

[dan (ddp)]

I have some logs at http://code.google.com/p/wip-ossec-rules

[Christopher Moraes]

Hi Folks, 

Here are some results of another performance test round - 

1.  Generated 6000 EPS in the log files

2.  OSSEC processed at the same rate i.e. 6000 EPS

3.  CPU utilization increased to 30% (avg) for analysisd and 10% (avg) for log-collector

4.  Memory utilization is steady at 1%

Test Server:

RHEL VM setup with dual CPU and 4 GB RAM.

OSSEC has been installed in "Local" mode

OSSEC seems to be vertically scaling quite well - with an increase in load it is consuming more server resources.

I am now testing with around 10,000 EPS and will publish the results.

Comments/Feedback are welcome and much appreciated.

Regards,

Chris

[dan (ddp)]

Awesome information, thanks!
I wonder how the local mode compares to the server mode with a couple of agents.

[Christopher Moraes]

Hi Everyone, 

Here are the results from Day 2 of my Performance test - 

1.  Generated 11,000 EPS in the log files being monitored.  

2.  OSSEC scaled beautifully to process at exactly the same rate - 11,000 EPS.  

3.  CPU utilization increased to 50% (avg) for analysisd and 20% (avg) for log-collector

4.  Memory utilization is steady at 1-2%

Test Setup - 

-  The hardware (VM) remains same as mentioned below.

-  Load is generated across 4 log files - messages (linux syslog), maillog, apache access log and apache error log

-  Alert severity is set to 1

-  While testing with 11,000 EPS the log files were approx 6-8 GB in size.  I have not seen any degradation in performance with increase in the size of the log file being monitored.

-  CPU utilization is reported as % of total CPU time (in this case % of combined 2 core CPU setup)

I will tabulate the results from all the test runs and will share with the group.

Dan, I will try to test client/server mode either tomorrow or Mon.

[Jeremy Rossi]

Thank you for this work and for sharing with the list/community. 

I would like to add your testing results and methods into ossec.net/doc/ (beta docs for up and coming release). If you give permission could you also send me any scripts/tools and method details to the list also to be included?

Thank you again,

Jeremy Rossi

Sent from my iPhone

[Christopher Moraes]

Hi Jeremy,

Sure, I will send you the scripts with some documentation on how they are to be used.  Also, please feel free to use whatever I have posted here for the OSSEC documentation.  

If you need me to add a writeup directly to the wiki/documentation, please let me know.

Regards,

Chris

[dan (ddp)]

If you have a blog or something, there are plans for the second "Week
of OSSEC, The" in October. If you're interested in writing something
up, performance testing sounds like an interesting topic (to me
anyways).

[Christopher Moraes]

Hi everyone, 

Here are the detailed results of the performance tests that I ran.

Summary:

-  The tests covered only the log-analysis functionality of OSSEC

-  A total of 4 test runs were done.  Each with a different rate of EPS generated in the log files.

-  Each test was run for 2-3 hours and the results were averaged out.

-  The test machine is a 2 Core, 4 GB RAM, RHEL Virtual Machine.

-  OSSEC is installed in local mode and is monitoring 4 log files - syslog, maillog, 2 apache logs.

Observations:

-  OSSEC scales vertically, using more CPU as the load (EPS) increases

-  At 11,000 EPS, log-collector and analysisd used a combined average of 70% of the CPU

-  Memory utilization was between 1-2% irrespective of the EPS

-  Size of the log files has no effect on the performance of OSSEC (8.5 GB log file was used during the test)

Detailed Test Results:

Run Generated EPS Processed EPS % Alerts generated Analysisd CPU Avg Logcollector Avg CPU1 2700 2700 18.56% 11 3 2 3600 3800 18.56% 14 4 3 6000 6000 18.61% 26 10 4 11000 11000 18.72% 48 22

% increase in EPS v/s % increase in resource utilization

Processed EPS Combined Avg %CPU (analysisd + logcollector) % increase in EPS %increase in CPU utlization 2700 14 0.00% 0.00%   <----- baseline for comparison 3600 18 33.33% 28.57% 6000 36 66.67% 100.00% 11000 70 83.33% 94.44%

[Christopher Moraes]

Sorry the detailed results table got messed up when pasting it into the email.

I'm attaching the results in an .xls format.

Regards,

Chris

[kristianpaul]

hey thanks for all this tests, i really love OSSEC even more after
that : )

About memory i think not sure, it could be tested in deepth by
creating logs
that force more the ossec correlation engine to work

On 23 sep, 16:00, Christopher Moraes <cmoraes....@gmail.com> wrote:
> Hi everyone,
>
> Here are the detailed results of the performance tests that I ran.
>

> *Summary:*

> -  The tests covered only the log-analysis functionality of OSSEC
> -  A total of 4 test runs were done.  Each with a different rate of EPS
> generated in the log files.
> -  Each test was run for 2-3 hours and the results were averaged out.
> -  The test machine is a 2 Core, 4 GB RAM, RHEL Virtual Machine.
> -  OSSEC is installed in local mode and is monitoring 4 log files - syslog,
> maillog, 2 apache logs.
>

> *Observations:*

> -  OSSEC scales vertically, using more CPU as the load (EPS) increases
> -  At 11,000 EPS, log-collector and analysisd used a combined average of 70%
> of the CPU
> -  Memory utilization was between 1-2% irrespective of the EPS
> -  Size of the log files has no effect on the performance of OSSEC (8.5 GB
> log file was used during the test)

> *
> *
> *
> *
> *Detailed Test Results:*
> *
> *
> ** Run Generated EPS Processed EPS % Alerts generated Analysisd CPU

> Avg Logcollector
> Avg CPU1 2700 2700 18.56% 11 3 2 3600 3800 18.56% 14 4 3 6000 6000 18.61% 26
> 10 4 11000 11000 18.72% 48 22

> *
> *

> % increase in EPS v/s % increase in resource utilization

> *
> *
> ** Processed EPS Combined Avg %CPU (analysisd + logcollector) % increase in

> EPS %increase in CPU utlization 2700 14 0.00% 0.00%   <----- baseline for
> comparison 3600 18 33.33% 28.57% 6000 36 66.67% 100.00% 11000 70 83.33%
> 94.44%

> **

[Michael Starks]

On Thu, 23 Sep 2010 17:00:27 -0400, Christopher Moraes
<cmora...@gmail.com> wrote:
> Hi everyone,
>
> Here are the detailed results of the performance tests that I ran.
>

> *Summary:*

> - The tests covered only the log-analysis functionality of OSSEC
> - A total of 4 test runs were done. Each with a different rate of EPS
> generated in the log files.
> - Each test was run for 2-3 hours and the results were averaged out.
> - The test machine is a 2 Core, 4 GB RAM, RHEL Virtual Machine.
> - OSSEC is installed in local mode and is monitoring 4 log files -
syslog,
> maillog, 2 apache logs.

On Thu, 23 Sep 2010 17:00:27 -0400, Christopher Moraes
<cmora...@gmail.com> wrote:
> Hi everyone,
>
> Here are the detailed results of the performance tests that I ran.
>

> *Summary:*

> - The tests covered only the log-analysis functionality of OSSEC
> - A total of 4 test runs were done. Each with a different rate of EPS
> generated in the log files.
> - Each test was run for 2-3 hours and the results were averaged out.
> - The test machine is a 2 Core, 4 GB RAM, RHEL Virtual Machine.
> - OSSEC is installed in local mode and is monitoring 4 log files -
syslog,
> maillog, 2 apache logs.

Thanks again for your efforts. This is really impressive, but people
shouldn't necessarily extrapolate this to agent->manager performance. I
have these two lines in /etc/sysctl.conf, which should help with UDP
buffers and increase the potential eps for agent->manger communication. You
can play with the values to see what happens.

net.core.rmem_default = 5123840
net.core.rmem_max = 5123840

Would you like to do another test? :) I would be really interested to see
how many eps you can achieve with an agent installed on a laptop and
plugged into the server with a crossover cable. "netstat -su" could be
helpful to see if the buffers gets maxed. This could give us a theoretical
maximum on a set platform for a distributed environment. Of course, network
congestion and other factors would cause logs to be dropped.

--
[I] Immutable Security
Information Security, Privacy and Personal Liberty
http://www.immutablesecurity.com

[Michael Starks]

On Thu, 23 Sep 2010 17:00:27 -0400, Christopher Moraes
<cmora...@gmail.com> wrote:

> *Observations:*

> - OSSEC scales vertically, using more CPU as the load (EPS) increases
> - At 11,000 EPS, log-collector and analysisd used a combined average of
> 70%
> of the CPU
> - Memory utilization was between 1-2% irrespective of the EPS
> - Size of the log files has no effect on the performance of OSSEC (8.5
GB
> log file was used during the test)

I forgot to mention that I have also tuned the I/O a bit. OSSEC is on its
own partition and the fstab looks similar to this:

/dev/VolGroup00/ossec /log/ossec ext3 defaults,noatime 1
2

Notice the "noatime" option. This will make it so the atime of the files
are not updated, and since files such as alerts.log are constantly
changing, this can help.

[Christopher Moraes]

About memory i think not sure, it could be tested in deepth by
creating logs
that force more the ossec correlation engine to work

If you happen to have any logs that you know will stress the engine, send them to me and I will run another test.  It'll be interesting to see how the resource usage changes.