High CPU I/O load with syscheckd under FreeBSD

Carlos Lopez

unread,

Mar 31, 2020, 10:09:37 AM3/31/20

to wa...@googlegroups.com

HI all,

Using Wazuh’s agent 3.11.4 or 3.12, syscheckd process loads high I/O CPU under FreeBSD 12-CURRENT at start over 9 min, more or less:

last pid: 82063; load averages: 1.05, 0.65, 0.61 up 0+07:27:41 14:05:16

64 threads: 2 running, 62 sleeping

CPU: 0.8% user, 98.8% nice, 0.4% system, 0.0% interrupt, 0.0% idle

Mem: 119M Active, 2261M Inact, 197M Laundry, 931M Wired, 397M Buf, 781M Free

Swap: 2048M Total, 2048M Free

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND

10782 root 106 10 19M 9480K RUN 0:17 98.31% ossec-syscheckd{ossec-syscheckd}

41431 root 20 0 956M 269M uwait 2:01 0.42% suricata{FM#01}

41431 root 20 0 956M 269M nanslp 2:29 0.27% suricata{suricata}

Any idea? Is it a bug?

Current syscheckd config is:

<scan_on_start>yes</scan_on_start>

<alert_new_files>yes</alert_new_files>

<auto_ignore frequency="10" timeframe="3600">no</auto_ignore>

<directories report_changes="yes" check_all="yes">/etc,/usr/bin,/usr/sbin,/usr/local/bin,/usr/local/sbin</directories>

<ignore>/etc/random.seed</ignore>

<skip_nfs>yes</skip_nfs>

<skip_dev>yes</skip_dev>

<skip_proc>yes</skip_proc>

<skip_sys>yes</skip_sys>

<process_priority>10</process_priority>

<max_eps>100</max_eps>

<max_interval>1h</max_interval>

<max_eps>10</max_eps>

</synchronization>

</syscheck>

--

Regards,

C. L. Martinez

Nicolas Papp

unread,

Mar 31, 2020, 11:20:29 AM3/31/20

to Carlos Lopez, wa...@googlegroups.com

Hello Carlos,

Can you show me your rootcheck section of the configuration?

Because rootcheck can cause a very high CPU spike at start if you configure it to check for the whole system.

Best Regards,

Nicolas

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/FD3233BC-986A-4013-B9D6-96693E84F445%40outlook.com.

Carlos Lopez

unread,

Mar 31, 2020, 11:25:10 AM3/31/20

to Nicolas Papp, wa...@googlegroups.com

Hi Nicolas,

Yes, here it is:

________________________________________
From: Nicolas Papp <nicola...@wazuh.com>
Sent: 31 March 2020 17:20
To: Carlos Lopez
Cc: wa...@googlegroups.com
Subject: Re: High CPU I/O load with syscheckd under FreeBSD

Hello Carlos,
Can you show me your rootcheck section of the configuration?
Because rootcheck can cause a very high CPU spike at start if you configure it to check for the whole system.

Best Regards,
Nicolas

Current syscheckd config is:

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com<mailto:wazuh+un...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/FD3233BC-986A-4013-B9D6-96693E84F445%40outlook.com<https://groups.google.com/d/msgid/wazuh/FD3233BC-986A-4013-B9D6-96693E84F445%40outlook.com?utm_medium=email&utm_source=footer>.

Nicolas Papp

unread,

Mar 31, 2020, 11:46:21 AM3/31/20

to Carlos Lopez, wa...@googlegroups.com

Can you try changing the check_sys tag to the value no? Because with that configuration rootcheck is basically checking recursively all your file system.

This could generate an enormous amount of data for rootcheck to analyse, causing the spykes in CPU you see and a big delay on the first run.

This has been a change introduced in 3.9. Rootcheck 3.6.1 scans the target directories in one-level depth. As of 3.9.0, Rootcheck is scanning the target directories recursively.

If this changes solves the problem, you could add a few ignore folders to rootcheck to avoid having such a big overhead and to be able to set the value back to yes.

Hope this helps.

Nicolas

Carlos Lopez

unread,

Mar 31, 2020, 11:50:55 AM3/31/20

to Nicolas Papp, wa...@googlegroups.com

Hi Nicolas,

Nothing. Same result disabling check_sys:

last pid: 993; load averages: 0.66, 0.50, 0.44 up 0+09:12:32 15:50:07
62 threads: 3 running, 59 sleeping
CPU: 0.4% user, 93.4% nice, 4.7% system, 1.2% interrupt, 0.4% idle
Mem: 84M Active, 2303M Inact, 197M Laundry, 862M Wired, 326M Buf, 782M Free

Swap: 2048M Total, 2048M Free

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND

64631 root 101 10 19M 9420K RUN 0:09 97.62% ossec-syscheckd{ossec-syscheckd}
41431 root 20 0 956M 269M select 5:58 0.65% suricata{W#01-vtnet2}
41431 root 20 0 956M 269M uwait 2:30 0.43% suricata{FM#01}
41431 root 20 0 956M 269M RUN 2:57 0.29% suricata{suricata}
41431 root 20 0 956M 269M select 0:55 0.12% suricata{W#01-v

________________________________________
From: Nicolas Papp <nicola...@wazuh.com>

Sent: 31 March 2020 17:46

To: Carlos Lopez
Cc: wa...@googlegroups.com
Subject: Re: High CPU I/O load with syscheckd under FreeBSD

Can you try changing the check_sys tag to the value no? Because with that configuration rootcheck is basically checking recursively all your file system.

This could generate an enormous amount of data for rootcheck to analyse, causing the spykes in CPU you see and a big delay on the first run.

This has been a change introduced in 3.9. Rootcheck 3.6.1 scans the target directories in one-level depth. As of 3.9.0, Rootcheck is scanning the target directories recursively.
If this changes solves the problem, you could add a few ignore folders to rootcheck to avoid having such a big overhead and to be able to set the value back to yes.

Hope this helps.
Nicolas

On Tue, Mar 31, 2020 at 12:25 PM Carlos Lopez <clo...@outlook.com<mailto:clo...@outlook.com>> wrote:
Hi Nicolas,

Yes, here it is:

________________________________________
From: Nicolas Papp <nicola...@wazuh.com<mailto:nicola...@wazuh.com>>

Sent: 31 March 2020 17:20
To: Carlos Lopez

Cc: wa...@googlegroups.com<mailto:wa...@googlegroups.com>

Subject: Re: High CPU I/O load with syscheckd under FreeBSD

Hello Carlos,
Can you show me your rootcheck section of the configuration?
Because rootcheck can cause a very high CPU spike at start if you configure it to check for the whole system.

Best Regards,
Nicolas

Current syscheckd config is:

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com<mailto:wazuh%2Bunsu...@googlegroups.com><mailto:wazuh+un...@googlegroups.com<mailto:wazuh%2Bunsu...@googlegroups.com>>.

Nicolas Papp

unread,

Mar 31, 2020, 1:23:50 PM3/31/20

to Carlos Lopez, wa...@googlegroups.com

Ok then maybe it is indeed syscheck the one generating all the overhead,

Let´s try commenting this 2 lines:

<directories report_changes="yes" check_all="yes">/etc,/usr/bin,/usr/sbin,/usr/local/bin,/usr/local/sbin</directories>
<directories report_changes="yes" check_all="yes">/bin,/sbin,/boot</directories>

Does it get any better?

Nicolas

Carlos Lopez

unread,

Apr 1, 2020, 1:42:30 AM4/1/20

to Nicolas Papp, wa...@googlegroups.com

Good morning Nicolas,

No. Same behavior,

--

Regards,

C. L. Martinez

Nicolas Papp

unread,

Apr 1, 2020, 10:21:13 AM4/1/20

to Carlos Lopez, wa...@googlegroups.com

Hello again Carlos, can you confirm me whether the Wazuh version you are using is 3.11.4 or 3.12? Because there are radical changes between both versions.

While the version 3.11.4 analizes 100 files at a time, 3.12 realizes a complete scan at start without interruptions. This puts the systems at its limit until the scan has ended, allowing who-data and real-time services (that are not available in freeBSD but are used in Linux) to start ASAP since they depend on that scan.

The good thing about this is that in this case the process will try to use all the resources but with a very low scheduling priority, so in case there is another process that needs to use CPU, or disk, it will be able to access those resources without a problem.

In the case you are using 3.12, you should expect a High CPU and disk ussage at start until the scan is finished. But it should not get in the way of your other process.

Hope this information helps!

Nicolas

Carlos Lopez

unread,

Apr 1, 2020, 10:58:59 AM4/1/20

to Nicolas Papp, wa...@googlegroups.com

HI NIcolas,

This FreeBSD agent is in 3.12 release … But syscheckd process peaks CPU until 98% during 5 min or more … It is not normal and nice launched from wazuh’s agent doesn’t have any effect, almost in FreeBSD platform …

Reply all

Reply to author

Forward