Cf-agent is taking very high CPU

[Jagdish Ahir]

Hi 

We are using  cfengine-community 3.6.0-1 on debian wheezy servers. 

also we use design-center under /var/cfegine/masterfiles/sketches

Every cf-agent run it take 95 to 99% of CPU.

What other information required about the same to explain more?

[Marco Marongiu]

On 25/08/15 12:37, Jagdish Ahir wrote:
> Every cf-agent run it take 95 to 99% of CPU.

First of all, how many cf-agent instances do you see on each host?

[Jagdish Ahir]

It is one only cf-agent instance 

[Neil Watson]

On Tue, Aug 25, 2015 at 03:37:30AM -0700, Jagdish Ahir wrote:
> We are using cfengine-community 3.6.0-1 on debian wheezy servers.
> also we use design-center under /var/cfegine/masterfiles/sketches
> Every cf-agent run it take 95 to 99% of CPU.

Any process will consumer as much CPU as it can. What you left out was
the time at which cf-agent consumes CPU.

You can try profiling your policy:
https://github.com/lpefferkorn/cfe-profiler


--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[Marco Marongiu]

cf-agent is usually quite lean in both memory and CPU consumption. The
only time I've seen a greedy cf-agent was when a colleague was
checksumming a huge tree of files by mistake, which of course was CPU
intensive. Otherwise, I barely notice that an agent is running on our
servers. Could it be your case?

Ciao
-- bronto

[Aleksey Tsalolikhin]

I quite commonly get alarmed reports that "CFEngine is consuming 100% of the CPU when it runs!".

Upon investigation it usually turns out that this happens every 5 minutes and usually only lasts seconds.  In other words this is just CFEngine doing its job.  It doesn't hurt anything.  CFEngine runs at a normal priority and if there are other processes that need CPU during this time, the kernel will share the CPU between CFEngine and the other processes.  This is just how time-sharing works.  If nobody else needs it, cf-agent gets all the CPU, finishes its job and EXITS.  Nothing to get alarmed about.  CFEngine actually has a small memory and CPU footprint compared to similar tools.

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

--

Aleksey Tsalolikhin

CFEngine Training & Consulting

Vertical Sysadmin, Inc.

+1-323-393-0779

[Jagdish Ahir]

Thanks for reply,

I think this can be the case. we use file copy function in many sketches. while run cf-agent it might be checking all the files then if required it copies new file.

but I think it is required to check file checksum. 

1. How i will make sure we are using checksum ?

2. Is it not required while we we use copy function ?

[Nick Anderson]

On 08/25/2015 08:34 AM, Jagdish Ahir wrote:
>
> 1. How i will make sure we are using checksum ?
> 2. Is it not required while we we use copy function ?

You will need to inspect the specific policy in use to see the details
of the copy_body.

Specifically you will want to check out what is set for the compare
attribute. You might also want to see if you are using the verify
attribute or not.

mtime without verification is probably the lightest on the CPU.
Also you could further guard the copy_from so that it doesn't try to
talk to a remote server every execution. Maybe you could limit it with
ifelapsed, or only if the file doesn't exist. It really depends on your
requirements.

https://docs.cfengine.com/latest/reference-promise-types-files.html#compare
https://docs.cfengine.com/latest/reference-promise-types-files.html#verify

[Jagdish Ahir]

Yes we observed with another process, cf-agent is sharing CPU. and as soon as process completes cf-agent takes 99 to 100% CPU. it is really cool.. 

Thanks for sharing :)

[Jagdish Ahir]

Thanks Nick,

I will try this too. 

[Ted Zlatanov]

On Tue, 25 Aug 2015 14:42:51 +0200 Marco Marongiu <bront...@gmail.com> wrote:

MM> cf-agent is usually quite lean in both memory and CPU consumption. The
MM> only time I've seen a greedy cf-agent was when a colleague was
MM> checksumming a huge tree of files by mistake, which of course was CPU
MM> intensive. Otherwise, I barely notice that an agent is running on our
MM> servers.

I've seen it with LARGE classic CFEngine arrays, e.g.

"array[x][y][1]" string => "1";
"array[x][y][2]" string => "2";
"array[x][y][3]" string => "3";

repeated a few thousand times, since each entry is a new variable...
Really really slow and cf-promises also spikes to 100% CPU.

Ted

[Gregory Matthews]

we can confirm! Our package promising bundle reads a largish file (850
lines, 70k) into a multidimensional array and requires 4GB of memory
installed to not bail out!

This (along with rhel7) is one of our major upgrade drivers (we on 3.5.x
currently).

GREG

>
> Ted
>


--
Greg Matthews 01235 778658
Scientific Computing Group Leader
Diamond Light Source Ltd. OXON UK

--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

[Danny Sauer]

On Tuesday, August 25, 2015 at 8:30:10 AM UTC-5, Aleksey Tsalolikhin wrote:

I quite commonly get alarmed reports that "CFEngine is consuming 100% of the CPU when it runs!".

I basically have a form letter for these emails that I send out now, explaining that a single-threaded reduced-priority process showing 98% CPU on a 32-core system for 30-45 seconds every five minutes is probably not the reason your web app is slow - but your "SELECT * | grep | sed" approach to database utilization might be.

--Danny, who's recently been considering removing the world execute bit on top :)

[Nick Anderson]

Do you have an issue open for this?

[Nick Anderson]

-----Original Message-----
From: Gregory Matthews .

we can confirm! Our package promising bundle reads a largish file (850
lines, 70k) into a multidimensional array and requires 4GB of memory
installed to not bail out!

Have you opened a ticket for this? Can you link me to it?

[Aleksey Tsalolikhin]

I told my executive today that we could use cgroups to guarantee resource constraints on CPU, network and memory utilization.

Not just for cfengine but other infrastructure services like Splunk.

--

[Gregory Matthews]

Hi Nick...

most of our testing was done in https://dev.cfengine.com/issues/1875
which in hindsight was unfortunate as we were concerned with memory
usage but Neil was concerned with performance.

GREG

> --
> You received this message because you are subscribed to the Google
> Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to help-cfengin...@googlegroups.com

> <mailto:help-cfengin...@googlegroups.com>.

> To post to this group, send email to help-c...@googlegroups.com

> <mailto:help-c...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/help-cfengine.
> For more options, visit https://groups.google.com/d/optout.

[Ted Zlatanov]

On Thu, 17 Sep 2015 12:36:02 -0700 (PDT) Danny Sauer <dannysa...@gmail.com> wrote:

DS> On Tuesday, August 25, 2015 at 8:30:10 AM UTC-5, Aleksey Tsalolikhin wrote:
>>
>> I quite commonly get alarmed reports that "CFEngine is consuming 100% of
>> the CPU when it runs!".

DS> I basically have a form letter for these emails that I send out now,
DS> explaining that a single-threaded reduced-priority process showing 98% CPU
DS> on a 32-core system for 30-45 seconds every five minutes is probably not
DS> the reason your web app is slow - but your "SELECT * | grep | sed" approach
DS> to database utilization might be.

In theory you're right, but in practice vmstat might show a high number
of interrupts when this happens, and there are other ways in which a
consumed core might affect a multi-core system, especially in Linux.
I recall Solaris was significantly better at handling this situation.

Ted

[Dimitrios Apostolou]

On Fri, Sep 18, 2015 at 12:21 PM, Gregory Matthews
<greg.m...@diamond.ac.uk> wrote:
> Hi Nick...
>
> most of our testing was done in https://dev.cfengine.com/issues/1875 which
> in hindsight was unfortunate as we were concerned with memory usage but Neil
> was concerned with performance.
>
> GREG

Hi Greg, that ticket is closed, and contains to much info probably
irrelevant to your problem. if you are still having the same problem
(consuming GBs of memory when reading a small file) I'd suggest you
file a separate bug report, with minimal policy and example file
attached that reproduces it. Please link the ticket here if you do so.


Dimitris