Does cf-execd automatically reload changes to body executor control?

[Jason Tucker]

I'm working on testing out the setting of a specific value for agent_expireafter, rather than the default value which we have been using. I was hoping cf-execd would automatically detect this change, but it doesn't seem to be. Do I need to manually restart cf-execd everywhere to get this change to take effect?

__Jason

[Neil Watson]

cf-execd and cf-serverd are supposed to detect changes in inputs and
reload automatically, but these have not been reliable. I'm in the habit
of configuring a restart when inputs are updated.

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[Jason Tucker]

Yes, I've seen cf-serverd pick up changes automatically before, which is why I was kind of surprised to see that cf-execd doesn't seem to be.

This is 3.6.3, BTW.

__Jason

[Jason Tucker]

OK, now I'm just starting to think that 'agent_expireafter' just doesn't work. Or, perhaps I'm testing it wrong somehow?

As a test, I set this in 'body executor control':

    agent_expireafter => "10";

At the same time, I set this in my policy:

  classes:
    second_pass::
      "test_expire" expression  => returnszero("/bin/sleep 99999999999", "noshell");

    any::
      "second_pass" expression => "any";

The 'second_pass' guard just keeps it from getting hung up in pre-validation.

After a full restart of cfengine, I watched as cf-agent eventually ran, and I saw the sleep process that was spawned... but that's it. Its been running now for 45 minutes, with no sign that it's doing anything other than waiting for that impossibly long sleep to finish.

Is anyone else using this... and have you confirmed that it's working the way you would expect?

__Jason

[Jason Tucker]

One more observation (for now): it *does* seem to be timing out after all:

cf-execd: !! Timeout waiting for output from agent (agent_expireafter=10) - terminating it

However, it seems to be taking much longer than the 10 minutes that I have agent_expireafter configured for (as in, over an hour).

Strange. I need to play with this a bit more.

__Jason

[Jason Tucker]

OK, well, after changing this to agent_expireafter=60, adding some reporting so I can better monitor things, and letting things run for a while, I *think* this is working OK (although, sometimes I see the timeout message logged, and sometimes I don't... which is a bit odd). Also, I did confirm that cf-execd is picking up the config change without having to restart the process.

At any rate, the desired result seems to be achieved: keeping long-running cf-agents from filling up the process table. So, I'm satisfied at this point.

On a related note, I see that NIck was asking for input from the community on the default value for 'agent_expireafter' back when the feature was added. I know I'm a bit late to the party, but the default that we ended up with (7 days) seems rather excessive. I'm having a hard time imagining a scenario where you might expect cf-agent to run for *days* without any output. If you're using the default 5-minute schedule, the default 'agent_expireafter' value means that you could potentially end up with over 2,000 cf-agent processes running at once if something goes awry.

A typical execution time in our environment is well under 1 minute, so I'm thinking a 60 minute limit (for us, anyway) is more than adequate.

__Jason