Puppet agent memory usage on the master

[Chris Ritson]

I've been trying for a while to work out why my puppet agent run on the puppet master was consuming so much time and memory. With a single webrick master running puppet puppet-3.6.2-1.el6 and serving about 100 clients, I was seeing agent runs on the master of the order of 4000 seconds. I was keeping, but not making use of the agent reports, so the puppet master was tidying these away after they reached the an age of 36 hours to avoid wasting disk space. Agent runs were quite often getting OOM errors and being killed as this was on a virtual machine with 2G of memory. Increasing this limit and watching the agent run grow it became clear that stored state was part of the problem. Memory consumption was sometimes reaching 3G.

In the end, even after abandoning report generation and allowing the remaining report files to be tidied out of existence, I was still seeing a large memory footprint and long delays when the puppet run appeared to be doing nothing or saving its previous state. Looking in the saved state.yaml file, I saw that there were still many hundreds of references to report files and directories that used to exist. Only when I finally removed this file altogether, taking the risk that a few services would be restarted when they shouldn't be, did my puppet run get back to a reasonable time and memory footprint.

It seems from this that the puppet agent is reading and preserving a lot of historic state information, even when this is no longer of any use. Is this a bug/feature? Is this something I could/should have been able to avoid with better knowledge of puppet's configuration options?

Chris Ritson
Newcastle University IT,
c/o School of Computing Science,
Floor 8,
Claremont Tower,
Newcastle University,
NE1 7RU, UK.

[Ramin K]

You're not alone with this problem.
http://ask.puppetlabs.com/question/13092/puppetmaster-process-memory-usage-constantly-increases-until-kernel-complains-and-starts-killing-off-other-processes/

I do think that there is something not quite right about whatever Puppet
is doing in more recent versions. That said moving your master to the
Apache/Passenger setup should solve it and is the correct solution
regardless of the memory problems you're having. webrick masters were
never meant to be the means to run Puppet in production or at more than
10 nodes in my opinion.

Ramin