Puppet agent hogging CPU/RAM

[ala...@op5.com]

Hi,

So the subject might be a bit generic, but couldn't really come up with anything better. Anyway, on to the point: having a problem with two servers in my environment, where when running the puppet agent, it hogs all memory and CPU; the weird thing is though that this only happens on to those two specific servers, which manifest varies quite a bit. To make it a bit easier to describe this issue, I'll call these servers server 1 and server 2. Server 1 runs puppet agent 3.4.2, while server 2 runs puppet agent 3.3.2. Both are running Debian 6 x86_64.

When running the agent in debug mode (puppet agent --test --debug --trace), it looks like the issue appears at:

Debug: Prefetching apt resources for package
Debug: Executing '/usr/bin/dpkg-query-W--showformat'${Status} ${Package} ${Version} :DESC: ${Description}\n:DESC:\n''

What happens when this comes up in the log is that server 1 stands there until it runs out of ram and starts killing processes (i.e. standard OOM behaviour) while server 2 takes about half a minute to complete (with the puppet agent hogging quite a bit of ram and 100% CPU), but then actually completes properly. A detail regarding the execution time is that the execution time reported to Puppet Dashboard when this happens isn't even near the time it actually takes to run the command (for example, for server 2, it is reporting times somewhere around 10-15 sec).

So, anyone have any idea what might be causing this? Have anyone seen this previously? (I was able to find quite a lot of posts regarding the puppet agent eating a lot of resources, although I couldn't find anything which seemed related to this specific issue)

Regards,
Aron Lander

[Felix Frank]

Hi,

that may be a red herring. Whatever is causing your contention may well
be the action taken *after* what is logged at that moment.

I advise to either
* watch the `ps auxwf` output for child processes or failing that
* emply `strace -f` to try and determine what is going on, roughly

HTH,
Felix