puppetserver 2.0.0: out of memory with Xmx3g and 1 client

[Thomas Müller]

hi

I've started trying to run our puppet3 environments on puppet4 (puppet-agent 1.1.0 with puppetserver 2.0.0). It failed with some out of memory problem. I've started with 1g and increased it up to 3g - same error. It's a standalone RHEL6 Vm with total 4GB memory and only itself as client connected.

# puppet agent --test --noop
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: end of file reached
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Connection refused - connect(2) for "puppet" port 8140

(it takes 3m30s until the "Could not send report" line)

puppetserver-daemon.log says: 

/var/log/puppetlabs/puppetserver-daemon.log:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /var/log/puppetlabs/puppetserver/java_pid14091.hprof ...
Heap dump file created [1051509827 bytes in 6.793 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 14091"...

puppetserver.log (with enabled level="debug" in logback.xml):

/var/log/puppetlabs/puppetserver/puppetserver.log:
...
2015-05-26 15:40:08,144 DEBUG [puppet-server] Puppet importing '/etc/puppetlabs/code/environments/production/modules/nsswitch/manifests/init.pp' in environment production
2015-05-26 15:40:08,152 DEBUG [puppet-server] Puppet Automatically imported nsswitch from nsswitch into production
2015-05-26 15:40:08,157 DEBUG [puppet-server] Puppet importing '/etc/puppetlabs/code/environments/production/modules/sysctl/manifests/init.pp' in environment production
2015-05-26 15:40:08,172 DEBUG [puppet-server] Puppet Automatically imported sysctl from sysctl into production
2015-05-26 15:40:08,177 DEBUG [puppet-server] Puppet importing '/etc/puppetlabs/code/environments/production/modules/sysctl/manifests/value.pp' in environment production
2015-05-26 15:40:08,254 DEBUG [puppet-server] Puppet Automatically imported sysctl::value from sysctl/value into production
2015-05-26 15:40:08,260 DEBUG [puppet-server] Puppet importing '/etc/puppetlabs/code/environments/production/modules/rsyslog/manifests/init.pp' in environment production
2015-05-26 15:40:08,289 DEBUG [puppet-server] Puppet Automatically imported rsyslog from rsyslog into production
2015-05-26 15:40:08,304 DEBUG [puppet-server] Puppet importing '/etc/puppetlabs/code/environments/production/modules/openssl/manifests/init.pp' in environment production
2015-05-26 15:40:08,309 DEBUG [puppet-server] Puppet Automatically imported openssl from openssl into production
2015-05-26 15:40:08,313 DEBUG [puppet-server] Puppet importing '/etc/puppetlabs/code/environments/production/modules/nscd/manifests/init.pp' in environment production
2015-05-26 15:40:08,328 DEBUG [puppet-server] Puppet Automatically imported nscd from nscd into production
2015-05-26 15:40:08,336 DEBUG [puppet-server] Puppet importing '/etc/puppetlabs/code/environments/production/modules/openldap/manifests/init.pp' in environment production
2015-05-26 15:40:08,348 DEBUG [puppet-server] Puppet Automatically imported openldap from openldap into production
2015-05-26 15:40:08,355 DEBUG [puppet-server] Puppet importing '/etc/puppetlabs/code/environments/production/modules/x11/manifests/xauth.pp' in environment production
2015-05-26 15:40:08,369 DEBUG [puppet-server] Puppet Automatically imported x11::xauth from x11/xauth into production

It ends reproducible with the last line. but after the line is printed it takes about 2 minutes until it runs out of memory.

Xms/Xmx in /etc/sysconfig/puppetserver JAVA_ARGS is set to "-Xms2g -Xmx3g" (verified with "ps -ef" that it is actually running with these args).

 Any hints on how to debug further?

- Thomas

[Thomas Müller]

it seems it was caused by an 15 line custom function which worked for the last few years. replaced it by some stdlib funcntions and now it works.

other lessons learned:

- run puppet-lint on all manifests (and take advantage of the new nice --fix option) - fixes a lot of problems in old code

- run puppet parser validate on all manifests - helped to discover some utf8 problems

- Thomas

[Olivier Hanesse]

Hello Thomas, 

What was your function like ? I think I got the same issue of my side. 

Olivier