Outofmemory error for puppetserver process
889 views
Skip to first unread message
Jim
May 24, 2016, 2:54:07 AM5/24/16
to Puppet Users
Hey Guys,
I am wondering what is the problem with the following puppetserver process and how to have it resolved.
I am wondering what is the problem with the following puppetserver process and how to have it resolved.
Interesting thing is puppetserver is still running ! and is able to communicate it with agents so I am not sure what is the problem !
I still have 2GB of free memory on my VM
root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# ps -aef|grep -i puppet|grep -v grep
root 742 1 0 09:39 ? 00:00:02 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
root 1369 1 0 09:39 ? 00:00:08 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived --config=/etc/puppetlabs/mcollective/server.cfg --pidfile=/var/run/puppetlabs/mcollective.pid --daemonize
puppet 8044 1 0 09:53 ? 00:01:09 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xms256M -Xmx256M -XX:MaxPermSize=256m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# free -g
total used free shared buffers cached
Mem: 3 1 2 0 0 0
-/+ buffers/cache: 1 2
Swap: 3 0 3
root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
As you can see I have set puppetserver memory to as low as 256MB. I tried to change it to 512MB as well but still it reports "OutOfMemory" so kept it on 256MB anyway.
Here is the output which confirms that puppetserver is running.
root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# service puppetserver status
â— puppetserver.service - LSB: puppetserver
Loaded: loaded (/etc/init.d/puppetserver)
Active: active (running) since Tue 2016-05-24 12:53:06 AEST; 16s ago
Docs: man:systemd-sysv-generator(8)
Process: 10986 ExecStart=/etc/init.d/puppetserver start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/puppetserver.service
└─10992 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xms256M -Xmx256M -XX:MaxPermSize=256m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-se...
May 24 12:52:22 jim-Ubuntu1504 systemd[1]: Starting LSB: puppetserver...
May 24 12:53:06 jim-Ubuntu1504 systemd[1]: Started LSB: puppetserver.
root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
Any advice would be much appreciated.
Cheers
Jim
Johan De Wit
May 24, 2016, 5:39:05 AM5/24/16
to puppet...@googlegroups.com
Hi Jim,
You should find some answers here : https://docs.puppet.com/puppetserver/2.3/tuning_guide.html
per jruby instance, we assign 512mb tot the VM, so 256mb is a little too less.
hth
Johan
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/60c8e923-10ca-4d01-a09c-7e87a5036fdb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jim
May 24, 2016, 12:44:03 PM5/24/16
to Puppet Users
Hi Johan,
I did try 512m option not just 256m before posting. and it didn't work. The link which you have provided talks basically about
1. Changing parameters for java options
2. max active instances(Its commented out by default)
3.num-cpus configuration. I did recursive grep but couldn't find it. Puppetlabs says that they ship it anyway with low value so I don't think that should be cause or concerb.
I would probably get hold of logs and see if I can find any valuable information.
Kind Regards,
Jim
Jim
May 24, 2016, 12:51:58 PM5/24/16
to Puppet Users
Any idea which log file to look at when I run the service puppetserver start command?
Trevor Vaughan
May 25, 2016, 9:56:43 AM5/25/16
to puppet...@googlegroups.com
Hi Jim,
Your logs should be in /var/log/puppetserver/puppetserver.log.
Also, I'm guessing that you need to change the Xmx java option in your puppetserver start script.
Trevor
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/375e3af0-4592-4ac8-b87d-473b85107b70%40googlegroups.com.
Trevor Vaughan
Vice President, Onyx Point, Inc
Vice President, Onyx Point, Inc
-- This account not approved for unencrypted proprietary information --
Jim
May 26, 2016, 1:56:01 PM5/26/16
to Puppet Users
Hi Trevor,
No luck.
2016-05-26 23:46:26,745 INFO [Thread-1] [p.t.internal] Shutting down due to JVM shutdown hook.
2016-05-26 23:46:26,747 INFO [Thread-1] [p.t.internal] Beginning shutdown sequence
2016-05-26 23:46:26,761 INFO [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-agents] Flush request received; creating new JRuby pool.
2016-05-26 23:46:26,765 INFO [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-agents] Replacing old JRuby pool with new instance.
2016-05-26 23:46:26,765 INFO [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-agents] Swapped JRuby pools, beginning cleanup of old pool.
2016-05-26 23:46:26,770 INFO [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-internal] Cleaned up old JRuby instance with id 1.
2016-05-26 23:46:26,772 INFO [Thread-1] [p.t.s.w.jetty9-service] Shutting down web server(s).
2016-05-26 23:46:26,781 INFO [Thread-1] [p.t.s.w.jetty9-core] Shutting down web server.
2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.ServerConnector] Stopped ServerConnector@79303966{SSL-HTTP/1.1}{0.0.0.0:8140}
2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@2b87522b{/,null,UNAVAILABLE}
2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@1f2ffb51{/puppet-admin-api,null,UNAVAILABLE}
2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@609b2b8a{/puppet,null,UNAVAILABLE}
2016-05-26 23:46:26,825 INFO [Thread-1] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@5b1cbf6d{/puppet-ca,null,UNAVAILABLE}
2016-05-26 23:46:26,827 INFO [Thread-1] [p.t.s.w.jetty9-core] Web server shutdown
2016-05-26 23:46:26,837 INFO [Thread-1] [p.t.internal] Finished shutdown sequence
2016-05-26 23:49:33,878 INFO [main] [o.e.j.u.log] Logging initialized @14524ms
2016-05-26 23:49:41,413 INFO [async-dispatch-2] [p.s.v.versioned-code-service] No code-id-command set for versioned-code-service. Code-id will be nil.
2016-05-26 23:49:41,414 INFO [async-dispatch-2] [p.s.v.versioned-code-service] No code-content-command set for versioned-code-service. Attempting to fetch code content will fail.
2016-05-26 23:49:41,417 INFO [async-dispatch-2] [p.t.s.w.jetty9-service] Initializing web server(s).
2016-05-26 23:49:41,457 INFO [async-dispatch-2] [p.s.j.jruby-puppet-service] Initializing the JRuby service
2016-05-26 23:49:41,457 WARN [async-dispatch-2] [p.s.j.jruby-puppet-service] The 'jruby-puppet.use-legacy-auth-conf' setting is set to 'true'. Support for the legacy Puppet auth.conf file is deprecated and will be removed in a future release. Change this setting to 'false' and migrate your authorization rule definitions in the /etc/puppetlabs/puppet/auth.conf file to the /etc/puppetlabs/puppetserver/conf.d/auth.conf file.
2016-05-26 23:49:41,500 INFO [clojure-agent-send-pool-0] [p.s.j.jruby-puppet-internal] Creating JRuby instance with id 1.
2016-05-26 23:49:58,448 INFO [clojure-agent-send-pool-0] [puppet-server] Puppet Puppet settings initialized; run mode: master
2016-05-26 23:49:59,624 INFO [clojure-agent-send-pool-0] [p.s.j.jruby-puppet-agents] Finished creating JRubyPuppet instance 1 of 1
2016-05-26 23:49:59,636 INFO [async-dispatch-2] [p.s.c.puppet-server-config-core] Initializing webserver settings from core Puppet
2016-05-26 23:49:59,672 INFO [async-dispatch-2] [p.p.certificate-authority] CA already initialized for SSL
2016-05-26 23:49:59,674 INFO [async-dispatch-2] [p.s.c.certificate-authority-service] CA Service adding a ring handler
2016-05-26 23:49:59,749 INFO [async-dispatch-2] [p.p.certificate-authority] Master already initialized for SSL
2016-05-26 23:49:59,750 INFO [async-dispatch-2] [p.s.m.master-service] Master Service adding ring handlers
2016-05-26 23:49:59,813 INFO [async-dispatch-2] [p.s.p.puppet-admin-service] Starting Puppet Admin web app
2016-05-26 23:49:59,844 WARN [async-dispatch-2] [o.e.j.s.h.ContextHandler] Empty contextPath
2016-05-26 23:49:59,849 INFO [async-dispatch-2] [p.t.s.w.jetty9-service] Starting web server(s).
2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] webserver config overridden for key 'ssl-cert'
2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] webserver config overridden for key 'ssl-key'
2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] webserver config overridden for key 'ssl-ca-cert'
2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] webserver config overridden for key 'ssl-crl-path'
2016-05-26 23:50:00,249 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] Starting web server.
2016-05-26 23:50:00,260 INFO [async-dispatch-2] [o.e.j.s.Server] jetty-9.2.z-SNAPSHOT
2016-05-26 23:50:00,360 INFO [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@62f22fd{/puppet-ca,null,AVAILABLE}
2016-05-26 23:50:00,360 INFO [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@dc925ec{/puppet,null,AVAILABLE}
2016-05-26 23:50:00,360 INFO [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@55c827e3{/puppet-admin-api,null,AVAILABLE}
2016-05-26 23:50:00,360 INFO [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@3b82279{/,null,AVAILABLE}
2016-05-26 23:50:00,472 INFO [async-dispatch-2] [o.e.j.s.ServerConnector] Started ServerConnector@2fcb900d{SSL-HTTP/1.1}{0.0.0.0:8140}
2016-05-26 23:50:00,473 INFO [async-dispatch-2] [o.e.j.s.Server] Started @41120ms
2016-05-26 23:50:00,486 INFO [async-dispatch-2] [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
2016-05-26 23:50:00,488 INFO [async-dispatch-2] [p.s.l.legacy-routes-service] The legacy routing service has successfully started and is now ready to handle requests
2016-05-26 23:50:00,602 INFO [clojure-agent-send-off-pool-0] [p.d.version-check] Newer version 2.4.0 is available! Visit https://github.com/puppetlabs/puppetserver/tree/stable for details.
root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# service puppetserver status
â— puppetserver.service - LSB: puppetserver
Loaded: loaded (/etc/init.d/puppetserver)
Active: active (running) since Thu 2016-05-26 23:50:01 AEST; 4min 55s ago
Docs: man:systemd-sysv-generator(8)
Process: 4397 ExecStart=/etc/init.d/puppetserver start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/puppetserver.service
└─4403 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xms512m -Xmx512m -XX:MaxPermSize=512m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-ser...
May 26 23:49:19 jim-Ubuntu1504 systemd[1]: Starting LSB: puppetserver...
May 26 23:50:01 jim-Ubuntu1504 systemd[1]: Started LSB: puppetserver.
root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# cat /etc/default/puppetserver
###########################################
# Init settings for puppetserver
###########################################
# Location of your Java binary (version 7 or higher)
JAVA_BIN="/usr/bin/java"
# Modify this if you'd like to change the memory allocation, enable JMX, etc
JAVA_ARGS="-Xms512m -Xmx512m -XX:MaxPermSize=512m"
# These normally shouldn't need to be edited if using OS packages
USER="puppet"
GROUP="puppet"
INSTALL_DIR="/opt/puppetlabs/server/apps/puppetserver"
CONFIG="/etc/puppetlabs/puppetserver/conf.d"
BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetserver/bootstrap.cfg"
SERVICE_STOP_RETRIES=60
# START_TIMEOUT can be set here to alter the default startup timeout in
# seconds. This is used in System-V style init scripts only, and will have no
# effect in systemd.
# START_TIMEOUT=120
root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
Cheers
Jim
Matthaus Owens
May 26, 2016, 4:46:14 PM5/26/16
to Puppet Users
Jim,
For debugging failures on ubuntu 15.04, I would recommend trying `/opt/puppetlabs/bin/puppetserver foreground`, it may show you the errors that are causing the service to shutdown. Ubuntu 15.04 is a bit difficult to debug on, because while it has systemd support, the puppetserver packages for that platform still ship sysv init scripts. In my experience, the sysv-systemd integration seems to swallow some startup errors and messages that would usually go to journalctl on a systemd platform. We started shipping systemd service files for puppetserver starting in Ubuntu 16.04.
HTH
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/97f14473-d7cc-4cab-be6f-9e726a686dad%40googlegroups.com.
Jim
May 30, 2016, 12:46:30 PM5/30/16
to Puppet Users
Hi Matthaus,
My hostname is misleading. I upgraded at the time of posting to Ubuntu 15.10. Now I have upgraded to 16.04.
I ran the command you suggested instead of service puppetserver start and I don't see OutOfMemory errors process.
root@jim-Ubuntu1504:~# ps -aef|grep -i puppet|grep -v grep
root 1116 1 0 19:34 ? 00:00:02 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
root 1719 1 0 19:38 ? 00:00:08 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived --config=/etc/puppetlabs/mcollective/server.cfg --pidfile=/var/run/puppetlabs/mcollective.pid --daemonize
root 2644 1544 0 22:35 pts/0 00:00:00 bash /opt/puppetlabs/server/apps/puppetserver/cli/apps/foreground
root 2650 2644 0 22:35 pts/0 00:00:00 runuser puppet -s /bin/bash -c /usr/bin/java -Xms512m -Xmx512m -XX:MaxPermSize=512m -Dlogappender=STDOUT -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/bootstrap.cfg
puppet 2651 2650 55 22:35 ? 00:00:42 /usr/bin/java -Xms512m -Xmx512m -XX:MaxPermSize=512m -Dlogappender=STDOUT -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/bootstrap.cfg
root@jim-Ubuntu1504:~#
Also when I do see that error as like below when I do Ctrl + C and execute service puppetserver start I get the OutOfMemory error like below.
root@jim-Ubuntu1504:~# ps -aef|grep -i puppet|grep -v grep
root 1116 1 0 19:34 ? 00:00:02 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
root 1719 1 0 19:38 ? 00:00:08 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived --config=/etc/puppetlabs/mcollective/server.cfg --pidfile=/var/run/puppetlabs/mcollective.pid --daemonize
puppet 3386 1 66 22:41 ? 00:00:45 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xms512m -Xmx512m -XX:MaxPermSize=512m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
root@jim-Ubuntu1504:~#
I wonder if I am doing start-up incorrectly ? Either even in this state where it sayOutOfMemoryError my puppetserver still serves all of its agents. The reason I say so is I changed motd file on all agents + installed vim package on agents.
I am not sure which is the right method to start the server ? Also instead of running in foreground can I run it in background using nohup ?
Regards,
Jim
Matthaus Owens
May 30, 2016, 3:07:39 PM5/30/16
to puppet...@googlegroups.com
Jim,
The 'XX:OnOutOfMemoryError=kill -9 %p' in the process list doesn't indicate that puppet server is out of memory, it is a Java flag that tells Java what to do if puppet server hits an outofmemory error. The output you have pasted looks totally normal to me.
HTH
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/da1a5527-cd82-4b6d-adfd-5789b96ab32f%40googlegroups.com.
Jim
May 31, 2016, 2:37:15 PM5/31/16
to Puppet Users
Hi Matthaus,
Assume its a production environment would we have allowed that ? What I'm interested in clean state of process listing when I run ps command.
Is that possible?
Regards,
Jim
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
Matthaus Owens
May 31, 2016, 5:31:06 PM5/31/16
to Puppet Users
Jim,
What do you consider clean state of process listing? Puppetserver is run with those options in many production environments.
It certainly is possible to change how puppetserver starts up, but I wouldn't recommend it. The OnOutOfMemoryError setting is there so that when such an error occurs, the process comes down right away without attempting any recovery (you can see https://tickets.puppetlabs.com/browse/SERVER-1368 for more details about this).
If you do want to change how puppetserver starts up, you'll probably want to setup an override for the ExecStart of puppetserver. http://askubuntu.com/questions/659267/how-do-i-override-or-configure-systemd-services has some details on how to do this.
HTH
HTH
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/da1a5527-cd82-4b6d-adfd-5789b96ab32f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/abe4ab64-0ad6-48ae-8ac5-80ffc0800317%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages