Puppet Server Crashing often

[Makrand]

Hi All,

I am doing POC for puppet enterprise. Master is on CentOS 6.4 VM with 2GB ram. Puppet server is going down on its own.

[root@mss-pup-mst2 ~]# /etc/init.d/pe-puppetserver status
pe-puppetserver dead but pid file exists

I can start it without any issues, but it stays for like 30-60 Mins up. then It goes down again.

Logs are not hinting anything, but for one of shutdown, I found following lines. Surprisingly they appeared for only one shutdown incident. Not for others

2015-12-13 04:38:21,100 INFO  [p.t.internal] Shutting down due to JVM shutdown hook.
2015-12-13 04:38:21,363 INFO  [p.t.internal] Beginning shutdown sequence
2015-12-13 04:38:21,640 INFO  [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopping metrics sampler job
2015-12-13 04:38:21,990 INFO  [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopped metrics sampler job
2015-12-13 04:38:22,724 INFO  [p.t.s.w.jetty9-service] Shutting down web server(s).
2015-12-13 04:38:22,808 INFO  [p.t.s.w.jetty9-core] Shutting down web server.
2015-12-13 04:38:23,243 INFO  [o.e.j.s.ServerConnector] Stopped ServerConnector@4369ad74{SSL-HTTP/1.1}{0.0.0.0:8140}
2015-12-13 04:38:23,262 INFO  [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@3749adc6{/status,null,UNAVAILABLE}

After bit of googling, I found following

https://github.com/puppetlabs/puppet-server/blob/master/documentation/install_from_packages.markdown#memory-allocation

Thing is, I am not able to find  init file under /etc/sysconfig/puppetserver

to ask Puppetserver to use less RAM than 2GB. (This proabably seems reason for this)

Do anyone know, where I can set the RAM usage for puppetserver (PE 2015.2.3)?

--

Best,
Makrand

[Jeremy Barlow]

Has your Puppet Server written any content into the "/var/log/puppetlabs/puppetserver/puppetserver-daemon.log" file?  Memory-related errors will often be written here instead of to the "/var/log/puppetlabs/puppetserver/puppetserver.log" file.  For example, you may see something like:

#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 34514"...

I agree that the problem is very likely that 2 GB is too small for the needs of the basic Puppet Enterprise configuration.  This link, http://docs.puppetlabs.com/pe/2015.2/install_system_requirements.html#evaluation-environment, for example, recommends 6 GB of RAM for an monolithic evaluation configuration.

You might take a look at this link if you have not already - http://docs.puppetlabs.com/pe/2015.2/config_puppetserver.html.  This has some general advice on how to tune Puppet Server in PE, including how to configure the heap size for the Puppet Server JVM process.  I wouldn't recommend dropping the maximum heap size below the default of 2 GB but this documentation may be helpful to you for evaluating what the optimal configuration for your environment may be.

[Makrand]

Hi Jermy,

Just saw your email.

a) Surprisingly puppetserver-daemon.log is blank.....size zero

b) Found the init file /etc/sysconfig/pe-puppetserver. Its showing thing as below.

# Modify this if you'd like to change the memory allocation, enable JMX, etc
JAVA_ARGS="-Xms1024m -Xmx1024m"

Upon restart of puppet server service, the edited values will be reset to default 1024(tried setting it to 512MB).

c) Upon checking with PS found following.

[root@mss-pup-mst2 puppetserver]# ps -ef | grep java
493       1526     1  0 Dec17 ?        00:11:59 /opt/puppetlabs/server/apps/java/lib/jvm/java/jre/bin/java -Xms512m -Xmx512m -Dorg.apache.activemq.UseDedicatedTaskRunner=false -Djava.util.logging.config.file=logging.properties -Dhawtio.realm=activemq -Dhawtio.role=admins -Dhawtio.rolePrincipalClasses=org.apache.activemq.jaas.GroupPrincipal -Djava.security.auth.login.config=/etc/puppetlabs/activemq/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/puppetlabs/server/apps/activemq/tmp -Dactivemq.classpath=/etc/puppetlabs/activemq; -Dactivemq.home=/opt/puppetlabs/server/apps/activemq -Dactivemq.base=/opt/puppetlabs/server/apps/activemq -Dactivemq.conf=/etc/puppetlabs/activemq -Dactivemq.data=/opt/puppetlabs/server/data/activemq -jar /opt/puppetlabs/server/apps/activemq/bin/activemq.jar start
497       1618     1  0 Dec17 ?        00:21:55 /opt/puppetlabs/server/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xmx256m -Xms256m -cp /opt/puppetlabs/server/apps/puppetdb/puppetdb.jar clojure.main -m puppetlabs.puppetdb.main --config /etc/puppetlabs/puppetdb/conf.d -b /etc/puppetlabs/puppetdb/bootstrap.cfg
495       2129     1  0 Dec17 ?        00:08:54 /opt/puppetlabs/server/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xmx256m -Xms256m -cp /opt/puppetlabs/server/apps/console-services/console-services-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/console-services/conf.d -b /etc/puppetlabs/console-services/bootstrap.cfg

Thanks for the links. But I saw some people (mostly youtube demos) are using VM with 1Gigs and working.

With current resources in hand, I can allow 4GB for master node.

Makrand

 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/a3e57049-21da-41d4-a438-a03a1c24fb11%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best,
Makrand

[Chris Price]

On Wednesday, December 16, 2015 at 4:26:26 AM UTC-8, Makrand Sanap wrote:

Hi All,

I am doing POC for puppet enterprise. Master is on CentOS 6.4 VM with 2GB ram. Puppet server is going down on its own.

[root@mss-pup-mst2 ~]# /etc/init.d/pe-puppetserver status
pe-puppetserver dead but pid file exists

I can start it without any issues, but it stays for like 30-60 Mins up. then It goes down again.

Logs are not hinting anything, but for one of shutdown, I found following lines. Surprisingly they appeared for only one shutdown incident. Not for others

2015-12-13 04:38:21,100 INFO  [p.t.internal] Shutting down due to JVM shutdown hook.
2015-12-13 04:38:21,363 INFO  [p.t.internal] Beginning shutdown sequence
2015-12-13 04:38:21,640 INFO  [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopping metrics sampler job
2015-12-13 04:38:21,990 INFO  [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopped metrics sampler job
2015-12-13 04:38:22,724 INFO  [p.t.s.w.jetty9-service] Shutting down web server(s).
2015-12-13 04:38:22,808 INFO  [p.t.s.w.jetty9-core] Shutting down web server.
2015-12-13 04:38:23,243 INFO  [o.e.j.s.ServerConnector] Stopped ServerConnector@4369ad74{SSL-HTTP/1.1}{0.0.0.0:8140}
2015-12-13 04:38:23,262 INFO  [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@3749adc6{/status,null,UNAVAILABLE}

This is very unusual, I would not expect for you to see the message "Shutting down due to JVM shutdown hook" unless something on your system sent a signal to the service to stop or restart.

I presume you have a Puppet agent running on the same node; can you check its report and see if it shows anything about it trying to restart Puppet Server?  Or maybe try disabling the agent (`service puppet stop`) for a period of time and see if the weird behavior of the server only occurs when the agent is running?

 

[Makrand]

Chris,

Latest from my side:- I've upgraded memory of VM to 4GB and things are looking stable as of now. The JVM for Pupper sever is now using 2048MB (previous 1024MB) as default (according to init file for puppet server)

[root@mss-pup-mst2 ~]# /etc/init.d/pe-puppetserver status

pe-puppetserver (pid  2975) is running...
[root@mss-pup-mst2 ~]# ps -p 2975 -o etime=
   22:20:31

Seems increasing memory worked.

FYI, the last message for JVM last appeared on

2015-12-22 13:10:05,322 INFO  [puppet-server] Puppet Caching catalog for mss-pup-mst2.cs2cloud.internal
2015-12-22 13:10:05,766 INFO  [puppet-server] Puppet 'replace catalog' command for mss-pup-mst2.cs2cloud.internal submitted to PuppetDB wit
h UUID 5cd265a2-296b-4046-a4ca-f2933c26337d
2015-12-22 13:10:20,239 INFO  [p.t.internal] Shutting down due to JVM shutdown hook.
2015-12-22 13:10:20,242 INFO  [p.t.internal] Beginning shutdown sequence
2015-12-22 13:10:20,247 INFO  [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopping metrics sampler job
2015-12-22 13:10:20,252 INFO  [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopped metrics sampler job
2015-12-22 13:10:20,265 INFO  [p.t.s.w.jetty9-service] Shutting down web server(s).

After this I upgraded the memory and puppet server is running since then w/o any issues. Sorry I am too new to puppet and exploring it. Not sure how would I check the report. If you can give some pointers I would try seeing report in that time when JVM was crashing.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/59958791-b989-484e-add0-09d7298eb328%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Best,
Makrand