Puppet under passenger constantly crashing

[treydock]

I recently moved from manually configured Puppetmaster under passenger to fully managed using theforeman/puppet module.  Now I am experiencing constant crashes (every few minutes) of the passenger process that runs the puppetmaster.

Host is CentOS 6.5 running Puppet 3.4.3.

This is the entry I see in /var/log/httpd/puppet_error_ssl.log:

[Tue Mar 25 16:25:26 2014] [error] [client 127.0.0.1] Premature end of script headers: production

This is the entry I see in /var/log/httpd/error_log

/usr/lib/ruby/site_ruby/1.8/puppet/parser/ast.rb:49: [BUG] rb_gc_mark(): unknown data type 0x20(0x2e6b230) non object

ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

[ pid=28256 thr=139906534451168 file=ext/apache2/Hooks.cpp:841 time=2014-03-25 16:25:26.86 ]: The backend application (process 32724) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whe.

/etc/httpd/conf.d/passenger.conf:

LoadModule passenger_module modules/mod_passenger.so

<IfModule mod_passenger.c>

   PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.19

   PassengerRuby /usr/bin/ruby

   PassengerTempDir /var/run/rubygem-passenger

</IfModule>

/etc/httpd/conf.d/25-puppet.conf

# ************************************

# Vhost template in module puppetlabs-apache

# Managed by Puppet

# ************************************

<VirtualHost *:8140>

  ServerName puppet

  ## Vhost docroot

  DocumentRoot "/etc/puppet/rack/public/"

  ## Directories, there should at least be a declaration for /etc/puppet/rack/public/

  <Directory "/etc/puppet/rack/public/">

    AllowOverride None

    Order allow,deny

    Allow from all

    PassengerEnabled On

  </Directory>

  ## Load additional static includes

  ## Logging

  ErrorLog "/var/log/httpd/puppet_error_ssl.log"

  ServerSignature Off

  CustomLog "/var/log/httpd/puppet_access_ssl.log" combined

  ## SSL directives

  SSLEngine on

  SSLCertificateFile      "/var/lib/puppet/ssl/certs/puppet.<DOMAIN>.pem"

  SSLCertificateKeyFile   "/var/lib/puppet/ssl/private_keys/puppet.<DOMAIN>.pem"

  SSLCertificateChainFile "/var/lib/puppet/ssl/ca/ca_crt.pem"

  SSLCACertificatePath    "/etc/pki/tls/certs"

  SSLCACertificateFile    "/var/lib/puppet/ssl/ca/ca_crt.pem"

  SSLCARevocationFile     "/var/lib/puppet/ssl/ca/ca_crl.pem"

  SSLProtocol             -ALL +SSLv3 +TLSv1

  SSLCipherSuite          ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

  SSLVerifyClient         optional

  SSLVerifyDepth          1

  SSLOptions +StdEnvVars +ExportCertData

  ## Request header rules

  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader

  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e

  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e

  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

  RequestHeader unset X-Forwarded-For

  ## Custom fragment

</VirtualHost>

Any suggestions or means to work around this issue?

Thanks

- Trey

[treydock]

As an update, I tried running 'puppet master --no-daemonize --debug' and am seeing a segmentation fault running outside of passenger/apache...

/usr/lib/ruby/site_ruby/1.8/puppet/parser/scope.rb:555: [BUG] Segmentation fault

ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

<another run>

/usr/lib/ruby/1.8/pathname.rb:287: [BUG] rb_gc_mark(): unknown data type 0x10(0x935ce90) non object

ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

This only seems occur most frequently when I run "puppet agent --test" from the puppet master server.  Remote clients do not seem to crash puppet master as frequently.

[paul.gomersbach]

Hi Treydock,

Did you ever resolve this problem?

Thanks!

Op dinsdag 25 maart 2014 23:24:17 UTC+1 schreef treydock:

[Trey Dockendorf]

Sort of. The normal Puppet+Passenger configuration still crashes, but
for some odd reason if I add the following to the Puppet config.ru
(after the --confdir and --vardir lines) the crashes stop...

ARGV << "--debug"
ARGV << "--trace"
ARGV << "--profile"
ARGV << "--logdest" << "/var/log/puppet/puppetmaster.log"

I made those changes to try and debug the problem with help on IRC
some time ago, and while the actual problem couldn't be found, those
options kept Passenger from crashing when running Puppet. The full
config is at the end [1].

I have an almost identical Puppetmaster setup for a different
infrastructure, using same versions and same Puppet modules
(theforeman), that does not exhibit this behavior. My hunch is that
this problematic Puppetmaster is suffering from configuration drift.
It was setup by hand a long time ago with various Rubygems installed
manually that are likely conflicting. The system is now fully managed
but there are still some Rubygems installed that are not managed and
I've not taken the time to clean it up, but rather will eventually
rebuild this VM.

- Trey

[1]:
### Next part of the file is managed by a different template ###
## Module: 'puppet'
# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $LOAD_PATH.unshift('/opt/puppet/lib')

$0 = "master"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"

# Rack applications typically don't start as root. Set --confdir and --vardir
# to prevent reading configuration from ~puppet/.puppet/puppet.conf and writing
# to ~puppet/.puppet
ARGV << "--confdir" << "/etc/puppet"
ARGV << "--vardir" << "/var/lib/puppet"
ARGV << "--debug"
ARGV << "--trace"
ARGV << "--profile"
ARGV << "--logdest" << "/var/log/puppet/puppetmaster.log"

# NOTE: it's unfortunate that we have to use the "CommandLine" class
# here to launch the app, but it contains some initialization logic
# (such as triggering the parsing of the config file) that is very
# important. We should do something less nasty here when we've
# gotten our API and settings initialization logic cleaned up.
#
# Also note that the "$0 = master" line up near the top here is
# the magic that allows the CommandLine class to know that it's
# supposed to be running master.
#
# --cprice 2012-05-22

require 'puppet/util/command_line'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Util::CommandLine.new.execute

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/qWN6j-eNiZ0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/9389521a-1f19-4dff-8414-69816ce45dae%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dietrich, Stefan]

+1 here.
We see this problem kind of problem on Puppet 3.4.3 and Scientific Linux
6.5.

Only the files causing the segfault are different from your case:
/usr/lib/ruby/site_ruby/1.8/puppet/util/autoload.rb:88: [BUG]

Segmentation fault
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

/usr/lib/ruby/site_ruby/1.8/puppet/file_system/file.rb:152: [BUG]

Segmentation fault
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

/usr/lib/ruby/site_ruby/1.8/puppet/file_system/file.rb:152: [BUG]
rb_gc_mark(): unknown data type 0x38(0xf90be90) non object

ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

I am able to reproduce this issue with our manifests on a fresh
installation from scratch.

So far, only Ubuntu agents trigger this segfault on the master.
I found some workarounds:
- Backporting the patch from PUP-1592
- Upgrading to Puppet 3.6.x
- Removing a specific ordering from the manifests

I was even able to reproduce this on Puppet 3.6.2, if I _remove_ the
patch from PUP-1592.

So, I think this is an ruby 1.8.7 issue and the mentioned patch only
makes it harder to run into this issue.

Regards,
Stefan

[Cristian Falcas]

Incredible. This made it work for me also.

[Aimon Bustardo]

Interestingly it appears to be the '--profile' that makes it work for me. The rest are not needed.