[phusion-passenger] Apache2 web server segmentation fault with Passenger

[Rob]

Hi,

I am trying to debug an apache2 segmentation fault when using
Passenger. I have tried both Passenger 2.2.4 and 2.2.11.

Here is the stack trace leading up to the segmentation fault:

Program received signal SIGSEGV, Segmentation fault.
0xfdd5762a in Passenger::DirConfig::isEnabled (this=0x0) at
ext/apache2/Configuration.h:183
183                                     return enabled != DISABLED;
(gdb) where
#0  0xfdd5762a in Passenger::DirConfig::isEnabled (this=0x0) at
ext/apache2/Configuration.h:183
#1  0xfdd56250 in Hooks::prepareRequestWhenInHighPerformanceMode
(this=0x9a65f40, r=0x9ad0368) at ext/apache2/Hooks.cpp:1247
#2  0xfdd42eb1 in prepare_request_when_in_high_performance_mode
(r=0x9ad0368) at
ext/apache2/Hooks.cpp:1575
#3  0x0807f54b in ap_process_request_internal ()
#4  0x0809a0f1 in ap_process_request ()
#5  0x08095cc9 in ap_process_http_connection ()
#6  0x0809143e in ap_process_connection ()
#7  0x080a1eec in child_main ()
#8  0x080a3a0d in ap_mpm_run ()
#9  0x0806c2a9 in main ()

Looks like the call to getDirconfig() in Hooks.cpp line 1246 returns
NULL unexpectedly:

    int prepareRequestWhenInHighPerformanceMode(request_rec *r) {
        DirConfig *config =
getDirConfig(r); <- line
1246
        if (config->isEnabled() && config->highPerformanceMode()) {
            if (prepareRequest(r, config, r->filename, true)) {
                return OK;
            } else {
                return DECLINED;
            }
        } else {
            return DECLINED;
        }
    }

Looking at the implementation of getDirconfig():

    inline DirConfig *getDirConfig(request_rec *r) {
        return (DirConfig *) ap_get_module_config(r->per_dir_config,
&passenger_module);
    }

it appears as if the call to ap_get_module_config() returns NULL.

Any thoughts why NULL would be returned?

I am using Sun's Coolstack Apache v2.2.15 compiled from source using
Sun Studio 12 compiler on a Solaris 10 x86 platform.

I notice that mod_passenger.so is compiled using GNU g++. Would
mod_passenger.so generated using GNU g++ be compatible with Apache
compiled using Sun Studio 12?

I did manage to get Passenger to compile with Sun Studio 12 and I get
the segmentation fault in exactly the same place.

I have Passenger running rock solid on other Solaris servers using
Sun's "Webstack" packaged version of Apache.

Thanks in advance for any insights you can provide.

--Rob

Here is a little more environment information:

uname -a
SunOS coiweb1-d 5.10 Generic_141445-09 i86pc i386 i86pc Solaris

/opt/coolstack/apache2/bin/httpd -V
Server version: Apache/2.2.15 (Unix)
Server built:   Apr 16 2010 15:51:09
Server's Module Magic Number: 20051115:24
Server loaded:  APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_PROC_PTHREAD_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/opt/coolstack/apache2"
 -D SUEXEC_BIN="/opt/coolstack/apache2/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

gem list

*** LOCAL GEMS ***

actionmailer (2.2.2, 2.0.2)
actionpack (2.2.2, 2.0.2)
activerecord (2.2.2, 2.0.2)
activerecord-oracle_enhanced-adapter (1.2.2)
activeresource (2.2.2, 2.0.2)
activesupport (2.2.2, 2.0.2)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10, 1.0.9)
fastthread (1.0.7, 1.0.1)
gem_plugin (0.2.3)
mongrel (1.1.5, 1.1.3)
passenger (2.2.11)
rack (1.1.0)
rails (2.2.2, 2.0.2)
rake (0.8.7, 0.8.1)
ruby-oci8 (1.0.6)

--
You received this message because you are subscribed to the Google Groups "Phusion Passenger Discussions" group.
To post to this group, send email to phusion-...@googlegroups.com.
To unsubscribe from this group, send email to phusion-passen...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/phusion-passenger?hl=en.

[Hongli Lai]

This is the first time I've seen getDirConfig() returning NULL, I
didn't think that would be possible.

Could you post your entire Apache config? I wonder whether that might
have anything to do with it.
I don't think the compiler is at fault. The Apache API is pure C and C
generally has a stable ABI even across different compilers.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Rob]

Yes, I agree this could be configuration related. The actual number of
Passenger related configuration lines is pretty small, I would think a
configuration problem would jump out at me. I'm concerned the problem
has something to do with how our Apache instance is compiled (which is
one big difference from the other environments where Passenger is
working for me) or possibly file permission related.

Thank you for taking a look at my configuration: Here are the Apache
configuration files:

--Rob

httpd.conf:
ServerRoot "/opt/coolstack/apache2"

Listen 0.0.0.0:80

#CoreDumpDirectory /tmp/apache2-gdb-dump

ServerSignature Off
ServerTokens Full
Options -Indexes
Options -Includes

LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_anon_module modules/mod_authn_anon.so

LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_cert_module modules/mod_auth_cert.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule headers_module modules/mod_headers.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule mime_module modules/mod_mime.so
LoadModule status_module modules/mod_status.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dir_module modules/mod_dir.so

<IfModule !mpm_netware_module>
User websrvr
Group dev
</IfModule>

ServerAdmin he...@example.com

ServerName coi.example.com:80

DocumentRoot "/export/home/websrvr/rails/current/public"

LoadModule passenger_module /opt/coolstack/lib/ruby/gems/1.8/gems/
passenger-2.2.11/ext/apache2/mod_passenger.so
PassengerRoot /opt/coolstack/lib/ruby/gems/1.8/gems/passenger-2.2.11
PassengerRuby /opt/coolstack/bin/ruby
RailsEnv development
PassengerLogLevel 9

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Directory "/export/home/websrvr/rails/current/public">
    Options Indexes FollowSymLinks -MultiViews
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>


<Directory "/opt/coolstack/apache2/htdocs">
    Options Indexes FollowSymLinks

    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

ErrorLog "/opt/coolstack/apache2/logs/error_log"

LogLevel debug

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-
Agent}i\""
combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-
Agent}i\" %I
%O" combinedio
    </IfModule>

    CustomLog "/opt/coolstack/apache2/logs/access_log" common

</IfModule>

<IfModule rewrite_module>
#RewriteEngine On

#Disable TRACE Method
#RewriteCond "%{REQUEST_METHOD}" ^TRACE
#RewriteRule ".*" - [F]

# Redirect all port 80 traffic to secure HTTPS/port 443
#RewriteCond "%{SERVER_PORT}" "^80$"
#RewriteRule "^(.*)$" "https://%{SERVER_NAME}$1" [R,L]

</IfModule>

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/opt/coolstack/apache2/cgi-bin/"
</IfModule>

<Directory "/opt/coolstack/apache2/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

DefaultType text/plain

<IfModule mime_module>
    TypesConfig conf/mime.types

    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
</IfModule>

ErrorDocument 400 "400 Error - Please contact your administrator"
ErrorDocument 401 "401 Error - Please contact your administrator"
ErrorDocument 403 "403 Error - Please contact your administrator"
ErrorDocument 404 "404 Error - Please contact your administrator"
ErrorDocument 405 "405 Error - Please contact your administrator"
ErrorDocument 406 "406 Error - Please contact your administrator"
ErrorDocument 500 "500 Error - Please contact your administrator"
ErrorDocument 501 "501 Error - Please contact your administrator"
ErrorDocument 503 "503 Error - Please contact your administrator"

# Server-pool management (MPM specific)
Include conf/extra/httpd-mpm.conf

<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>


conf/extra/httpd-mpm.conf:
#
# Server-Pool Management (MPM specific)
#

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
#
# Note that this is the default PidFile for most MPMs.
#
<IfModule !mpm_netware_module>
    PidFile "logs/httpd.pid"
</IfModule>

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
<IfModule !mpm_winnt_module>
<IfModule !mpm_netware_module>
LockFile "logs/accept.lock"
</IfModule>
</IfModule>

#
# Only one of the below sections will be relevant on your
# installed httpd.  Use "apachectl -l" to find out the
# active mpm.
#

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept
spare
# MaxSpareServers: maximum number of server processes which are kept
spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process
serves
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept
spare
# MaxSpareThreads: maximum number of worker threads which are kept
spare
# ThreadsPerChild: constant number of worker threads in each server
process
# MaxRequestsPerChild: maximum number of requests a server process
serves
<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

# BeOS MPM
# StartThreads: how many threads do we initially spawn?
# MaxClients:   max number of threads we can have (1 thread == 1
client)
# MaxRequestsPerThread: maximum number of requests each thread will
process
<IfModule mpm_beos_module>
    StartThreads            10
    MaxClients              50
    MaxRequestsPerThread 10000
</IfModule>

# NetWare MPM
# ThreadStackSize: Stack size allocated for each worker thread
# StartThreads: Number of worker threads launched at server startup
# MinSpareThreads: Minimum number of idle threads, to handle request
spikes
# MaxSpareThreads: Maximum number of idle threads
# MaxThreads: Maximum number of worker threads alive at the same time
# MaxRequestsPerChild: Maximum  number of requests a thread serves. It
is
#                      recommended that the default value of 0 be set
for this
#                      directive on NetWare.  This will allow the
thread to
#                      continue to service requests indefinitely.
<IfModule mpm_netware_module>
    ThreadStackSize      65536
    StartThreads           250
    MinSpareThreads         25
    MaxSpareThreads        250
    MaxThreads            1000
    MaxRequestsPerChild      0
    MaxMemFree             100
</IfModule>

# OS/2 MPM
# StartServers: Number of server processes to maintain
# MinSpareThreads: Minimum number of idle threads per process,
#                  to handle request spikes
# MaxSpareThreads: Maximum number of idle threads per process
# MaxRequestsPerChild: Maximum number of connections per server
process
<IfModule mpm_mpmt_os2_module>
    StartServers           2
    MinSpareThreads        5
    MaxSpareThreads       10
    MaxRequestsPerChild    0
</IfModule>

# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server
process
# MaxRequestsPerChild: maximum  number of requests a server process
serves
<IfModule mpm_winnt_module>
    ThreadsPerChild      150
    MaxRequestsPerChild    0
</IfModule>

On May 5, 4:34 pm, Hongli Lai <hon...@phusion.nl> wrote:

> E-mail: i...@phusion.nl

[Hongli Lai]

On Thu, May 6, 2010 at 3:55 PM, Rob <acts....@gmail.com> wrote:
> Yes, I agree this could be configuration related. The actual number of
> Passenger related configuration lines is pretty small, I would think a
> configuration problem would jump out at me. I'm concerned the problem
> has something to do with how our Apache instance is compiled (which is
> one big difference from the other environments where Passenger is
> working for me) or possibly file permission related.
>
> Thank you for taking a look at my configuration: Here are the Apache
> configuration files:

I don't see anything obviously wrong in the config. I've looked on the
Internet and it looks like ap_get_module_config is never supposed to
return NULL.

The dir config is created in Configuration.cpp,
create_dir_config_struct, where it is created with the C++ new
operator. This operator never returns NULL, and instead throws an
exception if it cannot allocate memory, so that cannot be it either.

Maybe it really is a compiler problem. I've actually seen a bunch of
compiler problems on Solaris while testing Phusion Passenger for Nginx
on Solaris, but so far I've been able to work around them. This one
blows my mind though.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/

E-mail: in...@phusion.nl