segmentation fault 11 when loading mod_wsgi from apache

[Allen Lee]

(also crossposted to https://bugs.launchpad.net/ius/+bug/1341325 but wasn't sure where the appropriate place for this is) 

I've been running python27-mod_wsgi 4.2.4 with python27 from the IUS repositories alongside apache 2.2.3 on RHEL 5 to host a Django app running within a virtualenv. I haven't had any problems until this weekend when Apache has started to segfault constantly when trying to load the python interpreter - messages like (child pid 12574 exit signal Segmentation fault (11)).

I've read through https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions#Apache_Process_Crashes and can confirm that:

1/ mod_python is not installed, and 

2/ httpd is loaded with expat 1.95.8 and my virtualenv python is running 2.0.1 but I didn't encounter a segfault when running:

(in-virtualenv) % LD_PRELOAD=/lib64/libexpat.so.0 python
Python 2.7.7 (default, Jun 4 2014, 17:09:35)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyexpat
>>> pyexpat.version_info
(1, 95, 8)
>>>

I dumped the core and extracted the following from gdb but am at a loss at this point, any additional pointers to try to identify the source of the problem would be much appreciated! 

Results of gdb /usr/sbin/httpd /path/to/core.dump:
<snipped>
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff1dffd000
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6
(gdb) where
#0 0x00002aed3f67ef40 in strlen () from /lib64/libc.so.6
#1 0x00002aed4a24e43b in PyString_FromString () from /usr/lib64/libpython2.7.so.1.0
#2 0x00002aed49f8124b in ?? ()
#3 0x00002aed00005168 in ?? ()
#4 0x00002aed49f97190 in ?? ()
#5 0x00002aed4a552e00 in ?? () from /usr/lib64/libpython2.7.so.1.0
#6 0x00000000ffffffff in ?? ()
#7 0x00002aed49f97190 in ?? ()
#8 0x00002aed3d826cf0 in ?? ()
#9 0x00002aed4a1a0b98 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x00002aed3d826cf0 in ?? ()
#12 0x0000000000000000 in ?? ()
(gdb)

# httpd -V
Server version: Apache/2.2.3
Server built: Mar 26 2014 08:47:55
Server's Module Magic Number: 20051115:3
Server loaded: APR 1.5.0, APR-Util 1.5.3
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

[Graham Dumpleton]

What did you do just prior to the weekend that changed anything?

Did you upgrade the Python installation you are dependent on, but not recreate the Python virtual environment against the updated Python installation?

Did you apply any update to the Apache version being used?

Did you change the Apache configuration to load any additional Apache modules?

Did you change the Apache configuration for any modules you were already using?

How many Python installations exist on the system and where are they installed?

What Python installation is your Python virtual environment based off?

What Python installation was mod_wsgi compiled against?

Is the mod_wsgi.so finding the correct Python shared library for the Python installation you expect it to use?

http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library

Are you using mod_wsgi daemon mode?

Is Apache segfaulting on startup in all child processes, or only for mod_wsgi daemon mode processes?

Is the segfault truly on startup, or only on the first request against a process?

Sorry for all the questions, but there isn't a lot to go on at this point.

The point of these questions is to prompt you to validate certain things about your installation and ensure they are correct, but responses to them will also help me to understand your setup and so be able to work out what the issue may be.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Allen Lee]

Hi Graham,

Thank you for the fast response! I've been checking to see what happened prior and it looks the relevant packages that changed between the working -> non-working state were from mod_wsgi 4.1.3 to 4.2.4 on 7/8  - the errors started after logrotate reloaded apache over the weekend. Responses to questions included inline:

On Sunday, July 13, 2014 11:34:15 AM UTC-7, Graham Dumpleton wrote:

What did you do just prior to the weekend that changed anything?

This must have been caused by a package update earlier in the weekj that manifested itself after httpd restarted at 4 AM this morning from a weekly logrotate. Possibly relevant entries from yum.log:

Jun 04 14:16:42 Installed: python27-backports-1.0-1.ius.el5.x86_64

Jun 04 14:16:43 Installed: python27-backports-ssl_match_hostname-3.4.0.2-1.ius.el5.noarch

Jun 04 14:16:47 Installed: python27-setuptools-3.6-1.ius.el5.noarch

Jun 04 14:16:52 Erased: python27-distribute

Jun 09 00:11:07 Updated: python27-libs-2.7.6-3.ius.el5.x86_64

Jun 09 00:11:07 Updated: python27-2.7.6-3.ius.el5.x86_64

Jun 09 00:11:08 Updated: python27-devel-2.7.6-3.ius.el5.x86_64

Jun 09 15:04:21 Installed: python27-virtualenv-1.11.6-1.ius.el5.noarch

Jun 09 15:05:59 Erased: python27-virtualenv

Jun 09 15:06:10 Erased: python-virtualenv

Jun 20 14:53:48 Updated: python27-libs-2.7.7-1.ius.el5.x86_64

Jun 20 14:53:48 Updated: python27-2.7.7-1.ius.el5.x86_64

Jun 20 14:53:48 Updated: python27-backports-1.0-2.ius.el5.x86_64

Jun 20 14:53:48 Updated: python27-mod_wsgi-4.1.3-1.ius.el5.x86_64

Jun 20 14:53:49 Updated: python27-backports-ssl_match_hostname-3.4.0.2-2.ius.el5.noarch

Jun 20 14:54:07 Updated: python27-setuptools-4.0.1-1.ius.el5.noarch

Jun 20 14:54:08 Updated: python27-devel-2.7.7-1.ius.el5.x86_64

Jul 03 16:05:09 Updated: python27-setuptools-5.1-1.ius.el5.noarch

Jul 08 16:21:50 Updated: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64

Jul 11 17:58:11 Updated: python27-setuptools-5.2-1.ius.el5.noarch

Jul 13 08:49:36 Installed: python27-mod_wsgi-4.2.4-1.ius.el5.x86_64

Jul 13 08:53:16 Installed: python27-mod_wsgi-debuginfo-4.2.4-1.ius.el5.x86_64

Jul 13 09:06:15 Erased: subversion-python

 

Did you upgrade the Python installation you are dependent on, but not recreate the Python virtual environment against the updated Python installation?

I just checked this and noticed that my virtualenv Python was indeed different from the updated Python installation. I recreated the Python virtualenv using the updated system Python 2.7 (now there is no diff between virtualenv Python and system Python) and restarted the server but it still continually segfaults like it used to.

 

Did you apply any update to the Apache version being used?

Nope.

 

Did you change the Apache configuration to load any additional Apache modules?

Nope. 

Did you change the Apache configuration for any modules you were already using?

Nope. 

How many Python installations exist on the system and where are they installed?

There are actually 3 installs of Python on this antiquated VM, the stock RHEL 5 python 2.4, python2.6 (which I can remove though it would take byobu with it which I rather like), and python2.7 from the IUS repositories. Python 2.4 and 2.6 are RHEL stock installs, 2.7 is from the IUS repo.

 

What Python installation is your Python virtual environment based off?

Python 2.7

 

What Python installation was mod_wsgi compiled against?

Python2.7, using the python27-mod_wsgi from the IUS repo as well.

 

Is the mod_wsgi.so finding the correct Python shared library for the Python installation you expect it to use? 

http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library

Yes, I did check this and it appears to be correct:

# ldd /usr/lib64/httpd/modules/python27-mod_wsgi.so 

        linux-vdso.so.1 =>  (0x00007fff7e577000)

        libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00002b99ffd5d000)

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b9a00121000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00002b9a0033e000)

        libutil.so.1 => /lib64/libutil.so.1 (0x00002b9a00542000)

        libm.so.6 => /lib64/libm.so.6 (0x00002b9a00745000)

        libc.so.6 => /lib64/libc.so.6 (0x00002b9a009c9000)

        /lib64/ld-linux-x86-64.so.2 (0x0000003822a00000)

 

Are you using mod_wsgi daemon mode?

Yes. I tried switching to embedded mode but it exhibits the same behavior. Here's the wsgi related config from my Apache vhost:

WSGISocketPrefix run/wsgi

WSGIPythonHome /opt/virtualenvs/vcweb

<VirtualHost *:80>

    ...

    WSGIDaemonProcess vcweb user=apache group=commons threads=25 python-path=/opt/vcweb:/opt/virtualenvs/vcweb/lib/python2.7/site-packages

    WSGIProcessGroup vcweb

    WSGIScriptAlias / /opt/vcweb/wsgi.py

    ...

</VirtualHost>

Is Apache segfaulting on startup in all child processes, or only for mod_wsgi daemon mode processes?

It appears that it's only for mod_wsgi daemon mode processes, php and Java sites also served by the same webserver are still up.

Is the segfault truly on startup, or only on the first request against a process?

Yes, as soon as the server starts up my base httpd error.log starts spamming child pid segfaults (exit 11) and the vhost error.log spams messages along these lines:

[Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Attach interpreter ''.

[Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding '/opt/vcweb' to path.

[Sun Jul 13 12:00:21 2014] [info] mod_wsgi (pid=24718): Adding '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.

Sorry for all the questions, but there isn't a lot to go on at this point.

The point of these questions is to prompt you to validate certain things about your installation and ensure they are correct, but responses to them will also help me to understand your setup and so be able to work out what the issue may be.

Thank you for taking a look, much appreciated! I've been planning to upgrade these VMs to RHEL 7 this summer anyways so there's no need to plunge a lot of resources into this if you don't see anything immediately untoward. 

[Graham Dumpleton]

One quick question while I think about what changed between 4.1.3 and 4.2.4.

When you install a newer version of mod_wsgi, does it get compiled from source code against the specific Apache installation you are using, or are you using a binary package?

If using a binary package for mod_wsgi and it was compiled for a newer version of Apache than you are actually using, that can be a source of problems.

Is the Apache version you are using (which is actually very old), the latest Apache version available for your distribution?

In general I would only recommend using at least Apache 2.2.18ish as minimum as there were some issues in Apache which were fixed around Apache 2.2.16-2.2.17 which were causing some issues for mod_wsgi. At least the reports seemed to vanish for newer versions. Exactly what the problem was I am not sure, but believe it was related to SSL changes in Apache.

Graham

[Graham Dumpleton]

Can you ensure that LogLevel directive in Apache is set to 'debug'.

Also add the directive:

WSGIVerboseDebugging On

at global scope.

The give me a snippet of the log messages from Apache/mod_wsgi that lead up to the segfault.

I can possibly narrow it down from that.

Graham

[Allen Lee]

I'm pretty sure it is a binary package that gets installed, though I'm not sure what version of httpd it is getting compiled against, I would presume it is the stock RHEL 5 httpd though. It is the latest version of httpd for RHEL 5. I've provisioned a new RHEL 7 instance and mod_wsgi is working fine there, so I may just chalk this up as the final straw for finally upgrading our VMs. 

The LogLevel is set to debug and I added the WSGIVerboseDebugging but didn't get any different output in the root error.log (constant segfaults from child pid) and vhost error.log.. 

[Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Attach interpreter ''.

[Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding '/opt/vcweb' to path.

[Sun Jul 13 04:04:13 2014] [info] mod_wsgi (pid=5845): Adding '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.

[Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Attach interpreter ''.

[Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding '/opt/vcweb' to path.

[Sun Jul 13 04:04:14 2014] [info] mod_wsgi (pid=5854): Adding '/opt/virtualenvs/vcweb/lib/python2.7/site-packages' to path.

[Graham Dumpleton]

Can you add to the directory:

/opt/vcweb

two empty files called:

mod_wsgi.py

apache.py

and restart and look then to see if the debug logging outputs:

mod_wsgi (pid=%d): Imported 'mod_wsgi'.

mod_wsgi (pid=%d): Imported 'apache'.

If they are output, then it I know it is getting past that point.

I suspect it may be dying in the following code which was changed in the newer version.

    str = ap_get_server_description();                                          

#if PY_MAJOR_VERSION >= 3                                                       

    object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);                    

#else                                                                           

    object = PyString_FromString(str);                                          

#endif                                                                          

    PyModule_AddObject(module, "description", object);                          

                                                                                

    str = MPM_NAME;                                                             

#if PY_MAJOR_VERSION >= 3                                                       

    object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);                    

#else                                                                           

    object = PyString_FromString(str);                                          

#endif                                                                          

    PyModule_AddObject(module, "mpm_name", object);                             

                                                                                

    str = ap_get_server_built();                                                

#if PY_MAJOR_VERSION >= 3                                                       

    object = PyUnicode_DecodeLatin1(str, strlen(str), NULL);                    

#else                                                                           

    object = PyString_FromString(str);                                          

#endif                                                                          

    PyModule_AddObject(module, "build_date", object);                           

Either the old Apache is returning NULL for some of the Apache calls, or Redhat in modifying the server to suppress the server description, is causing a NULL to be returned.

What you might do is check if your Apache configuration has:

ServerSignature Off 

and set it back to:

ServerSignature On

and try again.

Even with that as Off it still should return an empty string and not a NULL though. But then, for a standard Apache description, that flag should affect the response from ap_get_server_description() anyway.

Graham

[A Lee]

Hi Graham,

I added the two files - it does emit "Imported 'mod_wsgi'." but *not*
"Imported 'apache'."

My apache config does indeed have

ServerSignature On

set properly.

One thing strange I noticed related to ap_get_server_description -
after I removed mod_perl v2.0.4-6 in an effort to remove
unused/unneeded modules, httpd wouldn't even start if
python27-mod_wsgi.so was loaded, complaining "Cannot load
/etc/httpd/modules/python27-mod_wsgi.so into server: undefined symbol:
ap_get_server_description". Removing the mod_wsgi load without
mod_perl installed allowed httpd to start up normally. I'm not sure if
this is related or not though...




On Sun, Jul 13, 2014 at 8:39 PM, Graham Dumpleton

> You received this message because you are subscribed to a topic in the
> Google Groups "modwsgi" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/modwsgi/DnvZ8lWokYs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Graham Dumpleton]

Arrrgggh.

Changes with Apache 2.2.4

*) The full server version information is now included in the error log at
startup as well as server status reports, irrespective of the setting
of the ServerTokens directive. ap_get_server_version() is now
deprecated, and is replaced by ap_get_server_banner() and
ap_get_server_description(). [Jeff Trawick]

So two possibilities here.

Whoever compiled the mod_wsgi binary, if they did in fact compile against Apache 2.2.3, didn't see the warning at compilation time about the prototype for that function not being found and also did no testing of mod_wsgi against that Apache version so as to notice that it would crash things on startup.

Or, it was compiled against a newer Apache version which has that function and did work okay.

I will need to change the mod_wsgi code to check for the Apache module magic number of 20051115.4 to see if the function is available in the version being compiled against and if not use other means to generate the information, or simply not include it.

You might try and find out via the Redhat issue you created what version of Apache the mod_wsgi binary was compiled against and point out that it has been found that mod_wsgi will not work with Apache 2.2.3 and earlier and mod_wsgi will need a fix to work with such old versions.

I will try and create a new version of mod_wsgi tomorrow.

Graham

[Graham Dumpleton]

If you were using binary packages, compiling from source code may not be an option. If it is, can you try:

https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.tar.gz

That addresses the issue with the missing ap_get_server_description() on older Apache versions.

If you can't easily check then let me know and I will officially release it anyway.

Graham

[A Lee]

Just downloaded & installed it and it worked like a charm - thanks!!

Steps (in case anyone has the same obscure problem):
1. sudo yum install httpd-devel.x86_64
2. ./configure --with-python=python2.7 && make
3. sudo cp src/server/.libs/mod_wsgi.so
/usr/lib64/httpd/modules/py27-mod_wsgi.so
4. replaced the module load statement in the conf.d/mod_wsgi.conf to
point at py27-mod_wsgi.so
5. restarted apache

[cwg...@gmail.com]

Howdy Graham,

I work on the IUS Community project, and was responsible for packaging up the affected version of python27-mod_wsgi.  We use a custom build farm based around mock to create the rpm packages.  I check back and compared the build logs between CentOS 5 and 6 and found these warnings in 5 only.

src/server/wsgi_interp.c: In function 'newInterpreterObject':

src/server/wsgi_interp.c:1171: warning: implicit declaration of function 'ap_get_server_description'

src/server/wsgi_interp.c:1171: warning: assignment makes pointer from integer without a cast

The build still completed in spite of the warnings.  Looking through github, it looks like this was introduced in 4.2.0.  I can't find the same line in 4.1.3.

https://github.com/GrahamDumpleton/mod_wsgi/blame/4.2.0/src/server/wsgi_interp.c#L1171

Once we create a package, we place it in our testing repos for about two weeks for our community to test.  During this time we also ensure that the package installs cleanly in a mock environment.  If we don't get any negative feedback during this testing period, we push the package into our stable repos.  I packaged python27-mod_wsgi-4.2.4-1.ius on 2014-06-19 and placed it into the testing repos.  Two weeks later was a Friday, so I waited a few more days until 2014-07-07 to push the package to the stable repos.  The first problem with it we heard of was Allen's bug report on 2014-07-13.

We look forward to packaging up 4.2.6 to resolve this bug.  Will you be tagging that release soon?

[Graham Dumpleton]

I am at an internal work conference in a foreign land for a couple of days so may be hard, but now that Allen has confirmed that it fixes the issues, I will try and get the release tagged at least.

BTW, what are the up to date URLs I should be using for finding mod_wsgi on the IUS repos. The links in docs are broken but I don't know what to use.

Thanks.

Graham

[cwg...@gmail.com]

You can browse our master mirror.  Here is the full path to the rpm in question.

http://dl.iuscommunity.org/pub/ius/stable/CentOS/5/x86_64/python27-mod_wsgi-4.2.4-1.ius.centos5.x86_64.rpm

You can also query dmirr to get a list of mirrors.

http://dmirr.iuscommunity.org/mirrorlist/?repo=ius-centos5&arch=x86_64

[cwg...@gmail.com]

Also, where in the docs are you referring to?  I would love to correct a broken link if needed.

[Graham Dumpleton]

Is there a link to a web page I can use which doesn't bind to a specific mod_wsgi version and which lists all available module versions.

I would rather not be linking to specific version RPMs.

The old links, which are now all broken can be found at:

http://code.google.com/p/modwsgi/wiki/InstallationOnLinux#RHEL/Centos_Packages

I simply have no idea what it is safe to change it to which will not be immediately made out of date when a new package comes along.

Graham

[Graham Dumpleton]

Version 4.2.6 of mod_wsgi with Apache <=2.2.3 fix tagged/released and available from:

https://github.com/GrahamDumpleton/mod_wsgi/releases/tag/4.2.6

Getting it up on PyPi for pip install will have to wait until later as conference break is over.

Graham

[cwg...@gmail.com]

Ahh I see.  The links are broken because both python26 and python31 are end-of-life and have moved to the archive repos.  Here are some current links.

http://dl.iuscommunity.org/pub/ius/stable/Redhat/5/x86_64/repoview/python27-mod_wsgi.html

http://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/repoview/python27-mod_wsgi.html

http://dl.iuscommunity.org/pub/ius/stable/CentOS/5/x86_64/repoview/python27-mod_wsgi.html

http://dl.iuscommunity.org/pub/ius/stable/CentOS/6/x86_64/repoview/python27-mod_wsgi.html

We also have versions for python32 and python33 (and python34u in testing).

[Graham Dumpleton]

Thanks. Have updated:

http://code.google.com/p/modwsgi/wiki/InstallationOnLinux

with the newer links.

The pip installable version of mod_wsgi 4.2.6 is now also available on PyPi.

https://pypi.python.org/pypi/mod_wsgi

Graham

[cwg...@gmail.com]

Awesome, I have rebuilt our packages and we will release them later today.

...