Do Apache and mod_wsgi need to use the same python while build

430 views
Skip to first unread message

Cate Liu

unread,
Jan 8, 2015, 2:59:09 PM1/8/15
to mod...@googlegroups.com
Hi,

I have a small Django web application.  If I build mod_wsgi (tried both version 3.4 and 4.4.5) using the system python (2.6.6), then I can run my app on a web page.
But if I use a python package (say, Ureka with python 2.7.5) to build mod_wsgi, then it doesn't work.  
I've triple checked the check list from https://code.google.com/p/modwsgi/wiki/CheckingYourInstallation, which shows that the mod_wsgi.so is indeed using the
correct libypython.so.  What am I missing here?

In my apache conf file (attached below), if I comment out the wsgi module loading, then I can get to the static web page.  So it has to do with mod_wsgi.so.
Where is the mismatch that prevents this app to work?

Thanks in advance for any insights?

Cate

----------------------------------------------------------------------------------------

Here is my environment:
>>>httpd -V
Server version: Apache/2.2.15 (Unix)
Server built:   Aug  2 2013 08:02:15
Server's Module Magic Number: 20051115:25
Server loaded:  APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

My apache conf file (only part of it):
######
User apache
Group apache

LoadModule wsgi_module /usr/lib64/httpd/modules/mod_wsgi.so


# what port number will we listen on
#####  Listen 134.4.122.204:5000
Listen 4500

# Where is the root of the url name space?  DocumentRoot
# implies that http:/ is in this directory, except where
# some <Location> overrides it.
DocumentRoot /home/xliu/bmg_devel/bgmodel/web/static_html

<Directory "/home/xliu/bmg_devel/bgmodel/web/static_html">
Options Indexes FollowSymLinks +Includes
AllowOverride None
Order allow,deny
Allow from all
AddType text/html .shtml
AddHandler server-parsed .shtml
</Directory>

# 
DirectoryIndex index.shtml index.html

AccessFileName .htaccess

# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>


# Do not allow a single request to be larger than about 20 MB.  This is
# principally here to limit file uploads.
limitRequestBody 20000000

# where are the server config/log files
#ServerRoot /etc/apache2
ServerRoot /home/xliu/bmg_devel/bgmodel/web/apache

PidFile "/home/xliu/bmg_devel/bgmodel/web/apache/run/httpd.pid"

ErrorLog "/home/xliu/bmg_devel/bgmodel/web/apache/log/error_log"

## wsgi to django - 

WSGIScriptAlias /bmg /home/xliu/bmg_devel/bgmodel/web/webservice/wsgi/django.wsgi

<Directory /home/xliu/bmg_devel/bgmodel/web/webservice/wsgi>
Order allow,deny
Allow from all
</Directory>


Graham Dumpleton

unread,
Jan 8, 2015, 3:13:34 PM1/8/15
to mod...@googlegroups.com
Define 'it doesn't work'.

You do not actually describe what the problem is.

What is happening when you try and access the Python web application?

What errors do you get in the browser, or does it just show a blank page?

What errors do you see in the Apache error log around the time of the web request?

What error messages do you get in the Apache error log when you restart Apache that may indicate a mismatch with Python versions?

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

Cate Liu

unread,
Jan 8, 2015, 3:23:42 PM1/8/15
to mod...@googlegroups.com
Yes, you are correct!  Let me be more specific (forgot to mention it in my first post, but remember right away when hit the post key :).

I get seg fault when starting the apache (shown in the error log):  child pid 4816 exit signal Segmentation fault (11)

Graham Dumpleton

unread,
Jan 8, 2015, 3:42:51 PM1/8/15
to mod...@googlegroups.com
Have you read:


and specifically:


Try the directive:

    WSGIApplicationGroup %{GLOBAL}

as explained in the latter.

Graham

Cate Liu

unread,
Jan 8, 2015, 4:05:11 PM1/8/15
to mod...@googlegroups.com
I do not have mod_python installed.

I am not even trying to run my django app.  The apahe failed at the first line of  the conf file:
LoadModule wsgi_module /usr/lib64/httpd/modules/mod_wsgi.so

If I comment this line out, I can get to the static html page.

Graham Dumpleton

unread,
Jan 8, 2015, 4:32:18 PM1/8/15
to mod...@googlegroups.com
To clarify, when you say Apache is showing a segmentation fault on startup, does that mean Apache doesn't start at all, or are you seeing many of these segmentation fault messages with Apache looping on them with different process IDs in each case.

Please provide the complete Apache error log contents from when you attempt to start Apache until either you start seeing this looping segmentation fault messages or until there is no more log messages as it has stopped showing anything.

Also, if you have been compiling mod_wsgi from source code, did you ensure that you used fresh source code or ran 'make distclean' between trying to do configure/make for different Python versions when installing it.

Can you also better explain the steps you are taking to build and install mod_wsgi, and show me the output from the 'ldd' command so I can see what version of Python is being found and where from. Ensure that you do not have LD_LIBRARY_PATH set in your user environment when running ldd.

Graham

Cate Liu

unread,
Jan 8, 2015, 4:52:24 PM1/8/15
to mod...@googlegroups.com
Graham, thanks for sticking with me!!  I appreciate it.

How are some more info you asked.

I just re-build the wsgi4.4.5.  
Steps:  
make disclean
which python 
         /home/xliu/Ureka1_4_1/Ureka/variants/common/bin/python
python -V
         Python 2.7.5
./configure
$echo $LD_RUN_PATH 
         /home/xliu/Ureka1_4_1/Ureka/python/lib/
make
echo $LD_LIBRARY_PATH
        /usr/lib64/alliance/lib:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man:/usr/lib64/alliance/man:/usr/lib64.  Note, no Ureka python lib is included.
sudo make install
ldd /usr/lib64/httpd/modules/mod_wsgi.so
linux-vdso.so.1 =>  (0x00007fffb07ff000)
libpython2.7.so.1.0 => /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0 (0x00007f3bc420e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3bc3f8d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f3bc3d89000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f3bc3b86000)
libm.so.6 => /lib64/libm.so.6 (0x00007f3bc3901000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3bc356e000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007f3bc3231000)
libXft.so.2 => /usr/lib64/libXft.so.2 (0x00007f3bc301b000)
/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007f3bc2dfd000)
libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007f3bc2bc6000)
libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007f3bc2929000)
libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007f3bc271f000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007f3bc251c000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f3bc22f3000)
ls /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0
        /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0

Then I start the apache with my conf:
   /usr/sbin/httpd -f /home/xliu/bmg_devel/bgmodel/web/apache/conf/httpd.conf

Here is the error log (start looping):
[Thu Jan 08 13:48:20 2015] [notice] Digest: generating secret for digest authentication ...
[Thu Jan 08 13:48:20 2015] [notice] Digest: done
[Thu Jan 08 13:48:20 2015] [notice] Apache/2.2.15 (Unix) mod_wsgi/4.4.5 Python/2.7.5 DAV/2 c
onfigured -- resuming normal operations
[Thu Jan 08 13:48:21 2015] [notice] child pid 3149 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:21 2015] [notice] child pid 3151 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:21 2015] [notice] child pid 3153 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:23 2015] [notice] child pid 3168 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:24 2015] [notice] child pid 3171 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:24 2015] [notice] child pid 3173 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:25 2015] [notice] child pid 3175 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:25 2015] [notice] child pid 3177 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:25 2015] [notice] child pid 3178 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 13:48:25 2015] [notice] child pid 3179 exit signal Segmentation fault (11), poss
ible coredump in /home/xliu/bmg_devel/bgmodel/web/apache

The web page (localhost:4500) show:  This webpage is not available.

Graham Dumpleton

unread,
Jan 8, 2015, 5:00:24 PM1/8/15
to mod...@googlegroups.com
Because your Python installation is in a non standard location, you need to tell mod_wsgi where the runtime is. This is independent of how it finds the shared library.

So add:

    WSGIPythonHome /home/xliu/Ureka1_4_1/Ureka/variants/common

BTW, if just playing around or doing development, consider using mod_wsgi-express instead. You don't have to worry about these sorts of issues with it as it does it all properly for you.


Graham

Cate Liu

unread,
Jan 8, 2015, 5:09:09 PM1/8/15
to mod...@googlegroups.com
I've tried this option before, and just add it to the top of my conf file.  Nope, it still seg faults!!

Yes, I am considering the mod_wsgi-express.  But thought the traditional way should still work.  Just don't want to give up :-)  But I'm about to pull my hair......

Cate Liu

unread,
Jan 8, 2015, 5:12:23 PM1/8/15
to mod...@googlegroups.com
Why is that if I build the wsgi using /usr/bin/python (2.6.6), then it would work????

Graham Dumpleton

unread,
Jan 8, 2015, 5:44:01 PM1/8/15
to mod...@googlegroups.com
Add the WSGIPythonHome directive back and also ensure the LogLevel directive in Apache is set to 'debug' rather than 'warn' or some other value.

Capture the Apache error log when starting with that.

Graham

Cate Liu

unread,
Jan 8, 2015, 7:00:29 PM1/8/15
to mod...@googlegroups.com
Not much more info, but here it is the error log:

[Thu Jan 08 15:58:59 2015] [notice] Digest: generating secret for digest authentication ...
[Thu Jan 08 15:58:59 2015] [notice] Digest: done
[Thu Jan 08 15:58:59 2015] [notice] Apache/2.2.15 (Unix) mod_wsgi/4.4.5 Python/2.7.5 DAV/2 c
onfigured -- resuming normal operations
[Thu Jan 08 15:58:59 2015] [info] Server built: Aug  2 2013 08:02:15
[Thu Jan 08 15:58:59 2015] [debug] prefork.c(1018): AcceptMutex: sysvsem (default: sysvsem)
[Thu Jan 08 15:59:00 2015] [notice] child pid 23378 exit signal Segmentation fault (11), pos
sible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 15:59:00 2015] [notice] child pid 23380 exit signal Segmentation fault (11), pos
sible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 15:59:00 2015] [notice] child pid 23382 exit signal Segmentation fault (11), pos
sible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 15:59:02 2015] [notice] child pid 23394 exit signal Segmentation fault (11), pos
sible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 15:59:03 2015] [notice] child pid 23397 exit signal Segmentation fault (11), pos
sible coredump in /home/xliu/bmg_devel/bgmodel/web/apache
[Thu Jan 08 15:59:03 2015] [notice] child pid 23399 exit signal Segmentation fault (11), pos
sible coredump in /home/xliu/bmg_devel/bgmodel/web/apache

Cate Liu

unread,
Jan 8, 2015, 7:27:58 PM1/8/15
to mod...@googlegroups.com
Does it matter what version of Apache is?  Is there any conflict between apache 2.2.15 and wsgi 4.4.5. or with python 2.7.5?

Graham Dumpleton

unread,
Jan 8, 2015, 9:08:11 PM1/8/15
to mod...@googlegroups.com

On 09/01/2015, at 11:27 AM, Cate Liu <remot...@gmail.com> wrote:

ldd /usr/lib64/httpd/modules/mod_wsgi.so
linux-vdso.so.1 =>  (0x00007fffb07ff000)
libpython2.7.so.1.0 => /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0 (0x00007f3bc420e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3bc3f8d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f3bc3d89000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f3bc3b86000)
libm.so.6 => /lib64/libm.so.6 (0x00007f3bc3901000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3bc356e000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007f3bc3231000)
libXft.so.2 => /usr/lib64/libXft.so.2 (0x00007f3bc301b000)
/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007f3bc2dfd000)
libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007f3bc2bc6000)
libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007f3bc2929000)
libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007f3bc271f000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007f3bc251c000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f3bc22f3000)

Can you give me the complete output from running the 'make distclean; configure; make' for when you build mod_wsgi from source code?

It makes absolutely no sense why mod_wsgi has dependencies on these libraries:

libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007f3bc3231000)
libXft.so.2 => /usr/lib64/libXft.so.2 (0x00007f3bc301b000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007f3bc2dfd000)
libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007f3bc2bc6000)
libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007f3bc2929000)
libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007f3bc271f000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007f3bc251c000)

Also provide the output from running:

    env | grep FLAGS

It would look like something strange is going on at compile time to have those be listed as dependencies of mod_wsgi.so.

The only way I can see that they would be linked is if apxs build configuration had been fiddled with, or you had environment variables listing compiler flags with names that configure is automatically picking up and using.

Graham

Cate Liu

unread,
Jan 9, 2015, 1:21:45 PM1/9/15
to mod...@googlegroups.com
Morning.  Here is the screen dump attached.
screencapture.txt

Graham Dumpleton

unread,
Jan 9, 2015, 9:49:14 PM1/9/15
to mod...@googlegroups.com
That really isn't making much sense.

The problem is that you are seeing various X Windows and FreeType font libraries being linked into mod_wsgi.so and that really shouldn't be the case.

Three more things to do and then I will have to see if you can capture a stack trace for the crash if that doesn't help.

What do you get when you run:

    ldd .libs/mod_wsgi.so

in the mod_wsgi source code after building it? This is running ldd on the mod_wsgi.so before it gets installed. When it gets installed the apxs script can do some funny relinking and maybe it is libtool, called by apxs on installation that is adding them.

What do you get when you run:

    ldd /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0

This is just to make sure the Python library itself doesn't have any strange libraries linked into it.

Finally, for good measure send the 'Makefile' from the mod_wsgi source code which is generated when you run 'configure'.

Thanks.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
<screencapture.txt>

Cate Liu

unread,
Jan 10, 2015, 8:19:35 PM1/10/15
to mod...@googlegroups.com
Hi Graham,

Here they are:

nhsctest5:/home/xliu/mod_wsgi-4.4.5/src/server/.libs$echo $LD_LIBRARY_PATH
/usr/lib64/alliance/lib:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man:/usr/lib64/alliance/man:/usr/lib64
nhsctest5:/home/xliu/mod_wsgi-4.4.5/src/server/.libs$cd ..
nhsctest5:/home/xliu/mod_wsgi-4.4.5/src/server$ldd .libs/mod_wsgi.so
linux-vdso.so.1 =>  (0x00007fffc3d6c000)
libpython2.7.so.1.0 => /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0 (0x00007f1266c6a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f12669e9000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f12667e5000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f12665e2000)
libm.so.6 => /lib64/libm.so.6 (0x00007f126635d000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1265fca000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007f1265c8d000)
libXft.so.2 => /usr/lib64/libXft.so.2 (0x00007f1265a77000)
/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007f1265859000)
libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007f1265622000)
libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007f1265385000)
libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007f126517b000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007f1264f78000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f1264d4f000)

It looks the same as what is installed.

ldd /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0
linux-vdso.so.1 =>  (0x00007fff557a6000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007fb24d5a3000)
libXft.so.2 => /usr/lib64/libXft.so.2 (0x00007fb24d38d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb24d10d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fb24cf09000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fb24cd05000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb24ca81000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb24c6ee000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007fb24c4cf000)
libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007fb24c299000)
libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007fb24bffc000)
libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007fb24bdf1000)
/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007fb24bbee000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fb24b9c6000)


Makefile
#  Copyright 2007-2011 GRAHAM DUMPLETON
# 
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
# 
# 
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.

APXS = /usr/sbin/apxs
PYTHON = /home/xliu/Ureka1_4_1/Ureka/variants/common/bin/python

DESTDIR = 
LIBEXECDIR = /usr/lib64/httpd/modules

CPPFLAGS =  -I/home/xliu/Ureka1_4_1/Ureka/python/include/python2.7 -DNDEBUG 
CFLAGS =  -Wc,-g -Wc,-O2 
LDFLAGS =  -L/home/xliu/Ureka1_4_1/Ureka/python/lib -L/home/xliu/Ureka1_4_1/Ureka/python/lib
/python2.7/config 
LDLIBS =  -lpython2.7 -lpthread -ldl  -lutil -lm

SRCFILES = src/server/mod_wsgi.c src/server/wsgi_*.c

all : src/server/mod_wsgi.la

src/server/mod_wsgi.la : $(SRCFILES)
$(APXS) -c $(CPPFLAGS) $(CFLAGS) $(SRCFILES) $(LDFLAGS) $(LDLIBS)

$(DESTDIR)$(LIBEXECDIR) :
mkdir -p $@

install : all $(DESTDIR)$(LIBEXECDIR)
$(APXS) -i -S LIBEXECDIR=$(DESTDIR)$(LIBEXECDIR) -n 'mod_wsgi' src/server/mod_wsgi.l
a

clean :
-rm -rf src/server/.libs
-rm -f src/server/*.o
-rm -f src/server/*.la
-rm -f src/server/*.lo
-rm -f src/server/*.slo
-rm -f src/server/*.loT
-rm -f config.log
-rm -f config.status
-rm -rf autom4te.cache
-rm -rf mod_wsgi.egg-info
-rm -rf build
-rm -rf dist

distclean : clean
-rm -f Makefile
-rm -f apxs libtool
-rm -rf .Python bin lib include
-rm -rf .tox

realclean : distclean
-rm -f configure
===================================================================

Just so you know, I installed a personal copy of Apache 2.2.29, but it has the same seg fault problem.
So the problem got to be in mod_wsgi.so, no?

Thanks!!

Cate

Graham Dumpleton

unread,
Jan 10, 2015, 8:33:30 PM1/10/15
to mod...@googlegroups.com
It doesn't need be a problem with mod_wsgi. That all those extra shared libraries are being pulled in looks to a be a big problem, especially the libexpat library.

Since they are still there before the mod_wsgi is even installed, do this instead.

Presuming you have pip installed in /home/xliu/Ureka1_4_1/Ureka/python/bin, run:

    APXS=/usr/sbin/apxs /home/xliu/Ureka1_4_1/Ureka/python/bin/pip install mod_wsgi

Then run:

    /home/xliu/Ureka1_4_1/Ureka/python/bin/mod_wsgi-express module-location

This will output a path like:

    $ mod_wsgi-express module-location
    /Users/graham/Projects/mod_wsgi/venv/lib/python2.7/site-packages/mod_wsgi-4.4.6-py2.7-macosx-10.8-intel.egg/mod_wsgi/server/mod_wsgi-py27.so

Then run:

    ldd `/home/xliu/Ureka1_4_1/Ureka/python/bin/mod_wsgi-express module-location`

This will run ldd on the mod_wsgi.so generated by the mod_wsgi-express install. Show me what you get.

This build method by passes apxs/libtool and if libtool is somehow screwed up in its configuration so as to cause those libraries to be linked when they shouldn't, this will avoid that problem.

If the ldd doesn't show any dependencies on X Windows and FreeType libraries run:

    sudo /home/xliu/Ureka1_4_1/Ureka/python/bin/mod_wsgi-express install-module

This will copy this mod_wsgi.so into the Apache installation and output something like:

    $ sudo mod_wsgi-express install-module
    LoadModule wsgi_module /usr/local/httpd-2.4.x/modules/mod_wsgi-py27.so
    WSGIPythonHome /Users/graham/Projects/mod_wsgi/venv

Edit your Apache configuration and replace your existing LoadModule like for mod_wsgi with that one and any existing WSGIPythonHome directive, if different with that one.

If that all works, there is likely something broken with the 'libtool' installation on your system.

Can you also send the output of running:

    ldd /usr/sbin/httpd

If that isn't the path to your httpd executable then substitute as necessary.

One concern is that httpd links a different version of libexpat than mod_wsgi was for some reason linking. If it does that will cause lots of problems.

Graham

Cate Liu

unread,
Jan 10, 2015, 8:53:23 PM1/10/15
to mod...@googlegroups.com
Arh.  Syntax error?

nhsctest5:/home/xliu$APXS=/usr/sbin/apxs /home/xliu/Ureka1_4_1/Ureka/python/bin/pip install mod_wsgi
APXS=/usr/sbin/apxs: Command not found.

Graham Dumpleton

unread,
Jan 10, 2015, 8:58:26 PM1/10/15
to mod...@googlegroups.com
You would appear not to be using a Bourne style shell like bash but must be using csh or zsh.

Just run:

    /home/xliu/Ureka1_4_1/Ureka/python/bin/pip install mod_wsgi

It still should find it in the standard location but wanted to avoid possibility it picked up your self compiled Apache.

If it does pick up wrong Apache then try:

    setenv APXS /usr/sbin/apxs
    /home/xliu/Ureka1_4_1/Ureka/python/bin/pip install mod_wsgi
    unsetenv APXS

Cate Liu

unread,
Jan 10, 2015, 9:05:34 PM1/10/15
to mod...@googlegroups.com
ldd `Ureka1_4_1/Ureka/variants/common/bin/mod_wsgi-express module-location`
linux-vdso.so.1 =>  (0x00007fff385ff000)
libpython2.7.so.1.0 => /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0 (0x00007fb2fa459000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb2fa1d8000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb2f9e45000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007fb2f9b08000)
libXft.so.2 => /usr/lib64/libXft.so.2 (0x00007fb2f98f2000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fb2f96ee000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fb2f94eb000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb2f9266000)
/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007fb2f9048000)
libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007fb2f8e12000)
libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007fb2f8b74000)
libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007fb2f896a000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007fb2f8767000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fb2f853e000)

Cate Liu

unread,
Jan 10, 2015, 9:07:33 PM1/10/15
to mod...@googlegroups.com
ldd /usr/sbin/httpd
linux-vdso.so.1 =>  (0x00007fff1a1fe000)
libm.so.6 => /lib64/libm.so.6 (0x00007f7da14cc000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00007f7da129f000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f7da1080000)
libaprutil-1.so.0 => /usr/lib64/libaprutil-1.so.0 (0x00007f7da0e5c000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f7da0c24000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f7da09fc000)
libdb-4.7.so => /usr/lib64/libdb-4.7.so (0x00007f7da0688000)
libapr-1.so.0 => /usr/lib64/libapr-1.so.0 (0x00007f7da045b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7da023e000)
libc.so.6 => /lib64/libc.so.6 (0x00007f7d9feab000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f7d9fca6000)
/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f7d9faa2000)
libfreebl3.so => /usr/lib64/libfreebl3.so (0x00007f7d9f836000)

Graham Dumpleton

unread,
Jan 10, 2015, 9:17:48 PM1/10/15
to mod...@googlegroups.com
Whoops. I missed that you already said:

ldd /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0
linux-vdso.so.1 =>  (0x00007fff557a6000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007fb24d5a3000)
libXft.so.2 => /usr/lib64/libXft.so.2 (0x00007fb24d38d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb24d10d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fb24cf09000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fb24cd05000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb24ca81000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb24c6ee000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007fb24c4cf000)
libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007fb24c299000)
libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007fb24bffc000)
libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007fb24bdf1000)
/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007fb24bbee000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fb24b9c6000)

So those extra libraries are coming from the Python library.

Did you install this Python version yourself from source code, or is it some package binary distribution?

Do you know why the X Windows and FreeType libraries have been linked into Python?

Graham

Cate Liu

unread,
Jan 10, 2015, 9:21:16 PM1/10/15
to mod...@googlegroups.com
it's a binary distribution

Cate Liu

unread,
Jan 10, 2015, 9:26:30 PM1/10/15
to mod...@googlegroups.com
Not sure why, but it's a python package mainly for astronomy including image display, etc. It's web site says, there are currently no provisions for building Ureka from source.

Cate Liu

unread,
Jan 10, 2015, 9:31:28 PM1/10/15
to mod...@googlegroups.com
Am I doomed?  I'm delivering a web service to the folks who use Ureka as their production environment.  Arrrh!!

Graham Dumpleton

unread,
Jan 10, 2015, 9:59:20 PM1/10/15
to mod...@googlegroups.com
There is still a way we can confirm whether it is caused by what that Python distribution is doing. There still could be other reasons why it is failing, but again due to things the Python distribution has done.

What they have done with forcing the linking of those libraries into Python itself is the wrong way of going about things.

If they are distributing Python C extension modules which require those libraries, they should be linking the libraries when building those specific modules and not to Python as a whole. By linking to Python as a whole, there is a good chance they will break anything that wants to use that Python library in an embedded system.

To try and confirm the exact reason for crashing try doing the following.

1. Run:

    sudo gdb /usr/sbin/httpd

2. Then at the gdb prompt type:

    run -D FOREGOUND -D ONE_PROCESS

3. If that results in a crash and you get back the prompt, then enter at gdb prompt:

    where

If it doesn't crash, if you were using WSGIDaemonProcess directive in Apache configuration, try again but with that directive commented out.

Provide the output from the gdb session.

For background reading on what is being done here read:


To provide a bit more information about how they did configure that Python installation, you can also run this Python script against 'python' for that distribution.

    from distutils.sysconfig import get_config_var
    print 'CONFIG_ARGS:', get_config_var('CONFIG_ARGS')
    print 'LIBS:', get_config_var('LIBS')
    print 'SYSLIBS:', get_config_var('SYSLIBS')
    print 'SHLIBS:', get_config_var('SHLIBS')

This should yield something like:

CONFIG_ARGS: '--prefix=/usr' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--disable-dependency-tracking' '--enable-ipv6' '--with-system-expat' '--with-threads' '--enable-framework=/System/Library/Frameworks' '--enable-toolbox-glue' '--with-system-ffi' '--with-gcc=clang' 'CC=clang' 'CXX=clang++' 'CFLAGS=-g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32' 'LDFLAGS=-Wl,-F.'
LIBS: -ldl  -framework CoreFoundation
SYSLIBS:
SHLIBS: -ldl  -framework CoreFoundation

This still may not show how the extra libraries got linked as they could well have hand modified the generated makefile used when Python was built.

Graham

Cate Liu

unread,
Jan 10, 2015, 11:33:20 PM1/10/15
to mod...@googlegroups.com
assume you'd like me to run "sudo gdb /usr/sbin/httpd -f myconfigfile"???
So I tried:    sudo gdb --args /usr/sbin/httpd -f /home/xliu/bmg_devel/bgmodel/web/apache/conf/httpd.conf


GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.

Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-29.el6_4.x86_64

(gdb) run -D FOREGOUND -D ONE_PROCESS

Starting program: /usr/sbin/httpd -D FOREGOUND -D ONE_PROCESS

[Thread debugging using libthread_db enabled]

httpd: Syntax error on line 152 of /etc/httpd/conf/httpd.conf: Cannot load /usr/lib64/httpd/modules/mod_python.so into server: /usr/lib64/httpd/modules/mod_python.so: cannot open shared object file: No such file or directory

Program exited with code 01.

(gdb) where

No stack.

Why didn't it read my own config file??



Cate Liu

unread,
Jan 10, 2015, 11:38:59 PM1/10/15
to mod...@googlegroups.com

nhsctest5:/home/xliu/bmg_devel/bgmodel/web$python

Python 2.7.5 (default, Jun 19 2014, 11:22:38) 

[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> from distutils.sysconfig import get_config_var

>>> print 'CONFIG_ARGS:', get_config_var('CONFIG_ARGS')

CONFIG_ARGS: '-prefix=/home/xliu/Ureka1_4_1/Ureka/python' 'CFLAGS=-I/home/xliu/Ureka1_4_1/Ureka/python/include -I/home/xliu/Ureka1_4_1/Ureka/python/include  ' 'OPT=-I/home/xliu/Ureka1_4_1/Ureka/python/include -I/home/xliu/Ureka1_4_1/Ureka/python/include  ' 'LDFLAGS= -L/home/xliu/Ureka1_4_1/Ureka/python/lib  ' '--enable-shared' 'CC=cc -I/home/xliu/Ureka1_4_1/Ureka/python/include -L/home/xliu/Ureka1_4_1/Ureka/python/lib'

>>> print 'LIBS:', get_config_var('LIBS')

LIBS: -lpthread -ldl  -lutil

>>> print 'SYSLIBS:', get_config_var('SYSLIBS')

SYSLIBS: -lm

>>> print 'SHLIBS:', get_config_var('SHLIBS')

SHLIBS: -lpthread -ldl  -lutil

Graham Dumpleton

unread,
Jan 10, 2015, 11:40:04 PM1/10/15
to mod...@googlegroups.com
You should not have to pass it the config file, if you are editing it in the normal place Apache has the config file it would know where to find it automatically.

If you have modified the Apache startup script to look in a different location, inside of gdb type:

    run -D FOREGOUND -D ONE_PROCESS -f /home/xliu/bmg_devel/bgmodel/web/apache/conf/httpd.conf

Do not supply any args to gdb itself.

Please tell me you aren't loading mod_python as well as mod_wsgi. Because mod_python wouldn't be using the same Python it would crash and because mod_wsgi will now block mod_python being used at the same time you can't use them together anyway.

Graham

Cate Liu

unread,
Jan 10, 2015, 11:57:15 PM1/10/15
to mod...@googlegroups.com
It looks like there is problem for gdb to read my own config file.  So I copied it to the system default location /etc/httpd/httpd.conf.

sudo gdb /usr/sbin/httpd

[sudo] password for xliu: 

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.

Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-29.el6_4.x86_64

(gdb) run -D FOREGOUND -D ONE_PROCESS

Starting program: /usr/sbin/httpd -D FOREGOUND -D ONE_PROCESS

[Thread debugging using libthread_db enabled]

httpd: Syntax error on line 11 of /etc/httpd/conf/httpd.conf: Cannot load /usr/lib64/httpd/modules/mod_wsgi.so into server: libpython2.7.so.1.0: cannot open shared object file: Permission denied


Program exited with code 01.

(gdb) where

No stack.


Ha!  Is it because the python can't open my mod_wsgi.so?  I do have permission for that file though.

No worry about mod_python (it's loaded from the previous system config file).  But there may be problem at the production end.  What is they also use mod_python?

...

Graham Dumpleton

unread,
Jan 11, 2015, 12:18:01 AM1/11/15
to mod...@googlegroups.com
Are you run SELinux?

This can occur where SELinux is used as the httpd process will not have privileges to link a shared library outside of standard locations.

You can try temporarily disable SELinux by following procedure in:


Graham

Cate Liu

unread,
Jan 11, 2015, 12:25:34 AM1/11/15
to mod...@googlegroups.com
this seems a very dangerous action.  plus I don't have root privilege.
...

Cate Liu

unread,
Jan 11, 2015, 12:28:19 AM1/11/15
to mod...@googlegroups.com
I don't know if I have SELinux.  How to find out?  All our machines are behind firewall.
...

Graham Dumpleton

unread,
Jan 11, 2015, 12:35:48 AM1/11/15
to mod...@googlegroups.com
You know that when you run sudo it is giving you the ability to run commands as root? :-)

Based on that document I linked, if you are worried about disabling, then start out by looking to see if the file:

    /etc/selinux/config

exists and what 'SELINUX' is set to.

Do be aware my knowledge of SELinux is limited at I generally just point people at that page and they work it out.

Anyway, to create a root shell, given you have sudo access, you would run:

    sudo -s

Graham

Cate Liu

unread,
Jan 11, 2015, 12:51:53 AM1/11/15
to mod...@googlegroups.com
found the /etc/selinux/config.  it has following two lines:

SELINUX=disabled

SELINUXTYPE=targeted


Does it mean it's disabled?

...

Cate Liu

unread,
Jan 11, 2015, 12:54:29 AM1/11/15
to mod...@googlegroups.com
I don't think it's enforced, as I don't find /selinux/enforce file as stated in that link.
...

Cate Liu

unread,
Jan 11, 2015, 1:00:27 AM1/11/15
to mod...@googlegroups.com
Graham,

Please tell me about mod_python and mod_wsgi.  Are you saying they can't co-exist?
...

Graham Dumpleton

unread,
Jan 11, 2015, 1:41:26 AM1/11/15
to mod...@googlegroups.com
The short answer is if you are not using mod_python, don't load it.

If you are using mod_python, you are going to have a number of problems with what you are trying to do.

First up. Both mod_python and mod_wsgi will link to the Python library separately.

When Apache loads both mod_python and mod_wsgi the OS dynamic library loader will only allow one instance of the Python library to load.

This means they must be the same version of the Python library on disk and both mod_python and mod_wsgi must be compiled against the same Python installation.

In your case where you are recompiling mod_wsgi to use a different Python installation, it now wants a different Python library.

So which ever of mod_python or mod_wsgi is loaded first will win as far as what Python library is loaded. Because they which Python libraries they want were different and incompatible, as soon as the module that lost tried to call into Python code it will likely crash.

So that is the first problem.

The second problem is that the coexistence of mod_python and mod_wsgi in the same Apache is only supported for mod_wsgi < 4.0.

Even if both were compiled against the same Python installation, when mod_wsgi 4.X initialises, if it sees that mod_python is also loaded it will refuse to let Apache start and give an error that mod_python cannot be loaded concurrently.

This exclusion now exists because mod_python usage of the Python C APIs is somewhat broken. I maintained various hacks to allow mod_wsgi to peacefully coexist with mod_python, but the hacks becomes a pain, especially when Python 3.2+ cracked down on how the C APIs were meant to be used. The way mod_python did things, and the hacks I had to use to be compatible with mod_python usage became technical violations of the C API and would be flag as run time violations and from memory cause Python to exit the interpreter with assertion failures.

So it just became to much burden to try and maintain things so both could be used, especially since mod_python wouldn't even work properly on most recent Apache 2.2 versions and Apache 2.4 without having to change the mod_python code, something that Linux package maintainers were doing but which I wasn't prepared to do on MacOS X where I develop things.

Even when mod_python and mod_wsgi could run together, because mod_python override things and took control of initialising the interpreter, various features of mod_wsgi to control the interpreter environment wouldn't work. So a loss of functionality as well.

With the fact that mod_python is hardly used any more, there simply wasn't a point in trying to accommodate it.

Is the fact you are asking mean you are trying to load mod_python at the same time?

If you are, lets do a test.

Try running:

    mod_wsgi-express start-server

Does that start up and can you access:


If that works, then loading of mod_python in your main Apache could well be the cause of the problems.

Graham

You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

Cate Liu

unread,
Jan 11, 2015, 2:13:41 AM1/11/15
to mod...@googlegroups.com
Graham,

I know I have taken much of your time and thank you for the detail explanation and your help!!  No, the reason I'm asking about mod_python and mod_wsgi 
is because I fear my counter-partner (to whom I'm delivering code) might be using mod_python.  I know at least they used to use mod_python for the reason 
back then mod_python was a 'better' option.  They may have to switch if you say that mod_python won't work properly on newer Apache.

BTW, I've tried to set selinux to permissive, it says SELinux is disabled.  So then why the python couldn't load the mod_wsgi.so in gdb?  Are there other ways
moving forward in the debugging?

Thanks.

Cate
...

Graham Dumpleton

unread,
Jan 11, 2015, 2:58:56 AM1/11/15
to mod...@googlegroups.com
I am not sure yet why you would get permission denied for libpython2.7.so.

Can you try:

     sudo ldd /usr/lib64/httpd/modules/mod_wsgi.so

and see what you get.

There is only one reason I can think of why root wouldn't be able to access the library.

That is that if the path:

    /home/xliu/Ureka1_4_1/Ureka/python

is not actually located on your local disk, but is coming from a NFS mounted file system which has root squash attributes and the directories down to that location are not searchable or files not readable to other users.

Running ldd under sudo on the mod_wsgi.so should tell us if it might be such a problem as it wouldn't be able to resolve libpython.so if that is the case.

Also still run that:

    mod_wsgi-express start-server 

test I mentioned. This is still useful to just see whether that separately compiled mod_wsgi.so will work with an Apache configuration that was generated by mod_wsgi-express, which I know shouldn't itself have any issues.

Also, running mod_wsgi-express will do so as you and not root, so if is an issue caused by anything like SELinux or NFS, it wouldn't be affected and should still run.

Graham

Cate Liu

unread,
Jan 11, 2015, 3:08:04 AM1/11/15
to mod...@googlegroups.com
HA!  think you are onto something here:

sudo ldd /usr/lib64/httpd/modules/mod_wsgi.so

[sudo] password for xliu: 

linux-vdso.so.1 =>  (0x00007fffe25ff000)

libpython2.7.so.1.0 => not found

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1dcfe44000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007f1dcfc40000)

libutil.so.1 => /lib64/libutil.so.1 (0x00007f1dcfa3d000)

libm.so.6 => /lib64/libm.so.6 (0x00007f1dcf7b8000)

libc.so.6 => /lib64/libc.so.6 (0x00007f1dcf425000)

/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)


Yes, /home/xliu is NFS mounted.

...

Graham Dumpleton

unread,
Jan 11, 2015, 3:14:41 AM1/11/15
to mod...@googlegroups.com
Then it likely will be due to /home/xliu not having read/execute permissions to others. That is quite typical for Linux systems. With that and it being NFS mounted, root wouldn't be able to access inside of root squash for that NFS partition is on.

As I said though, trying 'mod_wsgi-express start-server' should give you an idea straight away if it will even work as it will not run as root and will not be affected by that.

Graham

Cate Liu

unread,
Jan 11, 2015, 3:17:02 AM1/11/15
to mod...@googlegroups.com
OK, started the server:

mod_wsgi-express start-server

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:5099

Server Conf        : /tmp/mod_wsgi-localhost:8000:5099/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:5099/error_log

Locale Setting     : en_US.ISO8859-1

Daemon Processes   : 1

Daemon Threads     : 5


But the web page is not available.  This may not be your problem.  I'm running on a machine behind a firewall remotely from home.  Should I see the page anyway?

...

Cate Liu

unread,
Jan 11, 2015, 3:23:30 AM1/11/15
to mod...@googlegroups.com
But I don't understand.  It says it can't load /usr/lib64/httpd/modules/mod_wsgi.so, which IS on local disk.  Yes, the python lib is on NFS mounted disk.
I'm confused about who is accessing whom now.
...

Cate Liu

unread,
Jan 11, 2015, 3:25:32 AM1/11/15
to mod...@googlegroups.com
Nope, the server died.

m /tmp/mod_wsgi-localhost:8000:5099/error_log

[Sun Jan 11 00:11:29 2015] [info] mod_wsgi (pid=9363): Starting process 'localhost:800

0' with threads=5.

[Sun Jan 11 00:11:29 2015] [notice] Apache/2.2.15 (Unix) mod_wsgi/4.4.5 Python/2.7.5 c

onfigured -- resuming normal operations

[Sun Jan 11 00:11:29 2015] [info] Server built: Aug  2 2013 08:02:15

[Sun Jan 11 00:11:30 2015] [notice] child pid 9363 exit signal Segmentation fault (11)

, possible coredump in /tmp/mod_wsgi-localhost:8000:5099

[Sun Jan 11 00:11:30 2015] [info] mod_wsgi (pid=9363): Process 'localhost:8000' has di

ed, deregister and restart it.

[Sun Jan 11 00:11:30 2015] [info] mod_wsgi (pid=9363): Process 'localhost:8000' termin

ated by signal 11

[Sun Jan 11 00:11:30 2015] [info] mod_wsgi (pid=9363): Process 'localhost:8000' has be

en deregistered and will no longer be monitored.

[Sun Jan 11 00:11:30 2015] [info] mod_wsgi (pid=9377): Starting process 'localhost:800

0' with threads=5.

[Sun Jan 11 00:11:31 2015] [notice] child pid 9377 exit signal Segmentation fault (11)

, possible coredump in /tmp/mod_wsgi-localhost:8000:5099

[Sun Jan 11 00:11:31 2015] [info] mod_wsgi (pid=9377): Process 'localhost:8000' has di

ed, deregister and restart it.

[Sun Jan 11 00:11:31 2015] [info] mod_wsgi (pid=9377): Process 'localhost:8000' termin

ated by signal 11

...

Graham Dumpleton

unread,
Jan 11, 2015, 3:38:10 AM1/11/15
to mod...@googlegroups.com
By default when have:

mod_wsgi-express start-server

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:5099

Server Conf        : /tmp/mod_wsgi-localhost:8000:5099/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:5099/error_log

Locale Setting     : en_US.ISO8859-1

Daemon Processes   : 1

Daemon Threads     : 5


it is only listen for socket connections coming from localhost.

If you need to access it from another machine, you would need to use:

    mod_wsgi start-server --host=public.name.of.host

or on some systems you can say:

    mod_wsgi start-server --host=0.0.0.0

Anyway, you at least have it crashing when running as yourself now. We can therefore run gdb again and not have the root squash issue.

First, run:

    mod_wsgi-express start-server --debug-mode --setup-only

Look in the file:

    /tmp/mod_wsgi-localhost:8000:5099/apachectl

At the top you run see a HTTPD variable, with a setting like:

    HTTPD="/usr/local/httpd-2.4.x/bin/httpd -f /tmp/mod_wsgi-localhost:8000:502/httpd.conf -DONE_PROCESS"

Yours may list other -D options.

Run gdb on the 'httpd' executable path which is first value in that.

    gdb /usr/local/httpd-2.4.x/bin/httpd

Then in gdb, use the rest of that value:

    run -f /tmp/mod_wsgi-localhost:8000:502/httpd.conf -DONE_PROCESS"

If crashes, enter:

    where

at gdb prompt.

Makes me think I should add an option to mod_wsgi-express specifically to run httpd under gdb to make that easier. :-)

Graham

Cate Liu

unread,
Jan 11, 2015, 2:30:14 PM1/11/15
to mod...@googlegroups.com
OK, back at it.   Morning!

mod_wsgi-express start-server --debug-mode --setup-only

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:5099

Server Conf        : /tmp/mod_wsgi-localhost:8000:5099/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:5099/error_log

Environ Variables  : /tmp/mod_wsgi-localhost:8000:5099/envvars

Control Script     : /tmp/mod_wsgi-localhost:8000:5099/apachectl

Locale Setting     : en_US.ISO8859-1

Daemon Processes   : 1

Daemon Threads     : 5

nhsctest5:/home/xliu$


nhsctest5:/home/xliu$gdb /usr/sbin/httpd

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.

Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-29.el6_4.x86_64

(gdb) run -f /tmp/mod_wsgi-localhost:8000:5099/httpd.conf -DONE_PROCESS

Starting program: /usr/sbin/httpd -f /tmp/mod_wsgi-localhost:8000:5099/httpd.conf -DONE_PROCESS

------------------

/home/xliu/Ureka1_4_1/Ureka/variants/common/bin /home/xliu/Ureka1_4_1/Ureka/bin /home/xliu/Ureka1_4_1/Ureka/python/bin /apps/java/jdk1.7.0_17/bin /home/xliu/pkg/eclipse /home/xliu/pkg/xxe-perso-3_8_0/bin /home/xliu/hcss_installer /home/xliu/pacssoft/hcss.dp/bin /home/xliu/pkg/openmpi-1.4.3/installation/bin /home/xliu/bin /usr/lib64/qt-3.3/bin /usr/kerberos/sbin /usr/kerberos/bin /usr/local/bin /bin /usr/bin /usr/lib64/alliance/bin /usr/sbin /apps/usr/local/bin /usr/lib64/alliance/bin

------------------

path=

/apps/java/jdk1.7.0_17/bin

/home/xliu/pkg/eclipse

/home/xliu/pkg/xxe-perso-3_8_0/bin

/home/xliu/hcss_installer

/home/xliu/pacssoft/hcss.dp/bin

/home/xliu/pkg/openmpi-1.4.3/installation/bin

/home/xliu/bin

/home/xliu/Ureka1_4_1/Ureka/variants/common/bin

/home/xliu/Ureka1_4_1/Ureka/bin

/home/xliu/Ureka1_4_1/Ureka/python/bin

/apps/java/jdk1.7.0_17/bin

/home/xliu/pkg/eclipse

/home/xliu/pkg/xxe-perso-3_8_0/bin

/home/xliu/hcss_installer

/home/xliu/pacssoft/hcss.dp/bin

/home/xliu/pkg/openmpi-1.4.3/installation/bin

/home/xliu/bin

/usr/lib64/qt-3.3/bin

/usr/kerberos/sbin

/usr/kerberos/bin

/usr/local/bin

/bin

/usr/bin

/usr/lib64/alliance/bin

/usr/sbin

/apps/usr/local/bin

/usr/lib64/alliance/bin

/usr/sbin

/apps/usr/local/bin

LD_LIBRARY_PATH=

[Thread debugging using libthread_db enabled]

httpd: bad user name ${WSGI_RUN_USER}


Program exited with code 01.

(gdb) where

No stack.

(gdb) 



The /tmp/mod_wsgi-localhost:8000:5099/apachectl looks like this:


#!/usr/bin/env bash


# ['/home/xliu/Ureka1_4_1/Ureka/variants/common/bin/mod_wsgi-express', 'start-server',

 '--debug-mode', '--setup-only']


HTTPD="/usr/sbin/httpd -f /tmp/mod_wsgi-localhost:8000:5099/httpd.conf -DONE_PROCESS"


SHLIBPATH=""


if [ "x$SHLIBPATH" != "x" ]; then

    LD_LIBRARY_PATH="$SHLIBPATH:$LD_LIBRARY_PATH"

    export LD_LIBRARY_PATH

fi


WSGI_RUN_USER="${WSGI_RUN_USER:-xliu}"

WSGI_RUN_GROUP="${WSGI_RUN_GROUP:-nhsc}"


export WSGI_RUN_USER

export WSGI_RUN_GROUP


if [ `id -u` = "0" -a ${WSGI_RUN_USER} = "root" ]; then

    cat << EOF


WARNING: When running as the 'root' user, it is required that the options

'--user' and '--group' be specified to mod_wsgi-express. These should

define a non 'root' user and group under which the Apache child worker

processes and mod_wsgi daemon processes should be run. Failure to specify

these options will result in Apache and/or the mod_wsgi daemon processes

failing to start. See the mod_wsgi-express documentation for further

information on this restriction.


EOF


fi


LANG='en_US.ISO8859-1'

LC_ALL='en_US.ISO8859-1'


export LANG

export LC_ALL


ACMD="$1"

ARGV="$@"


if test -f /tmp/mod_wsgi-localhost:8000:5099/envvars; then

    . /tmp/mod_wsgi-localhost:8000:5099/envvars

fi


STATUSURL="http://localhost:8000/server-status"


if [ "x$ARGV" = "x" ]; then

    ARGV="-h"

fi


PROCESS_NAME="httpd (mod_wsgi-express)  "


case $ACMD in

start|stop|restart|graceful|graceful-stop)

    exec -a "$PROCESS_NAME" $HTTPD -k $ARGV

    ;;

configtest)

    exec $HTTPD -t

    ;;

status)

    exec /home/xliu/Ureka1_4_1/Ureka/variants/common/bin/python -m webbrowser -t $STAT

USURL

    ;;

*)

    exec $HTTPD $ARGV

esac


...back 1 page

#!/usr/bin/env bash


# ['/home/xliu/Ureka1_4_1/Ureka/variants/common/bin/mod_wsgi-express', 'start-server',

 '--debug-mode', '--setup-only']


HTTPD="/usr/sbin/httpd -f /tmp/mod_wsgi-localhost:8000:5099/httpd.conf -DONE_PROCESS"


SHLIBPATH=""


if [ "x$SHLIBPATH" != "x" ]; then

    LD_LIBRARY_PATH="$SHLIBPATH:$LD_LIBRARY_PATH"

    export LD_LIBRARY_PATH

fi


WSGI_RUN_USER="${WSGI_RUN_USER:-xliu}"

WSGI_RUN_GROUP="${WSGI_RUN_GROUP:-nhsc}"


export WSGI_RUN_USER

export WSGI_RUN_GROUP


if [ `id -u` = "0" -a ${WSGI_RUN_USER} = "root" ]; then

    cat << EOF


WARNING: When running as the 'root' user, it is required that the options

'--user' and '--group' be specified to mod_wsgi-express. These should

define a non 'root' user and group under which the Apache child worker

processes and mod_wsgi daemon processes should be run. Failure to specify

these options will result in Apache and/or the mod_wsgi daemon processes

failing to start. See the mod_wsgi-express documentation for further

information on this restriction.


EOF


fi


LANG='en_US.ISO8859-1'

LC_ALL='en_US.ISO8859-1'


export LANG

export LC_ALL


ACMD="$1"

ARGV="$@"


if test -f /tmp/mod_wsgi-localhost:8000:5099/envvars; then

    . /tmp/mod_wsgi-localhost:8000:5099/envvars

fi


STATUSURL="http://localhost:8000/server-status"


if [ "x$ARGV" = "x" ]; then

    ARGV="-h"

fi


PROCESS_NAME="httpd (mod_wsgi-express)  "


case $ACMD in

start|stop|restart|graceful|graceful-stop)

    exec -a "$PROCESS_NAME" $HTTPD -k $ARGV

    ;;

configtest)

    exec $HTTPD -t

    ;;

status)

    exec /home/xliu/Ureka1_4_1/Ureka/variants/common/bin/python -m webbrowser -t $STAT

USURL

    ;;

*)

    exec $HTTPD $ARGV

esac

...

Graham Dumpleton

unread,
Jan 11, 2015, 6:01:19 PM1/11/15
to mod...@googlegroups.com
Reinstall mod_wsgi-express using latest development version.

    /home/xliu/Ureka1_4_1/Ureka/python/bin/pip install -U https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip

Ensure that -U option is used to force the install.

Now run:

    mod_wsgi-express start-server --debug-mode --enable-gdb

That newer version now has the --enable-gdb option for debug mode and will run up gdb for you so you don't have to worry about getting it right. It ensures that all environment variables required are also set automatically.

I don't quite understand why it complained about WSGI_RUN_USER if you were running gdb as yourself and not under gdb.

So remember, you do not need to be running this under sudo, just as yourself.

Graham

Cate Liu

unread,
Jan 11, 2015, 7:49:04 PM1/11/15
to mod...@googlegroups.com


mod_wsgi-express start-server --debug-mode --enable-gdb

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:5099

Server Conf        : /tmp/mod_wsgi-localhost:8000:5099/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:5099/error_log

Request Capacity   : 5 (1 process * 5 threads)

Request Timeout    : 60 (seconds)

Queue Backlog      : 100 (connections)

Queue Timeout      : 45 (seconds)

Server Capacity    : 20 (event/worker), 20 (prefork)

Server Backlog     : 500 (connections)

Locale Setting     : en_US.ISO8859-1

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.

------------------

Program received signal SIGSEGV, Segmentation fault.

0x00007ffff4c25c27 in wsgi_python_init (p=0x7ffff82ed5b8) at src/server/wsgi_interp.c:1963

1963 src/server/wsgi_interp.c: No such file or directory.

in src/server/wsgi_interp.c

Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-29.el6_4.x86_64

(gdb)  


...

Graham Dumpleton

unread,
Jan 11, 2015, 7:58:20 PM1/11/15
to mod...@googlegroups.com
Which equates to the code:

        /* Check for Python paths and optimisation flag. */

        if (wsgi_server_config->python_optimize > 0)
            Py_OptimizeFlag = wsgi_server_config->python_optimize;
        else
            Py_OptimizeFlag = 0;

Specifically the line:

            Py_OptimizeFlag = 0;

Not quite what I was expecting.

Can you run:

    file `mod_wsgi-express module-location`

and:

    file /home/xliu/Ureka1_4_1/Ureka/variants/common/lib/libpython2.7.so

Change path to libpython27.so if got this wrong. Wasn't in scroll back of current message history.

What I am wanting to check here is that both mod_wsgi and the Python library are compiled as both 64 bit or both 32 bit.

The output of file should tell you this.

Graham

Cate Liu

unread,
Jan 11, 2015, 8:12:31 PM1/11/15
to mod...@googlegroups.com

mod_wsgi-express module-location

/home/xliu/Ureka1_4_1/Ureka/variants/common/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so


file /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so

/home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so: symbolic link to `libpython2.7.so.1.0'

file /home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0 

/home/xliu/Ureka1_4_1/Ureka/python/lib/libpython2.7.so.1.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

...

Graham Dumpleton

unread,
Jan 11, 2015, 8:21:24 PM1/11/15
to mod...@googlegroups.com
Was actually wanting:

    file `mod_wsgi-express module-location`

for the first.

I was more concerned about the Python library being 32 bit, which it isn't, than mod_wsgi.so not being 64 bit.

But if can get 'file' output from running it on the mod_wsgi.so, that will still confirm.

Am a bit stumped right now I will then have to think about next step.

You could though try at the gdb prompt when it appears print out the address of Py_OptimizeFlag and its value.

(gdb) print &Py_OptimizeFlag
$1 = (<data variable, no debug info> *) 0x1005988e8
(gdb) print (int)Py_OptimizeFlag
$2 = 0

The latter should see gdb complain.

Graham

Cate Liu

unread,
Jan 11, 2015, 8:28:04 PM1/11/15
to mod...@googlegroups.com

file `mod_wsgi-express module-location`

/home/xliu/Ureka1_4_1/Ureka/variants/common/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

...

Cate Liu

unread,
Jan 11, 2015, 8:29:53 PM1/11/15
to mod...@googlegroups.com

(gdb) print &Py_OptimizeFlag

$1 = (int *) 0x655c20

(gdb) print (int)Py_OptimizeFlag

Cannot access memory at address 0x655c20

(gdb) 

...

Cate Liu

unread,
Jan 11, 2015, 11:22:59 PM1/11/15
to mod...@googlegroups.com
Graham,

In case you wonder why Ureka doesn't work with mod_wsgi-express, perhaps you can download it from http://ssb.stsci.edu/ureka/.
Just a thought.  Cheers,

Cate
...

Graham Dumpleton

unread,
Jan 11, 2015, 11:28:48 PM1/11/15
to mod...@googlegroups.com
Okay. I hadn't asked whether it was available as I assumed it was a private release.

At this point, when I have the time, trying that may well be the next step. Only issue will be if can get it working inside of a Docker container rather than requiring me to install a full Linux VM. Probably not due to the decencies on X window and FreeType libraries.

Thanks for telling me where it comes from.

Graham

Cate Liu

unread,
Jan 13, 2015, 7:27:25 PM1/13/15
to mod...@googlegroups.com
Graham,

I am very happy to report that after switching to another python package (anaconda), now the web server starts with NO problem!
I wonder what is causing the seg fault problem in the Ureka lib.  Here is the comparison of the two libpython2.7.so.  Yes, Ureka has
much more stuff bundled than Anaconda.  But which one is the culprit?  I'd like to know if you have any inklings.
Thank you sooo very much for taking the time with me debugging this crazy error.

Cate


ldd anaconda2_1_0linux/lib/libpython2.7.so

linux-vdso.so.1 =>  (0x00007fff625ff000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f14d18fb000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007f14d16f6000)

libutil.so.1 => /lib64/libutil.so.1 (0x00007f14d14f3000)

libm.so.6 => /home/xliu/anaconda2_1_0linux/lib/./libm.so.6 (0x00007f14d1270000)

libc.so.6 => /lib64/libc.so.6 (0x00007f14d0edc000)

/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)


ldd Ureka1_4_1/Ureka/python/lib/libpython2.7.so

linux-vdso.so.1 =>  (0x00007fff717ff000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc0b52c3000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007fc0b50bf000)

libutil.so.1 => /lib64/libutil.so.1 (0x00007fc0b4ebb000)

libm.so.6 => /lib64/libm.so.6 (0x00007fc0b4c37000)

libc.so.6 => /lib64/libc.so.6 (0x00007fc0b48a4000)

/lib64/ld-linux-x86-64.so.2 (0x0000003bc1600000)

libX11.so.6 => /usr/lib64/libX11.so.6 (0x00007fc0b5759000)

libXft.so.2 => /usr/lib64/libXft.so.2 (0x00007fc0b5543000)

libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x00007fc0b4685000)

libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007fc0b444f000)

libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007fc0b41b2000)

libXrender.so.1 => /usr/lib64/libXrender.so.1 (0x00007fc0b3fa7000)

libXau.so.6 => /usr/lib64/libXau.so.6 (0x00007fc0b3da4000)

libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fc0b3b7c000)

...

Graham Dumpleton

unread,
Jan 13, 2015, 7:30:36 PM1/13/15
to mod...@googlegroups.com
Just so you know, this still is in my inbox, but it will be a little while before I get back to looking at this. Am having to get a mod_wsgi release out and all the checking I need to do for that is taking time, plus have my normal job as well. :-)

Graham

Reply all
Reply to author
Forward
0 new messages