Mod_python's child segmentation fault
Clodoaldo
try to run a helloworld with mod_python's publisher I get a
segmentation fault. Without the wsgi_module loaded the publisher page
works. This already happened with modwsgi 204.
#helloworld.py
def index(req):
return """\
<html><body>
<p>hello world</p>
</body></html>
"""
#app.wsgi
def application(environ, start_response):
status = '200 OK'
output = 'Hello World!'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]
<VirtualHost 10.1.1.101:80>
ServerName wsgiapp.dkt
DocumentRoot /var/www/html/wsgiapp
WSGIScriptAlias / /var/www/html/wsgiapp/
<Directory /var/www/html/wsgiapp>
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
<VirtualHost 10.1.1.101:80>
ServerName test.dkt
DocumentRoot /var/www/html/test.dkt
ErrorLog logs/test.dkt-error_log
CustomLog logs/test.dkt-access_log combined
PythonDebug On
<Directory /var/www/html/test.dkt/modpython>
SetHandler mod_python
PythonHandler mod_python.publisher
PythonOption mod_python.importer.path "['~/mod']"
Order allow,deny
Allow from all
</Directory>
Options -Indexes
</VirtualHost>
[Thu May 10 08:11:08 2007] [notice] caught SIGTERM, shutting down
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27492): Terminating
Python.
[Thu May 10 08:11:08 2007] [notice] suEXEC mechanism enabled
(wrapper: /usr/sbin/suexec)
[Thu May 10 08:11:08 2007] [notice] Digest: generating secret for
digest authentication ...
[Thu May 10 08:11:08 2007] [notice] Digest: done
[Thu May 10 08:11:08 2007] [notice] mod_python: Creating 8 session
mutexes based on 256 max processes and 0 max threads.
[Thu May 10 08:11:08 2007] [notice] mod_python: using mutex_directory /
tmp
[Thu May 10 08:11:08 2007] [notice] mod_wsgi: Initializing Python.
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27602): Attach
interpreter ''.
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27603): Attach
interpreter ''.
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27604): Attach
interpreter ''.
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27605): Attach
interpreter ''.
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27606): Attach
interpreter ''.
[Thu May 10 08:11:08 2007] [notice] Apache/2.2.2 (Fedora) configured
-- resuming normal operations
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27607): Attach
interpreter ''.
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27608): Attach
interpreter ''.
[Thu May 10 08:11:08 2007] [notice] mod_wsgi (pid=27609): Attach
interpreter ''.
[Thu May 10 08:11:16 2007] [notice] mod_wsgi (pid=27602): Create
interpreter 'wsgiapp.dkt|/app.wsgi'.
[Thu May 10 08:11:16 2007] [notice] mod_wsgi (pid=27602, daemon='',
group='wsgiapp.dkt|/app.wsgi'): Loading WSGI script '/var/www/html/
wsgiapp/app.wsgi'.
[Thu May 10 08:11:19 2007] [notice] child pid 27603 exit signal
Segmentation fault (11)
# httpd -V
Server version: Apache/2.2.2
Server built: Jul 26 2006 11:12:08
Server's Module Magic Number: 20051115:2
Server loaded: APR 1.2.2, APR-Util 1.2.7
Compiled using: APR 1.2.2, APR-Util 1.2.7
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
# uname -a
Linux dkt.localdomain 2.6.20-1.2312.fc5 #1 Tue Apr 10 15:09:44 EDT
2007 i686 athlon i386 GNU/Linux
# python -V
Python 2.4.3
Regards, Clodoaldo Pinto Neto
Graham Dumpleton
>
> I have modwsgi 215 installed and working and mod_python 3.3. When I
> try to run a helloworld with mod_python's publisher I get a
> segmentation fault. Without the wsgi_module loaded the publisher page
> works. This already happened with modwsgi 204.
>
>
> [Thu May 10 08:11:16 2007] [notice] mod_wsgi (pid=27602, daemon='',
> group='wsgiapp.dkt|/app.wsgi'): Loading WSGI script '/var/www/html/
> wsgiapp/app.wsgi'.
> [Thu May 10 08:11:19 2007] [notice] child pid 27603 exit signal
> Segmentation fault (11)
This particular message isn't from rev 215 as format was changed, so
must be from rev 204. If you believe it is from when you were running
rev 215, then must be picking up old module somehow. Also may be
necessary to explicitly 'stop' Apache and 'start' it again, ie., don't
use 'restart' in case Apache isn't unloading modules properly when
version being switched, something which I have seen occasionally and
has for some reason also resulted in me seeing random crashes as well
for some reason after doing the 'restart'.
Now, not saying this to discount there being a problem as it has been
a while since I have done tests with running mod_python and mod_wsgi
together. Getting late here now, but I will run some fresh tests
tomorrow with both loaded.
Thanks.
Graham
Clodoaldo
wrote:
> On 10/05/07, Clodoaldo <clodoaldo.pi...@gmail.com> wrote:
>
>
>
> > I have modwsgi 215 installed and working and mod_python 3.3. When I
> > try to run a helloworld with mod_python's publisher I get a
> > segmentation fault. Without the wsgi_module loaded the publisher page
> > works. This already happened with modwsgi 204.
>
> > ...
>
> > [Thu May 10 08:11:16 2007] [notice] mod_wsgi (pid=27602, daemon='',
> > group='wsgiapp.dkt|/app.wsgi'): Loading WSGI script '/var/www/html/
> > wsgiapp/app.wsgi'.
> > [Thu May 10 08:11:19 2007] [notice] child pid 27603 exit signal
> > Segmentation fault (11)
>
> This particular message isn't from rev 215 as format was changed, so
> must be from rev 204. If you believe it is from when you were running
> rev 215, then must be picking up old module somehow. Also may be
> necessary to explicitly 'stop' Apache and 'start' it again, ie., don't
> use 'restart' in case Apache isn't unloading modules properly when
> version being switched, something which I have seen occasionally and
> has for some reason also resulted in me seeing random crashes as well
> for some reason after doing the 'restart'.
I stoped and started apache in instead of restarting and the message
now is different:
[Thu May 10 09:07:49 2007] [notice] mod_wsgi (pid=28968, process='',
application='wsgiapp.dkt|/app.wsgi'): Loading WSGI script '/var/www/
html/wsgiapp/app.wsgi'.
> Now, not saying this to discount there being a problem as it has been
> a while since I have done tests with running mod_python and mod_wsgi
> together. Getting late here now, but I will run some fresh tests
> tomorrow with both loaded.
Now running 215 the seg fault persists.
Clodoaldo
Graham Dumpleton
Will try Ubuntu with Apache 2.2 and Python 2.5.1 and see if I can
duplicate it.
If you have the opportunity can you shutdown your Apache and then run
it in single process mode under 'gdb' and catch a stack trace for the
crash.
sudo apachectl stop
sudo gdb /usr/sbin/httpd
# In gdb
run -X
# Then request against URL, when it crashes, in gdb
where
Thanks.
Graham
On 10/05/07, Clodoaldo <clodoal...@gmail.com> wrote:
>
Clodoaldo
>
> Can't find any problems on Mac OS X with Apache 2.2 and Python 2.3.5.
> Will try Ubuntu with Apache 2.2 and Python 2.5.1 and see if I can
> duplicate it.
>
> If you have the opportunity can you shutdown your Apache and then run
> it in single process mode under 'gdb' and catch a stack trace for the
> crash.
>
> sudo apachectl stop
> sudo gdb /usr/sbin/httpd
>
> # In gdb
>
> run -X
>
> # Then request against URL, when it crashes, in gdb
>
> where
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208289600 (LWP 32719)]
0x010748af in PyImport_GetModuleDict () from /etc/httpd/modules/mod_wsgi.so
(gdb) where
#0 0x010748af in PyImport_GetModuleDict () from /etc/httpd/modules/mod_wsgi.so
#1 0x01074902 in PyImport_AddModule () from /etc/httpd/modules/mod_wsgi.so
#2 0x0107b018 in Py_InitModule4 () from /etc/httpd/modules/mod_wsgi.so
#3 0x00bd926c in inittime () from /usr/lib/python2.4/lib-dynload/timemodule.so
#4 0x7190ab28 in _PyImport_LoadDynamicModule () from
/etc/httpd/modules/mod_python.so
#5 0x7190871e in load_module () from /etc/httpd/modules/mod_python.so
#6 0x71908ed3 in import_submodule () from /etc/httpd/modules/mod_python.so
#7 0x71909414 in load_next () from /etc/httpd/modules/mod_python.so
#8 0x719095a8 in PyImport_ImportModuleEx () from
/etc/httpd/modules/mod_python.so
#9 0x7193fa6e in builtin___import__ () from /etc/httpd/modules/mod_python.so
#10 0x718bf28d in PyCFunction_Call () from /etc/httpd/modules/mod_python.so
#11 0x718a0e27 in PyObject_Call () from /etc/httpd/modules/mod_python.so
#12 0x718ec7ec in PyEval_CallObjectWithKeywords () from
/etc/httpd/modules/mod_python.so
#13 0x718ee764 in PyEval_EvalFrame () from /etc/httpd/modules/mod_python.so
#14 0x718f3088 in PyEval_EvalCodeEx () from /etc/httpd/modules/mod_python.so
#15 0x718f3113 in PyEval_EvalCode () from /etc/httpd/modules/mod_python.so
#16 0x7190784d in PyImport_ExecCodeModuleEx () from
/etc/httpd/modules/mod_python.so
#17 0x71907df3 in load_source_module () from /etc/httpd/modules/mod_python.so
#18 0x71908ed3 in import_submodule () from /etc/httpd/modules/mod_python.so
#19 0x71909392 in load_next () from /etc/httpd/modules/mod_python.so
#20 0x719095f6 in PyImport_ImportModuleEx () from
/etc/httpd/modules/mod_python.so
#21 0x7193fa6e in builtin___import__ () from /etc/httpd/modules/mod_python.so
#22 0x718bf28d in PyCFunction_Call () from /etc/httpd/modules/mod_python.so
#23 0x718a0e27 in PyObject_Call () from /etc/httpd/modules/mod_python.so
#24 0x718a1900 in PyObject_CallFunction () from /etc/httpd/modules/mod_python.so
#25 0x71909a26 in PyImport_Import () from /etc/httpd/modules/mod_python.so
#26 0x71909c05 in PyImport_ImportModule () from /etc/httpd/modules/mod_python.so
#27 0x7189bdf3 in get_interpreter (name=0x81d1a3a8 "test.dkt") at
mod_python.c:107
#28 0x7189fbe2 in python_handler (req=0x81e3f958, phase=0x71943e35
"PythonHandler") at mod_python.c:1628
#29 0x800251fd in ap_run_handler () from /usr/sbin/httpd
#30 0x80028b98 in ap_invoke_handler () from /usr/sbin/httpd
#31 0x80034a9e in ap_process_request () from /usr/sbin/httpd
#32 0x8003189f in ap_register_input_filter () from /usr/sbin/httpd
#33 0x8002d07d in ap_run_process_connection () from /usr/sbin/httpd
#34 0x8002d17c in ap_process_connection () from /usr/sbin/httpd
#35 0x80039052 in ap_graceful_stop_signalled () from /usr/sbin/httpd
#36 0x800392c4 in ap_graceful_stop_signalled () from /usr/sbin/httpd
#37 0x8003a1d9 in ap_mpm_run () from /usr/sbin/httpd
#38 0x80010a9a in main () from /usr/sbin/httpd
Regards,
--
Clodoaldo Pinto Neto
Graham Dumpleton
gdb does that.
Specifically:
#0 0x010748af in PyImport_GetModuleDict () from /etc/httpd/modules/mod_wsgi.so
#1 0x01074902 in PyImport_AddModule () from /etc/httpd/modules/mod_wsgi.so
#2 0x0107b018 in Py_InitModule4 () from /etc/httpd/modules/mod_wsgi.so
#3 0x00bd926c in inittime () from /usr/lib/python2.4/lib-dynload/timemodule.so
#4 0x7190ab28 in _PyImport_LoadDynamicModule () from
/etc/httpd/modules/mod_python.so
#5 0x7190871e in load_module () from /etc/httpd/modules/mod_python.so
The calls can't be jumping from mod_python to 'time.so' to mod_wsgi
like that. :-(
Will just have to see if anything shows up on Ubuntu.
Thanks.
Graham
Graham Dumpleton
can try. :-(
Only significant difference in Apache is that Ubuntu uses worker MPM
instead of prefork.
Does it crash every time or does it depend some how on whether a
mod_wsgi request was processed first?
Graham
Clodoaldo
>
> And Ubuntu works fine for me as well. Will have to think what else I
> can try. :-(
>
> Only significant difference in Apache is that Ubuntu uses worker MPM
> instead of prefork.
>
> Does it crash every time or does it depend some how on whether a
> mod_wsgi request was processed first?
It crashes every time and it is not necessary to run a wsgi app first.
The only requirement is to have the wsgi_module loaded. This is with
FC5. I will try FC6.
--
Clodoaldo Pinto Neto
Clodoaldo
It is the same in FC6. This time the log shows the modpython seg fault
first and then the wsgi app working:
[Fri May 11 09:14:51 2007] [notice] suEXEC mechanism enabled (wrapper:
/usr/sbin/suexec)
[Fri May 11 09:14:51 2007] [notice] Digest: generating secret for
digest authentication ...
[Fri May 11 09:14:51 2007] [notice] Digest: done
[Fri May 11 09:14:51 2007] [notice] mod_python: Creating 8 session
mutexes based on 256 max processes and 0 max threads.
[Fri May 11 09:14:51 2007] [notice] mod_python: using mutex_directory /tmp
[Fri May 11 09:14:51 2007] [notice] mod_wsgi: Initializing Python.
[Fri May 11 09:14:51 2007] [notice] mod_wsgi (pid=10053): Attach
interpreter ''.[Fri May 11 09:14:51 2007] [notice] Apache/2.2.4 (Unix)
DAV/2 PHP/5.1.6 mod_python/3.3.1 Python/2.4.4 mod_wsgi/1.0-TRUNK
Python/2.4.4 configured -- resuming normal operations
[Fri May 11 09:14:58 2007] [notice] child pid 10053 exit signal
Segmentation fault (11)
[Fri May 11 09:14:58 2007] [notice] mod_wsgi (pid=10054): Attach
interpreter ''.[Fri May 11 09:15:09 2007] [notice] mod_wsgi
(pid=10054): Create interpreter 'wsgiapp.s0|/app.wsgi'.
[Fri May 11 09:15:09 2007] [notice] mod_wsgi (pid=10054, process='',
application='wsgiapp.s0|/app.wsgi'): Loading WSGI script
'/var/www/html/wsgiapp/app.wsgi'.
[Fri May 11 09:15:09 2007] [notice] mod_wsgi (pid=10055): Attach interpreter ''.
# httpd -V
Server version: Apache/2.2.4 (Unix)
Server built: Apr 27 2007 06:34:56
Server's Module Magic Number: 20051115:4
Server loaded: APR 1.2.7, APR-Util 1.2.8
Compiled using: APR 1.2.7, APR-Util 1.2.8
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
# uname -a
Linux s0.localdomain 2.6.20-1.2925.fc6 #1 SMP Sat Mar 10 18:20:58 EST
2007 i686 athlon i386 GNU/Linux
# python -V
Python 2.4.4
--
Clodoaldo Pinto Neto
Clodoaldo
> 2007/5/11, Clodoaldo <clodoaldo.pi...@gmail.com>:
>
> > 2007/5/10, Graham Dumpleton <graham.dumple...@gmail.com>:
>
> > > And Ubuntu works fine for me as well. Will have to think what else I
> > > can try. :-(
>
> > > Only significant difference in Apache is that Ubuntu uses worker MPM
> > > instead of prefork.
>
> > > Does it crash every time or does it depend some how on whether a
> > > mod_wsgi request was processed first?
>
> > It crashes every time and it is not necessary to run a wsgi app first.
> > The only requirement is to have the wsgi_module loaded. This is with
> > FC5. I will try FC6.
>
> It is the same in FC6. This time the log shows the modpython seg fault
> first and then the wsgi app working:
And gdb output in FC6:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208584496 (LWP 10086)]
0x010781af in PyImport_GetModuleDict () from /etc/httpd/modules/
mod_wsgi.so
(gdb) where
#0 0x010781af in PyImport_GetModuleDict () from /etc/httpd/modules/
mod_wsgi.so
#1 0x01078202 in PyImport_AddModule () from /etc/httpd/modules/
mod_wsgi.so
#2 0x0107e968 in Py_InitModule4 () from /etc/httpd/modules/
mod_wsgi.so
#3 0x00be401c in inittime () from /usr/lib/python2.4/lib-dynload/
timemodule.so
#4 0x6cabb358 in _PyImport_LoadDynamicModule () from /etc/httpd/
modules/mod_python.so
#5 0x6cab8f4e in load_module () from /etc/httpd/modules/mod_python.so
#6 0x6cab96e3 in import_submodule () from /etc/httpd/modules/
mod_python.so
#7 0x6cab9c24 in load_next () from /etc/httpd/modules/mod_python.so
#8 0x6cab9db8 in PyImport_ImportModuleEx () from /etc/httpd/modules/
mod_python.so
#9 0x6caf072e in builtin___import__ () from /etc/httpd/modules/
mod_python.so
#10 0x6ca6facd in PyCFunction_Call () from /etc/httpd/modules/
mod_python.so
#11 0x6ca51417 in PyObject_Call () from /etc/httpd/modules/
mod_python.so
#12 0x6ca9d08c in PyEval_CallObjectWithKeywords () from /etc/httpd/
modules/mod_python.so
#13 0x6ca9efdb in PyEval_EvalFrame () from /etc/httpd/modules/
mod_python.so
#14 0x6caa3865 in PyEval_EvalCodeEx () from /etc/httpd/modules/
mod_python.so
#15 0x6caa3943 in PyEval_EvalCode () from /etc/httpd/modules/
mod_python.so
#16 0x6cab807d in PyImport_ExecCodeModuleEx () from /etc/httpd/modules/
mod_python.so
#17 0x6cab861c in load_source_module () from /etc/httpd/modules/
mod_python.so
#18 0x6cab96e3 in import_submodule () from /etc/httpd/modules/
mod_python.so
#19 0x6cab9ba2 in load_next () from /etc/httpd/modules/mod_python.so
#20 0x6cab9e06 in PyImport_ImportModuleEx () from /etc/httpd/modules/
mod_python.so
#21 0x6caf072e in builtin___import__ () from /etc/httpd/modules/
mod_python.so
#22 0x6ca6facd in PyCFunction_Call () from /etc/httpd/modules/
mod_python.so
#23 0x6ca51417 in PyObject_Call () from /etc/httpd/modules/
mod_python.so
#24 0x6ca51ef0 in PyObject_CallFunction () from /etc/httpd/modules/
mod_python.so
#25 0x6caba236 in PyImport_Import () from /etc/httpd/modules/
mod_python.so
#26 0x6caba415 in PyImport_ImportModule () from /etc/httpd/modules/
mod_python.so
#27 0x6ca4c3e3 in get_interpreter (name=0x806a0178 "test.s0") at
mod_python.c:107
#28 0x6ca501d2 in python_handler (req=0x80a38238, phase=0x6caf4b55
"PythonHandler") at mod_python.c:1628
#29 0x80024a2d in ap_run_handler () from /usr/sbin/httpd
#30 0x800283e7 in ap_invoke_handler () from /usr/sbin/httpd
#31 0x800344ee in ap_process_request () from /usr/sbin/httpd
#32 0x800312df in ap_register_input_filter () from /usr/sbin/httpd
#33 0x8002ca1d in ap_run_process_connection () from /usr/sbin/httpd
#34 0x8002cb1c in ap_process_connection () from /usr/sbin/httpd
#35 0x80038aa2 in ap_graceful_stop_signalled () from /usr/sbin/httpd
#36 0x80038d14 in ap_graceful_stop_signalled () from /usr/sbin/httpd
#37 0x80039c29 in ap_mpm_run () from /usr/sbin/httpd
#38 0x800101fa in main () from /usr/sbin/httpd
Clodoaldo
Graham Dumpleton
What are the sizes of your mod_python.so and mod_wsgi.so files?
I am thinking that if your Python was installed so as not to be using
a shared library, both the mod_python.so and mod_wsgi.so files will
embed their own copy of the Python library. Depending on how the run
time linker merges the symbols in each, there could actually be two
disparate instances of Python global data in the process and this
could be causing problems especially if Python initialisation happens
twice.
I would expect mod_wsgi.so to be about 100Kbytes and mod_python.so to
be about 300Kbytes if a shared library is being used, or over 1.5Mb if
Python was built with only a static library.
You can also run 'ldd' on each .so file and see if they have a
dependency on libpython2.4.so or not.
I only thought of this possibility after I had left work for the day,
so can't check this theory on Ubuntu until next week.
Graham
On 11/05/07, Clodoaldo <clodoal...@gmail.com> wrote:
>
Clodoaldo
>
> I only have one theory at present.
>
> What are the sizes of your mod_python.so and mod_wsgi.so files?
# ll /usr/lib/httpd/modules/mod_python.so
-rwxr-xr-x 1 root root 1610813 May 10 08:22 /usr/lib/httpd/modules/mod_python.so
# ll /usr/lib/httpd/modules/mod_wsgi.so
-rwxr-xr-x 1 root root 1388179 May 10 08:19 /usr/lib/httpd/modules/mod_wsgi.so
> I am thinking that if your Python was installed so as not to be using
> a shared library, both the mod_python.so and mod_wsgi.so files will
> embed their own copy of the Python library. Depending on how the run
> time linker merges the symbols in each, there could actually be two
> disparate instances of Python global data in the process and this
> could be causing problems especially if Python initialisation happens
> twice.
>
> I would expect mod_wsgi.so to be about 100Kbytes and mod_python.so to
> be about 300Kbytes if a shared library is being used, or over 1.5Mb if
> Python was built with only a static library.
>
> You can also run 'ldd' on each .so file and see if they have a
> dependency on libpython2.4.so or not.
# ldd /usr/lib/httpd/modules/mod_python.so
linux-gate.so.1 => (0x00a1f000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00535000)
libdl.so.2 => /lib/libdl.so.2 (0x0015e000)
libutil.so.1 => /lib/libutil.so.1 (0x00110000)
libm.so.6 => /lib/libm.so.6 (0x00654000)
libc.so.6 => /lib/libc.so.6 (0x00330000)
/lib/ld-linux.so.2 (0x80000000)
# ldd /usr/lib/httpd/modules/mod_wsgi.so
linux-gate.so.1 => (0x00972000)
libpthread.so.0 => /lib/libpthread.so.0 (0x006af000)
libdl.so.2 => /lib/libdl.so.2 (0x00b5b000)
libutil.so.1 => /lib/libutil.so.1 (0x00110000)
libc.so.6 => /lib/libc.so.6 (0x00114000)
/lib/ld-linux.so.2 (0x80000000)
Graham Dumpleton
They definitely are both making use of static Python library rather
than shared library being available.
I did a test on an FC2 box I can get to when I need to, although is a
live site so don't like fiddling it too much, and when I duplicate
arrangement where both mod_python.so and mod_wsgi.so are using static
library I still can't get it to crash anything when using Python 2.3.
That includes running complicated stuff like MoinMoin from mod_python.
Now I know for sure that both of mine are compiled with exactly the
same version of Python. Were both of your modules compiled with exact
same Python, or was mod_python.so from a prebuilt binary package?
If you have some more time, I wouldn't mind if you could try one more
thing for me. After you have run configure for mod_wsgi, edit the
Makefile and change the LDLIBS line so that it isn't linking in
-lpythonX.Y. For example:
#LDLIBS = -lpython2.3 -lpthread -ldl -lutil
LDLIBS = -lpthread -ldl -lutil
Then rebuild mod_wsgi, install it and restart Apache and see if
anything is different.
By doing this, one will have:
1724 -rwxr-xr-x 1 root root 1759306 Dec 13 05:03 mod_python.so
120 -rwxr-xr-x 1 root root 116215 May 12 00:32 mod_wsgi.so
Note how mod_wsgi.so is now so much smaller. This is because it hasn't
got a copy of the Python library embedded in it. Instead, it will use
the copy which mod_python.so has already loaded into Apache. Just make
sure that the LoadModule line for mod_python.so comes before
mod_wsgi.so in the Apache configuration.
By doing this, it will absolutely guarantee that the problem can't be
caused by two separate versions of the Python library objects being
loaded by the two different modules.
Let me know if that makes a difference or not.
Thanks.
Graham
Clodoaldo
That fixes it. Now both run whithout errors.
# ll /usr/lib/httpd/modules/mod_wsgi.so
-rwxr-xr-x 1 root root 136825 May 12 06:35 /usr/lib/httpd/modules/mod_wsgi.so
The installed mod_python.so was compiled from source within a few
minutes difference (if that much) from when mod_wsgi.so was compiled.
Why would they use a different library?
Graham Dumpleton
memory of the same library. This is because the library being static
rather than shared, has to be embedded in the .so file. This occurs
for both. On some systems the symbols may not get merged together
seemlessly so there is only one actual copy in memory being used and
that may as a result cause problems.
Just another reason why having a Python installation which wasn't
configured with --enable-shared option when being installed is bad.
:-(
Anyway, I'll have to update the documentation to warn of this and
possibly mention the workaround for those who don't have the option or
rebuilding their Python installation to use shared libraries.
Thanks for trying the different things and giving the feedback on your
configuration etc.
Graham
Clodoaldo
>
> Not a different version of the library but each have their own copy in
> memory of the same library. This is because the library being static
> rather than shared, has to be embedded in the .so file. This occurs
> for both. On some systems the symbols may not get merged together
> seemlessly so there is only one actual copy in memory being used and
> that may as a result cause problems.
>
> Just another reason why having a Python installation which wasn't
> configured with --enable-shared option when being installed is bad.
> :-(
>
> Anyway, I'll have to update the documentation to warn of this and
> possibly mention the workaround for those who don't have the option or
> rebuilding their Python installation to use shared libraries.
If that is a problem in Fedora 5/6 it is probably also a problem with
the new Redhat and Centos, in which case a large user base will be
affected :(
> Thanks for trying the different things and giving the feedback on your
> configuration etc.
I wish the best to modwsgi and am willing to test to make it better.
Clodoaldo
Clodoaldo
> 2007/5/11, Graham Dumpleton <graham.d...@gmail.com>:
> >
> > I only have one theory at present.
> >
> > What are the sizes of your mod_python.so and mod_wsgi.so files?
>
> # ll /usr/lib/httpd/modules/mod_python.so
> -rwxr-xr-x 1 root root 1610813 May 10 08:22 /usr/lib/httpd/modules/mod_python.so
>
> # ll /usr/lib/httpd/modules/mod_wsgi.so
> -rwxr-xr-x 1 root root 1388179 May 10 08:19 /usr/lib/httpd/modules/mod_wsgi.so
FWIW those sizes where for Fedora Core 5/6. Now I have Fedora 7
installed and the sizes are much smaller:
# ll /usr/lib/httpd/modules/mod_python.so
-rwxr-xr-x 1 root root 114712 2007-02-19 13:35
/usr/lib/httpd/modules/mod_python.so
# ll /usr/lib/httpd/modules/mod_wsgi.so
-rwxr-xr-x 1 root root 165119 2007-06-13 09:45
/usr/lib/httpd/modules/mod_wsgi.so
Fedora 7:
# ldd /usr/lib/httpd/modules/mod_python.so
linux-gate.so.1 => (0x007a4000)
libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0x00547000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00110000)
libdl.so.2 => /lib/libdl.so.2 (0x0049c000)
libutil.so.1 => /lib/libutil.so.1 (0x003cb000)
libm.so.6 => /lib/libm.so.6 (0x00472000)
libc.so.6 => /lib/libc.so.6 (0x00128000)
/lib/ld-linux.so.2 (0x80000000)
# ldd /usr/lib/httpd/modules/mod_wsgi.so
linux-gate.so.1 => (0x00377000)
libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0x00a8e000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00a2a000)
libdl.so.2 => /lib/libdl.so.2 (0x006ad000)
libutil.so.1 => /lib/libutil.so.1 (0x0067e000)
libc.so.6 => /lib/libc.so.6 (0x00dbe000)
libm.so.6 => /lib/libm.so.6 (0x00d49000)
/lib/ld-linux.so.2 (0x80000000)
As expected, in Fedora 7 it is not necessary to apply the workaround
in the Makefile to avoid libraries problems.
Regards, Clodoaldo Pinto Neto
Graham Dumpleton
>
> 2007/5/11, Clodoaldo <clodoal...@gmail.com>:
> > 2007/5/11, Graham Dumpleton <graham.d...@gmail.com>:
> > >
> > > I only have one theory at present.
> > >
> > > What are the sizes of your mod_python.so and mod_wsgi.so files?
> >
> > # ll /usr/lib/httpd/modules/mod_python.so
> > -rwxr-xr-x 1 root root 1610813 May 10 08:22 /usr/lib/httpd/modules/mod_python.so
> >
> > # ll /usr/lib/httpd/modules/mod_wsgi.so
> > -rwxr-xr-x 1 root root 1388179 May 10 08:19 /usr/lib/httpd/modules/mod_wsgi.so
>
> FWIW those sizes where for Fedora Core 5/6. Now I have Fedora 7
> installed and the sizes are much smaller:
>
> # ll /usr/lib/httpd/modules/mod_python.so
> -rwxr-xr-x 1 root root 114712 2007-02-19 13:35
> /usr/lib/httpd/modules/mod_python.so
> # ll /usr/lib/httpd/modules/mod_wsgi.so
> -rwxr-xr-x 1 root root 165119 2007-06-13 09:45
> /usr/lib/httpd/modules/mod_wsgi.so
Hmmm, that mod_python.so is smaller is odd. Can only image that
mod_wsgi.so must have been compiled with combination of -O -g flags or
perhaps just -g depending on how apxs is setup, whereas mod_python.so
is just -O with .so file also being stripped.
On Ubuntu the binary packaged mod_python.so is 149018 with mod_wsgi.so
being 99148. This is with neither stripped. If I strip mod_wsgi.so
then it drops further to 82536.
You can tell if a .so is stripped by running:
nm /usr/lib/httpd/modules/mod_python.so
If stripped it will give back message saying 'no symbols', otherwise
it shows all the symbols.
Stripping symbols only means you can get meaningful tracebacks in
crash dumps or in debugger if need to debug it. Stripping symbols does
stop a shared object from being loaded as it still retains enough
information for that to work.
Anyway, I'll have to look at whether I should at least add a -O flag
to mod_wsgi makefile and whether I can pass that to apxs when
compiling. At the moment I ignore such compiler flags from Python
setup as different platforms may use arbitrary options which apxs will
reject if passed to it.
Graham
Graham Dumpleton
Can't add -O in Makefile as apxs rejects it. Only way is to change
apxs setup. On my Ubuntu box this means editing:
/usr/share/apache2/build/config_vars.mk
and adding -O into CFLAGS definition.
End result is that size of mod_wsgi.so when compiled is then 85859 and
70320 if also stripped.
In the bigger scheme of things, since the module should be shared,
will not drop memory use any noticeable amount. Not sure whether
turning on optimisation will help or not as mod_wsgi doesn't exactly
do that much itself, mainly utilising Apache and Python libraries.
Might be worthwhile for me to add some notes about this on my
performance tuning page still when I write it since it looks like
different Apache distributions may or may not have at least -O defined
within apxs configuration. If people want to feel that they are
squeezing out as much performance as possible and reducing memory use
as much as possible as well, they may want to play with the apxs
compiler setup.
Graham
Clodoaldo
>
> On 14/06/07, Clodoaldo <clodoal...@gmail.com> wrote:
> >
> > 2007/5/11, Clodoaldo <clodoal...@gmail.com>:
> > > 2007/5/11, Graham Dumpleton <graham.d...@gmail.com>:
> > > >
> > > > I only have one theory at present.
> > > >
> > > > What are the sizes of your mod_python.so and mod_wsgi.so files?
> > >
> > > # ll /usr/lib/httpd/modules/mod_python.so
> > > -rwxr-xr-x 1 root root 1610813 May 10 08:22 /usr/lib/httpd/modules/mod_python.so
> > >
> > > # ll /usr/lib/httpd/modules/mod_wsgi.so
> > > -rwxr-xr-x 1 root root 1388179 May 10 08:19 /usr/lib/httpd/modules/mod_wsgi.so
> >
> > FWIW those sizes where for Fedora Core 5/6. Now I have Fedora 7
> > installed and the sizes are much smaller:
> >
> > # ll /usr/lib/httpd/modules/mod_python.so
> > -rwxr-xr-x 1 root root 114712 2007-02-19 13:35
> > /usr/lib/httpd/modules/mod_python.so
> > # ll /usr/lib/httpd/modules/mod_wsgi.so
> > -rwxr-xr-x 1 root root 165119 2007-06-13 09:45
> > /usr/lib/httpd/modules/mod_wsgi.so
>
> Hmmm, that mod_python.so is smaller is odd. Can only image that
> mod_wsgi.so must have been compiled with combination of -O -g flags or
> perhaps just -g depending on how apxs is setup, whereas mod_python.so
> is just -O with .so file also being stripped.
mod_wsgi.so was compiled without any parameters, just pure ./configure
and make. mod_python is the default 3.3.1 binary from F7.
> On Ubuntu the binary packaged mod_python.so is 149018 with mod_wsgi.so
> being 99148. This is with neither stripped. If I strip mod_wsgi.so
> then it drops further to 82536.
>
> You can tell if a .so is stripped by running:
>
> nm /usr/lib/httpd/modules/mod_python.so
# nm /usr/lib/httpd/modules/mod_python.so
nm: /usr/lib/httpd/modules/mod_python.so: no symbols
nm for mod_wsgi.so shows a long list.
Clodoaldo