AH00052: child pid 14477 exit signal Segmentation fault (11)
2,307 views
Skip to first unread message
Abraham Varricatt
May 19, 2016, 8:59:32 AM5/19/16
to modwsgi
Hello,
This is related to the stackoverflow question I posted,
Basically, if I configure a simple wsgi script, I'm able to get a response from my apache server (a vagrant/virtualbox system). But if I connect it to my django application, it logs a segmentation fault on my error_log. Here's the latest output,
[Thu May 19 12:20:23.253545 2016] [mpm_worker:notice] [pid 14475:tid 139748788078336] AH00292: Apache/2.4.20 (Unix) mod_wsgi/4.5.2 Python/2.7.11 configured -- resuming normal operations
[Thu May 19 12:20:23.253666 2016] [mpm_worker:info] [pid 14475:tid 139748788078336] AH00293: Server built: May 19 2016 12:17:51
[Thu May 19 12:20:23.253690 2016] [core:notice] [pid 14475:tid 139748788078336] AH00094: Command line: '/usr/local/apache2/bin/httpd'
[Thu May 19 12:20:23.254588 2016] [wsgi:info] [pid 14477:tid 139748788078336] mod_wsgi (pid=14477): Python home /srv/VENVs/mybottle.
[Thu May 19 12:20:23.253969 2016] [wsgi:info] [pid 14478:tid 139748788078336] mod_wsgi (pid=14478): Python home /srv/VENVs/mybottle.
[Thu May 19 12:20:23.254731 2016] [wsgi:info] [pid 14478:tid 139748788078336] mod_wsgi (pid=14478): Initializing Python.
[Thu May 19 12:20:23.257086 2016] [wsgi:info] [pid 14477:tid 139748788078336] mod_wsgi (pid=14477): Initializing Python.
[Thu May 19 12:20:23.261390 2016] [wsgi:info] [pid 14476:tid 139748788078336] mod_wsgi (pid=14476): Python home /srv/VENVs/mybottle.
[Thu May 19 12:20:23.261493 2016] [wsgi:info] [pid 14476:tid 139748788078336] mod_wsgi (pid=14476): Initializing Python.
[Thu May 19 12:20:23.280954 2016] [wsgi:info] [pid 14477:tid 139748788078336] mod_wsgi (pid=14477): Attach interpreter ''.
[Thu May 19 12:20:23.285482 2016] [wsgi:info] [pid 14476:tid 139748788078336] mod_wsgi (pid=14476): Attach interpreter ''.
[Thu May 19 12:20:23.288636 2016] [wsgi:info] [pid 14478:tid 139748788078336] mod_wsgi (pid=14478): Attach interpreter ''.
[Thu May 19 12:20:41.047746 2016] [wsgi:info] [pid 14477:tid 139748683515648] mod_wsgi (pid=14477): Create interpreter 'local.mybottlebusiness.com|'.
[Thu May 19 12:20:41.050139 2016] [wsgi:info] [pid 14477:tid 139748683515648] [client 192.168.33.1:45061] mod_wsgi (pid=14477, process='', application='local.mybottlebusiness.com|'): Loading WSGI script '/srv/sources/mybottle/waterbottle/apache/mybottle.wsgi'.
[Thu May 19 12:20:41.277396 2016] [wsgi:info] [pid 14560:tid 139748788078336] mod_wsgi (pid=14560): Python home /srv/VENVs/mybottle.
[Thu May 19 12:20:41.277475 2016] [wsgi:info] [pid 14560:tid 139748788078336] mod_wsgi (pid=14560): Initializing Python.
[Thu May 19 12:20:41.293067 2016] [wsgi:info] [pid 14560:tid 139748788078336] mod_wsgi (pid=14560): Attach interpreter ''.
[Thu May 19 12:20:43.974586 2016] [wsgi:info] [pid 14476:tid 139748694005504] mod_wsgi (pid=14476): Create interpreter 'local.mybottlebusiness.com|'.
[Thu May 19 12:20:43.975762 2016] [wsgi:info] [pid 14476:tid 139748694005504] [client 192.168.33.1:45062] mod_wsgi (pid=14476, process='', application='local.mybottlebusiness.com|'): Loading WSGI script '/srv/sources/mybottle/waterbottle/apache/mybottle.wsgi'.
[Thu May 19 12:20:44.283417 2016] [core:notice] [pid 14475:tid 139748788078336] AH00052: child pid 14477 exit signal Segmentation fault (11)
[Thu May 19 12:20:45.284548 2016] [wsgi:info] [pid 14601:tid 139748788078336] mod_wsgi (pid=14601): Python home /srv/VENVs/mybottle.
[Thu May 19 12:20:45.284626 2016] [wsgi:info] [pid 14601:tid 139748788078336] mod_wsgi (pid=14601): Initializing Python.
[Thu May 19 12:20:45.292571 2016] [wsgi:info] [pid 14601:tid 139748788078336] mod_wsgi (pid=14601): Attach interpreter ''.
[Thu May 19 12:20:47.286421 2016] [core:notice] [pid 14475:tid 139748788078336] AH00052: child pid 14476 exit signal Segmentation fault (11)
[Thu May 19 12:20:51.799245 2016] [wsgi:info] [pid 14560:tid 139748788078336] mod_wsgi (pid=14560): Destroying interpreters.
[Thu May 19 12:20:51.799283 2016] [wsgi:info] [pid 14560:tid 139748788078336] mod_wsgi (pid=14560): Cleanup interpreter ''.
[Thu May 19 12:20:51.801764 2016] [wsgi:info] [pid 14478:tid 139748788078336] mod_wsgi (pid=14478): Destroying interpreters.
[Thu May 19 12:20:51.801798 2016] [wsgi:info] [pid 14478:tid 139748788078336] mod_wsgi (pid=14478): Cleanup interpreter ''.
[Thu May 19 12:20:51.804274 2016] [wsgi:info] [pid 14601:tid 139748788078336] mod_wsgi (pid=14601): Destroying interpreters.
[Thu May 19 12:20:51.804300 2016] [wsgi:info] [pid 14601:tid 139748788078336] mod_wsgi (pid=14601): Cleanup interpreter ''.
[Thu May 19 12:20:51.807480 2016] [wsgi:info] [pid 14601:tid 139748788078336] mod_wsgi (pid=14601): Terminating Python.
[Thu May 19 12:20:51.808908 2016] [wsgi:info] [pid 14601:tid 139748788078336] mod_wsgi (pid=14601): Python has shutdown.
[Thu May 19 12:20:51.811662 2016] [wsgi:info] [pid 14560:tid 139748788078336] mod_wsgi (pid=14560): Terminating Python.
[Thu May 19 12:20:51.813568 2016] [wsgi:info] [pid 14478:tid 139748788078336] mod_wsgi (pid=14478): Terminating Python.
[Thu May 19 12:20:51.815993 2016] [wsgi:info] [pid 14478:tid 139748788078336] mod_wsgi (pid=14478): Python has shutdown.
[Thu May 19 12:20:51.817315 2016] [wsgi:info] [pid 14560:tid 139748788078336] mod_wsgi (pid=14560): Python has shutdown.
[Thu May 19 12:20:51.880068 2016] [core:info] [pid 14475:tid 139748788078336] AH00096: removed PID file /usr/local/apache2/logs/httpd.pid (pid=14475)
[Thu May 19 12:20:51.880116 2016] [mpm_worker:notice] [pid 14475:tid 139748788078336] AH00295: caught SIGTERM, shutting downI've tried a LOT of things to get this to work - recompiling different versions of mod_wsgi (versions 4.4.x, 4.3.0 .. etc). Recompiling httpd with mpm of either event, prefork and lastly worker. And a lot of different httpd.conf stuff.
According to the instructions here,
I've tried getting a gdb stacktrace to ask for help. Any idea what the following means? Or at least, any suggestions on how I might get around the segmentation fault?
[vagrant@thebusinessdev bin]$
[vagrant@thebusinessdev bin]$ sudo gdb ./httpd
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/apache2/bin/httpd...done.
(gdb) run -X
Starting program: /usr/local/apache2/bin/httpd -X
[Thread debugging using libthread_db enabled]
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
[New Thread 0x7fffed1a9700 (LWP 14660)]
[New Thread 0x7fffe7fff700 (LWP 14661)]
[New Thread 0x7fffe75fe700 (LWP 14662)]
[New Thread 0x7fffe6bfd700 (LWP 14663)]
[New Thread 0x7fffe61fc700 (LWP 14664)]
[New Thread 0x7fffe57fb700 (LWP 14665)]
[New Thread 0x7fffe4dfa700 (LWP 14666)]
[New Thread 0x7fffe43f9700 (LWP 14667)]
[New Thread 0x7fffe39f8700 (LWP 14668)]
[New Thread 0x7fffe2ff7700 (LWP 14669)]
[New Thread 0x7fffe25f6700 (LWP 14670)]
[New Thread 0x7fffe1bf5700 (LWP 14671)]
[New Thread 0x7fffe11f4700 (LWP 14672)]
[New Thread 0x7fffe07f3700 (LWP 14673)]
[New Thread 0x7fffdfdf2700 (LWP 14674)]
[New Thread 0x7fffdf3f1700 (LWP 14675)]
[New Thread 0x7fffde9f0700 (LWP 14676)]
[New Thread 0x7fffddfef700 (LWP 14677)]
[New Thread 0x7fffdd5ee700 (LWP 14678)]
[New Thread 0x7fffdcbed700 (LWP 14679)]
[New Thread 0x7fffdc1ec700 (LWP 14680)]
[New Thread 0x7fffdb7eb700 (LWP 14681)]
[New Thread 0x7fffdadea700 (LWP 14682)]
[New Thread 0x7fffda3e9700 (LWP 14683)]
[New Thread 0x7fffd99e8700 (LWP 14684)]
[New Thread 0x7fffd8fe7700 (LWP 14685)]
[New Thread 0x7fffd85e6700 (LWP 14686)]
[Thread 0x7fffed1a9700 (LWP 14660) exited]
Detaching after fork from child process 14687.
Detaching after fork from child process 14689.
Detaching after fork from child process 14691.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd8fe7700 (LWP 14685)]
0x00007ffff6c746bf in __strlen_sse42 () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install GeoIP-1.6.5-1.el6.x86_64 bzip2-libs-1.0.5-7.el6_0.x86_64 cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libffi-3.0.5-3.2.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-2.el6_7.1.x86_64 libstdc++-4.4.7-16.el6.x86_64 libtidy-0.99.0-19.20070615.1.el6.x86_64 libuuid-2.17.2-12.18.el6.x86_64 nspr-4.11.0-0.1.el6_7.x86_64 nss-3.21.0-0.3.el6_7.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 nss-util-3.21.0-0.3.el6_7.x86_64 openldap-2.4.40-7.el6_7.x86_64 openssl-1.0.1e-42.el6_7.4.x86_64 pcre-7.8-7.el6.x86_64 sqlite-3.6.20-1.el6_7.2.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) backtrace
#0 0x00007ffff6c746bf in __strlen_sse42 () from /lib64/libc.so.6
#1 0x00007ffff376efeb in PyString_FromString (str=0x0) at Objects/stringobject.c:121
#2 0x00007ffff37e7eb1 in PyModule_AddStringConstant (m=0x7fffc817fec0, name=0x7fffc5fd5188 "XML_ERROR_UNDECLARING_PREFIX",
value=<value optimized out>) at Python/modsupport.c:637
#3 0x00007fffc5fb3db7 in initpyexpat () at extensions/pyexpat.c:1979
#4 0x00007ffff37e47b5 in _PyImport_LoadDynamicModule (name=0x7fffeae66340 "xml.parsers.pyexpat",
pathname=0x7fffeaf30c70 "/srv/VENVs/mybottle/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so", fp=<value optimized out>)
at ./Python/importdl.c:53
#5 0x00007ffff37e2e31 in import_submodule (mod=0x7fffc817fb08, subname=0x7fffeae6634c "pyexpat",
fullname=0x7fffeae66340 "xml.parsers.pyexpat") at Python/import.c:2704
#6 0x00007ffff37e30a4 in load_next (mod=0x7fffc817fb08, altmod=0x7ffff3a80a50, p_name=<value optimized out>,
buf=0x7fffeae66340 "xml.parsers.pyexpat", p_buflen=0x7fffd8fe3290) at Python/import.c:2519
#7 0x00007ffff37e36d0 in import_module_level (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc86da990, level=<value optimized out>) at Python/import.c:2228
#8 PyImport_ImportModuleLevel (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc86da990, level=<value optimized out>) at Python/import.c:2292
#9 0x00007ffff37c311f in builtin___import__ (self=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>)
at Python/bltinmodule.c:49
#10 0x00007ffff3719ce3 in PyObject_Call (func=0x7ffff7f9efc8, arg=<value optimized out>, kw=<value optimized out>)
at Objects/abstract.c:2546
#11 0x00007ffff37c3603 in PyEval_CallObjectWithKeywords (func=0x7ffff7f9efc8, arg=0x7fffc8428ba8, kw=<value optimized out>)
at Python/ceval.c:4219
#12 0x00007ffff37c826e in PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2622
#13 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffc82cf830, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#14 0x00007ffff37cdb12 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>)
at Python/ceval.c:669
#15 0x00007ffff37df932 in PyImport_ExecCodeModuleEx (name=0x7fffeaebf360 "xml.parsers.expat", co=0x7fffc82cf830,
pathname=0x7fffeae65330 "/srv/VENVs/mybottle/lib/python2.7/site-packages/_xmlplus/parsers/expat.pyc") at Python/import.c:713
#16 0x00007ffff37e207e in load_source_module (name=0x7fffeaebf360 "xml.parsers.expat",
pathname=0x7fffeae65330 "/srv/VENVs/mybottle/lib/python2.7/site-packages/_xmlplus/parsers/expat.pyc", fp=<value optimized out>)
at Python/import.c:1103
#17 0x00007ffff37e2e31 in import_submodule (mod=0x7fffc817fb08, subname=0x7fffeaebf36c "expat", fullname=0x7fffeaebf360 "xml.parsers.expat")
at Python/import.c:2704
#18 0x00007ffff37e30a4 in load_next (mod=0x7fffc817fb08, altmod=0x7fffc817fb08, p_name=<value optimized out>,
buf=0x7fffeaebf360 "xml.parsers.expat", p_buflen=0x7fffd8fe3830) at Python/import.c:2519
#19 0x00007ffff37e3710 in import_module_level (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7ffff3a80a50, level=<value optimized out>) at Python/import.c:2236
#20 PyImport_ImportModuleLevel (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7ffff3a80a50, level=<value optimized out>) at Python/import.c:2292
#21 0x00007ffff37c311f in builtin___import__ (self=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>)
at Python/bltinmodule.c:49
#22 0x00007ffff3719ce3 in PyObject_Call (func=0x7ffff7f9efc8, arg=<value optimized out>, kw=<value optimized out>)
at Objects/abstract.c:2546
#23 0x00007ffff37c3603 in PyEval_CallObjectWithKeywords (func=0x7ffff7f9efc8, arg=0x7fffc8166e10, kw=<value optimized out>)
at Python/ceval.c:4219
#24 0x00007ffff37c826e in PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2622
#25 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffc8429c30, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#26 0x00007ffff37cdb12 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>)
at Python/ceval.c:669
#27 0x00007ffff37df932 in PyImport_ExecCodeModuleEx (name=0x7fffeae7a980 "flight.xmlparser", co=0x7fffc8429c30,
---Type <return> to continue, or q <return> to quit---
pathname=0x7fffeaf39a40 "/srv/sources/mybottle/waterbottle/flight/xmlparser.pyc") at Python/import.c:713
#28 0x00007ffff37e207e in load_source_module (name=0x7fffeae7a980 "flight.xmlparser",
pathname=0x7fffeaf39a40 "/srv/sources/mybottle/waterbottle/flight/xmlparser.pyc", fp=<value optimized out>) at Python/import.c:1103
#29 0x00007ffff37e2e31 in import_submodule (mod=0x7fffcca7dfa0, subname=0x7fffeae7a987 "xmlparser",
fullname=0x7fffeae7a980 "flight.xmlparser") at Python/import.c:2704
#30 0x00007ffff37e30a4 in load_next (mod=0x7fffcca7dfa0, altmod=0x7ffff3a80a50, p_name=<value optimized out>,
buf=0x7fffeae7a980 "flight.xmlparser", p_buflen=0x7fffd8fe3dd0) at Python/import.c:2519
#31 0x00007ffff37e36d0 in import_module_level (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7ffff3a80a50, level=<value optimized out>) at Python/import.c:2228
#32 PyImport_ImportModuleLevel (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7ffff3a80a50, level=<value optimized out>) at Python/import.c:2292
#33 0x00007ffff37c311f in builtin___import__ (self=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>)
at Python/bltinmodule.c:49
#34 0x00007ffff3719ce3 in PyObject_Call (func=0x7ffff7f9efc8, arg=<value optimized out>, kw=<value optimized out>)
at Objects/abstract.c:2546
#35 0x00007ffff37c3603 in PyEval_CallObjectWithKeywords (func=0x7ffff7f9efc8, arg=0x7fffc86758e8, kw=<value optimized out>)
at Python/ceval.c:4219
#36 0x00007ffff37c826e in PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2622
#37 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffc86790b0, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#38 0x00007ffff37cdb12 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>)
at Python/ceval.c:669
#39 0x00007ffff37df932 in PyImport_ExecCodeModuleEx (name=0x7fffeae6c680 "flight.via", co=0x7fffc86790b0,
pathname=0x7fffeae5cb30 "/srv/sources/mybottle/waterbottle/flight/via.pyc") at Python/import.c:713
#40 0x00007ffff37e207e in load_source_module (name=0x7fffeae6c680 "flight.via",
pathname=0x7fffeae5cb30 "/srv/sources/mybottle/waterbottle/flight/via.pyc", fp=<value optimized out>) at Python/import.c:1103
#41 0x00007ffff37e2e31 in import_submodule (mod=0x7fffcca7dfa0, subname=0x7fffeae6c687 "via", fullname=0x7fffeae6c680 "flight.via")
at Python/import.c:2704
#42 0x00007ffff37e30a4 in load_next (mod=0x7fffcca7dfa0, altmod=0x7ffff3a80a50, p_name=<value optimized out>,
buf=0x7fffeae6c680 "flight.via", p_buflen=0x7fffd8fe4370) at Python/import.c:2519
#43 0x00007ffff37e36d0 in import_module_level (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc86ba550, level=<value optimized out>) at Python/import.c:2228
#44 PyImport_ImportModuleLevel (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc86ba550, level=<value optimized out>) at Python/import.c:2292
#45 0x00007ffff37c311f in builtin___import__ (self=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>)
at Python/bltinmodule.c:49
#46 0x00007ffff3719ce3 in PyObject_Call (func=0x7ffff7f9efc8, arg=<value optimized out>, kw=<value optimized out>)
at Objects/abstract.c:2546
#47 0x00007ffff37c3603 in PyEval_CallObjectWithKeywords (func=0x7ffff7f9efc8, arg=0x7fffc86c6260, kw=<value optimized out>)
at Python/ceval.c:4219
#48 0x00007ffff37c826e in PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2622
#49 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffc86c5eb0, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#50 0x00007ffff37cdb12 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>)
at Python/ceval.c:669
#51 0x00007ffff37df932 in PyImport_ExecCodeModuleEx (name=0x7fffeae512b0 "flight.controller", co=0x7fffc86c5eb0,
pathname=0x7fffeae532d0 "/srv/sources/mybottle/waterbottle/flight/controller.pyc") at Python/import.c:713
#52 0x00007ffff37e207e in load_source_module (name=0x7fffeae512b0 "flight.controller",
pathname=0x7fffeae532d0 "/srv/sources/mybottle/waterbottle/flight/controller.pyc", fp=<value optimized out>) at Python/import.c:1103
#53 0x00007ffff37e2e31 in import_submodule (mod=0x7fffcca7dfa0, subname=0x7fffeae512b7 "controller",
fullname=0x7fffeae512b0 "flight.controller") at Python/import.c:2704
#54 0x00007ffff37e30a4 in load_next (mod=0x7fffcca7dfa0, altmod=0x7ffff3a80a50, p_name=<value optimized out>,
---Type <return> to continue, or q <return> to quit---
buf=0x7fffeae512b0 "flight.controller", p_buflen=0x7fffd8fe4910) at Python/import.c:2519
#55 0x00007ffff37e36d0 in import_module_level (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc89a8890, level=<value optimized out>) at Python/import.c:2228
#56 PyImport_ImportModuleLevel (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc89a8890, level=<value optimized out>) at Python/import.c:2292
#57 0x00007ffff37c311f in builtin___import__ (self=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>)
at Python/bltinmodule.c:49
#58 0x00007ffff3719ce3 in PyObject_Call (func=0x7ffff7f9efc8, arg=<value optimized out>, kw=<value optimized out>)
at Objects/abstract.c:2546
#59 0x00007ffff37c3603 in PyEval_CallObjectWithKeywords (func=0x7ffff7f9efc8, arg=0x7fffc86abd60, kw=<value optimized out>)
at Python/ceval.c:4219
#60 0x00007ffff37c826e in PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2622
#61 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffc86bc030, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#62 0x00007ffff37cdb12 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>)
at Python/ceval.c:669
#63 0x00007ffff37df932 in PyImport_ExecCodeModuleEx (name=0x7fffeada21c0 "flight.flight_default_addons", co=0x7fffc86bc030,
pathname=0x7fffeada41e0 "/srv/sources/mybottle/waterbottle/flight/flight_default_addons.pyc") at Python/import.c:713
#64 0x00007ffff37e207e in load_source_module (name=0x7fffeada21c0 "flight.flight_default_addons",
pathname=0x7fffeada41e0 "/srv/sources/mybottle/waterbottle/flight/flight_default_addons.pyc", fp=<value optimized out>)
at Python/import.c:1103
#65 0x00007ffff37e2e31 in import_submodule (mod=0x7fffcca7dfa0, subname=0x7fffeada21c7 "flight_default_addons",
fullname=0x7fffeada21c0 "flight.flight_default_addons") at Python/import.c:2704
#66 0x00007ffff37e30a4 in load_next (mod=0x7fffcca7dfa0, altmod=0x7fffcca7dfa0, p_name=<value optimized out>,
buf=0x7fffeada21c0 "flight.flight_default_addons", p_buflen=0x7fffd8fe4eb0) at Python/import.c:2519
#67 0x00007ffff37e3710 in import_module_level (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc8a779d0, level=<value optimized out>) at Python/import.c:2236
#68 PyImport_ImportModuleLevel (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc8a779d0, level=<value optimized out>) at Python/import.c:2292
#69 0x00007ffff37c311f in builtin___import__ (self=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>)
at Python/bltinmodule.c:49
#70 0x00007ffff3719ce3 in PyObject_Call (func=0x7ffff7f9efc8, arg=<value optimized out>, kw=<value optimized out>)
at Objects/abstract.c:2546
#71 0x00007ffff37c3603 in PyEval_CallObjectWithKeywords (func=0x7ffff7f9efc8, arg=0x7fffc8a23a48, kw=<value optimized out>)
at Python/ceval.c:4219
#72 0x00007ffff37c826e in PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2622
#73 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffc89a9130, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#74 0x00007ffff37cdb12 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>)
at Python/ceval.c:669
#75 0x00007ffff37df932 in PyImport_ExecCodeModuleEx (name=0x7fffead67060 "flight.views", co=0x7fffc89a9130,
pathname=0x7fffead647e0 "/srv/sources/mybottle/waterbottle/flight/views.pyc") at Python/import.c:713
#76 0x00007ffff37e207e in load_source_module (name=0x7fffead67060 "flight.views",
pathname=0x7fffead647e0 "/srv/sources/mybottle/waterbottle/flight/views.pyc", fp=<value optimized out>) at Python/import.c:1103
#77 0x00007ffff37e2e31 in import_submodule (mod=0x7fffcca7dfa0, subname=0x7fffead67067 "views", fullname=0x7fffead67060 "flight.views")
at Python/import.c:2704
#78 0x00007ffff37e30a4 in load_next (mod=0x7fffcca7dfa0, altmod=0x7fffcca7dfa0, p_name=<value optimized out>,
buf=0x7fffead67060 "flight.views", p_buflen=0x7fffd8fe5450) at Python/import.c:2519
#79 0x00007ffff37e3710 in import_module_level (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc993cc00, level=<value optimized out>) at Python/import.c:2236
#80 PyImport_ImportModuleLevel (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x7fffc993cc00, level=<value optimized out>) at Python/import.c:2292
---Type <return> to continue, or q <return> to quit---
#81 0x00007ffff37c311f in builtin___import__ (self=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>)
at Python/bltinmodule.c:49
#82 0x00007ffff3719ce3 in PyObject_Call (func=0x7ffff7f9efc8, arg=<value optimized out>, kw=<value optimized out>)
at Objects/abstract.c:2546
#83 0x00007ffff37c3603 in PyEval_CallObjectWithKeywords (func=0x7ffff7f9efc8, arg=0x7fffc8b07c00, kw=<value optimized out>)
at Python/ceval.c:4219
#84 0x00007ffff37c826e in PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2622
#85 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffc8a6ab30, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#86 0x00007ffff37cdb12 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>)
at Python/ceval.c:669
#87 0x00007ffff37df932 in PyImport_ExecCodeModuleEx (name=0x7fffe980c780 "flight.admin", co=0x7fffc8a6ab30,
pathname=0x7fffead37a60 "/srv/sources/mybottle/waterbottle/flight/admin.pyc") at Python/import.c:713
#88 0x00007ffff37e207e in load_source_module (name=0x7fffe980c780 "flight.admin",
pathname=0x7fffead37a60 "/srv/sources/mybottle/waterbottle/flight/admin.pyc", fp=<value optimized out>) at Python/import.c:1103
#89 0x00007ffff37e2e31 in import_submodule (mod=0x7fffcca7dfa0, subname=0x7fffe980c787 "admin", fullname=0x7fffe980c780 "flight.admin")
at Python/import.c:2704
#90 0x00007ffff37e30a4 in load_next (mod=0x7fffcca7dfa0, altmod=0x7fffcca7dfa0, p_name=<value optimized out>,
buf=0x7fffe980c780 "flight.admin", p_buflen=0x7fffd8fe59f0) at Python/import.c:2519
#91 0x00007ffff37e3710 in import_module_level (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>,
fromlist=0x0, level=<value optimized out>) at Python/import.c:2236
#92 PyImport_ImportModuleLevel (name=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, fromlist=0x0,
level=<value optimized out>) at Python/import.c:2292
#93 0x00007ffff37c311f in builtin___import__ (self=<value optimized out>, args=<value optimized out>, kwds=<value optimized out>)
at Python/bltinmodule.c:49
#94 0x00007ffff37ccde5 in call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4350
#95 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2987
#96 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffd4163ab0, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=1, kws=0x7fffeacff380, kwcount=0, defs=0x7fffd4161568, defcount=1, closure=0x0)
at Python/ceval.c:3582
#97 0x00007ffff37cca25 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4446
#98 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4371
#99 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2987
#100 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffd417e9b0, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=1, kws=0x7fffc92daa98, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#101 0x00007ffff37cca25 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4446
#102 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4371
#103 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2987
#104 0x00007ffff37ccb65 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4436
#105 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4371
#106 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2987
#107 0x00007ffff37ccb65 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4436
#108 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4371
#109 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2987
#110 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7fffd40254b0, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=2, kws=0x7fffceac6cd0, kwcount=0, defs=0x7fffd402b4a8, defcount=1, closure=0x0)
at Python/ceval.c:3582
#111 0x00007ffff37cca25 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4446
#112 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4371
#113 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2987
#114 0x00007ffff37ccb65 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4436
#115 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4371
---Type <return> to continue, or q <return> to quit---
#116 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2987
#117 0x00007ffff37ccb65 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4436
#118 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4371
#119 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2987
#120 0x00007ffff37cd9fe in PyEval_EvalCodeEx (co=0x7ffff7eb7930, globals=<value optimized out>, locals=<value optimized out>,
args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3582
#121 0x00007ffff37cdb12 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>)
at Python/ceval.c:669
#122 0x00007ffff37df932 in PyImport_ExecCodeModuleEx (name=0x7fffe803fc98 "_mod_wsgi_d63c64b161f8c1e55025830e443670e9", co=0x7ffff7eb7930,
pathname=0x7fffe803c530 "/srv/sources/mybottle/waterbottle/apache/mybottle.wsgi") at Python/import.c:713
#123 0x00007ffff3acf16f in wsgi_load_source (pool=0x7fffe803ab98, r=0x7fffe803ac10,
name=0x7fffe803fc98 "_mod_wsgi_d63c64b161f8c1e55025830e443670e9", exists=<value optimized out>,
filename=0x7fffe803c530 "/srv/sources/mybottle/waterbottle/apache/mybottle.wsgi", process_group=0x7ffff3ae46ac "",
application_group=0x7fffe803efc0 "local.mybottlebusiness.com|") at src/server/mod_wsgi.c:3602
#124 0x00007ffff3ad76c3 in wsgi_execute_script (r=0x7fffe803ac10) at src/server/mod_wsgi.c:3966
#125 0x00007ffff3ae1f19 in wsgi_hook_handler (r=<value optimized out>) at src/server/mod_wsgi.c:7025
#126 0x000000000044daf0 in ap_run_handler (r=0x7fffe803ac10) at config.c:171
#127 0x0000000000451c6e in ap_invoke_handler (r=0x7fffe803ac10) at config.c:434
#128 0x000000000046530a in ap_process_async_request (r=0x7fffe803ac10) at http_request.c:410
#129 0x000000000046546f in ap_process_request (r=0x7fffe803ac10) at http_request.c:445
#130 0x00000000004615ee in ap_process_http_sync_connection (c=0x7fffd0000c48) at http_core.c:210
#131 ap_process_http_connection (c=0x7fffd0000c48) at http_core.c:251
#132 0x0000000000458a50 in ap_run_process_connection (c=0x7fffd0000c48) at connection.c:42
#133 0x000000000046c118 in process_socket (thd=0x6d0f40, dummy=<value optimized out>) at worker.c:631
#134 worker_thread (thd=0x6d0f40, dummy=<value optimized out>) at worker.c:990
#135 0x00007ffff6edcaa1 in start_thread () from /lib64/libpthread.so.0
#136 0x00007ffff6c2993d in clone () from /lib64/libc.so.6
(gdb)
(gdb) quit
A debugging session is active.
Inferior 1 [process 14657] will be killed.
Quit anyway? (y or n) n
Not confirmed.
(gdb) cont
Continuing.
[Thread 0x7fffd8fe7700 (LWP 14685) exited]
[Thread 0x7fffd99e8700 (LWP 14684) exited]
[Thread 0x7fffda3e9700 (LWP 14683) exited]
[Thread 0x7fffdadea700 (LWP 14682) exited]
[Thread 0x7fffdb7eb700 (LWP 14681) exited]
[Thread 0x7fffdc1ec700 (LWP 14680) exited]
[Thread 0x7fffdcbed700 (LWP 14679) exited]
[Thread 0x7fffdd5ee700 (LWP 14678) exited]
[Thread 0x7fffddfef700 (LWP 14677) exited]
[Thread 0x7fffde9f0700 (LWP 14676) exited]
[Thread 0x7fffdf3f1700 (LWP 14675) exited]
[Thread 0x7fffdfdf2700 (LWP 14674) exited]
[Thread 0x7fffe07f3700 (LWP 14673) exited]
[Thread 0x7fffe11f4700 (LWP 14672) exited]
[Thread 0x7fffe1bf5700 (LWP 14671) exited]
[Thread 0x7fffe25f6700 (LWP 14670) exited]
[Thread 0x7fffe2ff7700 (LWP 14669) exited]
[Thread 0x7fffe39f8700 (LWP 14668) exited]
[Thread 0x7fffe43f9700 (LWP 14667) exited]
[Thread 0x7fffe4dfa700 (LWP 14666) exited]
[Thread 0x7fffe57fb700 (LWP 14665) exited]
[Thread 0x7fffe61fc700 (LWP 14664) exited]
[Thread 0x7fffe6bfd700 (LWP 14663) exited]
[Thread 0x7fffe75fe700 (LWP 14662) exited]
[Thread 0x7fffe7fff700 (LWP 14661) exited]
[Thread 0x7fffd85e6700 (LWP 14686) exited]
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb) quit
[vagrant@thebusinessdev bin]$
[vagrant@thebusinessdev bin]$
[vagrant@thebusinessdev bin]$ Yours sincerely,
Abraham V.
Graham Dumpleton
May 19, 2016, 7:57:25 PM5/19/16
to mod...@googlegroups.com
Can you try some of the tests described in:
Has been a while since have seen expat library conflicts, plus you are saying to use the system expat library. Only cause for it to die inside of expat would be if PHP was also being used and it had dragged in an incompatible version of expat first.
Graham
--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
Abraham Varricatt
May 20, 2016, 8:51:54 AM5/20/16
to modwsgi
I'm a little surprised to hear that PHP might be an issue - this is a django/python app I'm trying to run. Nevertheless, I tried the tests mentioned on the link and think that I've made some kind of progress.
First off,
def application(environ, start_response):
status = '200 OK'
output = 'without expat\n'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]Worked for me when I used it in my wsgi file. I was able to see 'without expat' on my browser. Then I tried,
import pyexpat
def application(environ, start_response):
status = '200 OK'
output = 'with expat\n'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]in my wsgi file and got a 500 error. This gave me some hope to figure things out. The logs indicated a python error importing pyexpat. It looked like the library was not available. Investigation led me to suspect my python compilation. I changed things to the following, (earlier I passed the flags --with-system-ffi --with-system-expat to ./configure )
./configure --enable-shared --enable-unicode=ucs4 --prefix=$(pwd)/../COMPILED_PYTHON LDFLAGS="-L$(pwd)/../COMPILED_PYTHON/extlib/lib -Wl,--rpath=$(pwd)/../COMPILED_PYTHON/lib -Wl,--rpath=$(pwd)/../COMPILED_PYTHON/extlib/lib" CPPFLAGS="-I$(pwd)/../COMPILED_PYTHON/extlib/include"
make
make altinstallThe above is how I'm compiling Python 2.7.11 from the official sources. After this is created, I use it as the basis for a virtualenv created on a separate location which I point to from apache.
With the new python builds, I'm able to get 'with expat' on my browser, something I like to think of as progress. :)
The bad news, is that when I edit my wsgi file to point to my django app, the segmentation fault comes back.
At this point, I grew suspicious of the django app - could something be wrong with it? To investigate, I logged into my test server as the apache user, activated the same virtualenv that I point httpd/mod_wsgi to use, navigated to the sources folder and tried running django's 'runserver' command. To my befuddlement, it worked. Sure, the page had some CSS/JS rendering issues, but I did get back some HTML on my browser.
Looking over the backtrace, there are a lot of <value optimized out> statements. How might I get more debug info from this? Other suggestions also welcome.
Yours sincerely,
Abraham V.
Graham Dumpleton
May 20, 2016, 5:21:12 PM5/20/16
to mod...@googlegroups.com
On 20 May 2016, at 10:51 PM, 'Abraham Varricatt' via modwsgi <mod...@googlegroups.com> wrote:I'm a little surprised to hear that PHP might be an issue - this is a django/python app I'm trying to run. Nevertheless, I tried the tests mentioned on the link and think that I've made some kind of progress.First off,
status = '200 OK'
output = 'without expat\n'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]Worked for me when I used it in my wsgi file. I was able to see 'without expat' on my browser. Then I tried,
def application(environ, start_response):
status = '200 OK'
output = 'with expat\n'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]in my wsgi file and got a 500 error.
What was the full error messages from the Apache error logs about this?
Actual details is better to help work out what is going on.
This gave me some hope to figure things out. The logs indicated a python error importing pyexpat. It looked like the library was not available. Investigation led me to suspect my python compilation. I changed things to the following, (earlier I passed the flags --with-system-ffi --with-system-expat to ./configure )
make
make altinstallThe above is how I'm compiling Python 2.7.11 from the official sources. After this is created, I use it as the basis for a virtualenv created on a separate location which I point to from apache.With the new python builds, I'm able to get 'with expat' on my browser, something I like to think of as progress. :)The bad news, is that when I edit my wsgi file to point to my django app, the segmentation fault comes back.
You are possibly mixing Python installations when running mod_wsgi at this point if mod_wsgi wasn’t recompiled for new Python installation and WSGIPythonHome set to force use of Python in non standard location.
Abraham Varricatt
May 22, 2016, 1:56:41 PM5/22/16
to modwsgi
On Saturday, May 21, 2016 at 2:51:12 AM UTC+5:30, Graham Dumpleton wrote:
Actual details is better to help work out what is going on.
I'm probably going to sound like an idiot, but this got me stumped - I'm aware of the symptoms of my problem, but do not have enough understanding to figure out what might be relevant. After a lot of thinking (and experimentation) I came up with this,
It's a Virtualbox-based vagrant environment which replicates the issue (as far as I can tell). Here is they WSGI script which I'm using,
This configuration took me a lot of time, and while I'm not sure how to fix the segmentation fault, I think I've figured out what causes it. Specifically it is the PyXML module. For better or worse, I'm stuck maintaining a lot of code that depends on PyXML. The module isn't available on PyPi and is being installed from the archive. Here is a copy of it,
Why am I so sure the issue is with this module? Because if I load up my provisioning script without adding PyXML to my virtualenv, the wsgi page loads up perfectly fine! The moment I install PyXML, the segmentation fault comes back.
I'm certain that I'm not mixing up my Python installations, because I'm destroying my virtualbox instance between experiments and not setting it up by hand - all the setup is automate with ansible, including the compiling, configuration and installation.
It's late for me, but after I get back to work tomorrow morning, I'll post the apache logs I'm getting as well as the GDB stack-trace.
Yours,
Abraham V.
Graham Dumpleton
May 22, 2016, 9:06:29 PM5/22/16
to mod...@googlegroups.com
I would recommend not using embedded mode of mod_wsgi but use daemon mode. Delegate each application to a separate mod_wsgi daemon process group and then set the WSGIApplicationGroup directive to force the use of the main Python interpreter context. This avoids problems with Python extension modules which haven’t been implemented correctly to work in sub interpreters. The PyXML package is one such package.
Some reading:
Graham
Abraham Varricatt
May 23, 2016, 3:23:36 AM5/23/16
to modwsgi
On Monday, 23 May 2016 06:36:29 UTC+5:30, Graham Dumpleton wrote:
I would recommend not using embedded mode of mod_wsgi but use daemon mode. Delegate each application to a separate mod_wsgi daemon process group and then set the WSGIApplicationGroup directive to force the use of the main Python interpreter context. This avoids problems with Python extension modules which haven’t been implemented correctly to work in sub interpreters. The PyXML package is one such package.
Alright, I got daemon mode configured and have set WSGIApplicationGroup to use the main Python interpreter. Here is how the config looks like,
<VirtualHost *:80>
ServerAdmin y...@example.com
ServerName 192.168.33.10
ServerAlias 192.168.33.10
<Directory /srv/sources/bottle/myproject>
<Files bottle.wsgi>
Require all granted
</Files>
</Directory>
WSGIDaemonProcess bottle.debug processes=1 threads=1 display-name=%{GROUP}
WSGIProcessGroup bottle.debug
WSGIApplicationGroup %{GLOBAL}
WSGIScriptAlias / /srv/sources/bottle/myproject/bottle.wsgi
Alias /static/ /srv/STATIC_FILES/
<Directory /srv/STATIC_FILES/>
Require all granted
</Directory>
</VirtualHost>
Using the test WSGI scripts, I can confirm that the setup is running in daemon mode, with multi-threading and apache MPM as worker. Sadly, I'm getting segmentation faults in my apache logs when I install PyXML into my virtualenv. Here is the apache log when I access the site,
To make the gdb stack-trace a bit more useful, I recompiled my httpd server with debug flags. Here are the exact instructions I use,
[Mon May 23 06:52:21.026470 2016] [wsgi:debug] [pid 15745:tid 139682201560832] src/server/mod_wsgi.c(8070): mod_wsgi (pid=15745): Socket for 'bottle.debug' is '/var/run/wsgi.15745.0.1.sock'.
[Mon May 23 06:52:21.026637 2016] [wsgi:debug] [pid 15745:tid 139682201560832] src/server/mod_wsgi.c(8128): mod_wsgi (pid=15745): Listen backlog for socket '/var/run/wsgi.15745.0.1.sock' is '100'.
[Mon May 23 06:52:21.027182 2016] [mpm_worker:notice] [pid 15745:tid 139682201560832] AH00292: Apache/2.4.20 (Unix) mod_wsgi/4.5.2 Python/2.7.11 configured -- resuming normal operations
[Mon May 23 06:52:21.027208 2016] [mpm_worker:info] [pid 15745:tid 139682201560832] AH00293: Server built: May 23 2016 06:36:52
[Mon May 23 06:52:21.027236 2016] [core:notice] [pid 15745:tid 139682201560832] AH00094: Command line: '/usr/local/apache2/bin/httpd'
[Mon May 23 06:52:21.027250 2016] [core:debug] [pid 15745:tid 139682201560832] log.c(1546): AH02639: Using SO_REUSEPORT: yes (1)
[Mon May 23 06:52:21.027270 2016] [mpm_worker:debug] [pid 15745:tid 139682201560832] worker.c(1874): AH00294: Accept mutex: none (default: sysvsem)
[Mon May 23 06:52:21.027578 2016] [wsgi:info] [pid 15749:tid 139682201560832] mod_wsgi (pid=15749): Python home /srv/VENVs/bottle.
[Mon May 23 06:52:21.027626 2016] [wsgi:info] [pid 15749:tid 139682201560832] mod_wsgi (pid=15749): Initializing Python.
[Mon May 23 06:52:21.030045 2016] [wsgi:info] [pid 15748:tid 139682201560832] mod_wsgi (pid=15748): Python home /srv/VENVs/bottle.
[Mon May 23 06:52:21.030116 2016] [wsgi:info] [pid 15748:tid 139682201560832] mod_wsgi (pid=15748): Initializing Python.
[Mon May 23 06:52:21.037471 2016] [wsgi:info] [pid 15748:tid 139682201560832] mod_wsgi (pid=15748): Attach interpreter ''.
[Mon May 23 06:52:21.037805 2016] [wsgi:info] [pid 15747:tid 139682201560832] mod_wsgi (pid=15747): Python home /srv/VENVs/bottle.
[Mon May 23 06:52:21.037861 2016] [wsgi:info] [pid 15747:tid 139682201560832] mod_wsgi (pid=15747): Initializing Python.
[Mon May 23 06:52:21.040964 2016] [wsgi:info] [pid 15746:tid 139682201560832] mod_wsgi (pid=15746): Starting process 'bottle.debug' with uid=501, gid=501 and threads=1.
[Mon May 23 06:52:21.041235 2016] [wsgi:info] [pid 15746:tid 139682201560832] mod_wsgi (pid=15746): Python home /srv/VENVs/bottle.
[Mon May 23 06:52:21.041273 2016] [wsgi:info] [pid 15746:tid 139682201560832] mod_wsgi (pid=15746): Initializing Python.
[Mon May 23 06:52:21.048470 2016] [wsgi:info] [pid 15746:tid 139682201560832] mod_wsgi (pid=15746): Attach interpreter ''.
[Mon May 23 06:52:21.057295 2016] [wsgi:info] [pid 15749:tid 139682201560832] mod_wsgi (pid=15749): Attach interpreter ''.
[Mon May 23 06:52:21.063182 2016] [wsgi:info] [pid 15747:tid 139682201560832] mod_wsgi (pid=15747): Attach interpreter ''.
[Mon May 23 06:52:21.065709 2016] [wsgi:debug] [pid 15746:tid 139682096916224] src/server/mod_wsgi.c(8681): mod_wsgi (pid=15746): Started thread 0 in daemon process 'bottle.debug'.
[Mon May 23 06:52:33.285940 2016] [authz_core:debug] [pid 15749:tid 139682107406080] mod_authz_core.c(809): [client 192.168.33.1:55097] AH01626: authorization result of Require all granted: granted
[Mon May 23 06:52:33.286070 2016] [authz_core:debug] [pid 15749:tid 139682107406080] mod_authz_core.c(809): [client 192.168.33.1:55097] AH01626: authorization result of <RequireAny>: granted
[Mon May 23 06:52:33.290302 2016] [authz_core:debug] [pid 15749:tid 139682107406080] mod_authz_core.c(809): [client 192.168.33.1:55097] AH01626: authorization result of Require all granted: granted
[Mon May 23 06:52:33.290412 2016] [authz_core:debug] [pid 15749:tid 139682107406080] mod_authz_core.c(809): [client 192.168.33.1:55097] AH01626: authorization result of <RequireAny>: granted
[Mon May 23 06:52:33.292140 2016] [wsgi:info] [pid 15746:tid 139682096916224] [remote 192.168.33.1:50245] mod_wsgi (pid=15746, process='bottle.debug', application=''): Loading WSGI script '/srv/sources/bottle/myproject/bottle.wsgi'.
[Mon May 23 06:52:33.508113 2016] [wsgi:error] [pid 15749:tid 139682107406080] [client 192.168.33.1:55097] Truncated or oversized response headers received from daemon process 'bottle.debug': /srv/sources/bottle/myproject/bottle.wsgi
[Mon May 23 06:52:33.635001 2016] [authz_core:debug] [pid 15748:tid 139682107406080] mod_authz_core.c(809): [client 192.168.33.1:55098] AH01626: authorization result of Require all granted: granted, referer: http://192.168.33.10/
[Mon May 23 06:52:33.635069 2016] [authz_core:debug] [pid 15748:tid 139682107406080] mod_authz_core.c(809): [client 192.168.33.1:55098] AH01626: authorization result of <RequireAny>: granted, referer: http://192.168.33.10/
[Mon May 23 06:52:33.637438 2016] [authz_core:debug] [pid 15748:tid 139682107406080] mod_authz_core.c(809): [client 192.168.33.1:55098] AH01626: authorization result of Require all granted: granted, referer: http://192.168.33.10/
[Mon May 23 06:52:33.637737 2016] [authz_core:debug] [pid 15748:tid 139682107406080] mod_authz_core.c(809): [client 192.168.33.1:55098] AH01626: authorization result of <RequireAny>: granted, referer: http://192.168.33.10/
[Mon May 23 06:52:34.043628 2016] [core:notice] [pid 15745:tid 139682201560832] AH00052: child pid 15746 exit signal Segmentation fault (11)
[Mon May 23 06:52:34.043714 2016] [wsgi:info] [pid 15745:tid 139682201560832] mod_wsgi (pid=15746): Process 'bottle.debug' has died, deregister and restart it.
[Mon May 23 06:52:34.043728 2016] [wsgi:info] [pid 15745:tid 139682201560832] mod_wsgi (pid=15746): Process 'bottle.debug' terminated by signal 11
[Mon May 23 06:52:34.043743 2016] [wsgi:info] [pid 15745:tid 139682201560832] mod_wsgi (pid=15746): Process 'bottle.debug' has been deregistered and will no longer be monitored.
[Mon May 23 06:52:34.044279 2016] [wsgi:info] [pid 15837:tid 139682201560832] mod_wsgi (pid=15837): Starting process 'bottle.debug' with uid=501, gid=501 and threads=1.
[Mon May 23 06:52:34.044742 2016] [wsgi:info] [pid 15837:tid 139682201560832] mod_wsgi (pid=15837): Python home /srv/VENVs/bottle.
[Mon May 23 06:52:34.044807 2016] [wsgi:info] [pid 15837:tid 139682201560832] mod_wsgi (pid=15837): Initializing Python.
[Mon May 23 06:52:34.045346 2016] [wsgi:info] [pid 15836:tid 139682201560832] mod_wsgi (pid=15836): Python home /srv/VENVs/bottle.
[Mon May 23 06:52:34.051369 2016] [wsgi:info] [pid 15836:tid 139682201560832] mod_wsgi (pid=15836): Initializing Python.
[Mon May 23 06:52:34.066996 2016] [wsgi:info] [pid 15837:tid 139682201560832] mod_wsgi (pid=15837): Attach interpreter ''.
[Mon May 23 06:52:34.071135 2016] [wsgi:debug] [pid 15837:tid 139682096916224] src/server/mod_wsgi.c(8681): mod_wsgi (pid=15837): Started thread 0 in daemon process 'bottle.debug'.
[Mon May 23 06:52:34.072572 2016] [wsgi:info] [pid 15836:tid 139682201560832] mod_wsgi (pid=15836): Attach interpreter ''.
[Mon May 23 06:52:34.074336 2016] [wsgi:info] [pid 15837:tid 139682096916224] [remote 192.168.33.1:50245] mod_wsgi (pid=15837, process='bottle.debug', application=''): Loading WSGI script '/srv/sources/bottle/myproject/bottle.wsgi'.
[Mon May 23 06:52:34.275683 2016] [wsgi:error] [pid 15748:tid 139682107406080] [client 192.168.33.1:55098] Truncated or oversized response headers received from daemon process 'bottle.debug': /srv/sources/bottle/myproject/bottle.wsgi, referer: http://192.168.33.10/
[Mon May 23 06:52:35.045087 2016] [core:notice] [pid 15745:tid 139682201560832] AH00052: child pid 15837 exit signal Segmentation fault (11)
[Mon May 23 06:52:35.045161 2016] [wsgi:info] [pid 15745:tid 139682201560832] mod_wsgi (pid=15837): Process 'bottle.debug' has died, deregister and restart it.
[Mon May 23 06:52:35.045174 2016] [wsgi:info] [pid 15745:tid 139682201560832] mod_wsgi (pid=15837): Process 'bottle.debug' terminated by signal 11
[Mon May 23 06:52:35.045184 2016] [wsgi:info] [pid 15745:tid 139682201560832] mod_wsgi (pid=15837): Process 'bottle.debug' has been deregistered and will no longer be monitored.
[Mon May 23 06:52:35.045819 2016] [wsgi:info] [pid 15870:tid 139682201560832] mod_wsgi (pid=15870): Starting process 'bottle.debug' with uid=501, gid=501 and threads=1.
[Mon May 23 06:52:35.046278 2016] [wsgi:info] [pid 15870:tid 139682201560832] mod_wsgi (pid=15870): Python home /srv/VENVs/bottle.
[Mon May 23 06:52:35.046343 2016] [wsgi:info] [pid 15870:tid 139682201560832] mod_wsgi (pid=15870): Initializing Python.
[Mon May 23 06:52:35.059924 2016] [wsgi:info] [pid 15870:tid 139682201560832] mod_wsgi (pid=15870): Attach interpreter ''.
[Mon May 23 06:52:35.060263 2016] [wsgi:debug] [pid 15870:tid 139682096916224] src/server/mod_wsgi.c(8681): mod_wsgi (pid=15870): Started thread 0 in daemon process 'bottle.debug'.
$ cd /opt/oss_sources/httpd-2.4.20./configure --enable-so --with-included-apr --enable-mods-shared=reallyall --with-mpm=worker --enable-debugger-mode
make
sudo make install
$ cd /opt/oss_sources/mod_wsgi-4.5.2./configure --with-python=/opt/oss_sources/COMPILED_PYTHON/bin/python2.7 LDFLAGS="-Wl,-rpath=/opt/oss_sources/COMPILED_PYTHON/lib" --with-apxs=/usr/local/apache2/bin/apxs CFLAGS="-g"make
sudo make install
And finally here is the stack trace. I hope captured it correctly?
If you would like to replicate my environment, you can clone the following repo,,
https://github.com/abrahamvarricatt/modwsgi_env_setup
and run,
$ vagrant up
This will create a virtualbox image, and provision it to be exactly as my setup. You can then run,
$ vagrant ssh
to login to the system. The host dependencies for this to work are - vagrant, ansible, virtualbox
Yours sincerely,
Abraham V.
Graham Dumpleton
May 23, 2016, 6:27:39 AM5/23/16
to mod...@googlegroups.com
What do you get when you run:
ldd /srv/VENVs/bottle/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so
and:
ldd /usr/local/apache2/bin/httpd
Graham
Abraham Varricatt
May 23, 2016, 12:05:50 PM5/23/16
to modwsgi
Some good news! I got the system to work! More by accident than deliberate problem-solving. But before I get to the 'fix', let me answer your questions,
What do you get when you run:ldd /srv/VENVs/bottle/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.soand:ldd /usr/local/apache2/bin/httpd
[vagrant@bottle httpd]$ ldd /srv/VENVs/bottle/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so
linux-vdso.so.1 => (0x00007ffc723f1000)
libpython2.7.so.1.0 => /opt/oss_sources/Python-2.7.11/../COMPILED_PYTHON/lib/libpython2.7.so.1.0 (0x00007f05298ae000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f052968b000)
libc.so.6 => /lib64/libc.so.6 (0x00007f05292f7000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f05290f3000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f0528eef000)
libm.so.6 => /lib64/libm.so.6 (0x00007f0528c6b000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0529ecd000)
[vagrant@bottle httpd]$
[vagrant@bottle httpd]$ ldd /usr/local/apache2/bin/httpd
linux-vdso.so.1 => (0x00007fff00720000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fe6871d2000)
libaprutil-1.so.0 => /usr/local/apache2/lib/libaprutil-1.so.0 (0x00007fe686fac000)
libexpat.so.0 => /usr/local/apache2/lib/libexpat.so.0 (0x00007fe686d85000)
libapr-1.so.0 => /usr/local/apache2/lib/libapr-1.so.0 (0x00007fe686b52000)
librt.so.1 => /lib64/librt.so.1 (0x00007fe686949000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fe686712000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe6864f5000)
libc.so.6 => /lib64/libc.so.6 (0x00007fe686160000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe687405000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007fe685f5d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fe685d58000)
[vagrant@bottle httpd]$An overly simplified view of my provisioning script is as follows,
1. Update centos6 packages (this is NOT upgrading to centos7, just individual package updates)
2. Download python2.7 sources, compile and install
3. Download httpd sources, compile and install
4. Download mod_wsgi sources, compile and install
5. Clone my project repository to local file-system
6. Copy over different config files and restart httpd service
What you need to realize is that none of the above steps displayed any error message to me. They all seemed to flow just fine. Today afternoon, in an effort to get more debugging information, I updated the provisioning steps to compile debug versions of ALL the OSS sources I use - python, apache and mod_wsgi. Also included the expat-devel package from the yum repositories. At first I was shocked to discover that the debug builds did not produce any segmentation faults. Eventually I narrowed down the cause to the missing expat-devel package.
Graham, I'm really grateful to you for helping me out. If you hadn't commented about something funny in expat, I would have never figured this out. Thank you! :)
On a different note, any idea what could explain this behavior? My experience with *nix-based utilities is that if you don't have an underlying dependency installed, the utility will throw an error or attempt to resolve the dependency itself. This is the first time I'm seeing something proceed without issues through both compilation and installation, but only to fail at run-time.
Puzzled,
Abraham V.
Graham Dumpleton
May 23, 2016, 7:28:46 PM5/23/16
to mod...@googlegroups.com
Because you compile Apache from source code, there was nothing to trigger installation of expat-devel. The result was that Apache build couldn’t find it and so used its own internal copy.
libexpat.so.0 => /usr/local/apache2/lib/libexpat.so.0 (0x00007fe686d85000)
When pyexpat.so was built, it also used its own copy, but the problem may be that when using its own copy, it doesn’t namespace prefix symbols so they will not clash with a proper libexpat.so linked in by something else.
This used to be an issue with the expat module in Python itself until they started namespace prefixing their own version. I don’t understand why the one in the libXML package hasn’t learnt from that.
At least that appears to be what occurred. After you install expat-devel and rebuild everything, does pyexpat.so then link to system libexpat.so?
Graham
Puzzled,Abraham V.
Abraham Varricatt
May 24, 2016, 12:57:28 PM5/24/16
to modwsgi
Here's the output after building when expat-devel is installed,
Apache's expat is now pointing to the system one.
[vagrant@bottle ~]$ ldd /srv/VENVs/bottle/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so linux-vdso.so.1 => (0x00007ffe57da5000) libpython2.7.so.1.0 => /opt/oss_sources/Python-2.7.11/../COMPILED_PYTHON/lib/libpython2.7.so.1.0 (0x00007f2b29f7d000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2b29d5a000) libc.so.6 => /lib64/libc.so.6 (0x00007f2b299c6000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f2b297c2000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f2b295be000) libm.so.6 => /lib64/libm.so.6 (0x00007f2b2933a000) /lib64/ld-linux-x86-64.so.2 (0x00007f2b2a59c000)[vagrant@bottle ~]$ ldd /usr/local/apache2/bin/httpd linux-vdso.so.1 => (0x00007ffd4bb1f000) libpcre.so.0 => /lib64/libpcre.so.0 (0x00007f1ac6657000) libaprutil-1.so.0 => /usr/local/apache2/lib/libaprutil-1.so.0 (0x00007f1ac6431000) libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f1ac6209000) libapr-1.so.0 => /usr/local/apache2/lib/libapr-1.so.0 (0x00007f1ac5fd6000) librt.so.1 => /lib64/librt.so.1 (0x00007f1ac5dcd000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f1ac5b96000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1ac5979000) libc.so.6 => /lib64/libc.so.6 (0x00007f1ac55e4000) /lib64/ld-linux-x86-64.so.2 (0x00007f1ac688a000) libfreebl3.so => /lib64/libfreebl3.so (0x00007f1ac53e1000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f1ac51dc000)[vagrant@bottle ~]$
Apache's expat is now pointing to the system one.
Yours,
Abraham V.
Graham Dumpleton
May 24, 2016, 11:35:09 PM5/24/16
to mod...@googlegroups.com
Hmm, I am not sure what was happening before then since there is no libexpat.so dependency in pyexpat.so. That actually seems wrong to me.
The only last thing I can think of is that pyexpat.so was compiled against system header definitions but then didn’t actually link the libexpat library as it should and was relying on application it is loaded in linking it. When Apache brought in its own libexpat and it was different, it crashed. Now Apache is using system one, is okay.
Important thing is that it is working.
Reply all
Reply to author
Forward
0 new messages