Segmentation fault caused by libcrypto?
Alex Clark
I have an Apache buildout in which I recently upgraded openssl (to fix
another unrelated issue).
After the upgrade, I noticed mod_wsgi was seg faulting Apache (when
running Trac).
Many hairs pulled later, and after much double checking of the basics, I
fired up gdb and got this:
0x00007f9435d47c90 in EVP_PKEY_CTX_dup () from
/srv/apache/parts/openssl/lib/libcrypto.so.1.0.0
Anyone know what's going on here? Seems like I may have run across some
incompat between the newer openssl and mod_wsgi, but I have no idea.
Here's the full traceback:
----
aclark@orlok:/srv/apache/ > sudo su
orlok:/srv/apache# set -o vi
orlok:/srv/apache# export LD_LIBRARY_PATH=/srv/apache/parts/openssl/lib
orlok:/srv/apache# gdb parts/apache/bin/httpd
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(gdb) run -f /srv/apache/etc/httpd.conf -X
Starting program: /srv/apache/parts/apache/bin/httpd -f
/srv/apache/etc/httpd.conf -X
[Thread debugging using libthread_db enabled]
[New Thread 0x7fa1cc4a7730 (LWP 30894)]
[Wed Feb 23 18:25:24 2011] [notice] mod_transform: Cached Precompiled
XSL: /blog.xsl
[Wed Feb 23 18:25:24 2011] [notice] mod_transform: Cached Precompiled
XSL: /blog.xsl
[Wed Feb 23 18:25:24 2011] [notice] mod_transform: Cached Precompiled
XSL: /trac.xsl
[Wed Feb 23 18:25:24 2011] [notice] mod_transform: Cached Precompiled
XSL: /trac-support.xsl
httpd: Could not reliably determine the server's fully qualified domain
name, using 173.203.222.165 for ServerName
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fa1cc4a7730 (LWP 30894)]
0x00007fa1c3c17c90 in EVP_PKEY_CTX_dup () from
/srv/apache/parts/openssl/lib/libcrypto.so.1.0.0
Current language: auto; currently asm
(gdb) where
#0 0x00007fa1c3c17c90 in EVP_PKEY_CTX_dup () from
/srv/apache/parts/openssl/lib/libcrypto.so.1.0.0
#1 0x00007fa1c3c0b74f in EVP_MD_CTX_copy_ex () from
/srv/apache/parts/openssl/lib/libcrypto.so.1.0.0
#2 0x00007fa1ba93b916 in EVPnew (name_obj=0x10fee10, digest=<value
optimized out>, initial_ctx=0x7fa1bab3dba0, cp=0x0, len=0)
at /home/aclark/Python-2.6.6/Modules/_hashopenssl.c:365
#3 0x00007fa1ba93bc66 in EVP_new_md5 (self=<value optimized out>,
args=<value optimized out>)
at /home/aclark/Python-2.6.6/Modules/_hashopenssl.c:464
#4 0x00007fa1be3a036d in PyEval_EvalFrameEx (f=0xda0890,
throwflag=<value optimized out>) at Python/ceval.c:3750
#5 0x00007fa1be3a20b9 in PyEval_EvalCodeEx (co=0x11054e0,
globals=<value optimized out>, locals=<value optimized out>,
args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:3000
#6 0x00007fa1be3a2232 in PyEval_EvalCode (co=0x10fee40,
globals=0xd86fc4, locals=0x0) at Python/ceval.c:541
#7 0x00007fa1be3b4800 in PyImport_ExecCodeModuleEx (name=0x7fff9d59a71a
"hashlib", co=0x11054e0,
pathname=0x7fff9d598540 "/usr/local/lib/python2.6/hashlib.pyc") at
Python/import.c:677
#8 0x00007fa1be3b65d8 in load_source_module (name=0x7fff9d59a71a
"hashlib",
pathname=0x7fff9d598540 "/usr/local/lib/python2.6/hashlib.pyc",
fp=<value optimized out>) at Python/import.c:1017
#9 0x00007fa1be3b7413 in import_submodule (mod=0x7fa1be63c290,
subname=0x7fff9d59a71a "hashlib",
fullname=0x7fff9d59a71a "hashlib") at Python/import.c:2592
#10 0x00007fa1be3b772c in load_next (mod=0x1091910,
altmod=0x7fa1be63c290, p_name=<value optimized out>,
buf=0x7fff9d59a710 "trac.util.hashlib", p_buflen=0x7fff9d59b718) at
Python/import.c:2416
#11 0x00007fa1be3b7cdc in import_module_level (name=0x0,
globals=0xd81d90, locals=<value optimized out>, fromlist=0x10d4710,
level=-1) at Python/import.c:2133
#12 0x00007fa1be3b8265 in PyImport_ImportModuleLevel (name=0x10fe774
"hashlib", globals=0xd81d90, locals=0xd81d90,
fromlist=0x10d4710, level=-1) at Python/import.c:2185
#13 0x00007fa1be399d0d in builtin___import__ (self=<value optimized
out>, args=<value optimized out>,
kwds=<value optimized out>) at Python/bltinmodule.c:48
#14 0x00007fa1be2f8028 in PyObject_Call (func=0xcbe0e0, arg=0x1101208,
kw=0x0) at Objects/abstract.c:2492
#15 0x00007fa1be39acb6 in PyEval_CallObjectWithKeywords (func=0xcbe0e0,
arg=0x1101208, kw=0x0) at Python/ceval.c:3619
#16 0x00007fa1be39df13 in PyEval_EvalFrameEx (f=0x1079a80,
throwflag=<value optimized out>) at Python/ceval.c:2159
#17 0x00007fa1be3a20b9 in PyEval_EvalCodeEx (co=0x1105030,
globals=<value optimized out>, locals=<value optimized out>,
args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:3000
#18 0x00007fa1be3a2232 in PyEval_EvalCode (co=0x10fee40,
globals=0xd86fc4, locals=0x0) at Python/ceval.c:541
#19 0x00007fa1be3b4800 in PyImport_ExecCodeModuleEx (name=0x7fff9d59dd30
"trac.util.compat", co=0x1105030,
pathname=0x7fff9d59bb60
"/usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg/trac/util/compat.pyc")
at Python/import.c:677
#20 0x00007fa1be3b65d8 in load_source_module (name=0x7fff9d59dd30
"trac.util.compat",
pathname=0x7fff9d59bb60
"/usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg/trac/util/compat.pyc",
---Type <return> to continue, or q <return> to quit---
fp=<value optimized out>) at Python/import.c:1017
#21 0x00007fa1be3b7413 in import_submodule (mod=0x1091910,
subname=0x7fff9d59dd3a "compat",
fullname=0x7fff9d59dd30 "trac.util.compat") at Python/import.c:2592
#22 0x00007fa1be3b76a2 in load_next (mod=0x1091910, altmod=0x1091910,
p_name=<value optimized out>,
buf=0x7fff9d59dd30 "trac.util.compat", p_buflen=0x7fff9d59ed38) at
Python/import.c:2412
#23 0x00007fa1be3b7d25 in import_module_level (name=0x0, globals=<value
optimized out>, locals=<value optimized out>,
fromlist=0x1088890, level=17373456) at Python/import.c:2141
#24 0x00007fa1be3b8265 in PyImport_ImportModuleLevel (name=0x109bdac
"trac.util.compat", globals=0x10710a0, locals=0x10710a0,
fromlist=0x1088890, level=-1) at Python/import.c:2185
#25 0x00007fa1be399d0d in builtin___import__ (self=<value optimized
out>, args=<value optimized out>,
kwds=<value optimized out>) at Python/bltinmodule.c:48
#26 0x00007fa1be2f8028 in PyObject_Call (func=0xcbe0e0, arg=0x109e5d0,
kw=0x0) at Objects/abstract.c:2492
#27 0x00007fa1be39acb6 in PyEval_CallObjectWithKeywords (func=0xcbe0e0,
arg=0x109e5d0, kw=0x0) at Python/ceval.c:3619
#28 0x00007fa1be39df13 in PyEval_EvalFrameEx (f=0x10c9670,
throwflag=<value optimized out>) at Python/ceval.c:2159
#29 0x00007fa1be3a20b9 in PyEval_EvalCodeEx (co=0x10a3b70,
globals=<value optimized out>, locals=<value optimized out>,
args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:3000
#30 0x00007fa1be3a2232 in PyEval_EvalCode (co=0x10fee40,
globals=0xd86fc4, locals=0x0) at Python/ceval.c:541
#31 0x00007fa1be3b4800 in PyImport_ExecCodeModuleEx (name=0x7fff9d5a23a0
"trac.util", co=0x10a3b70,
pathname=0x7fff9d59f180
"/usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg/trac/util/__init__.pyc")
at Python/import.c:677
#32 0x00007fa1be3b65d8 in load_source_module (name=0x7fff9d5a23a0
"trac.util",
pathname=0x7fff9d59f180
"/usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg/trac/util/__init__.pyc",
fp=<value optimized out>) at Python/import.c:1017
#33 0x00007fa1be3b6e55 in load_package (name=0x7fff9d5a23a0 "trac.util",
pathname=<value optimized out>)
at Python/import.c:1073
#34 0x00007fa1be3b7413 in import_submodule (mod=0xdcdd70,
subname=0x7fff9d5a23a5 "util", fullname=0x7fff9d5a23a0 "trac.util")
at Python/import.c:2592
#35 0x00007fa1be3b76a2 in load_next (mod=0xdcdd70, altmod=0xdcdd70,
p_name=<value optimized out>,
buf=0x7fff9d5a23a0 "trac.util", p_buflen=0x7fff9d5a33a8) at
Python/import.c:2412
#36 0x00007fa1be3b7d25 in import_module_level (name=0x0, globals=<value
optimized out>, locals=<value optimized out>,
fromlist=0xddd320, level=-1) at Python/import.c:2141
#37 0x00007fa1be3b8265 in PyImport_ImportModuleLevel (name=0xfbb5cc
"trac.util", globals=0xe27780, locals=0xe27780,
fromlist=0xddd320, level=-1) at Python/import.c:2185
#38 0x00007fa1be399d0d in builtin___import__ (self=<value optimized
out>, args=<value optimized out>,
kwds=<value optimized out>) at Python/bltinmodule.c:48
---Type <return> to continue, or q <return> to quit---
#39 0x00007fa1be2f8028 in PyObject_Call (func=0xcbe0e0, arg=0xfbd1b0,
kw=0x0) at Objects/abstract.c:2492
#40 0x00007fa1be39acb6 in PyEval_CallObjectWithKeywords (func=0xcbe0e0,
arg=0xfbd1b0, kw=0x0) at Python/ceval.c:3619
#41 0x00007fa1be39df13 in PyEval_EvalFrameEx (f=0xf8e250,
throwflag=<value optimized out>) at Python/ceval.c:2159
#42 0x00007fa1be3a20b9 in PyEval_EvalCodeEx (co=0xfbcf30, globals=<value
optimized out>, locals=<value optimized out>,
args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:3000
#43 0x00007fa1be3a2232 in PyEval_EvalCode (co=0x10fee40,
globals=0xd86fc4, locals=0x0) at Python/ceval.c:541
#44 0x00007fa1be3b4800 in PyImport_ExecCodeModuleEx (name=0x7fff9d5a59c0
"trac.web.api", co=0xfbcf30,
pathname=0x7fff9d5a37f0
"/usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg/trac/web/api.pyc")
at Python/import.c:677
#45 0x00007fa1be3b65d8 in load_source_module (name=0x7fff9d5a59c0
"trac.web.api",
pathname=0x7fff9d5a37f0
"/usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg/trac/web/api.pyc",
fp=<value optimized out>) at Python/import.c:1017
#46 0x00007fa1be3b7413 in import_submodule (mod=0xdcde88,
subname=0x7fff9d5a59c9 "api", fullname=0x7fff9d5a59c0 "trac.web.api")
at Python/import.c:2592
#47 0x00007fa1be3b76a2 in load_next (mod=0xdcde88, altmod=0xdcde88,
p_name=<value optimized out>,
buf=0x7fff9d5a59c0 "trac.web.api", p_buflen=0x7fff9d5a69c8) at
Python/import.c:2412
#48 0x00007fa1be3b7d25 in import_module_level (name=0x0, globals=<value
optimized out>, locals=<value optimized out>,
fromlist=0xdc7d10, level=14474888) at Python/import.c:2141
#49 0x00007fa1be3b8265 in PyImport_ImportModuleLevel (name=0xdcfa64
"trac.web.api", globals=0xf207a0, locals=0xf207a0,
fromlist=0xdc7d10, level=-1) at Python/import.c:2185
#50 0x00007fa1be399d0d in builtin___import__ (self=<value optimized
out>, args=<value optimized out>,
kwds=<value optimized out>) at Python/bltinmodule.c:48
#51 0x00007fa1be2f8028 in PyObject_Call (func=0xcbe0e0, arg=0xdc2940,
kw=0x0) at Objects/abstract.c:2492
#52 0x00007fa1be39acb6 in PyEval_CallObjectWithKeywords (func=0xcbe0e0,
arg=0xdc2940, kw=0x0) at Python/ceval.c:3619
#53 0x00007fa1be39df13 in PyEval_EvalFrameEx (f=0xe86e30,
throwflag=<value optimized out>) at Python/ceval.c:2159
#54 0x00007fa1be3a20b9 in PyEval_EvalCodeEx (co=0xdb8738, globals=<value
optimized out>, locals=<value optimized out>,
args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:3000
#55 0x00007fa1be3a2232 in PyEval_EvalCode (co=0x10fee40,
globals=0xd86fc4, locals=0x0) at Python/ceval.c:541
#56 0x00007fa1be3b4800 in PyImport_ExecCodeModuleEx (name=0x7fff9d5aa030
"trac.web", co=0xdb8738,
pathname=0x7fff9d5a6e10
"/usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg/trac/web/__init__.pyc")
at Python/import.c:677
#57 0x00007fa1be3b65d8 in load_source_module (name=0x7fff9d5aa030
"trac.web",
pathname=0x7fff9d5a6e10
"/usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg/trac/web/__init__.pyc",
fp=<value optimized out>) at Python/import.c:1017
#58 0x00007fa1be3b6e55 in load_package (name=0x7fff9d5aa030 "trac.web",
pathname=<value optimized out>) at Python/import.c:1073
---Type <return> to continue, or q <return> to quit---
#59 0x00007fa1be3b7413 in import_submodule (mod=0xdcdd70,
subname=0x7fff9d5aa035 "web", fullname=0x7fff9d5aa030 "trac.web")
at Python/import.c:2592
#60 0x00007fa1be3b76a2 in load_next (mod=0xdcdd70, altmod=0xdcdd70,
p_name=<value optimized out>,
buf=0x7fff9d5aa030 "trac.web", p_buflen=0x7fff9d5ab038) at
Python/import.c:2412
#61 0x00007fa1be3b7d25 in import_module_level (name=0xdcdc65 "main",
globals=<value optimized out>,
locals=<value optimized out>, fromlist=0x7fa1be63c290, level=-1) at
Python/import.c:2141
#62 0x00007fa1be3b8265 in PyImport_ImportModuleLevel (name=0xdcdc5c
"trac.web.main", globals=0xe0a7a0, locals=0xe0a7a0,
fromlist=0x7fa1be63c290, level=-1) at Python/import.c:2185
#63 0x00007fa1be399d0d in builtin___import__ (self=<value optimized
out>, args=<value optimized out>,
kwds=<value optimized out>) at Python/bltinmodule.c:48
#64 0x00007fa1be2f8028 in PyObject_Call (func=0xcbe0e0, arg=0xdc28e8,
kw=0x0) at Objects/abstract.c:2492
#65 0x00007fa1be39acb6 in PyEval_CallObjectWithKeywords (func=0xcbe0e0,
arg=0xdc28e8, kw=0x0) at Python/ceval.c:3619
#66 0x00007fa1be39df13 in PyEval_EvalFrameEx (f=0xe10240,
throwflag=<value optimized out>) at Python/ceval.c:2159
#67 0x00007fa1be3a20b9 in PyEval_EvalCodeEx (co=0xdb8468, globals=<value
optimized out>, locals=<value optimized out>,
args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:3000
#68 0x00007fa1be3a2232 in PyEval_EvalCode (co=0x10fee40,
globals=0xd86fc4, locals=0x0) at Python/ceval.c:541
#69 0x00007fa1be3b4800 in PyImport_ExecCodeModuleEx (name=0xe25b30
"_mod_wsgi_c86079b7ad1998a33323fc73bfc176ae", co=0xdb8468,
pathname=0xe214a8 "/srv/trac-deploy/aclark/cgi-bin/trac.wsgi") at
Python/import.c:677
#70 0x00007fa1be67fc51 in wsgi_load_source (pool=0xe1d008, r=0xe1d080,
name=0xe25b30 "_mod_wsgi_c86079b7ad1998a33323fc73bfc176ae",
exists=<value optimized out>,
filename=0xe214a8 "/srv/trac-deploy/aclark/cgi-bin/trac.wsgi",
process_group=0x7fa1be68c54a "",
application_group=0x7fa1be68c54a "") at mod_wsgi.c:6157
#71 0x00007fa1be685d94 in wsgi_execute_script (r=0xe1d080) at
mod_wsgi.c:6514
#72 0x000000000043c4b3 in ap_run_handler (r=0xe1d080) at config.c:158
#73 0x000000000043f9ee in ap_invoke_handler (r=0xe1d080) at config.c:376
#74 0x000000000044ab6e in ap_process_request (r=0xe1d080) at
http_request.c:282
#75 0x0000000000447a78 in ap_process_http_connection (c=0xc875b0) at
http_core.c:190
#76 0x0000000000443a33 in ap_run_process_connection (c=0xc875b0) at
connection.c:43
#77 0x000000000044f1b0 in child_main (child_num_arg=<value optimized
out>) at prefork.c:662
#78 0x000000000044f448 in make_child (s=0x83e848, slot=0) at prefork.c:707
#79 0x000000000044fa60 in ap_mpm_run (_pconf=<value optimized out>,
plog=<value optimized out>, s=<value optimized out>)
at prefork.c:983
#80 0x0000000000428e45 in main (argc=4, argv=0x7fff9d5abb98) at main.c:739
----
Any thoughts I appreciated! I'd had to have to revert to the older openssl.
Thanks,
Alex
--
Alex Clark � http://aclark.net/help-me-help-plone
Joonas Lehtolahti
I can't provide any solution, but after examining the GDB's backtrace, it
looks like the segfault happens when importing hashlib (being imported by
trac) and hashlib tries to get MD5 implementation from OpenSSL's libcrypto.
So at least if no one knows a direct solution, it might help debugging to
make a very simple wsgi application that just imports hashlib, so that
this particular error has its own test case.
One suggestion I can think of is that the newer OpenSSL has some things
done differently enough that the Python used by mod_wsgi should be rebuilt
for the new OpenSSL and therefore also mod_wsgi then recompiled for this
recompiled Python installation.
Cheers,
Joonas
Alex Clark
> Hello,
>
> I can't provide any solution, but after examining the GDB's backtrace, it
> looks like the segfault happens when importing hashlib (being imported by
> trac) and hashlib tries to get MD5 implementation from OpenSSL's libcrypto.
>
> So at least if no one knows a direct solution, it might help debugging to
> make a very simple wsgi application that just imports hashlib, so that
> this particular error has its own test case.
>
> One suggestion I can think of is that the newer OpenSSL has some things
> done differently enough that the Python used by mod_wsgi should be rebuilt
> for the new OpenSSL and therefore also mod_wsgi then recompiled for this
> recompiled Python installation.
Thanks for the reply, Joonas! Yeah, I'm wondering if this is a
compilation issue (i.e. mismatched libs) or if there is an actual
problem (i.e. incompatibility).
I've reverted to compiling against Debian 5's "libssl0.9.8" package and
that works fine.
Maybe I will try again with the newer openssl (outside of production) to
see if I can figure out what's going on.
Alex
Graham Dumpleton
extension and a Python extension module all at the same time. If they
haven't all been compiled against the same openssl installation and
expecting to use the same version of the openssl libraries you can
have problems.
I would start out by disabling mod_php if enabled and eliminate it as
a conflict. Then look at what libraries mod_ssl.so is expecting to
use. Eg, on a Mac I get:
$ otool -L /usr/libexec/apache2/mod_ssl.so
/usr/libexec/apache2/mod_ssl.so:
/usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current
version 0.9.8)
/usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current
version 0.9.8)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.2.0)
On Linux systems use 'ldd' instead of 'otool -L'.
Similarly check what _ssl.so in Python is expecting to use:
$ otool -L /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/lib-dynload/_ssl.so
/System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/lib-dynload/_ssl.so:
/usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current
version 0.9.8)
/usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current
version 0.9.8)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.2.0)
Graham
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to
> modwsgi+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/modwsgi?hl=en.
>
>