Weird Segfaults

25 views
Skip to first unread message

aiden bell

unread,
Sep 27, 2010, 4:48:53 PM9/27/10
to modwsgi
Hi All,

Getting odd segfaults in mod_wsgi/3.2, python 2.6.4, apache 2.2.16 on
Fedora 13

Down to the point ... CodeA segfaults and CodeB doesn't whereas they
should both be semantically the same. The segfault
is the standard message "child pid 1234 exit signal Segmentation fault
(11)"

Everything ran fine until this one change (tracked down through
Eclipse's local history). There *was* an actual computed value
where `True` resides ... but I took it out to make the issue as simple
as possible.

CodeA:

...
if self.some_var:
# do something
else:
if(True):
raise SomeException("Some Message")
...

CodeB:

...
if self.some_var:
# do something
else:
raise SomeException("Some Message")
...

Thanks in advance!
Aiden
http://stackoverflow.com/users/104040/aiden-bell

sste...@gmail.com

unread,
Sep 27, 2010, 5:04:24 PM9/27/10
to mod...@googlegroups.com

On Sep 27, 2010, at 4:48 PM, aiden bell wrote:

> Hi All,
>
> Getting odd segfaults in mod_wsgi/3.2, python 2.6.4, apache 2.2.16 on
> Fedora 13
>
> Down to the point ... CodeA segfaults and CodeB doesn't whereas they
> should both be semantically the same. The segfault
> is the standard message "child pid 1234 exit signal Segmentation fault
> (11)"
>
> Everything ran fine until this one change

What one change?


> (tracked down through
> Eclipse's local history). There *was* an actual computed value
> where `True` resides ... but I took it out to make the issue as simple
> as possible.
>
> CodeA:
>
> ...
> if self.some_var:
> # do something
> else:
> if(True):
> raise SomeException("Some Message")
> ...
>
> CodeB:
>
> ...
> if self.some_var:
> # do something
> else:
> raise SomeException("Some Message")
> ...
>
> Thanks in advance!
> Aiden
> http://stackoverflow.com/users/104040/aiden-bell
>

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>

Aiden Bell

unread,
Sep 27, 2010, 5:06:58 PM9/27/10
to mod...@googlegroups.com
On 27 September 2010 22:04, sste...@gmail.com <sste...@gmail.com> wrote:

On Sep 27, 2010, at 4:48 PM, aiden bell wrote:

> Hi All,
>
> Getting odd segfaults in mod_wsgi/3.2, python 2.6.4, apache 2.2.16 on
> Fedora 13
>
> Down to the point ... CodeA segfaults and CodeB doesn't whereas they
> should both be semantically the same. The segfault
> is the standard message "child pid 1234 exit signal Segmentation fault
> (11)"
>
> Everything ran fine until this one change

What one change?

The `if(True):` 



--
------------------------------------------------------------------
Never send sensitive or private information via email unless it is encrypted. http://www.gnupg.org

aiden bell

unread,
Sep 27, 2010, 5:10:47 PM9/27/10
to modwsgi


On Sep 27, 10:04 pm, "sstein...@gmail.com" <sstein...@gmail.com>
wrote:
> On Sep 27, 2010, at 4:48 PM, aiden bell wrote:
>
> > Hi All,
>
> > Getting odd segfaults in mod_wsgi/3.2, python 2.6.4, apache 2.2.16 on
> > Fedora 13
>
> > Down to the point ... CodeA segfaults and CodeB doesn't whereas they
> > should both be semantically the same. The segfault
> > is the standard message "child pid 1234 exit signal Segmentation fault
> > (11)"
>
> > Everything ran fine until this one change
>
> What one change?
>
As an aside to the `if(True):` I also can't catch the error with gdb,
the request
just hangs. Also getting segfaults with any subsequent edit that
involves adding
an if statement! Odd stuff.

Graham Dumpleton

unread,
Sep 27, 2010, 6:02:43 PM9/27/10
to mod...@googlegroups.com
Read:

http://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions

Section about crashes.

Also see:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques

In short, try using main interpreter. Ensure mod_python not being
loaded. Disable php if you can.

Otherwise use gdb as documented to get stack trace and investigate.

Graham

aiden bell

unread,
Sep 28, 2010, 10:46:12 AM9/28/10
to modwsgi


On Sep 27, 11:02 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> Read:
>
> http://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions
>
Done and ruled out any MySQL conflict, mod_python and other modules
mentioned

> Section about crashes.
>
> Also see:
>
> http://code.google.com/p/modwsgi/wiki/ApplicationIssueshttp://code.google.com/p/modwsgi/wiki/DebuggingTechniques
>
> In short, try using main interpreter. Ensure mod_python not being
> loaded. Disable php if you can.
>
> Otherwise use gdb as documented to get stack trace and investigate.
>
using gdb as stated was unsuccessful, taking all the steps in
DebuggingTechniques, I only get:
"Script timed out before returning headers"
In the log and no segfault in gdb when attaching to either the
mod_wsgi process or apache.

However, the segfault only happens on the first request in each apache
process,
subsequent requests run fine.

aiden bell

unread,
Sep 28, 2010, 11:50:20 AM9/28/10
to modwsgi


On Sep 28, 3:46 pm, aiden bell <aiden...@gmail.com> wrote:
> On Sep 27, 11:02 pm, Graham Dumpleton <graham.dumple...@gmail.com>
> wrote:> Read:
>
> >http://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions
>
> Done and ruled out any MySQL conflict, mod_python and other modules
> mentioned
>
> > Section about crashes.
>
> > Also see:
>
> >http://code.google.com/p/modwsgi/wiki/ApplicationIssueshttp://code.go...
>
> > In short, try using main interpreter. Ensure mod_python not being
> > loaded. Disable php if you can.
>
> > Otherwise use gdb as documented to get stack trace and investigate.
>
> using gdb as stated was unsuccessful, taking all the steps in
> DebuggingTechniques, I only get:
> "Script timed out before returning headers"
> In the log and no segfault in gdb when attaching to either the
> mod_wsgi process or apache.
>
Update: It is the mod_wsgi process crashing (by PID and LogLevel info
in apache).
Attaching gdb to the process just causes the request to hang even
after increasing
Timeout in Apache to an obscene figure.
As soon as gdb is detached, the request advances to segfault. List of
loaded

modules:
core_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
auth_basic_module (shared)
auth_digest_module (shared)
authn_file_module (shared)
authn_alias_module (shared)
authn_anon_module (shared)
authn_dbm_module (shared)
authn_default_module (shared)
authz_host_module (shared)
authz_user_module (shared)
authz_owner_module (shared)
authz_groupfile_module (shared)
authz_dbm_module (shared)
authz_default_module (shared)
ldap_module (shared)
authnz_ldap_module (shared)
include_module (shared)
log_config_module (shared)
logio_module (shared)
env_module (shared)
ext_filter_module (shared)
mime_magic_module (shared)
expires_module (shared)
deflate_module (shared)
headers_module (shared)
usertrack_module (shared)
setenvif_module (shared)
mime_module (shared)
dav_module (shared)
status_module (shared)
autoindex_module (shared)
info_module (shared)
dav_fs_module (shared)
vhost_alias_module (shared)
negotiation_module (shared)
dir_module (shared)
actions_module (shared)
speling_module (shared)
userdir_module (shared)
alias_module (shared)
rewrite_module (shared)
proxy_module (shared)
proxy_balancer_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_connect_module (shared)
cache_module (shared)
suexec_module (shared)
disk_cache_module (shared)
cgi_module (shared)
proxy_ajp_module (shared)
dav_svn_module (shared)
authz_svn_module (shared)
wsgi_module (shared)
Syntax OK

Maybe a fedora specific problem, which I will take to their bugzilla
if
I can debug it. Any hint on getting a backtrace?

sste...@gmail.com

unread,
Sep 28, 2010, 12:26:46 PM9/28/10
to mod...@googlegroups.com

On Sep 28, 2010, at 11:50 AM, aiden bell wrote:
> Update: It is the mod_wsgi process crashing (by PID and LogLevel info
> in apache).
> Attaching gdb to the process just causes the request to hang even
> after increasing
> Timeout in Apache to an obscene figure.
> As soon as gdb is detached, the request advances to segfault. List of
> loaded
>
> modules:
> core_module (static)
[snip]

Are you really using all that stuff?

This sure would be easier to narrow down if you removed anything you weren't actually using...

S

aiden bell

unread,
Sep 28, 2010, 1:33:32 PM9/28/10
to modwsgi
Removed as many as didn't break apache. I think I *may* have tracked
it down to importing
the uuid module in Python. If I remove that line, the segfaults *seem*
to stop; But that may
just be correlation rather than causation. Ring true? Anyone else had
"import uuid" problems?

Graham Dumpleton

unread,
Sep 28, 2010, 10:58:25 PM9/28/10
to mod...@googlegroups.com

If the uuid module on Python is an external C module, ie., has a .so,
or even if it uses some, then what are the library dependencies the
.so files have. ie., result from running:

ldd *.so

on the .so for or used by uuid.

Do the library versions, paths differe to what Apache httpd executable
uses, or which are used by and PHP modules if being used, or other
third party Pache modules.

Also ensure you post your mod_wsgi configuration snippet you are
using. Ie., WSGIDaemonProcess, WSGIProcessGroup, WSGIApplictionGroup,
WSGIScriptAlias etc.

I want to confirm you are actually running in main interpreter.

Can you also try it in embedded mode if you are using daemon mode and
verify if it occurs there as well.

You should really be able to capture it in gdb if reproducible.

Graham

aiden bell

unread,
Oct 4, 2010, 6:56:55 PM10/4/10
to modwsgi
Still tracking this down. The segfault is in _ctypes.so, but proving
tricky to get a bt on,
as it seems not to segfault in gdb. I think UUID was a red herring.

On 29 Sep, 03:58, Graham Dumpleton <graham.dumple...@gmail.com> wrote:

Deron Meranda

unread,
Oct 5, 2010, 2:14:18 AM10/5/10
to mod...@googlegroups.com
I can replicate this SEGV by just doing:

import ctypes

in a wsgi script using just what comes with Fedora 13.

From what I can tell right off the bat, it looks like this may be an SELinux
permissions problem. I'm seeing httpd creating temporary files under
/tmp, /var/tmp, and /dev/shm and and then trying to execute them --
which SELinux is preventing. I don't know yet if this is due to ctypes.

By chance are you running with SLinux in enforcing mode? Use:

# sestatus


FYI about versions...

The mod_wsgi packaged with Fedora 13 is 3.1. Since you're using 3.2, I assume
you've compiled from source By the way, Fedora 14 which is due in a month
is set to ship with mod_wsgi 3.2 as well as Python 2.7.

The Python uuid module potentially has a lot of crazy external dependencies,
most of which are determined at run-time (import time). Under Fedora, it
will use the ctypes module to load in the shared library libuuid.so.1
(which is either under /lib64 or /lib depending on your cpu). So, importing
uuid will import and invoke ctypes.

--
Deron Meranda
http://deron.meranda.us/

Deron Meranda

unread,
Oct 5, 2010, 2:29:52 AM10/5/10
to mod...@googlegroups.com
Yes, this does appear to be caused by SELinux. Under the default
policy rules, importing the python ctypes module when running under
the httpd security context will cause a security violation.

For now, the best way to make this work is to change one of the
SELinux booleans (do not disable SELinux entirely). As root, do

# setsebool -P httpd_tmp_exec 1

Deron Meranda

unread,
Oct 5, 2010, 4:16:22 AM10/5/10
to mod...@googlegroups.com
> For now, the best way to make this work is to change one of the
> SELinux booleans (do not disable SELinux entirely).  As root, do
>
>   # setsebool -P httpd_tmp_exec 1


Another more secure option, if all you need is to generate a uuid and you
don't need to use ctypes for anything else, is to generate an
RFC 4122-compliant random uuid as follows. This allows you to omit
importing the python uuid module, which in turn imports ctypes.

(This works because SELinux allows reading from /dev/urandom under
the default policy rules)


def make_random_uuid():
bytes = open('/dev/urandom','rb').read(16)
num = long(('%02x'*16) % tuple(map(ord, bytes)), 16)
# Set the variant to RFC 4122.
num &= ~(0xc000 << 48L)
num |= 0x8000 << 48L
# Set the version number (4 = random uuid).
num &= ~(0xf000 << 64L)
num |= 4 << 76L
# Convert to hex-string format
hex = '%032x' % num
return '%s-%s-%s-%s-%s' % (
hex[:8], hex[8:12], hex[12:16], hex[16:20], hex[20:])

aiden bell

unread,
Oct 5, 2010, 7:01:17 AM10/5/10
to modwsgi


On 5 Oct, 09:16, Deron Meranda <deron.mera...@gmail.com> wrote:
> > For now, the best way to make this work is to change one of the
> > SELinux booleans (do not disable SELinux entirely).  As root, do
>
> >   # setsebool -P httpd_tmp_exec 1
>

You sir, are a sage. I am on F13. Bloody SELinux! I shall give this a
good
test, but it sounds like you hit the nail there.
Reply all
Reply to author
Forward
0 new messages