Pango causing mod_wsgi to crash - Segmentation fault (11)

[pbzRPA]

Hi

I was wondering if anyone had a similar problem when trying to use
pango in django running under mod_wsgi on apache.

I am running

|20:11:06|# cat /etc/debian_version
6.0.3

When I run a runserver on the same server it works fine but the moment
I "import pango" in any of my modules the mod_wsgi apache server
raises a 500 error.

I did the whole installation using apt-get and installed pygtk.

I would appreciate any help to try and debug this issue. The only
thing in my apache error.log is :

child pid 3642 exit signal Segmentation fault (11)

Thanks

[Graham Dumpleton]

At a guess it has an C extension component and that component is not
designed properly to work in Python sub interpreters. Force the use of
the main interpreter as documented in:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API

IOW, try adding:

WSGIApplicationGroup %{GLOBAL}

into your Apache configuration.

Alternatively and as explained elsewhere in that document, you have a
shared library version or symbol conflict with other stuff being
loaded into Apache. This can especially happen if you are also loading
PHP into the same Apache.

Graham

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>

[pbzRPA]

Hi Graham,

When I try to add WSGIApplicationGroup %{GLOBAL} i get

OSError: [Errno 13] Permission denied: '/srv/djweb/fileserv/etalent/
django_session/sessionid9b52b55137982353c696b8673b30f714'

What user should have permission to this?

Karol

ps: I do not run any php stuff on my server, well at least none that I
am aware of :)

[Graham Dumpleton]

Read this section in same document:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Access_Rights_Of_Apache_User

Code would normally run as special Apache user unless you have
configured it otherwise.

Apache user need wrote access to the directory, or you need to run
daemon mode of mod_wsgi and tell the daemon process to run as
different user that does have access to that directory.

Graham

[pbzRPA]

Thank you very much Graham,

My server is not crashing any more when I import pango. I still have
some other issues to sort out but that is no longer relevant in this
thread. :)

Regards
Karol

[pbzRPA]

Well I thought the problem was gone but it is not. After restarting
the server I now have a blank page show up in the browser and "child
pid XXX exit signal Segmentation fault (11) " in the error.log file.

I have tried using the gdb debugger that they suggest you use on the
mod_wsgi site but it simply does not work on my server (not sure if
their instructions are old). I do not have a httpd command to attach
gdb to. Is there no other way to try and see what is causing the
problem? I must admit these on days like these I wish I was not
running a linux server.

Graham, I am not sure if you are the one that replied to this post
http://www.mail-archive.com/mod...@googlegroups.com/msg06687.html but
it is exactly the same problem as what I am having.

Regards.

[Graham Dumpleton]

They may have been also getting segmentation fault, but unlikely to be
the same root cause. There are many many reasons stuff can crash.

For debugging, suggest you setup mod_wsgi in daemon mode with default
of a single process. Name the process using display-name option to
WSGIDaemonProcess so can easily identify the process using 'ps'.
Attach gdb to the daemon process by pid as explained in documentation.
Trigger request to crash and get the stack traces.

In other words, you haven't read far enough through the documentation
for other ways to use debugger.

This presumes you are talking about:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB

Read from:

"""Note that you should ensure that you have not assigned your WSGI
application to run in a mod_wsgi daemon process using the
WSGIDaemonProcess and WSGIProcessGroup directives. This is because the
above procedure will only catch crashes which occur when the
application is running in embedded mode. If it turns out that the
application only crashes when run in mod_wsgi daemon mode, an
alternate method of using 'gdb' will be required."""

in the document.

Graham

> Regards.

[pbzRPA]

Hi Graham,

I did read further but I am not a linux guru and the instructions
simple are not detailed enough to know what should be done :(

In the instructions it says:

$ sudo gdb /usr/local/apache/bin/httpd 666

I do not have a "/usr/local/apache/bin/httpd" path or command. I have
no idea what I should replace this with . I have already changed my
apache to run in single daemon mode and I get the PID

11:24:51|/etc/apache2/sites-available$ ps aux | grep "ETServer"
etalent 5676 0.0 0.8 12620 4564 ? Sl 11:17 0:00
ETServer -k start

When I try to use this in gdb:

karol@vz4703|11:24:11|~$ sudo gdb
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) attach 5676
Attaching to process 5676
warning: The current binary is a PIE (Position Independent
Executable), which
GDB does NOT currently support. Most debugger features will fail if
used
in this session.

So I don't know what to do further.

[Graham Dumpleton]

On 16 October 2011 22:33, pbzRPA <pbz...@gmail.com> wrote:
> Hi Graham,
>
> I did read further but I am not a linux guru and the instructions
> simple are not detailed enough to know what should be done :(
>
> In the instructions it says:
>
> $ sudo gdb /usr/local/apache/bin/httpd 666

You should be able to do:

sudo gdb -p 666

as well, where '666' is replaced with actual process ID.

I used the prior form as older gdb versions don't from memory accept
the latter way.

The path '/usr/local/apache/bin/httpd' just had to be replace with the
actual path of your Apache executable.

If your gdb can't handle connecting to the process then there is
something very wrong with your gdb or the way your Apache is compiled
is very weird. Not sure I can help you if that doesn't work.

> So I don't know what to do further.

Verify you are actually running in the main interpreter. See:

http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Sub_Interpreter_Being_Used

Does the crash happen on the very first request which hits the
application or at some random time later.

Another option is that the pango code is not thread safe. Try ensuring
stuff is in a single threaded process by specifying threads=1 to
WSGIDaemonProcess.

Also make sure you aren't mixing Python binaries/installations from
different versions and make sure mod_python not loaded into same
Apache. Go through some of the checks in:

http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library

to make sure mod_wsgi is using Python version you expect and not
different to any Python virtual environment you might be using.

Graham

[pbzRPA]

I tried all your above suggestions and everything seems to be fine.

I don't think the problem lies in my apache configuration as I have a
trac server running on the same server using mog_wsgi and it is still
running fine. I am positive the problem lies with pygtk being used by
mod_wsgi.

I updated my gdb to 7.3 but when I load the pid all the modules come
back with "no debugging symbols found". I don't know if this might
tell you something or not.

karol@vz4703|12:56:21|~$ sudo gdb -p 5921
GNU gdb (GDB) 7.3
Copyright (C) 2011 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.

This GDB was configured as "i686-pc-linux-gnu".

For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.

Attaching to process 5921
Reading symbols from /usr/lib/apache2/mpm-worker/apache2...(no
debugging symbols found)...done.
Reading symbols from /lib/libpcre.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /usr/lib/libaprutil-1.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libapr-1.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libapr-1.so.0
Reading symbols from /lib/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7660b70 (LWP 5929)]
[New Thread 0xb7690b70 (LWP 5923)]
[New Thread 0xb76c0b70 (LWP 5922)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libuuid.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/librt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libexpat.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/apache2/modules/mod_alias.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_alias.so
Reading symbols from /usr/lib/apache2/modules/mod_auth_basic.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_auth_basic.so
Reading symbols from /usr/lib/apache2/modules/mod_authn_file.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authn_file.so
Reading symbols from /usr/lib/apache2/modules/mod_authz_default.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_default.so
Reading symbols from /usr/lib/apache2/modules/mod_authz_groupfile.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_groupfile.so
Reading symbols from /usr/lib/apache2/modules/mod_authz_host.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_host.so
Reading symbols from /usr/lib/apache2/modules/mod_authz_user.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_user.so
Reading symbols from /usr/lib/apache2/modules/mod_autoindex.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_autoindex.so
Reading symbols from /usr/lib/apache2/modules/mod_cgid.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_cgid.so
Reading symbols from /usr/lib/apache2/modules/mod_deflate.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_deflate.so
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/apache2/modules/mod_dir.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_dir.so
Reading symbols from /usr/lib/apache2/modules/mod_env.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_env.so
Reading symbols from /usr/lib/apache2/modules/mod_expires.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_expires.so
Reading symbols from /usr/lib/apache2/modules/mod_headers.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_headers.so
Reading symbols from /usr/lib/apache2/modules/mod_mime.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_mime.so
Reading symbols from /usr/lib/apache2/modules/mod_negotiation.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_negotiation.so
Reading symbols from /usr/lib/apache2/modules/mod_reqtimeout.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_reqtimeout.so
Reading symbols from /usr/lib/apache2/modules/mod_rewrite.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_rewrite.so
Reading symbols from /usr/lib/apache2/modules/mod_setenvif.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_setenvif.so
Reading symbols from /usr/lib/apache2/modules/mod_status.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_status.so
Reading symbols from /usr/lib/apache2/modules/mod_wsgi.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/apache2/modules/mod_wsgi.so
Reading symbols from /usr/lib/libpython2.6.so.1.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/libpython2.6.so.1.0
Reading symbols from /lib/libutil.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libutil.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8
Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8
0xb7d6829c in poll () from /lib/libc.so.6
(gdb) cont
Continuing.

Program received signal SIGINT, Interrupt.
0xb7d6829c in poll () from /lib/libc.so.6
(gdb) backtrace
#0 0xb7d6829c in poll () from /lib/libc.so.6
#1 0xb7e2aa66 in apr_poll () from /usr/lib/libapr-1.so.0
#2 0xb7bb3464 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so
#3 0xb7bb8689 in ?? () from /usr/lib/apache2/modules/mod_wsgi.so
#4 0xb7ee950e in ap_run_post_config ()
#5 0xb7ed410d in main ()
(gdb)

Karol

[Graham Dumpleton]

All you have caught there is the SIGINT received by the process as
sent by Apache parent process telling it to restart. Unless pygtk or
something else is sending it SIGINT for some reason. So, you haven't
caught any crash of process at all. You may need to try a few times.

Also make sure you don't have inactivity-timeout or maximum-requests
set when doing this.

Graham