Segmentation fault - premature end of script headers

[Pigletto]

I'm using Django 1.0, mod_wsgi 2.3 in daemon mode. My application is
hosted at Webfaction.

Sometimes (it is not deterministic) I get errors like:

[Sat Sep 27 01:51:06 2008] [error] [client 127.0.0.1] Premature end of
script headers: rek.wsgi
[Sat Sep 27 01:51:06 2008] [notice] child pid 15592 exit signal
Segmentation fault (11)

My apache is not running mod_python. It uses mpm-worker threads.
Apache conf is like:

(...)
WSGIDaemonProcess rek-prod user=xyz group=xyz processes=2 threads=1
maximum-requests=500 inactivity-timeout=7200 stack-size=524288 display-
name=%{GROUP}
(...)
<VirtualHost *:2867>
ServerName my-domain.xyz

WSGIProcessGroup rek-prod
WSGIScriptAlias / /home2/(...)/rek.wsgi

<Directory /home2/(...)/rek_project/>
Order deny,allow
Allow from all
</Directory>

</VirtualHost>


User that apache is running is, say, 'xyz', same as set to
WSGIDaemonProcess directive.

I've set additional logging for wsgi (as described at
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques).
What I see is that output headers and output content are both empty (0
bytes).
Interesting is that the errors logged are appearing when daemon-
processes are reching max-limit. Saved files are:
size | date | filename

0 Sep 27 00:15 1222492508.42-26504-498.ocontent
136 Sep 27 00:15 1222492508.42-26504-498.oheaders
0 Sep 27 00:20 1222492841.42-26505-500.icontent
1813 Sep 27 00:20 1222492841.42-26505-500.iheaders
0 Sep 27 00:20 1222492841.42-26505-500.ocontent
136 Sep 27 00:20 1222492841.42-26505-500.oheaders
0 Sep 27 00:30 1222493410.86-26504-499.icontent
1813 Sep 27 00:30 1222493410.86-26504-499.iheaders
0 Sep 27 00:30 1222493410.86-26504-499.ocontent
136 Sep 27 00:30 1222493410.86-26504-499.oheaders
0 Sep 27 00:35 1222493743.3-9633-1.icontent
1813 Sep 27 00:35 1222493743.3-9633-1.iheaders
0 Sep 27 00:35 1222493743.3-9633-1.ocontent
0 Sep 27 00:35 1222493743.3-9633-1.oheaders
0 Sep 27 00:45 1222494313.46-26504-500.icontent
1813 Sep 27 00:45 1222494313.46-26504-500.iheaders
0 Sep 27 00:45 1222494313.46-26504-500.ocontent
136 Sep 27 00:45 1222494313.46-26504-500.oheaders
0 Sep 27 00:50 1222494648.64-10774-1.icontent
1813 Sep 27 00:50 1222494648.64-10774-1.iheaders
0 Sep 27 00:50 1222494648.64-10774-1.ocontent
0 Sep 27 00:50 1222494648.64-10774-1.oheaders
0 Sep 27 01:00 1222495215.96-11502-1.icontent
1813 Sep 27 01:00 1222495215.96-11502-1.iheaders
0 Sep 27 01:00 1222495215.96-11502-1.ocontent
0 Sep 27 01:00 1222495215.96-11502-1.oheaders
0 Sep 27 01:05 1222495558.97-11941-1.icontent
1813 Sep 27 01:05 1222495558.97-11941-1.iheaders
0 Sep 27 01:05 1222495558.97-11941-1.ocontent
0 Sep 27 01:05 1222495558.97-11941-1.oheaders
0 Sep 27 01:15 1222496119.36-12730-1.icontent
1813 Sep 27 01:15 1222496119.36-12730-1.iheaders
0 Sep 27 01:15 1222496119.36-12730-1.ocontent
0 Sep 27 01:15 1222496119.36-12730-1.oheaders
0 Sep 27 01:21 1222496460.87-13182-1.icontent
1813 Sep 27 01:21 1222496460.87-13182-1.iheaders
0 Sep 27 01:21 1222496460.87-13182-1.ocontent
0 Sep 27 01:21 1222496460.87-13182-1.oheaders
0 Sep 27 01:30 1222497021.53-13961-1.icontent
1813 Sep 27 01:30 1222497021.53-13961-1.iheaders
0 Sep 27 01:30 1222497021.53-13961-1.ocontent
0 Sep 27 01:30 1222497021.53-13961-1.oheaders
0 Sep 27 01:36 1222497362.97-14450-1.icontent
1813 Sep 27 01:36 1222497362.97-14450-1.iheaders
0 Sep 27 01:36 1222497362.97-14450-1.ocontent
0 Sep 27 01:36 1222497362.97-14450-1.oheaders
0 Sep 27 01:45 1222497924.66-15209-1.icontent
1813 Sep 27 01:45 1222497924.66-15209-1.iheaders
0 Sep 27 01:45 1222497924.66-15209-1.ocontent
0 Sep 27 01:45 1222497924.66-15209-1.oheaders
0 Sep 27 01:51 1222498265.33-15592-1.icontent
1813 Sep 27 01:51 1222498265.33-15592-1.iheaders
0 Sep 27 01:51 1222498265.33-15592-1.ocontent
0 Sep 27 01:51 1222498265.33-15592-1.oheaders
0 Sep 27 01:56 1222498563.45-16320-1.icontent
2300 Sep 27 01:56 1222498563.45-16320-1.iheaders
24720 Sep 27 01:56 1222498563.45-16320-1.ocontent
136 Sep 27 01:56 1222498563.45-16320-1.oheaders
0 Sep 27 01:56 1222498565.66-16794-1.icontent
2275 Sep 27 01:56 1222498565.66-16794-1.iheaders
3641 Sep 27 01:56 1222498565.66-16794-1.ocontent
127 Sep 27 01:56 1222498565.66-16794-1.oheaders
0 Sep 27 02:00 1222498840.89-16320-2.icontent
1813 Sep 27 02:00 1222498840.89-16320-2.iheaders
0 Sep 27 02:00 1222498840.89-16320-2.ocontent
136 Sep 27 02:00 1222498840.89-16320-2.oheaders
0 Sep 27 02:06 1222499167.12-16794-2.icontent
1813 Sep 27 02:06 1222499167.12-16794-2.iheaders
0 Sep 27 02:06 1222499167.12-16794-2.ocontent
136 Sep 27 02:06 1222499167.12-16794-2.oheaders

What can be read from the above is that after reaching count 500 (by
daemon process) there is a number of errors (premature end of script
headers in apache logs) causing that output headers are 0 length.
All the requests (iheaders) are same - from the application that is
monitoring my server every minute.

Additionally I sometimes see situations like:
1. After restart of wsgi daemon (by changing content of rek.wsgi) I
get segmentation fault
2. Then I change the code of my application (eg. I comment out
cache.set and cache.get lines)
3. I do restart of wsgi daemon again.
4. Everything works now
5. I'm changing the code of my application back
6. restart of daemon
7. I get segmentation fault this time
8. I'm adding logging to rek.wsgi
9. Everything works again
10. I remove logging from rek.wsgi (caching is on - everyting as in
point 1. above)
11. Everything works...

So.. this is very strange, unpredictable and not recurrent. Any ideas
what else can I check and how?

[Graham Dumpleton]

Are you using the Python installation and Apache modules that
WebFaction supplies or have you built any yourself from source code?
Have you installed any third party Python modules yourself? Are you
running PHP in the same Apache installation?

Does your Django application create any background threads to perform
processing.

Am asking just to collect a bit more background information while I
think about what to suggest.

Graham

2008/9/28 Pigletto <pigl...@gmail.com>:

[Pigletto]

> Are you using the Python installation and Apache modules that
> WebFaction supplies or have you built any yourself from source code?

I've built Apache 2.2 and mod_wsgi 2.3 myself. Python is webfaction's
python2.5

> Have you installed any third party Python modules yourself? Are you
> running PHP in the same Apache installation?

I've installed mod_auth_tkt. There is no PHP. Just django
applications. Begining of apache.conf is:

ServerRoot "/home2/(...)/apache2.2"
LoadModule wsgi_module /home2/(...)/apache2.2/modules/mod_wsgi.so
LoadModule log_config_module /home2/(...)/apache2.2/modules/
mod_log_config.so
LoadModule env_module /home2/(...)/apache2.2/modules/mod_env.so
LoadModule auth_basic_module /home2/(...)/apache2.2/modules/
mod_auth_basic.so
LoadModule authz_user_module /home2/(...)/apache2.2/modules/
mod_authz_user.so
LoadModule authz_host_module /home2/(...)/apache2.2/modules/
mod_authz_host.so
LoadModule alias_module /home2/(...)/apache2.2/modules/mod_alias.so
LoadModule auth_tkt_module modules/mod_auth_tkt.so


One more thing. While pasting this config
I've noticed that I had path to mod_wsgi.so same as another apache
instance (my test environment) has.
So both of these Apache instances were using same file. I'm not sure
if this matters??

> Does your Django application create any background threads to perform
> processing.

Yes. I'm using external threads to send e-mail messages.
Hm... I'll try to perform some actions that will cause e-mails to be
sent
and then I'll see whether process crashes after reaching maximum-
requests.

> Am asking just to collect a bit more background information while I
> think about what to suggest.

Feel free to ask :) Thanks for the answers so far.

[Graham Dumpleton]

2008/9/29 Pigletto <pigl...@gmail.com>:

>> Does your Django application create any background threads to perform
>> processing.
> Yes. I'm using external threads to send e-mail messages.
> Hm... I'll try to perform some actions that will cause e-mails to be
> sent
> and then I'll see whether process crashes after reaching maximum-
> requests.

Distinct threads can be an issue on shutdown because if they are doing
a lot of stuff in C code, they might still be running and attempting
to do stuff when the Python interpreter instance is being destroyed.
That though you are seeing problems on startup though means it would
have to be something different.

All I can suggest at this point are:

1. Disable the background email thread and see if the problem still occurs.

2. Force application to run in main interpreter by setting:

WSGIApplicationGroup %{GLOBAL}

This will eliminate possibility that problems are caused by a third
party C extension module which isn't implemented correctly so as to
work in secondary sub interpreters.

3. Try and attach gdb, if available, to running daemon process and see
if you can catch traceback for what causes process to crash. See:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB

Graham

Graham

[Pigletto]

> All I can suggest at this point are:
>
> 1. Disable the background email thread and see if the problem still occurs.
>
> 2. Force application to run in main interpreter by setting:
>
>   WSGIApplicationGroup %{GLOBAL}
>
> This will eliminate possibility that problems are caused by a third
> party C extension module which isn't implemented correctly so as to
> work in secondary sub interpreters.
>
> 3. Try and attach gdb, if available, to running daemon process and see
> if you can catch traceback for what causes process to crash. See:
>

>  http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_C...
Thanks for these tips so far.
You're writing about external threads... so I started wondering about
django-compress. This app uses external applications eg. yui
compressor or csstidy to compress all js and css files into one, big
resource file (one for js and one for css). It does merging during
startup of django processes but, AFAIK, only if there are any changes
in js or css files, so this shouldn't be an issue when recent
segmentation fault happened in my Apache, as there were no changes,
but maybe it is somehow connected. I don't know the internals of
django-compress too much.

--
Maciej Wisniowski

[Pigletto]

Today I was not able to start my application as I got segmentation
faults constantly.
I've attached gdb and that is the result:

(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1212216416 (LWP 29850)]
PyErr_Occurred () at Python/errors.c:80
80 Python/errors.c: No such file or directory.
in Python/errors.c
(gdb) bt
#0 PyErr_Occurred () at Python/errors.c:80
#1 0x002ce167 in _PyObject_GC_Malloc (basicsize=40) at Modules/
gcmodule.c:1326
#2 0x002ce21c in _PyObject_GC_NewVar (tp=0x3083c0, nitems=7) at
Modules/gcmodule.c:1352
#3 0x00267c33 in PyTuple_New (size=7) at Objects/tupleobject.c:68
#4 0x0041cdc0 in ?? ()
#5 0x00000007 in ?? ()
#6 0x0000001c in ?? ()
#7 0xb7beab18 in ?? ()
#8 0x0041cd4e in ?? ()
#9 0xb7be9af8 in ?? ()
#10 0xb758b22c in ?? ()
#11 0xb7be9b80 in ?? ()
#12 0x0042fed4 in ?? ()
#13 0x0042fed4 in ?? ()
#14 0xb7be9acc in ?? ()
#15 0xb7be9a2c in ?? ()
#16 0xfbad8001 in ?? ()
#17 0xb7be9ca0 in ?? ()
#18 0xb7be9ca0 in ?? ()
#19 0xb7be9ca0 in ?? ()
#20 0xb7be9ca0 in ?? ()
#21 0x0042fed4 in ?? ()
#22 0x08098608 in apr_bucket_type_eos ()
#23 0x09ba7920 in ?? ()
#24 0x0000002c in ?? ()
#25 0xffffffff in ?? ()
#26 0x00000413 in ?? ()
#27 0x00000000 in ?? ()
(gdb) thread apply all bt

Thread 4 (Thread -1211159648 (LWP 29848)):
#0 0x00ad57a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00bb33b1 in ___newselect_nocancel () from /lib/tls/libc.so.6
#2 0x0097826b in apr_sleep (t=299000053) at time/unix/time.c:246
#3 0x00a76110 in wsgi_monitor_thread (thd=0x9a93420, data=0x9a92dd0)
at mod_wsgi.c:8367
#4 0x0097783c in dummy_worker (opaque=0xfffffdfe) at threadproc/unix/
thread.c:142
#5 0x00c723cc in start_thread () from /lib/tls/libpthread.so.0
#6 0x00bba96e in clone () from /lib/tls/libc.so.6

Thread 3 (Thread -1211688032 (LWP 29849)):
#0 0x00ad57a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00bb33b1 in ___newselect_nocancel () from /lib/tls/libc.so.6
#2 0x0097826b in apr_sleep (t=1000000) at time/unix/time.c:246
#3 0x00a75f6a in wsgi_deadlock_thread (thd=0x9a93440, data=0x9a92dd0)
at mod_wsgi.c:8279
#4 0x0097783c in dummy_worker (opaque=0xfffffdfe) at threadproc/unix/
thread.c:142
#5 0x00c723cc in start_thread () from /lib/tls/libpthread.so.0
#6 0x00bba96e in clone () from /lib/tls/libc.so.6

Thread 2 (Thread -1212216416 (LWP 29850)):
#0 PyErr_Occurred () at Python/errors.c:80
#1 0x002ce167 in _PyObject_GC_Malloc (basicsize=40) at Modules/
gcmodule.c:1326
#2 0x002ce21c in _PyObject_GC_NewVar (tp=0x3083c0, nitems=7) at
Modules/gcmodule.c:1352
#3 0x00267c33 in PyTuple_New (size=7) at Objects/tupleobject.c:68
#4 0x0041cdc0 in ?? ()
#5 0x00000007 in ?? ()
#6 0x0000001c in ?? ()
#7 0xb7beab18 in ?? ()
#8 0x0041cd4e in ?? ()
#9 0xb7be9af8 in ?? ()
#10 0xb758b22c in ?? ()
#11 0xb7be9b80 in ?? ()
#12 0x0042fed4 in ?? ()
#13 0x0042fed4 in ?? ()
#14 0xb7be9acc in ?? ()
#15 0xb7be9a2c in ?? ()
#16 0xfbad8001 in ?? ()
#17 0xb7be9ca0 in ?? ()
#18 0xb7be9ca0 in ?? ()
#19 0xb7be9ca0 in ?? ()
#20 0xb7be9ca0 in ?? ()
#21 0x0042fed4 in ?? ()
#22 0x08098608 in apr_bucket_type_eos ()
#23 0x09ba7920 in ?? ()
#24 0x0000002c in ?? ()
#25 0xffffffff in ?? ()
#26 0x00000413 in ?? ()
#27 0x00000000 in ?? ()

---Type <return> to continue, or q <return> to quit---
Thread 1 (Thread -1208453440 (LWP 29847)):
#0 0x00ad57a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00c787c7 in do_sigwait () from /lib/tls/libpthread.so.0
#2 0x00c7888f in sigwait () from /lib/tls/libpthread.so.0
#3 0x009775ea in apr_signal_thread (signal_handler=0xa75e30
<wsgi_check_signal>) at threadproc/unix/signals.c:383
#4 0x00a76b61 in wsgi_start_process (p=0x9a0d0a8, daemon=0x9a92dd0)
at mod_wsgi.c:8483
#5 0x00a7707a in wsgi_manage_process (reason=0, data=0x9a92dd0,
status=11) at mod_wsgi.c:7708
#6 0x009703c8 in apr_proc_other_child_alert (proc=0xbfea8f80,
reason=0, status=11) at misc/unix/otherchild.c:115
#7 0x080817ad in ap_mpm_run (_pconf=0x9a0d0a8, plog=0x9a3b160,
s=0x9a0ef48) at worker.c:1611
#8 0x08061d9c in main (argc=3, argv=0xbfea90e4) at main.c:730
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
PyErr_Occurred () at Python/errors.c:80
80 in Python/errors.c
(gdb) cont
Continuing.

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb) quit


After switching to WSGIApplicationGroup %{GLOBAL} my application
started, but I have few more applications on this apache instance so I
can't use this kind of setup.
Is there anything interesting in the above gdb log? Any other commands
that I can use next time?

--
Maciej Wisniowski

[Graham Dumpleton]

Not particularly useful unfortunately.

Next thing would be to determine if crash happens as a result of
import WSGI script file itself, or due to call of WSGI application.

Thus at head of WSGI script file add:

import sys
print >> sys.stderr, "START OF WSGI SCRIPT FILE"

and at end of WSGI script file add:

print >> sys.stderr, "END OF WSGI SCRIPT FILE"

If it isn't crashing at load of WSGI script file, both should appear
in Apache error log.

If does crash, add more debug output like that to ascertain which
module being imported causes it to crash.

If that is a big module, then need to recursively work out what module
that module imports and do the import at start of WSGI script file and
try and narrow down which module causes crash.

I can't remember, but will test later, if one can manage to set
environment variable to force Python to log all imports. This will
help narrow it down quicker.

Other option is since works in %{GLOBAL}, once everything imported,
iterate over modules in sys.modules and find all that have __file__
referencing a .so file and print that out. That will tell you which C
extension modules are being used. Standard ones should be okay, but
third party ones would be worth a closer look.

More later.

Graham

2008/9/30 Pigletto <pigl...@gmail.com>:

[Graham Dumpleton]

2008/9/30 Pigletto <pigl...@gmail.com>:

> After switching to WSGIApplicationGroup %{GLOBAL} my application
> started, but I have few more applications on this apache instance so I
> can't use this kind of setup.

Can you explain to me how WebFaction process/memory limits work?

If you don't have issues with number of processes and only overall
memory usage, then create a separate daemon process group for each
application with it being forced to run in main interpreter of its own
process. Thus:

<VirtualHost *:2867>

ServerName my-domain.xyz

WSGIDaemonProcess rek-prod-app-1 user=xyz group=xyz processes=2 threads=1 \
maximum-requests=500 inactivity-timeout=7200 stack-size=524288 \
display-name=%{GROUP}

WSGIScriptAlias / /home2/(...)/rek_project-1.wsgi

<Directory /home2/(...)/rek_project-1/>
WSGIProcessGroup rek-prod-app-1
WSGIApplicationGroup %{GLOBAL}

Order deny,allow
Allow from all
</Directory>

WSGIDaemonProcess rek-prod-app-1 user=xyz group=xyz processes=2 threads=1 \
maximum-requests=500 inactivity-timeout=7200 stack-size=524288 \
display-name=%{GROUP}

WSGIScriptAlias /suburl /home2/(...)/rek_project-2.wsgi

<Directory /home2/(...)/rek_project-2/>
WSGIProcessGroup rek-prod-app-2
WSGIApplicationGroup %{GLOBAL}

Order deny,allow
Allow from all
</Directory>

</VirtualHost>

This would end up with similar memory usage, the difference being that
the application instances are in separate processes rather than
separate sub interpreters of same process.

Graham

[Pigletto]

Now, again, my application is working with the same setup as before
(without GLOBAL). I don't know why this started without segfault now.
Nothing has changed.
I have to mention that the issue that caused I was not able to start
my application today morning was
because my memory was over the limit (before this I was disconnected
while gdb'ing my app on another Apache instance and gdb process was
hung using too much memory)
so webfaction killed my processes. After my processes were killed I
had to start everything and I was not albe to make one of my apps
running (as you have seen already).
So, important thing is that there were no changes in application code
and no changes in apache configuration. Currently it works again and I
can't do more debugging - it doesn't want to segfault.


I've added some print statements as you've suggested but I think that
wsgi script was imported properlywhen segmentation fault has occured
becouse LoggingMiddleware had written empty oheaders.. and ocontent..
files.

> Can you explain to me how WebFaction process/memory limits work?

There are no limits for number of processes only for memory usage.

> If you don't have issues with number of processes and only overall
> memory usage, then create a separate daemon process group for each
> application with it being forced to run in main interpreter of its own
> process. Thus:

>  <Directory /home2/(...)/rek_project-2/>
>    WSGIProcessGroup rek-prod-app-2
>     WSGIApplicationGroup %{GLOBAL}
>    Order deny,allow
>    Allow from all
>  </Directory>
>
> </VirtualHost>
>
> This would end up with similar memory usage, the difference being that
> the application instances are in separate processes rather than
> separate sub interpreters of same process.

OK I'll try this.

Strange thing is that I had no segmentation faults for two days (since
my previous post), and today morning I've seen them one after one.
I think about things like: maximum requests per child setting in
apache, something with threading in apache, memcached - was not
started while I was trying to start my application, but when I've
switched to %{GLOBAL}, memcached was still down and it worked...
I had segmentation faults before (with locmem caching, so it is not
issue with memcached). AFAIR I saw some segfaults before using django-
compress. Maybe this is something nasty in psycopg2. I think about
adding print statements to all my middlewares and functions. This
thing is really hard to debug especially on the server that is used by
real users.

Thank you very much for your help so far.

--
Maciej Wisniowski

[Graham Dumpleton]

What do you get if you run:

ulimit -a

Maybe they have some sort of hard memory limits in place and you are
hitting that.

Graham

2008/9/30 Pigletto <pigl...@gmail.com>:

[Pigletto]

On 30 Wrz, 14:41, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

> What do you get if you run:
>
> ulimit -a
>
> Maybe they have some sort of hard memory limits in place and you are
> hitting that.

Output of ulimit -a is:
-------------------------------------------------------
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 200
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
-----------------------------------------------------------

AFAIK there is no hard limit at Webfaction. I have 160 MB memory limit
but my processes were killed when memory usage was above 220 MB
(ups..). Additionaly after every such incident I'm notified by
Webfaction about this issue. So other segmentation faults I've seen
before are not connected with process killing due to memory problems.

One more question as I'm a bit confused about WSGIApplicationGroup
directive. So far I was not using this at all. Does this mean that %
{GLOBAL} was used implicitly - by default? I only had WSGIProcessGroup
directives in use.

I've added a lot of print>>sys.stderr statements into my application
and I will try to raise segmentation fault somehow...

--
Maciej Wisniowski

[Pigletto]

I've managed to get segmentation fault (I was just clicking around my
application, I forced few reloads of mod_wsgi by changing wsgi script,
etc.), and I was able to reproduce this few times.
Again, I've connected to it with gdb but this time I've issued command
'share' before 'bt'. Thanks to this I was able to see much more
interesting things.

WSGI script is executed, processing reaches my function (view in
Django) and exception is raised inside the view. Below is long output
of gdb. Seems to me that it is psycopg2 issue...?
In my code it is like:

class OrManager(models.Manager):
def latest(self, count=5):
latest = cache.get('latest-offers')
if latest is None:
latest = self.filter(is_active=True).order_by('-
date_added')[:count]
print >> sys.stderr, latest # <<<--------------------
THIS LINE FAILS - real execution of the SQL

I wonder whether this issue might be solved by using %{GLOBAL}?


GDB session:

(...)

(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread -1212707936 (LWP 9463)]

PyErr_Occurred () at Python/errors.c:80
80 Python/errors.c: No such file or directory.
in Python/errors.c
(gdb) bt
#0 PyErr_Occurred () at Python/errors.c:80

#1 0x00d65167 in _PyObject_GC_Malloc (basicsize=40) at Modules/
gcmodule.c:1326
#2 0x00d6521c in _PyObject_GC_NewVar (tp=0xd9f3c0, nitems=7) at
Modules/gcmodule.c:1352
#3 0x00cfec33 in PyTuple_New (size=7) at Objects/tupleobject.c:68
#4 0x00400dc0 in ?? ()

#5 0x00000007 in ?? ()

#6 0x00000009 in ?? ()
#7 0xb7b74aa8 in ?? ()
#8 0x00400d4e in ?? ()
#9 0x00d95980 in PyExc_IndexError () from /usr/lib/libpython2.5.so.
1.0
#10 0x00000000 in ?? ()
(gdb) share
Symbols already loaded for /lib/tls/libm.so.6
Symbols already loaded for /home2/(...)/apache2.2//lib/libaprutil-1.so.
0
Symbols already loaded for /usr/lib/libsqlite3.so.0
Symbols already loaded for /usr/lib/libexpat.so.0
Symbols already loaded for /home2/(...)/apache2.2//lib/libapr-1.so.0
Symbols already loaded for /lib/libuuid.so.1
Symbols already loaded for /lib/tls/librt.so.1
Symbols already loaded for /lib/libcrypt.so.1
Symbols already loaded for /lib/tls/libpthread.so.0
Symbols already loaded for /lib/libdl.so.2
Symbols already loaded for /lib/tls/libc.so.6
Symbols already loaded for /lib/ld-linux.so.2
Symbols already loaded for /lib/libnss_files.so.2
Symbols already loaded for /home2/(...)/apache2.2/modules/mod_wsgi.so
Symbols already loaded for /usr/lib/libpython2.5.so.1.0
Symbols already loaded for /lib/libutil.so.1
Symbols already loaded for /home2/(...)/apache2.2/modules/
mod_log_config.so
Symbols already loaded for /home2/(...)/apache2.2/modules/
mod_auth_basic.so
Symbols already loaded for /home2/(...)/apache2.2/modules/
mod_authz_user.so
Symbols already loaded for /home2/(...)/apache2.2/modules/
mod_authz_host.so
Symbols already loaded for /home2/(...)/apache2.2/modules/mod_env.so
Symbols already loaded for /home2/(...)/modules/mod_alias.so
Symbols already loaded for /home2/(...)/modules/mod_auth_tkt.so
Symbols already loaded for /home2/(...)/modules/mod_rewrite.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
time.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/time.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
collections.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/collections.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
cStringIO.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/cStringIO.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
strop.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/strop.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
cPickle.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/cPickle.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_socket.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_socket.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_ssl.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_ssl.so
Reading symbols from /lib/libssl.so.4...done.
Loaded symbols for /lib/libssl.so.4
Reading symbols from /lib/libcrypto.so.4...done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
operator.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/operator.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
math.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/math.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
binascii.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/binascii.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_random.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_random.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
fcntl.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/fcntl.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
datetime.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/datetime.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_locale.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_locale.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_struct.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_struct.so
Reading symbols from /usr/local/lib/python2.5/site-packages/_xmlplus/
parsers/pyexpat.so...done.
Loaded symbols for /usr/local/lib/python2.5/site-packages/_xmlplus/
parsers/pyexpat.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
array.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/array.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_weakref.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_weakref.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_hashlib.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_hashlib.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_sha256.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_sha256.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_sha512.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_sha512.so
Reading symbols from /home2/(...)/lib/python2.5/psycopg2/
_psycopg.so...done.
Loaded symbols for /home2/(...)/lib/python2.5/psycopg2/_psycopg.so
Reading symbols from /usr/lib/libpq.so.5...done.
Loaded symbols for /usr/lib/libpq.so.5
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_functools.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_functools.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
itertools.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/itertools.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
_bisect.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/_bisect.so
Reading symbols from /usr/local/lib/python2.5/lib-dynload/
zlib.so...done.
Loaded symbols for /usr/local/lib/python2.5/lib-dynload/zlib.so
Reading symbols from /usr/local/lib/python2.5/site-packages/PIL/
_imaging.so...done.
Loaded symbols for /usr/local/lib/python2.5/site-packages/PIL/
_imaging.so
Reading symbols from /usr/lib/libjpeg.so.62...done.
Loaded symbols for /usr/lib/libjpeg.so.62

(gdb) bt
#0 PyErr_Occurred () at Python/errors.c:80

#1 0x00d65167 in _PyObject_GC_Malloc (basicsize=40) at Modules/
gcmodule.c:1326
#2 0x00d6521c in _PyObject_GC_NewVar (tp=0xd9f3c0, nitems=7) at
Modules/gcmodule.c:1352
#3 0x00cfec33 in PyTuple_New (size=7) at Objects/tupleobject.c:68
#4 0x00400dc0 in pq_fetch (curs=0xb7642ec0) at psycopg/pqpath.c:691
#5 0x004018cb in pq_execute (curs=0xb7642ec0,
query=0x88647fc "SELECT \"rek_advobject\".\"id\", \"rek_advobject
\".\"user_id\", \"rek_advobject\".\"gnmid\", \"reklamy_advobject\".
\"lata\", \"reklamy_advobject\".\"wg\","..., async=0) at psycopg/
pqpath.c:620
#6 0x00406660 in _psyco_curs_execute (self=0xb7642ec0,
operation=0x8864220, vars=0xb75e510c, async=0) at psycopg/
cursor_type.c:428
#7 0x00406b07 in psyco_curs_execute (self=0xb7642ec0,
args=0xb7650a8c, kwargs=0x0) at psycopg/cursor_type.c:483
#8 0x00ced4fa in PyCFunction_Call (func=0xb756a1ac, arg=0xb7650a8c,
kw=0x0) at Objects/methodobject.c:108
#9 0x00d366f0 in PyEval_EvalFrameEx (f=0x877e4c4, throwflag=0) at
Python/ceval.c:3566
#10 0x00d37865 in PyEval_EvalCodeEx (co=0xb7928a88,
globals=0xb79208ac, locals=0x0, args=0x87738f4, argcount=3,
kws=0x8773900, kwcount=0, defs=0xb7931758, defcount=1, closure=0x0) at
Python/ceval.c:2833
#11 0x00d3576d in PyEval_EvalFrameEx (f=0x87737a4, throwflag=0) at
Python/ceval.c:3661
#12 0x00d37865 in PyEval_EvalCodeEx (co=0xb78790b0,
globals=0xb787368c, locals=0x0, args=0x8773778, argcount=2,
kws=0x8773780, kwcount=0, defs=0xb787f098, defcount=1, closure=0x0) at
Python/ceval.c:2833
#13 0x00d3576d in PyEval_EvalFrameEx (f=0x877362c, throwflag=0) at
Python/ceval.c:3661
#14 0x00cd21ad in gen_send_ex (gen=0xb758136c, arg=0x0, exc=0) at
Objects/genobject.c:82
#15 0x00d31006 in PyEval_EvalFrameEx (f=0x8773494, throwflag=0) at
Python/ceval.c:2166
#16 0x00cd21ad in gen_send_ex (gen=0xb77aab0c, arg=0x0, exc=0) at
Objects/genobject.c:82
#17 0x00cdfe01 in listextend (self=0xb77fa6cc, b=0xb77aab0c) at
Objects/listobject.c:808
#18 0x00ce275f in list_init (self=0xb77fa6cc, args=0xb75e004c, kw=0x0)
at Objects/listobject.c:2377
#19 0x00d0057f in type_call (type=Variable "type" is not available.
) at Objects/typeobject.c:436
#20 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb75e004c,
kw=0x0) at Objects/abstract.c:1860
#21 0x00d31654 in PyEval_EvalFrameEx (f=0x8773334, throwflag=0) at
Python/ceval.c:3777
#22 0x00d37865 in PyEval_EvalCodeEx (co=0xb79049b0,
globals=0xb788d714, locals=0x0, args=0xb77ae098, argcount=1, kws=0x0,
kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2833
#23 0x00cdba70 in function_call (func=0xb788425c, arg=0xb77ae08c,
kw=0x0) at Objects/funcobject.c:517
#24 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb77ae08c,
kw=0x0) at Objects/abstract.c:1860
#25 0x00cc78ad in instancemethod_call (func=0xb75e8eb4,
arg=0xb77ae08c, kw=0x0) at Objects/classobject.c:2493
#26 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb7ec502c,
kw=0x0) at Objects/abstract.c:1860
#27 0x00d01081 in call_method (o=0xb758142c, name=0xd7b23f "__len__",
nameobj=0xdbcfe4, format=0xd843b0 "()") at Objects/typeobject.c:918
#28 0x00d03cb6 in slot_sq_length (self=0xb758142c) at Objects/
typeobject.c:4112
#29 0x00cb99de in PyObject_Size (o=0xdbdaec) at Objects/abstract.c:72
#30 0x00d2c5b0 in builtin_len (self=0x0, v=0xb758142c) at Python/
bltinmodule.c:1175
#31 0x00d36ad3 in PyEval_EvalFrameEx (f=0x88619b4, throwflag=0) at
Python/ceval.c:3554
#32 0x00d37125 in PyEval_EvalFrameEx (f=0x886182c, throwflag=0) at
Python/ceval.c:3652
#33 0x00d37865 in PyEval_EvalCodeEx (co=0xb7eb35c0,
globals=0xb7ea49bc, locals=0x0, args=0xb7650738, argcount=2, kws=0x0,
kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2833
#34 0x00cdba70 in function_call (func=0xb7eb5f7c, arg=0xb765072c,
kw=0x0) at Objects/funcobject.c:517
#35 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb765072c,
kw=0x0) at Objects/abstract.c:1860
#36 0x00d2fa04 in PyEval_CallObjectWithKeywords (func=0xb7eb5f7c,
arg=0xb765072c, kw=0x0) at Python/ceval.c:3435
#37 0x00d55381 in PyEval_CallMethod (obj=0xb7ec5584,
methodname=0xd813e3 "_reduce_ex", format=0xd813de "(Oi)") at Python/
modsupport.c:585
#38 0x00d01dab in object_reduce_ex (self=0xb758142c, args=0xb77ae06c)
at Objects/typeobject.c:2745
#39 0x00ced4fa in PyCFunction_Call (func=0xb756a16c, arg=0xb77ae06c,
kw=0x0) at Objects/methodobject.c:108
#40 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb77ae06c,
kw=0x0) at Objects/abstract.c:1860
#41 0x0028187f in save (self=0xb75e791c, args=0xb758142c, pers_save=0)
at /root/Downloads/Python-2.5/Modules/cPickle.c:2491
#42 0x00283ef0 in Pickler_dump (self=0xb75e791c, args=0xb777146c) at /
root/Downloads/Python-2.5/Modules/cPickle.c:2573
#43 0x00ced4fa in PyCFunction_Call (func=0xb756a34c, arg=0xb777146c,
kw=0x0) at Objects/methodobject.c:108
#44 0x00d366f0 in PyEval_EvalFrameEx (f=0x8861694, throwflag=0) at
Python/ceval.c:3566
#45 0x00d37125 in PyEval_EvalFrameEx (f=0x88628b4, throwflag=0) at
Python/ceval.c:3652
#46 0x00d37865 in PyEval_EvalCodeEx (co=0xb783c410,
globals=0xb782ea44, locals=0x0, args=0x8861658, argcount=6,
kws=0x8861670, kwcount=0, defs=0xb7834bf8, defcount=1, closure=0x0) at
Python/ceval.c:2833
#47 0x00d3576d in PyEval_EvalFrameEx (f=0x886150c, throwflag=0) at
Python/ceval.c:3661
#48 0x00d37865 in PyEval_EvalCodeEx (co=0xb783c2f0,
globals=0xb782ea44, locals=0x0, args=0x88626d4, argcount=4,
kws=0x88626e4, kwcount=0, defs=0xb7839b38, defcount=2, closure=0x0) at
Python/ceval.c:2833
#49 0x00d3576d in PyEval_EvalFrameEx (f=0x886258c, throwflag=0) at
Python/ceval.c:3661
#50 0x00d37865 in PyEval_EvalCodeEx (co=0xb782a968,
globals=0xb782e824, locals=0x0, args=0x884cde0, argcount=3,
kws=0x884cdec, kwcount=0, defs=0xb7839258, defcount=1, closure=0x0) at
Python/ceval.c:2833
#51 0x00d3576d in PyEval_EvalFrameEx (f=0x884cc9c, throwflag=0) at
Python/ceval.c:3661
#52 0x00d37865 in PyEval_EvalCodeEx (co=0xb772a728,
globals=0xb772d4f4, locals=0x0, args=0x88573dc, argcount=1,
kws=0x88573e0, kwcount=0, defs=0xb77d89f8, defcount=1, closure=0x0) at
Python/ceval.c:2833
#53 0x00d3576d in PyEval_EvalFrameEx (f=0x885728c, throwflag=0) at
Python/ceval.c:3661
#54 0x00d37865 in PyEval_EvalCodeEx (co=0xb7644f50,
globals=0xb7679dfc, locals=0x0, args=0x8817f40, argcount=1,
kws=0x8817f44, kwcount=1, defs=0xb758d8d8, defcount=1, closure=0x0) at
Python/ceval.c:2833
#55 0x00d3576d in PyEval_EvalFrameEx (f=0x8817df4, throwflag=0) at
Python/ceval.c:3661
#56 0x00d37865 in PyEval_EvalCodeEx (co=0xb7644f08,
globals=0xb7679dfc, locals=0x0, args=0xb7ea5c18, argcount=1,
kws=0x87dc948, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/
ceval.c:2833
#57 0x00cdba70 in function_call (func=0xb75890d4, arg=0xb7ea5c0c,
kw=0xb756957c) at Objects/funcobject.c:517

---Type <return> to continue, or q <return> to quit---

#58 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb7ea5c0c,
kw=0xb756957c) at Objects/abstract.c:1860
#59 0x00d3434b in PyEval_EvalFrameEx (f=0x86c9234, throwflag=0) at
Python/ceval.c:3846
#60 0x00d37125 in PyEval_EvalFrameEx (f=0x8476cc4, throwflag=0) at
Python/ceval.c:3652
#61 0x00d37865 in PyEval_EvalCodeEx (co=0xb7acb728,
globals=0xb7eb8a44, locals=0x0, args=0xb7986c90, argcount=3, kws=0x0,
kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2833
#62 0x00cdba70 in function_call (func=0xb798ca04, arg=0xb7986c84,
kw=0x0) at Objects/funcobject.c:517
#63 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb7986c84,
kw=0x0) at Objects/abstract.c:1860
#64 0x00cc78ad in instancemethod_call (func=0xb7950eb4,
arg=0xb7986c84, kw=0x0) at Objects/classobject.c:2493
#65 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb7aed3ac,
kw=0x0) at Objects/abstract.c:1860
#66 0x00d06acb in slot_tp_call (self=0xb7eb958c, args=0xb7aed3ac,
kwds=0x0) at Objects/typeobject.c:4590
#67 0x00cbd69c in PyObject_Call (func=0xd9091c, arg=0xb7aed3ac,
kw=0x0) at Objects/abstract.c:1860
#68 0x00d2fa04 in PyEval_CallObjectWithKeywords (func=0xb7eb958c,
arg=0xb7aed3ac, kw=0x0) at Python/ceval.c:3435
#69 0x001500bf in wsgi_execute_script (r=0x83eada8) at mod_wsgi.c:3045
#70 0x00156b30 in wsgi_daemon_thread (thd=0x8408228, data=0x8408218)
at mod_wsgi.c:9940
#71 0x0069d83c in dummy_worker (opaque=0x0) at threadproc/unix/
thread.c:142
#72 0x00c723cc in start_thread () from /lib/tls/libpthread.so.0
#73 0x00bba96e in clone () from /lib/tls/libc.so.6
(gdb) quit

[Pigletto]

Currently I use psycopg2 from the svn - version dated at January 2008.
I've just looked at initd.org's svn and I see there is psycopg2-2.0.8
and in change log from march I found:

2008-03-07 James Henstridge <ja...@jamesh.id.au>

* psycopg/pqpath.c (_pq_fetch_tuples): Don't call Python APIs
without holding the GIL.

Maybe that is the problem? I'll give a try to newest psycopg2

--
Maciej Wisniowski

[Brett Hoerner]

On Tue, Sep 30, 2008 at 4:22 PM, Pigletto <pigl...@gmail.com> wrote:
> Maybe that is the problem? I'll give a try to newest psycopg2

2.0.8 definitely fixed some segfaults on my end.

Brett

[Graham Dumpleton]

2008/10/1 Pigletto <pigl...@gmail.com>:

> One more question as I'm a bit confused about WSGIApplicationGroup
> directive. So far I was not using this at all. Does this mean that %
> {GLOBAL} was used implicitly - by default? I only had WSGIProcessGroup
> directives in use.

The default for WSGIApplicationGroup is %{RESOURCE}. This means the
named sub interpreter used is either:

server_hostname|/mount/point

for ports 80/443. Or:

server_hostname:server_port|/mount/point

for other ports.

You can see the value in WSGI environment as mod_wsgi.application_group.

What this means is that by default every WSGI application mounted at
different mount point, whether or not in different virtual hosts, get
a distinct sub interpreter. The mount point here is what SCRIPT_NAME
would be that is passed to application. The server hostname is what
SERVER_NAME is and the port SERVER_PORT.

Thus, applications are by default isolated from each other.

Using %{GLOBAL} forces any request handled in context that directive
is in, to be handled in the first/main Python interpreter. This is
effectively the same Python interpreter instance that command line
code always runs in.

This first/main interpreter results in mod_wsgi.application_group in
WSGI environment being empty string.

Graham

[Graham Dumpleton]

2008/10/1 Brett Hoerner <bretth...@gmail.com>:

>
> On Tue, Sep 30, 2008 at 4:22 PM, Pigletto <pigl...@gmail.com> wrote:
>> Maybe that is the problem? I'll give a try to newest psycopg2
>
> 2.0.8 definitely fixed some segfaults on my end.

Good to know. The psycopg2 package has always been a bit of a pain in
sub interpreters. Hopefully newer versions have eliminated the
problems. psycopg2 was one of the third party packages I had in mind
when was thinking about third party C extensions that could be causing
problem.

Graham

[Pigletto]

> 2.0.8 definitely fixed some segfaults on my end.

So far, since September 30 I have not seen any segmentation faults.
Seems that upgrade to 2.0.8 and using %{GLOBAL} solved the problem.

Graham, thank you for the support! :)

--
Maciej Wisniowski