[web2py] Web2py crashing

[David Zejda]

Hi,

my web2py instance (about 10000 pageviews, 300000+ ajax requests
daily) crashes about daily. As a workaround I have a script which
(re)starts the server when either memory usage exceeds certain
threshold or when the server is completely down. I run the script from
cron every minute.

#! /bin/sh

A=`netstat -tlnp | grep 8000`
TIME=`date`
LOG='/var/log/web2py/keepalive.log'

if [ "$A" ];
then
B=`echo $A | cut -d " " -f 7`
PID=${B%/*}
MEM=`ps -p $PID -o vsz | tail -n 1`
if [ "$MEM" -gt 2000000 ];
then
echo "$TIME $PID $MEM MEMORY" >> $LOG
echo "Web2py memory $MEM on $TIME => restart." | mail
m...@mail.com -s 'Web2py fail!'
/etc/init.d/web2py restart
else
echo "$TIME $PID $MEM OK" >> $LOG
fi
else
echo "$TIME FAIL" >> $LOG
echo "Web2py failed on $TIME" | mail m...@mail.com -s 'Web2py fail!'
/etc/init.d/web2py restart
fi

I know I should find time to try to examine cause of memory leaking
with guppy-heapy. But I think the crashing is another issue, not
directly related to the leaking. Access logs revealed no direct
relation to any particular controller or function. Also, after web2py
upgrade it has not changed.

Do you have any idea how to reveal the cause of crashing?

Thanks!
David

[Timothy Farrell]

Can you be more specific on what you mean by "crashing"? Interpreted
languages shouldn't crash and generally if they do it's something wrong
with the interpreter. Are you using cPython?

[David Zejda]

AFAIK web2py uses cherrypy webserver, which listens as a daemon on
certain port. With crashing I mean that the listening process (web2py
with cherrypy at the background) suddenly terminates. I use standard
Python v 2.5.2 as packaged for Debian.

David

[Timothy Farrell]

Is this standard hardware? The only place I've seen Python crash is on
non-standard hardware (AS400) with a non-supported build of Python.

Web2py recent switched from Cherrypy's wsgiserver to Rocket (which I
wrote) as of version 1.77.2 (I think). Which version are you running?

With built-in web-servers, the listening for connections and the
response processing all happens in the same Linux Process. So if it
stops listening, do you still see the process in 'top' or 'px ax'? If
so, then, Yes, the whole process crashed (in which case it is more
likely the Python interpreter); otherwise, there would likely be a bug
somewhere in web2py that we can address.

You could also try Apache+mod_wsgi to see if that changes things.

-tim

[David Zejda]

Yes, vanilla hardware (some dual core Intel). I'm running the latest
web2py now, but it was crashing even with about half year old version
(1.73 or so), so the issue is probably not webserver specific. The
process is in ps ax no more when crashes. Currently the server is
being accessed through Apache proxy. Maybe I'll find a while to test
it with mod_wsgi.

Thanks :)

[mdipierro]

Let us know what you discover.