Error in wsgi/apache

[Michael Toomim]

I'm getting errors like these in my apache error logs:


[Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end
of script headers: wsgihandler.py, referer:
http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4VLSQ&hitId=1NAV09D0NWNU2X87QR3I6RXXG0ER8N&workerId=A37YC0D90LZF2M&turkSubmitTo=https%3A%2F%2Fwww.mturk.com
[Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature
end of script headers: wsgihandler.py, referer:
http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1O3I&hitId=1G15BSUI1DBLMZPV54KGZFTE6JM0Z3&workerId=A3I5DLZHYT46GS&turkSubmitTo=https%3A%2F%2Fwww.mturk.com
[Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi
(pid=7730): Exception occurred processing WSGI script '/home/toomim/
projects/utility/web2py/wsgihandler.py'.
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
(pid=7730): Exception occurred processing WSGI script '/home/toomim/
projects/utility/web2py/wsgihandler.py'.
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
(pid=7730): Exception occurred processing WSGI script '/home/toomim/
projects/utility/web2py/wsgihandler.py'.
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError:
failed to write data
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
(pid=7730): Exception occurred processing WSGI script '/home/toomim/
projects/utility/web2py/wsgihandler.py'.
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError:
failed to write data
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
(pid=7730): Exception occurred processing WSGI script '/home/toomim/
projects/utility/web2py/wsgihandler.py'.
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError:
failed to write data
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
(pid=7730): Exception occurred processing WSGI script '/home/toomim/
projects/utility/web2py/wsgihandler.py'.
[Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError:
failed to write data


My web app gets about 7 requests per second. At first, things work
fine. Then after a while it seems like every request gets handled by
MULTIPLE threads, because my logging.debug() statements print multiple
copies of each message and it seems my database gets multiple entries.
And I get these errors in the apache logs (with LogLevel debug).

Any idea what to do? Where to look? I'm on ubuntu.

[Michael Toomim]

This message about "bucket brigade" is also appearing in the apache
error log:

[Mon Jul 19 19:01:53 2010] [error] [client 183.87.223.111] (9)Bad file
descriptor: mod_wsgi (pid=7940): Unable to get bucket brigade for
request., referer:
http://yuno.us/init/hits/hit?assignmentId=1WL68USPJR0HY1ENS50GN6IJ33ZY32&hitId=1TK6NH2ZSBU3RCI3F8FK7JE1YXMG96&workerId=AJ8R357DF74FF&turkSubmitTo=https%3A%2F%2Fwww.mturk.com

On Jul 19, 7:01 pm, Michael Toomim <too...@gmail.com> wrote:
> I'm getting errors like these in my apache error logs:
>
> [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end

> of script headers: wsgihandler.py, referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V...

> [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature

> end of script headers: wsgihandler.py, referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1...

[Michael Toomim]

And after a while apache completely freezes.

On Jul 19, 7:05 pm, Michael Toomim <too...@gmail.com> wrote:
> This message about "bucket brigade" is also appearing in the apache
> error log:
>
> [Mon Jul 19 19:01:53 2010] [error] [client 183.87.223.111] (9)Bad file
> descriptor: mod_wsgi (pid=7940): Unable to get bucket brigade for

> request., referer:http://yuno.us/init/hits/hit?assignmentId=1WL68USPJR0HY1ENS50GN6IJ33Z...

[Graham Dumpleton]

On Jul 20, 12:01 pm, Michael Toomim <too...@gmail.com> wrote:
> I'm getting errors like these in my apache error logs:
>
> [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end

> of script headers: wsgihandler.py, referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V...

> [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature

> end of script headers: wsgihandler.py, referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1...

The above is because the daemon process you are running web2py in
crashed.

> [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi
> (pid=7730): Exception occurred processing WSGI script '/home/toomim/
> projects/utility/web2py/wsgihandler.py'.
> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
> (pid=7730): Exception occurred processing WSGI script '/home/toomim/
> projects/utility/web2py/wsgihandler.py'.
> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
> (pid=7730): Exception occurred processing WSGI script '/home/toomim/
> projects/utility/web2py/wsgihandler.py'.
> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError:
> failed to write data

In the case of daemon mode being used, this is because the Apache
server child process crashed.

> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
> (pid=7730): Exception occurred processing WSGI script '/home/toomim/
> projects/utility/web2py/wsgihandler.py'.
> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError:
> failed to write data
> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
> (pid=7730): Exception occurred processing WSGI script '/home/toomim/
> projects/utility/web2py/wsgihandler.py'.
> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError:
> failed to write data
> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi
> (pid=7730): Exception occurred processing WSGI script '/home/toomim/
> projects/utility/web2py/wsgihandler.py'.
> [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError:
> failed to write data
>
> My web app gets about 7 requests per second. At first, things work
> fine. Then after a while it seems like every request gets handled by
> MULTIPLE threads, because my logging.debug() statements print multiple
> copies of each message and it seems my database gets multiple entries.
> And I get these errors in the apache logs (with LogLevel debug).
>
> Any idea what to do? Where to look? I'm on ubuntu.

Look at your systems resource usage, ie., memory, open files etc. The
above are symptomatic of your operating system running out of
resources and processes not coping too well with that.

Graham

[Michael Toomim]

Thanks! I tried rebooting the OS. Now my resources seem ok (but I
didn't check before the reboot):

Files used: 1376 out of 75556
Mem used: 580mb out of 796mb
Swap used: 0
CPU: 88-99% idle

And I know longer see the "Exception occurred" or "IOError" messages,
however I DO still see "Premature end of script headers". These
errors come in batches, every 10-20 seconds or so I get a continuous
block of 10-20 "Premature end of script headers" errors from different
clients. These are followed by errors notifying me that clients' ajax
requests failed.

I also found three of these in my web2py tickets:

Traceback (most recent call last):
File "gluon/main.py", line 337, in wsgibase
parse_get_post_vars(request, environ)
File "gluon/main.py", line 222, in parse_get_post_vars
request.body = copystream_progress(request) ### stores request
body
File "gluon/main.py", line 95, in copystream_progress
copystream(source, dest, size, chunk_size)
File "gluon/fileutils.py", line 301, in copystream
data = src.read(size)
IOError: request data read error

However, I've gotten around 3000 "premature end of script" errors, and
only 3 of these IOErrors.

Is there a way to identify what is causing the "Premature end of
script" errors?

On Jul 19, 7:50 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

[Graham Dumpleton]

They can also occur if you have set maximum-requests to a very low
value and have long running requests which are being interrupted when
process is forcibly restarted.

You should not use maximum-requests in a production system if you can
avoid it.

If you have memory/resource leaks in your application then fix them
rather than relying on periodic restarts.

Graham

[mdipierro]

The problem with IOError, I can understand. As Graham says, if the
client closes the connection before the server responds or if the
server timesout the socket is closed and apache logs the IOError.

What I really do not understand is why some requests are handled by
multiple threads. web2py is agnostic to this (unless you use Rocket
which you do not). web2py only provides a wsgi application which is
executed - per thread - by the web server. It is the web server (in
your case apache) that spans the thread, maps requests to threads,
calls the web2py wsgi application for each of them.

If this is happening it is a problem with apache or with mod_wsgi. Can
you tell us more about the version of ubuntu, apache and mod_wsgi that
you are using? Any additional information will be very useful.

Massimo

On Jul 19, 9:01 pm, Michael Toomim <too...@gmail.com> wrote:
> I'm getting errors like these in my apache error logs:
>
> [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end

> of script headers: wsgihandler.py, referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V...

> [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature

> end of script headers: wsgihandler.py, referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1...

[Graham Dumpleton]

On Jul 20, 5:17 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> The problem with IOError, I can understand. As Graham says, if the
> client closes the connection before the server responds or if the
> server timesout the socket is closed and apache logs the IOError.

That isn't what I said. If you see that message when using daemon
mode, the Apache server process that is proxying to the daemon process
is crashing. This is different to the HTTP client closing the
connection. You would only see that message if HTTP client closed
connection if using embedded mode.

I know they are using daemon mode as that is the only situation where
they could also see the message about premature end of script headers.

> What I really do not understand is why some requests are handled by
> multiple threads. web2py is agnostic to this (unless you use Rocket
> which you do not). web2py only provides a wsgi application which is
> executed - per thread - by the web server. It is the web server (in
> your case apache) that spans the thread, maps requests to threads,
> calls the web2py wsgi application for each of them.
>
> If this is happening it is a problem with apache or with mod_wsgi.

More likely the problem is that they are registering the logging
module from multiple places and that is why logging is displayed more
than once. They should log the thread ID as well as that would confirm
whether actually from the same thread where logging module handler has
been registered multiple times.

Multiple registrations of logging handler could occur if it isn't done
in a thread safe why, ie., so as to avoid multiple threads doing it at
the same time.

Graham

[mdipierro]

Thanks for the clarification.

@Michael, do you use the logging module? How?


On Jul 20, 4:00 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

[Michael Toomim]

Thank you for the clarification.

My wsgi.conf has default values, so I have not set maximum-requests.
Perhaps there are settings there I should look into?

I still have free memory, so perhaps there is not a memory leak
issue. I'm also not sure how one would get memory leaks in web2py,
since isn't the environment wiped clean with each request?

This looks similar to the issue here:
http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4910bcc/b6fac1806ffebcd1?lnk=gst&q=ioerror#b6fac1806ffebcd1
Was there any resolution?

I use logging by having the following file in models/0_log.py:

import logging
def get_log():
try:
f = open(logging.handlers[0].baseFilename, 'r')
c = f.readlines()
f.close()
return {'log':TABLE(*[TR(str(item)) for item in c])}
except:
return ()


# This model file defines some magic to implement app_wide_log.
def _init_log(): # Does not work on GAE
import os,logging,logging.handlers
logger = logging.getLogger(request.application)
logger.setLevel(logging.DEBUG)
handler = logging.handlers.RotatingFileHandler(
os.path.join( # so that it can be served as http://.../yourapp/static/applog.txt
request.folder,'static','applog.txt'),'a',1024*1024,1)
handler.setLevel(logging.DEBUG)
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %
(filename)s:%(lineno)d %(funcName)s(): %(message)s"))
logger.addHandler(handler)
return logger

logging =
cache.ram('app_wide_log',lambda:_init_log(),time_expire=None)

[Michael Toomim]

Let me also summarize the issues so far.

Originally:
- I got three types of error messages in apache logs
- Logging messages were often duplicated 2, 3, 5 times
- I got the IOError ticket a few times
- After a while the web serving slowed (some requests took up to a
minute) and then quit completely

After rebooting:
- I get one type of error message in apache logs, in big batches
- I get the IOError ticket once or twice
- After a while web serving slows (sometimes 150s per request) and
stops

So I haven't been seeing the duplicate log messages anymore.

I upgraded to a bigger machine and am changing my code to remove ajax
(will reduce load by 60x by decreasing functionality). I don't know
what else to do.

On Jul 20, 2:03 am, mdipierro <mdipie...@cs.depaul.edu> wrote:

[mdipierro]

Can you comment on memory usage?

I have see this once: "after a while web serving slows"
it appeared to be due to a memory leak somewhere (did not experience
it with web2py+Rocket but only in web2py+mod_wsgi+apache).

I googled it and I found Django was having the same problem on some
hosts:

http://stackoverflow.com/questions/2293333/django-memory-usage-going-up-with-every-request
http://blog.webfaction.com/tips-to-keep-your-django-mod-python-memory-usage-down

I followed the advice from a comment in the last post to limit the
number of requests served by each process:

# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 500
ServerLimit 30

instead of the default:

# prefork MPM
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256

The problem disappeared. The exact values that fix the problem may
depend on the ram available.

Massimo

[Graham Dumpleton]

On Jul 21, 7:17 am, Michael Toomim <too...@gmail.com> wrote:
> Thank you for the clarification.
>
> My wsgi.conf has default values, so I have not set maximum-requests.
> Perhaps there are settings there I should look into?
>
> I still have free memory, so perhaps there is not a memory leak
> issue.  I'm also not sure how one would get memory leaks in web2py,
> since isn't the environment wiped clean with each request?

The environment is not wiped clean on each request, the process stays
persistent.

Both memory leaks and resource leaks can occur to blow out memory
usage in Python web application processes.

A memory leak is where at C code level memory is allocated and never
freed, ie., traditional concept of a memory leak.

At Python level you can get what I'll call resource leaks. This can
arise for a couple of reasons, all under the control of the
programmer.

The first is where one caches objects in some sort of data structure
and don't clean them up at the end of the request. If the way in which
that data is stored is unique per request, then the number of cached
objects will grow continually. Such a data structure may be an
explicit global variable in a Python module, or indirectly as part of
an object which is itself referenced form a global variable. You also
have thread local data storage. These are still global objects, but
each thread has its own instance of them. In either case, you can get
memory increase through growing numbers of objects, but also could
have problems if references to open file objects were held in them. In
that case can also run out of file descriptors on the system.

As well as incorrect usage of caches, ie., not cleaning them up, you
can also get cyclical object structures. In certain cases where these
have __del__ methods, the Python garbage collector cannot break the
cycle and so they stay in memory and never get reclaimed. This is a
nasty problem, may worse if these objects store file objects, again
leading to exhaustion of file descriptors on the system.

Graham

> This looks similar to the issue here:http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4...

> Was there any resolution?
>
> I use logging by having the following file in models/0_log.py:
>
> import logging
> def get_log():
>     try:
>         f = open(logging.handlers[0].baseFilename, 'r')
>         c = f.readlines()
>         f.close()
>         return {'log':TABLE(*[TR(str(item)) for item in c])}
>     except:
>         return ()
>
> # This model file defines some magic to implement app_wide_log.
> def _init_log(): # Does not work on GAE
>     import os,logging,logging.handlers
>     logger = logging.getLogger(request.application)
>     logger.setLevel(logging.DEBUG)
>     handler = logging.handlers.RotatingFileHandler(

>       os.path.join( # so that it can be served ashttp://.../yourapp/static/applog.txt

[Graham Dumpleton]

On Jul 21, 8:18 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
> Can you comment on memory usage?
>
> I have see this once: "after a while web serving slows"
> it appeared to be due to a memory leak somewhere (did not experience
> it with web2py+Rocket but only in web2py+mod_wsgi+apache).
>
> I googled it and I found Django was having the same problem on some
> hosts:

Not sure how you can draw a parallel to that as it is a completely
different framework and just because another framework, or more
specifically one persons code, has issues, doesn't imply there is an
issue with underlying web hosting.

These sorts of problems are isolated cases. If there was an issue with
memory leakage in the hosting mechanism it would be affecting everyone
and there have been no such reports of mod_wsgi itself leaking memory.
That said, ensure you read:

http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html

This describes how Python itself leaks memory. For mod_wsgi 2.X and
older, or if you are still loading mod_python into your Apache server,
then you can be affected by this, but not if using mod_wsgi 3.X.

That post also explains how to completely disable initialisation of
Python in Apache server child processes, ie., embedded, if you aren't
using it.

> http://stackoverflow.com/questions/2293333/django-memory-usage-going-...http://blog.webfaction.com/tips-to-keep-your-django-mod-python-memory...

>
> I followed the advice from a comment in the last post to limit the
> number of requests served by each process:

Which is actually a useless thing to do if you are using daemon mode
which I understood you were, as MaxRequestsPerChild directive only
affects Apache server child process, ie., those use for embedded mode,
and not daemon mode processes.

If using that directive helped and you were using daemon mode, then
you likely have a memory leak in some other Apache module.

What you should have ensured you were doing was using display-name
option to WSGIDaemonProcess to name the process. That way in 'ps' you
can easily distinguish the mod_wsgi daemon mode processes from the
Apache processes and work out which is leaking memory. If it is the
daemon processes, it is likely to be a Python web application issue.
If the Apache parent process is getting fatter and you perform a lot
of Apache restart/reloads, then it could be that you are still using
mod_wsgi 2.X or mod_python is loaded at same time, and you are using a
version of Python that has lots of memory leaks on restarts.

If your daemon processes are not getting fat and the Apache server
child processes are, then you may through incorrect configuration not
even be running Python web application in daemon mode. This is where
WSGIRestrictEmbedded as described in my post is good, as it will
highlight when the configuration is screwed up.

> # prefork MPM
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 256
> MaxRequestsPerChild 500
> ServerLimit 30
>
> instead of the default:
>
> # prefork MPM
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 256
>
> The problem disappeared. The exact values that fix the problem may
> depend on the ram available.

The other difference with above is that I think by setting ServerLimit
to 30, you have effectively overridden MaxClients down to 30 even
though set to 256. You have thus in part limited the exact problems
described in:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

if it so happens you were using embedded mode and not daemon mode.

Graham

[Thadeus Burgess]

The solution: I switched to Flask.

And the problems dissipated completely, without modifying any configuration of the web server.

I would not, and will not use web2py for any application that is mission critical. For personal sites, or quick projects that I know won't receive that much attention, web2py is fine. For quickly prototyping something, web2py excels. For stability, reliability, and scalability, use Flask or Django.

The DAL is great though, nothing quite like it, thats why I am working on a Flask-DAL extension, and I am working to re-write parts of the DAL and strip out the web2py cohesion (such as SQLFORM validators).

Using the DAL inside of flask works fine, and I do not run into these errors. This means that the DAL is not the cause of these errors, but web2py core code. Most likely in dealing with pages that have forms is where these errors arise. Web2py core is messy, and it ignores the wsgi specification for the most part. I am sure that these errors arise from the fact that web2py uses execfile in many places over and over again, which is a discouraged practice among the python community, and you see why now.

--
Thadeus

[Michael Toomim]

THANK YOU ALL SO MUCH for your help!

I just learned a LOT. It looks like resource consumption was the
problem, because things are doing better on the bigger machine and
scaled down code. I've also added the MaxRequestsPerChild directive.

I am soooooo happy to have this web server working, and very pleased
to know what to do when I hit a such a scaling wall again!

And flask looks interesting, but I must say I really really like how
web2py's execfile puts things into global scope from the controllers
and automatically reloads code with each request.

On Jul 20, 5:02 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

> On Jul 21, 8:18 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > Can you comment on memory usage?
>
> > I have see this once: "after a while web serving slows"
> > it appeared to be due to a memory leak somewhere (did not experience
> > it with web2py+Rocket but only in web2py+mod_wsgi+apache).
>
> > I googled it and I found Django was having the same problem on some
> > hosts:
>
> Not sure how you can draw a parallel to that as it is a completely
> different framework and just because another framework, or more
> specifically one persons code, has issues, doesn't imply there is an
> issue with underlying web hosting.
>
> These sorts of problems are isolated cases. If there was an issue with
> memory leakage in the hosting mechanism it would be affecting everyone
> and there have been no such reports of mod_wsgi itself leaking memory.
> That said, ensure you read:
>
>  http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html
>
> This describes how Python itself leaks memory. For mod_wsgi 2.X and
> older, or if you are still loading mod_python into your Apache server,
> then you can be affected by this, but not if using mod_wsgi 3.X.
>
> That post also explains how to completely disable initialisation of
> Python in Apache server child processes, ie., embedded, if you aren't
> using it.
>

> >http://stackoverflow.com/questions/2293333/django-memory-usage-going-......

>  http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa...

[Graham Dumpleton]

On Jul 21, 1:03 pm, Michael Toomim <too...@gmail.com> wrote:
> THANK YOU ALL SO MUCH for your help!
>
> I just learned a LOT.  It looks like resource consumption was the
> problem, because things are doing better on the bigger machine and
> scaled down code.  I've also added the MaxRequestsPerChild directive.

Are you using mod_wsgi embedded mode or daemon mode? That directive
should not be required if you are using daemon mode of mod_wsgi.

It is generally a bad idea to make arbitrary changes without
understanding whether they are necessary. Changing to a larger machine
without understanding why your application is using lots of memory in
the first place is also questionable. All you have done is increased
your head room but potentially not solved the original underlying
problem. You may well just find that it all just blows up again when
you get hit with a larger amount of traffic or a request which
generates a lot of data. What are you going to do then, get an even
bigger machine?

Graham

> ...
>
> read more »

[Michael Toomim]

I'm using daemon mode... I didn't realize that the directive won't
matter in daemon mode.

Yes, I think I probably will run into the problem again when I get
more usage. However, I'm still not convinced it's a memory problem,
because I had 30mb free on my 740mb machine when I was having the
problem, with 0 swap usage.

I don't know what I'll do if I this happens again. My code just does
simple database lookups and updates, it doesn't create circular
references nor store anything in global variables, so if there's a
memory leak I worry it's somewhere further up the stack. I don't know
any ways to investigate memory consumption to see where it's being
used.

On Jul 20, 8:23 pm, Graham Dumpleton <graham.dumple...@gmail.com>

> > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi...
>
> read more »

[Graham Dumpleton]

On Jul 21, 1:41 pm, Michael Toomim <too...@gmail.com> wrote:
> I'm using daemon mode... I didn't realize that the directive won't
> matter in daemon mode.
>
> Yes, I think I probably will run into the problem again when I get
> more usage.  However, I'm still not convinced it's a memory problem,
> because I had 30mb free on my 740mb machine when I was having the
> problem, with 0 swap usage.

Well, as I explained before, it perhaps is a resource leakage such as
file descriptors. You can exhaust kernel file descriptors and still
have lots of memory available.

I have seen various cases before for different peoples applications on
different frameworks where file objects weren't being explicitly
closed, or database connection pools not being managed properly, such
that the number of open file descriptors went up and up and eventually
they ran out. This can cause all sorts of weird errors to manifest
when it occurs, including it impacting other applications if is
exhausted system wide. For example, in a shell, not being able to
execute commands to even debug the problem.

I would suggest you become familiar with some of the basic monitoring
commands such as 'lsof' or 'ofiles', depending on what your system
provides. You can then use these to monitor file descriptor usage by
your processes.

Also be aware that such problems may only arise when multithreading
kicks in and concurrent requests run. In other words, due to code
which isn't thread safe. If you don't get the concurrency, you may
well see your application run quite happily.

Thus, one suggestion is to not use multiple threads for daemon mode
processes and instead use something like 'processes=5 threads=1'. This
will avoid the potential of it being caused by multithreading issues
at least.

Graham

> ...
>
> read more »

[Michael Toomim]

Ah, preventing multithreading is a good idea to try too.

It wasn't a file descriptor problem either, I had

Files used: 1376 out of 75556

On Jul 20, 9:14 pm, Graham Dumpleton <graham.dumple...@gmail.com>

> > > > > > > > > connection. You would only see that message if HTTP...
>
> read more »

[mdipierro]

Hi Graham,

I did not say this is a problem with apache or mod_wsgi. I said I
found people using other frameworks having the problem and the
solution they proposed worked for me.

Massimo

On Jul 20, 7:02 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

> On Jul 21, 8:18 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > Can you comment on memory usage?
>
> > I have see this once: "after a while web serving slows"
> > it appeared to be due to a memory leak somewhere (did not experience
> > it with web2py+Rocket but only in web2py+mod_wsgi+apache).
>
> > I googled it and I found Django was having the same problem on some
> > hosts:
>
> Not sure how you can draw a parallel to that as it is a completely
> different framework and just because another framework, or more
> specifically one persons code, has issues, doesn't imply there is an
> issue with underlying web hosting.
>
> These sorts of problems are isolated cases. If there was an issue with
> memory leakage in the hosting mechanism it would be affecting everyone
> and there have been no such reports of mod_wsgi itself leaking memory.
> That said, ensure you read:
>
>  http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html
>
> This describes how Python itself leaks memory. For mod_wsgi 2.X and
> older, or if you are still loading mod_python into your Apache server,
> then you can be affected by this, but not if using mod_wsgi 3.X.
>
> That post also explains how to completely disable initialisation of
> Python in Apache server child processes, ie., embedded, if you aren't
> using it.
>

> >http://stackoverflow.com/questions/2293333/django-memory-usage-going-......

>  http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa...

[mdipierro]

Flask uses less resources

> >http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4...

[mdipierro]

Glad it worked for you.
I also noticed big improvement when going from 256KB to 384KB on my
server.

> ...
>
> read more »

[mdipierro]

I still want to understand why you are having this problem. I see the
following possibility:

1) There is a threshold in requests/seconds that depends on memory
available and it is different for different frameworks. web2py does
more than Flask (which is a microframework by definitions) and this
threshold may be lower.
If this is the case the problem should go away with more ram.

2) There is a bug in web2py or one of its modules that causes a memory
leak (did you experience a memory leak?) or a locking issues (for
example a process crashes before a locked resource is released).

I remember you mentioning have this problem exclusively with actions
using SQLFORM under heavy load. Is that correct? This could help us
narrow down the problem.

Massimo

On Jul 20, 7:17 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:

> >http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4...

[mdipierro]

I know you got it. ;-)

But for everybody else, I responded to this in a different thread:

http://groups.google.com/group/web2py/msg/5458046f6c985045

Massimo


On Jul 20, 7:17 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:

> >http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4...

[Michael Toomim]

Great! I want to understand it too!

Your breakdown helps me think about how to look into this. I will do
some more analysis, but for now:
- I'm not using forms
- I had migrate=True and the app was not compiled earlier. now it's
compiled and migrate=False and I have a bigger machine, things are
running ok, but I changed quite a few variables. Would migrate=True
create locking issues? The problems seemed similar to locking issues.
- My machine's ram slowly filled up from ~700mb/1500mb to 1400mb/
1500mb over 5 hours. But I haven't looked to see which process grew
yet, I will investigate.

[Thadeus Burgess]

For the record, I never really had any issues with web2py filling up my RAM, mostly just crashing while processing a certain number of requests.

Even after the ab tests, when looking at RAM usage, it would only spike to maybe 150MB for all apache+mod_wsgi+web2py processes, and then go back down to the normal 36MB load after the ab test finished.

What else do you have installed on the server? Is it really web2py that is causing the memory usage, or a rouge daemon running in the background? I have not seen any of my web2py processes ever make it over 150MB usage unless I did some massive caching.

--
Thadeus

[mdipierro]

I have seen this once. I fixed is by limiting the number of requests
served by each apache process.

migrate=True will lock the app for the time it takes to check whether
tables have migrated. For slow traffic it is not an issue but for high
traffic it can be.

> ...
>
> read more »

[Michael Toomim]

Great! I want to understand it too!

Your breakdown helps me think about how to look into this. I will do
some more analysis, but for now:
- I'm not using forms
- I had migrate=True and the app was not compiled earlier. now it's
compiled and migrate=False and I have a bigger machine, things are
running ok, but I changed quite a few variables. Would migrate=True
create locking issues? The problems seemed similar to locking issues.
- My machine's ram slowly filled up from ~700mb/1500mb to 1400mb/
1500mb over 5 hours. But I haven't looked to see which process grew
yet, I will investigate.


On Jul 21, 1:13 am, mdipierro <mdipie...@cs.depaul.edu> wrote: